Update Accelerator revisited

Help on building IPFire & Feature Requests
BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 15th, 2015, 12:51 pm

New version 1.1-1 available.

There was an error with vendors, when used the first time ( what is normal for a clean repository :-[ ).
Due the changed calculation of disk usage and free space, these directories must exist to start the download. ( Thx to N0man for giving me the hint ).

Just two tips for usage:
  • Definition of new vendors
    • vendorid must adhere to the regular expression '[A-Z][A-Za-z0-9_]+'
    • after definition you must run '/var/ipfire/updatexlrator/bin/sources.pl' to compile the defs and restart squid by '/etc/init.d/squid restart'
  • Transfer of repositories from existing systems
    • copy the files to /var/updatecache
    • if the originating system used the standard version, convert the metadata to DB format by '/var/ipfire/updatexlrator/bin/conv2db'
    • A transfer from the new version to system with the standard version can be done by copying the files after converting the metadata to old format by '/var/ipfire/updatexlrator/bin/conv2files
Image
Unitymedia Cable Internet ( 32MBit )

RedneckMother
Posts: 94
Joined: June 21st, 2014, 1:34 am
Location: USA

Re: Update Accelerator revisited

Post by RedneckMother » May 15th, 2015, 5:19 pm

Results for 1.1-1 install:

Code: Select all

[root@harpo ~]# md5sum updx_new.1.1-1.tgz 
3eb822001da035c15e32d5f89d9985cb  updx_new.1.1-1.tgz
[root@harpo ~]# tar -vpxf updx_new.1.1-1.tgz
updx_new.1.1-1/
updx_new.1.1-1/updx_new
updx_new.1.1-1/files.tgz
[root@harpo ~]# cd updx_new.1.1-1
[root@harpo updx_new.1.1-1]# ls
files.tgz  updx_new
[root@harpo updx_new.1.1-1]# ./updx_new uninstall
writing meta data from database to standard files
restoring standard update accelerator
removing files generated by new version
(without database)
new updatexlrator is uninstalled.
Thank you for testing.
[root@harpo updx_new.1.1-1]# ./updx_new install
new updatexlrator is installed yet.
Uninstall first, please.
[root@harpo updx_new.1.1-1]# rm /usr/sbin/updxlrator.std
rm: remove regular file ‘/usr/sbin/updxlrator.std’? y
[root@harpo updx_new.1.1-1]# ./updx_new install
copying files

converting the metadata to new format

generate cache of sources
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator/bin/sources.pl line 182.

updatexlrator is installed.
You can disable update sources by moving definition files from
/var/ipfire/updatexlrator/sources to /var/ipfire/updatexlrator/sources_disabled.
(Don't forget to run /var/ipfire/updatexlrator/bin/sources.pl again!)
[root@harpo updx_new.1.1-1]# /etc/init.d/squid restart
Stopping Squid Proxy Server...                                         [  OK  ]
Creating Squid swap directories...                                     [  OK  ]
Starting Squid Proxy Server...                                         [  OK  ]
[root@harpo updx_new.1.1-1]# /var/ipfire/updatexlrator/bin/sources.pl
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator/bin/sources.pl line 182.



I also noticed an entry int the system log:

Code: Select all

11:54:02   ipfire:    UPDXL: no vendors defined
-- Paul

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 15th, 2015, 5:34 pm

To solve the main problem, edit the file updx_new ( the installer ) please.
Line 32ff

Code: Select all

            if ($cpdir) { #copy directory contents
#                print "system(\"cp -pr files/$entry* /$entry*\")\n";
                system("cp -pr files/$entry* /$entryKx");
            }
should read

Code: Select all

            if ($cpdir) { #copy directory contents
#                print "system(\"cp -pr files/$entry* /$entry*\")\n";
                system("cp -pr files/$entry* /$entry");
            }


Don't know how this typo resisted all editing. :(
The installer presumes a definitions directory, but it is not there because of the false cp command.

Thanks for the hint.
Image
Unitymedia Cable Internet ( 32MBit )

RedneckMother
Posts: 94
Joined: June 21st, 2014, 1:34 am
Location: USA

Re: Update Accelerator revisited

Post by RedneckMother » May 15th, 2015, 6:01 pm

Thanks for the hint.

No, no... thanks for the quick fix, and your diligence!
-- Paul

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 15th, 2015, 8:13 pm

RedneckMother wrote:No, no... thanks for the quick fix, and your diligence!

That wasn't really a quick fix. I knew this typo very well and had corrected it several times since I found it. Perhaps it is a problem of release time. Many corrections were made at times, when "my main tester" N0man was active. He lives also in the US and that means his 'activity time' is after midnight here in Germany. ;)

BTW: Does it work now?
Image
Unitymedia Cable Internet ( 32MBit )

RedneckMother
Posts: 94
Joined: June 21st, 2014, 1:34 am
Location: USA

Re: Update Accelerator revisited

Post by RedneckMother » May 15th, 2015, 8:30 pm

BTW: Does it work now?


The uninstall / install sequence worked after the edit.

I'm having an issue with linux updates, though. I haven't had any luck tracing what's going on, but during a fedora update (after the 1.1-1 fix), I could see various rpm & drpm downloads in progress on the updx web interface, but they never show up in the stats or maintenance views.

I'll try to troubleshoot it as best I can.

I'm in the US, too (sorry 'bout that). Get some sleep!

Thanks again!
-- Paul

RedneckMother
Posts: 94
Joined: June 21st, 2014, 1:34 am
Location: USA

Re: Update Accelerator revisited

Post by RedneckMother » May 18th, 2015, 4:44 pm

A couple of issues:

I downloaded updx_new.1.1-1.tgz with the md5sum of 7487adec4e3e470f1af27cbe5361a7de and extracted it.

I changed to the created directory, and ran "./updx_new uninstall", which resulted in the following messages:

Code: Select all

restoring standard update accelerator
removing files generated by new version
(without database)
new updatexlrator is uninstalled.
Thank you for testing.    


I ran "./updx_new install", which generated:

Code: Select all

new updatexlrator is installed yet.
Uninstall first, please.


I then rm'd "/usr/sbin/updxlrator.std" and reran "./updx_new install", which generated:

Code: Select all

copying files

converting the metadata to new format

generate cache of sources

updatexlrator is installed.
You can disable update sources by moving definition files from
/var/ipfire/updatexlrator/sources to /var/ipfire/updatexlrator/sources_disabled.
(Don't forget to run /var/ipfire/updatexlrator/bin/sources.pl again!)


After a "/etc/init.d/squid restart", I attempted a few client software updates. The stats and maintenance views in the WUI do not reflect the new downloads.

What must I do to assist in troubleshooting?

Thank you,
Paul

Edit:
Also, saw "UPDXL: no vendors defined" in system logs.
In the maintenance WUI, existing entries won't delete... possible file/directory owner/permission problems?

p.

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 18th, 2015, 9:40 pm

I just checked the files of version 1.1-1 with the given MD5sum.
I can't see a reason for the your issues.
The error 'no sources defined' should printed to stdout also, if it is logged to the system log.
Your output of the install doesn't show this error.

Also the error of 'uninstall' not deleting the .std files was there in a eaier version, but not in the latest.

To check for owner/rights problems do a

Code: Select all

ls -lR files
from the installer directory.
All files a copied with 'cp -p'. Therefore the attributes of the files in the archive should be propagated to the destination directories.

Thx in advance
Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

RedneckMother
Posts: 94
Joined: June 21st, 2014, 1:34 am
Location: USA

Re: Update Accelerator revisited

Post by RedneckMother » May 20th, 2015, 5:03 pm

Hello Bernhard.

I discovered that various files for update accelerator were corrupted, most likely from earlier attempts to update from N0man's thread and previous iterations of this thread. I recovered the files from an install ISO, and will try 1.1-1 again.

Sorry for the confusion.

Paul

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 20th, 2015, 8:45 pm

Paul,

this may be truely a reason for your issues. I know this situation from development, but in most cases I know what I've changed. ;)

It is true, the installer grew up to a more or less stable version, that should allow a clean switch between the old and the new version. I am sorry for bothering my "earlybirds". :(

I'll try to check the installer once more deeply for the release of the next version.

Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

vkykam
Posts: 9
Joined: April 17th, 2015, 1:16 am

Re: Update Accelerator revisited

Post by vkykam » May 28th, 2015, 2:43 am

Hi Bernhard,

Thanks for the great work.

Some feedback, and I don't know whether it's a bug or of my own doing... I've tested an earlier version back in April, and now 1.1-1 as well. If you look at the attached .jpg, you'll see that it appears the icon files are either missing for my install (or maybe I uploaded with the wrong permissions?), as this happened I believe after testing 1.0-1. There's also a snippet of what is a very long list (probably in the 1000's or 10000's) of Microsoft .cab updates that are miniature, 10kB or less, which only started happening after I tested 1.0-1. I may also have messed with the configuration files prior to testing 1.0-1, so I don't know whether it's of my own doing, or whether this is expected behavior, as this is my only IPFire install.

The other thing of note, and I believe this is also true of the old Update Accelerator, is that max external download rate is per update and not total aggregate of Update Accelerator, based on my observations. If my observations are correct, then I think the GUI should be changed to indicate that to be per update, as it's currently ambiguous. In an ideal world, it should be total aggregate, but I don't know if that's technically feasible. We are using Update Accelerator in a BYOD environment in a very rural part of the world, with connection speeds of 0.25Mbps shared among 20 or 30 people at times. The limit becomes useless, at least to us, when there are large quantity of update files to be downloaded, and the number of files multiplied by the speed limit will often saturate the entire bandwidth available, but we can't set the max update speed so low (like 5kbps) that the updates will take a week or more to download.

Capture.JPG

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 28th, 2015, 10:06 am

Thanks for testing.

The first issue is generated by wrong permissions for the image files, I suppose.
The images are located in /srv/web/ipfire/html/images/updbooster. In my system they all have root/root 0644.

The second isssue is a true bug. Download is limited for each file. You are right, this doesn't make sense for multiple concurrent downloads. To solve this problem, we need a UAdownload demon. I'll try to implement this in one of the next releases.
Thanks for the hint.

Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

vkykam
Posts: 9
Joined: April 17th, 2015, 1:16 am

Re: Update Accelerator revisited

Post by vkykam » May 29th, 2015, 2:40 am

I checked that already, and confirmed that it's root/root 0644, so I don't know why the image icons aren't being served.


BeBiMa wrote:Thanks for testing.
The first issue is generated by wrong permissions for the image files, I suppose.
The images are located in /srv/web/ipfire/html/images/updbooster. In my system they all have root/root 0644.

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 29th, 2015, 2:27 pm

I fear, there were and are some problems with rights and ownership.
I'm checking this at the moment after a fresh install of new UA because of IPFire upgrade, which restored the standard UA. :(

I hope, I can release a new version tomorrow.
Image
Unitymedia Cable Internet ( 32MBit )

RDaemon
Posts: 1
Joined: September 23rd, 2012, 2:31 pm

Re: Update Accelerator revisited

Post by RDaemon » June 29th, 2015, 9:56 am

The limit becomes useless, at least to us, when there are large quantity of update files to be downloaded, and the number of files multiplied by the speed limit will often saturate the entire bandwidth available, but we can't set the max update speed so low (like 5kbps) that the updates will take a week or more to download.


I faced the same situation and use "quick and dirty" fix for now:

1) find following code in "download" script:

Code: Select all

    &UPDXLT::writehash("$UPDXLT::repository/download/$vendorid/$updatefile.info", \%dlinfo);

    my $cmd = "$UPDXLT::wget $login $dlrate --user-agent=\"$UPDXLT::useragent\" -q -P $UPDXLT::repository/download/$vendorid $wgetContinueFlag $sourceurl";
   
    $ENV{'http_proxy'} = $proxysettings{'UPSTREAM_PROXY'};
    $_ = system("$cmd");
    $ENV{'http_proxy'} = '';

    if ($_ == 0) {
        &writelog("Download finished with result code: OK");


2) place this right before string " $_ = system("$cmd");"

Code: Select all

my $limit_num_wgets = "ps -ef |grep -c wget |grep -v grep";
while(system("$limit_num_wgets") > 4){
<------>&writelog("Concurent: $limit_num_wgets, wait 360...");
sleep(360);
}


Resulting code must looks like this:

Code: Select all

    &UPDXLT::writehash("$UPDXLT::repository/download/$vendorid/$updatefile.info", \%dlinfo);

    my $cmd = "$UPDXLT::wget $login $dlrate --user-agent=\"$UPDXLT::useragent\" -q -P $UPDXLT::repository/download/$vendorid $wgetContinueFlag $sourceurl";
   
    $ENV{'http_proxy'} = $proxysettings{'UPSTREAM_PROXY'};
my $limit_num_wgets = "ps -ef |grep -c wget |grep -v grep";
while(system("$limit_num_wgets") > 3){
<------>&writelog("Concurent: $limit_num_wgets, wait 360...");
sleep(360);
}
    $_ = system("$cmd");
    $ENV{'http_proxy'} = '';

    if ($_ == 0) {
        &writelog("Download finished with result code: OK");
       


This limits number of concurrent WGET tasks to 4, total speed limit is "max update speed"*4, another download tasks remains inactive, checking with 10-minute interval when total number of WGET instances is less than 4 and waits, if more.

Post Reply