Update Accelerator revisited

Help on building IPFire & Feature Requests
BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Update Accelerator revisited

Post by BeBiMa » March 27th, 2015, 11:39 pm

As announced some days before, I've made some enhancements to Updatexlrator.

Main goal
Updatexlrator has grown more and more. There were added new update sources ( software vendors ), the URLs of existing sources were updated too.
This produced the "if mountains of Axel", not easy to maintain and to verify. Each addition increases complexity.
Another aspect of complexity is the abstraction of a set of update URLs to a regular expression.

Therefore some time ago the idea was born to simplify this process:
  • Addition are not made in the source, but in a description file for these new sources
  • The user must not compute a regular expression for the URLs. Only the URLs must be given, the regex is computed by Ipfire
  • Not all sources are used by all users. Thus a kind of selection should be possible.
  • The decision whether the updatexlrator is responsible for handling the URL should be quick. Some sort of history of update requests could help to speed up the process.


Realization ( till now)
The update sources ( vendors ) are described by files in directory /var/ipfire/updatexlrator/sources.
Filename is the vendor, e.g. Microsoft.
Each file contains positive matches, negative matches and the source type.

How to find them?
Updatexlrator ( file /usr/sbin/updatexlrator) contains if statements with the schematic ( =~ means "matches", !~ means "doesn't match" )

Code: Select all

if ( $url =~ m@<exp 1>@i | 
     ...
     $url =~ m@<exp n>@i
&&($url !~ m@<exp n+1>@i  |
       ...
       $url !~ m@<exp m>@i )) {
           $xurl = check(...,$url,...,<vendor>, <type>)
}

From this code sequence <exp 1> ... <exp n> are the positive matches and ( if existent) <exp n+1> ... <exp m> are the negative matches, <type> is either $unique or $mirror.

Thus the descriptive file contains a couple of lines adhereing to the syntax
<command>,<expression>
with the meaning
  • <command>:
    • p ... regexp that should match as whole (^<expression>$)
    • P ... regexp that should match ( without ^$ )
    • n ... regexp that should not match as whole (^<expression>$)
    • N ... regexp that should not match ( without ^$ )
    • a ... URL that should match (accept )
    • d ... URL that should not match ( deny )
    • t ... type of source ( unique, mirror )
  • <expression>:
    • repexp for p,P,n,N commands ( our positive and negative matches from above)
    • URL for a,d commands
    • type for t command ( u=unique, m=mirror )
The p and n commands take the <exp i> from our code segment ( without ^ and $ brackets for ease of use! ). These are the "normal" commands ( till now ).
Commands a and d allow to add plain URLs ( no knowledge of regexp necessary, I hope ).

This database of defined vendors is compiled to data structure holding one regexp for positive match and negative match for each vendor. The condition can be reduced by rules of logic to

Code: Select all

if ( ($url =~ <p_match>) & ($url !~ <n_match> ) ) {
    check(.......);
}
This is iterated over all know vendors.

The compilation is done by call of

Code: Select all

/var/ipfire/updatexlrator/bin/sources.pl
(This is inspired by the processing for language files in Ipfire ).
This script generates a list of known vendors also ( file /var/ipfire/updatexlrator/sources-used ).
If you add new vendor descriptions, you should add an image file in /srv/web/ipfire/html/images/updbooster also. The web interface will honor it.
For names I tried to follow the rule: vendor names begin with upper case, the image file names contain the vendor in lowercase. This is the state found in the running system.

For acceleration of the search process the list of known vendors is resorted by the updatexlrator process: If an URL yields an update request, the corresponding vendor is put at top of the list. Thus seldomly or never used vendors slowly drop to the bottom.
Further one can disable vendors, which should not be cached or aren't used at all by moving the definition file from /var/ipfire/updatexlrator/sources to /var/ipfire/updatexlrator/sources_disabled and rerunning the compilation process.
The file /var/ipfire/updatexlrator/sources_used represents a somewhat recent state of the internal list, therefore it is sorted according to the strategy from above, but not alphabetical. It is located in the tmpfs /var/log/rrd and saved/restored at restart.
While there are several updxlrator processes ( defined in proxy configuration ), each process has its own local copy. Further each process sends each vendorid found to a server process, which sorts the system wide list sources_used. A synchronisation of local lists and system wide list is done at restart of updatexlrator only ( yet ).

Installation for test
Download the attached file, transfer it to your Ipfire system.
Unpack and install with

Code: Select all

tar xzf updx_new.<version>tgz
cd updx_new
./updx_new install

restart updatexlrator


Uhhh, a bit lengthy this explanation, but the process for realization wasn't that short. ;)

Please test and ask.

-Bernhard

P.S. There are some other things to optimize for this module. I'm working on it. This should be a first version only.

P.P.S. Thx to N0man, who initiated the "revisit" to the updatexlrator "construction site" and added a bunch of new vendors. I hope I've integrated all without errors. ( viewtopic.php?f=50&t=12868 )

EDIT: correct installer.
v 0.9-2 (28-03-15 19:00) installer creates perl sub dir, deleted ESET

EDIT: new version
v 1.0-0 (16-04-15 18:15) disable by moving to directory sources_disabled, some new vendors
v 1.0-1 (19-04-15 20:11) typos corrected. Corrected WUI
v 1.0-2 (20-04-15 00:25) Processing of HTTP error responses in download, TODO: check in updxlrator
v 1.0-3 (21-04-15 13:45) Corrected addition to an empty repository, illegal end of line chars in source definition
v 1.0-3a (22-04-15 02:15) fixed rights / ownership of files
v 1.1-0 (13-05-15 19:03) New database for meta data, new structure of installer ( see post http://forum.ipfire.org/viewtopic.php?f ... 935#p84935 )
v 1.1.-0a fixed typos :-[
v 1.1-1 (15-05-15 23:56) Directories in repository for vendors are generated if not existent yet.
Attachments
updx_new.1.1-1.tgz
MD5: 7487adec4e3e470f1af27cbe5361a7de
(90.59 KiB) Downloaded 917 times
updx_new.1.1-0a.tgz
MD5: 5b5976c15ec088505c51685bc5f78f9e
(90.47 KiB) Downloaded 218 times
updx_new.1.1-0.tgz
MD5: 86532cfa16dc9139e57f2a500c9855d8
(90.46 KiB) Downloaded 210 times
updx_new.1.0-3a.tgz
(82.27 KiB) Downloaded 256 times
updx_new.1.0-3.tgz
(82.19 KiB) Downloaded 267 times
Last edited by BeBiMa on May 15th, 2015, 10:00 pm, edited 20 times in total.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » March 28th, 2015, 3:36 am

I'm having issues with the installer script.

It keeps giving several errors after line 46.

BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » March 28th, 2015, 9:09 am

Sorry, didn't check this piece of software :-[

I've added the corrected version to the top post.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » March 28th, 2015, 4:32 pm

Thank you, most of the installer issues seem fixed.
There is only one thing.

You need to make a directory called

/usr/lib/perl5/5.12.3/Regexp/

before you try to copy Assemble.pm to it.

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » March 28th, 2015, 5:00 pm

I've tested update accelerator after I made the new directory, It works well.

I also noticed something. In the maintenance page, there is still ESET as a supported vendor.

I removed it in the last revision because ESET uses authentication which the update accelerator couldn't use.

BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » March 28th, 2015, 6:06 pm

N0man wrote:Thank you, most of the installer issues seem fixed.
There is only one thing.

You need to make a directory called

/usr/lib/perl5/5.12.3/Regexp/

before you try to copy Assemble.pm to it.

I also noticed something. In the maintenance page, there is still ESET as a supported vendor.

I removed it in the last revision because ESET uses authentication which the update accelerator couldn't use


Thx. Done.

BTW: The file /var/ipfire/updatexlrator is not really updated by the compilation process. New vendors are added, but deleted vendors are not deleted ( yet ). Just remove the file before recompilation or delete the vendor with an editor.
Image
Unitymedia Cable Internet ( 32MBit )

BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » April 11th, 2015, 2:22 pm

Some state about the work.

I'll try to eliminate the ugly issues with weird file names. This is a bit more complex than expected. All files belonging to update accelerator must be inspected.

Having problems to update my Avira Free in the last days, I detected a problem with source definitions. Many sources/vendors are marked as "mirror" at the moment ( in standard accelerator and extensions from N0man ). In most cases this doesn't really matter, but for Avira it did!

The classifications used are:
  • unique ... file is identified by the URL as a whole ( a.b/v1/prog.exe, a.b/v2/prog.exe)
  • mirror ... file is identified by its name (a.b/upd/prog_v1.exe, a.b/upd/prog_v2.exe)

I can't really check, whether all classifications are right. Therefore it would be nice to get reports from users of the various update sources.
( Avira is corrected ;) )

- Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » April 11th, 2015, 3:51 pm

Hi,
I have been trying adding two sources to update accelerator.

The software are Libreoffice and Openoffice.

The issue is that Libreoffice seems to use metadata to redirect to a mirror.
Openoffice redirects to a sourceforge page which uses mirrors.

What would the URL source look like?

BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » April 11th, 2015, 4:34 pm

What kind of updates do want to cache?

Libreoffice usually does an update as a whole. The user can fetch it from the Libreoffice sites directly ( possibly redirected ) or from one of the mirrors. I personally prefer to get it as a torrent ( an unpredictable bunch of "mirrors" ).

EDIT: An inspection of the download pages of Libreoffive shows URLs adhereing to the RegExp
^http:\/\/.*\/libreoffice\/stable\/.*\/LibreOffice_.*\.(msi|tar\.gz)$
The file name seems to be unambiguous ( e.g. LibreOffice_4.4.2_Win_x86.msi ), thus we can classify as "mirror".
The endings must be completed for all supported OSs.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » April 11th, 2015, 8:55 pm

Thank you.

I corrected the unique vs mirror entries.
I made Avira support Professional Updates - I think it has different definitions between the free and paid versions.
I added Libreoffice support.

bloater99
Posts: 476
Joined: October 13th, 2014, 3:47 pm

Re: Update Accelerator revisited

Post by bloater99 » April 13th, 2015, 2:25 pm

LibreOffice does not have an automated update system as far as I know. You either have to manually apply the update or create a custom .msi and do it through Group Policy. I suppose, though, you can still cache the download so others who are also manually downloading can get it from the cache.

bloater99
Posts: 476
Joined: October 13th, 2014, 3:47 pm

Re: Update Accelerator revisited

Post by bloater99 » April 13th, 2015, 2:28 pm

I have not tested this as my only ipfire install is in a production setting. But I do use the default update accelerator and wondered if in your version have you separated Firefox and Thunderbird? Currently it just says Mozilla with no indication of which Mozilla package. I don't know if it's possible, but it would nice to some day be able to see separate entries for Firefox and Thunderbird :)

BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » April 13th, 2015, 5:10 pm

If you can separate the download URL of Firefox and Thunderbird, no problem.
My concept allows you define "packages" ( now named "vendors" or "sources" ) with their dedicated set of URLs.
Therefore if you want to divide the Mozilla repository into two, it is your choice. But I think we'll support the big bunch "Mozilla" only.

About the LibreOffice you are right. But Update Accelerator is not a cache for automated download only, but a cache for multiple downloads in the local network.
BTW: If it would cache "updates" only, there weren't the problem with blanks in file name. The "prototype example" is the setup file for Firefox. ;)
Image
Unitymedia Cable Internet ( 32MBit )

BeBiMa
Posts: 2694
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » April 16th, 2015, 4:29 pm

Finally, the first version which implements most ideas.
Main change is that vendors are disabled/enabled by moving between directories sources and sources_disabled.
Directory sources_disabled is only a storage for unused vendors. All vendors in sources are active!

First trial to handle a repository distributed over several drives/partitions. Calculations for free/used disk space should work. Output in web interface isn't really nice ( yet ).

EDIT: "Beautified" updatexlrator.cgi and description in wiki coming soon.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » April 17th, 2015, 10:03 pm

I tested the new version on a clean version of IPFire.

Here is what I've found so far:

The install script has a typo:
(Don't forget to run /var/ipfire/updatexlrator/bin/soources.pl again!)

The database seems to use the old, original sources rather than the new ones

There seems to be an error with permissions. Many files seem to have their permissions changed.

when I try to start the script I get:

Code: Select all

/usr/sbin/updxlrator
Name "Updxsrc::VendorPipe" used only once: possible typo at /usr/sbin/updxlrator line 100.
Name "Updxsrc::match_all" used only once: possible typo at /usr/sbin/updxlrator line 125.
Name "Updxsrc::SrcUsed" used only once: possible typo at /usr/sbin/updxlrator line 81.
Name "Updxsrc::SrvVendors" used only once: possible typo at /usr/sbin/updxlrator line 97.
Can't locate /root/work/usrc in @INC (@INC contains: /usr/lib/perl5/site_perl/5.12.3/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.12.3 /usr/lib/perl5/5.12.3/i586-linux-thread-multi /usr/lib/perl5/5.12.3 .) at /usr/sbin/updxlrator line 34.
Last edited by N0man on April 20th, 2015, 12:17 am, edited 1 time in total.

Post Reply