Update Accelerator revisited

Help on building IPFire & Feature Requests
BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 5th, 2015, 10:11 am

ckleuser wrote:How to clear the update accelerator cache without having to delete each file or waiting at least a week ??? Any command line?

At the moment the only way is to delete the contents of the directory /var/updatecache/<vendorid>.
Maybe there are any ideas to solve this in more "elegant" way.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » May 5th, 2015, 10:58 am

zcworld wrote:like an que system so it
like for updates


I think he wants a queue system where each update is downloaded one-at-a-time in-order.

Currently the update accelerator downloads all updates at once. This can be an issue when 30 updates are trying to download at once.

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 5th, 2015, 11:38 am

Thanks for the explanation. ;)

Yes, you are right. I had this idea yet.
At the moment each UpdxAcc process forks a download process for a new file. And WindowsUpdate may be very "aggresive" in requesting files simultaniously. Seems like the strategy "Download as much as we can get, then look what we need".
A queue for download would help to reduce e.g. 60 downloads ( 30 to repository, 30 to client(s) ) to 31 ( 1 to repository, 30 to clients).

I'll think of an implementation for this feature.

- Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

New version 1.1

Post by BeBiMa » May 12th, 2015, 10:07 pm

After some struggle between intented and written source code( ;) ), I happily release version 1.1 of the new Update Accelerator for test.

What's new:
  • the sorting of vendorids by usage is done at two places now. Each Updxlr process maintains it's own private copy. A server task maintains the overall list, getting information about chosen vendorids from the Updxlr processes through a pipe. At (re)start each process reads this global list.
  • The meta data for each file in the update repository is now held in a data base. Thus the sampling of information for statistics and maintenance in the WUI should be faster (and hopefully more memory efficient).
  • The installer was rewritten to use a more system conform structure. Install generates the new data base. Uninstall writes the information in the data base back to the file base meta data.

Further the nameing of vendorids is more strict:
Vendorids in the definition directories must start with an uppercase letter, allowed characters are letters, digits and '_'.
Vendorids in the repository are all lowercase. Thus defined vendorids must be unambiquous after converting to lower case.

Next steps will be:
  • More strict definition of repository contents. Only directories which are the lower case equivalent of a defined vendorid are part of the repository and can be managed by the WUI. Vendorids in directory sources_disabled are defined, but not used.
    This should speed up the repository file walk.
  • Tidy up the sources to eliminate obsolete and double functions.
  • Speed up the maintance page of the WUI
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » May 13th, 2015, 2:00 am

I tried the new update.

The update accelerator doesn't seem to start.

When I run:
/usr/sbin/updxlrator

It gives:
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator/bin/sources.pl line 182.

cardins2u
Posts: 9
Joined: April 25th, 2015, 6:25 pm

Re: Update Accelerator revisited

Post by cardins2u » May 13th, 2015, 6:11 am

same I got the same errors

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 13th, 2015, 10:45 am

Are there any messages, when you start /var/ipfire/updatexlrator/bin?
What is the contents of /var/ipfire/updatexlrator/cache-sources.pl?

The files in the installer archive are identical with my running version. And I've got no issues so far.
Maybe I've got some file permissions wrong again. I'll check this next.
Image
Unitymedia Cable Internet ( 32MBit )

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 13th, 2015, 4:07 pm

:-[ I think I've found the bug. :-[
The source definitions were not installed, due to a typo in the installer.
New file is in the top post.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Update Accelerator revisited

Post by N0man » May 14th, 2015, 1:43 am

I tried the new version 1-1-0

This is the error I get when I install it on a fresh IPFire Virtual Machine:

Code: Select all

./updx_new install
copying files

converting the metadata to new format

generate cache of sources
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator/bin/sources.pl line 182.

updatexlrator is installed.
You can disable update sources by moving definition files from
/var/ipfire/updatexlrator/sources to /var/ipfire/updatexlrator/sources_disabled.
(Don't forget to run /var/ipfire/updatexlrator/bin/sources.pl again!)

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 14th, 2015, 12:32 pm

@N0man: You've got a PM.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
binky
Posts: 6
Joined: May 14th, 2015, 1:18 pm

Re: Update Accelerator revisited

Post by binky » May 14th, 2015, 6:57 pm

I'm seeing exactly the same as N0man:

Code: Select all

[root@ipfire updx_new.1.1-0]# ./updx_new install
copying files

converting the metadata to new format

generate cache of sources
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator/bin/sources.pl line 182.

updatexlrator is installed.
You can disable update sources by moving definition files from
/var/ipfire/updatexlrator/sources to /var/ipfire/updatexlrator/sources_disabled.

(Don't forget to run /var/ipfire/updatexlrator/bin/sources.pl again!)
[root@ipfire updx_new.1.1-0]# /var/ipfire/updatexlrator/bin/sources.pl
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator
/bin/sources.pl line 182.
[root@ipfire updx_new.1.1-0]#


I only found ipfire today and I've got it all working as I'd hoped for my home network apart from the Update Accelerator not working, so I thought I'd try your 'revisited' version as you're asking for testers in a different thread and I'm only using it at home so absolutely no issue if I kill it 8)

Any suggestions?

Thanks,
Binky


N0man wrote:I tried the new version 1-1-0

This is the error I get when I install it on a fresh IPFire Virtual Machine:

Code: Select all

./updx_new install
copying files

converting the metadata to new format

generate cache of sources
Can't call method "as_string" on an undefined value at /var/ipfire/updatexlrator/bin/sources.pl line 182.

updatexlrator is installed.
You can disable update sources by moving definition files from
/var/ipfire/updatexlrator/sources to /var/ipfire/updatexlrator/sources_disabled.
(Don't forget to run /var/ipfire/updatexlrator/bin/sources.pl again!)
Image

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 14th, 2015, 8:47 pm

binky wrote:I only found ipfire today and I've got it all working as I'd hoped for my home network apart from the Update Accelerator not working, so I thought I'd try your 'revisited' version as you're asking for testers in a different thread and I'm only using it at home so absolutely no issue if I kill it 8)

That's interesting.Did you have a hint, why the normal Update Accelerator does work. My version is just a enhancement of the existing framework.
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
binky
Posts: 6
Joined: May 14th, 2015, 1:18 pm

Re: Update Accelerator revisited

Post by binky » May 14th, 2015, 9:17 pm

BeBiMa wrote:Did you have a hint, why the normal Update Accelerator does work. My version is just a enhancement of the existing framework.


Sorry, no idea of why it wasn't working, but I do now have a solution as I've used the install.tar.gz from N0man from here:

http://forum.ipfire.org/viewtopic.php?f=50&t=12868

I'm not a stranger to Update Accelerator, in fact I used if in IPCop for many many months and even extended the provided version to have more sets of data to be added to the cache and added in icons for other sources, so I'm going to review what has been written for ipfire to see if I can extend it. My IPCOP additions are here: https://github.com/pbinks/update-accelerator

I don't suppose you've got a Git repo for the current rewrite of the Update Accelerator that I could perhaps contribute to, or are you just passing updates between yourself and N0man?

Thanks,
Binky
Image

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 14th, 2015, 9:37 pm

No, I haven't saved my files to the IPFire Git, yet.
This will be one of the next steps, if installation and thus tests by others work.

At the moment I do the additions of new vendors by extracting them from N0man's archives. But this isn't really uptodate. If someone runs my new version, it is very easy to add new definitions.
I'll collect those to add them to the 'official' revision. Only the definition files will be needed.

I would be glad, if I would know where the install doesn't work properly.
I've installed to a separate directory and checked the files copied there. As far as I can see all files are transfered and are identical to my running version.
Image
Unitymedia Cable Internet ( 32MBit )

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » May 14th, 2015, 9:49 pm

For a check of integrity, I've added the MD5sum to the latest archive

EDIT new version 1.1-0a with correction of severe typos is available. The directories with the vendor definitions should be copied now.
Next version will check for a empty definition dir.

Post Reply