Rewrote Update Accelerator

Help on building IPFire & Feature Requests
BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Rewrote Update Accelerator

Post by BeBiMa » April 17th, 2015, 7:30 am

Which version of "new Update Accelerator" did you install?
If it is my "revisited version" just install version 1.0 from viewtopic.php?f=50&t=12984 and report errors in that thread.

-Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
makara
Posts: 11
Joined: March 16th, 2013, 8:25 pm
Location: New Zealand

Re: Rewrote Update Accelerator

Post by makara » May 6th, 2015, 8:41 pm

Hi -

I hope you don't mind me asking a question about Update Accelerator here.

I've been using update accelerator on a network with around 100 computers, over the last 7 months the statistics show 66Gb in the cache and 46Tb delivered from the cache.

Over the same period the data going through the red and green network interfaces is about the same. I would have expected a lot more traffic on Green to reflect the data coming from the Update Accelerator cache, or have I got that wrong?

David

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Rewrote Update Accelerator

Post by BeBiMa » May 6th, 2015, 9:17 pm

Hi,

it's okay asking your question here. I'm listening to both threads. ;)

The relation between traffic of red/green and traffic from Update Accelerator depends heavily on the kind of the traffic.
Update Accelerator don't cache all downloads. Only these files which are defined in source ( old version ) or definition files ( new version ) are cached. Furthermore an initial download counts twice: one download to the repository, one download to the requesting client.

The efficiency index in the statistics is one hint for this situation. It is defined as the ratio #requests/#files. A value near 1.0 means that most files are requested only once. ( Traffic for this file: 2*filesize on red, 1*filesize on green = 3*filesize )
Furthermore the traffic on red and green is measured outside Update Accelerator and contains other traffic ( e.g. the files exchanged to check what files should be updated are not cached ).

Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
makara
Posts: 11
Joined: March 16th, 2013, 8:25 pm
Location: New Zealand

Re: Rewrote Update Accelerator

Post by makara » May 6th, 2015, 10:01 pm

Thanks for the quick reply Bernard.

I'm still using the version of Update Accelerator (UA) bundled with IPfire, which I guess is the old version.

With the cache efficiency currently at around 16, I expected to see a lot more traffic on the green interface than the red, given UA reports it has delivered around 45Tb of data from a 67Gb cache size.

Regards


David

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Rewrote Update Accelerator

Post by N0man » May 7th, 2015, 12:12 am

I fixed Java Updates.

The package is in the first post.

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Rewrote Update Accelerator

Post by BeBiMa » May 7th, 2015, 7:06 am

makara wrote:With the cache efficiency currently at around 16, I expected to see a lot more traffic on the green interface than the red, given UA reports it has delivered around 45Tb of data from a 67Gb cache size.

How do you look up over-all traffic on green and red?
It's true, a index of 16 implies many deliveries from repository. Unfortunately the index is calculated on file numbers, not on file size. Thus many small files result in a big index value, but low traffic.
Nevertheless traffic from repository is only a part of the traffic on green.
How are the figures for red and green in this period?
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
makara
Posts: 11
Joined: March 16th, 2013, 8:25 pm
Location: New Zealand

Re: Rewrote Update Accelerator

Post by makara » May 7th, 2015, 8:46 am

Hi there.

I have used "Net-Traffic" option in the "Status" tab of the IPFire GUI to monitor Red and Green network use. Over the last 7 months they are very similar (red data throughput is almost the same as green).

That's why I was doubting that Update Accelerator had delivered 46Tb more than the cache size, as the difference in red and green data is less than a 500Gb over the same period.

D

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Rewrote Update Accelerator

Post by BeBiMa » May 7th, 2015, 9:27 am

Didn't check how the traffic stats are calculated by vnstat. I suppose it is the real traffic flowing through the interface, which would be exact, and I don't remember posts about issues with this figures.
The amount "Total data delivered from cache" in the Update Accelerator statistics isn't measured traffic, but calculated as sum of number_of_access*file_size over all files in repository. Thus an request for an file, which is stopped in progress of downloading, counts for the full file size not only the transmitted amount. This situation could be generated by "impatient" updaters, which do multiple requests for the same file and cancel all but one when data for this file is received.


Example:
  • file with 20MB
  • three requests are started
  • after data flow of 500 bytes for each request two requests are stopped
  • "delivered" = 3*20MB = 60MB
  • tx on green = 2*500+20MB = 21MB
Makes a difference in Update Accelerator statistics and NetTraffic of 39MB.
Image
Unitymedia Cable Internet ( 32MBit )

bloater99
Posts: 482
Joined: October 13th, 2014, 3:47 pm

Re: Rewrote Update Accelerator

Post by bloater99 » May 7th, 2015, 7:37 pm

Nice work everyone! If I decide to install this before it's officially included in a Core update:
1) will anything special need to be done when the Core Update that eventually includes this is released?
2) I currently have 16GB of files in my Update Accelerator. Will the existing update accelerator cache remain, or does it wipe it out and start from scratch when the update is installed?

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Rewrote Update Accelerator

Post by BeBiMa » May 7th, 2015, 8:49 pm

bloater99 wrote:1) will anything special need to be done when the Core Update that eventually includes this is released?
I'll try to implement that "smoothly". The data structures should be identical for the last test version and the released version.
bloater99 wrote:2) I currently have 16GB of files in my Update Accelerator. Will the existing update accelerator cache remain, or does it wipe it out and start from scratch when the update is installed?

At the moment it is planned to leave the basic directory structure ( one directory for each vendor/source with subdirectories for each update file ) untouched. The meta data will move from the subdirs <vendorid>/<uuid> to data base in <vendorid>. A conversion program will be supplied.
Other main change will be the definition of sources, but that doesn't influence the repository. Leaving the old meta data on place should make it possible to work on the repository with the old and the new set of programs.

EDIT: The installer for version 1.1 contains a converter from "old" file based meta data to db based metas. The uninstaller updates the file based meta data with the information hold in the db. Access log is somewhat "faked". The DB holds last access and number of accesses only ( only these data are used in WUI), while the file based access.log holds a list of accesses. The converter just fills access.log with number of access entries of the last access time stamp.
Last edited by BeBiMa on May 13th, 2015, 10:05 am, edited 1 time in total.
Image
Unitymedia Cable Internet ( 32MBit )

cardins2u
Posts: 9
Joined: April 25th, 2015, 6:25 pm

Re: Rewrote Update Accelerator

Post by cardins2u » May 13th, 2015, 8:31 am

thank you for the great work. your seem more stables.

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Rewrote Update Accelerator

Post by N0man » May 20th, 2015, 1:54 am

I added support for Adobe zip packages
Removed overlapping Trend Micro Update

Narrowed down GIMP and Virtualbox (Used to cause issues browsing directory index of download websites)

Added Ubiquiti Products
Added Spotify

Removed Playstation Games (Did not like it)

BeBiMa
Posts: 2842
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Rewrote Update Accelerator

Post by BeBiMa » May 20th, 2015, 11:40 am

I'll integrate your definitions into the next version of my new UA. ( I've found a solution to simplify the transfer from the updxlrator script to my vendors definitions ;) )

Two things I've found during analysis:
  • 'mcafee' doesn't adhere to the rule "vendor names begin with an uppercase character
  • 'Ad-Aware' contains a character '-' which isn't allowed in vendor names in the new UA. Further the corresponding icon file is named 'updxl-src-adaware.gif', which doesn't match the vendor name 'Ad-Aware'

Regards,
Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Rewrote Update Accelerator

Post by N0man » May 21st, 2015, 12:14 am

I changed Mcafee and Adaware in my version so the sources fit your format.

User avatar
N0man
Posts: 299
Joined: July 26th, 2013, 1:56 am

Re: Rewrote Update Accelerator

Post by N0man » May 27th, 2015, 8:20 pm

I added the Steam client and Evernote:

I also fixed Apple Updates.

Post Reply