Update Accelerator revisited

Help on building IPFire & Feature Requests
BeBiMa
Posts: 2563
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » February 27th, 2016, 1:32 pm

The cow isn't dead! ;)

It only had no time to check and try the handling for real management of the download bandwidth. ( A solution isn't implemented already :( ).

But I hope to release a new version with updated source lists ( and the code running for months now in my installation here ), soon.

Greetings,
Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

tt22tt
Posts: 3
Joined: August 24th, 2016, 10:49 am

Re: Update Accelerator revisited

Post by tt22tt » August 24th, 2016, 11:04 am

Hi, thank you for your work :-)
Is this an official addon which I can download in pakfire?

deswong
Posts: 3
Joined: December 26th, 2012, 9:21 am

Re: Update Accelerator revisited

Post by deswong » September 25th, 2016, 10:21 am

Just checking in to see if there is an ETA on a newer version? I have been having issues with my current install - and would like to try a later version rather than clean install and re-setup.

JDSponky
Posts: 57
Joined: September 19th, 2015, 6:35 pm

Re: Update Accelerator revisited

Post by JDSponky » November 27th, 2016, 9:41 pm

Hello!

What's going on with this Tool? Will it soon replace the existing Updater?
Image

Mahagon
Posts: 5
Joined: December 5th, 2013, 10:29 am

Re: Update Accelerator revisited

Post by Mahagon » July 17th, 2017, 8:03 am

Just found this Project

BeBIMa hasn't been active since Mon Aug 22, 2016.
I just sent him an pm hoping he gets a mail notification ^^

Uploading it to GitHub or sth like that would be a solution. Maybe someone could continue his work.

robertgpeterson
Posts: 33
Joined: April 21st, 2014, 11:47 am

Re: Update Accelerator revisited

Post by robertgpeterson » October 9th, 2017, 9:01 pm

Is this rewrite reasonably ready for production use? Is there an uninstall/revert if necessary?

Update Accelerator has been a distinctive and key feature of IPFire for our context in Africa on a limited bandwidth satellite connection. It is a game-changer in comparison to any other firewall solution out there. I hope that it continues to be an integral and high priority part of IPFire's development. Our live version has 180+ GB of updates cached and used to average around 18:1 ratio. Windows 10 change to cumulative updates doesn't function as well with it and the ratio is closer to 12:1 now.

DNS filtering is another aspect that would be so helpful at the GUI level. I'm using external DNS filtering for the moment with selective URL filtering as backup.

Rob

jlgtx
Posts: 28
Joined: August 19th, 2016, 4:23 am

Re: Update Accelerator revisited

Post by jlgtx » December 2nd, 2017, 5:27 pm

One of the most significant problems with Update Accelerator is that it doesn't manage its cached files well. Even the manual maintenance tool is incomplete; e.g., there's no way to delete cached updates older than a certain date or smaller than a certain size, clean up duplicate updates, etc. If hard drive space were infinite, this would be no problem...but I'm running IPfire on a mini-PC with a fairly small CF filesystem, and Update Accelerator eats it up space pretty quickly. If you don't mind doing a bit of command-line stuff, you can use this shell script and crontab entry to clean up the "big two" problems--old files and duplicates.

Login to your IPfire host as root (replace 1.2.3.4 with your IPfire host IP):

# ssh root@1.2.3.4 -p 222

Now create the script...

# vi cleanupUA.sh

Code: Select all

#!/bin/bash
# change this to the number of days you want to keep!
age=365
#
# This only works on the default cache directory. Things will break if you've moved it to a deeper or shallower path.
cachedir=/var/updatecache
# Touch the base update directories to avoid deleting them!
touch $cachedir/*
echo "Removing old updates..."
for upd in `find $cachedir -type f -mtime +$age -print|cut -f1-5 -d'/'`
do
  echo "  $upd"
  rm -rf $upd
done
#
# Now get rid of duplicate updates, using an md5sum to be sure they're really duplicates.
#
echo "Removing duplicate updates..."
done=0
while [ $done = 0 ]
do
# The find command spits out a list of all duplicates. We want to keep one of them, and the easiest way
# is to delete one file at a time, the first one in the list, then redo the find. A bit slow, but it works.
upd=`find $cachedir ! -empty -type f \( -iname "*.cab" -o -iname "*.exe" \) -exec md5sum {} + | sort | uniq -w32 -dD | head -1 | cut -c34-100 | cut -f1-5 -d'/'`
if [ "$upd" != "" ]
then
  echo "  $upd"
  rm -rf $upd
else
  done=1
fi
done
echo "Finished."
Make it executable:

# chmod 700 cleanupUA.sh

Run it to see it do its thing:

# ./cleanupUA.sh

To run the script automatically on a monthly basis, run "fcrontab -e" and insert the following rows at the end of the file:

Code: Select all

# Clean up the Update Accelerator cache once a month.
%monthly * * * /root/cleanupUA.sh >/dev/null 2>&1

robertgpeterson
Posts: 33
Joined: April 21st, 2014, 11:47 am

Re: Update Accelerator revisited

Post by robertgpeterson » December 4th, 2017, 7:48 pm

Thank you. Appreciate the very clear instructions. Will run this soon.

jluth@mail.com
Posts: 9
Joined: March 24th, 2015, 4:28 pm

Re: Update Accelerator revisited

Post by jluth@mail.com » December 6th, 2017, 8:52 am

robertgpeterson wrote:
October 9th, 2017, 9:01 pm
Is this rewrite reasonably ready for production use?
History has shown (in both ipcop and ipfire) that you can't wait for pending updates to updxlrator to actually arrive. If you are capable of fixing any problems by yourself, then just move forward on your own.

An in regards to additional changes needed in updxlrator, many microsoft patches now ought to be matched on filename ($mirror) rather than url ($unique); specifically the files with an SHA1 hash in their name. That's really hitting me hard with Windows 10 updates being pulled from different mirrors.

Of course, this means that the existing cache files won't match, but some scripting can rename that. I've created a bug report for this at https://bugzilla.ipfire.org/show_bug.cgi?id=11558

robertgpeterson
Posts: 33
Joined: April 21st, 2014, 11:47 am

Re: Update Accelerator revisited

Post by robertgpeterson » December 6th, 2017, 7:22 pm

I'm not a programmer but this sounds very promising. I wonder if our situation is particularly disfunctional with being in Africa, the system can never decide which mirror to use and seems to pull from Australia, Europe and the US...sometimes all at the same time.

I think I could manage changing the code but the symbolic link section would require line-by-line instructions for me to try it.

jluth@mail.com
Posts: 9
Joined: March 24th, 2015, 4:28 pm

Re: Update Accelerator revisited

Post by jluth@mail.com » January 9th, 2018, 6:54 am

The next release of ipfire will include fixes for updxlrator.
1.) With Microsoft's new way of updating, it can easily make multiple requests for the same URL within one second. This was exposing a flaw that allowed multiple downloads of the same file to occur. Have you ever seen a download climb to greater than 100% of the file size? This is likely the reason - that multiple downloads were dumping on top of the same filename. Obviously in the end this is corrupt anyway, so that file, downloaded multiple times, ends up being useless, wasting a lot of bandwidth. That has been fixed. https://bugzilla.ipfire.org/show_bug.cgi?id=11567

2.) Updatxlrator has two modes - a mirror mode which requires that the filename be unique regardless of the server that it downloads from, or URL mode, where the entire URL must be matched exactly. ALL Microsoft downloads were based on URL mode. With the huge (5+GB) windows 10 updates that come each month, they could be downloaded multiple times, depending on which mirrors the client requested it from. Sometimes a single client will request from multiple mirrors for the same file. Most of Microsoft's updates now include an SHA1 number in their filename, indicating that the file is unique. This allows updatexlrator to just use the filename as the identifiable part, instead of the entire URL. So for Microsoft updates, updxlrator now notices when an SHA1 number is present. If so, it first looks to see if the file is already cached using the old method (the entire URL). If so, it uses that cache. If not, it will switch over to filename mode, check if the download exists in that cache, and if not it will download and save as filename mode. https://bugzilla.ipfire.org/show_bug.cgi?id=11558
This means that for any NEW updates that come in, they will only ever be downloaded one time. For 2017 updates, it is possible that you STILL might download an update once more IF a client requests it from a mirror that wasn't accessed previously. (Of course, that would also have been true before this update...) So there isn't really any need for the end user to do anything, but for any programming type of users who REALLY want to avoid a potential extra download, the bug report has a script that can be modified to rename the existing URL-based caches into filename-based caches. (It needs to be modified so that it renames the folder instead of creating a link.)

3.) filenames containing a + or a ~ were never being retrieved from the cache. That has been fixed. https://bugzilla.ipfire.org/show_bug.cgi?id=10504

@everyone who tinkers with updxlrator: Make sure you have a backup of your updxlrator files if you have modified them (to add custom caches etc.) I'm quite sure that the upgrade process will replace your custom version, wiping out your customizations.

@Robert. I'm also in Africa (South Sudan) and these issues over satellite really hit us hard, so I dug into it until I figured out what the problems / solutions were. The latency in getting a download started opens up a multi-second window for simultaneous downloads (problem #1). So the sites that need updxlrator the most were also the most affected by the bug. These updates should come automatically in Core 118, so no need for you to do any scripting any more.

@jlgtx: deleting duplicates is not a good idea, since the cache (for Microsoft updates) is not based on filename, but on the URL. The deleted duplicates would have downloaded again if a client asks for that particular URL. (But of course, that has now changed for SHA1-named files).

dnl
Posts: 311
Joined: June 28th, 2013, 11:03 am

Re: Update Accelerator revisited

Post by dnl » February 24th, 2018, 4:37 am

Hello,

I'm glad to see this is still being used and updated.
Is somebody officially maintaining it now?


I was hoping to ask if it would be possible to add a feature. I'd like to split the functionality in to multiple files so that I can maintain my own "source_url" sections without them being overwritten every time the script is updated.

I have a network where I only want to cache downloads for one specific operating system. This is because there is at most 2 systems of any other operating system and the Update Accelerator results in two complete downloads anyway. The default settings for the Update Accelerator can result in less efficient usage of the download quota than not using it at all.


So I was wondering if it was possible to make the script modular. It could have 3 major parts:
  1. The main perl script, which includes a reference to
  2. a configuration file, listing all of the files which contain source_urls (today these are in "Section" heading)
    in which the main while loop
  3. and split today's "Section" headings with source_urls in to multiple files. Let's call them "Sources".
The config file would allow a user to turn on and off individual modules ("Section" files containing source_urls). If necessary a user (like me) could add reference to their own custom "Source" file.

The idea being that the main perl script and the source_url files (modules) could be updated by the maintainer, while the config file can be left alone after initial install.

I don't know Perl. Would that be complex to do?

Thank you!
Image

dnl
Posts: 311
Joined: June 28th, 2013, 11:03 am

Re: Update Accelerator revisited

Post by dnl » February 24th, 2018, 4:43 am

PS: I am NOT expecting a web interface for this. While that would be nice the update accelerator is incomplete in other important ways, like automatically pruning cache files (see jlgtx's post above).
Image

scaredycrow
Posts: 8
Joined: February 9th, 2015, 6:50 am

Re: Update Accelerator revisited

Post by scaredycrow » April 4th, 2018, 12:57 pm

Should we be expecting all Windows 10 updates to cache properly now?

I'm seeing pretty much everything except the Cumulative Updates (which are the largest) cached. Example KB4088776 is not caching.
Image

jluth@mail.com
Posts: 9
Joined: March 24th, 2015, 4:28 pm

Re: Update Accelerator revisited

Post by jluth@mail.com » April 4th, 2018, 1:08 pm

Yes, I would expect it to be cached. Do you see it in the updatexlrator page being downloaded? What makes you think that it isn't being cached?
My only guess would be that your connection isn't stable enough to complete a very long download, so it aborts before finishing.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests