Update Accelerator revisited

Help on building IPFire & Feature Requests
BeBiMa
Posts: 2509
Joined: July 30th, 2011, 12:55 pm
Location: Mannheim

Re: Update Accelerator revisited

Post by BeBiMa » February 27th, 2016, 1:32 pm

The cow isn't dead! ;)

It only had no time to check and try the handling for real management of the download bandwidth. ( A solution isn't implemented already :( ).

But I hope to release a new version with updated source lists ( and the code running for months now in my installation here ), soon.

Greetings,
Bernhard
Image
Unitymedia Cable Internet ( 32MBit )

tt22tt
Posts: 3
Joined: August 24th, 2016, 10:49 am

Re: Update Accelerator revisited

Post by tt22tt » August 24th, 2016, 11:04 am

Hi, thank you for your work :-)
Is this an official addon which I can download in pakfire?

deswong
Posts: 3
Joined: December 26th, 2012, 9:21 am

Re: Update Accelerator revisited

Post by deswong » September 25th, 2016, 10:21 am

Just checking in to see if there is an ETA on a newer version? I have been having issues with my current install - and would like to try a later version rather than clean install and re-setup.

JDSponky
Posts: 56
Joined: September 19th, 2015, 6:35 pm

Re: Update Accelerator revisited

Post by JDSponky » November 27th, 2016, 9:41 pm

Hello!

What's going on with this Tool? Will it soon replace the existing Updater?
Image

Mahagon
Posts: 5
Joined: December 5th, 2013, 10:29 am

Re: Update Accelerator revisited

Post by Mahagon » July 17th, 2017, 8:03 am

Just found this Project

BeBIMa hasn't been active since Mon Aug 22, 2016.
I just sent him an pm hoping he gets a mail notification ^^

Uploading it to GitHub or sth like that would be a solution. Maybe someone could continue his work.

robertgpeterson
Posts: 33
Joined: April 21st, 2014, 11:47 am

Re: Update Accelerator revisited

Post by robertgpeterson » October 9th, 2017, 9:01 pm

Is this rewrite reasonably ready for production use? Is there an uninstall/revert if necessary?

Update Accelerator has been a distinctive and key feature of IPFire for our context in Africa on a limited bandwidth satellite connection. It is a game-changer in comparison to any other firewall solution out there. I hope that it continues to be an integral and high priority part of IPFire's development. Our live version has 180+ GB of updates cached and used to average around 18:1 ratio. Windows 10 change to cumulative updates doesn't function as well with it and the ratio is closer to 12:1 now.

DNS filtering is another aspect that would be so helpful at the GUI level. I'm using external DNS filtering for the moment with selective URL filtering as backup.

Rob

jlgtx
Posts: 14
Joined: August 19th, 2016, 4:23 am

Re: Update Accelerator revisited

Post by jlgtx » December 2nd, 2017, 5:27 pm

One of the most significant problems with Update Accelerator is that it doesn't manage its cached files well. Even the manual maintenance tool is incomplete; e.g., there's no way to delete cached updates older than a certain date or smaller than a certain size, clean up duplicate updates, etc. If hard drive space were infinite, this would be no problem...but I'm running IPfire on a mini-PC with a fairly small CF filesystem, and Update Accelerator eats it up space pretty quickly. If you don't mind doing a bit of command-line stuff, you can use this shell script and crontab entry to clean up the "big two" problems--old files and duplicates.

Login to your IPfire host as root (replace 1.2.3.4 with your IPfire host IP):

# ssh root@1.2.3.4 -p 222

Now create the script...

# vi cleanupUA.sh

Code: Select all

#!/bin/bash
# change this to the number of days you want to keep!
age=365
#
# This only works on the default cache directory. Things will break if you've moved it to a deeper or shallower path.
cachedir=/var/updatecache
# Touch the base update directories to avoid deleting them!
touch $cachedir/*
echo "Removing old updates..."
for upd in `find $cachedir -type f -mtime +$age -print|cut -f1-5 -d'/'`
do
  echo "  $upd"
  rm -rf $upd
done
#
# Now get rid of duplicate updates, using an md5sum to be sure they're really duplicates.
#
echo "Removing duplicate updates..."
done=0
while [ $done = 0 ]
do
# The find command spits out a list of all duplicates. We want to keep one of them, and the easiest way
# is to delete one file at a time, the first one in the list, then redo the find. A bit slow, but it works.
upd=`find $cachedir ! -empty -type f \( -iname "*.cab" -o -iname "*.exe" \) -exec md5sum {} + | sort | uniq -w32 -dD | head -1 | cut -c34-100 | cut -f1-5 -d'/'`
if [ "$upd" != "" ]
then
  echo "  $upd"
  rm -rf $upd
else
  done=1
fi
done
echo "Finished."
Make it executable:

# chmod 700 cleanupUA.sh

Run it to see it do its thing:

# ./cleanupUA.sh

To run the script automatically on a monthly basis, run "fcrontab -e" and insert the following rows at the end of the file:

Code: Select all

# Clean up the Update Accelerator cache once a month.
%monthly * * * /root/cleanupUA.sh >/dev/null 2>&1

robertgpeterson
Posts: 33
Joined: April 21st, 2014, 11:47 am

Re: Update Accelerator revisited

Post by robertgpeterson » December 4th, 2017, 7:48 pm

Thank you. Appreciate the very clear instructions. Will run this soon.

jluth@mail.com
Posts: 8
Joined: March 24th, 2015, 4:28 pm

Re: Update Accelerator revisited

Post by jluth@mail.com » December 6th, 2017, 8:52 am

robertgpeterson wrote:
October 9th, 2017, 9:01 pm
Is this rewrite reasonably ready for production use?
History has shown (in both ipcop and ipfire) that you can't wait for pending updates to updxlrator to actually arrive. If you are capable of fixing any problems by yourself, then just move forward on your own.

An in regards to additional changes needed in updxlrator, many microsoft patches now ought to be matched on filename ($mirror) rather than url ($unique); specifically the files with an SHA1 hash in their name. That's really hitting me hard with Windows 10 updates being pulled from different mirrors.

Of course, this means that the existing cache files won't match, but some scripting can rename that. I've created a bug report for this at https://bugzilla.ipfire.org/show_bug.cgi?id=11558

robertgpeterson
Posts: 33
Joined: April 21st, 2014, 11:47 am

Re: Update Accelerator revisited

Post by robertgpeterson » December 6th, 2017, 7:22 pm

I'm not a programmer but this sounds very promising. I wonder if our situation is particularly disfunctional with being in Africa, the system can never decide which mirror to use and seems to pull from Australia, Europe and the US...sometimes all at the same time.

I think I could manage changing the code but the symbolic link section would require line-by-line instructions for me to try it.

jluth@mail.com
Posts: 8
Joined: March 24th, 2015, 4:28 pm

Re: Update Accelerator revisited

Post by jluth@mail.com » January 9th, 2018, 6:54 am

The next release of ipfire will include fixes for updxlrator.
1.) With Microsoft's new way of updating, it can easily make multiple requests for the same URL within one second. This was exposing a flaw that allowed multiple downloads of the same file to occur. Have you ever seen a download climb to greater than 100% of the file size? This is likely the reason - that multiple downloads were dumping on top of the same filename. Obviously in the end this is corrupt anyway, so that file, downloaded multiple times, ends up being useless, wasting a lot of bandwidth. That has been fixed. https://bugzilla.ipfire.org/show_bug.cgi?id=11567

2.) Updatxlrator has two modes - a mirror mode which requires that the filename be unique regardless of the server that it downloads from, or URL mode, where the entire URL must be matched exactly. ALL Microsoft downloads were based on URL mode. With the huge (5+GB) windows 10 updates that come each month, they could be downloaded multiple times, depending on which mirrors the client requested it from. Sometimes a single client will request from multiple mirrors for the same file. Most of Microsoft's updates now include an SHA1 number in their filename, indicating that the file is unique. This allows updatexlrator to just use the filename as the identifiable part, instead of the entire URL. So for Microsoft updates, updxlrator now notices when an SHA1 number is present. If so, it first looks to see if the file is already cached using the old method (the entire URL). If so, it uses that cache. If not, it will switch over to filename mode, check if the download exists in that cache, and if not it will download and save as filename mode. https://bugzilla.ipfire.org/show_bug.cgi?id=11558
This means that for any NEW updates that come in, they will only ever be downloaded one time. For 2017 updates, it is possible that you STILL might download an update once more IF a client requests it from a mirror that wasn't accessed previously. (Of course, that would also have been true before this update...) So there isn't really any need for the end user to do anything, but for any programming type of users who REALLY want to avoid a potential extra download, the bug report has a script that can be modified to rename the existing URL-based caches into filename-based caches. (It needs to be modified so that it renames the folder instead of creating a link.)

3.) filenames containing a + or a ~ were never being retrieved from the cache. That has been fixed. https://bugzilla.ipfire.org/show_bug.cgi?id=10504

@everyone who tinkers with updxlrator: Make sure you have a backup of your updxlrator files if you have modified them (to add custom caches etc.) I'm quite sure that the upgrade process will replace your custom version, wiping out your customizations.

@Robert. I'm also in Africa (South Sudan) and these issues over satellite really hit us hard, so I dug into it until I figured out what the problems / solutions were. The latency in getting a download started opens up a multi-second window for simultaneous downloads (problem #1). So the sites that need updxlrator the most were also the most affected by the bug. These updates should come automatically in Core 118, so no need for you to do any scripting any more.

@jlgtx: deleting duplicates is not a good idea, since the cache (for Microsoft updates) is not based on filename, but on the URL. The deleted duplicates would have downloaded again if a client asks for that particular URL. (But of course, that has now changed for SHA1-named files).

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests