Page 1 of 1

continued troubles with backups

Posted: September 30th, 2019, 4:07 pm
by cwensink
Everyone,

I've been struggling with getting consistent backups to our remote site to complete over the weekends.

Here's the basic structure of backups:

Local:
Linux servers -> rsync -> Daisy(Linux)
Windows servers ->Windows server Backup software via samba share-> Daisy (Linux)

Remote:
Daisy -> rsync -> Donkeykong(Western Digital Mycloud Ex2 running busybox Linux)

The local backups work fine every day / week for backups, but the Transfer of data from the local backup server (Daisy) to remote NAS share (Donkeykong) is constantly failing each week.

I re-run the cron job scripts manually to get the backups to complete, which takes several attempts each time and eventually they all work.

Running Linux backups via rsync is done via a php command line script which passes through these rsync options:
--archive --compress --delete --delete-during --devices --hard-links --keep-dirlinks --numeric-ids --one-file-system --partial --specials --stats --timeout=3600 --update --quiet

The result of running the linux backup commands is a lot of little files that are transferred over

The result of the Windows backups is a lot of little files bundled up into very large .vhdx files

The transfer of the little files from the local back up server to the remote backup server typically kicks off and completes without issues

The transfer of the large windows backup files almost always times out with a variety of error messages:
code 20 rsync.c 543
code 12 io.c 760
code 12 io.c 601
code 30 io.c 200

The files on the Windows Servers are transferring very little week to week, but the md5sum changes on the large .vhdx files so there's a huge transfer that occurs every week that often times fails.

Backups are pushed from Daisy (local) to Donkeykong (remote)

There's a point to point vpn tunnel set up between office locations. using ipsec, w/ IKEv2 256 bit AES, lifetime 8 hours, Dead peer detection action: reset, timeout 120 delay 30

The local location has a Fiber connection that's 50/50 with an ipfire router
The remote location has a cable connection that's 30/5 with an ipfire router

As part of troubleshooting this issue I have tried:
-changing the date / time of when backups kick off
-changing switches that daisy is connected to
-changing switch ports that both sides are connected to
-resetting both internet connections on both sides
-resetting switches on both sides
-rebooting ipfire routers on both sides
-rebooting servers on both sides
-upgrading to different versions of rsync
-Adjusting the MTU on both sides of the tunnel to be 1300

Adjusting rsync settings:
--tried --delete-before, --delete-after for reliability adjustments
--tried --bwlimit=XXX to limit bandwidth

Sometimes the backups fail due to network timeouts, sometimes they fail due to io timeouts sometimes they fail because I run out of disk space.

Does anyone have any suggestions for improving reliability for regularly scheduled cron job backups, or suggestions for alternative programs / strategies for backups as a whole?

Chris

Re: continued troubles with backups

Posted: December 23rd, 2019, 3:05 pm
by SmithJ
Same issue here backup fails in between :(