desaster upgrading from 119 over 120 => 121 => 122 to 123 (no web interface, many missing system files)

General questions.
Post Reply
wech
Posts: 8
Joined: September 12th, 2018, 2:21 pm

desaster upgrading from 119 over 120 => 121 => 122 to 123 (no web interface, many missing system files)

Post by wech » September 12th, 2018, 2:57 pm

Hi there,

I have updated two ipfire firewalls today, the first one did not produce any problems (120=>123), but the second one (a production system) resulted in a near-desaster.

I hadn't upgraded the production firewall for some time (because of fear of problems), so I had to do several upgrade steps from 119 to 123.

Everything seemed fine until the last upgrade step (I think it was 122 to 123, but it might have been 121 to 123). The web interface was no longer available.

When I logged in trough ssh I found that some essential files like /bin/tar, /usr/sbin/apachectl, usr/sbin/httpd were missing, and after copying them from the installation iso image, even more files were missing.

When examining the update-core-upgrade-123.log i noticed these errors

./update.sh: line 58: rebuild-initrd: command not found
./update.sh: line 69: grub-mkconfig: command not found

After some hours of work, I managed to get the web interface running again.

I then set the patchlevel to 122 and tried an pakfire upgrade, which installed 123 without further problems. Unfortunately the boot environment still was bad.

Therefore I set the patchlevel to 121 and again tried an pakfire upgrade, which downloaded core-upgrade-122 and core-upgrade-123 and then produced the following output and error:

---

PAKFIRE RESV: linux-pae: Resolving dependencies...

PAKFIRE UPGR: We are going to install all packages listed above.
PAKFIRE INFO: Is this okay? [y/N]
y

PAKFIRE UPGR: linux-pae: Decrypting...
PAKFIRE UPGR: linux-pae: Upgrading files and running post-upgrading scripts...


Message from syslogd@fw at Wed Sep 12 14:52:24 2018 ...

fw root[27260]: pakfire linux-pae: no pae support found, aborted!

PAKFIRE ERROR: Returncode: 1. Sorry. Please search our forum to find a solution for this problem.

---

After this again many files were missing.

The bad system has pae support:
Linux fw 3.14.79-ipfire-pae #1 SMP Thu Feb 2 00:43:29 GMT 2017 i686 Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz GenuineIntel GNU/Linux

grep "^flags.* pae " /proc/cpuinfo

flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
...


Looking at /var/log/pakfire/update-linux-pae.log shows that the update process removed all the files and THEN found a problem with pae (because grep could no longer load libpcre.so.1 and therefore could not grep cpuinfo) and then stopped the upgrade - leaving the system in an inconsistent state.

The check should IMHO really be before it removes old versions as this leads to this fatal situation:

removed '/var/log/httpd/ssl_request_log'
removed '/var/log/httpd/error_log'
removed directory '/var/log/httpd/captive'
removed directory '/var/log/httpd'
...Finished.
./update.sh: line 30: grub-mkconfig: command not found
grep: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory

I don't know how to best handle the grep libpcre-problem - maybe a statically linked grep or not removing libpcre at all during upgrade. I also don't know why it worked fine on the other ipfire installation.

May this be a result of the new slimmer update packages process?

If yes, then in cases like this it would help if it would be possible to install a full upgrade or some kind of system-restore to a defined state. In my case installing from an iso image was only possible as a last resort, because the necessary downtime would have produced too many complaints.

Some kind of "pakfire reinstall" while keeping configuration files would have saved me many hours of manual work.

Epilog: An old saying here is "Second time it's much faster to do.", so I managed to repair the system again as I did now know how where to look and what to copy. Still, as the machine is located pretty far away, I am not sure if the machine would reboot - I just don't dare to try ;-)

In the hope that this helps somebody.

side notice: I also had problems with the ssh keys which could not be loaded (maybe old format?), but regenerating helped. Still got an warning because it could not load the dsa key, because the /etc/ssh/sshd_config did not specifiy the host keys (commented out).

Post Reply