yum update times out after upgrade to 131

General questions.
cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

yum update times out after upgrade to 131

Post by cwensink » May 28th, 2019, 4:29 pm

Hello Everyone,

I just upgraded to IPFire 2.23 (x86_64) - Core Update 131, and after rebooting the ipfire box, all CentOS Servers that are in the LAN / and ones in the DMZ when trying to run yum update are timing out with slow connections:
----------------------------------
[root@peach etc]# yum update --noplugins
Setting up Update Process
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6 ... nfra=stock error was
12: Timeout on http://mirrorlist.centos.org/?release=6 ... nfra=stock: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds')
Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again
[root@peach etc]#
-----------------------

I've tried clearing the proxy cache, adding *.centos.org to the custom white list. I've tried setting the proxy to run not in transparent mode, disabling and enabling the url filter and update accelerator options, and I've tried commenting out the proxy line in each of the servers. Nothing has made a difference.

It's like the proxy is somehow filtering or limiting bandwidth to centos yum mirrored sites, but I can't figure out where or how.

Any suggestions?

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 28th, 2019, 4:57 pm

We've not experienced this issue at all with any of our CentOS servers, connected to numerous firewalls on various ISP's. If this just started happening, could it be something changed in iptables? Have you checked the iptables to see if it has the Apply button displayed?

I'm curious as to why you are using --noplugins option with yum. One of the default plugins specifically addresses finding the quickest mirror and passing by ones which don't respond.

Best regards,
Fred

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 28th, 2019, 5:42 pm

I haven't changed any iptables rules, Image
ipfire.JPG
I don't see the apply button listed. Just to be safe I turned a rule on and off so the apply button did show up, then tried yum update again, and I had the same result.

I had the --noplugins option running as part of troubleshooting. Normally I just run #

Code: Select all

yum update
without any extra parameters first to see what updates are available.

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 28th, 2019, 6:45 pm

I see you are running C6-x64. I tried the mirrors link listed in the log you provided on our local IPFire system. A response was returned in less than a second. We have a few C6-x64 systems still around and I tried it on one of them and no issues were reported as well.

You said the server is in your DMZ. Are you sure this is just the DMZ which is affected? Have you tried this from Green to see if it works there?

Have you made sure this is not a DNS issue? We have seen a LOT more DNS issues of late with IPFire, particularly with Cloudflare.

Best regards,
Fred

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 28th, 2019, 8:00 pm

We're having the issues under Green and Orange, so the LAN and the DMZ.

It does not appear to be a DNS issue at first glance, I can ping by name and IP mirror.centos.org:
-------------------------
[root@peach etc]# ping mirror.centos.org
PING mirror.centos.org (72.9.156.254) 56(84) bytes of data.
64 bytes from 72.9.156.254.tailormadeservers.com (72.9.156.254): icmp_seq=1 ttl=52 time=36.4 ms
64 bytes from 72.9.156.254.tailormadeservers.com (72.9.156.254): icmp_seq=2 ttl=52 time=36.4 ms
64 bytes from 72.9.156.254.tailormadeservers.com (72.9.156.254): icmp_seq=3 ttl=52 time=36.5 ms
64 bytes from 72.9.156.254.tailormadeservers.com (72.9.156.254): icmp_seq=4 ttl=52 time=36.5 ms
^C
--- mirror.centos.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3717ms
rtt min/avg/max/mdev = 36.403/36.509/36.590/0.244 ms
--------------------

As I'm writing this post, the issue seems to have resolved itself, and it very well may have been a DNS issue. From an internal CentOS 6 server I resolved mirror.centos.org to 72.9.156.254.

When I ran the ping from the command prompt on ipfire, it resolved to 69.167.138.255.

in /etc/resolv.conf of ipfire it just has it's own server:

search (internal domain)
nameserver: 127.0.0.1

If I run setup from the command line I can see the Primary, and Secondary DNS servers and the default gateway of our ISP, which has not changed.

As part of troubleshooting this I did uncheck the enabled on green checkbox for the proxy, and the transparent on green checkbox. Then I hit save, then save and reload, then clear cache, then save and restart.

Then I was able to run yum update without issues.

I then re-enabled the proxy on green and made it transparent on green, then ran another ping on mirror.centos.org and I got 66.241.106.180, and yum update works fine.

Just for my knowledge of troubleshooting this in the future, if it is a DNS issue / bad cache issue in the future, what's the right procedure for using the four buttons for the proxy: save, save and reload, save and restart, and clear cache? Should only one need to be done, or do all need to be done, and if so what order?

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 29th, 2019, 2:08 pm

Seems like the problem is back today. The problem seems to occur whenever the proxy is on, and set to be transparent. Whenever it's on and transparent it breaks centos servers' ability to run yum update. I need a way to keep the transparent proxy on all the time but set ipfire to exclude all servers from being tucked behind the proxy. Any thoughts?

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 29th, 2019, 3:06 pm

On the Web Proxy page, try adding your server IP addresses in the block labeled "Unrestricted IP addresses (one per line):"

Best regards,
Fred

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 29th, 2019, 3:10 pm

To correct the DNS issue you can either do:
- do a restart the IPFire hardware
- at the command prompt on the IPFire hardware, type "/etc/init.d/unbound restart"

Either one will force the unbound to clear and reconnect to the DNS servers you have specified.

Best regards,
Fred

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 30th, 2019, 4:09 pm

Adding the ip address to the unrestricted line did not help.

I tried doing a /etc/init.d/unbound restart, and at the command line on ipfire this is what I got:

[root@ipfire log]# /etc/init.d/unbound restart
Stopping Unbound DNS Proxy... [ OK ]
Starting Unbound DNS Proxy... [ OK ]
Ignoring broken upstream name server(s): 208.67.222.222 [ WARN ]
Configuring upstream name server(s): 12.127.16.67 [ OK ]

So, I thought maybe there's an issue with the DNS server (opendns) - 208.67.222.222

I tried changing to our local ISP's secondary DNS server, the one given to us by our provider, then ran the same unbound option and I got this:

[root@ipfire log]# /etc/init.d/unbound restart
Stopping Unbound DNS Proxy... [ OK ]
Starting Unbound DNS Proxy... [ OK ]
Ignoring broken upstream name server(s): 12.12.17.71 [ WARN ]
Configuring upstream name server(s): 12.127.16.67 [ OK ]

It seems like somehow whatever name server is named second in the configuration, unbound thinks' it's invalid.

In /etc/resolv.conf I don't see anything except:

search <localdomain>
nameserver 127.0.0.1

Where else can I look for the invalid configuration of unbound?

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 30th, 2019, 5:08 pm

To test this, try specifying Google DNS (8.8.8.8 and 8.8.4.4), try restarting, and and see what messages you get at the command line. If everything works as it should, your issues are with the upline DNS providers. It's NOT uncommon to have one work and another not even with the SAME DNS provider. If you still get warning messages, something is wrong with your IPFire software or setup.

Best regards,
Fred

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 30th, 2019, 5:27 pm

I tried it, then restarted unbound and it's the same result:

[root@ipfire log]# /etc/init.d/unbound restart
Stopping Unbound DNS Proxy... [ OK ]
Starting Unbound DNS Proxy... [ OK ]
Ignoring broken upstream name server(s): 4.4.4.4 [ WARN ]
Configuring upstream name server(s): 8.8.8.8 [ OK ]
[root@ipfire log]#


This is a production ipfire server for a business that's hosting 60 desktops, running 24/5 so I can't just pop in a new ipfire server or re-install easily. I'd like to fix the problem by fixing the config file, any idea on where I look?

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 30th, 2019, 5:49 pm

You should look in the unbound Log files for hints. They should be able to tell you what is failing.

Best regards,
Fred

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 31st, 2019, 2:05 pm

I have found nothing in /var/log/messages unbound related that seems to tie in a correlation of what is happening. In /var/log/squid/access.log when the proxy is on I can see entries like this when I try to run yum update:

1559246983.310 0 10.5.1.104 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -
1559246983.310 0 10.5.1.104 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -
1559246983.310 0 10.5.1.104 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -
1559246983.310 0 10.5.1.104 NONE/000 0 NONE error:transaction-end-before-headers - HIER_NONE/- -
1559246989.311 10876 10.5.1.18 TCP_REFRESH_ABORTED/000 0 GET http://mirrorlist.centos.org/?release=7 ... nfra=stock - ORIGINAL_DST/212.69.166.138 -

cwensink
Posts: 25
Joined: August 7th, 2018, 2:18 pm

Re: yum update times out after upgrade to 131

Post by cwensink » May 31st, 2019, 3:53 pm

With the proxy line commented out, but with it on and in transparent mode I see this in access.log of squid:

[root@ipfire squid]# tail -f access.log | grep 10.5.1.18
1559317786.805 30001 10.5.1.18 TCP_MISS_ABORTED/000 0 GET http://mirrorlist.centos.org/?release=7 ... nfra=stock - ORIGINAL_DST/85.236.43.108 -
1559317848.393 30000 10.5.1.18 TCP_MISS_ABORTED/000 0 GET http://mirrorlist.centos.org/?release=7 ... nfra=stock - ORIGINAL_DST/85.236.43.108 -

This is the output of the command line from the server requesting yum updates.

[root@moodle /]# yum update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7 ... nfra=stock error was
12: Timeout on http://mirrorlist.centos.org/?release=7 ... nfra=stock: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')


One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:

1. Contact the upstream for the repository and get them to fix the problem.

2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).

3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...

4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:

yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>

5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:

yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/7/x86_64

fkienker
Posts: 126
Joined: March 3rd, 2011, 4:59 pm

Re: yum update times out after upgrade to 131

Post by fkienker » May 31st, 2019, 4:20 pm

The best approach at this point to start with is to get the proxy COMPLETELY out of the loop. Disable it completely and see what happens.
If you have Geo Filtering turned on, turn it off.

If yum can't access a mirror with the proxy disabled, then your problem is with DNS, some basic network configuration, an iptables rule, or something outside of your network is blocking it.

Turn off the URL filter and the Update Accelerator. Turn on the proxy and enable transparent mode if you want. if you now can't access yum your problem is with the basic proxy setup.

Turn on the URL filter. If yum stops working then some blacklist is stopping it.

Turn on the Update Accelerator. If yum stops working you will have to reset it and try again.

After this I'm out of ideas. I know for CERTAIN, with a properly configured IPFire system it is possible access CentOS and Redhat mirror via yum.

Best regards,
Fred

Post Reply