some valid domains getting blocked

General questions.
axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » September 24th, 2019, 11:31 pm

JonM wrote:
September 24th, 2019, 11:24 pm
jmpentney wrote:
September 24th, 2019, 10:29 pm
I've tried changing my NTP time server settings and update frequency, without success.
NTP always seemed "off" to me. To fix drift issues I added the servers to my ntp.conf:

Code: Select all

[root@ipfire ~]# cat /etc/ntp.conf
disable monitor
restrict default kod nomodify notrap nopeer
restrict 127.0.0.1
server  0.us.pool.ntp.org prefer
server  1.us.pool.ntp.org
server  127.127.1.0
fudge   127.127.1.0 stratum 10
driftfile /etc/ntp/drift
[root@ipfire ~]# 
Now all works as NTP should. Instead of the time being off by 12 to 20 seconds a day, now its less then 0.1 seconds. I was going to suggest this to @axel2078 since there are DNS (and pakfire) things that seem to be very time dependent.

see viewtopic.php?f=27&t=21861
I’m currently running OPNsense but if this works for jmpentney, I’d be glad to switch back and give it a shot.
Image

Alorotom
Posts: 429
Joined: March 30th, 2015, 6:56 am

Re: some valid domains getting blocked

Post by Alorotom » September 28th, 2019, 2:09 pm

Hello axel2078,

today I experienced again a problem to resolve and load a website here. www.bremeneins.de could not be loaded with 'page not found'. Also ping did not work. 'Host not known'. I've experienced this several times now, also with ipfire.org, and usually I can help along with 'nslookup <domain> 8.8.8.8'. However this triggers something, I can load the site then. Just to ad this info to your thread.

Regards
Alorotom
Image
Image

axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » September 30th, 2019, 3:32 am

Alorotom wrote:
September 28th, 2019, 2:09 pm
Hello axel2078,

today I experienced again a problem to resolve and load a website here. www.bremeneins.de could not be loaded with 'page not found'. Also ping did not work. 'Host not known'. I've experienced this several times now, also with ipfire.org, and usually I can help along with 'nslookup <domain> 8.8.8.8'. However this triggers something, I can load the site then. Just to ad this info to your thread.

Regards
Alorotom
So if I'm understanding you correctly, this means you also have the problem where some websites do not load in a web browser when you are on a system behind IPfire....is that correct? It sounds like you see the same DNS errors that I do with some websites, but yet when you specify the DNS server, things seem to work, is that correct? I've noticed oddities myself...like the fact that my system was pointed to 9.9.9.9 for DNS resolution and sometimes when I would ping a website, I'd get an error about no record found. However, if I ran nslookup and then specified the DNS server to query (9.9.9.9) the same lookup worked fine and I got a ping reply back, but in my case I still couldn't get the page to load in a browser.

It sounds to me like this problem is affecting others too. There are at least 2 others in this thread now that have run into the same issue, although it was for different websites. Is there no fix for this?
Image

Alorotom
Posts: 429
Joined: March 30th, 2015, 6:56 am

Re: some valid domains getting blocked

Post by Alorotom » September 30th, 2019, 4:02 pm

Hello axel2078,
So if I'm understanding you correctly, this means you also have the problem where some websites do not load in a web browser when you are on a system behind IPfire....is that correct? It sounds like you see the same DNS errors that I do with some websites, but yet when you specify the DNS server, things seem to work, is that correct? I've noticed oddities myself...like the fact that my system was pointed to 9.9.9.9 for DNS resolution and sometimes when I would ping a website, I'd get an error about no record found. However, if I ran nslookup and then specified the DNS server to query (9.9.9.9) the same lookup worked fine and I got a ping reply back, but in my case I still couldn't get the page to load in a browser.
Yes.

Sometimes reload in the browser fixes it, or 2 times or 3 times. If not, nslookup ... The occurence is intermittent. Some sites more often, most sites never, a few rarely. Hard to reproduce.

In Jan / Feb I had problems with forum.ipfire.org: viewtopic.php?f=22&t=22245 which turned out to result from attacks on the DNS infrastrukture on one hand and for my part from a provider-change where I missed to change the configured DNS in one place. Quite tricky.

I experienced also problems with alias domains. No one ever answered on that.

Another one. Finally reason was his blacklisted IP.

One more. We had Telekom DNS in common.

These ones are from my list of user posts only. If you query the board, you'll surely find more. There are not much but a significant number of users that report this. All cases that I remember are circling around DNS / unbound somehow.
Is there no fix for this?
As there is no known reason in your case, there can't be a fix yet. All these continuing problems seem to have in common that they occour only behind IPFire and not behind another firewal. Unfortunately I've no idea for an approach to dig deeper into this. Maybe someone else can take over and use your good reproduceable case to unravel this mystery.

Regards
Alorotom
Image
Image

jmpentney
Posts: 4
Joined: September 24th, 2019, 8:14 am

Re: some valid domains getting blocked

Post by jmpentney » October 1st, 2019, 4:40 pm

I am still experiencing this problem. I've tried adjusting my NTP settings as per JonM's suggestion from 25th September, but it has not made any difference.

I have also found that I can't access Pakfire to check for updates - the log report is:

17:35:30 pakfire: PAKFIRE INFO: IPFire Pakfire 2.23 started!
17:35:30 pakfire: MIRROR INFO: server-list.db is 1847921 seconds old. - DEBUG: force
17:35:30 pakfire: DOWNLOAD STARTED: 2.23/lists/server-list.db
17:35:30 pakfire: DOWNLOAD INFO: Host: pakfire.ipfire.org (HTTP) - File: 2.23/lists/server-list.d b
17:35:30 pakfire: DOWNLOAD INFO: 2.23/lists/server-list.db has size of bytes
17:35:30 pakfire: DOWNLOAD INFO: HTTP-Status-Code: 500 - 500 Can't connect to pakfire.ipfire.org: 80 (Bad hostname 'pakfire.ipfire.org')
17:35:30 pakfire: Giving up: There was no chance to get the file 2.23/lists/server-list.db from a ny available server. There was an error on the way. Please fix it.
17:35:30 pakfire: DB INFO: packages_list.db is 1847921 seconds old. - DEBUG: force
17:35:30 pakfire: DOWNLOAD STARTED: lists/packages_list.db
17:35:30 pakfire: MIRROR INFO: 20 servers found in list
17:35:30 pakfire: DOWNLOAD INFO: Host: ftp.gwdg.de (HTTPS) - File: pub/linux/ipfire/pakfire2/2.23 /lists/packages_list.db
17:35:30 pakfire: DOWNLOAD INFO: pub/linux/ipfire/pakfire2/2.23/lists/packages_list.db has size o f bytes
17:35:30 pakfire: DOWNLOAD INFO: HTTP-Status-Code: 500 - 500 Can't connect to ftp.gwdg.de:443 (Ba d hostname 'ftp.gwdg.de')
17:35:30 pakfire: Giving up: There was no chance to get the file lists/packages_list.db from any available server. There was an error on the way. Please fix it.
17:35:30 pakfire: CORE INFO: core-list.db is 1847918 seconds old. - DEBUG: force
17:35:30 pakfire: DOWNLOAD STARTED: lists/core-list.db
17:35:30 pakfire: MIRROR INFO: 20 servers found in list
17:35:30 pakfire: DOWNLOAD INFO: Host: www.mirrorservice.org (HTTPS) - File: sites/downloads.ipfi re.org/pakfire2/2.23/lists/core-list.db
17:35:30 pakfire: DOWNLOAD INFO: sites/downloads.ipfire.org/pakfire2/2.23/lists/core-list.db has size of bytes
17:35:31 pakfire: DOWNLOAD INFO: HTTP-Status-Code: 500 - 500 Can't connect to www.mirrorservice.o rg:443 (Bad hostname 'www.mirrorservice.org')
17:35:31 pakfire: Giving up: There was no chance to get the file lists/core-list.db from any available server. There was an error on the way. Please fix it.
17:35:31 pakfire: PAKFIRE INFO: Pakfire has finished. Closing.

Again this is a recent problem. I had been running core 131 without any problem, then did an update (in essentially 1 step) to core 135 and have been having trouble ever since.

Any ideas about how to fix it?

Thanks!

axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » October 2nd, 2019, 12:15 am

Alorotom wrote:
September 30th, 2019, 4:02 pm

As there is no known reason in your case, there can't be a fix yet. All these continuing problems seem to have in common that they occour only behind IPFire and not behind another firewal. Unfortunately I've no idea for an approach to dig deeper into this. Maybe someone else can take over and use your good reproduceable case to unravel this mystery.

Regards
Alorotom
Actually, I switched to OPNsense after running into this problem on IPfire and the problem still persisted. Here's something else that's odd. As of last night, all of the *.webster.edu domains that I had problems with previously load fine in all browsers in all computers on my network. Why this is, I'm not sure. i haven't made any modifications to OPNsense in several days, nor have I had time. I suspect this will be a temporary thing and it will eventually stop working, but we'll see.
Image

axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » October 2nd, 2019, 12:20 am

jmpentney wrote:
October 1st, 2019, 4:40 pm
I am still experiencing this problem. I've tried adjusting my NTP settings as per JonM's suggestion from 25th September, but it has not made any difference.

I have also found that I can't access Pakfire to check for updates - the log report is:

17:35:30 pakfire: PAKFIRE INFO: IPFire Pakfire 2.23 started!
17:35:30 pakfire: MIRROR INFO: server-list.db is 1847921 seconds old. - DEBUG: force
17:35:30 pakfire: DOWNLOAD STARTED: 2.23/lists/server-list.db
17:35:30 pakfire: DOWNLOAD INFO: Host: pakfire.ipfire.org (HTTP) - File: 2.23/lists/server-list.d b
17:35:30 pakfire: DOWNLOAD INFO: 2.23/lists/server-list.db has size of bytes
17:35:30 pakfire: DOWNLOAD INFO: HTTP-Status-Code: 500 - 500 Can't connect to pakfire.ipfire.org: 80 (Bad hostname 'pakfire.ipfire.org')
17:35:30 pakfire: Giving up: There was no chance to get the file 2.23/lists/server-list.db from a ny available server. There was an error on the way. Please fix it.
17:35:30 pakfire: DB INFO: packages_list.db is 1847921 seconds old. - DEBUG: force
17:35:30 pakfire: DOWNLOAD STARTED: lists/packages_list.db
17:35:30 pakfire: MIRROR INFO: 20 servers found in list
17:35:30 pakfire: DOWNLOAD INFO: Host: ftp.gwdg.de (HTTPS) - File: pub/linux/ipfire/pakfire2/2.23 /lists/packages_list.db
17:35:30 pakfire: DOWNLOAD INFO: pub/linux/ipfire/pakfire2/2.23/lists/packages_list.db has size o f bytes
17:35:30 pakfire: DOWNLOAD INFO: HTTP-Status-Code: 500 - 500 Can't connect to ftp.gwdg.de:443 (Ba d hostname 'ftp.gwdg.de')
17:35:30 pakfire: Giving up: There was no chance to get the file lists/packages_list.db from any available server. There was an error on the way. Please fix it.
17:35:30 pakfire: CORE INFO: core-list.db is 1847918 seconds old. - DEBUG: force
17:35:30 pakfire: DOWNLOAD STARTED: lists/core-list.db
17:35:30 pakfire: MIRROR INFO: 20 servers found in list
17:35:30 pakfire: DOWNLOAD INFO: Host: www.mirrorservice.org (HTTPS) - File: sites/downloads.ipfi re.org/pakfire2/2.23/lists/core-list.db
17:35:30 pakfire: DOWNLOAD INFO: sites/downloads.ipfire.org/pakfire2/2.23/lists/core-list.db has size of bytes
17:35:31 pakfire: DOWNLOAD INFO: HTTP-Status-Code: 500 - 500 Can't connect to www.mirrorservice.o rg:443 (Bad hostname 'www.mirrorservice.org')
17:35:31 pakfire: Giving up: There was no chance to get the file lists/core-list.db from any available server. There was an error on the way. Please fix it.
17:35:31 pakfire: PAKFIRE INFO: Pakfire has finished. Closing.

Again this is a recent problem. I had been running core 131 without any problem, then did an update (in essentially 1 step) to core 135 and have been having trouble ever since.

Any ideas about how to fix it?

Thanks!
I had been running IPfire since 2013 without a single hiccup until I upgraded to Core 134. That's when I started having random web page issues. It seems to me like something changed between the different builds, but I haven't been able to nail it down.
Image

User avatar
Deepcuts
Posts: 461
Joined: March 1st, 2016, 3:18 pm
Location: Romania

Re: some valid domains getting blocked

Post by Deepcuts » October 2nd, 2019, 6:15 am

I have been having some similar issues lately.
Some website just did not load. Could not connect to various services over UDP.
In the end, it seems it is my Aquantia AQC107 network card or its drivers.
The only fix was to disable "tcp/udp checksum offload" for IPv4

I have no clue what network card you have or if it applies to you, but worth a shot.
Image
Image

jmpentney
Posts: 4
Joined: September 24th, 2019, 8:14 am

Re: some valid domains getting blocked

Post by jmpentney » October 2nd, 2019, 8:30 am

I don't recall off the top of my head what network cards I have in my ipfire box, but adjusting the checksum offload settings had no effect.

I've noticed that if my browser is connected to a website before I re-enable the transparent proxy and web filtering, it will continue to load new pages from that site without any problems. But I can't establish a new connection with a different web browser - the connection times out. The ipfire error message lists the actual ip address of the site I am trying to reach, so it does not seem to be a DNS problem. I'm running out of ideas about what might fix this issue, and I also still have the problem of not being able to check for updates with pakfire...

I think I'll have to either go back to core 133 or switch to pfsense/opnsense.

axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » October 9th, 2019, 1:57 am

A couple weeks ago, I decided to completely reinstall OPNsense to give me a fresh start and see if it fixed the problem. It was a fresh, vanilla install and I still could not get to any of the *webster.edu websites. Sometimes they would respond to pings and sometimes they wouldn't. Sometimes I got A records returned via nslookup and other times I got a timeout or no record found. I was stumped. Since I'm a glutton for punishment, I decided to give pfSense a try. I did a fresh, new install of pfSense and still, I could not get any of the *webster.edu websites to load in a browser (without a VPN). I was exhausted and extremely frustrated, so I just gave up on it and left it alone, accepting that I will just have to use a VPN client to get to some websites. Well, a couple days later, everything started working again. I hadn't made any changes, but I was able to access all of the *webster.edu websites that I couldn't get to before. It's been about 2 1/2 weeks now and I'm still not having any issues getting to any websites. I still don't know why this happened or why it seemed to just go away, but I had the same problem with 3 different firewalls. For now, I think I'm going to stick with pfSense...not because I don't like IPfire, but because I want to try something new for a while. Since all three of my firewalls are virtualized, I can switch back and forth as needed relatively quickly.

Incidentally, there were a couple folks in the OPNsense forum that had the same problem I did. A co-worker of mine runs pfSense at home and I had told him all about the problems I've been having. A few days ago, he noticed that he couldn't get to a payment processing website anymore...unless he turned on his VPN client. Weird.
Image

Alorotom
Posts: 429
Joined: March 30th, 2015, 6:56 am

Re: some valid domains getting blocked

Post by Alorotom » October 9th, 2019, 10:19 am

So it's after all a problem with blocked IPs or IP-Ranges? Nothing that the involved firewalls are accountable for. But how to dertermine that?
Image
Image

axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » October 10th, 2019, 3:34 am

Alorotom wrote:
October 9th, 2019, 10:19 am
So it's after all a problem with blocked IPs or IP-Ranges? Nothing that the involved firewalls are accountable for. But how to dertermine that?
That's a very good question. I'm not sure how to determine the answer.
Image

User avatar
Arne.F
Core Developer
Core Developer
Posts: 8516
Joined: May 7th, 2006, 8:57 am
Location: BS <-> NDH
Contact:

Re: some valid domains getting blocked

Post by Arne.F » October 10th, 2019, 5:19 am

Maybee the sites has installed something like fail2ban that blocks the ip after some failed logins for a while to prevent brute force passwords.

If there are some more users behind a firewall this is often a problem.
Arne

Support the project on the donation!

Image

Image

Image
PS: I will not answer support questions via email and ignore IPFire related messages on my non IPFire.org mail addresses.

gpatel-fr
Posts: 51
Joined: July 24th, 2019, 7:59 am

Re: some valid domains getting blocked

Post by gpatel-fr » October 26th, 2019, 2:07 pm

Sorry to revive this old thread. In fact I think it should stay buried since the subject is so vague and I suspect that some people posting 'I have the same problem' may not have the same problem, since it's in fact a very generic problem. Even the OP seemed to have different problems in different posts. But I did just stumble on this astonishing post (on a site from a very well-known competent and serious techie)

https://rachelbythebay.com/w/2016/03/27/wonky/

what is said is that basically a Vmware OS was caught spoofing IP connections. Yes it's IPv6 connections while for Ipfire it's IPv4 only - but, with a closed source product like Vmware, one can never be sure of what is done or not in a given situation. So unless you are a Vmware wizard and know exactly how this kind of situation arises and how to turn off for sure any kind of IP connection spoofing, I think it's best to stay away of Vmware for firewalls. Even it has been 'rock solid for years' like the OP was saying. With TCP spoofing, weird stuff can happen at any time depending on network delays or software bugs. It's just bad.

axel2078
Posts: 294
Joined: January 30th, 2013, 3:53 am
Location: IL, USA

Re: some valid domains getting blocked

Post by axel2078 » October 29th, 2019, 2:32 am

That article is in regard to VMware Fusion, not ESXi. One is a layer 2 hypervisor (hosted on MacOS) and the other is layer 1 running on bare metal. It adds another layer of complexity when you are running a layer 2 hypervisor because then you have the quirks of the host OS to deal with, but I'm running a type 1. Plus, the inner workings of the products are different. I’m still not sure what caused the issue, but I haven’t had any problems for several weeks now.
Image

Post Reply