pfSense stopped re-sending packets mysteriously...
Translation of the original in Catalan. Read it in: CatalàCastellano
I will try to explain an 'X-File' of the kind I have already solved, but of which I do not have a 100% explanation of the behaviour pfSense had with the problem. The conclusion is that pfSense was dropping packets because it had too many concurrent connections, but since they were UDP connections they were hard to identify. The problem is that in a completely mysterious way pfSense was dropping the first X packets towards any IP. This indeterminate number could range from 2, 3 or 4 packets up to 100 or 200. That is, there were moments when it gave the impression that the link had been lost. As a peculiarity of the firewall (pfSense) configuration, let me mention that the WAN card was in bridge mode with the internet servers card. That is, the DMZ receives the traffic from the public IPs that are there, filtered through a bridge of these two network cards I mentioned.
An example of the packet loss I was talking about. A ping made to an internet IP (the one of oriol.joor.net) from the firewall itself. Really that operation not permitted left me very, very puzzled.
# ping 80.35.31.228
PING 80.35.31.228 (80.35.31.228): 56 data bytes
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
ping: sendto: Operation not permitted
64 bytes from 80.35.31.228: icmp_seq=9 ttl=57 time=117.668 ms
64 bytes from 80.35.31.228: icmp_seq=10 ttl=57 time=86.728 ms
From any of the other legs of the firewall the same effect was suffered, even if the pings were made towards the LAN. It took me almost 48h to find the solution, it was madness. Since in the end, to locate the problem, we isolated the firewall from the internet and the phenomenon stopped happening. That is, that operation not permitted thing never happened again no matter how much traffic we injected into the firewall, it did not happen. So, thanks to a very good idea from Manu we closed all traffic from the internet and we started opening services towards the DMZ, one by one. Until I discovered that the NTP service (123/upd) was the one causing this strange effect on pfSense.
After several tests the conclusion is that there were too many concurrent connections against the service I mention. So, right now what I have done is filter the number of connections I allow against that port:
pfsense.png
You can find these options by editing the Rule, specifically in the section where it says Advaced Options. As you can see you can leave very well limited which connections and which ones you do not let through to a service.
Perhaps the most important thing I have learned from this story is never to set up a firewall that does not comply with the golden rule: by default block all. That is, always close everything and you will open up as needed. But sometimes, out of a wish to be a bit more permissive, we do it the other way round and in the end that ends up costing us, as in this case. It will not happen to me again, I hope it will not happen to you either.