[SOLVED] UDP Blocking by Server

[benfnjovi]

Is there anything inside Windows Server 2008 R2, other than the firewall, that would cause UDP traffic to not be received by server side application? I have a 30 client environment, all communicating with one server side app. The server side application does not receive traffic from specific IP's. The server OS can be completely restarted, and the blocked client IP's move to other clients. In this deployment, it seems to always be 2 clients that are blocked. I can change the IP address of the affected clients, and UDP communication will then resume(this is a patch that I do not wish to constantly implement upon server OS restart). TCP communications work as they should on affected clients. Its almost as if, on every restart, UDP traffic from specific IP addresses are being blocked by the NIC for some unknown reason, and is cleared on the next restart, and then others are being blocked. I've researched the issue for the last 3 days and cant find anything to help solve the issue.
Server Specifics:
HP Proliant G6 DL380 running Windows Server 2008 R2
Broadcom Netxtreme ii Nics

Server Roles:
Active Directory Directory Domain Services
DNS Server
File Services

 

[bill001g]

The easy test to see if it is the firewall is to disable it for a period of time...the shorter the better but long enough for your to test.

In general the way the firewall rules work is traffic is allowed to return on the same udp path. So the server would send the first packet to the client using the proper ports and then the firewall would leave this open as long is there is some traffic.

Getting a udp packet from a unkown machine tends to be risky and is blocked by many firewalls. This is a common problem for example in VoIP calls where the voice stream is on random ports. VoIP is a special case and firewall will intercept the call setup message and extract out the negotiated ports.

Firewall generally have a log of traffic they block, you might have to adjust the logging level. In worst case you can load wireshark on the server. You will see the data even if the firewall later blocks it.

 

[benfnjovi]

Thats the first thing I tried. Well, I created an inbound rule for the port first. When that didn't work, I turned it off completely. Still no cigar. The fact that it changes affected ip addresses on server restart, leads me to believe this is something else. A buffer maybe... idk.

 

[benfnjovi]

I also want to add that there are no smart switches in the network, so it wouldn't be related to anything in a switch configuration.

 

[bill001g]

This could be a application issue. Network equipment by default blocks nothing. When you have eliminated the firewall on the server you do not have many other things. Can you tell if it is just UDP being blocked or is all traffic being blocked to those ip. Can you ping them. It could be a duplicate IP address or maybe some ARP entry problem. You can look at the arp table with the "arp" command.

 

[benfnjovi]

Its only UDP traffic being blocked. A restart of the server OS, will almost always cause the problem to move to other clients and free up the previously affected clients. Changing the Static IP of those affected clients will then allow the client communication to resume. There are no duplicate IP addresses. Each client IP has been statically set to correspond with the client ID. The ARP entries look correct. A Wireshark capture, shows no udp traffic being received from the affected client IP's, only TCP traffic. As if it never makes it past the server NIC buffer. I'm at a loss.

 

[bill001g]

Capture on the client and see if it is really being sent.

Wireshark hooks at a very low level so it almost has to see it. I guess it could be a driver error. This is where a smart switch with mirror/monitor ability helps when you can't be sure the server is not telling lies.

 

[benfnjovi]

Client captures show that the clients are receiving UDP packets as well as sending them. It definitely stops at the server NIC.

 

[bill001g]

Very strange indeed. A nic card has no concept of ip address or port numbers or TCP/UDP. It pretty much looks for its mac address in destination field of a packet and copies it to the input buffer.

 

[benfnjovi]

I'm sure you know more than I about the subject, but server NIC's offer many advanced features that I am unaware of how to use, so I just wanted to be sure I wasn't missing something there due to my lack of education on those features. I know that it handles offload checksums, which is protocol specific, correct? Anyway...

I may not have completely solved the original issue, but I found a solution that seemingly backs up my theory of it being NIC hardware or driver related. I added a pcix Intel NIC. I switched my LAN to it, rebooted, and the problem seems to have completely disappeared. It may not have only been an issue with the Broadcom NIC, but I'm pretty sure either a feature of the NIC that I'm completely uneducated on how to use properly or a flaw, contributed to the problem.

 

[Alabalcho]

Your server OS is pretty old to allow for running of Microsoft' NETMON. This where you can see whether these packets reach the network stack.

I'd also check the server application whether it does not have some capacity limit, let say allow no more than 20 clients at a time.

 

[benfnjovi]

We have the same application running in a 75 machine environment. Both the server side application and client side, are developed by us and isn't limited.
I will look into NETMON.