Network Throughput Issue - Packet Loss

[Damin]

I have a 2821 router connected to a 100Mb Ethernet WAN Circuit . The far end of my Ethernet is a 7600.

On both ends I had a QoS policy prioritising a couple of apps. I noticed a small amount of packet loss, 3 pings every 1000 or so. Having removed the QoS at both ends it made no difference through the day, however, on an evening the circuit seems to run clear. In peak business hrs the 2821 shows upto approx 4000- 5000pps rx & tx on it's WAN port (no more than 127/255). The processor shows no more than 10% utilisation. I can't figure out why we may be getting packet loss.We called the carrier and they removed their rate limiting on the circuit, they see no interface errors, drops etc on their switches connected to our routers. Our router likewise show no errors on the WAN interface between us and the NTU's at either end. The last mile is provided by another carrier. They apparently provide a 100mb fibre to our carrier and have no active kit in the middle. Our carrier was the one rate limiting.

I am stumped as to what this could be. Is it feasible that although the 2800 shows low utilisation, the processor / box could still be dropping packets before making it to the output queue. Furthermore, that these 'processor drops' would not record at the interface level ?

[Marco-D]

if you are not seeing any packet drops on your side (via a show interface) then your end isn't dropping the packets the carrier is dropping packets, but this still could be your problem.You need to implement a traffic-shaping policy to shape the traffic to less than what the carrier isproviding, i.e. if you connect at 100MB/s but it is really a 30Mb/s circuit, you need to shape it to 29MB/s or less. Even if your 5 minute rate is significantly less than 30MB/s, the 1 second rate can significantly exceed 30MB/s and that traffic will get dropped by the carrier.

I would also check to make sure that the duplex settings on your side match what the carrier is using. If your carrier is set to auto/auto you MUST also use auto/auto, if the carrier has hard-coded full/100, you MUST also configure full/100 or you will have a duplex mismatch and packets can get dropped that don't show up as dropped packets. Verify the duplex with your carrier by having both of you do a "show interface" and verify that the actual duplex settings match on both ends of your circuit.

[DAIJIRO]

if you are not seeing any packet drops on your side (via a show interface) then your end isn't dropping the packets the carrier is dropping packets, but this still could be your problem.You need to implement a traffic-shaping policy to shape the traffic to less than what the carrier isproviding, i.e. if you connect at 100MB/s but it is really a 30Mb/s circuit, you need to shape it to 29MB/s or less. Even if your 5 minute rate is significantly less than 30MB/s, the 1 second rate can significantly exceed 30MB/s and that traffic will get dropped by the carrier.

I would also check to make sure that the duplex settings on your side match what the carrier is using. If your carrier is set to auto/auto you MUST also use auto/auto, if the carrier has hard-coded full/100, you MUST also configure full/100 or you will have a duplex mismatch and packets can get dropped that don't show up as dropped packets. Verify the duplex with your carrier by having both of you do a "show interface" and verify that the actual duplex settings match on both ends of your circuit.

I would disagree with only one point,

Sometimes you have to experiment with autonegotiation. Just because one side is set to "auto" doesn't mean that the other side must be set to auto. Yeah, the Cisco guides say this is kosher, but sometimes reality's a bit different. I can't recall how many times I've ran auto/auto with Cisco gear, only to have it blow up in my face. This situation has been noted more than once on Catalyst-to-IOS router interconnects in particular. Otherwise, I think your thoughts are cool.

[Marco-D]

I would disagree with only one point,

Sometimes you have to experiment with autonegotiation. Just because one side is set to "auto" doesn't mean that the other side must be set to auto. Yeah, the Cisco guides say this is kosher, but sometimes reality's a bit different. I can't recall how many times I've ran auto/auto with Cisco gear, only to have it blow up in my face. This situation has been noted more than once on Catalyst-to-IOS router interconnects in particular. Otherwise, I think your thoughts are cool.

If one side is set to auto-negotiate and the other is set to 100/full you will have a duplex mismatch. The side that is set to auto will attempt to negotiate duplex, the other side which is hard-coded will not participate. The auto side will then set the interface to half duplex since the other side did not respond. You now have a duplex mismatch because one side is configured for full duplex, and the auto side will fall-back to half. This is how the standard is written. There are devices that will not negotiate properly when both sides are set to auto, but this is very rare. The only way to fix this, is to manually configure duplex and speed on both sides.

In a nutshell, both ends must be configured the same, both auto/auto, both 100/full, or both 100/half. No matter how they are set, you should always verify that both ends are set correctly.