Windows 2003 Server Slow Network Throughput

[drew.flint@gmail.com]

Whenever I try to move files between servers through a Windows File
Copy I can never get more than 25% NIC utilization on a Gigabit NIC.
Here are all of the things I've tried to diagnose the problem.
I made sure that name resolution was working properly (both DNS and
WINS). I've tried performing the copy with both the netbios name as
well as the IP address directly.
I've verified the NICs have the most recent driver from the
manufacturers. In this case HP, these are Proliant DLXXX series with
the NC 77XX series gigabit NICs installed. I have verified the card's
link speed and duplex settings match how the switch port is configured
(auto if the switch port is auto, and hard coded if it's beens
specifically configured to something else).
I've tried different network cables, different grades of network
cables (CAT 5e vs CAT 6), and different switch ports. I've even tried
a direct connection between servers with a cross over cable. In the
direct connect scenario I let the cards autonegotiate to 1Gbps full
duplex.
The switch that all the servers are connected to is a Cisco 6509,
with low CPU utilization. That doesn't seem to be the bottleneck.
No matter what I try, I can't break the 25% utilization barrier.
Does anyone have any recommendations for me. Some random thoughts I
have x64 Enterprise Edition R2 servers talking to x86 Standard Edition
servers without the R2 feature set. Is it possible there is something
in R2 that's causing this limitation. I've looked through the various
event logs (system and application) and there just doesn't seem to be
any clues.
I would appreciate any advice you have to offer. Thanks in advance
for your help

Drew Flint

[scorbishley@fsa.com]

On 28 Feb, 13:04, drew.fl...@gmail.com wrote:
> Whenever I try to move filesbetweenserversthrough a WindowsFileCopyI can never get more than 25% NIC utilization on a Gigabit NIC.
> Here are all of the things I've tried to diagnose the problem.
> I made sure that name resolution was working properly (both DNS and
> WINS). I've tried performing thecopywith both the netbios name as
> well as the IP address directly.
> I've verified the NICs have the most recent driver from the
> manufacturers. In this case HP, these are Proliant DLXXX series with
> the NC 77XX series gigabit NICs installed. I have verified the card's
> link speed and duplex settings match how the switch port is configured
> (auto if the switch port is auto, and hard coded if it's beens
> specifically configured to something else).
> I've tried different network cables, different grades of network
> cables (CAT 5e vs CAT 6), and different switch ports. I've even tried
> a direct connectionbetweenserverswith a cross over cable. In the
> direct connect scenario I let the cards autonegotiate to 1Gbps full
> duplex.
> The switch that all theserversare connected to is a Cisco 6509,
> with low CPU utilization. That doesn't seem to be the bottleneck.
> No matter what I try, I can't break the 25% utilization barrier.
> Does anyone have any recommendations for me. Some random thoughts I
> have x64 Enterprise Edition R2serverstalking to x86 Standard Editionserverswithout the R2 feature set. Is it possible there is something
> in R2 that's causing this limitation. I've looked through the various
> event logs (system and application) and there just doesn't seem to be
> any clues.
> I would appreciate any advice you have to offer. Thanks in advance
> for your help
>
> Drew Flint

Drew, can help you at this point but we have the exact same issue, we
are working on it and if we get a fix I'll let you know, if you hear
of anything I would appreciate a heads up.

Cheers.....Si

[drew.flint@gmail.com]

On Mar 6, 6:57 am, scorbish...@fsa.com wrote:
> On 28 Feb, 13:04, drew.fl...@gmail.com wrote:
>
>
>
>
>
> > Whenever I try to move filesbetweenserversthrough a WindowsFileCopyIcan never get more than 25% NIC utilization on a Gigabit NIC.
> > Here are all of the things I've tried to diagnose the problem.
> > I made sure that name resolution was working properly (both DNS and
> > WINS). I've tried performing thecopywith both the netbios name as
> > well as the IP address directly.
> > I've verified the NICs have the most recent driver from the
> > manufacturers. In this case HP, these are Proliant DLXXX series with
> > the NC 77XX series gigabit NICs installed. I have verified the card's
> > link speed and duplex settings match how the switch port is configured
> > (auto if the switch port is auto, and hard coded if it's beens
> > specifically configured to something else).
> > I've tried different network cables, different grades of network
> > cables (CAT 5e vs CAT 6), and different switch ports. I've even tried
> > a direct connectionbetweenserverswith a cross over cable. In the
> > direct connect scenario I let the cards autonegotiate to 1Gbps full
> > duplex.
> > The switch that all theserversare connected to is a Cisco 6509,
> > with low CPU utilization. That doesn't seem to be the bottleneck.
> > No matter what I try, I can't break the 25% utilization barrier.
> > Does anyone have any recommendations for me. Some random thoughts I
> > have x64 Enterprise Edition R2serverstalking to x86 Standard Editionserverswithout the R2 feature set. Is it possible there is something
> > in R2 that's causing this limitation. I've looked through the various
> > event logs (system and application) and there just doesn't seem to be
> > any clues.
> > I would appreciate any advice you have to offer. Thanks in advance
> > for your help
>
> > Drew Flint
>
> Drew, can help you at this point but we have the exact same issue, we
> are working on it and if we get a fix I'll let you know, if you hear
> of anything I would appreciate a heads up.
>
> Cheers.....Si- Hide quoted text -
>
> - Show quoted text -

We've done a lot of investigation into this. It would seem that
equivalent servers with a network card that implements the TCP Offload
Engine, or TOE, to do all of the checksumming, achieve a much higher
throughput on larger files (> 1GB). In order to achieve this
increased throughput you must have a TOE compatible NIC on both sides
of the transmission.
For example with non-TOE NICs we regularly only achieve throughput
of 25 to 30% utilization of the NIC. With TOE-enabled NICs we
regularly achieve 50-65% utilization. This is an amazing difference
for a pretty minimal investment (~$250) to get a TOE enabled card for
each server. If you're moving a lot of large files around your
network, you should definitely consider getting a couple of these
cards to see if it makes a difference for you. If you're moving a lot
of smaller files around I don't know how much it will change your
transmission time as files with a size of < 1GB didn't show much
difference between the two NIC types.
My personal hypothesis is that Microsoft has limited the number of
CPU cycles that the network checksumming is allowed to have in the OS.
For arguments sake let's say it's limited to 5% overall CPU
utilization. That would translate to the ~25% utilization we're seeing
because there is a governor in place that you can't change. As you can
imagine if there wasn't some sort of ceiling in place the server could
easily be overwhelmed by network transfers.
This is what we've found in our environment. Perhaps it will be of
use to you.

Drew

[Andrew L]

We tried all this .................
"To manually disable RSS and TCP Offload, follow these steps:1. Click Start,
click Run, type regedit, and then click OK.
2. Locate the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. Right-click EnableTCPChimney, and then click Modify.
4. In the Value data box, type 0, and then click OK.
5. Right-click EnableRSS, and then click Modify.
6. In the Value data box, type 0, and then click OK.
7. Right-click EnableTCPA, and then click Modify.
8. In the Value data box, type 0, and then click OK.
9. Exit Registry Editor, and then restart the computer.
If you set the value of the EnableTCPChimney registry entry to 1, the TCP
Chimney functionality is enabled. Similarly, if you set the values of the
EnableRSS and EnableTCPA registry entries to 1, the RSS functionality and the
TCPA functionality are enabled."


but This one fixed it for us.........flying network speeds

don't forget to reboot afterwards!

From the registry you can do:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Click Edit, point to New, and then click DWORD Value.

Type DisableTaskOffload as the entry name, and then press ENTER.

Right-click DisableTaskOffload, and then click Modify.

In the Value data box, type a value of 1 , and then click OK.

Disable RSS (Receive side scaling)


1. Click Start, click Run, type regedit , and then click OK.
2. Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. On the Edit menu, point to New, click DWORD Value, and then type
EnableRSS .
4. Double-click EnableRSS, type 0 , and then click OK.


regards

Andrew



> >
> >
> >
> > > Whenever I try to move filesbetweenserversthrough a WindowsFileCopyI can never get more than 25% NIC utilization on a Gigabit NIC.
> > > Here are all of the things I've tried to diagnose the problem.
> > > I made sure that name resolution was working properly (both DNS and
> > > WINS). I've tried performing thecopywith both the netbios name as
> > > well as the IP address directly.
> > > I've verified the NICs have the most recent driver from the
> > > manufacturers. In this case HP, these are Proliant DLXXX series with
> > > the NC 77XX series gigabit NICs installed. I have verified the card's
> > > link speed and duplex settings match how the switch port is configured
> > > (auto if the switch port is auto, and hard coded if it's beens
> > > specifically configured to something else).
> > > I've tried different network cables, different grades of network
> > > cables (CAT 5e vs CAT 6), and different switch ports. I've even tried
> > > a direct connectionbetweenserverswith a cross over cable. In the
> > > direct connect scenario I let the cards autonegotiate to 1Gbps full
> > > duplex.
> > > The switch that all theserversare connected to is a Cisco 6509,
> > > with low CPU utilization. That doesn't seem to be the bottleneck.
> > > No matter what I try, I can't break the 25% utilization barrier.
> > > Does anyone have any recommendations for me. Some random thoughts I
> > > have x64 Enterprise Edition R2serverstalking to x86 Standard Editionserverswithout the R2 feature set. Is it possible there is something
> > > in R2 that's causing this limitation. I've looked through the various
> > > event logs (system and application) and there just doesn't seem to be
> > > any clues.
> > > I would appreciate any advice you have to offer. Thanks in advance
> > > for your help
> >
> > > Drew Flint
> >
> > Drew, can help you at this point but we have the exact same issue, we
> > are working on it and if we get a fix I'll let you know, if you hear
> > of anything I would appreciate a heads up.
> >
> > Cheers.....Si- Hide quoted text -
> >
> > - Show quoted text -
>
> We've done a lot of investigation into this. It would seem that
> equivalent servers with a network card that implements the TCP Offload
> Engine, or TOE, to do all of the checksumming, achieve a much higher
> throughput on larger files (> 1GB). In order to achieve this
> increased throughput you must have a TOE compatible NIC on both sides
> of the transmission.
> For example with non-TOE NICs we regularly only achieve throughput
> of 25 to 30% utilization of the NIC. With TOE-enabled NICs we
> regularly achieve 50-65% utilization. This is an amazing difference
> for a pretty minimal investment (~$250) to get a TOE enabled card for
> each server. If you're moving a lot of large files around your
> network, you should definitely consider getting a couple of these
> cards to see if it makes a difference for you. If you're moving a lot
> of smaller files around I don't know how much it will change your
> transmission time as files with a size of < 1GB didn't show much
> difference between the two NIC types.
> My personal hypothesis is that Microsoft has limited the number of
> CPU cycles that the network checksumming is allowed to have in the OS.
> For arguments sake let's say it's limited to 5% overall CPU
> utilization. That would translate to the ~25% utilization we're seeing
> because there is a governor in place that you can't change. As you can
> imagine if there wasn't some sort of ceiling in place the server could
> easily be overwhelmed by network transfers.
> This is what we've found in our environment. Perhaps it will be of
> use to you.
>
> Drew
>