Network Share on File Server Cluster being Interrupted.

When I perform heavy disk operations, like deleting 10k files at a time, the network share becomes unresponsive and won't serve out files for a short time.

Here's my configuration. I have a failover file server cluster composed of two Windows 2008 R2 Enterprise servers. Each server is a VM running on top of two independent Dell Poweredge servers running Windows Hyper-V. Both of the Dell servers have dedicated NICs to a Dell MD3000i SAN. Each of the file server VMs route their iSCSI connections through this dedicated NIC for their connection to the volume on the SAN where the files reside.

If I run a batch file that performs 10k deletes from a remote machine that references the file by share name (ie. \\fileserver\sharename\folder\filename.jpg), it may do 1,000 or 8,000 deletes before the share gives out. It's random each time. Ironically, the batch file will continue deleting the files, but other servers accessing files on that same share will get held up. The files I'm deleting would not be accessed by other servers, so locking of those specific files is not an issue.

If I run the same batch file on the master server of the file cluster and reference the files by their local path (ie. x:\folder\filename.jpg), the share cuts out immediately and the other servers sit and wait. Access to that share will resume when I terminate the batch file running.

Anyone have an idea as to the cause of the share cutting out or what I could do to diagnose this issue further? Any suggestions are greatly appreciated.

---

Updated Note: I've isolated this problem to occurring only within the boundaries of the host box. None of the network traffic involved to replicate this problem with the VMs reaches the physical switch the host box connects to other than the iSCSI connection to the SAN. The iSCSI connection has it's own dedicated switch and private subnet to the SAN outside of standard network traffic.

From serverfault Adam Winter

• Good lord. Is there anything in the event viewer to indicate the OS is seeing some sort of resource depletion? Can you inspect with perfmon?

  From mfinni

• This screams resource depletion of some kind. If this were a Linux host I'd be thinking, "this sounds like a boat load of IO-Wait." Check OS level performance monitors like mfinni pointed out. You have two areas that could be bottle-necking, and that's logical/physical disk performance, and network performance on the iSCSI network connection. PerfMon can give you this. I don't know HyperV at all, but if it is anything like VMWare then you have some performance metrics on the Hypervisor side you can look into as well. Do so.

  As a theory, my guess is that the very high level of metadata updates you're doing is causing some inherent latency in your iSCSI stack to magnify. This in turn crowds out other I/O or metadata requests, which results in the symptoms you describe, other processes can get a word in edgewise as the MFT blocks are being hammered by this other process. iSCSI itself can cause this, but the VM layer is probably adding its own internal delays. If this is indeed the problem, you might want to consider presenting the iSCSI LUN to the hypervisor instead and then presenting the resulting disk to the VM; that way you're not relying on a virtualized network adapter for iSCSI, you're relying on a physical one.

  Edit: It seems that you probably have this kind of fault on your hands. The PerfMon counters I'd pay attention to are "Bytes Sent/sec" and "Packets Sent/sec" for the interface running the iSCSI connection. The combination of the two should give you your average packet SIZE. (alternately, if you have the ability, throw a sniffer into the loop and see what the packets look like at the network switch. This is the more reliable method if you can do it) If that packet size is pretty small (say, under 800 bytes) then there is not much you can do about this other than get down to the TCP level and see what kind of optimizations can be made between your cluster nodes and the iSCSI target. Server 2008 is picky with its TCP settings, so there may be gains to be made here.

  Adam Winter : What you're saying makes sense and has credibility. Any suggestions on how I can prove it? What would I monitor in perfmon to validate this theory? I can't move the iSCSI LUNs to the host box because this is a cluster. If the host box goes down, so do the LUNs, so I have to keep them attached to the VM.

  Adam Winter : Large files copies do not suffer the same problem, but doing a ton of small file deletes locally on the master server of the file cluster kills the share to other servers. If the problem were with the iSCSI going through the virtual switches of Hyper-V, why would it not suffer with big data writes as it does with small requests? I agree that some type of requests are queuing up somewhere, and I need a means of finding out.

  sysadmin1138 : If large-file options don't do it but lots of itty bitty files do, then it is almost definitely meta-data related in some way. What's doing it is probably lots of little data operations on the same few blocks of the file-system. This is the kind of thing that write-combining in RAID cards is designed to help with, but in this case you've got a network stack between you and your RAID cache, and this kind of operation is HIGHLY sensitive to network latency.

  tony roth : do you have the nic's set to auto or manual, how about the switch ports? They need to match.

  Adam Winter : Just had another thought. We setup this file server cluster because we wanted to expand our web hosting environment from 1 server to multiple. The volume on the SAN which the file cluster points to used to be connected to a physical machine using iSCSI running Windows Web 2008. At that time, file ops to this volume were super fast. When we expanded, we mounted this volume to the file cluster running as VMs so that the data would be highly available. Could the move to a VM negatively affect the iSCSI connections? That would imply the virtual switch within Hyper-V is the main culprit.

  sysadmin1138 : The virtual switch in Hyper-V is my top suspect right now. The kind of op you're doing will seriously magnify even small slowdowns in there.

  Adam Winter : One more thought on this subject. Before moving this volume from the physical machine as described above, we converted that physical machine to a virtual machine on the same host as the file cluster node. The iSCSI connection on the web server VM still performed great with the batch file. What would have caused performance to drop when moving that volume from the web server VM to the file server VM on the same host, using the same NICs and virtual switch on the host? Something to do with the cluster service?
  From sysadmin1138