Poor network bandwidth on VM (failover cluster)

[scottalanmiller]

So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

[LAH3385]

@scottalanmiller said:

So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

[Dashrender]

When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

[Dashrender]

@LAH3385 said:

@scottalanmiller said:

So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

In the case of RDP - what do the system resources utilization look like?

[LAH3385]

@Dashrender said:

When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

How to check if the bandwidth is saturated on cross over? It does not seem to use much at all.

@Dashrender said:

@LAH3385 said:

@scottalanmiller said:

So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

In the case of RDP - what do the system resources utilization look like?

resources utilization on the my PC, hypervisor, or VM?

[Dashrender]

@LAH3385 said:

@Dashrender said:

In the case of RDP - what do the system resources utilization look like?

resources utilization on the my PC, hypervisor, or VM?

That would depend, are you RDPing into the hypervisor or VM? in either case the hypervisor could be at fault. I'd start by looking at the hypervisor. I know in VMWare you can see the resource utilization of all on a chart, so you should be able to see everything you need from within the hypervisor I would guess.

[scottalanmiller]

@LAH3385 said:

@scottalanmiller said:

So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

Could be. Does the image update quickly?

[marcinozga]

@LAH3385 said:

@Dashrender said:

When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

How to check if the bandwidth is saturated on cross over? It does not seem to use much at all.

Starwind console should have graphs available for all kinds of resources utilisation. Btw, you don't need crossover cable on 1Gbit and faster ethernet cards.

[LAH3385]

@Dashrender said:

@LAH3385 said:

@Dashrender said:

In the case of RDP - what do the system resources utilization look like?

resources utilization on the my PC, hypervisor, or VM?

That would depend, are you RDPing into the hypervisor or VM? in either case the hypervisor could be at fault. I'd start by looking at the hypervisor. I know in VMWare you can see the resource utilization of all on a chart, so you should be able to see everything you need from within the hypervisor I would guess.

I just gave the VM more virtual processor from 1 to 4. It does seem more responsive while RDP. But the progress bar when saving still there. The graph on Performance (Task Manager) does not seem to be indicating anything out of ordinary. The graph goes up and down like a heart beat on cross over ethernet and wobbling on internet.

@scottalanmiller said:

@LAH3385 said:

@scottalanmiller said:

So when you say saving the file is slow... this is when you are saving the file locally on the server or remotely from a workstation?

2 scenarios. 1st I noticed that opening files, saving, deleting, moving seem laggy. 2nd is when I remote into the VM via RDC it seem very laggy and delay. The mouse seem fine but clicking anything seem delayed. This is how I came to the thought of poor network bandwidth

Could be. Does the image update quickly?

How do I check? It should be within couple of seconds.. or every second.

@marcinozga said:

@LAH3385 said:

@Dashrender said:

When a file is written to your vSAN, the process doing the writing most likely won't reply that the process is completed until all nodes in the vSAN report that the file has been written.

If there is a bottleneck on the communication between the vSAN servers, this could introduce your delay.

I believe you said you are using x-over cables between the servers for the vSAN, so you likely don't have a switch related problem there. But you could still have bandwidth/latency issues there.

How to check if the bandwidth is saturated on cross over? It does not seem to use much at all.

Starwind console should have graphs available for all kinds of resources utilisation. Btw, you don't need crossover cable on 1Gbit and faster ethernet cards.

I couldn't find where the graph would be located at. But this is the setting on Synchronization priority

[marcinozga]

What antivirus (if any) is running on that server? What happens if you disable it?

[LAH3385]

@marcinozga said:

What antivirus (if any) is running on that server? What happens if you disable it?

No AntiVirus. If any.. it would be w/e built-in into Windows server 2K12 R2

[dafyre]

You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

[LAH3385]

@dafyre said:

You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

I moved it to 9/10 client access. Very little to no different.

[dafyre]

@LAH3385 said:

@dafyre said:

You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

I moved it to 9/10 client access. Very little to no different.

Might be a wise thing to set it back closer to the defaults.

Look at the Perfmon Counters for Disk Read / Writes and Queue Length for both of your servers that are running Starwind?

[LAH3385]

@dafyre said:

@LAH3385 said:

@dafyre said:

You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

I moved it to 9/10 client access. Very little to no different.

Might be a wise thing to set it back closer to the defaults.

Look at the Perfmon Counters for Disk Read / Writes and Queue Length for both of your servers that are running Starwind?

What am I looking for? How long should I run the test?

[Dashrender]

set the counters, then move some files around, then take a screen shot and post.

[LAH3385]

@Dashrender said:

set the counters, then move some files around, then take a screen shot and post.

Idle 1min

File Transfer 11 minutes

[LAH3385]

@LAH3385

These are some tests I conduct.
visor1
visor2 (the host where VM resides at the moment)
VM
File size 1.8GB (contain 450 files of 4MB each)

visor1 to visor2

visor2 to visor1

visor1 to VM

visor2 to VM

Connection from my PC to visor1/2 or VM is around 6MB/s to 11MB/s (average around 7MB/s)

[dafyre]

@LAH3385 said:

@dafyre said:

@LAH3385 said:

@dafyre said:

You could also try to move that synchronization slider a few notches towards the middle. That should give you a balance of sync and client access speed. It looks like you have it set to just focus on syncing. This could likely be what is hurting you.

I moved it to 9/10 client access. Very little to no different.

Might be a wise thing to set it back closer to the defaults.

Look at the Perfmon Counters for Disk Read / Writes and Queue Length for both of your servers that are running Starwind?

What am I looking for? How long should I run the test?

I saw you post your permon screen... Look under the disk performance tabs and see what you are getting?

[LAH3385]

@dafyre

Weird thing is only hypervisor1 able to view report. visor2 and VM return with error

I really do not think my disk is the bottle neck here.
I did found a setting on GPO [Network/Background Intelligent Transfer Service (BITS)] It should not have impact on Foreground transfer.