Hyper-V host connectivity lost but VMs still communicate

[DustinB3403]

The only time I've seen something similar is on XenServer if XAPI is hung for one reason or another. The VM's will still operate, but accessing the management console is down.

[JaredBusch]

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

I see that, but I've never seen a server lose its management interface with team'd nics and not also lose access to the VM's.

This is the second time this has happened, and is now something I need to troubleshoot.

Once I honestly could care less about because it could have just been some freak thing. I did check a few logs then, but nothing in detail, and I did not see anything off hand.

The NICS are standard Dell Broadcoms. The team was made via powershell from the Hyper-V console back when it was all setup.

[JaredBusch]

The Hyper-V dom0 is available locally, but I am not. Hence wanting iDRAC console.

When this happened last time I stepped a user through shutting down the VMs and rebooting the server.

But it was also at 4:30pm, not 9am.

[DustinB3403]

This is the only thing I can find that might be the culprit.

Take a look

[JaredBusch]

This is the workload on the host

1. Old SBS 2008 finally decommisioned 3 months ago, just not deleted
2. Current DC
3. local exchange 2013 /sigh

[JaredBusch]

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

This is the only thing I can find that might be the culprit.

Take a look

I found that one too, but I am not using the broadcom teaming. I am using native Hyper-V teaming.

[DustinB3403]

Just to ask, did you see those event logs on the host at all (I know I know you're using the hyper-v teaming)

[DustinB3403]

So this says at the VM level the intermittent connection issue, but people here have reported it affecting the host as well..

Edit: Update the drivers to the NICs

[JaredBusch]

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

Just to ask, did you see those event logs on the host at all (I know I know you're using the hyper-v teaming)

No there are no even logs at all related to networking. In fact, the host still says it has internet access even though it cannot ping out.

Also this is Server 2012 R2 +GUI + Hyper-V Role. I forgot about this client using this. I am so used to everything I have setup being Hyper-V Server.

The Errors are all related to the Veeam 4am backup crashing.

[DustinB3403]

There is also the recommendation to disable "Virtual Machine Queue" on both the host and the VM's to address this issue.

[JaredBusch]

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

There is also the recommendation to disable "Enable Virtual Machine Queue" on both the host and the VM's to address this issue.

Always and forever disabled...

[JaredBusch]

I just rebooted a CentOS 7 VM on the host and it lost network connectivity, cannot even ping the local gateway.

So it looks like networking is shitting itself, but online VMs are still working.

But it pulled a DHCP address...

[DustinB3403]

I bet if you rebooted the host the issue goes away (at least for a short while) what version drivers are installed for these NICs?

[JaredBusch]

@JaredBusch said in Hyper-V host connectivity lost but VMs still communicate:

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

There is also the recommendation to disable "Enable Virtual Machine Queue" on both the host and the VM's to address this issue.

Always and forever disabled...

btw, I checked again just be to sure and it is still/back on when I swear I had turned it off at the Hyper-V level as well as in the settings for each VM over a year ago.

[JaredBusch]

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

I bet if you rebooted the host the issue goes away (at least for a short while) what version drivers are installed for these NICs?

Of course it will as that is how I fixed it 3 weeks ago.

[JaredBusch]

@JaredBusch said in Hyper-V host connectivity lost but VMs still communicate:

@JaredBusch said in Hyper-V host connectivity lost but VMs still communicate:

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

There is also the recommendation to disable "Enable Virtual Machine Queue" on both the host and the VM's to address this issue.

Always and forever disabled...

btw, I checked again just be to sure and it is still/back on when I swear I had turned it off at the Hyper-V level as well as in the settings for each VM over a year ago.

Just checked the VMs themselves. all disabled. cannot even comprehend how this is on and those are off.

[JaredBusch]

@JaredBusch said in Hyper-V host connectivity lost but VMs still communicate:

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

There is also the recommendation to disable "Enable Virtual Machine Queue" on both the host and the VM's to address this issue.

Always and forever disabled...

Get-NetAdapterVmq | Disable-NetAdapterVmq

and everything starts running.

[JaredBusch]

According to my ScreenConnect timeline, the Hyper-V host lost connectivity just after midnight. There is not a damned thing in the logs for the time frame.

Up yours VMQ.....

[DustinB3403]

VQM to the rescue, am I right?

Maybe there is a bug in Hyper-V that enables it after a patch... I honestly don't know... just throwing darts...

[JaredBusch]

@DustinB3403 said in Hyper-V host connectivity lost but VMs still communicate:

VQM to the rescue, am I right?

Maybe there is a bug in Hyper-V that enables it after a patch... I honestly don't know... just throwing darts...

Quite possible because I patched the servers after November patch Tuesday and that first failure was the following week.