Hyper-V Failover Cluster FAILURE(S)
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
What size pipe is between the switch and the router? Can you see if the pipe was tapped out around the time of the failure?
It's never tapped out. We keep the Tegile Metrics up on our monitor that shows our SQL server performance and TotalMail server which communicates with the fleet trucks.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks (RAID 1). All VMs are stored in 1 LUN on the SAN.
-
6 hosts, failover cluster with local storage might be challenging, I don't really know anything about it.
I'm sure @scottalanmiller can give some info.
Depending on the age of the hosts, you might find yourself much better off with a two host setup with internal storage and something like StarWinds VSAN. 7 TB internal storage shouldn't be that hard to come by - though the performance needed might require some caching, etc.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
So you don't have real HA anyhow? i.e. if either SAN fails, the VMs on the failed SAN are down.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
So you don't have real HA anyhow? i.e. if either SAN fails, the VMs on the failed SAN are down.
Exactly!
There's also a SQL server that's still running in bare metal.
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
So you don't have real HA anyhow? i.e. if either SAN fails, the VMs on the failed SAN are down.
Exactly! There's also a SQL server running in bare metal.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
So you don't have real HA anyhow? i.e. if either SAN fails, the VMs on the failed SAN are down.
Exactly!
There's also a SQL server that's still running in bare metal.
So you have at least 2 SQL servers? One bare metal and one VM?
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
So you don't have real HA anyhow? i.e. if either SAN fails, the VMs on the failed SAN are down.
Exactly!
There's also a SQL server that's still running in bare metal.
So you have at least 2 SQL servers? One bare metal and one VM?
3 SQL servers. 2 VM 1 bare metal that is going to be migrated in the next month.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
If downtime is a no no, why separate the VMs?
Do you have one or two SANs?
We have 2 but the 2nd is for SQL data.
So you don't have real HA anyhow? i.e. if either SAN fails, the VMs on the failed SAN are down.
Exactly!
There's also a SQL server that's still running in bare metal.
So you have at least 2 SQL servers? One bare metal and one VM?
The entire environment is bad practice after bad practice.
-
What are your end goals here exactly?
To just fix the error/main problem and be done?
To achieve true HA?
If not HA, then to actually set things up in a practical way that makes sense and is good for the business?
Host redundancy?
Network redundancy?
Host Storage / VM redundancy?
-
@tim_g said in Hyper-V Failover Cluster FAILURE(S):
What are your end goals here exactly?
To just fix the error/main problem and be done?
To achieve true HA?
If not HA, then to actually set things up in a practical way that makes sense and is good for the business?
Host redundancy?
Network redundancy?
Host Storage / VM redundancy?
Fix the I/O issue to start. Deprecating all the old bare metal is on the list but is taking time as we have to work with vendors and some upgrades are contingent on future upgrades that are due soon.
-
We rolled out Class B networking and then 48 hours later made the IP changes on the DFS farm and then 2 hours later we ended up having identical Event ID 5120 Where the Cluster lost connection to the SAN.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
We rolled out Class B networking and then 48 hours later made the IP changes on the DFS farm and then 2 hours later we ended up having identical Event ID 5120 Where the Cluster lost connection to the SAN.
DFS? What exactly are you using DFS for in relation to the cluster?
-
@tim_g said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
We rolled out Class B networking and then 48 hours later made the IP changes on the DFS farm and then 2 hours later we ended up having identical Event ID 5120 Where the Cluster lost connection to the SAN.
DFS? What exactly are you using DFS for in relation to the cluster?
Nothing as far as the Clustering goes. But that's not saying the MSP didn't change something else when they did the IP address changes on the DFS server after going from a /24 to a /16. I've read several things about Subnetting causing Auto Pause issues in a Hyper-V environment and 2 huge IP changes were made in the environment in a short amount of time.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
We rolled out Class B networking and then 48 hours later made the IP changes on the DFS farm and then 2 hours later we ended up having identical Event ID 5120 Where the Cluster lost connection to the SAN.
But that never happened before?
An issue here is that changing the networking means a lot of things were changed, not just the subnet mask size.
