XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.
That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.
Does this mean it's irreparable?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.
That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.
Does this mean it's irreparable?
No, just really REALLY silly.
-
@momurda from a quick glance those appear to be all CIFS ISO repos that are AD tied, so we should be able to just remove those to fix that, right? I don't see anything else.
-
@CitrixNewbJD
No i dont think so.
You got some work to do though
Probably need to do:
log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the sr -
I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.
-
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD
No i dont think so.
You got some work to do though
Probably need to do:
log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the srI didn't see it on the iSCSI, darn it. Still fixable, but more work.
-
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.
That, and never have AD on top of something that depends on it. That's like locking your keys in your car.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.
That, and never have AD on top of something that depends on it. That's like locking your keys in your car.
Yup. Our SAN is isolated from the rest of the house network physically. All authentication between the SAN and the host should never even touch AD just in case of things like this.
Sometimes playing what-if is a good exercise to keep bad things from happening as far as security, reliability, and recovery.
-
@scottalanmiller
Yes the cifs repos can be forgotten for now. I doubt there is anything important there in the iso repo.
@CitrixNewbJD
Before you do anything else, make sure you have the root pw for your xs servers as disabling AD integration will force you to use them. -
Unless you have a DC that you can get working, ie one not attached to your shared storage. Perhaps you have a physical dc somewhere? Unlikely i know.
-
@CitrixNewbJD This is Frank. I hope Scott and the gang here can help get you back operational quickly. To touch on a topic that was mentioned earlier, VMs from Xen can be imported and converted on the fly. As long as the Scale nodes can browse to the VM files (.vhd) on the storage, the XenServer functional state doesn't matter.
-
Having been through this once before, and learning the hard way, I do normally have a physical DC. Despite my warnings, because I know that we do not currently have one here, I was told to bring it all down. And here we are. We do not have a physical DC.
-
I've been using the root authentication for everything.
-
Frank, can we speak on the phone for a minute so I can be sure I can intelligently talk to the guy when demanding his money?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Having been through this once before, and learning the hard way, I do normally have a physical DC.
This is absolutely the wrong response. You should never have a physical DC, ever. There is zero issues here with virtualization. There are two problems....
- Zero AD redundancy
- An inverted pyramid of doom (single storage for all systems)
Fixing either of those anti-practices would have saved you. Physical would have zero benefit and is the polar opposite of the reaction that you should have.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
I've been using the root authentication for everything.
So we are safe there.
-
More on the IPOD: http://www.smbitjournal.com/2013/06/the-inverted-pyramid-of-doom/
And in video form from MangoCon:
-
So, when looking for places to turn off AD integration, I see this...
-
It's not pool integration that is the issue, it's SAN integration. Check the SAN (PowerVault) interface instead.
-
@seal Just came across these two items on the SAN interface. Dental_Data, Spindlemedia, are critical and it looks like those VDs failed.
PROFILE FOR STORAGE ARRAY: MDS-Spindle01 (12/27/16 3:28:58 PM) STANDARD VIRTUAL DISKS------------------------------ SUMMARY Number of standard virtual disks: 3 See other Virtual Disks sub-tabs for premium feature information. NAME STATUS CAPACITY RAID LEVEL DISK GROUP DRIVE TYPE Dental_Data Failed 1.495 TB 5 0 SAS SpindleMedia Failed 2.862 TB 5 0 SAS Virtual Failed 1.367 TB 5 0 SAS DETAILS Virtual Disk name: Dental_Data Virtual Disk status: Failed Capacity: 1.495 TB Virtual Disk world-wide identifier: 60:02:4e:80:00:7b:78:6a:00:00:04:13:4a:96:70:f3 Subsystem ID (SSID): 1 Associated disk group: 0 RAID level: 5 Physical Disk type: Serial Attached SCSI (SAS) Enclosure loss protection: No Preferred owner: RAID Controller Module in slot 1 Current owner: RAID Controller Module in slot 1 Segment size: 128 KB Capacity reserved for future segment size changes: Yes Maximum future segment size: 2,048 KB Modification priority: High Read cache: Enabled Write cache: Enabled Write cache without batteries: Disabled Write cache with mirroring: Enabled Flush write cache after (in seconds): 10.00 Dynamic cache read prefetch: Enabled Enable background media scan: Enabled Media scan with consistency check: Enabled Pre-Read consistency check: Disabled Virtual Disk name: SpindleMedia Virtual Disk status: Failed Capacity: 2.862 TB Virtual Disk world-wide identifier: 60:02:4e:80:00:70:ed:06:00:00:07:f5:4d:ba:7b:fb Subsystem ID (SSID): 2 Associated disk group: 0 RAID level: 5 Physical Disk type: Serial Attached SCSI (SAS) Enclosure loss protection: No Preferred owner: RAID Controller Module in slot 0 Current owner: RAID Controller Module in slot 1 Segment size: 128 KB Capacity reserved for future segment size changes: Yes Maximum future segment size: 2,048 KB Modification priority: High Read cache: Enabled Write cache: Enabled Write cache without batteries: Disabled Write cache with mirroring: Enabled Flush write cache after (in seconds): 10.00 Dynamic cache read prefetch: Enabled Enable background media scan: Enabled Media scan with consistency check: Enabled Pre-Read consistency check: Disabled Virtual Disk name: Virtual Virtual Disk status: Failed Capacity: 1.367 TB Virtual Disk world-wide identifier: 60:02:4e:80:00:70:ed:06:00:00:04:31:4a:96:73:09 Subsystem ID (SSID): 0 Associated disk group: 0 RAID level: 5 Physical Disk type: Serial Attached SCSI (SAS) Enclosure loss protection: No Preferred owner: RAID Controller Module in slot 0 Current owner: RAID Controller Module in slot 1 Segment size: 128 KB Capacity reserved for future segment size changes: Yes Maximum future segment size: 2,048 KB Modification priority: High Read cache: Enabled Write cache: Enabled Write cache without batteries: Disabled Write cache with mirroring: Enabled Flush write cache after (in seconds): 10.00 Dynamic cache read prefetch: Enabled Enable background media scan: Enabled Media scan with consistency check: Enabled Pre-Read consistency check: Disabled
