@stacksofplates said in Domain Controller Down (VM):

@DustinB3403 said in Domain Controller Down (VM):

Here is a physical system

Here is a virtual system.

Now tell me which system would you prefer to use if IOPS performance was an issue.

Ha. If you look at the timeline it's the same thing from the same dates (9-12:11 - 9-12:12). Good try.

This graph is also worthless as it doesn't show us latency or queue depth so we don't actually know if the app just doesn't do anything or if it has actual demands. This would be like me showing you how many RPM's I used on my car, and without any other context you don't know if I drove from Waco to Houston at 100MPH this weekend (I did) or if I just sat in a parking lot in neutral. RAWR IOPS GRAPH TIME!

@Dashrender said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@Dashrender said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@Dashrender said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

If you're running on something using PV drivers that they don't understand...

Then your critical app vendor is below the home line. THAT'S how scary this should be to companies.

When your "business critical support" lacks the knowledge and skills of your first year help desk people, you need to be worried about their ability to support. Sure, when nothing goes wrong, everything is fine. But if anything goes wrong, you are suggesting these people don't have even the most rudimentary knowledge of systems today. That's worrisome. And it's why so many systems simply have no support options - relying on software and hardware that is out of support meaning that while the app might call itself supported, they depend on non-production systems making the whole thing out of support by extension.

So when running with a preallocated qcow2 image, which caching mode do you use for your disk? Writethrough, writeback, directsync, none?

What about IO mode? native, threads, default?

No one can support every hypervisor at that level.

Also, none of those things need to be supported by the app vendor. They just need to support the app and stop looking for meaningless excuses to block support. I understand some vendors want to support all the way down the stack, but if they don't know how to do that with virtualization, they don't know how to do it. The skills to support the stack would give them the skills to do it virtually even better (fewer variables.) So that logic doesn't hold up.

You still haven't provided a single healthcare vendor that does any of what you say is appropriate.

I know Greenway didn't have a virtualization plan 3 years ago when we were looking at them. It's why I had to build a ridiculous $100K two server failover system. Today the performance needed could be done for $25k.
The sad thing is that the vendor could not provide any IOPs requirements, etc. They only had this generic hardware requirement.
SQL Dual Proc Xeon 4 cores each two drive boot, 4 drive RAID 10 SQL, 4 drive log
RDS single proc xeon 4 core 2 drive boot, 2 drive data
IIS application dual proc xeon 4 cores each, 2 drive boot, 6 drive RAID 10 data
etc
etc

Because... no support

eh? yeah Greenway didn't bother to do the right thing for their customers and have support for hypervisors! Shit, how can they really support their customers on bare metal if they don't know the IOPs requirements, etc? Just keep stabbing hardware until they "get lucky"?

That's my guess. Lacking support of VMs isn't exactly the big issue... it's WHY they lack that support that is the big issue.

LOL - Short of someone like Epic, from what I can tell, they are mostly software developers, who don't care about the hardware/VM it's running on. They don't approach the software holistically.

In healthcare you'll find a LOT that take this stance, for liability reasons (they want something they can provide support, or to reduce the chance of an SLA miss from something that their GSS isn't familiar with). Most healthcare systems are going hosted for this reason. I had a nice chat with the Cerner guys at VMworld and they mentioned that they offer SLA's all the way to how quick a patient note pulls up (7 seconds worst case I think). They in many cases actually take over on site support end to end (and act as a MSP in addition to an EMR). Realistically for EMR's given their horizontal integration of features, the next logical step is vertical integration of the hardware and end user computing support.

@stacksofplates said in Domain Controller Down (VM):

@DustinB3403 said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

If you're running on something using PV drivers that they don't understand...

Then your critical app vendor is below the home line. THAT'S how scary this should be to companies.

When your "business critical support" lacks the knowledge and skills of your first year help desk people, you need to be worried about their ability to support. Sure, when nothing goes wrong, everything is fine. But if anything goes wrong, you are suggesting these people don't have even the most rudimentary knowledge of systems today. That's worrisome. And it's why so many systems simply have no support options - relying on software and hardware that is out of support meaning that while the app might call itself supported, they depend on non-production systems making the whole thing out of support by extension.

So when running with a preallocated qcow2 image, which caching mode do you use for your disk? Writethrough, writeback, directsync, none?

What about IO mode? native, threads, default?

No one can support every hypervisor at that level.

Also, none of those things need to be supported by the app vendor. They just need to support the app and stop looking for meaningless excuses to block support. I understand some vendors want to support all the way down the stack, but if they don't know how to do that with virtualization, they don't know how to do it. The skills to support the stack would give them the skills to do it virtually even better (fewer variables.) So that logic doesn't hold up.

So they don't need to be fully supported, but let's say the IT guy down the street who's used Linux twice in his life installs the software in a VM with a non preallocated QCOW2 with an rtl8139 NIC. It's going to run slower than anything. So he calls the vendor for support and they try to help him. Nothing they are going to be able to tell him is going to help him, because it's nothing to do with their software. It's in their best interest to try to control what you're installing on to to mitigate stupid issues like that.

At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.

That is the issue of the IT Guy not understanding the system requirements, the fact that it is virtual means nothing. He could install that image to a bare metal system and have just as poor performance!

No, those are specific to a hypervisor. Bare metal would be much faster than that, you woudln't have those issues.

If bare metal was with a single ATA 66 drive, it might not be... Virtualization doesn't have a monopoly on stupid non-supported configurations.

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.

Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.

Right, and they do. VMware.

Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.

Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.

I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.

To be clear, requiring VMware ESXi in a supported configuration is at odds with the 100Mb/s for vMotion and iSCSI (VMware does NOT support this abomination of a configuration).

@scottalanmiller said in Domain Controller Down (VM):

What if your critical enterprise app ONLY supported Mac OSX based on similar logic. Would you start deploying Mac Minis without a car as your enterprise platform? We would not normally consider that an enterprise option. But it's a similar approach. They would know the hardware and the OS. Is them knowing something more important than it being something good to know?

Oddly I'm in a conversation with a (Unicorn) over a 64 node All Flash VSAN cluster because they have software that is critical that requires OS X (They do a heavy amount of mobile development with heavy xCode dependencies). I've worked with (Gaming development company) who did FC to Mac's running ESXi for test/dev environments. You can run HA clusters on Mac's and VMware does support this.

That said if a medical EMR had this requirement I would tell them to GTFO.

@scottalanmiller said in Domain Controller Down (VM):

@DustinB3403 said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

@scottalanmiller said in Domain Controller Down (VM):

@stacksofplates said in Domain Controller Down (VM):

At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.

Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.

Right, and they do. VMware.

Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.

Ya, this whole thing started because Dustin said @wirestyle22 should drop them since they don't support anything else. That's ridiculous.

I specifically said I'd look for alternative software if an appliance vendor said they only supported a single hypervisor.

Big difference.

Although the client SHOULD consider the high cost of VMware for such a small system. They are looking at a $40K SAN to support that one application now based on that one app. And that's a lot of VMware costs. We don't know how much that one app costs, but holy cow is that a huge budget for a tiny company just as support costs for a single app. SMBs don't normally have total budgets that big, let alone that much to spend as ancillary costs to a single app!

You'd "hope" that this was a $200K+ application to make that make sense.

In healthcare there's a strong chance that the cost of the application, the migration, and the accompanied support agreements make a 40K storage array "cheap". Combined the fact that he likely has 4-5 applications in this situation (at a minimum) and a small HDS or a VxRAIL appliance (~$65K starting) could be a rounding error.

@scottalanmiller said in Domain Controller Down (VM):

@John-Nicholson said in Domain Controller Down (VM):

@coliver said in Domain Controller Down (VM):

Looks like an ipod... This is going to be interesting in the long term. Those Cisco chassis can do some expending though so you may be able to get to a more reliable system with what you have.

Actually UCS can't really expand much from a storage perspective. They don't have any native DAS JBOD support, and the MegaRaids on them they do little in integration or customization. UCS was never really designed to use local storage in RAID I'm convinced (at scale anyways). They are useful if your using them in true JBOD (VSAN, they are certified for use) or with HBA's to talk to an external disk array.

Definitely not designed for that use.

Other thing is get some training. There's some rookie mistakes lurking in that config that scare me about other things...

http://vmware.stanly.edu is the poor mans path to a VCP. Grab a copy of mastering vSphere. The HA deep dive book is free now if you know where to look. and increasingly we'll have storage and availability documents lurking at storagehub.vmware.com

@scottalanmiller I see your on 45 coming back from the airport. You still in town up for a beer?

Steps everyone else missed.

vCenter came up after the DC's so we restarted the vCenter Services.

Some of the VM's were running as zombie's (storage had dropped to long, OS had crashed) so the VM's had to be reset once storage came back up (Note APD detection in 5.5 and 6.x is better and this isn't as common).

Storage latency over 100Mbps iSCSI is awful. 500ms max on one VM, and average of 45ms. This is like running your hard drive remote from Houston to Atlanta. Hence my recommendation of some proper enterprise direct FC attached storage to deal with this mess.

This stuff isn't supported (and has NEVER been supported to use vMotion or iSCSI over 100Mbps) in the past 8 years I've worked with VMware.

As we discussed my home lab from 5 years ago was in a better state for availability and performance and supportability.

@JaredBusch said in Domain Controller Down (VM):

@wirestyle22 said in Domain Controller Down (VM):

Thanks to @John-Nicholson for the help! A lot of great information.

The problem is now that we stepped up to help a member of our community and the bosses know nothing of how much this should have cost you to get repaired.

VMware GSS is around 24/7 to help with stuff like this.
Just call them next time. 1 (877) 486-9273
Make sure to add yourself to the authorized list ahead of time for faster service, and write down your customer support. This is why you use enterprise products, so you can get help quickly.

@coliver said in Domain Controller Down (VM):

Looks like an ipod... This is going to be interesting in the long term. Those Cisco chassis can do some expending though so you may be able to get to a more reliable system with what you have.

Actually UCS can't really expand much from a storage perspective. They don't have any native DAS JBOD support, and the MegaRaids on them they do little in integration or customization. UCS was never really designed to use local storage in RAID I'm convinced (at scale anyways). They are useful if your using them in true JBOD (VSAN, they are certified for use) or with HBA's to talk to an external disk array.

@wirestyle22 said in Domain Controller Down (VM):

@John-Nicholson

I am definitely interested in any education you are willing to offer but I think we are actually almost finished with this currently. Are you willing to sit with me for literally any amount of time tomorrow? I am a very eager learner.

A bit tied up with prep for VMworld Barcelona but might find a few minutes to talk.

One thing is as soon as humpty is back together again is run RVTools against the vCenter. Get a XLS dump. The health tab on the end will find all kinds of fun misconfigurations but I can go over it with you (or if you can sanitize it and post it hear I can give you a tear down).

Next up your on essentials Plus. You can have 24/7 production support, use it. GSS likes to help.

100Mbps switching is NOT supported for storage or vMotion.
Get some real storage. [email protected] can help you get a small FC DAS HUS with good support to avoid a lot of this mess.

Upgrade to the VCSA (use the migrate2VCSA tool!), and 6.x ESXi. ESXi 5.1 is not in general support anymore as of last week or so.

Before you do that upgrade the BIOS/FIrmware on those UCS boxes.
Fix the NTP serve config to start the service, and make sure to have 3 (not 2!) NTP servers so you can fix drift.
Get into the UCS CIMC (Set that up!) and fix the clocks if needed.

Get a pair of cheap but fast/good TOR 1Gbps switches. ICX 6450's Brocade's are solid, speak proper RSTP, and fast enough to handle iSCSI for the migration, and vMotion once done.

Get Veeam for backups. Possibly beef up the backup storage.
Replace that Gen6 HP. Its out of support, and unsupportable.

Don't worry a lot on resource balance, your CPU and memory usage are good and those cisco hosts (M3's) are current enough. Check the smartness' on them though, and get CIMC alerting setup.

All in FC, labor (week at 10K) Host, migration of a few VM's Veeam setup, small host, switches, upgrade, your looking 50-60K?

Still got other challenges (banish 2003, get VDI deployed, replace campus switching) but I'd give Howard a ring. He's seen worse I promise...

Other people to talk to if your in the SW, Sigma (Nigel Hickey's a good guy there). Does a lot of VDI work.

@wirestyle22 said in Domain Controller Down (VM):

@Dashrender unsure of the IP of that host. It has nothing running on it typically, but I can just assume that.

Hit what you think it might be on 443, you should be greeted with the ESXi landing page.

@wirestyle22 said in Domain Controller Down (VM):

@John-Nicholson Does VMware offer any services for suicide watch?

If You'd like join http://vm.......
I can walk you down from the ledge (for a little bit of time before dinner here)

VMware SABU checking in... You need a webex to fix this?

@scottalanmiller said in How Are You Measured?:

There is only one useful way to measure IT.... Its impact to the way that the CEO is measured. Anything else like SLA, uptime, tickets, etc is misleading, actively undermines the organization and leads to gamification.

RIOC, Average Revenue per customer/whatever the metric of your industry is, IT should be serving improving that metric.

Couple thoughts...

1. Explain the width of the field. Like medicine we have 100 directions you can go with sub specialization to the point that two professionals in the same room might not understand a single thing the other does. (IE Developer vs. Sysadmin, and within Sysadmin a AIX admin vs. a Windows admin will still have a lot of different concepts and tools).

2. Explain how NOT to get in the field. AVOID all for profit scam shops. Note that a large number of people work in IT without holding degree's, or holding non-traditional degree's. Explain how you can learn many concepts at home with a small lab.

3. Explain the role IT plays (Operations, Finance etc). In a lot of ways IT is becoming less of a stand alone field, and discuss how understanding business and operations are critical to the job (It isn't for people who just want to hide in a basement).

@DustinB3403 said in What are you listening to? What would you recommend?:

And moderated....

Love it... Follow up to Bud G. but I honestly don't want to continue this thread.. Ninja is a PITA. Plain and simple, and only when called out does he provide context.

You have to realize that that guy in the afternoon is often so lets assume.

1. Possibly posting from an iPhone which on spiceworks is epically painful to post long posts on.
2. Possible not sober.
3. Posting between games of Dota II.

@DustinB3403 said in What are you listening to? What would you recommend?:

FYI I just called out that p***k storageninja

Spice Pe**** forum.

Sorry if you can't read it.

here is a screen grab (does google read these?)

That guys a dick!

@scottalanmiller

@scottalanmiller said in Simplivity - anyone use them?:

But that's not a viable customer anyway. Don't hurt good, real possible customers in order to protect someone too stupid to operate in IT anyway. That's not sound logic. You are protecting the wrong people... punishing the qualified buyers to assist the unqualified ones.

Also known as the majority of the people purchasing IT equipment?

@scottalanmiller said in Simplivity - anyone use them?:

That means that that customer can't do things like order food in restaurants, buy a car, buy a house, etc. This is a level of incompetence that is so bad, that there is no way that they could be an operational company.

Yah about that. From my consulting days I was confused how people remembered to put their pants on their legs and not their heads some days...

Withholding pricing till its qualified is part of a game in the enterprise that fixes the following situations...

Sometimes its malicious...

1. Cisco Fanboy network admin decides he wants to buy a new 5K switch he does't need... He is required to get 3 bids so he calls Broacde and asks for the biggest baddest VDX config so they will be too expensive.

Sometimes its just someone in a hurry....

2.HCI dude In a mid market company has decided he want some HCI! He mistakenly assumes that Simplivity does RAID 10 on top of RAID 10 like he used to do with LeftHand and vastly oversizes the solution. At the same time he quotes vendor xxx he assumes they have dedupe (even though its large block and doesn't really work or only scans the first couple GB of disk before it gives up). He quotes vendor Z and because he only got pricing missed the fact that they don't have a (Magical FPGA card) like simplivity and their VSA will require 8 core's be assigned (and hard reserve a good chunk of them!).

There is HUGE differentiation in a lot of solutions.

SMB's Quote first, learn later attitude works when your buying commodity low friction products (Printer Ink, generic rack mount servers etc). When you buying stuff like HCI or fancy Networking gear, or storage arrays I'd argue that getting quotes too quickly can (and I have seen) can lead to poor outcomes if someone doesn't stop in the quoting process and educate the customer. (When you end up running your entire business on a VNXe with 6 SATA disks because it was the only thing you thought you could afford). HCI appliance pricing is even trickier as with meet in the channel type solutions you are at the mercy of the underlying OEM for prices (and Dell's SSD prices have been changing almost daily in some cases due to changes in their supply chain!). Getting a price today carries the risk that another solution may appear cheaper if you get your quotes to far apart.

Disclaimer, for what its worth my employer posts most of their list prices on the internet (or doesn't do a very good job of suppressing them). Thankfully we don't sell hardware (So that risk/shifting price) isn't part of our exposure and prices stay pretty constant with a fairly well known multiplier for support and updates till the end of time. We do sell primarily though channel though, and require you call someone to get a quote on anything but the smallest packages (like essentials plus, and even then you'll still pay less from a partner). The reason is to see if they can qualify and maybe get you something else that you might not know you need, or make sure your purchasing the right "SKU". It could be something basic like an academic institution not realizing we have special pricing, or someone deploying for a VDI pilot not knowing we have per user licensing that will cost them 20% of normal socket pricing, service providers missing per GB pricing, or ROBO missing per VM pricing stuff.

I remember many years back someone called out Steve Balmer on Microsoft licensing being confusing with all the options. He calmly responded that he recognized it as a problem having lots of options to buy something, but he contented that to simplify it would lead to someone being angry.

Complicated purchasing and pricing and packaging is fundamental to any class of products that doesn't have a laser like focus on a single vertical.