Testing oVirt...
-
@dyasny said in Testing oVirt...:
In the real world, Fedora is supported and works. EL does not for production workloads I deal with. When I have the choice between the two, Fedora every time.
In the real world, especially at scale, nobody has the time to deal with the bugs you get in the new untested code. Not everything can be solved by restarting a microservice, and not many organizations keep talent onboard, who can fix a kernel bug on the fly. This is why people rely on EL and other enterprise grade software.
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
-
@scottalanmiller said in Testing oVirt...:
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.
Having used both in the real world, what problems do you see with Fedora and what old code are you running that EL always has what you need?
I can install EL, keep it updated without moving versions, and be certain everything I build on top of it will keep working throughout the OS lifecycle, which is very long. That means I can concentrate on developing my software instead of wasting time on keeping up with what the underlying OS is doing, introducing bugs into my code which used to work before the OS update. Without losing the safety of important updates coming in on time. Yes, some of the prepackaged stuff is outdated, but I can get the newer code, if I need it, from repositories that are code-specific (in my case - mostly pypi). I've come into a place once, where Fedora was installed on several hundred servers which were rendering CGI. They had two people there just making sure the cluster task dispatcher was able to work after every update. Two expensive coders, doing nothing but test fedora releases versus their task scheduler. A test cluster on CentOS turned out to be able to do exactly the same job, but the guys dealing with fedora quirks became free to develop the task scheduler, which boosted their bottom line productivity after a few months.
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
-
@dyasny said in Testing oVirt...:
This is why people rely on EL and other enterprise grade software.
Not really. In every case I've ever encountered, people use EL due to a policy of "buying vendor support" without actual investigation into a need. It's political finger pointing in nearly all cases. That's how businesses work. Go up the chain and dig in at real world customers, how many actually evaluated risk and got support for that reason, versus get it for every product and never even look into why for a specific one.
When I got into the hedge fund world, the idea of "evaluating support" was new and just started happening. And there too, RH lost out because they found that there was no need for support. It just wasn't a thing that came up in the real world.
People running into those kinds of bugs are edge cases and generally causing them by doing other things poorly.
-
@dyasny Yep, a great example
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
And vice versa. Having worked at the vendor in question, it feels like you are seeing this through the eyes of the sales team, and not thinking about it from the perspective of the customers needing to actually run the product. Where is your conviction coming from, other than it being the sales mantra of the vendor?
Never worked in sales, but I have worked for other enterprise vendors and their customers quite a bit. From government to automotive and aerospace, to oil and gas to hipster-ish startups. No Fedora anywhere but desktops sometimes. Ever.
This is a time thing, though.
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
When I was in banking, EL was king. But the entire point of the article I linked was that things have changed, code is more mature today, projects move faster, priorities in the world have changed.
Your looking to past experiences tells me you are thinking about the past, not the environment today. And you are missing that we had the same experiences back then, too. So it is based on the same experience, same knowledge, but bringing it up to date to today's situations, that led us from EL to Fedora.
-
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
-
@dyasny said in Testing oVirt...:
Are you really seeing Fedora instabilities that the rest of us are not? Are you really not running any modern code that benefits from current libraries and packages?
Of course I am. pip, cpan, (whatever php has for that same purpose) etc do the job perfectly. There isn't much I cannot do on an EL distribution, one way or another
- But you can't do it as well, or as performant.
- If you replace any libraries to do it, I consider that acknowledgement that Fedora was better, but you were only on EL to make a point.
CAN you do things on EL? Of course. That's not the point. It's whether it is better than Fedora most of the time. And that answer seems to be "no". You make some good points about testing, but don't address that we had already considered that before coming to the conclusion that they weren't good enough to overcome the significant deficits of the process.
-
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
-
@dyasny said in Testing oVirt...:
@scottalanmiller this is foul play indeed, and knowing the company from within, I'm pretty sure such practices aren't smiled upon. I'm not with RH any longer, but in my 10 years there, they have always struck me as the most morally positive company I've ever worked with and for. I've seen business needs sacrificed to doing the right and moral thing quite a few times there. In the end - this is why they succeed, where those who attempt to be business sharks fail (waves at Canonical).
Do keep in mind, you have one single story, I've lived in there for years. I'd say I do have more stats
Inside stats are misleading, though. Vendors paint a different picture for employees than what customers see. That's not an RH thing, that's universal.
I guarantee as an employee, there were zero stats kept about how they were doing unethical, maybe illegal, things with customers. Account managers don't report to engineering.
That they had a process for this at all means it wasn't casual. This was a major effort to do what they did. If you didn't hear about it while there, then your stats aren't valid, if that makes sense. Because I bet you were told that nothing like that was happening with major customers (we had to be one of the five biggest customers.)
We had the same thing internally at where I was, so I get how it happens. We happen to catch it in our case. HR was trying to sabotage some departments and telling people that the jobs were awful before those applicants went fully into the system. So the "stats" internally said one thing, but if you could find someone who had turned down a job with the company, you learned all kinds of secrets that weren't officially recorded anywhere.
Same with RH - there is no chance that that activity was on the books. But it was a major effort to manipulate the customer. They could easily guarantee that their account would not be dropped if the heads of the engineering departments were working at RH and not at the customer.
-
@dyasny said in Testing oVirt...:
The apps we run (and develop) are tested against Fedora, so.... where do you see the concern? Why would the customer(s) need to deal with these problems, what's the source of your worries?
The apps don't run in a vacuum, they rely on layers of software. Do you test all those layers?
Individually? No. Together, yes.
-
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
But in the real world, you don't have these problems with Fedora. You are correct, those are things you don't want happening. Thankfully, Fedora protects you from that. That's the point.
How does Fedora protect you? It's a distro packaged, with some bugfixes here and there, no formal QA besides the very basics, no support, nothing. I'm fine with that on my laptop, but on a thousand servers?
How doesn't matter, that it does it and does it reliably is what matters.
We can postulate any number of reasons "why" or "how". Maybe having the individual packages more up to date simply matters that much. Maybe extensive testing is a waste and not as valuable as it sounds. Maybe Fedora tests more than you realize (maybe unofficially.)
Bottom line, though, is that Fedora has reliably protected. EL has, too. It's not that EL is bad, it's that Fedora just has come out better in real world use.
There are lots of things that matter at the end of the day...
- Reliability
- Speed
- Security
- Compatibility
And more, of course. When taken as a whole, EL might win on reliability, but the margin of winning is nominal. But Fedora seems to win on all of the others. Sometimes nominally, sometimes by quite a margin.
-
@obsolesce said in Testing oVirt...:
@dyasny said in Testing oVirt...:
@black3dynamite said in Testing oVirt...:
Because of Fedora release schedule, I don't have to rely to much on using additional repos for stuff like php, databases, etc.
Well, if you need the latest bleeding edge releases, of course an EL distro isn't for you. Why use Fedora though, when you can use something more lightweight, like Alpine, in a container?
Please tell me what the point is in CentOS running PHP 5.6?
I mean, Look how old it is, and look when it looses support!
Fedora 28 uses 7.2.x, FAR FROM BLEEDING EDGE (ffs!). And oh looky, supported for longer than 2 more months lol.
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
One COULD argue that RHEL goes out of support when PHP does. From an application perspective, using RHEL 7 would be "unsupported".
By that logic, which is pretty solid in reality, Fedora is the supported OS, not RHEL, come January.
-
@scottalanmiller said in Testing oVirt...:
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
So one second you say business is slow to change, and then you say that is moot. OK.
When I was in banking, EL was king. But the entire point of the article I linked was that things have changed, code is more mature today, projects move faster, priorities in the world have changed.
Code is, if anything, less mature today. With the current craze of moving as fast as possible, release as fast as possible and skip testing everywhere ("we've got devs writing unit tests for that" - sounds familiar?) code is getting crappier every day. And with solutions like k8s and mesos, people stopped debugging, their answer is "just respawn, who cares if it fails every 5 minutes".
Your looking to past experiences tells me you are thinking about the past, not the environment today. And you are missing that we had the same experiences back then, too. So it is based on the same experience, same knowledge, but bringing it up to date to today's situations, that led us from EL to Fedora.
Oh leave the slogans aside, neither of us has any future experience, just the past. You can guess what will happen in the future, but you dont know, no matter how much you "look to the future". You found Fedora to be working for your use case - well and good. This doesn't mean EL is less stable or has less support, you simply found a niche use case which works for you. Huge difference
-
@obsolesce said in Testing oVirt...:
Have fun upgrading the CentOS LTS servers you use to the next CentOS LTS... EVERYTHING will break, including all of your PHP apps.
This is a big one. Fedora has a consistently rock solid update path. EL does not. Moving EL versions has always been a hassle and tends to cause apps to break like crazy. Fedora isn't perfect, nothing is, but the difference is shocking. Keeping Fedora up to date is trivial, EL is an effort (akin to Windows.)
-
@dyasny said in Testing oVirt...:
@scottalanmiller said in Testing oVirt...:
Remember you are talking to loads of people that were passionate about CentOS / EL in the past and switched recently because times have changed. So unless you worked for all of these places in the last 24 months, your experience is moot. All of our experiences would match that, as well.
So one second you say business is slow to change, and then you say that is moot. OK.
You are arguing for what businesses "do", which is irrelevant. That most companies do things badly is of no concern to us. What we care about is how should we do things to do them well.
That "what good looks like" has changed and that's what we as IT pros care about. Doing what "everyone else does" is a recipe for disaster. We discuss this all the time. The "average" IT shop is terrible, and the average business loses money and fails. So what "everyone else does" is interesting to note, and worth looking at, but never a reason to not evaluate needs and look at the real world.
But the point was, you used an example of "what people do" that exactly matched both cases - those that would switch to Fedora and those that don't. It's moot because you equally supported both points.
-
@scottalanmiller said in Testing oVirt...:
Inside stats are misleading, though. Vendors paint a different picture for employees than what customers see. That's not an RH thing, that's universal.
I've spent plenty of time on the customer side.
I guarantee as an employee, there were zero stats kept about how they were doing unethical, maybe illegal, things with customers. Account managers don't report to engineering.
That they had a process for this at all means it wasn't casual. This was a major effort to do what they did. If you didn't hear about it while there, then your stats aren't valid, if that makes sense. Because I bet you were told that nothing like that was happening with major customers (we had to be one of the five biggest customers.)
We had the same thing internally at where I was, so I get how it happens. We happen to catch it in our case. HR was trying to sabotage some departments and telling people that the jobs were awful before those applicants went fully into the system. So the "stats" internally said one thing, but if you could find someone who had turned down a job with the company, you learned all kinds of secrets that weren't officially recorded anywhere.
Same with RH - there is no chance that that activity was on the books. But it was a major effort to manipulate the customer. They could easily guarantee that their account would not be dropped if the heads of the engineering departments were working at RH and not at the customer.
In a large company, stuff done by some members can be unethical. I seriously doubt it was something decided at the top company level though, and if you cared to complain all the way to the top, heads would have flown.
when I worked at a hardware vendor we all know quite well, there was a sales team hired somewhere in Asia to do UK sales. They were told to sell as much as possible, provided training and left alone. A month later, a call came in from some old lady who was sold a rack of SAN equipment when she called for a printer. After an investigation, the entire team was sacked and the customers reimbursed. Does that make that vendor evil?
-
@dyasny said in Testing oVirt...:
This doesn't mean EL is less stable or has less support, you simply found a niche use case which works for you. Huge difference
Same in reverse. You started this by pushing that EL was so much better because it gets testing and more support. You made us explain why that doesn't work for us (and anyone of whom we know.) It has a bigger support process, but we aren't convinced the support is actually better in end results. We've shown that it may not even be able to claim to have support technically in a few weeks any longer. And that "EL works for you" is fine, but we are saying that they both work, but one works "better".
You are requiring the position be that EL doesn't work at all, but that's not the case. It's that we believe Fedora has pulled ahead. And that's where I feel you aren't reading what we are writing. You are giving lots of great examples of why EL has some great value and advantages. But you aren't explaining why you feel they are more important than the reasons why Fedora has advantages.
Everything you are saying about EL we already knew and accepted and used and believe fell in value in the last two years to the point where Fedora pulled ahead in real world desirability for most production environments. And in all of this, I feel like you are only arguing for getting up to the starting point of the discussion where we were before we first looked into this years ago.
So consider it that we accept everything you said, and thought that it was all assumed, and didn't realize it needed to be stated. Now from this point, explain why you feel that outweigh's Fedora's advantages such as actual package support, working applications, no significant kernel issues, higher speed, more mature code, more packages, more features, working upgrade paths, etc.
-
@dyasny said in Testing oVirt...:
In a large company, stuff done by some members can be unethical. I seriously doubt it was something decided at the top company level though, and if you cared to complain all the way to the top, heads would have flown.
Maybe, but they probably should have been following up with customers, too. This went very, very high at the customer and ended up affecting the RH relationship at multiple F100 companies. And has been published publicly more than a few times. RH has never once acknowledged, reprimanded, explained or apologized.
I think it's unreasonable that senior management doesn't know. Either they are putting their heads in the sand and don't want to know what customers are experiencing; or they are complacently allowing stuff like this to happen.
Maybe they realized the risk was too large and backed off. Maybe they didn't realize they would get caught. Who knows. What I know is, there's no plausible way they didn't either ignore it or outright condone it.
-
@dyasny said in Testing oVirt...:
when I worked at a hardware vendor we all know quite well, there was a sales team hired somewhere in Asia to do UK sales. They were told to sell as much as possible, provided training and left alone. A month later, a call came in from some old lady who was sold a rack of SAN equipment when she called for a printer. After an investigation, the entire team was sacked and the customers reimbursed. Does that make that vendor evil?
So the did something to fix the situation? That's the difference between them and RH. RH has never once attempting to rectify it to the customer, or to the employees.
So you see why I see RH as very, very different.
And overselling to someone is the job of sales, violating professional and business ethics is not. Nor is attempting to poach customer staff.
