Exchange 365 Down?
-
We got the issue resolved.
The low-down.
We have Exchange 2010 in a Hybrid environment. AD onsite and MS manages Exchange.
In the EMC our Office 365 Exchange management was disconnected, we could no longer manage Exchange Online from our on premise server.
Submitted a ticket into MS to determine why, and the technician recommends running the Hybrid configuration wizard and confirms the details of what needs to be in the "Wizard".
What this Wizard does is put a new connector onto your Office365 Exchange account, which in our case change the route for our email to flow.
Rather than straight from us to MS to the outside, it was going from Us to MS back to Us and then attempting to go out.
A second ticket with Microsoft reporting the outage found the issue after about 4 hours of downtime, and the MS team even admitting that the tech who was trying to help fix the initial issue needs further training.
Summary, MS updated something on O365 Exchange that is no longer compatible with Exchange 2010 on premise, using the wizard broke our mail flow connector. I not being an exchange admin (nor office 365 admin) would have known where this issue was or how to fix it.
-
Interesting, so in this case, while an MS tech was involved (we've had a lot of issues tied to MS support screwing things up) the issue was on premises Exchange causing the outage, not Office 365? So this actually supports why hosted email is important rather than the opposite.
And supports what we keep saying that SMBs rarely have highly trained Exchange admins, you need to be doing Exchange all day, every day to really be good at it.
-
Our on premise server only acts as an Active Directory relay for our User / Room accounts that need email.
Being in a hybrid environment is what messed this up, and the tech not knowing what the hell he was doing.
-
@DustinB3403 said:
Being in a hybrid environment is what messed this up, and the tech not knowing what the hell he was doing.
That was my point, it's the on premises stuff that created the risk.
-
@scottalanmiller You could also look at it as 365 was the problem too
That being said, I will be migrating from on-premise to 365 within the next year.
-
@wrx7m said:
@scottalanmiller You could also look at it as 365 was the problem too
In what way could you, though?
-
@scottalanmiller If he didn't have 365, the problem wouldn't have existed because MS introduced a change to the 365 side that broke his hybrid setup. Either, full hosted or full on-premise would not have been impacted.
-
@wrx7m said:
@scottalanmiller If he didn't have 365, the problem wouldn't have existed because MS introduced a change to the 365 side that broke his hybrid setup. Either, full hosted or full on-premise would not have been impacted.
Sort of, if he didn't have O365 he'd have no email at all
That the change was made on the O365 side doesn't mean that the break wasn't on the local side, though. No one said it was a mistake on the hosted side. -
@DustinB3403 said:
Summary,** MS updated something on O365 Exchange **that is no longer compatible with Exchange 2010 on premise, using the wizard broke our mail flow connector. I not being an exchange admin (nor office 365 admin) would have known where this issue was or how to fix it.
-
@wrx7m said:
@DustinB3403 said:
Summary, MS updated something on O365 Exchange that is no longer compatible with Exchange 2010 on premise, using the wizard broke our mail flow connector. I not being an exchange admin (nor office 365 admin) would have known where this issue was or how to fix it.
Exactly.... using the wizard broke our mail flow connector. The on site wizard broke it, not the change to O365.
-
@scottalanmiller Ha! But first MS updated something on O365 Exchange that was incompatible with Exchange 2010. It was then followed by the tech running the wizard, which broke the on-premise side. Insult to injury performed by MS.
-
@wrx7m said:
@scottalanmiller Ha! But first MS updated something on O365 Exchange that was incompatible with Exchange 2010.
You have no proof of that, because the correct troubleshooting steps were not taken. It could easily have been restarting the services or properly reauthenticating. We will never know because the tech broke it immediately.
-
@JaredBusch said:
@wrx7m said:
@scottalanmiller Ha! But first MS updated something on O365 Exchange that was incompatible with Exchange 2010.
You have no proof of that, because the correct troubleshooting steps were not taken. It could easily have been restarting the services or properly reauthenticating. We will never know because the tech broke it immediately.
Right, the only thing that we are nearly sure of is that the tech broke something on the local side. Almost totally certain that this is all MS's fault based on what we know, but the fault of MS support (which we've had terrible experiences with in general ourselves so this follows on what we've seen first hand, but that's pretty anecdotal) but nothing here suggests any reason to think that the issue is in any way on the hosted side. Is there the possibility that it is? Yes, it is possible. Do we have any indicator to make us pin any blame there? No, not without more information and forensics. All of the known issues are on the local side (and caused by an MS tech working locally.)
-
@scottalanmiller It's OK. I can't say that there is enough "evidence" that points exclusively to the on-premise side of things but I guess it doesn't matter. I am still planning on migrating to hosted Exchange.
-
@wrx7m said:
@scottalanmiller It's OK. I can't say that there is enough "evidence" that points exclusively to the on-premise side of things but I guess it doesn't matter. I am still planning on migrating to hosted Exchange.
No, but in this case they stated that there was a break on the local side. We don't have the evidence ourselves, but the people involved pointed to it being a local issue. So what little we know is that it is an on premises problem. Is it possible that an on premises problems was covering up a hosted one too, of course. But we could suppose a lot of things break and get fixed that we don't know about. What is important is that we have no reason to suspect a break on the hosted side.
-
What's also important is that people jump on these threads and see "issue with hosted" and start to assume that on premises never breaks and that hosted always breaks. But often the hosted issues are very isolated (we've had big outages, but limited to our account), are actually misreported local on premises issues (like here) or are just sensational and public when tons of companies have Exchange outages every day on premises but report it to no one. It all makes hosted look far more fragile than it is and on premises look far more reliable than it is.
-
@scottalanmiller I can only go by what the OP said in his summary, which did indicate a change on the 365 side they claimed MS told them was the root cause and then was further borked by a tech that was in over his head.
-
@wrx7m said:
@scottalanmiller I can only go by what the OP said in his summary, which did indicate a change on the 365 side they claimed MS told them was the root cause and then was further borked by a tech that was in over his head.
I read what he said as stating something quite different. I read it as saying that a change on the O365 side exposed a borked on premises install. That explains why we are seeing it differently.
-
@scottalanmiller I agree we are seeing it differently. I still can't see your side after rereading it. LOL - Oh well, agree to disagree as some say.
-
@scottalanmiller said:
What's also important is that people jump on these threads and see "issue with hosted" and start to assume that on premises never breaks and that hosted always breaks.
I wouldn't have thought so. Everything breaks. It would be a very naive IT bod who claims that on-premise never breaks. It's about which breaks the most, or which is more likely to break.
The best thing about hosted is that when it breaks it's normally someone else's fault, whereas when on-premise breaks it is usually my fault. And I always like the ability to deflect blame away from myself!
