Nobody Tests the Parachute: Microsoft, the FAA, and the Failover That Made it Worse

Network servers in data room

Schedule a Portnox Cloud demo today.

Contents

Last Thursday, a lot of us spent the afternoon refreshing Outlook and wondering if it was just us. It wasn’t; Microsoft 365’s email side was down for the better part of a day. Microsoft just showed the world exactly how crippling a day without business email can be, and a few hours after it gave the all-clear, the FAA looked over and said, “Hold my beer.” Then it stopped every departing flight in the United States for almost five hours, on Friday the 13th no less.

Two outages that close together, both hitting things people can’t work without, and the internet did what the internet does. One senator went on TV with a theory about AI rewriting its own code. Plenty of people assumed hackers. Both organizations said no on both counts, and the preliminary explanations they’ve published are a lot more boring than that. They’re also a lot more useful, if you’re responsible for anything other people depend on, because both outages started with a routine change, and in both cases, the systems built to contain failure are exactly what spread it.

Microsoft: the safety net did exactly what it was told

Microsoft’s preliminary post-incident report is worth reading in full, because it’s unusually candid. The short version goes back to July, when Microsoft shipped a Windows fix built specifically for the gateway servers that sit in front of Exchange Online and a big chunk of M365. It went through their safe deployment process, rolled out globally over several weeks, and then ran clean for more than three months. The catch was in the load testing. Their lab gear was older hardware with fewer cores, and the bug (lock contention inside a Windows process) only really bites on newer, many-core machines under heavy load. So the test passed, and it legitimately passed—but on hardware that doesn’t look like what Microsoft runs where it counts. That’s the kind of gap I’d ding even a mid-size company for in an audit, and Microsoft runs mail for a huge share of the world’s businesses.

Then, early on November 12, an automated failover moved Teams off its alternate gateway and onto the primary one Exchange uses. Microsoft describes that kind of move as normal operations and part of how they stay resilient, and I believe them. At the same time, about 14% of gateway capacity was already offline from an unrelated problem the day before. That was fine on its own too, since there was headroom to spare.

When U.S. business hours peaked, servers started locking up, and the traffic management system did its job: it pulled users off the struggling servers and sent them to healthy ones. That extra load pushed the healthy ones into the same bug, so they locked up too, and the cycle repeated itself across the fleet. In under two hours, more than 80% of the servers were unresponsive. The bulkheads between regions that should have stopped the spread didn’t, for a separate reason Microsoft hasn’t explained yet.

Recovery is where it gets painful to read. Every server they brought back got buried by waiting traffic and fell over again. The normal rollback tools were no help because the bad change was months old. And around 2 a.m. UTC on Friday, part of the mitigation was turning off some of the automated traffic redirection. When your fix list includes disabling the safety net, it’s worth asking what that net was actually tested against.

The FAA: a “routine” job with a 17-year paper trail

The FAA’s story is less complete (the investigation is still open), but the outline is familiar. Its contractor, L3Harris, was swapping old copper lines for fiber at a radio facility in El Dorado, Arkansas, which meant installing a new router. Similar work had wrapped up at a radar site in Amarillo about three weeks earlier. The FAA’s own Q&A makes a point of saying this kind of replacement has been done routinely for decades without trouble.

This time, the network started getting hammered with roughly 1,000 connection attempts every four to five seconds, and according to AVweb’s writeup of the FAA statement, that destabilized things across 17 of the 22 centers that handle traffic en route. Anyone who has run a modern enterprise network is going to look at that number and think it sounds small, and that’s sort of the point. It was enough here, which tells you how little slack that network had.

Controllers described frequencies dropping in and out and having to bounce between primary and backup. The FAA says a backup radio switch normally takes under a minute, and that procedure is built for one frequency going bad. My read (until the final report is released) is that nobody built it for the network underneath both primary and backup getting wobbly at the same time. Things only settled once L3Harris cut off both sites, and the FAA then disabled the same function on about 60 similar routers as a precaution. It also says it will rethink when it schedules this kind of work, and the controllers’ union has been pretty blunt that it belongs in the hours with the fewest planes in the sky.

None of that gets anyone off the hook, though, because this was a boneheaded, self-inflicted wound with a paper trail. In November 2009, a temporary router that an engineer for Harris (L3Harris’s predecessor) had misconfigured during, wait for it, the move onto the new fiber network blocked about three quarters of that network’s routes for five hours and hit 129 facilities. The inspector general’s review the following year said the FAA tended to handle network weaknesses reactively, and an independent panel told the agency to build automated tooling for router config changes plus a model of the network it could test changes against. Then in September of this year, the GAO warned that the sped-up rebuild was booking projects into the same sites without an integrated schedule, buying speed at the risk of disruption. That’s seventeen years of warnings, across every administration in between, and the work still went into a live national network on a weekday afternoon.

We don’t know yet whether Friday came down to a bad config or a bad box. That’s a big question for L3Harris and a much smaller one for the FAA, which owns that network’s resilience no matter whose badge the installer is wearing. Pointing at the contractor from a podium is a fine way to start an investigation but I’d be a lot more interested in hearing what happened to those 2010 recommendations.

Same story, different industries

Back up far enough to see the forest for the trees and these turn out to be pretty much the same outage. A change gets labeled routine, because the last 50 of them went fine. It lands on a system that is a little more stressed than anyone is tracking. Then the mechanism designed to make a failure small (automated redirection at Microsoft, primary/backup switching at the FAA) takes a local problem and hands it to everything nearby. A failover moves load, and if the problem is load, you’ve built a very efficient way to share it.

“Routine” deserves a little suspicion here. It usually means “nothing bad happened last time,” which is a statement about history and says nothing about how much margin you have today. Microsoft’s July fix was routine for three months right up until the day it wasn’t.

So why don’t we test this stuff? Mostly because the tests we run are built to pass. We pull a node, watch traffic move, confirm the dashboard went green, and check the box that says failover works. Far fewer teams ask what happens to the place the traffic lands, or what happens when the thing that failed is the one component every path shares. Microsoft ran a load test; the load test was the wrong shape. I’m in the middle of tabletop planning right now, and the same blind spot shows up in incident exercises constantly. Teams love “the primary goes down” scenario. They almost never script “the backup works perfectly and makes it worse,” because it feels unfair to the people playing, and I’m as guilty of softening scenarios that way as anybody.

Fail open or fail closed, and who decided?

For everything the FAA botched on the way in, it got the response right: it failed closed, on purpose. Its leadership couldn’t rule out a radar or radio site losing contact with aircraft, so it stopped departures. Analysts are putting the airline cost somewhere around $345 million, more than 18,000 flights got canceled, and it was still the right call, because the alternative was guessing with airplanes. Somebody had the authority to make that decision, made it fast, and owned it.

Exchange also effectively failed closed, since almost nobody could get to their email. The difference is that no one chose that outcome during the incident. Microsoft says Exchange can’t fail out of that gateway at all for scale reasons, so its failure mode was settled by an architecture decision made years ago. Teams, OneDrive, and SharePoint could route around the problem and came back sooner. Exchange just had to wait.

Okay, time to land the plane (pun fully intended). For most of us, access to anything (the network, SaaS apps, VPN, sometimes the front door) runs through cloud dependencies: an identity provider, MFA, conditional access, increasingly network authentication too. So when one of those is unreachable at 9 a.m. on a Monday, what actually happens? Does a wired port drop a device into a quarantine VLAN or straight onto production? Do laptops that authenticated an hour ago keep working off a cached decision, or does everything expire and pile back on at once when the service comes up, which is the exact reconnect stampede that kept knocking Microsoft’s servers over? Does the RADIUS “server dead” fallback say permit or deny, and does anybody remember who set it?

In my experience, the honest answer is often, “It fails open, because somebody got yelled at during an outage years ago and quietly changed it.” I don’t think there’s one right answer here. A clinical floor, a factory line, and a guest Wi-Fi network deserve three different plans, and failing closed everywhere just trades a security problem for an availability problem. The wrong answer is the one you learn about in the middle of the incident.

Go break it on purpose

None of this needs a new tool or a big program. Start by listing the three or four cloud services your access decisions depend on, and for each one, write down in plain language what is supposed to happen when it disappears. If two people on your team give different answers, you’ve already found something.

Then go test it during a maintenance window you picked. Block the dependency at the firewall and watch what happens with real devices on the wired side and the wireless side, for people who were already connected and people showing up fresh. Keep watching when you unblock it, because recovery is its own failure mode. Microsoft’s outage dragged on partly because returning servers kept getting flattened, so check whether your clients back off and retry politely or all slam the service at the same second. If your setup can cache recent authentication decisions so known devices ride through a short outage while unknown ones wait, find out how long that cache actually lasts and whether anyone ever turned it on.

Lastly, decide ahead of time who gets to make the FAA-style call. If access is failing open during an outage and someone needs to flip it closed (or the reverse), that person and that procedure should exist before the day you need them. Same goes for your change reviews: next to “What’s the rollback plan?”, add a question about what your automated failover will do if this change goes sideways. It’s a five-minute conversation, and it’s a lot cheaper than finding out on Friday the 13th that someone else picked for you.

Share

About the Author

Picture of Garrett Gross

Garrett Gross

Garrett Gross is Field CISO at Portnox, where he leads pre- and post-sales strategy and serves as the company's public-facing voice, representing Portnox through speaking engagements and press commentary on identity, access, and zero trust.

About the Author

Picture of Garrett Gross

Garrett Gross

Garrett Gross is Field CISO at Portnox, where he leads pre- and post-sales strategy and serves as the company's public-facing voice, representing Portnox through speaking engagements and press commentary on identity, access, and zero trust.

Related Reading

Artificial IntelligenceSecurity Trends

DeepSeek and Ye Shall Find: What the DHS Ban Can’t See

October 29, 2026
Artificial IntelligenceSecurity Trends

Your AI Agent Is Logging In As You, and Your Bank Can’t Tell the Difference

October 26, 2026
Network SecurityNewsSecurity Trends

PHP2Shell: Patch Tonight, Then Figure Out What Your Web Servers Can Reach

October 22, 2026

New research: detection alone isn't enough for AI access.

X