Las Vegas didn’t fold during CrowdStrike outage

LAS VEGAS — When CrowdStrike pushed a few bits of errant code last month, Michael Sherwood, Las Vegas’s chief information officer, watched as seemingly random networks around the city shut down. Meanwhile, digital security tools stayed quiet and it was unclear what — or perhaps who — was the cause of the outage.

“We started seeing what everybody else saw — machines dropping off, going into a blue screen mode,” Sherwood said during an interview at the Black Hat hacker conference. “I’d say for the first half hour, we didn’t have an indication what the problem was.”

It wasn’t until news reports started to come out that Sherwood learned it wasn’t malicious hackers that had burrowed into Sin City’s sensitive networks; it was just a really bad update from the software that was supposed to keep the unwanted out. 

A faulty driver in a CrowdStrike security software update caused millions of Windows machines to crash. Around the world planes were grounded, television stations went quiet, and banks and supermarkets and other vital services shut down.

Las Vegas was no different but did not appear to be as badly impacted. Local reporting highlighted quiet slot machines in casinos and not-so-quiet customers in lines at the Harry Reid International Airport. False rumors even began to spread that the Sphere, the technological marvel of a concert bowl, was hit by the blue screen of death.

The CrowdStrike incident offered a preview of what a devastating cyberattack might accomplish, and Sherwood looks at it as a helpful exercise in anticipation of what’s to come. “We learned a lot. We learned our plans worked,” he said. “We learned a little more about the logistics, and how we’re going to plan out for the future.”

It wasn’t until 3 a.m. the next day that some of those back-end services for critical infrastructure were back up, said Sherwood, who cited security protocols for not sharing more when asked for details about those impacted systems. But safeguards, including multiple vendors and backups, were in place to prevent major disruptions, and within “eight to 10 hours,” most systems were back up and operational, he said.

“By Saturday afternoon, we were completely recovered,” Sherwood said.

In the past few months, the Biden administration has warned critical infrastructure owners and operators to assume compromise by malicious actors and to build resilience for when a worst-case scenario cyberattack occurs. National security officials have warned that Beijing is positioning for possible disruptive attacks against critical infrastructure to destabilize the supply chain or troop movement. The impacts of malware targeting critical systems may look like what happened after CrowdStrike released the update.

Sherwood noted that his job is increasingly having to balance risks found in software, be it extensive testing of security updates or patching immediately to prevent potential intrusions. Consequently, that means increasingly relying on external tools to make real-time decisions, like artificial intelligence and machine learning.

Now, Sherwood said, Las Vegas is looking at how to further diversify its systems and stay resilient in case another service goes down. While Sherwood’s team is staying with CrowdStrike in some areas, he said the city plans on pursuing a “layered approach.” For example, services like Slack and Microsoft 360 were operational during the outage, but that might not always be the case.

“What if that wasn’t available?” he said. “How much harder would it have been to recover from an event like this?”