Jordan Wirfs-Brock / Inside Energy
Steve Metheny, of the Delta-Montrose Electric Association, illustrates the utility’s back-up to the back-up: a floor to ceiling paper map of their service area and infrastructure.
January 21, 2016: Story updated, see below.
In the infamous 2007 Aurora Test, the U.S. Department of Homeland Security simulated a remote computer attack on electric grid infrastructure. In the video of the test, a generator the size of a minibus shakes and sputters. Tiny pieces begin flying off. Then thick black smoke pours out the top and sides, engulfing the entire machine. This was a watershed moment for the electric grid: the DHS showed that it’s possible to destroy physical infrastructure with a cyber attack.
Cybersecurity experts like to divide the world into two categories: Those who have been hacked, and those who have been hacked but don’t yet know it. Electric utilities are being hacked: The Department of Homeland Security’s cyber response team reported 79 cyber incidents in the energy sector in 2014 and 145 in 2013, an average of one every three days. As our grid gets digitized – with technologies like smart meters and automated controls that run power plants and substations – it becomes more and more vulnerable to computer attacks by anyone from activist groups like “Anonymous” to nation-states like China and Russia.
But the utility industry maintains that hackers have not taken out power in this country. “There has never been a cyber attack that has had an impact on the reliability of the grid,” said Scott Aaronson of the Edison Electric Institute, a utility industry interest group. “It has not resulted in a power outage.”
Aaronson’s statement isn’t simply an industry dodge. Cybersecurity experts can point to examples where hackers have infiltrated grid software, or launched ransom attacks on utility companies, but never to the extent of actually creating power disruption. At a hearing before Congress earlier this month, Granger Morgan, an electrical engineering professor at Carnegie Mellon University, said of cyber attacks, “There haven’t actually been any successful ones that I’ve been aware of.”
So how big of a threat do cyber attacks pose to the electric grid?
Jordan Wirfs-Brock / Inside Energy
Since 2009, the Department of Homeland Security has been tracking cyber incidents reported on critical infrastructure like emergency services, water treatment, and communications networks. The electric sector, which includes utilities and power plants, accounts for 40% of the 887 reported attacks.
Get the data: CSV | XLS | Google Sheets | Source and notes: Github
Cybersecurity: Something utilities aren’t eager to speak about
Understanding the risks hackers pose to the grid is challenging because there’s a culture of secrecy in the utility industry, although Aaronson says that’s changing. Still, investor-owned utilities face a Catch-22: Mention potential cyber threats, they risk scaring their shareholders and driving prices down; mention how secure they are, they risk daring potential attackers to prove them wrong.
But investor-owned utilities have a smaller cousin in the electric utility world: Rural electric cooperatives.
The 900 rural co-ops spread across the country are non-profits; they have members, not stockholders. They serve only about 12 percent of the population, but cover three-quarters of U.S. land area. Even though they are rural, they still provide electricity to critical services like hospitals, dams, factories and mines. They have smaller staffs and budgets to deal with cyber attacks than their investor-owned counterparts, which actually makes them more willing to talk about it. They have to. Sharing information and resources is the only way they can survive.
Maurice Martin, a former strategist for the National Rural Electric Cooperative Association, pointed out, “The average time between when a cyber infection happens and the time it is detected is 205 days.” That gives hackers plenty of time to cause trouble.
They could steal customer information – things like credit card numbers and addresses – just like they have from Target or Home Depot. Or hackers could potentially shut off power, damage infrastructure, or injure workers, “if a line were to be energized at a time when a repairman was on it,” said Martin.
A rural co-op’s cybersecurity department is often small or non-existent, which means they have to stretch their resources as far as they can. “Part of the co-op culture is cooperation. It’s enshrined in our principles: cooperatives cooperate with each other,” said Martin.
As a result, “This is one area where co-ops have an advantage,” said Martin. “They are more willing to share information and to talk openly about problems they are seeing, including cyber security problems.”
Behind the scenes: How one rural co-op addresses cybersecurity
The Delta-Montrose Electric Association on Colorado’s Western Slope serves an area where the stars shine so bright it was recently designated an International Dark Sky Park. Though it may be rural, DMEA has had smart grid technologies since 1983. They’ve also made security a priority: Recently, they conducted a “penetration test” to simulate an attack on their systems and identify vulnerabilities.
Steve Metheny, DMEA’s chief of operations, took me behind the scenes to see how the utility’s computer systems work. Standing in one of DMEA’s hydroelectric plants, Metheny pointed out the control panel, which had analog switches and a lattice of boxy status lights you might find on Apollo-era electronics. It also had a digital computer monitor. “This same exact computer screen shows up on our dispatch center, miles away, so they can see exactly what you see here,” said Metheny.
The power plant’s digital data – and all of DMEA’s data, including that from the 32,000 smart meters in homes and businesses – is not transmitted via the Internet or a wireless network. Instead, it travels on hard-wired fiber optic cable. That cable carries data underground, then up along power lines, past cattle feedlots and fields of beans and sweet Olathe corn, to DMEA’s headquarters. Metheny calls it a “closed loop,” which means the only way to tap into the system would be to physically break in.
Few utilities use only this kind of closed loop fiber optic system. Most use a combination of hard-wired and wireless mesh networks. But either way, physical and cyber security are intertwined, and always will be.
Jordan Wirfs-Brock / Inside Energy
Steve Metheny, chief of operations at the Delta-Montrose Electric Association, points to the power lines that also carry fiber optic cable that transfers data from a hydroelectric plant to their headquarters.
The Delta-Montrose Electric Association hydroelectric turbine draws water from a canal built in 1909, and sits across from the crumbling ruins of a testbed for Hoover Dam components, but the plant itself is less than five years old. It’s outfitted with digital sensors and computer monitoring systems. But if DMEA’s computers got hacked, they have back up. They could shut all of those systems off and the hydroelectric plant would still run. It has physical systems, like a gravity-powered shutoff in the canal, as a back-up. “The plant would operate just fine without any digital controls at all,” Metheny said.
Jordan Wirfs-Brock / Inside Energy
A view from inside the dispatch control room at Delta-Montrose Electric Association in Montrose, Colorado.
He said that’s also true of the infrastructure that delivers power to customers: “It looks like an amazing control center, and it is. We’ve got our alarm systems here, we can monitor our power flow, we can monitor our power plants, we can tweak things.” But throw the breaker to that control room, and the lights stay on, he said.
The difference is, “All of a sudden it’s like we used to be. Until somebody from Grand Mesa calls in and says, ‘I don’t have any power.’ Well where are you?”
Metheny walked over and pointed out a paper floor-to-ceiling map of their service area. “That’s our ‘back-up back-up.’ And we still keep it.”
The most important things to know about cybersecurity and the electric grid
Not all electric utilities are as prepared for a cyber attack as DMEA. But DMEA’s ability to fall back on its older, pre-computer systems illustrates one of the grid’s paradoxes: Our 21st century smart grid is built right on top of 20th century technology that never went away. The grid is fragmented, diverse and old. Oddly, that’s also what makes is hard for hackers to attack.
That property, “security by obscurity,” is disappearing, noted Utilities Telecom Council cybersecurity expert Nadya Bartol in her testimony to Congress. As utilities update their older systems to newer, digital ones, they often are using third-party software so there’s more overlap from utility-to-utility, which makes things easier for hackers.
Still, the grid itself is designed to survive a lot of stress and keep running – whether that stress comes from falling branches or computer hackers.
“I know how to really cause a lot of disruption and inconvenience with cyber attacks,” said Granger Morgan before Congress. “But I don’t know how to cause large spread physical damage.”
Andy Bochman, a grid security expert with Idaho National Laboratory, compares the threat posed by hackers to that of a large hurricane. But Bochman says there’s a key difference: “With cyber, you don’t see it coming, you don’t know what it is at first. In fact, the fact that it’s taking some of your systems out may make it even harder to know it’s a cyber attack.”
And unlike storms, hackers are constantly getting smarter.
“As the defenders get stronger, the attackers see that, and they develop ways around some of those defenses,” said Bochman. “It’s a real cat and mouse game.”
Finding people with the right training to keep up with that cat and mouse game is a real challenge. Utilities have to deal with protecting their business information, including the personal information of their customers, and protecting the software systems that operate power plants and substations. When you consider those qualifications, “Now you’re really getting down to a comparative handful of people in the world,” said Bochman. “And we are trying to grow them as fast as we can.”
Even the U.S. military has had trouble finding staff with the qualifications to prepare the grid for cyber attacks.
It’s impossible, and prohibitively expensive, to eliminate cyber threats entirely. So while hackers won’t launch us into a Mad Max scenario, dealing with them will be a constant, uphill battle. Which brings us back to that question: If utilities are constantly reporting cyber attacks, how come no one can point to an example of a hacker taking out the power?
Aaronson of the Edison Electric Institute said this: “Think of it as the thief walking down the street jiggling doorknobs, looking for a crime of opportunity.”
Hackers are inside the software that runs our grid, snooping around, watching, and waiting. But for now, at least, that’s all they are doing.
Update: Since publication of this story, the first known cyber attack to cause a power outage has been reported. The attack occurred in Ukraine in December 2015 and resulted in 80,000 customers losing power.
What’s Next
- Search Inside Energy’s database of major power outages based on information from the Department of Energy, which includes several incidents labeled as cyber attacks. Those incident reports either said no customers were affected, or excluded customer information.
- Watch the recent Congressional hearing on the electric grid’s vulnerabilities, including cyber attacks and solar storms, and read the testimony from experts.
- The Department of Energy has itself been the target of cyber attacks. Read about it in USA Today’s recent investigation.
- Read other stories in our series, Blackout: Reinventing The Grid.
