Two days before Christmas 2015, a distribution control center operator at an electric utility in Ukraine watched stupefied as the cursor on his screen moved by itself to a breaker control box and, with two clicks, took a substation off-line. The puzzled operator tried to intervene but was promptly logged off of his computer, and his password was disabled.

When the December 23 cyber attack was complete, three electric distribution utilities in Western Ukraine had been simultaneously taken off-line, leaving 225,000 customers in the dark for up to six hours. A total of 22 substations were shut down, and one of the utilities’ customer service lines was rendered useless by a denial-of-service virus.

Utility employees regained control of the system by disabling the Supervisory Control and Data Acquisition (SCADA) systems, which the hackers had commandeered, and driving to each substation to manually close the breakers.

These coordinated attacks were accomplished by “a sophisticated player” who could have been anywhere in the world, says Tony Thomas, NRECA principal engineer and a certified global industrial cyber security professional. “This wasn’t a kid sitting in a basement.”

Within days of the attack, NRECA’s Business & Technology Strategies (BTS) unit issued an advisory stating the attack was the work of cyber criminals using BlackEnergy, an infamous malware that is “highly modular and under continuous development.”

New analyses of the event by BTS, the Electricity Information Sharing and Analysis Center, and SANS Industrial Control Systems have shed more light on how the attack unfolded.

Investigations confirmed that neither new techniques nor original software were used in Ukraine. An employee responding to a phishing e-mail started a cascade of events that led to the utilities losing control of their operations systems.

What happened in Western Ukraine “could happen to co-ops here today,” says Craig Miller, NRECA chief scientist. “The Ukrainian utilities were not incompetent. They received an e-mail that appeared to be from the Ukrainian government; they did not recognize it as a fake e-mail or phish.”

Evidence suggests the hackers likely breached the utilities’ network up to nine months before the attack, says Cynthia Hsu, NRECA cyber security program manager. “That gave the attackers plenty of time to figure out how to get from the business side of the utility to the industrial-control system side.”

In a more recent BTS advisory on the attack, Miller and other experts conclude that “industry best practices, as widely employed by NRECA members, could have prevented this specific attack from being successful.”

Lessons Learned

In the Ukraine attack, hackers exploited a link between the utilities’ business and industrial-control systems. Once inside, they took their time learning how to access and run the SCADA software.

Thomas says one clear lesson emerges from the Ukraine event: Industrial-control systems like SCADA should be totally separated from the Internet, the cloud, and business enterprise networks. Additionally, any remote-access capability to SCADA should be protected with very strong usernames and passwords and multi-factor authentication.

“SCADA networks should be segregated, and there should be no link to move information from enterprise networks to the SCADA networks,” Thomas says. “The Ukraine hacks are a really good argument why that needs to happen.”

Hsu says co-ops should review their internal network and system connections to determine if any can be used by hackers to gain access to operational networks.

In addition, a co-op’s external Internet presence, like a website or social media accounts, can be used to prepare a targeted cyber-attack, says Barry Lawson, NRECA senior director for power delivery and reliability.

“Look at what you publish out on the Internet, and ask yourself, if you were a bad guy, would any of that help you figure out how to infiltrate a system to do harm?” he says.

He recommends co-ops conduct annual drills to shore up their cyber defenses and ensure they are implementing industry best practices.

Hsu adds: “You need layers of defense to prevent unauthorized access, detect intruders quickly, minimize any damage they can do if they get in, and have a recovery plan to get the system back up and running.” That includes an alert, trained staff, CEO, and board that will deploy and enforce good cyber security policies.

Lawson says the Ukraine attack will remain a bellwether for the industry.

“The big takeaway is, regardless of how the bad guys got in, they got in and caused outages through a cyber event,” Lawson says. “All co-ops should be focused on mitigating cyber risks. We can’t eliminate all risks, but we can act to minimize them and the likelihood of bad results.”