This story appears in the November 2023 RE Magazine Cybersecurity Special Insert. RE Magazine subscribers have access to the full insert in the archive.

“This was my boom.”

Jay Suckey has the sinking feeling memorialized in a chat message he saved from 5:13 a.m. Sunday, Nov. 7, 2021: A Delta-Montrose Electric Association dispatcher couldn’t log in. Suckey, DMEA’s system administrator at the time, also was denied access.

It would soon become clear that the Montrose, Colorado-based co-op’s systems were in the middle of a highly invasive cyberattack, a boom that would alter the co-op forever.

Suckey, now DMEA’s IT manager, and Chief Information Officer Bob Farmer told the gutwrenching tale and accompanying four-month comeback at the NRECA Co-op Cyber Tech conference.

As DMEA moves toward full recovery, the two IT professionals are sharing their story as a warning for co-ops to bolster their cyberdefenses to stay left of boom and to provide tips on recovery should disaster strike.

“We are not the same co-op,” says Farmer. “We’re an improved and more secure DMEA because of it.”

Boom!

The ransomware text message outlined the damage done and the potential for more with no option for recourse. In 18 minutes, the hacker had taken down DMEA’s phone and email system and its customer management platform while also disabling the meter data management system, mapping system, file servers and active directory.

DMEA’s backup files were stored offsite miles away but not off network. They were obliterated.

The IT team called the CEO, who informed the board president. They then reported the attack to state and federal authorities, including the FBI and the Department of Homeland Security. They also contacted their primary technology vendor, NISC.

The hacker likely penetrated a single server that had yet to receive the latest security patch. From there, the domain administrator’s login information was stolen, and the attack escalated to “data annihilation,” Suckey says.

In the days and weeks that followed, the co-op learned a “new normal.” Recovery was a moving target, dynamic rather than linear—and at times unknowable.

“There wasn’t a ‘We will be back to normal on such and such day,’” Suckey says. “It was, ‘This is the new way that this will work’ because most systems and processes needed to be rebuilt from the ground up.”

Crawling from the wreckage

DMEA’s electric and broadband services never shut down from the attack, but system operations for the co-op initially functioned at a base level and did not resume full operation until mid-December 2021.

Automated integration between the meter data management system and the customer information system was not restored until January 2022. DMEA initially used the paper service territory maps on its dispatch department walls to track service outages while its outage management system was offline.

Billing was the biggest challenge. Without the ability to grab data off members’ meters, DMEA was unable to send out invoices for November 2021.

“If your meter data management system is down, that means you’re not getting any of the reads from your meters,” says Farmer. “Think about how important billing is for a co-op to survive, and how dependent billing is on meter reads.”

Customer service reps took over data entry, while other staff, including the CEO, worked shifts on the phones. Data entry carried on until the billing was restored at the end of the year.

Basic phone service was up and running by the end of the week, but DMEA’s full phone server wasn’t restored until December. A new cloud-based email system was deployed within several days.

As word of the attack got out, members began flooding the co-op with calls. Staff worked to ensure they had the most accurate and up-to-date information to provide to members about recovery efforts.

“What do you tell the members, and when do you tell them?” says Farmer. “My encouragement would be for co-ops to think about that topic in advance and be prepared ahead of time.”

The overall cost of the cyberattack is difficult for DMEA to quantify, but it involves lost productivity, temporary reputational harm, new equipment purchases and lots of staff overtime.

Suckey recalls several weeks of coming home from the co-op after his kids were already in bed and then leaving in the early morning before they woke up.

“The emotional impact is way more than we realize,” says Farmer. “I have been impressed with how resourceful and resilient DMEA’s employees are, but an event like this certainly takes its toll.”

'Don't go it alone'

Farmer and Suckey say they learned a lot being “right of boom.” First and foremost, “Don’t go it alone. Don’t reinvent the wheel. And be open to feedback.”

Fortunately, DMEA had a cyber insurance policy that included an incident response team to launch its own investigation and recovery efforts.

“Insurance is not a magic fix-all, but it is a huge help during the incident, and it helps you to recover to your new normal,” says Farmer.

In the wake of the attack, the co-op’s CEO also hired a third-party cybersecurity firm, which remains a trusted technical partner today.

DMEA is making use of many of the co-op cyber tools and programs offered by NRECA and participates in tabletop exercises fairly frequently now. That includes some of the most significant cyber drills hosted by NRECA, plus the biennial GridEx held by the North American Electric Reliability Corp.’s Electricity Information Sharing and Analysis Center (E-ISAC).

“Tabletops are a huge help by allowing you to prepare for cyber or physical security incidents,” says Farmer. “They allow you to recognize where you have gaps in your existing plan or when you lack plans entirely. Tabletops provide us with the forum for necessary practice to be ready to better respond to the next incident.”

DMEA also constantly measures its cyber performance against a host of assessments. That’s where it helps to accept the need for better solutions, Suckey adds.

“From the perspective of living through this, the only way that we’re going to learn and grow is to have an open mind and focus on improving ourselves and our organization.”

The co-op has since improved its vulnerability management and patching process with tools for timely automatic upgrades of devices, including iPads and iPhones.

DMEA has started to focus on the nontechnical side of cybersecurity, such as crafting a cyber incident response plan and an IT disaster recovery plan and updating applicable board and administrative policies.

“We’ve gone as far as creating an acceptable use policy for our board to protect them while they use our technology,” says Farmer. “We appreciate their ability to provide valuable feedback and support for our cybersecurity efforts.”

Story of warning

Sharing DMEA’s story isn’t easy, say Farmer and Suckey, but they’re driven to tell it based on the principle of cooperation among cooperatives. CEO Jack Johnston’s belief in the cooperative principles encourages DMEA to continue to keep cybersecurity at the forefront and share its message with others, they said.

“When Jack joined us in January 2023, he certainly could have said, ‘It’s time to move on, let’s forget this happened,’” says Farmer. “Instead, he has embraced it and allowed us to share this important message.”

DMEA hopes that destigmatizing the cyberattack it endured will lead to greater willingness among co-ops to talk about threats, preparedness and response and will make everyone safer in the long run.

“We hope that no other co-op goes through what we went through,” says Farmer. “By telling our story, we hope others share their own incidents and near misses. Each one offers valuable lessons learned, and if co-ops are willing to share, we will all be safer for it.”