Cyber security investments helped mitigate Johnson City ransomware attack

David Floyd • Oct 22, 2019 at 11:00 PM

Johnson City officials say quick thinking and a pre-emptive investment in city’s cyber security infrastructure helped prevent lasting damage from a ransomware attack on city government early Monday morning.

The city said a message from hackers appeared on city desktops on Monday notifying employees that files on computers had been encrypted. The message asked staff to contact the hackers by email to have the files unlocked. The city said staff did not engage the hackers.

Lisa Sagona, Johnson City’s IT director, said the date and time of the encryption of the files indicates that the attack began at 12:08 a.m. on Monday morning. Sagona said she happened to be in the office at around 5 a.m. on Monday and noticed that one of the files on her computer had been locked. Sagona said she rallied her staff and that the city secured the network within a short period of time.

“Once we assessed, it was easy to determine based on the logs where they got in and shut that down immediately,” she said.

Sagona said the city verified the attack was malicious and started damage control, which included cutting off internet access, turning off all computers, shutting down the email system and disabling outside accounts.

“Once we did that we were comfortable in knowing that the risk was very, very small,” she said. “There was really no way for anybody to get in.”

Because the hackers had indicated the city’s backup system had been infected, Sagona said the city also took a close look at its backup server.

The attack occurred just three weeks after the city rolled out a hyperconverged storage area network, a tool that the city said enabled it to restore files in less than a day. That upgrade cost the city about $650,000, which Sagona said included five years of maintenance and support that accounts for $185,000 of that figure.

Sagona said the city’s financial system and credit card information were not compromised during the attack, and although it did have to complete tasks by paper in some cases, she said city services remained operational.

While the city’s system was down, Sagona said police officers had to issue paper tickets, and staff wasn’t able to handle credit card payments at City Hall for water and tax bills because that system is connected to the internet. In those cases, Sagona said the city directed customers to an online payment system.

“The lingering effect is the painful process of assessing 600-plus computers in 30 physical locations with a small staff,” she said. “That’s the difficult part.”

All affected computers will have to be re-imaged, she said, and although they won’t be a damage to the computers, she said any locked files will have to be addressed at some point. She estimated that the city will have to perform work on about 90% of its computers before everything can be considered back to normal.

The city has prioritized computers that interface with the public, take payments and are involved in permitting, licenses and inspections, she said.

“Everyone mission-critical was brought back online very, very quickly,” she said. “Not even a day. But, it will take a better part of a week to do the assessment and the better part of a few extra weeks to do all of the imaging for every computer. It’s not a quick process.”

Sagona said city staff conferred with members of the city’s police department following the hack, but determined that it would be beyond the jurisdiction of a U.S. law enforcement agency to investigate the perpetrator.

“It is undoubtedly somebody from overseas,” she said. “It is undoubtedly someone who we cannot detect and have no way of doing so.”

Referencing a recent ransomware attack on Atlanta that has cost the city millions of dollars, Sagona said Johnson City recognized that it needed to be proactive.

“It became very clear us that we’re not different from Atlanta other than by size,” Sagona said. “And as you know, so many municipalities and private entities have been attacked. We see local attacks — one or two a month — just from our partners. We were adamant that this was just a matter of when, and we had to be prepared.”