More than eight months after a ransomware attack severely impacted services at the Resort Municipality of Whistler (RMOW), it remains unclear what the incident cost the municipality—or taxpayers.
While the RMOW has previously said much of the costs were covered by insurance, the full financial impact won’t be shared until the new year, as some items like staff overtime are still being calculated, a municipal spokesperson said after a Technology Advisory Committee (TAC) meeting on Dec. 22.
At the meeting itself, RMOW IT manager Phil Cartwright walked committee members through the cyber security event from a technical perspective, sharing how the municipality handled the late-April attack on municipal servers.
“You hear people say, a cyber breach is not if, but when, and I’m going to reiterate that,” Cartwright said, though he noted that the attackers didn’t technically complete their ransomware attack.
“It’s possible that my team’s actions were taken in a timely manner to prevent any ransomware components, but a portion of our files were accessed and extracted and an extortion attempt was made.”
The RMOW has previously stated that it did not engage with or send any payment to the criminals, believed to be a group known as HelloKitty.
But with the new tactics cyber criminals are using, tracking their actions with any certainty is next to impossible.
“Threat actors are becoming increasingly sophisticated at covering their tracks in deploying anti-forensic measures,” Cartwright said.
“They are deleting logs, rebooting servers, deleting memory and then going after the backups to make detection, attribution and identifying indicators of compromise harder—and they are highly motivated.”
As Pique reported in May, the criminals likely gained access to the RMOW’s servers through a vulnerability found in SonicWall VPN, a service used by the municipality, Cartwright said, though logging limitations make it difficult to verify that.
One of the challenges the RMOW ran up against after the breach was logs that cycle every 30 days, or in some cases 60 or 90 days, Cartwright said.
Attackers can sometimes “lay dormant on systems for months,” he said, adding that “we don’t have a comprehensive picture of their lateral movements [through municipal systems], mostly because of the limitations of logs exceeding 90 days.
“And we don’t know if they were able to leverage other vulnerabilities … [We have] very limited information on what we could confirm.”
System breach
The morning of April 26 started out as a regular day, but turned into one Cartwright will not soon forget, he said.
“We came into the office with reports of various systems being down; definitely concerning, but we weren’t suspicious at that particular moment,” he said.
“We learned that all services had been rebooted at approximately 4 a.m., and that led us to believe a suspected hardware issue. And as the team was investigating, we discovered that our primary backup appliance and subsequent backups on that appliance had been wiped.”
Though a secondary backup was still intact, “at that point, we were immediately suspicious of a breach,” Cartwright said.
The alarm was raised and an incident response team was assembled, which focused on containment and preserving evidence as its first steps.
“It’s really important to note at this point, that if you do go through something like this, it’s really important to not shut systems down, but to disconnect them from the network. The idea here is to preserve memory artifacts in the current state as best as possible,” Cartwright said.
“You should also resist the urge to remove or delete or remediate any suspicious files or software. Again, we want to preserve as much information as possible.”
Next, the municipality reset all its passwords, and took the independent network supporting the wastewater treatment plant and drinking water systems offline.
“That was critical from a public health and safety perspective. We took all precautions to secure that environment immediately,” Cartwright said.
An “IT war room” was established, as well as teams focused on incident response and business continuity.
Working around the clock, staff collected as much intel as they could before conducting an in-depth forensic review of municipal systems.
Following that, the municipality moved into recovery mode.
“This was arguably the biggest body of work for the team, and the first challenge was managing expectations,” Cartwright said, adding that there seemed to be an expectation from some senior staff that the system would be offline for only a couple of days.
“Once we realized that things would be offline for a greater period of time measuring in weeks to months, that changed the temperature around staff and their reliance on technology, and how they could continue providing services to the public in the absence of these critical tools,” he said.
“So it’s really important to get ahead of those expectations early on to support the planning process.”
Part of the recovery included prioritizing what systems needed to be rebooted to support critical municipal processes—some necessary for legal obligations, others for public health and safety requirements.
“So really trying to define where should we be spending our resources to recover these systems to support those critical processes,” Cartwright said.
At the same time, temporary communication tools were put in place, as well as a plan to rebuild and remediate municipal servers.
“We had a healthy level of paranoia at this stage, and one of the biggest things that we wanted to avoid was finding ourselves in a position where we were re-compromised because we missed a tool or avenue to close off that the threat actor could regain access,” Cartwright said.
“So at this point, we were trying to balance the speed in which we recovered with the thoroughness of rebuilding our systems.”
From the ground up
At a high level, the RMOW’s recovery plan was to re-establish basic network services, such as internet access, in a trusted environment (or one known to be clear of potential threat actors).
The process involved rebuilding all municipal workstations from the ground up, and remediating all systems based on priority.
To do that, IT staff created four “zones”: red, yellow, blue and green.
The red zone referred to the RMOW’s existing corporate network at the time of the breach.
“We had zero trust in that zone, and we wanted to make sure that we didn’t contaminate any other areas, potentially, with red-zone devices, servers or workstations,” Cartwright said.
The yellow zone was used to remediate servers from the red zone, providing very restricted internet access to run anti-virus scans or install updates, “and a full battery of tests and evaluations were performed on each server moving through the yellow zone,” he said.
Remediated servers were moved to the blue zone, while the green zone was reserved for all rebuilt servers or systems in which the RMOW had full trust.
“Part of this was the redesign of our network architecture with improved [Virtual Local Area Network] and subnet segmentation,” Cartwright said.
“We moved our default gateways from our core router to the firewall so we had greater visibility across the traffic between segments.”
A majority of the municipality’s core IT services were also rebuilt from the ground up.
“We needed to ensure that these pieces, which were critical to restoring basic network access, were cleaned and trusted,” Cartwright said.
The RMOW wasn’t working alone through the process, he added, noting that the municipality had assistance from its cyber insurance team, IT peers at another municipality, the RCMP and Canadian Centre for Cyber Security, Deloitte consulting and the office of the privacy commissioner.
Eight months after the fact, the municipality considers itself out of recovery mode, Cartwright said, though a handful of internal systems remain out of commission.
“Where this has left us is a backlog of work as a result of the impact of the cyber security breach impacting our 2021 work plan. We’re now factoring in some of the delayed projects into 2022,” he said, adding that staff burnout is another element he’s mindful of.
In the wake of the incident, the RMOW has taken time to review and reflect on the entire ordeal, while looking for ways to strengthen systems and processes to protect the organization from future threats.
“We’ve also taken some opportunities to implement some significant improvements in the architecture and design as a foundational piece to improving our cyber security posture,” Cartwright said.
In terms of key lessons for other organizations, Cartwright recommends having appropriate cyber security insurance; preserving evidence and ensuring you have adequate logging; keeping multiple copies of backups in multiple locations, as well as having a copy offline; seeking the help of partners and experts; and having an incident response plan that can be implemented in a hurry.
The work of the RMOW team left an impression on the other organizations, said general manager of corporate and community services Ted Battiston.
“Every one of those organizations, when they’ve circled back with us, has been absolutely blown away by the quality of work, the dedication, the extra hours, and really the ingenuity that the team brought to this,” Battiston said.
“So I just want to thank you and your team for all the hard work that you put in. It was very impressive during a really tricky time to get through.”
