A call comes in one April morning and the voice on the other end says: “Nate we have been hacked. They want $20,000 USD in bitcoin. We have been locked out of our server, and our reservation system is offline. What do we do?”
Cory was the General Manager of a hotel in the San Juan Islands who had a serious problem on his hands and a ticking clock. Pay up, or lose the data forever. After chatting for about 30 minutes to understand what was happening, we quickly understood the severity of the problem, and the potential dollar value attached. With an agreement in place, we hit the road to Friday Harbor to get eyes on the situation.
A series of decade-old machines hummed along as we entered the server room. A ratty patchwork of cables connecting them to the internet via a labyrinthine maze of network devices.
Upon arrival and surveying their server room, it was quickly ascertained the vector of attack hackers took to gain control of their network. Running unpatched operating systems long since abandoned, cluttered with consumer applications unrelated to the tasks they were there to perform. As we suspected, the users of this machine were using a single admin login with full permissions and privileges for personal use through a patchwork of network gear with minimal to no security. And the cherry on top? a piece of paper posted on the wall with current passwords could be visibly seen through a window from the outside of the building. This was a powder keg primed to blow through haphazard one-off hacks to keep things running and is an all too common scenario amongst older establishments. Business owners that don’t understand the ramifications of bad design, failing to see the wisdom in consolidation and upgrading choose to prolong investment until there is no other choice left.
The first step was to get them back up and running with the software they had been using to manage the hotel without reinventing the wheel. This software employed what’s known as a “hub & spoke” design, where a series of client machines connect to a single server for their data across a local network.
As you might imagine, a hotel’s life blood are its reservation books. Standing up a new server would fix things moving forward, but it does not account for past reservations. With the server containing this data being ransomware locked by the hackers, finding any backup record of reservations was of paramount importance. The first place we investigated was any receipts sent over email that might contain this information. Unfortunately, the email hosting they were using had such small limits that the only way to properly function out of them was to set outlook to download the mail, and remove it from the server to make room for new email. This meant that the only option left was to locate a machine with outlook that might have connected to the GM’s user account recently.
Snake eyes – Cory explained that he had just upgraded his machine a week before this all happened due to what later would be diagnosed as a failed hard drive and had not considered migrating his old mail as he considered it not possible.
With some luck and the right tools, we were able to successfully clone the failed drive and fix the boot issue. Once in Windows on Cory’s old machine, we were able to successfully export his mailbox from Outlook and access the resulting .PST file on his new computer, giving him the ability to confidently process guest check-ins with the original email records. It was a manual process, but business continuity had been restored. The hacker’s gambit was for nought.
Now that we had a copy of the reservation data, it was time to restore the hub and spoke management software. Speaking with the producers of the software, it turns out they had just launched a cloud-based version of their software that no longer employed the hub and spoke architecture, but could only be activated via a migration. With no data to migrate, and a pushy sales team insisting that we setup the on-premise software first, we employed some clever negotiating tactics and after some elbow jabbing with the sales guys, we were able to bypass the hub and spoke setup and migration (and the exorbitant fees associated with transitioning to their cloud product.) This meant we were starting fresh on their new cloud based product, passively neutralizing any future attacks on the hotel’s reservations lifeblood. As a bonus, moving to a cloud-based credit card processing service to work with the now cloud-based management system meant that operations for the hotel were now 100% location independent. As long as there was internet, everything would “just work”. No phone lines for the credit card machine to worry about, no server room, no problems.
With day-to-day operations of their reservation software restored, the decision was made to double down on a cloud-first approach in an effort to eliminate any attack surface for hackers to gain a footing. In this case, we looked back on the events that opened the attack vector to begin with which was rooted in individual machines using a single admin logon with no password security. This had to be addressed, and we were prepared with a fix that fixed all others.
After a demo and whiteboard sessions detailing Microsoft365 services and Teams, Cory and company were hooked. This would change everything for them, and preemptively neutralize future attacks on their network. No more single admin logons. Everyone received credentials to tie their actions on the network to their employee ID, and were trained on security basics. All machines were bound to Azure Active Directory, and as a result the network and its machines were now secured with one single global admin with limited permissions granted to staff. Enough to do their jobs, but not enough to compromise the network by installing malicious software.
What came next, was the decision to migrate all data into Teams and permanently dismantle their server room decommissioning or repurposing all devices contained within it. Most of the network gear was redundant and archaic with several desktops being repurposed as guest browsing machines in Kiosk mode, a staff computer in the break room, and a machine now located in the engineering area for research and communications.
With some quick action, clever negotiating and forward thinking, this and future crises of this type were now pre-emptively thwarted. It is an unfortunate reality that sometimes business owners miss the value of a solution until it hits them and they have no alternative but to fix it the right way. As a result, this crisis cost nearly as much to fix as the hackers requested in Bitcoin as a result of round-the-clock emergency work being performed with specific task sets done at night so as not to disrupt business flow. In total, this crisis took nearly a week to fully rectify for which we were on-site. Luckily, they had a spare room for us 🙂
Moral of the story? You don’t negotiate with terrorists. It might cost you just as much to prevent future attacks than it does to pay them, but at least you rob them of the satisfaction of extorting another victim. Finally, make sure your insurance policy has adequate cyberterrorism coverage.