We recently assisted a medium-sized hospitality business facing potential network breaches. Find out how we handled it.
Introduction
The client is a successful medium-sized business operating in the hospitality sector with a network of owned or managed outlets across the Northwest. When the client became aware of suspected unauthorised access to its network, Solis was brought in to find out what was going on, and fix it.
Challenge
Initially, this looked like a simple, cut-and-dried case. The intrusion appeared to be limited to one or two servers on the client’s network. Our role was to step in quickly - and kick the ‘threat actor’ off the client’s network before they could do real damage.
Investigation
The client first became suspicious when its security software flagged a couple of unexpected logins, one of which appeared to come from a Russian IP address.
That was the first thing we looked at. But it wasn’t the only thing. At the outset of any investigation, we aim to pull in data from as many sources as possible to get a full picture of any and all unauthorised activity across the client’s IT environment.
We also typically deploy a powerful managed detection response (MDR) tool, SentinelOne, which detects suspicious activity and uses AI to monitor unusual patterns of behaviour which could indicate a threat actor at work. This tool enables us to pick up any new suspicious activity, but it can’t show us what’s already happened. We’re reliant on logs stored on the client’s network for that.
In many cases, checking the logs kept by the client’s antivirus and endpoint detection and response (EDR) products enables us to build up a clear picture quickly. In this case, the client had Microsoft Defender installed on all endpoint devices connecting to its network, and a different product, Sophos Antivirus, running on its servers. Through a third-party managed service provider (MSP), the client also had access to a portal where it could view firewall and virtual private network (VPN) logs.
The downside was that the licences both the client and its MSP held were relatively basic and only allowed access to a very limited amount of data. The use of two different products for endpoints and servers also meant that our ability to follow a particular investigative path went cold where one product ended and the other began.
In a couple of instances, we were able to establish that data might have left the client’s system and gone to a suspicious IP address, but we couldn’t access sufficient data to say for certain what that data might have been, or to be sure exactly what the threat actor was trying to do.
We were able to see that Microsoft Defender had detected an attempt to run an ‘rclone’ command, often used to exfiltrate data from a target organisation’s environment. We could also see that someone had run a tool called CrackMapExec. This is typically used for tasks like mapping an organisation’s network, scanning for users and devices, and gathering user credentials. But, again, we weren’t able to access enough log data to see exactly what the attackers had been trying to do.
Response
One of the first actions we took was to get the client to change all its passwords to prevent further unauthorised logins. We then focused on establishing what the threat actors had accessed - and what data they might have exfiltrated.
During this phase, however, SentinelOne detected that an unauthorised user had compromised an admin account belonging to a managed service provider and was running commands remotely on the client’s environment, setting up scheduled tasks with the aim of transferring data from the client’s network. Again, we acted fast to shut down all unauthorised activity.
As we attempted to find the source of this new (or persistent) unauthorised access, we established that several additional accounts had also been compromised. This suggested that some passwords had not been updated. So that exercise had to be repeated, taking care to include all users.
The client’s network comprised close to 300 endpoints and servers, some of which were only occasionally connected (for tasks such as software updates or remote servicing and repair). Clearly, that’s a lot for a small internal IT than to keep track of. But this investigation underlines the importance of keeping a complete and up-to-date inventory of all connecting users and devices.
Solution
Having upgraded the client’s security product licenses and installed our own tool SentinelOne, we now were able to follow the data trail. The new alerts coming through from SentinelOne had disproved the initial assumption that this was quite a limited attack, affecting just one or two endpoints. Before we shut them out, the threat actors had been able to compromise half a dozen accounts and were well on their way to achieving a broad and persistent infiltration of the client’s network.
Individually examining hundreds of connected devices would have been impracticably time-consuming, but we deployed a data forensics tool that looks at Office 365 logins to see whether it showed up anything suspicious. This came back with an alert relating to an external IT services provider, which suggested that the client’s MSP-managed virtual private network (VPN) had been compromised - and that this was how the threat actors had gained access.
Without a data timeline stretching further back, it was impossible to be completely certain, but it seemed the original point of compromise had been a generically named shared account whose user name the threat actors had correctly guessed and then simply ‘brute-forced’ the password (automatically submitting multiple alternatives until one works).
Our investigations revealed a number of concerning factors (all of which have since been resolved):
-
No multi-factor authentication (MFA) in place for VPN access
-
Devices connected to the client’s network that didn’t relate to registered users (highlighting the need for a network asset inventory)
-
A lack of central control over what software users uploaded to network-connected devices
-
Multiple software vulnerabilities including applications running on endpoint devices not being regularly updated or patched
-
Inadequate data recorded by endpoint detection and response (EDR) tools
-
Gaps created by use of different EDR tools for endpoints and servers respectively.
Outcome
Because the attack was detected at a reasonably early stage, and because we acted fast to shut down unauthorised activity, we were able to prevent the threat actor’s incursions escalating into a full-blown ransomware attack. The activity found suggested an attack in its early exploratory phases and we found no evidence of tooling that could have been used to encrypt data held on the client’s network.
The limited evidence available suggested that any data exfiltrated related to network reconnaissance and gathering user credentials, rather than being anything the threat actors could monetise, for example, by selling on the dark web or making a ransom demand. Any usernames and passwords captured would quickly have become useless, as the client changed all login details several times over during the course of our investigation.
We provided the client with a detailed set of all unauthorised activity we had been able to find, along with comprehensive recommendations on how to make its network more secure. Needless to say, the client was greatly relieved that had been able to stop the attack progressing further. Older and wiser, recommendations acted upon, the client now has the tools in place to detect and detail any evidence of future attacks.