Login
Part 13
On Detection: Tactical to Functional
Why a Single Test Case is Insufficient
Introduction
In my previous post, I explored the idea that different tools can implement the same operation chain (behavior) in various ways. I referred to these various ways as execution modalities. In that post, we explored five tools that allowed us to understand some of the most common modalities that one would expect to encounter and concluded with an image of a function call stack that represented the Session Enumeration operation with overlaid tools.
In this post, I want to explore the implications of execution modalities on detection engineering. I’m particularly interested in how diverse modalities affect our ability to evaluate detection coverage. Evaluating detection coverage is a problem we’ve seen rise to industry attention with the ATT&CK EDR Evaluations. While the objective of the evaluations is not necessarily to assess detection coverage, that is undoubtedly a question that industry consumers are interested in, and rightfully so. This post will explore why a test not specifically designed to answer coverage questions fails to provide the necessary evidence. But before we do that, let’s revisit what we learned in the previous post. This refresh will set us up for a hypothetical scenario that will allow us to understand the problems that execution modalities create for us.
NetSessionEnum Function Call Stack
In the previous post, we analyzed the netapi32!NetSessionEnum function to generate its function call stack. Based on our analysis, we know that when an application calls netapi32!NetSessionEnum, it calls the lower level srvcli!NetSessionEnum and ms-srvs!NetrSessionEnum functions behind the scenes. We then identified that we could use the “Session Enumeration” operation to group the functions.
We represent the relationship between these functions as a function call stack below:
Tool Samples
In the previous post, we analyzed five Session Enumeration tool samples; however, we will reduce our scope for this post to just two samples so we can explore how changes to the execution modality impact detectability even if the behavior remains static. I’ve selected two samples, PowerView Get-NetSession and BOF get-netsession, that overlap maximally. I selected these samples because they have the same name (Get-NetSession) and execute the same API function (netapi32!NetSessionEnum). We can, therefore, say they perform the same behavior; however, their authors implemented that behavior via different execution modalities (PowerShell Script vs. Beacon Object File [BOF]). We should expect the rule to detect both samples if we use a behavior-based approach. The problem lies in our definition of behavior-based detection, which I hope this post will help reify. Before we get going, let’s quickly review these particular samples.
Sample 1: PowerView Get-NetSession (PowerShell Script)
The first sample is the Get-NetSession function from Will Schroeder’s PowerView project. Will implemented this sample as a PowerShell script, which confers certain advantages and disadvantages to its users. When we view its source code, we quickly find that the bulk of the script’s interaction with the operating system occurs when it calls the netapi32!NetSessionEnum Windows API function.
Sample 2: get-netsession (Beacon Object File)
The second sample is a BOF called get-netsession, part of TrustedSec’s CS-Situational-Awareness-BOF project. BOFs offer a distinct advantage over PowerShell scripts because they tightly integrate with the agent. As a result, there is no need to spawn a new process, as we typically see during PowerShell execution. Since this BOF is open-source, we can review the code to understand its implementation. Very quickly, we saw a similar call to the netapi32!NetSessionEnum function, so we know that the underlying behavior of this BOF will be identical to that of the first sample we analyzed.
Integrating the Tools into the Function Call Stack
After introducing the samples and performing a quick analysis, we’ve identified that both tools call the netapi32!NetSessionEnum API function. We can now add these samples to our graphic to demonstrate where they reside in the function call stack. Notice that both tools point to the same function, netapi32!NetSessionEnum, which indicates that while the superficial details (e.g., programming language, variable names, etc.) of each tool may differ, they should be considered “functionally equivalent.” Functional equivalency is important because, given the minimal differences between the samples, we expect that a rule written to detect one sample would also detect the other. This resiliency to change differentiates between a “signature” and a “behavior-based detection.” If two tools are equivalent at the functional level, but a rule does not detect both, the rule is not behavior-based. Put another way, the rule focuses on tool-specific details rather than the tool’s behavior.
Detection Rules
Next, we will look at two detection rules. Of course, we all know that there are myriad possible detections. For instance, some rules are built using hashes, some are built using strings or other details associated with malware, and some are more focused on the behavior itself. MITRE Engenuity recently released its Summiting The Pyramid project to explore this proposition. however, for the sake of this particular exercise, we will act as if there are only two possible detection rules. We will use publicly available rules through the Sigma project for this exercise. One rule will depend on the modality, while the other will be behaviorally focused. The goal is not to say that one rule is better; instead, our goal is to understand the tradeoffs between different detection strategies. Let’s take a look at the two detection rule options.
Rule 1: Malicious PowerView PowerShell Commandlets
Rule 1, written by Bhabesh Raj, was written to leverage a potent telemetry source called PowerShell ScriptBlock Logging. If you aren’t familiar with ScriptBlock Logging, I recommend the original PowerShell ❤ the Blue Team article, which describes what it is and how it works. If we look at line 28 of the rule, which is shared below, we see that the rule is also predicated on explicitly including a specific string (Get-NetSession) in the PowerShell ScriptBlock Log. I’ve included a version of the rule below for your review:
https://medium.com/media/fcd09de414418c3a3da2542220fc7044/href
It is helpful to explicitly lay out the conditions a sample must meet for this rule to fire. I’ve listed the alert conditions for Rule 1 below:
1️⃣: PowerShell ScriptBlock Logging (SBL) is enabled
logsource:
product: windows
category: ps_script
definition: 'Requirements: Script Block Logging must be enabled'
2️⃣: The tool is implemented in PowerShell (and therefore triggers SBL)
logsource:
product: windows
category: ps_script
definition: 'Requirements: Script Block Logging must be enabled'
3️⃣: The Get-NetSession string is included in the script
detection:
selection:
ScriptBlockText|contains:
- 'Get-NetSession'
condition: selection
This rule is great for detecting the default implementation of PowerView’s Get-NetSession function because it is named “Get-NetSession” and, therefore, will appear in the ScriptBlock when executed. However, the rule assumes that the attacker will not alter the function’s name AND that the attacker will execute this behavior via a PowerShell script.
Rule 2: SharpHound Recon Sessions
This second rule, by Sagie Dulce and Dekel Paz, actually undersells its capability. It is titled “SharpHound Recon Sessions,” but is generic enough to cover all implementations, so long as they call the NetrSessionEnum RPC Procedure (which, as discussed earlier in this post, is true of all our samples). The rule depends on the Zero Network’s RPC Firewall project to function. It then looks for any process attempting to call the NetrSessionEnum RPC procedure, which the protocol defines as Opnum 12 of the 4b324fc8–1670–01d3–1278–5a47bf6ee188 interface.
https://medium.com/media/cc6a5aa601cf83b64df766b2dc7e3f36/href
The following conditions must be met for this rule to trigger an alert:
1️⃣: RPC Firewall is installed on all relevant processes
logsource:
product: rpc_firewall
category: application
definition: 'Requirements: install and apply the RPC Firewall to all processes with "audit:true action:block uuid:4b324fc8-1670-01d3-1278-5a47bf6ee188 opnum:12'
2️⃣: The RPC Procedure is implemented by the MS-SRVS RPC Interface (4b324fc8–1670–01d3–1278–5a47bf6ee188)
InterfaceUuid: 4b324fc8-1670-01d3-1278-5a47bf6ee188
3️⃣: The RPC Procedure is NetrSessionEnum (OpNum 12)
OpNum: 12
Here, we see that the rule focuses on the execution of a specific RPC procedure, NetrSessionEnum. It may be difficult to immediately understand the viability of such a rule, especially if you are unfamiliar with RPC and its integration into standard Windows API functions. We can refer to the Session Enumeration function call stack, where we see the ms-srvs!NetrSessionEnum RPC procedure sits at the bottom of the stack. Its position indicates that all code paths will eventually result in a call to the procedure, which is good news for defenders who choose this rule.
Rule Analysis
I want to analyze both samples quickly in the context of each rule. The goal is to identify whether the sample(s) meet the conditions of each rule, as described in the previous section. Remember that the sample must meet ALL conditions for the rule to produce an alert.
Note: Both rules rely on telemetry, which may not be enabled by default in your environment. However, for the sake of this post, we will assume that our target environment has enabled all of the necessary telemetry for both rules. This assumption means that each sample will meet Condition 1 for both rules as it evaluates the state of telemetry collection, not the sample or behavior itself.
Sample 1 + Rule 1
1️⃣: PowerShell ScriptBlock Logging (SBL) is enabled ✅
- We have assumed that our hypothetical environment has SBL enabled. The first condition passes.
2️⃣: The tool is implemented in PowerShell (and therefore triggers SBL) ✅
- PowerView’s Get-NetSession is implemented as a PowerShell script. Therefore, it would be subject to SBL, satisfying the second condition.
3️⃣: The Get-NetSession string is included in the script that is executed ✅
- The function is named Get-NetSession, which means the string Get-NetSession would exist in the relevant logs, satisfying the third condition.
Overall Result ✅
- Based on our analysis of the conditions of Rule 1 and the features of Sample 1, I found that Sample 1 WOULD (✅) trigger Rule 1 to produce an alert.
Sample 2 + Rule 1
1️⃣: PowerShell ScriptBlock Logging (SBL) is enabled ✅
- We have assumed that our hypothetical environment has SBL enabled. The first condition passes.
2️⃣: The tool is implemented in PowerShell (and therefore triggers SBL) ⛔️
- TrustedSec’s get-netsession is implemented as a BOF and, therefore, would not be subject to SBL, failing to satisfy the second condition.
3️⃣: The Get-NetSession string is included in the script that is executed ✅
- The BOF’s name is get-netsession; therefore, assuming the string comparison is case-agnostic, the third condition would be satisfied.
Overall Result ⛔️
- Based on our analysis of Rule 1’s conditions and Sample 2’s features, Sample 2 WOULD NOT (⛔️) trigger Rule 1 to produce an alert.
Sample 1 + Rule 2
1️⃣: RPC Firewall is installed on all relevant processes ✅
- We have assumed that the RPC Firewall is installed in the hypothetical environment, satisfying the first condition.
2️⃣: The RPC Procedure is implemented by the MS-SRVS RPC Interface ✅
- The same RPC call specifies OpNum 12, which corresponds with the NetrSessionEnum procedure and satisfies the third condition.
3️⃣: The RPC Procedure is NetSessionEnum ✅
- The same RPC call specifies OpNum 12 which corresponds with the NetrSessionEnum procedure, satisfying the third condition.
Overall Result ✅
- Based on our analysis of the conditions of Rule 2 and the features of Sample 1, I found that Sample 1 WOULD (✅) trigger Rule 2 to produce an alert.
Sample 2 + Rule 2 ✅
1️⃣: RPC Firewall is installed on all relevant processes ✅
- We have assumed that the RPC Firewall is installed in the hypothetical environment, satisfying the first condition.
2️⃣: The RPC Procedure is implemented by the MS-SRVS RPC Interface ✅
- TrustedSec’s get-netsession calls netapi32!NetSessionEnum. According to the function call stack, this function leads to an RPC call to the MS-SRVS Interface, satisfying the second condition.
3️⃣: The RPC Procedure is NetrSessionEnum ✅
- The same RPC call specifies OpNum 12, which corresponds with the NetrSessionEnum procedure and satisfies the third condition.
Overall Result ✅
- Based on our analysis of the conditions of Rule 2 and the features of Sample 2, I found that Sample 2 WOULD (✅) trigger Rule 2 to produce an alert.
Integrating Rules into the Function Call Stack
Now that we understand both rules, we can integrate them into our graph. Remember that both Sample 1 (PowerView’s Get-NetSession) and Sample 2 (BOF get-netsession) call the netapi32!NetSessionEnum function; in other words, the samples are “functionally equivalent.” When two samples are functionally equivalent, we can attribute any difference in detection results to differences in modality or some other tool-specific detail or toolmark.
Recall that Rule 1 is based on PowerShell ScriptBlock Logging, so its detection scope will be limited to implementations that rely on the PowerShell engine. This reliance means it is married to the PowerShell Script modality discussed in the previous post. Our analysis shows that Sample 1 is implemented in PowerShell, but Sample 2 is implemented as a BOF instead of in PowerShell. Therefore, Rule 1 detects Sample 1 but not Sample 2.
Rule 2, on the other hand, is focused on the execution of the ms-srvs!NetrSessionEnum RPC procedure. Based on our understanding of the function call stack, we know that when a sample calls the netapi32!NetSessionEnum function, as both samples do, it will eventually reach the bottom of the function call stack and call ms-srvs!NetrSessionEnum. As a result, both samples will ultimately execute the ms-srvs!NetrSessionEnum procedure and thus trigger the alert. We can confidently say that Rule 1 is focused on modality (PowerShell Script), while Rule 2 focuses on behavior (Session Enumeration).
It is also worth noting that in RPC-based function call stacks, the RPC procedure sits at the base of the stack and, therefore, represents the optimal location for telemetry. Client-side telemetry implementations risk missing direct RPC implementations, like impacket, while server-side implementations lose important client-side context.
I have added both rules to the function call stack in the image below. Notice that Rule 1 is associated with a specific sample or, more generically, with the PowerShell Script execution modality. In contrast, Rule 2 sits at the base of the function call stack (at the RPC procedure). This position in the stack indicates that Rule 2 offers a more holistic approach to detecting this behavior.
Detection Analytic Categories
When we complete this analysis, we find that at least three categories of detection analytics exist.
The first, “tool-based” detections (signatures), is a well-known approach that represents a relatively solved problem. We are all familiar with different ways to build detection rules that focus on the specific details or toolmarks of known malware samples or attacker tools.
With the rise of Endpoint Detection and Response (EDR) products, we saw an effort to move beyond tool-based detections to a new approach that can better deal with the uncertainty that we face. This new category was commonly referred to as “behavior-based” detection. Behavioral detections purported to focus on what the sample does instead of what it is. Unfortunately, the term “behavior” has been poorly defined traditionally within the industry, but in this blog series, I argue that behavior should be considered identical to the operation chain. Therefore, behavioral detections for a tool like PowerView’s Get-NetSession should focus on the Session Enumeration operation.
A problem that I often wrestled with was that detection rules would present themselves as behavior-based because they had moved beyond hashes and static strings, but they didn’t make it to focus on the behavior. This disconnect was evident, especially when Red Teamers migrated from PowerShell tools to C#. It didn’t make sense to me that one could change the programming language used to write malware, which was sufficient to bypass behavior-based detections. The introduction of execution modalities as a concept helps to bridge the gap. A third detection category, which I call “modality-based” detection, seems to exist.
Many modern detection analytics fall into this third modality-based detection category. This focus on modality-based detection strategies can largely be attributed to the poor definition of “behavior” and our lack of resolution, which did not allow us to differentiate between the execution modality and the behavior. I hope this post helps detection engineers make deliberate decisions between modal and behavior detections.
Determining Which Category a Rule Belongs To
The best way to determine which category a detection rule fits into is to execute multiple test cases using samples that emphasize each category. For example, if one were to execute two PowerShell scripts that both call netapi32!NetSessionEnum, but maybe the name of the function was changed, and perhaps some comments with the author’s name were removed, and the detection rule fires for one sample but not the other, so it is likely that the rule is tool-based. If, however, a rule detects both PowerShell scripts but does not detect a BOF that also calls netapi32!NetSessionEnum, then the detection rule is likely modality-based. Finally, suppose the detection rule detects all five tool samples from the previous post. In that case, the rule is likely sufficient for detecting any variation of the behavior and would fit in the behavior-based category.
Understanding that most rules do not fit nicely into one category is essential. Instead, we find that the rule’s logic often involves elements of each approach but is predominantly aligned to one approach. For example, Rule 1, “Malicious PowerView PowerShell Commandlets”, is modality-based due to its reliance on PowerShell ScriptBlock Logging and tool-based due to its use of the string Get-NetSession. Meanwhile, Rule 2 is a pure behavior-based rule due to its location at the bottom of the function call stack.
Modality-based vs. Behavior-based Detection Approaches
An interesting dichotomy exists between modality-based and behavior-based, which makes it difficult to determine whether one approach is “better” than the other. It is possible for a modality-based approach to make executing ANY behavior challenging using that particular modality. For instance, one could imagine a detection engineer or product that learns how to spot proxied RPC. If that were the case, then it would be possible for that solution to eliminate the ability of attackers to execute any behavior via that modality. The downside, of course, is that it would be possible for attackers to execute each behavior using a different modality. It is unclear whether it is possible to eliminate all execution modalities despite the likelihood of fewer modalities than behaviors.
Similarly, we can easily imagine a scenario where a behavior-based approach eliminates a specific behavior regardless of the selected modality. We expect this from a behavior-based detection rule targeting the Session Enumeration operation. In practice, we expect to see some combination of detection approaches, primarily when multiple products exist in many environments.
Example: Detecting the Direct Syscall Modality
Gijs Hollestelle at FalconForce provides an excellent example of modality-based detection in this blog post. Gijs describes his approach to detect direct syscalls (an execution modality) where one looks for syscalls where the calling module is not ntdll.dll, win32u.dll, or wow64win.dll. The thing to notice here is that this detection strategy does not focus on a single behavior, like Session Enumeration, but instead on detecting the use of direct syscalls. This focus makes it a modality-based detection. Again, modality-based detection strategies are fine. The important thing is that we understand the strengths and limitations of whichever strategy we choose to implement.
Why a Single Test Case is Not Enough
I opened the post by explaining that execution modalities complicate our ability to evaluate detection coverage. This problem arises due to the diversity of forms of any given behavior. In his recent post, Luke Paine describes the mathematical underpinnings of detection coverage. He focused on explaining how we can calculate the coverage of a Technique based on the results of its Procedures (operation chains or behaviors). We realized that while this is correct, determining the results of a procedure is a complex problem. As I’ve described, attackers can use numerous execution modalities to implement a procedure, and each modality adds its own wrinkle. This post focuses on the problem of whether it is possible to detect any single variation using many analytics. These analytics range from specific (brittle) to generic (robust) concerning the procedure. It is not always possible to determine where a particular solution exists on the spectrum, especially when evaluating proprietary analytics. You can only build confidence in your procedure coverage when you test the same procedure via many different modalities (built-in tool, PowerShell script, Beacon Object File, Direct RPC).
Your confidence should roughly follow the same function that Luke described for the procedure to technique coverage estimate with the exception that the first test is as good as meaningless because even if you successfully detect it, you have no way of knowing whether your analytic is more like Rule 1 or Rule 2 (as described in this post). Robust detections are resilient to change. This resilience, first and foremost, should be true regarding execution modality. Still, it also should include changes at any level of resolution that is more fine grain than the functional level.
Scenario
Imagine that you are conducting a Red Team assessment. You gain initial access as an unprivileged user and are interested in determining your next step. One common approach is to pursue “user hunting,” where the attacker identifies an interesting target account (e.g., SQL Administrators), typically due to the account’s access to resources germane to the Red Team’s objectives. Once that target user is selected, the attacker must determine where the user account is logged in in the network. The red teamer may enumerate network sessions to glean this information, but they have a choice. They can use PowerView’s Get-NetSession PowerShell function or the get-netsession BOF from TrustedSec’s SA repository.
Note: In real life, there are more than two implementation options. Constraining the possibilities makes the point easier to demonstrate.
The red teamer is a PowerShell fanatic and, as a result, chooses PowerView’s Get-NetSession. Now imagine that later that day, in an exchange with the client, the red teamer is informed that their Session Enumeration was detected. The question is, “What broader assertions can we make as a result of this detection?” Here’s the rub. Remember our two detection rules? Like our two tool implementations, these represent only two of the many ways that either of the tools could be detected. However, I selected these two rules on purpose. Rule 1 only works to detect Sample 1, but Rule 2 works to detect both.
When it comes to dynamic testing, especially in cases where proprietary analytics are in play, we do not necessarily know how the analytic works under the hood. We are using our test cases as a means to reverse engineer or glean the alerting conditions of the analytic itself. When we run a test, the binary result abstracts details of how the analytic works. The test case is either detected (i.e., the rule produces an alert) or not detected (i.e., the rule does not produce an alert). Therefore, when we run our PowerView Get-NetSession test case, if the detection rule generates an alert, we often have no way of knowing whether the first rule caught it (explicitly focused on the tool itself) or the second rule (focused on the underlying behavior and therefore tool agnostic). The best or most efficient way of ascertaining is to use multiple test cases. In a sense, we would then be “triangulating” the analytics logic.
Triangulation
The process of triangulation would look like this. The first test case would be something like executing PowerView’s Get-NetSession function. Assuming it is detected, we now have two example analytics that could be responsible for the alert. We should then select a second test case to eliminate any residual uncertainty (concerning how the analytic works), so we might choose to execute TrustedSec’s get-netsession BOF. The BOF code is almost identical to the Get-NetSession PowerShell function. The difference is that the BOF is written in C and integrates directly into Beacon, eliminating the relevance of PowerShell-specific data sources such as ScriptBlock Logging. In the case of a different result, the similarity between the test cases allows us to pinpoint the cause. If the second test case is not detected, we can infer that a holistic detection rule, like Rule 2, is not in play, and the likelihood that the active rule is similar to Rule 1 increases tremendously.
One final observation is that our ability to triangulate coverage depends on the results from a single detection rule. During this type of testing, the null hypothesis is that all detection rules are tool-based until proven otherwise. If, for instance, we detected both PowerView’s Get-NetSession and the get-netsession BOF, but we depended on different rules to do so, we must assume those rules are tool-based and do not generalize to other implementations. We can, of course, continue testing to determine whether that is, in fact, the truth, but we must operate from the tool-based starting point as it is the most likely result.
Conclusion
In these two posts (Parts 12 and 13), we’ve explored how a behavior implementation matters as much as the behavior itself. We discovered that in the transition from tool-based to behavior-based detections we made an unintended discovery of modality-based detections. These detections complement a behavior-based strategy because they can detect unknown or unaccounted-for behaviors. We identified that it is impossible to evaluate detection coverage using a single test case. This fact makes red team exercises non-ideal for this particular task (although they are quite valuable for other pursuits). Purple team exercises fill this gap because they primarily implement a different testing protocol. One that focuses on presenting multiple test cases. However, the devil is in the details. Consumers should be mindful of understanding the vendor’s testing protocol including how many test cases they will execute per behavior, how those test cases are selected, and whether the selected test cases represent the range of behavioral and modal variability.
On Detection: Tactical to Functional Series
- Understanding the Function Call Graph
- Part 1: Discovering API Function Usage through Source Code Review
- Part 2: Operations
- Part 3: Expanding the Function Call Graph
- Part 4: Compound Functions
- Part 5: Expanding the Operational Graph
- Part 6: What is a Procedure?
- Part 7: Synonyms
- Part 8: Tool Graph
- Part 9: Perception vs. Conception
- Part 10: Implicit Process Create
- Part 11: Functional Composition
- Part 12: Behavior vs. Execution Modality
Part 13 was originally published in Posts By SpecterOps Team Members on Medium, where people are continuing the conversation by highlighting and responding to this story.
The post Part 13 appeared first on Security Boulevard.
