Security teams can automate the discovery of incident-related events by chaining pivoted searches in Splunk or any other event-based data stores, without writing a single line of code.

Fighting “Zombie Workflows”

Combing through security logs and events in response to an alert or a detection takes a lion’s share of time for analysts. Being able to pivot -- to run a search with parameters based on results of a previous search, is vital to quickly discover critical activity around a host or account suspected of compromise.

However, as anyone who spent any time in security operations knows, this process can be downright painful. Some call these manual executions of prescribed steps “zombie workflows,” and the name seems fitting.

In most “machine data” stores, including Splunk, ElasticSearch, or others, complex searches and queries take time, even in an environment tuned for performance. On top of that, there is typically a good number of these canned queries, and their use depends on the prior results in an investigation.

Luckily, this is where CRFT can help. Let’s build an automated flow of conditional Splunk searches to speed things up for security analysts.

Building an Automated Search Pivot

Allow us to present ‘FishTech Financial,' a hypothetical Fin-tech company:

Among other technologies, FishTech uses Splunk Cloud, Slack, and ServiceNow for Incident Management.

For the sake of brevity, let's keep the scenario simple:

The local SecOps team relies on three saved searches in Splunk for daily investigations:

  • 'Detect Persistence' is set up to generate actionable alerts.
  • 'Command and Control' and 'Defense Evasion' searches are launched ad-hoc by analysts to discover additional activity based on the alert.

The analysts’ “playbook” is straightforward:

  1. When the alert is triggered, run the 'Command and Control' search with the Host reported by the alert as a parameter, in order to discover any C2 activity associated with the compromised system.
  2. If such activity is found, open an incident in ServiceNow.
  3. If not, run 'Defense Evasion' search with UserID as a parameter, to discover if the user in the alert has shown any activity attributed to defense evasion.
  4. If events are found, create an incident in ServiceNow.
  5. If not, send a message to the SecOps channel in Slack, acknowledging the investigated alert.

Let’s build it

Building this use case in CRFT is relatively straightforward:

Configure Webhook

We begin by defining the trigger for the workflow. While we could trigger ‘Run Saved Search’ Action with a 'Schedule' Action, we’ll opt for using a webhook instead.

Once it’s dragged onto the canvas in Flow Builder, the ‘Webhook’ Action generates a unique URL. We’ll need it, along with a unique CRFT API key, to configure our Alert in Splunk:

Extract Host

So far, so good. Next, we’ll need to pull the values for our first search from the webhook request body.

We can use a ‘Transform Data’ Action to find and extract the host field we’ll use as a parameter:

In most mature Splunk environments, the number of saved searches could reach hundreds. Our ‘Splunk: Run Saved Search’ Action helps to locate the right one by narrowing down the choices with the app context.

It also supports parameters, so let’s take advantage of that and pass on the token, which contains the extracted host value from the alert:

Add ‘List Has Entries’

Now we need to check if our search produced results. We can quickly evaluate successful empty results by using the Core Action ‘List Has Items’. We’ll use the ‘Data’ object from ‘Run Search’ Action here for evaluation.

Wrapping Up

Now, let’s configure ‘ServiceNow: Create New Task’ Action, and tie it to the green “positive” outcome of the ‘List Has Items’. For now, let's simply dump the Splunk events into a comment.

According to our “playbook”, we should be launching the ‘Defense Evasion’ search in case no data is found for C2 activity. To accomplish this, we need to repeat the 3 previous steps for the UserID and lead the new Actions from the red “negative” outcome of the ‘List Has Items’.

We can also use Slack to let the SecOps team know that we’ve investigated the Splunk alert and found nothing.

Here’s what the completed Flow looks like:

Testing the Results

We can test our Flow by introducing an event that should trigger our alert in Splunk. Uploading a log file containing alert-matching events would do the trick.

After the Alert is triggered, we can now observe a ServiceNow task with the results of the ‘Defense Evasion’ search (pardon the event formatting):

CRFT has triggered the Flow based on the alert sent via webhook, ran the ‘Command and Control’ search, found no results, and then ran and reported on the ‘Defense Evasion’ search.

As a result, we've helped the team to move away from manual, time-consuming tasks to an automated process with high-value results, all in a matter of minutes, and without writing a single line of code.

Thoughts?

We hope you enjoyed our walk-through. Let us know what you think!

If you'd like to take a closer look at CRFT, feel free to add your email to our waiting list here, or on the main website: https://crft.app.

All The Best,

Anton Goncharov