Author: ajtap

Whether you’re part of a SOC, CSIRT, a Threat Hunter, or involved in DFIR—you’re likely familiar with the challenge of dealing with various log files from different sources, each with its format.

Logs can come in many forms: structured (like CSV, JSON, ELF, XML), semi-structured (such as Syslog or EVTX), or even unstructured (like CLF). When working with structured logs, we’re fortunate because we can often rely on traditional tools, like our trusty friend Excel, especially when handling CSV files. However, this approach has its limitations. For instance, what happens when the CSV logs you’re dealing with are massive, say 5GB or more, where Excel starts to struggle? Or when you’re facing more complex formats like JSON files or even PCAPs?

This is the reality analysts face—navigating through different log structures where time is of the essence. The diverse formats can make it difficult to correlate the data, turning log carving into a time-consuming and challenging task.

Managing different raw log formats can be overwhelming. The diversity in formats can make data correlation challenging, turning log analysis into a time-consuming and complex task. Is there a tool that acts like a Swiss Army knife for handling these diverse log formats?

Zed Language

This is where ZUI comes into play. ZUI is the graphical user interface for the Zed language, a powerful tool designed to handle various data formats with ease.

According to the Zed website,

 Zed is to remove the tabular and schema concepts from the underlying data model altogether and replace them with a granular and modern type system inspired by general-purpose programming languages. Instead of defining a single, composite schema to which all values must conform, the Zed type system allows each value to freely express its type in accordance with the type system. In this approach, Zed is neither tabular nor semi-structured. Zed is “super-structured”.

The Zed Project takes a different approach to data manipulation and management by using a structured data model that blends aspects of relational tables and JSON. The Zed model is inspired by Linux Bash’s pattern syntax, which makes it flexible and well-suited for working with various data formats. To see how this works in practice, let’s look at a case where it is used in an investigation, starting with some logs stored in the /var/logs directory.

Rapid Triage Windows Event logs with Chainsaw and Zui

For the first use case, imagine you need to quickly triage Windows EVTX files. There’s a tool called Chainsaw that excels at rapidly searching and hunting through Windows forensic artifacts.

For our demonstration, we’ll use Windows Event Attack Samples. The following command shows how Chainsaw can hunt through all EVTX files using Sigma rules for detection logic and save the results in JSON format:

./chainsaw hunt evtx_attack_samples/ -s sigma/ --mapping mappings/sigma-event-logs-all.yml --json

Once the JSON file is generated, you can simply drag it into ZUI to view the data in a SQL-style tabular format, making it much easier to analyze and correlate complex information.

To further refine the data, you can use the following Zed shaper script:

over this | fuse | timestamp:=time(timestamp)

In Zed, this process is called “shaping” the data. Zed’s powerful type system allows for efficient data transformation, including converting field types, adding or removing fields, and reordering them to fit a desired structure.

We can now perform analyses like long-tail analysis or specify the time range of events. ZUI also allows you to filter data using a GUI, similar to Wireshark, making it easy to refine and explore your data interactively.

Triaging Linux logs

Imagine you’re investigating and encounter unstructured raw logs, such as Linux’s auth logs in the /var/logs directory. ZUI can efficiently handle these formats by using a shaper script to extract the necessary fields. For this demonstration, I’ll use an auth.log file containing suspicious login activities. The Zed language offers a function called grok, which parses a string using a pattern and returns a record with the extracted fields. First, we’ll set the data format to line manually, then apply the shaper script.

Here’s a sample script to parse the auth.log file:

yield grok("%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{DATA:process}(?:\\[%{NUMBER:pid}\\])?: %{GREEDYDATA:message}", this) | fuse

This script extracts key fields such as timestamp, hostname, and process from the log entries, transforming the data into a structured format that’s easier to analyze.

With unstructured raw logs like auth.logs, ZUI enables us to perform various data aggregations, such as counting occurrences or filtering specific fields, significantly enhancing our ability to analyze the log data efficiently.

Pcap Analysis

For the final demonstration, one of the most powerful features of ZUI is its ability to analyze PCAP files. ZUI has a built-in tool called Brimcap, which automatically generates Zeek and Suricata summary logs from the PCAP data. In this demonstration, we’ll use a sample PCAP file from January 8, 2024 (Monday), which captures a GOOTLOADER infection.

You don’t need to manually run Zeek and sift through the resulting logs one by one. ZUI automates this process, streamlining your workflow. Additionally, any detection, such as a connection (conn), can be directly opened in Wireshark for deeper analysis. This integration makes ZUI a highly efficient tool for handling complex network traffic investigations.

Conclusion

ZUI, powered by the Zed language, is a valuable addition to your log analysis toolkit. It offers faster triage and log carving capabilities, making it a powerful ally in handling complex data. The only caveat is the need to become familiar with Zed’s syntax, which comes with a moderate learning curve. However, once mastered, it significantly enhances your efficiency in log analysis. While ZUI isn’t a perfect tool, it serves as a robust addition to your arsenal. Stay tuned for more tips and tricks on mastering Zed Language.

References

[1] “The Zui Desktop Application | Zui.” [Online]. Available: https://zui.brimdata.io/docs
[2] “Chainsaw | WithSecure™ Labs.” [Online]. Available: https://labs.withsecure.com/tools/chainsaw
[3] “GitHub – sbousseaden/EVTX-ATTACK-SAMPLES: Windows Events Attack Samples.” [Online]. Available: https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES
[4] “The Zed Project | Zed.” [Online]. Available: https://zed.brimdata.io/docs