I’m planning on a talk discussing opensource C2 (command & control) frameworks like Havoc and Mythic. To be complete, there really needs to be a good environment to test and demonstrate against. And to be thorough, this environment should at least have some basic security controls in place. As I keep finding poor documentation online, these blog posts will help to consolodate some lessons learned.

Logging

Logging is the most critical security control to implement. Since it’s a given endpoints will get compromised, a centralized logging system is the only way to get insight into how the compromise occured. Proper logging is also critical to semi-automating common alerts and responses. For this series of blogs, the logging solution is Fluent Bit (log collector) feeding Data-Prepper (a replacement for LogStash) and OpenSearch (Amazon forked ElasticSearch) combined with OpenSearch Dashboards (Amazon forked Kibana).

A simple four virtual machine lab environment will be the reference:

1. Logging VM: Docker containers with data-prepper, opensearch, and opensearch dashboards
2. Windows Server 2022 VM: Simple Active Directory
3. Windows 10 Pro domain joined VM: the main target of our logging adventure
4. Kali Linux VM: Interface to the web front end of the logging system

Logging with Fluent-Bit, Data-Prepper, OpenSearch, and OpenSearch Dashboards

There’s a wide range of free* (with varying levels of ability before incuring charges) and opensource logging solutions. A very common one is Elastic Stack (aka ELK). I’ve decided to base this environment and blog posts on the Amazon forked version called OpenSearch. This is not intended to be advocating for one solution or another. I’m interested in learning how this forked version has changed and what it’s capabilities are. This post should hopefully help others with their initial setup when they’re evaluating it.

See Data-Prepper with Fluent-Bit

Fluent-Bit is our very light-weight log shipper that comes with pre-compiled version for Windows and Linux. Data-Prepper is our LogStash replacement and will enable us to define log pipelines that can standardize our logs and enrich them. It’s also an intermediate stage that can be scaled to accomodate an extremely large system of log collection. The only stage not discussed in these blogs, that would be needed to scale very large, is an intermediate buffering system. Some common solutions for that would be RabbitMQ, Kafka, or others. The logs would then be shipped into OpenSearch (Amazon forked ElasticSearch). At last we would navigate and manage the logs using OpenSearch Dashboards (Amazon forked Kibana).

Setup

A complete step-by-step description of setting up the VMs is beyond the scope of this blog post. I’m going to be focusing on logging.

Throughout this blog series, it is assumed this setup is being used in an isolated test environment. As such, we’ll be disabling some basic security controls the same as default examples. Of course don’t do this in a production system or if you’re going to be running over internet-accessible VMs. The two biggest controls being defered are TLS and usernames/passwords. The default OpenSearch user and password is admin/admin.

Logging VM

After setting up a Debian server VM with Docker, we’ll use the following docker-compose.yml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

version: '3'
services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1 # Nodes eligible to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    networks:
      intnet:
        ipv4_address: 172.29.0.2
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    networks:
      intnet:
        ipv4_address: 172.29.0.3
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
  data-prepper:
    container_name: data-prepper
    image: opensearchproject/data-prepper:2
    networks:
      intnet:
        ipv4_address: 172.29.0.4
    volumes:
      - ./log-pipeline.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml
      - ./data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
    ports:
      - "2021:2021" # Linux pipeline
      - "2022:2022" # Windows pipeline
    depends_on:
      - opensearch-node1

volumes:
  opensearch-data1:

networks:
  intnet:
    ipam:
      config:
        - subnet: 172.29.0.0/24

This is very similar to the example docker-compose.yml file supplied by the OpenSearch project. I’ve added static IPs to facilitate logging from the base OS as well as two different Data-Prepper ports to separate out Linux and Windows loging pipelines.

We also need (at least) two configuration files for Data-Prepper. One is for configuring the service and the other is for defining our logging pipelines. For now, we’ll just use the very basic Data-Prepper configuration that disables SSL (data-prepper-config.yaml):

1

ssl: false

Our pipeline is going to be complicated enough to demonstrate features, but not complete by any means (log-pipeline.yaml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

log-pipeline:
  source:
    http:
      ssl: false
      port: 2021 #default
  processor:
    - grok:
        match:
          log: [ "%{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} %{SYSLOGPROG}: %{GREEDYDATA:data}" ]
    - grok:
        match:
          program: [ "sshd" ]
          data: [ "Accepted %{WORD:login/type} for %{USERNAME:user} from %{IP:source/ip} port %{POSINT:source/port} %{WORD:ssh_version}: %{WORD:crypt_type} %{WORD:hash_type}:%{GREEDYDATA:hash}" ]
# Accepted publickey for debian from 10.10.10.1 port 56586 ssh2: RSA SHA256:5LI78lG+j36pswZip/VN9sNPv9bV28JQOQX+h/EnAf0
  sink:
    - opensearch:
        hosts: [ "https://opensearch-node1:9200" ]
        insecure: true
        username: admin
        password: admin
        index: "logs-linux-%{YYYY-MM-dd}"

windows-pipeline:
  source:
    http:
      ssl: false
      port: 2022
  processor:
    - date:
        match:
          - key: TimeCreated
            patterns: ["YYYY-MM-dd HH:mm:ss -0800"]
        destination: "timestamp"
        locale: "en_US"
#2024-01-25 00:32:04 -0800
  sink:
    - opensearch:
        hosts: [ "https://opensearch-node1:9200" ]
        insecure: true
        username: admin
        password: admin
        index: "logs-windows-%{YYYY-MM-dd}"

There’s a bunch of important things to discuss here. First off, this does not demonstrate chaining pipelines together, which is likely to be needed when standardizing and enriching logs. See this official example for how that works.

This file contains two pipelines. It will be necessary at some point to spread your configuration between many files.

The number one most important thing that must be done in these pipelines is correctly calling out or creating an ISO8601 timestamp. If that is not done, your logs will not be in order and the important and useful features for searching and filtering will not be available. Also, if any logs enter your pipeline without the timestamp, they would then not be visible (in most cases) within Opensearch Dashboards.

The OpenSearch project generally uses ‘/’ notation within variables instead of ‘.’ - although, it will look like a ‘.’ in some places. So when you might want to use something like “login.type” you’ll often need to use “login/type”.

For the Windows log pipeline, make sure to use ‘TimeCreated’ as that is the field Fluent-Bit ‘winevtlog’ inputs will spit out. DO NOT USE ‘winlog’ as that is not compatible with application logs like “microsoft-windows-powershell”. You will not see an error if you try using ‘winlog’, but you will get very wierd and mostly incomplete logs. Technically, the timezone offset should probably be “xx” instead of “-0800”, but I did not get that to work correctly.

A very important and useful feature is to add the current date to your index. This lets you later archive or delete old data which will become critical as you run out of space. This index templating is not documented anywhere I’ve seen, except in the Github issue that originated it.

The last bit needed for the logging VM, is installing Fluent-Bit and creating a configuration file (fluent-bit.conf):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[SERVICE]
  log_level warning

[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/syslog, /var/log/auth.log, /var/log/kern.log
  read_from_head        true

[OUTPUT]
  Name http
  Match *
  Host 172.29.0.4
  Port 2021
  URI /log/ingest
  Format json

Notice we’re using the docker container IP here. That’s important to make sure running Fluent-Bit on the same host as the docker container will work. On any other host, use the IP assigned to the VM’s exernal network interface.

Windows VMs

There’s a lot of configuration I’m leaving out here as it’s not directly needed for logging. I will also assume you’ve installed sysmon. The one piece I will mention is enabling powershell script-block logging. Since our configuration is pulling from “Microsoft-Windows-PowerShell/Operational” where these logs come from. To do that, run

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

get-item -path "HKLM:\Software\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging\EnableScriptBlockLogging"

# Set variables to indicate value and key to set
$RegistryPath = 'HKLM:\Software\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging'
$Name         = 'EnableScriptBlockLogging'
$Value        = '1'
# Create the key if it does not exist
If (-NOT (Test-Path $RegistryPath)) {
  New-Item -Path $RegistryPath -Force | Out-Null
}  
# Now set the value
New-ItemProperty -Path $RegistryPath -Name $Name -Value $Value -PropertyType DWORD -Force 

A simple way to install Fluent-Bit on Windows is to run the following commands (replace the executable filename with whatever version you’ve downloaded):

1
2
3
4
5

.\fluent-bit-2.2.2-win64.exe /S /D=C:\fluent-bit
# create the configuration file \fluent-bit\conf\fluent-bit.conf as described below
sc.exe create fluent-bit binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf"
sc.exe start fluent-bit
sc.exe config fluent-bit start= auto

Make sure to set the logging VM IP under the Output section. This is the fluent-bit.conf file being used on Windows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

[INPUT]
  Name         winevtlog
  Channels     Microsoft-Windows-Sysmon/Operational,Microsoft-Windows-PowerShell/Operational,Security,System,Application,Setup,Windows PowerShell,Microsoft-Windows-Windows Defender/Operational
  Interval_Sec 1
  DB           winevtlog.sqlite

[OUTPUT]
  Name http
  Match *
  Host 10.200.1.237
  Port 2022
  URI /log/ingest
  Format json

If you modify this file later, make sure to stop Fluent-Bit and delete the SQLite database (under \fluent-bit\bin\winevtlog.sqlite), or logs will often not get generated in OpenSearch correctly.

Conclusion

This blog post discussed most of the configuration needed for a basic Fluent-Bit/OpenSearch logging lab. In the next blog post, I’ll be discussing set up of OpenSearch Dashboards and how to use it. Spoiler - it’s still almost identical to Kibana.