Aaron's Class Projects

IT366 Lab 3 - Setting up an ELK Stack

IT 366 Information Assurance and Security

February 17, 2019

Executive Summary

               Google Cloud Platform can be used to create and manage a VM Instance, which is a Virtual Machine ran on Google’s hardware elsewhere. It can be accessed and managed through SSH. This instance would be a great opportunity to test out the setup of a SIEM program. The ELK Stack is a great, free SIEM program used to examine and manage log files and other system details, and to visualize the results. This stack consists of Elasticsearch (which is a search and analytics engine), Logsearch (which gathers and processes data to send to Elasticsearch), and Kibana (allows users to setup charts and graphs to visualize data). Additional beats, such as Filebeat (which monitors log files), Metricbeat (which collects metrics, the numbers and stats of a system), and Auditbeat (examines audit data which checks the integrity of files) can be added to this system to obtain additional information to get processed. After these programs are all installed, the web interface of Kibana can be used to create graphs and charts with visualizations, and these can be combined and displayed in dashboards. Firewall rules, which allow whitelisting IP addresses that can access the system, helps make this setup secure.

Diagram 1 - Diagram showing the topology of the ELK system setup

 

Setting up Google Cloud Platform

               Using Google Cloud Platform, a computer server can be setup to provide an environment perfect for gathering log data. With this service, physical hardware can be used that is stored at a Google facility. It can be accessed through SSH from the Google Cloud website. To begin using this service, a Google account should be signed into at http://cloud.google.com. In addition, a method of billing should be added as well. This can be inputted by clicking the menu icon on the top left corner and selecting the billing option.

 

Creating an Instance

               An “instance” is a customizable computer that you can purchase that is located somewhere else. It involves setting up a hard drive, memory, and an operating system (such as Linux). On the Google Cloud Platform website, click the menu button then highlight “Compute Engine” and select “VM instances”. Create an instance, configuring it with the settings you would like and within your budget. The name would be the name of your computer (such as ‘it366-cloud-vm”). The region is the physical location where the computer will reside (such as us-west1-b). Note that some regions (such as California) costs more than others so plan accordingly. Set the machine’s CPU and RAM (1 vCPU with 3GB memory) and configure the boot disk (20 GB disk with Debian GNU/Linux 9 OS). When this is set, check that the budget is what you are willing to pay (which will only be charged while the VM is running, when it is stopped, there will be no charge), create the engine, and wait for it to setup.

Image 1 - Configuring the VM Instance

 

Setup Firewall

               While the VM instance is booting up for the first time, the firewall should be configured to allow access to the ELK Stack on a network. In the menu button, highlight “VPC network”, and select “Firewall rules”. On the top of the page, press “CREATE FIREWALL RULE”. Give it a name, such as “home-network” and make sure the direction is set to “Ingress”. This ensures that the IP(s) you will enter will be whitelisted and allowed to access the web interfaces and ports. “Targets” should be set to “All instances in the network” and “Source filter” should be set to “IP ranges”. Enter your IP address in the “IP ranges” (such as “128.187.48.247/32”; your personal IP can be found by simply googling “what’s my ip”). Under “Protocols and ports” enable the ports you will use with the stack (“tcp:80,9200,9300,9600,5601,5044”). Logstash runs on port receives data from 9600, Elasticsearch receives data on port 9200 or 9300, and Kibana runs on port 5601. The beats send data over port 5044. Press “Create” to save the firewall and allow access from your computer.

Image 2 - Configuring the Firewall

 

Starting up the VM Instance

               After configuring the firewall settings, the VM Instance is likely ready to start. Click back on the menu, highlight “Compute Engine”, and select “VM Instances”. The system created should appear in the list. To start the VM, check the box and select “START” on the top of the page. It will begin to bill your Google account. When you are finished using it, and would not need it running in the background, check the button and select “STOP” to avoid additional charges. After the spinning circle next to the instance’s name turns to a green circle, it is ready to use. Take note of the “External IP” as this will be used to connect to the related webpages. Select “SSH” to connect and begin working on the machine.

Image 3 - Starting the VM Instance

Installing the ELK Stack

               The ELK stack consists of Elasticsearch, Logstash, and Kibana (and additional beats). It is a type of a security information and event management (SIEM) software product. These products provide a real-time collection, analysis, and report of security alerts and logs that can be gathered from multiple sources. There are many SIEM products available that provide different levels of functionality, value, and support. LogRhythm is a type of SIEM that is known to be easier to deploy than some other top SIEM products, but it does not support environments with a high volume of events. It is great for small and mid-sized organizations. SolarWinds Log & Event Manager is another SIEM product that is very easy to deploy, has good support and performance, and at a reasonable cost. However, it lacks the fully security options of many other competitors. It is a great choice for larger organizations with limited resources. Splunk Enterprise Security is another SIEM that has high popularity, but is comes at a high licensing cost. It is a good choice for larger organizations. (https://www.esecurityplanet.com/products/top-siem-products.html) The ELK Stack uses Logstash to process data that passes through a system. Beats can be used to collect additional information from a system. The information gathered is then sent through Elasticsearch which is a search and analytics engine. Kibana lets users set up charts and graphs to visualize the Elasticsearch data. (https://www.elastic.co/elk-stack)

 

Installing Elasticsearch

               The first program of the ELK stack to install is Elasticsearch. We will be installing version 6.5.4, evn though it isn’t the newest version available. Regardless of the version being installed, it is important to install the same version to all products in the ELK Stack. Each program will also be installed manually through the Debian package so it does not get automatically updated. First, install Java through the command “sudo apt install default-jdk”. The Debian package for Elasticsearch can be installed through the company’s main website (https://www.elastic.co/guide/en/elasticsearch/reference/6.5/deb.html). Enter in the command “wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.deb” then “sudo dpkg -i elasticsearch-6.5.4.deb”. To set this (and other services) to automatically turn on with bootup, enter the commands “sudo /bin/systemctl daemon-reload” and “sudo /bin/systemctl enable elasticsearch.service”. Start the service by typing either “sudo systemctl start elasticsearch.service” or “sudo service elasticsearch start”. To verify it is running, you can type “sudo journalctl --unit elasticsearch” and “curl -X GET "localhost:9200/"” to see data. Finally, to see nice, location-based data in Kibana visualizations (such as location of IP addreses), you need to enable ingest-geoip. This can be done by typing the commands “cd /usr/share/elasticsearch”, “sudo bin/elasticsearch-plugin install ingest-geoip”, and “sudo bin/elasticsearch-plugin install ingest-user-agent”.

Image 4 - Elasticsearch is Verified Running

 

Installing Logstash

               The next part of the ELK stack to enable is Logstash. This will be performed in a similar method. Type the command “wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.deb” and then “sudo dpkg -i logstash-6.5.4.deb”. Set it to start automatically by “sudo /bin/systemctl daemon-reload” and “sudo /bin/systemctl enable logstash.service”. Begin Logstash with “sudo service logstash start”. Verify it is up and running with “sudo service logstash status”. Logstash will need to be prepared to listen for input from the beats. Go into the Logstash Configuration directory with the command “cd /etc/logstash/conf.d/” and then make a new “.conf” file with “sudo touch <filename>.conf”. Copy and paste the code listed in the appendix named “it366logstash.conf”.

Image 5 - Logstash is Up and Running

 

Installing Kibana

               Once again, Kibana will be installed just like Elasticsearch and Kibana. Type the command “wget wget https://artifacts.elastic.co/downloads/kibana/kibana-6.5.4-amd64.deb” and then “sudo dpkg -i kibana-6.5.4-amd64.deb”. Set it to start automatically by “sudo /bin/systemctl daemon-reload” and “sudo /bin/systemctl enable kibana.service”. Begin Logstash with “sudo service kibana start”. Verify it is up and running with “sudo service kibana status”. To see the web interface, the Kibana yml configuration file needs to be changed. Edit with the command “Sudo nano /etc/kibana/kibana.yml” and change the 6th line down from “#server.host: “localhost” into “server.host: “0.0.0.0””. Kibana can now be opened in the web browser from “<instance external ip>:5601”. If this specific port is forgotten, the command “sudo netstat -plt” will show a list of what ports can be accessed through the system.

Image 6 - Kibana is Up and Running

 

Installing and Configuring Beats

               Beats are used for gathering data to share with Logstash or to send directly to Elasticsearch. They contain configurable modules that can simplify collecting, parsing, and visualizing common log formats. Filebeat is a shipper for log files. It can collect and examine log files from Apache, auditd, NGINX, MySQL, and the System logs. It can even read the logs of Kibana and Elasticsearch to identify problems and what is occurring through them. Metricbeat is another beat that collects metrics from your systems and services. It can monitor system-level CPU usage, memory, network, and disk data and statistics for every process running on the system. This beat is good for keeping track of the “numbers and stats” of a system. Auditbeat is another beat that examines audit data. This collects (and works directly with) Linux audit framework data and it can watch the integrity of files. It collects the same data as “auditd” (in the Linux Auditing System that works closely with the kernel and watches for system calls) and it is also able to be setup to carefully watch directories for any sorts of change. This beat can watch the activities of users and processes on the system. Other beats are also available such as Packetbeat (network data), Winlogbeat (Windows Event Logs), Heartbeat (uptime Monitoring), and Functionbeat (watch cloud services). (https://www.elastic.co/products/beats)

 

Filebeat

               The beats will be installed manually, just like the rest of the ELK stack, to keep the same version number. Like before, download the Debian package with the command “curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-amd64.deb” and install it with “sudo dpkg -i filebeat-6.5.4-amd64.deb”. Run the command “sudo filebeat setup” to prepare Filebeat and to copy templates to Kibana, which will provide easy access to visualizations and Filebeat data. Now that Filebeat is setup, it will need to be configured a little bit to run through Logstash rather than Elasticsearch (to allow for additional transformation and parsing). This is done by editing the Filebeat YML file (the full file can be viewed in the appendix) by typing the command “sudo nano /etc/filebeat/filebeat.yml”. Navigate down until the “output” section and comment out all lines under the Elasticsearch section, and then under the Logstash section, uncomment “output.logstash:” and “hosts: [“localhost:5044”]”. Have the service run at bootup by typing the commands by “sudo /bin/systemctl daemon-reload” and “sudo /bin/systemctl enable filebeat.service”. Begin the service with “sudo service filebeat start”.

               Now that Filebeat is installed, additional modules should be added to it to collect more log files. To see what modules are currently enabled, type in “sudo filebeat modules list”. To add any of the additional modules to this beat, type “Sudo filebeat modules enable system”. The module of “system“ should be enabled, as well as “elasticsearch”, “logstash”, and “kibana” (following the same method). Feel free to add any other modules that would be useful (such as the Apache one if Apache2 is being used on the system). After adding modules, to be safe, restart the Filebeat service with “Sudo service filebeat restart”.

 

Metricbeat

               Metricbeat will be installed the exact same way as Filebeat. Download the Debian package with the command “curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.5.4-amd64.deb” and install it with “sudo dpkg -i metricbeat-6.5.4-amd64.deb”. Prepare Metricbeat and its templates with “sudo metricbeat setup”. Again, edit the YML file (the full version is in the appendix) through the command “sudo nano /etc/metricbeat/metricbeat.yml”. Navigate down until the “output” section and comment out all lines under the Elasticsearch section, and then under the Logstash section, uncomment “output.logstash:” and “hosts: [“localhost:5044”]”. Have the service run at bootup by typing the commands by “sudo /bin/systemctl daemon-reload” and “sudo /bin/systemctl enable metricbeat.service”. Begin the service with “sudo service metricbeat  start”.

               Once again, add modules to Metricbeat to gather system and service metric data. Type in “sudo metricbeat modules list” to see what is already being collected. To add any of the additional modules to this beat, type “Sudo metricbeat modules enable elasticsearch”. Repeat the process to add the module metrics of “logstash” and “kibana”. Any other modules can also be added if you are curious to see its statistics. Restart the service with service with “Sudo service metricbeat restart”.

 

Auditbeat

               Yet again, Auditbeat will be installed just like Filebeat and Metricbeat. Download the Debian package with the command “curl -L -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-6.5.4-amd64.deb” and install it with “sudo dpkg -i auditbeat-6.5.4-amd64.deb”. Prepare Auditbeat and its templates with “sudo auditbeat setup”. Auditbeat is setup slightly different than the other beats, and it typically comes with a configuration file that will not allow it to run properly on the system. To fix this, you simply need to remove this file. Type in “cd /etc/auditbeat/audit.rules.d” and then remove the 32-bit configuration file (if you are using a 64-bit machine) with “sudo rm sample-rules-linux-32bit.conf”.

Auditbeat does provide modules but they are not added the same way as Filebeat and Metricbeat. Instead of going through a modules list, these modules are added directly in the YML file. Navigate to this file with “sudo nano /etc/auditbeat/auditbeat.yml”. To perform all necessary changes, simply delete everything in this file and copy the YML that is provided in the appendix here. First, like before, the outputs should be changed, so comment out all lines under the Elasticsearch section, and then under the Logstash section, uncomment “output.logstash:” and “hosts: [“localhost:5044”]”. Near the top of the YML, uncomment and add additional lines under “auditbeat.modules”. The module “auditd” and “file_integrity” should be added just like in the YML in the sources page. The list of paths under the “file_integrity” module are the directories you wish to monitor. Another module named “system” is available, but only to those using Auditbeat 6.0 and later. Do not try to add this module if running 6.5.4.

Now that Auditbeat is configured properly, have the service run at bootup by typing the commands “sudo /bin/systemctl daemon-reload” and “sudo /bin/systemctl enable auditbeat.service”. Begin the service with “sudo service auditbeat start”.

 

Configuring Kibana Visualizations

               Now that the ELK stack and beats are installed, and they have all been configured to collect desired data and run at startup, you are ready to begin working with this data and visualizing it using the Kibana web interface. On a computer that is within the IP range specified in the Firewall, go to the address “<External IP Address>:5601”, where the External IP Address is the value shown next to the Instance (shown in the “Starting up the VM Instance” section). This should load up the Kibana interface. To verify all the modules are working, click “Management” and then under “Elasticsearch” select “Index Management”. You should see files here from Filebeat, Auditbeat, and Metricbeat. The data of these files can be seen in the “Discover” tab, where filters can be added or by clicking the “*” button you can view only a specific beat’s information. From here, patterns can be observed to create visualizations.

Image 7 - Verifying that all Beats are Available in Kibana

 

Auditbeat Visualizations

               By looking through the Discover tab, you can see what information is collected with auditbeat. Press the ‘*’ dropdown underneath “Add a filter” to change the selection to only “auditbeat-*”. Now, under “Available fields”, you can see the type of gathered data. Select “process.name”, for example, to see a list of the top running processes. This can be built into a word cloud. Select “Visualize” on the left side and the plus button to create a new one. Then select “Tag Cloud” and use the “audtibeat-*” search filter. Under Metrics set the Aggregation to “Count”. Under Buckets, set the Aggregation to “Terms” and Field to “process.name” then you can change the size to 15 to show more processes. To save this visualization, press the save button on the very top of the page, move the slider over to save as a new visualization (uncheck this slider to only rename or make changes to the original visualization), and name it to a title that you will easily be able to find (such as IT366 – Top Processes).

Image 8 - Auditbeat Top Processes Word Cloud VIsualization

 

Metricbeat Visualizations

               Some visualizations are already setup and available to use. For example, Metricbeat provides a visualization for memory usage and cpu usage. When you click on the Visualize tab, you can search for, and find “Memory Usage [Metricbeat System]” with the type of Visual Builder. This premade visualization shows the total amount of RAM and how much is available. Again, you can Save this visualization with a new name to make it easier to find later (check the button “Save as a new visualization” to make a copy under another name). The visualization “CPU usage [Metricbeat System]” with the type of Visual Builder is also a great option to add.

Image 9 - The Metricbeat Memory Usage and CPU Usage Visualizations

               Preset Visualizations can also be adjusted to show the data you want. Initially, the visualization “Requests [Metricbeat HAProxy]” with type Visual Builder doesn’t seem to show any information on the graph. This can be adjusted by changing the Aggregation type. Under the Requests category, which should show the amount of valid requests, but does not, has the aggregation of “Max”. Change this to “Count” to populate the graph with data.

Image 10 - Visualization of Requests (original visualization was altered)

 

Filebeat Visualizations

               Filebeat can show a visualization based on the top sudo commands typed into the VM Instance. This is already premade if you type “Top sudo commands [Filebeat System] and it is of the type Data Table. As can be seen in the Buckets, this gathers data from the field “system.auth.sudo.command”. This can be adjusted by increasing the size to show more commands on the list.

Image 11 - Top Sudo Commands Visualization (Premade)

               Visualizations can also be created to show when a SSH connection was successful. By examining the log information in the Discover tab, you can make the connection that each time a SSH request worked, and only when it works, the “message” shows the key words “Accepted publickey”. You can create a graph visualization to show this by going to the Visualize tab, and add a new one with the type of vertical bar. For Metrics, set the Aggregation to “Count” and under Buckets, set an Aggregation of “Date Histogram” with the Field of “@timestamp”. To filter this to only show the SSH connections, add a filter on the top bar. In the filter, select “message” “is one of” “accepted publickey”. The graph of connections will appear. Another visualization can be made to show a table of times when these connections are made. Create another new visualization with the type of data table. Add the same filter of “message” “is one of” “accepted publickey”. Metric Aggregation should be set to “Unique Count”, with the Field of “@timestamp” and Buckets should be set to show the Aggregation of “Date Histogram” and Field of “@timestamp”.

Image 12 - Searching for "accepted publickey" shows when SSH requests were successful

Image 13 - Accepted SSH Attempts Graph and Table

 

Dashboards

               Now that these visualizations have been created, they can be added to a dashboard to easily see them side-by-side. Like with the visualizations, the Dashboard tab has some preset options that provide lots of information. A neat dashboard to examine is “[Filebeat System] SSH login attempts”. This dashboard shows a lot of information from pre-made visualizations about the time and date of SSH connections and if they were successful. What is particularly interesting here is the list of failed login attempts, where it will show the username used to login and the map of the physical location where these attempts were made. Just like a visualization, the Save button on the top of the screen can be used to rename the dashboard and find it easier later.

Image 14 - Part of the SSH Login Attempts dashboard (premade)

               Another dashboard can be created to show all the visualizations created earlier. Press the Dashboard tab again and click the create new dashboard button. From here, select the Add button on the top and begin by searching the name of a created visualization and clicking on the ones to add (if the visualizations were renamed differently, such as by putting the class name before it, it will be easier to find). Close the add panel and then visualizations can be dragged around to their desired positions. It is very important to press the Save button on the top or else the changes will be lost in the future. If you wish to add a new visualization to a dashboard or change the positioning, press the Edit button on the top first and remember to Save afterwards. It is important to realize that visualizations cannot have custom timestamps attached to them. In the very top right corner, you can edit the timeframe for the data to show. Some visualizations, such as the Memory Usage, CPU Usage, and Requests charts, are easier to see with a smaller timeframe, such as only the last 15 minutes. Other visualizations such as SSH Attempts and top processes work better when you have a larger time range, such as the last 7 days. 

Image 15 - Dashboard of Created Visualizations

 


 

Conclusion and Review

               In this lab, a VM Instance was created with Google Cloud Platform. A firewall rule was established to only allow certain computers access to it. In the instance, the ELK Stack, a type of SIEM, was setup by installing Elasticsearch, Logstash, and Kibana. Filebeat, Metricbeat, and Auditbeat were also installed to gather data from more sources. Visualizations were created using data from each of these beats and these were added to a Kibana dashboard. All together, these make a pretty secure stack. Using Google Cloud Platform, the SSH terminal is primarily opened through the website itself. If you wish to connect through a different system, you would need to download a SSH key to connect. This makes the instance itself as secure as your Google account is. If it has a strong password, and especially if it uses two-step verification, then the data within the instance is well protected. Without the special key, when trying to login through a different SSH client, the connection will automatically be rejected, which is good for security. The Kibana interface can also be connected to securely through the use of the Google Cloud Platform firewall rules. With it, you can limit connections to certain ports which will only allow specified IP addresses to connect. To improve the security here, you should use the very specific IP address value, but to account for subnets, you may wish to add additional source filters so other computers on the network aren’t able to connect. Finally, the security of the data can be further improved if there was the ability to add a password to the web platform. All in all though, the option to whitelist certain IPs to access specific ports, and the SSH key you need to obtain from Google, makes for a well-secured setup.

               My favorite part of this lab was creating the visualizations in Kibana. At first the interface was a little confusing, but once I examined some pre-set visualizations, I got a good feel for how they work. I really enjoyed creating my own and seeing the data show up. In the discover tab, I learned to search for patterns to find what I wanted and then I was able to chart them with a histogram of data. My favorite set of visualizations was included in the pre-set SSH dashboard. The failed login attempts map and usernames was fun, and shocking, to look at. It was also fun to attempt to connect to other kid’s IPs to show up on their map with whatever name I set. My least favorite part of the lab, and definitely the hardest part, was trying to figure out the setup process for everything. There was no clear and direct set of instructions located anywhere (even on the elastic website) and many additional configurations were needed. For example, with Auditbeat, we had to delete the 32-bit rules file (which I couldn’t figure out without the help of others in the class). Enabling the HTTP module in Metricbeat stopped it from working, and it wasn’t until I disabled this module (hours later), that the beat started working again. Trying to use the System module in Auditbeat also stopped it from working, and it wasn’t until I removed this, it worked again (because it is not compatible in 6.5.4). There were many additional instructions that I did not find on tutorials, so it was very difficult to figure out which changes I needed to undo to make my beats work again. I, and many other students, had never installed programs manually with Linux or configured yml files, so having clear instructions where everything is specifically laid out (such as in this writeup) would have been very beneficial.

               As can be seen in the Firewall rules, I have allowed access to 128.187.48.247:

                

Appendix

Code

Elasticsearch YML

(located at /etc/elasticsearch.elasticsearch.yml)

# ======================== Elasticsearch Configuration =========================

# ----------------------------------- Paths ------------------------------------

#

# Path to directory where to store the data (separate multiple locations by comma):

#

path.data: /var/lib/elasticsearch

#

# Path to log files:

#

path.logs: /var/log/elasticsearch

#

 

 

Logstash YML

(located at /etc/logstash/logstash.yml)

# Settings file in YAML

# ------------ Data path ------------------

#

# Which directory should be used by logstash and its plugins

# for any persistent needs. Defaults to LOGSTASH_HOME/data

#

path.data: /var/lib/logstash

#

# ------------ Debugging Settings --------------

#

# Options for log.level:

#   * fatal

#   * error

#   * warn

#   * info (default)

#   * debug

#   * trace

#

# log.level: info

path.logs: /var/log/logstash

#

 

 

It366logstash.conf

(stored in /etc/logstash/conf.d/)

input {

  beats {

    port => 5044

  }

}

 

# The filter part of this file is commented out to indicate that it

# is optional.

filter {

  if [fileset][module] == "system" {

    if [fileset][name] == "auth" {

      grok {

        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",

                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",

                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",

                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",

                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",

                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",

                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }

        pattern_definitions => {

          "GREEDYMULTILINE"=> "(.|\n)*"

        }

        remove_field => "message"

      }

      date {

        match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

      }

      geoip {

        source => "[system][auth][ssh][ip]"

        target => "[system][auth][ssh][geoip]"

      }

    }

    else if [fileset][name] == "syslog" {

      grok {

        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }

        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }

        remove_field => "message"

      }

      date {

        match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

      }

    }

  }

}

 

output {

  elasticsearch {

    hosts => "localhost:9200"

    manage_template => false

    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

  }

}

 

 

Kibana YML

(located at /etc/kibana/kibana.yml)

# Kibana is served by a back end server. This setting specifies the port to use.

#server.port: 5601

 

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

# The default is 'localhost', which usually means remote machines will not be able to connect.

# To allow connections from remote users, set this parameter to a non-loopback address.

server.host: "0.0.0.0"

 

 

Filebeat YML

(located at /etc/filebeat/filebeat.yml)

###################### Filebeat Configuration Example #########################

 

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

 

#=========================== Filebeat inputs =============================

 

filebeat.inputs:

 

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

 

- type: log

 

  # Change to true to enable this input configuration.

  enabled: false

 

  # Paths that should be crawled and fetched. Glob based paths.

  paths:

    - /var/log/*.log

    #- c:\programdata\elasticsearch\logs\*

 

#============================= Filebeat modules ===============================

 

filebeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

 

  # Set to true to enable config reloading

  reload.enabled: false

 

  # Period on which files under path should be checked for changes

  #reload.period: 10s

 

#==================== Elasticsearch template setting ==========================

 

setup.template.settings:

  index.number_of_shards: 1

  index.number_of_replicas: 0

  #index.codec: best_compression

  #_source.enabled: false

 

#================================ General =====================================

 

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

name: elk-01

 

# The tags of the shipper are included in their own field with each

# transaction published.

tags: ["my-elk-stack"]

 

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

 

#============================== Kibana =====================================

 

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

 

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  host: "localhost:5601"

 

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

 

#================================ Outputs =====================================

 

# Configure what output to use when sending the data collected by the beat.

 

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

  # Array of hosts to connect to.

#  hosts: ["localhost:9200"]

 

  # Optional protocol and basic auth credentials.

  #protocol: "https"

  #username: "elastic"

  #password: "changeme"

 

#----------------------------- Logstash output --------------------------------

output.logstash:

  # The Logstash hosts

  hosts: ["localhost:5044"]

 

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

 

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

 

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

 

#================================ Procesors =====================================

 

# Configure processors to enhance or manipulate events generated by the beat.

 

processors:

  - add_host_metadata: ~

  - add_cloud_metadata: ~

 

 

Metricbeat YML

(located at /etc/metricbeat/metricbeat.yml)

###################### Metricbeat Configuration Example #######################

 

# This file is an example configuration file highlighting only the most common

# options. The metricbeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/metricbeat/index.html

 

#==========================  Modules configuration ============================

 

metricbeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

 

  # Set to true to enable config reloading

  reload.enabled: false

 

  # Period on which files under path should be checked for changes

  #reload.period: 10s

 

#==================== Elasticsearch template setting ==========================

 

setup.template.settings:

  index.number_of_shards: 1

  index.number_of_replicas: 0

  index.codec: best_compression

  #_source.enabled: false

 

#================================ General =====================================

 

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

name: elk-01

 

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: ["service-X", "web-tier"]

tags: ["elk-stack"]

 

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

 

#============================== Kibana =====================================

 

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

 

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  #host: "localhost:5601"

 

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

#================================ Outputs =====================================

 

# Configure what output to use when sending the data collected by the beat.

 

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

  # Array of hosts to connect to.

#  hosts: ["localhost:9200"]

 

  # Optional protocol and basic auth credentials.

  #protocol: "https"

  #username: "elastic"

  #password: "changeme"

 

#----------------------------- Logstash output --------------------------------

output.logstash:

  # The Logstash hosts

  hosts: ["localhost:5044"]

 

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

 

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

 

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

 

#================================ Procesors =====================================

 

# Configure processors to enhance or manipulate events generated by the beat.

 

processors:

  - add_host_metadata: ~

  - add_cloud_metadata: ~

 

 

Auditbeat YML

(located at /etc/auditbeat/auditbeat.yml)

auditbeat.modules:

 

- module: auditd

  audit_rules: |

    -a always,exit -F arch=b32 -S all -F key=32bit-abi

    -a always,exit -F arch=b64 -S execve,execveat -k exec

 

    ## Identity changes.

    -w /etc/group -p wa -k identity

    -w /etc/passwd -p wa -k identity

    -w /etc/gshadow -p wa -k identity

 

    ## Unauthorized access attempts.

    -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access

    -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access

 

- module: file_integrity

  paths:

  - /bin

  - /usr/bin

  - /sbin

  - /usr/sbin

  - /etc

 

 

 

setup.template.settings:

  index.number_of_shards: 1

  index.number_of_replicas: 0

 

#tags: ["service-X", "web-tier"]

 

setup.kibana:

  host: "localhost:5601"

 

output.logstash:

  hosts: ["localhost:5044"]

 

processors:

  - add_host_metadata: ~

  - add_cloud_metadata: ~

 

 

Sources

·        https://www.esecurityplanet.com/products/top-siem-products.html - Explanation of SIEMs and popular examples available.

·        https://www.elastic.co/elk-stack - The Elastic Stack website with details and support for all ELK stack programs.

·        https://www.elastic.co/guide/en/elasticsearch/reference/6.5/deb.html – Setup details for Elasticsearch.

·        https://www.elastic.co/guide/en/logstash/6.6/installing-logstash.html - Setup details for Logstash (but install version 6.5.4).

·        https://www.elastic.co/guide/en/elastic-stack-get-started/6.6/get-started-elastic-stack.html#logstash-setup – Running Logstash and preparing it to listen for beats input (configuration file that needs to be copied).

·        https://www.elastic.co/guide/en/kibana/6.5/deb.html - Setup details for Kibana.

·        https://www.elastic.co/products/beats - Details about Beats and what each one does.

·        https://docs.bitnami.com/aws/apps/elk/get-started/understand-default-config/ - Understanding ELK Stack port numbers and what goes to which port.