Logstash Load Data with Docker

Today, we will pass our logs to elasticsearch in Json and Csv format with the help of logstash.

Project Structure should be created as in the image.

elasticsearch

logstash.conf

data-json.log

docker-compose

Run the command and move the “data-json.log” data to elasticsearch.

$ docker-compose up
  or
$ docker-compose up -d

You can provide controls with Elasticsearch Rest API.

curl -X GET 'http://localhost:9200/_cat/indices?v' 

To view your index data;

curl -X GET 'http://localhost:9200/your-index-name/_search'

Under the source section, we see the data you type into elasticsearch via logstash. We see that “message”, “timestamp”, “version” fields have been added even though it is not in our log file.

While examining Logstash Filter Plugins, we will examine the removal of these fields by using “remove_field”.

Logstash Filter Plugin

Drop

Used in conjunction with if. When this condition is fulfilled, this document is not attached. For example; Payment Type Sodexo ones are not added.

filter {
   json {
      source => "message"
   }
   if([orderPayment] == "Sodexo"){
      drop {}
   }
}

Mutate

It allows us to make general changes on the fields.We can name, change or remove fields.

Rename:

filter {
      mutate {
        rename => { "orderDate" => "timestamp" }
      }

Update: If there is no field, no action is taken.

filter {
      mutate {
        update => { "orderPaymentType" => "Multinet" }
      }
    }

Coerce: Gives a default value to an empty field.

filter {
      mutate {
        coerce => { "field1" => "default_value" }
      }
    }

Add_Field:

filter {
      mutate {
        add_field => { "Email}" => "cihattpskl@gmail.com" }
      }
    }

Remove_Field:

filter {
      mutate {
        remove_field => [ "message","host","@timestamp","@version"]
      }
    }

We have removed the fields that are not in the log file with remove_field.

Logstash-CSV

If we want to add data in CSV format, we need to make the following updates on “logstash.conf” and “docker-compose.yml”.

logstash.conf:

docker-compose (logstash volume update):

Cihat-TOPSAKAL/Logstash-Load-Data