I am trying to write a regular expression of the logstash grok filter for the log of my web application. The format log itself:

2015-12-02 14:5728,763 POST 1.1.1.1 627472 start_one windows 7 x64

Decryption:

Date method address number function system

I can handle the "normal" log

 %{TIMESTAMP_ISO8601:date} %{WORD:method} %{IPV4:client} %{NUMBER:size} %{WORD:def} (?<VerOs>.*)

At the exit we have

 { "date": [ "2015-12-02 14:5728,763" ], "method": [ "POST" ], "client": [ "1.1.1.1" ], "size": [ "627472" ], "def": [ "start_one" ], "VerOs": [ "windows 7 x64" ] }

But with a log in which there is an error or an empty variable, I can not

error log:

 2015-12-3 12:16:43,631 POST 2.2.2.2 637263 error=Bad request - 2015-12-3 12:16:43,631 POST 2.2.2.2 error=No found this address - - 2015-12-3 12:16:46,631 GET - - - 2015-12-3 12:16:46,631 POST 3.3.3.3 del -

I would like to change the names of variables in the field where an error occurred on error and exclude empty fields (indicated by the "-" symbol) from filtering

I want to get something like this. I can change the logging in the system

 { "date": [ "2015-12-02 14:5728,763" ], "method": [ "POST" ], "client": [ "1.1.1.1" ], "size": [ "627472" ], "error": [ "Bad request" ] }

    1 answer 1

    The main problem that I see is that you do not have a fixed log format ie The format of the string with an error and without differ.

    There are two approaches to solving:

    1. Change the format of the log to a uniform one when each element is in its place even if absent.

    For example:

     2015-12-02 14:57:28,763;POST;1.1.1.1;627472;start_one;windows 7 x64 2015-12-02 14:57:28,763;POST;1.1.1.2;;;;error=No found this address
    1. Describe both options and the rules by which to choose them, examples .