Logstash vs. NXLog: Best Practices for Collecting Windows Event Logs and CSV Files in a Distributed Environment


2 views

When it comes to log collection in heterogeneous environments, two prominent solutions emerge: Logstash (often paired with Graylog) and NXLog. While both can handle Windows Event Logs and CSV files, their architectures differ significantly.

Logstash operates as a centralized processing pipeline, while NXLog functions more efficiently as a lightweight log forwarder. For Windows systems specifically, NXLog's native Windows service integration often proves more robust than Logstash's Winlogbeat alternative.

In high-volume Windows environments, NXLog typically demonstrates better performance due to its optimized Windows Event Log reader. Here's a basic NXLog configuration for forwarding Security logs:


<Input eventlog>
    Module      im_msvistalog
    Query       <QueryList>\n
                  <Query Id="0">\n
                    <Select Path="Security">*</Select>\n
                  </Query>\n
                </QueryList>
</Input>

<Output tcp>
    Module      om_tcp
    Host        192.168.1.100
    Port        514
</Output>

For CSV files, Logstash provides more flexible parsing capabilities. A sample Logstash configuration for CSV might look like:


input {
  file {
    path => "C:/logs/applications/*.csv"
    start_position => "beginning"
  }
}

filter {
  csv {
    columns => ["timestamp", "user", "action", "status"]
    separator => ","
  }
}

Many enterprises adopt a hybrid model where NXLog handles Windows Event Log collection and forwards to Logstash for processing. This combines NXLog's Windows optimization with Logstash's powerful transformation capabilities.

When using Graylog as the visualization layer, both NXLog and Logstash can feed data into it. Graylog's native GELF format often works better with NXLog for Windows environments:


<Output graylog>
    Module      om_gelf
    Host        192.168.1.100
    Port        12201
</Output>

Key considerations when choosing between these tools include:
- Windows Server count
- Log volume
- Need for complex parsing
- Existing infrastructure
- Team expertise


When building a centralized logging solution for Windows environments, three main contenders emerge: Logstash (part of the ELK stack), Graylog, and NxLog. Each has distinct advantages for log collection and processing.

Logstash: Provides end-to-end log processing with input/filter/output plugins. Written in JRuby, it can handle Windows Event Logs natively using the winlogbeat input:

input {
  winlogbeat {
    port => 5044
  }
}

Graylog: Offers a more turnkey solution with built-in UI and alerting. Uses collectors like Sidecar or NxLog for Windows event forwarding.

NxLog: Specializes in Windows log collection with minimal resource usage. Its configuration for CSV files demonstrates efficiency:

define CSV_HEADER Hostname,EventID,Message

<Input csv_logs>
    Module      im_file
    File        "C:\\logs\\*.csv"
    Exec        parse_csv();
</Input>

For Windows environments, NxLog provides several advantages:

  • Native Windows service integration
  • Efficient binary event log reading
  • Small memory footprint compared to Java-based alternatives

Example NxLog configuration for forwarding to Logstash:

<Extension json>
    Module      xm_json
</Extension>

<Input event_logs>
    Module      im_msvistalog
    Exec        to_json();
</Input>

<Output logstash>
    Module      om_tcp
    Host        192.168.1.100
    Port        5044
</Output>

Logstash shines when handling structured data like CSV files with its powerful filter plugins:

filter {
  csv {
    columns => ["timestamp","level","message"]
    separator => ","
  }
  date {
    match => ["timestamp", "ISO8601"]
    target => "@timestamp"
  }
}

Many enterprises combine these tools:

  1. NxLog as lightweight forwarders on Windows hosts
  2. Logstash for heavy processing and pipeline management
  3. Graylog for visualization and alerting (optional)

Key metrics to evaluate:

Tool Memory Usage Throughput Windows Native
Logstash High Medium No
Graylog Medium High Via Sidecar
NxLog Low High Yes

For your specific requirements of Windows Event Logs and CSV files:

  • Use NxLog for Windows server log collection
  • Process CSV files either directly with Logstash or via NxLog forwarding
  • Consider Graylog if you need built-in dashboards and don't want to configure Kibana

Combining NxLog and Logstash for optimal performance:

# NxLog config (agent)
<Input application_logs>
    Module      im_file
    File        "C:\\app\\logs\\*.csv"
    Exec        $raw_event =~ s/(\r|\n)*$//;
    Exec        if $raw_event =~ /^#/ drop();
</Input>

# Logstash pipeline (server)
input {
  tcp {
    port => 5050
    codec => json_lines
  }
}
filter {
  if [type] == "csv" {
    csv {
      columns => ["date","user","action"]
      convert => { "user" => "string" }
    }
  }
}