Indexing data with Filebeat

Filebeat can ship log data directly to Manticore Search using the Elasticsearch-compatible output.

Let's look at a real log file that we'll use as a data source — the dpkg.log, a standard log file of the Debian package manager:

cat /var/log/dpkg.log

Now let's examine the Filebeat configuration:

cat filebeat.yml

Key settings in this config:

• output.elasticsearch.hosts — points to Manticore's HTTP port (9308)
• setup.ilm.enabled: false — disables Index Lifecycle Management, which Manticore doesn't support
• setup.template — controls the index name so Filebeat doesn't try to manage templates
• allow_older_versions: true — prevents version compatibility errors between Filebeat and Manticore

Now let's start Filebeat. We run it in the background using nohup (the strict.perms flag avoids file permission issues):

nohup filebeat -c filebeat.yml -e -strict.perms=false 2>/dev/null &

Let's wait a few seconds and then check if Manticore has received the data:

mysql -P9306 -h0

Run SHOW TABLES to see when the table appears (you may need to wait and retry):

SHOW TABLES;

Once the testlog table appears, let's examine its schema:

DESCRIBE testlog;

And look at the actual data:

SELECT * FROM testlog LIMIT 3\G

The log data from dpkg.log has been successfully indexed by Manticore through Filebeat.

exit;

Querying Filebeat data

Now that we have real log data in Manticore, let's explore what we can do with it.

mysql -P9306 -h0

Full-text search on the log messages:

SELECT * FROM testlog WHERE MATCH('install') LIMIT 5\G

SELECT * FROM testlog WHERE MATCH('configure') LIMIT 5\G

Count total records:

SELECT COUNT(*) FROM testlog;

View table status:

SHOW TABLE testlog STATUS;

exit;

Filebeat version differences

The configuration we used works with Filebeat 7.x. If you use a different Filebeat version, there are a few important differences.

Filebeat 7.x — 8.10:

The most compatible version range. Use type: log for input:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.elasticsearch:
hosts: ["http://localhost:9308"]
allow_older_versions: true
setup.ilm.enabled: false
setup.template.enabled: false

Filebeat 8.11+:

Starting from 8.11, Filebeat enables HTTP compression by default. Manticore does not support compressed input, so you need to disable it with compression_level: 0:

output.elasticsearch:
  hosts: ["http://localhost:9308"]
  allow_older_versions: true
  compression_level: 0

Filebeat 9.0+:

Filebeat 9.0 removed the log input type entirely. You must use filestream instead. Also, fingerprinting is enabled by default and requires files to be at least 1024 bytes. For smaller files, disable it:

filebeat.inputs:
- type: filestream
  id: my-filestream
  paths:
    - /var/log/*.log
  prospector:
    scanner:
      fingerprint:
        enabled: false
output.elasticsearch:
hosts: ["http://localhost:9308"]
allow_older_versions: true
compression_level: 0
setup.ilm.enabled: false
setup.template.enabled: false

For Ubuntu 22.04+ environments, you may also need seccomp settings if you encounter glibc errors:

seccomp:
  default_action: allow
  syscalls:
  - action: allow
    names:
    - rseq