Introduction

Manticore Search supports integration with Logstash and Filebeat through Manticore Buddy, which provides Elasticsearch-compatible endpoints. This allows you to use Manticore as a drop-in replacement for Elasticsearch in your logging pipelines.

Supported Logstash versions: 7.6 - 9.2+ Supported Filebeat versions: 7.17 - 9.2+

The key configuration requirements are:

• The hosts option must point to Manticore's HTTP port (default: localhost:9308)
• ilm_enabled must be set to false (Manticore doesn't support Index Lifecycle Management)
• manage_template must be set to false (Manticore doesn't support Log Template Management)

Indexing data with Logstash

Let's index the dpkg.log file — a standard Debian package manager log:

head -5 /var/log/dpkg.log

Here is the Logstash configuration we will use:

cat /logstash.conf

Key settings in the output section:

• hosts => ["http://localhost:9308"] — connects to Manticore's HTTP endpoint
• ilm_enabled => false — disables Index Lifecycle Management
• manage_template => false — disables Log Template Management

Now let's run Logstash to index the log data:

logstash -f /logstash.conf

Once Logstash finishes processing, let's verify the data was indexed in Manticore:

mysql -P9306 -h0

SHOW TABLES;

Let's examine the schema of the table created by Logstash:

DESCRIBE dpkg_log;

And query some of the indexed data:

SELECT * FROM dpkg_log LIMIT 3\G

The log data along with metadata added by Logstash has been successfully indexed by Manticore.

exit;

Indexing data with Filebeat

Filebeat is a lightweight alternative to Logstash for shipping log data. Let's use it to index the same log file into a separate table.

Here is the Filebeat configuration:

cat /filebeat.yml

Key settings:

• compression_level: 0 — required for Filebeat 8.11+ (compression is enabled by default in these versions)
• allow_older_versions: true — required for Filebeat 8.x to work with Manticore
• setup.ilm.enabled: false — disables Index Lifecycle Management
• setup.template.enabled: false — disables template management

Run Filebeat in the background:

nohup filebeat -c /filebeat.yml -e -strict.perms=false 2>/dev/null &

Wait a few seconds for Filebeat to process the log file, then check if the data was indexed:

mysql -P9306 -h0

Run SHOW TABLES to see when the new table appears (you may need to wait a few seconds and re-run):

SHOW TABLES;

Once the dpkg_log_beats table appears, let's check its schema:

DESCRIBE dpkg_log_beats;

And query the data:

SELECT * FROM dpkg_log_beats LIMIT 3\G

The log data has been successfully indexed through Filebeat.

exit;

Version-specific configurations

Logstash and Filebeat configurations vary depending on the version. Here is a reference guide.

Logstash versions

Logstash 7.17 — minimal config, ILM and template management are not enabled by default:

output {
  elasticsearch {
    index => "my_index"
    hosts => ["http://localhost:9308"]
  }
}

Logstash 8.0 - 9.1 — ILM and template management must be explicitly disabled:

output {
  elasticsearch {
    index => "my_index"
    hosts => ["http://localhost:9308"]
    ilm_enabled => false
    manage_template => false
  }
}

Logstash 9.0 - 9.1 — additionally requires superuser mode:

export ALLOW_SUPERUSER=1

Logstash 9.2+ — uses a config file instead of environment variable. Add to /etc/logstash/logstash.yml:

allow_superuser: true

Filebeat versions

Filebeat 7.17, 8.0, 8.1 — may require a seccomp workaround on systems with glibc 2.35+ (e.g. Ubuntu 22.04):

seccomp:
  default_action: allow
  syscalls:
  - action: allow
    names:
    - rseq

Filebeat 8.1 - 8.10 — requires allow_older_versions: true in the output config.

Filebeat 8.11 - 8.19 — additionally requires compression_level: 0 since output compression is enabled by default.

Filebeat 9.0+ — uses filestream input type instead of log:

filebeat.inputs:
- type: filestream
  id: my-log-input
  paths:
    - /var/log/my.log
  prospector.scanner.fingerprint.enabled: false

Note: prospector.scanner.fingerprint.enabled: false is required for files smaller than 1024 bytes.

Best practices

• Always set ilm_enabled: false and manage_template: false for Logstash 8.0+
• Always set setup.ilm.enabled: false and setup.template.enabled: false for all Filebeat versions
• Use Manticore's HTTP endpoint (port 9308) for the hosts setting
• For production, ensure the manticore-extra package is installed (provides Manticore Buddy)
• Manticore auto-creates tables based on incoming data — no need to pre-define schemas