Introduction

Manticore allows you to index data collected with Logstash/Filebeat.

To do this, you must have Logstash or/and a Filebeat agent installed on your local machine.

Indexing data with Logstash

Let's look at the following simple example that demonstrates the indexing of the dpkg.log, a standard log file of the Debian package manager:

cat /var/log/dpkg.log

Take note of a possible caveat that need to be dealt with before proceeding further:

• Manticore doesn't support the Log Template Management and the Index Lifecycle Management features of Elasticsearch.
  Since they are enabled by default in Logstash, we need to disable them in the config explicitly.

Also, the hosts option in the output section must correspond to the Manticore's listen port (by default, it's localhost:9308).

cat logstash.conf

After we've modified the config to prevent the above-mentioned issues, we can run Logstash:

logstash -f logstash.conf

Let's wait a few seconds until Logstash finishes its work and then check if log data has been properly passed to Manticore and indexed.

mysql -P9306 -h0

SHOW TABLES;

Let's look at the schema and the content of the testlog_1 table created:

DESCRIBE testlog_1;

SELECT * FROM testlog_1 LIMIT 3\G

Indeed, data from the dpkg log file along with metadata added by Logstash has been correctly indexed by Manticore.

exit;

Indexing data with Filebeat

The alternative way of collecting raw data is the use of a beat agent, e.g., Filebeat.

Here is a simple example of the Filebeat's config:

cat filebeat.yml

Now, we'll store log data in the testlog_2 table so we've updated the setup.templates options accordingly. Also note of setting the allow_older_versions option to true to prevent errors in case you use a newer Filebeat version incompatible by default with the Elasticsearch version emulated by Manticore.

Run Filebeat in the background (we use the strict.perms option here to avoid issues with the config file permissions):

nohup filebeat -c filebeat.yml -e -strict.perms=false 2>/dev/null &

Now when Filebeat has started we'll check if data from the dpkg log is correctly processed and indexed by Manticore.

mysql -P9306 -h0

Run the SHOW TABLES every few seconds to see when the second table is created:

SHOW TABLES;

After we see that testlog_2 has been created, we can run the following queries to make sure that log data has been processed correctly:

DESCRIBE testlog_2;

SELECT * FROM testlog_2 LIMIT 3\G

exit;

Again, original data from the log file along with metadata added by Filebeat has been succesfully indexed.