Terminal Terminal | Web Web
Home  //  Play

Manticore indexes replication

Difficulty: Beginner
Estimated Time: 10 minutes

Manticoresearch - Introduction in replication

In this tutorial we will study basics of replication in Manticore Search.

Manticore indexes replication

Step 1 of 4

Shortly about replication

Manticore search daemon can replicate a write transaction in a Real-Time or Percolate index to other nodes in the cluster.

To use replication with the daemon you need to make sure:

  • your build supports replication (on by default in Manticore official builds - https://manticoresearch.com/downloads/)
  • set data_dir option in searchd section of config. This is where incoming indexes are stored
  • set listen option with a range of at least two ports per cluster for the replication protocol
  • set listen option for the SphinxAPI protocol
  • set listen option for the SphinxQL protocol (to execute cluster manipulation statements)
  • optionally set server_id option. The values must be unique across the cluster. When it's not set the server_ids will be auto-generated

You can take a look at the example of Manticore config used for this demo:

cat /etc/sphinxsearch/sphinx1.conf

We have two instances of Manticore search daemon running on different ports (9306 and 9307) here to represent two nodes in our future cluster. The other one's config is:

cat /etc/sphinxsearch/sphinx2.conf

Let's connect to the first one:

mysql -P 9306 -h0

and create a new cluster in it.

Creating new cluster

CREATE CLUSTER posts;

To make sure the cluster's been created successfully, use SHOW STATUS command:

SHOW STATUS LIKE 'cluster%';

You can see the information of our new cluster appeared in the status fields.

Let's insert some data into our testrt index.

INSERT INTO testrt VALUES(1,'List of HP business laptops','Elitebook Probook',10);

SELECT * FROM testrt;

Note that all write statements such as INSERT, REPLACE, DELETE, TRUNCATE that are supposed to change the index in the cluster must use cluster_name:index_name format instead of a plain index_name to ensure the changes are propagated to all the replicas in the cluster. An error will be triggered otherwise (see below). But until an index is in a cluster you can insert into it without any prefix:

INSERT INTO testrt VALUES(2,'List of Dell business laptops','Latitude Precision Vostro',10);

When you try to insert a document into an index within a cluster you need to prepend the cluster name to the index name:

INSERT INTO posts:testrt VALUES(3,'List of Dell business laptops','Latitude Precision Vostro',10);

As the index is not in the cluster yet we see the error. To be able to replicate the new document, we should first add the index to the cluster. It can be done with the following statement:

ALTER CLUSTER posts ADD testrt;

Our testrt index is now in the cluster, which can be checked by looking at cluster_posts_indexes status field (posts being the name of our cluster).

SHOW STATUS LIKE 'cluster_posts_indexes';

Let's insert more data into our testrt index:

INSERT INTO testrt VALUES(3,'List of Dell business laptops','Latitude Precision Vostro',10);

Now we see the error, because the correct query is:

INSERT INTO posts:testrt VALUES(3,'List of Dell business laptops','Latitude Precision Vostro',10);

And it works as expected.

SELECT * FROM testrt;

Joining a node to a cluster

Let's go another other node we have.

exit;

mysql -P 9307 -h0

We know this node is not in our cluster yet so it doesn't contain any data in the testrt index we've added before at the other node. There's no even index testrt

SELECT * FROM testrt;

Let's join to the cluster. To do that, use a JOIN CLUSTER statement. You'll need to specify cluster name (as there may be multiple clusters) and the node you want to connect to:

JOIN CLUSTER posts at '127.0.0.1:9312';

Now let's try again:

SELECT * FROM testrt;

We see that all the index data has been successfully replicated. Let's insert one more value to the index.

INSERT INTO posts:testrt VALUES(4,'List of Dell gaming laptops','Inspirion Alienware',20);

SELECT * FROM testrt;

And now switch back to our first node:

exit;

mysql -P 9306 -h0

SELECT * FROM testrt;

As we see, the changes were replicated successfully.

Removing cluster indexes and clusters

Now we'll remove our index from the cluster:

ALTER CLUSTER posts DROP testrt;

SHOW STATUS LIKE 'cluster_posts_indexes';

It worked!

SELECT * from testrt;

testrt index has become just a local non-replicated index. Let's ensure that:

INSERT INTO testrt VALUES(5,'Lenovo laptops list','Yoga IdeaPad',30);

No warning about missing 'posts:', so 'testrt' is not in the cluster any more.

SELECT * from testrt;

Let' switch to our second node:

exit;

mysql -P 9307 -h0

SELECT * from testrt;

As expected, the changes that we made at the first node were not replicated to the second node.

To remove a cluster completely use DELETE CLUSTER statement:

DELETE CLUSTER posts;

SHOW STATUS LIKE 'cluster%';

The cluster gets removed from all the nodes, but its indexes are left intact and become plain local non-replicated indexes.

SELECT * FROM testrt;

To make sure it's really removed from everywhere we can check the status of the other node:

exit;

mysql -P 9306 -h0

SHOW STATUS LIKE 'cluster%';

And we see that posts doesn't exist here either.