Home  //  Play

Manticore indexes replication

Difficulty: Beginner
Estimated Time: 10 minutes

Manticoresearch - Introduction in replication

In this tutorial we will study the basics of replication of percolate indexes in Manticore Search.

Manticore indexes replication

Step 1 of 4

Shortly about replication

Manticore search daemon can replicate a write transaction in a percolate index to other nodes in the cluster.

To use replication with the daemon:

  • build the daemon with replication support (this option is enabled in the Manticore builds by default)
  • set a data_dir option in searchd section of config. This is where incoming indexes are stored.
  • set a listen option with a range of at least two ports per cluster for the replication protocol
  • set a listen option for the SphinxAPI protocol
  • set a listen option for the SphinxQL protocol (to execute cluster manipulation statements)

You can take a look at the example of Manticore config used for this demo:

cat /etc/sphinxsearch/sphinx1.conf

We have two instances of Manticore search daemon run on different ports (9306 and 9307) here to represent two nodes in our future cluster.

Let's connect to the first one:

mysql -P 9306 -h0

and create a new cluster at it.

Creating new cluster

CREATE CLUSTER posts;

To make sure the cluster's been created successfully, use SHOW STATUS command:

SHOW STATUS LIKE 'cluster%';

You can see the information of our new cluster appeared in status fields.

Let's insert some data into our pq index.

INSERT INTO pq VALUES('value 1');

SELECT * FROM pq;

Note that all write statements such as INSERT, REPLACE, DELETE, TRUNCATE that change the index in the cluster should use cluster_name:index_name instead of a plain index_name to ensure the changes are propagated to all the replicas in the cluster. An error will be triggered otherwise.

INSERT INTO posts:pq VALUES('value 2');

As we see, the cluster we've created doesn't know about our pq index so far. To be able to replicate it, we should add the index to the cluster. This can be done with the following statement:

ALTER CLUSTER posts ADD pq;

Our pq index is now in the cluster, which can be checked by looking at cluster_posts_indexes status field (posts being the name of our cluster).

`SHOW STATUS LIKE 'cluster_posts_indexes;'

Insert some more data into our pq index.

INSERT INTO posts:pq VALUES('value 2');

And it works as expected.

SELECT * FROM pq;

Joining a node to a cluster

Let's go to the other node we have.

exit;

mysql -P 9307 -h0

SELECT * FROM pq;

We know this node is not in our cluster yet so it doesn't contain any data in the pq index we've added before at the other node. Let's join it to the cluster too.

To do this, use a JOIN CLUSTER statement with a nodes option. It defines the addresses of cluster nodes it'll be connected to.

JOIN CLUSTER posts '127.0.0.1:9312' as nodes;

(If there's only one node in the cluster, JOIN CLUSTER posts at 'node_ip:node_port' can be used instead)

Now let's try again:

SELECT * FROM pq;

We see that all the index data has been successfully replicated. Let's insert one more value to the index.

INSERT INTO posts:pq VALUES('value 3');

SELECT * FROM pq;

And now switch back to our first node:

exit;

mysql -P 9306 -h0

SELECT * FROM pq;

As we see, the changes were replicated successfully.

Removing cluster indexes and clusters

Now we'll remove our index from the cluster:

ALTER CLUSTER posts DROP pq;

`SHOW STATUS LIKE 'cluster_posts_indexes;'

INSERT INTO posts:pq VALUES('value 4');

SELECT * from pq;

pq index has become just a local non-replicated index. Let's check this:

INSERT INTO pq VALUES('value 4');

SELECT * from pq;

Switch to our second node:

exit;

mysql -P 9307 -h0

SELECT * from pq;

As expected, the changes that we made at the first node were not replicated to the second node.

To remove a cluster use a DELETE CLUSTER statement:

DELETE CLUSTER posts;

SHOW STATUS;

The cluster gets removed from all the nodes, but its indexes are left intact and become plain local non-replicated indexes.

SELECT * FROM pq;

To make sure it's really removed from everywhere we can check the status of the other node:

exit;

mysql -P 9306 -h0

SHOW STATUS;

And we see that posts doesn't exist here either.