Terminal Terminal | Web Web
Home  //  Play

Manticore Document Storage

Difficulty: Beginner
Estimated Time: 10 minutes

Manticoresearch - Document Storage

In this course you will learn how to use indexes which stores document content, not just index them.

Manticore Document Storage

Step 1 of 4

Introduction to Document Storage

By default, for full-text fields Manticore Search stores only index data, not the original text. Starting with 3.2 it is also possible to store the original content of the fields and retrieve it in the results. In this course we're going to use 2 RealTime indexes, with and without document storage, to see the difference between them.

index testrt
{
    type = rt
    path = /usr/local/sphinx/data/rt
    rt_field = title
    rt_field = content
    rt_attr_uint = gid
    rt_attr_string = title
}
index testrtstore
{
    type = rt
    path = /usr/local/sphinx/data/rtstore
    rt_field = title
    rt_field = content
    rt_attr_uint = gid
    rt_attr_string = title
    stored_fields = content
}

Inserting documents

The 2 indexes have no data, we are going to insert several documents in them. First, connect to Manticore by SphinxQL:

mysql -P9306 -h0

Now let's add several documents in the index without document storage:

INSERT INTO testrt VALUES(1,'Samsung Galaxy A50','The Samsung galaxy A50 is a complete device that provides for a onscreen fingerprint sensor along with the 6.4-inch (16.21 centimeters) super AMOLED - infinity u cut display, FHD+ resolution (2340 x 1080), 404 ppi with 16m colours and triple camera setup - 16mp (f1.9)+ 5mp (2.2) wide angle camera + 5mp (2.2) with flash and 25mp (f2.0) front facing camera.',10,'Samsung Galaxy A50');

INSERT INTO testrt VALUES(2,'Apple iPhone 6S','This update to the iPhone 6 upgrades the camera to 12 megapixels, bumps up the processor power, and adds a pressure-sensitive screen with haptic feedback. Other features are similar to the 6, including its rounded metal design, NFC, fingerprint reader, Siri voice assistant, front camera.',10,'Apple iPhone 6S');

INSERT INTO testrt VALUES(3,'Xiaomi Redmi Note 7','The global version of Xiaomi Redmi Note 7 features 2.5D Corning Gorilla 5 glass panels with an additional 0.8mm thickness at the front and rear. It has a 6.3 inch screen with an aspect ratio of 19.5: 9 and Full HD + resolution of 2340 x 1080 pixels.The screen supports the brightness of 450 nits, the contrast ratio of 1,500: 1 .',20,'Xiaomi Redmi Note 7');

INSERT INTO testrt VALUES(4,'Moto G7','12 MP dual camera system. Go beyond photos with a creative 12 MP dual camera system, creative photo software, and Google Lens.',30,'Moto G7');

INSERT INTO testrt VALUES(5,'ASUS ZenFone 5Z','Specs: Android 8.0 (Oreo); glass back; 6.2Inch FHD+ 2246x1080 display; 8MP front camera; 12 + 8 MP rear cameras; Qualcomm Snapdragon S845; 8GB RAM; 64GB storage',30,'ASUS ZenFone 5Z');

And now let's insert the same documents in the index with storage enabled:

INSERT INTO testrtstore VALUES(1,'Samsung Galaxy A50','The Samsung galaxy A50 is a complete device that provides for a onscreen fingerprint sensor along with the 6.4-inch (16.21 centimeters) super AMOLED - infinity u cut display, FHD+ resolution (2340 x 1080), 404 ppi with 16m colours and triple camera setup - 16mp (f1.9)+ 5mp (2.2) wide angle camera + 5mp (2.2) with flash and 25mp (f2.0) front facing camera.',10,'Samsung Galaxy A50');

INSERT INTO testrtstore VALUES(2,'Apple iPhone 6S','This update to the iPhone 6 upgrades the camera to 12 megapixels, bumps up the processor power, and adds a pressure-sensitive screen with haptic feedback. Other features are similar to the 6, including its rounded metal design, NFC, fingerprint reader, Siri voice assistant, front camera.',10,'Apple iPhone 6S');

INSERT INTO testrtstore VALUES(3,'Xiaomi Redmi Note 7','The global version of Xiaomi Redmi Note 7 features 2.5D Corning Gorilla 5 glass panels with an additional 0.8mm thickness at the front and rear. It has a 6.3 inch screen with an aspect ratio of 19.5: 9 and Full HD + resolution of 2340 x 1080 pixels.The screen supports the brightness of 450 nits, the contrast ratio of 1,500: 1 .',20,'Xiaomi Redmi Note 7');

INSERT INTO testrtstore VALUES(4,'Moto G7','12 MP dual camera system. Go beyond photos with a creative 12 MP dual camera system, creative photo software, and Google Lens.',30,'Moto G7');

INSERT INTO testrtstore VALUES(5,'ASUS ZenFone 5Z','Specs: Android 8.0 (Oreo); glass back; 6.2Inch FHD+ 2246x1080 display; 8MP front camera; 12 + 8 MP rear cameras; Qualcomm Snapdragon S845; 8GB RAM; 64GB storage',30,'ASUS ZenFone 5Z');

Executing search queries

To see the differences between the indexes let's execute a simple query on both indexes:

SELECT * FROM testrt WHERE MATCH('camera');

SELECT * FROM testrtstore WHERE MATCH('camera');

Highlighting

Without docstore highlighting required to get the original text from it's source and add it in the query returning back highlighted text.

With docstore now to highlighting function it can be simple passed the name of the field.

Taking the example from previous step, if we want to get highlighting on the 'content' stored fields we can simply do:

SELECT SNIPPET(content,'camera') FROM testrtstore WHERE MATCH('camera')\G