Elasticsearch Multisearch API

In projects, when performing search operations on Elasticsearch, sometimes we need to make multiple requests to gather the required data. In some cases, the number of these requests can increase, and we might need to make many requests for a single search result.

This problem can be solved using the Multisearch API. Of course, this usage has its pros and cons, and it can be used as needed based on these factors.

You can also access the Elasticsearch documentation via this link.

Pros:

• As mentioned, we can execute multiple requests in a single request.

• Instead of creating a complex code structure for many requests, we can use it in a more simplified manner.

Cons:

• We may encounter some difficulties in catching and distinguishing errors.

• By default, the general request size is 10MB. This limit encompasses the total size of all sub-requests. You can change this limit with the max_content_length parameter. The general request size error is “request_size_limit_exceeded”.

• The default size limit for each sub-request is 1MB. You can change this limit with the max_docsize parameter.

• If the total size of your sub-requests exceeds the general request size limit, or if the size of a sub-request exceeds the sub-request size limit, Elasticsearch will return an error message. The sub-request size error is “search_size_limit_exceeded”.

Usage Recommendations:

• If you have large sub-requests, you can divide them into multiple Multisearch requests.

• We can optimize the indices by using smaller documents or excluding unnecessary fields.

You can also find answers to questions about limits here.

Let's go through an example:

First, let's create a sample index.

PUT /test-multisearch
{
    "settings" : {
        "number_of_shards" : 2,
        "number_of_replicas" : 1
    },
    "mappings" : {
        "properties" : {
            "name" : { "type" : "text" },
            "categoryId": { "type" : "integer" }
        }
    }
}

Let's add some sample data.

POST /test-multisearch/_doc
{
  "name":"Coffee Maker",
  "categoryId" : 1
}

POST /test-multisearch/_doc
{
  "name":"Monitor",
  "categoryId" : 2
}

...

If we perform a regular search operation after all these steps;

GET test-multisearch/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "name": "Monitor"
          }
        }
      ]
    }
  }
}

Data we receive is as follows:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.9808291,
    "hits" : [
      {
        "_index" : "test-multisearch",
        "_type" : "_doc",
        "_id" : "_Himm5AB8XOTTnS2Mpjl",
        "_score" : 0.9808291,
        "_source" : {
          "name" : "Monitor",
          "categoryId" : 2
        }
      }
    ]
  }
}

Now let's move on to the Multisearch API section where we can receive multiple responses. Here, the "_msearch" endpoint is used.

GET test-multisearch/_msearch
{}
{ "query": { "bool": { "must": [ { "match": { "name": "Monitor" } } ] } } }
{}
{ "query": { "bool": { "must": [ { "match": { "name": "Laptop" } } ] } } }

The response we receive is as follows:

{
  "took" : 1,
  "responses" : [
    {
      "took" : 1,
      "timed_out" : false,
      "_shards" : {
        "total" : 2,
        "successful" : 2,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 1,
          "relation" : "eq"
        },
        "max_score" : 0.9808291,
        "hits" : [
          {
            "_index" : "test-multisearch",
            "_type" : "_doc",
            "_id" : "_Himm5AB8XOTTnS2Mpjl",
            "_score" : 0.9808291,
            "_source" : {
              "name" : "Monitor",
              "categoryId" : 2
            }
          }
        ]
      },
      "status" : 200
    },
    {
      "took" : 1,
      "timed_out" : false,
      "_shards" : {
        "total" : 2,
        "successful" : 2,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 1,
          "relation" : "eq"
        },
        "max_score" : 1.5697745,
        "hits" : [
          {
            "_index" : "test-multisearch",
            "_type" : "_doc",
            "_id" : "AHimm5AB8XOTTnS23Zkw",
            "_score" : 1.5697745,
            "_source" : {
              "name" : "Laptop",
              "categoryId" : 3
            }
          }
        ]
      },
      "status" : 200
    }
  ]
}

As you can see, we can access multiple search results within the "responses" field.

Let's now implement this in .NET after these steps.

using ElasticSearchMultisearchAPI;
using Nest;

// Connection
ConnectionSettings connSettings = new ConnectionSettings(new Uri("http://localhost:9200"))
                                      .DisableDirectStreaming();

ElasticClient elasticClient = new ElasticClient(connSettings);


// Quer
var firstCategoryQuery = new SearchDescriptor<Product>()
    .Index("test-multisearch")
    .Query(q => q
      .Bool(b => b
        .Must(m => m
          .Term(mm => mm.Field(mmf => mmf.CategoryId).Value(1)))));

var secondCategoryQuery = new SearchDescriptor<Product>()
    .Index("test-multisearch")
    .Query(q => q
      .Bool(b => b
        .Must(m => m
          .Term(mm => mm.Field(mmf => mmf.CategoryId).Value(2)))));

// Multisearch
var multiSearchRequest = new MultiSearchRequest()
{
    Operations = new Dictionary<string, ISearchRequest>
    {
        {
            "first-match", firstCategoryQuery
        },
        {
            "second-match", secondCategoryQuery
        }
    }
};

var multiSearchResponse = elasticClient.MultiSearch(multiSearchRequest);

// Responses
var firstResponse = multiSearchResponse.GetResponse<Product>("first-match");
var secondResponse = multiSearchResponse.GetResponse<Product>("second-match");

Console.WriteLine(firstResponse);
Console.WriteLine(secondResponse);