"Fossies" - the Fresh Open Source Software Archive

Member "elasticsearch-6.8.23/docs/reference/index.x.asciidoc" (29 Dec 2021, 26 Bytes) of package /linux/www/elasticsearch-6.8.23-src.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming AsciiDoc format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field.

Unresolved directive in ../Versions.asciidoc - include::{asciidoc-dir}/../../shared/versions/stack/{source_branch}.asciidoc[]

Unresolved directive in ../Versions.asciidoc - include::{asciidoc-dir}/../../shared/attributes.asciidoc[]

Elasticsearch introduction

Data in: documents and indices

{es} is a distributed document store. Instead of storing information as rows of columnar data, {es} stores complex data structures that have been serialized as JSON documents. When you have multiple {es} nodes in a cluster, stored documents are distributed across the cluster and can be accessed immediately from any node.

When a document is stored, it is indexed and fully searchable in near real-time—​within 1 second. {es} uses a data structure called an inverted index that supports very fast full-text searches. An inverted index lists every unique word that appears in any document and identifies all of the documents each word occurs in.

An index can be thought of as an optimized collection of documents and each document is a collection of fields, which are the key-value pairs that contain your data. By default, {es} indexes all data in every field and each indexed field has a dedicated, optimized data structure. For example, text fields are stored in inverted indices, and numeric and geo fields are stored in BKD trees. The ability to use the per-field data structures to assemble and return search results is what makes {es} so fast.

{es} also has the ability to be schema-less, which means that documents can be indexed without explicitly specifying how to handle each of the different fields that might occur in a document. When dynamic mapping is enabled, {es} automatically detects and adds new fields to the index. This default behavior makes it easy to index and explore your data—​just start indexing documents and {es} will detect and map booleans, floating point and integer values, dates, and strings to the appropriate {es} datatypes.

Ultimately, however, you know more about your data and how you want to use it than {es} can. You can define rules to control dynamic mapping and explicitly define mappings to take full control of how fields are stored and indexed.

Defining your own mappings enables you to:

  • Distinguish between full-text string fields and exact value string fields

  • Perform language-specific text analysis

  • Optimize fields for partial matching

  • Use custom date formats

  • Use data types such as geo_point and geo_shape that cannot be automatically detected

It’s often useful to index the same field in different ways for different purposes. For example, you might want to index a string field as both a text field for full-text search and as a keyword field for sorting or aggregating your data. Or, you might choose to use more than one language analyzer to process the contents of a string field that contains user input.

The analysis chain that is applied to a full-text field during indexing is also used at search time. When you query a full-text field, the query text undergoes the same analysis before the terms are looked up in the index.

Information out: search and analyze

While you can use {es} as a document store and retrieve documents and their metadata, the real power comes from being able to easily access the full suite of search capabilities built on the Apache Lucene search engine library.

{es} provides a simple, coherent REST API for managing your cluster and indexing and searching your data. For testing purposes, you can easily submit requests directly from the command line or through the Developer Console in {kib}. From your applications, you can use the {es} client for your language of choice: Java, JavaScript, Go, .NET, PHP, Perl, Python or Ruby.

Searching your data

The {es} REST APIs support structured queries, full text queries, and complex queries that combine the two. Structured queries are similar to the types of queries you can construct in SQL. For example, you could search the gender and age fields in your employee index and sort the matches by the hire_date field. Full-text queries find all documents that match the query string and return them sorted by relevance—how good a match they are for your search terms.

In addition to searching for individual terms, you can perform phrase searches, similarity searches, and prefix searches, and get autocomplete suggestions.

Have geospatial or other numerical data that you want to search? {es} indexes non-textual data in optimized data structures that support high-performance geo and numerical queries.

You can access all of these search capabilities using {es}'s comprehensive JSON-style query language (Query DSL). You can also construct SQL-style queries to search and aggregate data natively inside {es}, and JDBC and ODBC drivers enable a broad range of third-party applications to interact with {es} via SQL.

Analyzing your data

{es} aggregations enable you to build complex summaries of your data and gain insight into key metrics, patterns, and trends. Instead of just finding the proverbial “needle in a haystack”, aggregations enable you to answer questions like:

  • How many needles are in the haystack?

  • What is the average length of the needles?

  • What is the median length of the needles, broken down by manufacturer?

  • How many needles were added to the haystack in each of the last six months?

You can also use aggregations to answer more subtle questions, such as:

  • What are your most popular needle manufacturers?

  • Are there any unusual or anomalous clumps of needles?

Because aggregations leverage the same data-structures used for search, they are also very fast. This enables you to analyze and visualize your data in real time. Your reports and dashboards update as your data changes so you can take action based on the latest information.

What’s more, aggregations operate alongside search requests. You can search documents, filter results, and perform analytics at the same time, on the same data, in a single request. And because aggregations are calculated in the context of a particular search, you’re not just displaying a count of all size 7 needles, you’re displaying a count of the size 7 needles that match your users' search criteria—​for example, all size 7 non-stick embroidery needles.

But wait, there’s more

Want to automate the analysis of your time-series data? You can use {ml-docs}/ml-overview.html[machine learning] features to create accurate baselines of normal behavior in your data and identify anomalous patterns. With machine learning, you can detect:

  • Anomalies related to temporal deviations in values, counts, or frequencies

  • Statistical rarity

  • Unusual behaviors for a member of a population

And the best part? You can do this without having to specify algorithms, models, or other data science-related configurations.

Scalability and resilience: clusters, nodes, and shards

Scalability and resilience

{es} is built to be always available and to scale with your needs. It does this by being distributed by nature. You can add servers (nodes) to a cluster to increase capacity and {es} automatically distributes your data and query load across all of the available nodes. No need to overhaul your application, {es} knows how to balance multi-node clusters to provide scale and high availability. The more nodes, the merrier.

How does this work? Under the covers, an {es} index is really just a logical grouping of one or more physical shards, where each shard is actually a self-contained index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple nodes, {es} can ensure redundancy, which both protects against hardware failures and increases query capacity as nodes are added to a cluster. As the cluster grows (or shrinks), {es} automatically migrates shards to rebalance the cluster.

There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. A replica shard is a copy of a primary shard. Replicas provide redundant copies of your data to protect against hardware failure and increase capacity to serve read requests like searching or retrieving a document.

The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time, without interrupting indexing or query operations.

It depends…​

There are a number of performance considerations and trade offs with respect to shard size and the number of primary shards configured for an index. The more shards, the more overhead there is simply in maintaining those indices. The larger the shard size, the longer it takes to move shards around when {es} needs to rebalance a cluster.

Querying lots of small shards makes the processing per shard faster, but more queries means more overhead, so querying a smaller number of larger shards might be faster. In short…​it depends.

As a starting point:

  • Aim to keep the average shard size between a few GB and a few tens of GB. For use cases with time-based data, it is common to see shards in the 20GB to 40GB range.

  • Avoid the gazillion shards problem. The number of shards a node can hold is proportional to the available heap space. As a general rule, the number of shards per GB of heap space should be less than 20.

The best way to determine the optimal configuration for your use case is through testing with your own data and queries.

In case of disaster

For performance reasons, the nodes within a cluster need to be on the same network. Balancing shards in a cluster across nodes in different data centers simply takes too long. But high-availability architectures demand that you avoid putting all of your eggs in one basket. In the event of a major outage in one location, servers in another location need to be able to take over. Seamlessly. The answer? {ccr-cap} (CCR).

CCR provides a way to automatically synchronize indices from your primary cluster to a secondary remote cluster that can serve as a hot backup. If the primary cluster fails, the secondary cluster can take over. You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users.

{ccr-cap} is active-passive. The index on the primary cluster is the active leader index and handles all write requests. Indices replicated to secondary clusters are read-only followers.

Care and feeding

As with any enterprise system, you need tools to secure, manage, and monitor your {es} clusters. Security, monitoring, and administrative features that are integrated into {es} enable you to use {kibana-ref}/introduction.html[{kib}] as a control center for managing a cluster. Features like data rollups and index lifecycle management help you intelligently manage your data over time.

Getting started with {es}

Get {es} up and running

To take {es} for a test drive, you can create a hosted deployment on the {ess} or set up a multi-node {es} cluster on your own Linux, macOS, or Windows machine.

Run {es} on Elastic Cloud

When you create a deployment on the {es} Service, the service provisions a three-node {es} cluster along with Kibana and APM.

To create a deployment:

  1. Sign up for a free trial and verify your email address.

  2. Set a password for your account.

  3. Click Create Deployment.

Once you’ve created a deployment, you’re ready to Index some documents.

Run {es} locally on Linux, macOS, or Windows

When you create a deployment on the {ess}, a master node and two data nodes are provisioned automatically. By installing from the tar or zip archive, you can start multiple instances of {es} locally to see how a multi-node cluster behaves.

To run a three-node {es} cluster locally:

  1. Download the {es} archive for your OS:

    curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.tar.gz
  2. Extract the archive:

    Linux and macOS:

    tar -xvf elasticsearch-{version}.tar.gz

    Windows PowerShell:

    Expand-Archive elasticsearch-{version}-windows-x86_64.zip
  3. Start {es} from the bin directory:

    Linux and macOS:

    cd elasticsearch-{version}/bin
    ./elasticsearch

    Windows:

    cd %PROGRAMFILES%\Elastic\Elasticsearch\bin
    .\elasticsearch.exe

    You now have a single-node {es} cluster up and running!

  4. Start two more instances of {es} so you can see how a typical multi-node cluster behaves. You need to specify unique data and log paths for each node.

    Linux and macOS:

    ./elasticsearch -Epath.data=data2 -Epath.logs=log2
    ./elasticsearch -Epath.data=data3 -Epath.logs=log3

    Windows:

    .\elasticsearch.exe -Epath.data=data2 -Epath.logs=log2
    .\elasticsearch.exe -Epath.data=data3 -Epath.logs=log3

    The additional nodes are assigned unique IDs. Because you’re running all three nodes locally, they automatically join the cluster with the first node.

  5. Use the cat health API to verify that your three-node cluster is up running. The cat APIs return information about your cluster and indices in a format that’s easier to read than raw JSON.

    You can interact directly with your cluster by submitting HTTP requests to the {es} REST API. Most of the examples in this guide enable you to copy the appropriate cURL command and submit the request to your local {es} instance from the command line. If you have Kibana installed and running, you can also open Kibana and submit requests through the Dev Console.

    Tip
    You’ll want to check out the {es} language clients when you’re ready to start using {es} in your own applications.
    GET /_cat/health?v

    The response should indicate that the status of the elasticsearch cluster is green and it has three nodes:

    epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
    1565052807 00:53:27  elasticsearch green           3         3      6   3    0    0        0             0                  -                100.0%
    Note
    The cluster status will remain yellow if you are only running a single instance of {es}. A single node cluster is fully functional, but data cannot be replicated to another node to provide resiliency. Replica shards must be available for the cluster status to be green. If the cluster status is red, some data is unavailable.

Other installation options

Installing {es} from an archive file enables you to easily install and run multiple instances locally so you can try things out. To run a single instance, you can run {es} in a Docker container, install {es} using the DEB or RPM packages on Linux, install using Homebrew on macOS, or install using the MSI package installer on Windows. See Installing Elasticsearch for more information.

Index some documents

Once you have a cluster up and running, you’re ready to index some data. There are a variety of ingest options for {es}, but in the end they all do the same thing: put JSON documents into an {es} index.

You can do this directly with a simple PUT request that specifies the index you want to add the document, a unique document ID, and one or more "field": "value" pairs in the request body:

PUT /customer/_doc/1
{
  "name": "John Doe"
}

This request automatically creates the customer index if it doesn’t already exist, adds a new document that has an ID of 1, and stores and indexes the name field.

Since this is a new document, the response shows that the result of the operation was that version 1 of the document was created:

{
  "_index" : "customer",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 26,
  "_primary_term" : 4
}

The new document is available immediately from any node in the cluster. You can retrieve it with a GET request that specifies its document ID:

GET /customer/_doc/1

The response indicates that a document with the specified ID was found and shows the original source fields that were indexed.

{
  "_index" : "customer",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 26,
  "_primary_term" : 4,
  "found" : true,
  "_source" : {
    "name": "John Doe"
  }
}

Indexing documents in bulk

If you have a lot of documents to index, you can submit them in batches with the {ref}/docs-bulk.html[bulk API]. Using bulk to batch document operations is significantly faster than submitting requests individually as it minimizes network roundtrips.

The optimal batch size depends a number of factors: the document size and complexity, the indexing and search load, and the resources available to your cluster. A good place to start is with batches of 1,000 to 5,000 documents and a total payload between 5MB and 15MB. From there, you can experiment to find the sweet spot.

To get some data into {es} that you can start searching and analyzing:

  1. Download the accounts.json sample data set. The documents in this randomly-generated data set represent user accounts with the following information:

    {
        "account_number": 0,
        "balance": 16623,
        "firstname": "Bradshaw",
        "lastname": "Mckenzie",
        "age": 29,
        "gender": "F",
        "address": "244 Columbus Place",
        "employer": "Euron",
        "email": "bradshawmckenzie@euron.com",
        "city": "Hobucken",
        "state": "CO"
    }
  2. Index the account data into the bank index with the following _bulk request:

    curl -H "Content-Type: application/json" -XPOST "localhost:9200/bank/_doc/_bulk?pretty&refresh" --data-binary "@accounts.json"
    curl "localhost:9200/_cat/indices?v"

    The response indicates that 1,000 documents were indexed successfully.

    health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    green open   bank  l7sSYV2cQXmu6_4rJWVIww   5   1       1000            0    128.6kb        128.6kb

Once you have ingested some data into an {es} index, you can search it by sending requests to the _search endpoint. To access the full suite of search capabilities, you use the {es} Query DSL to specify the search criteria in the request body. You specify the name of the index you want to search in the request URI.

For example, the following request retrieves all documents in the bank index sorted by account number:

GET /bank/_search
{
  "query": { "match_all": {} },
  "sort": [
    { "account_number": "asc" }
  ]
}

By default, the hits section of the response includes the first 10 documents that match the search criteria:

{
  "took" : 63,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score": null,
    "hits" : [ {
      "_index" : "bank",
      "_type" : "_doc",
      "_id" : "0",
      "sort": [0],
      "_score" : null,
      "_source" : {"account_number":0,"balance":16623,"firstname":"Bradshaw","lastname":"Mckenzie","age":29,"gender":"F","address":"244 Columbus Place","employer":"Euron","email":"bradshawmckenzie@euron.com","city":"Hobucken","state":"CO"}
    }, {
      "_index" : "bank",
      "_type" : "_doc",
      "_id" : "1",
      "sort": [1],
      "_score" : null,
      "_source" : {"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
    }, ...
    ]
  }
}

The response also provides the following information about the search request:

  • took – how long it took {es} to run the query, in milliseconds

  • timed_out – whether or not the search request timed out

  • _shards – how many shards were searched and a breakdown of how many shards succeeded, failed, or were skipped.

  • max_score – the score of the most relevant document found

  • hits.total.value - how many matching documents were found

  • hits.sort - the document’s sort position (when not sorting by relevance score)

  • hits._score - the document’s relevance score (not applicable when using match_all)

Each search request is self-contained: {es} does not maintain any state information across requests. To page through the search hits, specify the from and size parameters in your request.

For example, the following request gets hits 10 through 19:

GET /bank/_search
{
  "query": { "match_all": {} },
  "sort": [
    { "account_number": "asc" }
  ],
  "from": 10,
  "size": 10
}

Now that you’ve seen how to submit a basic search request, you can start to construct queries that are a bit more interesting than match_all.

To search for specific terms within a field, you can use a match query. For example, the following request searches the address field to find customers whose addresses contain mill or lane:

GET /bank/_search
{
  "query": { "match": { "address": "mill lane" } }
}

To perform a phrase search rather than matching individual terms, you use match_phrase instead of match. For example, the following request only matches addresses that contain the phrase mill lane:

GET /bank/_search
{
  "query": { "match_phrase": { "address": "mill lane" } }
}

To construct more complex queries, you can use a bool query to combine multiple query criteria. You can designate criteria as required (must match), desirable (should match), or undesirable (must not match).

For example, the following request searches the bank index for accounts that belong to customers who are 40 years old, but excludes anyone who lives in Idaho (ID):

GET /bank/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "age": "40" } }
      ],
      "must_not": [
        { "match": { "state": "ID" } }
      ]
    }
  }
}

Each must, should, and must_not element in a Boolean query is referred to as a query clause. How well a document meets the criteria in each must or should clause contributes to the document’s relevance score. The higher the score, the better the document matches your search criteria. By default, {es} returns documents ranked by these relevance scores.

The criteria in a must_not clause is treated as a filter. It affects whether or not the document is included in the results, but does not contribute to how documents are scored. You can also explicitly specify arbitrary filters to include or exclude documents based on structured data.

For example, the following request uses a range filter to limit the results to accounts with a balance between $20,000 and $30,000 (inclusive).

GET /bank/_search
{
  "query": {
    "bool": {
      "must": { "match_all": {} },
      "filter": {
        "range": {
          "balance": {
            "gte": 20000,
            "lte": 30000
          }
        }
      }
    }
  }
}

Analyze results with aggregations

{es} aggregations enable you to get meta-information about your search results and answer questions like, "How many account holders are in Texas?" or "What’s the average balance of accounts in Tennessee?" You can search documents, filter hits, and use aggregations to analyze the results all in one request.

For example, the following request uses a terms aggregation to group all of the accounts in the bank index by state, and returns the ten states with the most accounts in descending order:

GET /bank/_search
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      }
    }
  }
}

The buckets in the response are the values of the state field. The doc_count shows the number of accounts in each state. For example, you can see that there are 27 accounts in ID (Idaho). Because the request set size=0, the response only contains the aggregation results.

{
  "took": 29,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits" : {
     "total" : 1000,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "group_by_state" : {
      "doc_count_error_upper_bound": 20,
      "sum_other_doc_count": 770,
      "buckets" : [ {
        "key" : "ID",
        "doc_count" : 27
      }, {
        "key" : "TX",
        "doc_count" : 27
      }, {
        "key" : "AL",
        "doc_count" : 25
      }, {
        "key" : "MD",
        "doc_count" : 25
      }, {
        "key" : "TN",
        "doc_count" : 23
      }, {
        "key" : "MA",
        "doc_count" : 21
      }, {
        "key" : "NC",
        "doc_count" : 21
      }, {
        "key" : "ND",
        "doc_count" : 21
      }, {
        "key" : "ME",
        "doc_count" : 20
      }, {
        "key" : "MO",
        "doc_count" : 20
      } ]
    }
  }
}

You can combine aggregations to build more complex summaries of your data. For example, the following request nests an avg aggregation within the previous group_by_state aggregation to calculate the average account balances for each state.

GET /bank/_search
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      },
      "aggs": {
        "average_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}

Instead of sorting the results by count, you could sort using the result of the nested aggregation by specifying the order within the terms aggregation:

GET /bank/_search
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword",
        "order": {
          "average_balance": "desc"
        }
      },
      "aggs": {
        "average_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}

In addition to basic bucketing and metrics aggregations like these, {es} provides specialized aggregations for operating on multiple fields and analyzing particular types of data such as dates, IP addresses, and geo data. You can also feed the results of individual aggregations into pipeline aggregations for further analysis.

The core analysis capabilities provided by aggregations enable advanced features such as using machine learning to detect anomalies.

Where to go from here

Now that you’ve set up a cluster, indexed some documents, and run some searches and aggregations, you might want to:

  • {stack-gs}/get-started-elastic-stack.html#install-kibana[Dive in to the Elastic Stack Tutorial] to install Kibana, Logstash, and Beats and set up a basic system monitoring solution.

  • {kibana-ref}/add-sample-data.html[Load one of the sample data sets into Kibana] to see how you can use {es} and Kibana together to visualize your data.

  • Try out one of the Elastic search solutions:

Set up Elasticsearch

Installing Elasticsearch

Hosted Elasticsearch

{es} can be run on your own hardware or using our hosted {ess} on {ecloud}, which is available on AWS, GCP and Azure. You can {ess-trial}[try out the hosted service] for free.

Installing Elasticsearch Yourself

Elasticsearch is provided in the following package formats:

zip/tar.gz

The zip and tar.gz packages are suitable for installation on any system and are the easiest choice for getting started with Elasticsearch on most systems.

deb

The deb package is suitable for Debian, Ubuntu, and other Debian-based systems. Debian packages may be downloaded from the Elasticsearch website or from our Debian repository.

rpm

The rpm package is suitable for installation on Red Hat, Centos, SLES, OpenSuSE and other RPM-based systems. RPMs may be downloaded from the Elasticsearch website or from our RPM repository.

msi

beta[] The msi package is suitable for installation on Windows 64-bit systems with at least .NET 4.5 framework installed, and is the easiest choice for getting started with Elasticsearch on Windows. MSIs may be downloaded from the Elasticsearch website.

docker

Images are available for running Elasticsearch as Docker containers. They may be downloaded from the Elastic Docker Registry.

{ref}/docker.html[Install {es} with Docker]

Configuration Management Tools

We also provide the following configuration management tools to help with large deployments:

Install Elasticsearch with .zip or .tar.gz

Elasticsearch is provided as a .zip and as a .tar.gz package. These packages can be used to install Elasticsearch on any system and are the easiest package format to use when trying out Elasticsearch.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Download and install the .zip package

The .zip archive for Elasticsearch v{version} can be downloaded and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip.sha512
shasum -a 512 -c elasticsearch-{version}.zip.sha512 <1>
unzip elasticsearch-{version}.zip
cd elasticsearch-{version}/ <2>
  1. Compares the SHA of the downloaded .zip archive and the published checksum, which should output elasticsearch-{version}.zip: OK.

  2. This directory is known as $ES_HOME.

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.zip

Download and install the .tar.gz package

The .tar.gz archive for Elasticsearch v{version} can be downloaded and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.tar.gz.sha512
shasum -a 512 -c elasticsearch-{version}.tar.gz.sha512 <1>
tar -xzf elasticsearch-{version}.tar.gz
cd elasticsearch-{version}/ <2>
  1. Compares the SHA of the downloaded .tar.gz archive and the published checksum, which should output elasticsearch-{version}.tar.gz: OK.

  2. This directory is known as $ES_HOME.

Alternatively, you can download the following package, which includes only Apache 2.0 licensed code: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.tar.gz

Enable automatic creation of {xpack} indices

{xpack} will try to automatically create a number of indices within {es}. By default, {es} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {es}, you must configure action.auto_create_index in elasticsearch.yml to allow {xpack} to create the following indices:

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
Important

If you are using Logstash or Beats then you will most likely require additional index names in your action.auto_create_index setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to * which will allow automatic creation of all indices.

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

./bin/elasticsearch

By default, Elasticsearch runs in the foreground, prints its logs to the standard output (stdout), and can be stopped by pressing Ctrl-C.

Note
All scripts packaged with Elasticsearch require a version of Bash that supports arrays and assume that Bash is available at /bin/bash. As such, Bash should be available at this path either directly or via a symbolic link.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Log printing to stdout can be disabled using the -q or --quiet option on the command line.

Running as a daemon

To run Elasticsearch as a daemon, specify -d on the command line, and record the process ID in a file using the -p option:

./bin/elasticsearch -d -p pid

Log messages can be found in the $ES_HOME/logs/ directory.

To shut down Elasticsearch, kill the process ID recorded in the pid file:

pkill -F pid
Note
The startup scripts provided in the RPM and Debian packages take care of starting and stopping the Elasticsearch process for you.

Configuring Elasticsearch on the command line

Elasticsearch loads its configuration from the $ES_HOME/config/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

Any settings that can be specified in the config file can also be specified on the command line, using the -E syntax as follows:

./bin/elasticsearch -d -Ecluster.name=my_cluster -Enode.name=node_1
Tip
Typically, any cluster-wide settings (like cluster.name) should be added to the elasticsearch.yml config file, while any node-specific settings such as node.name could be specified on the command line.

Directory layout of .zip and .tar.gz archives

The .zip and .tar.gz packages are entirely self-contained. All files and directories are, by default, contained within $ES_HOME — the directory created when unpacking the archive.

This is very convenient because you don’t have to create any directories to start using Elasticsearch, and uninstalling Elasticsearch is as easy as removing the $ES_HOME directory. However, it is advisable to change the default locations of the config directory, the data directory, and the logs directory so that you do not delete important data later on.

Type Description Default Location Setting

home

Elasticsearch home directory or $ES_HOME

Directory created by unpacking the archive

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

$ES_HOME/bin

conf

Configuration files including elasticsearch.yml

$ES_HOME/config

ES_PATH_CONF

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

$ES_HOME/data

path.data

logs

Log files location.

$ES_HOME/logs

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

$ES_HOME/plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

script

Location of script files.

$ES_HOME/scripts

path.scripts

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with .zip on Windows

Elasticsearch can be installed on Windows using the .zip package. This comes with a elasticsearch-service.bat command which will setup Elasticsearch to run as a service.

Tip
Elasticsearch has historically been installed on Windows using the .zip archive. An MSI installer package is available that provides the easiest getting started experience for Windows. You can continue using the .zip approach if you prefer.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Download and install the .zip package

Download the .zip archive for Elasticsearch v{version} from: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.zip

Unzip it with your favourite unzip tool. This will create a folder called elasticsearch-{version}, which we will refer to as %ES_HOME%. In a terminal window, cd to the %ES_HOME% directory, for instance:

cd c:\elasticsearch-{version}

Enable automatic creation of {xpack} indices

{xpack} will try to automatically create a number of indices within {es}. By default, {es} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {es}, you must configure action.auto_create_index in elasticsearch.yml to allow {xpack} to create the following indices:

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
Important

If you are using Logstash or Beats then you will most likely require additional index names in your action.auto_create_index setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to * which will allow automatic creation of all indices.

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

.\bin\elasticsearch.bat

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl-C.

Configuring Elasticsearch on the command line

Elasticsearch loads its configuration from the %ES_HOME%\config\elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

Any settings that can be specified in the config file can also be specified on the command line, using the -E syntax as follows:

.\bin\elasticsearch.bat -Ecluster.name=my_cluster -Enode.name=node_1
Note
Values that contain spaces must be surrounded with quotes. For instance -Epath.logs="C:\My Logs\logs".
Tip
Typically, any cluster-wide settings (like cluster.name) should be added to the elasticsearch.yml config file, while any node-specific settings such as node.name could be specified on the command line.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Installing Elasticsearch as a Service on Windows

Elasticsearch can be installed as a service to run in the background or start automatically at boot time without any user interaction. This can be achieved through the elasticsearch-service.bat script in the bin\ folder which allows one to install, remove, manage or configure the service and potentially start and stop the service, all from the command-line.

c:\elasticsearch-{version}\bin>elasticsearch-service.bat

Usage: elasticsearch-service.bat install|remove|start|stop|manager [SERVICE_ID]

The script requires one parameter (the command to execute) followed by an optional one indicating the service id (useful when installing multiple Elasticsearch services).

The commands available are:

install

Install Elasticsearch as a service

remove

Remove the installed Elasticsearch service (and stop the service if started)

start

Start the Elasticsearch service (if installed)

stop

Stop the Elasticsearch service (if started)

manager

Start a GUI for managing the installed service

The name of the service and the value of JAVA_HOME will be made available during install:

c:\elasticsearch-{version}\bin>elasticsearch-service.bat install
Installing service      :  "elasticsearch-service-x64"
Using JAVA_HOME (64-bit):  "c:\jvm\jdk1.8"
The service 'elasticsearch-service-x64' has been installed.
Note
While a JRE can be used for the Elasticsearch service, due to its use of a client VM (as opposed to a server JVM which offers better performance for long-running applications) its usage is discouraged and a warning will be issued.
Note
The system environment variable JAVA_HOME should be set to the path to the JDK installation that you want the service to use. If you upgrade the JDK, you are not required to the reinstall the service but you must set the value of the system environment variable JAVA_HOME to the path to the new JDK installation. However, upgrading across JVM types (e.g. JRE versus SE) is not supported, and does require the service to be reinstalled.

Customizing service settings

The Elasticsearch service can be configured prior to installation by setting the following environment variables (either using the set command from the command line, or through the System Properties→Environment Variables GUI).

SERVICE_ID

A unique identifier for the service. Useful if installing multiple instances on the same machine. Defaults to elasticsearch-service-x64.

SERVICE_USERNAME

The user to run as, defaults to the local system account.

SERVICE_PASSWORD

The password for the user specified in %SERVICE_USERNAME%.

SERVICE_DISPLAY_NAME

The name of the service. Defaults to Elasticsearch <version> %SERVICE_ID%.

SERVICE_DESCRIPTION

The description of the service. Defaults to Elasticsearch <version> Windows Service - https://elastic.co.

JAVA_HOME

The installation directory of the desired JVM to run the service under.

SERVICE_LOG_DIR

Service log directory, defaults to %ES_HOME%\logs. Note that this does not control the path for the Elasticsearch logs; the path for these is set via the setting path.logs in the elasticsearch.yml configuration file, or on the command line.

ES_PATH_CONF

Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files), defaults to %ES_HOME%\config.

ES_JAVA_OPTS

Any additional JVM system properties you may want to apply.

ES_START_TYPE

Startup mode for the service. Can be either auto or manual (default).

ES_STOP_TIMEOUT

The timeout in seconds that procrun waits for service to exit gracefully. Defaults to 0.

Note
At its core, elasticsearch-service.bat relies on Apache Commons Daemon project to install the service. Environment variables set prior to the service installation are copied and will be used during the service lifecycle. This means any changes made to them after the installation will not be picked up unless the service is reinstalled.
Note
On Windows, the heap size can be configured as for any other Elasticsearch installation when running Elasticsearch from the command line, or when installing Elasticsearch as a service for the first time. To adjust the heap size for an already installed service, use the service manager: bin\elasticsearch-service.bat manager.
Note
The service automatically configures a private temporary directory for use by Elasticsearch when it is running. This private temporary directory is configured as a sub-directory of the private temporary directory for the user running the installation. If the service will run under a different user, you can configure the location of the temporary directory that the service should use by setting the environment variable ES_TMPDIR to the preferred location before you execute the service installation.
Using the Manager GUI

It is also possible to configure the service after it’s been installed using the manager GUI (elasticsearch-service-mgr.exe), which offers insight into the installed service, including its status, startup type, JVM, start and stop settings amongst other things. Simply invoking elasticsearch-service.bat manager from the command-line will open up the manager window:

Windows Service Manager GUI

Most changes (like JVM settings) made through the manager GUI will require a restart of the service in order to take affect.

Directory layout of .zip archive

The .zip package is entirely self-contained. All files and directories are, by default, contained within %ES_HOME% — the directory created when unpacking the archive.

This is very convenient because you don’t have to create any directories to start using Elasticsearch, and uninstalling Elasticsearch is as easy as removing the %ES_HOME% directory. However, it is advisable to change the default locations of the config directory, the data directory, and the logs directory so that you do not delete important data later on.

Type Description Default Location Setting

home

Elasticsearch home directory or %ES_HOME%

Directory created by unpacking the archive

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

%ES_HOME%\bin

conf

Configuration files including elasticsearch.yml

%ES_HOME%\config

ES_PATH_CONF

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

%ES_HOME%\data

path.data

logs

Log files location.

%ES_HOME%\logs

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

%ES_HOME%\plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with Debian Package

The Debian package for Elasticsearch can be downloaded from our website or from our APT repository. It can be used to install Elasticsearch on any Debian-based system such as Debian and Ubuntu.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Import the Elasticsearch PGP Key

We sign all of our packages with the Elasticsearch Signing Key (PGP key D88E42B4, available from https://pgp.mit.edu) with fingerprint:

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

Download and install the public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Installing from the APT repository

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https

Save the repository definition to /etc/apt/sources.list.d/elastic-{major-version}.list:

Note

These instructions do not use add-apt-repository for several reasons:

  1. add-apt-repository adds entries to the system /etc/apt/sources.list file rather than a clean per-repository file in /etc/apt/sources.list.d

  2. add-apt-repository is not part of the default install on many distributions and requires a number of non-default dependencies.

  3. Older versions of add-apt-repository always add a deb-src entry which will cause errors because we do not provide a source package. If you have added the deb-src entry, you will see an error like the following until you delete the deb-src line:

    Unable to find expected entry 'main/source/Sources' in Release file
    (Wrong sources.list entry or malformed file)

You can install the Elasticsearch Debian package with:

sudo apt-get update && sudo apt-get install elasticsearch
Warning

If two entries exist for the same Elasticsearch repository, you will see an error like this during apt-get update:

Duplicate sources.list entry https://artifacts.elastic.co/packages/{major-version}/apt/ ...`

Examine /etc/apt/sources.list.d/elasticsearch-{major-version}.list for the duplicate entry or locate the duplicate entry amongst the files in /etc/apt/sources.list.d/ and the /etc/apt/sources.list file.

Note
On systemd-based distributions, the installation scripts will attempt to set kernel parameters (e.g., vm.max_map_count); you can skip this by masking the systemd-sysctl.service unit.
Note

An alternative package which contains only features that are available under the Apache 2.0 license is also available. To install it, use the following sources list:

echo "deb https://artifacts.elastic.co/packages/oss-{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list

Download and install the Debian package manually

The Debian package for Elasticsearch v{version} can be downloaded from the website and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.deb
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.deb.sha512
shasum -a 512 -c elasticsearch-{version}.deb.sha512 <1>
sudo dpkg -i elasticsearch-{version}.deb
  1. Compares the SHA of the downloaded Debian package and the published checksum, which should output elasticsearch-{version}.deb: OK.

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.deb

Enable automatic creation of {xpack} indices

{xpack} will try to automatically create a number of indices within Elasticsearch. By default, {es} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {es}, you must configure action.auto_create_index in elasticsearch.yml to allow {xpack} to create the following indices:

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
Important

If you are using Logstash or Beats then you will most likely require additional index names in your action.auto_create_index setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to * which will allow automatic creation of all indices.

SysV init vs systemd

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the update-rc.d command to configure Elasticsearch to start automatically when the system boots up:

sudo update-rc.d elasticsearch defaults 95 10

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Configuring Elasticsearch

Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. The ownership of this directory and all files in this directory are set to root:elasticsearch on package installation and the directory has the setgid flag set so that any files and subdirectories created under /etc/elasticsearch are created with this ownership as well (e.g., if a keystore is created using the keystore tool). It is expected that this be maintained so that the Elasticsearch process can read the files under this directory via the group permissions.

Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

The Debian package also has a system configuration file (/etc/default/elasticsearch), which allows you to set the following parameters:

JAVA_HOME

Set a custom Java path to be used.

MAX_OPEN_FILES

Maximum number of open files, defaults to 65535.

MAX_LOCKED_MEMORY

Maximum locked memory size. Set to unlimited if you use the bootstrap.memory_lock option in elasticsearch.yml.

MAX_MAP_COUNT

Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting Elasticsearch. Defaults to 262144.

ES_PATH_CONF

Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files); defaults to /etc/elasticsearch.

ES_JAVA_OPTS

Any additional JVM system properties you may want to apply.

RESTART_ON_UPGRADE

Configure restart on package upgrade, defaults to false. This means you will have to restart your Elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster.

Note
Distributions that use systemd require that system resource limits be configured via systemd rather than via the /etc/sysconfig/elasticsearch file. See Systemd configuration for more information.

Directory layout of Debian package

The Debian package places config files, logs, and the data directory in the appropriate locations for a Debian-based system:

Type Description Default Location Setting

home

Elasticsearch home directory or $ES_HOME

/usr/share/elasticsearch

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

/usr/share/elasticsearch/bin

conf

Configuration files including elasticsearch.yml

/etc/elasticsearch

ES_PATH_CONF

conf

Environment variables including heap size, file descriptors.

/etc/default/elasticsearch

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

/var/lib/elasticsearch

path.data

logs

Log files location.

/var/log/elasticsearch

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

/usr/share/elasticsearch/plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with RPM

The RPM for Elasticsearch can be downloaded from our website or from our RPM repository. It can be used to install Elasticsearch on any RPM-based system such as OpenSuSE, SLES, Centos, Red Hat, and Oracle Enterprise.

Note
RPM install is not supported on distributions with old versions of RPM, such as SLES 11 and CentOS 5. Please see Install Elasticsearch with .zip or .tar.gz instead.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Import the Elasticsearch PGP Key

We sign all of our packages with the Elasticsearch Signing Key (PGP key D88E42B4, available from https://pgp.mit.edu) with fingerprint:

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

Download and install the public signing key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Installing from the RPM repository

Create a file called elasticsearch.repo in the /etc/yum.repos.d/ directory for RedHat based distributions, or in the /etc/zypp/repos.d/ directory for OpenSuSE based distributions, containing:

And your repository is ready for use. You can now install Elasticsearch with one of the following commands:

sudo yum install elasticsearch (1)
sudo dnf install elasticsearch (2)
sudo zypper install elasticsearch (3)
  1. Use yum on CentOS and older Red Hat based distributions.

  2. Use dnf on Fedora and other newer Red Hat distributions.

  3. Use zypper on OpenSUSE based distributions

Note

An alternative package which contains only features that are available under the Apache 2.0 license is also available. To install it, use the following baseurl in your elasticsearch.repo file:

baseurl=https://artifacts.elastic.co/packages/oss-{major-version}/yum

Download and install the RPM manually

The RPM for Elasticsearch v{version} can be downloaded from the website and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.rpm.sha512
shasum -a 512 -c elasticsearch-{version}.rpm.sha512 <1>
sudo rpm --install elasticsearch-{version}.rpm
  1. Compares the SHA of the downloaded RPM and the published checksum, which should output elasticsearch-{version}.rpm: OK.

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.rpm

Note
On systemd-based distributions, the installation scripts will attempt to set kernel parameters (e.g., vm.max_map_count); you can skip this by masking the systemd-sysctl.service unit.

Enable automatic creation of {xpack} indices

{xpack} will try to automatically create a number of indices within {es}. By default, {es} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {es}, you must configure action.auto_create_index in elasticsearch.yml to allow {xpack} to create the following indices:

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
Important

If you are using Logstash or Beats then you will most likely require additional index names in your action.auto_create_index setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to * which will allow automatic creation of all indices.

SysV init vs systemd

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the chkconfig command to configure Elasticsearch to start automatically when the system boots up:

sudo chkconfig --add elasticsearch

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Configuring Elasticsearch

Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. The ownership of this directory and all files in this directory are set to root:elasticsearch on package installation and the directory has the setgid flag set so that any files and subdirectories created under /etc/elasticsearch are created with this ownership as well (e.g., if a keystore is created using the keystore tool). It is expected that this be maintained so that the Elasticsearch process can read the files under this directory via the group permissions.

Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

The RPM also has a system configuration file (/etc/sysconfig/elasticsearch), which allows you to set the following parameters:

JAVA_HOME

Set a custom Java path to be used.

MAX_OPEN_FILES

Maximum number of open files, defaults to 65535.

MAX_LOCKED_MEMORY

Maximum locked memory size. Set to unlimited if you use the bootstrap.memory_lock option in elasticsearch.yml.

MAX_MAP_COUNT

Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting Elasticsearch. Defaults to 262144.

ES_PATH_CONF

Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files); defaults to /etc/elasticsearch.

ES_JAVA_OPTS

Any additional JVM system properties you may want to apply.

RESTART_ON_UPGRADE

Configure restart on package upgrade, defaults to false. This means you will have to restart your Elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster.

Note
Distributions that use systemd require that system resource limits be configured via systemd rather than via the /etc/sysconfig/elasticsearch file. See Systemd configuration for more information.

Directory layout of RPM

The RPM places config files, logs, and the data directory in the appropriate locations for an RPM-based system:

Type Description Default Location Setting

home

Elasticsearch home directory or $ES_HOME

/usr/share/elasticsearch

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

/usr/share/elasticsearch/bin

conf

Configuration files including elasticsearch.yml

/etc/elasticsearch

ES_PATH_CONF

conf

Environment variables including heap size, file descriptors.

/etc/sysconfig/elasticsearch

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

/var/lib/elasticsearch

path.data

logs

Log files location.

/var/log/elasticsearch

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

/usr/share/elasticsearch/plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with Windows MSI Installer

beta[]

Elasticsearch can be installed on Windows using the .msi package. This can install Elasticsearch as a Windows service or allow it to be run manually using the included elasticsearch.exe executable.

Tip
Elasticsearch has historically been installed on Windows using the .zip archive. You can continue using the .zip approach if you prefer.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Download the .msi package

Download the .msi package for Elasticsearch v{version} from https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.msi

Install using the graphical user interface (GUI)

Double-click the downloaded .msi package to launch a GUI wizard that will guide you through the installation process. You can view help on any step by clicking the ? button, which reveals an aside panel with additional information for each input:

msi installer help

Within the first screen, select the directory for the installation. In addition, select directories for where data, logs and configuration will be placed or use the default locations:

msi installer locations

Then select whether to install as a service or start Elasticsearch manually as needed. When installing as a service, you can also configure the Windows account to run the service with, whether the service should be started after installation and the Windows startup behaviour:

msi installer service
Important
When selecting a Windows account to run the service with, be sure that the chosen account has sufficient privileges to access the installation and other deployment directories chosen. Also ensure the account is able to run Windows services.

Common configuration settings are exposed within the Configuration section, allowing the cluster name, node name and roles to be set, in addition to memory and network settings:

msi installer configuration

A list of common plugins that can be downloaded and installed as part of the installation, with the option to configure an HTTPS proxy through which to download these plugins.

Tip
Ensure the installation machine has access to the internet and that any corporate firewalls in place are configured to allow downloads from artifacts.elastic.co:
msi installer selected plugins

As of version 6.3.0, {xpack} is now bundled by default. The final step allows a choice of the type of license to install, in addition to security configuration and built-in user configuration:

msi installer xpack
Note
{xpack} includes a choice of a Trial or Basic license. A Trial license is valid for 30 days, after which you can obtain one of the available subscriptions. The Basic license is free and perpetual. Consult the available subscriptions for further details on which features are available under which license.

After clicking the install button, the installation will begin:

msi installer installing

…​and will indicate when it has been successfully installed:

msi installer success

Install using the command line

The .msi can also install Elasticsearch using the command line. The simplest installation using the same defaults as the GUI is achieved by first navigating to the download directory, then running:

msiexec.exe /i elasticsearch-{version}.msi /qn

By default, msiexec.exe does not wait for the installation process to complete, since it runs in the Windows subsystem. To wait on the process to finish and ensure that %ERRORLEVEL% is set accordingly, it is recommended to use start /wait to create a process and wait for it to exit

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn

As with any MSI installation package, a log file for the installation process can be found within the %TEMP% directory, with a randomly generated name adhering to the format MSI<random>.LOG. The path to a log file can be supplied using the /l command line argument

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn /l install.log

Supported Windows Installer command line arguments can be viewed using

msiexec.exe /help

…​or by consulting the Windows Installer SDK Command-Line Options.

Command line options

All settings exposed within the GUI are also available as command line arguments (referred to as properties within Windows Installer documentation) that can be passed to msiexec.exe:

INSTALLDIR

The installation directory. The final directory in the path must be the version of Elasticsearch. Defaults to %ProgramW6432%\Elastic\Elasticsearch{backslash}{version}.

DATADIRECTORY

The directory in which to store your data. Defaults to %ALLUSERSPROFILE%\Elastic\Elasticsearch\data

CONFIGDIRECTORY

The directory in which to store your configuration. Defaults to %ALLUSERSPROFILE%\Elastic\Elasticsearch\config

LOGSDIRECTORY

The directory in which to store your logs. Defaults to %ALLUSERSPROFILE%\Elastic\Elasticsearch\logs

PLACEWRITABLELOCATIONSINSAMEPATH

Whether the data, configuration and logs directories should be created under the installation directory. Defaults to false

INSTALLASSERVICE

Whether Elasticsearch is installed and configured as a Windows Service. Defaults to true

STARTAFTERINSTALL

Whether the Windows Service is started after installation finishes. Defaults to true

STARTWHENWINDOWSSTARTS

Whether the Windows Service is started when Windows is started. Defaults to true

USELOCALSYSTEM

Whether the Windows service runs under the LocalSystem Account. Defaults to true

USENETWORKSERVICE

Whether the Windows service runs under the NetworkService Account. Defaults to false

USEEXISTINGUSER

Whether the Windows service runs under a specified existing account. Defaults to false

USER

The username for the account under which the Windows service runs. Defaults to ""

PASSWORD

The password for the account under which the Windows service runs. Defaults to ""

CLUSTERNAME

The name of the cluster. Defaults to elasticsearch

NODENAME

The name of the node. Defaults to %COMPUTERNAME%

MASTERNODE

Whether Elasticsearch is configured as a master node. Defaults to true

DATANODE

Whether Elasticsearch is configured as a data node. Defaults to true

INGESTNODE

Whether Elasticsearch is configured as an ingest node. Defaults to true

SELECTEDMEMORY

The amount of memory to allocate to the JVM heap for Elasticsearch. Defaults to 2048 unless the target machine has less than 4GB in total, in which case it defaults to 50% of total memory.

LOCKMEMORY

Whether bootstrap.memory_lock should be used to try to lock the process address space into RAM. Defaults to false

UNICASTNODES

A comma separated list of hosts in the form host:port or host to be used for unicast discovery. Defaults to ""

MINIMUMMASTERNODES

The minimum number of master-eligible nodes that must be visible in order to form a cluster. Defaults to ""

NETWORKHOST

The hostname or IP address to bind the node to and publish (advertise) this host to other nodes in the cluster. Defaults to ""

HTTPPORT

The port to use for exposing Elasticsearch APIs over HTTP. Defaults to 9200

TRANSPORTPORT

The port to use for internal communication between nodes within the cluster. Defaults to 9300

PLUGINS

A comma separated list of the plugins to download and install as part of the installation. Defaults to ""

HTTPSPROXYHOST

The proxy host to use to download plugins over HTTPS. Defaults to ""

HTTPSPROXYPORT

The proxy port to use to download plugins over HTTPS. Defaults to 443

HTTPPROXYHOST

The proxy host to use to download plugins over HTTP. Defaults to ""

HTTPPROXYPORT

The proxy port to use to download plugins over HTTP. Defaults to 80

XPACKLICENSE

The type of license to install, either Basic or Trial. Defaults to Basic

XPACKSECURITYENABLED

When installing with a Trial license, whether {security-features} are enabled. Defaults to true

BOOTSTRAPPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to used to bootstrap the cluster and persisted as the bootstrap.password setting in the keystore. Defaults to a randomized value.

SKIPSETTINGPASSWORDS

When installing with a Trial license and {security-features} enabled, whether the installation should skip setting up the built-in users. Defaults to false

ELASTICUSERPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to use for the built-in user elastic. Defaults to ""

KIBANAUSERPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to use for the built-in user kibana. Defaults to ""

LOGSTASHSYSTEMUSERPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to use for the built-in user logstash_system. Defaults to ""

To pass a value, simply append the property name and value using the format <PROPERTYNAME>="<VALUE>" to the installation command. For example, to use a different installation directory to the default one:

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn INSTALLDIR="C:\Custom Install Directory{version}"

Consult the Windows Installer SDK Command-Line Options for additional rules related to values containing quotation marks.

Enable automatic creation of {xpack} indices

The {stack} features try to automatically create a number of indices within {es}. By default, {es} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {es}, you must configure action.auto_create_index in elasticsearch.yml to allow {xpack} to create the following indices:

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
Important

If you are using Logstash or Beats then you will most likely require additional index names in your action.auto_create_index setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to * which will allow automatic creation of all indices.

Running Elasticsearch from the command line

Once installed, Elasticsearch can be started from the command line, if not installed as a service and configured to start when installation completes, as follows:

.\bin\elasticsearch.exe

The command line terminal will display output similar to the following:

elasticsearch exe

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT in addition to the <cluster name>.log file within LOGSDIRECTORY, and can be stopped by pressing Ctrl-C.

Configuring Elasticsearch on the command line

Elasticsearch loads its configuration from the %ES_PATH_CONF%\elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

Any settings that can be specified in the config file can also be specified on the command line, using the -E syntax as follows:

.\bin\elasticsearch.exe -E cluster.name=my_cluster -E node.name=node_1
Note
Values that contain spaces must be surrounded with quotes. For instance -E path.logs="C:\My Logs\logs".
Tip
Typically, any cluster-wide settings (like cluster.name) should be added to the elasticsearch.yml config file, while any node-specific settings such as node.name could be specified on the command line.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.3",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Installing Elasticsearch as a Service on Windows

Elasticsearch can be installed as a service to run in the background or start automatically at boot time without any user interaction. This can be achieved upon installation using the following command line options

  • INSTALLASSERVICE=true

  • STARTAFTERINSTALL=true

  • STARTWHENWINDOWSSTARTS=true

Once installed, Elasticsearch will appear within the Services control panel:

msi installer installed service

and can be stopped and restarted from within the control panel, or from the command line using:

with Command Prompt:

sc.exe stop Elasticsearch
sc.exe start Elasticsearch

with PowerShell:

Get-Service Elasticsearch | Stop-Service
Get-Service Elasticsearch | Start-Service

Changes can be made to jvm.options and elasticsearch.yml configuration files to configure the service after installation. Most changes (like JVM settings) will require a restart of the service in order to take effect.

Upgrade using the graphical user interface (GUI)

The .msi package supports upgrading an installed version of Elasticsearch to a newer version. The upgrade process through the GUI handles upgrading all installed plugins as well as retaining both your data and configuration.

Downloading and double-clicking on a newer version of the .msi package will launch the GUI wizard. The first step will list the read-only properties from the previous installation:

msi installer upgrade notice

The next step allows certain configuration options to be changed:

msi installer upgrade configuration

Finally, the plugins step allows currently installed plugins to be upgraded or removed, and for plugins not currently installed, to be downloaded and installed:

msi installer upgrade plugins

Upgrade using the command line

The .msi can also upgrade Elasticsearch using the command line.

Important

A command line upgrade requires passing the same command line properties as used at first install time; the Windows Installer does not remember these properties.

For example, if you originally installed with the command line options PLUGINS="ingest-geoip" and LOCKMEMORY="true", then you must pass these same values when performing an upgrade from the command line.

The exception to this is the INSTALLDIR parameter (if originally specified), which must be a different directory to the current installation. If setting INSTALLDIR, the final directory in the path must be the version of Elasticsearch e.g.

C:\Program Files\Elastic\Elasticsearch{backslash}{version}

The simplest upgrade, assuming Elasticsearch was installed using all defaults, is achieved by first navigating to the download directory, then running:

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn

Similar to the install process, a path to a log file for the upgrade process can be passed using the /l command line argument

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn /l upgrade.log

Uninstall using Add/Remove Programs

The .msi package handles uninstallation of all directories and files added as part of installation.

Warning
Uninstallation will remove all contents created as part of installation, except for data, config or logs directories. It is recommended that you make a copy of your data directory before upgrading or consider using the snapshot API.

MSI installer packages do not provide a GUI for uninstallation. An installed program can be uninstalled by pressing the Windows key and typing add or remove programs to open the system settings.

Once opened, find the Elasticsearch installation within the list of installed applications, click and choose Uninstall:

msi installer uninstall

This will launch the uninstallation process.

Uninstall using the command line

Uninstallation can also be performed from the command line by navigating to the directory containing the .msi package and running:

start /wait msiexec.exe /x elasticsearch-{version}.msi /qn

Similar to the install process, a path to a log file for the uninstallation process can be passed using the /l command line argument

start /wait msiexec.exe /x elasticsearch-{version}.msi /qn /l uninstall.log

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install {es} with Docker

{es} is also available as Docker images. The images use centos:7 as the base image.

A list of all published Docker images and tags is available at www.docker.elastic.co. The source files are in Github.

These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

Pulling the image

Obtaining {es} for Docker is as simple as issuing a docker pull command against the Elastic Docker registry.

docker pull docker.elastic.co/elasticsearch/elasticsearch:{version}

Alternatively, you can download other Docker images that contain only features available under the Apache 2.0 license. To download the images, go to www.docker.elastic.co.

Running {es} from the command line

Development mode

{es} can be quickly started for development or testing use with the following command:

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:{version}
Production mode
Important

The vm.max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:

  • Linux

    The vm.max_map_count setting should be set permanently in /etc/sysctl.conf:

    $ grep vm.max_map_count /etc/sysctl.conf
    vm.max_map_count=262144

    To apply the setting on a live system type: sysctl -w vm.max_map_count=262144

  • macOS with Docker for Mac

    The vm.max_map_count setting must be set within the xhyve virtual machine:

    $ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

    Just press enter and configure the sysctl setting as you would for Linux:

    sysctl -w vm.max_map_count=262144
  • Windows and macOS with Docker Toolbox

    The vm.max_map_count setting must be set via docker-machine:

    docker-machine ssh
    sudo sysctl -w vm.max_map_count=262144

The following example brings up a cluster comprising two {es} nodes. To bring up the cluster, use the docker-compose.yml and just type:

docker-compose up
Note
docker-compose is not pre-installed with Docker on Linux. Instructions for installing it can be found on the Docker Compose webpage.

The node elasticsearch listens on localhost:9200 while elasticsearch2 talks to elasticsearch over a Docker network.

This example also uses Docker named volumes, called esdata1 and esdata2 which will be created if not already present.

docker-compose.yml:

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:{version}
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:{version}
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

To stop the cluster, type docker-compose down. Data volumes will persist, so it’s possible to start the cluster again with the same data using docker-compose up. To destroy the cluster and the data volumes, just type docker-compose down -v.

Inspect status of cluster:
curl http://127.0.0.1:9200/_cat/health
1472225929 15:38:49 docker-cluster green 2 2 4 2 0 0 0 0 - 100.0%

Log messages go to the console and are handled by the configured Docker logging driver. By default you can access logs with docker logs.

Configuring {es} with Docker

{es} loads its configuration from files under /usr/share/elasticsearch/config/. These configuration files are documented in Configuring Elasticsearch and Setting JVM options.

The image offers several methods for configuring {es} settings with the conventional approach being to provide customized files, that is to say, elasticsearch.yml. It’s also possible to use environment variables to set options:

A. Present the parameters via Docker environment variables

For example, to define the cluster name with docker run you can pass -e "cluster.name=mynewclustername". Double quotes are required.

B. Bind-mounted configuration

Create your custom config file and mount this over the image’s corresponding file. For example, bind-mounting a custom_elasticsearch.yml with docker run can be accomplished with the parameter:

-v full_path_to/custom_elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
Important
The container runs {es} as user elasticsearch using uid:gid 1000:1000. Bind mounted host directories and files, such as custom_elasticsearch.yml above, need to be accessible by this user. For the data and log dirs, such as /usr/share/elasticsearch/data, write access is required as well. Also see note 1 below.
C. Customized image

In some environments, it may make more sense to prepare a custom image containing your configuration. A Dockerfile to achieve this may be as simple as:

FROM docker.elastic.co/elasticsearch/elasticsearch:{version}
COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/

You could then build and try the image with something like:

docker build --tag=elasticsearch-custom .
docker run -ti -v /usr/share/elasticsearch/data elasticsearch-custom

Some plugins require additional security permissions. You have to explicitly accept them either by attaching a tty when you run the Docker image and accepting yes at the prompts, or inspecting the security permissions separately and if you are comfortable with them adding the --batch flag to the plugin install command. See {plugins}/_other_command_line_parameters.html[Plugin Management documentation] for more details.

D. Override the image’s default CMD

Options can be passed as command-line options to the {es} process by overriding the default command for the image. For example:

docker run <various parameters> bin/elasticsearch -Ecluster.name=mynewclustername

Configuring SSL/TLS with the {es} Docker image

Notes for production use and defaults

We have collected a number of best practices for production use. Any Docker parameters mentioned below assume the use of docker run.

  1. By default, {es} runs inside the container as user elasticsearch using uid:gid 1000:1000.

    Caution
    One exception is Openshift which runs containers using an arbitrarily assigned user ID. Openshift will present persistent volumes with the gid set to 0 which will work without any adjustments.

    If you are bind-mounting a local directory or file, ensure it is readable by this user, while the data and log dirs additionally require write access. A good strategy is to grant group access to gid 1000 or 0 for the local directory. As an example, to prepare a local directory for storing data through a bind-mount:

    mkdir esdatadir
    chmod g+rwx esdatadir
    chgrp 1000 esdatadir

    As a last resort, you can also force the container to mutate the ownership of any bind-mounts used for the data and log dirs through the environment variable TAKE_FILE_OWNERSHIP. In this case, they will be owned by uid:gid 1000:0 providing read/write access to the {es} process as required.

  2. It is important to ensure increased ulimits for nofile and nproc are available for the {es} containers. Verify the init system for the Docker daemon is already setting those to acceptable values and, if needed, adjust them in the Daemon, or override them per container, for example using docker run:

    --ulimit nofile=65535:65535
    Note
    One way of checking the Docker daemon defaults for the aforementioned ulimits is by running:
    docker run --rm centos:7 /bin/bash -c 'ulimit -Hn && ulimit -Sn && ulimit -Hu && ulimit -Su'
  3. Swapping needs to be disabled for performance and node stability. This can be achieved through any of the methods mentioned in the {es} docs. If you opt for the bootstrap.memory_lock: true approach, apart from defining it through any of the configuration methods, you will additionally need the memlock: true ulimit, either defined in the Docker Daemon or specifically set for the container. This is demonstrated above in the docker-compose.yml. If using docker run:

    -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1
  4. The image exposes TCP ports 9200 and 9300. For clusters it is recommended to randomize the published ports with --publish-all, unless you are pinning one container per host.

  5. Use the ES_JAVA_OPTS environment variable to set heap size. For example, to use 16GB use -e ES_JAVA_OPTS="-Xms16g -Xmx16g" with docker run. Note that while the default configuration file jvm.options sets a default heap of 1GB, any value you set in ES_JAVA_OPTS will override it.

    Note
    You still need to configure the heap size even if you are limiting memory access to the container.

    While setting the heap size via an environment variable is the recommended method, you can also configure this by bind-mounting your own jvm.options file under /usr/share/elasticsearch/config/. The file that {es} provides contains some important settings, so you should start by taking a copy of jvm.options from an {es} container and editing it as you require.

  6. Pin your deployments to a specific version of the {es} Docker image. For example, docker.elastic.co/elasticsearch/elasticsearch:{version}.

  7. Always use a volume bound on /usr/share/elasticsearch/data, as shown in the production example, for the following reasons:

    1. The data of your elasticsearch node won’t be lost if the container is killed

    2. {es} is I/O sensitive and the Docker storage driver is not ideal for fast I/O

    3. It allows the use of advanced Docker volume plugins

  8. If you are using the devicemapper storage driver, make sure you are not using the default loop-lvm mode. Configure docker-engine to use direct-lvm instead.

  9. Consider centralizing your logs by using a different logging driver. Also note that the default json-file logging driver is not ideally suited for production use.

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Configuring Elasticsearch

Elasticsearch ships with good defaults and requires very little configuration. Most settings can be changed on a running cluster using the Cluster Update Settings API.

The configuration files should contain settings which are node-specific (such as node.name and paths), or settings which a node requires in order to be able to join a cluster, such as cluster.name and network.host.

Config files location

Elasticsearch has three configuration files:

  • elasticsearch.yml for configuring Elasticsearch

  • jvm.options for configuring Elasticsearch JVM settings

  • log4j2.properties for configuring Elasticsearch logging

These files are located in the config directory, whose default location depends on whether or not the installation is from an archive distribution (tar.gz or zip) or a package distribution (Debian or RPM packages).

For the archive distributions, the config directory location defaults to $ES_HOME/config. The location of the config directory can be changed via the ES_PATH_CONF environment variable as follows:

ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch

Alternatively, you can export the ES_PATH_CONF environment variable via the command line or via your shell profile.

For the package distributions, the config directory location defaults to /etc/elasticsearch. The location of the config directory can also be changed via the ES_PATH_CONF environment variable, but note that setting this in your shell is not sufficient. Instead, this variable is sourced from /etc/default/elasticsearch (for the Debian package) and /etc/sysconfig/elasticsearch (for the RPM package). You will need to edit the ES_PATH_CONF=/etc/elasticsearch entry in one of these files accordingly to change the config directory location.

Config file format

The configuration format is YAML. Here is an example of changing the path of the data and logs directories:

path:
    data: /var/lib/elasticsearch
    logs: /var/log/elasticsearch

Settings can also be flattened as follows:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Environment variable substitution

Environment variables referenced with the ${…​} notation within the configuration file will be replaced with the value of the environment variable. For example:

node.name:    ${HOSTNAME}
network.host: ${ES_NETWORK_HOST}

Prompting for settings

Note
Prompting for settings is deprecated. Please use Secure settings for sensitive property values. Not all settings can be converted to use Secure settings.

For settings that you do not wish to store in the configuration file, you can use the value ${prompt.text} or ${prompt.secret} and start Elasticsearch in the foreground. ${prompt.secret} has echoing disabled so that the value entered will not be shown in your terminal; ${prompt.text} will allow you to see the value as you type it in. For example:

node:
  name: ${prompt.text}

When starting Elasticsearch, you will be prompted to enter the actual value like so:

Enter value for [node.name]:
Note
Elasticsearch will not start if ${prompt.text} or ${prompt.secret} is used in the settings and the process is run as a service or in the background.

Values for environment variables must be simple strings. Use a comma-separated string to provide values that Elasticsearch will parse as a list. For example, Elasticsearch will split the following string into a list of values for the ${HOSTNAME} environment variable:

export HOSTNAME=“host1,host2"

Cluster and node setting types

Cluster and node settings can be categorized based on how they are configured:

Dynamic

You can configure and update dynamic settings on a running cluster using the cluster update settings API.

You can also configure dynamic settings locally on an unstarted or shut down node using elasticsearch.yml.

Tip
It’s best to set dynamic, cluster-wide settings with the cluster update settings API and use elasticsearch.yml only for local configurations. Using the cluster update settings API ensures the setting is the same on all nodes. If you accidentally configure different settings in elasticsearch.yml on different nodes, it can be difficult to notice discrepancies.
Static

Static settings can only be configured on an unstarted or shut down node using elasticsearch.yml.

Static settings must be set on every relevant node in the cluster.

Setting JVM options

You should rarely need to change Java Virtual Machine (JVM) options. If you do, the most likely change is setting the heap size. The remainder of this document explains in detail how to set JVM options.

The preferred method of setting JVM options (including system properties and JVM flags) is via the jvm.options configuration file. The default location of this file is config/jvm.options (when installing from the tar or zip distributions) and /etc/elasticsearch/jvm.options (when installing from the Debian or RPM packages).

This file contains a line-delimited list of JVM arguments following a special syntax:

  • lines consisting of whitespace only are ignored

  • lines beginning with # are treated as comments and are ignored

    # this is a comment
  • lines beginning with a - are treated as a JVM option that applies independent of the version of the JVM

    -Xmx2g
  • lines beginning with a number followed by a : followed by a - are treated as a JVM option that applies only if the version of the JVM matches the number

    8:-Xmx2g
  • lines beginning with a number followed by a - followed by a : are treated as a JVM option that applies only if the version of the JVM is greater than or equal to the number

    8-:-Xmx2g
  • lines beginning with a number followed by a - followed by a number followed by a : are treated as a JVM option that applies only if the version of the JVM falls in the range of the two numbers

    8-9:-Xmx2g
  • all other lines are rejected

You can add custom JVM flags to this file and check this configuration into your version control system.

An alternative mechanism for setting Java Virtual Machine options is via the ES_JAVA_OPTS environment variable. For instance:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djava.io.tmpdir=/path/to/temp/dir"
./bin/elasticsearch

When using the RPM or Debian packages, ES_JAVA_OPTS can be specified in the system configuration file.

The JVM has a built-in mechanism for observing the JAVA_TOOL_OPTIONS environment variable. We intentionally ignore this environment variable in our packaging scripts. The primary reason for this is that on some OS (e.g., Ubuntu) there are agents installed by default via this environment variable that we do not want interfering with Elasticsearch.

Additionally, some other Java programs support the JAVA_OPTS environment variable. This is not a mechanism built into the JVM but instead a convention in the ecosystem. However, we do not support this environment variable, instead supporting setting JVM options via the jvm.options file or the environment variable ES_JAVA_OPTS as above.

Secure settings

Some settings are sensitive, and relying on filesystem permissions to protect their values is not sufficient. For this use case, Elasticsearch provides a keystore and the elasticsearch-keystore tool to manage the settings in the keystore.

Note
All commands here should be run as the user which will run Elasticsearch.
Important
Only some settings are designed to be read from the keystore. However, the keystore has no validation to block unsupported settings. Adding unsupported settings to the keystore will cause {es} to fail to start. See documentation for each setting to see if it is supported as part of the keystore.
Note
All the modifications to the keystore take affect only after restarting Elasticsearch.
Note
The elasticsearch keystore currently only provides obfuscation. In the future, password protection will be added.

These settings, just like the regular ones in the elasticsearch.yml config file, need to be specified on each node in the cluster. Currently, all secure settings are node-specific settings that must have the same value on every node.

Creating the keystore

To create the elasticsearch.keystore, use the create command:

bin/elasticsearch-keystore create

The file elasticsearch.keystore will be created alongside elasticsearch.yml.

Listing settings in the keystore

A list of the settings in the keystore is available with the list command:

bin/elasticsearch-keystore list

Adding string settings

Sensitive string settings, like authentication credentials for cloud plugins, can be added using the add command:

bin/elasticsearch-keystore add the.setting.name.to.set

The tool will prompt for the value of the setting. To pass the value through stdin, use the --stdin flag:

cat /file/containing/setting/value | bin/elasticsearch-keystore add --stdin the.setting.name.to.set

Adding file settings

You can add sensitive files, like authentication key files for cloud plugins, using the add-file command. Be sure to include your file path as an argument after the setting name.

bin/elasticsearch-keystore add-file the.setting.name.to.set /path/example-file.json

Removing settings

To remove a setting from the keystore, use the remove command:

bin/elasticsearch-keystore remove the.setting.name.to.remove

Upgrading the keystore

Occasionally, the internal format of the keystore changes. When Elasticsearch is installed from a package manager, an upgrade of the on-disk keystore to the new format is done during package upgrade. In other cases, Elasticsearch will perform such an upgrade during node startup. This requires that Elasticsearch have write permissions to the directory that contains the keystore. Alternatively, you can manually perform such an upgrade by using the upgrade command:

bin/elasticsearch-keystore upgrade

Reloadable secure settings

Just like the settings values in elasticsearch.yml, changes to the keystore contents are not automatically applied to the running elasticsearch node. Re-reading settings requires a node restart. However, certain secure settings are marked as reloadable. Such settings can be re-read and applied on a running node.

The values of all secure settings, reloadable or not, must be identical across all cluster nodes. After making the desired secure settings changes, using the bin/elasticsearch-keystore add command, call:

POST _nodes/reload_secure_settings

This API will decrypt and re-read the entire keystore, on every cluster node, but only the reloadable secure settings will be applied. Changes to other settings will not go into effect until the next restart. Once the call returns, the reload has been completed, meaning that all internal datastructures dependent on these settings have been changed. Everything should look as if the settings had the new value from the start.

When changing multiple reloadable secure settings, modify all of them, on each cluster node, and then issue a reload_secure_settings call, instead of reloading after each modification.

Logging configuration

Elasticsearch uses Log4j 2 for logging. Log4j 2 can be configured using the log4j2.properties file. Elasticsearch exposes three properties, ${sys:es.logs.base_path}, ${sys:es.logs.cluster_name}, and ${sys:es.logs.node_name} (if the node name is explicitly set via node.name) that can be referenced in the configuration file to determine the location of the log files. The property ${sys:es.logs.base_path} will resolve to the log directory, ${sys:es.logs.cluster_name} will resolve to the cluster name (used as the prefix of log filenames in the default configuration), and ${sys:es.logs.node_name} will resolve to the node name (if the node name is explicitly set).

For example, if your log directory (path.logs) is /var/log/elasticsearch and your cluster is named production then ${sys:es.logs.base_path} will resolve to /var/log/elasticsearch and ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log will resolve to /var/log/elasticsearch/production.log.

appender.rolling.type = RollingFile (1)
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log (2)
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n
appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz (3)
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy (4)
appender.rolling.policies.time.interval = 1 (5)
appender.rolling.policies.time.modulate = true (6)
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy (7)
appender.rolling.policies.size.size = 256MB (8)
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.action.type = Delete (9)
appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.action.condition.type = IfFileName (10)
appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-* (11)
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize (12)
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB (13)
  1. Configure the RollingFile appender

  2. Log to /var/log/elasticsearch/production.log

  3. Roll logs to /var/log/elasticsearch/production-yyyy-MM-dd-i.log; logs will be compressed on each roll and i will be incremented

  4. Use a time-based roll policy

  5. Roll logs on a daily basis

  6. Align rolls on the day boundary (as opposed to rolling every twenty-four hours)

  7. Using a size-based roll policy

  8. Roll logs after 256 MB

  9. Use a delete action when rolling logs

  10. Only delete logs matching a file pattern

  11. The pattern is to only delete the main logs

  12. Only delete if we have accumulated too many compressed logs

  13. The size condition on the compressed logs is 2 GB

Note
Log4j’s configuration parsing gets confused by any extraneous whitespace; if you copy and paste any Log4j settings on this page, or enter any Log4j configuration in general, be sure to trim any leading and trailing whitespace.

Note than you can replace .gz by .zip in appender.rolling.filePattern to compress the rolled logs using the zip format. If you remove the .gz extension then logs will not be compressed as they are rolled.

If you want to retain log files for a specified period of time, you can use a rollover strategy with a delete action.

appender.rolling.strategy.type = DefaultRolloverStrategy (1)
appender.rolling.strategy.action.type = Delete (2)
appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path} (3)
appender.rolling.strategy.action.condition.type = IfFileName (4)
appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-* (5)
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified (6)
appender.rolling.strategy.action.condition.nested_condition.age = 7D (7)
  1. Configure the DefaultRolloverStrategy

  2. Configure the Delete action for handling rollovers

  3. The base path to the Elasticsearch logs

  4. The condition to apply when handling rollovers

  5. Delete files from the base path matching the glob ${sys:es.logs.cluster_name}-*; this is the glob that log files are rolled to; this is needed to only delete the rolled Elasticsearch logs but not also delete the deprecation and slow logs

  6. A nested condition to apply to files matching the glob

  7. Retain logs for seven days

Multiple configuration files can be loaded (in which case they will get merged) as long as they are named log4j2.properties and have the Elasticsearch config directory as an ancestor; this is useful for plugins that expose additional loggers. The logger section contains the java packages and their corresponding log level. The appender section contains the destinations for the logs. Extensive information on how to customize logging and all the supported appenders can be found on the Log4j documentation.

Configuring logging levels

There are four ways to configuring logging levels, each having situations in which they are appropriate to use.

  1. Via the command-line: -E <name of logging hierarchy>=<level> (e.g., -E logger.org.elasticsearch.discovery=debug). This is most appropriate when you are temporarily debugging a problem on a single node (for example, a problem with startup, or during development).

  2. Via elasticsearch.yml: <name of logging hierarchy>: <level> (e.g., logger.org.elasticsearch.discovery: debug). This is most appropriate when you are temporarily debugging a problem but are not starting Elasticsearch via the command-line (e.g., via a service) or you want a logging level adjusted on a more permanent basis.

  3. Via cluster settings:

    PUT /_cluster/settings
    {
      "transient": {
        "<name of logging hierarchy>": "<level>"
      }
    }

    For example:

    PUT /_cluster/settings
    {
      "transient": {
        "logger.org.elasticsearch.discovery": "DEBUG"
      }
    }

    This is most appropriate when you need to dynamically need to adjust a logging level on an actively-running cluster.

  4. Via the log4j2.properties:

    logger.<unique_identifier>.name = <name of logging hierarchy>
    logger.<unique_identifier>.level = <level>

    For example:

    logger.discovery.name = org.elasticsearch.discovery
    logger.discovery.level = debug

    This is most appropriate when you need fine-grained control over the logger (for example, you want to send the logger to another file, or manage the logger differently; this is a rare use-case).

Deprecation logging

In addition to regular logging, Elasticsearch allows you to enable logging of deprecated actions. For example this allows you to determine early, if you need to migrate certain functionality in the future. By default, deprecation logging is enabled at the WARN level, the level at which all deprecation log messages will be emitted.

logger.deprecation.level = warn

This will create a daily rolling deprecation log file in your log directory. Check this file regularly, especially when you intend to upgrade to a new major version.

The default logging configuration has set the roll policy for the deprecation logs to roll and compress after 1 GB, and to preserve a maximum of five log files (four rolled logs, and the active log).

You can disable it in the config/log4j2.properties file by setting the deprecation log level to error like this:

logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = error

Important Elasticsearch configuration

While Elasticsearch requires very little configuration, there are a number of settings which need to be considered before going into production.

The following settings must be considered before going to production:

path.data and path.logs

If you are using the .zip or .tar.gz archives, the data and logs directories are sub-folders of $ES_HOME. If these important folders are left in their default locations, there is a high risk of them being deleted while upgrading Elasticsearch to a new version.

In production use, you will almost certainly want to change the locations of the data and log folder:

path:
  logs: /var/log/elasticsearch
  data: /var/data/elasticsearch

The RPM and Debian distributions already use custom paths for data and logs.

The path.data settings can be set to multiple paths, in which case all paths will be used to store data (although the files belonging to a single shard will all be stored on the same data path):

path:
  data:
    - /mnt/elasticsearch_1
    - /mnt/elasticsearch_2
    - /mnt/elasticsearch_3

cluster.name

A node can only join a cluster when it shares its cluster.name with all the other nodes in the cluster. The default name is elasticsearch, but you should change it to an appropriate name which describes the purpose of the cluster.

cluster.name: logging-prod

Make sure that you don’t reuse the same cluster names in different environments, otherwise you might end up with nodes joining the wrong cluster.

node.name

By default, Elasticsearch will use the first seven characters of the randomly generated UUID as the node id. Note that the node id is persisted and does not change when a node restarts and therefore the default node name will also not change.

It is worth configuring a more meaningful name which will also have the advantage of persisting after restarting the node:

node.name: prod-data-2

The node.name can also be set to the server’s HOSTNAME as follows:

node.name: ${HOSTNAME}

network.host

By default, Elasticsearch binds to loopback addresses only — e.g. 127.0.0.1 and [::1]. This is sufficient to run a single development node on a server.

Tip
In fact, more than one node can be started from the same $ES_HOME location on a single node. This can be useful for testing Elasticsearch’s ability to form clusters, but it is not a configuration recommended for production.

In order to form a cluster with nodes on other servers, your node will need to bind to a non-loopback address. While there are many network settings, usually all you need to configure is network.host:

network.host: 192.168.1.10

The network.host setting also understands some special values such as local, site, global and modifiers like :ip4 and :ip6, details of which can be found in Special values for network.host.

Important
As soon as you provide a custom setting for network.host, Elasticsearch assumes that you are moving from development mode to production mode, and upgrades a number of system startup checks from warnings to exceptions. See Development mode vs production mode for more information.

Discovery settings

Elasticsearch uses a custom discovery implementation called "Zen Discovery" for node-to-node clustering and master election. There are two important discovery settings that should be configured before going to production.

discovery.zen.ping.unicast.hosts

Out of the box, without any network configuration, Elasticsearch will bind to the available loopback addresses and will scan ports 9300 to 9305 to try to connect to other nodes running on the same server. This provides an auto- clustering experience without having to do any configuration.

When the moment comes to form a cluster with nodes on other servers, you have to provide a seed list of other nodes in the cluster that are likely to be live and contactable. This can be specified as follows:

discovery.zen.ping.unicast.hosts:
   - 192.168.1.10:9300
   - 192.168.1.11 (1)
   - seeds.mydomain.com (2)
  1. The port will default to transport.profiles.default.port and fallback to transport.port if not specified.

  2. A hostname that resolves to multiple IP addresses will try all resolved addresses.

discovery.zen.minimum_master_nodes

To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.

Without this setting, a cluster that suffers a network failure is at risk of having the cluster split into two independent clusters — a split brain — which will lead to data loss. A more detailed explanation is provided in Avoiding split brain with minimum_master_nodes.

To avoid a split brain, this setting should be set to a quorum of master-eligible nodes:

(master_eligible_nodes / 2) + 1

In other words, if there are three master-eligible nodes, then minimum master nodes should be set to (3 / 2) + 1 or 2:

discovery.zen.minimum_master_nodes: 2

Setting the heap size

By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. When moving to production, it is important to configure heap size to ensure that Elasticsearch has enough heap available.

Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings. These two settings must be equal to each other.

The value for these setting depends on the amount of RAM available on your server. Good rules of thumb are:

  • The more heap available to Elasticsearch, the more memory it can use for caching. But note that too much heap can subject you to long garbage collection pauses.

  • Set Xms and Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches.

  • Don’t set Xms and Xmx to above the cutoff that the JVM uses for compressed object pointers (compressed oops); the exact cutoff varies but is near 32 GB. You can verify that you are under the limit by looking for a line in the logs like the following:

    heap size [1.9gb], compressed ordinary object pointers [true]
  • Even better, try to stay below the threshold for zero-based compressed oops; the exact cutoff varies but 26 GB is safe on most systems, but can be as large as 30 GB on some systems. You can verify that you are under the limit by starting Elasticsearch with the JVM options -XX:+UnlockDiagnosticVMOptions -XX:+PrintCompressedOopsMode and looking for a line like the following:

    heap address: 0x000000011be00000, size: 27648 MB, zero based Compressed Oops

    showing that zero-based compressed oops are enabled instead of

    heap address: 0x0000000118400000, size: 28672 MB, Compressed Oops with base: 0x00000001183ff000

Here are examples of how to set the heap size via the jvm.options file:

-Xms2g (1)
-Xmx2g (2)
  1. Set the minimum heap size to 2g.

  2. Set the maximum heap size to 2g.

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:

ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch (1)
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch (2)
  1. Set the minimum and maximum heap size to 2 GB.

  2. Set the minimum and maximum heap size to 4000 MB.

Note
Configuring the heap for the Windows service is different than the above. The values initially populated for the Windows service can be configured as above but are different after the service has been installed. Consult the Windows service documentation for additional details.

JVM heap dump path

By default, Elasticsearch configures the JVM to dump the heap on out of memory exceptions to the default data directory (this is /var/lib/elasticsearch for the RPM and Debian package distributions, and the data directory under the root of the Elasticsearch installation for the tar and zip archive distributions). If this path is not suitable for receiving heap dumps, you should modify the entry -XX:HeapDumpPath=…​ in jvm.options. If you specify a directory, the JVM will generate a filename for the heap dump based on the PID of the running instance. If you specify a fixed filename instead of a directory, the file must not exist when the JVM needs to perform a heap dump on an out of memory exception, otherwise the heap dump will fail.

GC logging

By default, Elasticsearch enables GC logs. These are configured in jvm.options and default to the same default location as the Elasticsearch logs. The default configuration rotates the logs every 64 MB and can consume up to 2 GB of disk space.

Temp directory

By default, Elasticsearch uses a private temporary directory that the startup script creates immediately below the system temporary directory.

On some Linux distributions a system utility will clean files and directories from /tmp if they have not been recently accessed. This can lead to the private temporary directory being removed while Elasticsearch is running if features that require the temporary directory are not used for a long time. This causes problems if a feature that requires the temporary directory is subsequently used.

If you install Elasticsearch using the .deb or .rpm packages and run it under systemd then the private temporary directory that Elasticsearch uses is excluded from periodic cleanup.

However, if you intend to run the .tar.gz distribution on Linux for an extended period then you should consider creating a dedicated temporary directory for Elasticsearch that is not under a path that will have old files and directories cleaned from it. This directory should have permissions set so that only the user that Elasticsearch runs as can access it. Then set the $ES_TMPDIR environment variable to point to it before starting Elasticsearch.

JVM fatal error logs

By default, Elasticsearch configures the JVM to write fatal error logs to the default logging directory (this is /var/log/elasticsearch for the RPM and Debian package distributions, and the logs directory under the root of the Elasticsearch installation for the tar and zip archive distributions). These are logs produced by the JVM when it encounters a fatal error (e.g., a segmentation fault). If this path is not suitable for receiving logs, you should modify the entry -XX:ErrorFile=…​ in jvm.options to an alternate path.

Important System Configuration

Ideally, Elasticsearch should run alone on a server and use all of the resources available to it. In order to do so, you need to configure your operating system to allow the user running Elasticsearch to access more resources than allowed by default.

The following settings must be considered before going to production:

Development mode vs production mode

By default, Elasticsearch assumes that you are working in development mode. If any of the above settings are not configured correctly, a warning will be written to the log file, but you will be able to start and run your Elasticsearch node.

As soon as you configure a network setting like network.host, Elasticsearch assumes that you are moving to production and will upgrade the above warnings to exceptions. These exceptions will prevent your Elasticsearch node from starting. This is an important safety measure to ensure that you will not lose data because of a malconfigured server.

Configuring system settings

Where to configure systems settings depends on which package you have used to install Elasticsearch, and which operating system you are using.

When using the .zip or .tar.gz packages, system settings can be configured:

When using the RPM or Debian packages, most system settings are set in the system configuration file. However, systems which use systemd require that system limits are specified in a systemd configuration file.

ulimit

On Linux systems, ulimit can be used to change resource limits on a temporary basis. Limits usually need to be set as root before switching to the user that will run Elasticsearch. For example, to set the number of open file handles (ulimit -n) to 65,536, you can do the following:

sudo su  (1)
ulimit -n 65535 (2)
su elasticsearch (3)
  1. Become root.

  2. Change the max number of open files.

  3. Become the elasticsearch user in order to start Elasticsearch.

The new limit is only applied during the current session.

You can consult all currently applied limits with ulimit -a.

/etc/security/limits.conf

On Linux systems, persistent limits can be set for a particular user by editing the /etc/security/limits.conf file. To set the maximum number of open files for the elasticsearch user to 65,535, add the following line to the limits.conf file:

elasticsearch  -  nofile  65535

This change will only take effect the next time the elasticsearch user opens a new session.

Note
Ubuntu and limits.conf

Ubuntu ignores the limits.conf file for processes started by init.d. To enable the limits.conf file, edit /etc/pam.d/su and uncomment the following line:

# session    required   pam_limits.so

Sysconfig file

When using the RPM or Debian packages, system settings and environment variables can be specified in the system configuration file, which is located in:

RPM

/etc/sysconfig/elasticsearch

Debian

/etc/default/elasticsearch

However, for systems which uses systemd, system limits need to be specified via systemd.

Systemd configuration

When using the RPM or Debian packages on systems that use systemd, system limits must be specified via systemd.

The systemd service file (/usr/lib/systemd/system/elasticsearch.service) contains the limits that are applied by default.

To override them, add a file called /etc/systemd/system/elasticsearch.service.d/override.conf (alternatively, you may run sudo systemctl edit elasticsearch which opens the file automatically inside your default editor). Set any changes in this file, such as:

[Service]
LimitMEMLOCK=infinity

Once finished, run the following command to reload units:

sudo systemctl daemon-reload

Disable swapping

Most operating systems try to use as much memory as possible for file system caches and eagerly swap out unused application memory. This can result in parts of the JVM heap or even its executable pages being swapped out to disk.

Swapping is very bad for performance, for node stability, and should be avoided at all costs. It can cause garbage collections to last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. In a resilient distributed system, it’s more effective to let the operating system kill the node.

There are three approaches to disabling swapping. The preferred option is to completely disable swap. If this is not an option, whether or not to prefer minimizing swappiness versus memory locking is dependent on your environment.

Disable all swap files

Usually Elasticsearch is the only service running on a box, and its memory usage is controlled by the JVM options. There should be no need to have swap enabled.

On Linux systems, you can disable swap temporarily by running:

sudo swapoff -a

To disable it permanently, you will need to edit the /etc/fstab file and comment out any lines that contain the word swap.

On Windows, the equivalent can be achieved by disabling the paging file entirely via System Properties → Advanced → Performance → Advanced → Virtual memory.

Configure swappiness

Another option available on Linux systems is to ensure that the sysctl value vm.swappiness is set to 1. This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions.

Enable bootstrap.memory_lock

Another option is to use mlockall on Linux/Unix systems, or VirtualLock on Windows, to try to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out. This can be done, by adding this line to the config/elasticsearch.yml file:

bootstrap.memory_lock: true
Warning
mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available!

After starting Elasticsearch, you can see whether this setting was applied successfully by checking the value of mlockall in the output from this request:

GET _nodes?filter_path=**.mlockall

If you see that mlockall is false, then it means that the mlockall request has failed. You will also see a line with more information in the logs with the words Unable to lock JVM Memory.

The most probable reason, on Linux/Unix systems, is that the user running Elasticsearch doesn’t have permission to lock memory. This can be granted as follows:

.zip and .tar.gz

Set ulimit -l unlimited as root before starting Elasticsearch, or set memlock to unlimited in /etc/security/limits.conf.

RPM and Debian

Set MAX_LOCKED_MEMORY to unlimited in the system configuration file (or see below for systems using systemd).

Systems using systemd

Set LimitMEMLOCK to infinity in the systemd configuration.

Another possible reason why mlockall can fail is that the JNA temporary directory (usually a sub-directory of /tmp) is mounted with the noexec option. This can be solved by specifying a new temporary directory for JNA using the ES_JAVA_OPTS environment variable:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djna.tmpdir=<path>"
./bin/elasticsearch

or setting this JVM flag in the jvm.options configuration file.

File Descriptors

Note
This is only relevant for Linux and macOS and can be safely ignored if running Elasticsearch on Windows. On Windows that JVM uses an API limited only by available resources.

Elasticsearch uses a lot of file descriptors or file handles. Running out of file descriptors can be disastrous and will most probably lead to data loss. Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65,536 or higher.

For the .zip and .tar.gz packages, set ulimit -n 65535 as root before starting Elasticsearch, or set nofile to 65535 in /etc/security/limits.conf.

On macOS, you must also pass the JVM option -XX:-MaxFDLimit to Elasticsearch in order for it to make use of the higher file descriptor limit.

RPM and Debian packages already default the maximum number of file descriptors to 65535 and do not require further configuration.

You can check the max_file_descriptors configured for each node using the Nodes Stats API, with:

GET _nodes/stats/process?filter_path=**.max_file_descriptors

Virtual memory

Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.

On Linux, you can increase the limits by running the following command as root:

sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.

The RPM and Debian packages will configure this setting automatically. No further configuration is required.

Number of threads

Elasticsearch uses a number of thread pools for different types of operations. It is important that it is able to create new threads whenever needed. Make sure that the number of threads that the Elasticsearch user can create is at least 4096.

This can be done by setting ulimit -u 4096 as root before starting Elasticsearch, or by setting nproc to 4096 in /etc/security/limits.conf.

The package distributions when run as services under systemd will configure the number of threads for the Elasticsearch process automatically. No additional configuration is required.

DNS cache settings

Elasticsearch runs with a security manager in place. With a security manager in place, the JVM defaults to caching positive hostname resolutions indefinitely and defaults to caching negative hostname resolutions for ten seconds. Elasticsearch overrides this behavior with default values to cache positive lookups for sixty seconds, and to cache negative lookups for ten seconds. These values should be suitable for most environments, including environments where DNS resolutions vary with time. If not, you can edit the values es.networkaddress.cache.ttl and es.networkaddress.cache.negative.ttl in the JVM options. Note that the values networkaddress.cache.ttl=<timeout> and networkaddress.cache.negative.ttl=<timeout> in the Java security policy are ignored by Elasticsearch unless you remove the settings for es.networkaddress.cache.ttl and es.networkaddress.cache.negative.ttl.

JNA temporary directory not mounted with noexec

Note
This is only relevant for Linux.

Elasticsearch uses the Java Native Access (JNA) library for executing some platform-dependent native code. On Linux, the native code backing this library is extracted at runtime from the JNA archive. By default, this code is extracted to the Elasticsearch temporary directory which defaults to a sub-directory of /tmp. Alternatively, this location can be controlled with the JVM flag -Djna.tmpdir=<path>. As the native library is mapped into the JVM virtual address space as executable, the underlying mount point of the location that this code is extracted to must not be mounted with noexec as this prevents the JVM process from being able to map this code as executable. On some hardened Linux installations this is a default mount option for /tmp. One indication that the underlying mount is mounted with noexec is that at startup JNA will fail to load with a java.lang.UnsatisfiedLinkerError exception with a message along the lines of failed to map segment from shared object. Note that the exception message can differ amongst JVM versions. Additionally, the components of Elasticsearch that rely on execution of native code via JNA will fail with messages indicating that it is because JNA is not available. If you are seeing such error messages, you must remount the temporary directory used for JNA to not be mounted with noexec.

Bootstrap Checks

Collectively, we have a lot of experience with users suffering unexpected issues because they have not configured important settings. In previous versions of Elasticsearch, misconfiguration of some of these settings were logged as warnings. Understandably, users sometimes miss these log messages. To ensure that these settings receive the attention that they deserve, Elasticsearch has bootstrap checks upon startup.

These bootstrap checks inspect a variety of Elasticsearch and system settings and compare them to values that are safe for the operation of Elasticsearch. If Elasticsearch is in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If Elasticsearch is in production mode, any bootstrap checks that fail will cause Elasticsearch to refuse to start.

There are some bootstrap checks that are always enforced to prevent Elasticsearch from running with incompatible settings. These checks are documented individually.

Development vs. production mode

By default, Elasticsearch binds to loopback addresses for HTTP and transport (internal) communication. This is fine for downloading and playing with Elasticsearch as well as everyday development, but it’s useless for production systems. To join a cluster, an Elasticsearch node must be reachable via transport communication. To join a cluster via a non-loopback address, a node must bind transport to a non-loopback address and not be using single-node discovery. Thus, we consider an Elasticsearch node to be in development mode if it can not form a cluster with another machine via a non-loopback address, and is otherwise in production mode if it can join a cluster via non-loopback addresses.

Note that HTTP and transport can be configured independently via http.host and transport.host; this can be useful for configuring a single node to be reachable via HTTP for testing purposes without triggering production mode.

Single-node discovery

We recognize that some users need to bind transport to an external interface for testing their usage of the transport client. For this situation, we provide the discovery type single-node (configure it by setting discovery.type to single-node); in this situation, a node will elect itself master and will not join a cluster with any other node.

Forcing the bootstrap checks

If you are running a single node in production, it is possible to evade the bootstrap checks (either by not binding transport to an external interface, or by binding transport to an external interface and setting the discovery type to single-node). For this situation, you can force execution of the bootstrap checks by setting the system property es.enforce.bootstrap.checks to true (set this in Setting JVM options, or by adding -Des.enforce.bootstrap.checks=true to the environment variable ES_JAVA_OPTS). We strongly encourage you to do this if you are in this specific situation. This system property can be used to force execution of the bootstrap checks independent of the node configuration.

Heap size check

If a JVM is started with unequal initial and max heap size, it can be prone to pauses as the JVM heap is resized during system usage. To avoid these resize pauses, it’s best to start the JVM with the initial heap size equal to the maximum heap size. Additionally, if bootstrap.memory_lock is enabled, the JVM will lock the initial size of the heap on startup. If the initial heap size is not equal to the maximum heap size, after a resize it will not be the case that all of the JVM heap is locked in memory. To pass the heap size check, you must configure the heap size.

File descriptor check

File descriptors are a Unix construct for tracking open "files". In Unix though, everything is a file. For example, "files" could be a physical file, a virtual file (e.g., /proc/loadavg), or network sockets. Elasticsearch requires lots of file descriptors (e.g., every shard is composed of multiple segments and other files, plus connections to other nodes, etc.). This bootstrap check is enforced on OS X and Linux. To pass the file descriptor check, you might have to configure file descriptors.

Memory lock check

When the JVM does a major garbage collection it touches every page of the heap. If any of those pages are swapped out to disk they will have to be swapped back in to memory. That causes lots of disk thrashing that Elasticsearch would much rather use to service requests. There are several ways to configure a system to disallow swapping. One way is by requesting the JVM to lock the heap in memory through mlockall (Unix) or virtual lock (Windows). This is done via the Elasticsearch setting bootstrap.memory_lock. However, there are cases where this setting can be passed to Elasticsearch but Elasticsearch is not able to lock the heap (e.g., if the elasticsearch user does not have memlock unlimited). The memory lock check verifies that if the bootstrap.memory_lock setting is enabled, that the JVM was successfully able to lock the heap. To pass the memory lock check, you might have to configure bootstrap.memory_lock.

Maximum number of threads check

Elasticsearch executes requests by breaking the request down into stages and handing those stages off to different thread pool executors. There are different thread pool executors for a variety of tasks within Elasticsearch. Thus, Elasticsearch needs the ability to create a lot of threads. The maximum number of threads check ensures that the Elasticsearch process has the rights to create enough threads under normal use. This check is enforced only on Linux. If you are on Linux, to pass the maximum number of threads check, you must configure your system to allow the Elasticsearch process the ability to create at least 4096 threads. This can be done via /etc/security/limits.conf using the nproc setting (note that you might have to increase the limits for the root user too).

Max file size check

The segment files that are the components of individual shards and the translog generations that are components of the translog can get large (exceeding multiple gigabytes). On systems where the max size of files that can be created by the Elasticsearch process is limited, this can lead to failed writes. Therefore, the safest option here is that the max file size is unlimited and that is what the max file size bootstrap check enforces. To pass the max file check, you must configure your system to allow the Elasticsearch process the ability to write files of unlimited size. This can be done via /etc/security/limits.conf using the fsize setting to unlimited (note that you might have to increase the limits for the root user too).

Maximum size virtual memory check

Elasticsearch and Lucene use mmap to great effect to map portions of an index into the Elasticsearch address space. This keeps certain index data off the JVM heap but in memory for blazing fast access. For this to be effective, the Elasticsearch should have unlimited address space. The maximum size virtual memory check enforces that the Elasticsearch process has unlimited address space and is enforced only on Linux. To pass the maximum size virtual memory check, you must configure your system to allow the Elasticsearch process the ability to have unlimited address space. This can be done via /etc/security/limits.conf using the as setting to unlimited (note that you might have to increase the limits for the root user too).

Maximum map count check

Continuing from the previous point, to use mmap effectively, Elasticsearch also requires the ability to create many memory-mapped areas. The maximum map count check checks that the kernel allows a process to have at least 262,144 memory-mapped areas and is enforced on Linux only. To pass the maximum map count check, you must configure vm.max_map_count via sysctl to be at least 262144.

Alternatively, the maximum map count check is only needed if you are using mmapfs or hybridfs as the store type for your indices. If you do not allow the use of mmap then this bootstrap check will not be enforced.

Client JVM check

There are two different JVMs provided by OpenJDK-derived JVMs: the client JVM and the server JVM. These JVMs use different compilers for producing executable machine code from Java bytecode. The client JVM is tuned for startup time and memory footprint while the server JVM is tuned for maximizing performance. The difference in performance between the two VMs can be substantial. The client JVM check ensures that Elasticsearch is not running inside the client JVM. To pass the client JVM check, you must start Elasticsearch with the server VM. On modern systems and operating systems, the server VM is the default.

Use serial collector check

There are various garbage collectors for the OpenJDK-derived JVMs targeting different workloads. The serial collector in particular is best suited for single logical CPU machines or extremely small heaps, neither of which are suitable for running Elasticsearch. Using the serial collector with Elasticsearch can be devastating for performance. The serial collector check ensures that Elasticsearch is not configured to run with the serial collector. To pass the serial collector check, you must not start Elasticsearch with the serial collector (whether it’s from the defaults for the JVM that you’re using, or you’ve explicitly specified it with -XX:+UseSerialGC). Note that the default JVM configuration that ships with Elasticsearch configures Elasticsearch to use the CMS collector.

System call filter check

Elasticsearch installs system call filters of various flavors depending on the operating system (e.g., seccomp on Linux). These system call filters are installed to prevent the ability to execute system calls related to forking as a defense mechanism against arbitrary code execution attacks on Elasticsearch. The system call filter check ensures that if system call filters are enabled, then they were successfully installed. To pass the system call filter check you must either fix any configuration errors on your system that prevented system call filters from installing (check your logs), or at your own risk disable system call filters by setting bootstrap.system_call_filter to false.

OnError and OnOutOfMemoryError checks

The JVM options OnError and OnOutOfMemoryError enable executing arbitrary commands if the JVM encounters a fatal error (OnError) or an OutOfMemoryError (OnOutOfMemoryError). However, by default, Elasticsearch system call filters (seccomp) are enabled and these filters prevent forking. Thus, using OnError or OnOutOfMemoryError and system call filters are incompatible. The OnError and OnOutOfMemoryError checks prevent Elasticsearch from starting if either of these JVM options are used and system call filters are enabled. This check is always enforced. To pass this check do not enable OnError nor OnOutOfMemoryError; instead, upgrade to Java 8u92 and use the JVM flag ExitOnOutOfMemoryError. While this does not have the full capabilities of OnError nor OnOutOfMemoryError, arbitrary forking will not be supported with seccomp enabled.

Early-access check

The OpenJDK project provides early-access snapshots of upcoming releases. These releases are not suitable for production. The early-access check detects these early-access snapshots. To pass this check, you must start Elasticsearch on a release build of the JVM.

G1GC check

Early versions of the HotSpot JVM that shipped with JDK 8 are known to have issues that can lead to index corruption when the G1GC collector is enabled. The versions impacted are those earlier than the version of HotSpot that shipped with JDK 8u40. The G1GC check detects these early versions of the HotSpot JVM.

All permission check

The all permission check ensures that the security policy used during bootstrap does not grant the java.security.AllPermission to Elasticsearch. Running with the all permission granted is equivalent to disabling the security manager.

Starting Elasticsearch

The method for starting {es} varies depending on how you installed it.

Archive packages (.tar.gz)

If you installed {es} with a .tar.gz package, you can start {es} from the command line.

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

./bin/elasticsearch

By default, Elasticsearch runs in the foreground, prints its logs to the standard output (stdout), and can be stopped by pressing Ctrl-C.

Note
All scripts packaged with Elasticsearch require a version of Bash that supports arrays and assume that Bash is available at /bin/bash. As such, Bash should be available at this path either directly or via a symbolic link.

Running as a daemon

To run Elasticsearch as a daemon, specify -d on the command line, and record the process ID in a file using the -p option:

./bin/elasticsearch -d -p pid

Log messages can be found in the $ES_HOME/logs/ directory.

To shut down Elasticsearch, kill the process ID recorded in the pid file:

pkill -F pid
Note
The startup scripts provided in the RPM and Debian packages take care of starting and stopping the Elasticsearch process for you.

Archive packages (.zip)

If you installed {es} on Windows with a .zip package, you can start {es} from the command line. If you want {es} to start automatically at boot time without any user interaction, install {es} as a service.

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

.\bin\elasticsearch.bat

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl-C.

Debian packages

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the update-rc.d command to configure Elasticsearch to start automatically when the system boots up:

sudo update-rc.d elasticsearch defaults 95 10

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Docker images

If you installed a Docker image, you can start {es} from the command line. There are different methods depending on whether you’re using development mode or production mode. See Running {es} from the command line.

MSI packages

If you installed {es} on Windows using the .msi package, you can start {es} from the command line. If you want it to start automatically at boot time without any user interaction, install {es} as a Windows service.

Running Elasticsearch from the command line

Once installed, Elasticsearch can be started from the command line, if not installed as a service and configured to start when installation completes, as follows:

.\bin\elasticsearch.exe

The command line terminal will display output similar to the following:

elasticsearch exe

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT in addition to the <cluster name>.log file within LOGSDIRECTORY, and can be stopped by pressing Ctrl-C.

RPM packages

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the chkconfig command to configure Elasticsearch to start automatically when the system boots up:

sudo chkconfig --add elasticsearch

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Stopping Elasticsearch

An orderly shutdown of Elasticsearch ensures that Elasticsearch has a chance to cleanup and close outstanding resources. For example, a node that is shutdown in an orderly fashion will remove itself from the cluster, sync translogs to disk, and perform other related cleanup activities. You can help ensure an orderly shutdown by properly stopping Elasticsearch.

If you’re running Elasticsearch as a service, you can stop Elasticsearch via the service management functionality provided by your installation.

If you’re running Elasticsearch directly, you can stop Elasticsearch by sending control-C if you’re running Elasticsearch in the console, or by sending SIGTERM to the Elasticsearch process on a POSIX system. You can obtain the PID to send the signal to via various tools (e.g., ps or jps):

$ jps | grep Elasticsearch
14542 Elasticsearch

From the Elasticsearch startup logs:

[2016-07-07 12:26:18,908][INFO ][node                     ] [I8hydUG] version[5.0.0-alpha4], pid[15399], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Mac OS X/10.11.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_92/25.92-b14]

Or by specifying a location to write a PID file to on startup (-p <path>):

$ ./bin/elasticsearch -p /tmp/elasticsearch-pid -d
$ cat /tmp/elasticsearch-pid && echo
15516
$ kill -SIGTERM 15516

Stopping on Fatal Errors

During the life of the Elasticsearch virtual machine, certain fatal errors could arise that put the virtual machine in a questionable state. Such fatal errors include out of memory errors, internal errors in virtual machine, and serious I/O errors.

When Elasticsearch detects that the virtual machine has encountered such a fatal error Elasticsearch will attempt to log the error and then will halt the virtual machine. When Elasticsearch initiates such a shutdown, it does not go through an orderly shutdown as described above. The Elasticsearch process will also return with a special status code indicating the nature of the error.

JVM internal error

128

Out of memory error

127

Stack overflow error

126

Unknown virtual machine error

125

Serious I/O error

124

Unknown fatal error

1

Adding nodes to your cluster

When you start an instance of {es}, you are starting a node. An {es} cluster is a group of nodes that have the same cluster.name attribute. As nodes join or leave a cluster, the cluster automatically reorganizes itself to evenly distribute the data across the available nodes.

If you are running a single instance of {es}, you have a cluster of one node. All primary shards reside on the single node. No replica shards can be allocated, therefore the cluster state remains yellow. The cluster is fully functional but is at risk of data loss in the event of a failure.

A cluster with one node and three primary shards

You add nodes to a cluster to increase its capacity and reliability. By default, a node is both a data node and eligible to be elected as the master node that controls the cluster. You can also configure a new node for a specific purpose, such as handling ingest requests. For more information, see Nodes.

When you add more nodes to a cluster, it automatically allocates replica shards. When all primary and replica shards are active, the cluster state changes to green.

A cluster with three nodes

To add a node to a cluster:

  1. Set up a new {es} instance.

  2. Specify the name of the cluster in its cluster.name attribute. For example, to add a node to the logging-prod cluster, set cluster.name: "logging-prod" in elasticsearch.yml.

  3. Start {es}. The node automatically discovers and joins the specified cluster.

For more information about discovery and shard allocation, see Discovery and Cluster.

Installing X-Pack in Elasticsearch

Installing {xpack}

By default, when you install {es}, {xpack} is installed. See Installing Elasticsearch.

Set up {xpack}

{xpack} is an Elastic Stack extension that provides security, alerting, monitoring, reporting, machine learning, and many other capabilities. By default, when you install {es}, {xpack} is installed.

If you want to try all of the {xpack} features, you can {stack-ov}/license-management.html[start a 30-day trial]. At the end of the trial period, you can purchase a subscription to keep using the full functionality of the {xpack} components. For more information, see https://www.elastic.co/subscriptions.

Configuring {xpack} Java Clients

If you want to use a Java {javaclient}/transport-client.html[transport client] with a cluster where {xpack} is installed, then you must download and configure the {xpack} transport client.

Warning

We plan on deprecating the TransportClient in Elasticsearch 7.0 and removing it completely in 8.0. Instead, you should be using the {java-rest}/java-rest-high.html[Java High Level REST Client], which executes HTTP requests rather than serialized Java requests. The {java-rest}/java-rest-high-level-migration.html[migration guide] describes all the steps needed to migrate.

The Java High Level REST Client currently has support for the more commonly used APIs, but there are a lot more that still need to be added. You can help us prioritise by telling us which missing APIs you need for your application by adding a comment to this issue: Java high-level REST client completeness.

Any missing APIs can always be implemented today by using the {java-rest}/java-rest-low.html[low level Java REST Client] with JSON request and response bodies.

  1. Add the {xpack} transport JAR file to your CLASSPATH. You can download the {xpack} distribution and extract the JAR file manually or you can get it from the Elasticsearch Maven repository. As with any dependency, you will also need its transitive dependencies. Refer to the X-Pack POM file for your version when downloading for offline usage.

  2. If you are using Maven, you need to add the {xpack} JAR file as a dependency in your project’s pom.xml file:

    <project ...>
         <repositories>
         <!-- add the elasticsearch repo -->
          <repository>
             <id>elasticsearch-releases</id>
             <url>https://artifacts.elastic.co/maven</url>
             <releases>
              <enabled>true</enabled>
             </releases>
             <snapshots>
                <enabled>false</enabled>
             </snapshots>
          </repository>
          ...
       </repositories>
       ...
    
       <dependencies>
          <!-- add the x-pack jar as a dependency -->
          <dependency>
             <groupId>org.elasticsearch.client</groupId>
             <artifactId>x-pack-transport</artifactId>
                  <version>{version}</version>
          </dependency>
          ...
       </dependencies>
       ...
    
    </project>
  3. If you are using Gradle, you need to add the {xpack} JAR file as a dependency in your build.gradle file:

    repositories {
      /* ... Any other repositories ... */
    
      // Add the Elasticsearch Maven Repository
      maven {
        name "elastic"
        url "https://artifacts.elastic.co/maven"
      }
    }
    
    dependencies {
      compile "org.elasticsearch.client:x-pack-transport:{version}"
    
      /* ... */
    }
  4. If you are using a repository manager such as Nexus OSS within your company, you need to add the repository as per the following screenshot:

    Adding the Elastic repo in Nexus

    Then in your project’s pom.xml if using maven, add the following repositories and dependencies definitions:

    <dependencies>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>x-pack-transport</artifactId>
            <version>{version}</version>
        </dependency>
    </dependencies>
    
    <repositories>
        <repository>
            <id>local-nexus</id>
            <name>Elastic Local Nexus</name>
            <url>http://0.0.0.0:8081/repository/elasticsearch/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
      </repositories>
  5. If you are using {stack} {security-features}, there are more configuration steps. See {ref}/java-clients.html[Java Client and security].

{xpack} Settings in {es}

{xpack} Settings

Unresolved directive in settings/configuring-xes.asciidoc - include::{asciidoc-dir}/../../shared/settings.asciidoc[]

{ccr-cap} settings

These {ccr} settings can be dynamically updated on a live cluster with the cluster update settings API.

Remote recovery settings

The following setting can be used to rate-limit the data transmitted during remote recoveries:

ccr.indices.recovery.max_bytes_per_sec (Dynamic)

Limits the total inbound and outbound remote recovery traffic on each node. Since this limit applies on each node, but there may be many nodes performing remote recoveries concurrently, the total amount of remote recovery bytes may be much higher than this limit. If you set this limit too high then there is a risk that ongoing remote recoveries will consume an excess of bandwidth (or other resources) which could destabilize the cluster. This setting is used by both the leader and follower clusters. For example if it is set to 20mb on a leader, the leader will only send 20mb/s to the follower even if the follower is requesting and can accept 60mb/s. Defaults to 40mb.

Advanced remote recovery settings

The following expert settings can be set to manage the resources consumed by remote recoveries:

ccr.indices.recovery.max_concurrent_file_chunks (Dynamic)

Controls the number of file chunk requests that can be sent in parallel per recovery. As multiple remote recoveries might already running in parallel, increasing this expert-level setting might only help in situations where remote recovery of a single shard is not reaching the total inbound and outbound remote recovery traffic as configured by ccr.indices.recovery.max_bytes_per_sec. Defaults to 5. The maximum allowed value is 10.

ccr.indices.recovery.chunk_size(Dynamic)

Controls the chunk size requested by the follower during file transfer. Defaults to 1mb.

ccr.indices.recovery.recovery_activity_timeout(Dynamic)

Controls the timeout for recovery activity. This timeout primarily applies on the leader cluster. The leader cluster must open resources in-memory to supply data to the follower during the recovery process. If the leader does not receive recovery requests from the follower for this period of time, it will close the resources. Defaults to 60 seconds.

ccr.indices.recovery.internal_action_timeout (Dynamic)

Controls the timeout for individual network requests during the remote recovery process. An individual action timing out can fail the recovery. Defaults to 60 seconds.

{xpack} License Settings

License Settings

You can configure this licensing setting in the elasticsearch.yml file. For more information, see {stack-ov}/license-management.html[License management].

xpack.license.self_generated.type

Set to basic (default) to enable basic {xpack} features.

If set to trial, the self-generated license gives access only to all the features of a x-pack for 30 days. You can later downgrade the cluster to a basic license if needed.

Machine learning settings in Elasticsearch

Machine learning settings

You do not need to configure any settings to use {ml}. It is enabled by default.

All of these settings can be added to the elasticsearch.yml configuration file. The dynamic settings can also be updated across a cluster with the cluster update settings API.

Tip
Dynamic settings take precedence over settings in the elasticsearch.yml file.

General machine learning settings

node.ml

Set to true (default) to identify the node as a machine learning node.

If set to false in elasticsearch.yml, the node cannot run jobs. If set to true but xpack.ml.enabled is set to false, the node.ml setting is ignored and the node cannot run jobs. If you want to run jobs, there must be at least one machine learning node in your cluster.

Important
On dedicated coordinating nodes or dedicated master nodes, disable the node.ml role.
xpack.ml.enabled

Set to true (default) to enable {ml} on the node.

If set to false in elasticsearch.yml, the {ml} APIs are disabled on the node. Therefore the node cannot open jobs, start {dfeeds}, or receive transport (internal) communication requests related to {ml} APIs. It also affects all {kib} instances that connect to this {es} instance; you do not need to disable {ml} in those kibana.yml files. For more information about disabling {ml} in specific {kib} instances, see {kibana-ref}/ml-settings-kb.html[{kib} Machine Learning Settings].

Important
If you want to use {ml} features in your cluster, you must have xpack.ml.enabled set to true on all master-eligible nodes. This is the default behavior.
xpack.ml.max_machine_memory_percent

The maximum percentage of the machine’s memory that {ml} may use for running analytics processes. (These processes are separate to the {es} JVM.) Defaults to 30 percent. The limit is based on the total memory of the machine, not current free memory. Jobs will not be allocated to a node if doing so would cause the estimated memory use of {ml} jobs to exceed the limit.

xpack.ml.max_model_memory_limit

The maximum model_memory_limit property value that can be set for any job on this node. If you try to create a job with a model_memory_limit property value that is greater than this setting value, an error occurs. Existing jobs are not affected when you update this setting. For more information about the model_memory_limit property, see Analysis Limits.

xpack.ml.max_open_jobs

The maximum number of jobs that can run on a node. Defaults to 20. The maximum number of jobs is also constrained by memory usage, so fewer jobs than specified by this setting will run on a node if the estimated memory use of the jobs would be higher than allowed.

xpack.ml.node_concurrent_job_allocations

The maximum number of jobs that can concurrently be in the opening state on each node. Typically, jobs spend a small amount of time in this state before they move to open state. Jobs that must restore large models when they are opening spend more time in the opening state. Defaults to 2.

Advanced machine learning settings

These settings are for advanced use cases; the default values are generally sufficient:

xpack.ml.enable_config_migration (Dynamic)

Reserved.

xpack.ml.max_anomaly_records (Dynamic)

The maximum number of records that are output per bucket. The default value is 500.

xpack.ml.max_lazy_ml_nodes (Dynamic)

The number of lazily spun up Machine Learning nodes. Useful in situations where ML nodes are not desired until the first Machine Learning Job is opened. It defaults to 0 and has a maximum acceptable value of 3. If the current number of ML nodes is >= than this setting, then it is assumed that there are no more lazy nodes available as the desired number of nodes have already been provisioned. When a job is opened with this setting set at >0 and there are no nodes that can accept the job, then the job will stay in the OPENING state until a new ML node is added to the cluster and the job is assigned to run on that node.

Important
This setting assumes some external process is capable of adding ML nodes to the cluster. This setting is only useful when used in conjunction with such an external process.
xpack.ml.process_connect_timeout (Dynamic)

The connection timeout for {ml} processes that run separately from the {es} JVM. Defaults to 10s. Some {ml} processing is done by processes that run separately to the {es} JVM. When such processes are started they must connect to the {es} JVM. If such a process does not connect within the time period specified by this setting then the process is assumed to have failed. Defaults to 10s. The minimum value for this setting is 5s.

Monitoring settings in Elasticsearch

Monitoring settings

By default, monitoring is enabled but data collection is disabled. To enable data collection, use the xpack.monitoring.collection.enabled setting.

You can configure these monitoring settings in the elasticsearch.yml file. Some of them can also be set across the cluster by using the cluster update settings API.

Tip
Cluster settings take precedence over settings in the elasticsearch.yml file.

To adjust how monitoring data is displayed in the monitoring UI, configure {kibana-ref}/monitoring-settings-kb.html[xpack.monitoring settings] in kibana.yml. To control how monitoring data is collected from Logstash, configure {logstash-ref}/configuring-logstash.html#monitoring-settings[xpack.monitoring settings] in logstash.yml.

General Monitoring Settings

xpack.monitoring.enabled

Set to true (default) to enable {es} {monitoring} for {es} on the node.

Note
To enable data collection, you must also set xpack.monitoring.collection.enabled to true. Its default value is false.

Monitoring Collection Settings

The xpack.monitoring.collection settings control how data is collected from your Elasticsearch nodes.

xpack.monitoring.collection.enabled (Dynamic)

added[6.3.0] Set to true to enable the collection of monitoring data. When this setting is false (default), {es} monitoring data is not collected and all monitoring data from other sources such as {kib}, Beats, and Logstash is ignored.

xpack.monitoring.collection.interval (Dynamic)

Setting to -1 to disable data collection has been deprecated. deprecated[6.3.0, Use xpack.monitoring.collection.enabled set to false instead.]

Controls how often data samples are collected. Defaults to 10s. If you modify the collection interval, set the xpack.monitoring.min_interval_seconds option in kibana.yml to the same value.

xpack.monitoring.elasticsearch.collection.enabled (Dynamic)

Controls whether statistics about your {es} cluster should be collected. Defaults to true. This is different from xpack.monitoring.collection.enabled, which allows you to enable or disable all monitoring collection. However, this setting simply disables the collection of Elasticsearch data while still allowing other data (e.g., Kibana, Logstash, Beats, or APM Server monitoring data) to pass through this cluster.

xpack.monitoring.collection.cluster.stats.timeout

Sets the timeout for collecting the cluster statistics. Defaults to 10s.

xpack.monitoring.collection.node.stats.timeout

Sets the timeout for collecting the node statistics. Defaults to 10s.

xpack.monitoring.collection.indices (Dynamic)

Controls which indices Monitoring collects data from. Defaults to all indices. Specify the index names as a comma-separated list, for example test1,test2,test3. Names can include wildcards, for example test*. You can explicitly exclude indices by prepending -. For example test*,-test3 will monitor all indexes that start with test except for test3. System indices like .security* or .kibana* always start with a ., and generally should be monitored. Consider adding . to the list of indices ensure monitoring of system indices. For example .,test*,-test3

xpack.monitoring.collection.index.stats.timeout

Sets the timeout for collecting index statistics. Defaults to 10s.

xpack.monitoring.collection.index.recovery.active_only

Controls whether or not all recoveries are collected. Set to true to collect only active recoveries. Defaults to false.

xpack.monitoring.collection.index.recovery.timeout

Sets the timeout for collecting the recovery information. Defaults to 10s.

xpack.monitoring.history.duration

Sets the retention duration beyond which the indices created by a Monitoring exporter are automatically deleted. Defaults to 7d (7 days).

This setting has a minimum value of 1d (1 day) to ensure that something is being monitored, and it cannot be disabled.

Important
This setting currently only impacts local-type exporters. Indices created using the http exporter will not be deleted automatically.

If both {monitoring} and {watcher} are enabled, you can use this setting to affect the {watcher} cleaner service too. For more information, see the xpack.watcher.history.cleaner_service.enabled setting in the {watcher} settings in Elasticsearch.

xpack.monitoring.exporters

Configures where the agent stores monitoring data. By default, the agent uses a local exporter that indexes monitoring data on the cluster where it is installed. Use an HTTP exporter to send data to a separate monitoring cluster. For more information, see local exporter settings, HTTP exporter settings, and How monitoring works.

Local Exporter Settings

The local exporter is the default exporter used by Monitoring. As the name is meant to imply, it exports data to the local cluster, which means that there is not much needed to be configured.

If you do not supply any exporters, then Monitoring will automatically create one for you. If any exporter is provided, then no default is added.

xpack.monitoring.exporters.my_local:
  type: local
type

The value for a Local exporter must always be local and it is required.

use_ingest

Whether to supply a placeholder pipeline to the cluster and a pipeline processor with every bulk request. The default value is true. If disabled, then it means that it will not use pipelines, which means that a future release cannot automatically upgrade bulk requests to future-proof them.

cluster_alerts.management.enabled

Whether to create cluster alerts for this cluster. The default value is true. To use this feature, {watcher} must be enabled. If you have a basic license, cluster alerts are not displayed.

HTTP Exporter Settings

The following lists settings that can be supplied with the http exporter. All settings are shown as what follows the name you select for your exporter:

xpack.monitoring.exporters.my_remote:
  type: http
  host: ["host:port", ...]
type

The value for an HTTP exporter must always be http and it is required.

host

Host supports multiple formats, both as an array or as a single value. Supported formats include hostname, hostname:port, http://hostname http://hostname:port, https://hostname, and https://hostname:port. Hosts cannot be assumed. The default scheme is always http and the default port is always 9200 if not supplied as part of the host string.

xpack.monitoring.exporters:
  example1:
    type: http
    host: "10.1.2.3"
  example2:
    type: http
    host: ["http://10.1.2.4"]
  example3:
    type: http
    host: ["10.1.2.5", "10.1.2.6"]
  example4:
    type: http
    host: ["https://10.1.2.3:9200"]
auth.username

The username is required if a auth.password is supplied.

auth.password

The password for the auth.username.

connection.timeout

The amount of time that the HTTP connection is supposed to wait for a socket to open for the request. The default value is 6s.

connection.read_timeout

The amount of time that the HTTP connection is supposed to wait for a socket to send back a response. The default value is 10 * connection.timeout (60s if neither are set).

ssl

Each HTTP exporter can define its own TLS / SSL settings or inherit them. See the TLS / SSL section below.

proxy.base_path

The base path to prefix any outgoing request, such as /base/path (e.g., bulk requests would then be sent as /base/path/_bulk). There is no default value.

headers

Optional headers that are added to every request, which can assist with routing requests through proxies.

xpack.monitoring.exporters.my_remote:
  headers:
    X-My-Array: [abc, def, xyz]
    X-My-Header: abc123

Array-based headers are sent n times where n is the size of the array. Content-Type and Content-Length cannot be set. Any headers created by the Monitoring agent will override anything defined here.

index.name.time_format

A mechanism for changing the default date suffix for the, by default, daily Monitoring indices. The default value is YYYY.MM.DD, which is why the indices are created daily.

use_ingest

Whether to supply a placeholder pipeline to the monitoring cluster and a pipeline processor with every bulk request. The default value is true. If disabled, then it means that it will not use pipelines, which means that a future release cannot automatically upgrade bulk requests to future-proof them.

cluster_alerts.management.enabled

Whether to create cluster alerts for this cluster. The default value is true. To use this feature, {watcher} must be enabled. If you have a basic license, cluster alerts are not displayed.

cluster_alerts.management.blacklist

Prevents the creation of specific cluster alerts. It also removes any applicable watches that already exist in the current cluster.

You can add any of the following watch identifiers to the blacklist:

  • elasticsearch_cluster_status

  • elasticsearch_version_mismatch

  • elasticsearch_nodes

  • kibana_version_mismatch

  • logstash_version_mismatch

  • xpack_license_expiration

For example: ["elasticsearch_version_mismatch","xpack_license_expiration"].

{monitoring} TLS/SSL Settings

You can configure the following TLS/SSL settings. If the settings are not configured, the {ref}/security-settings.html#ssl-tls-settings[Default TLS/SSL Settings] are used.

{ssl-prefix}.ssl.supported_protocols

Supported protocols with versions. Valid protocols: SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2. Defaults to TLSv1.2, TLSv1.1, TLSv1. Defaults to the value of xpack.ssl.supported_protocols.

{ssl-prefix}.ssl.verification_mode

Controls the verification of certificates. Valid values are none, certificate, and full. See xpack.ssl.verification_mode for a description of these values. Defaults to the value of xpack.ssl.verification_mode.

{ssl-prefix}.ssl.cipher_suites

Supported cipher suites can be found in Oracle’s Java Cryptography Architecture documentation. Defaults to the value of xpack.ssl.cipher_suites.

{monitoring} TLS/SSL Key and Trusted Certificate Settings

The following settings are used to specify a private key, certificate, and the trusted certificates that should be used when communicating over an SSL/TLS connection. A private key and certificate are optional and would be used if the server requires client authentication for PKI authentication. If none of the settings below are specified, the {ref}/security-settings.html#ssl-tls-settings[Default TLS/SSL Settings] are used.

PEM Encoded Files

When using PEM encoded files, use the following settings:

{ssl-prefix}.ssl.key

Path to a PEM encoded file containing the private key.

{ssl-prefix}.ssl.key_passphrase

The passphrase that is used to decrypt the private key. This value is optional as the key might not be encrypted.

{ssl-prefix}.ssl.secure_key_passphrase (Secure)

The passphrase that is used to decrypt the private key. This value is optional as the key might not be encrypted.

{ssl-prefix}.ssl.certificate

Path to a PEM encoded file containing the certificate (or certificate chain) that will be presented when requested.

{ssl-prefix}.ssl.certificate_authorities

List of paths to the PEM encoded certificate files that should be trusted.

Java Keystore Files

When using Java keystore files (JKS), which contain the private key, certificate and certificates that should be trusted, use the following settings:

{ssl-prefix}.ssl.keystore.path

Path to the keystore that holds the private key and certificate.

{ssl-prefix}.ssl.keystore.password

Password to the keystore.

{ssl-prefix}.ssl.keystore.secure_password (Secure)

Password to the keystore.

{ssl-prefix}.ssl.keystore.key_password

Password for the private key in the keystore. Defaults to the same value as {ssl-prefix}.ssl.keystore.password.

{ssl-prefix}.ssl.keystore.secure_key_password (Secure)

Password for the private key in the keystore.

{ssl-prefix}.ssl.truststore.path

Path to the truststore file.

{ssl-prefix}.ssl.truststore.password

Password to the truststore.

{ssl-prefix}.ssl.truststore.secure_password (Secure)

Password to the truststore.

PKCS#12 Files

{es} can be configured to use PKCS#12 container files (.p12 or .pfx files) that contain the private key, certificate and certificates that should be trusted.

PKCS#12 files are configured in the same way as Java Keystore Files:

{ssl-prefix}.ssl.keystore.path

Path to the PKCS#12 file that holds the private key and certificate.

{ssl-prefix}.ssl.keystore.type

Set this to PKCS12 to indicate that the keystore is a PKCS#12 file.

{ssl-prefix}.ssl.keystore.password

Password to the PKCS#12 file.

{ssl-prefix}.ssl.keystore.secure_password (Secure)

Password to the PKCS#12 file.

{ssl-prefix}.ssl.keystore.key_password

Password for the private key stored in the PKCS#12 file. Defaults to the same value as {ssl-prefix}.ssl.keystore.password.

{ssl-prefix}.ssl.keystore.secure_key_password (Secure)

Password for the private key stored in the PKCS#12 file.

{ssl-prefix}.ssl.truststore.path

Path to the PKCS#12 file that holds the certificates to be trusted.

{ssl-prefix}.ssl.truststore.type

Set this to PKCS12 to indicate that the truststore is a PKCS#12 file.

{ssl-prefix}.ssl.truststore.password

Password to the PKCS#12 file.

{ssl-prefix}.ssl.truststore.secure_password (Secure)

Password to the PKCS#12 file.

PKCS#11 Tokens

{es} can be configured to use a PKCS#11 token that contains the private key, certificate and certificates that should be trusted.

PKCS#11 token require additional configuration on the JVM level and can be enabled via the following settings:

{ssl-prefix}.keystore.type

Set this to PKCS11 to indicate that the PKCS#11 token should be used as a keystore.

{ssl-prefix}.truststore.type

Set this to PKCS11 to indicate that the PKCS#11 token should be used as a truststore.

SQL Access Settings in Elasticsearch

SQL Access Settings

SQL Access is enabled by default. You can configure these SQL Access settings in the elasticsearch.yml file.

General SQL Access Settings

xpack.sql.enabled

Set to false to disable SQL Access on the node.

{watcher} settings in Elasticsearch

{watcher} Settings

You configure xpack.notification settings in elasticsearch.yml to send set up {watcher} and send notifications via email, HipChat, Slack, and PagerDuty. Dynamic settings can also be updated across a cluster with the cluster update settings API.

General Watcher Settings

xpack.watcher.enabled

Set to false to disable {watcher} on the node.

xpack.watcher.encrypt_sensitive_data

Set to true to encrypt sensitive data. If this setting is enabled, you must also specify the xpack.watcher.encryption_key setting. For more information, see [encrypting-data].

xpack.watcher.encryption_key (Secure)

Specifies the path to a file that contains a key for encrypting sensitive data. If xpack.watcher.encrypt_sensitive_data is set to true, this setting is required. For more information, see [encrypting-data].

xpack.watcher.history.cleaner_service.enabled

added:[6.3.0,Default changed to true.]

Set to true (default) to enable the cleaner service. If this setting is true, the xpack.monitoring.enabled setting must also be set to true with a local exporter enabled. The cleaner service removes previous versions of {watcher} indices (for example, .watcher-history*) when it determines that they are old. The duration of {watcher} indices is determined by the xpack.monitoring.history.duration setting, which defaults to 7 days. For more information about that setting, see Monitoring settings in Elasticsearch.

xpack.http.proxy.host

Specifies the address of the proxy server to use to connect to HTTP services.

xpack.http.proxy.port

Specifies the port number to use to connect to the proxy server.

xpack.http.default_connection_timeout

The maximum period to wait until abortion of the request, when a connection is being initiated.

xpack.http.default_read_timeout

The maximum period of inactivity between two data packets, before the request is aborted.

xpack.http.max_response_size

Specifies the maximum size an HTTP response is allowed to have, defaults to 10mb, the maximum configurable value is 50mb.

{watcher} TLS/SSL Settings

You can configure the following TLS/SSL settings. If the settings are not configured, the {ref}/security-settings.html#ssl-tls-settings[Default TLS/SSL Settings] are used.

{ssl-prefix}.ssl.supported_protocols

Supported protocols with versions. Valid protocols: SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2. Defaults to TLSv1.2, TLSv1.1, TLSv1. Defaults to the value of xpack.ssl.supported_protocols.

{ssl-prefix}.ssl.verification_mode

Controls the verification of certificates. Valid values are none, certificate, and full. See xpack.ssl.verification_mode for a description of these values. Defaults to the value of xpack.ssl.verification_mode.

{ssl-prefix}.ssl.cipher_suites

Supported cipher suites can be found in Oracle’s Java Cryptography Architecture documentation. Defaults to the value of xpack.ssl.cipher_suites.

{watcher} TLS/SSL Key and Trusted Certificate Settings

The following settings are used to specify a private key, certificate, and the trusted certificates that should be used when communicating over an SSL/TLS connection. A private key and certificate are optional and would be used if the server requires client authentication for PKI authentication. If none of the settings below are specified, the {ref}/security-settings.html#ssl-tls-settings[Default TLS/SSL Settings] are used.

PEM Encoded Files

When using PEM encoded files, use the following settings:

{ssl-prefix}.ssl.key

Path to a PEM encoded file containing the private key.

{ssl-prefix}.ssl.key_passphrase

The passphrase that is used to decrypt the private key. This value is optional as the key might not be encrypted.

{ssl-prefix}.ssl.secure_key_passphrase (Secure)

The passphrase that is used to decrypt the private key. This value is optional as the key might not be encrypted.

{ssl-prefix}.ssl.certificate

Path to a PEM encoded file containing the certificate (or certificate chain) that will be presented when requested.

{ssl-prefix}.ssl.certificate_authorities

List of paths to the PEM encoded certificate files that should be trusted.

Java Keystore Files

When using Java keystore files (JKS), which contain the private key, certificate and certificates that should be trusted, use the following settings:

{ssl-prefix}.ssl.keystore.path

Path to the keystore that holds the private key and certificate.

{ssl-prefix}.ssl.keystore.password

Password to the keystore.

{ssl-prefix}.ssl.keystore.secure_password (Secure)

Password to the keystore.

{ssl-prefix}.ssl.keystore.key_password

Password for the private key in the keystore. Defaults to the same value as {ssl-prefix}.ssl.keystore.password.

{ssl-prefix}.ssl.keystore.secure_key_password (Secure)

Password for the private key in the keystore.

{ssl-prefix}.ssl.truststore.path

Path to the truststore file.

{ssl-prefix}.ssl.truststore.password

Password to the truststore.

{ssl-prefix}.ssl.truststore.secure_password (Secure)

Password to the truststore.

PKCS#12 Files

{es} can be configured to use PKCS#12 container files (.p12 or .pfx files) that contain the private key, certificate and certificates that should be trusted.

PKCS#12 files are configured in the same way as Java Keystore Files:

{ssl-prefix}.ssl.keystore.path

Path to the PKCS#12 file that holds the private key and certificate.

{ssl-prefix}.ssl.keystore.type

Set this to PKCS12 to indicate that the keystore is a PKCS#12 file.

{ssl-prefix}.ssl.keystore.password

Password to the PKCS#12 file.

{ssl-prefix}.ssl.keystore.secure_password (Secure)

Password to the PKCS#12 file.

{ssl-prefix}.ssl.keystore.key_password

Password for the private key stored in the PKCS#12 file. Defaults to the same value as {ssl-prefix}.ssl.keystore.password.

{ssl-prefix}.ssl.keystore.secure_key_password (Secure)

Password for the private key stored in the PKCS#12 file.

{ssl-prefix}.ssl.truststore.path

Path to the PKCS#12 file that holds the certificates to be trusted.

{ssl-prefix}.ssl.truststore.type

Set this to PKCS12 to indicate that the truststore is a PKCS#12 file.

{ssl-prefix}.ssl.truststore.password

Password to the PKCS#12 file.

{ssl-prefix}.ssl.truststore.secure_password (Secure)

Password to the PKCS#12 file.

PKCS#11 Tokens

{es} can be configured to use a PKCS#11 token that contains the private key, certificate and certificates that should be trusted.

PKCS#11 token require additional configuration on the JVM level and can be enabled via the following settings:

{ssl-prefix}.keystore.type

Set this to PKCS11 to indicate that the PKCS#11 token should be used as a keystore.

{ssl-prefix}.truststore.type

Set this to PKCS11 to indicate that the PKCS#11 token should be used as a truststore.

Email Notification Settings

You can configure the following email notification settings in elasticsearch.yml. For more information about sending notifications via email, see [configuring-email-actions].

xpack.notification.email.account

Specifies account information for sending notifications via email. You can specify the following email account attributes:

profile (Dynamic)

The email profile to use to build the MIME messages that are sent from the account. Valid values: standard, gmail and outlook. Defaults to standard.

email_defaults.* (Dynamic)

An optional set of email attributes to use as defaults for the emails sent from the account. See [email-action-attributes] for the supported attributes.

smtp.auth (Dynamic)

Set to true to attempt to authenticate the user using the AUTH command. Defaults to false.

smtp.host (Dynamic)

The SMTP server to connect to. Required.

smtp.port (Dynamic)

The SMTP server port to connect to. Defaults to 25.

smtp.user (Dynamic)

The user name for SMTP. Required.

smtp.password (Dynamic)

The password for the specified SMTP user. deprecated[6.6.0]

smtp.secure_password (Secure)

The password for the specified SMTP user.

smtp.starttls.enable (Dynamic)

Set to true to enable the use of the STARTTLS command (if supported by the server) to switch the connection to a TLS-protected connection before issuing any login commands. Note that an appropriate trust store must configured so that the client will trust the server’s certificate. Defaults to false.

smtp.starttls.required (Dynamic)

If true, then STARTTLS will be required. If that command fails, the connection will fail. Defaults to false.

smtp.ssl.trust (Dynamic)

A list of SMTP server hosts that are assumed trusted and for which certificate verification is disabled. If set to "*", all hosts are trusted. If set to a whitespace separated list of hosts, those hosts are trusted. Otherwise, trust depends on the certificate the server presents.

smtp.timeout (Dynamic)

The socket read timeout. Default is two minutes.

smtp.connection_timeout (Dynamic)

The socket connection timeout. Default is two minutes.

smtp.write_timeout (Dynamic)

The socket write timeout. Default is two minutes.

smtp.local_address (Dynamic)

A configurable local address when sending emails. Not configured by default.

smtp.local_port (Dynamic)

A configurable local port when sending emails. Not configured by default.

smtp.send_partial (Dynamic)

Send an email, despite one of the receiver addresses being invalid.

smtp.wait_on_quit (Dynamic)

If set to false the QUIT command is sent and the connection closed. If set to true, the QUIT command is sent and a reply is waited for. True by default.

xpack.notification.email.html.sanitization.allow

Specifies the HTML elements that are allowed in email notifications. For more information, see [email-html-sanitization]. You can specify individual HTML elements and the following HTML feature groups:

_tables

All table related elements: <table>, <th>, <tr> and <td>.

_blocks

The following block elements: <p>, <div>, <h1>, <h2>, <h3>, <h4>, <h5>, <h6>, <ul>, <ol>, <li>, and <blockquote>.

_formatting

The following inline formatting elements: <b>, <i>, <s>, <u>, <o>, <sup>, <sub>, <ins>, <del>, <strong>, <strike>, <tt>, <code>, <big>, <small>, <br>, <span>, and <em>.

_links

The <a> element with an href attribute that points to a URL using the following protocols: http, https and mailto.

_styles

The style attribute on all elements. Note that CSS attributes are also sanitized to prevent XSS attacks.

img
img:all

All images (external and embedded).

img:embedded

Only embedded images. Embedded images can only use the cid: URL protocol in their src attribute.

xpack.notification.email.html.sanitization.disallow

Specifies the HTML elements that are NOT allowed in email notifications. You can specify individual HTML elements and HTML feature groups.

xpack.notification.email.html.sanitization.enabled

Set to false to completely disable HTML sanitation. Not recommended. Defaults to true.

HipChat notification settings

You can configure the following HipChat notification settings in elasticsearch.yml. For more information about sending notifications via HipChat, see [configuring-hipchat-actions].

deprecated[6.7.0,Hipchat notification has been deprecated as Hipchat has ceased operation]

xpack.notification.hipchat

Specifies account information for sending notifications via HipChat. You can specify the following HipChat account attributes:

profile

The HipChat account profile to use: integration, user, or v1. Required.

auth_token

The authentication token to use to access the HipChat API. Required. deprecated[6.6.0]

secure_auth_token (Secure)

The authentication token to use to access the HipChat API. Required.

host

The HipChat server hostname. Defaults to api.hipchat.com.

port

The HipChat server port number. Defaults to 443.

room

The room you want to send messages to. Must be specified if the profile is set to integration. Not valid for the user or vi profiles.

user

The HipChat user account to use to send messages. Specified as an email address. Must be specified if the profile is set to user. Not valid for the integration or v1 profiles.

message.format

The format of the message: text or html. Defaults to html.

message.color

The background color of the notification in the room. Defaults to yellow.

message.notify

Indicates whether people in the room should be actively notified. Defaults to false.

Slack Notification Settings

You can configure the following Slack notification settings in elasticsearch.yml. For more information about sending notifications via Slack, see [configuring-slack-actions].

xpack.notification.slack

Specifies account information for sending notifications via Slack. You can specify the following Slack account attributes:

url

The Incoming Webhook URL to use to post messages to Slack. Required. deprecated[6.6.0]

secure_url (Secure)

The Incoming Webhook URL to use to post messages to Slack. Required.

message_defaults.from

The sender name to display in the Slack message. Defaults to the watch ID.

message_defaults.to

The default Slack channels or groups you want to send messages to.

message_defaults.icon

The icon to display in the Slack messages. Overrides the incoming webhook’s configured icon. Accepts a public URL to an image.

message_defaults.text

The default message content.

message_defaults.attachment

Default message attachments. Slack message attachments enable you to create more richly-formatted messages. Specified as an array as defined in the Slack attachments documentation.

Jira Notification Settings

You can configure the following Jira notification settings in elasticsearch.yml. For more information about using notifications to create issues in Jira, see [configuring-jira-actions].

xpack.notification.jira

Specifies account information for using notifications to create issues in Jira. You can specify the following Jira account attributes:

url

The URL of the Jira Software server. Required. deprecated[6.6.0]

secure_url (Secure)

The URL of the Jira Software server. Required.

user

The name of the user to connect to the Jira Software server. Required. deprecated[6.6.0]

secure_user (Secure)

The name of the user to connect to the Jira Software server. Required.

password

The password of the user to connect to the Jira Software server. Required. deprecated[6.6.0]

secure_password (Secure)

The password of the user to connect to the Jira Software server. Required.

issue_defaults

Default fields values for the issue created in Jira. See [jira-action-attributes] for more information. Optional.

PagerDuty Notification Settings

You can configure the following PagerDuty notification settings in elasticsearch.yml. For more information about sending notifications via PagerDuty, see [configuring-pagerduty-actions].

xpack.notification.pagerduty

Specifies account information for sending notifications via PagerDuty. You can specify the following PagerDuty account attributes:

name

A name for the PagerDuty account associated with the API key you are using to access PagerDuty. Required.

service_api_key

The PagerDuty API key to use to access PagerDuty. Required. deprecated[6.6.0]

secure_service_api_key (Secure)

The PagerDuty API key to use to access PagerDuty. Required.

event_defaults

Default values for PagerDuty event attributes. Optional.

description

A string that contains the default description for PagerDuty events. If no default is configured, each PagerDuty action must specify a description.

incident_key

A string that contains the default incident key to use when sending PagerDuty events.

client

A string that specifies the default monitoring client.

client_url

The URL of the default monitoring client.

event_type

The default event type. Valid values: trigger,resolve, acknowledge.

attach_payload

Whether or not to provide the watch payload as context for the event by default. Valid values: true, false.

Bootstrap Checks for {xpack}

In addition to the {es} bootstrap checks, there are checks that are specific to {xpack} features.

Encrypt sensitive data check

If you use {watcher} and have chosen to encrypt sensitive data (by setting xpack.watcher.encrypt_sensitive_data to true), you must also place a key in the secure settings store.

To pass this bootstrap check, you must set the xpack.watcher.encryption_key on each node in the cluster. For more information, see [encrypting-data].

PKI realm check

If you use {es} {security-features} and a Public Key Infrastructure (PKI) realm, you must configure Transport Layer Security (TLS) on your cluster and enable client authentication on the network layers (either transport or http). For more information, see [pki-realm] and [ssl-tls].

To pass this bootstrap check, if a PKI realm is enabled, you must configure TLS and enable client authentication on at least one network communication layer.

Role mappings check

If you authenticate users with realms other than native or file realms, you must create role mappings. These role mappings define which roles are assigned to each user.

If you use files to manage the role mappings, you must configure a YAML file and copy it to each node in the cluster. By default, role mappings are stored in ES_PATH_CONF/role_mapping.yml. Alternatively, you can specify a different role mapping file for each type of realm and specify its location in the elasticsearch.yml file. For more information, see [mapping-roles-file].

To pass this bootstrap check, the role mapping files must exist and must be valid. The Distinguished Names (DNs) that are listed in the role mappings files must also be valid.

SSL/TLS check

In 6.0 and later releases, if you have a gold, platinum, or enterprise license and {es} {security-features} are enabled, you must configure SSL/TLS for internode-communication.

Note
Single-node clusters that use a loopback interface do not have this requirement. For more information, see [encrypting-communications].

To pass this bootstrap check, you must [ssl-tls].

Token SSL check

If you use {es} {security-features} and the built-in token service is enabled, you must configure your cluster to use SSL/TLS for the HTTP interface. HTTPS is required in order to use the token service.

In particular, if xpack.security.authc.token.enabled and http.enabled are set to true in the elasticsearch.yml file, you must also set xpack.security.http.ssl.enabled to true. For more information about these settings, see [security-settings] and HTTP.

To pass this bootstrap check, you must enable HTTPS or disable the built-in token service.

Upgrade Elasticsearch

Rolling upgrades

A rolling upgrade allows an Elasticsearch cluster to be upgraded one node at a time so upgrading does not interrupt service. Running multiple versions of Elasticsearch in the same cluster beyond the duration of an upgrade is not supported, as shards cannot be replicated from upgraded nodes to nodes running the older version.

Rolling upgrades can be performed between minor versions. Elasticsearch 6.x supports rolling upgrades from Elasticsearch 5.6. Upgrading from earlier 5.x versions requires a full cluster restart. You must reindex to upgrade from versions prior to 5.x.

Warning
If the {es} {security-features} are enabled on your 5.x cluster, before you can do a rolling upgrade you must encrypt the internode-communication with SSL/TLS, which requires a full cluster restart. For more information about this requirement and the associated bootstrap check, see SSL/TLS check.
Warning
The format used for the internal indices used by Kibana and {xpack} has changed in 6.x. When upgrading from 5.6 to 6.x, these internal indices have to be {stack-ref}/upgrading-elastic-stack.html#upgrade-internal-indices[upgraded] before the rolling upgrade procedure can start. Otherwise the upgraded node will refuse to join the cluster.

To perform a rolling upgrade:

  1. Disable shard allocation.

    When you shut down a node, the allocation process waits for index.unassigned.node_left.delayed_timeout (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by disabling allocation of replicas before shutting down the node:

    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": "primaries"
      }
    }
  2. Stop non-essential indexing and perform a synced flush. (Optional)

    While you can continue indexing during the upgrade, shard recovery is much faster if you temporarily stop non-essential indexing and perform a synced-flush.

    POST _flush/synced

    When you perform a synced flush, check the response to make sure there are no failures. Synced flush operations that fail due to pending indexing operations are listed in the response body, although the request itself still returns a 200 OK status. If there are failures, reissue the request.

  3. Stop any machine learning jobs that are running.

    If your {ml} indices were created earlier than the previous major version, they must be reindexed. In those circumstances, there must be no machine learning jobs running during the upgrade.

    In all other circumstances, there is no requirement to close your {ml} jobs. There are, however, advantages to doing so. If you choose to leave your jobs running during the upgrade, they are affected when you stop the {ml} nodes. The jobs move to another {ml} node and restore the model states. This scenario has the least disruption to the active {ml} jobs but incurs the highest load on the cluster.

    To close all {ml} jobs before you upgrade, see {ml-docs}/stopping-ml.html[Stopping {ml}]. This method persists the model state at the moment of closure, which means that when you open your jobs after the upgrade, they use the exact same model. This scenario takes the most time, however, especially if you have many jobs or jobs with large model states.

    To temporarily halt the tasks associated with your {ml} jobs and {dfeeds} and prevent new jobs from opening, use the set upgrade mode API:

    POST _ml/set_upgrade_mode?enabled=true

    This method does not persist the absolute latest model state, rather it uses the last model state that was automatically saved. By halting the tasks, you avoid incurring the cost of managing active jobs during the upgrade and it’s quicker than stopping {dfeeds} and closing jobs.

  4. Shut down a single node.

    • If you are running Elasticsearch with systemd:

      sudo systemctl stop elasticsearch.service
    • If you are running Elasticsearch with SysV init:

      sudo -i service elasticsearch stop
    • If you are running Elasticsearch as a daemon:

      kill $(cat pid)
  5. Upgrade the node you shut down.

    Important
    If you are upgrading from a version prior to 6.3 and use {xpack} then you must remove the {xpack} plugin before upgrading with bin/elasticsearch-plugin remove x-pack. As of 6.3, {xpack} is included in the default distribution so make sure to upgrade to that one. If you upgrade without removing the {xpack} plugin first the node will fail to start. If you did not remove the {xpack} plugin and the node fails to start then you must downgrade to your previous version, remove {xpack}, and then upgrade again. In general downgrading is not supported but in this particular case it is.

    To upgrade using a Debian or RPM package:

    • Use rpm or dpkg to install the new package. All files are installed in the appropriate location for the operating system and Elasticsearch config files are not overwritten.

    To upgrade using a zip or compressed tarball:

    1. Extract the zip or tarball to a new directory. This is critical if you are not using external config and data directories.

    2. Set the ES_PATH_CONF environment variable to specify the location of your external config directory and jvm.options file. If you are not using an external config directory, copy your old configuration over to the new installation.

    3. Set path.data in config/elasticsearch.yml to point to your external data directory. If you are not using an external data directory, copy your old data directory over to the new installation.

      Important
      If you use {monitoring}, re-use the data directory when you upgrade {es}. Monitoring identifies unique {es} nodes by using the persistent UUID, which is stored in the data directory.
    4. Set path.logs in config/elasticsearch.yml to point to the location where you want to store your logs. If you do not specify this setting, logs are stored in the directory you extracted the archive to.

    When you extract the zip or tarball packages, the elasticsearch-n.n.n directory contains the Elasticsearch config, data, and logs directories.

    We recommend moving these directories out of the Elasticsearch directory so that there is no chance of deleting them when you upgrade Elasticsearch. To specify the new locations, use the ES_PATH_CONF environment variable and the path.data and path.logs settings. For more information, see Important Elasticsearch configuration.

    The Debian and RPM packages place these directories in the appropriate place for each operating system. In production, we recommend installing using the deb or rpm package.

  6. Upgrade any plugins.

    Use the elasticsearch-plugin script to install the upgraded version of each installed Elasticsearch plugin. All plugins must be upgraded when you upgrade a node.

  7. Start the upgraded node.

    Start the newly-upgraded node and confirm that it joins the cluster by checking the log file or by submitting a _cat/nodes request:

    GET _cat/nodes
  8. Reenable shard allocation.

    Once the node has joined the cluster, remove the cluster.routing.allocation.enable setting to enable shard allocation and start using the node:

    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": null
      }
    }
  9. Wait for the node to recover.

    Before upgrading the next node, wait for the cluster to finish shard allocation. You can check progress by submitting a _cat/health request:

    GET _cat/health?v

    Wait for the status column to switch to green. Once the node is green, all primary and replica shards have been allocated.

    Important

    During a rolling upgrade, primary shards assigned to a node running the new version cannot have their replicas assigned to a node with the old version. The new version might have a different data format that is not understood by the old version.

    If it is not possible to assign the replica shards to another node (there is only one upgraded node in the cluster), the replica shards remain unassigned and status stays yellow.

    In this case, you can proceed once there are no initializing or relocating shards (check the init and relo columns).

    As soon as another node is upgraded, the replicas can be assigned and the status will change to green.

    Shards that were not sync-flushed might take longer to recover. You can monitor the recovery status of individual shards by submitting a _cat/recovery request:

    GET _cat/recovery

    If you stopped indexing, it is safe to resume indexing as soon as recovery completes.

  10. Repeat

    When the node has recovered and the cluster is stable, repeat these steps for each node that needs to be updated. You can monitor the health of the cluster with a _cat/health request:

    GET /_cat/health?v

    And check which nodes have been upgraded with a _cat/nodes request:

    GET /_cat/nodes?h=ip,name,version&v
  11. Restart machine learning jobs.

    If you closed all {ml} jobs before the upgrade, you must open them. Use {kib} or the open jobs API.

    Alternatively, if you temporarily halted the tasks associated with your {ml} jobs, use the set upgrade mode API to return them to active states:

    POST _ml/set_upgrade_mode?enabled=false
Important

During a rolling upgrade, the cluster continues to operate normally. However, any new functionality is disabled or operates in a backward compatible mode until all nodes in the cluster are upgraded. New functionality becomes operational once the upgrade is complete and all nodes are running the new version. Once that has happened, there’s no way to return to operating in a backward compatible mode. Nodes running the previous major version will not be allowed to join the fully-updated cluster.

In the unlikely case of a network malfunction during the upgrade process that isolates all remaining old nodes from the cluster, you must take the old nodes offline and upgrade them to enable them to join the cluster.

Full cluster restart upgrade

A full cluster restart upgrade requires that you shut all nodes in the cluster down, upgrade them, and restart the cluster. A full cluster restart was required when upgrading to major versions prior to 6.x. Elasticsearch 6.x supports rolling upgrades from Elasticsearch 5.6. Upgrading to 6.x from earlier versions requires a full cluster restart. See the Upgrade paths table to verify the type of upgrade you need to perform.

To perform a full cluster restart upgrade:

  1. Disable shard allocation.

    When you shut down a node, the allocation process waits for index.unassigned.node_left.delayed_timeout (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by disabling allocation of replicas before shutting down the node:

    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": "primaries"
      }
    }
  2. Stop indexing and perform a synced flush.

    Performing a synced-flush speeds up shard recovery.

    POST _flush/synced

    When you perform a synced flush, check the response to make sure there are no failures. Synced flush operations that fail due to pending indexing operations are listed in the response body, although the request itself still returns a 200 OK status. If there are failures, reissue the request.

  3. Stop any machine learning jobs that are running.

    If your {ml} indices were created earlier than the previous major version, they must be reindexed. In those circumstances, there must be no machine learning jobs running during the upgrade.

    In all other circumstances, there is no requirement to close your {ml} jobs. There are, however, advantages to doing so. If you choose to leave your jobs running during the upgrade, they are affected when you stop the {ml} nodes. The jobs move to another {ml} node and restore the model states. This scenario has the least disruption to the active {ml} jobs but incurs the highest load on the cluster.

    To close all {ml} jobs before you upgrade, see {ml-docs}/stopping-ml.html[Stopping {ml}]. This method persists the model state at the moment of closure, which means that when you open your jobs after the upgrade, they use the exact same model. This scenario takes the most time, however, especially if you have many jobs or jobs with large model states.

    To temporarily halt the tasks associated with your {ml} jobs and {dfeeds} and prevent new jobs from opening, use the set upgrade mode API:

    POST _ml/set_upgrade_mode?enabled=true

    This method does not persist the absolute latest model state, rather it uses the last model state that was automatically saved. By halting the tasks, you avoid incurring the cost of managing active jobs during the upgrade and it’s quicker than stopping {dfeeds} and closing jobs.

  4. Shutdown all nodes.

    • If you are running Elasticsearch with systemd:

      sudo systemctl stop elasticsearch.service
    • If you are running Elasticsearch with SysV init:

      sudo -i service elasticsearch stop
    • If you are running Elasticsearch as a daemon:

      kill $(cat pid)
  5. Upgrade all nodes.

    Important
    If you are upgrading from a version prior to 6.3 and use {xpack} then you must remove the {xpack} plugin before upgrading with bin/elasticsearch-plugin remove x-pack. As of 6.3, {xpack} is included in the default distribution so make sure to upgrade to that one. If you upgrade without removing the {xpack} plugin first the node will fail to start. If you did not remove the {xpack} plugin and the node fails to start then you must downgrade to your previous version, remove {xpack}, and then upgrade again. In general downgrading is not supported but in this particular case it is.

    To upgrade using a Debian or RPM package:

    • Use rpm or dpkg to install the new package. All files are installed in the appropriate location for the operating system and Elasticsearch config files are not overwritten.

    To upgrade using a zip or compressed tarball:

    1. Extract the zip or tarball to a new directory. This is critical if you are not using external config and data directories.

    2. Set the ES_PATH_CONF environment variable to specify the location of your external config directory and jvm.options file. If you are not using an external config directory, copy your old configuration over to the new installation.

    3. Set path.data in config/elasticsearch.yml to point to your external data directory. If you are not using an external data directory, copy your old data directory over to the new installation.

      Important
      If you use {monitoring}, re-use the data directory when you upgrade {es}. Monitoring identifies unique {es} nodes by using the persistent UUID, which is stored in the data directory.
    4. Set path.logs in config/elasticsearch.yml to point to the location where you want to store your logs. If you do not specify this setting, logs are stored in the directory you extracted the archive to.

    Tip

    When you extract the zip or tarball packages, the elasticsearch-n.n.n directory contains the Elasticsearch config, data, and logs directories.

    We recommend moving these directories out of the Elasticsearch directory so that there is no chance of deleting them when you upgrade Elasticsearch. To specify the new locations, use the ES_PATH_CONF environment variable and the path.data and path.logs settings. For more information, see Important Elasticsearch configuration.

    The Debian and RPM packages place these directories in the appropriate place for each operating system. In production, we recommend installing using the deb or rpm package.

  6. Upgrade any plugins.

    Use the elasticsearch-plugin script to install the upgraded version of each installed Elasticsearch plugin. All plugins must be upgraded when you upgrade a node.

  7. Start each upgraded node.

    If you have dedicated master nodes, start them first and wait for them to form a cluster and elect a master before proceeding with your data nodes. You can check progress by looking at the logs.

    As soon as the minimum number of master-eligible nodes have discovered each other, they form a cluster and elect a master. At that point, you can use _cat/health and _cat/nodes to monitor nodes joining the cluster:

    GET _cat/health
    
    GET _cat/nodes

    The status column returned by _cat/health shows the health of each node in the cluster: red, yellow, or green.

  8. Wait for all nodes to join the cluster and report a status of yellow.

    When a node joins the cluster, it begins to recover any primary shards that are stored locally. The _cat/health API initially reports a status of red, indicating that not all primary shards have been allocated.

    Once a node recovers its local shards, the cluster status switches to yellow, indicating that all primary shards have been recovered, but not all replica shards are allocated. This is to be expected because you have not yet reenabled allocation. Delaying the allocation of replicas until all nodes are yellow allows the master to allocate replicas to nodes that already have local shard copies.

  9. Reenable allocation.

    When all nodes have joined the cluster and recovered their primary shards, reenable allocation by restoring cluster.routing.allocation.enable to its default:

    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": null
      }
    }

    Once allocation is reenabled, the cluster starts allocating replica shards to the data nodes. At this point it is safe to resume indexing and searching, but your cluster will recover more quickly if you can wait until all primary and replica shards have been successfully allocated and the status of all nodes is green.

    You can monitor progress with the _cat/health and _cat/recovery APIs:

    GET _cat/health
    
    GET _cat/recovery
  10. If you use {security} and are upgrading directly to {version} from 5.5 or earlier, you must upgrade the .security index after you restart {es}.

    Important
    Native realm users cannot authenticate until the index is upgraded. For instructions, see {stack-ref}/upgrading-elastic-stack.html#upgrade-internal-indices[Upgrading internal indices]. You also need to upgrade the .security index if you restore a pre-5.6 snapshot to a fresh 6.0 install.
  11. Restart machine learning jobs.

    If you closed all {ml} jobs before the upgrade, you must open them. Use {kib} or the open jobs API.

    Alternatively, if you temporarily halted the tasks associated with your {ml} jobs, use the set upgrade mode API to return them to active states:

    POST _ml/set_upgrade_mode?enabled=false

Reindex before upgrading

Elasticsearch can read indices created in the previous major version. Older indices must be reindexed or deleted. Elasticsearch 6.x can use indices created in Elasticsearch 5.x, but not those created in Elasticsearch 2.x or before. Elasticsearch 5.x can use indices created in Elasticsearch 2.x, but not those created in 1.x or before.

Elasticsearch nodes will fail to start if incompatible indices are present.

To upgrade an Elasticsearch 5.x cluster that contains indices created in 2.x, you must reindex or delete them before upgrading to 6.x. For more information, see Reindex in place.

To upgrade an Elasticsearch cluster running 2.x, you have two options:

  • Perform a full cluster restart upgrade to 5.6, reindex the 2.x indices, then perform a rolling upgrade to 6.x. If your Elasticsearch 2.x cluster contains indices that were created before 2.x, you must either delete or reindex them before upgrading to 5.6. For more information about upgrading from 2.x to 5.6, see Upgrading Elasticsearch in the Elasticsearch 5.6 Reference.

  • Create a new 6.x cluster and reindex from remote to import indices directly from the 2.x cluster.

To upgrade an Elasticsearch 1.x cluster, you have two options:

Upgrading time-based indices

If you use time-based indices, you likely won’t need to carry pre-5.x indices forward to 6.x. Data in time-based indices generally becomes less useful as time passes and are deleted as they age past your retention period.

Unless you have an unusually long retention period, you can just wait to upgrade to 6.x until all of your pre-5.x indices have been deleted.

Reindex in place

To manually reindex your old indices with the reindex API:

  1. Create a new index and copy the mappings and settings from the old index.

  2. Set the refresh_interval to -1 and the number_of_replicas to 0 for efficient reindexing.

  3. Reindex all documents from the old index into the new index using the reindex API.

  4. Reset the refresh_interval and number_of_replicas to the values used in the old index.

  5. Wait for the index status to change to green.

  6. In a single update aliases request:

    1. Delete the old index.

    2. Add an alias with the old index name to the new index.

    3. Add any aliases that existed on the old index to the new index.

Migration assistance and upgrade tools

{xpack} 5.6 provides migration assistance and upgrade tools that simplify reindexing and upgrading to 6.x. These tools are free with the X-Pack trial and Basic licenses and you can use them to upgrade whether or not X-Pack is a regular part of your Elastic Stack. For more information, see {stack-ref}/upgrading-elastic-stack.html.

Reindex from a remote cluster

You can use reindex from remote to migrate indices from your old cluster to a new 6.x cluster. This enables you move to 6.x from a pre-5.6 cluster without interrupting service.

Warning

Elasticsearch provides backwards compatibility support that enables indices from the previous major version to be upgraded to the current major version. Skipping a major version means that you must resolve any backward compatibility issues yourself.

{es} does not support forward compatibility across major versions. For example, you cannot reindex from a 7.x cluster into a 6.x cluster.

To migrate your indices:

  1. Set up a new 6.x cluster alongside your old cluster. Enable it to access your old cluster by adding your old cluster to the reindex.remote.whitelist in elasticsearch.yml:

    reindex.remote.whitelist: oldhost:9200
    Note

    The new cluster doesn’t have to start fully-scaled out. As you migrate indices and shift the load to the new cluster, you can add nodes to the new cluster and remove nodes from the old one.

  2. For each index that you need to migrate to the 6.x cluster:

    1. Create a new index in 6.x with the appropriate mappings and settings. Set the refresh_interval to -1 and set number_of_replicas to 0 for faster reindexing.

    2. Reindex from remote to pull documents from the old index into the new 6.x index:

      POST _reindex
      {
        "source": {
          "remote": {
            "host": "http://oldhost:9200",
            "username": "user",
            "password": "pass"
          },
          "index": "source",
          "query": {
            "match": {
              "test": "data"
            }
          }
        },
        "dest": {
          "index": "dest"
        }
      }

      If you run the reindex job in the background by setting wait_for_completion to false, the reindex request returns a task_id you can use to monitor progress of the reindex job with the task API: GET _tasks/TASK_ID.

    3. When the reindex job completes, set the refresh_interval and number_of_replicas to the desired values (the default settings are 30s and 1).

    4. Once replication is complete and the status of the new index is green, you can delete the old index.

API Conventions

Multiple Indices

Most APIs that refer to an index parameter support execution across multiple indices, using simple test1,test2,test3 notation (or _all for all indices). It also supports wildcards, for example: test* or test or te*t or *test, and the ability to "exclude" (-), for example: test*,-test3.

All multi indices APIs support the following url query string parameters:

ignore_unavailable

Controls whether to ignore if any specified indices are unavailable, including indices that don’t exist or closed indices. Either true or false can be specified.

allow_no_indices

Controls whether to fail if a wildcard indices expression results in no concrete indices. Either true or false can be specified. For example if the wildcard expression foo* is specified and no indices are available that start with foo, then depending on this setting the request will fail. This setting is also applicable when _all, *, or no index has been specified. This settings also applies for aliases, in case an alias points to a closed index.

expand_wildcards

Controls what kind of concrete indices that wildcard indices expressions can expand to. If open is specified then the wildcard expression is expanded to only open indices. If closed is specified then the wildcard expression is expanded only to closed indices. Also both values (open,closed) can be specified to expand to all indices.

If none is specified then wildcard expansion will be disabled. If all is specified, wildcard expressions will expand to all indices (this is equivalent to specifying open,closed).

The defaults settings for the above parameters depend on the API being used.

Note
Single index APIs such as the Document APIs and the single-index alias APIs do not support multiple indices.

Date math support in index names

Date math index name resolution enables you to search a range of time-series indices, rather than searching all of your time-series indices and filtering the results or maintaining aliases. Limiting the number of indices that are searched reduces the load on the cluster and improves execution performance. For example, if you are searching for errors in your daily logs, you can use a date math name template to restrict the search to the past two days.

Almost all APIs that have an index parameter support date math in the index parameter value.

A date math index name takes the following form:

<static_name{date_math_expr{date_format|time_zone}}>

Where:

static_name

is the static text part of the name

date_math_expr

is a dynamic date math expression that computes the date dynamically

date_format

is the optional format in which the computed date should be rendered. Defaults to YYYY.MM.dd.

time_zone

is the optional time zone. Defaults to utc.

Note
Pay attention to the usage of small vs capital letters used in the date_format. For example: mm denotes minute of hour, while MM denotes month of year. Similarly hh denotes the hour in the 1-12 range in combination with AM/PM, while HH denotes the hour in the 0-23 24-hour range.

Date math expressions are resolved locale-independent. Consequently, it is not possible to use any other calendars than the Gregorian calendar.

You must enclose date math index name expressions within angle brackets, and all special characters should be URI encoded. For example:

# GET /<logstash-{now/d}>/_search
GET /%3Clogstash-%7Bnow%2Fd%7D%3E/_search
{
  "query" : {
    "match": {
      "test": "data"
    }
  }
}
Note
Percent encoding of date math characters

The special characters used for date rounding must be URI encoded as follows:

<

%3C

>

%3E

/

%2F

{

%7B

}

%7D

|

%7C

+

%2B

:

%3A

,

%2C

The following example shows different forms of date math index names and the final index names they resolve to given the current time is 22nd March 2024 noon utc.

Expression Resolves to

<logstash-{now/d}>

logstash-2024.03.22

<logstash-{now/M}>

logstash-2024.03.01

<logstash-{now/M{YYYY.MM}}>

logstash-2024.03

<logstash-{now/M-1M{YYYY.MM}}>

logstash-2024.02

<logstash-{now/d{YYYY.MM.dd|+12:00}}>

logstash-2024.03.23

To use the characters { and } in the static part of an index name template, escape them with a backslash \, for example:

  • <elastic\\{ON\\}-{now/M}> resolves to elastic{ON}-2024.03.01

The following example shows a search request that searches the Logstash indices for the past three days, assuming the indices use the default Logstash index name format, logstash-YYYY.MM.dd.

# GET /<logstash-{now/d-2d}>,<logstash-{now/d-1d}>,<logstash-{now/d}>/_search
GET /%3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogstash-%7Bnow%2Fd%7D%3E/_search
{
  "query" : {
    "match": {
      "test": "data"
    }
  }
}

Common options

The following options can be applied to all of the REST APIs.

Pretty Results

When appending ?pretty=true to any request made, the JSON returned will be pretty formatted (use it for debugging only!). Another option is to set ?format=yaml which will cause the result to be returned in the (sometimes) more readable yaml format.

Human readable output

Statistics are returned in a format suitable for humans (e.g. "exists_time": "1h" or "size": "1kb") and for computers (e.g. "exists_time_in_millis": 3600000 or "size_in_bytes": 1024). The human readable values can be turned off by adding ?human=false to the query string. This makes sense when the stats results are being consumed by a monitoring tool, rather than intended for human consumption. The default for the human flag is false.

Date Math

Most parameters which accept a formatted date value — such as gt and lt in range queries, or from and to in daterange aggregations — understand date maths.

The expression starts with an anchor date, which can either be now, or a date string ending with ||. This anchor date can optionally be followed by one or more maths expressions:

  • +1h: Add one hour

  • -1d: Subtract one day

  • /d: Round down to the nearest day

The supported time units differ from those supported by time units for durations. The supported units are:

y

Years

M

Months

w

Weeks

d

Days

h

Hours

H

Hours

m

Minutes

s

Seconds

Assuming now is 2001-01-01 12:00:00, some examples are:

now+1h

now in milliseconds plus one hour. Resolves to: 2001-01-01 13:00:00

now-1h

now in milliseconds minus one hour. Resolves to: 2001-01-01 11:00:00

now-1h/d

now in milliseconds minus one hour, rounded down to UTC 00:00. Resolves to: 2001-01-01 00:00:00

2001.02.01\|\|+1M/d

2001-02-01 in milliseconds plus one month. Resolves to: 2001-03-01 00:00:00

Response Filtering

All REST APIs accept a filter_path parameter that can be used to reduce the response returned by Elasticsearch. This parameter takes a comma separated list of filters expressed with the dot notation:

GET /_search?q=elasticsearch&filter_path=took,hits.hits._id,hits.hits._score

Responds:

{
  "took" : 3,
  "hits" : {
    "hits" : [
      {
        "_id" : "0",
        "_score" : 1.6375021
      }
    ]
  }
}

It also supports the * wildcard character to match any field or part of a field’s name:

GET /_cluster/state?filter_path=metadata.indices.*.stat*

Responds:

{
  "metadata" : {
    "indices" : {
      "twitter": {"state": "open"}
    }
  }
}

And the ** wildcard can be used to include fields without knowing the exact path of the field. For example, we can return the Lucene version of every segment with this request:

GET /_cluster/state?filter_path=routing_table.indices.**.state

Responds:

{
  "routing_table": {
    "indices": {
      "twitter": {
        "shards": {
          "0": [{"state": "STARTED"}, {"state": "UNASSIGNED"}],
          "1": [{"state": "STARTED"}, {"state": "UNASSIGNED"}],
          "2": [{"state": "STARTED"}, {"state": "UNASSIGNED"}],
          "3": [{"state": "STARTED"}, {"state": "UNASSIGNED"}],
          "4": [{"state": "STARTED"}, {"state": "UNASSIGNED"}]
        }
      }
    }
  }
}

It is also possible to exclude one or more fields by prefixing the filter with the char -:

GET /_count?filter_path=-_shards

Responds:

{
  "count" : 5
}

And for more control, both inclusive and exclusive filters can be combined in the same expression. In this case, the exclusive filters will be applied first and the result will be filtered again using the inclusive filters:

GET /_cluster/state?filter_path=metadata.indices.*.state,-metadata.indices.logstash-*

Responds:

{
  "metadata" : {
    "indices" : {
      "index-1" : {"state" : "open"},
      "index-2" : {"state" : "open"},
      "index-3" : {"state" : "open"}
    }
  }
}

Note that Elasticsearch sometimes returns directly the raw value of a field, like the _source field. If you want to filter _source fields, you should consider combining the already existing _source parameter (see Get API for more details) with the filter_path parameter like this:

POST /library/book?refresh
{"title": "Book #1", "rating": 200.1}
POST /library/book?refresh
{"title": "Book #2", "rating": 1.7}
POST /library/book?refresh
{"title": "Book #3", "rating": 0.1}
GET /_search?filter_path=hits.hits._source&_source=title&sort=rating:desc
{
  "hits" : {
    "hits" : [ {
      "_source":{"title":"Book #1"}
    }, {
      "_source":{"title":"Book #2"}
    }, {
      "_source":{"title":"Book #3"}
    } ]
  }
}

Flat Settings

The flat_settings flag affects rendering of the lists of settings. When the flat_settings flag is true, settings are returned in a flat format:

GET twitter/_settings?flat_settings=true

Returns:

{
  "twitter" : {
    "settings": {
      "index.number_of_replicas": "1",
      "index.number_of_shards": "1",
      "index.creation_date": "1474389951325",
      "index.uuid": "n6gzFZTgS664GUfx0Xrpjw",
      "index.version.created": ...,
      "index.provided_name" : "twitter"
    }
  }
}

When the flat_settings flag is false, settings are returned in a more human readable structured format:

GET twitter/_settings?flat_settings=false

Returns:

{
  "twitter" : {
    "settings" : {
      "index" : {
        "number_of_replicas": "1",
        "number_of_shards": "1",
        "creation_date": "1474389951325",
        "uuid": "n6gzFZTgS664GUfx0Xrpjw",
        "version": {
          "created": ...
        },
        "provided_name" : "twitter"
      }
    }
  }
}

By default flat_settings is set to false.

Parameters

Rest parameters (when using HTTP, map to HTTP URL parameters) follow the convention of using underscore casing.

Boolean Values

All REST API parameters (both request parameters and JSON body) support providing boolean "false" as the value false and boolean "true" as the value true. All other values will raise an error.

Number Values

All REST APIs support providing numbered parameters as string on top of supporting the native JSON number types.

Time units

Whenever durations need to be specified, e.g. for a timeout parameter, the duration must specify the unit, like 2d for 2 days. The supported units are:

d

Days

h

Hours

m

Minutes

s

Seconds

ms

Milliseconds

micros

Microseconds

nanos

Nanoseconds

Byte size units

Whenever the byte size of data needs to be specified, e.g. when setting a buffer size parameter, the value must specify the unit, like 10kb for 10 kilobytes. Note that these units use powers of 1024, so 1kb means 1024 bytes. The supported units are:

b

Bytes

kb

Kilobytes

mb

Megabytes

gb

Gigabytes

tb

Terabytes

pb

Petabytes

Unit-less quantities

Unit-less quantities means that they don’t have a "unit" like "bytes" or "Hertz" or "meter" or "long tonne".

If one of these quantities is large we’ll print it out like 10m for 10,000,000 or 7k for 7,000. We’ll still print 87 when we mean 87 though. These are the supported multipliers:

k

Kilo

m

Mega

g

Giga

t

Tera

p

Peta

Distance Units

Wherever distances need to be specified, such as the distance parameter in the Geo Distance Query), the default unit is meters if none is specified. Distances can be specified in other units, such as "1km" or "2mi" (2 miles).

The full list of units is listed below:

Mile

mi or miles

Yard

yd or yards

Feet

ft or feet

Inch

in or inch

Kilometer

km or kilometers

Meter

m or meters

Centimeter

cm or centimeters

Millimeter

mm or millimeters

Nautical mile

NM, nmi, or nauticalmiles

Fuzziness

Some queries and APIs support parameters to allow inexact fuzzy matching, using the fuzziness parameter.

When querying text or keyword fields, fuzziness is interpreted as a Levenshtein Edit Distance — the number of one character changes that need to be made to one string to make it the same as another string.

The fuzziness parameter can be specified as:

0, 1, 2

The maximum allowed Levenshtein Edit Distance (or number of edits)

AUTO

Generates an edit distance based on the length of the term. Low and high distance arguments may be optionally provided AUTO:[low],[high]. If not specified, the default values are 3 and 6, equivalent to AUTO:3,6 that make for lengths:

0..2

Must match exactly

3..5

One edit allowed

>5

Two edits allowed

AUTO should generally be the preferred value for fuzziness.

Enabling stack traces

By default when a request returns an error Elasticsearch doesn’t include the stack trace of the error. You can enable that behavior by setting the error_trace url parameter to true. For example, by default when you send an invalid size parameter to the _search API:

POST /twitter/_search?size=surprise_me

The response looks like:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Failed to parse int parameter [size] with value [surprise_me]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Failed to parse int parameter [size] with value [surprise_me]",
    "caused_by" : {
      "type" : "number_format_exception",
      "reason" : "For input string: \"surprise_me\""
    }
  },
  "status" : 400
}

But if you set error_trace=true:

POST /twitter/_search?size=surprise_me&error_trace=true

The response looks like:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Failed to parse int parameter [size] with value [surprise_me]",
        "stack_trace": "Failed to parse int parameter [size] with value [surprise_me]]; nested: IllegalArgumentException..."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Failed to parse int parameter [size] with value [surprise_me]",
    "stack_trace": "java.lang.IllegalArgumentException: Failed to parse int parameter [size] with value [surprise_me]\n    at org.elasticsearch.rest.RestRequest.paramAsInt(RestRequest.java:175)...",
    "caused_by": {
      "type": "number_format_exception",
      "reason": "For input string: \"surprise_me\"",
      "stack_trace": "java.lang.NumberFormatException: For input string: \"surprise_me\"\n    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)..."
    }
  },
  "status": 400
}

Request body in query string

For libraries that don’t accept a request body for non-POST requests, you can pass the request body as the source query string parameter instead. When using this method, the source_content_type parameter should also be passed with a media type value that indicates the format of the source, such as application/json.

Content-Type Requirements

The type of the content sent in a request body must be specified using the Content-Type header. The value of this header must map to one of the supported formats that the API supports. Most APIs support JSON, YAML, CBOR, and SMILE. The bulk and multi-search APIs support NDJSON, JSON, and SMILE; other types will result in an error response.

Additionally, when using the source query string parameter, the content type must be specified using the source_content_type query string parameter.

URL-based access control

Many users use a proxy with URL-based access control to secure access to Elasticsearch indices. For multi-search, multi-get, and bulk requests, the user has the choice of specifying an index in the URL and on each individual request within the request body. This can make URL-based access control challenging.

To prevent the user from overriding the index which has been specified in the URL, add this setting to the elasticsearch.yml file:

rest.action.multi.allow_explicit_index: false

The default value is true, but when set to false, Elasticsearch will reject requests that have an explicit index specified in the request body.

Document APIs

Reading and Writing documents

Introduction

Each index in Elasticsearch is divided into shards and each shard can have multiple copies. These copies are known as a replication group and must be kept in sync when documents are added or removed. If we fail to do so, reading from one copy will result in very different results than reading from another. The process of keeping the shard copies in sync and serving reads from them is what we call the data replication model.

Elasticsearch’s data replication model is based on the primary-backup model and is described very well in the PacificA paper of Microsoft Research. That model is based on having a single copy from the replication group that acts as the primary shard. The other copies are called replica shards. The primary serves as the main entry point for all indexing operations. It is in charge of validating them and making sure they are correct. Once an index operation has been accepted by the primary, the primary is also responsible for replicating the operation to the other copies.

This purpose of this section is to give a high level overview of the Elasticsearch replication model and discuss the implications it has for various interactions between write and read operations.

Basic write model

Every indexing operation in Elasticsearch is first resolved to a replication group using routing, typically based on the document ID. Once the replication group has been determined, the operation is forwarded internally to the current primary shard of the group. The primary shard is responsible for validating the operation and forwarding it to the other replicas. Since replicas can be offline, the primary is not required to replicate to all replicas. Instead, Elasticsearch maintains a list of shard copies that should receive the operation. This list is called the in-sync copies and is maintained by the master node. As the name implies, these are the set of "good" shard copies that are guaranteed to have processed all of the index and delete operations that have been acknowledged to the user. The primary is responsible for maintaining this invariant and thus has to replicate all operations to each copy in this set.

The primary shard follows this basic flow:

  1. Validate incoming operation and reject it if structurally invalid (Example: have an object field where a number is expected)

  2. Execute the operation locally i.e. indexing or deleting the relevant document. This will also validate the content of fields and reject if needed (Example: a keyword value is too long for indexing in Lucene).

  3. Forward the operation to each replica in the current in-sync copies set. If there are multiple replicas, this is done in parallel.

  4. Once all replicas have successfully performed the operation and responded to the primary, the primary acknowledges the successful completion of the request to the client.

Failure handling

Many things can go wrong during indexing — disks can get corrupted, nodes can be disconnected from each other, or some configuration mistake could cause an operation to fail on a replica despite it being successful on the primary. These are infrequent but the primary has to respond to them.

In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by default) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See here for more details.

Once the operation has been successfully performed on the primary, the primary has to deal with potential failures when executing it on the replica shards. This may be caused by an actual failure on the replica or due to a network issue preventing the operation from reaching the replica (or preventing the replica from responding). All of these share the same end result: a replica which is part of the in-sync replica set misses an operation that is about to be acknowledged. In order to avoid violating the invariant, the primary sends a message to the master requesting that the problematic shard be removed from the in-sync replica set. Only once removal of the shard has been acknowledged by the master does the primary acknowledge the operation. Note that the master will also instruct another node to start building a new shard copy in order to restore the system to a healthy state.

While forwarding an operation to the replicas, the primary will use the replicas to validate that it is still the active primary. If the primary has been isolated due to a network partition (or a long GC) it may continue to process incoming indexing operations before realising that it has been demoted. Operations that come from a stale primary will be rejected by the replicas. When the primary receives a response from the replica rejecting its request because it is no longer the primary then it will reach out to the master and will learn that it has been replaced. The operation is then routed to the new primary.

What happens if there are no replicas?

This is a valid scenario that can happen due to index configuration or simply because all the replicas have failed. In that case the primary is processing operations without any external validation, which may seem problematic. On the other hand, the primary cannot fail other shards on its own but request the master to do so on its behalf. This means that the master knows that the primary is the only single good copy. We are therefore guaranteed that the master will not promote any other (out-of-date) shard copy to be a new primary and that any operation indexed into the primary will not be lost. Of course, since at that point we are running with only single copy of the data, physical hardware issues can cause data loss. See Wait For Active Shards for some mitigation options.

Basic read model

Reads in Elasticsearch can be very lightweight lookups by ID or a heavy search request with complex aggregations that take non-trivial CPU power. One of the beauties of the primary-backup model is that it keeps all shard copies identical (with the exception of in-flight operations). As such, a single in-sync copy is sufficient to serve read requests.

When a read request is received by a node, that node is responsible for forwarding it to the nodes that hold the relevant shards, collating the responses, and responding to the client. We call that node the coordinating node for that request. The basic flow is as follows:

  1. Resolve the read requests to the relevant shards. Note that since most searches will be sent to one or more indices, they typically need to read from multiple shards, each representing a different subset of the data.

  2. Select an active copy of each relevant shard, from the shard replication group. This can be either the primary or a replica. By default, Elasticsearch will simply round robin between the shard copies.

  3. Send shard level read requests to the selected copies.

  4. Combine the results and respond. Note that in the case of get by ID look up, only one shard is relevant and this step can be skipped.

Shard failures

When a shard fails to respond to a read request, the coordinating node sends the request to another shard copy in the same replication group. Repeated failures can result in no available shard copies.

To ensure fast responses, the following APIs will respond with partial results if one or more shards fail:

Responses containing partial results still provide a 200 OK HTTP status code. Shard failures are indicated by the timed_out and _shards fields of the response header.

A few simple implications

Each of these basic flows determines how Elasticsearch behaves as a system for both reads and writes. Furthermore, since read and write requests can be executed concurrently, these two basic flows interact with each other. This has a few inherent implications:

Efficient reads

Under normal operation each read operation is performed once for each relevant replication group. Only under failure conditions do multiple copies of the same shard execute the same search.

Read unacknowledged

Since the primary first indexes locally and then replicates the request, it is possible for a concurrent read to already see the change before it has been acknowledged.

Two copies by default

This model can be fault tolerant while maintaining only two copies of the data. This is in contrast to quorum-based system where the minimum number of copies for fault tolerance is 3.

Failures

Under failures, the following is possible:

A single shard can slow down indexing

Because the primary waits for all replicas in the in-sync copies set during each operation, a single slow shard can slow down the entire replication group. This is the price we pay for the read efficiency mentioned above. Of course a single slow shard will also slow down unlucky searches that have been routed to it.

Dirty reads

An isolated primary can expose writes that will not be acknowledged. This is caused by the fact that an isolated primary will only realize that it is isolated once it sends requests to its replicas or when reaching out to the master. At that point the operation is already indexed into the primary and can be read by a concurrent read. Elasticsearch mitigates this risk by pinging the master every second (by default) and rejecting indexing operations if no master is known.

The Tip of the Iceberg

This document provides a high level overview of how Elasticsearch deals with data. Of course, there is much much more going on under the hood. Things like primary terms, cluster state publishing, and master election all play a role in keeping this system behaving correctly. This document also doesn’t cover known and important bugs (both closed and open). We recognize that GitHub is hard to keep up with. To help people stay on top of those, we maintain a dedicated resiliency page on our website. We strongly advise reading it.

Index API

Important
See Removal of mapping types.

The index API adds or updates a typed JSON document in a specific index, making it searchable. The following example inserts the JSON document into the "twitter" index, under a type called _doc with an id of 1:

PUT twitter/_doc/1
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

The result of the above index operation is:

{
    "_shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 2
    },
    "_index" : "twitter",
    "_type" : "_doc",
    "_id" : "1",
    "_version" : 1,
    "_seq_no" : 0,
    "_primary_term" : 1,
    "result" : "created"
}

The _shards header provides information about the replication process of the index operation:

total

Indicates how many shard copies (primary and replica shards) the index operation should be executed on.

successful

Indicates the number of shard copies the index operation succeeded on.

failed

An array that contains replication-related errors in the case an index operation failed on a replica shard.

The index operation is successful in the case successful is at least 1.

Note
Replica shards may not all be started when an indexing operation successfully returns (by default, only the primary is required, but this behavior can be changed). In that case, total will be equal to the total shards based on the number_of_replicas setting and successful will be equal to the number of shards started (primary plus replicas). If there were no failures, the failed will be 0.

Automatic Index Creation

The index operation automatically creates an index if it does not already exist, and applies any index templates that are configured. The index operation also creates a dynamic type mapping for the specified type if one does not already exist. By default, new fields and objects will automatically be added to the mapping definition for the specified type if needed. Check out the mapping section for more information on mapping definitions, and the put mapping API for information about updating type mappings manually.

Automatic index creation is controlled by the action.auto_create_index setting. This setting defaults to true, meaning that indices are always automatically created. Automatic index creation can be permitted only for indices matching certain patterns by changing the value of this setting to a comma-separated list of these patterns. It can also be explicitly permitted and forbidden by prefixing patterns in the list with a + or -. Finally it can be completely disabled by changing this setting to false.

PUT _cluster/settings
{
    "persistent": {
        "action.auto_create_index": "twitter,index10,-index1*,+ind*" (1)
    }
}

PUT _cluster/settings
{
    "persistent": {
        "action.auto_create_index": "false" (2)
    }
}

PUT _cluster/settings
{
    "persistent": {
        "action.auto_create_index": "true" (3)
    }
}
  1. Permit only the auto-creation of indices called twitter, index10, no other index matching index1*, and any other index matching ind*. The patterns are matched in the order in which they are given.

  2. Completely disable the auto-creation of indices.

  3. Permit the auto-creation of indices with any name. This is the default.

Operation Type

The index operation also accepts an op_type that can be used to force a create operation, allowing for "put-if-absent" behavior. When create is used, the index operation will fail if a document by that id already exists in the index.

Here is an example of using the op_type parameter:

PUT twitter/_doc/1?op_type=create
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

Another option to specify create is to use the following uri:

PUT twitter/_doc/1/_create
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

Automatic ID Generation

The index operation can be executed without specifying the id. In such a case, an id will be generated automatically. In addition, the op_type will automatically be set to create. Here is an example (note the POST used instead of PUT):

POST twitter/_doc/
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

The result of the above index operation is:

{
    "_shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 2
    },
    "_index" : "twitter",
    "_type" : "_doc",
    "_id" : "W0tpsmIBdwcYyG50zbta",
    "_version" : 1,
    "_seq_no" : 0,
    "_primary_term" : 1,
    "result": "created"
}

Optimistic concurrency control

Index operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters. If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409. See Optimistic concurrency control for more details.

Routing

By default, shard placement — or routing — is controlled by using a hash of the document’s id value. For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing parameter. For example:

POST twitter/_doc?routing=kimchy
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

In the example above, the "_doc" document is routed to a shard based on the routing parameter provided: "kimchy".

When setting up explicit mapping, the _routing field can be optionally used to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the _routing mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted.

Distributed

The index operation is directed to the primary shard based on its route (see the Routing section above) and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas.

Wait For Active Shards

To improve the resiliency of writes to the system, indexing operations can be configured to wait for a certain number of active shard copies before proceeding with the operation. If the requisite number of active shard copies are not available, then the write operation must wait and retry, until either the requisite shard copies have started or a timeout occurs. By default, write operations only wait for the primary shards to be active before proceeding (i.e. wait_for_active_shards=1). This default can be overridden in the index settings dynamically by setting index.write.wait_for_active_shards. To alter this behavior per operation, the wait_for_active_shards request parameter can be used.

Valid values are all or any positive integer up to the total number of configured copies per shard in the index (which is number_of_replicas+1). Specifying a negative value or a number greater than the number of shard copies will throw an error.

For example, suppose we have a cluster of three nodes, A, B, and C and we create an index index with the number of replicas set to 3 (resulting in 4 shard copies, one more copy than there are nodes). If we attempt an indexing operation, by default the operation will only ensure the primary copy of each shard is available before proceeding. This means that even if B and C went down, and A hosted the primary shard copies, the indexing operation would still proceed with only one copy of the data. If wait_for_active_shards is set on the request to 3 (and all 3 nodes are up), then the indexing operation will require 3 active shard copies before proceeding, a requirement which should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard. However, if we set wait_for_active_shards to all (or to 4, which is the same), the indexing operation will not proceed as we do not have all 4 copies of each shard active in the index. The operation will timeout unless a new node is brought up in the cluster to host the fourth copy of the shard.

It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation commences. Once the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary. The _shards section of the write operation’s response reveals the number of shard copies on which replication succeeded/failed.

{
    "_shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 2
    }
}

Refresh

Control when the changes made by this request are visible to search. See refresh.

Noop Updates

When updating a document using the index API a new version of the document is always created even if the document hasn’t changed. If this isn’t acceptable use the _update API with detect_noop set to true. This option isn’t available on the index API because the index API doesn’t fetch the old source and isn’t able to compare it against the new source.

There isn’t a hard and fast rule about when noop updates aren’t acceptable. It’s a combination of lots of factors like how frequently your data source sends updates that are actually noops and how many queries per second Elasticsearch runs on the shard receiving the updates.

Timeout

The primary shard assigned to perform the index operation might not be available when the index operation is executed. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the index operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. The timeout parameter can be used to explicitly specify how long it waits. Here is an example of setting it to 5 minutes:

PUT twitter/_doc/1?timeout=5m
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

Versioning

Each indexed document is given a version number. By default, internal versioning is used that starts at 1 and increments with each update, deletes included. Optionally, the version number can be set to an external value (for example, if maintained in a database). To enable this functionality, version_type should be set to external. The value provided must be a numeric, long value greater than or equal to 0, and less than around 9.2e+18.

When using the external version type, the system checks to see if the version number passed to the index request is greater than the version of the currently stored document. If true, the document will be indexed and the new version number used. If the value provided is less than or equal to the stored document’s version number, a version conflict will occur and the index operation will fail. For example:

PUT twitter/_doc/1?version=2&version_type=external
{
    "message" : "elasticsearch now has versioning support, double cool!"
}

NOTE: Versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, then the operation is executed without any version checks.

The above will succeed since the supplied version of 2 is higher than the current document version of 1. If the document was already updated and its version was set to 2 or higher, the indexing command will fail and result in a conflict (409 http status code).

A nice side effect is that there is no need to maintain strict ordering of async indexing operations executed as a result of changes to a source database, as long as version numbers from the source database are used. Even the simple case of updating the Elasticsearch index using data from a database is simplified if external versioning is used, as only the latest version will be used if the index operations arrive out of order for whatever reason.

Version types

Next to the external version type explained above, Elasticsearch also supports other types for specific use cases. Here is an overview of the different version types and their semantics.

internal

Only index the document if the given version is identical to the version of the stored document. deprecated:[6.7.0, "Please use if_seq_no & if_primary_term instead. See Optimistic concurrency control for more details."]

external or external_gt

Only index the document if the given version is strictly higher than the version of the stored document or if there is no existing document. The given version will be used as the new version and will be stored with the new document. The supplied version must be a non-negative long number.

external_gte

Only index the document if the given version is equal or higher than the version of the stored document. If there is no existing document the operation will succeed as well. The given version will be used as the new version and will be stored with the new document. The supplied version must be a non-negative long number.

NOTE: The external_gte version type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data. There is another option, force, which is deprecated because it can cause primary and replica shards to diverge.

Get API

The get API allows to get a typed JSON document from the index based on its id. The following example gets a JSON document from an index called twitter, under a type called _doc, with id valued 0:

GET twitter/_doc/0

The result of the above get operation is:

{
    "_index" : "twitter",
    "_type" : "_doc",
    "_id" : "0",
    "_version" : 1,
    "_seq_no" : 10,
    "_primary_term" : 1,
    "found": true,
    "_source" : {
        "user" : "kimchy",
        "date" : "2009-11-15T14:12:12",
        "likes": 0,
        "message" : "trying out Elasticsearch"
    }
}

The above result includes the _index, _type, _id, and _version of the document we wish to retrieve, including the actual _source of the document if it could be found (as indicated by the found field in the response).

The API also allows to check for the existence of a document using HEAD, for example:

HEAD twitter/_doc/0

Realtime

By default, the get API is realtime, and is not affected by the refresh rate of the index (when data will become visible for search). If a document has been updated but is not yet refreshed, the get API will issue a refresh call in-place to make the document visible. This will also make other documents changed since the last refresh visible. In order to disable realtime GET, one can set the realtime parameter to false.

Source filtering

By default, the get operation returns the contents of the _source field unless you have used the stored_fields parameter or if the _source field is disabled. You can turn off _source retrieval by using the _source parameter:

GET twitter/_doc/0?_source=false

If you only need one or two fields from the complete _source, you can use the _source_includes and _source_excludes parameters to include or filter out the parts you need. This can be especially helpful with large documents where partial retrieval can save on network overhead. Both parameters take a comma separated list of fields or wildcard expressions. Example:

GET twitter/_doc/0?_source_includes=*.id&_source_excludes=entities

If you only want to specify includes, you can use a shorter notation:

GET twitter/_doc/0?_source=*.id,retweeted

Stored Fields

The get operation allows specifying a set of stored fields that will be returned by passing the stored_fields parameter. If the requested fields are not stored, they will be ignored. Consider for instance the following mapping:

PUT twitter
{
   "mappings": {
      "_doc": {
         "properties": {
            "counter": {
               "type": "integer",
               "store": false
            },
            "tags": {
               "type": "keyword",
               "store": true
            }
         }
      }
   }
}

Now we can add a document:

PUT twitter/_doc/1
{
    "counter" : 1,
    "tags" : ["red"]
}

And then try to retrieve it:

GET twitter/_doc/1?stored_fields=tags,counter

The result of the above get operation is:

{
   "_index": "twitter",
   "_type": "_doc",
   "_id": "1",
   "_version": 1,
   "_seq_no" : 22,
   "_primary_term" : 1,
   "found": true,
   "fields": {
      "tags": [
         "red"
      ]
   }
}

Field values fetched from the document itself are always returned as an array. Since the counter field is not stored the get request simply ignores it when trying to get the stored_fields.

It is also possible to retrieve metadata fields like the _routing field:

PUT twitter/_doc/2?routing=user1
{
    "counter" : 1,
    "tags" : ["white"]
}
GET twitter/_doc/2?routing=user1&stored_fields=tags,counter

The result of the above get operation is:

{
   "_index": "twitter",
   "_type": "_doc",
   "_id": "2",
   "_version": 1,
   "_seq_no" : 13,
   "_primary_term" : 1,
   "_routing": "user1",
   "found": true,
   "fields": {
      "tags": [
         "white"
      ]
   }
}

Also only leaf fields can be returned via the stored_field option. So object fields can’t be returned and such requests will fail.

Getting the _source directly

Use the /{index}/{type}/{id}/_source endpoint to get just the _source field of the document, without any additional content around it. For example:

GET twitter/_doc/1/_source

You can also use the same source filtering parameters to control which parts of the _source will be returned:

GET twitter/_doc/1/_source?_source_includes=*.id&_source_excludes=entities

Note, there is also a HEAD variant for the _source endpoint to efficiently test for document _source existence. An existing document will not have a _source if it is disabled in the mapping.

HEAD twitter/_doc/1/_source

Routing

When indexing using the ability to control the routing, in order to get a document, the routing value should also be provided. For example:

GET twitter/_doc/2?routing=user1

The above will get a tweet with id 2, but will be routed based on the user. Note that issuing a get without the correct routing will cause the document not to be fetched.

Preference

Controls a preference of which shard replicas to execute the get request on. By default, the operation is randomized between the shard replicas.

The preference can be set to:

_primary

The operation will go and be executed only on the primary shards.

_local

The operation will prefer to be executed on a local allocated shard if possible.

Custom (string) value

A custom value will be used to guarantee that the same shards will be used for the same custom value. This can help with "jumping values" when hitting different shards in different refresh states. A sample value can be something like the web session id, or the user name.

Refresh

The refresh parameter can be set to true in order to refresh the relevant shard before the get operation and make it searchable. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slows down indexing).

Distributed

The get operation gets hashed into a specific shard id. It then gets redirected to one of the replicas within that shard id and returns the result. The replicas are the primary shard and its replicas within that shard id group. This means that the more replicas we have, the better GET scaling we will have.

Versioning support

You can use the version parameter to retrieve the document only if its current version is equal to the specified one. This behavior is the same for all version types with the exception of version type FORCE which always retrieves the document. Note that FORCE version type is deprecated.

Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn’t disappear immediately, although you won’t be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.

Delete API

The delete API allows to delete a typed JSON document from a specific index based on its id. The following example deletes the JSON document from an index called twitter, under a type called _doc, with id 1:

DELETE /twitter/_doc/1

The result of the above delete operation is:

{
    "_shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 2
    },
    "_index" : "twitter",
    "_type" : "_doc",
    "_id" : "1",
    "_version" : 2,
    "_primary_term": 1,
    "_seq_no": 5,
    "result": "deleted"
}

Optimistic concurrency control

Delete operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters. If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409. See Optimistic concurrency control for more details.

Versioning

Each document indexed is versioned. When deleting a document, the version can be specified to make sure the relevant document we are trying to delete is actually being deleted and it has not changed in the meantime. Every write operation executed on a document, deletes included, causes its version to be incremented. The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations. The length of time for which a deleted document’s version remains available is determined by the index.gc_deletes index setting and defaults to 60 seconds.

Routing

When indexing using the ability to control the routing, in order to delete a document, the routing value should also be provided. For example:

DELETE /twitter/_doc/1?routing=kimchy

The above will delete a tweet with id 1, but will be routed based on the user. Note that issuing a delete without the correct routing will cause the document to not be deleted.

When the _routing mapping is set as required and no routing value is specified, the delete API will throw a RoutingMissingException and reject the request.

Automatic index creation

If an external versioning variant is used, the delete operation automatically creates an index if it has not been created before (check out the create index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if it has not been created before (check out the put mapping API for manually creating type mapping).

Distributed

The delete operation gets hashed into a specific shard id. It then gets redirected into the primary shard within that id group, and replicated (if needed) to shard replicas within that id group.

Wait For Active Shards

When making delete requests, you can set the wait_for_active_shards parameter to require a minimum number of shard copies to be active before starting to process the delete request. See here for further details and a usage example.

Refresh

Control when the changes made by this request are visible to search. See ?refresh.

Timeout

The primary shard assigned to perform the delete operation might not be available when the delete operation is executed. Some reasons for this might be that the primary shard is currently recovering from a store or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. The timeout parameter can be used to explicitly specify how long it waits. Here is an example of setting it to 5 minutes:

DELETE /twitter/_doc/1?timeout=5m

Delete By Query API

The simplest usage of _delete_by_query just performs a deletion on every document that matches a query. Here is the API:

POST twitter/_delete_by_query
{
  "query": { (1)
    "match": {
      "message": "some message"
    }
  }
}
  1. The query must be passed as a value to the query key, in the same way as the Search API. You can also use the q parameter in the same way as the search API.

That will return something like this:

{
  "took" : 147,
  "timed_out": false,
  "deleted": 119,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "total": 119,
  "failures" : [ ]
}

_delete_by_query gets a snapshot of the index when it starts and deletes what it finds using internal versioning. That means that you’ll get a version conflict if the document changes between the time when the snapshot was taken and when the delete request is processed. When the versions match the document is deleted.

Note
Since internal versioning does not support the value 0 as a valid version number, documents with version equal to zero cannot be deleted using _delete_by_query and will fail the request.

During the _delete_by_query execution, multiple search requests are sequentially executed in order to find all the matching documents to delete. Every time a batch of documents is found, a corresponding bulk request is executed to delete all these documents. In case a search or bulk request got rejected, _delete_by_query relies on a default policy to retry rejected requests (up to 10 times, with exponential back off). Reaching the maximum retries limit causes the _delete_by_query to abort and all failures are returned in the failures of the response. The deletions that have been performed still stick. In other words, the process is not rolled back, only aborted. While the first failure causes the abort, all failures that are returned by the failing bulk request are returned in the failures element; therefore it’s possible for there to be quite a few failed entities.

If you’d like to count version conflicts rather than cause them to abort, then set conflicts=proceed on the url or "conflicts": "proceed" in the request body.

Back to the API format, this will delete tweets from the twitter index:

POST twitter/_doc/_delete_by_query?conflicts=proceed
{
  "query": {
    "match_all": {}
  }
}

It’s also possible to delete documents of multiple indexes and multiple types at once, just like the search API:

POST twitter,blog/_docs,post/_delete_by_query
{
  "query": {
    "match_all": {}
  }
}

If you provide routing then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

POST twitter/_delete_by_query?routing=1
{
  "query": {
    "range" : {
        "age" : {
           "gte" : 10
        }
    }
  }
}

By default _delete_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

POST twitter/_delete_by_query?scroll_size=5000
{
  "query": {
    "term": {
      "user": "kimchy"
    }
  }
}

URL Parameters

In addition to the standard parameters like pretty, the delete by query API also supports refresh, wait_for_completion, wait_for_active_shards, timeout, and scroll.

Sending the refresh will refresh all shards involved in the delete by query once the request completes. This is different than the delete API’s refresh parameter which causes just the shard that received the delete request to be refreshed. Also unlike the delete API it does not support wait_for.

If the request contains wait_for_completion=false then Elasticsearch will perform some preflight checks, launch the request, and then return a task which can be used with Tasks APIs to cancel or get the status of the task. Elasticsearch will also create a record of this task as a document at .tasks/task/${taskId}. This is yours to keep or remove as you see fit. When you are done with it, delete it so Elasticsearch can reclaim the space it uses.

wait_for_active_shards controls how many copies of a shard must be active before proceeding with the request. See here for details. timeout controls how long each write request waits for unavailable shards to become available. Both work exactly how they work in the Bulk API. As _delete_by_query uses scroll search, you can also specify the scroll parameter to control how long it keeps the "search context" alive, e.g. ?scroll=10m. By default it’s 5 minutes.

requests_per_second can be set to any positive decimal number (1.4, 6, 1000, etc.) and throttles the rate at which delete by query issues batches of delete operations by padding each batch with a wait time. The throttling can be disabled by setting requests_per_second to -1.

The throttling is done by waiting between batches so that scroll that _delete_by_query uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing. By default the batch size is 1000, so if the requests_per_second is set to 500:

target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds

Since the batch is issued as a single _bulk request, large batch sizes will cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth". The default is -1.

Response body

The JSON response looks like this:

{
  "took" : 147,
  "timed_out": false,
  "total": 119,
  "deleted": 119,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "failures" : [ ]
}
took

The number of milliseconds from start to end of the whole operation.

timed_out

This flag is set to true if any of the requests executed during the delete by query execution has timed out.

total

The number of documents that were successfully processed.

deleted

The number of documents that were successfully deleted.

batches

The number of scroll responses pulled back by the delete by query.

version_conflicts

The number of version conflicts that the delete by query hit.

noops

This field is always equal to zero for delete by query. It only exists so that delete by query, update by query, and reindex APIs return responses with the same structure.

retries

The number of retries attempted by delete by query. bulk is the number of bulk actions retried, and search is the number of search actions retried.

throttled_millis

Number of milliseconds the request slept to conform to requests_per_second.

requests_per_second

The number of requests per second effectively executed during the delete by query.

throttled_until_millis

This field should always be equal to zero in a _delete_by_query response. It only has meaning when using the Task API, where it indicates the next time (in milliseconds since epoch) a throttled request will be executed again in order to conform to requests_per_second.

failures

Array of failures if there were any unrecoverable errors during the process. If this is non-empty then the request aborted because of those failures. Delete by query is implemented using batches, and any failure causes the entire process to abort but all failures in the current batch are collected into the array. You can use the conflicts option to prevent reindex from aborting on version conflicts.

Works with the Task API

You can fetch the status of any running delete by query requests with the Task API:

GET _tasks?detailed=true&actions=*/delete/byquery

The response looks like:

{
  "nodes" : {
    "r1A2WoRbTwKZ516z6NEs5A" : {
      "name" : "r1A2WoR",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "attributes" : {
        "testattr" : "test",
        "portsfile" : "true"
      },
      "tasks" : {
        "r1A2WoRbTwKZ516z6NEs5A:36619" : {
          "node" : "r1A2WoRbTwKZ516z6NEs5A",
          "id" : 36619,
          "type" : "transport",
          "action" : "indices:data/write/delete/byquery",
          "status" : {    (1)
            "total" : 6154,
            "updated" : 0,
            "created" : 0,
            "deleted" : 3500,
            "batches" : 36,
            "version_conflicts" : 0,
            "noops" : 0,
            "retries": 0,
            "throttled_millis": 0
          },
          "description" : ""
        }
      }
    }
  }
}
  1. This object contains the actual status. It is just like the response JSON with the important addition of the total field. total is the total number of operations that the reindex expects to perform. You can estimate the progress by adding the updated, created, and deleted fields. The request will finish when their sum is equal to the total field.

With the task id you can look up the task directly:

GET /_tasks/r1A2WoRbTwKZ516z6NEs5A:36619

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set on it then it’ll come back with results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

Works with the Cancel Task API

Any delete by query can be canceled using the task cancel API:

POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel

The task ID can be found using the tasks API.

Cancellation should happen quickly but might take a few seconds. The task status API above will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.

Rethrottling

The value of requests_per_second can be changed on a running delete by query using the _rethrottle API:

POST _delete_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1

The task ID can be found using the tasks API.

Just like when setting it on the delete by query API, requests_per_second can be either -1 to disable throttling or any decimal number like 1.7 or 12 to throttle to that level. Rethrottling that speeds up the query takes effect immediately but rethrotting that slows down the query will take effect after completing the current batch. This prevents scroll timeouts.

Slicing

Delete by query supports sliced scroll to parallelize the deleting process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Manual slicing

Slice a delete by query manually by providing a slice id and total number of slices to each request:

POST twitter/_delete_by_query
{
  "slice": {
    "id": 0,
    "max": 2
  },
  "query": {
    "range": {
      "likes": {
        "lt": 10
      }
    }
  }
}
POST twitter/_delete_by_query
{
  "slice": {
    "id": 1,
    "max": 2
  },
  "query": {
    "range": {
      "likes": {
        "lt": 10
      }
    }
  }
}

Which you can verify works with:

GET _refresh
POST twitter/_search?size=0&filter_path=hits.total
{
  "query": {
    "range": {
      "likes": {
        "lt": 10
      }
    }
  }
}

Which results in a sensible total like this one:

{
  "hits": {
    "total": 0
  }
}

Automatic slicing

You can also let delete by query automatically parallelize using sliced scroll to slice on _uid. Use slices to specify the number of slices to use:

POST twitter/_delete_by_query?refresh&slices=5
{
  "query": {
    "range": {
      "likes": {
        "lt": 10
      }
    }
  }
}

Which you also can verify works with:

POST twitter/_search?size=0&filter_path=hits.total
{
  "query": {
    "range": {
      "likes": {
        "lt": 10
      }
    }
  }
}

Which results in a sensible total like this one:

{
  "hits": {
    "total": 0
  }
}

Setting slices to auto will let Elasticsearch choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source indices, it will choose the number of slices based on the index with the smallest number of shards.

Adding slices to _delete_by_query just automates the manual process used in the section above, creating sub-requests which means it has some quirks:

  • You can see these requests in the Tasks APIs. These sub-requests are "child" tasks of the task for the request with slices.

  • Fetching the status of the task for the request with slices only contains the status of completed slices.

  • These sub-requests are individually addressable for things like cancellation and rethrottling.

  • Rethrottling the request with slices will rethrottle the unfinished sub-request proportionally.

  • Canceling the request with slices will cancel each sub-request.

  • Due to the nature of slices each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.

  • Parameters like requests_per_second and size on a request with slices are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that the using size with slices might not result in exactly size documents being deleted.

  • Each sub-request gets a slightly different snapshot of the source index though these are all taken at approximately the same time.

Picking the number of slices

If slicing automatically, setting slices to auto will choose a reasonable number for most indices. If you’re slicing manually or otherwise tuning automatic slicing, use these guidelines.

Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large (for example, 500), choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.

Delete performance scales linearly across available resources with the number of slices.

Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources.

Update API

The update API allows to update a document based on a script provided. The operation gets the document (collocated with the shard) from the index, runs the script (with optional script language and parameters), and indexes back the result (also allows to delete, or ignore the operation). It uses versioning to make sure no updates have happened during the "get" and "reindex".

Note, this operation still means full reindex of the document, it just removes some network roundtrips and reduces chances of version conflicts between the get and the index. The _source field needs to be enabled for this feature to work.

For example, let’s index a simple doc:

PUT test/_doc/1
{
    "counter" : 1,
    "tags" : ["red"]
}

Scripted updates

Now, we can execute a script that would increment the counter:

POST test/_doc/1/_update
{
    "script" : {
        "source": "ctx._source.counter += params.count",
        "lang": "painless",
        "params" : {
            "count" : 4
        }
    }
}

We can add a tag to the list of tags (if the tag exists, it still gets added, since this is a list):

POST test/_doc/1/_update
{
    "script" : {
        "source": "ctx._source.tags.add(params.tag)",
        "lang": "painless",
        "params" : {
            "tag" : "blue"
        }
    }
}

We can remove a tag from the list of tags. Note that the Painless function to remove a tag takes as its parameter the array index of the element you wish to remove, so you need a bit more logic to locate it while avoiding a runtime error. Note that if the tag was present more than once in the list, this will remove only one occurrence of it:

POST test/_doc/1/_update
{
    "script" : {
        "source": "if (ctx._source.tags.contains(params.tag)) { ctx._source.tags.remove(ctx._source.tags.indexOf(params.tag)) }",
        "lang": "painless",
        "params" : {
            "tag" : "blue"
        }
    }
}

In addition to _source, the following variables are available through the ctx map: _index, _type, _id, _version, _routing, and _now (the current timestamp).

We can also add a new field to the document:

POST test/_doc/1/_update
{
    "script" : "ctx._source.new_field = 'value_of_new_field'"
}

Or remove a field from the document:

POST test/_doc/1/_update
{
    "script" : "ctx._source.remove('new_field')"
}

And, we can even change the operation that is executed. This example deletes the doc if the tags field contains green, otherwise it does nothing (noop):

POST test/_doc/1/_update
{
    "script" : {
        "source": "if (ctx._source.tags.contains(params.tag)) { ctx.op = 'delete' } else { ctx.op = 'none' }",
        "lang": "painless",
        "params" : {
            "tag" : "green"
        }
    }
}

Updates with a partial document

The update API also supports passing a partial document, which will be merged into the existing document (simple recursive merge, inner merging of objects, replacing core "keys/values" and arrays). To fully replace the existing document, the index API should be used instead. The following partial update adds a new field to the existing document:

POST test/_doc/1/_update
{
    "doc" : {
        "name" : "new_name"
    }
}

If both doc and script are specified, then doc is ignored. Best is to put your field pairs of the partial document in the script itself.

Detecting noop updates

If doc is specified its value is merged with the existing _source. By default updates that don’t change anything detect that they don’t change anything and return "result": "noop" like this:

POST test/_doc/1/_update
{
    "doc" : {
        "name" : "new_name"
    }
}

If name was new_name before the request was sent then the entire update request is ignored. The result element in the response returns noop if the request was ignored.

{
   "_shards": {
        "total": 0,
        "successful": 0,
        "failed": 0
   },
   "_index": "test",
   "_type": "_doc",
   "_id": "1",
   "_version": 7,
   "result": "noop"
}

You can disable this behavior by setting "detect_noop": false like this:

POST test/_doc/1/_update
{
    "doc" : {
        "name" : "new_name"
    },
    "detect_noop": false
}

Upserts

If the document does not already exist, the contents of the upsert element will be inserted as a new document. If the document does exist, then the script will be executed instead:

POST test/_doc/1/_update
{
    "script" : {
        "source": "ctx._source.counter += params.count",
        "lang": "painless",
        "params" : {
            "count" : 4
        }
    },
    "upsert" : {
        "counter" : 1
    }
}

scripted_upsert

If you would like your script to run regardless of whether the document exists or not — i.e. the script handles initializing the document instead of the upsert element — then set scripted_upsert to true:

POST sessions/session/dh3sgudg8gsrgl/_update
{
    "scripted_upsert":true,
    "script" : {
        "id": "my_web_session_summariser",
        "params" : {
            "pageViewEvent" : {
                "url":"foo.com/bar",
                "response":404,
                "time":"2014-01-01 12:32"
            }
        }
    },
    "upsert" : {}
}

doc_as_upsert

Instead of sending a partial doc plus an upsert doc, setting doc_as_upsert to true will use the contents of doc as the upsert value:

POST test/_doc/1/_update
{
    "doc" : {
        "name" : "new_name"
    },
    "doc_as_upsert" : true
}

Parameters

The update operation supports the following query-string parameters:

retry_on_conflict

In between the get and indexing phases of the update, it is possible that another process might have already updated the same document. By default, the update will fail with a version conflict exception. The retry_on_conflict parameter controls how many times to retry the update before finally throwing an exception.

routing

Routing is used to route the update request to the right shard and sets the routing for the upsert request if the document being updated doesn’t exist. Can’t be used to update the routing of an existing document.

timeout

Timeout waiting for a shard to become available.

wait_for_active_shards

The number of shard copies required to be active before proceeding with the update operation. See here for details.

refresh

Control when the changes made by this request are visible to search. See refresh.

_source

Allows to control if and how the updated source should be returned in the response. By default the updated source is not returned. See Source filtering for details.

version

The update API uses the Elasticsearch versioning support internally to make sure the document doesn’t change during the update. You can use the version parameter to specify that the document should only be updated if its version matches the one specified. deprecated:[6.7.0, "Please use if_seq_no & if_primary_term instead. See Optimistic concurrency control for more details."]

Note
The update API does not support versioning other than internal

External (version types external and external_gte) or forced (version type force) versioning is not supported by the update API as it would result in Elasticsearch version numbers being out of sync with the external system. Use the index API instead.

if_seq_no and if_primary_term

Update operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters. If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409. See Optimistic concurrency control for more details.

Update By Query API

The simplest usage of _update_by_query just performs an update on every document in the index without changing the source. This is useful to pick up a new property or some other online mapping change. Here is the API:

POST twitter/_update_by_query?conflicts=proceed

That will return something like this:

{
  "took" : 147,
  "timed_out": false,
  "updated": 120,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "total": 120,
  "failures" : [ ]
}

_update_by_query gets a snapshot of the index when it starts and indexes what it finds using internal versioning. That means you’ll get a version conflict if the document changes between the time when the snapshot was taken and when the index request is processed. When the versions match, the document is updated and the version number is incremented.

Note
Since internal versioning does not support the value 0 as a valid version number, documents with version equal to zero cannot be updated using _update_by_query and will fail the request.

All update and query failures cause the _update_by_query to abort and are returned in the failures of the response. The updates that have been performed still stick. In other words, the process is not rolled back, only aborted. While the first failure causes the abort, all failures that are returned by the failing bulk request are returned in the failures element; therefore it’s possible for there to be quite a few failed entities.

If you want to simply count version conflicts, and not cause the _update_by_query to abort, you can set conflicts=proceed on the url or "conflicts": "proceed" in the request body. The first example does this because it is just trying to pick up an online mapping change, and a version conflict simply means that the conflicting document was updated between the start of the _update_by_query and the time when it attempted to update the document. This is fine because that update will have picked up the online mapping update.

Back to the API format, this will update tweets from the twitter index:

POST twitter/_doc/_update_by_query?conflicts=proceed

You can also limit _update_by_query using the Query DSL. This will update all documents from the twitter index for the user kimchy:

POST twitter/_update_by_query?conflicts=proceed
{
  "query": { (1)
    "term": {
      "user": "kimchy"
    }
  }
}
  1. The query must be passed as a value to the query key, in the same way as the Search API. You can also use the q parameter in the same way as the search API.

So far we’ve only been updating documents without changing their source. That is genuinely useful for things like picking up new properties but it’s only half the fun. _update_by_query supports scripts to update the document. This will increment the likes field on all of kimchy’s tweets:

POST twitter/_update_by_query
{
  "script": {
    "source": "ctx._source.likes++",
    "lang": "painless"
  },
  "query": {
    "term": {
      "user": "kimchy"
    }
  }
}

Just as in Update API you can set ctx.op to change the operation that is executed:

noop

Set ctx.op = "noop" if your script decides that it doesn’t have to make any changes. That will cause _update_by_query to omit that document from its updates. This no operation will be reported in the noop counter in the response body.

delete

Set ctx.op = "delete" if your script decides that the document must be deleted. The deletion will be reported in the deleted counter in the response body.

Setting ctx.op to anything else is an error. Setting any other field in ctx is an error.

Note that we stopped specifying conflicts=proceed. In this case we want a version conflict to abort the process so we can handle the failure.

This API doesn’t allow you to move the documents it touches, just modify their source. This is intentional! We’ve made no provisions for removing the document from its original location.

It’s also possible to do this whole thing on multiple indexes and multiple types at once, just like the search API:

POST twitter,blog/_doc,post/_update_by_query

If you provide routing then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

POST twitter/_update_by_query?routing=1

By default _update_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

POST twitter/_update_by_query?scroll_size=100

_update_by_query can also use the Ingest Node feature by specifying a pipeline like this:

PUT _ingest/pipeline/set-foo
{
  "description" : "sets foo",
  "processors" : [ {
      "set" : {
        "field": "foo",
        "value": "bar"
      }
  } ]
}
POST twitter/_update_by_query?pipeline=set-foo

URL Parameters

In addition to the standard parameters like pretty, the Update By Query API also supports refresh, wait_for_completion, wait_for_active_shards, timeout, and scroll.

Sending the refresh will update all shards in the index being updated when the request completes. This is different than the Update API’s refresh parameter, which causes just the shard that received the new data to be indexed. Also unlike the Update API it does not support wait_for.

If the request contains wait_for_completion=false then Elasticsearch will perform some preflight checks, launch the request, and then return a task which can be used with Tasks APIs to cancel or get the status of the task. Elasticsearch will also create a record of this task as a document at .tasks/task/${taskId}. This is yours to keep or remove as you see fit. When you are done with it, delete it so Elasticsearch can reclaim the space it uses.

wait_for_active_shards controls how many copies of a shard must be active before proceeding with the request. See here for details. timeout controls how long each write request waits for unavailable shards to become available. Both work exactly how they work in the Bulk API. Because _update_by_query uses scroll search, you can also specify the scroll parameter to control how long it keeps the "search context" alive, e.g. ?scroll=10m. By default it’s 5 minutes.

requests_per_second can be set to any positive decimal number (1.4, 6, 1000, etc.) and throttles the rate at which _update_by_query issues batches of index operations by padding each batch with a wait time. The throttling can be disabled by setting requests_per_second to -1.

The throttling is done by waiting between batches so that scroll that _update_by_query uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing. By default the batch size is 1000, so if the requests_per_second is set to 500:

target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds

Since the batch is issued as a single _bulk request, large batch sizes will cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth". The default is -1.

Response body

The JSON response looks like this:

{
  "took" : 147,
  "timed_out": false,
  "total": 5,
  "updated": 5,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "failures" : [ ]
}
took

The number of milliseconds from start to end of the whole operation.

timed_out

This flag is set to true if any of the requests executed during the update by query execution has timed out.

total

The number of documents that were successfully processed.

updated

The number of documents that were successfully updated.

deleted

The number of documents that were successfully deleted.

batches

The number of scroll responses pulled back by the update by query.

version_conflicts

The number of version conflicts that the update by query hit.

noops

The number of documents that were ignored because the script used for the update by query returned a noop value for ctx.op.

retries

The number of retries attempted by update by query. bulk is the number of bulk actions retried, and search is the number of search actions retried.

throttled_millis

Number of milliseconds the request slept to conform to requests_per_second.

requests_per_second

The number of requests per second effectively executed during the update by query.

throttled_until_millis

This field should always be equal to zero in an _update_by_query response. It only has meaning when using the Task API, where it indicates the next time (in milliseconds since epoch) a throttled request will be executed again in order to conform to requests_per_second.

failures

Array of failures if there were any unrecoverable errors during the process. If this is non-empty then the request aborted because of those failures. Update by query is implemented using batches. Any failure causes the entire process to abort, but all failures in the current batch are collected into the array. You can use the conflicts option to prevent reindex from aborting on version conflicts.

Works with the Task API

You can fetch the status of all running update by query requests with the Task API:

GET _tasks?detailed=true&actions=*byquery

The responses looks like:

{
  "nodes" : {
    "r1A2WoRbTwKZ516z6NEs5A" : {
      "name" : "r1A2WoR",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "attributes" : {
        "testattr" : "test",
        "portsfile" : "true"
      },
      "tasks" : {
        "r1A2WoRbTwKZ516z6NEs5A:36619" : {
          "node" : "r1A2WoRbTwKZ516z6NEs5A",
          "id" : 36619,
          "type" : "transport",
          "action" : "indices:data/write/update/byquery",
          "status" : {    (1)
            "total" : 6154,
            "updated" : 3500,
            "created" : 0,
            "deleted" : 0,
            "batches" : 4,
            "version_conflicts" : 0,
            "noops" : 0,
            "retries": {
              "bulk": 0,
              "search": 0
            },
            "throttled_millis": 0
          },
          "description" : ""
        }
      }
    }
  }
}
  1. This object contains the actual status. It is just like the response JSON with the important addition of the total field. total is the total number of operations that the reindex expects to perform. You can estimate the progress by adding the updated, created, and deleted fields. The request will finish when their sum is equal to the total field.

With the task id you can look up the task directly. The following example retrieves information about task r1A2WoRbTwKZ516z6NEs5A:36619:

GET /_tasks/r1A2WoRbTwKZ516z6NEs5A:36619

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set on it, then it’ll come back with a results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

Works with the Cancel Task API

Any update by query can be cancelled using the Task Cancel API:

POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel

The task ID can be found using the tasks API.

Cancellation should happen quickly but might take a few seconds. The task status API above will continue to list the update by query task until this task checks that it has been cancelled and terminates itself.

Rethrottling

The value of requests_per_second can be changed on a running update by query using the _rethrottle API:

POST _update_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1

The task ID can be found using the tasks API.

Just like when setting it on the _update_by_query API, requests_per_second can be either -1 to disable throttling or any decimal number like 1.7 or 12 to throttle to that level. Rethrottling that speeds up the query takes effect immediately, but rethrotting that slows down the query will take effect after completing the current batch. This prevents scroll timeouts.

Slicing

Update by query supports Sliced Scroll to parallelize the updating process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Manual slicing

Slice an update by query manually by providing a slice id and total number of slices to each request:

POST twitter/_update_by_query
{
  "slice": {
    "id": 0,
    "max": 2
  },
  "script": {
    "source": "ctx._source['extra'] = 'test'"
  }
}
POST twitter/_update_by_query
{
  "slice": {
    "id": 1,
    "max": 2
  },
  "script": {
    "source": "ctx._source['extra'] = 'test'"
  }
}

Which you can verify works with:

GET _refresh
POST twitter/_search?size=0&q=extra:test&filter_path=hits.total

Which results in a sensible total like this one:

{
  "hits": {
    "total": 120
  }
}

Automatic slicing

You can also let update by query automatically parallelize using Sliced Scroll to slice on _uid. Use slices to specify the number of slices to use:

POST twitter/_update_by_query?refresh&slices=5
{
  "script": {
    "source": "ctx._source['extra'] = 'test'"
  }
}

Which you also can verify works with:

POST twitter/_search?size=0&q=extra:test&filter_path=hits.total

Which results in a sensible total like this one:

{
  "hits": {
    "total": 120
  }
}

Setting slices to auto will let Elasticsearch choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source indices, it will choose the number of slices based on the index with the smallest number of shards.

Adding slices to _update_by_query just automates the manual process used in the section above, creating sub-requests which means it has some quirks:

  • You can see these requests in the Tasks APIs. These sub-requests are "child" tasks of the task for the request with slices.

  • Fetching the status of the task for the request with slices only contains the status of completed slices.

  • These sub-requests are individually addressable for things like cancellation and rethrottling.

  • Rethrottling the request with slices will rethrottle the unfinished sub-request proportionally.

  • Canceling the request with slices will cancel each sub-request.

  • Due to the nature of slices each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.

  • Parameters like requests_per_second and size on a request with slices are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that the using size with slices might not result in exactly size documents being updated.

  • Each sub-request gets a slightly different snapshot of the source index though these are all taken at approximately the same time.

Picking the number of slices

If slicing automatically, setting slices to auto will choose a reasonable number for most indices. If you’re slicing manually or otherwise tuning automatic slicing, use these guidelines.

Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large, (for example, 500) choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.

Update performance scales linearly across available resources with the number of slices.

Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.

Pick up a new property

Say you created an index without dynamic mapping, filled it with data, and then added a mapping value to pick up more fields from the data:

PUT test
{
  "mappings": {
    "_doc": {
      "dynamic": false,   (1)
      "properties": {
        "text": {"type": "text"}
      }
    }
  }
}

POST test/_doc?refresh
{
  "text": "words words",
  "flag": "bar"
}
POST test/_doc?refresh
{
  "text": "words words",
  "flag": "foo"
}
PUT test/_mapping/_doc   (2)
{
  "properties": {
    "text": {"type": "text"},
    "flag": {"type": "text", "analyzer": "keyword"}
  }
}
  1. This means that new fields won’t be indexed, just stored in _source.

  2. This updates the mapping to add the new flag field. To pick up the new field you have to reindex all documents with it.

Searching for the data won’t find anything:

POST test/_search?filter_path=hits.total
{
  "query": {
    "match": {
      "flag": "foo"
    }
  }
}
{
  "hits" : {
    "total" : 0
  }
}

But you can issue an _update_by_query request to pick up the new mapping:

POST test/_update_by_query?refresh&conflicts=proceed
POST test/_search?filter_path=hits.total
{
  "query": {
    "match": {
      "flag": "foo"
    }
  }
}
{
  "hits" : {
    "total" : 1
  }
}

You can do the exact same thing when adding a field to a multifield.

Multi Get API

The Multi get API returns multiple documents based on an index, type, (optional) and id (and possibly routing). The response includes a docs array with all the fetched documents in order corresponding to the original multi-get request (if there was a failure for a specific get, an object containing this error is included in place in the response instead). The structure of a successful get is similar in structure to a document provided by the get API.

Here is an example:

GET /_mget
{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1"
        },
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "2"
        }
    ]
}

The mget endpoint can also be used against an index (in which case it is not required in the body):

GET /test/_mget
{
    "docs" : [
        {
            "_type" : "_doc",
            "_id" : "1"
        },
        {
            "_type" : "_doc",
            "_id" : "2"
        }
    ]
}

And type:

GET /test/_doc/_mget
{
    "docs" : [
        {
            "_id" : "1"
        },
        {
            "_id" : "2"
        }
    ]
}

In which case, the ids element can directly be used to simplify the request:

GET /test/_doc/_mget
{
    "ids" : ["1", "2"]
}

Source filtering

By default, the _source field will be returned for every document (if stored). Similar to the get API, you can retrieve only parts of the _source (or not at all) by using the _source parameter. You can also use the url parameters _source, _source_includes, and _source_excludes to specify defaults, which will be used when there are no per-document instructions.

For example:

GET /_mget
{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_source" : false
        },
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "2",
            "_source" : ["field3", "field4"]
        },
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "3",
            "_source" : {
                "include": ["user"],
                "exclude": ["user.location"]
            }
        }
    ]
}

Fields

Specific stored fields can be specified to be retrieved per document to get, similar to the stored_fields parameter of the Get API. For example:

GET /_mget
{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "stored_fields" : ["field1", "field2"]
        },
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "2",
            "stored_fields" : ["field3", "field4"]
        }
    ]
}

Alternatively, you can specify the stored_fields parameter in the query string as a default to be applied to all documents.

GET /test/_doc/_mget?stored_fields=field1,field2
{
    "docs" : [
        {
            "_id" : "1" (1)
        },
        {
            "_id" : "2",
            "stored_fields" : ["field3", "field4"] (2)
        }
    ]
}
  1. Returns field1 and field2

  2. Returns field3 and field4

Routing

You can also specify a routing value as a parameter:

GET /_mget?routing=key1
{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "routing" : "key2"
        },
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "2"
        }
    ]
}

In this example, document test/_doc/2 will be fetched from the shard corresponding to routing key key1 but document test/_doc/1 will be fetched from the shard corresponding to routing key key2.

Security

Partial responses

To ensure fast responses, the multi get API will respond with partial results if one or more shards fail. See Shard failures for more information.

Bulk API

The bulk API makes it possible to perform many index/delete operations in a single API call. This can greatly increase the indexing speed.

Client support for bulk requests

Some of the officially supported clients provide helpers to assist with bulk requests and reindexing of documents from one index to another:

The REST API endpoint is /_bulk, and it expects the following newline delimited JSON (NDJSON) structure:

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n

NOTE: The final line of data must end with a newline character \n. Each newline character may be preceded by a carriage return \r. When sending requests to this endpoint the Content-Type header should be set to application/x-ndjson.

The possible actions are index, create, delete, and update. index and create expect a source on the next line, and have the same semantics as the op_type parameter to the standard index API (i.e. create will fail if a document with the same index and type exists already, whereas index will add or replace a document as necessary). delete does not expect a source on the following line, and has the same semantics as the standard delete API. update expects that the partial doc, upsert and script and its options are specified on the next line.

If you’re providing text file input to curl, you must use the --data-binary flag instead of plain -d. The latter doesn’t preserve newlines. Example:

$ cat requests
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7, "errors": false, "items":[{"index":{"_index":"test","_type":"_doc","_id":"1","_version":1,"result":"created","forced_refresh":false}}]}

Because this format uses literal `\n’s as delimiters, please be sure that the JSON actions and sources are not pretty printed. Here is an example of a correct sequence of bulk commands:

POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "_doc", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "_doc", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "_doc", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

The result of this bulk operation is:

{
   "took": 30,
   "errors": false,
   "items": [
      {
         "index": {
            "_index": "test",
            "_type": "_doc",
            "_id": "1",
            "_version": 1,
            "result": "created",
            "_shards": {
               "total": 2,
               "successful": 1,
               "failed": 0
            },
            "status": 201,
            "_seq_no" : 0,
            "_primary_term": 1
         }
      },
      {
         "delete": {
            "_index": "test",
            "_type": "_doc",
            "_id": "2",
            "_version": 1,
            "result": "not_found",
            "_shards": {
               "total": 2,
               "successful": 1,
               "failed": 0
            },
            "status": 404,
            "_seq_no" : 1,
            "_primary_term" : 2
         }
      },
      {
         "create": {
            "_index": "test",
            "_type": "_doc",
            "_id": "3",
            "_version": 1,
            "result": "created",
            "_shards": {
               "total": 2,
               "successful": 1,
               "failed": 0
            },
            "status": 201,
            "_seq_no" : 2,
            "_primary_term" : 3
         }
      },
      {
         "update": {
            "_index": "test",
            "_type": "_doc",
            "_id": "1",
            "_version": 2,
            "result": "updated",
            "_shards": {
                "total": 2,
                "successful": 1,
                "failed": 0
            },
            "status": 200,
            "_seq_no" : 3,
            "_primary_term" : 4
         }
      }
   ]
}

The endpoints are /_bulk, /{index}/_bulk, and {index}/{type}/_bulk. When the index or the index/type are provided, they will be used by default on bulk items that don’t provide them explicitly.

A note on the format. The idea here is to make processing of this as fast as possible. As some of the actions will be redirected to other shards on other nodes, only action_meta_data is parsed on the receiving node side.

Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.

The response to a bulk action is a large JSON structure with the individual results of each action that was performed in the same order as the actions that appeared in the request. The failure of a single action does not affect the remaining actions.

There is no "correct" number of actions to perform in a single bulk call. You should experiment with different settings to find the optimum size for your particular workload.

If using the HTTP API, make sure that the client does not send HTTP chunks, as this will slow things down.

Optimistic Concurrency Control

Each index and delete action within a bulk API call may include the if_seq_no and if_primary_term parameters in their respective action and meta data lines. The if_seq_no and if_primary_term parameters control how operations are executed, based on the last modification to existing documents. See Optimistic concurrency control for more details.

Versioning

Each bulk item can include the version value using the version field. It automatically follows the behavior of the index / delete operation based on the _version mapping. It also support the version_type (see versioning).

Routing

Each bulk item can include the routing value using the routing field. It automatically follows the behavior of the index / delete operation based on the _routing mapping.

Wait For Active Shards

When making bulk calls, you can set the wait_for_active_shards parameter to require a minimum number of shard copies to be active before starting to process the bulk request. See here for further details and a usage example.

Refresh

Control when the changes made by this request are visible to search. See refresh.

Note
Only the shards that receive the bulk request will be affected by refresh. Imagine a _bulk?refresh=wait_for request with three documents in it that happen to be routed to different shards in an index with five shards. The request will only wait for those three shards to refresh. The other two shards that make up the index do not participate in the _bulk request at all.

Update

When using the update action, retry_on_conflict can be used as a field in the action itself (not in the extra payload line), to specify how many times an update should be retried in the case of a version conflict.

The update action payload supports the following options: doc (partial document), upsert, doc_as_upsert, script, params (for script), lang (for script), and _source. See update documentation for details on the options. Example with update actions:

POST _bulk
{ "update" : {"_id" : "1", "_type" : "_doc", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_type" : "_doc", "_index" : "index1", "retry_on_conflict" : 3} }
{ "script" : { "source": "ctx._source.counter += params.param1", "lang" : "painless", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_type" : "_doc", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }
{ "update" : {"_id" : "3", "_type" : "_doc", "_index" : "index1", "_source" : true} }
{ "doc" : {"field" : "value"} }
{ "update" : {"_id" : "4", "_type" : "_doc", "_index" : "index1"} }
{ "doc" : {"field" : "value"}, "_source": true}

Security

Partial responses

To ensure fast responses, the multi search API will respond with partial results if one or more shards fail. See Shard failures for more information.

Reindex API

Important
Reindex requires _source to be enabled for all documents in the source index.
Important
Reindex does not attempt to set up the destination index. It does not copy the settings of the source index. You should set up the destination index prior to running a _reindex action, including setting up mappings, shard counts, replicas, etc.

The most basic form of _reindex just copies documents from one index to another. This will copy documents from the twitter index into the new_twitter index:

POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}

That will return something like this:

{
  "took" : 147,
  "timed_out": false,
  "created": 120,
  "updated": 0,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1.0,
  "throttled_until_millis": 0,
  "total": 120,
  "failures" : [ ]
}

Just like _update_by_query, _reindex gets a snapshot of the source index but its target must be a different index so version conflicts are unlikely. The dest element can be configured like the index API to control optimistic concurrency control. Just leaving out version_type (as above) or setting it to internal will cause Elasticsearch to blindly dump documents into the target, overwriting any that happen to have the same type and id:

POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "internal"
  }
}

Setting version_type to external will cause Elasticsearch to preserve the version from the source, create any documents that are missing, and update any documents that have an older version in the destination index than they do in the source index:

POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "external"
  }
}

Settings op_type to create will cause _reindex to only create missing documents in the target index. All existing documents will cause a version conflict:

POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "op_type": "create"
  }
}

By default, version conflicts abort the _reindex process. The "conflicts" request body parameter can be used to instruct _reindex to proceed with the next document on version conflicts. It is important to note that the handling of other error types is unaffected by the "conflicts" parameter. When "conflicts": "proceed" is set in the request body, the _reindex process will continue on version conflicts and return a count of version conflicts encountered:

POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "op_type": "create"
  }
}

You can limit the documents by adding a type to the source or by adding a query. This will only copy tweets made by kimchy into new_twitter:

POST _reindex
{
  "source": {
    "index": "twitter",
    "type": "_doc",
    "query": {
      "term": {
        "user": "kimchy"
      }
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}

index and type in source can both be lists, allowing you to copy from lots of sources in one request. This will copy documents from the _doc and post types in the twitter and blog indices.

POST _reindex
{
  "source": {
    "index": ["twitter", "blog"],
    "type": ["_doc", "post"]
  },
  "dest": {
    "index": "all_together",
    "type": "_doc"
  }
}
Note
The Reindex API makes no effort to handle ID collisions so the last document written will "win" but the order isn’t usually predictable so it is not a good idea to rely on this behavior. Instead, make sure that IDs are unique using a script.

It’s also possible to limit the number of processed documents by setting size. This will only copy a single document from twitter to new_twitter:

POST _reindex
{
  "size": 1,
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}

If you want a particular set of documents from the twitter index you’ll need to use sort. Sorting makes the scroll less efficient but in some contexts it’s worth it. If possible, prefer a more selective query to size and sort. This will copy 10000 documents from twitter into new_twitter:

POST _reindex
{
  "size": 10000,
  "source": {
    "index": "twitter",
    "sort": { "date": "desc" }
  },
  "dest": {
    "index": "new_twitter"
  }
}

The source section supports all the elements that are supported in a search request. For instance, only a subset of the fields from the original documents can be reindexed using source filtering as follows:

POST _reindex
{
  "source": {
    "index": "twitter",
    "_source": ["user", "_doc"]
  },
  "dest": {
    "index": "new_twitter"
  }
}

Like _update_by_query, _reindex supports a script that modifies the document. Unlike _update_by_query, the script is allowed to modify the document’s metadata. This example bumps the version of the source document:

POST _reindex
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter",
    "version_type": "external"
  },
  "script": {
    "source": "if (ctx._source.foo == 'bar') {ctx._version++; ctx._source.remove('foo')}",
    "lang": "painless"
  }
}

Just as in _update_by_query, you can set ctx.op to change the operation that is executed on the destination index:

noop

Set ctx.op = "noop" if your script decides that the document doesn’t have to be indexed in the destination index. This no operation will be reported in the noop counter in the response body.

delete

Set ctx.op = "delete" if your script decides that the document must be deleted from the destination index. The deletion will be reported in the deleted counter in the response body.

Setting ctx.op to anything else will return an error, as will setting any other field in ctx.

Think of the possibilities! Just be careful; you are able to change:

  • _id

  • _type

  • _index

  • _version

  • _routing

Setting _version to null or clearing it from the ctx map is just like not sending the version in an indexing request; it will cause the document to be overwritten in the target index regardless of the version on the target or the version type you use in the _reindex request.

By default if _reindex sees a document with routing then the routing is preserved unless it’s changed by the script. You can set routing on the dest request to change this:

keep

Sets the routing on the bulk request sent for each match to the routing on the match. This is the default value.

discard

Sets the routing on the bulk request sent for each match to null.

=<some text>

Sets the routing on the bulk request sent for each match to all text after the =.

For example, you can use the following request to copy all documents from the source index with the company name cat into the dest index with routing set to cat.

POST _reindex
{
  "source": {
    "index": "source",
    "query": {
      "match": {
        "company": "cat"
      }
    }
  },
  "dest": {
    "index": "dest",
    "routing": "=cat"
  }
}

By default _reindex uses scroll batches of 1000. You can change the batch size with the size field in the source element:

POST _reindex
{
  "source": {
    "index": "source",
    "size": 100
  },
  "dest": {
    "index": "dest",
    "routing": "=cat"
  }
}

Reindex can also use the Ingest Node feature by specifying a pipeline like this:

POST _reindex
{
  "source": {
    "index": "source"
  },
  "dest": {
    "index": "dest",
    "pipeline": "some_ingest_pipeline"
  }
}

Reindex from Remote

Reindex supports reindexing from a remote Elasticsearch cluster:

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "username": "user",
      "password": "pass"
    },
    "index": "source",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}

The host parameter must contain a scheme, host, port (e.g. https://otherhost:9200), and optional path (e.g. https://otherhost:9200/proxy). The username and password parameters are optional, and when they are present _reindex will connect to the remote Elasticsearch node using basic auth. Be sure to use https when using basic auth or the password will be sent in plain text. There are a range of settings available to configure the behaviour of the https connection.

Remote hosts have to be explicitly whitelisted in elasticsearch.yaml using the reindex.remote.whitelist property. It can be set to a comma delimited list of allowed remote host and port combinations (e.g. otherhost:9200, another:9200, 127.0.10.:9200, localhost:). Scheme is ignored by the whitelist — only host and port are used, for example:

reindex.remote.whitelist: "otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*"

The whitelist must be configured on any nodes that will coordinate the reindex.

This feature should work with remote clusters of any version of Elasticsearch you are likely to find. This should allow you to upgrade from any version of Elasticsearch to the current version by reindexing from a cluster of the old version.

Warning
{es} does not support forward compatibility across major versions. For example, you cannot reindex from a 7.x cluster into a 6.x cluster.

To enable queries sent to older versions of Elasticsearch the query parameter is sent directly to the remote host without validation or modification.

Note
Reindexing from remote clusters does not support manual or automatic slicing.

Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb. If the remote index includes very large documents you’ll need to use a smaller batch size. The example below sets the batch size to 10 which is very, very small.

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200"
    },
    "index": "source",
    "size": 10,
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}

It is also possible to set the socket read timeout on the remote connection with the socket_timeout field and the connection timeout with the connect_timeout field. Both default to 30 seconds. This example sets the socket read timeout to one minute and the connection timeout to 10 seconds:

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "socket_timeout": "1m",
      "connect_timeout": "10s"
    },
    "index": "source",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}

Configuring SSL parameters

Reindex from remote supports configurable SSL settings. These must be specified in the elasticsearch.yml file, with the exception of the secure settings, which you add in the Elasticsearch keystore. It is not possible to configure SSL in the body of the _reindex request.

The following settings are supported:

reindex.ssl.certificate_authorities

List of paths to PEM encoded certificate files that should be trusted. You cannot specify both reindex.ssl.certificate_authorities and reindex.ssl.truststore.path.

reindex.ssl.truststore.path

The path to the Java Keystore file that contains the certificates to trust. This keystore can be in "JKS" or "PKCS#12" format. You cannot specify both reindex.ssl.certificate_authorities and reindex.ssl.truststore.path.

reindex.ssl.truststore.password

The password to the truststore (reindex.ssl.truststore.path). This setting cannot be used with reindex.ssl.truststore.secure_password.

reindex.ssl.truststore.secure_password (Secure)

The password to the truststore (reindex.ssl.truststore.path). This setting cannot be used with reindex.ssl.truststore.password.

reindex.ssl.truststore.type

The type of the truststore (reindex.ssl.truststore.path). Must be either jks or PKCS12. If the truststore path ends in ".p12", ".pfx" or "pkcs12", this setting defaults to PKCS12. Otherwise, it defaults to jks.

reindex.ssl.verification_mode

Indicates the type of verification to protect against man in the middle attacks and certificate forgery. One of full (verify the hostname and the certificate path), certificate (verify the certificate path, but not the hostname) or none (perform no verification - this is strongly discouraged in production environments). Defaults to full.

reindex.ssl.certificate

Specifies the path to the PEM encoded certificate (or certificate chain) to be used for HTTP client authentication (if required by the remote cluster) This setting requires that reindex.ssl.key also be set. You cannot specify both reindex.ssl.certificate and reindex.ssl.keystore.path.

reindex.ssl.key

Specifies the path to the PEM encoded private key associated with the certificate used for client authentication (reindex.ssl.certificate). You cannot specify both reindex.ssl.key and reindex.ssl.keystore.path.

reindex.ssl.key_passphrase

Specifies the passphrase to decrypt the PEM encoded private key (reindex.ssl.key) if it is encrypted. Cannot be used with reindex.ssl.secure_key_passphrase.

reindex.ssl.secure_key_passphrase (Secure)

Specifies the passphrase to decrypt the PEM encoded private key (reindex.ssl.key) if it is encrypted. Cannot be used with reindex.ssl.key_passphrase.

reindex.ssl.keystore.path

Specifies the path to the keystore that contains a private key and certificate to be used for HTTP client authentication (if required by the remote cluster). This keystore can be in "JKS" or "PKCS#12" format. You cannot specify both reindex.ssl.key and reindex.ssl.keystore.path.

reindex.ssl.keystore.type

The type of the keystore (reindex.ssl.keystore.path). Must be either jks or PKCS12. If the keystore path ends in ".p12", ".pfx" or "pkcs12", this setting defaults to PKCS12. Otherwise, it defaults to jks.

reindex.ssl.keystore.password

The password to the keystore (reindex.ssl.keystore.path). This setting cannot be used with reindex.ssl.keystore.secure_password.

reindex.ssl.keystore.secure_password (Secure)

The password to the keystore (reindex.ssl.keystore.path). This setting cannot be used with reindex.ssl.keystore.password.

reindex.ssl.keystore.key_password

The password for the key in the keystore (reindex.ssl.keystore.path). Defaults to the keystore password. This setting cannot be used with reindex.ssl.keystore.secure_key_password.

reindex.ssl.keystore.secure_key_password (Secure)

The password for the key in the keystore (reindex.ssl.keystore.path). Defaults to the keystore password. This setting cannot be used with reindex.ssl.keystore.key_password.

URL Parameters

In addition to the standard parameters like pretty, the Reindex API also supports refresh, wait_for_completion, wait_for_active_shards, timeout, scroll, and requests_per_second.

Sending the refresh url parameter will cause all indexes to which the request wrote to be refreshed. This is different than the Index API’s refresh parameter which causes just the shard that received the new data to be refreshed. Also unlike the Index API it does not support wait_for.

If the request contains wait_for_completion=false then Elasticsearch will perform some preflight checks, launch the request, and then return a task which can be used with Tasks APIs to cancel or get the status of the task. Elasticsearch will also create a record of this task as a document at .tasks/task/${taskId}. This is yours to keep or remove as you see fit. When you are done with it, delete it so Elasticsearch can reclaim the space it uses.

wait_for_active_shards controls how many copies of a shard must be active before proceeding with the reindexing. See here for details. timeout controls how long each write request waits for unavailable shards to become available. Both work exactly how they work in the Bulk API. As _reindex uses scroll search, you can also specify the scroll parameter to control how long it keeps the "search context" alive, (e.g. ?scroll=10m). The default value is 5 minutes.

requests_per_second can be set to any positive decimal number (1.4, 6, 1000, etc.) and throttles the rate at which _reindex issues batches of index operations by padding each batch with a wait time. The throttling can be disabled by setting requests_per_second to -1.

The throttling is done by waiting between batches so that the scroll which _reindex uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing. By default the batch size is 1000, so if the requests_per_second is set to 500:

target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds

Since the batch is issued as a single _bulk request, large batch sizes will cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth". The default value is -1.

Response body

The JSON response looks like this:

{
  "took": 639,
  "timed_out": false,
  "total": 5,
  "updated": 0,
  "created": 5,
  "deleted": 0,
  "batches": 1,
  "noops": 0,
  "version_conflicts": 2,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": 1,
  "throttled_until_millis": 0,
  "failures": [ ]
}
took

The total milliseconds the entire operation took.

timed_out

This flag is set to true if any of the requests executed during the reindex timed out.

total

The number of documents that were successfully processed.

updated

The number of documents that were successfully updated.

created

The number of documents that were successfully created.

deleted

The number of documents that were successfully deleted.

batches

The number of scroll responses pulled back by the reindex.

noops

The number of documents that were ignored because the script used for the reindex returned a noop value for ctx.op.

version_conflicts

The number of version conflicts that reindex hit.

retries

The number of retries attempted by reindex. bulk is the number of bulk actions retried and search is the number of search actions retried.

throttled_millis

Number of milliseconds the request slept to conform to requests_per_second.

requests_per_second

The number of requests per second effectively executed during the reindex.

throttled_until_millis

This field should always be equal to zero in a _reindex response. It only has meaning when using the Task API, where it indicates the next time (in milliseconds since epoch) a throttled request will be executed again in order to conform to requests_per_second.

failures

Array of failures if there were any unrecoverable errors during the process. If this is non-empty then the request aborted because of those failures. Reindex is implemented using batches and any failure causes the entire process to abort but all failures in the current batch are collected into the array. You can use the conflicts option to prevent reindex from aborting on version conflicts.

Works with the Task API

You can fetch the status of all running reindex requests with the Task API:

GET _tasks?detailed=true&actions=*reindex

The response looks like:

{
  "nodes" : {
    "r1A2WoRbTwKZ516z6NEs5A" : {
      "name" : "r1A2WoR",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "attributes" : {
        "testattr" : "test",
        "portsfile" : "true"
      },
      "tasks" : {
        "r1A2WoRbTwKZ516z6NEs5A:36619" : {
          "node" : "r1A2WoRbTwKZ516z6NEs5A",
          "id" : 36619,
          "type" : "transport",
          "action" : "indices:data/write/reindex",
          "status" : {    (1)
            "total" : 6154,
            "updated" : 3500,
            "created" : 0,
            "deleted" : 0,
            "batches" : 4,
            "version_conflicts" : 0,
            "noops" : 0,
            "retries": {
              "bulk": 0,
              "search": 0
            },
            "throttled_millis": 0,
            "requests_per_second": -1,
            "throttled_until_millis": 0
          },
          "description" : "",
          "start_time_in_millis": 1535149899665,
          "running_time_in_nanos": 5926916792,
          "cancellable": true,
          "headers": {}
        }
      }
    }
  }
}
  1. This object contains the actual status. It is identical to the response JSON except for the important addition of the total field. total is the total number of operations that the _reindex expects to perform. You can estimate the progress by adding the updated, created, and deleted fields. The request will finish when their sum is equal to the total field.

With the task id you can look up the task directly. The following example retrieves information about the task r1A2WoRbTwKZ516z6NEs5A:36619:

GET /_tasks/r1A2WoRbTwKZ516z6NEs5A:36619

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set, it will return a results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

Works with the Cancel Task API

Any reindex can be canceled using the Task Cancel API. For example:

POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel

The task ID can be found using the Tasks API.

Cancelation should happen quickly but might take a few seconds. The Tasks API will continue to list the task until it wakes to cancel itself.

Rethrottling

The value of requests_per_second can be changed on a running reindex using the _rethrottle API:

POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1

The task ID can be found using the tasks API.

Just like when setting it on the Reindex API, requests_per_second can be either -1 to disable throttling or any decimal number like 1.7 or 12 to throttle to that level. Rethrottling that speeds up the query takes effect immediately, but rethrottling that slows down the query will take effect after completing the current batch. This prevents scroll timeouts.

Reindex to change the name of a field

_reindex can be used to build a copy of an index with renamed fields. Say you create an index containing documents that look like this:

POST test/_doc/1?refresh
{
  "text": "words words",
  "flag": "foo"
}

but you don’t like the name flag and want to replace it with tag. _reindex can create the other index for you:

POST _reindex
{
  "source": {
    "index": "test"
  },
  "dest": {
    "index": "test2"
  },
  "script": {
    "source": "ctx._source.tag = ctx._source.remove(\"flag\")"
  }
}

Now you can get the new document:

GET test2/_doc/1

which will return:

{
  "found": true,
  "_id": "1",
  "_index": "test2",
  "_type": "_doc",
  "_version": 1,
  "_seq_no": 44,
  "_primary_term": 1,
  "_source": {
    "text": "words words",
    "tag": "foo"
  }
}

Slicing

Reindex supports Sliced Scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Manual slicing

Slice a reindex request manually by providing a slice id and total number of slices to each request:

POST _reindex
{
  "source": {
    "index": "twitter",
    "slice": {
      "id": 0,
      "max": 2
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}
POST _reindex
{
  "source": {
    "index": "twitter",
    "slice": {
      "id": 1,
      "max": 2
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}

You can verify this works by:

GET _refresh
POST new_twitter/_search?size=0&filter_path=hits.total

which results in a sensible total like this one:

{
  "hits": {
    "total": 120
  }
}

Automatic slicing

You can also let _reindex automatically parallelize using Sliced Scroll to slice on _uid. Use slices to specify the number of slices to use:

POST _reindex?slices=5&refresh
{
  "source": {
    "index": "twitter"
  },
  "dest": {
    "index": "new_twitter"
  }
}

You can also this verify works by:

POST new_twitter/_search?size=0&filter_path=hits.total

which results in a sensible total like this one:

{
  "hits": {
    "total": 120
  }
}

Setting slices to auto will let Elasticsearch choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source indices, it will choose the number of slices based on the index with the smallest number of shards.

Adding slices to _reindex just automates the manual process used in the section above, creating sub-requests which means it has some quirks:

  • You can see these requests in the Tasks APIs. These sub-requests are "child" tasks of the task for the request with slices.

  • Fetching the status of the task for the request with slices only contains the status of completed slices.

  • These sub-requests are individually addressable for things like cancelation and rethrottling.

  • Rethrottling the request with slices will rethrottle the unfinished sub-request proportionally.

  • Canceling the request with slices will cancel each sub-request.

  • Due to the nature of slices each sub-request won’t get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.

  • Parameters like requests_per_second and size on a request with slices are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that the using size with slices might not result in exactly size documents being reindexed.

  • Each sub-request gets a slightly different snapshot of the source index, though these are all taken at approximately the same time.

Picking the number of slices

If slicing automatically, setting slices to auto will choose a reasonable number for most indices. If slicing manually or otherwise tuning automatic slicing, use these guidelines.

Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large (e.g. 500), choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.

Indexing performance scales linearly across available resources with the number of slices.

Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources.

Reindexing many indices

If you have many indices to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up many indices. That way you can resume the process if there are any errors by removing the partially completed index and starting over at that index. It also makes parallelizing the process fairly simple: split the list of indices to reindex and run each list in parallel.

One-off bash scripts seem to work nicely for this:

for index in i1 i2 i3 i4 i5; do
  curl -HContent-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{
    "source": {
      "index": "'$index'"
    },
    "dest": {
      "index": "'$index'-reindexed"
    }
  }'
done

Reindex daily indices

Notwithstanding the above advice, you can use _reindex in combination with Painless to reindex daily indices to apply a new template to the existing documents.

Assuming you have indices consisting of documents as follows:

PUT metricbeat-2016.05.30/_doc/1?refresh
{"system.cpu.idle.pct": 0.908}
PUT metricbeat-2016.05.31/_doc/1?refresh
{"system.cpu.idle.pct": 0.105}

The new template for the metricbeat-* indices is already loaded into Elasticsearch, but it applies only to the newly created indices. Painless can be used to reindex the existing documents and apply the new template.

The script below extracts the date from the index name and creates a new index with -1 appended. All data from metricbeat-2016.05.31 will be reindexed into metricbeat-2016.05.31-1.

POST _reindex
{
  "source": {
    "index": "metricbeat-*"
  },
  "dest": {
    "index": "metricbeat"
  },
  "script": {
    "lang": "painless",
    "source": "ctx._index = 'metricbeat-' + (ctx._index.substring('metricbeat-'.length(), ctx._index.length())) + '-1'"
  }
}

All documents from the previous metricbeat indices can now be found in the *-1 indices.

GET metricbeat-2016.05.30-1/_doc/1
GET metricbeat-2016.05.31-1/_doc/1

The previous method can also be used in conjunction with changing a field name to load only the existing data into the new index and rename any fields if needed.

Extracting a random subset of an index

_reindex can be used to extract a random subset of an index for testing:

POST _reindex
{
  "size": 10,
  "source": {
    "index": "twitter",
    "query": {
      "function_score" : {
        "query" : { "match_all": {} },
        "random_score" : {}
      }
    },
    "sort": "_score"    (1)
  },
  "dest": {
    "index": "random_twitter"
  }
}
  1. _reindex defaults to sorting by _doc so random_score will not have any effect unless you override the sort to _score.

Term Vectors

Returns information and statistics on terms in the fields of a particular document. The document could be stored in the index or artificially provided by the user. Term vectors are realtime by default, not near realtime. This can be changed by setting the realtime parameter to false.

GET /twitter/_doc/1/_termvectors

Optionally, you can specify the fields for which the information is retrieved either with a parameter in the url

GET /twitter/_doc/1/_termvectors?fields=message

or by adding the requested fields in the request body (see example below). Fields can also be specified with wildcards in a similar way to the multi match query

Warning
Note that the usage of /_termvector is deprecated in 2.0, and replaced by /_termvectors.

Return values

Three types of values can be requested: term information, term statistics, and field statistics. By default, all term information and field statistics are returned for all fields but no term statistics.

Term information

  • term frequency in the field (always returned)

  • term positions (positions : true)

  • start and end offsets (offsets : true)

  • term payloads (payloads : true), as base64 encoded bytes

If the requested information wasn’t stored in the index, it will be computed on the fly if possible. Additionally, term vectors could be computed for documents not even existing in the index, but instead provided by the user.

Warning

Start and end offsets assume UTF-16 encoding is being used. If you want to use these offsets in order to get the original text that produced this token, you should make sure that the string you are taking a sub-string of is also encoded using UTF-16.

Term statistics

Setting term_statistics to true (default is false) will return

  • total term frequency (how often a term occurs in all documents)

  • document frequency (the number of documents containing the current term)

By default these values are not returned since term statistics can have a serious performance impact.

Field statistics

Setting field_statistics to false (default is true) will omit :

  • document count (how many documents contain this field)

  • sum of document frequencies (the sum of document frequencies for all terms in this field)

  • sum of total term frequencies (the sum of total term frequencies of each term in this field)

Terms Filtering

With the parameter filter, the terms returned could also be filtered based on their tf-idf scores. This could be useful in order find out a good characteristic vector of a document. This feature works in a similar manner to the second phase of the More Like This Query. See example 5 for usage.

The following sub-parameters are supported:

max_num_terms

Maximum number of terms that must be returned per field. Defaults to 25.

min_term_freq

Ignore words with less than this frequency in the source doc. Defaults to 1.

max_term_freq

Ignore words with more than this frequency in the source doc. Defaults to unbounded.

min_doc_freq

Ignore terms which do not occur in at least this many docs. Defaults to 1.

max_doc_freq

Ignore words which occur in more than this many docs. Defaults to unbounded.

min_word_length

The minimum word length below which words will be ignored. Defaults to 0.

max_word_length

The maximum word length above which words will be ignored. Defaults to unbounded (0).

Behaviour

The term and field statistics are not accurate. Deleted documents are not taken into account. The information is only retrieved for the shard the requested document resides in. The term and field statistics are therefore only useful as relative measures whereas the absolute numbers have no meaning in this context. By default, when requesting term vectors of artificial documents, a shard to get the statistics from is randomly selected. Use routing only to hit a particular shard.

Example: Returning stored term vectors

First, we create an index that stores term vectors, payloads, etc. :

PUT /twitter/
{ "mappings": {
    "_doc": {
      "properties": {
        "text": {
          "type": "text",
          "term_vector": "with_positions_offsets_payloads",
          "store" : true,
          "analyzer" : "fulltext_analyzer"
         },
         "fullname": {
          "type": "text",
          "term_vector": "with_positions_offsets_payloads",
          "analyzer" : "fulltext_analyzer"
        }
      }
    }
  },
  "settings" : {
    "index" : {
      "number_of_shards" : 1,
      "number_of_replicas" : 0
    },
    "analysis": {
      "analyzer": {
        "fulltext_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "type_as_payload"
          ]
        }
      }
    }
  }
}

Second, we add some documents:

PUT /twitter/_doc/1
{
  "fullname" : "John Doe",
  "text" : "twitter test test test "
}

PUT /twitter/_doc/2
{
  "fullname" : "Jane Doe",
  "text" : "Another twitter test ..."
}

The following request returns all information and statistics for field text in document 1 (John Doe):

GET /twitter/_doc/1/_termvectors
{
  "fields" : ["text"],
  "offsets" : true,
  "payloads" : true,
  "positions" : true,
  "term_statistics" : true,
  "field_statistics" : true
}

Response:

{
    "_id": "1",
    "_index": "twitter",
    "_type": "_doc",
    "_version": 1,
    "found": true,
    "took": 6,
    "term_vectors": {
        "text": {
            "field_statistics": {
                "doc_count": 2,
                "sum_doc_freq": 6,
                "sum_ttf": 8
            },
            "terms": {
                "test": {
                    "doc_freq": 2,
                    "term_freq": 3,
                    "tokens": [
                        {
                            "end_offset": 12,
                            "payload": "d29yZA==",
                            "position": 1,
                            "start_offset": 8
                        },
                        {
                            "end_offset": 17,
                            "payload": "d29yZA==",
                            "position": 2,
                            "start_offset": 13
                        },
                        {
                            "end_offset": 22,
                            "payload": "d29yZA==",
                            "position": 3,
                            "start_offset": 18
                        }
                    ],
                    "ttf": 4
                },
                "twitter": {
                    "doc_freq": 2,
                    "term_freq": 1,
                    "tokens": [
                        {
                            "end_offset": 7,
                            "payload": "d29yZA==",
                            "position": 0,
                            "start_offset": 0
                        }
                    ],
                    "ttf": 2
                }
            }
        }
    }
}

Example: Generating term vectors on the fly

Term vectors which are not explicitly stored in the index are automatically computed on the fly. The following request returns all information and statistics for the fields in document 1, even though the terms haven’t been explicitly stored in the index. Note that for the field text, the terms are not re-generated.

GET /twitter/_doc/1/_termvectors
{
  "fields" : ["text", "some_field_without_term_vectors"],
  "offsets" : true,
  "positions" : true,
  "term_statistics" : true,
  "field_statistics" : true
}

Example: Artificial documents

Term vectors can also be generated for artificial documents, that is for documents not present in the index. For example, the following request would return the same results as in example 1. The mapping used is determined by the index and type.

If dynamic mapping is turned on (default), the document fields not in the original mapping will be dynamically created.

GET /twitter/_doc/_termvectors
{
  "doc" : {
    "fullname" : "John Doe",
    "text" : "twitter test test test"
  }
}
Per-field analyzer

Additionally, a different analyzer than the one at the field may be provided by using the per_field_analyzer parameter. This is useful in order to generate term vectors in any fashion, especially when using artificial documents. When providing an analyzer for a field that already stores term vectors, the term vectors will be re-generated.

GET /twitter/_doc/_termvectors
{
  "doc" : {
    "fullname" : "John Doe",
    "text" : "twitter test test test"
  },
  "fields": ["fullname"],
  "per_field_analyzer" : {
    "fullname": "keyword"
  }
}

Response:

{
  "_index": "twitter",
  "_type": "_doc",
  "_version": 0,
  "found": true,
  "took": 6,
  "term_vectors": {
    "fullname": {
       "field_statistics": {
          "sum_doc_freq": 2,
          "doc_count": 4,
          "sum_ttf": 4
       },
       "terms": {
          "John Doe": {
             "term_freq": 1,
             "tokens": [
                {
                   "position": 0,
                   "start_offset": 0,
                   "end_offset": 8
                }
             ]
          }
       }
    }
  }
}

Example: Terms filtering

Finally, the terms returned could be filtered based on their tf-idf scores. In the example below we obtain the three most "interesting" keywords from the artificial document having the given "plot" field value. Notice that the keyword "Tony" or any stop words are not part of the response, as their tf-idf must be too low.

GET /imdb/_doc/_termvectors
{
    "doc": {
      "plot": "When wealthy industrialist Tony Stark is forced to build an armored suit after a life-threatening incident, he ultimately decides to use its technology to fight against evil."
    },
    "term_statistics" : true,
    "field_statistics" : true,
    "positions": false,
    "offsets": false,
    "filter" : {
      "max_num_terms" : 3,
      "min_term_freq" : 1,
      "min_doc_freq" : 1
    }
}

Response:

{
   "_index": "imdb",
   "_type": "_doc",
   "_version": 0,
   "found": true,
   "term_vectors": {
      "plot": {
         "field_statistics": {
            "sum_doc_freq": 3384269,
            "doc_count": 176214,
            "sum_ttf": 3753460
         },
         "terms": {
            "armored": {
               "doc_freq": 27,
               "ttf": 27,
               "term_freq": 1,
               "score": 9.74725
            },
            "industrialist": {
               "doc_freq": 88,
               "ttf": 88,
               "term_freq": 1,
               "score": 8.590818
            },
            "stark": {
               "doc_freq": 44,
               "ttf": 47,
               "term_freq": 1,
               "score": 9.272792
            }
         }
      }
   }
}

Multi termvectors API

The multi termvectors API allows to get multiple termvectors at once. The documents from which to retrieve the term vectors are specified by an index, type, and id. But the documents could also be artificially provided in the request itself.

The response includes a docs array with all the fetched termvectors, each element having the structure provided by the termvectors API. Here is an example:

POST /_mtermvectors
{
   "docs": [
      {
         "_index": "twitter",
         "_type": "_doc",
         "_id": "2",
         "term_statistics": true
      },
      {
         "_index": "twitter",
         "_type": "_doc",
         "_id": "1",
         "fields": [
            "message"
         ]
      }
   ]
}

See the termvectors API for a description of possible parameters.

The _mtermvectors endpoint can also be used against an index (in which case it is not required in the body):

POST /twitter/_mtermvectors
{
   "docs": [
      {
         "_type": "_doc",
         "_id": "2",
         "fields": [
            "message"
         ],
         "term_statistics": true
      },
      {
         "_type": "_doc",
         "_id": "1"
      }
   ]
}

And type:

POST /twitter/_doc/_mtermvectors
{
   "docs": [
      {
         "_id": "2",
         "fields": [
            "message"
         ],
         "term_statistics": true
      },
      {
         "_id": "1"
      }
   ]
}

If all requested documents are on same index and have the same type and the same parameters, the request can be simplified:

POST /twitter/_doc/_mtermvectors
{
    "ids" : ["1", "2"],
    "parameters": {
    	"fields": [
         	"message"
      	],
      	"term_statistics": true
    }
}

Additionally, just like for the termvectors API, term vectors could be generated for user provided documents. The mapping used is determined by _index and _type.

POST /_mtermvectors
{
   "docs": [
      {
         "_index": "twitter",
         "_type": "_doc",
         "doc" : {
            "user" : "John Doe",
            "message" : "twitter test test test"
         }
      },
      {
         "_index": "twitter",
         "_type": "_doc",
         "doc" : {
           "user" : "Jane Doe",
           "message" : "Another twitter test ..."
         }
      }
   ]
}

?refresh

The Index, Update, Delete, and Bulk APIs support setting refresh to control when changes made by this request are made visible to search. These are the allowed values:

Empty string or true

Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately. This should ONLY be done after careful thought and verification that it does not lead to poor performance, both from an indexing and a search standpoint.

wait_for

Wait for the changes made by the request to be made visible by a refresh before replying. This doesn’t force an immediate refresh, rather, it waits for a refresh to happen. Elasticsearch automatically refreshes shards that have changed every index.refresh_interval which defaults to one second. That setting is dynamic. Calling the Refresh API or setting refresh to true on any of the APIs that support it will also cause a refresh, in turn causing already running requests with refresh=wait_for to return.

false (the default)

Take no refresh related actions. The changes made by this request will be made visible at some point after the request returns.

Choosing which setting to use

Unless you have a good reason to wait for the change to become visible always use refresh=false, or, because that is the default, just leave the refresh parameter out of the URL. That is the simplest and fastest choice.

If you absolutely must have the changes made by a request visible synchronously with the request then you must pick between putting more load on Elasticsearch (true) and waiting longer for the response (wait_for). Here are a few points that should inform that decision:

  • The more changes being made to the index the more work wait_for saves compared to true. In the case that the index is only changed once every index.refresh_interval then it saves no work.

  • true creates less efficient indexes constructs (tiny segments) that must later be merged into more efficient index constructs (larger segments). Meaning that the cost of true is paid at index time to create the tiny segment, at search time to search the tiny segment, and at merge time to make the larger segments.

  • Never start multiple refresh=wait_for requests in a row. Instead batch them into a single bulk request with refresh=wait_for and Elasticsearch will start them all in parallel and return only when they have all finished.

  • If the refresh interval is set to -1, disabling the automatic refreshes, then requests with refresh=wait_for will wait indefinitely until some action causes a refresh. Conversely, setting index.refresh_interval to something shorter than the default like 200ms will make refresh=wait_for come back faster, but it’ll still generate inefficient segments.

  • refresh=wait_for only affects the request that it is on, but, by forcing a refresh immediately, refresh=true will affect other ongoing requests. In general, if you have a running system you don’t wish to disturb then refresh=wait_for is a smaller modification.

refresh=wait_for Can Force a Refresh

If a refresh=wait_for request comes in when there are already index.max_refresh_listeners (defaults to 1000) requests waiting for a refresh on that shard then that request will behave just as though it had refresh set to true instead: it will force a refresh. This keeps the promise that when a refresh=wait_for request returns that its changes are visible for search while preventing unchecked resource usage for blocked requests. If a request forced a refresh because it ran out of listener slots then its response will contain "forced_refresh": true.

Bulk requests only take up one slot on each shard that they touch no matter how many times they modify the shard.

Examples

These will create a document and immediately refresh the index so it is visible:

PUT /test/_doc/1?refresh
{"test": "test"}
PUT /test/_doc/2?refresh=true
{"test": "test"}

These will create a document without doing anything to make it visible for search:

PUT /test/_doc/3
{"test": "test"}
PUT /test/_doc/4?refresh=false
{"test": "test"}

This will create a document and wait for it to become visible for search:

PUT /test/_doc/4?refresh=wait_for
{"test": "test"}

Optimistic concurrency control

Elasticsearch is distributed. When documents are created, updated, or deleted, the new version of the document has to be replicated to other nodes in the cluster. Elasticsearch is also asynchronous and concurrent, meaning that these replication requests are sent in parallel, and may arrive at their destination out of sequence. Elasticsearch needs a way of ensuring that an older version of a document never overwrites a newer version.

To ensure an older version of a document doesn’t overwrite a newer version, every operation performed to a document is assigned a sequence number by the primary shard that coordinates that change. The sequence number is increased with each operation and thus newer operations are guaranteed to have a higher sequence number than older operations. Elasticsearch can then use the sequence number of operations to make sure a newer document version is never overridden by a change that has a smaller sequence number assigned to it.

For example, the following indexing command will create a document and assign it an initial sequence number and primary term:

PUT products/_doc/1567
{
    "product" : "r2d2",
    "details" : "A resourceful astromech droid"
}

You can see the assigned sequence number and primary term in the _seq_no and _primary_term fields of the response:

{
    "_shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 1
    },
    "_index" : "products",
    "_type" : "_doc",
    "_id" : "1567",
    "_version" : 1,
    "_seq_no" : 362,
    "_primary_term" : 2,
    "result" : "created"
}

Elasticsearch keeps tracks of the sequence number and primary term of the last operation to have changed each of the documents it stores. The sequence number and primary term are returned in the _seq_no and _primary_term fields in the response of the GET API:

GET products/_doc/1567

returns:

{
    "_index" : "products",
    "_type" : "_doc",
    "_id" : "1567",
    "_version" : 1,
    "_seq_no" : 362,
    "_primary_term" : 2,
    "found": true,
    "_source" : {
        "product" : "r2d2",
        "details" : "A resourceful astromech droid"
    }
}

Note: The Search API can return the _seq_no and _primary_term for each search hit by setting seq_no_primary_term parameter.

The sequence number and the primary term uniquely identify a change. By noting down the sequence number and primary term returned, you can make sure to only change the document if no other change was made to it since you retrieved it. This is done by setting the if_seq_no and if_primary_term parameters of either the Index API or the Delete API.

For example, the following indexing call will make sure to add a tag to the document without losing any potential change to the description or an addition of another tag by another API:

PUT products/_doc/1567?if_seq_no=362&if_primary_term=2
{
    "product" : "r2d2",
    "details" : "A resourceful astromech droid",
    "tags": ["droid"]
}

Search APIs

The search API allows you to execute a search query and get back search hits that match the query. The query can either be provided using a simple query string as a parameter, or using a request body.

Multi-Index

All search APIs can be applied across multiple indices with support for the multi index syntax. For example, we can search on all documents within the twitter index:

GET /twitter/_search?q=user:kimchy

We can also search all documents with a certain tag across several indices (for example, when there is one index per user):

GET /kimchy,elasticsearch/_search?q=tag:wow

Or we can search across all available indices using _all:

GET /_all/_search?q=tag:wow

Partial responses

To ensure fast responses, the search API will respond with partial results if one or more shards fail. See Shard failures for more information.

URI Search

A search request can be executed purely using a URI by providing request parameters. Not all search options are exposed when executing a search using this mode, but it can be handy for quick "curl tests". Here is an example:

GET twitter/_search?q=user:kimchy

And here is a sample response:

{
    "timed_out": false,
    "took": 62,
    "_shards":{
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
    },
    "hits":{
        "total" : 1,
        "max_score": 1.3862944,
        "hits" : [
            {
                "_index" : "twitter",
                "_type" : "_doc",
                "_id" : "0",
                "_score": 1.3862944,
                "_source" : {
                    "user" : "kimchy",
                    "date" : "2009-11-15T14:12:12",
                    "message" : "trying out Elasticsearch",
                    "likes": 0
                }
            }
        ]
    }
}

Parameters

The parameters allowed in the URI are:

Name Description

q

The query string (maps to the query_string query, see Query String Query for more details).

df

The default field to use when no field prefix is defined within the query.

analyzer

The analyzer name to be used when analyzing the query string.

analyze_wildcard

Should wildcard and prefix queries be analyzed or not. Defaults to false.

batched_reduce_size

The number of shard results that should be reduced at once on the coordinating node. This value should be used as a protection mechanism to reduce the memory overhead per search request if the potential number of shards in the request can be large.

default_operator

The default operator to be used, can be AND or OR. Defaults to OR.

lenient

If set to true will cause format based failures (like providing text to a numeric field) to be ignored. Defaults to false.

explain

For each hit, contain an explanation of how scoring of the hits was computed.

_source

Set to false to disable retrieval of the _source field. You can also retrieve part of the document by using _source_includes & _source_excludes (see the request body documentation for more details)

stored_fields

The selective stored fields of the document to return for each hit, comma delimited. Not specifying any value will cause no fields to return.

sort

Sorting to perform. Can either be in the form of fieldName, or fieldName:asc/fieldName:desc. The fieldName can either be an actual field within the document, or the special _score name to indicate sorting based on scores. There can be several sort parameters (order is important).

track_scores

When sorting, set to true in order to still track scores and return them as part of each hit.

track_total_hits

Set to false in order to disable the tracking of the total number of hits that match the query. (see Index Sorting for more details). Defaults to true.

timeout

A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Defaults to no timeout.

terminate_after

The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early. If set, the response will have a boolean field terminated_early to indicate whether the query execution has actually terminated_early. Defaults to no terminate_after.

from

The starting from index of the hits to return. Defaults to 0.

size

The number of hits to return. Defaults to 10.

search_type

The type of the search operation to perform. Can be dfs_query_then_fetch or query_then_fetch. Defaults to query_then_fetch. See Search Type for more details on the different types of search that can be performed.

allow_partial_search_results

Set to false to return an overall failure if the request would produce partial results. Defaults to true, which will allow partial results in the case of timeouts or partial failures. This default can be controlled using the cluster-level setting search.default_allow_partial_results.

Request Body Search

The search request can be executed with a search DSL, which includes the Query DSL, within its body. Here is an example:

GET /twitter/_search
{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

And here is a sample response:

{
    "took": 1,
    "timed_out": false,
    "_shards":{
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
    },
    "hits":{
        "total" : 1,
        "max_score": 1.3862944,
        "hits" : [
            {
                "_index" : "twitter",
                "_type" : "_doc",
                "_id" : "0",
                "_score": 1.3862944,
                "_source" : {
                    "user" : "kimchy",
                    "message": "trying out Elasticsearch",
                    "date" : "2009-11-15T14:12:12",
                    "likes" : 0
                }
            }
        ]
    }
}

Parameters

timeout

A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Search requests are canceled after the timeout is reached using the Search Cancellation mechanism. Defaults to no timeout. See Time units.

from

To retrieve hits from a certain offset. Defaults to 0.

size

The number of hits to return. Defaults to 10. If you do not care about getting some hits back but only about the number of matches and/or aggregations, setting the value to 0 will help performance.

search_type

The type of the search operation to perform. Can be dfs_query_then_fetch or query_then_fetch. Defaults to query_then_fetch. See Search Type for more.

request_cache

Set to true or false to enable or disable the caching of search results for requests where size is 0, ie aggregations and suggestions (no top hits returned). See Shard request cache.

allow_partial_search_results

Set to false to return an overall failure if the request would produce partial results. Defaults to true, which will allow partial results in the case of timeouts or partial failures. This default can be controlled using the cluster-level setting search.default_allow_partial_results.

terminate_after

The maximum number of documents to collect for each shard, upon reaching which the query execution will terminate early. If set, the response will have a boolean field terminated_early to indicate whether the query execution has actually terminated_early. Defaults to no terminate_after.

batched_reduce_size

The number of shard results that should be reduced at once on the coordinating node. This value should be used as a protection mechanism to reduce the memory overhead per search request if the potential number of shards in the request can be large.

Out of the above, the search_type, request_cache and the allow_partial_search_results settings must be passed as query-string parameters. The rest of the search request should be passed within the body itself. The body content can also be passed as a REST parameter named source.

Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well.

Fast check for any matching docs

Note
terminate_after is always applied after the post_filter and stops the query as well as the aggregation executions when enough hits have been collected on the shard. Though the doc count on aggregations may not reflect the hits.total in the response since aggregations are applied before the post filtering.

In case we only want to know if there are any documents matching a specific query, we can set the size to 0 to indicate that we are not interested in the search results. Also we can set terminate_after to 1 to indicate that the query execution can be terminated whenever the first matching document was found (per shard).

GET /_search?q=message:number&size=0&terminate_after=1

The response will not contain any hits as the size was set to 0. The hits.total will be either equal to 0, indicating that there were no matching documents, or greater than 0 meaning that there were at least as many documents matching the query when it was early terminated. Also if the query was terminated early, the terminated_early flag will be set to true in the response.

{
  "took": 3,
  "timed_out": false,
  "terminated_early": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.0,
    "hits": []
  }
}

The took time in the response contains the milliseconds that this request took for processing, beginning quickly after the node received the query, up until all search related work is done and before the above JSON is returned to the client. This means it includes the time spent waiting in thread pools, executing a distributed search across the whole cluster and gathering all the results.

Query

The query element within the search request body allows to define a query using the Query DSL.

GET /_search
{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

From / Size

Pagination of results can be done by using the from and size parameters. The from parameter defines the offset from the first result you want to fetch. The size parameter allows you to configure the maximum amount of hits to be returned.

Though from and size can be set as request parameters, they can also be set within the search body. from defaults to 0, and size defaults to 10.

GET /_search
{
    "from" : 0, "size" : 10,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Note that from + size can not be more than the index.max_result_window index setting which defaults to 10,000. See the Scroll or Search After API for more efficient ways to do deep scrolling.

Sort

Allows you to add one or more sorts on specific fields. Each sort can be reversed as well. The sort is defined on a per field level, with special field name for _score to sort by score, and _doc to sort by index order.

Assuming the following index mapping:

PUT /my_index
{
    "mappings": {
        "_doc": {
            "properties": {
                "post_date": { "type": "date" },
                "user": {
                    "type": "keyword"
                },
                "name": {
                    "type": "keyword"
                },
                "age": { "type": "integer" }
            }
        }
    }
}
GET /my_index/_search
{
    "sort" : [
        { "post_date" : {"order" : "asc"}},
        "user",
        { "name" : "desc" },
        { "age" : "desc" },
        "_score"
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
Note
_doc has no real use-case besides being the most efficient sort order. So if you don’t care about the order in which documents are returned, then you should sort by _doc. This especially helps when scrolling.

Sort Values

The sort values for each document returned are also returned as part of the response.

Sort Order

The order option can have the following values:

asc

Sort in ascending order

desc

Sort in descending order

The order defaults to desc when sorting on the _score, and defaults to asc when sorting on anything else.

Sort mode option

Elasticsearch supports sorting by array or multi-valued fields. The mode option controls what array value is picked for sorting the document it belongs to. The mode option can have the following values:

min

Pick the lowest value.

max

Pick the highest value.

sum

Use the sum of all values as sort value. Only applicable for number based array fields.

avg

Use the average of all values as sort value. Only applicable for number based array fields.

median

Use the median of all values as sort value. Only applicable for number based array fields.

The default sort mode in the ascending sort order is min — the lowest value is picked. The default sort mode in the descending order is max — the highest value is picked.

Sort mode example usage

In the example below the field price has multiple prices per document. In this case the result hits will be sorted by price ascending based on the average price per document.

PUT /my_index/_doc/1?refresh
{
   "product": "chocolate",
   "price": [20, 4]
}

POST /_search
{
   "query" : {
      "term" : { "product" : "chocolate" }
   },
   "sort" : [
      {"price" : {"order" : "asc", "mode" : "avg"}}
   ]
}

Sorting within nested objects.

Elasticsearch also supports sorting by fields that are inside one or more nested objects. The sorting by nested field support has a nested sort option with the following properties:

path

Defines on which nested object to sort. The actual sort field must be a direct field inside this nested object. When sorting by nested field, this field is mandatory.

filter

A filter that the inner objects inside the nested path should match with in order for its field values to be taken into account by sorting. Common case is to repeat the query / filter inside the nested filter or query. By default no nested_filter is active.

max_children

The maximum number of children to consider per root document when picking the sort value. Defaults to unlimited.

nested

Same as top-level nested but applies to another nested path within the current nested object.

Warning
Nested sort options before Elasticseach 6.1

The nested_path and nested_filter options have been deprecated in favor of the options documented above.

Nested sorting examples

In the below example offer is a field of type nested. The nested path needs to be specified; otherwise, Elasticsearch doesn’t know on what nested level sort values need to be captured.

POST /_search
{
   "query" : {
      "term" : { "product" : "chocolate" }
   },
   "sort" : [
       {
          "offer.price" : {
             "mode" :  "avg",
             "order" : "asc",
             "nested": {
                "path": "offer",
                "filter": {
                   "term" : { "offer.color" : "blue" }
                }
             }
          }
       }
    ]
}

In the below example parent and child fields are of type nested. The nested_path needs to be specified at each level; otherwise, Elasticsearch doesn’t know on what nested level sort values need to be captured.

POST /_search
{
   "query": {
      "nested": {
         "path": "parent",
         "query": {
            "bool": {
                "must": {"range": {"parent.age": {"gte": 21}}},
                "filter": {
                    "nested": {
                        "path": "parent.child",
                        "query": {"match": {"parent.child.name": "matt"}}
                    }
                }
            }
         }
      }
   },
   "sort" : [
      {
         "parent.child.age" : {
            "mode" :  "min",
            "order" : "asc",
            "nested": {
               "path": "parent",
               "filter": {
                  "range": {"parent.age": {"gte": 21}}
               },
               "nested": {
                  "path": "parent.child",
                  "filter": {
                     "match": {"parent.child.name": "matt"}
                  }
               }
            }
         }
      }
   ]
}

Nested sorting is also supported when sorting by scripts and sorting by geo distance.

Missing Values

The missing parameter specifies how docs which are missing the sort field should be treated: The missing value can be set to _last, _first, or a custom value (that will be used for missing docs as the sort value). The default is _last.

For example:

GET /_search
{
    "sort" : [
        { "price" : {"missing" : "_last"} }
    ],
    "query" : {
        "term" : { "product" : "chocolate" }
    }
}
Note
If a nested inner object doesn’t match with the nested_filter then a missing value is used.

Ignoring Unmapped Fields

By default, the search request will fail if there is no mapping associated with a field. The unmapped_type option allows you to ignore fields that have no mapping and not sort by them. The value of this parameter is used to determine what sort values to emit. Here is an example of how it can be used:

GET /_search
{
    "sort" : [
        { "price" : {"unmapped_type" : "long"} }
    ],
    "query" : {
        "term" : { "product" : "chocolate" }
    }
}

If any of the indices that are queried doesn’t have a mapping for price then Elasticsearch will handle it as if there was a mapping of type long, with all documents in this index having no value for this field.

Geo Distance Sorting

Allow to sort by _geo_distance. Here is an example, assuming pin.location is a field of type geo_point:

GET /_search
{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : [-70, 40],
                "order" : "asc",
                "unit" : "km",
                "mode" : "min",
                "distance_type" : "arc",
                "ignore_unmapped": true
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
distance_type

How to compute the distance. Can either be arc (default), or plane (faster, but inaccurate on long distances and close to the poles).

mode

What to do in case a field has several geo points. By default, the shortest distance is taken into account when sorting in ascending order and the longest distance when sorting in descending order. Supported values are min, max, median and avg.

unit

The unit to use when computing sort values. The default is m (meters).

ignore_unmapped

Indicates if the unmapped field should be treated as a missing value. Setting it to true is equivalent to specifying an unmapped_type in the field sort. The default is false (unmapped field cause the search to fail).

Note
geo distance sorting does not support configurable missing values: the distance will always be considered equal to Infinity when a document does not have values for the field that is used for distance computation.

The following formats are supported in providing the coordinates:

Lat Lon as Properties
GET /_search
{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : {
                    "lat" : 40,
                    "lon" : -70
                },
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
Lat Lon as String

Format in lat,lon.

GET /_search
{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : "40,-70",
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
Geohash
GET /_search
{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : "drm3btev3e86",
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
Lat Lon as Array

Format in [lon, lat], note, the order of lon/lat here in order to conform with GeoJSON.

GET /_search
{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : [-70, 40],
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Multiple reference points

Multiple geo points can be passed as an array containing any geo_point format, for example

GET /_search
{
    "sort" : [
        {
            "_geo_distance" : {
                "pin.location" : [[-70, 40], [-71, 42]],
                "order" : "asc",
                "unit" : "km"
            }
        }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

and so forth.

The final distance for a document will then be min/max/avg (defined via mode) distance of all points contained in the document to all points given in the sort request.

Script Based Sorting

Allow to sort based on custom scripts, here is an example:

GET /_search
{
    "query" : {
        "term" : { "user" : "kimchy" }
    },
    "sort" : {
        "_script" : {
            "type" : "number",
            "script" : {
                "lang": "painless",
                "source": "doc['field_name'].value * params.factor",
                "params" : {
                    "factor" : 1.1
                }
            },
            "order" : "asc"
        }
    }
}

Track Scores

When sorting on a field, scores are not computed. By setting track_scores to true, scores will still be computed and tracked.

GET /_search
{
    "track_scores": true,
    "sort" : [
        { "post_date" : {"order" : "desc"} },
        { "name" : "desc" },
        { "age" : "desc" }
    ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Memory Considerations

When sorting, the relevant sorted field values are loaded into memory. This means that per shard, there should be enough memory to contain them. For string based types, the field sorted on should not be analyzed / tokenized. For numeric types, if possible, it is recommended to explicitly set the type to narrower types (like short, integer and float).

Source filtering

Allows to control how the _source field is returned with every hit.

By default operations return the contents of the _source field unless you have used the stored_fields parameter or if the _source field is disabled.

You can turn off _source retrieval by using the _source parameter:

To disable _source retrieval set to false:

GET /_search
{
    "_source": false,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

The _source also accepts one or more wildcard patterns to control what parts of the _source should be returned:

For example:

GET /_search
{
    "_source": "obj.*",
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Or

GET /_search
{
    "_source": [ "obj1.*", "obj2.*" ],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Finally, for complete control, you can specify both includes and excludes patterns:

GET /_search
{
    "_source": {
        "includes": [ "obj1.*", "obj2.*" ],
        "excludes": [ "*.description" ]
    },
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Fields

Warning
The stored_fields parameter is about fields that are explicitly marked as stored in the mapping, which is off by default and generally not recommended. Use source filtering instead to select subsets of the original source document to be returned.

Allows to selectively load specific stored fields for each document represented by a search hit.

GET /_search
{
    "stored_fields" : ["user", "postDate"],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

* can be used to load all stored fields from the document.

An empty array will cause only the _id and _type for each hit to be returned, for example:

GET /_search
{
    "stored_fields" : [],
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

If the requested fields are not stored (store mapping set to false), they will be ignored.

Stored field values fetched from the document itself are always returned as an array. On the contrary, metadata fields like _routing are never returned as an array.

Also only leaf fields can be returned via the stored_fields option. If an object field is specified, it will be ignored.

Note
On its own, stored_fields cannot be used to load fields in nested objects — if a field contains a nested object in its path, then no data will be returned for that stored field. To access nested fields, stored_fields must be used within an inner_hits block.

Disable stored fields entirely

To disable the stored fields (and metadata fields) entirely use: none:

GET /_search
{
    "stored_fields": "_none_",
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
Note
source and version parameters cannot be activated if _none is used.

Script Fields

Allows to return a script evaluation (based on different fields) for each hit, for example:

GET /_search
{
    "query" : {
        "match_all": {}
    },
    "script_fields" : {
        "test1" : {
            "script" : {
                "lang": "painless",
                "source": "doc['price'].value * 2"
            }
        },
        "test2" : {
            "script" : {
                "lang": "painless",
                "source": "doc['price'].value * params.factor",
                "params" : {
                    "factor"  : 2.0
                }
            }
        }
    }
}

Script fields can work on fields that are not stored (my_field_name in the above case), and allow to return custom values to be returned (the evaluated value of the script).

Script fields can also access the actual _source document and extract specific elements to be returned from it by using params['_source']. Here is an example:

GET /_search
    {
        "query" : {
            "match_all": {}
        },
        "script_fields" : {
            "test1" : {
                "script" : "params['_source']['message']"
            }
        }
    }

Note the _source keyword here to navigate the json-like model.

It’s important to understand the difference between doc['my_field'].value and params['_source']['my_field']. The first, using the doc keyword, will cause the terms for that field to be loaded to memory (cached), which will result in faster execution, but more memory consumption. Also, the doc[…​] notation only allows for simple valued fields (you can’t return a json object from it) and makes sense only for non-analyzed or single term based fields. However, using doc is still the recommended way to access values from the document, if at all possible, because _source must be loaded and parsed every time it’s used. Using _source is very slow.

Doc value Fields

Allows to return the doc value representation of a field for each hit, for example:

GET /_search
{
    "query" : {
        "match_all": {}
    },
    "docvalue_fields" : [
        {
            "field": "my_ip_field", (1)
            "format": "use_field_mapping" (2)
        },
        {
            "field": "my_date_field",
            "format": "epoch_millis" (3)
        }
    ]
}
  1. the name of the field

  2. the special use_field_mapping format tells Elasticsearch to use the format from the mapping

  3. date fields may use a custom format

Doc value fields can work on fields that are not stored.

* can be used as a wild card, for example:

GET /_search
{
    "query" : {
        "match_all": {}
    },
    "docvalue_fields" : [
        {
            "field": "*field", (1)
            "format": "use_field_mapping" (2)
        }
    ]
}
  1. Match all fields ending with field

  2. Format to be applied to all matching fields.

Note that if the fields parameter specifies fields without docvalues it will try to load the value from the fielddata cache causing the terms for that field to be loaded to memory (cached), which will result in more memory consumption.

Custom formats

While most fields do not support custom formats, some of them do:

All fields support the special use_field_mapping format, which tells Elasticsearch to use the mappings to figure out a default format.

Note
The default is currently to return the same output as script fields. However it will change in 7.0 to behave as if the use_field_mapping format was provided.
Note
On its own, docvalue_fields cannot be used to load fields in nested objects — if a field contains a nested object in its path, then no data will be returned for that docvalue field. To access nested fields, docvalue_fields must be used within an inner_hits block.

Post filter

The post_filter is applied to the search hits at the very end of a search request, after aggregations have already been calculated. Its purpose is best explained by example:

Imagine that you are selling shirts that have the following properties:

PUT /shirts
{
    "mappings": {
        "_doc": {
            "properties": {
                "brand": { "type": "keyword"},
                "color": { "type": "keyword"},
                "model": { "type": "keyword"}
            }
        }
    }
}

PUT /shirts/_doc/1?refresh
{
    "brand": "gucci",
    "color": "red",
    "model": "slim"
}

Imagine a user has specified two filters:

color:red and brand:gucci. You only want to show them red shirts made by Gucci in the search results. Normally you would do this with a bool query:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  }
}

However, you would also like to use faceted navigation to display a list of other options that the user could click on. Perhaps you have a model field that would allow the user to limit their search results to red Gucci t-shirts or dress-shirts.

This can be done with a terms aggregation:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  },
  "aggs": {
    "models": {
      "terms": { "field": "model" } (1)
    }
  }
}
  1. Returns the most popular models of red shirts by Gucci.

But perhaps you would also like to tell the user how many Gucci shirts are available in other colors. If you just add a terms aggregation on the color field, you will only get back the color red, because your query returns only red shirts by Gucci.

Instead, you want to include shirts of all colors during aggregation, then apply the colors filter only to the search results. This is the purpose of the post_filter:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": { "brand": "gucci" } (1)
      }
    }
  },
  "aggs": {
    "colors": {
      "terms": { "field": "color" } (2)
    },
    "color_red": {
      "filter": {
        "term": { "color": "red" } (3)
      },
      "aggs": {
        "models": {
          "terms": { "field": "model" } (3)
        }
      }
    }
  },
  "post_filter": { (4)
    "term": { "color": "red" }
  }
}
  1. The main query now finds all shirts by Gucci, regardless of color.

  2. The colors agg returns popular colors for shirts by Gucci.

  3. The color_red agg limits the models sub-aggregation to red Gucci shirts.

  4. Finally, the post_filter removes colors other than red from the search hits.

Highlighting

Highlighters enable you to get highlighted snippets from one or more fields in your search results so you can show users where the query matches are. When you request highlights, the response contains an additional highlight element for each search hit that includes the highlighted fields and the highlighted fragments.

Note
Highlighters don’t reflect the boolean logic of a query when extracting terms to highlight. Thus, for some complex boolean queries (e.g nested boolean queries, queries using minimum_should_match etc.), parts of documents may be highlighted that don’t correspond to query matches.

Highlighting requires the actual content of a field. If the field is not stored (the mapping does not set store to true), the actual _source is loaded and the relevant field is extracted from _source.

Note
The _all field cannot be extracted from _source, so it can only be used for highlighting if it is explicitly stored.

For example, to get highlights for the content field in each search hit using the default highlighter, include a highlight object in the request body that specifies the content field:

GET /_search
{
    "query" : {
        "match": { "content": "kimchy" }
    },
    "highlight" : {
        "fields" : {
            "content" : {}
        }
    }
}

{es} supports three highlighters: unified, plain, and fvh (fast vector highlighter). You can specify the highlighter type you want to use for each field.

Unified highlighter

The unified highlighter uses the Lucene Unified Highlighter. This highlighter breaks the text into sentences and uses the BM25 algorithm to score individual sentences as if they were documents in the corpus. It also supports accurate phrase and multi-term (fuzzy, prefix, regex) highlighting. This is the default highlighter.

Plain highlighter

The plain highlighter uses the standard Lucene highlighter. It attempts to reflect the query matching logic in terms of understanding word importance and any word positioning criteria in phrase queries.

Warning
The plain highlighter works best for highlighting simple query matches in a single field. To accurately reflect query logic, it creates a tiny in-memory index and re-runs the original query criteria through Lucene’s query execution planner to get access to low-level match information for the current document. This is repeated for every field and every document that needs to be highlighted. If you want to highlight a lot of fields in a lot of documents with complex queries, we recommend using the unified highlighter on postings or term_vector fields.

Fast vector highlighter

The fvh highlighter uses the Lucene Fast Vector highlighter. This highlighter can be used on fields with term_vector set to with_positions_offsets in the mapping. The fast vector highlighter:

  • Can be customized with a boundary_scanner.

  • Requires setting term_vector to with_positions_offsets which increases the size of the index

  • Can combine matches from multiple fields into one result. See matched_fields

  • Can assign different weights to matches at different positions allowing for things like phrase matches being sorted above term matches when highlighting a Boosting Query that boosts phrase matches over term matches

Warning
The fvh highlighter does not support span queries. If you need support for span queries, try an alternative highlighter, such as the unified highlighter.

Offsets Strategy

To create meaningful search snippets from the terms being queried, the highlighter needs to know the start and end character offsets of each word in the original text. These offsets can be obtained from:

  • The postings list. If index_options is set to offsets in the mapping, the unified highlighter uses this information to highlight documents without re-analyzing the text. It re-runs the original query directly on the postings and extracts the matching offsets from the index, limiting the collection to the highlighted documents. This is important if you have large fields because it doesn’t require reanalyzing the text to be highlighted. It also requires less disk space than using term_vectors.

  • Term vectors. If term_vector information is provided by setting term_vector to with_positions_offsets in the mapping, the unified highlighter automatically uses the term_vector to highlight the field. It’s fast especially for large fields (> 1MB) and for highlighting multi-term queries like prefix or wildcard because it can access the dictionary of terms for each document. The fvh highlighter always uses term vectors.

  • Plain highlighting. This mode is used by the unified when there is no other alternative. It creates a tiny in-memory index and re-runs the original query criteria through Lucene’s query execution planner to get access to low-level match information on the current document. This is repeated for every field and every document that needs highlighting. The plain highlighter always uses plain highlighting.

Warning
Plain highlighting for large texts may require substantial amount of time and memory. To protect against this, the maximum number of text characters to be analyzed will be limited to 1000000 in the next major Elastic version. The default limit is not set for this version, but can be set for a particular index with the index setting index.highlight.max_analyzed_offset.

Highlighting Settings

Highlighting settings can be set on a global level and overridden at the field level.

boundary_chars

A string that contains each boundary character. Defaults to .,!? \t\n.

boundary_max_scan

How far to scan for boundary characters. Defaults to 20.

boundary_scanner

Specifies how to break the highlighted fragments: chars, sentence, or word. Only valid for the unified and fvh highlighters. Defaults to sentence for the unified highlighter. Defaults to chars for the fvh highlighter.

chars

Use the characters specified by boundary_chars as highlighting boundaries. The boundary_max_scan setting controls how far to scan for boundary characters. Only valid for the fvh highlighter.

sentence

Break highlighted fragments at the next sentence boundary, as determined by Java’s BreakIterator. You can specify the locale to use with boundary_scanner_locale.

Note
When used with the unified highlighter, the sentence scanner splits sentences bigger than fragment_size at the first word boundary next to fragment_size. You can set fragment_size to 0 to never split any sentence.
word

Break highlighted fragments at the next word boundary, as determined by Java’s BreakIterator. You can specify the locale to use with boundary_scanner_locale.

boundary_scanner_locale

Controls which locale is used to search for sentence and word boundaries. This parameter takes a form of a language tag, e.g. "en-US", "fr-FR", "ja-JP". More info can be found in the Locale Language Tag documentation. The default value is Locale.ROOT.

encoder

Indicates if the snippet should be HTML encoded: default (no encoding) or html (HTML-escape the snippet text and then insert the highlighting tags)

fields

Specifies the fields to retrieve highlights for. You can use wildcards to specify fields. For example, you could specify comment_* to get highlights for all text and keyword fields that start with comment_.

Note
Only text and keyword fields are highlighted when you use wildcards. If you use a custom mapper and want to highlight on a field anyway, you must explicitly specify that field name.
force_source

Highlight based on the source even if the field is stored separately. Defaults to false.

fragmenter

Specifies how text should be broken up in highlight snippets: simple or span. Only valid for the plain highlighter. Defaults to span.

simple

Breaks up text into same-sized fragments.

span

Breaks up text into same-sized fragments, but tries to avoid breaking up text between highlighted terms. This is helpful when you’re querying for phrases. Default.

fragment_offset

Controls the margin from which you want to start highlighting. Only valid when using the fvh highlighter.

fragment_size

The size of the highlighted fragment in characters. Defaults to 100.

highlight_query

Highlight matches for a query other than the search query. This is especially useful if you use a rescore query because those are not taken into account by highlighting by default.

Important
{es} does not validate that highlight_query contains the search query in any way so it is possible to define it so legitimate query results are not highlighted. Generally, you should include the search query as part of the highlight_query.
matched_fields

Combine matches on multiple fields to highlight a single field. This is most intuitive for multifields that analyze the same string in different ways. All matched_fields must have term_vector set to with_positions_offsets, but only the field to which the matches are combined is loaded so only that field benefits from having store set to yes. Only valid for the fvh highlighter.

no_match_size

The amount of text you want to return from the beginning of the field if there are no matching fragments to highlight. Defaults to 0 (nothing is returned).

number_of_fragments

The maximum number of fragments to return. If the number of fragments is set to 0, no fragments are returned. Instead, the entire field contents are highlighted and returned. This can be handy when you need to highlight short texts such as a title or address, but fragmentation is not required. If number_of_fragments is 0, fragment_size is ignored. Defaults to 5.

order

Sorts highlighted fragments by score when set to score. By default, fragments will be output in the order they appear in the field (order: none). Setting this option to score will output the most relevant fragments first. Each highlighter applies its own logic to compute relevancy scores. See the document How highlighters work internally for more details how different highlighters find the best fragments.

phrase_limit

Controls the number of matching phrases in a document that are considered. Prevents the fvh highlighter from analyzing too many phrases and consuming too much memory. When using matched_fields, phrase_limit phrases per matched field are considered. Raising the limit increases query time and consumes more memory. Only supported by the fvh highlighter. Defaults to 256.

pre_tags

Use in conjunction with post_tags to define the HTML tags to use for the highlighted text. By default, highlighted text is wrapped in <em> and </em> tags. Specify as an array of strings.

post_tags

Use in conjunction with pre_tags to define the HTML tags to use for the highlighted text. By default, highlighted text is wrapped in <em> and </em> tags. Specify as an array of strings.

require_field_match

By default, only fields that contains a query match are highlighted. Set require_field_match to false to highlight all fields. Defaults to true.

tags_schema

Set to styled to use the built-in tag schema. The styled schema defines the following pre_tags and defines post_tags as </em>.

<em class="hlt1">, <em class="hlt2">, <em class="hlt3">,
<em class="hlt4">, <em class="hlt5">, <em class="hlt6">,
<em class="hlt7">, <em class="hlt8">, <em class="hlt9">,
<em class="hlt10">
type

The highlighter to use: unified, plain, or fvh. Defaults to unified.

Highlighting Examples

Override global settings

You can specify highlighter settings globally and selectively override them for individual fields.

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "number_of_fragments" : 3,
        "fragment_size" : 150,
        "fields" : {
            "_all" : { "pre_tags" : ["<em>"], "post_tags" : ["</em>"] },
            "blog.title" : { "number_of_fragments" : 0 },
            "blog.author" : { "number_of_fragments" : 0 },
            "blog.comment" : { "number_of_fragments" : 5, "order" : "score" }
        }
    }
}

Specify a highlight query

You can specify a highlight_query to take additional information into account when highlighting. For example, the following query includes both the search query and rescore query in the highlight_query. Without the highlight_query, highlighting would only take the search query into account.

GET /_search
{
    "query" : {
        "match": {
            "comment": {
                "query": "foo bar"
            }
        }
    },
    "rescore": {
        "window_size": 50,
        "query": {
            "rescore_query" : {
                "match_phrase": {
                    "comment": {
                        "query": "foo bar",
                        "slop": 1
                    }
                }
            },
            "rescore_query_weight" : 10
        }
    },
    "_source": false,
    "highlight" : {
        "order" : "score",
        "fields" : {
            "comment" : {
                "fragment_size" : 150,
                "number_of_fragments" : 3,
                "highlight_query": {
                    "bool": {
                        "must": {
                            "match": {
                                "comment": {
                                    "query": "foo bar"
                                }
                            }
                        },
                        "should": {
                            "match_phrase": {
                                "comment": {
                                    "query": "foo bar",
                                    "slop": 1,
                                    "boost": 10.0
                                }
                            }
                        },
                        "minimum_should_match": 0
                    }
                }
            }
        }
    }
}

Set highlighter type

The type field allows to force a specific highlighter type. The allowed values are: unified, plain and fvh. The following is an example that forces the use of the plain highlighter:

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "fields" : {
            "comment" : {"type" : "plain"}
        }
    }
}

Configure highlighting tags

By default, the highlighting will wrap highlighted text in <em> and </em>. This can be controlled by setting pre_tags and post_tags, for example:

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "pre_tags" : ["<tag1>"],
        "post_tags" : ["</tag1>"],
        "fields" : {
            "_all" : {}
        }
    }
}

When using the fast vector highlighter, you can specify additional tags and the "importance" is ordered.

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "_all" : {}
        }
    }
}

You can also use the built-in styled tag schema:

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "tags_schema" : "styled",
        "fields" : {
            "comment" : {}
        }
    }
}

Highlight on source

Forces the highlighting to highlight fields based on the source even if fields are stored separately. Defaults to false.

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "fields" : {
            "comment" : {"force_source" : true}
        }
    }
}

Combine matches on multiple fields

Warning
This is only supported by the fvh highlighter

The Fast Vector Highlighter can combine matches on multiple fields to highlight a single field. This is most intuitive for multifields that analyze the same string in different ways. All matched_fields must have term_vector set to with_positions_offsets but only the field to which the matches are combined is loaded so only that field would benefit from having store set to yes.

In the following examples, comment is analyzed by the english analyzer and comment.plain is analyzed by the standard analyzer.

GET /_search
{
    "query": {
        "query_string": {
            "query": "comment.plain:running scissors",
            "fields": ["comment"]
        }
    },
    "highlight": {
        "order": "score",
        "fields": {
            "comment": {
                "matched_fields": ["comment", "comment.plain"],
                "type" : "fvh"
            }
        }
    }
}

The above matches both "run with scissors" and "running with scissors" and would highlight "running" and "scissors" but not "run". If both phrases appear in a large document then "running with scissors" is sorted above "run with scissors" in the fragments list because there are more matches in that fragment.

GET /_search
{
    "query": {
        "query_string": {
            "query": "running scissors",
            "fields": ["comment", "comment.plain^10"]
        }
    },
    "highlight": {
        "order": "score",
        "fields": {
            "comment": {
                "matched_fields": ["comment", "comment.plain"],
                "type" : "fvh"
            }
        }
    }
}

The above highlights "run" as well as "running" and "scissors" but still sorts "running with scissors" above "run with scissors" because the plain match ("running") is boosted.

GET /_search
{
    "query": {
        "query_string": {
            "query": "running scissors",
            "fields": ["comment", "comment.plain^10"]
        }
    },
    "highlight": {
        "order": "score",
        "fields": {
            "comment": {
                "matched_fields": ["comment.plain"],
                "type" : "fvh"
            }
        }
    }
}

The above query wouldn’t highlight "run" or "scissor" but shows that it is just fine not to list the field to which the matches are combined (comment) in the matched fields.

Note
Technically it is also fine to add fields to matched_fields that don’t share the same underlying string as the field to which the matches are combined. The results might not make much sense and if one of the matches is off the end of the text then the whole query will fail.
Note

There is a small amount of overhead involved with setting matched_fields to a non-empty array so always prefer

    "highlight": {
        "fields": {
            "comment": {}
        }
    }

to

    "highlight": {
        "fields": {
            "comment": {
                "matched_fields": ["comment"],
                "type" : "fvh"
            }
        }
    }

Explicitly order highlighted fields

Elasticsearch highlights the fields in the order that they are sent, but per the JSON spec, objects are unordered. If you need to be explicit about the order in which fields are highlighted specify the fields as an array:

GET /_search
{
    "highlight": {
        "fields": [
            { "title": {} },
            { "text": {} }
        ]
    }
}

None of the highlighters built into Elasticsearch care about the order that the fields are highlighted but a plugin might.

Control highlighted fragments

Each field highlighted can control the size of the highlighted fragment in characters (defaults to 100), and the maximum number of fragments to return (defaults to 5). For example:

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "fields" : {
            "comment" : {"fragment_size" : 150, "number_of_fragments" : 3}
        }
    }
}

On top of this it is possible to specify that highlighted fragments need to be sorted by score:

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "order" : "score",
        "fields" : {
            "comment" : {"fragment_size" : 150, "number_of_fragments" : 3}
        }
    }
}

If the number_of_fragments value is set to 0 then no fragments are produced, instead the whole content of the field is returned, and of course it is highlighted. This can be very handy if short texts (like document title or address) need to be highlighted but no fragmentation is required. Note that fragment_size is ignored in this case.

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "fields" : {
            "_all" : {},
            "blog.title" : {"number_of_fragments" : 0}
        }
    }
}

When using fvh one can use fragment_offset parameter to control the margin to start highlighting from.

In the case where there is no matching fragment to highlight, the default is to not return anything. Instead, we can return a snippet of text from the beginning of the field by setting no_match_size (default 0) to the length of the text that you want returned. The actual length may be shorter or longer than specified as it tries to break on a word boundary.

GET /_search
{
    "query" : {
        "match": { "user": "kimchy" }
    },
    "highlight" : {
        "fields" : {
            "comment" : {
                "fragment_size" : 150,
                "number_of_fragments" : 3,
                "no_match_size": 150
            }
        }
    }
}

Highlight using the postings list

Here is an example of setting the comment field in the index mapping to allow for highlighting using the postings:

PUT /example
{
  "mappings": {
    "doc" : {
      "properties": {
        "comment" : {
          "type": "text",
          "index_options" : "offsets"
        }
      }
    }
  }
}

Here is an example of setting the comment field to allow for highlighting using the term_vectors (this will cause the index to be bigger):

PUT /example
{
  "mappings": {
    "doc" : {
      "properties": {
        "comment" : {
          "type": "text",
          "term_vector" : "with_positions_offsets"
        }
      }
    }
  }
}

Specify a fragmenter for the plain highlighter

When using the plain highlighter, you can choose between the simple and span fragmenters:

GET twitter/_search
{
    "query" : {
        "match_phrase": { "message": "number 1" }
    },
    "highlight" : {
        "fields" : {
            "message" : {
                "type": "plain",
                "fragment_size" : 15,
                "number_of_fragments" : 3,
                "fragmenter": "simple"
            }
        }
    }
}

Response:

{
    ...
    "hits": {
        "total": 1,
        "max_score": 1.601195,
        "hits": [
            {
                "_index": "twitter",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.601195,
                "_source": {
                    "user": "test",
                    "message": "some message with the number 1",
                    "date": "2009-11-15T14:12:12",
                    "likes": 1
                },
                "highlight": {
                    "message": [
                        " with the <em>number</em>",
                        " <em>1</em>"
                    ]
                }
            }
        ]
    }
}
GET twitter/_search
{
    "query" : {
        "match_phrase": { "message": "number 1" }
    },
    "highlight" : {
        "fields" : {
            "message" : {
                "type": "plain",
                "fragment_size" : 15,
                "number_of_fragments" : 3,
                "fragmenter": "span"
            }
        }
    }
}

Response:

{
    ...
    "hits": {
        "total": 1,
        "max_score": 1.601195,
        "hits": [
            {
                "_index": "twitter",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.601195,
                "_source": {
                    "user": "test",
                    "message": "some message with the number 1",
                    "date": "2009-11-15T14:12:12",
                    "likes": 1
                },
                "highlight": {
                    "message": [
                        " with the <em>number</em> <em>1</em>"
                    ]
                }
            }
        ]
    }
}

If the number_of_fragments option is set to 0, NullFragmenter is used which does not fragment the text at all. This is useful for highlighting the entire contents of a document or field.

How highlighters work internally

Given a query and a text (the content of a document field), the goal of a highlighter is to find the best text fragments for the query, and highlight the query terms in the found fragments. For this, a highlighter needs to address several questions:

  • How break a text into fragments?

  • How to find the best fragments among all fragments?

  • How to highlight the query terms in a fragment?

How to break a text into fragments?

Relevant settings: fragment_size, fragmenter, type of highlighter, boundary_chars, boundary_max_scan, boundary_scanner, boundary_scanner_locale.

Plain highlighter begins with analyzing the text using the given analyzer, and creating a token stream from it. Plain highlighter uses a very simple algorithm to break the token stream into fragments. It loops through terms in the token stream, and every time the current term’s end_offset exceeds fragment_size multiplied by the number of created fragments, a new fragment is created. A little more computation is done with using span fragmenter to avoid breaking up text between highlighted terms. But overall, since the breaking is done only by fragment_size, some fragments can be quite odd, e.g. beginning with a punctuation mark.

Unified or FVH highlighters do a better job of breaking up a text into fragments by utilizing Java’s BreakIterator. This ensures that a fragment is a valid sentence as long as fragment_size allows for this.

How to find the best fragments?

Relevant settings: number_of_fragments.

To find the best, most relevant, fragments, a highlighter needs to score each fragment in respect to the given query. The goal is to score only those terms that participated in generating the 'hit' on the document. For some complex queries, this is still work in progress.

The plain highlighter creates an in-memory index from the current token stream, and re-runs the original query criteria through Lucene’s query execution planner to get access to low-level match information for the current text. For more complex queries the original query could be converted to a span query, as span queries can handle phrases more accurately. Then this obtained low-level match information is used to score each individual fragment. The scoring method of the plain highlighter is quite simple. Each fragment is scored by the number of unique query terms found in this fragment. The score of individual term is equal to its boost, which is by default is 1. Thus, by default, a fragment that contains one unique query term, will get a score of 1; and a fragment that contains two unique query terms, will get a score of 2 and so on. The fragments are then sorted by their scores, so the highest scored fragments will be output first.

FVH doesn’t need to analyze the text and build an in-memory index, as it uses pre-indexed document term vectors, and finds among them terms that correspond to the query. FVH scores each fragment by the number of query terms found in this fragment. Similarly to plain highlighter, score of individual term is equal to its boost value. In contrast to plain highlighter, all query terms are counted, not only unique terms.

Unified highlighter can use pre-indexed term vectors or pre-indexed terms offsets, if they are available. Otherwise, similar to Plain Highlighter, it has to create an in-memory index from the text. Unified highlighter uses the BM25 scoring model to score fragments.

How to highlight the query terms in a fragment?

Relevant settings: pre-tags, post-tags.

The goal is to highlight only those terms that participated in generating the 'hit' on the document. For some complex boolean queries, this is still work in progress, as highlighters don’t reflect the boolean logic of a query and only extract leaf (terms, phrases, prefix etc) queries.

Plain highlighter given the token stream and the original text, recomposes the original text to highlight only terms from the token stream that are contained in the low-level match information structure from the previous step.

FVH and unified highlighter use intermediate data structures to represent fragments in some raw form, and then populate them with actual text.

A highlighter uses pre-tags, post-tags to encode highlighted terms.

An example of the work of the unified highlighter

Let’s look in more details how unified highlighter works.

First, we create a index with a text field content, that will be indexed using english analyzer, and will be indexed without offsets or term vectors.

PUT test_index
{
    "mappings": {
        "_doc": {
            "properties": {
                "content" : {
                    "type" : "text",
                    "analyzer" : "english"
                }
            }
        }
    }
}

We put the following document into the index:

PUT test_index/_doc/doc1
{
  "content" : "For you I'm only a fox like a hundred thousand other foxes. But if you tame me, we'll need each other. You'll be the only boy in the world for me. I'll be the only fox in the world for you."
}

And we ran the following query with a highlight request:

GET test_index/_search
{
    "query": {
        "match_phrase" : {"content" : "only fox"}
    },
    "highlight": {
        "type" : "unified",
        "number_of_fragments" : 3,
        "fields": {
            "content": {}
        }
    }
}

After doc1 is found as a hit for this query, this hit will be passed to the unified highlighter for highlighting the field content of the document. Since the field content was not indexed either with offsets or term vectors, its raw field value will be analyzed, and in-memory index will be built from the terms that match the query:

{"token":"onli","start_offset":12,"end_offset":16,"position":3},
{"token":"fox","start_offset":19,"end_offset":22,"position":5},
{"token":"fox","start_offset":53,"end_offset":58,"position":11},
{"token":"onli","start_offset":117,"end_offset":121,"position":24},
{"token":"onli","start_offset":159,"end_offset":163,"position":34},
{"token":"fox","start_offset":164,"end_offset":167,"position":35}

Our complex phrase query will be converted to the span query: spanNear([text:onli, text:fox], 0, true), meaning that we are looking for terms "onli: and "fox" within 0 distance from each other, and in the given order. The span query will be run against the created before in-memory index, to find the following match:

{"term":"onli", "start_offset":159, "end_offset":163},
{"term":"fox", "start_offset":164, "end_offset":167}

In our example, we have got a single match, but there could be several matches. Given the matches, the unified highlighter breaks the text of the field into so called "passages". Each passage must contain at least one match. The unified highlighter with the use of Java’s BreakIterator ensures that each passage represents a full sentence as long as it doesn’t exceed fragment_size. For our example, we have got a single passage with the following properties (showing only a subset of the properties here):

Passage:
    startOffset: 147
    endOffset: 189
    score: 3.7158387
    matchStarts: [159, 164]
    matchEnds: [163, 167]
    numMatches: 2

Notice how a passage has a score, calculated using the BM25 scoring formula adapted for passages. Scores allow us to choose the best scoring passages if there are more passages available than the requested by the user number_of_fragments. Scores also let us to sort passages by order: "score" if requested by the user.

As the final step, the unified highlighter will extract from the field’s text a string corresponding to each passage:

"I'll be the only fox in the world for you."

and will format with the tags <em> and </em> all matches in this string using the passages’s matchStarts and matchEnds information:

I'll be the <em>only</em> <em>fox</em> in the world for you.

This kind of formatted strings are the final result of the highlighter returned to the user.

Rescoring

Rescoring can help to improve precision by reordering just the top (eg 100 - 500) documents returned by the query and post_filter phases, using a secondary (usually more costly) algorithm, instead of applying the costly algorithm to all documents in the index.

A rescore request is executed on each shard before it returns its results to be sorted by the node handling the overall search request.

Currently the rescore API has only one implementation: the query rescorer, which uses a query to tweak the scoring. In the future, alternative rescorers may be made available, for example, a pair-wise rescorer.

Note
An error will be thrown if an explicit sort (other than _score in descending order) is provided with a rescore query.
Note
when exposing pagination to your users, you should not change window_size as you step through each page (by passing different from values) since that can alter the top hits causing results to confusingly shift as the user steps through pages.

Query rescorer

The query rescorer executes a second query only on the Top-K results returned by the query and post_filter phases. The number of docs which will be examined on each shard can be controlled by the window_size parameter, which defaults to 10.

By default the scores from the original query and the rescore query are combined linearly to produce the final _score for each document. The relative importance of the original query and of the rescore query can be controlled with the query_weight and rescore_query_weight respectively. Both default to 1.

For example:

POST /_search
{
   "query" : {
      "match" : {
         "message" : {
            "operator" : "or",
            "query" : "the quick brown"
         }
      }
   },
   "rescore" : {
      "window_size" : 50,
      "query" : {
         "rescore_query" : {
            "match_phrase" : {
               "message" : {
                  "query" : "the quick brown",
                  "slop" : 2
               }
            }
         },
         "query_weight" : 0.7,
         "rescore_query_weight" : 1.2
      }
   }
}

The way the scores are combined can be controlled with the score_mode:

Score Mode Description

total

Add the original score and the rescore query score. The default.

multiply

Multiply the original score by the rescore query score. Useful for function query rescores.

avg

Average the original score and the rescore query score.

max

Take the max of original score and the rescore query score.

min

Take the min of the original score and the rescore query score.

Multiple Rescores

It is also possible to execute multiple rescores in sequence:

POST /_search
{
   "query" : {
      "match" : {
         "message" : {
            "operator" : "or",
            "query" : "the quick brown"
         }
      }
   },
   "rescore" : [ {
      "window_size" : 100,
      "query" : {
         "rescore_query" : {
            "match_phrase" : {
               "message" : {
                  "query" : "the quick brown",
                  "slop" : 2
               }
            }
         },
         "query_weight" : 0.7,
         "rescore_query_weight" : 1.2
      }
   }, {
      "window_size" : 10,
      "query" : {
         "score_mode": "multiply",
         "rescore_query" : {
            "function_score" : {
               "script_score": {
                  "script": {
                    "source": "Math.log10(doc.likes.value + 2)"
                  }
               }
            }
         }
      }
   } ]
}

The first one gets the results of the query then the second one gets the results of the first, etc. The second rescore will "see" the sorting done by the first rescore so it is possible to use a large window on the first rescore to pull documents into a smaller window for the second rescore.

Search Type

There are different execution paths that can be done when executing a distributed search. The distributed search operation needs to be scattered to all the relevant shards and then all the results are gathered back. When doing scatter/gather type execution, there are several ways to do that, specifically with search engines.

One of the questions when executing a distributed search is how much results to retrieve from each shard. For example, if we have 10 shards, the 1st shard might hold the most relevant results from 0 till 10, with other shards results ranking below it. For this reason, when executing a request, we will need to get results from 0 till 10 from all shards, sort them, and then return the results if we want to ensure correct results.

Another question, which relates to the search engine, is the fact that each shard stands on its own. When a query is executed on a specific shard, it does not take into account term frequencies and other search engine information from the other shards. If we want to support accurate ranking, we would need to first gather the term frequencies from all shards to calculate global term frequencies, then execute the query on each shard using these global frequencies.

Also, because of the need to sort the results, getting back a large document set, or even scrolling it, while maintaining the correct sorting behavior can be a very expensive operation. For large result set scrolling, it is best to sort by _doc if the order in which documents are returned is not important.

Elasticsearch is very flexible and allows to control the type of search to execute on a per search request basis. The type can be configured by setting the search_type parameter in the query string. The types are:

Query Then Fetch

Parameter value: query_then_fetch.

The request is processed in two phases. In the first phase, the query is forwarded to all involved shards. Each shard executes the search and generates a sorted list of results, local to that shard. Each shard returns just enough information to the coordinating node to allow it to merge and re-sort the shard level results into a globally sorted set of results, of maximum length size.

During the second phase, the coordinating node requests the document content (and highlighted snippets, if any) from only the relevant shards.

Note
This is the default setting, if you do not specify a search_type in your request.

Dfs, Query Then Fetch

Parameter value: dfs_query_then_fetch.

Same as "Query Then Fetch", except for an initial scatter phase which goes and computes the distributed term frequencies for more accurate scoring.

Scroll

While a search request returns a single `page'' of results, the `scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database.

Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration.

Client support for scrolling and reindexing

Some of the officially supported clients provide helpers to assist with scrolled searches and reindexing of documents from one index to another:

Note
The results that are returned from a scroll request reflect the state of the index at the time that the initial search request was made, like a snapshot in time. Subsequent changes to documents (index, update or delete) will only affect later search requests.

In order to use scrolling, the initial search request should specify the scroll parameter in the query string, which tells Elasticsearch how long it should keep the `search context'' alive (see Keeping the search context alive), eg `?scroll=1m.

POST /twitter/_search?scroll=1m
{
    "size": 100,
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}

The result from the above request includes a _scroll_id, which should be passed to the scroll API in order to retrieve the next batch of results.

POST /_search/scroll (1)
{
    "scroll" : "1m", (2)
    "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==" (3)
}
  1. GET or POST can be used and the URL should not include the index name — this is specified in the original search request instead.

  2. The scroll parameter tells Elasticsearch to keep the search context open for another 1m.

  3. The scroll_id parameter

The size parameter allows you to configure the maximum number of hits to be returned with each batch of results. Each call to the scroll API returns the next batch of results until there are no more results left to return, ie the hits array is empty.

Important
The initial search request and each subsequent scroll request each return a _scroll_id. While the _scroll_id may change between requests, it doesn’t always change — in any case, only the most recently received _scroll_id should be used.
Note
If the request specifies aggregations, only the initial search response will contain the aggregations results.
Note
Scroll requests have optimizations that make them faster when the sort order is _doc. If you want to iterate over all documents regardless of the order, this is the most efficient option:
GET /_search?scroll=1m
{
  "sort": [
    "_doc"
  ]
}

Keeping the search context alive

The scroll parameter (passed to the search request and to every scroll request) tells Elasticsearch how long it should keep the search context alive. Its value (e.g. 1m, see Time units) does not need to be long enough to process all data — it just needs to be long enough to process the previous batch of results. Each scroll request (with the scroll parameter) sets a new expiry time. If a scroll request doesn’t pass in the scroll parameter, then the search context will be freed as part of that scroll request.

Normally, the background merge process optimizes the index by merging together smaller segments to create new bigger segments, at which time the smaller segments are deleted. This process continues during scrolling, but an open search context prevents the old segments from being deleted while they are still in use. This is how Elasticsearch is able to return the results of the initial search request, regardless of subsequent changes to documents.

Tip
Keeping older segments alive means that more file handles are needed. Ensure that you have configured your nodes to have ample free file handles. See File Descriptors.
Note
To prevent against issues caused by having too many scrolls open, you can limit the number of open scrolls per node with the search.max_open_scroll_context cluster setting (defaults to unlimited).

You can check how many search contexts are open with the nodes stats API:

GET /_nodes/stats/indices/search

Clear scroll API

Search context are automatically removed when the scroll timeout has been exceeded. However keeping scrolls open has a cost, as discussed in the previous section so scrolls should be explicitly cleared as soon as the scroll is not being used anymore using the clear-scroll API:

DELETE /_search/scroll
{
    "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ=="
}

Multiple scroll IDs can be passed as array:

DELETE /_search/scroll
{
    "scroll_id" : [
      "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==",
      "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAABFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAAAxZrUllkUVlCa1NqNmRMaUhiQlZkMWFBAAAAAAAAAAIWa1JZZFFZQmtTajZkTGlIYkJWZDFhQQAAAAAAAAAFFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAABBZrUllkUVlCa1NqNmRMaUhiQlZkMWFB"
    ]
}

All search contexts can be cleared with the _all parameter:

DELETE /_search/scroll/_all

The scroll_id can also be passed as a query string parameter or in the request body. Multiple scroll IDs can be passed as comma separated values:

DELETE /_search/scroll/DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==,DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAABFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAAAxZrUllkUVlCa1NqNmRMaUhiQlZkMWFBAAAAAAAAAAIWa1JZZFFZQmtTajZkTGlIYkJWZDFhQQAAAAAAAAAFFmtSWWRRWUJrU2o2ZExpSGJCVmQxYUEAAAAAAAAABBZrUllkUVlCa1NqNmRMaUhiQlZkMWFB

Sliced Scroll

For scroll queries that return a lot of documents it is possible to split the scroll in multiple slices which can be consumed independently:

GET /twitter/_search?scroll=1m
{
    "slice": {
        "id": 0, (1)
        "max": 2 (2)
    },
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}
GET /twitter/_search?scroll=1m
{
    "slice": {
        "id": 1,
        "max": 2
    },
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}
  1. The id of the slice

  2. The maximum number of slices

The result from the first request returned documents that belong to the first slice (id: 0) and the result from the second request returned documents that belong to the second slice. Since the maximum number of slices is set to 2 the union of the results of the two requests is equivalent to the results of a scroll query without slicing. By default the splitting is done on the shards first and then locally on each shard using the _uid field with the following formula: slice(doc) = floorMod(hashCode(doc._uid), max) For instance if the number of shards is equal to 2 and the user requested 4 slices then the slices 0 and 2 are assigned to the first shard and the slices 1 and 3 are assigned to the second shard.

Each scroll is independent and can be processed in parallel like any scroll request.

Note
If the number of slices is bigger than the number of shards the slice filter is very slow on the first calls, it has a complexity of O(N) and a memory cost equals to N bits per slice where N is the total number of documents in the shard. After few calls the filter should be cached and subsequent calls should be faster but you should limit the number of sliced query you perform in parallel to avoid the memory explosion.

To avoid this cost entirely it is possible to use the doc_values of another field to do the slicing but the user must ensure that the field has the following properties:

  • The field is numeric.

  • doc_values are enabled on that field

  • Every document should contain a single value. If a document has multiple values for the specified field, the first value is used.

  • The value for each document should be set once when the document is created and never updated. This ensures that each slice gets deterministic results.

  • The cardinality of the field should be high. This ensures that each slice gets approximately the same amount of documents.

GET /twitter/_search?scroll=1m
{
    "slice": {
        "field": "date",
        "id": 0,
        "max": 10
    },
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}

For append only time-based indices, the timestamp field can be used safely.

Note
By default the maximum number of slices allowed per scroll is limited to 1024. You can update the index.max_slices_per_scroll index setting to bypass this limit.

Preference

Controls a preference of which shard copies on which to execute the search. By default, Elasticsearch selects from the available shard copies in an unspecified order, taking the allocation awareness and adaptive replica selection configuration into account. However, it may sometimes be desirable to try and route certain searches to certain sets of shard copies, for instance to make better use of per-copy caches.

The preference is a query string parameter which can be set to:

_primary

The operation will be executed only on primary shards. deprecated:[6.1.0, "will be removed in 7.0. See the warning below for more information."]

_primary_first

The operation will be executed on primary shards if possible, but will fall back to other shards if not. deprecated:[6.1.0, "will be removed in 7.0. See the warning below for more information."]

_replica

The operation will be executed only on replica shards. If there are multiple replicas then the order of preference between them is unspecified. deprecated:[6.1.0, "will be removed in 7.0. See the warning below for more information."]

_replica_first

The operation will be executed on replica shards if possible, but will fall back to other shards if not. If there are multiple replicas then the order of preference between them is unspecified. deprecated:[6.1.0, "will be removed in 7.0. See the warning below for more information."]

_only_local

The operation will be executed only on shards allocated to the local node.

_local

The operation will be executed on shards allocated to the local node if possible, and will fall back to other shards if not.

_prefer_nodes:abc,xyz

The operation will be executed on nodes with one of the provided node ids (abc or xyz in this case) if possible. If suitable shard copies exist on more than one of the selected nodes then the order of preference between these copies is unspecified.

_shards:2,3

Restricts the operation to the specified shards. (2 and 3 in this case). This preference can be combined with other preferences but it has to appear first: _shards:2,3|_local

_only_nodes:abc*,x*yz,…​

Restricts the operation to nodes specified according to the node specification. If suitable shard copies exist on more than one of the selected nodes then the order of preference between these copies is unspecified.

Custom (string) value

Any value that does not start with _. If two searches both give the same custom string value for their preference and the underlying cluster state does not change then the same ordering of shards will be used for the searches. This does not guarantee that the exact same shards will be used each time: the cluster state, and therefore the selected shards, may change for a number of reasons including shard relocations and shard failures, and nodes may sometimes reject searches causing fallbacks to alternative nodes. However, in practice the ordering of shards tends to remain stable for long periods of time. A good candidate for a custom preference value is something like the web session id or the user name.

For instance, use the user’s session ID xyzabc123 as follows:

GET /_search?preference=xyzabc123
{
    "query": {
        "match": {
            "title": "elasticsearch"
        }
    }
}
Note
The only_local preference guarantees only to use shard copies on the local node, which is sometimes useful for troubleshooting. All other options do not _fully guarantee that any particular shard copies are used in a search, and on a changing index this may mean that repeated searches may yield different results if they are executed on different shard copies which are in different refresh states.
Warning
The _primary, _primary_first, _replica and _replica_first are deprecated as their use is not recommended. They do not help to avoid inconsistent results that arise from the use of shards that have different refresh states, and Elasticsearch uses synchronous replication so the primary does not in general hold fresher data than its replicas. The _primary_first and _replica_first preferences silently fall back to non-preferred copies if it is not possible to search the preferred copies. The _primary and _replica preferences will silently change their preferred shards if a replica is promoted to primary, which can happen at any time. The _primary preference can also put undesirable extra load on the primary shards. The cache-related benefits of these options can also be obtained using _only_nodes, _prefer_nodes, or a custom string value instead.

Explain

Enables explanation for each hit on how its score was computed.

GET /_search
{
    "explain": true,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Sequence Numbers and Primary Term

Returns the sequence number and primary term of the last modification to each search hit. See Optimistic concurrency control for more details.

GET /_search
{
    "seq_no_primary_term": true,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Version

Returns a version for each search hit.

GET /_search
{
    "version": true,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Index Boost

Allows to configure different boost level per index when searching across more than one indices. This is very handy when hits coming from one index matter more than hits coming from another index (think social graph where each user has an index).

deprecated[5.2.0, This format is deprecated. Please use array format instead.]

GET /_search
{
    "indices_boost" : {
        "index1" : 1.4,
        "index2" : 1.3
    }
}

You can also specify it as an array to control the order of boosts.

GET /_search
{
    "indices_boost" : [
        { "alias1" : 1.4 },
        { "index*" : 1.3 }
    ]
}

This is important when you use aliases or wildcard expression. If multiple matches are found, the first match will be used. For example, if an index is included in both alias1 and index*, boost value of 1.4 is applied.

min_score

Exclude documents which have a _score less than the minimum specified in min_score:

GET /_search
{
    "min_score": 0.5,
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

Note, most times, this does not make much sense, but is provided for advanced use cases.

Named Queries

Each filter and query can accept a _name in its top level definition.

GET /_search
{
    "query": {
        "bool" : {
            "should" : [
                {"match" : { "name.first" : {"query" : "shay", "_name" : "first"} }},
                {"match" : { "name.last" : {"query" : "banon", "_name" : "last"} }}
            ],
            "filter" : {
                "terms" : {
                    "name.last" : ["banon", "kimchy"],
                    "_name" : "test"
                }
            }
        }
    }
}

The search response will include for each hit the matched_queries it matched on. The tagging of queries and filters only make sense for the bool query.

Inner hits

The parent-join and nested features allow the return of documents that have matches in a different scope. In the parent/child case, parent documents are returned based on matches in child documents or child documents are returned based on matches in parent documents. In the nested case, documents are returned based on matches in nested inner objects.

In both cases, the actual matches in the different scopes that caused a document to be returned are hidden. In many cases, it’s very useful to know which inner nested objects (in the case of nested) or children/parent documents (in the case of parent/child) caused certain information to be returned. The inner hits feature can be used for this. This feature returns per search hit in the search response additional nested hits that caused a search hit to match in a different scope.

Inner hits can be used by defining an inner_hits definition on a nested, has_child or has_parent query and filter. The structure looks like this:

"<query>" : {
    "inner_hits" : {
        <inner_hits_options>
    }
}

If inner_hits is defined on a query that supports it then each search hit will contain an inner_hits json object with the following structure:

"hits": [
     {
        "_index": ...,
        "_type": ...,
        "_id": ...,
        "inner_hits": {
           "<inner_hits_name>": {
              "hits": {
                 "total": ...,
                 "hits": [
                    {
                       "_type": ...,
                       "_id": ...,
                       ...
                    },
                    ...
                 ]
              }
           }
        },
        ...
     },
     ...
]

Options

Inner hits support the following options:

from

The offset from where the first hit to fetch for each inner_hits in the returned regular search hits.

size

The maximum number of hits to return per inner_hits. By default the top three matching hits are returned.

sort

How the inner hits should be sorted per inner_hits. By default the hits are sorted by the score.

name

The name to be used for the particular inner hit definition in the response. Useful when multiple inner hits have been defined in a single search request. The default depends in which query the inner hit is defined. For has_child query and filter this is the child type, has_parent query and filter this is the parent type and the nested query and filter this is the nested path.

Inner hits also supports the following per document features:

Nested inner hits

The nested inner_hits can be used to include nested inner objects as inner hits to a search hit.

PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "comments": {
          "type": "nested"
        }
      }
    }
  }
}

PUT test/_doc/1?refresh
{
  "title": "Test title",
  "comments": [
    {
      "author": "kimchy",
      "number": 1
    },
    {
      "author": "nik9000",
      "number": 2
    }
  ]
}

POST test/_search
{
  "query": {
    "nested": {
      "path": "comments",
      "query": {
        "match": {"comments.number" : 2}
      },
      "inner_hits": {} (1)
    }
  }
}
  1. The inner hit definition in the nested query. No other options need to be defined.

An example of a response snippet that could be generated from the above search request:

{
  ...,
  "hits": {
    "total": 1,
    "max_score": 1.0,
    "hits": [
      {
        "_index": "test",
        "_type": "_doc",
        "_id": "1",
        "_score": 1.0,
        "_source": ...,
        "inner_hits": {
          "comments": { (1)
            "hits": {
              "total": 1,
              "max_score": 1.0,
              "hits": [
                {
                  "_index": "test",
                  "_type": "_doc",
                  "_id": "1",
                  "_nested": {
                    "field": "comments",
                    "offset": 1
                  },
                  "_score": 1.0,
                  "_source": {
                    "author": "nik9000",
                    "number": 2
                  }
                }
              ]
            }
          }
        }
      }
    ]
  }
}
  1. The name used in the inner hit definition in the search request. A custom key can be used via the name option.

The _nested metadata is crucial in the above example, because it defines from what inner nested object this inner hit came from. The field defines the object array field the nested hit is from and the offset relative to its location in the _source. Due to sorting and scoring the actual location of the hit objects in the inner_hits is usually different than the location a nested inner object was defined.

By default the _source is returned also for the hit objects in inner_hits, but this can be changed. Either via _source filtering feature part of the source can be returned or be disabled. If stored fields are defined on the nested level these can also be returned via the fields feature.

An important default is that the _source returned in hits inside inner_hits is relative to the _nested metadata. So in the above example only the comment part is returned per nested hit and not the entire source of the top level document that contained the comment.

Nested inner hits and _source

Nested document don’t have a _source field, because the entire source of document is stored with the root document under its _source field. To include the source of just the nested document, the source of the root document is parsed and just the relevant bit for the nested document is included as source in the inner hit. Doing this for each matching nested document has an impact on the time it takes to execute the entire search request, especially when size and the inner hits' size are set higher than the default. To avoid the relatively expensive source extraction for nested inner hits, one can disable including the source and solely rely on doc values fields. Like this:

PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "comments": {
          "type": "nested"
        }
      }
    }
  }
}

PUT test/_doc/1?refresh
{
  "title": "Test title",
  "comments": [
    {
      "author": "kimchy",
      "text": "comment text"
    },
    {
      "author": "nik9000",
      "text": "words words words"
    }
  ]
}

POST test/_search
{
  "query": {
    "nested": {
      "path": "comments",
      "query": {
        "match": {"comments.text" : "words"}
      },
      "inner_hits": {
        "_source" : false,
        "docvalue_fields" : [
          {
            "field": "comments.text.keyword",
            "format": "use_field_mapping"
          }
        ]
      }
    }
  }
}

Hierarchical levels of nested object fields and inner hits.

If a mapping has multiple levels of hierarchical nested object fields each level can be accessed via dot notated path. For example if there is a comments nested field that contains a votes nested field and votes should directly be returned with the root hits then the following path can be defined:

PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "comments": {
          "type": "nested",
          "properties": {
            "votes": {
              "type": "nested"
            }
          }
        }
      }
    }
  }
}

PUT test/_doc/1?refresh
{
  "title": "Test title",
  "comments": [
    {
      "author": "kimchy",
      "text": "comment text",
      "votes": []
    },
    {
      "author": "nik9000",
      "text": "words words words",
      "votes": [
        {"value": 1 , "voter": "kimchy"},
        {"value": -1, "voter": "other"}
      ]
    }
  ]
}

POST test/_search
{
  "query": {
    "nested": {
      "path": "comments.votes",
        "query": {
          "match": {
            "comments.votes.voter": "kimchy"
          }
        },
        "inner_hits" : {}
    }
  }
}

Which would look like:

{
  ...,
  "hits": {
    "total": 1,
    "max_score": 0.6931472,
    "hits": [
      {
        "_index": "test",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.6931472,
        "_source": ...,
        "inner_hits": {
          "comments.votes": { (1)
            "hits": {
              "total": 1,
              "max_score": 0.6931472,
              "hits": [
                {
                  "_index": "test",
                  "_type": "_doc",
                  "_id": "1",
                  "_nested": {
                    "field": "comments",
                    "offset": 1,
                    "_nested": {
                      "field": "votes",
                      "offset": 0
                    }
                  },
                  "_score": 0.6931472,
                  "_source": {
                    "value": 1,
                    "voter": "kimchy"
                  }
                }
              ]
            }
          }
        }
      }
    ]
  }
}

This indirect referencing is only supported for nested inner hits.

Parent/child inner hits

The parent/child inner_hits can be used to include parent or child:

PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_join_field": {
          "type": "join",
          "relations": {
            "my_parent": "my_child"
          }
        }
      }
    }
  }
}

PUT test/_doc/1?refresh
{
  "number": 1,
  "my_join_field": "my_parent"
}

PUT test/_doc/2?routing=1&refresh
{
  "number": 1,
  "my_join_field": {
    "name": "my_child",
    "parent": "1"
  }
}

POST test/_search
{
  "query": {
    "has_child": {
      "type": "my_child",
      "query": {
        "match": {
          "number": 1
        }
      },
      "inner_hits": {}    (1)
    }
  }
}
  1. The inner hit definition like in the nested example.

An example of a response snippet that could be generated from the above search request:

{
    ...,
    "hits": {
        "total": 1,
        "max_score": 1.0,
        "hits": [
            {
                "_index": "test",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "number": 1,
                    "my_join_field": "my_parent"
                },
                "inner_hits": {
                    "my_child": {
                        "hits": {
                            "total": 1,
                            "max_score": 1.0,
                            "hits": [
                                {
                                    "_index": "test",
                                    "_type": "_doc",
                                    "_id": "2",
                                    "_score": 1.0,
                                    "_routing": "1",
                                    "_source": {
                                        "number": 1,
                                        "my_join_field": {
                                            "name": "my_child",
                                            "parent": "1"
                                        }
                                    }
                                }
                            ]
                        }
                    }
                }
            }
        ]
    }
}

Field Collapsing

Allows to collapse search results based on field values. The collapsing is done by selecting only the top sorted document per collapse key. For instance the query below retrieves the best tweet for each user and sorts them by number of likes.

GET /twitter/_search
{
    "query": {
        "match": {
            "message": "elasticsearch"
        }
    },
    "collapse" : {
        "field" : "user" (1)
    },
    "sort": ["likes"], (2)
    "from": 10 (3)
}
  1. collapse the result set using the "user" field

  2. sort the top docs by number of likes

  3. define the offset of the first collapsed result

Warning
The total number of hits in the response indicates the number of matching documents without collapsing. The total number of distinct group is unknown.

The field used for collapsing must be a single valued keyword or numeric field with doc_values activated

Note
The collapsing is applied to the top hits only and does not affect aggregations.

Expand collapse results

It is also possible to expand each collapsed top hits with the inner_hits option.

GET /twitter/_search
{
    "query": {
        "match": {
            "message": "elasticsearch"
        }
    },
    "collapse" : {
        "field" : "user", (1)
        "inner_hits": {
            "name": "last_tweets", (2)
            "size": 5, (3)
            "sort": [{ "date": "asc" }] (4)
        },
        "max_concurrent_group_searches": 4 (5)
    },
    "sort": ["likes"]
}
  1. collapse the result set using the "user" field

  2. the name used for the inner hit section in the response

  3. the number of inner_hits to retrieve per collapse key

  4. how to sort the document inside each group

  5. the number of concurrent requests allowed to retrieve the inner_hits` per group

See inner hits for the complete list of supported options and the format of the response.

It is also possible to request multiple inner_hits for each collapsed hit. This can be useful when you want to get multiple representations of the collapsed hits.

GET /twitter/_search
{
    "query": {
        "match": {
            "message": "elasticsearch"
        }
    },
    "collapse" : {
        "field" : "user", (1)
        "inner_hits": [
            {
                "name": "most_liked",  (2)
                "size": 3,
                "sort": ["likes"]
            },
            {
                "name": "most_recent", (3)
                "size": 3,
                "sort": [{ "date": "asc" }]
            }
        ]
    },
    "sort": ["likes"]
}
  1. collapse the result set using the "user" field

  2. return the three most liked tweets for the user

  3. return the three most recent tweets for the user

The expansion of the group is done by sending an additional query for each inner_hit request for each collapsed hit returned in the response. This can significantly slow things down if you have too many groups and/or inner_hit requests.

The max_concurrent_group_searches request parameter can be used to control the maximum number of concurrent searches allowed in this phase. The default is based on the number of data nodes and the default search thread pool size.

Warning
collapse cannot be used in conjunction with scroll, rescore or search after.

Second level of collapsing

Second level of collapsing is also supported and is applied to inner_hits. For example, the following request finds the top scored tweets for each country, and within each country finds the top scored tweets for each user.

GET /twitter/_search
{
    "query": {
        "match": {
            "message": "elasticsearch"
        }
    },
    "collapse" : {
        "field" : "country",
        "inner_hits" : {
            "name": "by_location",
            "collapse" : {"field" : "user"},
            "size": 3
        }
    }
}

Response:

{
    ...
    "hits": [
        {
            "_index": "twitter",
            "_type": "_doc",
            "_id": "9",
            "_score": ...,
            "_source": {...},
            "fields": {"country": ["UK"]},
            "inner_hits":{
                "by_location": {
                    "hits": {
                       ...,
                       "hits": [
                          {
                            ...
                            "fields": {"user" : ["user124"]}
                          },
                          {
                            ...
                            "fields": {"user" : ["user589"]}
                          },
                          {
                            ...
                             "fields": {"user" : ["user001"]}
                          }
                       ]
                    }
                 }
            }
        },
        {
            "_index": "twitter",
            "_type": "_doc",
            "_id": "1",
            "_score": ..,
            "_source": {...},
            "fields": {"country": ["Canada"]},
            "inner_hits":{
                "by_location": {
                    "hits": {
                       ...,
                       "hits": [
                          {
                            ...
                            "fields": {"user" : ["user444"]}
                          },
                          {
                            ...
                            "fields": {"user" : ["user1111"]}
                          },
                          {
                            ...
                             "fields": {"user" : ["user999"]}
                          }
                       ]
                    }
                 }
            }

        },
        ....
    ]
}
Note
Second level of collapsing doesn’t allow inner_hits.

Search After

Pagination of results can be done by using the from and size but the cost becomes prohibitive when the deep pagination is reached. The index.max_result_window which defaults to 10,000 is a safeguard, search requests take heap memory and time proportional to from + size. The Scroll api is recommended for efficient deep scrolling but scroll contexts are costly and it is not recommended to use it for real time user requests. The search_after parameter circumvents this problem by providing a live cursor. The idea is to use the results from the previous page to help the retrieval of the next page.

Suppose that the query to retrieve the first page looks like this:

GET twitter/_search
{
    "size": 10,
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    },
    "sort": [
        {"date": "asc"},
        {"tie_breaker_id": "asc"}      (1)
    ]
}
  1. A copy of the _id field with doc_values enabled

Important
A field with one unique value per document should be used as the tiebreaker of the sort specification. Otherwise the sort order for documents that have the same sort values would be undefined and could lead to missing or duplicate results. The _id field has a unique value per document but it is not recommended to use it as a tiebreaker directly. Beware that search_after looks for the first document which fully or partially matches tiebreaker’s provided value. Therefore if a document has a tiebreaker value of "654323" and you search_after for "654" it would still match that document and return results found after it. Doc values are disabled on this field so sorting on it requires to load a lot of data in memory. Instead it is advised to duplicate (client side or with a set ingest processor) the content of the _id field in another field that has doc value enabled and to use this new field as the tiebreaker for the sort.

The result from the above request includes an array of sort values for each document. These sort values can be used in conjunction with the search_after parameter to start returning results "after" any document in the result list. For instance we can use the sort values of the last document and pass it to search_after to retrieve the next page of results:

GET twitter/_search
{
    "size": 10,
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    },
    "search_after": [1463538857, "654323"],
    "sort": [
        {"date": "asc"},
        {"tie_breaker_id": "asc"}
    ]
}
Note
The parameter from must be set to 0 (or -1) when search_after is used.

search_after is not a solution to jump freely to a random page but rather to scroll many queries in parallel. It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher. For this reason the sort order may change during a walk depending on the updates and deletes of your index.

Search Template

The /_search/template endpoint allows to use the mustache language to pre render search requests, before they are executed and fill existing templates with template parameters.

GET _search/template
{
    "source" : {
      "query": { "match" : { "{{my_field}}" : "{{my_value}}" } },
      "size" : "{{my_size}}"
    },
    "params" : {
        "my_field" : "message",
        "my_value" : "some message",
        "my_size" : 5
    }
}

For more information on how Mustache templating and what kind of templating you can do with it check out the online documentation of the mustache project.

Note
The mustache language is implemented in Elasticsearch as a sandboxed scripting language, hence it obeys settings that may be used to enable or disable scripts per type and context as described in the scripting docs

Examples

Store a search template

You can store a search template using the stored scripts API.

POST _scripts/<templateid>
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}

This template can be retrieved by

GET _scripts/<templateid>

which is rendered as:

{
    "script" : {
        "lang" : "mustache",
        "source" : "{\"query\":{\"match\":{\"title\":\"{{query_string}}\"}}}",
        "options": {
          "content_type" : "application/json; charset=UTF-8"
        }
    },
    "_id": "<templateid>",
    "found": true
}

This template can be deleted by

DELETE _scripts/<templateid>
Use a stored search template

To use a stored template at search time use:

GET _search/template
{
    "id": "<templateid>", (1)
    "params": {
        "query_string": "search for these words"
    }
}
  1. Name of the stored template script.

Validate a search template

A template can be rendered in a response with given parameters using

GET _render/template
{
  "source": "{ \"query\": { \"terms\": {{#toJson}}statuses{{/toJson}} }}",
  "params": {
    "statuses" : {
        "status": [ "pending", "published" ]
    }
  }
}

This call will return the rendered template:

{
  "template_output": {
    "query": {
      "terms": {
        "status": [ (1)
          "pending",
          "published"
        ]
      }
    }
  }
}
  1. status array has been populated with values from the params object.

Stored templates can also be rendered using

GET _render/template/<template_name>
{
  "params": {
    "..."
  }
}
Explain

You can use explain parameter when running a template:

GET _search/template
{
  "id": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "explain": true
}
Profiling

You can use profile parameter when running a template:

GET _search/template
{
  "id": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "profile": true
}
Filling in a query string with a single value
GET _search/template
{
    "source": {
        "query": {
            "term": {
                "message": "{{query_string}}"
            }
        }
    },
    "params": {
        "query_string": "search for these words"
    }
}
Converting parameters to JSON

The {{#toJson}}parameter{{/toJson}} function can be used to convert parameters like maps and array to their JSON representation:

GET _search/template
{
  "source": "{ \"query\": { \"terms\": {{#toJson}}statuses{{/toJson}} }}",
  "params": {
    "statuses" : {
        "status": [ "pending", "published" ]
    }
  }
}

which is rendered as:

{
  "query": {
    "terms": {
      "status": [
        "pending",
        "published"
      ]
    }
  }
}

A more complex example substitutes an array of JSON objects:

GET _search/template
{
    "source": "{\"query\":{\"bool\":{\"must\": {{#toJson}}clauses{{/toJson}} }}}",
    "params": {
        "clauses": [
            { "term": { "user" : "foo" } },
            { "term": { "user" : "bar" } }
        ]
   }
}

which is rendered as:

{
    "query" : {
      "bool" : {
        "must" : [
          {
            "term" : {
                "user" : "foo"
            }
          },
          {
            "term" : {
                "user" : "bar"
            }
          }
        ]
      }
    }
}
Concatenating array of values

The {{#join}}array{{/join}} function can be used to concatenate the values of an array as a comma delimited string:

GET _search/template
{
  "source": {
    "query": {
      "match": {
        "emails": "{{#join}}emails{{/join}}"
      }
    }
  },
  "params": {
    "emails": [ "username@email.com", "lastname@email.com" ]
  }
}

which is rendered as:

{
    "query" : {
        "match" : {
            "emails" : "username@email.com,lastname@email.com"
        }
    }
}

The function also accepts a custom delimiter:

GET _search/template
{
  "source": {
    "query": {
      "range": {
        "born": {
            "gte"   : "{{date.min}}",
            "lte"   : "{{date.max}}",
            "format": "{{#join delimiter='||'}}date.formats{{/join delimiter='||'}}"
	    }
      }
    }
  },
  "params": {
    "date": {
        "min": "2016",
        "max": "31/12/2017",
        "formats": ["dd/MM/yyyy", "yyyy"]
    }
  }
}

which is rendered as:

{
    "query" : {
      "range" : {
        "born" : {
          "gte" : "2016",
          "lte" : "31/12/2017",
          "format" : "dd/MM/yyyy||yyyy"
        }
      }
    }
}
Default values

A default value is written as {{var}}{{^var}}default{{/var}} for instance:

{
  "source": {
    "query": {
      "range": {
        "line_no": {
          "gte": "{{start}}",
          "lte": "{{end}}{{^end}}20{{/end}}"
        }
      }
    }
  },
  "params": { ... }
}

When params is { "start": 10, "end": 15 } this query would be rendered as:

{
    "range": {
        "line_no": {
            "gte": "10",
            "lte": "15"
        }
  }
}

But when params is { "start": 10 } this query would use the default value for end:

{
    "range": {
        "line_no": {
            "gte": "10",
            "lte": "20"
        }
    }
}
Conditional clauses

Conditional clauses cannot be expressed using the JSON form of the template. Instead, the template must be passed as a string. For instance, let’s say we wanted to run a match query on the line field, and optionally wanted to filter by line numbers, where start and end are optional.

The params would look like:

{
    "params": {
        "text":      "words to search for",
        "line_no": { (1)
            "start": 10,
            "end":   20
        }
    }
}
  1. The line_no, start, and end parameters are optional.

We could write the query as:

{
  "query": {
    "bool": {
      "must": {
        "match": {
          "line": "{{text}}" (1)
        }
      },
      "filter": {
        {{#line_no}} (2)
          "range": {
            "line_no": {
              {{#start}} (3)
                "gte": "{{start}}" (4)
                {{#end}},{{/end}} (5)
              {{/start}}
              {{#end}} (6)
                "lte": "{{end}}" (7)
              {{/end}}
            }
          }
        {{/line_no}}
      }
    }
  }
}
  1. Fill in the value of param text

  2. Include the range filter only if line_no is specified

  3. Include the gte clause only if line_no.start is specified

  4. Fill in the value of param line_no.start

  5. Add a comma after the gte clause only if line_no.start AND line_no.end are specified

  6. Include the lte clause only if line_no.end is specified

  7. Fill in the value of param line_no.end

Note

As written above, this template is not valid JSON because it includes the section markers like {{#line_no}}. For this reason, the template should either be stored in a file (see Store a search template) or, when used via the REST API, should be written as a string:

"source": "{\"query\":{\"bool\":{\"must\":{\"match\":{\"line\":\"{{text}}\"}},\"filter\":{{{#line_no}}\"range\":{\"line_no\":{{{#start}}\"gte\":\"{{start}}\"{{#end}},{{/end}}{{/start}}{{#end}}\"lte\":\"{{end}}\"{{/end}}}}{{/line_no}}}}}}"
Encoding URLs

The {{#url}}value{{/url}} function can be used to encode a string value in a HTML encoding form as defined in by the HTML specification.

As an example, it is useful to encode a URL:

GET _render/template
{
    "source" : {
        "query" : {
            "term": {
                "http_access_log": "{{#url}}{{host}}/{{page}}{{/url}}"
            }
        }
    },
    "params": {
        "host": "https://www.elastic.co/",
        "page": "learn"
    }
}

The previous query will be rendered as:

{
    "template_output" : {
        "query" : {
            "term" : {
                "http_access_log" : "https%3A%2F%2Fwww.elastic.co%2F%2Flearn"
            }
        }
    }
}

Multi Search Template

The multi search template API allows to execute several search template requests within the same API using the _msearch/template endpoint.

The format of the request is similar to the Multi Search API format:

header\n
body\n
header\n
body\n

The header part supports the same index, search_type, preference, and routing options as the usual Multi Search API.

The body includes a search template body request and supports inline, stored and file templates. Here is an example:

$ cat requests
{"index": "test"}
{"source": {"query": {"match":  {"user" : "{{username}}" }}}, "params": {"username": "john"}} (1)
{"source": {"query": {"{{query_type}}": {"name": "{{name}}" }}}, "params": {"query_type": "match_phrase_prefix", "name": "Smith"}}
{"index": "_all"}
{"id": "template_1", "params": {"query_string": "search for these words" }} (2)

$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/_msearch/template --data-binary "@requests"; echo
  1. Inline search template request

  2. Search template request based on a stored template

The response returns a responses array, which includes the search template response for each search template request matching its order in the original multi search template request. If there was a complete failure for that specific search template request, an object with error message will be returned in place of the actual search response.

Search Shards API

The search shards api returns the indices and shards that a search request would be executed against. This can give useful feedback for working out issues or planning optimizations with routing and shard preferences. When filtered aliases are used, the filter is returned as part of the indices section [5.1.0] Added in 5.1.0.

The index may be a single value, or comma-separated.

Usage

Full example:

GET /twitter/_search_shards

This will yield the following result:

{
  "nodes": ...,
  "indices" : {
    "twitter": { }
  },
  "shards": [
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 0,
        "state": "STARTED",
        "allocation_id": {"id":"0TvkCyF7TAmM1wHP4a42-A"},
        "relocating_node": null
      }
    ],
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 1,
        "state": "STARTED",
        "allocation_id": {"id":"fMju3hd1QHWmWrIgFnI4Ww"},
        "relocating_node": null
      }
    ],
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 2,
        "state": "STARTED",
        "allocation_id": {"id":"Nwl0wbMBTHCWjEEbGYGapg"},
        "relocating_node": null
      }
    ],
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 3,
        "state": "STARTED",
        "allocation_id": {"id":"bU_KLGJISbW0RejwnwDPKw"},
        "relocating_node": null
      }
    ],
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 4,
        "state": "STARTED",
        "allocation_id": {"id":"DMs7_giNSwmdqVukF7UydA"},
        "relocating_node": null
      }
    ]
  ]
}

And specifying the same request, this time with a routing value:

GET /twitter/_search_shards?routing=foo,baz

This will yield the following result:

{
  "nodes": ...,
  "indices" : {
      "twitter": { }
  },
  "shards": [
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 0,
        "state": "STARTED",
        "allocation_id": {"id":"0TvkCyF7TAmM1wHP4a42-A"},
        "relocating_node": null
      }
    ],
    [
      {
        "index": "twitter",
        "node": "JklnKbD7Tyqi9TP3_Q_tBg",
        "primary": true,
        "shard": 1,
        "state": "STARTED",
        "allocation_id": {"id":"fMju3hd1QHWmWrIgFnI4Ww"},
        "relocating_node": null
      }
    ]
  ]
}

This time the search will only be executed against two of the shards, because routing values have been specified.

All parameters:

routing

A comma-separated list of routing values to take into account when determining which shards a request would be executed against.

preference

Controls a preference of which shard replicas to execute the search request on. By default, the operation is randomized between the shard replicas. See the preference documentation for a list of all acceptable values.

local

A boolean value whether to read the cluster state locally in order to determine where shards are allocated instead of using the Master node’s cluster state.

Suggesters

The suggest feature suggests similar looking terms based on a provided text by using a suggester. Parts of the suggest feature are still under development.

The suggest request part is defined alongside the query part in a _search request. If the query part is left out, only suggestions are returned.

Note
_suggest endpoint has been deprecated in favour of using suggest via _search endpoint. In 5.0, the _search endpoint has been optimized for suggest only search requests.
POST twitter/_search
{
  "query" : {
    "match": {
      "message": "tring out Elasticsearch"
    }
  },
  "suggest" : {
    "my-suggestion" : {
      "text" : "trying out Elasticsearch",
      "term" : {
        "field" : "message"
      }
    }
  }
}

Several suggestions can be specified per request. Each suggestion is identified with an arbitrary name. In the example below two suggestions are requested. Both my-suggest-1 and my-suggest-2 suggestions use the term suggester, but have a different text.

POST _search
{
  "suggest": {
    "my-suggest-1" : {
      "text" : "tring out Elasticsearch",
      "term" : {
        "field" : "message"
      }
    },
    "my-suggest-2" : {
      "text" : "kmichy",
      "term" : {
        "field" : "user"
      }
    }
  }
}

The below suggest response example includes the suggestion response for my-suggest-1 and my-suggest-2. Each suggestion part contains entries. Each entry is effectively a token from the suggest text and contains the suggestion entry text, the original start offset and length in the suggest text and if found an arbitrary number of options.

{
  "_shards": ...
  "hits": ...
  "took": 2,
  "timed_out": false,
  "suggest": {
    "my-suggest-1": [ {
      "text": "tring",
      "offset": 0,
      "length": 5,
      "options": [ {"text": "trying", "score": 0.8, "freq": 1 } ]
    }, {
      "text": "out",
      "offset": 6,
      "length": 3,
      "options": []
    }, {
      "text": "elasticsearch",
      "offset": 10,
      "length": 13,
      "options": []
    } ],
    "my-suggest-2": ...
  }
}

Each options array contains an option object that includes the suggested text, its document frequency and score compared to the suggest entry text. The meaning of the score depends on the used suggester. The term suggester’s score is based on the edit distance.

Global suggest text

To avoid repetition of the suggest text, it is possible to define a global text. In the example below the suggest text is defined globally and applies to the my-suggest-1 and my-suggest-2 suggestions.

POST _search
{
  "suggest": {
    "text" : "tring out Elasticsearch",
    "my-suggest-1" : {
      "term" : {
        "field" : "message"
      }
    },
    "my-suggest-2" : {
       "term" : {
        "field" : "user"
       }
    }
  }
}

The suggest text can in the above example also be specified as suggestion specific option. The suggest text specified on suggestion level override the suggest text on the global level.

Term suggester

Note
In order to understand the format of suggestions, please read the Suggesters page first.

The term suggester suggests terms based on edit distance. The provided suggest text is analyzed before terms are suggested. The suggested terms are provided per analyzed suggest text token. The term suggester doesn’t take the query into account that is part of request.

Common suggest options:

text

The suggest text. The suggest text is a required option that needs to be set globally or per suggestion.

field

The field to fetch the candidate suggestions from. This is a required option that either needs to be set globally or per suggestion.

analyzer

The analyzer to analyse the suggest text with. Defaults to the search analyzer of the suggest field.

size

The maximum corrections to be returned per suggest text token.

sort

Defines how suggestions should be sorted per suggest text term. Two possible values:

  • score: Sort by score first, then document frequency and then the term itself.

  • frequency: Sort by document frequency first, then similarity score and then the term itself.

suggest_mode

The suggest mode controls what suggestions are included or controls for what suggest text terms, suggestions should be suggested. Three possible values can be specified:

  • missing: Only provide suggestions for suggest text terms that are not in the index. This is the default.

  • popular: Only suggest suggestions that occur in more docs than the original suggest text term.

  • always: Suggest any matching suggestions based on terms in the suggest text.

Other term suggest options:

max_edits

The maximum edit distance candidate suggestions can have in order to be considered as a suggestion. Can only be a value between 1 and 2. Any other value results in a bad request error being thrown. Defaults to 2.

prefix_length

The number of minimal prefix characters that must match in order be a candidate for suggestions. Defaults to 1. Increasing this number improves spellcheck performance. Usually misspellings don’t occur in the beginning of terms. (Old name "prefix_len" is deprecated)

min_word_length

The minimum length a suggest text term must have in order to be included. Defaults to 4. (Old name "min_word_len" is deprecated)

shard_size

Sets the maximum number of suggestions to be retrieved from each individual shard. During the reduce phase only the top N suggestions are returned based on the size option. Defaults to the size option. Setting this to a value higher than the size can be useful in order to get a more accurate document frequency for spelling corrections at the cost of performance. Due to the fact that terms are partitioned amongst shards, the shard level document frequencies of spelling corrections may not be precise. Increasing this will make these document frequencies more precise.

max_inspections

A factor that is used to multiply with the shards_size in order to inspect more candidate spelling corrections on the shard level. Can improve accuracy at the cost of performance. Defaults to 5.

min_doc_freq

The minimal threshold in number of documents a suggestion should appear in. This can be specified as an absolute number or as a relative percentage of number of documents. This can improve quality by only suggesting high frequency terms. Defaults to 0f and is not enabled. If a value higher than 1 is specified, then the number cannot be fractional. The shard level document frequencies are used for this option.

max_term_freq

The maximum threshold in number of documents in which a suggest text token can exist in order to be included. Can be a relative percentage number (e.g., 0.4) or an absolute number to represent document frequencies. If a value higher than 1 is specified, then fractional can not be specified. Defaults to 0.01f. This can be used to exclude high frequency terms — which are usually spelled correctly — from being spellchecked. This also improves the spellcheck performance. The shard level document frequencies are used for this option.

string_distance

Which string distance implementation to use for comparing how similar suggested terms are. Five possible values can be specified:

  • internal: The default based on damerau_levenshtein but highly optimized for comparing string distance for terms inside the index.

  • damerau_levenshtein: String distance algorithm based on Damerau-Levenshtein algorithm.

  • levenshtein: String distance algorithm based on Levenshtein edit distance algorithm.

  • jaro_winkler: String distance algorithm based on Jaro-Winkler algorithm.

  • ngram: String distance algorithm based on character n-grams.

Phrase Suggester

Note
In order to understand the format of suggestions, please read the Suggesters page first.

The term suggester provides a very convenient API to access word alternatives on a per token basis within a certain string distance. The API allows accessing each token in the stream individually while suggest-selection is left to the API consumer. Yet, often pre-selected suggestions are required in order to present to the end-user. The phrase suggester adds additional logic on top of the term suggester to select entire corrected phrases instead of individual tokens weighted based on ngram-language models. In practice this suggester will be able to make better decisions about which tokens to pick based on co-occurrence and frequencies.

API Example

In general the phrase suggester requires special mapping up front to work. The phrase suggester examples on this page need the following mapping to work. The reverse analyzer is used only in the last example.

PUT test
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "analysis": {
        "analyzer": {
          "trigram": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase","shingle"]
          },
          "reverse": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase","reverse"]
          }
        },
        "filter": {
          "shingle": {
            "type": "shingle",
            "min_shingle_size": 2,
            "max_shingle_size": 3
          }
        }
      }
    }
  },
  "mappings": {
    "_doc": {
      "properties": {
        "title": {
          "type": "text",
          "fields": {
            "trigram": {
              "type": "text",
              "analyzer": "trigram"
            },
            "reverse": {
              "type": "text",
              "analyzer": "reverse"
            }
          }
        }
      }
    }
  }
}
POST test/_doc?refresh=true
{"title": "noble warriors"}
POST test/_doc?refresh=true
{"title": "nobel prize"}

Once you have the analyzers and mappings set up you can use the phrase suggester in the same spot you’d use the term suggester:

POST test/_search
{
  "suggest": {
    "text": "noble prize",
    "simple_phrase": {
      "phrase": {
        "field": "title.trigram",
        "size": 1,
        "gram_size": 3,
        "direct_generator": [ {
          "field": "title.trigram",
          "suggest_mode": "always"
        } ],
        "highlight": {
          "pre_tag": "<em>",
          "post_tag": "</em>"
        }
      }
    }
  }
}

The response contains suggestions scored by the most likely spell correction first. In this case we received the expected correction "nobel prize".

{
  "_shards": ...
  "hits": ...
  "timed_out": false,
  "took": 3,
  "suggest": {
    "simple_phrase" : [
      {
        "text" : "noble prize",
        "offset" : 0,
        "length" : 11,
        "options" : [ {
          "text" : "nobel prize",
          "highlighted": "<em>nobel</em> prize",
          "score" : 0.5962314
        }]
      }
    ]
  }
}

Basic Phrase suggest API parameters

field

The name of the field used to do n-gram lookups for the language model, the suggester will use this field to gain statistics to score corrections. This field is mandatory.

gram_size

Sets max size of the n-grams (shingles) in the field. If the field doesn’t contain n-grams (shingles), this should be omitted or set to 1. Note that Elasticsearch tries to detect the gram size based on the specified field. If the field uses a shingle filter, the gram_size is set to the max_shingle_size if not explicitly set.

real_word_error_likelihood

The likelihood of a term being a misspelled even if the term exists in the dictionary. The default is 0.95, meaning 5% of the real words are misspelled.

confidence

The confidence level defines a factor applied to the input phrases score which is used as a threshold for other suggest candidates. Only candidates that score higher than the threshold will be included in the result. For instance a confidence level of 1.0 will only return suggestions that score higher than the input phrase. If set to 0.0 the top N candidates are returned. The default is 1.0.

max_errors

The maximum percentage of the terms considered to be misspellings in order to form a correction. This method accepts a float value in the range [0..1) as a fraction of the actual query terms or a number >=1 as an absolute number of query terms. The default is set to 1.0, meaning only corrections with at most one misspelled term are returned. Note that setting this too high can negatively impact performance. Low values like 1 or 2 are recommended; otherwise the time spend in suggest calls might exceed the time spend in query execution.

separator

The separator that is used to separate terms in the bigram field. If not set the whitespace character is used as a separator.

size

The number of candidates that are generated for each individual query term. Low numbers like 3 or 5 typically produce good results. Raising this can bring up terms with higher edit distances. The default is 5.

analyzer

Sets the analyzer to analyze to suggest text with. Defaults to the search analyzer of the suggest field passed via field.

shard_size

Sets the maximum number of suggested terms to be retrieved from each individual shard. During the reduce phase, only the top N suggestions are returned based on the size option. Defaults to 5.

text

Sets the text / query to provide suggestions for.

highlight

Sets up suggestion highlighting. If not provided then no highlighted field is returned. If provided must contain exactly pre_tag and post_tag, which are wrapped around the changed tokens. If multiple tokens in a row are changed the entire phrase of changed tokens is wrapped rather than each token.

collate

Checks each suggestion against the specified query to prune suggestions for which no matching docs exist in the index. The collate query for a suggestion is run only on the local shard from which the suggestion has been generated from. The query must be specified and it can be templated, see search templates for more information. The current suggestion is automatically made available as the {{suggestion}} variable, which should be used in your query. You can still specify your own template params — the suggestion value will be added to the variables you specify. Additionally, you can specify a prune to control if all phrase suggestions will be returned; when set to true the suggestions will have an additional option collate_match, which will be true if matching documents for the phrase was found, false otherwise. The default value for prune is false.

POST _search
{
  "suggest": {
    "text" : "noble prize",
    "simple_phrase" : {
      "phrase" : {
        "field" :  "title.trigram",
        "size" :   1,
        "direct_generator" : [ {
          "field" :            "title.trigram",
          "suggest_mode" :     "always",
          "min_word_length" :  1
        } ],
        "collate": {
          "query": { (1)
            "source" : {
              "match": {
                "{{field_name}}" : "{{suggestion}}" (2)
              }
            }
          },
          "params": {"field_name" : "title"}, (3)
          "prune": true (4)
        }
      }
    }
  }
}
  1. This query will be run once for every suggestion.

  2. The {{suggestion}} variable will be replaced by the text of each suggestion.

  3. An additional field_name variable has been specified in params and is used by the match query.

  4. All suggestions will be returned with an extra collate_match option indicating whether the generated phrase matched any document.

Smoothing Models

The phrase suggester supports multiple smoothing models to balance weight between infrequent grams (grams (shingles) are not existing in the index) and frequent grams (appear at least once in the index). The smoothing model can be selected by setting the smoothing parameter to one of the following options. Each smoothing model supports specific properties that can be configured.

stupid_backoff

A simple backoff model that backs off to lower order n-gram models if the higher order count is 0 and discounts the lower order n-gram model by a constant factor. The default discount is 0.4. Stupid Backoff is the default model.

laplace

A smoothing model that uses an additive smoothing where a constant (typically 1.0 or smaller) is added to all counts to balance weights. The default alpha is 0.5.

linear_interpolation

A smoothing model that takes the weighted mean of the unigrams, bigrams, and trigrams based on user supplied weights (lambdas). Linear Interpolation doesn’t have any default values. All parameters (trigram_lambda, bigram_lambda, unigram_lambda) must be supplied.

POST _search
{
  "suggest": {
    "text" : "obel prize",
    "simple_phrase" : {
      "phrase" : {
        "field" : "title.trigram",
        "size" : 1,
        "smoothing" : {
          "laplace" : {
            "alpha" : 0.7
          }
        }
      }
    }
  }
}

Candidate Generators

The phrase suggester uses candidate generators to produce a list of possible terms per term in the given text. A single candidate generator is similar to a term suggester called for each individual term in the text. The output of the generators is subsequently scored in combination with the candidates from the other terms for suggestion candidates.

Currently only one type of candidate generator is supported, the direct_generator. The Phrase suggest API accepts a list of generators under the key direct_generator; each of the generators in the list is called per term in the original text.

Direct Generators

The direct generators support the following parameters:

field

The field to fetch the candidate suggestions from. This is a required option that either needs to be set globally or per suggestion.

size

The maximum corrections to be returned per suggest text token.

suggest_mode

The suggest mode controls what suggestions are included on the suggestions generated on each shard. All values other than always can be thought of as an optimization to generate fewer suggestions to test on each shard and are not rechecked when combining the suggestions generated on each shard. Thus missing will generate suggestions for terms on shards that do not contain them even if other shards do contain them. Those should be filtered out using confidence. Three possible values can be specified:

  • missing: Only generate suggestions for terms that are not in the shard. This is the default.

  • popular: Only suggest terms that occur in more docs on the shard than the original term.

  • always: Suggest any matching suggestions based on terms in the suggest text.

max_edits

The maximum edit distance candidate suggestions can have in order to be considered as a suggestion. Can only be a value between 1 and 2. Any other value results in a bad request error being thrown. Defaults to 2.

prefix_length

The number of minimal prefix characters that must match in order be a candidate suggestions. Defaults to 1. Increasing this number improves spellcheck performance. Usually misspellings don’t occur in the beginning of terms. (Old name "prefix_len" is deprecated)

min_word_length

The minimum length a suggest text term must have in order to be included. Defaults to 4. (Old name "min_word_len" is deprecated)

max_inspections

A factor that is used to multiply with the shards_size in order to inspect more candidate spelling corrections on the shard level. Can improve accuracy at the cost of performance. Defaults to 5.

min_doc_freq

The minimal threshold in number of documents a suggestion should appear in. This can be specified as an absolute number or as a relative percentage of number of documents. This can improve quality by only suggesting high frequency terms. Defaults to 0f and is not enabled. If a value higher than 1 is specified, then the number cannot be fractional. The shard level document frequencies are used for this option.

max_term_freq

The maximum threshold in number of documents in which a suggest text token can exist in order to be included. Can be a relative percentage number (e.g., 0.4) or an absolute number to represent document frequencies. If a value higher than 1 is specified, then fractional can not be specified. Defaults to 0.01f. This can be used to exclude high frequency terms — which are usually spelled correctly — from being spellchecked. This also improves the spellcheck performance. The shard level document frequencies are used for this option.

pre_filter

A filter (analyzer) that is applied to each of the tokens passed to this candidate generator. This filter is applied to the original token before candidates are generated.

post_filter

A filter (analyzer) that is applied to each of the generated tokens before they are passed to the actual phrase scorer.

The following example shows a phrase suggest call with two generators: the first one is using a field containing ordinary indexed terms, and the second one uses a field that uses terms indexed with a reverse filter (tokens are index in reverse order). This is used to overcome the limitation of the direct generators to require a constant prefix to provide high-performance suggestions. The pre_filter and post_filter options accept ordinary analyzer names.

POST _search
{
  "suggest": {
    "text" : "obel prize",
    "simple_phrase" : {
      "phrase" : {
        "field" : "title.trigram",
        "size" : 1,
        "direct_generator" : [ {
          "field" : "title.trigram",
          "suggest_mode" : "always"
        }, {
          "field" : "title.reverse",
          "suggest_mode" : "always",
          "pre_filter" : "reverse",
          "post_filter" : "reverse"
        } ]
      }
    }
  }
}

pre_filter and post_filter can also be used to inject synonyms after candidates are generated. For instance for the query captain usq we might generate a candidate usa for the term usq, which is a synonym for america. This allows us to present captain america to the user if this phrase scores high enough.

Completion Suggester

Note
In order to understand the format of suggestions, please read the Suggesters page first.

The completion suggester provides auto-complete/search-as-you-type functionality. This is a navigational feature to guide users to relevant results as they are typing, improving search precision. It is not meant for spell correction or did-you-mean functionality like the term or phrase suggesters.

Ideally, auto-complete functionality should be as fast as a user types to provide instant feedback relevant to what a user has already typed in. Hence, completion suggester is optimized for speed. The suggester uses data structures that enable fast lookups, but are costly to build and are stored in-memory.

Mapping

To use this feature, specify a special mapping for this field, which indexes the field values for fast completions.

PUT music
{
    "mappings": {
        "_doc" : {
            "properties" : {
                "suggest" : {
                    "type" : "completion"
                },
                "title" : {
                    "type": "keyword"
                }
            }
        }
    }
}

Mapping supports the following parameters:

analyzer

The index analyzer to use, defaults to simple.

search_analyzer

The search analyzer to use, defaults to value of analyzer.

preserve_separators

Preserves the separators, defaults to true. If disabled, you could find a field starting with Foo Fighters, if you suggest for foof.

preserve_position_increments

Enables position increments, defaults to true. If disabled and using stopwords analyzer, you could get a field starting with The Beatles, if you suggest for b. Note: You could also achieve this by indexing two inputs, Beatles and The Beatles, no need to change a simple analyzer, if you are able to enrich your data.

max_input_length

Limits the length of a single input, defaults to 50 UTF-16 code points. This limit is only used at index time to reduce the total number of characters per input string in order to prevent massive inputs from bloating the underlying datastructure. Most use cases won’t be influenced by the default value since prefix completions seldom grow beyond prefixes longer than a handful of characters.

Indexing

You index suggestions like any other field. A suggestion is made of an input and an optional weight attribute. An input is the expected text to be matched by a suggestion query and the weight determines how the suggestions will be scored. Indexing a suggestion is as follows:

PUT music/_doc/1?refresh
{
    "suggest" : {
        "input": [ "Nevermind", "Nirvana" ],
        "weight" : 34
    }
}

The following parameters are supported:

input

The input to store, this can be an array of strings or just a string. This field is mandatory.

weight

A positive integer or a string containing a positive integer, which defines a weight and allows you to rank your suggestions. This field is optional.

You can index multiple suggestions for a document as follows:

PUT music/_doc/1?refresh
{
    "suggest" : [
        {
            "input": "Nevermind",
            "weight" : 10
        },
        {
            "input": "Nirvana",
            "weight" : 3
        }
    ]
}

You can use the following shorthand form. Note that you can not specify a weight with suggestion(s) in the shorthand form.

PUT music/_doc/1?refresh
{
  "suggest" : [ "Nevermind", "Nirvana" ]
}

Querying

Suggesting works as usual, except that you have to specify the suggest type as completion. Suggestions are near real-time, which means new suggestions can be made visible by refresh and documents once deleted are never shown. This request:

POST music/_search?pretty
{
    "suggest": {
        "song-suggest" : {
            "prefix" : "nir", (1)
            "completion" : { (2)
                "field" : "suggest" (3)
            }
        }
    }
}
  1. Prefix used to search for suggestions

  2. Type of suggestions

  3. Name of the field to search for suggestions in

returns this response:

{
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits": ...
  "took": 2,
  "timed_out": false,
  "suggest": {
    "song-suggest" : [ {
      "text" : "nir",
      "offset" : 0,
      "length" : 3,
      "options" : [ {
        "text" : "Nirvana",
        "_index": "music",
        "_type": "_doc",
        "_id": "1",
        "_score": 1.0,
        "_source": {
          "suggest": ["Nevermind", "Nirvana"]
        }
      } ]
    } ]
  }
}
Important
_source meta-field must be enabled, which is the default behavior, to enable returning _source with suggestions.

The configured weight for a suggestion is returned as _score. The text field uses the input of your indexed suggestion. Suggestions return the full document _source by default. The size of the _source can impact performance due to disk fetch and network transport overhead. To save some network overhead, filter out unnecessary fields from the _source using source filtering to minimize _source size. Note that the _suggest endpoint doesn’t support source filtering but using suggest on the _search endpoint does:

POST music/_search
{
    "_source": "suggest", (1)
    "suggest": {
        "song-suggest" : {
            "prefix" : "nir",
            "completion" : {
                "field" : "suggest", (2)
                "size" : 5 (3)
            }
        }
    }
}
  1. Filter the source to return only the suggest field

  2. Name of the field to search for suggestions in

  3. Number of suggestions to return

Which should look like:

{
    "took": 6,
    "timed_out": false,
    "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
    },
    "hits": {
        "total" : 0,
        "max_score" : 0.0,
        "hits" : []
    },
    "suggest": {
        "song-suggest" : [ {
            "text" : "nir",
            "offset" : 0,
            "length" : 3,
            "options" : [ {
                "text" : "Nirvana",
                "_index": "music",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "suggest": ["Nevermind", "Nirvana"]
                }
            } ]
        } ]
    }
}

The basic completion suggester query supports the following parameters:

field

The name of the field on which to run the query (required).

size

The number of suggestions to return (defaults to 5).

skip_duplicates

Whether duplicate suggestions should be filtered out (defaults to false).

Note
The completion suggester considers all documents in the index. See Context Suggester for an explanation of how to query a subset of documents instead.
Note
In case of completion queries spanning more than one shard, the suggest is executed in two phases, where the last phase fetches the relevant documents from shards, implying executing completion requests against a single shard is more performant due to the document fetch overhead when the suggest spans multiple shards. To get best performance for completions, it is recommended to index completions into a single shard index. In case of high heap usage due to shard size, it is still recommended to break index into multiple shards instead of optimizing for completion performance.

Skip duplicate suggestions

Queries can return duplicate suggestions coming from different documents. It is possible to modify this behavior by setting skip_duplicates to true. When set, this option filters out documents with duplicate suggestions from the result.

POST music/_search?pretty
{
    "suggest": {
        "song-suggest" : {
            "prefix" : "nor",
            "completion" : {
                "field" : "suggest",
                "skip_duplicates": true
            }
        }
    }
}
Warning
When set to true, this option can slow down search because more suggestions need to be visited to find the top N.

Fuzzy queries

The completion suggester also supports fuzzy queries — this means you can have a typo in your search and still get results back.

POST music/_search?pretty
{
    "suggest": {
        "song-suggest" : {
            "prefix" : "nor",
            "completion" : {
                "field" : "suggest",
                "fuzzy" : {
                    "fuzziness" : 2
                }
            }
        }
    }
}

Suggestions that share the longest prefix to the query prefix will be scored higher.

The fuzzy query can take specific fuzzy parameters. The following parameters are supported:

fuzziness

The fuzziness factor, defaults to AUTO. See Fuzziness for allowed settings.

transpositions

if set to true, transpositions are counted as one change instead of two, defaults to true

min_length

Minimum length of the input before fuzzy suggestions are returned, defaults 3

prefix_length

Minimum length of the input, which is not checked for fuzzy alternatives, defaults to 1

unicode_aware

If true, all measurements (like fuzzy edit distance, transpositions, and lengths) are measured in Unicode code points instead of in bytes. This is slightly slower than raw bytes, so it is set to false by default.

Note
If you want to stick with the default values, but still use fuzzy, you can either use fuzzy: {} or fuzzy: true.

Regex queries

The completion suggester also supports regex queries meaning you can express a prefix as a regular expression

POST music/_search?pretty
{
    "suggest": {
        "song-suggest" : {
            "regex" : "n[ever|i]r",
            "completion" : {
                "field" : "suggest"
            }
        }
    }
}

The regex query can take specific regex parameters. The following parameters are supported:

flags

Possible flags are ALL (default), ANYSTRING, COMPLEMENT, EMPTY, INTERSECTION, INTERVAL, or NONE. See regexp-syntax for their meaning

max_determinized_states

Regular expressions are dangerous because it’s easy to accidentally create an innocuous looking one that requires an exponential number of internal determinized automaton states (and corresponding RAM and CPU) for Lucene to execute. Lucene prevents these using the max_determinized_states setting (defaults to 10000). You can raise this limit to allow more complex regular expressions to execute.

Context Suggester

The completion suggester considers all documents in the index, but it is often desirable to serve suggestions filtered and/or boosted by some criteria. For example, you want to suggest song titles filtered by certain artists or you want to boost song titles based on their genre.

To achieve suggestion filtering and/or boosting, you can add context mappings while configuring a completion field. You can define multiple context mappings for a completion field. Every context mapping has a unique name and a type. There are two types: category and geo. Context mappings are configured under the contexts parameter in the field mapping.

deprecated:[6.4.0, "Indexing and querying without context on a context enabled completion field is deprecated and will be removed in the next major release."]

The following defines types, each with two context mappings for a completion field:

PUT place
{
    "mappings": {
        "_doc" : {
            "properties" : {
                "suggest" : {
                    "type" : "completion",
                    "contexts": [
                        { (1)
                            "name": "place_type",
                            "type": "category"
                        },
                        { (2)
                            "name": "location",
                            "type": "geo",
                            "precision": 4
                        }
                    ]
                }
            }
        }
    }
}
PUT place_path_category
{
    "mappings": {
        "_doc" : {
            "properties" : {
                "suggest" : {
                    "type" : "completion",
                    "contexts": [
                        { (3)
                            "name": "place_type",
                            "type": "category",
                            "path": "cat"
                        },
                        { (4)
                            "name": "location",
                            "type": "geo",
                            "precision": 4,
                            "path": "loc"
                        }
                    ]
                },
                "loc": {
                    "type": "geo_point"
                }
            }
        }
    }
}
  1. Defines a category context named 'place_type' where the categories must be sent with the suggestions.

  2. Defines a geo context named 'location' where the categories must be sent with the suggestions.

  3. Defines a category context named 'place_type' where the categories are read from the cat field.

  4. Defines a geo context named 'location' where the categories are read from the loc field.

Note
Adding context mappings increases the index size for completion field. The completion index is entirely heap resident, you can monitor the completion field index size using Indices Stats.

Category Context

The category context allows you to associate one or more categories with suggestions at index time. At query time, suggestions can be filtered and boosted by their associated categories.

The mappings are set up like the place_type fields above. If path is defined then the categories are read from that path in the document, otherwise they must be sent in the suggest field like this:

PUT place/_doc/1
{
    "suggest": {
        "input": ["timmy's", "starbucks", "dunkin donuts"],
        "contexts": {
            "place_type": ["cafe", "food"] (1)
        }
    }
}
  1. These suggestions will be associated with 'cafe' and 'food' category.

If the mapping had a path then the following index request would be enough to add the categories:

PUT place_path_category/_doc/1
{
    "suggest": ["timmy's", "starbucks", "dunkin donuts"],
    "cat": ["cafe", "food"] (1)
}
  1. These suggestions will be associated with 'cafe' and 'food' category.

Note
If context mapping references another field and the categories are explicitly indexed, the suggestions are indexed with both set of categories.
Category Query

Suggestions can be filtered by one or more categories. The following filters suggestions by multiple categories:

POST place/_search?pretty
{
    "suggest": {
        "place_suggestion" : {
            "prefix" : "tim",
            "completion" : {
                "field" : "suggest",
                "size": 10,
                "contexts": {
                    "place_type": [ "cafe", "restaurants" ]
                }
            }
        }
    }
}
Note
If multiple categories or category contexts are set on the query they are merged as a disjunction. This means that suggestions match if they contain at least one of the provided context values.

Suggestions with certain categories can be boosted higher than others. The following filters suggestions by categories and additionally boosts suggestions associated with some categories:

POST place/_search?pretty
{
    "suggest": {
        "place_suggestion" : {
            "prefix" : "tim",
            "completion" : {
                "field" : "suggest",
                "size": 10,
                "contexts": {
                    "place_type": [ (1)
                        { "context" : "cafe" },
                        { "context" : "restaurants", "boost": 2 }
                     ]
                }
            }
        }
    }
}
  1. The context query filter suggestions associated with categories 'cafe' and 'restaurants' and boosts the suggestions associated with 'restaurants' by a factor of 2

In addition to accepting category values, a context query can be composed of multiple category context clauses. The following parameters are supported for a category context clause:

context

The value of the category to filter/boost on. This is mandatory.

boost

The factor by which the score of the suggestion should be boosted, the score is computed by multiplying the boost with the suggestion weight, defaults to 1

prefix

Whether the category value should be treated as a prefix or not. For example, if set to true, you can filter category of 'type1', 'type2' and so on, by specifying a category prefix of 'type'. Defaults to false

Note
If a suggestion entry matches multiple contexts the final score is computed as the maximum score produced by any matching contexts.

Geo location Context

A geo context allows you to associate one or more geo points or geohashes with suggestions at index time. At query time, suggestions can be filtered and boosted if they are within a certain distance of a specified geo location.

Internally, geo points are encoded as geohashes with the specified precision.

Geo Mapping

In addition to the path setting, geo context mapping accepts the following settings:

precision

This defines the precision of the geohash to be indexed and can be specified as a distance value (5m, 10km etc.), or as a raw geohash precision (1..12). Defaults to a raw geohash precision value of 6.

Note
The index time precision setting sets the maximum geohash precision that can be used at query time.
Indexing geo contexts

geo contexts can be explicitly set with suggestions or be indexed from a geo point field in the document via the path parameter, similar to category contexts. Associating multiple geo location context with a suggestion, will index the suggestion for every geo location. The following indexes a suggestion with two geo location contexts:

PUT place/_doc/1
{
    "suggest": {
        "input": "timmy's",
        "contexts": {
            "location": [
                {
                    "lat": 43.6624803,
                    "lon": -79.3863353
                },
                {
                    "lat": 43.6624718,
                    "lon": -79.3873227
                }
            ]
        }
    }
}
Geo location Query

Suggestions can be filtered and boosted with respect to how close they are to one or more geo points. The following filters suggestions that fall within the area represented by the encoded geohash of a geo point:

POST place/_search
{
    "suggest": {
        "place_suggestion" : {
            "prefix" : "tim",
            "completion" : {
                "field" : "suggest",
                "size": 10,
                "contexts": {
                    "location": {
                        "lat": 43.662,
                        "lon": -79.380
                    }
                }
            }
        }
    }
}
Note
When a location with a lower precision at query time is specified, all suggestions that fall within the area will be considered.
Note
If multiple categories or category contexts are set on the query they are merged as a disjunction. This means that suggestions match if they contain at least one of the provided context values.

Suggestions that are within an area represented by a geohash can also be boosted higher than others, as shown by the following:

POST place/_search?pretty
{
    "suggest": {
        "place_suggestion" : {
            "prefix" : "tim",
            "completion" : {
                "field" : "suggest",
                "size": 10,
                "contexts": {
                    "location": [ (1)
                        {
                            "lat": 43.6624803,
                            "lon": -79.3863353,
                            "precision": 2
                        },
                        {
                            "context": {
                                "lat": 43.6624803,
                                "lon": -79.3863353
                            },
                            "boost": 2
                        }
                     ]
                }
            }
        }
    }
}
  1. The context query filters for suggestions that fall under the geo location represented by a geohash of '(43.662, -79.380)' with a precision of '2' and boosts suggestions that fall under the geohash representation of '(43.6624803, -79.3863353)' with a default precision of '6' by a factor of 2

Note
If a suggestion entry matches multiple contexts the final score is computed as the maximum score produced by any matching contexts.

In addition to accepting context values, a context query can be composed of multiple context clauses. The following parameters are supported for a category context clause:

context

A geo point object or a geo hash string to filter or boost the suggestion by. This is mandatory.

boost

The factor by which the score of the suggestion should be boosted, the score is computed by multiplying the boost with the suggestion weight, defaults to 1

precision

The precision of the geohash to encode the query geo point. This can be specified as a distance value (5m, 10km etc.), or as a raw geohash precision (1..12). Defaults to index time precision level.

neighbours

Accepts an array of precision values at which neighbouring geohashes should be taken into account. precision value can be a distance value (5m, 10km etc.) or a raw geohash precision (1..12). Defaults to generating neighbours for index time precision level.

Returning the type of the suggester

Sometimes you need to know the exact type of a suggester in order to parse its results. The typed_keys parameter can be used to change the suggester’s name in the response so that it will be prefixed by its type.

Considering the following example with two suggesters term and phrase:

POST _search?typed_keys
{
  "suggest": {
    "text" : "some test mssage",
    "my-first-suggester" : {
      "term" : {
        "field" : "message"
      }
    },
    "my-second-suggester" : {
      "phrase" : {
        "field" : "message"
      }
    }
  }
}

In the response, the suggester names will be changed to respectively term#my-first-suggester and phrase#my-second-suggester, reflecting the types of each suggestion:

{
  "suggest": {
    "term#my-first-suggester": [ (1)
      {
        "text": "some",
        "offset": 0,
        "length": 4,
        "options": []
      },
      {
        "text": "test",
        "offset": 5,
        "length": 4,
        "options": []
      },
      {
        "text": "mssage",
        "offset": 10,
        "length": 6,
        "options": [
          {
            "text": "message",
            "score": 0.8333333,
            "freq": 4
          }
        ]
      }
    ],
    "phrase#my-second-suggester": [ (2)
      {
        "text": "some test mssage",
        "offset": 0,
        "length": 16,
        "options": [
          {
            "text": "some test message",
            "score": 0.030227963
          }
        ]
      }
    ]
  },
  ...
}
  1. The name my-first-suggester now contains the term prefix.

  2. The name my-second-suggester now contains the phrase prefix.

The multi search API allows to execute several search requests within the same API. The endpoint for it is _msearch.

The format of the request is similar to the bulk API format and makes use of the newline delimited JSON (NDJSON) format. The structure is as follows (the structure is specifically optimized to reduce parsing if a specific search ends up redirected to another node):

header\n
body\n
header\n
body\n

NOTE: the final line of data must end with a newline character \n. Each newline character may be preceded by a carriage return \r. When sending requests to this endpoint the Content-Type header should be set to application/x-ndjson.

The header part includes which index / indices to search on, the search_type, preference, and routing. The body includes the typical search body request (including the query, aggregations, from, size, and so on). Here is an example:

$ cat requests
{"index" : "test"}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10}
{"index" : "test", "search_type" : "dfs_query_then_fetch"}
{"query" : {"match_all" : {}}}
{}
{"query" : {"match_all" : {}}}

{"query" : {"match_all" : {}}}
{"search_type" : "dfs_query_then_fetch"}
{"query" : {"match_all" : {}}}
$ curl -H "Content-Type: application/x-ndjson" -XGET localhost:9200/_msearch --data-binary "@requests"; echo

Note, the above includes an example of an empty header (can also be just without any content) which is supported as well.

The response returns a responses array, which includes the search response and status code for each search request matching its order in the original multi search request. If there was a complete failure for that specific search request, an object with error message and corresponding status code will be returned in place of the actual search response.

The endpoint allows to also search against an index/indices in the URI itself, in which case it will be used as the default unless explicitly defined otherwise in the header. For example:

GET twitter/_msearch
{}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10}
{}
{"query" : {"match_all" : {}}}
{"index" : "twitter2"}
{"query" : {"match_all" : {}}}

The above will execute the search against the twitter index for all the requests that don’t define an index, and the last one will be executed against the twitter2 index.

The search_type can be set in a similar manner to globally apply to all search requests.

The msearch’s max_concurrent_searches request parameter can be used to control the maximum number of concurrent searches the multi search api will execute. This default is based on the number of data nodes and the default search thread pool size.

The request parameter max_concurrent_shard_requests can be used to control the maximum number of concurrent shard requests the each sub search request will execute. This parameter should be used to protect a single request from overloading a cluster (e.g., a default request will hit all indices in a cluster which could cause shard request rejections if the number of shards per node is high). This default is based on the number of data nodes in the cluster but at most 256.In certain scenarios parallelism isn’t achieved through concurrent request such that this protection will result in poor performance. For instance in an environment where only a very low number of concurrent search requests are expected it might help to increase this value to a higher number.

Security

Template support

Much like described in Search Template for the _search resource, _msearch also provides support for templates. Submit them like follows:

GET _msearch/template
{"index" : "twitter"}
{ "source" : "{ \"query\": { \"match\": { \"message\" : \"{{keywords}}\" } } } }", "params": { "query_type": "match", "keywords": "some message" } }
{"index" : "twitter"}
{ "source" : "{ \"query\": { \"match_{{template}}\": {} } }", "params": { "template": "all" } }

for inline templates.

You can also create search templates:

POST /_scripts/my_template_1
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "message": "{{query_string}}"
                }
            }
        }
    }
}
POST /_scripts/my_template_2
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "term": {
                    "{{field}}": "{{value}}"
                }
            }
        }
    }
}

and later use them in a _msearch:

GET _msearch/template
{"index" : "main"}
{ "id": "my_template_1", "params": { "query_string": "some message" } }
{"index" : "main"}
{ "id": "my_template_2", "params": { "field": "user", "value": "test" } }

Partial responses

To ensure fast responses, the multi search API will respond with partial results if one or more shards fail. See Shard failures for more information.

Count API

The count API allows to easily execute a query and get the number of matches for that query. It can be executed across one or more indices. The query can either be provided using a simple query string as a parameter, or using the Query DSL defined within the request body. Here is an example:

PUT /twitter/_doc/1?refresh
{
    "user": "kimchy"
}

GET /twitter/_doc/_count?q=user:kimchy

GET /twitter/_doc/_count
{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}
Note
The query being sent in the body must be nested in a query key, same as the search api works

Both examples above do the same thing, which is count the number of tweets from the twitter index for a certain user. The result is:

{
    "count" : 1,
    "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
    }
}

The query is optional, and when not provided, it will use match_all to count all the docs.

Multi index

The count API can be applied to multiple indices.

Request Parameters

When executing count using the query parameter q, the query passed is a query string using Lucene query parser. There are additional parameters that can be passed:

Name Description

df

The default field to use when no field prefix is defined within the query.

analyzer

The analyzer name to be used when analyzing the query string.

default_operator

The default operator to be used, can be AND or OR. Defaults to OR.

lenient

If set to true will cause format based failures (like providing text to a numeric field) to be ignored. Defaults to false.

analyze_wildcard

Should wildcard and prefix queries be analyzed or not. Defaults to false.

terminate_after

The maximum count for each shard, upon reaching which the query execution will terminate early. If set, the response will have a boolean field terminated_early to indicate whether the query execution has actually terminated_early. Defaults to no terminate_after.

Request Body

The count can use the Query DSL within its body in order to express the query that should be executed. The body content can also be passed as a REST parameter named source.

Both HTTP GET and HTTP POST can be used to execute count with body. Since not all clients support GET with body, POST is allowed as well.

Distributed

The count operation is broadcast across all shards. For each shard id group, a replica is chosen and executed against it. This means that replicas increase the scalability of count.

Routing

The routing value (a comma separated list of the routing values) can be specified to control which shards the count request will be executed on.

Validate API

The validate API allows a user to validate a potentially expensive query without executing it. We’ll use the following test data to explain _validate:

PUT twitter/_doc/_bulk?refresh
{"index":{"_id":1}}
{"user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch"}
{"index":{"_id":2}}
{"user" : "kimchi", "post_date" : "2009-11-15T14:12:13", "message" : "My username is similar to @kimchy!"}

When sent a valid query:

GET twitter/_validate/query?q=user:foo

The response contains valid:true:

{"valid":true,"_shards":{"total":1,"successful":1,"failed":0}}

Request Parameters

When executing exists using the query parameter q, the query passed is a query string using Lucene query parser. There are additional parameters that can be passed:

Name Description

df

The default field to use when no field prefix is defined within the query.

analyzer

The analyzer name to be used when analyzing the query string.

default_operator

The default operator to be used, can be AND or OR. Defaults to OR.

lenient

If set to true will cause format based failures (like providing text to a numeric field) to be ignored. Defaults to false.

analyze_wildcard

Should wildcard and prefix queries be analyzed or not. Defaults to false.

The query may also be sent in the request body:

GET twitter/_doc/_validate/query
{
  "query" : {
    "bool" : {
      "must" : {
        "query_string" : {
          "query" : "*:*"
        }
      },
      "filter" : {
        "term" : { "user" : "kimchy" }
      }
    }
  }
}
Note
The query being sent in the body must be nested in a query key, same as the search api works

If the query is invalid, valid will be false. Here the query is invalid because Elasticsearch knows the post_date field should be a date due to dynamic mapping, and 'foo' does not correctly parse into a date:

GET twitter/_doc/_validate/query
{
  "query": {
    "query_string": {
      "query": "post_date:foo",
      "lenient": false
    }
  }
}
{"valid":false,"_shards":{"total":1,"successful":1,"failed":0}}

An explain parameter can be specified to get more detailed information about why a query failed:

GET twitter/_doc/_validate/query?explain=true
{
  "query": {
    "query_string": {
      "query": "post_date:foo",
      "lenient": false
    }
  }
}

responds with:

{
  "valid" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "explanations" : [ {
    "index" : "twitter",
    "valid" : false,
    "error" : "twitter/IAEc2nIXSSunQA_suI0MLw] QueryShardException[failed to create query:...failed to parse date field [foo]"
  } ]
}

When the query is valid, the explanation defaults to the string representation of that query. With rewrite set to true, the explanation is more detailed showing the actual Lucene query that will be executed.

For More Like This:

GET twitter/_doc/_validate/query?rewrite=true
{
  "query": {
    "more_like_this": {
      "like": {
        "_id": "2"
      },
      "boost_terms": 1
    }
  }
}

Response:

{
   "valid": true,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "explanations": [
      {
         "index": "twitter",
         "valid": true,
         "explanation": "((user:terminator^3.71334 plot:future^2.763601 plot:human^2.8415773 plot:sarah^3.4193945 plot:kyle^3.8244398 plot:cyborg^3.9177752 plot:connor^4.040236 plot:reese^4.7133346 ... )~6) -ConstantScore(_uid:tweet#2)) #(ConstantScore(_type:_doc))^0.0"
      }
   ]
}

By default, the request is executed on a single shard only, which is randomly selected. The detailed explanation of the query may depend on which shard is being hit, and therefore may vary from one request to another. So, in case of query rewrite the all_shards parameter should be used to get response from all available shards.

For Fuzzy Queries:

GET twitter/_doc/_validate/query?rewrite=true&all_shards=true
{
  "query": {
    "match": {
      "user": {
        "query": "kimchy",
        "fuzziness": "auto"
      }
    }
  }
}

Response:

{
  "valid": true,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "explanations": [
    {
      "index": "twitter",
      "shard": 0,
      "valid": true,
      "explanation": "user:kimchy~2"
    },
    {
      "index": "twitter",
      "shard": 1,
      "valid": true,
      "explanation": "user:kimchy~2"
    },
    {
      "index": "twitter",
      "shard": 2,
      "valid": true,
      "explanation": "(user:kimchi)^0.8333333"
    },
    {
      "index": "twitter",
      "shard": 3,
      "valid": true,
      "explanation": "user:kimchy"
    },
    {
      "index": "twitter",
      "shard": 4,
      "valid": true,
      "explanation": "user:kimchy~2"
    }
  ]
}

Explain API

The explain api computes a score explanation for a query and a specific document. This can give useful feedback whether a document matches or didn’t match a specific query.

Note that a single index must be provided to the index parameter.

Usage

Full query example:

GET /twitter/_doc/0/_explain
{
      "query" : {
        "match" : { "message" : "elasticsearch" }
      }
}

This will yield the following result:

{
   "_index": "twitter",
   "_type": "_doc",
   "_id": "0",
   "matched": true,
   "explanation": {
      "value": 1.6943599,
      "description": "weight(message:elasticsearch in 0) [PerFieldSimilarity], result of:",
      "details": [
         {
            "value": 1.6943599,
            "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:",
            "details": [
               {
                  "value": 1.3862944,
                  "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                  "details": [
                     {
                        "value": 1.0,
                        "description": "docFreq",
                        "details": []
                     },
                     {
                        "value": 5.0,
                        "description": "docCount",
                        "details": []
                      }
                   ]
               },
                {
                  "value": 1.2222223,
                  "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
                  "details": [
                     {
                        "value": 1.0,
                        "description": "termFreq=1.0",
                        "details": []
                     },
                     {
                        "value": 1.2,
                        "description": "parameter k1",
                        "details": []
                     },
                     {
                        "value": 0.75,
                        "description": "parameter b",
                        "details": []
                     },
                     {
                        "value": 5.4,
                        "description": "avgFieldLength",
                        "details": []
                     },
                     {
                        "value": 3.0,
                        "description": "fieldLength",
                        "details": []
                     }
                  ]
               }
            ]
         }
      ]
   }
}

There is also a simpler way of specifying the query via the q parameter. The specified q parameter value is then parsed as if the query_string query was used. Example usage of the q parameter in the explain api:

GET /twitter/_doc/0/_explain?q=message:search

This will yield the same result as the previous request.

All parameters:

_source

Set to true to retrieve the _source of the document explained. You can also retrieve part of the document by using _source_includes & _source_excludes (see Get API for more details)

stored_fields

Allows to control which stored fields to return as part of the document explained.

routing

Controls the routing in the case the routing was used during indexing.

parent

Same effect as setting the routing parameter.

preference

Controls on which shard the explain is executed.

source

Allows the data of the request to be put in the query string of the url.

q

The query string (maps to the query_string query).

df

The default field to use when no field prefix is defined within the query. Defaults to _all field.

analyzer

The analyzer name to be used when analyzing the query string. Defaults to the analyzer of the _all field.

analyze_wildcard

Should wildcard and prefix queries be analyzed or not. Defaults to false.

lenient

If set to true will cause format based failures (like providing text to a numeric field) to be ignored. Defaults to false.

default_operator

The default operator to be used, can be AND or OR. Defaults to OR.

Profile API

Warning
The Profile API is a debugging tool and adds significant overhead to search execution.

The Profile API provides detailed timing information about the execution of individual components in a search request. It gives the user insight into how search requests are executed at a low level so that the user can understand why certain requests are slow, and take steps to improve them. Note that the Profile API, amongst other things, doesn’t measure network latency, time spent in the search fetch phase, time spent while the requests spends in queues or while merging shard responses on the coordinating node.

The output from the Profile API is very verbose, especially for complicated requests executed across many shards. Pretty-printing the response is recommended to help understand the output

Usage

Any _search request can be profiled by adding a top-level profile parameter:

GET /twitter/_search
{
  "profile": true,(1)
  "query" : {
    "match" : { "message" : "some number" }
  }
}
  1. Setting the top-level profile parameter to true will enable profiling for the search

This will yield the following result:

{
   "took": 25,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "skipped" : 0,
      "failed": 0
   },
   "hits": {
      "total": 4,
      "max_score": 0.5093388,
      "hits": [...] (1)
   },
   "profile": {
     "shards": [
        {
           "id": "[2aE02wS1R8q_QFnYu6vDVQ][twitter][0]",
           "searches": [
              {
                 "query": [
                    {
                       "type": "BooleanQuery",
                       "description": "message:some message:number",
                       "time_in_nanos": "1873811",
                       "breakdown": {
                          "score": 51306,
                          "score_count": 4,
                          "build_scorer": 2935582,
                          "build_scorer_count": 1,
                          "match": 0,
                          "match_count": 0,
                          "create_weight": 919297,
                          "create_weight_count": 1,
                          "next_doc": 53876,
                          "next_doc_count": 5,
                          "advance": 0,
                          "advance_count": 0
                       },
                       "children": [
                          {
                             "type": "TermQuery",
                             "description": "message:some",
                             "time_in_nanos": "391943",
                             "breakdown": {
                                "score": 28776,
                                "score_count": 4,
                                "build_scorer": 784451,
                                "build_scorer_count": 1,
                                "match": 0,
                                "match_count": 0,
                                "create_weight": 1669564,
                                "create_weight_count": 1,
                                "next_doc": 10111,
                                "next_doc_count": 5,
                                "advance": 0,
                                "advance_count": 0
                             }
                          },
                          {
                             "type": "TermQuery",
                             "description": "message:number",
                             "time_in_nanos": "210682",
                             "breakdown": {
                                "score": 4552,
                                "score_count": 4,
                                "build_scorer": 42602,
                                "build_scorer_count": 1,
                                "match": 0,
                                "match_count": 0,
                                "create_weight": 89323,
                                "create_weight_count": 1,
                                "next_doc": 2852,
                                "next_doc_count": 5,
                                "advance": 0,
                                "advance_count": 0
                             }
                          }
                       ]
                    }
                 ],
                 "rewrite_time": 51443,
                 "collector": [
                    {
                       "name": "CancellableCollector",
                       "reason": "search_cancelled",
                       "time_in_nanos": "304311",
                       "children": [
                         {
                           "name": "SimpleTopScoreDocCollector",
                           "reason": "search_top_hits",
                           "time_in_nanos": "32273"
                         }
                       ]
                    }
                 ]
              }
           ],
           "aggregations": []
        }
     ]
   }
}
  1. Search results are returned, but were omitted here for brevity

Even for a simple query, the response is relatively complicated. Let’s break it down piece-by-piece before moving to more complex examples.

First, the overall structure of the profile response is as follows:

{
   "profile": {
        "shards": [
           {
              "id": "[2aE02wS1R8q_QFnYu6vDVQ][twitter][0]",  (1)
              "searches": [
                 {
                    "query": [...],             (2)
                    "rewrite_time": 51443,      (3)
                    "collector": [...]          (4)
                 }
              ],
              "aggregations": [...]             (5)
           }
        ]
     }
}
  1. A profile is returned for each shard that participated in the response, and is identified by a unique ID

  2. Each profile contains a section which holds details about the query execution

  3. Each profile has a single time representing the cumulative rewrite time

  4. Each profile also contains a section about the Lucene Collectors which run the search

  5. Each profile contains a section which holds the details about the aggregation execution

Because a search request may be executed against one or more shards in an index, and a search may cover one or more indices, the top level element in the profile response is an array of shard objects. Each shard object lists its id which uniquely identifies the shard. The ID’s format is [nodeID][indexName][shardID].

The profile itself may consist of one or more "searches", where a search is a query executed against the underlying Lucene index. Most search requests submitted by the user will only execute a single search against the Lucene index. But occasionally multiple searches will be executed, such as including a global aggregation (which needs to execute a secondary "match_all" query for the global context).

Inside each search object there will be two arrays of profiled information: a query array and a collector array. Alongside the search object is an aggregations object that contains the profile information for the aggregations. In the future, more sections may be added, such as suggest, highlight, etc.

There will also be a rewrite metric showing the total time spent rewriting the query (in nanoseconds).

Note
As with other statistics apis, the Profile API supports human readable outputs. This can be turned on by adding ?human=true to the query string. In this case, the output contains the additional time field containing rounded, human readable timing information (e.g. "time": "391,9ms", "time": "123.3micros").

Profiling Queries

Note

The details provided by the Profile API directly expose Lucene class names and concepts, which means that complete interpretation of the results require fairly advanced knowledge of Lucene. This page attempts to give a crash-course in how Lucene executes queries so that you can use the Profile API to successfully diagnose and debug queries, but it is only an overview. For complete understanding, please refer to Lucene’s documentation and, in places, the code.

With that said, a complete understanding is often not required to fix a slow query. It is usually sufficient to see that a particular component of a query is slow, and not necessarily understand why the advance phase of that query is the cause, for example.

query Section

The query section contains detailed timing of the query tree executed by Lucene on a particular shard. The overall structure of this query tree will resemble your original Elasticsearch query, but may be slightly (or sometimes very) different. It will also use similar but not always identical naming. Using our previous match query example, let’s analyze the query section:

"query": [
    {
       "type": "BooleanQuery",
       "description": "message:some message:number",
       "time_in_nanos": "1873811",
       "breakdown": {...},               (1)
       "children": [
          {
             "type": "TermQuery",
             "description": "message:some",
             "time_in_nanos": "391943",
             "breakdown": {...}
          },
          {
             "type": "TermQuery",
             "description": "message:number",
             "time_in_nanos": "210682",
             "breakdown": {...}
          }
       ]
    }
]
  1. The breakdown timings are omitted for simplicity

Based on the profile structure, we can see that our match query was rewritten by Lucene into a BooleanQuery with two clauses (both holding a TermQuery). The type field displays the Lucene class name, and often aligns with the equivalent name in Elasticsearch. The description field displays the Lucene explanation text for the query, and is made available to help differentiating between parts of your query (e.g. both message:search and message:test are TermQuery’s and would appear identical otherwise.

The time_in_nanos field shows that this query took ~1.8ms for the entire BooleanQuery to execute. The recorded time is inclusive of all children.

The breakdown field will give detailed stats about how the time was spent, we’ll look at that in a moment. Finally, the children array lists any sub-queries that may be present. Because we searched for two values ("search test"), our BooleanQuery holds two children TermQueries. They have identical information (type, time, breakdown, etc). Children are allowed to have their own children.

Timing Breakdown

The breakdown component lists detailed timing statistics about low-level Lucene execution:

"breakdown": {
   "score": 51306,
   "score_count": 4,
   "build_scorer": 2935582,
   "build_scorer_count": 1,
   "match": 0,
   "match_count": 0,
   "create_weight": 919297,
   "create_weight_count": 1,
   "next_doc": 53876,
   "next_doc_count": 5,
   "advance": 0,
   "advance_count": 0
}

Timings are listed in wall-clock nanoseconds and are not normalized at all. All caveats about the overall time_in_nanos apply here. The intention of the breakdown is to give you a feel for A) what machinery in Lucene is actually eating time, and B) the magnitude of differences in times between the various components. Like the overall time, the breakdown is inclusive of all children times.

The meaning of the stats are as follows:

All parameters:

create_weight

A Query in Lucene must be capable of reuse across multiple IndexSearchers (think of it as the engine that executes a search against a specific Lucene Index). This puts Lucene in a tricky spot, since many queries need to accumulate temporary state/statistics associated with the index it is being used against, but the Query contract mandates that it must be immutable.

To get around this, Lucene asks each query to generate a Weight object which acts as a temporary context object to hold state associated with this particular (IndexSearcher, Query) tuple. The weight metric shows how long this process takes

build_scorer

This parameter shows how long it takes to build a Scorer for the query. A Scorer is the mechanism that iterates over matching documents and generates a score per-document (e.g. how well does "foo" match the document?). Note, this records the time required to generate the Scorer object, not actually score the documents. Some queries have faster or slower initialization of the Scorer, depending on optimizations, complexity, etc.

This may also show timing associated with caching, if enabled and/or applicable for the query

next_doc

The Lucene method next_doc returns Doc ID of the next document matching the query. This statistic shows the time it takes to determine which document is the next match, a process that varies considerably depending on the nature of the query. Next_doc is a specialized form of advance() which is more convenient for many queries in Lucene. It is equivalent to advance(docId() + 1)

advance

advance is the "lower level" version of next_doc: it serves the same purpose of finding the next matching doc, but requires the calling query to perform extra tasks such as identifying and moving past skips, etc. However, not all queries can use next_doc, so advance is also timed for those queries.

Conjunctions (e.g. must clauses in a boolean) are typical consumers of advance

match

Some queries, such as phrase queries, match documents using a "two-phase" process. First, the document is "approximately" matched, and if it matches approximately, it is checked a second time with a more rigorous (and expensive) process. The second phase verification is what the match statistic measures.

For example, a phrase query first checks a document approximately by ensuring all terms in the phrase are present in the doc. If all the terms are present, it then executes the second phase verification to ensure the terms are in-order to form the phrase, which is relatively more expensive than just checking for presence of the terms.

Because this two-phase process is only used by a handful of queries, the match statistic is often zero

score

This records the time taken to score a particular document via its Scorer

*_count

Records the number of invocations of the particular method. For example, "next_doc_count": 2, means the nextDoc() method was called on two different documents. This can be used to help judge how selective queries are, by comparing counts between different query components.

collectors Section

The Collectors portion of the response shows high-level execution details. Lucene works by defining a "Collector" which is responsible for coordinating the traversal, scoring, and collection of matching documents. Collectors are also how a single query can record aggregation results, execute unscoped "global" queries, execute post-query filters, etc.

Looking at the previous example:

"collector": [
   {
      "name": "CancellableCollector",
      "reason": "search_cancelled",
      "time_in_nanos": "304311",
      "children": [
        {
          "name": "SimpleTopScoreDocCollector",
          "reason": "search_top_hits",
          "time_in_nanos": "32273"
        }
      ]
   }
]

We see a single collector named SimpleTopScoreDocCollector wrapped into CancellableCollector. SimpleTopScoreDocCollector is the default "scoring and sorting" Collector used by Elasticsearch. The reason field attempts to give a plain English description of the class name. The time_in_nanos is similar to the time in the Query tree: a wall-clock time inclusive of all children. Similarly, children lists all sub-collectors. The CancellableCollector that wraps SimpleTopScoreDocCollector is used by Elasticsearch to detect if the current search was cancelled and stop collecting documents as soon as it occurs.

It should be noted that Collector times are independent from the Query times. They are calculated, combined, and normalized independently! Due to the nature of Lucene’s execution, it is impossible to "merge" the times from the Collectors into the Query section, so they are displayed in separate portions.

For reference, the various collector reasons are:

search_sorted

A collector that scores and sorts documents. This is the most common collector and will be seen in most simple searches

search_count

A collector that only counts the number of documents that match the query, but does not fetch the source. This is seen when size: 0 is specified

search_terminate_after_count

A collector that terminates search execution after n matching documents have been found. This is seen when the terminate_after_count query parameter has been specified

search_min_score

A collector that only returns matching documents that have a score greater than n. This is seen when the top-level parameter min_score has been specified.

search_multi

A collector that wraps several other collectors. This is seen when combinations of search, aggregations, global aggs, and post_filters are combined in a single search.

search_timeout

A collector that halts execution after a specified period of time. This is seen when a timeout top-level parameter has been specified.

aggregation

A collector that Elasticsearch uses to run aggregations against the query scope. A single aggregation collector is used to collect documents for all aggregations, so you will see a list of aggregations in the name rather.

global_aggregation

A collector that executes an aggregation against the global query scope, rather than the specified query. Because the global scope is necessarily different from the executed query, it must execute its own match_all query (which you will see added to the Query section) to collect your entire dataset

rewrite Section

All queries in Lucene undergo a "rewriting" process. A query (and its sub-queries) may be rewritten one or more times, and the process continues until the query stops changing. This process allows Lucene to perform optimizations, such as removing redundant clauses, replacing one query for a more efficient execution path, etc. For example a Boolean → Boolean → TermQuery can be rewritten to a TermQuery, because all the Booleans are unnecessary in this case.

The rewriting process is complex and difficult to display, since queries can change drastically. Rather than showing the intermediate results, the total rewrite time is simply displayed as a value (in nanoseconds). This value is cumulative and contains the total time for all queries being rewritten.

A more complex example

To demonstrate a slightly more complex query and the associated results, we can profile the following query:

GET /twitter/_search
{
  "profile": true,
  "query": {
    "term": {
      "user": {
        "value": "test"
      }
    }
  },
  "aggs": {
    "my_scoped_agg": {
      "terms": {
        "field": "likes"
      }
    },
    "my_global_agg": {
      "global": {},
      "aggs": {
        "my_level_agg": {
          "terms": {
            "field": "likes"
          }
        }
      }
    }
  },
  "post_filter": {
    "match": {
      "message": "some"
    }
  }
}

This example has:

  • A query

  • A scoped aggregation

  • A global aggregation

  • A post_filter

And the response:

{
   ...
   "profile": {
         "shards": [
            {
               "id": "[P6-vulHtQRWuD4YnubWb7A][test][0]",
               "searches": [
                  {
                     "query": [
                        {
                           "type": "TermQuery",
                           "description": "message:some",
                           "time_in_nanos": "409456",
                           "breakdown": {
                              "score": 0,
                              "build_scorer_count": 1,
                              "match_count": 0,
                              "create_weight": 31584,
                              "next_doc": 0,
                              "match": 0,
                              "create_weight_count": 1,
                              "next_doc_count": 2,
                              "score_count": 1,
                              "build_scorer": 377872,
                              "advance": 0,
                              "advance_count": 0
                           }
                        },
                        {
                           "type": "TermQuery",
                           "description": "user:test",
                           "time_in_nanos": "303702",
                           "breakdown": {
                              "score": 0,
                              "build_scorer_count": 1,
                              "match_count": 0,
                              "create_weight": 185215,
                              "next_doc": 5936,
                              "match": 0,
                              "create_weight_count": 1,
                              "next_doc_count": 2,
                              "score_count": 1,
                              "build_scorer": 112551,
                              "advance": 0,
                              "advance_count": 0
                           }
                        }
                     ],
                     "rewrite_time": 7208,
                     "collector": [
                        {
                          "name": "CancellableCollector",
                          "reason": "search_cancelled",
                          "time_in_nanos": 2390,
                          "children": [
                            {
                              "name": "MultiCollector",
                              "reason": "search_multi",
                              "time_in_nanos": 1820,
                              "children": [
                                {
                                  "name": "FilteredCollector",
                                  "reason": "search_post_filter",
                                  "time_in_nanos": 7735,
                                  "children": [
                                    {
                                      "name": "SimpleTopScoreDocCollector",
                                      "reason": "search_top_hits",
                                      "time_in_nanos": 1328
                                    }
                                  ]
                                },
                                {
                                  "name": "MultiBucketCollector: [[my_scoped_agg, my_global_agg]]",
                                  "reason": "aggregation",
                                  "time_in_nanos": 8273
                                }
                              ]
                            }
                          ]
                        }
                     ]
                  }
               ],
               "aggregations": [...] (1)
            }
         ]
      }
}
  1. The "aggregations" portion has been omitted because it will be covered in the next section

As you can see, the output is significantly more verbose than before. All the major portions of the query are represented:

  1. The first TermQuery (user:test) represents the main term query

  2. The second TermQuery (message:some) represents the post_filter query

The Collector tree is fairly straightforward, showing how a single CancellableCollector wraps a MultiCollector which also wraps a FilteredCollector to execute the post_filter (and in turn wraps the normal scoring SimpleCollector), a BucketCollector to run all scoped aggregations.

Understanding MultiTermQuery output

A special note needs to be made about the MultiTermQuery class of queries. This includes wildcards, regex, and fuzzy queries. These queries emit very verbose responses, and are not overly structured.

Essentially, these queries rewrite themselves on a per-segment basis. If you imagine the wildcard query b*, it technically can match any token that begins with the letter "b". It would be impossible to enumerate all possible combinations, so Lucene rewrites the query in context of the segment being evaluated, e.g., one segment may contain the tokens [bar, baz], so the query rewrites to a BooleanQuery combination of "bar" and "baz". Another segment may only have the token [bakery], so the query rewrites to a single TermQuery for "bakery".

Due to this dynamic, per-segment rewriting, the clean tree structure becomes distorted and no longer follows a clean "lineage" showing how one query rewrites into the next. At present time, all we can do is apologize, and suggest you collapse the details for that query’s children if it is too confusing. Luckily, all the timing statistics are correct, just not the physical layout in the response, so it is sufficient to just analyze the top-level MultiTermQuery and ignore its children if you find the details too tricky to interpret.

Hopefully this will be fixed in future iterations, but it is a tricky problem to solve and still in-progress :)

Profiling Aggregations

aggregations Section

The aggregations section contains detailed timing of the aggregation tree executed by a particular shard. The overall structure of this aggregation tree will resemble your original Elasticsearch request. Let’s execute the previous query again and look at the aggregation profile this time:

GET /twitter/_search
{
  "profile": true,
  "query": {
    "term": {
      "user": {
        "value": "test"
      }
    }
  },
  "aggs": {
    "my_scoped_agg": {
      "terms": {
        "field": "likes"
      }
    },
    "my_global_agg": {
      "global": {},
      "aggs": {
        "my_level_agg": {
          "terms": {
            "field": "likes"
          }
        }
      }
    }
  },
  "post_filter": {
    "match": {
      "message": "some"
    }
  }
}

This yields the following aggregation profile output:

{
  "profile" : {
    "shards" : [
      {
        "aggregations" : [
          {
            "type" : "LongTermsAggregator",
            "description" : "my_scoped_agg",
            "time_in_nanos" : 195386,
            "breakdown" : {
              "reduce" : 0,
              "build_aggregation" : 81171,
              "build_aggregation_count" : 1,
              "initialize" : 22753,
              "initialize_count" : 1,
              "reduce_count" : 0,
              "collect" : 91456,
              "collect_count" : 4
            }
          },
          {
            "type" : "GlobalAggregator",
            "description" : "my_global_agg",
            "time_in_nanos" : 190430,
            "breakdown" : {
              "reduce" : 0,
              "build_aggregation" : 59990,
              "build_aggregation_count" : 1,
              "initialize" : 29619,
              "initialize_count" : 1,
              "reduce_count" : 0,
              "collect" : 100815,
              "collect_count" : 4
            },
            "children" : [
              {
                "type" : "LongTermsAggregator",
                "description" : "my_level_agg",
                "time_in_nanos" : 160329,
                "breakdown" : {
                  "reduce" : 0,
                  "build_aggregation" : 55712,
                  "build_aggregation_count" : 1,
                  "initialize" : 10559,
                  "initialize_count" : 1,
                  "reduce_count" : 0,
                  "collect" : 94052,
                  "collect_count" : 4
                }
              }
            ]
          }
        ]
      }
    ]
  }
}

From the profile structure we can see that the my_scoped_agg is internally being run as a LongTermsAggregator (because the field it is aggregating, likes, is a numeric field). At the same level, we see a GlobalAggregator which comes from my_global_agg. That aggregation then has a child LongTermsAggregator which comes from the second term’s aggregation on likes.

The time_in_nanos field shows the time executed by each aggregation, and is inclusive of all children. While the overall time is useful, the breakdown field will give detailed stats about how the time was spent.

Timing Breakdown

The breakdown component lists detailed timing statistics about low-level Lucene execution:

"breakdown": {
  "reduce": 0,
  "reduce_count": 0,
  "build_aggregation": 49765,
  "build_aggregation_count": 300,
  "initialize": 52785,
  "initialize_count": 300,
  "reduce_count": 0,
  "collect": 3155490036,
  "collect_count": 1800
}

Timings are listed in wall-clock nanoseconds and are not normalized at all. All caveats about the overall time apply here. The intention of the breakdown is to give you a feel for A) what machinery in Elasticsearch is actually eating time, and B) the magnitude of differences in times between the various components. Like the overall time, the breakdown is inclusive of all children times.

The meaning of the stats are as follows:

All parameters:

initialise

This times how long it takes to create and initialise the aggregation before starting to collect documents.

collect

This represents the cumulative time spent in the collect phase of the aggregation. This is where matching documents are passed to the aggregation and the state of the aggregator is updated based on the information contained in the documents.

build_aggregation

This represents the time spent creating the shard level results of the aggregation ready to pass back to the reducing node after the collection of documents is finished.

reduce

This is not currently used and will always report 0. Currently aggregation profiling only times the shard level parts of the aggregation execution. Timing of the reduce phase will be added later.

*_count

Records the number of invocations of the particular method. For example, "collect_count": 2, means the collect() method was called on two different documents.

Profiling Considerations

Performance Notes

Like any profiler, the Profile API introduces a non-negligible overhead to search execution. The act of instrumenting low-level method calls such as collect, advance, and next_doc can be fairly expensive, since these methods are called in tight loops. Therefore, profiling should not be enabled in production settings by default, and should not be compared against non-profiled query times. Profiling is just a diagnostic tool.

There are also cases where special Lucene optimizations are disabled, since they are not amenable to profiling. This could cause some queries to report larger relative times than their non-profiled counterparts, but in general should not have a drastic effect compared to other components in the profiled query.

Limitations

  • Profiling currently does not measure the search fetch phase nor the network overhead

  • Profiling also does not account for time spent in the queue, merging shard responses on the coordinating node, or additional work such as building global ordinals (an internal data structure used to speed up search)

  • Profiling statistics are currently not available for suggestions, highlighting, dfs_query_then_fetch

  • Profiling of the reduce phase of aggregation is currently not available

  • The Profiler is still highly experimental. The Profiler is instrumenting parts of Lucene that were never designed to be exposed in this manner, and so all results should be viewed as a best effort to provide detailed diagnostics. We hope to improve this over time. If you find obviously wrong numbers, strange query structures, or other bugs, please report them!

Field Capabilities API

The field capabilities API allows to retrieve the capabilities of fields among multiple indices.

The field capabilities API by default executes on all indices:

GET _field_caps?fields=rating

The request can also be restricted to specific indices:

GET twitter/_field_caps?fields=rating

Alternatively the fields option can also be defined in the request body. deprecated[6.4.0, Please use a request parameter instead.]

POST _field_caps
{
   "fields" : ["rating"]
}

This is equivalent to the previous request.

Supported request options:

fields

A list of fields to compute stats for. The field name supports wildcard notation. For example, using text_* will cause all fields that match the expression to be returned.

Field Capabilities

The field capabilities API returns the following information per field:

searchable

Whether this field is indexed for search on all indices.

aggregatable

Whether this field can be aggregated on all indices.

indices

The list of indices where this field has the same type, or null if all indices have the same type for the field.

non_searchable_indices

The list of indices where this field is not searchable, or null if all indices have the same definition for the field.

non_aggregatable_indices

The list of indices where this field is not aggregatable, or null if all indices have the same definition for the field.

Response format

Request:

GET _field_caps?fields=rating,title
{
    "fields": {
        "rating": { (1)
            "long": {
                "searchable": true,
                "aggregatable": false,
                "indices": ["index1", "index2"],
                "non_aggregatable_indices": ["index1"] (2)
            },
            "keyword": {
                "searchable": false,
                "aggregatable": true,
                "indices": ["index3", "index4"],
                "non_searchable_indices": ["index4"] (3)
            }
        },
        "title": { (4)
            "text": {
                "searchable": true,
                "aggregatable": false

            }
        }
    }
}
  1. The field rating is defined as a long in index1 and index2 and as a keyword in index3 and index4.

  2. The field rating is not aggregatable in index1.

  3. The field rating is not searchable in index4.

  4. The field title is defined as text in all indices.

Ranking Evaluation API

experimental["The ranking evaluation API is experimental and may be changed or removed completely in a future release, as well as change in non-backwards compatible ways on minor versions updates. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features."]

The ranking evaluation API allows to evaluate the quality of ranked search results over a set of typical search queries. Given this set of queries and a list of manually rated documents, the rank_eval endpoint calculates and returns typical information retrieval metrics like _mean reciprocal rank, precision or discounted cumulative gain.

Overview

Search quality evaluation starts with looking at the users of your search application, and the things that they are searching for. Users have a specific information need, e.g. they are looking for gift in a web shop or want to book a flight for their next holiday. They usually enters some search terms into a search box or some other web form. All of this information, together with meta information about the user (e.g. the browser, location, earlier preferences etc…​) then gets translated into a query to the underlying search system.

The challenge for search engineers is to tweak this translation process from user entries to a concrete query in such a way, that the search results contain the most relevant information with respect to the users information need. This can only be done if the search result quality is evaluated constantly across a representative test suite of typical user queries, so that improvements in the rankings for one particular query doesn’t negatively effect the ranking for other types of queries.

In order to get started with search quality evaluation, three basic things are needed:

  1. a collection of documents you want to evaluate your query performance against, usually one or more indices

  2. a collection of typical search requests that users enter into your system

  3. a set of document ratings that judge the documents relevance with respect to a search request+ It is important to note that one set of document ratings is needed per test query, and that the relevance judgements are based on the information need of the user that entered the query.

The ranking evaluation API provides a convenient way to use this information in a ranking evaluation request to calculate different search evaluation metrics. This gives a first estimation of your overall search quality and give you a measurement to optimize against when fine-tuning various aspect of the query generation in your application.

Ranking evaluation request structure

In its most basic form, a request to the _rank_eval endpoint has two sections:

GET /my_index/_rank_eval
{
    "requests": [ ... ], (1)
    "metric": { (2)
      "mean_reciprocal_rank": { ... } (3)
   }
}
  1. a set of typical search requests, together with their provided ratings

  2. definition of the evaluation metric to calculate

  3. a specific metric and its parameters

The request section contains several search requests typical to your application, along with the document ratings for each particular search request, e.g.

    "requests": [
        {
            "id": "amsterdam_query", (1)
            "request": { (2)
                "query": { "match": { "text": "amsterdam" }}
            },
            "ratings": [ (3)
                 { "_index": "my_index", "_id": "doc1", "rating": 0 },
                 { "_index": "my_index", "_id": "doc2", "rating": 3},
                 { "_index": "my_index", "_id": "doc3", "rating": 1 }
            ]
        },
        {
            "id": "berlin_query",
            "request": {
                "query": { "match": { "text": "berlin" }}
            },
            "ratings": [
                { "_index": "my_index", "_id": "doc1", "rating": 1 }
            ]
        }
    ]
  1. the search requests id, used to group result details later

  2. the query that is being evaluated

  3. a list of document ratings, each entry containing the documents _index and _id together with the rating of the documents relevance with regards to this search request

A document rating can be any integer value that expresses the relevance of the document on a user defined scale. For some of the metrics, just giving a binary rating (e.g. 0 for irrelevant and 1 for relevant) will be sufficient, other metrics can use a more fine grained scale.

Note
To use the ranking evaluation API with indices that use multiple types, you should add a filter on the _type field to the query in the request. Otherwise, if your index uses multiple types with the same id, the provided document rating might be ambiguous.

Template based ranking evaluation

As an alternative to having to provide a single query per test request, it is possible to specify query templates in the evaluation request and later refer to them. Queries with similar structure that only differ in their parameters don’t have to be repeated all the time in the requests section this way. In typical search systems where user inputs usually get filled into a small set of query templates, this helps making the evaluation request more succinct.

GET /my_index/_rank_eval
{
   [...]
  "templates": [
     {
        "id": "match_one_field_query",  (1)
        "template": { (2)
            "inline": {
                "query": {
                  "match": { "{{field}}": { "query": "{{query_string}}" }}
                }
            }
        }
     }
  ],
  "requests": [
      {
         "id": "amsterdam_query",
         "ratings": [ ... ],
         "template_id": "match_one_field_query", (3)
         "params": { (4)
            "query_string": "amsterdam",
            "field": "text"
          }
     },
    [...]
}
  1. the template id

  2. the template definition to use

  3. a reference to a previously defined template

  4. the parameters to use to fill the template

Available evaluation metrics

The metric section determines which of the available evaluation metrics is going to be used. Currently, the following metrics are supported:

Precision at K (P@k)

This metric measures the number of relevant results in the top k search results. Its a form of the well known Precision metric that only looks at the top k documents. It is the fraction of relevant documents in those first k search. A precision at 10 (P@10) value of 0.6 then means six out of the 10 top hits are relevant with respect to the users information need.

P@k works well as a simple evaluation metric that has the benefit of being easy to understand and explain. Documents in the collection need to be rated either as relevant or irrelevant with respect to the current query. P@k does not take into account where in the top k results the relevant documents occur, so a ranking of ten results that contains one relevant result in position 10 is equally good as a ranking of ten results that contains one relevant result in position 1.

GET /twitter/_rank_eval
{
    "requests": [
    {
        "id": "JFK query",
        "request": { "query": { "match_all": {}}},
        "ratings": []
    }],
    "metric": {
      "precision": {
        "k" : 20,
        "relevant_rating_threshold": 1,
        "ignore_unlabeled": false
      }
   }
}

The precision metric takes the following optional parameters

Parameter Description

k

sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query. Defaults to 10.

relevant_rating_threshold

sets the rating threshold above which documents are considered to be "relevant". Defaults to 1.

ignore_unlabeled

controls how unlabeled documents in the search results are counted. If set to 'true', unlabeled documents are ignored and neither count as relevant or irrelevant. Set to 'false' (the default), they are treated as irrelevant.

Mean reciprocal rank

For every query in the test suite, this metric calculates the reciprocal of the rank of the first relevant document. For example finding the first relevant result in position 3 means the reciprocal rank is 1/3. The reciprocal rank for each query is averaged across all queries in the test suite to give the mean reciprocal rank.

GET /twitter/_rank_eval
{
    "requests": [
    {
        "id": "JFK query",
        "request": { "query": { "match_all": {}}},
        "ratings": []
    }],
    "metric": {
        "mean_reciprocal_rank": {
            "k" : 20,
            "relevant_rating_threshold" : 1
        }
    }
}

The mean_reciprocal_rank metric takes the following optional parameters

Parameter Description

k

sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query. Defaults to 10.

relevant_rating_threshold

Sets the rating threshold above which documents are considered to be "relevant". Defaults to 1.

Discounted cumulative gain (DCG)

In contrast to the two metrics above, discounted cumulative gain takes both, the rank and the rating of the search results, into account.

The assumption is that highly relevant documents are more useful for the user when appearing at the top of the result list. Therefore, the DCG formula reduces the contribution that high ratings for documents on lower search ranks have on the overall DCG metric.

GET /twitter/_rank_eval
{
    "requests": [
    {
        "id": "JFK query",
        "request": { "query": { "match_all": {}}},
        "ratings": []
    }],
    "metric": {
       "dcg": {
            "k" : 20,
            "normalize": false
       }
    }
}

The dcg metric takes the following optional parameters:

Parameter Description

k

sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query. Defaults to 10.

normalize

If set to true, this metric will calculate the Normalized DCG.

Expected Reciprocal Rank (ERR)

Expected Reciprocal Rank (ERR) is an extension of the classical reciprocal rank for the graded relevance case (Olivier Chapelle, Donald Metzler, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance.)

It is based on the assumption of a cascade model of search, in which a user scans through ranked search results in order and stops at the first document that satisfies the information need. For this reason, it is a good metric for question answering and navigation queries, but less so for survey oriented information needs where the user is interested in finding many relevant documents in the top k results.

The metric models the expectation of the reciprocal of the position at which a user stops reading through the result list. This means that relevant document in top ranking positions will contribute much to the overall score. However, the same document will contribute much less to the score if it appears in a lower rank, even more so if there are some relevant (but maybe less relevant) documents preceding it. In this way, the ERR metric discounts documents which are shown after very relevant documents. This introduces a notion of dependency in the ordering of relevant documents that e.g. Precision or DCG don’t account for.

GET /twitter/_rank_eval
{
    "requests": [
    {
        "id": "JFK query",
        "request": { "query": { "match_all": {}}},
        "ratings": []
    }],
    "metric": {
       "expected_reciprocal_rank": {
            "maximum_relevance" : 3,
            "k" : 20
       }
    }
}

The expected_reciprocal_rank metric takes the following parameters:

Parameter Description

maximum_relevance

Mandatory parameter. The highest relevance grade used in the user supplied relevance judgments.

k

sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query. Defaults to 10.

Response format

The response of the _rank_eval endpoint contains the overall calculated result for the defined quality metric, a details section with a breakdown of results for each query in the test suite and an optional failures section that shows potential errors of individual queries. The response has the following format:

{
    "rank_eval": {
        "metric_score": 0.4, (1)
        "details": {
            "my_query_id1": { (2)
                "metric_score": 0.6, (3)
                "unrated_docs": [ (4)
                    {
                        "_index": "my_index",
                        "_id": "1960795"
                    }, [...]
                ],
                "hits": [
                    {
                        "hit": { (5)
                            "_index": "my_index",
                            "_type": "page",
                            "_id": "1528558",
                            "_score": 7.0556192
                        },
                        "rating": 1
                    }, [...]
                ],
                "metric_details": { (6)
                    "precision" : {
                        "relevant_docs_retrieved": 6,
                        "docs_retrieved": 10
                    }
                }
            },
            "my_query_id2" : { [...] }
        },
        "failures": { [...] }
    }
}
  1. the overall evaluation quality calculated by the defined metric

  2. the details section contains one entry for every query in the original requests section, keyed by the search request id

  3. the metric_score in the details section shows the contribution of this query to the global quality metric score

  4. the unrated_docs section contains an _index and _id entry for each document in the search result for this query that didn’t have a ratings value. This can be used to ask the user to supply ratings for these documents

  5. the hits section shows a grouping of the search results with their supplied rating

  6. the metric_details give additional information about the calculated quality metric (e.g. how many of the retrieved documents where relevant). The content varies for each metric but allows for better interpretation of the results

Aggregations

Metrics Aggregations

The aggregations in this family compute metrics based on values extracted in one way or another from the documents that are being aggregated. The values are typically extracted from the fields of the document (using the field data), but can also be generated using scripts.

Numeric metrics aggregations are a special type of metrics aggregation which output numeric values. Some aggregations output a single numeric metric (e.g. avg) and are called single-value numeric metrics aggregation, others generate multiple metrics (e.g. stats) and are called multi-value numeric metrics aggregation. The distinction between single-value and multi-value numeric metrics aggregations plays a role when these aggregations serve as direct sub-aggregations of some bucket aggregations (some bucket aggregations enable you to sort the returned buckets based on the numeric metrics in each bucket).

Avg Aggregation

A single-value metrics aggregation that computes the average of numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

Assuming the data consists of documents representing exams grades (between 0 and 100) of students we can average their scores with:

POST /exams/_search?size=0
{
    "aggs" : {
        "avg_grade" : { "avg" : { "field" : "grade" } }
    }
}

The above aggregation computes the average grade over all documents. The aggregation type is avg and the field setting defines the numeric field of the documents the average will be computed on. The above will return the following:

{
    ...
    "aggregations": {
        "avg_grade": {
            "value": 75.0
        }
    }
}

The name of the aggregation (avg_grade above) also serves as the key by which the aggregation result can be retrieved from the returned response.

Script

Computing the average grade based on a script:

POST /exams/_search?size=0
{
    "aggs" : {
        "avg_grade" : {
            "avg" : {
                "script" : {
                    "source" : "doc.grade.value"
                }
            }
        }
    }
}

This will interpret the script parameter as an inline script with the painless script language and no script parameters. To use a stored script use the following syntax:

POST /exams/_search?size=0
{
    "aggs" : {
        "avg_grade" : {
            "avg" : {
                "script" : {
                    "id": "my_script",
                    "params": {
                        "field": "grade"
                    }
                }
            }
        }
    }
}
Value Script

It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new average:

POST /exams/_search?size=0
{
    "aggs" : {
        "avg_corrected_grade" : {
            "avg" : {
                "field" : "grade",
                "script" : {
                    "lang": "painless",
                    "source": "_value * params.correction",
                    "params" : {
                        "correction" : 1.2
                    }
                }
            }
        }
    }
}

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

POST /exams/_search?size=0
{
    "aggs" : {
        "grade_avg" : {
            "avg" : {
                "field" : "grade",
                "missing": 10 (1)
            }
        }
    }
}
  1. Documents without a value in the grade field will fall into the same bucket as documents that have the value 10.

Weighted Avg Aggregation

A single-value metrics aggregation that computes the weighted average of numeric values that are extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents.

When calculating a regular average, each datapoint has an equal "weight" …​ it contributes equally to the final value. Weighted averages, on the other hand, weight each datapoint differently. The amount that each datapoint contributes to the final value is extracted from the document, or provided by a script.

As a formula, a weighted average is the ∑(value * weight) / ∑(weight)

A regular average can be thought of as a weighted average where every value has an implicit weight of 1.

Table 1. weighted_avg Parameters
Parameter Name Description Required Default Value

value

The configuration for the field or script that provides the values

Required

weight

The configuration for the field or script that provides the weights

Required

format

The numeric response formatter

Optional

value_type

A hint about the values for pure scripts or unmapped fields

Optional

The value and weight objects have per-field specific configuration:

Table 2. value Parameters
Parameter Name Description Required Default Value

field

The field that values should be extracted from

Required

missing

A value to use if the field is missing entirely

Optional

Table 3. weight Parameters
Parameter Name Description Required Default Value

field

The field that weights should be extracted from

Required

missing

A weight to use if the field is missing entirely

Optional

Examples

If our documents have a "grade" field that holds a 0-100 numeric score, and a "weight" field which holds an arbitrary numeric weight, we can calculate the weighted average using:

POST /exams/_search
{
    "size": 0,
    "aggs" : {
        "weighted_grade": {
            "weighted_avg": {
                "value": {
                    "field": "grade"
                },
                "weight": {
                    "field": "weight"
                }
            }
        }
    }
}

Which yields a response like:

{
    ...
    "aggregations": {
        "weighted_grade": {
            "value": 70.0
        }
    }
}

While multiple values-per-field are allowed, only one weight is allowed. If the aggregation encounters a document that has more than one weight (e.g. the weight field is a multi-valued field) it will throw an exception. If you have this situation, you will need to specify a script for the weight field, and use the script to combine the multiple values into a single value to be used.

This single weight will be applied independently to each value extracted from the value field.

This example show how a single document with multiple values will be averaged with a single weight:

POST /exams/_doc?refresh
{
    "grade": [1, 2, 3],
    "weight": 2
}

POST /exams/_search
{
    "size": 0,
    "aggs" : {
        "weighted_grade": {
            "weighted_avg": {
                "value": {
                    "field": "grade"
                },
                "weight": {
                    "field": "weight"
                }
            }
        }
    }
}

The three values (1, 2, and 3) will be included as independent values, all with the weight of 2:

{
    ...
    "aggregations": {
        "weighted_grade": {
            "value": 2.0
        }
    }
}

The aggregation returns 2.0 as the result, which matches what we would expect when calculating by hand: 1*2) + (2*2) + (3*2 / (2+2+2) == 2

Script

Both the value and the weight can be derived from a script, instead of a field. As a simple example, the following will add one to the grade and weight in the document using a script:

POST /exams/_search
{
    "size": 0,
    "aggs" : {
        "weighted_grade": {
            "weighted_avg": {
                "value": {
                    "script": "doc.grade.value + 1"
                },
                "weight": {
                    "script": "doc.weight.value + 1"
                }
            }
        }
    }
}

Missing values

The missing parameter defines how documents that are missing a value should be treated. The default behavior is different for value and weight:

By default, if the value field is missing the document is ignored and the aggregation moves on to the next document. If the weight field is missing, it is assumed to have a weight of 1 (like a normal average).

Both of these defaults can be overridden with the missing parameter:

POST /exams/_search
{
    "size": 0,
    "aggs" : {
        "weighted_grade": {
            "weighted_avg": {
                "value": {
                    "field": "grade",
                    "missing": 2
                },
                "weight": {
                    "field": "weight",
                    "missing": 3
                }
            }
        }
    }
}

Cardinality Aggregation

A single-value metrics aggregation that calculates an approximate count of distinct values. Values can be extracted either from specific fields in the document or generated by a script.

Assume you are indexing store sales and would like to count the unique number of sold products that match a query:

POST /sales/_search?size=0
{
    "aggs" : {
        "type_count" : {
            "cardinality" : {
                "field" : "type"
            }
        }
    }
}

Response:

{
    ...
    "aggregations" : {
        "type_count" : {
            "value" : 3
        }
    }
}

Precision control

This aggregation also supports the precision_threshold option:

POST /sales/_search?size=0
{
    "aggs" : {
        "type_count" : {
            "cardinality" : {
                "field" : "_doc",
                "precision_threshold": 100 (1)
            }
        }
    }
}
  1. The precision_threshold options allows to trade memory for accuracy, and defines a unique count below which counts are expected to be close to accurate. Above this value, counts might become a bit more fuzzy. The maximum supported value is 40000, thresholds above this number will have the same effect as a threshold of 40000. The default value is 3000.

Counts are approximate

Computing exact counts requires loading values into a hash set and returning its size. This doesn’t scale when working on high-cardinality sets and/or large values as the required memory usage and the need to communicate those per-shard sets between nodes would utilize too many resources of the cluster.

This cardinality aggregation is based on the HyperLogLog++ algorithm, which counts based on the hashes of the values with some interesting properties:

  • configurable precision, which decides on how to trade memory for accuracy,

  • excellent accuracy on low-cardinality sets,

  • fixed memory usage: no matter if there are tens or billions of unique values, memory usage only depends on the configured precision.

For a precision threshold of c, the implementation that we are using requires about c * 8 bytes.

The following chart shows how the error varies before and after the threshold:

cardinality error

For all 3 thresholds, counts have been accurate up to the configured threshold. Although not guaranteed, this is likely to be the case. Accuracy in practice depends on the dataset in question. In general, most datasets show consistently good accuracy. Also note that even with a threshold as low as 100, the error remains very low (1-6% as seen in the above graph) even when counting millions of items.

The HyperLogLog++ algorithm depends on the leading zeros of hashed values, the exact distributions of hashes in a dataset can affect the accuracy of the cardinality.

Please also note that even with a threshold as low as 100, the error remains very low, even when counting millions of items.

Pre-computed hashes

On string fields that have a high cardinality, it might be faster to store the hash of your field values in your index and then run the cardinality aggregation on this field. This can either be done by providing hash values from client-side or by letting Elasticsearch compute hash values for you by using the {plugins}/mapper-murmur3.html[mapper-murmur3] plugin.

Note
Pre-computing hashes is usually only useful on very large and/or high-cardinality fields as it saves CPU and memory. However, on numeric fields, hashing is very fast and storing the original values requires as much or less memory than storing the hashes. This is also true on low-cardinality string fields, especially given that those have an optimization in order to make sure that hashes are computed at most once per unique value per segment.

Script

The cardinality metric supports scripting, with a noticeable performance hit however since hashes need to be computed on the fly.

POST /sales/_search?size=0
{
    "aggs" : {
        "type_promoted_count" : {
            "cardinality" : {
                "script": {
                    "lang": "painless",
                    "source": "doc['type'].value + ' ' + doc['promoted'].value"
                }
            }
        }
    }
}

This will interpret the script parameter as an inline script with the painless script language and no script parameters. To use a stored script use the following syntax:

POST /sales/_search?size=0
{
    "aggs" : {
        "type_promoted_count" : {
            "cardinality" : {
                "script" : {
                    "id": "my_script",
                    "params": {
                        "type_field": "_doc",
                        "promoted_field": "promoted"
                    }
                }
            }
        }
    }
}

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

POST /sales/_search?size=0
{
    "aggs" : {
        "tag_cardinality" : {
            "cardinality" : {
                "field" : "tag",
                "missing": "N/A" (1)
            }
        }
    }
}
  1. Documents without a value in the tag field will fall into the same bucket as documents that have the value N/A.

Extended Stats Aggregation

A multi-value metrics aggregation that computes stats over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

The extended_stats aggregations is an extended version of the stats aggregation, where additional metrics are added such as sum_of_squares, variance, std_deviation and std_deviation_bounds.

Assuming the data consists of documents representing exams grades (between 0 and 100) of students

GET /exams/_search
{
    "size": 0,
    "aggs" : {
        "grades_stats" : { "extended_stats" : { "field" : "grade" } }
    }
}

The above aggregation computes the grades statistics over all documents. The aggregation type is extended_stats and the field setting defines the numeric field of the documents the stats will be computed on. The above will return the following:

{
    ...

    "aggregations": {
        "grades_stats": {
           "count": 2,
           "min": 50.0,
           "max": 100.0,
           "avg": 75.0,
           "sum": 150.0,
           "sum_of_squares": 12500.0,
           "variance": 625.0,
           "std_deviation": 25.0,
           "std_deviation_bounds": {
            "upper": 125.0,
            "lower": 25.0
           }
        }
    }
}

The name of the aggregation (grades_stats above) also serves as the key by which the aggregation result can be retrieved from the returned response.

Standard Deviation Bounds

By default, the extended_stats metric will return an object called std_deviation_bounds, which provides an interval of plus/minus two standard deviations from the mean. This can be a useful way to visualize variance of your data. If you want a different boundary, for example three standard deviations, you can set sigma in the request:

GET /exams/_search
{
    "size": 0,
    "aggs" : {
        "grades_stats" : {
            "extended_stats" : {
                "field" : "grade",
                "sigma" : 3 (1)
            }
        }
    }
}
  1. sigma controls how many standard deviations +/- from the mean should be displayed

sigma can be any non-negative double, meaning you can request non-integer values such as 1.5. A value of 0 is valid, but will simply return the average for both upper and lower bounds.

Note
Standard Deviation and Bounds require normality

The standard deviation and its bounds are displayed by default, but they are not always applicable to all data-sets. Your data must be normally distributed for the metrics to make sense. The statistics behind standard deviations assumes normally distributed data, so if your data is skewed heavily left or right, the value returned will be misleading.

Script

Computing the grades stats based on a script:

GET /exams/_search
{
    "size": 0,
    "aggs" : {
        "grades_stats" : {
            "extended_stats" : {
                "script" : {
                    "source" : "doc['grade'].value",
                    "lang" : "painless"
                 }
             }
         }
    }
}

This will interpret the script parameter as an inline script with the painless script language and no script parameters. To use a stored script use the following syntax:

GET /exams/_search
{
    "size": 0,
    "aggs" : {
        "grades_stats" : {
            "extended_stats" : {
                "script" : {
                    "id": "my_script",
                    "params": {
                        "field": "grade"
                    }
                }
            }
        }
    }
}
Value Script

It turned out that the exam was way above the level of the students and a grade correction needs to be applied. We can use value script to get the new stats:

GET /exams/_search
{
    "size": 0,
    "aggs" : {
        "grades_stats" : {
            "extended_stats" : {
                "field" : "grade",
                "script" : {
                    "lang" : "painless",
                    "source": "_value * params.correction",
                    "params" : {
                        "correction" : 1.2
                    }
                }
            }
        }
    }
}

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

GET /exams/_search
{
    "size": 0,
    "aggs" : {
        "grades_stats" : {
            "extended_stats" : {
                "field" : "grade",
                "missing": 0 (1)
            }
        }
    }
}
  1. Documents without a value in the grade field will fall into the same bucket as documents that have the value 0.

Geo Bounds Aggregation

A metric aggregation that computes the bounding box containing all geo_point values for a field.

Example:

PUT /museums
{
    "mappings": {
        "_doc": {
            "properties": {
                "location": {
                    "type": "geo_point"
                }
            }
        }
    }
}

POST /museums/_doc/_bulk?refresh
{"index":{"_id":1}}
{"location": "52.374081,4.912350", "name": "NEMO Science Museum"}
{"index":{"_id":2}}
{"location": "52.369219,4.901618", "name": "Museum Het Rembrandthuis"}
{"index":{"_id":3}}
{"location": "52.371667,4.914722", "name": "Nederlands Scheepvaartmuseum"}
{"index":{"_id":4}}
{"location": "51.222900,4.405200", "name": "Letterenhuis"}
{"index":{"_id":5}}
{"location": "48.861111,2.336389", "name": "Musée du Louvre"}
{"index":{"_id":6}}
{"location": "48.860000,2.327000", "name": "Musée d'Orsay"}

POST /museums/_search?size=0
{
    "query" : {
        "match" : { "name" : "musée" }
    },
    "aggs" : {
        "viewport" : {
            "geo_bounds" : {
                "field" : "location", (1)
                "wrap_longitude" : true (2)
            }
        }
    }
}
  1. The geo_bounds aggregation specifies the field to use to obtain the bounds

  2. wrap_longitude is an optional parameter which specifies whether the bounding box should be allowed to overlap the international date line. The default value is true

The above aggregation demonstrates how one would compute the bounding box of the location field for all documents with a business type of shop

The response for the above aggregation:

{
    ...
    "aggregations": {
        "viewport": {
            "bounds": {
                "top_left": {
                    "lat": 48.86111099738628,
                    "lon": 2.3269999679178
                },
                "bottom_right": {
                    "lat": 48.85999997612089,
                    "lon": 2.3363889567553997
                }
            }
        }
    }
}

Geo Centroid Aggregation

A metric aggregation that computes the weighted centroid from all coordinate values for a Geo-point datatype field.

Example:

PUT /museums
{
    "mappings": {
        "_doc": {
            "properties": {
                "location": {
                    "type": "geo_point"
                }
            }
        }
    }
}

POST /museums/_doc/_bulk?refresh
{"index":{"_id":1}}
{"location": "52.374081,4.912350", "city": "Amsterdam", "name": "NEMO Science Museum"}
{"index":{"_id":2}}
{"location": "52.369219,4.901618", "city": "Amsterdam", "name": "Museum Het Rembrandthuis"}
{"index":{"_id":3}}
{"location": "52.371667,4.914722", "city": "Amsterdam", "name": "Nederlands Scheepvaartmuseum"}
{"index":{"_id":4}}
{"location": "51.222900,4.405200", "city": "Antwerp", "name": "Letterenhuis"}
{"index":{"_id":5}}
{"location": "48.861111,2.336389", "city": "Paris", "name": "Musée du Louvre"}
{"index":{"_id":6}}
{"location": "48.860000,2.327000", "city": "Paris", "name": "Musée d'Orsay"}

POST /museums/_search?size=0
{
    "aggs" : {
        "centroid" : {
            "geo_centroid" : {
                "field" : "location" (1)
            }
        }
    }
}
  1. The geo_centroid aggregation specifies the field to use for computing the centroid. (NOTE: field must be a Geo-point datatype type)

The above aggregation demonstrates how one would compute the centroid of the location field for all documents with a crime type of burglary

The response for the above aggregation:

{
    ...
    "aggregations": {
        "centroid": {
            "location": {
                "lat": 51.00982963107526,
                "lon": 3.9662130922079086
            },
            "count": 6
        }
    }
}

The geo_centroid aggregation is more interesting when combined as a sub-aggregation to other bucket aggregations.

Example:

POST /museums/_search?size=0
{
    "aggs" : {
        "cities" : {
            "terms" : { "field" : "city.keyword" },
            "aggs" : {
                "centroid" : {
                    "geo_centroid" : { "field" : "location" }
                }
            }
        }
    }
}

The above example uses geo_centroid as a sub-aggregation to a terms bucket aggregation for finding the central location for museums in each city.

The response for the above aggregation:

{
    ...
    "aggregations": {
        "cities": {
            "sum_other_doc_count": 0,
            "doc_count_error_upper_bound": 0,
            "buckets": [
               {
                   "key": "Amsterdam",
                   "doc_count": 3,
                   "centroid": {
                      "location": {
                         "lat": 52.371655656024814,
                         "lon": 4.909563297405839
                      },
                      "count": 3
                   }
               },
               {
                   "key": "Paris",
                   "doc_count": 2,
                   "centroid": {
                      "location": {
                         "lat": 48.86055548675358,
                         "lon": 2.3316944623366
                      },
                      "count": 2
                   }
                },
                {
                    "key": "Antwerp",
                    "doc_count": 1,
                    "centroid": {
                       "location": {
                          "lat": 51.22289997059852,
                          "lon": 4.40519998781383
                       },
                       "count": 1
                    }
                 }
            ]
        }
    }
}

Max Aggregation

A single-value metrics aggregation that keeps track and returns the maximum value among the numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

Note
The min and max aggregation operate on the double representation of the data. As a consequence, the result may be approximate when running on longs whose absolute value is greater than 2^53.

Computing the max price value across all documents

POST /sales/_search?size=0
{
    "aggs" : {
        "max_price" : { "max" : { "field" : "price" } }
    }
}

Response:

{
    ...
    "aggregations": {
        "max_price": {
            "value": 200.0
        }
    }
}

As can be seen, the name of the aggregation (max_price above) also serves as the key by which the aggregation result can be retrieved from the returned response.

Script

The max aggregation can also calculate the maximum of a script. The example below computes the maximum price:

POST /sales/_search
{
    "aggs" : {
        "max_price" : {
            "max" : {
                "script" : {
                    "source" : "doc.price.value"
                }
            }
        }
    }
}

This will use the Painless scripting language and no script parameters. To use a stored script use the following syntax:

POST /sales/_search
{
    "aggs" : {
        "max_price" : {
            "max" : {
                "script" : {
                    "id": "my_script",
                    "params": {
                        "field": "price"
                    }
                }
            }
        }
    }
}

Value Script

Let’s say that the prices of the documents in our index are in USD, but we would like to compute the max in EURO (and for the sake of this example, let’s say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it is aggregated:

POST /sales/_search
{
    "aggs" : {
        "max_price_in_euros" : {
            "max" : {
                "field" : "price",
                "script" : {
                    "source" : "_value * params.conversion_rate",
                    "params" : {
                        "conversion_rate" : 1.2
                    }
                }
            }
        }
    }
}

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

POST /sales/_search
{
    "aggs" : {
        "grade_max" : {
            "max" : {
                "field" : "grade",
                "missing": 10 (1)
            }
        }
    }
}
  1. Documents without a value in the grade field will fall into the same bucket as documents that have the value 10.

Min Aggregation

A single-value metrics aggregation that keeps track and returns the minimum value among numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

Note
The min and max aggregation operate on the double representation of the data. As a consequence, the result may be approximate when running on longs whose absolute value is greater than 2^53.

Computing the min price value across all documents:

POST /sales/_search?size=0
{
    "aggs" : {
        "min_price" : { "min" : { "field" : "price" } }
    }
}

Response:

{
    ...

    "aggregations": {
        "min_price": {
            "value": 10.0
        }
    }
}

As can be seen, the name of the aggregation (min_price above) also serves as the key by which the aggregation result can be retrieved from the returned response.

Script

The min aggregation can also calculate the minimum of a script. The example below computes the minimum price:

POST /sales/_search
{
    "aggs" : {
        "min_price" : {
            "min" : {
                "script" : {
                    "source" : "doc.price.value"
                }
            }
        }
    }
}

This will use the Painless scripting language and no script parameters. To use a stored script use the following syntax:

POST /sales/_search
{
    "aggs" : {
        "min_price" : {
            "min" : {
                "script" : {
                    "id": "my_script",
                    "params": {
                        "field": "price"
                    }
                }
            }
        }
    }
}

Value Script

Let’s say that the prices of the documents in our index are in USD, but we would like to compute the min in EURO (and for the sake of this example, let’s say the conversion rate is 1.2). We can use a value script to apply the conversion rate to every value before it is aggregated:

POST /sales/_search
{
    "aggs" : {
        "min_price_in_euros" : {
            "min" : {
                "field" : "price",
                "script" : {
                    "source" : "_value * params.conversion_rate",
                    "params" : {
                        "conversion_rate" : 1.2
                    }
                }
            }
        }
    }
}

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

POST /sales/_search
{
    "aggs" : {
        "grade_min" : {
            "min" : {
                "field" : "grade",
                "missing": 10 (1)
            }
        }
    }
}
  1. Documents without a value in the grade field will fall into the same bucket as documents that have the value 10.

Percentiles Aggregation

A multi-value metrics aggregation that calculates one or more percentiles over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

Percentiles show the point at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values.

Percentiles are often used to find outliers. In normal distributions, the 0.13th and 99.87th percentiles represents three standard deviations from the mean. Any data which falls outside three standard deviations is often considered an anomaly.

When a range of percentiles are retrieved, they can be used to estimate the data distribution and determine if the data is skewed, bimodal, etc.

Assume your data consists of website load times. The average and median load times are not overly useful to an administrator. The max may be interesting, but it can be easily skewed by a single slow response.

Let’s look at a range of percentiles representing load time:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time" (1)
            }
        }
    }
}
  1. The field load_time must be a numeric field

By default, the percentile metric will generate a range of percentiles: [ 1, 5, 25, 50, 75, 95, 99 ]. The response will look like this:

{
    ...

   "aggregations": {
      "load_time_outlier": {
         "values" : {
            "1.0": 5.0,
            "5.0": 25.0,
            "25.0": 165.0,
            "50.0": 445.0,
            "75.0": 725.0,
            "95.0": 945.0,
            "99.0": 985.0
         }
      }
   }
}

As you can see, the aggregation will return a calculated value for each percentile in the default range. If we assume response times are in milliseconds, it is immediately obvious that the webpage normally loads in 10-725ms, but occasionally spikes to 945-985ms.

Often, administrators are only interested in outliers — the extreme percentiles. We can specify just the percents we are interested in (requested percentiles must be a value between 0-100 inclusive):

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time",
                "percents" : [95, 99, 99.9] (1)
            }
        }
    }
}
  1. Use the percents parameter to specify particular percentiles to calculate

Keyed Response

By default the keyed flag is set to true which associates a unique string key with each bucket and returns the ranges as a hash rather than an array. Setting the keyed flag to false will disable this behavior:

GET latency/_search
{
    "size": 0,
    "aggs": {
        "load_time_outlier": {
            "percentiles": {
                "field": "load_time",
                "keyed": false
            }
        }
    }
}

Response:

{
    ...

    "aggregations": {
        "load_time_outlier": {
            "values": [
                {
                    "key": 1.0,
                    "value": 5.0
                },
                {
                    "key": 5.0,
                    "value": 25.0
                },
                {
                    "key": 25.0,
                    "value": 165.0
                },
                {
                    "key": 50.0,
                    "value": 445.0
                },
                {
                    "key": 75.0,
                    "value": 725.0
                },
                {
                    "key": 95.0,
                    "value": 945.0
                },
                {
                    "key": 99.0,
                    "value": 985.0
                }
            ]
        }
    }
}

Script

The percentile metric supports scripting. For example, if our load times are in milliseconds but we want percentiles calculated in seconds, we could use a script to convert them on-the-fly:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "script" : {
                    "lang": "painless",
                    "source": "doc['load_time'].value / params.timeUnit", (1)
                    "params" : {
                        "timeUnit" : 1000   (2)
                    }
                }
            }
        }
    }
}
  1. The field parameter is replaced with a script parameter, which uses the script to generate values which percentiles are calculated on

  2. Scripting supports parameterized input just like any other script

This will interpret the script parameter as an inline script with the painless script language and no script parameters. To use a stored script use the following syntax:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "script" : {
                    "id": "my_script",
                    "params": {
                        "field": "load_time"
                    }
                }
            }
        }
    }
}

Percentiles are (usually) approximate

There are many different algorithms to calculate percentiles. The naive implementation simply stores all the values in a sorted array. To find the 50th percentile, you simply find the value that is at my_array[count(my_array) * 0.5].

Clearly, the naive implementation does not scale — the sorted array grows linearly with the number of values in your dataset. To calculate percentiles across potentially billions of values in an Elasticsearch cluster, approximate percentiles are calculated.

The algorithm used by the percentile metric is called TDigest (introduced by Ted Dunning in Computing Accurate Quantiles using T-Digests).

When using this metric, there are a few guidelines to keep in mind:

  • Accuracy is proportional to q(1-q). This means that extreme percentiles (e.g. 99%) are more accurate than less extreme percentiles, such as the median

  • For small sets of values, percentiles are highly accurate (and potentially 100% accurate if the data is small enough).

  • As the quantity of values in a bucket grows, the algorithm begins to approximate the percentiles. It is effectively trading accuracy for memory savings. The exact level of inaccuracy is difficult to generalize, since it depends on your data distribution and volume of data being aggregated

The following chart shows the relative error on a uniform distribution depending on the number of collected values and the requested percentile:

percentiles error

It shows how precision is better for extreme percentiles. The reason why error diminishes for large number of values is that the law of large numbers makes the distribution of values more and more uniform and the t-digest tree can do a better job at summarizing it. It would not be the case on more skewed distributions.

Warning

Percentile aggregations are also non-deterministic. This means you can get slightly different results using the same data.

Compression

Approximate algorithms must balance memory utilization with estimation accuracy. This balance can be controlled using a compression parameter:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time",
                "tdigest": {
                  "compression" : 200 (1)
                }
            }
        }
    }
}
  1. Compression controls memory usage and approximation error

The TDigest algorithm uses a number of "nodes" to approximate percentiles — the more nodes available, the higher the accuracy (and large memory footprint) proportional to the volume of data. The compression parameter limits the maximum number of nodes to 20 * compression.

Therefore, by increasing the compression value, you can increase the accuracy of your percentiles at the cost of more memory. Larger compression values also make the algorithm slower since the underlying tree data structure grows in size, resulting in more expensive operations. The default compression value is 100.

A "node" uses roughly 32 bytes of memory, so under worst-case scenarios (large amount of data which arrives sorted and in-order) the default settings will produce a TDigest roughly 64KB in size. In practice data tends to be more random and the TDigest will use less memory.

HDR Histogram

Note
This setting exposes the internal implementation of HDR Histogram and the syntax may change in the future.

HDR Histogram (High Dynamic Range Histogram) is an alternative implementation that can be useful when calculating percentiles for latency measurements as it can be faster than the t-digest implementation with the trade-off of a larger memory footprint. This implementation maintains a fixed worse-case percentage error (specified as a number of significant digits). This means that if data is recorded with values from 1 microsecond up to 1 hour (3,600,000,000 microseconds) in a histogram set to 3 significant digits, it will maintain a value resolution of 1 microsecond for values up to 1 millisecond and 3.6 seconds (or better) for the maximum tracked value (1 hour).

The HDR Histogram can be used by specifying the method parameter in the request:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_outlier" : {
            "percentiles" : {
                "field" : "load_time",
                "percents" : [95, 99, 99.9],
                "hdr": { (1)
                  "number_of_significant_value_digits" : 3 (2)
                }
            }
        }
    }
}
  1. hdr object indicates that HDR Histogram should be used to calculate the percentiles and specific settings for this algorithm can be specified inside the object

  2. number_of_significant_value_digits specifies the resolution of values for the histogram in number of significant digits

The HDRHistogram only supports positive values and will error if it is passed a negative value. It is also not a good idea to use the HDRHistogram if the range of values is unknown as this could lead to high memory usage.

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "grade_percentiles" : {
            "percentiles" : {
                "field" : "grade",
                "missing": 10 (1)
            }
        }
    }
}
  1. Documents without a value in the grade field will fall into the same bucket as documents that have the value 10.

Percentile Ranks Aggregation

A multi-value metrics aggregation that calculates one or more percentile ranks over numeric values extracted from the aggregated documents. These values can be extracted either from specific numeric fields in the documents, or be generated by a provided script.

Note

Please see Percentiles are (usually) approximate and Compression for advice regarding approximation and memory use of the percentile ranks aggregation

Percentile rank show the percentage of observed values which are below certain value. For example, if a value is greater than or equal to 95% of the observed values it is said to be at the 95th percentile rank.

Assume your data consists of website load times. You may have a service agreement that 95% of page loads completely within 500ms and 99% of page loads complete within 600ms.

Let’s look at a range of percentiles representing load time:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_ranks" : {
            "percentile_ranks" : {
                "field" : "load_time", (1)
                "values" : [500, 600]
            }
        }
    }
}
  1. The field load_time must be a numeric field

The response will look like this:

{
    ...

   "aggregations": {
      "load_time_ranks": {
         "values" : {
            "500.0": 90.01,
            "600.0": 100.0
         }
      }
   }
}

From this information you can determine you are hitting the 99% load time target but not quite hitting the 95% load time target

Keyed Response

By default the keyed flag is set to true associates a unique string key with each bucket and returns the ranges as a hash rather than an array. Setting the keyed flag to false will disable this behavior:

GET latency/_search
{
    "size": 0,
    "aggs": {
        "load_time_ranks": {
            "percentile_ranks": {
                "field": "load_time",
                "values": [500, 600],
                "keyed": false
            }
        }
    }
}

Response:

{
    ...

    "aggregations": {
        "load_time_ranks": {
            "values": [
                {
                    "key": 500.0,
                    "value": 90.01
                },
                {
                    "key": 600.0,
                    "value": 100.0
                }
            ]
        }
    }
}

Script

The percentile rank metric supports scripting. For example, if our load times are in milliseconds but we want to specify values in seconds, we could use a script to convert them on-the-fly:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_ranks" : {
            "percentile_ranks" : {
                "values" : [500, 600],
                "script" : {
                    "lang": "painless",
                    "source": "doc['load_time'].value / params.timeUnit", (1)
                    "params" : {
                        "timeUnit" : 1000   (2)
                    }
                }
            }
        }
    }
}
  1. The field parameter is replaced with a script parameter, which uses the script to generate values which percentile ranks are calculated on

  2. Scripting supports parameterized input just like any other script

This will interpret the script parameter as an inline script with the painless script language and no script parameters. To use a stored script use the following syntax:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_ranks" : {
            "percentile_ranks" : {
                "values" : [500, 600],
                "script" : {
                    "id": "my_script",
                    "params": {
                        "field": "load_time"
                    }
                }
            }
        }
    }
}

HDR Histogram

Note
This setting exposes the internal implementation of HDR Histogram and the syntax may change in the future.

HDR Histogram (High Dynamic Range Histogram) is an alternative implementation that can be useful when calculating percentile ranks for latency measurements as it can be faster than the t-digest implementation with the trade-off of a larger memory footprint. This implementation maintains a fixed worse-case percentage error (specified as a number of significant digits). This means that if data is recorded with values from 1 microsecond up to 1 hour (3,600,000,000 microseconds) in a histogram set to 3 significant digits, it will maintain a value resolution of 1 microsecond for values up to 1 millisecond and 3.6 seconds (or better) for the maximum tracked value (1 hour).

The HDR Histogram can be used by specifying the method parameter in the request:

GET latency/_search
{
    "size": 0,
    "aggs" : {
        "load_time_ranks" : {
            "percentile_ranks" : {
                "field" : "load_time",
                "values" : [500, 600],
                "hdr": { (1)
                  "number_of_significant_value_digits" : 3 (2)
                }
            }
        }
    }
}
  1. hdr object indicates that HDR Histogram should be used to calculate the percentiles and specific settings for this algorithm can be specified inside the object

  2. number_of_significant_value_digits specifies the resolution of values for the histogram in number of significant digits

The HDRHistogram only supports positive values and will error if it is passed a negative value. It is also not a good idea to use the HDRHistogram if the range of values is unknown as this could lead to high memory usage.

Missing value

The missing parameter defines how documents that are missing a value should be treated. By default they will be ignored but it is also possible to treat them as if they had a value.

GET latency/data/_search
{
    "size": 0,
    "aggs" : {
        "load_time_ranks" : {
            "percentile_ranks" : {
                "field" : "load_time",
                "values" : [500, 600],
                "missing": 10 (1)
            }
        }
    }
}
  1. Documents without a value in the load_time field will fall into the same bucket as documents that have the value 10.

Scripted Metric Aggregation

A metric aggregation that executes using scripts to provide a metric output.

Example:

POST ledger/_search?size=0
{
    "query" : {
        "match_all" : {}
    },
    "aggs": {
        "profit": {
            "scripted_metric": {
                "init_script" : "state.transactions = []",
                "map_script" : "state.transactions.add(doc.type.value == 'sale' ? doc.amount.value : -1 * doc.amount.value)", (1)
                "combine_script" : "double profit = 0; for (t in state.transactions) { profit += t } return profit",
                "reduce_script" : "double profit = 0; for (a in states) { profit += a } return profit"
            }
        }
    }
}
  1. map_script is the only required parameter

The above aggregation demonstrates how one would use the script aggregation compute the total profit from sale and cost transactions.

The response for the above aggregation:

{
    "took": 218,
    ...
    "aggregations": {
        "profit": {