Document APIs

Each index in Elasticsearch is divided into shards and each shard can have multiple copies. These copies are known as a replication group and must be kept in sync when documents are added or removed. If we fail to do so, reading from one copy will result in very different results than reading from another. The process of keeping the shard copies in sync and serving reads from them is what we call the data replication model.

Elasticsearch’s data replication model is based on the primary-backup model and is described very well in the PacificA paper of Microsoft Research. That model is based on having a single copy from the replication group that acts as the primary shard. The other copies are called replica shards. The primary serves as the main entry point for all indexing operations. It is in charge of validating them and making sure they are correct. Once an index operation has been accepted by the primary, the primary is also responsible for replicating the operation to the other copies.

This purpose of this section is to give a high level overview of the Elasticsearch replication model and discuss the implications it has for various interactions between write and read operations.

Every indexing operation in Elasticsearch is first resolved to a replication group using routing, typically based on the document ID. Once the replication group has been determined, the operation is forwarded internally to the current primary shard of the group. The primary shard is responsible for validating the operation and forwarding it to the other replicas. Since replicas can be offline, the primary is not required to replicate to all replicas. Instead, Elasticsearch maintains a list of shard copies that should receive the operation. This list is called the in-sync copies and is maintained by the master node. As the name implies, these are the set of "good" shard copies that are guaranteed to have processed all of the index and delete operations that have been acknowledged to the user. The primary is responsible for maintaining this invariant and thus has to replicate all operations to each copy in this set.

The primary shard follows this basic flow:

Many things can go wrong during indexing — disks can get corrupted, nodes can be disconnected from each other, or some configuration mistake could cause an operation to fail on a replica despite it being successful on the primary. These are infrequent but the primary has to respond to them.

In the case that the primary itself fails, the node hosting the primary will send a message to the master about it. The indexing operation will wait (up to 1 minute, by default) for the master to promote one of the replicas to be a new primary. The operation will then be forwarded to the new primary for processing. Note that the master also monitors the health of the nodes and may decide to proactively demote a primary. This typically happens when the node holding the primary is isolated from the cluster by a networking issue. See here for more details.

Once the operation has been successfully performed on the primary, the primary has to deal with potential failures when executing it on the replica shards. This may be caused by an actual failure on the replica or due to a network issue preventing the operation from reaching the replica (or preventing the replica from responding). All of these share the same end result: a replica which is part of the in-sync replica set misses an operation that is about to be acknowledged. In order to avoid violating the invariant, the primary sends a message to the master requesting that the problematic shard be removed from the in-sync replica set. Only once removal of the shard has been acknowledged by the master does the primary acknowledge the operation. Note that the master will also instruct another node to start building a new shard copy in order to restore the system to a healthy state.

While forwarding an operation to the replicas, the primary will use the replicas to validate that it is still the active primary. If the primary has been isolated due to a network partition (or a long GC) it may continue to process incoming indexing operations before realising that it has been demoted. Operations that come from a stale primary will be rejected by the replicas. When the primary receives a response from the replica rejecting its request because it is no longer the primary then it will reach out to the master and will learn that it has been replaced. The operation is then routed to the new primary.

Reads in Elasticsearch can be very lightweight lookups by ID or a heavy search request with complex aggregations that take non-trivial CPU power. One of the beauties of the primary-backup model is that it keeps all shard copies identical (with the exception of in-flight operations). As such, a single in-sync copy is sufficient to serve read requests.

When a read request is received by a node, that node is responsible for forwarding it to the nodes that hold the relevant shards, collating the responses, and responding to the client. We call that node the coordinating node for that request. The basic flow is as follows:

When a shard fails to respond to a read request, the coordinating node sends the request to another shard copy in the same replication group. Repeated failures can result in no available shard copies.

To ensure fast responses, the following APIs will respond with partial results if one or more shards fail:

Responses containing partial results still provide a 200 OK HTTP status code. Shard failures are indicated by the timed_out and _shards fields of the response header.

Each of these basic flows determines how Elasticsearch behaves as a system for both reads and writes. Furthermore, since read and write requests can be executed concurrently, these two basic flows interact with each other. This has a few inherent implications:

Under failures, the following is possible:

This document provides a high level overview of how Elasticsearch deals with data. Of course, there is much much more going on under the hood. Things like primary terms, cluster state publishing, and master election all play a role in keeping this system behaving correctly. This document also doesn’t cover known and important bugs (both closed and open). We recognize that GitHub is hard to keep up with. To help people stay on top of those, we maintain a dedicated resiliency page on our website. We strongly advise reading it.

The index API adds or updates a typed JSON document in a specific index, making it searchable. The following example inserts the JSON document into the "twitter" index, under a type called _doc with an id of 1:

The result of the above index operation is:

The _shards header provides information about the replication process of the index operation:

The index operation is successful in the case successful is at least 1.

The index operation automatically creates an index if it does not already exist, and applies any index templates that are configured. The index operation also creates a dynamic type mapping for the specified type if one does not already exist. By default, new fields and objects will automatically be added to the mapping definition for the specified type if needed. Check out the mapping section for more information on mapping definitions, and the put mapping API for information about updating type mappings manually.

Automatic index creation is controlled by the action.auto_create_index setting. This setting defaults to true, meaning that indices are always automatically created. Automatic index creation can be permitted only for indices matching certain patterns by changing the value of this setting to a comma-separated list of these patterns. It can also be explicitly permitted and forbidden by prefixing patterns in the list with a + or -. Finally it can be completely disabled by changing this setting to false.

The index operation also accepts an op_type that can be used to force a create operation, allowing for "put-if-absent" behavior. When create is used, the index operation will fail if a document by that id already exists in the index.

Here is an example of using the op_type parameter:

Another option to specify create is to use the following uri:

The index operation can be executed without specifying the id. In such a case, an id will be generated automatically. In addition, the op_type will automatically be set to create. Here is an example (note the POST used instead of PUT):

The result of the above index operation is:

Index operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters. If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409. See Optimistic concurrency control for more details.

By default, shard placement — or routing — is controlled by using a hash of the document’s id value. For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing parameter. For example:

In the example above, the "_doc" document is routed to a shard based on the routing parameter provided: "kimchy".

When setting up explicit mapping, the _routing field can be optionally used to direct the index operation to extract the routing value from the document itself. This does come at the (very minimal) cost of an additional document parsing pass. If the _routing mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted.

The index operation is directed to the primary shard based on its route (see the Routing section above) and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas.

To improve the resiliency of writes to the system, indexing operations can be configured to wait for a certain number of active shard copies before proceeding with the operation. If the requisite number of active shard copies are not available, then the write operation must wait and retry, until either the requisite shard copies have started or a timeout occurs. By default, write operations only wait for the primary shards to be active before proceeding (i.e. wait_for_active_shards=1). This default can be overridden in the index settings dynamically by setting index.write.wait_for_active_shards. To alter this behavior per operation, the wait_for_active_shards request parameter can be used.

Valid values are all or any positive integer up to the total number of configured copies per shard in the index (which is number_of_replicas+1). Specifying a negative value or a number greater than the number of shard copies will throw an error.

For example, suppose we have a cluster of three nodes, A, B, and C and we create an index index with the number of replicas set to 3 (resulting in 4 shard copies, one more copy than there are nodes). If we attempt an indexing operation, by default the operation will only ensure the primary copy of each shard is available before proceeding. This means that even if B and C went down, and A hosted the primary shard copies, the indexing operation would still proceed with only one copy of the data. If wait_for_active_shards is set on the request to 3 (and all 3 nodes are up), then the indexing operation will require 3 active shard copies before proceeding, a requirement which should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard. However, if we set wait_for_active_shards to all (or to 4, which is the same), the indexing operation will not proceed as we do not have all 4 copies of each shard active in the index. The operation will timeout unless a new node is brought up in the cluster to host the fourth copy of the shard.

It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation commences. Once the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary. The _shards section of the write operation’s response reveals the number of shard copies on which replication succeeded/failed.

Control when the changes made by this request are visible to search. See refresh.

When updating a document using the index API a new version of the document is always created even if the document hasn’t changed. If this isn’t acceptable use the _update API with detect_noop set to true. This option isn’t available on the index API because the index API doesn’t fetch the old source and isn’t able to compare it against the new source.

There isn’t a hard and fast rule about when noop updates aren’t acceptable. It’s a combination of lots of factors like how frequently your data source sends updates that are actually noops and how many queries per second Elasticsearch runs on the shard receiving the updates.

The primary shard assigned to perform the index operation might not be available when the index operation is executed. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the index operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. The timeout parameter can be used to explicitly specify how long it waits. Here is an example of setting it to 5 minutes:

Each indexed document is given a version number. By default, internal versioning is used that starts at 1 and increments with each update, deletes included. Optionally, the version number can be set to an external value (for example, if maintained in a database). To enable this functionality, version_type should be set to external. The value provided must be a numeric, long value greater than or equal to 0, and less than around 9.2e+18.

When using the external version type, the system checks to see if the version number passed to the index request is greater than the version of the currently stored document. If true, the document will be indexed and the new version number used. If the value provided is less than or equal to the stored document’s version number, a version conflict will occur and the index operation will fail. For example:

NOTE: Versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, then the operation is executed without any version checks.

The above will succeed since the supplied version of 2 is higher than the current document version of 1. If the document was already updated and its version was set to 2 or higher, the indexing command will fail and result in a conflict (409 http status code).

A nice side effect is that there is no need to maintain strict ordering of async indexing operations executed as a result of changes to a source database, as long as version numbers from the source database are used. Even the simple case of updating the Elasticsearch index using data from a database is simplified if external versioning is used, as only the latest version will be used if the index operations arrive out of order for whatever reason.

Next to the external version type explained above, Elasticsearch also supports other types for specific use cases. Here is an overview of the different version types and their semantics.

NOTE: The external_gte version type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data. There is another option, force, which is deprecated because it can cause primary and replica shards to diverge.

The get API allows to get a typed JSON document from the index based on its id. The following example gets a JSON document from an index called twitter, under a type called _doc, with id valued 0:

The result of the above get operation is:

The above result includes the _index, _type, _id, and _version of the document we wish to retrieve, including the actual _source of the document if it could be found (as indicated by the found field in the response).

The API also allows to check for the existence of a document using HEAD, for example:

By default, the get API is realtime, and is not affected by the refresh rate of the index (when data will become visible for search). If a document has been updated but is not yet refreshed, the get API will issue a refresh call in-place to make the document visible. This will also make other documents changed since the last refresh visible. In order to disable realtime GET, one can set the realtime parameter to false.

By default, the get operation returns the contents of the _source field unless you have used the stored_fields parameter or if the _source field is disabled. You can turn off _source retrieval by using the _source parameter:

If you only need one or two fields from the complete _source, you can use the _source_includes and _source_excludes parameters to include or filter out the parts you need. This can be especially helpful with large documents where partial retrieval can save on network overhead. Both parameters take a comma separated list of fields or wildcard expressions. Example:

If you only want to specify includes, you can use a shorter notation:

The get operation allows specifying a set of stored fields that will be returned by passing the stored_fields parameter. If the requested fields are not stored, they will be ignored. Consider for instance the following mapping:

Now we can add a document:

And then try to retrieve it:

The result of the above get operation is:

Field values fetched from the document itself are always returned as an array. Since the counter field is not stored the get request simply ignores it when trying to get the stored_fields.

It is also possible to retrieve metadata fields like the _routing field:

The result of the above get operation is:

Also only leaf fields can be returned via the stored_field option. So object fields can’t be returned and such requests will fail.

Use the /{index}/{type}/{id}/_source endpoint to get just the _source field of the document, without any additional content around it. For example:

You can also use the same source filtering parameters to control which parts of the _source will be returned:

Note, there is also a HEAD variant for the _source endpoint to efficiently test for document _source existence. An existing document will not have a _source if it is disabled in the mapping.

When indexing using the ability to control the routing, in order to get a document, the routing value should also be provided. For example:

The above will get a tweet with id 2, but will be routed based on the user. Note that issuing a get without the correct routing will cause the document not to be fetched.

Controls a preference of which shard replicas to execute the get request on. By default, the operation is randomized between the shard replicas.

The preference can be set to:

The refresh parameter can be set to true in order to refresh the relevant shard before the get operation and make it searchable. Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slows down indexing).

The get operation gets hashed into a specific shard id. It then gets redirected to one of the replicas within that shard id and returns the result. The replicas are the primary shard and its replicas within that shard id group. This means that the more replicas we have, the better GET scaling we will have.

You can use the version parameter to retrieve the document only if its current version is equal to the specified one. This behavior is the same for all version types with the exception of version type FORCE which always retrieves the document. Note that FORCE version type is deprecated.

Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn’t disappear immediately, although you won’t be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.

The delete API allows to delete a typed JSON document from a specific index based on its id. The following example deletes the JSON document from an index called twitter, under a type called _doc, with id 1:

The result of the above delete operation is:

Delete operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters. If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409. See Optimistic concurrency control for more details.

Each document indexed is versioned. When deleting a document, the version can be specified to make sure the relevant document we are trying to delete is actually being deleted and it has not changed in the meantime. Every write operation executed on a document, deletes included, causes its version to be incremented. The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations. The length of time for which a deleted document’s version remains available is determined by the index.gc_deletes index setting and defaults to 60 seconds.

When indexing using the ability to control the routing, in order to delete a document, the routing value should also be provided. For example:

The above will delete a tweet with id 1, but will be routed based on the user. Note that issuing a delete without the correct routing will cause the document to not be deleted.

When the _routing mapping is set as required and no routing value is specified, the delete API will throw a RoutingMissingException and reject the request.

If an external versioning variant is used, the delete operation automatically creates an index if it has not been created before (check out the create index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if it has not been created before (check out the put mapping API for manually creating type mapping).

The delete operation gets hashed into a specific shard id. It then gets redirected into the primary shard within that id group, and replicated (if needed) to shard replicas within that id group.

When making delete requests, you can set the wait_for_active_shards parameter to require a minimum number of shard copies to be active before starting to process the delete request. See here for further details and a usage example.

Control when the changes made by this request are visible to search. See ?refresh.

The primary shard assigned to perform the delete operation might not be available when the delete operation is executed. Some reasons for this might be that the primary shard is currently recovering from a store or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error. The timeout parameter can be used to explicitly specify how long it waits. Here is an example of setting it to 5 minutes:

The simplest usage of _delete_by_query just performs a deletion on every document that matches a query. Here is the API:

That will return something like this:

_delete_by_query gets a snapshot of the index when it starts and deletes what it finds using internal versioning. That means that you’ll get a version conflict if the document changes between the time when the snapshot was taken and when the delete request is processed. When the versions match the document is deleted.

During the _delete_by_query execution, multiple search requests are sequentially executed in order to find all the matching documents to delete. Every time a batch of documents is found, a corresponding bulk request is executed to delete all these documents. In case a search or bulk request got rejected, _delete_by_query relies on a default policy to retry rejected requests (up to 10 times, with exponential back off). Reaching the maximum retries limit causes the _delete_by_query to abort and all failures are returned in the failures of the response. The deletions that have been performed still stick. In other words, the process is not rolled back, only aborted. While the first failure causes the abort, all failures that are returned by the failing bulk request are returned in the failures element; therefore it’s possible for there to be quite a few failed entities.

If you’d like to count version conflicts rather than cause them to abort, then set conflicts=proceed on the url or "conflicts": "proceed" in the request body.

Back to the API format, this will delete tweets from the twitter index:

It’s also possible to delete documents of multiple indexes and multiple types at once, just like the search API:

If you provide routing then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

By default _delete_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

In addition to the standard parameters like pretty, the delete by query API also supports refresh, wait_for_completion, wait_for_active_shards, timeout, and scroll.

Sending the refresh will refresh all shards involved in the delete by query once the request completes. This is different than the delete API’s refresh parameter which causes just the shard that received the delete request to be refreshed. Also unlike the delete API it does not support wait_for.

If the request contains wait_for_completion=false then Elasticsearch will perform some preflight checks, launch the request, and then return a task which can be used with Tasks APIs to cancel or get the status of the task. Elasticsearch will also create a record of this task as a document at .tasks/task/${taskId}. This is yours to keep or remove as you see fit. When you are done with it, delete it so Elasticsearch can reclaim the space it uses.

wait_for_active_shards controls how many copies of a shard must be active before proceeding with the request. See here for details. timeout controls how long each write request waits for unavailable shards to become available. Both work exactly how they work in the Bulk API. As _delete_by_query uses scroll search, you can also specify the scroll parameter to control how long it keeps the "search context" alive, e.g. ?scroll=10m. By default it’s 5 minutes.

requests_per_second can be set to any positive decimal number (1.4, 6, 1000, etc.) and throttles the rate at which delete by query issues batches of delete operations by padding each batch with a wait time. The throttling can be disabled by setting requests_per_second to -1.

The throttling is done by waiting between batches so that scroll that _delete_by_query uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing. By default the batch size is 1000, so if the requests_per_second is set to 500:

Since the batch is issued as a single _bulk request, large batch sizes will cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth". The default is -1.

The JSON response looks like this:

You can fetch the status of any running delete by query requests with the Task API:

The response looks like:

With the task id you can look up the task directly:

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set on it then it’ll come back with results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

Any delete by query can be canceled using the task cancel API:

The task ID can be found using the tasks API.

Cancellation should happen quickly but might take a few seconds. The task status API above will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.

The value of requests_per_second can be changed on a running delete by query using the _rethrottle API:

The task ID can be found using the tasks API.

Just like when setting it on the delete by query API, requests_per_second can be either -1 to disable throttling or any decimal number like 1.7 or 12 to throttle to that level. Rethrottling that speeds up the query takes effect immediately but rethrotting that slows down the query will take effect after completing the current batch. This prevents scroll timeouts.

Delete by query supports sliced scroll to parallelize the deleting process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Slice a delete by query manually by providing a slice id and total number of slices to each request:

Which you can verify works with:

Which results in a sensible total like this one:

You can also let delete by query automatically parallelize using sliced scroll to slice on _uid. Use slices to specify the number of slices to use:

Which you also can verify works with:

Which results in a sensible total like this one:

Setting slices to auto will let Elasticsearch choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source indices, it will choose the number of slices based on the index with the smallest number of shards.

Adding slices to _delete_by_query just automates the manual process used in the section above, creating sub-requests which means it has some quirks:

If slicing automatically, setting slices to auto will choose a reasonable number for most indices. If you’re slicing manually or otherwise tuning automatic slicing, use these guidelines.

Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large (for example, 500), choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.

Delete performance scales linearly across available resources with the number of slices.

Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources.

The update API allows to update a document based on a script provided. The operation gets the document (collocated with the shard) from the index, runs the script (with optional script language and parameters), and indexes back the result (also allows to delete, or ignore the operation). It uses versioning to make sure no updates have happened during the "get" and "reindex".

Note, this operation still means full reindex of the document, it just removes some network roundtrips and reduces chances of version conflicts between the get and the index. The _source field needs to be enabled for this feature to work.

For example, let’s index a simple doc:

Now, we can execute a script that would increment the counter:

We can add a tag to the list of tags (if the tag exists, it still gets added, since this is a list):

We can remove a tag from the list of tags. Note that the Painless function to remove a tag takes as its parameter the array index of the element you wish to remove, so you need a bit more logic to locate it while avoiding a runtime error. Note that if the tag was present more than once in the list, this will remove only one occurrence of it:

In addition to _source, the following variables are available through the ctx map: _index, _type, _id, _version, _routing, and _now (the current timestamp).

We can also add a new field to the document:

Or remove a field from the document:

And, we can even change the operation that is executed. This example deletes the doc if the tags field contains green, otherwise it does nothing (noop):

The update API also supports passing a partial document, which will be merged into the existing document (simple recursive merge, inner merging of objects, replacing core "keys/values" and arrays). To fully replace the existing document, the index API should be used instead. The following partial update adds a new field to the existing document:

If both doc and script are specified, then doc is ignored. Best is to put your field pairs of the partial document in the script itself.

If doc is specified its value is merged with the existing _source. By default updates that don’t change anything detect that they don’t change anything and return "result": "noop" like this:

If name was new_name before the request was sent then the entire update request is ignored. The result element in the response returns noop if the request was ignored.

You can disable this behavior by setting "detect_noop": false like this:

If the document does not already exist, the contents of the upsert element will be inserted as a new document. If the document does exist, then the script will be executed instead:

If you would like your script to run regardless of whether the document exists or not — i.e. the script handles initializing the document instead of the upsert element — then set scripted_upsert to true:

Instead of sending a partial doc plus an upsert doc, setting doc_as_upsert to true will use the contents of doc as the upsert value:

The update operation supports the following query-string parameters:

The simplest usage of _update_by_query just performs an update on every document in the index without changing the source. This is useful to pick up a new property or some other online mapping change. Here is the API:

That will return something like this:

_update_by_query gets a snapshot of the index when it starts and indexes what it finds using internal versioning. That means you’ll get a version conflict if the document changes between the time when the snapshot was taken and when the index request is processed. When the versions match, the document is updated and the version number is incremented.

All update and query failures cause the _update_by_query to abort and are returned in the failures of the response. The updates that have been performed still stick. In other words, the process is not rolled back, only aborted. While the first failure causes the abort, all failures that are returned by the failing bulk request are returned in the failures element; therefore it’s possible for there to be quite a few failed entities.

If you want to simply count version conflicts, and not cause the _update_by_query to abort, you can set conflicts=proceed on the url or "conflicts": "proceed" in the request body. The first example does this because it is just trying to pick up an online mapping change, and a version conflict simply means that the conflicting document was updated between the start of the _update_by_query and the time when it attempted to update the document. This is fine because that update will have picked up the online mapping update.

Back to the API format, this will update tweets from the twitter index:

You can also limit _update_by_query using the Query DSL. This will update all documents from the twitter index for the user kimchy:

So far we’ve only been updating documents without changing their source. That is genuinely useful for things like picking up new properties but it’s only half the fun. _update_by_query supports scripts to update the document. This will increment the likes field on all of kimchy’s tweets:

Just as in Update API you can set ctx.op to change the operation that is executed:

Setting ctx.op to anything else is an error. Setting any other field in ctx is an error.

Note that we stopped specifying conflicts=proceed. In this case we want a version conflict to abort the process so we can handle the failure.

This API doesn’t allow you to move the documents it touches, just modify their source. This is intentional! We’ve made no provisions for removing the document from its original location.

It’s also possible to do this whole thing on multiple indexes and multiple types at once, just like the search API:

If you provide routing then the routing is copied to the scroll query, limiting the process to the shards that match that routing value:

By default _update_by_query uses scroll batches of 1000. You can change the batch size with the scroll_size URL parameter:

_update_by_query can also use the [ingest] feature by specifying a pipeline like this:

In addition to the standard parameters like pretty, the Update By Query API also supports refresh, wait_for_completion, wait_for_active_shards, timeout, and scroll.

Sending the refresh will update all shards in the index being updated when the request completes. This is different than the Update API’s refresh parameter, which causes just the shard that received the new data to be indexed. Also unlike the Update API it does not support wait_for.

If the request contains wait_for_completion=false then Elasticsearch will perform some preflight checks, launch the request, and then return a task which can be used with Tasks APIs to cancel or get the status of the task. Elasticsearch will also create a record of this task as a document at .tasks/task/${taskId}. This is yours to keep or remove as you see fit. When you are done with it, delete it so Elasticsearch can reclaim the space it uses.

wait_for_active_shards controls how many copies of a shard must be active before proceeding with the request. See here for details. timeout controls how long each write request waits for unavailable shards to become available. Both work exactly how they work in the Bulk API. Because _update_by_query uses scroll search, you can also specify the scroll parameter to control how long it keeps the "search context" alive, e.g. ?scroll=10m. By default it’s 5 minutes.

requests_per_second can be set to any positive decimal number (1.4, 6, 1000, etc.) and throttles the rate at which _update_by_query issues batches of index operations by padding each batch with a wait time. The throttling can be disabled by setting requests_per_second to -1.

The throttling is done by waiting between batches so that scroll that _update_by_query uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing. By default the batch size is 1000, so if the requests_per_second is set to 500:

Since the batch is issued as a single _bulk request, large batch sizes will cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth". The default is -1.

The JSON response looks like this:

You can fetch the status of all running update by query requests with the Task API:

The responses looks like:

With the task id you can look up the task directly. The following example retrieves information about task r1A2WoRbTwKZ516z6NEs5A:36619:

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set on it, then it’ll come back with a results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

Any update by query can be cancelled using the Task Cancel API:

The task ID can be found using the tasks API.

Cancellation should happen quickly but might take a few seconds. The task status API above will continue to list the update by query task until this task checks that it has been cancelled and terminates itself.

The value of requests_per_second can be changed on a running update by query using the _rethrottle API:

The task ID can be found using the tasks API.

Just like when setting it on the _update_by_query API, requests_per_second can be either -1 to disable throttling or any decimal number like 1.7 or 12 to throttle to that level. Rethrottling that speeds up the query takes effect immediately, but rethrotting that slows down the query will take effect after completing the current batch. This prevents scroll timeouts.

Update by query supports [sliced-scroll] to parallelize the updating process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Slice an update by query manually by providing a slice id and total number of slices to each request:

Which you can verify works with:

Which results in a sensible total like this one:

You can also let update by query automatically parallelize using [sliced-scroll] to slice on _uid. Use slices to specify the number of slices to use:

Which you also can verify works with:

Which results in a sensible total like this one:

Setting slices to auto will let Elasticsearch choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source indices, it will choose the number of slices based on the index with the smallest number of shards.

Adding slices to _update_by_query just automates the manual process used in the section above, creating sub-requests which means it has some quirks:

If slicing automatically, setting slices to auto will choose a reasonable number for most indices. If you’re slicing manually or otherwise tuning automatic slicing, use these guidelines.

Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large, (for example, 500) choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.

Update performance scales linearly across available resources with the number of slices.

Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.

Say you created an index without dynamic mapping, filled it with data, and then added a mapping value to pick up more fields from the data:

Searching for the data won’t find anything:

But you can issue an _update_by_query request to pick up the new mapping:

You can do the exact same thing when adding a field to a multifield.

The Multi get API returns multiple documents based on an index, type, (optional) and id (and possibly routing). The response includes a docs array with all the fetched documents in order corresponding to the original multi-get request (if there was a failure for a specific get, an object containing this error is included in place in the response instead). The structure of a successful get is similar in structure to a document provided by the get API.

Here is an example:

The mget endpoint can also be used against an index (in which case it is not required in the body):

And type:

In which case, the ids element can directly be used to simplify the request:

By default, the _source field will be returned for every document (if stored). Similar to the get API, you can retrieve only parts of the _source (or not at all) by using the _source parameter. You can also use the url parameters _source, _source_includes, and _source_excludes to specify defaults, which will be used when there are no per-document instructions.

For example:

Specific stored fields can be specified to be retrieved per document to get, similar to the stored_fields parameter of the Get API. For example:

Alternatively, you can specify the stored_fields parameter in the query string as a default to be applied to all documents.

You can also specify a routing value as a parameter:

In this example, document test/_doc/2 will be fetched from the shard corresponding to routing key key1 but document test/_doc/1 will be fetched from the shard corresponding to routing key key2.

See [url-access-control].

To ensure fast responses, the multi get API will respond with partial results if one or more shards fail. See Shard failures for more information.

The bulk API makes it possible to perform many index/delete operations in a single API call. This can greatly increase the indexing speed.

The REST API endpoint is /_bulk, and it expects the following newline delimited JSON (NDJSON) structure:

NOTE: The final line of data must end with a newline character \n. Each newline character may be preceded by a carriage return \r. When sending requests to this endpoint the Content-Type header should be set to application/x-ndjson.

The possible actions are index, create, delete, and update. index and create expect a source on the next line, and have the same semantics as the op_type parameter to the standard index API (i.e. create will fail if a document with the same index and type exists already, whereas index will add or replace a document as necessary). delete does not expect a source on the following line, and has the same semantics as the standard delete API. update expects that the partial doc, upsert and script and its options are specified on the next line.

If you’re providing text file input to curl, you must use the --data-binary flag instead of plain -d. The latter doesn’t preserve newlines. Example:

Because this format uses literal `\n’s as delimiters, please be sure that the JSON actions and sources are not pretty printed. Here is an example of a correct sequence of bulk commands:

The result of this bulk operation is:

The endpoints are /_bulk, /{index}/_bulk, and {index}/{type}/_bulk. When the index or the index/type are provided, they will be used by default on bulk items that don’t provide them explicitly.

A note on the format. The idea here is to make processing of this as fast as possible. As some of the actions will be redirected to other shards on other nodes, only action_meta_data is parsed on the receiving node side.

Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.

The response to a bulk action is a large JSON structure with the individual results of each action that was performed in the same order as the actions that appeared in the request. The failure of a single action does not affect the remaining actions.

There is no "correct" number of actions to perform in a single bulk call. You should experiment with different settings to find the optimum size for your particular workload.

If using the HTTP API, make sure that the client does not send HTTP chunks, as this will slow things down.

Each index and delete action within a bulk API call may include the if_seq_no and if_primary_term parameters in their respective action and meta data lines. The if_seq_no and if_primary_term parameters control how operations are executed, based on the last modification to existing documents. See Optimistic concurrency control for more details.

Each bulk item can include the version value using the version field. It automatically follows the behavior of the index / delete operation based on the _version mapping. It also support the version_type (see versioning).

Each bulk item can include the routing value using the routing field. It automatically follows the behavior of the index / delete operation based on the _routing mapping.

When making bulk calls, you can set the wait_for_active_shards parameter to require a minimum number of shard copies to be active before starting to process the bulk request. See here for further details and a usage example.

Control when the changes made by this request are visible to search. See refresh.

When using the update action, retry_on_conflict can be used as a field in the action itself (not in the extra payload line), to specify how many times an update should be retried in the case of a version conflict.

The update action payload supports the following options: doc (partial document), upsert, doc_as_upsert, script, params (for script), lang (for script), and _source. See update documentation for details on the options. Example with update actions:

See [url-access-control].

To ensure fast responses, the multi search API will respond with partial results if one or more shards fail. See Shard failures for more information.

The most basic form of _reindex just copies documents from one index to another. This will copy documents from the twitter index into the new_twitter index:

That will return something like this:

Just like _update_by_query, _reindex gets a snapshot of the source index but its target must be a different index so version conflicts are unlikely. The dest element can be configured like the index API to control optimistic concurrency control. Just leaving out version_type (as above) or setting it to internal will cause Elasticsearch to blindly dump documents into the target, overwriting any that happen to have the same type and id:

Setting version_type to external will cause Elasticsearch to preserve the version from the source, create any documents that are missing, and update any documents that have an older version in the destination index than they do in the source index:

Settings op_type to create will cause _reindex to only create missing documents in the target index. All existing documents will cause a version conflict:

By default, version conflicts abort the _reindex process. The "conflicts" request body parameter can be used to instruct _reindex to proceed with the next document on version conflicts. It is important to note that the handling of other error types is unaffected by the "conflicts" parameter. When "conflicts": "proceed" is set in the request body, the _reindex process will continue on version conflicts and return a count of version conflicts encountered:

You can limit the documents by adding a type to the source or by adding a query. This will only copy tweets made by kimchy into new_twitter:

index and type in source can both be lists, allowing you to copy from lots of sources in one request. This will copy documents from the _doc and post types in the twitter and blog indices.

It’s also possible to limit the number of processed documents by setting size. This will only copy a single document from twitter to new_twitter:

If you want a particular set of documents from the twitter index you’ll need to use sort. Sorting makes the scroll less efficient but in some contexts it’s worth it. If possible, prefer a more selective query to size and sort. This will copy 10000 documents from twitter into new_twitter:

The source section supports all the elements that are supported in a search request. For instance, only a subset of the fields from the original documents can be reindexed using source filtering as follows:

Like _update_by_query, _reindex supports a script that modifies the document. Unlike _update_by_query, the script is allowed to modify the document’s metadata. This example bumps the version of the source document:

Just as in _update_by_query, you can set ctx.op to change the operation that is executed on the destination index:

Setting ctx.op to anything else will return an error, as will setting any other field in ctx.

Think of the possibilities! Just be careful; you are able to change:

Setting _version to null or clearing it from the ctx map is just like not sending the version in an indexing request; it will cause the document to be overwritten in the target index regardless of the version on the target or the version type you use in the _reindex request.

By default if _reindex sees a document with routing then the routing is preserved unless it’s changed by the script. You can set routing on the dest request to change this:

For example, you can use the following request to copy all documents from the source index with the company name cat into the dest index with routing set to cat.

By default _reindex uses scroll batches of 1000. You can change the batch size with the size field in the source element:

Reindex can also use the [ingest] feature by specifying a pipeline like this:

Reindex supports reindexing from a remote Elasticsearch cluster:

The host parameter must contain a scheme, host, port (e.g. https://otherhost:9200), and optional path (e.g. https://otherhost:9200/proxy). The username and password parameters are optional, and when they are present _reindex will connect to the remote Elasticsearch node using basic auth. Be sure to use https when using basic auth or the password will be sent in plain text. There are a range of settings available to configure the behaviour of the https connection.

Remote hosts have to be explicitly whitelisted in elasticsearch.yaml using the reindex.remote.whitelist property. It can be set to a comma delimited list of allowed remote host and port combinations (e.g. otherhost:9200, another:9200, 127.0.10.:9200, localhost:). Scheme is ignored by the whitelist — only host and port are used, for example:

The whitelist must be configured on any nodes that will coordinate the reindex.

This feature should work with remote clusters of any version of Elasticsearch you are likely to find. This should allow you to upgrade from any version of Elasticsearch to the current version by reindexing from a cluster of the old version.

To enable queries sent to older versions of Elasticsearch the query parameter is sent directly to the remote host without validation or modification.

Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb. If the remote index includes very large documents you’ll need to use a smaller batch size. The example below sets the batch size to 10 which is very, very small.

It is also possible to set the socket read timeout on the remote connection with the socket_timeout field and the connection timeout with the connect_timeout field. Both default to 30 seconds. This example sets the socket read timeout to one minute and the connection timeout to 10 seconds:

Reindex from remote supports configurable SSL settings. These must be specified in the elasticsearch.yml file, with the exception of the secure settings, which you add in the Elasticsearch keystore. It is not possible to configure SSL in the body of the _reindex request.

The following settings are supported:

In addition to the standard parameters like pretty, the Reindex API also supports refresh, wait_for_completion, wait_for_active_shards, timeout, scroll, and requests_per_second.

Sending the refresh url parameter will cause all indexes to which the request wrote to be refreshed. This is different than the Index API’s refresh parameter which causes just the shard that received the new data to be refreshed. Also unlike the Index API it does not support wait_for.

If the request contains wait_for_completion=false then Elasticsearch will perform some preflight checks, launch the request, and then return a task which can be used with Tasks APIs to cancel or get the status of the task. Elasticsearch will also create a record of this task as a document at .tasks/task/${taskId}. This is yours to keep or remove as you see fit. When you are done with it, delete it so Elasticsearch can reclaim the space it uses.

wait_for_active_shards controls how many copies of a shard must be active before proceeding with the reindexing. See here for details. timeout controls how long each write request waits for unavailable shards to become available. Both work exactly how they work in the Bulk API. As _reindex uses scroll search, you can also specify the scroll parameter to control how long it keeps the "search context" alive, (e.g. ?scroll=10m). The default value is 5 minutes.

requests_per_second can be set to any positive decimal number (1.4, 6, 1000, etc.) and throttles the rate at which _reindex issues batches of index operations by padding each batch with a wait time. The throttling can be disabled by setting requests_per_second to -1.

The throttling is done by waiting between batches so that the scroll which _reindex uses internally can be given a timeout that takes into account the padding. The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing. By default the batch size is 1000, so if the requests_per_second is set to 500:

Since the batch is issued as a single _bulk request, large batch sizes will cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth". The default value is -1.

The JSON response looks like this:

You can fetch the status of all running reindex requests with the Task API:

The response looks like:

With the task id you can look up the task directly. The following example retrieves information about the task r1A2WoRbTwKZ516z6NEs5A:36619:

The advantage of this API is that it integrates with wait_for_completion=false to transparently return the status of completed tasks. If the task is completed and wait_for_completion=false was set, it will return a results or an error field. The cost of this feature is the document that wait_for_completion=false creates at .tasks/task/${taskId}. It is up to you to delete that document.

Any reindex can be canceled using the Task Cancel API. For example:

The task ID can be found using the Tasks API.

Cancelation should happen quickly but might take a few seconds. The Tasks API will continue to list the task until it wakes to cancel itself.

The value of requests_per_second can be changed on a running reindex using the _rethrottle API:

The task ID can be found using the tasks API.

Just like when setting it on the Reindex API, requests_per_second can be either -1 to disable throttling or any decimal number like 1.7 or 12 to throttle to that level. Rethrottling that speeds up the query takes effect immediately, but rethrottling that slows down the query will take effect after completing the current batch. This prevents scroll timeouts.

_reindex can be used to build a copy of an index with renamed fields. Say you create an index containing documents that look like this:

but you don’t like the name flag and want to replace it with tag. _reindex can create the other index for you:

Now you can get the new document:

which will return:

Reindex supports [sliced-scroll] to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.

Slice a reindex request manually by providing a slice id and total number of slices to each request:

You can verify this works by:

which results in a sensible total like this one:

You can also let _reindex automatically parallelize using [sliced-scroll] to slice on _uid. Use slices to specify the number of slices to use:

You can also this verify works by:

which results in a sensible total like this one:

Setting slices to auto will let Elasticsearch choose the number of slices to use. This setting will use one slice per shard, up to a certain limit. If there are multiple source indices, it will choose the number of slices based on the index with the smallest number of shards.

Adding slices to _reindex just automates the manual process used in the section above, creating sub-requests which means it has some quirks:

If slicing automatically, setting slices to auto will choose a reasonable number for most indices. If slicing manually or otherwise tuning automatic slicing, use these guidelines.

Query performance is most efficient when the number of slices is equal to the number of shards in the index. If that number is large (e.g. 500), choose a lower number as too many slices will hurt performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.

Indexing performance scales linearly across available resources with the number of slices.

Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources.

If you have many indices to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up many indices. That way you can resume the process if there are any errors by removing the partially completed index and starting over at that index. It also makes parallelizing the process fairly simple: split the list of indices to reindex and run each list in parallel.

One-off bash scripts seem to work nicely for this:

Notwithstanding the above advice, you can use _reindex in combination with Painless to reindex daily indices to apply a new template to the existing documents.

Assuming you have indices consisting of documents as follows:

The new template for the metricbeat-* indices is already loaded into Elasticsearch, but it applies only to the newly created indices. Painless can be used to reindex the existing documents and apply the new template.

The script below extracts the date from the index name and creates a new index with -1 appended. All data from metricbeat-2016.05.31 will be reindexed into metricbeat-2016.05.31-1.

All documents from the previous metricbeat indices can now be found in the *-1 indices.

The previous method can also be used in conjunction with changing a field name to load only the existing data into the new index and rename any fields if needed.

_reindex can be used to extract a random subset of an index for testing:

Returns information and statistics on terms in the fields of a particular document. The document could be stored in the index or artificially provided by the user. Term vectors are realtime by default, not near realtime. This can be changed by setting the realtime parameter to false.

Optionally, you can specify the fields for which the information is retrieved either with a parameter in the url

or by adding the requested fields in the request body (see example below). Fields can also be specified with wildcards in a similar way to the multi match query

Three types of values can be requested: term information, term statistics, and field statistics. By default, all term information and field statistics are returned for all fields but no term statistics.

If the requested information wasn’t stored in the index, it will be computed on the fly if possible. Additionally, term vectors could be computed for documents not even existing in the index, but instead provided by the user.

Setting term_statistics to true (default is false) will return

By default these values are not returned since term statistics can have a serious performance impact.

Setting field_statistics to false (default is true) will omit :

With the parameter filter, the terms returned could also be filtered based on their tf-idf scores. This could be useful in order find out a good characteristic vector of a document. This feature works in a similar manner to the second phase of the More Like This Query. See example 5 for usage.

The following sub-parameters are supported:

The term and field statistics are not accurate. Deleted documents are not taken into account. The information is only retrieved for the shard the requested document resides in. The term and field statistics are therefore only useful as relative measures whereas the absolute numbers have no meaning in this context. By default, when requesting term vectors of artificial documents, a shard to get the statistics from is randomly selected. Use routing only to hit a particular shard.

First, we create an index that stores term vectors, payloads, etc. :

Second, we add some documents:

The following request returns all information and statistics for field text in document 1 (John Doe):

Response:

Term vectors which are not explicitly stored in the index are automatically computed on the fly. The following request returns all information and statistics for the fields in document 1, even though the terms haven’t been explicitly stored in the index. Note that for the field text, the terms are not re-generated.

Term vectors can also be generated for artificial documents, that is for documents not present in the index. For example, the following request would return the same results as in example 1. The mapping used is determined by the index and type.

If dynamic mapping is turned on (default), the document fields not in the original mapping will be dynamically created.

Additionally, a different analyzer than the one at the field may be provided by using the per_field_analyzer parameter. This is useful in order to generate term vectors in any fashion, especially when using artificial documents. When providing an analyzer for a field that already stores term vectors, the term vectors will be re-generated.

Response:

Finally, the terms returned could be filtered based on their tf-idf scores. In the example below we obtain the three most "interesting" keywords from the artificial document having the given "plot" field value. Notice that the keyword "Tony" or any stop words are not part of the response, as their tf-idf must be too low.

Response:

The multi termvectors API allows to get multiple termvectors at once. The documents from which to retrieve the term vectors are specified by an index, type, and id. But the documents could also be artificially provided in the request itself.

The response includes a docs array with all the fetched termvectors, each element having the structure provided by the termvectors API. Here is an example:

See the termvectors API for a description of possible parameters.

The _mtermvectors endpoint can also be used against an index (in which case it is not required in the body):

And type:

If all requested documents are on same index and have the same type and the same parameters, the request can be simplified:

Additionally, just like for the termvectors API, term vectors could be generated for user provided documents. The mapping used is determined by _index and _type.

The Index, Update, Delete, and Bulk APIs support setting refresh to control when changes made by this request are made visible to search. These are the allowed values:

Unless you have a good reason to wait for the change to become visible always use refresh=false, or, because that is the default, just leave the refresh parameter out of the URL. That is the simplest and fastest choice.

If you absolutely must have the changes made by a request visible synchronously with the request then you must pick between putting more load on Elasticsearch (true) and waiting longer for the response (wait_for). Here are a few points that should inform that decision:

If a refresh=wait_for request comes in when there are already index.max_refresh_listeners (defaults to 1000) requests waiting for a refresh on that shard then that request will behave just as though it had refresh set to true instead: it will force a refresh. This keeps the promise that when a refresh=wait_for request returns that its changes are visible for search while preventing unchecked resource usage for blocked requests. If a request forced a refresh because it ran out of listener slots then its response will contain "forced_refresh": true.

Bulk requests only take up one slot on each shard that they touch no matter how many times they modify the shard.

These will create a document and immediately refresh the index so it is visible:

These will create a document without doing anything to make it visible for search:

This will create a document and wait for it to become visible for search:

Elasticsearch is distributed. When documents are created, updated, or deleted, the new version of the document has to be replicated to other nodes in the cluster. Elasticsearch is also asynchronous and concurrent, meaning that these replication requests are sent in parallel, and may arrive at their destination out of sequence. Elasticsearch needs a way of ensuring that an older version of a document never overwrites a newer version.

To ensure an older version of a document doesn’t overwrite a newer version, every operation performed to a document is assigned a sequence number by the primary shard that coordinates that change. The sequence number is increased with each operation and thus newer operations are guaranteed to have a higher sequence number than older operations. Elasticsearch can then use the sequence number of operations to make sure a newer document version is never overridden by a change that has a smaller sequence number assigned to it.

For example, the following indexing command will create a document and assign it an initial sequence number and primary term:

You can see the assigned sequence number and primary term in the _seq_no and _primary_term fields of the response:

Elasticsearch keeps tracks of the sequence number and primary term of the last operation to have changed each of the documents it stores. The sequence number and primary term are returned in the _seq_no and _primary_term fields in the response of the GET API:

returns:

Note: The Search API can return the _seq_no and _primary_term for each search hit by setting seq_no_primary_term parameter.

The sequence number and the primary term uniquely identify a change. By noting down the sequence number and primary term returned, you can make sure to only change the document if no other change was made to it since you retrieved it. This is done by setting the if_seq_no and if_primary_term parameters of either the Index API or the Delete API.

For example, the following indexing call will make sure to add a tag to the document without losing any potential change to the description or an addition of another tag by another API: