Modules

One of the main roles of the master is to decide which shards to allocate to which nodes, and when to move shards between nodes in order to rebalance the cluster.

There are a number of settings available to control the shard allocation process:

Besides these, there are a few other miscellaneous cluster-level settings.

All of the settings in this section are dynamic settings which can be updated on a live cluster with the cluster-update-settings API.

The discovery module is responsible for discovering nodes within a cluster, as well as electing a master node.

Note, Elasticsearch is a peer to peer based system, nodes communicate with one another directly if operations are delegated / broadcast. All the main APIs (index, delete, search) do not communicate with the master node. The responsibility of the master node is to maintain the global cluster state, and act if nodes join or leave the cluster by reassigning shards. Each time a cluster state is changed, the state is made known to the other nodes in the cluster (the manner depends on the actual discovery implementation).

The cluster.name allows to create separated clusters from one another. The default value for the cluster name is elasticsearch, though it is recommended to change this to reflect the logical group name of the cluster running.

The local gateway module stores the cluster state and shard data across full cluster restarts.

The following static settings, which must be set on every master node, control how long a freshly elected master should wait before it tries to recover the cluster state and the cluster’s data:

Once the recover_after_time duration has timed out, recovery will start as long as the following conditions are met:

The http module allows to expose Elasticsearch APIs over HTTP.

The http mechanism is completely asynchronous in nature, meaning that there is no blocking thread waiting for a response. The benefit of using asynchronous communication for HTTP is solving the C10k problem.

When possible, consider using HTTP keep alive when connecting for better performance and try to get your favorite client not to do HTTP chunking.

The settings in the table below can be configured for HTTP. Note that none of them are dynamically updatable so for them to take effect they should be set in the Elasticsearch configuration file.

It also uses the common network settings.

The http module can be completely disabled and not started by setting http.enabled to false. Elasticsearch nodes (and Java clients) communicate internally using the transport interface, not HTTP. It might make sense to disable the http layer entirely on nodes which are not meant to serve REST requests directly. For instance, you could disable HTTP on data-only nodes if you also have client nodes which are intended to serve all REST requests. Be aware, however, that you will not be able to send any REST requests (eg to retrieve node stats) directly to nodes which have HTTP disabled.

The indices module controls index-related settings that are globally managed for all indices, rather than being configurable at a per-index level.

Available settings include:

Elasticsearch binds to localhost only by default. This is sufficient for you to run a local development server (or even a development cluster, if you start multiple nodes on the same machine), but you will need to configure some basic network settings in order to run a real production cluster across multiple servers.

The following special values may be passed to network.host:

These special values will work over both IPv4 and IPv6 by default, but you can also limit this with the use of :ipv4 of :ipv6 specifiers. For example, en0:ipv4 would only bind to the IPv4 addresses of interface en0.

The network.host setting explained in Commonly used network settings is a shortcut which sets the bind host and the publish host at the same time. In advanced used cases, such as when running behind a proxy server, you may need to set these settings to different values:

Both of the above settings can be configured just like network.host — they accept IP addresses, host names, and special values.

Any component that uses TCP (like the HTTP and Transport modules) share the following settings:

An Elasticsearch node exposes two network protocols which inherit the above settings, but may be further configured independently:

Any time that you start an instance of Elasticsearch, you are starting a node. A collection of connected nodes is called a cluster. If you are running a single node of Elasticsearch, then you have a cluster of one node.

Every node in the cluster can handle HTTP and Transport traffic by default. The transport layer is used exclusively for communication between nodes and the {javaclient}/transport-client.html[Java TransportClient]; the HTTP layer is used only by external REST clients.

All nodes know about all the other nodes in the cluster and can forward client requests to the appropriate node.

By default, a node is all of the following types: master-eligible, data, ingest, and machine learning (if available).

By default a node is a master-eligible node and a data node, plus it can pre-process documents through ingest pipelines. This is very convenient for small clusters but, as the cluster grows, it becomes important to consider separating dedicated master-eligible nodes from dedicated data nodes.

The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. It is important for cluster health to have a stable master node.

Any master-eligible node (all nodes by default) may be elected to become the master node by the master election process.

Indexing and searching your data is CPU-, memory-, and I/O-intensive work which can put pressure on a node’s resources. To ensure that your master node is stable and not under pressure, it is a good idea in a bigger cluster to split the roles between dedicated master-eligible nodes and dedicated data nodes.

While master nodes can also behave as coordinating nodes and route search and indexing requests from clients to data nodes, it is better not to use dedicated master nodes for this purpose. It is important for the stability of the cluster that master-eligible nodes do as little work as possible.

To create a dedicated master-eligible node in the {default-dist}, set:

To create a dedicated master-eligible node in the {oss-dist}, set:

To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting (which defaults to 1) so that each master-eligible node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.

To explain, imagine that you have a cluster consisting of two master-eligible nodes. A network failure breaks communication between these two nodes. Each node sees one master-eligible node…​ itself. With minimum_master_nodes set to the default of 1, this is sufficient to form a cluster. Each node elects itself as the new master (thinking that the other master-eligible node has died) and the result is two clusters, or a split brain. These two nodes will never rejoin until one node is restarted. Any data that has been written to the restarted node will be lost.

Now imagine that you have a cluster with three master-eligible nodes, and minimum_master_nodes set to 2. If a network split separates one node from the other two nodes, the side with one node cannot see enough master-eligible nodes and will realise that it cannot elect itself as master. The side with two nodes will elect a new master (if needed) and continue functioning correctly. As soon as the network split is resolved, the single node will rejoin the cluster and start serving requests again.

This setting should be set to a quorum of master-eligible nodes:

In other words, if there are three master-eligible nodes, then minimum master nodes should be set to (3 / 2) + 1 or 2:

To be able to remain available when one of the master-eligible nodes fails, clusters should have at least three master-eligible nodes, with minimum_master_nodes set accordingly. A rolling upgrade, performed without any downtime, also requires at least three master-eligible nodes to avoid the possibility of data loss if a network split occurs while the upgrade is in progress.

This setting can also be changed dynamically on a live cluster with the cluster update settings API:

Data nodes hold the shards that contain the documents you have indexed. Data nodes handle data related operations like CRUD, search, and aggregations. These operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more data nodes if they are overloaded.

The main benefit of having dedicated data nodes is the separation of the master and data roles.

To create a dedicated data node in the {default-dist}, set:

To create a dedicated data node in the {oss-dist}, set:

Ingest nodes can execute pre-processing pipelines, composed of one or more ingest processors. Depending on the type of operations performed by the ingest processors and the required resources, it may make sense to have dedicated ingest nodes, that will only perform this specific task.

To create a dedicated ingest node in the {default-dist}, set:

To create a dedicated ingest node in the {oss-dist}, set:

If you take away the ability to be able to handle master duties, to hold data, and pre-process documents, then you are left with a coordinating node that can only route requests, handle the search reduce phase, and distribute bulk indexing. Essentially, coordinating only nodes behave as smart load balancers.

Coordinating only nodes can benefit large clusters by offloading the coordinating node role from data and master-eligible nodes. They join the cluster and receive the full cluster state, like every other node, and they use the cluster state to route requests directly to the appropriate place(s).

To create a dedicated coordinating node in the {default-dist}, set:

To create a dedicated coordinating node in the {oss-dist}, set:

The {ml-features} provide {ml} nodes, which run jobs and handle {ml} API requests. If xpack.ml.enabled is set to true and node.ml is set to false, the node can service API requests but it cannot run jobs.

If you want to use {ml-features} in your cluster, you must enable {ml} (set xpack.ml.enabled to true) on all master-eligible nodes. If you have the {oss-dist}, do not use these settings.

For more information about these settings, see [ml-settings].

To create a dedicated {ml} node in the {default-dist}, set:

Every data and master-eligible node requires access to a data directory where shards and index and cluster metadata will be stored. The path.data defaults to $ES_HOME/data but can be configured in the elasticsearch.yml config file an absolute path or a path relative to $ES_HOME as follows:

Like all node settings, it can also be specified on the command line as:

The data path can be shared by multiple nodes, even by nodes from different clusters. This is very useful for testing failover and different configurations on your development machine. In production, however, it is recommended to run only one node of Elasticsearch per server.

By default, Elasticsearch is configured to prevent more than one node from sharing the same data path. To allow for more than one node (e.g., on your development machine), use the setting node.max_local_storage_nodes and set this to a positive integer larger than one.

More node settings can be found in Modules. Of particular note are the cluster.name, the node.name and the network settings.

Plugins are a way to enhance the basic Elasticsearch functionality in a custom manner. They range from adding custom mapping types, custom analyzers (in a more built in fashion), custom script engines, custom discovery and more.

See the {plugins}/index.html[Plugins documentation] for more.

A snapshot is a backup taken from a running Elasticsearch cluster. You can take a snapshot of individual indices or of the entire cluster and store it in a repository on a shared filesystem, and there are plugins that support remote repositories on S3, HDFS, Azure, Google Cloud Storage and more.

Snapshots are taken incrementally. This means that when creating a snapshot of an index Elasticsearch will avoid copying any data that is already stored in the repository as part of an earlier snapshot of the same index. Therefore it can be efficient to take snapshots of your cluster quite frequently.

Snapshots can be restored into a running cluster via the restore API. When restoring an index it is possible to alter the name of the restored index as well as some of its settings, allowing a great deal of flexibility in how the snapshot and restore functionality can be used.

A snapshot contains a copy of the on-disk data structures that make up an index. This means that snapshots can only be restored to versions of Elasticsearch that can read the indices:

Conversely, snapshots of indices created in 1.x cannot be restored to 5.x or 6.x, and snapshots of indices created in 2.x cannot be restored to 6.x.

Each snapshot can contain indices created in various versions of Elasticsearch, and when restoring a snapshot it must be possible to restore all of the indices into the target cluster. If any indices in a snapshot were created in an incompatible version, you will not be able restore the snapshot.

If you end up in a situation where you need to restore a snapshot of an index that is incompatible with the version of the cluster you are currently running, you can restore it on the latest compatible version and use reindex-from-remote to rebuild the index on the current version. Reindexing from remote is only possible if the original index has source enabled. Retrieving and reindexing the data can take significantly longer than simply restoring a snapshot. If you have a large amount of data, we recommend testing the reindex from remote process with a subset of your data to understand the time requirements before proceeding.

You must register a snapshot repository before you can perform snapshot and restore operations. We recommend creating a new snapshot repository for each major version. The valid repository settings depend on the repository type.

If you register same snapshot repository with multiple clusters, only one cluster should have write access to the repository. All other clusters connected to that repository should set the repository to readonly mode.

To retrieve information about a registered repository, use a GET request:

which returns:

To retrieve information about multiple repositories, specify a comma-delimited list of repositories. You can also use the * wildcard when specifying repository names. For example, the following request retrieves information about all of the snapshot repositories that start with repo or contain backup:

To retrieve information about all registered snapshot repositories, omit the repository name or specify _all:

or

The shared file system repository ("type": "fs") uses the shared file system to store snapshots. In order to register the shared file system repository it is necessary to mount the same shared filesystem to the same location on all master and data nodes. This location (or one of its parent directories) must be registered in the path.repo setting on all master and data nodes.

Assuming that the shared filesystem is mounted to /mount/backups/my_fs_backup_location, the following setting should be added to elasticsearch.yml file:

The path.repo setting supports Microsoft Windows UNC paths as long as at least server name and share are specified as a prefix and back slashes are properly escaped:

After all nodes are restarted, the following command can be used to register the shared file system repository with the name my_fs_backup:

If the repository location is specified as a relative path this path will be resolved against the first path specified in path.repo:

The following settings are supported:

The URL repository ("type": "url") can be used as an alternative read-only way to access data created by the shared file system repository. The URL specified in the url parameter should point to the root of the shared filesystem repository. The following settings are supported:

URL Repository supports the following protocols: "http", "https", "ftp", "file" and "jar". URL repositories with http:, https:, and ftp: URLs has to be whitelisted by specifying allowed URLs in the repositories.url.allowed_urls setting. This setting supports wildcards in the place of host, path, query, and fragment. For example:

URL repositories with file: URLs can only point to locations registered in the path.repo setting similar to shared file system repository.

A source repository enables you to create minimal, source-only snapshots that take up to 50% less space on disk. Source only snapshots contain stored fields and index metadata. They do not include index or doc values structures and are not searchable when restored. After restoring a source-only snapshot, you must reindex the data into a new index.

Source repositories delegate to another snapshot repository for storage.

When you create a source repository, you must specify the type and name of the delegate repository where the snapshots will be stored:

Other repository backends are available in these official plugins:

When a repository is registered, it’s immediately verified on all master and data nodes to make sure that it is functional on all nodes currently present in the cluster. The verify parameter can be used to explicitly disable the repository verification when registering or updating a repository:

The verification process can also be executed manually by running the following command:

It returns a list of nodes where repository was successfully verified or an error message if verification process failed.

A repository can contain multiple snapshots of the same cluster. Snapshots are identified by unique names within the cluster. A snapshot with the name snapshot_1 in the repository my_backup can be created by executing the following command:

The wait_for_completion parameter specifies whether or not the request should return immediately after snapshot initialization (default) or wait for snapshot completion. During snapshot initialization, information about all previous snapshots is loaded into the memory, which means that in large repositories it may take several seconds (or even minutes) for this command to return even if the wait_for_completion parameter is set to false.

By default a snapshot of all open and started indices in the cluster is created. This behavior can be changed by specifying the list of indices in the body of the snapshot request.

The list of indices that should be included into the snapshot can be specified using the indices parameter that supports multi index syntax. The snapshot request also supports the ignore_unavailable option. Setting it to true will cause indices that do not exist to be ignored during snapshot creation. By default, when ignore_unavailable option is not set and an index is missing the snapshot request will fail. By setting include_global_state to false it’s possible to prevent the cluster global state to be stored as part of the snapshot. By default, the entire snapshot will fail if one or more indices participating in the snapshot don’t have all primary shards available. This behaviour can be changed by setting partial to true.

Snapshot names can be automatically derived using date math expressions, similarly as when creating new indices. Note that special characters need to be URI encoded.

For example, creating a snapshot with the current day in the name, like snapshot-2018.05.11, can be achieved with the following command:

The index snapshot process is incremental. In the process of making the index snapshot Elasticsearch analyses the list of the index files that are already stored in the repository and copies only files that were created or changed since the last snapshot. That allows multiple snapshots to be preserved in the repository in a compact form. Snapshotting process is executed in non-blocking fashion. All indexing and searching operation can continue to be executed against the index that is being snapshotted. However, a snapshot represents the point-in-time view of the index at the moment when snapshot was created, so no records that were added to the index after the snapshot process was started will be present in the snapshot. The snapshot process starts immediately for the primary shards that has been started and are not relocating at the moment. Before version 1.2.0, the snapshot operation fails if the cluster has any relocating or initializing primaries of indices participating in the snapshot. Starting with version 1.2.0, Elasticsearch waits for relocation or initialization of shards to complete before snapshotting them.

Besides creating a copy of each index the snapshot process can also store global cluster metadata, which includes persistent cluster settings and templates. The transient settings and registered snapshot repositories are not stored as part of the snapshot.

Only one snapshot process can be executed in the cluster at any time. While snapshot of a particular shard is being created this shard cannot be moved to another node, which can interfere with rebalancing process and allocation filtering. Elasticsearch will only be able to move a shard to another node (according to the current allocation filtering settings and rebalancing algorithm) once the snapshot is finished.

Once a snapshot is created information about this snapshot can be obtained using the following command:

This command returns basic information about the snapshot including start and end time, version of Elasticsearch that created the snapshot, the list of included indices, the current state of the snapshot and the list of failures that occurred during the snapshot. The snapshot state can be

Similar as for repositories, information about multiple snapshots can be queried in one go, supporting wildcards as well:

All snapshots currently stored in the repository can be listed using the following command:

The command fails if some of the snapshots are unavailable. The boolean parameter ignore_unavailable can be used to return all snapshots that are currently available.

Getting all snapshots in the repository can be costly on cloud-based repositories, both from a cost and performance perspective. If the only information required is the snapshot names/uuids in the repository and the indices in each snapshot, then the optional boolean parameter verbose can be set to false to execute a more performant and cost-effective retrieval of the snapshots in the repository. Note that setting verbose to false will omit all other information about the snapshot such as status information, the number of snapshotted shards, etc. The default value of the verbose parameter is true.

A currently running snapshot can be retrieved using the following command:

A snapshot can be deleted from the repository using the following command:

When a snapshot is deleted from a repository, Elasticsearch deletes all files that are associated with the deleted snapshot and not used by any other snapshots. If the deleted snapshot operation is executed while the snapshot is being created the snapshotting process will be aborted and all files created as part of the snapshotting process will be cleaned. Therefore, the delete snapshot operation can be used to cancel long running snapshot operations that were started by mistake.

A repository can be unregistered using the following command:

When a repository is unregistered, Elasticsearch only removes the reference to the location where the repository is storing the snapshots. The snapshots themselves are left untouched and in place.

A snapshot can be restored using the following command:

By default, all indices in the snapshot are restored, and the cluster state is not restored. It’s possible to select indices that should be restored as well as to allow the global cluster state from being restored by using indices and include_global_state options in the restore request body. The list of indices supports multi index syntax. The rename_pattern and rename_replacement options can be also used to rename indices on restore using regular expression that supports referencing the original text as explained here. Set include_aliases to false to prevent aliases from being restored together with associated indices

The restore operation can be performed on a functioning cluster. However, an existing index can be only restored if it’s closed and has the same number of shards as the index in the snapshot. The restore operation automatically opens restored indices if they were closed and creates new indices if they didn’t exist in the cluster. If cluster state is restored with include_global_state (defaults to false), the restored templates that don’t currently exist in the cluster are added and existing templates with the same name are replaced by the restored templates. The restored persistent settings are added to the existing persistent settings.

By default, the entire restore operation will fail if one or more indices participating in the operation don’t have snapshots of all shards available. It can occur if some shards failed to snapshot for example. It is still possible to restore such indices by setting partial to true. Please note, that only successfully snapshotted shards will be restored in this case and all missing shards will be recreated empty.

Most of index settings can be overridden during the restore process. For example, the following command will restore the index index_1 without creating any replicas while switching back to default refresh interval:

Please note, that some settings such as index.number_of_shards cannot be changed during restore operation.

The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore it’s possible to restore a snapshot made from one cluster into another cluster. All that is required is registering the repository containing the snapshot in the new cluster and starting the restore process. The new cluster doesn’t have to have the same size or topology. However, the version of the new cluster should be the same or newer (only 1 major version newer) than the cluster that was used to create the snapshot. For example, you can restore a 1.x snapshot to a 2.x cluster, but not a 1.x snapshot to a 5.x cluster.

If the new cluster has a smaller size additional considerations should be made. First of all it’s necessary to make sure that new cluster have enough capacity to store all indices in the snapshot. It’s possible to change indices settings during restore to reduce the number of replicas, which can help with restoring snapshots into smaller cluster. It’s also possible to select only subset of the indices using the indices parameter.

If indices in the original cluster were assigned to particular nodes using shard allocation filtering, the same rules will be enforced in the new cluster. Therefore if the new cluster doesn’t contain nodes with appropriate attributes that a restored index can be allocated on, such index will not be successfully restored unless these index allocation settings are changed during restore operation.

The restore operation also checks that restored persistent settings are compatible with the current cluster to avoid accidentally restoring an incompatible settings such as discovery.zen.minimum_master_nodes and as a result disable a smaller cluster until the required number of master eligible nodes is added. If you need to restore a snapshot with incompatible persistent settings, try restoring it without the global cluster state.

A list of currently running snapshots with their detailed status information can be obtained using the following command:

In this format, the command will return information about all currently running snapshots. By specifying a repository name, it’s possible to limit the results to a particular repository:

If both repository name and snapshot id are specified, this command will return detailed status information for the given snapshot even if it’s not currently running:

The output looks similar to the following:

The output is composed of different sections. The stats sub-object provides details on the number and size of files that were snapshotted. As snapshots are incremental, copying only the Lucene segments that are not already in the repository, the stats object contains a total section for all the files that are referenced by the snapshot, as well as an incremental section for those files that actually needed to be copied over as part of the incremental snapshotting. In case of a snapshot that’s still in progress, there’s also a processed section that contains information about the files that are in the process of being copied.

Note: Properties number_of_files, processed_files, total_size_in_bytes and processed_size_in_bytes are used for backward compatibility reasons with older 5.x and 6.x versions. These fields will be removed in Elasticsearch v7.0.0.

Multiple ids are also supported:

There are several ways to monitor the progress of the snapshot and restores processes while they are running. Both operations support wait_for_completion parameter that would block client until the operation is completed. This is the simplest method that can be used to get notified about operation completion.

The snapshot operation can be also monitored by periodic calls to the snapshot info:

Please note that snapshot info operation uses the same resources and thread pool as the snapshot operation. So, executing a snapshot info operation while large shards are being snapshotted can cause the snapshot info operation to wait for available resources before returning the result. On very large shards the wait time can be significant.

To get more immediate and complete information about snapshots the snapshot status command can be used instead:

While snapshot info method returns only basic information about the snapshot in progress, the snapshot status returns complete breakdown of the current state for each shard participating in the snapshot.

The restore process piggybacks on the standard recovery mechanism of the Elasticsearch. As a result, standard recovery monitoring services can be used to monitor the state of restore. When restore operation is executed the cluster typically goes into red state. It happens because the restore operation starts with "recovering" primary shards of the restored indices. During this operation the primary shards become unavailable which manifests itself in the red cluster state. Once recovery of primary shards is completed Elasticsearch is switching to standard replication process that creates the required number of replicas at this moment cluster switches to the yellow state. Once all required replicas are created, the cluster switches to the green states.

The cluster health operation provides only a high level status of the restore process. It’s possible to get more detailed insight into the current state of the recovery process by using indices recovery and cat recovery APIs.

The snapshot and restore framework allows running only one snapshot or one restore operation at a time. If a currently running snapshot was executed by mistake, or takes unusually long, it can be terminated using the snapshot delete operation. The snapshot delete operation checks if the deleted snapshot is currently running and if it does, the delete operation stops that snapshot before deleting the snapshot data from the repository.

The restore operation uses the standard shard recovery mechanism. Therefore, any currently running restore operation can be canceled by deleting indices that are being restored. Please note that data for all deleted indices will be removed from the cluster as a result of this operation.

Many snapshot and restore operations are affected by cluster and index blocks. For example, registering and unregistering repositories require write global metadata access. The snapshot operation requires that all indices and their metadata as well as the global metadata were readable. The restore operation requires the global metadata to be writable, however the index level blocks are ignored during restore because indices are essentially recreated during restore. Please note that a repository content is not part of the cluster and therefore cluster blocks don’t affect internal repository operations such as listing or deleting snapshots from an already registered repository.

A node holds several thread pools in order to improve how threads memory consumption are managed within a node. Many of these pools also have queues associated with them, which allow pending requests to be held instead of discarded.

There are several thread pools, but the important ones include:

Changing a specific thread pool can be done by setting its type-specific parameters; for example, changing the index thread pool to have more threads:

The following are the types of thread pools and their respective parameters:

The fixed thread pool holds a fixed size of threads to handle the requests with a queue (optionally bounded) for pending requests that have no threads to service them.

The size parameter controls the number of threads, and defaults to the number of cores times 5.

The queue_size allows to control the size of the queue of pending requests that have no threads to execute them. By default, it is set to -1 which means its unbounded. When a request comes in and the queue is full, it will abort the request.

experimental[]

The fixed_auto_queue_size thread pool holds a fixed size of threads to handle the requests with a bounded queue for pending requests that have no threads to service them. It’s similar to the fixed threadpool, however, the queue_size automatically adjusts according to calculations based on Little’s Law. These calculations will potentially adjust the queue_size up or down by 50 every time auto_queue_frame_size operations have been completed.

The size parameter controls the number of threads, and defaults to the number of cores times 5.

The queue_size allows to control the initial size of the queue of pending requests that have no threads to execute them.

The min_queue_size setting controls the minimum amount the queue_size can be adjusted to.

The max_queue_size setting controls the maximum amount the queue_size can be adjusted to.

The auto_queue_frame_size setting controls the number of operations during which measurement is taken before the queue is adjusted. It should be large enough that a single operation cannot unduly bias the calculation.

The target_response_time is a time value setting that indicates the targeted average response time for tasks in the thread pool queue. If tasks are routinely above this time, the thread pool queue will be adjusted down so that tasks are rejected.

The scaling thread pool holds a dynamic number of threads. This number is proportional to the workload and varies between the value of the core and max parameters.

The keep_alive parameter determines how long a thread should be kept around in the thread pool without it doing any work.

The number of processors is automatically detected, and the thread pool settings are automatically set based on it. In some cases it can be useful to override the number of detected processors. This can be done by explicitly setting the processors setting.

There are a few use-cases for explicitly overriding the processors setting:

In order to check the number of processors detected, use the nodes info API with the os flag.

The transport module is used for internal communication between nodes within the cluster. Each call that goes from one node to the other uses the transport module (for example, when an HTTP GET request is processed by one node, and should actually be processed by another node that holds the data). The transport module is also used for the TransportClient in the {es} Java API.

The transport mechanism is completely asynchronous in nature, meaning that there is no blocking thread waiting for a response. The benefit of using asynchronous communication is first solving the C10k problem, as well as being the ideal solution for scatter (broadcast) / gather operations such as search in Elasticsearch.

The internal transport communicates over TCP. You can configure it with the following settings:

It also uses the common network settings.

Elasticsearch allows you to bind to multiple ports on different interfaces by the use of transport profiles. See this example configuration

The default profile is special. It is used as a fallback for any other profiles, if those do not have a specific configuration setting set, and is how this node connects to other nodes in the cluster.

The following parameters can be configured on each transport profile, as in the example above:

Elasticsearch opens a number of long-lived TCP connections between each pair of nodes in the cluster, and some of these connections may be idle for an extended period of time. Nonetheless, Elasticsearch requires these connections to remain open, and it can disrupt the operation of the cluster if any inter-node connections are closed by an external influence such as a firewall. It is important to configure your network to preserve long-lived idle connections between Elasticsearch nodes, for instance by leaving tcp.keep_alive enabled and ensuring that the keepalive interval is shorter than any timeout that might cause idle connections to be closed, or by setting transport.ping_schedule if keepalives cannot be configured.

By default, the transport.compress setting is false and network-level request compression is disabled between nodes in the cluster. This default normally makes sense for local cluster communication as compression has a noticeable CPU cost and local clusters tend to be set up with fast network connections between nodes.

The transport.compress setting always configures local cluster request compression and is the fallback setting for remote cluster request compression. If you want to configure remote request compression differently than local request compression, you can set it on a per-remote cluster basis using the cluster.remote.${cluster_alias}.transport.compress setting.

The compression settings do not configure compression for responses. {es} will compress a response if the inbound request was compressed—​even when compression is not enabled. Similarly, {es} will not compress a response if the inbound request was uncompressed—​even when compression is enabled.

The transport module has a dedicated tracer logger which, when activated, logs incoming and out going requests. The log can be dynamically activated by settings the level of the org.elasticsearch.transport.TransportService.tracer logger to TRACE:

You can also control which actions will be traced, using a set of include and exclude wildcard patterns. By default every request will be traced except for fault detection pings:

deprecated[5.4.0,The tribe node is deprecated in favour of {ccs-cap} and will be removed in Elasticsearch 7.0.]

The tribes feature allows a tribe node to act as a federated client across multiple clusters.

The tribe node works by retrieving the cluster state from all connected clusters and merging them into a global cluster state. With this information at hand, it is able to perform read and write operations against the nodes in all clusters as if they were local. Note that a tribe node needs to be able to connect to each single node in every configured cluster.

The elasticsearch.yml config file for a tribe node just needs to list the clusters that should be joined, for instance:

The example above configures connections to two clusters, name t1 and t2 respectively. The tribe node will create a node client to connect each cluster using unicast discovery by default. Any other settings for the connection can be configured under tribe.{name}, just like the cluster.name in the example.

The merged global cluster state means that almost all operations work in the same way as a single cluster: distributed search, suggest, percolation, indexing, etc.

However, there are a few exceptions:

The tribe node can be configured to block all write operations and all metadata operations with:

The tribe node can also configure blocks on selected indices:

When there is a conflict and multiple clusters hold the same index, by default the tribe node will pick one of them. This can be configured using the tribe.on_conflict setting. It defaults to any, but can be set to drop (drop indices that have a conflict), or prefer_[tribeName] to prefer the index from a specific tribe.

The tribe node starts a node client for each listed cluster. The following configuration options are passed down from the tribe node to each node client:

Almost any setting (except for path.*) may be configured at the node client level itself, in which case it will override any passed through setting from the tribe node. Settings you may want to set at the node client level include:

The remote clusters module enables you to establish uni-directional connections to a remote cluster. This functionality is used in {ccs}.

Remote cluster connections work by configuring a remote cluster and connecting only to a limited number of nodes in that remote cluster. Each remote cluster is referenced by a name and a list of seed nodes. When a remote cluster is registered, its cluster state is retrieved from one of the seed nodes and up to three gateway nodes are selected to be connected to as part of remote cluster requests. All the communication required between different clusters goes through the transport layer. Remote cluster connections consist of uni-directional connections from the coordinating node to the selected remote gateway nodes only.

The gateway nodes selection depends on the following criteria:

You can configure remote clusters globally by using cluster settings, which you can update dynamically. Alternatively, you can configure them locally on individual nodes by using the elasticsearch.yml file.

If you specify the settings in elasticsearch.yml files, only the nodes with those settings can connect to the remote cluster. In other words, functionality that relies on remote cluster requests must be driven specifically from those nodes. For example:

For more information about the optional transport settings, see Transport.

If you use cluster settings, the remote clusters are available on every node in the cluster. For example:

You can dynamically update the compression and ping schedule settings. However, you must re-include seeds in the settings update request. For example:

A remote cluster can be deleted from the cluster settings by setting its seeds and optional settings to null :

You can use the remote cluster info API to retrieve information about the configured remote clusters, as well as the remote nodes that the node is connected to.

The {ccs} feature allows any node to act as a federated client across multiple clusters. In contrast to the tribe node feature, a {ccs} node won’t join the remote cluster, instead it connects to a remote cluster in a light fashion in order to execute federated search requests. For details on communication and compatibility between different clusters, see Remote clusters.

{ccs-cap} requires configuring remote clusters.

To search the twitter index on remote cluster cluster_one the index name must be prefixed with the alias of the remote cluster followed by the : character:

In contrast to the tribe feature cross cluster search can also search indices with the same name on different clusters:

Search results are disambiguated the same way as the indices are disambiguated in the request. Indices with same names are treated as different indices when results are merged. All results retrieved from an index located in a remote cluster are prefixed with their corresponding cluster alias:

By default, all remote clusters that are searched via {ccs} need to be available when the search request is executed. Otherwise, the whole request fails; even if some of the clusters are available, no search results are returned. You can use the boolean skip_unavailable setting to make remote clusters optional. By default, it is set to false.