"Fossies" - the Fresh Open Source Software Archive

Member "elasticsearch-6.8.23/docs/reference/setup.asciidoc" (29 Dec 2021, 1682 Bytes) of package /linux/www/elasticsearch-6.8.23-src.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming AsciiDoc format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field.

Installing Elasticsearch

Hosted Elasticsearch

{es} can be run on your own hardware or using our hosted {ess} on {ecloud}, which is available on AWS, GCP and Azure. You can {ess-trial}[try out the hosted service] for free.

Installing Elasticsearch Yourself

Elasticsearch is provided in the following package formats:

zip/tar.gz

The zip and tar.gz packages are suitable for installation on any system and are the easiest choice for getting started with Elasticsearch on most systems.

deb

The deb package is suitable for Debian, Ubuntu, and other Debian-based systems. Debian packages may be downloaded from the Elasticsearch website or from our Debian repository.

rpm

The rpm package is suitable for installation on Red Hat, Centos, SLES, OpenSuSE and other RPM-based systems. RPMs may be downloaded from the Elasticsearch website or from our RPM repository.

msi

beta[] The msi package is suitable for installation on Windows 64-bit systems with at least .NET 4.5 framework installed, and is the easiest choice for getting started with Elasticsearch on Windows. MSIs may be downloaded from the Elasticsearch website.

docker

Images are available for running Elasticsearch as Docker containers. They may be downloaded from the Elastic Docker Registry.

{ref}/docker.html[Install {es} with Docker]

Configuration Management Tools

We also provide the following configuration management tools to help with large deployments:

Install Elasticsearch with .zip or .tar.gz

Elasticsearch is provided as a .zip and as a .tar.gz package. These packages can be used to install Elasticsearch on any system and are the easiest package format to use when trying out Elasticsearch.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Download and install the .zip package

The .zip archive for Elasticsearch v{version} can be downloaded and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip.sha512
shasum -a 512 -c elasticsearch-{version}.zip.sha512 <1>
unzip elasticsearch-{version}.zip
cd elasticsearch-{version}/ <2>
  1. Compares the SHA of the downloaded .zip archive and the published checksum, which should output elasticsearch-{version}.zip: OK.

  2. This directory is known as $ES_HOME.

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.zip

Download and install the .tar.gz package

The .tar.gz archive for Elasticsearch v{version} can be downloaded and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.tar.gz.sha512
shasum -a 512 -c elasticsearch-{version}.tar.gz.sha512 <1>
tar -xzf elasticsearch-{version}.tar.gz
cd elasticsearch-{version}/ <2>
  1. Compares the SHA of the downloaded .tar.gz archive and the published checksum, which should output elasticsearch-{version}.tar.gz: OK.

  2. This directory is known as $ES_HOME.

Alternatively, you can download the following package, which includes only Apache 2.0 licensed code: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.tar.gz

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

./bin/elasticsearch

By default, Elasticsearch runs in the foreground, prints its logs to the standard output (stdout), and can be stopped by pressing Ctrl-C.

Note
All scripts packaged with Elasticsearch require a version of Bash that supports arrays and assume that Bash is available at /bin/bash. As such, Bash should be available at this path either directly or via a symbolic link.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "{build_flavor}",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "{lucene_version}",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Log printing to stdout can be disabled using the -q or --quiet option on the command line.

Running as a daemon

To run Elasticsearch as a daemon, specify -d on the command line, and record the process ID in a file using the -p option:

./bin/elasticsearch -d -p pid

Log messages can be found in the $ES_HOME/logs/ directory.

To shut down Elasticsearch, kill the process ID recorded in the pid file:

pkill -F pid
Note
The startup scripts provided in the RPM and Debian packages take care of starting and stopping the Elasticsearch process for you.

Configuring Elasticsearch on the command line

Elasticsearch loads its configuration from the $ES_HOME/config/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

Any settings that can be specified in the config file can also be specified on the command line, using the -E syntax as follows:

./bin/elasticsearch -d -Ecluster.name=my_cluster -Enode.name=node_1
Tip
Typically, any cluster-wide settings (like cluster.name) should be added to the elasticsearch.yml config file, while any node-specific settings such as node.name could be specified on the command line.

Directory layout of .zip and .tar.gz archives

The .zip and .tar.gz packages are entirely self-contained. All files and directories are, by default, contained within $ES_HOME — the directory created when unpacking the archive.

This is very convenient because you don’t have to create any directories to start using Elasticsearch, and uninstalling Elasticsearch is as easy as removing the $ES_HOME directory. However, it is advisable to change the default locations of the config directory, the data directory, and the logs directory so that you do not delete important data later on.

Type Description Default Location Setting

home

Elasticsearch home directory or $ES_HOME

Directory created by unpacking the archive

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

$ES_HOME/bin

conf

Configuration files including elasticsearch.yml

$ES_HOME/config

ES_PATH_CONF

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

$ES_HOME/data

path.data

logs

Log files location.

$ES_HOME/logs

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

$ES_HOME/plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

script

Location of script files.

$ES_HOME/scripts

path.scripts

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with .zip on Windows

Elasticsearch can be installed on Windows using the .zip package. This comes with a elasticsearch-service.bat command which will setup Elasticsearch to run as a service.

Tip
Elasticsearch has historically been installed on Windows using the .zip archive. An MSI installer package is available that provides the easiest getting started experience for Windows. You can continue using the .zip approach if you prefer.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Download and install the .zip package

Download the .zip archive for Elasticsearch v{version} from: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.zip

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.zip

Unzip it with your favourite unzip tool. This will create a folder called elasticsearch-{version}, which we will refer to as %ES_HOME%. In a terminal window, cd to the %ES_HOME% directory, for instance:

cd c:\elasticsearch-{version}

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

.\bin\elasticsearch.bat

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl-C.

Configuring Elasticsearch on the command line

Elasticsearch loads its configuration from the %ES_HOME%\config\elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

Any settings that can be specified in the config file can also be specified on the command line, using the -E syntax as follows:

.\bin\elasticsearch.bat -Ecluster.name=my_cluster -Enode.name=node_1
Note
Values that contain spaces must be surrounded with quotes. For instance -Epath.logs="C:\My Logs\logs".
Tip
Typically, any cluster-wide settings (like cluster.name) should be added to the elasticsearch.yml config file, while any node-specific settings such as node.name could be specified on the command line.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "{build_flavor}",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "{lucene_version}",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Installing Elasticsearch as a Service on Windows

Elasticsearch can be installed as a service to run in the background or start automatically at boot time without any user interaction. This can be achieved through the elasticsearch-service.bat script in the bin\ folder which allows one to install, remove, manage or configure the service and potentially start and stop the service, all from the command-line.

c:\elasticsearch-{version}\bin>elasticsearch-service.bat

Usage: elasticsearch-service.bat install|remove|start|stop|manager [SERVICE_ID]

The script requires one parameter (the command to execute) followed by an optional one indicating the service id (useful when installing multiple Elasticsearch services).

The commands available are:

install

Install Elasticsearch as a service

remove

Remove the installed Elasticsearch service (and stop the service if started)

start

Start the Elasticsearch service (if installed)

stop

Stop the Elasticsearch service (if started)

manager

Start a GUI for managing the installed service

The name of the service and the value of JAVA_HOME will be made available during install:

c:\elasticsearch-{version}\bin>elasticsearch-service.bat install
Installing service      :  "elasticsearch-service-x64"
Using JAVA_HOME (64-bit):  "c:\jvm\jdk1.8"
The service 'elasticsearch-service-x64' has been installed.
Note
While a JRE can be used for the Elasticsearch service, due to its use of a client VM (as opposed to a server JVM which offers better performance for long-running applications) its usage is discouraged and a warning will be issued.
Note
The system environment variable JAVA_HOME should be set to the path to the JDK installation that you want the service to use. If you upgrade the JDK, you are not required to the reinstall the service but you must set the value of the system environment variable JAVA_HOME to the path to the new JDK installation. However, upgrading across JVM types (e.g. JRE versus SE) is not supported, and does require the service to be reinstalled.

Customizing service settings

The Elasticsearch service can be configured prior to installation by setting the following environment variables (either using the set command from the command line, or through the System Properties→Environment Variables GUI).

SERVICE_ID

A unique identifier for the service. Useful if installing multiple instances on the same machine. Defaults to elasticsearch-service-x64.

SERVICE_USERNAME

The user to run as, defaults to the local system account.

SERVICE_PASSWORD

The password for the user specified in %SERVICE_USERNAME%.

SERVICE_DISPLAY_NAME

The name of the service. Defaults to Elasticsearch <version> %SERVICE_ID%.

SERVICE_DESCRIPTION

The description of the service. Defaults to Elasticsearch <version> Windows Service - https://elastic.co.

JAVA_HOME

The installation directory of the desired JVM to run the service under.

SERVICE_LOG_DIR

Service log directory, defaults to %ES_HOME%\logs. Note that this does not control the path for the Elasticsearch logs; the path for these is set via the setting path.logs in the elasticsearch.yml configuration file, or on the command line.

ES_PATH_CONF

Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files), defaults to %ES_HOME%\config.

ES_JAVA_OPTS

Any additional JVM system properties you may want to apply.

ES_START_TYPE

Startup mode for the service. Can be either auto or manual (default).

ES_STOP_TIMEOUT

The timeout in seconds that procrun waits for service to exit gracefully. Defaults to 0.

Note
At its core, elasticsearch-service.bat relies on Apache Commons Daemon project to install the service. Environment variables set prior to the service installation are copied and will be used during the service lifecycle. This means any changes made to them after the installation will not be picked up unless the service is reinstalled.
Note
On Windows, the heap size can be configured as for any other Elasticsearch installation when running Elasticsearch from the command line, or when installing Elasticsearch as a service for the first time. To adjust the heap size for an already installed service, use the service manager: bin\elasticsearch-service.bat manager.
Note
The service automatically configures a private temporary directory for use by Elasticsearch when it is running. This private temporary directory is configured as a sub-directory of the private temporary directory for the user running the installation. If the service will run under a different user, you can configure the location of the temporary directory that the service should use by setting the environment variable ES_TMPDIR to the preferred location before you execute the service installation.
Using the Manager GUI

It is also possible to configure the service after it’s been installed using the manager GUI (elasticsearch-service-mgr.exe), which offers insight into the installed service, including its status, startup type, JVM, start and stop settings amongst other things. Simply invoking elasticsearch-service.bat manager from the command-line will open up the manager window:

Windows Service Manager GUI

Most changes (like JVM settings) made through the manager GUI will require a restart of the service in order to take affect.

Directory layout of .zip archive

The .zip package is entirely self-contained. All files and directories are, by default, contained within %ES_HOME% — the directory created when unpacking the archive.

This is very convenient because you don’t have to create any directories to start using Elasticsearch, and uninstalling Elasticsearch is as easy as removing the %ES_HOME% directory. However, it is advisable to change the default locations of the config directory, the data directory, and the logs directory so that you do not delete important data later on.

Type Description Default Location Setting

home

Elasticsearch home directory or %ES_HOME%

Directory created by unpacking the archive

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

%ES_HOME%\bin

conf

Configuration files including elasticsearch.yml

%ES_HOME%\config

ES_PATH_CONF

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

%ES_HOME%\data

path.data

logs

Log files location.

%ES_HOME%\logs

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

%ES_HOME%\plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with Debian Package

The Debian package for Elasticsearch can be downloaded from our website or from our APT repository. It can be used to install Elasticsearch on any Debian-based system such as Debian and Ubuntu.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Import the Elasticsearch PGP Key

We sign all of our packages with the Elasticsearch Signing Key (PGP key D88E42B4, available from https://pgp.mit.edu) with fingerprint:

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

Download and install the public signing key:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Installing from the APT repository

You may need to install the apt-transport-https package on Debian before proceeding:

sudo apt-get install apt-transport-https

Save the repository definition to /etc/apt/sources.list.d/elastic-{major-version}.list:

Note

These instructions do not use add-apt-repository for several reasons:

  1. add-apt-repository adds entries to the system /etc/apt/sources.list file rather than a clean per-repository file in /etc/apt/sources.list.d

  2. add-apt-repository is not part of the default install on many distributions and requires a number of non-default dependencies.

  3. Older versions of add-apt-repository always add a deb-src entry which will cause errors because we do not provide a source package. If you have added the deb-src entry, you will see an error like the following until you delete the deb-src line:

    Unable to find expected entry 'main/source/Sources' in Release file
    (Wrong sources.list entry or malformed file)

You can install the Elasticsearch Debian package with:

sudo apt-get update && sudo apt-get install elasticsearch
Warning

If two entries exist for the same Elasticsearch repository, you will see an error like this during apt-get update:

Duplicate sources.list entry https://artifacts.elastic.co/packages/{major-version}/apt/ ...`

Examine /etc/apt/sources.list.d/elasticsearch-{major-version}.list for the duplicate entry or locate the duplicate entry amongst the files in /etc/apt/sources.list.d/ and the /etc/apt/sources.list file.

Note
On systemd-based distributions, the installation scripts will attempt to set kernel parameters (e.g., vm.max_map_count); you can skip this by masking the systemd-sysctl.service unit.
Note

An alternative package which contains only features that are available under the Apache 2.0 license is also available. To install it, use the following sources list:

echo "deb https://artifacts.elastic.co/packages/oss-{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list

Download and install the Debian package manually

The Debian package for Elasticsearch v{version} can be downloaded from the website and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.deb
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.deb.sha512
shasum -a 512 -c elasticsearch-{version}.deb.sha512 <1>
sudo dpkg -i elasticsearch-{version}.deb
  1. Compares the SHA of the downloaded Debian package and the published checksum, which should output elasticsearch-{version}.deb: OK.

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.deb

SysV init vs systemd

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the update-rc.d command to configure Elasticsearch to start automatically when the system boots up:

sudo update-rc.d elasticsearch defaults 95 10

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "{build_flavor}",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "{lucene_version}",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Configuring Elasticsearch

Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. The ownership of this directory and all files in this directory are set to root:elasticsearch on package installation and the directory has the setgid flag set so that any files and subdirectories created under /etc/elasticsearch are created with this ownership as well (e.g., if a keystore is created using the keystore tool). It is expected that this be maintained so that the Elasticsearch process can read the files under this directory via the group permissions.

Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

The Debian package also has a system configuration file (/etc/default/elasticsearch), which allows you to set the following parameters:

JAVA_HOME

Set a custom Java path to be used.

MAX_OPEN_FILES

Maximum number of open files, defaults to 65535.

MAX_LOCKED_MEMORY

Maximum locked memory size. Set to unlimited if you use the bootstrap.memory_lock option in elasticsearch.yml.

MAX_MAP_COUNT

Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting Elasticsearch. Defaults to 262144.

ES_PATH_CONF

Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files); defaults to /etc/elasticsearch.

ES_JAVA_OPTS

Any additional JVM system properties you may want to apply.

RESTART_ON_UPGRADE

Configure restart on package upgrade, defaults to false. This means you will have to restart your Elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster.

Note
Distributions that use systemd require that system resource limits be configured via systemd rather than via the /etc/sysconfig/elasticsearch file. See Systemd configuration for more information.

Directory layout of Debian package

The Debian package places config files, logs, and the data directory in the appropriate locations for a Debian-based system:

Type Description Default Location Setting

home

Elasticsearch home directory or $ES_HOME

/usr/share/elasticsearch

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

/usr/share/elasticsearch/bin

conf

Configuration files including elasticsearch.yml

/etc/elasticsearch

ES_PATH_CONF

conf

Environment variables including heap size, file descriptors.

/etc/default/elasticsearch

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

/var/lib/elasticsearch

path.data

logs

Log files location.

/var/log/elasticsearch

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

/usr/share/elasticsearch/plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with RPM

The RPM for Elasticsearch can be downloaded from our website or from our RPM repository. It can be used to install Elasticsearch on any RPM-based system such as OpenSuSE, SLES, Centos, Red Hat, and Oracle Enterprise.

Note
RPM install is not supported on distributions with old versions of RPM, such as SLES 11 and CentOS 5. Please see Install Elasticsearch with .zip or .tar.gz instead.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Import the Elasticsearch PGP Key

We sign all of our packages with the Elasticsearch Signing Key (PGP key D88E42B4, available from https://pgp.mit.edu) with fingerprint:

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

Download and install the public signing key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Installing from the RPM repository

Create a file called elasticsearch.repo in the /etc/yum.repos.d/ directory for RedHat based distributions, or in the /etc/zypp/repos.d/ directory for OpenSuSE based distributions, containing:

And your repository is ready for use. You can now install Elasticsearch with one of the following commands:

sudo yum install elasticsearch (1)
sudo dnf install elasticsearch (2)
sudo zypper install elasticsearch (3)
  1. Use yum on CentOS and older Red Hat based distributions.

  2. Use dnf on Fedora and other newer Red Hat distributions.

  3. Use zypper on OpenSUSE based distributions

Note

An alternative package which contains only features that are available under the Apache 2.0 license is also available. To install it, use the following baseurl in your elasticsearch.repo file:

baseurl=https://artifacts.elastic.co/packages/oss-{major-version}/yum

Download and install the RPM manually

The RPM for Elasticsearch v{version} can be downloaded from the website and installed as follows:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.rpm.sha512
shasum -a 512 -c elasticsearch-{version}.rpm.sha512 <1>
sudo rpm --install elasticsearch-{version}.rpm
  1. Compares the SHA of the downloaded RPM and the published checksum, which should output elasticsearch-{version}.rpm: OK.

Alternatively, you can download the following package, which contains only features that are available under the Apache 2.0 license: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{version}.rpm

Note
On systemd-based distributions, the installation scripts will attempt to set kernel parameters (e.g., vm.max_map_count); you can skip this by masking the systemd-sysctl.service unit.

SysV init vs systemd

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the chkconfig command to configure Elasticsearch to start automatically when the system boots up:

sudo chkconfig --add elasticsearch

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "{build_flavor}",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "{lucene_version}",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Configuring Elasticsearch

Elasticsearch defaults to using /etc/elasticsearch for runtime configuration. The ownership of this directory and all files in this directory are set to root:elasticsearch on package installation and the directory has the setgid flag set so that any files and subdirectories created under /etc/elasticsearch are created with this ownership as well (e.g., if a keystore is created using the keystore tool). It is expected that this be maintained so that the Elasticsearch process can read the files under this directory via the group permissions.

Elasticsearch loads its configuration from the /etc/elasticsearch/elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

The RPM also has a system configuration file (/etc/sysconfig/elasticsearch), which allows you to set the following parameters:

JAVA_HOME

Set a custom Java path to be used.

MAX_OPEN_FILES

Maximum number of open files, defaults to 65535.

MAX_LOCKED_MEMORY

Maximum locked memory size. Set to unlimited if you use the bootstrap.memory_lock option in elasticsearch.yml.

MAX_MAP_COUNT

Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting Elasticsearch. Defaults to 262144.

ES_PATH_CONF

Configuration file directory (which needs to include elasticsearch.yml, jvm.options, and log4j2.properties files); defaults to /etc/elasticsearch.

ES_JAVA_OPTS

Any additional JVM system properties you may want to apply.

RESTART_ON_UPGRADE

Configure restart on package upgrade, defaults to false. This means you will have to restart your Elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster.

Note
Distributions that use systemd require that system resource limits be configured via systemd rather than via the /etc/sysconfig/elasticsearch file. See Systemd configuration for more information.

Directory layout of RPM

The RPM places config files, logs, and the data directory in the appropriate locations for an RPM-based system:

Type Description Default Location Setting

home

Elasticsearch home directory or $ES_HOME

/usr/share/elasticsearch

bin

Binary scripts including elasticsearch to start a node and elasticsearch-plugin to install plugins

/usr/share/elasticsearch/bin

conf

Configuration files including elasticsearch.yml

/etc/elasticsearch

ES_PATH_CONF

conf

Environment variables including heap size, file descriptors.

/etc/sysconfig/elasticsearch

data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

/var/lib/elasticsearch

path.data

logs

Log files location.

/var/log/elasticsearch

path.logs

plugins

Plugin files location. Each plugin will be contained in a subdirectory.

/usr/share/elasticsearch/plugins

repo

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

Not configured

path.repo

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install Elasticsearch with Windows MSI Installer

beta[]

Elasticsearch can be installed on Windows using the .msi package. This can install Elasticsearch as a Windows service or allow it to be run manually using the included elasticsearch.exe executable.

Tip
Elasticsearch has historically been installed on Windows using the .zip archive. You can continue using the .zip approach if you prefer.

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

The latest stable version of Elasticsearch can be found on the Download Elasticsearch page. Other versions can be found on the Past Releases page.

Note
Elasticsearch requires Java 8 or later. Use the official Oracle distribution or an open-source distribution such as OpenJDK.

Download the .msi package

Download the .msi package for Elasticsearch v{version} from https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{version}.msi

Install using the graphical user interface (GUI)

Double-click the downloaded .msi package to launch a GUI wizard that will guide you through the installation process. You can view help on any step by clicking the ? button, which reveals an aside panel with additional information for each input:

msi installer help

Within the first screen, select the directory for the installation. In addition, select directories for where data, logs and configuration will be placed or use the default locations:

msi installer locations

Then select whether to install as a service or start Elasticsearch manually as needed. When installing as a service, you can also configure the Windows account to run the service with, whether the service should be started after installation and the Windows startup behaviour:

msi installer service
Important
When selecting a Windows account to run the service with, be sure that the chosen account has sufficient privileges to access the installation and other deployment directories chosen. Also ensure the account is able to run Windows services.

Common configuration settings are exposed within the Configuration section, allowing the cluster name, node name and roles to be set, in addition to memory and network settings:

msi installer configuration

A list of common plugins that can be downloaded and installed as part of the installation, with the option to configure an HTTPS proxy through which to download these plugins.

Tip
Ensure the installation machine has access to the internet and that any corporate firewalls in place are configured to allow downloads from artifacts.elastic.co:
msi installer selected plugins

As of version 6.3.0, {xpack} is now bundled by default. The final step allows a choice of the type of license to install, in addition to security configuration and built-in user configuration:

msi installer xpack
Note
{xpack} includes a choice of a Trial or Basic license. A Trial license is valid for 30 days, after which you can obtain one of the available subscriptions. The Basic license is free and perpetual. Consult the available subscriptions for further details on which features are available under which license.

After clicking the install button, the installation will begin:

msi installer installing

…​and will indicate when it has been successfully installed:

msi installer success

Install using the command line

The .msi can also install Elasticsearch using the command line. The simplest installation using the same defaults as the GUI is achieved by first navigating to the download directory, then running:

msiexec.exe /i elasticsearch-{version}.msi /qn

By default, msiexec.exe does not wait for the installation process to complete, since it runs in the Windows subsystem. To wait on the process to finish and ensure that %ERRORLEVEL% is set accordingly, it is recommended to use start /wait to create a process and wait for it to exit

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn

As with any MSI installation package, a log file for the installation process can be found within the %TEMP% directory, with a randomly generated name adhering to the format MSI<random>.LOG. The path to a log file can be supplied using the /l command line argument

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn /l install.log

Supported Windows Installer command line arguments can be viewed using

msiexec.exe /help

…​or by consulting the Windows Installer SDK Command-Line Options.

Command line options

All settings exposed within the GUI are also available as command line arguments (referred to as properties within Windows Installer documentation) that can be passed to msiexec.exe:

INSTALLDIR

The installation directory. The final directory in the path must be the version of Elasticsearch. Defaults to %ProgramW6432%\Elastic\Elasticsearch{backslash}{version}.

DATADIRECTORY

The directory in which to store your data. Defaults to %ALLUSERSPROFILE%\Elastic\Elasticsearch\data

CONFIGDIRECTORY

The directory in which to store your configuration. Defaults to %ALLUSERSPROFILE%\Elastic\Elasticsearch\config

LOGSDIRECTORY

The directory in which to store your logs. Defaults to %ALLUSERSPROFILE%\Elastic\Elasticsearch\logs

PLACEWRITABLELOCATIONSINSAMEPATH

Whether the data, configuration and logs directories should be created under the installation directory. Defaults to false

INSTALLASSERVICE

Whether Elasticsearch is installed and configured as a Windows Service. Defaults to true

STARTAFTERINSTALL

Whether the Windows Service is started after installation finishes. Defaults to true

STARTWHENWINDOWSSTARTS

Whether the Windows Service is started when Windows is started. Defaults to true

USELOCALSYSTEM

Whether the Windows service runs under the LocalSystem Account. Defaults to true

USENETWORKSERVICE

Whether the Windows service runs under the NetworkService Account. Defaults to false

USEEXISTINGUSER

Whether the Windows service runs under a specified existing account. Defaults to false

USER

The username for the account under which the Windows service runs. Defaults to ""

PASSWORD

The password for the account under which the Windows service runs. Defaults to ""

CLUSTERNAME

The name of the cluster. Defaults to elasticsearch

NODENAME

The name of the node. Defaults to %COMPUTERNAME%

MASTERNODE

Whether Elasticsearch is configured as a master node. Defaults to true

DATANODE

Whether Elasticsearch is configured as a data node. Defaults to true

INGESTNODE

Whether Elasticsearch is configured as an ingest node. Defaults to true

SELECTEDMEMORY

The amount of memory to allocate to the JVM heap for Elasticsearch. Defaults to 2048 unless the target machine has less than 4GB in total, in which case it defaults to 50% of total memory.

LOCKMEMORY

Whether bootstrap.memory_lock should be used to try to lock the process address space into RAM. Defaults to false

UNICASTNODES

A comma separated list of hosts in the form host:port or host to be used for unicast discovery. Defaults to ""

MINIMUMMASTERNODES

The minimum number of master-eligible nodes that must be visible in order to form a cluster. Defaults to ""

NETWORKHOST

The hostname or IP address to bind the node to and publish (advertise) this host to other nodes in the cluster. Defaults to ""

HTTPPORT

The port to use for exposing Elasticsearch APIs over HTTP. Defaults to 9200

TRANSPORTPORT

The port to use for internal communication between nodes within the cluster. Defaults to 9300

PLUGINS

A comma separated list of the plugins to download and install as part of the installation. Defaults to ""

HTTPSPROXYHOST

The proxy host to use to download plugins over HTTPS. Defaults to ""

HTTPSPROXYPORT

The proxy port to use to download plugins over HTTPS. Defaults to 443

HTTPPROXYHOST

The proxy host to use to download plugins over HTTP. Defaults to ""

HTTPPROXYPORT

The proxy port to use to download plugins over HTTP. Defaults to 80

XPACKLICENSE

The type of license to install, either Basic or Trial. Defaults to Basic

XPACKSECURITYENABLED

When installing with a Trial license, whether {security-features} are enabled. Defaults to true

BOOTSTRAPPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to used to bootstrap the cluster and persisted as the bootstrap.password setting in the keystore. Defaults to a randomized value.

SKIPSETTINGPASSWORDS

When installing with a Trial license and {security-features} enabled, whether the installation should skip setting up the built-in users. Defaults to false

ELASTICUSERPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to use for the built-in user elastic. Defaults to ""

KIBANAUSERPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to use for the built-in user kibana. Defaults to ""

LOGSTASHSYSTEMUSERPASSWORD

When installing with a Trial license and {security-features} are enabled, the password to use for the built-in user logstash_system. Defaults to ""

To pass a value, simply append the property name and value using the format <PROPERTYNAME>="<VALUE>" to the installation command. For example, to use a different installation directory to the default one:

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn INSTALLDIR="C:\Custom Install Directory{version}"

Consult the Windows Installer SDK Command-Line Options for additional rules related to values containing quotation marks.

Running Elasticsearch from the command line

Once installed, Elasticsearch can be started from the command line, if not installed as a service and configured to start when installation completes, as follows:

.\bin\elasticsearch.exe

The command line terminal will display output similar to the following:

elasticsearch exe

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT in addition to the <cluster name>.log file within LOGSDIRECTORY, and can be stopped by pressing Ctrl-C.

Configuring Elasticsearch on the command line

Elasticsearch loads its configuration from the %ES_PATH_CONF%\elasticsearch.yml file by default. The format of this config file is explained in Configuring Elasticsearch.

Any settings that can be specified in the config file can also be specified on the command line, using the -E syntax as follows:

.\bin\elasticsearch.exe -E cluster.name=my_cluster -E node.name=node_1
Note
Values that contain spaces must be surrounded with quotes. For instance -E path.logs="C:\My Logs\logs".
Tip
Typically, any cluster-wide settings (like cluster.name) should be added to the elasticsearch.yml config file, while any node-specific settings such as node.name could be specified on the command line.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

GET /

which should give you a response something like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "{version}",
    "build_flavor" : "{build_flavor}",
    "build_type" : "zip",
    "build_hash" : "f27399d",
    "build_date" : "2016-03-30T09:51:41.449Z",
    "build_snapshot" : false,
    "lucene_version" : "{lucene_version}",
    "minimum_wire_compatibility_version" : "1.2.3",
    "minimum_index_compatibility_version" : "1.2.3"
  },
  "tagline" : "You Know, for Search"
}

Installing Elasticsearch as a Service on Windows

Elasticsearch can be installed as a service to run in the background or start automatically at boot time without any user interaction. This can be achieved upon installation using the following command line options

  • INSTALLASSERVICE=true

  • STARTAFTERINSTALL=true

  • STARTWHENWINDOWSSTARTS=true

Once installed, Elasticsearch will appear within the Services control panel:

msi installer installed service

and can be stopped and restarted from within the control panel, or from the command line using:

with Command Prompt:

sc.exe stop Elasticsearch
sc.exe start Elasticsearch

with PowerShell:

Get-Service Elasticsearch | Stop-Service
Get-Service Elasticsearch | Start-Service

Changes can be made to jvm.options and elasticsearch.yml configuration files to configure the service after installation. Most changes (like JVM settings) will require a restart of the service in order to take effect.

Upgrade using the graphical user interface (GUI)

The .msi package supports upgrading an installed version of Elasticsearch to a newer version. The upgrade process through the GUI handles upgrading all installed plugins as well as retaining both your data and configuration.

Downloading and double-clicking on a newer version of the .msi package will launch the GUI wizard. The first step will list the read-only properties from the previous installation:

msi installer upgrade notice

The next step allows certain configuration options to be changed:

msi installer upgrade configuration

Finally, the plugins step allows currently installed plugins to be upgraded or removed, and for plugins not currently installed, to be downloaded and installed:

msi installer upgrade plugins

Upgrade using the command line

The .msi can also upgrade Elasticsearch using the command line.

Important

A command line upgrade requires passing the same command line properties as used at first install time; the Windows Installer does not remember these properties.

For example, if you originally installed with the command line options PLUGINS="ingest-geoip" and LOCKMEMORY="true", then you must pass these same values when performing an upgrade from the command line.

The exception to this is the INSTALLDIR parameter (if originally specified), which must be a different directory to the current installation. If setting INSTALLDIR, the final directory in the path must be the version of Elasticsearch e.g.

C:\Program Files\Elastic\Elasticsearch{backslash}{version}

The simplest upgrade, assuming Elasticsearch was installed using all defaults, is achieved by first navigating to the download directory, then running:

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn

Similar to the install process, a path to a log file for the upgrade process can be passed using the /l command line argument

start /wait msiexec.exe /i elasticsearch-{version}.msi /qn /l upgrade.log

Uninstall using Add/Remove Programs

The .msi package handles uninstallation of all directories and files added as part of installation.

Warning
Uninstallation will remove all contents created as part of installation, except for data, config or logs directories. It is recommended that you make a copy of your data directory before upgrading or consider using the snapshot API.

MSI installer packages do not provide a GUI for uninstallation. An installed program can be uninstalled by pressing the Windows key and typing add or remove programs to open the system settings.

Once opened, find the Elasticsearch installation within the list of installed applications, click and choose Uninstall:

msi installer uninstall

This will launch the uninstallation process.

Uninstall using the command line

Uninstallation can also be performed from the command line by navigating to the directory containing the .msi package and running:

start /wait msiexec.exe /x elasticsearch-{version}.msi /qn

Similar to the install process, a path to a log file for the uninstallation process can be passed using the /l command line argument

start /wait msiexec.exe /x elasticsearch-{version}.msi /qn /l uninstall.log

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Install {es} with Docker

{es} is also available as Docker images. The images use centos:7 as the base image.

A list of all published Docker images and tags is available at www.docker.elastic.co. The source files are in Github.

These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. {stack-ov}/license-management.html[Start a 30-day trial] to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

Pulling the image

Obtaining {es} for Docker is as simple as issuing a docker pull command against the Elastic Docker registry.

docker pull {docker-repo}:{version}

Alternatively, you can download other Docker images that contain only features available under the Apache 2.0 license. To download the images, go to www.docker.elastic.co.

Running {es} from the command line

Development mode

{es} can be quickly started for development or testing use with the following command:

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" {docker-image}
Production mode
Important

The vm.max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:

  • Linux

    The vm.max_map_count setting should be set permanently in /etc/sysctl.conf:

    $ grep vm.max_map_count /etc/sysctl.conf
    vm.max_map_count=262144

    To apply the setting on a live system type: sysctl -w vm.max_map_count=262144

  • macOS with Docker for Mac

    The vm.max_map_count setting must be set within the xhyve virtual machine:

    $ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

    Just press enter and configure the sysctl setting as you would for Linux:

    sysctl -w vm.max_map_count=262144
  • Windows and macOS with Docker Toolbox

    The vm.max_map_count setting must be set via docker-machine:

    docker-machine ssh
    sudo sysctl -w vm.max_map_count=262144

The following example brings up a cluster comprising two {es} nodes. To bring up the cluster, use the docker-compose.yml and just type:

docker-compose up
Note
docker-compose is not pre-installed with Docker on Linux. Instructions for installing it can be found on the Docker Compose webpage.

The node elasticsearch listens on localhost:9200 while elasticsearch2 talks to elasticsearch over a Docker network.

This example also uses Docker named volumes, called esdata1 and esdata2 which will be created if not already present.

docker-compose.yml:

version: '2.2'
services:
  elasticsearch:
    image: {docker-image}
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: {docker-image}
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

To stop the cluster, type docker-compose down. Data volumes will persist, so it’s possible to start the cluster again with the same data using docker-compose up. To destroy the cluster and the data volumes, just type docker-compose down -v.

Inspect status of cluster:
curl http://127.0.0.1:9200/_cat/health
1472225929 15:38:49 docker-cluster green 2 2 4 2 0 0 0 0 - 100.0%

Log messages go to the console and are handled by the configured Docker logging driver. By default you can access logs with docker logs.

Configuring {es} with Docker

{es} loads its configuration from files under /usr/share/elasticsearch/config/. These configuration files are documented in Configuring Elasticsearch and Setting JVM options.

The image offers several methods for configuring {es} settings with the conventional approach being to provide customized files, that is to say, elasticsearch.yml. It’s also possible to use environment variables to set options:

A. Present the parameters via Docker environment variables

For example, to define the cluster name with docker run you can pass -e "cluster.name=mynewclustername". Double quotes are required.

B. Bind-mounted configuration

Create your custom config file and mount this over the image’s corresponding file. For example, bind-mounting a custom_elasticsearch.yml with docker run can be accomplished with the parameter:

-v full_path_to/custom_elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
Important
The container runs {es} as user elasticsearch using uid:gid 1000:1000. Bind mounted host directories and files, such as custom_elasticsearch.yml above, need to be accessible by this user. For the data and log dirs, such as /usr/share/elasticsearch/data, write access is required as well. Also see note 1 below.
C. Customized image

In some environments, it may make more sense to prepare a custom image containing your configuration. A Dockerfile to achieve this may be as simple as:

FROM docker.elastic.co/elasticsearch/elasticsearch:{version}
COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/

You could then build and try the image with something like:

docker build --tag=elasticsearch-custom .
docker run -ti -v /usr/share/elasticsearch/data elasticsearch-custom

Some plugins require additional security permissions. You have to explicitly accept them either by attaching a tty when you run the Docker image and accepting yes at the prompts, or inspecting the security permissions separately and if you are comfortable with them adding the --batch flag to the plugin install command. See {plugins}/_other_command_line_parameters.html[Plugin Management documentation] for more details.

D. Override the image’s default CMD

Options can be passed as command-line options to the {es} process by overriding the default command for the image. For example:

docker run <various parameters> bin/elasticsearch -Ecluster.name=mynewclustername

Configuring SSL/TLS with the {es} Docker image

Notes for production use and defaults

We have collected a number of best practices for production use. Any Docker parameters mentioned below assume the use of docker run.

  1. By default, {es} runs inside the container as user elasticsearch using uid:gid 1000:1000.

    Caution
    One exception is Openshift which runs containers using an arbitrarily assigned user ID. Openshift will present persistent volumes with the gid set to 0 which will work without any adjustments.

    If you are bind-mounting a local directory or file, ensure it is readable by this user, while the data and log dirs additionally require write access. A good strategy is to grant group access to gid 1000 or 0 for the local directory. As an example, to prepare a local directory for storing data through a bind-mount:

    mkdir esdatadir
    chmod g+rwx esdatadir
    chgrp 1000 esdatadir

    As a last resort, you can also force the container to mutate the ownership of any bind-mounts used for the data and log dirs through the environment variable TAKE_FILE_OWNERSHIP. In this case, they will be owned by uid:gid 1000:0 providing read/write access to the {es} process as required.

  2. It is important to ensure increased ulimits for nofile and nproc are available for the {es} containers. Verify the init system for the Docker daemon is already setting those to acceptable values and, if needed, adjust them in the Daemon, or override them per container, for example using docker run:

    --ulimit nofile=65535:65535
    Note
    One way of checking the Docker daemon defaults for the aforementioned ulimits is by running:
    docker run --rm centos:7 /bin/bash -c 'ulimit -Hn && ulimit -Sn && ulimit -Hu && ulimit -Su'
  3. Swapping needs to be disabled for performance and node stability. This can be achieved through any of the methods mentioned in the {es} docs. If you opt for the bootstrap.memory_lock: true approach, apart from defining it through any of the configuration methods, you will additionally need the memlock: true ulimit, either defined in the Docker Daemon or specifically set for the container. This is demonstrated above in the docker-compose.yml. If using docker run:

    -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1
  4. The image exposes TCP ports 9200 and 9300. For clusters it is recommended to randomize the published ports with --publish-all, unless you are pinning one container per host.

  5. Use the ES_JAVA_OPTS environment variable to set heap size. For example, to use 16GB use -e ES_JAVA_OPTS="-Xms16g -Xmx16g" with docker run. Note that while the default configuration file jvm.options sets a default heap of 1GB, any value you set in ES_JAVA_OPTS will override it.

    Note
    You still need to configure the heap size even if you are limiting memory access to the container.

    While setting the heap size via an environment variable is the recommended method, you can also configure this by bind-mounting your own jvm.options file under /usr/share/elasticsearch/config/. The file that {es} provides contains some important settings, so you should start by taking a copy of jvm.options from an {es} container and editing it as you require.

  6. Pin your deployments to a specific version of the {es} Docker image. For example, docker.elastic.co/elasticsearch/elasticsearch:{version}.

  7. Always use a volume bound on /usr/share/elasticsearch/data, as shown in the production example, for the following reasons:

    1. The data of your elasticsearch node won’t be lost if the container is killed

    2. {es} is I/O sensitive and the Docker storage driver is not ideal for fast I/O

    3. It allows the use of advanced Docker volume plugins

  8. If you are using the devicemapper storage driver, make sure you are not using the default loop-lvm mode. Configure docker-engine to use direct-lvm instead.

  9. Consider centralizing your logs by using a different logging driver. Also note that the default json-file logging driver is not ideally suited for production use.

Next steps

You now have a test {es} environment set up. Before you start serious development or go into production with {es}, you must do some additional setup:

Configuring Elasticsearch

Elasticsearch ships with good defaults and requires very little configuration. Most settings can be changed on a running cluster using the [cluster-update-settings] API.

The configuration files should contain settings which are node-specific (such as node.name and paths), or settings which a node requires in order to be able to join a cluster, such as cluster.name and network.host.

Config files location

Elasticsearch has three configuration files:

  • elasticsearch.yml for configuring Elasticsearch

  • jvm.options for configuring Elasticsearch JVM settings

  • log4j2.properties for configuring Elasticsearch logging

These files are located in the config directory, whose default location depends on whether or not the installation is from an archive distribution (tar.gz or zip) or a package distribution (Debian or RPM packages).

For the archive distributions, the config directory location defaults to $ES_HOME/config. The location of the config directory can be changed via the ES_PATH_CONF environment variable as follows:

ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch

Alternatively, you can export the ES_PATH_CONF environment variable via the command line or via your shell profile.

For the package distributions, the config directory location defaults to /etc/elasticsearch. The location of the config directory can also be changed via the ES_PATH_CONF environment variable, but note that setting this in your shell is not sufficient. Instead, this variable is sourced from /etc/default/elasticsearch (for the Debian package) and /etc/sysconfig/elasticsearch (for the RPM package). You will need to edit the ES_PATH_CONF=/etc/elasticsearch entry in one of these files accordingly to change the config directory location.

Config file format

The configuration format is YAML. Here is an example of changing the path of the data and logs directories:

path:
    data: /var/lib/elasticsearch
    logs: /var/log/elasticsearch

Settings can also be flattened as follows:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Environment variable substitution

Environment variables referenced with the ${…​} notation within the configuration file will be replaced with the value of the environment variable. For example:

node.name:    ${HOSTNAME}
network.host: ${ES_NETWORK_HOST}

Prompting for settings

Note
Prompting for settings is deprecated. Please use Secure settings for sensitive property values. Not all settings can be converted to use Secure settings.

For settings that you do not wish to store in the configuration file, you can use the value ${prompt.text} or ${prompt.secret} and start Elasticsearch in the foreground. ${prompt.secret} has echoing disabled so that the value entered will not be shown in your terminal; ${prompt.text} will allow you to see the value as you type it in. For example:

node:
  name: ${prompt.text}

When starting Elasticsearch, you will be prompted to enter the actual value like so:

Enter value for [node.name]:
Note
Elasticsearch will not start if ${prompt.text} or ${prompt.secret} is used in the settings and the process is run as a service or in the background.

Values for environment variables must be simple strings. Use a comma-separated string to provide values that Elasticsearch will parse as a list. For example, Elasticsearch will split the following string into a list of values for the ${HOSTNAME} environment variable:

export HOSTNAME=“host1,host2"

Cluster and node setting types

Cluster and node settings can be categorized based on how they are configured:

Dynamic

You can configure and update dynamic settings on a running cluster using the cluster update settings API.

You can also configure dynamic settings locally on an unstarted or shut down node using elasticsearch.yml.

Tip
It’s best to set dynamic, cluster-wide settings with the cluster update settings API and use elasticsearch.yml only for local configurations. Using the cluster update settings API ensures the setting is the same on all nodes. If you accidentally configure different settings in elasticsearch.yml on different nodes, it can be difficult to notice discrepancies.
Static

Static settings can only be configured on an unstarted or shut down node using elasticsearch.yml.

Static settings must be set on every relevant node in the cluster.

Setting JVM options

You should rarely need to change Java Virtual Machine (JVM) options. If you do, the most likely change is setting the heap size. The remainder of this document explains in detail how to set JVM options.

The preferred method of setting JVM options (including system properties and JVM flags) is via the jvm.options configuration file. The default location of this file is config/jvm.options (when installing from the tar or zip distributions) and /etc/elasticsearch/jvm.options (when installing from the Debian or RPM packages).

This file contains a line-delimited list of JVM arguments following a special syntax:

  • lines consisting of whitespace only are ignored

  • lines beginning with # are treated as comments and are ignored

    # this is a comment
  • lines beginning with a - are treated as a JVM option that applies independent of the version of the JVM

    -Xmx2g
  • lines beginning with a number followed by a : followed by a - are treated as a JVM option that applies only if the version of the JVM matches the number

    8:-Xmx2g
  • lines beginning with a number followed by a - followed by a : are treated as a JVM option that applies only if the version of the JVM is greater than or equal to the number

    8-:-Xmx2g
  • lines beginning with a number followed by a - followed by a number followed by a : are treated as a JVM option that applies only if the version of the JVM falls in the range of the two numbers

    8-9:-Xmx2g
  • all other lines are rejected

You can add custom JVM flags to this file and check this configuration into your version control system.

An alternative mechanism for setting Java Virtual Machine options is via the ES_JAVA_OPTS environment variable. For instance:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djava.io.tmpdir=/path/to/temp/dir"
./bin/elasticsearch

When using the RPM or Debian packages, ES_JAVA_OPTS can be specified in the system configuration file.

The JVM has a built-in mechanism for observing the JAVA_TOOL_OPTIONS environment variable. We intentionally ignore this environment variable in our packaging scripts. The primary reason for this is that on some OS (e.g., Ubuntu) there are agents installed by default via this environment variable that we do not want interfering with Elasticsearch.

Additionally, some other Java programs support the JAVA_OPTS environment variable. This is not a mechanism built into the JVM but instead a convention in the ecosystem. However, we do not support this environment variable, instead supporting setting JVM options via the jvm.options file or the environment variable ES_JAVA_OPTS as above.

Secure settings

Some settings are sensitive, and relying on filesystem permissions to protect their values is not sufficient. For this use case, Elasticsearch provides a keystore and the elasticsearch-keystore tool to manage the settings in the keystore.

Note
All commands here should be run as the user which will run Elasticsearch.
Important
Only some settings are designed to be read from the keystore. However, the keystore has no validation to block unsupported settings. Adding unsupported settings to the keystore will cause {es} to fail to start. See documentation for each setting to see if it is supported as part of the keystore.
Note
All the modifications to the keystore take affect only after restarting Elasticsearch.
Note
The elasticsearch keystore currently only provides obfuscation. In the future, password protection will be added.

These settings, just like the regular ones in the elasticsearch.yml config file, need to be specified on each node in the cluster. Currently, all secure settings are node-specific settings that must have the same value on every node.

Creating the keystore

To create the elasticsearch.keystore, use the create command:

bin/elasticsearch-keystore create

The file elasticsearch.keystore will be created alongside elasticsearch.yml.

Listing settings in the keystore

A list of the settings in the keystore is available with the list command:

bin/elasticsearch-keystore list

Adding string settings

Sensitive string settings, like authentication credentials for cloud plugins, can be added using the add command:

bin/elasticsearch-keystore add the.setting.name.to.set

The tool will prompt for the value of the setting. To pass the value through stdin, use the --stdin flag:

cat /file/containing/setting/value | bin/elasticsearch-keystore add --stdin the.setting.name.to.set

Adding file settings

You can add sensitive files, like authentication key files for cloud plugins, using the add-file command. Be sure to include your file path as an argument after the setting name.

bin/elasticsearch-keystore add-file the.setting.name.to.set /path/example-file.json

Removing settings

To remove a setting from the keystore, use the remove command:

bin/elasticsearch-keystore remove the.setting.name.to.remove

Upgrading the keystore

Occasionally, the internal format of the keystore changes. When Elasticsearch is installed from a package manager, an upgrade of the on-disk keystore to the new format is done during package upgrade. In other cases, Elasticsearch will perform such an upgrade during node startup. This requires that Elasticsearch have write permissions to the directory that contains the keystore. Alternatively, you can manually perform such an upgrade by using the upgrade command:

bin/elasticsearch-keystore upgrade

Reloadable secure settings

Just like the settings values in elasticsearch.yml, changes to the keystore contents are not automatically applied to the running elasticsearch node. Re-reading settings requires a node restart. However, certain secure settings are marked as reloadable. Such settings can be re-read and applied on a running node.

The values of all secure settings, reloadable or not, must be identical across all cluster nodes. After making the desired secure settings changes, using the bin/elasticsearch-keystore add command, call:

POST _nodes/reload_secure_settings

This API will decrypt and re-read the entire keystore, on every cluster node, but only the reloadable secure settings will be applied. Changes to other settings will not go into effect until the next restart. Once the call returns, the reload has been completed, meaning that all internal datastructures dependent on these settings have been changed. Everything should look as if the settings had the new value from the start.

When changing multiple reloadable secure settings, modify all of them, on each cluster node, and then issue a reload_secure_settings call, instead of reloading after each modification.

Logging configuration

Elasticsearch uses Log4j 2 for logging. Log4j 2 can be configured using the log4j2.properties file. Elasticsearch exposes three properties, ${sys:es.logs.base_path}, ${sys:es.logs.cluster_name}, and ${sys:es.logs.node_name} (if the node name is explicitly set via node.name) that can be referenced in the configuration file to determine the location of the log files. The property ${sys:es.logs.base_path} will resolve to the log directory, ${sys:es.logs.cluster_name} will resolve to the cluster name (used as the prefix of log filenames in the default configuration), and ${sys:es.logs.node_name} will resolve to the node name (if the node name is explicitly set).

For example, if your log directory (path.logs) is /var/log/elasticsearch and your cluster is named production then ${sys:es.logs.base_path} will resolve to /var/log/elasticsearch and ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log will resolve to /var/log/elasticsearch/production.log.

appender.rolling.type = RollingFile (1)
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log (2)
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n
appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz (3)
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy (4)
appender.rolling.policies.time.interval = 1 (5)
appender.rolling.policies.time.modulate = true (6)
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy (7)
appender.rolling.policies.size.size = 256MB (8)
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.action.type = Delete (9)
appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.action.condition.type = IfFileName (10)
appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-* (11)
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize (12)
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB (13)
  1. Configure the RollingFile appender

  2. Log to /var/log/elasticsearch/production.log

  3. Roll logs to /var/log/elasticsearch/production-yyyy-MM-dd-i.log; logs will be compressed on each roll and i will be incremented

  4. Use a time-based roll policy

  5. Roll logs on a daily basis

  6. Align rolls on the day boundary (as opposed to rolling every twenty-four hours)

  7. Using a size-based roll policy

  8. Roll logs after 256 MB

  9. Use a delete action when rolling logs

  10. Only delete logs matching a file pattern

  11. The pattern is to only delete the main logs

  12. Only delete if we have accumulated too many compressed logs

  13. The size condition on the compressed logs is 2 GB

Note
Log4j’s configuration parsing gets confused by any extraneous whitespace; if you copy and paste any Log4j settings on this page, or enter any Log4j configuration in general, be sure to trim any leading and trailing whitespace.

Note than you can replace .gz by .zip in appender.rolling.filePattern to compress the rolled logs using the zip format. If you remove the .gz extension then logs will not be compressed as they are rolled.

If you want to retain log files for a specified period of time, you can use a rollover strategy with a delete action.

appender.rolling.strategy.type = DefaultRolloverStrategy (1)
appender.rolling.strategy.action.type = Delete (2)
appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path} (3)
appender.rolling.strategy.action.condition.type = IfFileName (4)
appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-* (5)
appender.rolling.strategy.action.condition.nested_condition.type = IfLastModified (6)
appender.rolling.strategy.action.condition.nested_condition.age = 7D (7)
  1. Configure the DefaultRolloverStrategy

  2. Configure the Delete action for handling rollovers

  3. The base path to the Elasticsearch logs

  4. The condition to apply when handling rollovers

  5. Delete files from the base path matching the glob ${sys:es.logs.cluster_name}-*; this is the glob that log files are rolled to; this is needed to only delete the rolled Elasticsearch logs but not also delete the deprecation and slow logs

  6. A nested condition to apply to files matching the glob

  7. Retain logs for seven days

Multiple configuration files can be loaded (in which case they will get merged) as long as they are named log4j2.properties and have the Elasticsearch config directory as an ancestor; this is useful for plugins that expose additional loggers. The logger section contains the java packages and their corresponding log level. The appender section contains the destinations for the logs. Extensive information on how to customize logging and all the supported appenders can be found on the Log4j documentation.

Configuring logging levels

There are four ways to configuring logging levels, each having situations in which they are appropriate to use.

  1. Via the command-line: -E <name of logging hierarchy>=<level> (e.g., -E logger.org.elasticsearch.discovery=debug). This is most appropriate when you are temporarily debugging a problem on a single node (for example, a problem with startup, or during development).

  2. Via elasticsearch.yml: <name of logging hierarchy>: <level> (e.g., logger.org.elasticsearch.discovery: debug). This is most appropriate when you are temporarily debugging a problem but are not starting Elasticsearch via the command-line (e.g., via a service) or you want a logging level adjusted on a more permanent basis.

  3. Via cluster settings:

    PUT /_cluster/settings
    {
      "transient": {
        "<name of logging hierarchy>": "<level>"
      }
    }

    For example:

    PUT /_cluster/settings
    {
      "transient": {
        "logger.org.elasticsearch.discovery": "DEBUG"
      }
    }

    This is most appropriate when you need to dynamically need to adjust a logging level on an actively-running cluster.

  4. Via the log4j2.properties:

    logger.<unique_identifier>.name = <name of logging hierarchy>
    logger.<unique_identifier>.level = <level>

    For example:

    logger.discovery.name = org.elasticsearch.discovery
    logger.discovery.level = debug

    This is most appropriate when you need fine-grained control over the logger (for example, you want to send the logger to another file, or manage the logger differently; this is a rare use-case).

Deprecation logging

In addition to regular logging, Elasticsearch allows you to enable logging of deprecated actions. For example this allows you to determine early, if you need to migrate certain functionality in the future. By default, deprecation logging is enabled at the WARN level, the level at which all deprecation log messages will be emitted.

logger.deprecation.level = warn

This will create a daily rolling deprecation log file in your log directory. Check this file regularly, especially when you intend to upgrade to a new major version.

The default logging configuration has set the roll policy for the deprecation logs to roll and compress after 1 GB, and to preserve a maximum of five log files (four rolled logs, and the active log).

You can disable it in the config/log4j2.properties file by setting the deprecation log level to error like this:

logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = error

Important Elasticsearch configuration

While Elasticsearch requires very little configuration, there are a number of settings which need to be considered before going into production.

The following settings must be considered before going to production:

path.data and path.logs

If you are using the .zip or .tar.gz archives, the data and logs directories are sub-folders of $ES_HOME. If these important folders are left in their default locations, there is a high risk of them being deleted while upgrading Elasticsearch to a new version.

In production use, you will almost certainly want to change the locations of the data and log folder:

path:
  logs: /var/log/elasticsearch
  data: /var/data/elasticsearch

The RPM and Debian distributions already use custom paths for data and logs.

The path.data settings can be set to multiple paths, in which case all paths will be used to store data (although the files belonging to a single shard will all be stored on the same data path):

path:
  data:
    - /mnt/elasticsearch_1
    - /mnt/elasticsearch_2
    - /mnt/elasticsearch_3

cluster.name

A node can only join a cluster when it shares its cluster.name with all the other nodes in the cluster. The default name is elasticsearch, but you should change it to an appropriate name which describes the purpose of the cluster.

cluster.name: logging-prod

Make sure that you don’t reuse the same cluster names in different environments, otherwise you might end up with nodes joining the wrong cluster.

node.name

By default, Elasticsearch will use the first seven characters of the randomly generated UUID as the node id. Note that the node id is persisted and does not change when a node restarts and therefore the default node name will also not change.

It is worth configuring a more meaningful name which will also have the advantage of persisting after restarting the node:

node.name: prod-data-2

The node.name can also be set to the server’s HOSTNAME as follows:

node.name: ${HOSTNAME}

network.host

By default, Elasticsearch binds to loopback addresses only — e.g. 127.0.0.1 and [::1]. This is sufficient to run a single development node on a server.

Tip
In fact, more than one node can be started from the same $ES_HOME location on a single node. This can be useful for testing Elasticsearch’s ability to form clusters, but it is not a configuration recommended for production.

In order to form a cluster with nodes on other servers, your node will need to bind to a non-loopback address. While there are many network settings, usually all you need to configure is network.host:

network.host: 192.168.1.10

The network.host setting also understands some special values such as local, site, global and modifiers like :ip4 and :ip6, details of which can be found in [network-interface-values].

Important
As soon as you provide a custom setting for network.host, Elasticsearch assumes that you are moving from development mode to production mode, and upgrades a number of system startup checks from warnings to exceptions. See Development mode vs production mode for more information.

Discovery settings

Elasticsearch uses a custom discovery implementation called "Zen Discovery" for node-to-node clustering and master election. There are two important discovery settings that should be configured before going to production.

discovery.zen.ping.unicast.hosts

Out of the box, without any network configuration, Elasticsearch will bind to the available loopback addresses and will scan ports 9300 to 9305 to try to connect to other nodes running on the same server. This provides an auto- clustering experience without having to do any configuration.

When the moment comes to form a cluster with nodes on other servers, you have to provide a seed list of other nodes in the cluster that are likely to be live and contactable. This can be specified as follows:

discovery.zen.ping.unicast.hosts:
   - 192.168.1.10:9300
   - 192.168.1.11 (1)
   - seeds.mydomain.com (2)
  1. The port will default to transport.profiles.default.port and fallback to transport.port if not specified.

  2. A hostname that resolves to multiple IP addresses will try all resolved addresses.

discovery.zen.minimum_master_nodes

To prevent data loss, it is vital to configure the discovery.zen.minimum_master_nodes setting so that each master-eligible node knows the minimum number of master-eligible nodes that must be visible in order to form a cluster.

Without this setting, a cluster that suffers a network failure is at risk of having the cluster split into two independent clusters — a split brain — which will lead to data loss. A more detailed explanation is provided in [split-brain].

To avoid a split brain, this setting should be set to a quorum of master-eligible nodes:

(master_eligible_nodes / 2) + 1

In other words, if there are three master-eligible nodes, then minimum master nodes should be set to (3 / 2) + 1 or 2:

discovery.zen.minimum_master_nodes: 2

Setting the heap size

By default, Elasticsearch tells the JVM to use a heap with a minimum and maximum size of 1 GB. When moving to production, it is important to configure heap size to ensure that Elasticsearch has enough heap available.

Elasticsearch will assign the entire heap specified in jvm.options via the Xms (minimum heap size) and Xmx (maximum heap size) settings. These two settings must be equal to each other.

The value for these setting depends on the amount of RAM available on your server. Good rules of thumb are:

  • The more heap available to Elasticsearch, the more memory it can use for caching. But note that too much heap can subject you to long garbage collection pauses.

  • Set Xms and Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches.

  • Don’t set Xms and Xmx to above the cutoff that the JVM uses for compressed object pointers (compressed oops); the exact cutoff varies but is near 32 GB. You can verify that you are under the limit by looking for a line in the logs like the following:

    heap size [1.9gb], compressed ordinary object pointers [true]
  • Even better, try to stay below the threshold for zero-based compressed oops; the exact cutoff varies but 26 GB is safe on most systems, but can be as large as 30 GB on some systems. You can verify that you are under the limit by starting Elasticsearch with the JVM options -XX:+UnlockDiagnosticVMOptions -XX:+PrintCompressedOopsMode and looking for a line like the following:

    heap address: 0x000000011be00000, size: 27648 MB, zero based Compressed Oops

    showing that zero-based compressed oops are enabled instead of

    heap address: 0x0000000118400000, size: 28672 MB, Compressed Oops with base: 0x00000001183ff000

Here are examples of how to set the heap size via the jvm.options file:

-Xms2g (1)
-Xmx2g (2)
  1. Set the minimum heap size to 2g.

  2. Set the maximum heap size to 2g.

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:

ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch (1)
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch (2)
  1. Set the minimum and maximum heap size to 2 GB.

  2. Set the minimum and maximum heap size to 4000 MB.

Note
Configuring the heap for the Windows service is different than the above. The values initially populated for the Windows service can be configured as above but are different after the service has been installed. Consult the Windows service documentation for additional details.

JVM heap dump path

By default, Elasticsearch configures the JVM to dump the heap on out of memory exceptions to the default data directory (this is /var/lib/elasticsearch for the RPM and Debian package distributions, and the data directory under the root of the Elasticsearch installation for the tar and zip archive distributions). If this path is not suitable for receiving heap dumps, you should modify the entry -XX:HeapDumpPath=…​ in jvm.options. If you specify a directory, the JVM will generate a filename for the heap dump based on the PID of the running instance. If you specify a fixed filename instead of a directory, the file must not exist when the JVM needs to perform a heap dump on an out of memory exception, otherwise the heap dump will fail.

GC logging

By default, Elasticsearch enables GC logs. These are configured in jvm.options and default to the same default location as the Elasticsearch logs. The default configuration rotates the logs every 64 MB and can consume up to 2 GB of disk space.

Temp directory

By default, Elasticsearch uses a private temporary directory that the startup script creates immediately below the system temporary directory.

On some Linux distributions a system utility will clean files and directories from /tmp if they have not been recently accessed. This can lead to the private temporary directory being removed while Elasticsearch is running if features that require the temporary directory are not used for a long time. This causes problems if a feature that requires the temporary directory is subsequently used.

If you install Elasticsearch using the .deb or .rpm packages and run it under systemd then the private temporary directory that Elasticsearch uses is excluded from periodic cleanup.

However, if you intend to run the .tar.gz distribution on Linux for an extended period then you should consider creating a dedicated temporary directory for Elasticsearch that is not under a path that will have old files and directories cleaned from it. This directory should have permissions set so that only the user that Elasticsearch runs as can access it. Then set the $ES_TMPDIR environment variable to point to it before starting Elasticsearch.

JVM fatal error logs

By default, Elasticsearch configures the JVM to write fatal error logs to the default logging directory (this is /var/log/elasticsearch for the RPM and Debian package distributions, and the logs directory under the root of the Elasticsearch installation for the tar and zip archive distributions). These are logs produced by the JVM when it encounters a fatal error (e.g., a segmentation fault). If this path is not suitable for receiving logs, you should modify the entry -XX:ErrorFile=…​ in jvm.options to an alternate path.

Important System Configuration

Ideally, Elasticsearch should run alone on a server and use all of the resources available to it. In order to do so, you need to configure your operating system to allow the user running Elasticsearch to access more resources than allowed by default.

The following settings must be considered before going to production:

Development mode vs production mode

By default, Elasticsearch assumes that you are working in development mode. If any of the above settings are not configured correctly, a warning will be written to the log file, but you will be able to start and run your Elasticsearch node.

As soon as you configure a network setting like network.host, Elasticsearch assumes that you are moving to production and will upgrade the above warnings to exceptions. These exceptions will prevent your Elasticsearch node from starting. This is an important safety measure to ensure that you will not lose data because of a malconfigured server.

Configuring system settings

Where to configure systems settings depends on which package you have used to install Elasticsearch, and which operating system you are using.

When using the .zip or .tar.gz packages, system settings can be configured:

When using the RPM or Debian packages, most system settings are set in the system configuration file. However, systems which use systemd require that system limits are specified in a systemd configuration file.

ulimit

On Linux systems, ulimit can be used to change resource limits on a temporary basis. Limits usually need to be set as root before switching to the user that will run Elasticsearch. For example, to set the number of open file handles (ulimit -n) to 65,536, you can do the following:

sudo su  (1)
ulimit -n 65535 (2)
su elasticsearch (3)
  1. Become root.

  2. Change the max number of open files.

  3. Become the elasticsearch user in order to start Elasticsearch.

The new limit is only applied during the current session.

You can consult all currently applied limits with ulimit -a.

/etc/security/limits.conf

On Linux systems, persistent limits can be set for a particular user by editing the /etc/security/limits.conf file. To set the maximum number of open files for the elasticsearch user to 65,535, add the following line to the limits.conf file:

elasticsearch  -  nofile  65535

This change will only take effect the next time the elasticsearch user opens a new session.

Note
Ubuntu and limits.conf

Ubuntu ignores the limits.conf file for processes started by init.d. To enable the limits.conf file, edit /etc/pam.d/su and uncomment the following line:

# session    required   pam_limits.so

Sysconfig file

When using the RPM or Debian packages, system settings and environment variables can be specified in the system configuration file, which is located in:

RPM

/etc/sysconfig/elasticsearch

Debian

/etc/default/elasticsearch

However, for systems which uses systemd, system limits need to be specified via systemd.

Systemd configuration

When using the RPM or Debian packages on systems that use systemd, system limits must be specified via systemd.

The systemd service file (/usr/lib/systemd/system/elasticsearch.service) contains the limits that are applied by default.

To override them, add a file called /etc/systemd/system/elasticsearch.service.d/override.conf (alternatively, you may run sudo systemctl edit elasticsearch which opens the file automatically inside your default editor). Set any changes in this file, such as:

[Service]
LimitMEMLOCK=infinity

Once finished, run the following command to reload units:

sudo systemctl daemon-reload

Disable swapping

Most operating systems try to use as much memory as possible for file system caches and eagerly swap out unused application memory. This can result in parts of the JVM heap or even its executable pages being swapped out to disk.

Swapping is very bad for performance, for node stability, and should be avoided at all costs. It can cause garbage collections to last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. In a resilient distributed system, it’s more effective to let the operating system kill the node.

There are three approaches to disabling swapping. The preferred option is to completely disable swap. If this is not an option, whether or not to prefer minimizing swappiness versus memory locking is dependent on your environment.

Disable all swap files

Usually Elasticsearch is the only service running on a box, and its memory usage is controlled by the JVM options. There should be no need to have swap enabled.

On Linux systems, you can disable swap temporarily by running:

sudo swapoff -a

To disable it permanently, you will need to edit the /etc/fstab file and comment out any lines that contain the word swap.

On Windows, the equivalent can be achieved by disabling the paging file entirely via System Properties → Advanced → Performance → Advanced → Virtual memory.

Configure swappiness

Another option available on Linux systems is to ensure that the sysctl value vm.swappiness is set to 1. This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions.

Enable bootstrap.memory_lock

Another option is to use mlockall on Linux/Unix systems, or VirtualLock on Windows, to try to lock the process address space into RAM, preventing any Elasticsearch memory from being swapped out. This can be done, by adding this line to the config/elasticsearch.yml file:

bootstrap.memory_lock: true
Warning
mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available!

After starting Elasticsearch, you can see whether this setting was applied successfully by checking the value of mlockall in the output from this request:

GET _nodes?filter_path=**.mlockall

If you see that mlockall is false, then it means that the mlockall request has failed. You will also see a line with more information in the logs with the words Unable to lock JVM Memory.

The most probable reason, on Linux/Unix systems, is that the user running Elasticsearch doesn’t have permission to lock memory. This can be granted as follows:

.zip and .tar.gz

Set ulimit -l unlimited as root before starting Elasticsearch, or set memlock to unlimited in /etc/security/limits.conf.

RPM and Debian

Set MAX_LOCKED_MEMORY to unlimited in the system configuration file (or see below for systems using systemd).

Systems using systemd

Set LimitMEMLOCK to infinity in the systemd configuration.

Another possible reason why mlockall can fail is that the JNA temporary directory (usually a sub-directory of /tmp) is mounted with the noexec option. This can be solved by specifying a new temporary directory for JNA using the ES_JAVA_OPTS environment variable:

export ES_JAVA_OPTS="$ES_JAVA_OPTS -Djna.tmpdir=<path>"
./bin/elasticsearch

or setting this JVM flag in the jvm.options configuration file.

File Descriptors

Note
This is only relevant for Linux and macOS and can be safely ignored if running Elasticsearch on Windows. On Windows that JVM uses an API limited only by available resources.

Elasticsearch uses a lot of file descriptors or file handles. Running out of file descriptors can be disastrous and will most probably lead to data loss. Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65,536 or higher.

For the .zip and .tar.gz packages, set ulimit -n 65535 as root before starting Elasticsearch, or set nofile to 65535 in /etc/security/limits.conf.

On macOS, you must also pass the JVM option -XX:-MaxFDLimit to Elasticsearch in order for it to make use of the higher file descriptor limit.

RPM and Debian packages already default the maximum number of file descriptors to 65535 and do not require further configuration.

You can check the max_file_descriptors configured for each node using the [cluster-nodes-stats] API, with:

GET _nodes/stats/process?filter_path=**.max_file_descriptors

Virtual memory

Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.

On Linux, you can increase the limits by running the following command as root:

sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.

The RPM and Debian packages will configure this setting automatically. No further configuration is required.

Number of threads

Elasticsearch uses a number of thread pools for different types of operations. It is important that it is able to create new threads whenever needed. Make sure that the number of threads that the Elasticsearch user can create is at least 4096.

This can be done by setting ulimit -u 4096 as root before starting Elasticsearch, or by setting nproc to 4096 in /etc/security/limits.conf.

The package distributions when run as services under systemd will configure the number of threads for the Elasticsearch process automatically. No additional configuration is required.

DNS cache settings

Elasticsearch runs with a security manager in place. With a security manager in place, the JVM defaults to caching positive hostname resolutions indefinitely and defaults to caching negative hostname resolutions for ten seconds. Elasticsearch overrides this behavior with default values to cache positive lookups for sixty seconds, and to cache negative lookups for ten seconds. These values should be suitable for most environments, including environments where DNS resolutions vary with time. If not, you can edit the values es.networkaddress.cache.ttl and es.networkaddress.cache.negative.ttl in the JVM options. Note that the values networkaddress.cache.ttl=<timeout> and networkaddress.cache.negative.ttl=<timeout> in the Java security policy are ignored by Elasticsearch unless you remove the settings for es.networkaddress.cache.ttl and es.networkaddress.cache.negative.ttl.

JNA temporary directory not mounted with noexec

Note
This is only relevant for Linux.

Elasticsearch uses the Java Native Access (JNA) library for executing some platform-dependent native code. On Linux, the native code backing this library is extracted at runtime from the JNA archive. By default, this code is extracted to the Elasticsearch temporary directory which defaults to a sub-directory of /tmp. Alternatively, this location can be controlled with the JVM flag -Djna.tmpdir=<path>. As the native library is mapped into the JVM virtual address space as executable, the underlying mount point of the location that this code is extracted to must not be mounted with noexec as this prevents the JVM process from being able to map this code as executable. On some hardened Linux installations this is a default mount option for /tmp. One indication that the underlying mount is mounted with noexec is that at startup JNA will fail to load with a java.lang.UnsatisfiedLinkerError exception with a message along the lines of failed to map segment from shared object. Note that the exception message can differ amongst JVM versions. Additionally, the components of Elasticsearch that rely on execution of native code via JNA will fail with messages indicating that it is because JNA is not available. If you are seeing such error messages, you must remount the temporary directory used for JNA to not be mounted with noexec.

Bootstrap Checks

Collectively, we have a lot of experience with users suffering unexpected issues because they have not configured important settings. In previous versions of Elasticsearch, misconfiguration of some of these settings were logged as warnings. Understandably, users sometimes miss these log messages. To ensure that these settings receive the attention that they deserve, Elasticsearch has bootstrap checks upon startup.

These bootstrap checks inspect a variety of Elasticsearch and system settings and compare them to values that are safe for the operation of Elasticsearch. If Elasticsearch is in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If Elasticsearch is in production mode, any bootstrap checks that fail will cause Elasticsearch to refuse to start.

There are some bootstrap checks that are always enforced to prevent Elasticsearch from running with incompatible settings. These checks are documented individually.

Development vs. production mode

By default, Elasticsearch binds to loopback addresses for HTTP and transport (internal) communication. This is fine for downloading and playing with Elasticsearch as well as everyday development, but it’s useless for production systems. To join a cluster, an Elasticsearch node must be reachable via transport communication. To join a cluster via a non-loopback address, a node must bind transport to a non-loopback address and not be using single-node discovery. Thus, we consider an Elasticsearch node to be in development mode if it can not form a cluster with another machine via a non-loopback address, and is otherwise in production mode if it can join a cluster via non-loopback addresses.

Note that HTTP and transport can be configured independently via http.host and transport.host; this can be useful for configuring a single node to be reachable via HTTP for testing purposes without triggering production mode.

Single-node discovery

We recognize that some users need to bind transport to an external interface for testing their usage of the transport client. For this situation, we provide the discovery type single-node (configure it by setting discovery.type to single-node); in this situation, a node will elect itself master and will not join a cluster with any other node.

Forcing the bootstrap checks

If you are running a single node in production, it is possible to evade the bootstrap checks (either by not binding transport to an external interface, or by binding transport to an external interface and setting the discovery type to single-node). For this situation, you can force execution of the bootstrap checks by setting the system property es.enforce.bootstrap.checks to true (set this in Setting JVM options, or by adding -Des.enforce.bootstrap.checks=true to the environment variable ES_JAVA_OPTS). We strongly encourage you to do this if you are in this specific situation. This system property can be used to force execution of the bootstrap checks independent of the node configuration.

Heap size check

If a JVM is started with unequal initial and max heap size, it can be prone to pauses as the JVM heap is resized during system usage. To avoid these resize pauses, it’s best to start the JVM with the initial heap size equal to the maximum heap size. Additionally, if bootstrap.memory_lock is enabled, the JVM will lock the initial size of the heap on startup. If the initial heap size is not equal to the maximum heap size, after a resize it will not be the case that all of the JVM heap is locked in memory. To pass the heap size check, you must configure the heap size.

File descriptor check

File descriptors are a Unix construct for tracking open "files". In Unix though, everything is a file. For example, "files" could be a physical file, a virtual file (e.g., /proc/loadavg), or network sockets. Elasticsearch requires lots of file descriptors (e.g., every shard is composed of multiple segments and other files, plus connections to other nodes, etc.). This bootstrap check is enforced on OS X and Linux. To pass the file descriptor check, you might have to configure file descriptors.

Memory lock check

When the JVM does a major garbage collection it touches every page of the heap. If any of those pages are swapped out to disk they will have to be swapped back in to memory. That causes lots of disk thrashing that Elasticsearch would much rather use to service requests. There are several ways to configure a system to disallow swapping. One way is by requesting the JVM to lock the heap in memory through mlockall (Unix) or virtual lock (Windows). This is done via the Elasticsearch setting bootstrap.memory_lock. However, there are cases where this setting can be passed to Elasticsearch but Elasticsearch is not able to lock the heap (e.g., if the elasticsearch user does not have memlock unlimited). The memory lock check verifies that if the bootstrap.memory_lock setting is enabled, that the JVM was successfully able to lock the heap. To pass the memory lock check, you might have to configure bootstrap.memory_lock.

Maximum number of threads check

Elasticsearch executes requests by breaking the request down into stages and handing those stages off to different thread pool executors. There are different thread pool executors for a variety of tasks within Elasticsearch. Thus, Elasticsearch needs the ability to create a lot of threads. The maximum number of threads check ensures that the Elasticsearch process has the rights to create enough threads under normal use. This check is enforced only on Linux. If you are on Linux, to pass the maximum number of threads check, you must configure your system to allow the Elasticsearch process the ability to create at least 4096 threads. This can be done via /etc/security/limits.conf using the nproc setting (note that you might have to increase the limits for the root user too).

Max file size check

The segment files that are the components of individual shards and the translog generations that are components of the translog can get large (exceeding multiple gigabytes). On systems where the max size of files that can be created by the Elasticsearch process is limited, this can lead to failed writes. Therefore, the safest option here is that the max file size is unlimited and that is what the max file size bootstrap check enforces. To pass the max file check, you must configure your system to allow the Elasticsearch process the ability to write files of unlimited size. This can be done via /etc/security/limits.conf using the fsize setting to unlimited (note that you might have to increase the limits for the root user too).

Maximum size virtual memory check

Elasticsearch and Lucene use mmap to great effect to map portions of an index into the Elasticsearch address space. This keeps certain index data off the JVM heap but in memory for blazing fast access. For this to be effective, the Elasticsearch should have unlimited address space. The maximum size virtual memory check enforces that the Elasticsearch process has unlimited address space and is enforced only on Linux. To pass the maximum size virtual memory check, you must configure your system to allow the Elasticsearch process the ability to have unlimited address space. This can be done via /etc/security/limits.conf using the as setting to unlimited (note that you might have to increase the limits for the root user too).

Maximum map count check

Continuing from the previous point, to use mmap effectively, Elasticsearch also requires the ability to create many memory-mapped areas. The maximum map count check checks that the kernel allows a process to have at least 262,144 memory-mapped areas and is enforced on Linux only. To pass the maximum map count check, you must configure vm.max_map_count via sysctl to be at least 262144.

Alternatively, the maximum map count check is only needed if you are using mmapfs or hybridfs as the store type for your indices. If you do not allow the use of mmap then this bootstrap check will not be enforced.

Client JVM check

There are two different JVMs provided by OpenJDK-derived JVMs: the client JVM and the server JVM. These JVMs use different compilers for producing executable machine code from Java bytecode. The client JVM is tuned for startup time and memory footprint while the server JVM is tuned for maximizing performance. The difference in performance between the two VMs can be substantial. The client JVM check ensures that Elasticsearch is not running inside the client JVM. To pass the client JVM check, you must start Elasticsearch with the server VM. On modern systems and operating systems, the server VM is the default.

Use serial collector check

There are various garbage collectors for the OpenJDK-derived JVMs targeting different workloads. The serial collector in particular is best suited for single logical CPU machines or extremely small heaps, neither of which are suitable for running Elasticsearch. Using the serial collector with Elasticsearch can be devastating for performance. The serial collector check ensures that Elasticsearch is not configured to run with the serial collector. To pass the serial collector check, you must not start Elasticsearch with the serial collector (whether it’s from the defaults for the JVM that you’re using, or you’ve explicitly specified it with -XX:+UseSerialGC). Note that the default JVM configuration that ships with Elasticsearch configures Elasticsearch to use the CMS collector.

System call filter check

Elasticsearch installs system call filters of various flavors depending on the operating system (e.g., seccomp on Linux). These system call filters are installed to prevent the ability to execute system calls related to forking as a defense mechanism against arbitrary code execution attacks on Elasticsearch. The system call filter check ensures that if system call filters are enabled, then they were successfully installed. To pass the system call filter check you must either fix any configuration errors on your system that prevented system call filters from installing (check your logs), or at your own risk disable system call filters by setting bootstrap.system_call_filter to false.

OnError and OnOutOfMemoryError checks

The JVM options OnError and OnOutOfMemoryError enable executing arbitrary commands if the JVM encounters a fatal error (OnError) or an OutOfMemoryError (OnOutOfMemoryError). However, by default, Elasticsearch system call filters (seccomp) are enabled and these filters prevent forking. Thus, using OnError or OnOutOfMemoryError and system call filters are incompatible. The OnError and OnOutOfMemoryError checks prevent Elasticsearch from starting if either of these JVM options are used and system call filters are enabled. This check is always enforced. To pass this check do not enable OnError nor OnOutOfMemoryError; instead, upgrade to Java 8u92 and use the JVM flag ExitOnOutOfMemoryError. While this does not have the full capabilities of OnError nor OnOutOfMemoryError, arbitrary forking will not be supported with seccomp enabled.

Early-access check

The OpenJDK project provides early-access snapshots of upcoming releases. These releases are not suitable for production. The early-access check detects these early-access snapshots. To pass this check, you must start Elasticsearch on a release build of the JVM.

G1GC check

Early versions of the HotSpot JVM that shipped with JDK 8 are known to have issues that can lead to index corruption when the G1GC collector is enabled. The versions impacted are those earlier than the version of HotSpot that shipped with JDK 8u40. The G1GC check detects these early versions of the HotSpot JVM.

All permission check

The all permission check ensures that the security policy used during bootstrap does not grant the java.security.AllPermission to Elasticsearch. Running with the all permission granted is equivalent to disabling the security manager.

Starting Elasticsearch

The method for starting {es} varies depending on how you installed it.

Archive packages (.tar.gz)

If you installed {es} with a .tar.gz package, you can start {es} from the command line.

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

./bin/elasticsearch

By default, Elasticsearch runs in the foreground, prints its logs to the standard output (stdout), and can be stopped by pressing Ctrl-C.

Note
All scripts packaged with Elasticsearch require a version of Bash that supports arrays and assume that Bash is available at /bin/bash. As such, Bash should be available at this path either directly or via a symbolic link.

Running as a daemon

To run Elasticsearch as a daemon, specify -d on the command line, and record the process ID in a file using the -p option:

./bin/elasticsearch -d -p pid

Log messages can be found in the $ES_HOME/logs/ directory.

To shut down Elasticsearch, kill the process ID recorded in the pid file:

pkill -F pid
Note
The startup scripts provided in the RPM and Debian packages take care of starting and stopping the Elasticsearch process for you.

Archive packages (.zip)

If you installed {es} on Windows with a .zip package, you can start {es} from the command line. If you want {es} to start automatically at boot time without any user interaction, install {es} as a service.

Running Elasticsearch from the command line

Elasticsearch can be started from the command line as follows:

.\bin\elasticsearch.bat

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl-C.

Debian packages

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the update-rc.d command to configure Elasticsearch to start automatically when the system boots up:

sudo update-rc.d elasticsearch defaults 95 10

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Docker images

If you installed a Docker image, you can start {es} from the command line. There are different methods depending on whether you’re using development mode or production mode. See Running {es} from the command line.

MSI packages

If you installed {es} on Windows using the .msi package, you can start {es} from the command line. If you want it to start automatically at boot time without any user interaction, install {es} as a Windows service.

Running Elasticsearch from the command line

Once installed, Elasticsearch can be started from the command line, if not installed as a service and configured to start when installation completes, as follows:

.\bin\elasticsearch.exe

The command line terminal will display output similar to the following:

elasticsearch exe

By default, Elasticsearch runs in the foreground, prints its logs to STDOUT in addition to the <cluster name>.log file within LOGSDIRECTORY, and can be stopped by pressing Ctrl-C.

RPM packages

Elasticsearch is not started automatically after installation. How to start and stop Elasticsearch depends on whether your system uses SysV init or systemd (used by newer distributions). You can tell which is being used by running this command:

ps -p 1

Running Elasticsearch with SysV init

Use the chkconfig command to configure Elasticsearch to start automatically when the system boots up:

sudo chkconfig --add elasticsearch

Elasticsearch can be started and stopped using the service command:

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

If Elasticsearch fails to start for any reason, it will print the reason for failure to STDOUT. Log files can be found in /var/log/elasticsearch/.

Running Elasticsearch with systemd

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Elasticsearch can be started and stopped as follows:

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

When systemd logging is enabled, the logging information are available using the journalctl commands:

To tail the journal:

sudo journalctl -f

To list journal entries for the elasticsearch service:

sudo journalctl --unit elasticsearch

To list journal entries for the elasticsearch service starting from a given time:

sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"

Check man journalctl or https://www.freedesktop.org/software/systemd/man/journalctl.html for more command line options.

Stopping Elasticsearch

An orderly shutdown of Elasticsearch ensures that Elasticsearch has a chance to cleanup and close outstanding resources. For example, a node that is shutdown in an orderly fashion will remove itself from the cluster, sync translogs to disk, and perform other related cleanup activities. You can help ensure an orderly shutdown by properly stopping Elasticsearch.

If you’re running Elasticsearch as a service, you can stop Elasticsearch via the service management functionality provided by your installation.

If you’re running Elasticsearch directly, you can stop Elasticsearch by sending control-C if you’re running Elasticsearch in the console, or by sending SIGTERM to the Elasticsearch process on a POSIX system. You can obtain the PID to send the signal to via various tools (e.g., ps or jps):

$ jps | grep Elasticsearch
14542 Elasticsearch

From the Elasticsearch startup logs:

[2016-07-07 12:26:18,908][INFO ][node                     ] [I8hydUG] version[5.0.0-alpha4], pid[15399], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Mac OS X/10.11.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_92/25.92-b14]

Or by specifying a location to write a PID file to on startup (-p <path>):

$ ./bin/elasticsearch -p /tmp/elasticsearch-pid -d
$ cat /tmp/elasticsearch-pid && echo
15516
$ kill -SIGTERM 15516

Stopping on Fatal Errors

During the life of the Elasticsearch virtual machine, certain fatal errors could arise that put the virtual machine in a questionable state. Such fatal errors include out of memory errors, internal errors in virtual machine, and serious I/O errors.

When Elasticsearch detects that the virtual machine has encountered such a fatal error Elasticsearch will attempt to log the error and then will halt the virtual machine. When Elasticsearch initiates such a shutdown, it does not go through an orderly shutdown as described above. The Elasticsearch process will also return with a special status code indicating the nature of the error.

JVM internal error

128

Out of memory error

127

Stack overflow error

126

Unknown virtual machine error

125

Serious I/O error

124

Unknown fatal error

1

Adding nodes to your cluster

When you start an instance of {es}, you are starting a node. An {es} cluster is a group of nodes that have the same cluster.name attribute. As nodes join or leave a cluster, the cluster automatically reorganizes itself to evenly distribute the data across the available nodes.

If you are running a single instance of {es}, you have a cluster of one node. All primary shards reside on the single node. No replica shards can be allocated, therefore the cluster state remains yellow. The cluster is fully functional but is at risk of data loss in the event of a failure.

A cluster with one node and three primary shards

You add nodes to a cluster to increase its capacity and reliability. By default, a node is both a data node and eligible to be elected as the master node that controls the cluster. You can also configure a new node for a specific purpose, such as handling ingest requests. For more information, see Nodes.

When you add more nodes to a cluster, it automatically allocates replica shards. When all primary and replica shards are active, the cluster state changes to green.

A cluster with three nodes

To add a node to a cluster:

  1. Set up a new {es} instance.

  2. Specify the name of the cluster in its cluster.name attribute. For example, to add a node to the logging-prod cluster, set cluster.name: "logging-prod" in elasticsearch.yml.

  3. Start {es}. The node automatically discovers and joins the specified cluster.

For more information about discovery and shard allocation, see [modules-discovery] and [modules-cluster].