"Fossies" - the Fresh Open Source Software Archive

Member "elasticsearch-6.8.23/docs/reference/how-to.asciidoc" (29 Dec 2021, 698 Bytes) of package /linux/www/elasticsearch-6.8.23-src.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming AsciiDoc format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field.

General recommendations

Don’t return large result sets

Elasticsearch is designed as a search engine, which makes it very good at getting back the top documents that match a query. However, it is not as good for workloads that fall into the database domain, such as retrieving all documents that match a particular query. If you need to do this, make sure to use the Scroll API.

Avoid large documents

Given that the default http.max_content_length is set to 100MB, Elasticsearch will refuse to index any document that is larger than that. You might decide to increase that particular setting, but Lucene still has a limit of about 2GB.

Even without considering hard limits, large documents are usually not practical. Large documents put more stress on network, memory usage and disk, even for search requests that do not request the _source since Elasticsearch needs to fetch the _id of the document in all cases, and the cost of getting this field is bigger for large documents due to how the filesystem cache works. Indexing this document can use an amount of memory that is a multiplier of the original size of the document. Proximity search (phrase queries for instance) and highlighting also become more expensive since their cost directly depends on the size of the original document.

It is sometimes useful to reconsider what the unit of information should be. For instance, the fact you want to make books searchable doesn’t necesarily mean that a document should consist of a whole book. It might be a better idea to use chapters or even paragraphs as documents, and then have a property in these documents that identifies which book they belong to. This does not only avoid the issues with large documents, it also makes the search experience better. For instance if a user searches for two words foo and bar, a match across different chapters is probably very poor, while a match within the same paragraph is likely good.

Recipes

This section includes a few recipes to help with common problems:

Mixing exact search with stemming

When building a search application, stemming is often a must as it is desirable for a query on skiing to match documents that contain ski or skis. But what if a user wants to search for skiing specifically? The typical way to do this would be to use a multi-field in order to have the same content indexed in two different ways:

PUT index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "english_exact": {
          "tokenizer": "standard",
          "filter": [
            "lowercase"
          ]
        }
      }
    }
  },
  "mappings": {
    "_doc": {
      "properties": {
        "body": {
          "type": "text",
          "analyzer": "english",
          "fields": {
            "exact": {
              "type": "text",
              "analyzer": "english_exact"
            }
          }
        }
      }
    }
  }
}

PUT index/_doc/1
{
  "body": "Ski resort"
}

PUT index/_doc/2
{
  "body": "A pair of skis"
}

POST index/_refresh

With such a setup, searching for ski on body would return both documents:

GET index/_search
{
  "query": {
    "simple_query_string": {
      "fields": [ "body" ],
      "query": "ski"
    }
  }
}
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "_doc",
        "_id": "2",
        "_score": 0.2876821,
        "_source": {
          "body": "A pair of skis"
        }
      },
      {
        "_index": "index",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "Ski resort"
        }
      }
    ]
  }
}

On the other hand, searching for ski on body.exact would only return document 1 since the analysis chain of body.exact does not perform stemming.

GET index/_search
{
  "query": {
    "simple_query_string": {
      "fields": [ "body.exact" ],
      "query": "ski"
    }
  }
}
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "Ski resort"
        }
      }
    ]
  }
}

This is not something that is easy to expose to end users, as we would need to have a way to figure out whether they are looking for an exact match or not and redirect to the appropriate field accordingly. Also what to do if only parts of the query need to be matched exactly while other parts should still take stemming into account?

Fortunately, the query_string and simple_query_string queries have a feature that solves this exact problem: quote_field_suffix. This tells Elasticsearch that the words that appear in between quotes are to be redirected to a different field, see below:

GET index/_search
{
  "query": {
    "simple_query_string": {
      "fields": [ "body" ],
      "quote_field_suffix": ".exact",
      "query": "\"ski\""
    }
  }
}
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped" : 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "index",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "body": "Ski resort"
        }
      }
    ]
  }
}

In the above case, since ski was in-between quotes, it was searched on the body.exact field due to the quote_field_suffix parameter, so only document 1 matched. This allows users to mix exact search with stemmed search as they like.

Getting consistent scoring

The fact that Elasticsearch operates with shards and replicas adds challenges when it comes to having good scoring.

Scores are not reproducible

Say the same user runs the same request twice in a row and documents do not come back in the same order both times, this is a pretty bad experience isn’t it? Unfortunately this is something that can happen if you have replicas (index.number_of_replicas is greater than 0). The reason is that Elasticsearch selects the shards that the query should go to in a round-robin fashion, so it is quite likely if you run the same query twice in a row that it will go to different copies of the same shard.

Now why is it a problem? Index statistics are an important part of the score. And these index statistics may be different across copies of the same shard due to deleted documents. As you may know when documents are deleted or updated, the old document is not immediately removed from the index, it is just marked as deleted and it will only be removed from disk on the next time that the segment this old document belongs to is merged. However for practical reasons, those deleted documents are taken into account for index statistics. So imagine that the primary shard just finished a large merge that removed lots of deleted documents, then it might have index statistics that are sufficiently different from the replica (which still have plenty of deleted documents) so that scores are different too.

The recommended way to work around this issue is to use a string that identifies the user that is logged is (a user id or session id for instance) as a preference. This ensures that all queries of a given user are always going to hit the same shards, so scores remain more consistent across queries.

This work around has another benefit: when two documents have the same score, they will be sorted by their internal Lucene doc id (which is unrelated to the _id or _uid) by default. However these doc ids could be different across copies of the same shard. So by always hitting the same shard, we would get more consistent ordering of documents that have the same scores.

Relevancy looks wrong

If you notice that two documents with the same content get different scores or that an exact match is not ranked first, then the issue might be related to sharding. By default, Elasticsearch makes each shard responsible for producing its own scores. However since index statistics are an important contributor to the scores, this only works well if shards have similar index statistics. The assumption is that since documents are routed evenly to shards by default, then index statistics should be very similar and scoring would work as expected. However in the event that you either:

  • use routing at index time,

  • query multiple indices,

  • or have too little data in your index

then there are good chances that all shards that are involved in the search request do not have similar index statistics and relevancy could be bad.

If you have a small dataset, the easiest way to work around this issue is to index everything into an index that has a single shard (index.number_of_shards: 1). Then index statistics will be the same for all documents and scores will be consistent.

Otherwise the recommended way to work around this issue is to use the dfs_query_then_fetch search type. This will make Elasticsearch perform an inital round trip to all involved shards, asking them for their index statistics relatively to the query, then the coordinating node will merge those statistics and send the merged statistics alongside the request when asking shards to perform the query phase, so that shards can use these global statistics rather than their own statistics in order to do the scoring.

In most cases, this additional round trip should be very cheap. However in the event that your query contains a very large number of fields/terms or fuzzy queries, beware that gathering statistics alone might not be cheap since all terms have to be looked up in the terms dictionaries in order to look up statistics.

Tune for indexing speed

Use bulk requests

Bulk requests will yield much better performance than single-document index requests. In order to know the optimal size of a bulk request, you should run a benchmark on a single node with a single shard. First try to index 100 documents at once, then 200, then 400, etc. doubling the number of documents in a bulk request in every benchmark run. When the indexing speed starts to plateau then you know you reached the optimal size of a bulk request for your data. In case of tie, it is better to err in the direction of too few rather than too many documents. Beware that too large bulk requests might put the cluster under memory pressure when many of them are sent concurrently, so it is advisable to avoid going beyond a couple tens of megabytes per request even if larger requests seem to perform better.

Use multiple workers/threads to send data to Elasticsearch

A single thread sending bulk requests is unlikely to be able to max out the indexing capacity of an Elasticsearch cluster. In order to use all resources of the cluster, you should send data from multiple threads or processes. In addition to making better use of the resources of the cluster, this should help reduce the cost of each fsync.

Make sure to watch for TOO_MANY_REQUESTS (429) response codes (EsRejectedExecutionException with the Java client), which is the way that Elasticsearch tells you that it cannot keep up with the current indexing rate. When it happens, you should pause indexing a bit before trying again, ideally with randomized exponential backoff.

Similarly to sizing bulk requests, only testing can tell what the optimal number of workers is. This can be tested by progressively increasing the number of workers until either I/O or CPU is saturated on the cluster.

Increase the refresh interval

The default index.refresh_interval is 1s, which forces Elasticsearch to create a new segment every second. Increasing this value (to say, 30s) will allow larger segments to flush and decreases future merge pressure.

Disable refresh and replicas for initial loads

If you need to load a large amount of data at once, you should disable refresh by setting index.refresh_interval to -1 and set index.number_of_replicas to 0. This will temporarily put your index at risk since the loss of any shard will cause data loss, but at the same time indexing will be faster since documents will be indexed only once. Once the initial loading is finished, you can set index.refresh_interval and index.number_of_replicas back to their original values.

Disable swapping

You should make sure that the operating system is not swapping out the java process by disabling swapping.

Give memory to the filesystem cache

The filesystem cache will be used in order to buffer I/O operations. You should make sure to give at least half the memory of the machine running Elasticsearch to the filesystem cache.

Use auto-generated ids

When indexing a document that has an explicit id, Elasticsearch needs to check whether a document with the same id already exists within the same shard, which is a costly operation and gets even more costly as the index grows. By using auto-generated ids, Elasticsearch can skip this check, which makes indexing faster.

Use faster hardware

If indexing is I/O bound, you should investigate giving more memory to the filesystem cache (see above) or buying faster drives. In particular SSD drives are known to perform better than spinning disks. Always use local storage, remote filesystems such as NFS or SMB should be avoided. Also beware of virtualized storage such as Amazon’s Elastic Block Storage. Virtualized storage works very well with Elasticsearch, and it is appealing since it is so fast and simple to set up, but it is also unfortunately inherently slower on an ongoing basis when compared to dedicated local storage. If you put an index on EBS, be sure to use provisioned IOPS otherwise operations could be quickly throttled.

Stripe your index across multiple SSDs by configuring a RAID 0 array. Remember that it will increase the risk of failure since the failure of any one SSD destroys the index. However this is typically the right tradeoff to make: optimize single shards for maximum performance, and then add replicas across different nodes so there’s redundancy for any node failures. You can also use snapshot and restore to backup the index for further insurance.

Indexing buffer size

If your node is doing only heavy indexing, be sure indices.memory.index_buffer_size is large enough to give at most 512 MB indexing buffer per shard doing heavy indexing (beyond that indexing performance does not typically improve). Elasticsearch takes that setting (a percentage of the java heap or an absolute byte-size), and uses it as a shared buffer across all active shards. Very active shards will naturally use this buffer more than shards that are performing lightweight indexing.

The default is 10% which is often plenty: for example, if you give the JVM 10GB of memory, it will give 1GB to the index buffer, which is enough to host two shards that are heavily indexing.

Disable _field_names

The _field_names field introduces some index-time overhead, so you might want to disable it if you never need to run exists queries.

Additional optimizations

Many of the strategies outlined in Tune for disk usage also provide an improvement in the speed of indexing.

Tune for search speed

Give memory to the filesystem cache

Elasticsearch heavily relies on the filesystem cache in order to make search fast. In general, you should make sure that at least half the available memory goes to the filesystem cache so that Elasticsearch can keep hot regions of the index in physical memory.

Use faster hardware

If your search is I/O bound, you should investigate giving more memory to the filesystem cache (see above) or buying faster drives. In particular SSD drives are known to perform better than spinning disks. Always use local storage, remote filesystems such as NFS or SMB should be avoided. Also beware of virtualized storage such as Amazon’s Elastic Block Storage. Virtualized storage works very well with Elasticsearch, and it is appealing since it is so fast and simple to set up, but it is also unfortunately inherently slower on an ongoing basis when compared to dedicated local storage. If you put an index on EBS, be sure to use provisioned IOPS otherwise operations could be quickly throttled.

If your search is CPU-bound, you should investigate buying faster CPUs.

Document modeling

Documents should be modeled so that search-time operations are as cheap as possible.

In particular, joins should be avoided. nested can make queries several times slower and parent-child relations can make queries hundreds of times slower. So if the same questions can be answered without joins by denormalizing documents, significant speedups can be expected.

Search as few fields as possible

The more fields a query_string or multi_match query targets, the slower it is. A common technique to improve search speed over multiple fields is to copy their values into a single field at index time, and then use this field at search time. This can be automated with the copy-to directive of mappings without having to change the source of documents. Here is an example of an index containing movies that optimizes queries that search over both the name and the plot of the movie by indexing both values into the name_and_plot field.

PUT movies
{
  "mappings": {
    "_doc": {
      "properties": {
        "name_and_plot": {
          "type": "text"
        },
        "name": {
          "type": "text",
          "copy_to": "name_and_plot"
        },
        "plot": {
          "type": "text",
          "copy_to": "name_and_plot"
        }
      }
    }
  }
}

Pre-index data

You should leverage patterns in your queries to optimize the way data is indexed. For instance, if all your documents have a price field and most queries run range aggregations on a fixed list of ranges, you could make this aggregation faster by pre-indexing the ranges into the index and using a terms aggregations.

For instance, if documents look like:

PUT index/_doc/1
{
  "designation": "spoon",
  "price": 13
}

and search requests look like:

GET index/_search
{
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 10 },
          { "from": 10, "to": 100 },
          { "from": 100 }
        ]
      }
    }
  }
}

Then documents could be enriched by a price_range field at index time, which should be mapped as a keyword:

PUT index
{
  "mappings": {
    "_doc": {
      "properties": {
        "price_range": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT index/_doc/1
{
  "designation": "spoon",
  "price": 13,
  "price_range": "10-100"
}

And then search requests could aggregate this new field rather than running a range aggregation on the price field.

GET index/_search
{
  "aggs": {
    "price_ranges": {
      "terms": {
        "field": "price_range"
      }
    }
  }
}

Consider mapping identifiers as keyword

The fact that some data is numeric does not mean it should always be mapped as a numeric field. The way that Elasticsearch indexes numbers optimizes for range queries while keyword fields are better at term queries. Typically, fields storing identifiers such as an ISBN or any number identifying a record from another database are rarely used in range queries or aggregations. This is why they might benefit from being mapped as keyword rather than as integer or long.

Avoid scripts

In general, scripts should be avoided. If they are absolutely needed, you should prefer the painless and expressions engines.

Search rounded dates

Queries on date fields that use now are typically not cacheable since the range that is being matched changes all the time. However switching to a rounded date is often acceptable in terms of user experience, and has the benefit of making better use of the query cache.

For instance the below query:

PUT index/_doc/1
{
  "my_date": "2016-05-11T16:30:55.328Z"
}

GET index/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "range": {
          "my_date": {
            "gte": "now-1h",
            "lte": "now"
          }
        }
      }
    }
  }
}

could be replaced with the following query:

GET index/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "range": {
          "my_date": {
            "gte": "now-1h/m",
            "lte": "now/m"
          }
        }
      }
    }
  }
}

In that case we rounded to the minute, so if the current time is 16:31:29, the range query will match everything whose value of the my_date field is between 15:31:00 and 16:31:59. And if several users run a query that contains this range in the same minute, the query cache could help speed things up a bit. The longer the interval that is used for rounding, the more the query cache can help, but beware that too aggressive rounding might also hurt user experience.

Note
It might be tempting to split ranges into a large cacheable part and smaller not cacheable parts in order to be able to leverage the query cache, as shown below:
GET index/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "should": [
            {
              "range": {
                "my_date": {
                  "gte": "now-1h",
                  "lte": "now-1h/m"
                }
              }
            },
            {
              "range": {
                "my_date": {
                  "gt": "now-1h/m",
                  "lt": "now/m"
                }
              }
            },
            {
              "range": {
                "my_date": {
                  "gte": "now/m",
                  "lte": "now"
                }
              }
            }
          ]
        }
      }
    }
  }
}

However such practice might make the query run slower in some cases since the overhead introduced by the bool query may defeat the savings from better leveraging the query cache.

Force-merge read-only indices

Indices that are read-only would benefit from being merged down to a single segment. This is typically the case with time-based indices: only the index for the current time frame is getting new documents while older indices are read-only.

Important
Don’t force-merge indices that are still being written to — leave merging to the background merge process.

Warm up global ordinals

Global ordinals are a data-structure that is used in order to run terms aggregations on keyword fields. They are loaded lazily in memory because Elasticsearch does not know which fields will be used in terms aggregations and which fields won’t. You can tell Elasticsearch to load global ordinals eagerly at refresh-time by configuring mappings as described below:

PUT index
{
  "mappings": {
    "_doc": {
      "properties": {
        "foo": {
          "type": "keyword",
          "eager_global_ordinals": true
        }
      }
    }
  }
}

Warm up the filesystem cache

If the machine running Elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. You can explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the index.store.preload setting.

Warning
Loading data into the filesystem cache eagerly on too many indices or too many files will make search slower if the filesystem cache is not large enough to hold all the data. Use with caution.

Use index sorting to speed up conjunctions

Index sorting can be useful in order to make conjunctions faster at the cost of slightly slower indexing. Read more about it in the index sorting documentation.

Use preference to optimize cache utilization

There are multiple caches that can help with search performance, such as the filesystem cache, the request cache or the query cache. Yet all these caches are maintained at the node level, meaning that if you run the same request twice in a row, have 1 replica or more and use round-robin, the default routing algorithm, then those two requests will go to different shard copies, preventing node-level caches from helping.

Since it is common for users of a search application to run similar requests one after another, for instance in order to analyze a narrower subset of the index, using a preference value that identifies the current user or session could help optimize usage of the caches.

Replicas might help with throughput, but not always

In addition to improving resiliency, replicas can help improve throughput. For instance if you have a single-shard index and three nodes, you will need to set the number of replicas to 2 in order to have 3 copies of your shard in total so that all nodes are utilized.

Now imagine that you have a 2-shards index and two nodes. In one case, the number of replicas is 0, meaning that each node holds a single shard. In the second case the number of replicas is 1, meaning that each node has two shards. Which setup is going to perform best in terms of search performance? Usually, the setup that has fewer shards per node in total will perform better. The reason for that is that it gives a greater share of the available filesystem cache to each shard, and the filesystem cache is probably Elasticsearch’s number 1 performance factor. At the same time, beware that a setup that does not have replicas is subject to failure in case of a single node failure, so there is a trade-off between throughput and availability.

So what is the right number of replicas? If you have a cluster that has num_nodes nodes, num_primaries primary shards in total and if you want to be able to cope with max_failures node failures at once at most, then the right number of replicas for you is max(max_failures, ceil(num_nodes / num_primaries) - 1).

Turn on adaptive replica selection

When multiple copies of data are present, elasticsearch can use a set of criteria called adaptive replica selection to select the best copy of the data based on response time, service time, and queue size of the node containing each copy of the shard. This can improve query throughput and reduce latency for search-heavy applications.

Tune your queries with the Profile API

You can also analyse how expensive each component of your queries and aggregations are using the {ref}/search-profile.html[Profile API]. This might allow you to tune your queries to be less expensive, resulting in a positive performance result and reduced load. Also note that Profile API payloads can be easily visualised for better readability in the {kibana-ref}/xpack-profiler.html[Search Profiler], which is a Kibana dev tools UI available in all X-Pack licenses, including the free X-Pack Basic license.

Some caveats to the Profile API are that:

  • the Profile API as a debugging tool adds significant overhead to search execution and can also have a very verbose output

  • given the added overhead, the resulting took times are not reliable indicators of actual took time, but can be used comparatively between clauses for relative timing differences

  • the Profile API is best for exploring possible reasons behind the most costly clauses of a query but isn’t intended for accurately measuring absolute timings of each clause

Tune for disk usage

Disable the features you do not need

By default Elasticsearch indexes and adds doc values to most fields so that they can be searched and aggregated out of the box. For instance if you have a numeric field called foo that you need to run histograms on but that you never need to filter on, you can safely disable indexing on this field in your mappings:

PUT index
{
  "mappings": {
    "_doc": {
      "properties": {
        "foo": {
          "type": "integer",
          "index": false
        }
      }
    }
  }
}

text fields store normalization factors in the index in order to be able to score documents. If you only need matching capabilities on a text field but do not care about the produced scores, you can configure Elasticsearch to not write norms to the index:

PUT index
{
  "mappings": {
    "_doc": {
      "properties": {
        "foo": {
          "type": "text",
          "norms": false
        }
      }
    }
  }
}

text fields also store frequencies and positions in the index by default. Frequencies are used to compute scores and positions are used to run phrase queries. If you do not need to run phrase queries, you can tell Elasticsearch to not index positions:

PUT index
{
  "mappings": {
    "_doc": {
      "properties": {
        "foo": {
          "type": "text",
          "index_options": "freqs"
        }
      }
    }
  }
}

Furthermore if you do not care about scoring either, you can configure Elasticsearch to just index matching documents for every term. You will still be able to search on this field, but phrase queries will raise errors and scoring will assume that terms appear only once in every document.

PUT index
{
  "mappings": {
    "_doc": {
      "properties": {
        "foo": {
          "type": "text",
          "norms": false,
          "index_options": "freqs"
        }
      }
    }
  }
}

Don’t use default dynamic string mappings

The default dynamic string mappings will index string fields both as text and keyword. This is wasteful if you only need one of them. Typically an id field will only need to be indexed as a keyword while a body field will only need to be indexed as a text field.

This can be disabled by either configuring explicit mappings on string fields or setting up dynamic templates that will map string fields as either text or keyword.

For instance, here is a template that can be used in order to only map string fields as keyword:

PUT index
{
  "mappings": {
    "_doc": {
      "dynamic_templates": [
        {
          "strings": {
            "match_mapping_type": "string",
            "mapping": {
              "type": "keyword"
            }
          }
        }
      ]
    }
  }
}

Watch your shard size

Larger shards are going to be more efficient at storing data. To increase the size of your shards, you can decrease the number of primary shards in an index by creating indices with less primary shards, creating less indices (e.g. by leveraging the Rollover API), or modifying an existing index using the Shrink API.

Keep in mind that large shard sizes come with drawbacks, such as long full recovery times.

Disable _all

The _all field indexes the value of all fields of a document and can use significant space. If you never need to search against all fields at the same time, it can be disabled.

Disable _source

The _source field stores the original JSON body of the document. If you don’t need access to it you can disable it. However, APIs that needs access to _source such as update and reindex won’t work.

Use best_compression

The _source and stored fields can easily take a non negligible amount of disk space. They can be compressed more aggressively by using the best_compression codec.

Force Merge

Indices in Elasticsearch are stored in one or more shards. Each shard is a Lucene index and made up of one or more segments - the actual files on disk. Larger segments are more efficient for storing data.

The _forcemerge API can be used to reduce the number of segments per shard. In many cases, the number of segments can be reduced to one per shard by setting max_num_segments=1.

Shrink Index

The Shrink API allows you to reduce the number of shards in an index. Together with the Force Merge API above, this can significantly reduce the number of shards and segments of an index.

Use the smallest numeric type that is sufficient

The type that you pick for numeric data can have a significant impact on disk usage. In particular, integers should be stored using an integer type (byte, short, integer or long) and floating points should either be stored in a scaled_float if appropriate or in the smallest type that fits the use-case: using float over double, or half_float over float will help save storage.

Use index sorting to colocate similar documents

When Elasticsearch stores _source, it compresses multiple documents at once in order to improve the overall compression ratio. For instance it is very common that documents share the same field names, and quite common that they share some field values, especially on fields that have a low cardinality or a zipfian distribution.

By default documents are compressed together in the order that they are added to the index. If you enabled index sorting then instead they are compressed in sorted order. Sorting documents with similar structure, fields, and values together should improve the compression ratio.

Put fields in the same order in documents

Due to the fact that multiple documents are compressed together into blocks, it is more likely to find longer duplicate strings in those _source documents if fields always occur in the same order.