"Fossies" - the Fresh Open Source Software Archive

Member "elasticsearch-6.8.8/docs/reference/release-notes.asciidoc" (18 Mar 2020, 1642 Bytes) of package /linux/www/elasticsearch-6.8.8-src.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format (assuming AsciiDoc format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field. See also the latest Fossies "Diffs" side-by-side code changes report for "release-notes.asciidoc": 6.8.7_vs_6.8.8.

{es} version 6.8.7

Bug fixes

Authentication
  • Preserve ApiKey credentials for async verification {pull}51244[#51244]

CCR
  • Fix shard follow task cleaner under security {pull}52347[#52347] (issues: {issue}44702[#44702], {issue}51971[#51971])

Engine
  • Account soft-deletes in FrozenEngine {pull}51192[#51192] (issue: {issue}50775[#50775])

Features/Java Low Level REST Client
  • Improve warning value extraction performance in Response {pull}50208[#50208] (issue: {issue}24114[#24114])

Infra/Core
  • Ignore virtual ethernet devices that disappear {pull}51581[#51581] (issue: {issue}49914[#49914])

Machine Learning
  • Include out-of-order as well as in-order terms in categorization reverse searches {ml-pull}950[#950] (issue: {ml-issue}949[#949])

SQL
  • Fix ORDER BY on aggregates and GROUPed BY fields {pull}51894[#51894] (issue: {issue}50355[#50355])

  • Fix ORDER BY YEAR() function {pull}51562[#51562] (issue: {issue}51224[#51224])

  • Fix issue with CAST and NULL checking. {pull}50371[#50371] (issue: {issue}50191[#50191])

Search
  • Fix NPE bug inner_hits {pull}50709[#50709] (issue: {issue}50539[#50539])

Enhancements

SQL
  • Verify Full-Text Search functions not allowed in SELECT {pull}51568[#51568] (issue: {issue}47446[#47446])

Upgrades

Infra/Packaging
  • Upgrade the bundled JDK to JDK 13.0.2 {pull}51511[#51511]

{es} version 6.8.6

Enhancements

Features/Watcher
  • Log attachment generation failures {pull}50080[#50080]

Infra/Packaging
  • Restrict support for CMS to pre-JDK 14 {pull}49123[#49123] (issue: {issue}46973[#46973])

Bug fixes

Aggregations
  • Avoid precision loss in DocValueFormat.RAW#parseLong {pull}49063[#49063] (issue: {issue}38692[#38692])

Audit
  • Audit log filter and marker {pull}45456[#45456] (issue: {issue}47251[#47251])

Authentication
  • Fix iterate-from-1 bug in smart realm order {pull}49473[#49473]

CCR
  • CCR should auto-retry rejected execution exceptions {pull}49213[#49213]

Features/Java High Level REST Client
  • Support es7 node http publish_address format {pull}49279[#49279] (issue: {issue}48950[#48950])

Machine Learning
  • Fixes for stop datafeed edge cases {pull}49191[#49191] (issues: {issue}43670[#43670], {issue}48931[#48931])

Recovery
  • Ignore Lucene index in peer recovery if translog corrupted {pull}49114[#49114]

Reindex
  • Fix delete- and update-by-query on indices without sequence numbers {pull}50077[#50077]

{es} version 6.8.5

Enhancements

Engine
  • Avoid unneeded refresh with concurrent realtime gets {pull}47895[#47895]

Infra/Settings
  • Do not reference values for filtered settings {pull}48066[#48066]

Machine Learning
  • The {ml} native processes are now arranged in a .app directory structure on macOS to allow for notarization on macOS Catalina {ml-pull}593[#593]

Bug fixes

Allocation
  • Handle negative free disk space in deciders {pull}48392[#48392] (issue: {issue}48380[#48380])

Analysis
  • Actually close IndexAnalyzers contents {pull}43914[#43914]

CCR
  • Handle lower retaining sequence number retention lease error {pull}46420[#46420] (issues: {issue}46013[#46013], {issue}46416[#46416])

CRUD
  • Close query cache on index service creation failure {pull}48230[#48230] (issue: {issue}48186[#48186])

Engine
  • Fix refresh optimization for realtime get in mixed cluster {pull}48151[#48151] (issue: {issue}48114[#48114])

Features/Java High Level REST Client
  • Fix ILM HLRC Javadoc→documentation links {pull}48083[#48083]

  • Prevent deadlock in BulkProcessor by using separate schedulers {pull}48697[#48697] (issue: {issue}47599[#47599])

Infra/Logging
  • Slow log must use separate underlying logger for each index {pull}47234[#47234] (issue: {issue}42432[#42432])

Machine Learning
  • Restore from checkpoint could damage seasonality modeling. For example, it could cause seasonal components to be overwritten in error {ml-pull}821[#821]

SQL
  • Fix issue with negative literels and parentheses {pull}48113[#48113] (issue: {issue}48009[#48009])

  • Failing group by queries due to different ExpressionIds {pull}43072[#43072] (issues: {issue}33361[#33361], {issue}34543[#34543], {issue}36074[#36074], {issue}37044[#37044], {issue}40001[#40001], {issue}40240[#40240], {issue}41159[#41159], {issue}42041[#42041], {issue}46316[#46316])

Snapshot/Restore
  • Cleanup concurrent RepositoryData loading {pull}48329[#48329] (issue: {issue}48122[#48122])

{es} version 6.8.4

Enhancements

Engine
  • Limit number of retaining translog files for peer recovery {pull}47414[#47414]

Infra/Settings
  • Add more meaningful keystore version mismatch errors {pull}46291[#46291] (issue: {issue}44624[#44624])

Machine Learning
  • Throttle the delete-by-query of expired results {pull}47177[#47177] (issues: {issue}47003[#47003])

Bug fixes

Authentication
  • Fallback to realm authc if ApiKey fails {pull}46538[#46538]

CRUD
  • Do not use ifSeqNo if doc does not have seq_no {pull}46198[#46198]

Distributed
  • Fix false positive out of sync warning in synced-flush {pull}46576[#46576] (issues: {issue}28464[#28464], {issue}30244[#30244])

  • Suppress warning logs from background sync on relocated primary {pull}46247[#46247] (issues: {issue}40800[#40800], {issue}42241[#42241])

Engine
  • Improve translog corruption detection {pull}47873[#47873] (issues: {issue}41480[#41480], {issue}42744[#42744], {issue}42980[#42980], {issue}44217[#44217])

  • Always rebuild checkpoint tracker for old indices {pull}46340[#46340] (issues: {issue}38879[#38879], {issue}46311[#46311])

Features/Indices APIs
  • Fix Rollover error when alias has closed indices {pull}47839[#47839] (issue: {issue}47148[#47148])

Features/Ingest
  • Allow dropping documents with auto-generated ID {pull}46773[#46773] (issue: {issue}46678[#46678])

Features/Watcher
  • Watcher - workaround for potential deadlock {pull}47603[#47603] (issues: {issue}41451[#41451], {issue}47599[#47599])

  • Fix class used to initialize logger in Watcher {pull}46467[#46467]

Machine Learning
  • Reinstate ML daily maintenance actions {pull}47103[#47103] (issue: {issue}47003[#47003])

  • Fix two datafeed flush lockup bugs {pull}46982[#46982]

  • Throw an error when a datafeed needs CCS but it is not enabled for the node {pull}46044[#46044]

  • Fix possibility of crash when calculating forecasts that overflow to disk {ml-pull}688[#688]

SQL
  • SQL: Allow whitespaces in escape patterns {pull}47577[#47577] (issue: {issue}47401[#47401])

  • SQL: Fix issue with common type resolution {pull}46565[#46565] (issue: {issue}46551[#46551])

  • SQL: wrong number of values for columns {pull}42122[#42122]

Snapshot/Restore
  • Fix Bug in Azure Repo Exception Handling {pull}47968[#47968]

  • Fix Snapshot Corruption in Edge Case {pull}47552[#47552] (issues: {issue}46250[#46250], {issue}47550[#47550])

  • Fix TransportSnapshotsStatusAction ThreadPool Use {pull}45824[#45824]

Upgrades

Engine
  • Upgrade to lucene 7.7.2 {pull}47901[#47901]

{es} version 6.8.3

Enhancements

CCR
  • Include leases in error message when operations no longer available {pull}45681[#45681]

Infra/Core
  • Add OCI annotations and adjust existing annotations {pull}45167[#45167]

Infra/Settings
  • Normalize environment paths {pull}45179[#45179] (issue: {issue}45176[#45176])

Bug fixes

Aggregations
  • Fix early termination of aggregators that run with breadth-first mode {pull}44963[#44963] (issue: {issue}44909[#44909])

Allocation
  • Avoid overshooting watermarks during relocation {pull}46079[#46079] (issue: {issue}45177[#45177])

CCR
  • Clean up ShardFollowTasks for deleted indices {pull}44702[#44702] (issue: {issue}34404[#34404])

CRUD
  • Allow _update on write alias {pull}45318[#45318] (issue: {issue}31520[#31520])

Distributed
  • Ensure AsyncTask#isScheduled remain false after close {pull}45687[#45687] (issue: {issue}45576[#45576])

Engine
  • Handle delete document level failures {pull}46100[#46100]

  • Handle no-op document level failures {pull}46083[#46083]

Features/Indices APIs
  • Check shard limit after applying index templates {pull}44619[#44619] (issue: {issue}44567[#44567])

  • Prevent NullPointerException in TransportRolloverAction {pull}43353[#43353] (issue: {issue}43296[#43296])

Features/Java High Level REST Client
  • Add rest_total_hits_as_int in HLRC’s search requests {pull}46076[#46076] (issue: {issue}43925[#43925])

Infra/Core
  • Add default CLI JVM options {pull}44545[#44545] (issue: {issue}42021[#42021])

Infra/Plugins
  • Do not checksum all bytes at once in plugin install {pull}44649[#44649] (issue: {issue}44545[#44545])

SQL
  • Fix URI path being lost in case of hosted ES scenario {pull}44776[#44776] (issue: {issue}44721[#44721])

  • Interval arithmetics failure {pull}42014[#42014] (issue: {issue}41200[#41200])

Search
  • Prevent Leaking Search Tasks on Exceptions in FetchSearchPhase and DfsQueryPhase {pull}45500[#45500]

Security
  • Use system context for looking up connected nodes {pull}43991[#43991] (issue: {issue}43974[#43974])

Upgrades

Infra/Packaging
  • Upgrade to JDK 12.0.2 {pull}45172[#45172]

{es} version 6.8.2

Enhancements

Machine Learning
  • Improve message when native controller cannot connect {pull}43565[#43565] (issue: {issue}42341[#42341])

  • Introduce a setting for the process connect timeout {pull}43234[#43234]

Ranking
  • Fix parameter value for calling data.advanceExact {pull}44205[#44205]

Snapshot/Restore
  • Add SAS token authentication support to Azure Repository plugin {pull}42982[#42982] (issue: {issue}42117[#42117])

Bug fixes

Allocation
  • Do not copy initial recovery filter during an index split {pull}44053[#44053] (issue: {issue}43955[#43955])

Analysis
  • Fix varying responses for <index>/_analyze request {pull}44342[#44342] (issues: {issue}44078[#44078], {issue}44284[#44284])

CCR
  • Skip update if leader and follower settings are identical {pull}44535[#44535] (issue: {issue}44521[#44521])

  • Avoid stack overflow in auto-follow coordinator {pull}44421[#44421] (issue: {issue}43251[#43251])

  • Avoid NPE when checking for CCR index privileges {pull}44397[#44397] (issue: {issue}44172[#44172])

Engine
  • Preserve thread context in AsyncIOProcessor {pull}43729[#43729]

Features/ILM
  • Fix swapped variables in error message {pull}44300[#44300]

  • Account for node versions during allocation in ILM shrink {pull}43300[#43300] (issue: {issue}41879[#41879])

  • Narrow period of shrink action in which ILM prevents stopping {pull}43254[#43254] (issue: {issue}43253[#43253])

Features/Indices APIs
  • Validate index settings after applying templates {pull}44612[#44612] (issues: {issue}34021[#34021], {issue}44567[#44567])

Features/Stats
  • Return 0 for negative "free" and "total" memory reported by the OS {pull}42725[#42725] (issue: {issue}42157[#42157])

Machine Learning
  • Fix ML memory tracker lockup when inner step fails {pull}44158[#44158] (issue: {issue}44156[#44156])

  • Fix datafeed checks when a concrete remote index is present {pull}43923[#43923] (issue: {issue}42113[#42113])

  • Don’t write model size stats when job is closed without any input {ml-pull}512[#512] (issue: {ml-issue}394[#394])

  • Don’t persist model state at the end of lookback if the lookback did not generate any input {ml-pull}521[#521] (issue: {ml-issue}519[#519])

Mapping
  • Prevent types deprecation warning for indices.exists requests {pull}43963[#43963] (issue: {issue}43905[#43905])

  • Add include_type_name in indices.exists REST API specification {pull}43910[#43910] (issue: {issue}43905[#43905])

Network
  • Reconnect remote cluster when seeds are changed {pull}43379[#43379] (issue: {issue}37799[#37799])

SQL
  • Handle the edge case of an empty array of values to return from source {pull}43868[#43868] (issue: {issue}43863[#43863])

  • Fix NPE in case of subsequent scrolled requests for a CSV/TSV formatted response {pull}43365[#43365] (issue: {issue}43327[#43327])

  • Increase hard limit for sorting on aggregates {pull}43220[#43220] (issue: {issue}43168[#43168])

Search
  • Fix wrong logic in match_phrase query with multi-word synonyms {pull}43941[#43941] (issue: {issue}43308[#43308])

Security
  • Do not swallow I/O exception getting authentication {pull}44398[#44398] (issues: {issue}44172[#44172], {issue}44397[#44397])

  • SecurityIndexSearcherWrapper doesn’t always carry over caches and similarity {pull}43436[#43436]

{es} version 6.8.1

Enhancements

Java High Level REST Client
  • Added param ignore_throttled=false when indicesOptions.ignoreThrottle {pull}42393[#42393] (issue: {issue}42358[#42358])

Bug fixes

Allocation
  • Avoid bubbling up failures from a shard that is recovering {pull}42287[#42287] (issues: {issue}30919[#30919], {issue}40107[#40107])

Authentication
  • Only index into "doc" type in security index {pull}42563[#42563] (issue: {issue}42562[#42562])

Distributed
  • Do not use ifSeqNo for update requests on mixed cluster {pull}42596[#42596] (issue: {issue}42561[#42561])

  • Avoid unnecessary persistence of retention leases {pull}42299[#42299]

  • Execute actions under permit in primary mode only {pull}42241[#42241] (issues: {issue}40386[#40386], {issue}41686[#41686])

Engine
  • Account soft deletes for committed segments {pull}43126[#43126] (issue: {issue}43103[#43103])

Infra/Core
  • scheduleAtFixedRate would hang {pull}42993[#42993] (issue: {issue}38441[#38441])

  • Only ignore IOException when fsyncing on dirs {pull}42972[#42972] (issue: {issue}42950[#42950])

  • Deprecation info for joda-java migration {pull}41956[#41956] (issue: {issue}42010[#42010])

Machine Learning
  • Fix possible race condition when closing an opening job {pull}42506[#42506]

Mapping
  • Fix possible NPE in put mapping validators {pull}43000[#43000] (issue: {issue}37675[#37675])

Percolator
  • Fixed ignoring name parameter for percolator queries {pull}42598[#42598] (issue: {issue}40405[#40405])

  • Exclude nested documents {pull}42554[#42554] (issue: {issue}42361[#42361])

Recovery
  • Recovery with syncId should verify seqno infos {pull}41265[#41265]

Reindex
  • Remote reindex failure parse fix {pull}42928[#42928]

Rollup
  • Fix max boundary for rollup jobs that use a delay {pull}42158[#42158]

SQL
  • Fix wrong results when sorting on aggregate {pull}43154[#43154] (issue: {issue}42851[#42851])

  • Cover the Integer type when extracting values from _source {pull}42859[#42859] (issue: {issue}42858[#42858])

Search
  • SearchRequest#allowPartialSearchResults does not handle successful retries {pull}43095[#43095] (issue: {issue}40743[#40743])

  • Fix auto fuzziness in query_string query {pull}42897[#42897]

  • Wire query cache into sorting nested-filter computation {pull}42906[#42906] (issue: {issue}42813[#42813])

  • Fix sorting on nested field with unmapped {pull}42451[#42451] (issue: {issue}33644[#33644])

  • Fixes a bug in AnalyzeRequest.toXContent() {pull}42795[#42795] (issues: {issue}39670[#39670], {issue}42197[#42197])

  • Fix concurrent search and index delete {pull}42621[#42621] (issue: {issue}28053[#28053])

Snapshot/Restore
  • Fix Azure List by Prefix Bug {pull}42713[#42713]

{es} version 6.8.0

Enhancements

Security
Authentication
  • Log warning when unlicensed realms are skipped {pull}41778[#41778]

Infra/Settings
  • Drop distinction in entries for keystore {pull}41701[#41701]

Bug fixes

Features/Watcher
  • Fix Watcher deadlock that can cause in-abilty to index documents. {pull}41418[#41418] (issue: {issue}41390[#41390])

Network
  • Enforce transport TLS on Basic with Security {pull}42150[#42150]

Reindex
  • Allow reindexing into write alias {pull}41677[#41677] (issue: {issue}41667[#41667])

SQL
  • SQL: Fix issue regarding INTERVAL * number {pull}42014[#42014] (issue: {issue}41239[#41239])

  • SQL: Remove CircuitBreaker from parser {pull}41835[#41835] (issue: {issue}41471[#41471])

Security
  • Remove dynamic objects from security index {pull}40499[#40499] (issue: {issue}35460[#35460])

  • Add granular API key privileges {pull}41488[#41488] (issue: {issue}40031[#40031])

{es} version 6.7.2

Enhancements

CCR
  • Reduce security permissions in CCR plugin {pull}41391[#41391]

Distributed
  • Avoid background sync on relocated primary {pull}40800[#40800] (issue: {issue}40731[#40731])

Infra/Packaging
  • Be lenient when parsing build flavor and type on the wire {pull}40734[#40734] (issues: {issue}39378[#39378], {issue}40723[#40723])

Machine Learning
  • [ML] Allow xpack.ml.max_machine_memory_percent higher than 100% {pull}41193[#41193]

Snapshot/Restore
  • Mark Deleted Snapshot Directories with Tombstones {pull}40228[#40228] (issue: {issue}39852[#39852])

Bug fixes

Allocation
  • Short-circuit rebalancing when disabled {pull}40966[#40966] (issue: {issue}40942[#40942])

Authorization
  • Fix role mapping DN field wildcards for users with NULL DNs {pull}41343[#41343] (issue: {issue}41305[#41305])

  • Fix Has Privilege API check on restricted indices {pull}41226[#41226]

  • put mapping authorization for alias with write-index and multiple read indices {pull}40834[#40834] (issue: {issue}40831[#40831])

  • Use alias name from rollover request to query indices stats {pull}40774[#40774] (issue: {issue}40771[#40771])

CCR
  • Suppress lease background sync failures if stopping {pull}40902[#40902]

Distributed
  • Fix NPE on Stale Index in IndicesService {pull}38891[#38891] (issue: {issue}38845[#38845])

Features/ILM
  • Allow ILM to stop if indices have nonexistent policies {pull}40820[#40820] (issue: {issue}40824[#40824])

Features/Monitoring
  • Properly handle Monitoring exporters all disabled {pull}40920[#40920] (issue: {issue}40919[#40919])

Features/Watcher
  • Use setIfSeqNo(…​) and setIfPrimaryTerm(…​) for updating watch status if all nodes are at least on 6.7.0 {pull}40888[#40888] (issue: {issue}40841[#40841])

Highlighting
  • Unified highlighter should ignore terms that target the _id field {pull}41275[#41275] (issue: {issue}37525[#37525])

  • Unified highlighter should respect no_match_size with number_of_fragments set to 0 {pull}41069[#41069] (issue: {issue}41066[#41066])

Infra/Core
  • Allow single digit milliseconds in strict date parsing {pull}40676[#40676] (issue: {issue}40403[#40403])

  • Bat scripts to work with JAVA_HOME with parantheses {pull}39712[#39712] (issues: {issue}30606[#30606], {issue}33405[#33405], {issue}38578[#38578], {issue}38624[#38624])

Infra/Packaging
  • Suppress illegal access in plugin install {pull}41620[#41620] (issue: {issue}41478[#41478])

Infra/Scripting
  • Correct bug in ScriptDocValues {pull}40864[#40864] (issue: {issue}40488[#40488])

  • Correct bug in ScriptDocValues {pull}40488[#40488]

Infra/Settings
  • Always check for archiving broken index settings {pull}41209[#41209]

Machine Learning
  • [ML] checking if p-tasks metadata is null before updating state {pull}41091[#41091] (issue: {issue}41090[#41090])

  • [ML] Write header to autodetect before it is visible to other calls {pull}41085[#41085] (issue: {issue}40385[#40385])

  • Fix unsafe publication of invalid license enforcer {pull}40985[#40985] (issue: {issue}40957[#40957])

  • [ML] Scrolling datafeed should clear scroll contexts on error {pull}40773[#40773] (issue: {issue}40772[#40772])

Mapping
  • Optimise rejection of out-of-range long values {pull}40325[#40325] (issues: {issue}26137[#26137], {issue}40323[#40323])

Recovery
  • Resync should not send operations without sequence number {pull}40433[#40433]

Rollup
  • Remove timezone validation on rollup range queries {pull}40647[#40647]

SQL
  • SQL: Fix bug with optimization of null related conditionals {pull}41355[#41355]

  • SQL: Predicate diff takes into account all values {pull}41346[#41346] (issue: {issue}40835[#40835])

  • SQL: Fix LIMIT bug in agg sorting {pull}41258[#41258] (issue: {issue}40984[#40984])

  • SQL: Allow current_date/time/timestamp to be also used as a function escape pattern {pull}41254[#41254] (issue: {issue}41240[#41240])

  • SQL: Translate MIN/MAX on keyword fields as FIRST/LAST {pull}41247[#41247] (issues: {issue}37936[#37936], {issue}41195[#41195])

  • SQL: Tweak pattern matching in SYS TABLES {pull}41243[#41243] (issue: {issue}40775[#40775])

  • SQL: Change schema calls to empty set {pull}41034[#41034] (issue: {issue}41028[#41028])

  • SQL: Fix display size for DATE/DATETIME {pull}40669[#40669]

  • SQL: Use ResultSets over exceptions in metadata {pull}40641[#40641] (issue: {issue}40533[#40533])

  • SQL: Fix catalog filtering in SYS COLUMNS {pull}40583[#40583] (issue: {issue}40582[#40582])

Search
  • Full text queries should not always ignore unmapped fields {pull}41062[#41062] (issue: {issue}41022[#41022])

  • Fix rewrite of inner queries in DisMaxQueryBuilder {pull}40956[#40956] (issue: {issue}40953[#40953])

  • Fix alias resolution runtime complexity. {pull}40263[#40263] (issue: {issue}40248[#40248])

Security
  • Fix unsafe publication in opt-out query cache {pull}40957[#40957]

  • Remove dynamic objects from security index {pull}40499[#40499] (issue: {issue}35460[#35460])

Snapshot/Restore
  • Fix Broken Index Shard Snapshot File Preventing Snapshot Creation {pull}41310[#41310] (issue: {issue}41304[#41304])

  • Do not create missing directories in readonly repo {pull}41249[#41249] (issues: {issue}26909[#26909], {issue}41009[#41009])

Upgrades

Infra/Packaging
  • Bump the bundled JDK to 12.0.1 {pull}41627[#41627]

{es} version 6.7.1

Enhancements

Infra/Core
  • Enhancements to IndicesQueryCache. {pull}39099[#39099] (issue: {issue}37117[#37117])

SQL
  • SQL: add "fuzziness" option to QUERY and MATCH function predicates {pull}40529[#40529] (issue: {issue}40495[#40495])

Bug fixes

Authentication
  • Allow non super users to create API keys {pull}40028[#40028] (issue: {issue}40029[#40029])

CCR
  • Only run retention lease actions on active primary {pull}40386[#40386] (issues: {issue}40089[#40089], {issue}40373[#40373])

CRUD
  • Store Pending Deletions Fix {pull}40345[#40345] (issue: {issue}40249[#40249])

Features/ILM
  • Correct ILM metadata minimum compatibility version {pull}40569[#40569] (issue: {issue}40565[#40565])

  • Handle null retention leases in WaitForNoFollowersStep {pull}40477[#40477]

Geo
  • Geo Point parse error fix {pull}40447[#40447] (issue: {issue}17617[#17617])

Infra/Core
  • Parse composite patterns using ClassicFormat.parseObject {pull}40100[#40100] (issue: {issue}39916[#39916])

Infra/Packaging
  • Use TAR instead of DOCKER build type before 6.7.0 {pull}40723[#40723] (issues: {issue}39378[#39378], {issue}40511[#40511])

Machine Learning
  • [ML] Addressing bug streaming DatafeedConfig aggs from (⇐ 6.5.4) → 6.7.0 {pull}40610[#40610]

SQL
  • SQL: have LIKE/RLIKE use wildcard and regexp queries {pull}40628[#40628] (issue: {issue}40557[#40557])

  • SQL: Fix getTime() methods in JDBC {pull}40484[#40484]

  • SQL: Add missing handling of IP field in JDBC {pull}40384[#40384] (issue: {issue}40358[#40358])

  • SQL: Fix metric aggs on date/time to not return double {pull}40377[#40377] (issues: {issue}39492[#39492], {issue}40376[#40376])

  • SQL: CAST supports both SQL and ES types {pull}40365[#40365] (issue: {issue}40282[#40282])

  • SQL: Fix RLIKE bug and improve testing for RLIKE statement {pull}40354[#40354] (issues: {issue}34609[#34609], {issue}39931[#39931])

  • SQL: SYS TABLES: enumerate tables of requested types [ISSUE] {pull}40348[#40348]

  • SQL: JLine upgrade and polishing {pull}40321[#40321] (issue: {issue}40239[#40239])

  • SQL: unwrap the first value in an array in case of array leniency {pull}40318[#40318] (issue: {issue}40296[#40296])

  • SQL: fix LIKE function equality by considering its pattern as well {pull}40260[#40260] (issue: {issue}39931[#39931])

  • SQL: rewrite ROUND and TRUNCATE functions with a different optional parameter handling method {pull}40242[#40242] (issue: {issue}40001[#40001])

  • SQL: passing an input to the CLI "freezes" the CLI after displaying an error message [ISSUE] {pull}40164[#40164]

{es} version 6.7.0

Known issues

A new docker build_type value was introduced in 6.7.0 for Docker images, which caused two problems:

  • If you perform a rolling upgrade to 6.7.0 on nodes that are running the official Docker container, they cannot rejoin the cluster. Upgrade to 6.7.1 instead or perform a full cluster restart upgrade. (issue: {issue}/40784[#40784])

  • If you have a 6.6 or earlier Java High Level REST client, it cannot understand the new build_type value. For example, it cannot read main responses on / from a cluster that’s running 6.7.0. Upgrade your client to 6.7.0 or later. (issue: {issue}/40511[#40511])

    Features/ILM

    If an index is configured with an index.lifecycle.name that refers to a policy that does not exist, when {ilm} is stopped, {ilm}'s status will remain as STOPPING until all policies that do not exist are removed from any indices by using the Remove Policy API. (issue: {issue}/40824[#40824])

Breaking changes

Authentication
  • The TokenService no longer accepts tokens generated by Elasticsearch 6.1 or earlier {pull}38881[#38881]

CCR
  • Follow stats api should return a 404 when requesting stats for a non existing index {pull}37220[#37220] (issue: {issue}37021[#37021])

Distributed
  • Stop returning cluster state size by default {pull}40016[#40016] (issues: {issue}39806[#39806], {issue}39827[#39827], {issue}39951[#39951])

Infra/Packaging
  • Package ingest-user-agent as a module {pull}36956[#36956]

  • Package ingest-geoip as a module {pull}36898[#36898]

Ranking
  • Forbid negative field boosts in analyzed queries {pull}37930[#37930] (issue: {issue}33309[#33309])

Breaking Java changes

Infra/Core
  • Handle scheduler exceptions {pull}38014[#38014] (issues: {issue}28667[#28667], {issue}36137[#36137], {issue}37708[#37708])

Deprecations

Analysis
  • [Analysis] Deprecate standard html analyzer in 6.x {pull}37292[#37292] (issues: {issue}26719[#26719], {issue}4704[#4704])

Audit
  • Deprecate index audit output type {pull}37671[#37671] (issues: {issue}29881[#29881], {issue}37301[#37301])

  • Deprecate index audit output type {pull}37301[#37301] (issue: {issue}29881[#29881])

CRUD
  • Deprecate support for internal versioning for concurrency control {pull}38451[#38451] (issues: {issue}10708[#10708], {issue}38254[#38254])

Cluster Coordination
  • Deprecate size in cluster state response {pull}39951[#39951] (issue: {issue}39806[#39806])

Features/Features
  • Deprecate Migration Assistance and Upgrade APIs {pull}40072[#40072] (issue: {issue}40014[#40014])

Features/Java High Level REST Client
  • Deprecate HLRC EmptyResponse used by security {pull}37540[#37540] (issue: {issue}36938[#36938])

Features/Java Low Level REST Client
  • Deprecate the low-level REST client on JDK 7 {pull}38542[#38542] (issue: {issue}29607[#29607])

  • Deprecate maxRetryTimeout in RestClient and increase default value {pull}38425[#38425] (issue: {issue}38085[#38085])

Features/Watcher
  • Deprecate Hipchat Watcher actions {pull}39160[#39160]

Infra/Core
  • Core: Deprecate negative epoch timestamps {pull}36793[#36793]

Infra/Packaging
  • Deprecate fallback to java on PATH {pull}37990[#37990]

Infra/REST API
  • Deprecate requests that have an unconsumed body {pull}37534[#37534] (issue: {issue}37504[#37504])

Machine Learning
  • Add ml_settings entry to HLRC and Docs for deprecation_info {pull}38118[#38118]

  • Datafeed deprecation checks {pull}37932[#37932]

  • Adjust structure finder for Joda to Java time migration {pull}37306[#37306]

Mapping
  • Deprecate types in get field mapping API {pull}37667[#37667] (issue: {issue}35190[#35190])

  • Deprecate types in the put mapping API. {pull}37280[#37280] (issues: {issue}29453[#29453], {issue}37285[#37285])

  • Support include_type_name in the field mapping and index template APIs. {pull}37210[#37210]

  • Deprecate types in create index requests. {pull}37134[#37134] (issues: {issue}29453[#29453], {issue}37285[#37285])

Network
  • Add TLS version changes to deprecation checks {pull}37793[#37793] (issue: {issue}37512[#37512])

  • Issue deprecation warning if TLSv1.0 is used without explicit config {pull}37788[#37788] (issue: {issue}37512[#37512])

  • Add deprecation warnings for ssl config fallback {pull}36847[#36847] (issue: {issue}36846[#36846])

Security
  • Deprecate implicit security on trial licenses {pull}38295[#38295] (issues: {issue}38009[#38009], {issue}38075[#38075])

New features

Authentication
  • Add support for API keys to access Elasticsearch {pull}38291[#38291] (issue: {issue}34383[#34383])

Authorization
  • Allow custom authorization with an authorization engine {pull}38358[#38358] (issues: {issue}32435[#32435], {issue}36245[#36245], {issue}37328[#37328], {issue}37495[#37495], {issue}37785[#37785], {issue}38137[#38137], {issue}38219[#38219])

  • WIldcard IndicesPermissions don’t cover .security {pull}36765[#36765]

CCR
  • Add ccr follow info api {pull}37408[#37408] (issue: {issue}37127[#37127])

CRUD
  • Make _doc work as an alias of the actual type of an index. {pull}39505[#39505] (issue: {issue}39469[#39469])

Features/ILM
  • [ILM] Add unfollow action {pull}36970[#36970] (issue: {issue}34648[#34648])

Features/Ingest
  • Enable grok processor to support long, double and boolean {pull}27896[#27896]

Machine Learning
  • Add set_upgrade_mode API endpoint {pull}37837[#37837]

Mapping
  • Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. {pull}37871[#37871] (issue: {issue}37773[#37773])

  • Add an include_type_name option to 6.x. (#29453) {pull}37147[#37147] (issue: {issue}35190[#35190])

SQL
  • SQL: Allow sorting of groups by aggregates {pull}38042[#38042] (issue: {issue}35118[#35118])

  • SQL: Implement FIRST/LAST aggregate functions {pull}37936[#37936] (issue: {issue}35639[#35639])

  • SQL: Introduce SQL DATE data type {pull}37693[#37693] (issue: {issue}37340[#37340])

Security
  • Switch internal security index to ".security-7" {pull}39337[#39337] (issue: {issue}39284[#39284])

Enhancements

Aggregations
  • Add Composite to AggregationBuilders {pull}38207[#38207] (issue: {issue}38020[#38020])

  • Allow nested fields in the composite aggregation {pull}37178[#37178] (issue: {issue}28611[#28611])

  • Remove single shard optimization when suggesting shard_size {pull}37041[#37041] (issue: {issue}32125[#32125])

  • Use List instead of priority queue for stable sorting in bucket sort aggregator {pull}36748[#36748] (issue: {issue}36322[#36322])

  • Keys are compared in BucketSortPipelineAggregation so making key type… {pull}36407[#36407]

Audit
  • Security Audit includes HTTP method for requests {pull}37322[#37322] (issue: {issue}29765[#29765])

  • Add X-Forwarded-For to the logfile audit {pull}36427[#36427]

Authentication
  • Security: propagate auth result to listeners {pull}36900[#36900] (issue: {issue}30794[#30794])

  • Security: reorder realms based on last success {pull}36878[#36878]

  • Deprecation check for Auth realm setting structure {pull}36664[#36664] (issue: {issue}36024[#36024])

Authorization
  • Permission for restricted indices {pull}37577[#37577] (issue: {issue}34454[#34454])

  • Create snapshot role {pull}35820[#35820] (issue: {issue}34454[#34454])

CCR
  • Reduce retention lease sync intervals {pull}40302[#40302]

  • Introduce forget follower API {pull}39718[#39718] (issue: {issue}37165[#37165])

  • Renew retention leases while following {pull}39335[#39335] (issues: {issue}37165[#37165], {issue}38718[#38718])

  • Reduce refresh when lookup term in FollowingEngine {pull}39184[#39184]

  • Integrate retention leases to recovery from remote {pull}38829[#38829] (issue: {issue}37165[#37165])

  • Enable removal of retention leases {pull}38751[#38751] (issue: {issue}37165[#37165])

  • Concurrent file chunk fetching for CCR restore {pull}38495[#38495]

  • Tighten mapping syncing in ccr remote restore {pull}38071[#38071] (issues: {issue}36879[#36879], {issue}37887[#37887])

  • Do not allow put mapping on follower {pull}37675[#37675] (issue: {issue}30086[#30086])

  • Added ccr to xpack usage infrastructure {pull}37256[#37256] (issue: {issue}37221[#37221])

  • [CCR] FollowingEngine should fail with 403 if operation has no seqno assigned {pull}37213[#37213]

  • [CCR] Added auto_follow_exception.timestamp field to auto follow stats {pull}36947[#36947]

CRUD
  • Add Seq# based optimistic concurrency control to UpdateRequest {pull}37872[#37872] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Introduce ssl settings to reindex from remote {pull}37527[#37527] (issues: {issue}29755[#29755], {issue}37287[#37287])

  • Use Sequence number powered OCC for processing updates {pull}37308[#37308] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Document Seq No powered optimistic concurrency control {pull}37284[#37284] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Enable IPv6 URIs in reindex from remote {pull}36874[#36874]

  • Set acking timeout to 0 on dynamic mapping update {pull}31140[#31140] (issues: {issue}30672[#30672], {issue}30844[#30844])

Client
  • Fixed required fields and paths list {pull}39358[#39358]

Cluster Coordination
  • Expose minimum_master_nodes in cluster state {pull}37811[#37811] (issue: {issue}37701[#37701])

Distributed
  • Add BWC for retention leases {pull}39482[#39482] (issue: {issue}37165[#37165])

  • Allow retention lease operations under blocks {pull}39089[#39089] (issues: {issue}34648[#34648], {issue}37165[#37165])

  • Remove retention leases when unfollowing {pull}39088[#39088] (issues: {issue}34648[#34648], {issue}37165[#37165])

  • Introduce retention lease state file {pull}39004[#39004] (issues: {issue}37165[#37165], {issue}38588[#38588], {issue}39032[#39032])

  • Introduce retention lease actions {pull}38756[#38756] (issue: {issue}37165[#37165])

  • Add dedicated retention lease exceptions {pull}38754[#38754] (issue: {issue}37165[#37165])

  • Recover retention leases during peer recovery {pull}38435[#38435] (issue: {issue}37165[#37165])

  • Lift retention lease expiration to index shard {pull}38380[#38380] (issues: {issue}37165[#37165], {issue}37963[#37963], {issue}38070[#38070])

  • Introduce retention lease background sync {pull}38262[#38262] (issue: {issue}37165[#37165])

  • Copy retention leases when trim unsafe commits {pull}37995[#37995] (issue: {issue}37165[#37165])

  • Expose retention leases in shard stats {pull}37991[#37991] (issue: {issue}37165[#37165])

  • Introduce retention leases versioning {pull}37951[#37951] (issue: {issue}37165[#37165])

  • Soft-deletes policy should always fetch latest leases {pull}37940[#37940] (issues: {issue}37165[#37165], {issue}37375[#37375])

  • Sync retention leases on expiration {pull}37902[#37902] (issue: {issue}37165[#37165])

  • Ignore shard started requests when primary term does not match {pull}37899[#37899] (issue: {issue}33888[#33888])

  • Move update and delete by query to use seq# for optimistic concurrency control {pull}37857[#37857] (issues: {issue}10708[#10708], {issue}36148[#36148], {issue}37639[#37639])

  • Introduce retention lease serialization {pull}37447[#37447] (issues: {issue}37165[#37165], {issue}37398[#37398])

  • Add run under primary permit method {pull}37440[#37440] (issue: {issue}37398[#37398])

  • Introduce retention lease syncing {pull}37398[#37398] (issue: {issue}37165[#37165])

  • Introduce retention lease persistence {pull}37375[#37375] (issue: {issue}37165[#37165])

  • Add validation for retention lease construction {pull}37312[#37312] (issue: {issue}37165[#37165])

  • Introduce retention lease expiration {pull}37195[#37195] (issue: {issue}37165[#37165])

  • Introduce shard history retention leases {pull}37167[#37167] (issue: {issue}37165[#37165])

  • Always initialize the global checkpoint {pull}34381[#34381]

Docs Infrastructure
  • Align generated release notes with doc standards {pull}39234[#39234] (issue: {issue}39155[#39155])

Engine
  • Also mmap cfs files for hybridfs {pull}38940[#38940] (issue: {issue}36668[#36668])

  • Specialize pre-closing checks for engine implementations {pull}38702[#38702]

  • Ensure that max seq # is equal to the global checkpoint when creating ReadOnlyEngines {pull}37426[#37426]

  • Enable Bulk-Merge if all source remains {pull}37269[#37269]

  • Introduce time-based retention policy for soft-deletes {pull}34943[#34943] (issue: {issue}34908[#34908])

Features/CAT APIs
  • Expose search.throttled on _cat/indices {pull}37073[#37073] (issue: {issue}34352[#34352])

Features/Features
  • Deprecation check for No Master Block setting {pull}38383[#38383] (issue: {issue}36024[#36024])

  • Run Node deprecation checks locally {pull}38065[#38065] (issue: {issue}37845[#37845])

  • Watcher notification settings Upgrade checks {pull}36907[#36907]

Features/ILM
  • Ensure ILM policies run safely on leader indices {pull}38140[#38140] (issue: {issue}34648[#34648])

  • Skip Shrink when numberOfShards not changed {pull}37953[#37953] (issue: {issue}33275[#33275])

  • Inject Unfollow before Rollover and Shrink {pull}37625[#37625] (issue: {issue}34648[#34648])

  • Add set_priority action to ILM {pull}37397[#37397] (issue: {issue}36905[#36905])

  • [ILM] Add Freeze Action {pull}36910[#36910] (issue: {issue}34630[#34630])

Features/Indices APIs
  • New mapping signature and mapping string source fixed. {pull}37401[#37401]

Features/Ingest
  • minor updates for user-agent ecs for 6.7 {pull}39213[#39213] (issue: {issue}38757[#38757])

  • Dep. check for ECS changes to User Agent processor {pull}38362[#38362] (issue: {issue}36024[#36024])

  • Add ECS schema for user-agent ingest processor {pull}37727[#37727] (issue: {issue}37329[#37329])

  • ingest: compile mustache template only if field includes '{{'' {pull}37207[#37207] (issue: {issue}37120[#37120])

  • Move ingest-geoip default databases out of config {pull}36949[#36949] (issue: {issue}36898[#36898])

Features/Java High Level REST Client
  • HLRC: Fix strict setting exception handling {pull}37247[#37247] (issue: {issue}37090[#37090])

  • HLRC: Use nonblocking entity for requests {pull}32249[#32249]

Features/Watcher
  • Move watcher to use seq# and primary term for concurrency control {pull}37977[#37977] (issues: {issue}10708[#10708], {issue}37872[#37872])

Infra/Core
  • Use DateFormatter in monitoring instead of joda code {pull}38309[#38309]

  • Use dateformatter in ingest-common to log deprecations {pull}38099[#38099]

  • Add simple method to write collection of writeables {pull}37448[#37448] (issue: {issue}37398[#37398])

  • Date/Time parsing: Use java time API instead of exception handling {pull}37222[#37222]

  • [API] spelling: interruptible {pull}37049[#37049] (issue: {issue}37035[#37035])

  • restrict node start-up when cluster name in data path {pull}36519[#36519] (issue: {issue}32661[#32661])

Infra/Logging
  • Trim the JSON source in indexing slow logs {pull}38081[#38081] (issue: {issue}38080[#38080])

  • Optimize warning header de-duplication {pull}37725[#37725] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597], {issue}37622[#37622])

  • Remove warn-date from warning headers {pull}37622[#37622] (issues: {issue}35754[#35754], {issue}37530[#37530], {issue}37597[#37597])

  • Add some deprecation optimizations {pull}37597[#37597] (issues: {issue}35754[#35754], {issue}37530[#37530])

  • Only update response headers if we have a new one {pull}37590[#37590] (issues: {issue}35754[#35754], {issue}37530[#37530])

Infra/Packaging
  • Upgrade bundled JDK and Docker images to JDK 12 {pull}40229[#40229]

  • Change file descriptor limit to 65535 {pull}37537[#37537] (issue: {issue}35839[#35839])

  • Exit batch files explictly using ERRORLEVEL {pull}29583[#29583] (issue: {issue}29582[#29582])

Infra/Scripting
  • Add getZone to JodaCompatibleZonedDateTime {pull}37084[#37084]

Infra/Settings
  • Provide a clearer error message on keystore add {pull}39327[#39327] (issue: {issue}39324[#39324])

  • Separate out validation of groups of settings {pull}34184[#34184]

License
  • Handle malformed license signatures {pull}37137[#37137] (issue: {issue}35340[#35340])

Machine Learning
  • Allow stop unassigned datafeed and relax unset upgrade mode wait {pull}39034[#39034]

  • Move ML Optimistic Concurrency Control to Seq No {pull}38278[#38278] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Add upgrade mode docs, hlrc, and fix bug {pull}37942[#37942]

  • Tighten up use of aliases rather than concrete indices {pull}37874[#37874]

  • Add support for single bucket aggs in Datafeeds {pull}37544[#37544] (issue: {issue}36838[#36838])

  • Migrate unallocated jobs and datafeeds {pull}37536[#37536] (issue: {issue}32905[#32905])

  • Adjust seccomp filter for Fedora 29. {ml-pull}354[#354]

Mapping
  • Only issue a deprecation warning if include_type_name is not set. {pull}38825[#38825] (issue: {issue}35190[#35190])

  • Log document id when MapperParsingException occurs {pull}37800[#37800] (issue: {issue}37658[#37658])

  • Types removal - add constants for include_type_names {pull}37304[#37304]

  • Deprecation check for index_options on numeric fields {pull}37026[#37026] (issue: {issue}36024[#36024])

  • Deprecation check for indices with multiple types {pull}36952[#36952] (issues: {issue}35190[#35190], {issue}36024[#36024])

  • Use index-prefix fields for terms of length min_chars - 1 {pull}36703[#36703]

Recovery
  • Do not wait for advancement of checkpoint in recovery {pull}39006[#39006] (issues: {issue}38949[#38949], {issue}39000[#39000])

  • SyncedFlushService.getShardRoutingTable() should use metadata to check for index existence {pull}37691[#37691] (issue: {issue}33888[#33888])

  • Make prepare engine step of recovery source non-blocking {pull}37573[#37573] (issue: {issue}37174[#37174])

  • Make recovery source send operations non-blocking {pull}37503[#37503] (issue: {issue}37458[#37458])

  • Prepare to make send translog of recovery non-blocking {pull}37458[#37458] (issue: {issue}37291[#37291])

  • Make finalize step of recovery source non-blocking {pull}37388[#37388] (issue: {issue}37291[#37291])

  • Make recovery source partially non-blocking {pull}37291[#37291] (issue: {issue}36195[#36195])

  • Do not mutate RecoveryResponse {pull}37204[#37204] (issue: {issue}37174[#37174])

  • Don’t block on peer recovery on the target side {pull}37076[#37076] (issue: {issue}36195[#36195])

  • Reduce recovery time with compress or secure transport {pull}36981[#36981] (issue: {issue}33844[#33844])

Rollup
  • Replace the TreeMap in the composite aggregation {pull}36675[#36675]

SQL
  • SQL: Enhance checks for inexact fields {pull}39427[#39427] (issue: {issue}38501[#38501])

  • SQL: change the default precision for CURRENT_TIMESTAMP function {pull}39391[#39391] (issue: {issue}39288[#39288])

  • SQL: add "validate.properties" property to JDBC’s allowed list of settings {pull}39050[#39050] (issue: {issue}38068[#38068])

  • SQL: Allow look-ahead resolution of aliases for WHERE clause {pull}38450[#38450] (issue: {issue}29983[#29983])

  • SQL: Implement CURRENT_DATE {pull}38175[#38175] (issue: {issue}38160[#38160])

  • SQL: Generate relevant error message when grouping functions are not used in GROUP BY {pull}38017[#38017] (issue: {issue}37952[#37952])

  • SQL: Skip the nested and object field types in case of an ODBC request {pull}37948[#37948] (issue: {issue}37801[#37801])

  • SQL: Add protocol tests and remove jdbc_type from drivers response {pull}37516[#37516] (issues: {issue}36635[#36635], {issue}36882[#36882])

  • SQL: Remove slightly used meta commands {pull}37506[#37506] (issue: {issue}37409[#37409])

  • SQL: Describe aliases as views {pull}37496[#37496] (issue: {issue}37422[#37422])

  • SQL: Make FULL non-reserved keyword in the grammar {pull}37377[#37377] (issue: {issue}37376[#37376])

  • SQL: Use declared source for error messages {pull}37161[#37161]

  • SQL: Improve error message when unable to translate to ES query DSL {pull}37129[#37129] (issue: {issue}37040[#37040])

  • [API] spelling: subtract {pull}37055[#37055] (issue: {issue}37035[#37035])

  • [API] spelling: similar {pull}37054[#37054] (issue: {issue}37035[#37035])

  • [API] spelling: input {pull}37048[#37048] (issue: {issue}37035[#37035])

  • SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg {pull}36933[#36933] (issue: {issue}36903[#36903])

  • SQL: Preserve original source for each expression {pull}36912[#36912] (issue: {issue}36894[#36894])

Search
  • Add finalReduce flag to SearchRequest {pull}38104[#38104] (issues: {issue}37000[#37000], {issue}37838[#37838])

  • Expose sequence number and primary terms in search responses {pull}37639[#37639]

  • Allow field types to optimize phrase prefix queries {pull}37436[#37436] (issue: {issue}31921[#31921])

  • Add support for providing absolute start time to SearchRequest {pull}37142[#37142] (issue: {issue}32125[#32125])

  • Ensure that local cluster alias is never treated as remote {pull}37121[#37121] (issues: {issue}32125[#32125], {issue}36997[#36997])

  • [API] spelling: cacheable {pull}37047[#37047] (issue: {issue}37035[#37035])

  • Add ability to suggest shard_size on coord node rewrite {pull}37017[#37017] (issues: {issue}32125[#32125], {issue}36997[#36997], {issue}37000[#37000])

  • Skip final reduction if SearchRequest holds a cluster alias {pull}37000[#37000] (issues: {issue}32125[#32125], {issue}36997[#36997])

  • Add support for local cluster alias to SearchRequest {pull}36997[#36997] (issue: {issue}32125[#32125])

Security
  • Move CAS operations in TokenService to sequence numbers {pull}38311[#38311] (issues: {issue}10708[#10708], {issue}37872[#37872])

  • Cleanup construction of interceptors {pull}38294[#38294]

Snapshot/Restore
  • RestoreService should update primary terms when restoring shards of existing indices {pull}38177[#38177] (issue: {issue}33888[#33888])

  • Allow open indices to be restored {pull}37733[#37733]

  • Create specific exception for when snapshots are in progress {pull}37550[#37550] (issue: {issue}37541[#37541])

  • SNAPSHOT: Speed up HDFS Repository Writes {pull}37069[#37069]

  • Implement Atomic Blob Writes for HDFS Repository {pull}37066[#37066] (issue: {issue}37011[#37011])

  • [API] spelling: repositories {pull}37053[#37053] (issue: {issue}37035[#37035])

  • SNAPSHOT: Use CancellableThreads to Abort {pull}35901[#35901] (issue: {issue}21759[#21759])

Suggesters
  • [API] spelling: likelihood {pull}37052[#37052] (issue: {issue}37035[#37035])

Bug fixes

Aggregations
  • Skip sibling pipeline aggregators reduction during non-final reduce {pull}40101[#40101] (issue: {issue}40059[#40059])

  • Only create MatrixStatsResults on final reduction {pull}38130[#38130] (issue: {issue}37587[#37587])

  • Don’t load global ordinals with the map execution_hint {pull}37833[#37833] (issue: {issue}37705[#37705])

  • Issue #37303 - Invalid variance fix {pull}37384[#37384] (issue: {issue}37303[#37303])

Allocation
  • Fix _host based require filters {pull}38173[#38173]

  • Ignore obsolete dangling indices {pull}37824[#37824] (issue: {issue}27073[#27073])

  • ALLOC: Fail Stale Primary Alloc. Req. without Data {pull}37226[#37226] (issue: {issue}37098[#37098])

Analysis
  • Fix PreConfiguredTokenFilters getSynonymFilter() implementations {pull}38858[#38858] (issues: {issue}38793[#38793], {issue}38839[#38839])

  • Fix PreConfiguredTokenFilters getSynonymFilter() implementations {pull}38839[#38839] (issue: {issue}38793[#38793])

Audit
  • LoggingAuditTrail correctly handle ReplicatedWriteRequest {pull}39925[#39925] (issue: {issue}39555[#39555])

  • Fix IndexAuditTrail rolling upgrade on rollover edge - take 2 {pull}38286[#38286] (issues: {issue}33867[#33867], {issue}35988[#35988], {issue}37062[#37062])

  • Fix NPE in Logfile Audit Filter {pull}38120[#38120] (issue: {issue}38097[#38097])

Authentication
  • Correct authenticate response for API key {pull}39684[#39684]

  • Fix security index auto-create and state recovery race {pull}39582[#39582]

  • Use consistent view of realms for authentication {pull}38815[#38815] (issue: {issue}30301[#30301])

  • Enhance parsing of StatusCode in SAML Responses {pull}38628[#38628]

  • Limit token expiry to 1 hour maximum {pull}38244[#38244]

  • Fix expired token message in Exception header {pull}37196[#37196]

  • Fix NPE in CachingUsernamePasswordRealm {pull}36953[#36953] (issue: {issue}36951[#36951])

CCR
  • Safe publication of AutoFollowCoordinator {pull}40153[#40153] (issue: {issue}38560[#38560])

  • Enable reading auto-follow patterns from x-content {pull}40130[#40130] (issue: {issue}40128[#40128])

  • Stop auto-followers on shutdown {pull}40124[#40124]

  • Protect against the leader index being removed {pull}39351[#39351] (issue: {issue}39308[#39308])

  • Fix shard follow task startup error handling {pull}39053[#39053] (issue: {issue}38779[#38779])

  • Filter out upgraded version index settings when starting index following {pull}38838[#38838] (issue: {issue}38835[#38835])

  • Handle the fact that ShardStats instance may have no commit or seqno stats {pull}38782[#38782] (issue: {issue}38779[#38779])

  • Fix LocalIndexFollowingIT#testRemoveRemoteConnection() test {pull}38709[#38709] (issue: {issue}38695[#38695])

  • Prevent CCR recovery from missing documents {pull}38237[#38237]

  • Fix file reading in ccr restore service {pull}38117[#38117]

  • Correct argument names in update mapping/settings from leader {pull}38063[#38063]

  • Ensure changes requests return the latest mapping version {pull}37633[#37633]

  • Do not set fatal exception when shard follow task is stopped. {pull}37603[#37603]

  • Add fatal_exception field for ccr stats in monitoring mapping {pull}37563[#37563]

  • Do not add index event listener if CCR disabled {pull}37432[#37432]

  • When removing an AutoFollower also mark it as removed. {pull}37402[#37402] (issue: {issue}36761[#36761])

  • [CCR] Resume follow Api should not require a request body {pull}37217[#37217] (issue: {issue}37022[#37022])

CRUD
  • Cascading primary failure lead to MSU too low {pull}40249[#40249]

  • ShardBulkAction ignore primary response on primary {pull}38901[#38901]

  • Fix Reindex from remote query logic {pull}36908[#36908]

Cluster Coordination
  • Fixing the custom object serialization bug in diffable utils. {pull}39544[#39544]

  • Always return metadata version if metadata is requested {pull}37674[#37674]

Distributed
  • Enforce retention leases require soft deletes {pull}39922[#39922] (issue: {issue}39914[#39914])

  • Treat TransportService stopped error as node is closing {pull}39800[#39800] (issue: {issue}39584[#39584])

  • Use cause to determine if node with primary is closing {pull}39723[#39723] (issue: {issue}39584[#39584])

  • Don’t ack if unable to remove failing replica {pull}39584[#39584] (issue: {issue}39467[#39467])

  • Ignore waitForActiveShards when syncing leases {pull}39224[#39224] (issue: {issue}39089[#39089])

  • Fix synchronization in LocalCheckpointTracker#contains {pull}38755[#38755] (issues: {issue}33871[#33871], {issue}38633[#38633])

  • TransportVerifyShardBeforeCloseAction should force a flush {pull}38401[#38401] (issues: {issue}33888[#33888], {issue}37961[#37961])

  • Fix limit on retaining sequence number {pull}37992[#37992] (issue: {issue}37165[#37165])

  • Close Index API should force a flush if a sync is needed {pull}37961[#37961] (issues: {issue}33888[#33888], {issue}37426[#37426])

  • Force Refresh Listeners when Acquiring all Operation Permits {pull}36835[#36835]

  • Replaced the word 'shards' with 'replicas' in an error message. (#36234) {pull}36275[#36275] (issue: {issue}36234[#36234])

Engine
  • Bubble up exception when processing NoOp {pull}39338[#39338] (issue: {issue}38898[#38898])

  • ReadOnlyEngine should update translog recovery state information {pull}39238[#39238]

  • Advance max_seq_no before add operation to Lucene {pull}38879[#38879] (issue: {issue}31629[#31629])

Features/Features
  • Only count some fields types for deprecation check {pull}40166[#40166]

  • Deprecation check for indices with very large numbers of fields {pull}39869[#39869] (issue: {issue}39851[#39851])

  • Check for .watches that wasn’t upgraded properly {pull}39609[#39609]

  • Link to 7.0 documentation in deprecation checks {pull}39194[#39194]

  • Handle Null in FetchSourceContext#fetchSource {pull}36839[#36839] (issue: {issue}29293[#29293])

Features/ILM
  • Handle failure to release retention leases in ILM {pull}39281[#39281] (issue: {issue}39181[#39181])

  • Preserve ILM operation mode when creating new lifecycles {pull}38134[#38134] (issues: {issue}38229[#38229], {issue}38230[#38230])

  • Retry ILM steps that fail due to SnapshotInProgressException {pull}37624[#37624] (issues: {issue}37541[#37541], {issue}37552[#37552])

  • Remove indexing_complete when removing policy {pull}36620[#36620]

Features/Indices APIs
  • Add pre-upgrade check to test cluster routing allocation is enabled {pull}39340[#39340] (issue: {issue}39339[#39339])

  • Reject delete index requests with a body {pull}37501[#37501] (issue: {issue}8217[#8217])

  • Get Aliases with wildcard exclusion expression {pull}34230[#34230] (issues: {issue}33518[#33518], {issue}33805[#33805], {issue}34144[#34144])

Features/Ingest
  • Ingest ingest then create index {pull}39607[#39607] (issues: {issue}32758[#32758], {issue}32786[#32786], {issue}36545[#36545])

  • Support unknown fields in ingest pipeline map configuration {pull}38352[#38352] (issue: {issue}36938[#36938])

  • Ingest node - user_agent, move device parsing to an object {pull}38115[#38115] (issues: {issue}37329[#37329], {issue}38094[#38094])

Features/Java High Level REST Client
  • Allow setting of copy_settings in the HLRC {pull}39752[#39752] (issue: {issue}30255[#30255])

  • Update IndexTemplateMetaData to allow unknown fields {pull}38448[#38448] (issue: {issue}36938[#36938])

  • if_seq_no and if_primary_term parameters aren’t wired correctly in REST Client’s CRUD API {pull}38411[#38411]

  • Update Rollup Caps to allow unknown fields {pull}38339[#38339] (issue: {issue}36938[#36938])

  • Fix ILM explain response to allow unknown fields {pull}38054[#38054] (issue: {issue}36938[#36938])

  • Fix ILM status to allow unknown fields {pull}38043[#38043] (issue: {issue}36938[#36938])

  • Fix ILM Lifecycle Policy to allow unknown fields {pull}38041[#38041] (issue: {issue}36938[#36938])

  • Update authenticate to allow unknown fields {pull}37713[#37713] (issue: {issue}36938[#36938])

  • Update verify repository to allow unknown fields {pull}37619[#37619] (issue: {issue}36938[#36938])

  • Update get users to allow unknown fields {pull}37593[#37593] (issue: {issue}36938[#36938])

  • Update Execute Watch to allow unknown fields {pull}37498[#37498] (issue: {issue}36938[#36938])

  • Update Put Watch to allow unknown fields {pull}37494[#37494] (issue: {issue}36938[#36938])

  • Update Delete Watch to allow unknown fields {pull}37435[#37435] (issue: {issue}36938[#36938])

  • Fix weighted_avg parser not found for RestHighLevelClient {pull}37027[#37027] (issue: {issue}36861[#36861])

Features/Monitoring
  • Specify include_type_name in HTTP monitoring. {pull}38927[#38927] (issue: {issue}37442[#37442])

  • Allow built-in monitoring_user role to call GET _xpack API {pull}38060[#38060] (issue: {issue}37970[#37970])

Features/Watcher
  • Fix Watcher stats class cast exception {pull}39821[#39821] (issue: {issue}39780[#39780])

  • Use any index specified by .watches for Watcher {pull}39541[#39541] (issue: {issue}39478[#39478])

  • Resolve concurrency with watcher trigger service {pull}39092[#39092] (issue: {issue}39087[#39087])

  • Only flush Watcher’s bulk processor if Watcher is enabled {pull}38803[#38803] (issue: {issue}38798[#38798])

Geo
  • Geo: Do not normalize the longitude with value -180 for Lucene shapes {pull}37299[#37299] (issue: {issue}37297[#37297])

Highlighting
  • Bug fix for AnnotatedTextHighlighter {pull}39525[#39525] (issue: {issue}39395[#39395])

Infra/Core
  • Correct name of basic_date_time_no_millis {pull}39367[#39367]

  • Fix DateFormatters.parseMillis when no timezone is given {pull}39100[#39100] (issue: {issue}39067[#39067])

  • Prefix java formatter patterns with '8' {pull}38712[#38712] (issue: {issue}38567[#38567])

  • Bubble-up exceptions from scheduler {pull}38317[#38317] (issue: {issue}38014[#38014])

  • Core: Revert back to joda’s multi date formatters {pull}36814[#36814] (issues: {issue}36447[#36447], {issue}36602[#36602])

  • Propagate Errors in executors to uncaught exception handler {pull}36137[#36137] (issue: {issue}28667[#28667])

Infra/Packaging
  • Remove NOREPLACE for /etc/elasticsearch in rpm and deb {pull}37839[#37839]

  • Packaging: Remove permission editing in postinst {pull}37242[#37242] (issue: {issue}37143[#37143])

  • Suppress error message when /proc/sys/vm/max_map_count is not exists. {pull}35933[#35933]

Infra/Scripting
  • Fix Painless void return bug {pull}38046[#38046]

Infra/Settings
  • Fix setting by time unit {pull}37192[#37192]

  • Fix handling of fractional byte size value settings {pull}37172[#37172]

  • Fix handling of fractional time value settings {pull}37171[#37171]

Machine Learning
  • Fix race condition when creating multiple jobs {pull}40049[#40049] (issue: {issue}38785[#38785])

  • Fix datafeed skipping first bucket after lookback when aggs are used {pull}39859[#39859] (issue: {issue}39842[#39842])

  • Refactoring lazy query and agg parsing {pull}39776[#39776] (issue: {issue}39528[#39528])

  • Allow aliased .ml-anomalies* index on PUT Job {pull}38821[#38821] (issue: {issue}38773[#38773])

  • Report index unavailable instead of waiting for lazy node {pull}38423[#38423]

  • Prevent submit after autodetect worker is stopped {pull}37700[#37700] (issue: {issue}37108[#37108])

  • Fix ML datafeed CCS with wildcarded cluster name {pull}37470[#37470] (issue: {issue}36228[#36228])

  • Update error message for process update {pull}37363[#37363]

  • Make GetJobStats work with arbitrary wildcards and groups {pull}36683[#36683] (issue: {issue}34745[#34745])

Mapping
  • Make sure to reject mappings with type _doc when include_type_name is false. {pull}38270[#38270] (issue: {issue}38266[#38266])

  • Treat put-mapping calls with _doc as a top-level key as typed calls. {pull}38032[#38032]

  • Update the deprecation message for typed put mapping requests. {pull}37835[#37835]

  • Make sure PutMappingRequest accepts content types other than JSON. {pull}37720[#37720]

  • MAPPING: Improve Precision for scaled_float {pull}37169[#37169] (issue: {issue}32570[#32570])

  • Make sure to accept empty unnested mappings in create index requests. {pull}37089[#37089]

  • Stop automatically nesting mappings in index creation requests. {pull}36924[#36924]

Network
  • Rebuild remote connections on profile changes {pull}37678[#37678] (issue: {issue}37201[#37201])

  • Reload SSL context on file change for LDAP {pull}36937[#36937] (issues: {issue}30509[#30509], {issue}36923[#36923])

Ranking
  • QueryRescorer should keep the window size when rewriting {pull}36836[#36836]

Recovery
  • Create retention leases file during recovery {pull}39359[#39359] (issue: {issue}37165[#37165])

  • RecoveryMonitor#lastSeenAccessTime should be volatile {pull}36781[#36781]

SQL
  • SQL: Preserve original source for cast/convert function {pull}40271[#40271] (issue: {issue}40239[#40239])

  • SQL: Fix issue with optimization on queries with ORDER BY/LIMIT {pull}40256[#40256] (issue: {issue}40211[#40211])

  • SQL: Fix issue with getting DATE type in JDBC {pull}40207[#40207]

  • SQL: Fix issue with date columns returned always in UTC {pull}40163[#40163] (issue: {issue}40152[#40152])

  • SQL: Add multi_value_field_leniency inside FieldHitExtractor {pull}40113[#40113] (issue: {issue}39700[#39700])

  • SQL: fix incorrect ordering of groupings (GROUP BY) based on orderings (ORDER BY) {pull}40087[#40087] (issue: {issue}39956[#39956])

  • SQL: Fix bug with JDBC timezone setting and DATE type {pull}39978[#39978] (issue: {issue}39915[#39915])

  • SQL: Wrap ZonedDateTime parameters inside scripts {pull}39911[#39911] (issue: {issue}39877[#39877])

  • SQL: ConstantProcessor can now handle NamedWriteable {pull}39876[#39876] (issue: {issue}39875[#39875])

  • SQL: Extend the multi dot field notation extraction to lists of values {pull}39823[#39823] (issue: {issue}39738[#39738])

  • SQL: values in datetime script aggs should be treated as long {pull}39773[#39773] (issue: {issue}37042[#37042])

  • SQL: Don’t allow inexact fields for MIN/MAX {pull}39563[#39563] (issue: {issue}39427[#39427])

  • SQL: Fix merging of incompatible multi-fields {pull}39560[#39560] (issue: {issue}39547[#39547])

  • SQL: fix COUNT DISTINCT column name {pull}39537[#39537] (issue: {issue}39511[#39511])

  • SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' {pull}39518[#39518] (issue: {issue}39471[#39471])

  • SQL: Use underlying exact field for LIKE/RLIKE {pull}39443[#39443] (issue: {issue}39442[#39442])

  • SQL: enforce JDBC driver - ES server version parity {pull}38972[#38972] (issue: {issue}38775[#38775])

  • SQL: fall back to using the field name for column label {pull}38842[#38842] (issue: {issue}38831[#38831])

  • SQL: Prevent grouping over grouping functions {pull}38649[#38649] (issue: {issue}38308[#38308])

  • SQL: Relax StackOverflow circuit breaker for constants {pull}38572[#38572] (issue: {issue}38571[#38571])

  • SQL: Fix issue with IN not resolving to underlying keyword field {pull}38440[#38440] (issue: {issue}38424[#38424])

  • SQL: change the Intervals milliseconds precision to 3 digits {pull}38297[#38297] (issue: {issue}37423[#37423])

  • SQL: Fix esType for DATETIME/DATE and INTERVALS {pull}38179[#38179] (issue: {issue}38051[#38051])

  • SQL: Added SSL configuration options tests {pull}37875[#37875] (issue: {issue}37711[#37711])

  • SQL: Fix casting from date to numeric type to use millis {pull}37869[#37869] (issue: {issue}37655[#37655])

  • SQL: Fix BasicFormatter NPE {pull}37804[#37804]

  • SQL: Return Intervals in SQL format for CLI {pull}37602[#37602] (issues: {issue}29970[#29970], {issue}36186[#36186], {issue}36432[#36432])

  • SQL: fix object extraction from sources {pull}37502[#37502] (issue: {issue}37364[#37364])

  • SQL: Fix issue with field names containing "." {pull}37364[#37364] (issue: {issue}37128[#37128])

  • SQL: Fix bug regarding alias fields with dots {pull}37279[#37279] (issue: {issue}37224[#37224])

  • SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) {pull}37254[#37254] (issue: {issue}30285[#30285])

  • SQL: fix COUNT DISTINCT filtering {pull}37176[#37176] (issue: {issue}37086[#37086])

  • SQL: Fix issue with wrong NULL optimization {pull}37124[#37124] (issue: {issue}35872[#35872])

  • SQL: Fix issue with complex expression as args of PERCENTILE/_RANK {pull}37102[#37102] (issue: {issue}37099[#37099])

  • SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless {pull}37024[#37024] (issue: {issue}37023[#37023])

  • SQL: Fix bug regarding histograms usage in scripting {pull}36866[#36866]

  • SQL: Fix issue with always false filter involving functions {pull}36830[#36830] (issue: {issue}35980[#35980])

  • SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {pull}36800[#36800] (issue: {issue}36756[#36756])

  • SQL: Enhance Verifier to prevent aggregate or grouping functions from {pull}36799[#36799] (issue: {issue}36798[#36798])

  • SQL: normalized keywords shouldn’t be allowed for groupings and sorting [ISSUE] {pull}35203[#35203]

Search
  • Serialize top-level pipeline aggs as part of InternalAggregations {pull}40177[#40177] (issues: {issue}40059[#40059], {issue}40101[#40101])

  • Fix Fuzziness#asDistance(String) {pull}39643[#39643] (issue: {issue}39614[#39614])

  • Fix simple query string serialization conditional {pull}38960[#38960] (issues: {issue}21504[#21504], {issue}38889[#38889])

  • Ensure that maxConcurrentShardRequests is never defaulted to 0 {pull}38734[#38734]

  • Look up connection using the right cluster alias when releasing contexts {pull}38570[#38570]

  • Fix fetch source option in expand search phase {pull}37908[#37908] (issue: {issue}23829[#23829])

  • Throw if two inner_hits have the same name {pull}37645[#37645] (issue: {issue}37584[#37584])

  • Ensure either success or failure path for SearchOperationListener is called {pull}37467[#37467] (issue: {issue}37185[#37185])

  • Use executor SAME to handle search related handlers {pull}37427[#37427] (issues: {issue}33732[#33732], {issue}37392[#37392])

Security
  • Fix exit code for Security CLI tools {pull}37956[#37956] (issue: {issue}37841[#37841])

  • Fix potential NPE in UsersTool {pull}37660[#37660]

Snapshot/Restore
  • Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization {pull}38368[#38368] (issue: {issue}38226[#38226])

  • Fix Two Races that Lead to Stuck Snapshots {pull}37686[#37686] (issues: {issue}32265[#32265], {issue}32348[#32348])

  • Fix Race in Concurrent Snapshot Delete and Create {pull}37612[#37612] (issue: {issue}37581[#37581])

  • Streamline S3 Repository- and Client-Settings {pull}37393[#37393]

  • SNAPSHOTS: Upgrade GCS Dependencies to 1.55.0 {pull}36634[#36634] (issues: {issue}35229[#35229], {issue}35459[#35459])

Suggesters
  • Fix duplicate removal when merging completion suggestions {pull}36996[#36996] (issue: {issue}35836[#35836])

Task Management
  • Un-assign persistent tasks as nodes exit the cluster {pull}37656[#37656]

Regressions

Infra/Core
  • Speed up converting of temporal accessor to zoned date time {pull}37915[#37915] (issue: {issue}37826[#37826])

Upgrades

Discovery-Plugins
  • Bump jackson-databind version for AWS SDK {pull}39183[#39183]

Features/Ingest
  • Bump jackson-databind version for ingest-geoip {pull}39182[#39182]

Security
  • Upgrade the bouncycastle dependency to 1.61 {pull}40017[#40017] (issue: {issue}40011[#40011])

Snapshot/Restore
  • plugins/repository-gcs: Update google-cloud-storage/core to 1.59.0 {pull}39748[#39748] (issue: {issue}39366[#39366])

{es} version 6.6.2

Breaking changes

Authentication
  • Disable BWC mode in TokenService by default {pull}38881[#38881]

Enhancements

SQL
  • Enhance checks for inexact fields {pull}39427[#39427] (issue: {issue}38501[#38501])

  • Add "validate.properties" property to JDBC’s allowed list of settings {pull}39050[#39050] (issue: {issue}38068[#38068])

Bug fixes

Authentication
  • Use consistent view of realms for authentication {pull}38815[#38815] (issue: {issue}30301[#30301])

Engine
  • Bubble up exception when processing NoOp {pull}39338[#39338] (issue: {issue}38898[#38898])

  • Advance max_seq_no before add operation to Lucene {pull}38879[#38879] (issue: {issue}31629[#31629])

Machine Learning
  • Stop the ML memory tracker before closing nodes {pull}39111[#39111] (issue: {issue}37117[#37117])

  • Fix an issue where interim results would be calculated after advancing time into an empty bucket {ml-pull}416[#416] (issue: {ml-issue}324[#324])

SQL
  • Fix merging of incompatible multi-fields {pull}39560[#39560] (issue: {issue}39547[#39547])

  • Ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' {pull}39518[#39518] (issue: {issue}39471[#39471])

  • Use underlying exact field for LIKE/RLIKE {pull}39443[#39443] (issue: {issue}39442[#39442])

  • Enforce JDBC driver - ES server version parity {pull}38972[#38972] (issue: {issue}38775[#38775])

  • Fall back to using the field name for column label {pull}38842[#38842] (issue: {issue}38831[#38831])

  • Normalized keywords shouldn’t be allowed for groupings and sorting [ISSUE] {pull}35203[#35203]

Search
  • Fix Fuzziness#asDistance(String) {pull}39643[#39643] (issue: {issue}39614[#39614])

  • Fix simple query string serialization conditional {pull}38960[#38960] (issues: {issue}21504[#21504], {issue}38889[#38889])

Watcher
  • Only flush Watcher’s bulk processor if Watcher is enabled {pull}38803[#38803] (issue: {issue}38798[#38798])

{es} version 6.6.1

Enhancements

Machine Learning
  • Add explanation so far to file structure finder exceptions {pull}38191[#38191]

SQL
  • Generate relevant error message when grouping functions are not used in GROUP BY {pull}38017[#38017] (issue: {issue}37952[#37952])

  • Skip the nested and object field types in case of an ODBC request {pull}37948[#37948] (issue: {issue}37801[#37801])

  • Describe aliases as views {pull}37496[#37496] (issue: {issue}37422[#37422])

Security
  • Cleanup construction of interceptors {pull}38294[#38294]

Bug fixes

Audit
  • Fix IndexAuditTrail rolling upgrade on rollover edge - take 2 {pull}38286[#38286] (issues: {issue}33867[#33867], {issue}37062[#37062])

  • Fix NPE in Logfile Audit Filter {pull}38120[#38120] (issue: {issue}38097[#38097])

Authentication
  • Enhance parsing of StatusCode in SAML Responses {pull}38628[#38628]

  • Fix expired token message in Exception header {pull}37196[#37196]

CCR
  • Do not set fatal exception when shard follow task is stopped. {pull}37603[#37603]

Distributed
  • Fix synchronization in LocalCheckpointTracker#contains {pull}38755[#38755]

Geo
  • Fix GeoHash PrefixTree BWC {pull}38584[#38584] (issue: {issue}38494[#38494])

  • Geo: Do not normalize the longitude with value -180 for Lucene shapes {pull}37299[#37299] (issue: {issue}37297[#37297])

ILM
  • Preserve ILM operation mode when creating new lifecycles {pull}38134[#38134], {pull}38230[#38230])

Machine Learning
  • Fix error race condition on stop _all datafeeds and close _all jobs {pull}38113[#38113] (issue: {issue}37959[#37959])

  • Update ML results mappings on process start {pull}37706[#37706] (issue: {issue}37607[#37607])

Monitoring
  • Allow built-in monitoring_user role to call GET _xpack API {pull}38060[#38060] (issue: {issue}37970[#37970])

Network
  • Reload SSL context on file change for LDAP {pull}36937[#36937] (issues: {issue}30509[#30509], {issue}36923[#36923])

SQL
  • Prevent grouping over grouping functions {pull}38649[#38649] (issue: {issue}38308[#38308])

  • Relax StackOverflow circuit breaker for constants {pull}38572[#38572] (issue: {issue}38571[#38571])

  • Fix issue with IN not resolving to underlying keyword field {pull}38440[#38440] (issue: {issue}38424[#38424])

  • Change the Intervals milliseconds precision to 3 digits {pull}38297[#38297] (issue: {issue}37423[#37423])

  • Added SSL configuration options tests {pull}37875[#37875] (issue: {issue}37711[#37711])

  • Fix casting from date to numeric type to use millis {pull}37869[#37869] (issue: {issue}37655[#37655])

  • Fix object extraction from sources {pull}37502[#37502] (issue: {issue}37364[#37364])

  • Fix bug regarding alias fields with dots {pull}37279[#37279] (issue: {issue}37224[#37224])

  • Fix issue with complex expression as args of PERCENTILE/_RANK {pull}37102[#37102] (issue: {issue}37099[#37099])

Search
  • Use executor SAME to handle search related handlers {pull}37427[#37427] (issues: {issue}33732[#33732], {issue}37392[#37392])

Security
  • Fix an improper permission issue when a new name is attached to an index. (CVE-2019-7611). See Security issues.

  • Fix potential NPE in UsersTool {pull}37660[#37660]

ZenDiscovery
  • Always return metadata version if metadata is requested {pull}37674[#37674]

{es} version 6.6.0

Breaking changes

Cross-cluster replication
  • Change get autofollow patterns API response format {pull}36203[#36203] (issue: {issue}36049[#36049])

Ranking
  • Forbid negative scores in function_score query {pull}35709[#35709] (issue: {issue}33309[#33309])

Search
  • Remove the distinction between query and filter context in QueryBuilders {pull}35354[#35354] (issue: {issue}35293[#35293])

  • Throw a parsing exception when boost is set in span_or query (#28390) {pull}34112[#34112] (issue: {issue}28390[#28390])

Watcher
  • Fix metric stats names {pull}34951[#34951] (issue: {issue}34865[#34865])

Breaking Java changes

Infra/Core
  • Drop settings member from AbstractComponent {pull}35083[#35083]

Infra/Logging
  • Drop last deprecated logger function {pull}35082[#35082] (issue: {issue}32174[#32174])

Java High Level REST Client
  • HLRC XPack Protocol clean up: Migration; Graph; Watcher {pull}34639[#34639] (issue: {issue}34451[#34451])

ZenDiscovery
  • Make node field in JoinRequest private {pull}36405[#36405]

Deprecations

Aggregations
  • Deprecate dots in aggregation names {pull}31468[#31468] (issues: {issue}17600[#17600], {issue}19040[#19040])

Core
  • Deprecate use of scientific notation in epoch time parsing {pull}36691[#36691]

  • Add backcompat for joda time formats {pull}36531[#36531]

Features
  • Deprecation info API: 'fix' value for index.shard.check_on_startup {pull}36458[#36458] (issues: {issue}33194[#33194], {issue}36024[#36024])

  • Deprecation info API: negative index.unassigned.node_left.delayed_timeout {pull}36454[#36454] (issues: {issue}26828[#26828], {issue}36024[#36024])

Mapping
  • Add warning about upcoming expanded fields limit {pull}34906[#34906] (issue: {issue}26541[#26541])

Scripting
  • Adds deprecation logging to ScriptDocValues#getValues. {pull}34279[#34279] (issue: {issue}22919[#22919])

Search
  • Warn in multi-search on unknown keys in meatdata {pull}36104[#36104] (issue: {issue}35938[#35938])

  • Deprecate the _termvector endpoint. {pull}36098[#36098] (issue: {issue}8484[#8484])

  • Deprecate _source_include and _source_exclude url parameters {pull}33475[#33475] (issue: {issue}22792[#22792])

Security
  • Undeprecate /_license endpoints {pull}35974[#35974] (issue: {issue}35959[#35959])

Settings
  • Deprecate setting index.optimize_auto_generated_id in 6.x {pull}28862[#28862] (issue: {issue}27600[#27600])

Watcher
  • Watcher deprecate notification service settings {pull}36403[#36403]

  • Undeprecate /_watcher endpoints {pull}36269[#36269]

New features

Aggregations
  • Median absolute deviation agg {pull}34482[#34482] (issue: {issue}26681[#26681])

Analysis
  • Add support for inlined user dictionary in Nori {pull}36123[#36123] (issue: {issue}35842[#35842])

  • Add a prebuilt ICU Analyzer {pull}34958[#34958] (issue: {issue}34285[#34285])

Index Lifecycle Management
  • Adds Index lifecycle feature {pull}35193[#35193]

Java High Level REST Client
  • Add rollup search {pull}36334[#36334] (issue: {issue}29827[#29827])

Java Low Level REST Client
  • Make warning behavior pluggable per request {pull}36345[#36345]

  • Add PreferHasAttributeNodeSelector {pull}36005[#36005]

Geo
  • Fork Lucene’s LatLonShape Classes to local lucene package {pull}36794[#36794]

  • Integrate Lucene’s LatLonShape (BKD Backed GeoShapes) as default geo_shape indexing approach {pull}35320[#35320] (issue: {issue}32039[#32039])

Machine learning
  • Store job configuration information in the new .ml-config index {pull}36698[#36698] (issue: {issue}32905[#32905])

  • Determine when data is missing from a bucket due to ingest latency {pull}35387[#35387] (issue: {issue}35131[#35131])

Search
  • Added soft limit to open scroll contexts #25244 {pull}36009[#36009] (issue: {issue}25244[#25244])

  • Make lucene’s IntervalQuery available via the Query DSL {pull}32406[#32406] (issue: {issue}29636[#29636])

SQL
  • Introduce HISTOGRAM grouping function {pull}36510[#36510] (issue: {issue}36509[#36509])

  • DATABASE() and USER() system functions {pull}35946[#35946] (issue: {issue}35863[#35863])

  • Introduce INTERVAL support {pull}35521[#35521] (issue: {issue}29990[#29990])

Enhancements

Aggregations
  • Enforce max_buckets limit only in the final reduction phase {pull}36152[#36152] (issues: {issue}32125[#32125], {issue}35921[#35921])

  • Histogram aggs: add empty buckets only in the final reduce step {pull}35921[#35921]

  • Handles exists query in composite aggs {pull}35758[#35758]

  • Added parent validation for auto date histogram {pull}35670[#35670]

  • Allow unmapped fields in composite aggregations {pull}35331[#35331] (issue: {issue}35317[#35317])

  • Refactor children aggregator into a generic ParentJoinAggregator {pull}34845[#34845] (issue: {issue}34210[#34210])

  • Add parent-aggregation to parent-join module {pull}34210[#34210] (issue: {issue}9705[#9705])

Analysis
  • Ensure TokenFilters only produce single tokens when parsing synonyms {pull}34331[#34331] (issue: {issue}34298[#34298])

Audit
  • Add "request.id" to file audit logs {pull}35536[#35536]

Authentication
  • Invalidate Token API enhancements - HLRC {pull}36362[#36362]

  • Add DEBUG/TRACE logs for LDAP bind {pull}36028[#36028]

  • Add Tests for findSamlRealm {pull}35905[#35905]

  • Add realm information for Authenticate API {pull}35648[#35648]

  • Formal support for "password_hash" in Put User {pull}35242[#35242] (issue: {issue}34729[#34729])

  • Enhance Invalidate Token API {pull}35388[#35388] (issues: {issue}34556[#34556], {issue}35115[#35115])

Authorization
  • Add origin_address to authentication_success {pull}36409[#36409]

  • Improve exact index matching performance {pull}36017[#36017]

  • manage_token privilege for kibana_system {pull}35751[#35751]

  • Grant .tasks access to kibana_system role {pull}35573[#35573]

  • Native roles store uses mget to retrieve roles {pull}33531[#33531] (issue: {issue}33205[#33205])

Build
  • Sounds like typo in exception message {pull}35458[#35458]

  • Improve validation of yaml suites {pull}34957[#34957] (issue: {issue}34735[#34735])

  • Enforce skip headers when needed {pull}34735[#34735] (issue: {issue}34650[#34650])

  • Improve validation of do sections {pull}34734[#34734] (issue: {issue}34651[#34651])

Core
  • Override the JVM DNS cache policy {pull}36570[#36570]

  • Added wait_for_metadata_version parameter to cluster state api. {pull}35535[#35535]

  • Extract RunOnce into a dedicated class {pull}35489[#35489]

Cross-cluster replication
  • Add time since last auto follow fetch to auto follow stats {pull}36542[#36542] (issues: {issue}33007[#33007], {issue}35895[#35895])

  • Clean followed leader index UUIDs in auto follow metadata {pull}36408[#36408] (issue: {issue}33007[#33007])

  • Change AutofollowCoordinator to use wait_for_metadata_version {pull}36264[#36264] (issues: {issue}33007[#33007], {issue}35895[#35895])

  • Refactor AutoFollowCoordinator to track leader indices per remote cluster {pull}36031[#36031] (issues: {issue}33007[#33007], {issue}35895[#35895])

  • Refactor auto follow coordinator {pull}35895[#35895] (issue: {issue}33007[#33007])

CRUD
  • Document Seq No powered optimistic concurrency control {pull}37284[#37284] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm {pull}36757[#36757] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Expose Sequence Number based Optimistic Concurrency Control in the rest layer {pull}36721[#36721] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Add doc’s sequence number + primary term to GetResult and use it for updates {pull}36680[#36680] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Add seq no powered optimistic locking support to the index and delete transport actions {pull}36619[#36619] (issues: {issue}10708[#10708], {issue}36148[#36148])

Distributed
  • Allow asynchronous block operations to be delayed in IndexShardOperationPermits {pull}35999[#35999] (issues: {issue}35540[#35540], {issue}35850[#35850])

  • TransportResyncReplicationAction should not honour blocks {pull}35795[#35795] (issues: {issue}35332[#35332], {issue}35597[#35597])

  • Add global and index level blocks to IndexSettings {pull}35695[#35695] (issues: {issue}35332[#35332], {issue}35597[#35597])

  • Expose all permits acquisition in IndexShard and TransportReplicationAction {pull}35540[#35540] (issue: {issue}33888[#33888])

  • Add a java level freeze/unfreeze API {pull}35353[#35353] (issues: {issue}34352[#34352], {issue}34357[#34357])

  • Check blocks while having index shard permit in TransportReplicationAction {pull}35332[#35332] (issue: {issue}33888[#33888])

  • Apply masterNodeTimeout to MasterNodeRequest transmission {pull}35235[#35235]

Engine
  • Add sequence numbers based optimistic concurrency control support to Engine {pull}36467[#36467] (issues: {issue}10708[#10708], {issue}36148[#36148])

  • Require soft-deletes when access changes snapshot {pull}36446[#36446]

  • Use delCount of SegmentInfos to calculate numDocs {pull}36323[#36323]

  • Always configure soft-deletes field of IndexWriterConfig {pull}36196[#36196] (issue: {issue}36141[#36141])

  • Always return false from refreshNeeded on ReadOnlyEngine {pull}35837[#35837] (issue: {issue}35785[#35785])

  • Add a _freeze / _unfreeze API {pull}35592[#35592] (issue: {issue}34352[#34352])

  • Engine.newChangesSnapshot may cause unneeded refreshes if called concurrently {pull}35169[#35169]

  • Do not alloc full buffer for small change requests {pull}35158[#35158]

  • Add IndexShardOperationPermits.asyncBlockOperations(ActionListener<Releasable>) {pull}34902[#34902] (issue: {issue}33888[#33888])

  • Add a frozen engine implementation {pull}34357[#34357] (issue: {issue}34352[#34352])

Features
  • Deprecation check for discovery configuration {pull}36666[#36666] (issue: {issue}36024[#36024])

  • Simplify deprecation issue levels {pull}36326[#36326]

  • Deprecation check for tribe node {pull}36240[#36240] (issue: {issue}36024[#36024])

  • Deprecation check for : in Cluster/Index name {pull}36185[#36185] (issue: {issue}36024[#36024])

  • Deprecation check for renamed bulk threadpool settings {pull}36662[#36662] (issue: {issue}36024[#36024])

  • Deprecation check for audit log prefix settings {pull}36661[#36661] (issue: {issue}36024[#36024])

  • Deprecation check for classic similarity {pull}36577[#36577] (issue: {issue}36024[#36024])

  • Deprecation check for HTTP pipelining {pull}36521[#36521] (issue: {issue}36024[#36024])

  • Deprecation check for index threadpool {pull}36520[#36520] (issue: {issue}36024[#36024])

  • Deprecation check for percolator.map_unmapped_fields_as_string {pull}36460[#36460] (issue: {issue}36024[#36024])

  • Deprecation check for http.enabled setting {pull}36394[#36394] (issues: {issue}29601[#29601], {issue}36024[#36024])

  • Deprecation check for File Discovery plugin {pull}36190[#36190] (issue: {issue}36024[#36024])

Geo
  • Adds a name of the field to geopoint parsing errors {pull}36529[#36529] (issue: {issue}15965[#15965])

  • Add support to ShapeBuilders for building Lucene geometry {pull}35707[#35707] (issue: {issue}35320[#35320])

Ingest
  • Make the ingest-geoip databases even lazier to load {pull}36679[#36679]

  • ingest: grok fix duplicate patterns JAVACLASS and JAVAFILE {pull}35886[#35886]

Java High Level REST Client
  • Add HLRC support for pause follow API {pull}35216[#35216] (issue: {issue}33824[#33824])

  • Add security Create Token API {pull}34791[#34791]

  • Add start rollup job support to HL REST Client {pull}34623[#34623] (issue: {issue}29827[#29827])

  • Add security authenticate API {pull}33552[#33552]

  • Add get users action {pull}36332[#36332] (issue: {issue}29827[#29827])

  • Add delete template API {pull}36320[#36320] (issue: {issue}27205[#27205])

  • Implement get-user-privileges API {pull}36292[#36292]

  • Get Deprecation Info API {pull}36279[#36279] (issue: {issue}29827[#29827])

  • Added support for Follow Stats API {pull}36253[#36253] (issue: {issue}33824[#33824])

  • Added support for CCR Stats API {pull}36213[#36213] (issue: {issue}33824[#33824])

  • Put Role {pull}36209[#36209] (issue: {issue}29827[#29827])

  • Add index templates exist API {pull}36132[#36132] (issue: {issue}27205[#27205])

  • Add support for CCR Get Auto Follow Pattern apis {pull}36049[#36049] (issue: {issue}33824[#33824])

  • Add support for CCR Delete Auto Follow Pattern API {pull}35981[#35981] (issue: {issue}33824[#33824])

  • Remove fromXContent from IndexUpgradeInfoResponse {pull}35934[#35934]

  • Add delete expired data API {pull}35906[#35906] (issue: {issue}29827[#29827])

  • Execute watch API {pull}35868[#35868] (issue: {issue}29827[#29827])

  • Add ability to put user with a password hash {pull}35844[#35844] (issue: {issue}35242[#35242])

  • Add ML find file structure API {pull}35833[#35833] (issue: {issue}29827[#29827])

  • Add support for get roles API {pull}35787[#35787] (issue: {issue}29827[#29827])

  • Add support for CCR Put Auto Follow Pattern API {pull}35780[#35780] (issue: {issue}33824[#33824])

  • XPack ML info action {pull}35777[#35777] (issue: {issue}29827[#29827])

  • ML Delete event from Calendar {pull}35760[#35760] (issue: {issue}29827[#29827])

  • Add ML revert model snapshot API {pull}35750[#35750] (issue: {issue}29827[#29827])

  • ML Get Calendar Events {pull}35747[#35747] (issue: {issue}29827[#29827])

  • Add high-level REST client API for _freeze and _unfreeze {pull}35723[#35723] (issue: {issue}34352[#34352])

  • Fix issue in equals impl for GlobalOperationPrivileges {pull}35721[#35721]

  • ML Delete job from calendar {pull}35713[#35713] (issue: {issue}29827[#29827])

  • ML Add Event To Calendar API {pull}35704[#35704] (issue: {issue}29827[#29827])

  • Add ML update model snapshot API (#35537) {pull}35694[#35694] (issue: {issue}29827[#29827])

  • Add support for CCR Unfollow API {pull}35693[#35693] (issue: {issue}33824[#33824])

  • Clean up PutLicenseResponse {pull}35689[#35689] (issue: {issue}35547[#35547])

  • Clean up StartBasicResponse {pull}35688[#35688] (issue: {issue}35547[#35547])

  • Add support for put privileges API {pull}35679[#35679]

  • Add Job to Calendar API {pull}35666[#35666] (issue: {issue}29827[#29827])

  • Add support for CCR Resume Follow API {pull}35638[#35638] (issue: {issue}33824[#33824])

  • Add support for get application privileges API {pull}35556[#35556] (issue: {issue}29827[#29827])

  • Clean up XPackInfoResponse class and related tests {pull}35547[#35547]

  • Add parameters to stopRollupJob API {pull}35545[#35545] (issue: {issue}34811[#34811])

  • Add ML delete model snapshot API {pull}35537[#35537] (issue: {issue}29827[#29827])

  • Add get watch API {pull}35531[#35531] (issue: {issue}29827[#29827])

  • Add ML Update Filter API {pull}35522[#35522] (issue: {issue}29827[#29827])

  • Add ML get filters api {pull}35502[#35502] (issue: {issue}29827[#29827])

  • Add ML get model snapshots API {pull}35487[#35487] (issue: {issue}29827[#29827])

  • Add "_has_privileges" API to Security Client {pull}35479[#35479] (issue: {issue}29827[#29827])

  • Add Delete Privileges API to HLRC {pull}35454[#35454] (issue: {issue}29827[#29827])

  • Add support for CCR Put Follow API {pull}35409[#35409]

  • Add ML delete filter action {pull}35382[#35382] (issue: {issue}29827[#29827])

  • Add delete user action {pull}35294[#35294] (issue: {issue}29827[#29827])

  • HLRC for _mtermvectors {pull}35266[#35266] (issues: {issue}27205[#27205], {issue}33447[#33447])

  • Reindex API with wait_for_completion false {pull}35202[#35202] (issue: {issue}27205[#27205])

  • Add watcher stats API {pull}35185[#35185] (issue: {issue}29827[#29827])

  • Add ML API PUT filter {pull}35175[#35175] (issue: {issue}29827[#29827])

  • HLRC support for getTask {pull}35166[#35166] (issue: {issue}27205[#27205])

  • Add support for the clear realm cache API {pull}35163[#35163] (issue: {issue}29827[#29827])

  • Add InvalidateToken security API {pull}35114[#35114] (issue: {issue}29827[#29827])

  • Add GetRollupIndexCaps API {pull}35102[#35102] (issue: {issue}29827[#29827])

  • Migration api - upgrade {pull}34898[#34898] (issue: {issue}29827[#29827])

  • Add Update datafeed API {pull}34882[#34882] (issue: {issue}29827[#29827])

  • Add stop rollup job support to HL REST Client {pull}34702[#34702] (issue: {issue}29827[#29827])

  • Bulk Api support for global parameters {pull}34528[#34528] (issue: {issue}26026[#26026])

  • Add support for source exists API {pull}34519[#34519] (issue: {issue}27205[#27205])

  • Add document _count API support to Rest High Level Client. {pull}34267[#34267] (issue: {issue}27205[#27205])

  • Add delete rollup job support to HL REST Client {pull}34066[#34066] (issue: {issue}29827[#29827])

  • HLRC API for _termvectors {pull}33447[#33447] (issue: {issue}27205[#27205])

  • Add support for get license basic/trial status API {pull}33176[#33176] (issue: {issue}29827[#29827])

  • Small corrections to HLRC doc for _termvectors {pull}35221[#35221] (issue: {issue}33447[#33447])

Java Low Level REST Client
  • On retry timeout add root exception {pull}25576[#25576]

Machine Learning
  • Create the {ml} annotations index {pull}36731[#36731] (issue: {issue}33376[#33376])

  • Add cluster setting to enable/disable config migration {pull}36700[#36700] (issue: {issue}32905[#32905])

  • Enable the use of endpoints starting with _ml instead of _xpack/ml {pull}36373[#36373] (issue: {issue}36315[#36315])

  • Add audits when deprecation warnings occur while datafeeds start {pull}36233[#36233]

  • Add lazy parsing for DatafeedConfig:Aggs,Query {pull}36117[#36117]

  • Add support for rollup indexes in datafeeds {pull}34654[#34654]

Monitoring
  • Make Exporters Async {pull}35765[#35765] (issue: {issue}35743[#35743])

  • Add cluster metadata to cluster_stats docs (#33860) {pull}34023[#34023] (issues: {issue}33860[#33860], {issue}34040[#34040])

Network
  • Unify transport settings naming {pull}36623[#36623]

  • Move compression config to ConnectionProfile {pull}35357[#35357] (issue: {issue}34483[#34483])

  • Simplify Transport Compression Setting {pull}34959[#34959] (issue: {issue}33844[#33844])

  • Allow to enable pings for specific remote clusters {pull}34753[#34753] (issues: {issue}30247[#30247], {issue}34405[#34405])

Packaging
  • Introduce Docker images build {pull}36246[#36246]

  • Move creation of temporary directory to Java {pull}36002[#36002] (issue: {issue}31003[#31003])

  • Update procrun executables to version 1.1.0 {pull}35147[#35147]

Plugins
  • Plugin install: don’t print download progress in batch mode {pull}36361[#36361]

Recovery
  • Exposed engine must include all operations below global checkpoint during rollback {pull}36159[#36159] (issue: {issue}32867[#32867])

  • Use soft-deleted docs to resolve strategy for engine operation {pull}35230[#35230] (issues: {issue}0[#0], {issue}1[#1], {issue}33656[#33656], {issue}34474[#34474])

  • Put a fake allocation id on allocate stale primary command {pull}34140[#34140] (issue: {issue}33432[#33432])

Rollup
  • Add non-X-Pack centric rollup endpoints {pull}36383[#36383] (issues: {issue}35958[#35958], {issue}35962[#35962])

  • Add more diagnostic stats to job {pull}35471[#35471]

  • Add default fields to job configs {pull}34831[#34831]

  • Add wait_for_completion option to StopRollupJob API {pull}34811[#34811] (issue: {issue}34574[#34574])

Scripting
  • Update joda compat methods to use compat class {pull}36654[#36654]

  • [Painless] Add boxed type to boxed type casts for method/return {pull}36571[#36571]

  • [Painless] Add def to boxed type casts {pull}36506[#36506]

  • [Scripting] Make Max Script Length Setting Dynamic {pull}35184[#35184] (issue: {issue}23209[#23209])

  • [Painless] Add instance bindings {pull}34410[#34410]

Search
  • Use SearchRequest copy constructor in ExpandSearchPhase {pull}36772[#36772] (issue: {issue}36641[#36641])

  • Add copy constructor to SearchRequest {pull}36641[#36641] (issue: {issue}32125[#32125])

  • Add raw sort values to SearchSortValues transport serialization {pull}36617[#36617] (issue: {issue}32125[#32125])

  • Warn when using use_dis_max in multi_match {pull}36614[#36614] (issue: {issue}36488[#36488])

  • Add sort and collapse info to SearchHits transport serialization {pull}36555[#36555] (issue: {issue}32125[#32125])

  • Add default methods to DocValueFormat {pull}36480[#36480]

  • Extend field caps API to mark meta fields {pull}36309[#36309]

  • Respect indices options on _msearch {pull}35887[#35887]

  • Allow efficient can_match phases on frozen indices {pull}35431[#35431] (issues: {issue}34352[#34352], {issue}34357[#34357])

  • Apply ignore_throttled also to concrete indices {pull}35335[#35335] (issue: {issue}34354[#34354])

  • Adapt field limit deprecation warning {pull}35302[#35302] (issue: {issue}35284[#35284])

  • Upgrade 6.x to lucene-7.6.0-snapshot-f9598f335b {pull}35225[#35225]

  • Prevent throttled indices to be searched through wildcards by default {pull}34354[#34354] (issues: {issue}33732[#33732], {issue}34352[#34352])

  • check for null argument is already done in splitStringByCommaToArray {pull}34268[#34268]

  • has_parent builder: exception message/param fix {pull}31182[#31182]

Security
  • Option to use endpoints starting with _security {pull}36379[#36379] (issue: {issue}36293[#36293])

  • Make credentials mandatory when launching xpack/migrate {pull}36197[#36197] (issues: {issue}29847[#29847], {issue}33972[#33972])

  • Make credentials mandatory when launching x-pack/migrate {pull}33972[#33972] (issue: {issue}29847[#29847])

Snapshot/Restore
  • Allow Parallel Restore Operations {pull}36397[#36397]

  • Repo creation out of ClusterStateTask {pull}36157[#36157] (issue: {issue}9488[#9488])

  • Add read-only repository verification {pull}35731[#35731] (issue: {issue}35703[#35703])

SQL
  • Make FULL non-reserved keyword in the grammar {pull}37377[#37377] (issue: {issue}37376[#37376])

  • Extend the ODBC metric by differentiating between 32 and 64bit platforms {pull}36753[#36753] (issue: {issue}36740[#36740])

  • Fix wrong appliance of StackOverflow limit for IN {pull}36724[#36724] (issue: {issue}36592[#36592])

  • Introduce NOW/CURRENT_TIMESTAMP function {pull}36562[#36562] (issue: {issue}36534[#36534])

  • Move requests' parameters to requests JSON body {pull}36149[#36149] (issue: {issue}35992[#35992])

  • Make INTERVAL millis optional {pull}36043[#36043] (issue: {issue}36032[#36032])

  • Implement data type verification for conditionals {pull}35916[#35916] (issue: {issue}35907[#35907])

  • Implement GREATEST and LEAST functions {pull}35879[#35879] (issue: {issue}35878[#35878])

  • Implement null safe equality operator <⇒ {pull}35873[#35873] (issue: {issue}35871[#35871])

  • SYS COLUMNS returns ODBC specific schema {pull}35870[#35870] (issue: {issue}35376[#35376])

  • Polish grammar for intervals {pull}35853[#35853]

  • Add filtering to SYS TYPES {pull}35852[#35852] (issue: {issue}35342[#35342])

  • Implement NULLIF(expr1, expr2) function {pull}35826[#35826] (issue: {issue}35818[#35818])

  • Lock down JDBC driver {pull}35798[#35798] (issue: {issue}35437[#35437])

  • Implement NVL(expr1, expr2) {pull}35794[#35794] (issue: {issue}35782[#35782])

  • Implement ISNULL(expr1, expr2) {pull}35793[#35793] (issue: {issue}35781[#35781])

  • Implement IFNULL variant of COALESCE {pull}35762[#35762] (issue: {issue}35749[#35749])

  • XPack FeatureSet functionality {pull}35725[#35725] (issue: {issue}34821[#34821])

  • Perform lazy evaluation of mismatched mappings {pull}35676[#35676] (issues: {issue}35659[#35659], {issue}35675[#35675])

  • Improve validation of unsupported fields {pull}35675[#35675] (issue: {issue}35673[#35673])

  • Move internals from Joda to java.time {pull}35649[#35649] (issue: {issue}35633[#35633])

  • Improve CircuitBreaker logic for SqlParser {pull}35300[#35300] (issue: {issue}35299[#35299])

  • Upgrade jline to version 3.8.2 {pull}35288[#35288]

  • New SQL CLI logo {pull}35261[#35261]

  • Introduce Coalesce function {pull}35253[#35253] (issue: {issue}35060[#35060])

  • Optimizer rule for folding nullable expressions {pull}35080[#35080] (issue: {issue}34826[#34826])

  • Improve painless script generated from IN {pull}35055[#35055] (issue: {issue}34750[#34750])

  • Implement CAST between STRING and IP {pull}34949[#34949] (issue: {issue}34799[#34799])

  • Fix function args verification and error msgs {pull}34926[#34926] (issues: {issue}33469[#33469], {issue}34752[#34752])

  • Handle X-Pack or X-Pack SQL not being available in a more graceful way {pull}34736[#34736] (issue: {issue}30009[#30009])

Stats
  • Handle OS pretty name on old OS without OS release {pull}35453[#35453] (issue: {issue}35440[#35440])

  • Add more detailed OS name on Linux {pull}35352[#35352]

Task Management
  • Periodically try to reassign unassigned persistent tasks {pull}36069[#36069] (issue: {issue}35792[#35792])

  • Only require task permissions {pull}35667[#35667] (issue: {issue}35573[#35573])

  • Retry if task can’t be written {pull}35054[#35054] (issue: {issue}33764[#33764])

Bug fixes

Aggregations
  • Fix MultiValuesSourceFieldConfig toXContent {pull}36525[#36525] (issue: {issue}36474[#36474])

  • Cache the score of the parent document in the nested agg {pull}36019[#36019] (issues: {issue}34555[#34555], {issue}35985[#35985])

  • Correct implemented interface of ParsedReverseNested {pull}35455[#35455] (issue: {issue}35449[#35449])

  • Handle IndexOrDocValuesQuery in composite aggregation {pull}35392[#35392]

  • Preserve format when aggregation contains unmapped date fields {pull}35254[#35254] (issue: {issue}31760[#31760])

  • Check self references in metric agg after last doc collection (#33593) {pull}34001[#34001]

Audit
  • Fix origin.type for connection_* events {pull}36410[#36410]

  • Fix deprecation of audit log settings {pull}36175[#36175] (issue: {issue}36162[#36162])

  • Fix IndexAuditTrail rolling restart on rollover edge {pull}35988[#35988] (issue: {issue}33867[#33867])

Authentication
  • Fix NPE in CachingUsernamePasswordRealm {pull}36953[#36953] (issue: {issue}36951[#36951])

  • Add support for Kerberos V5 Oid {pull}35764[#35764] (issue: {issue}34763[#34763])

Build
  • Use explicit deps on test tasks for check {pull}36325[#36325]

  • Fix jdbc jar pom to not include deps {pull}36036[#36036] (issue: {issue}32014[#32014])

  • Fix official plugins list {pull}35661[#35661] (issue: {issue}35623[#35623])

Circuit Breakers
  • Modify BigArrays to take name of circuit breaker {pull}36461[#36461] (issue: {issue}31435[#31435])

Core
  • Revert back to joda’s multi date formatters {pull}36814[#36814] (issues: {issue}36447[#36447], {issue}36602[#36602])

  • Fix CompositeBytesReference#slice to not throw AIOOBE with legal offsets. {pull}35955[#35955] (issue: {issue}35950[#35950])

  • Suppress CachedTimeThread in hot threads output {pull}35558[#35558] (issue: {issue}23175[#23175])

  • Upgrade to Joda 2.10.1 {pull}35410[#35410] (issue: {issue}33749[#33749])

  • XContent: Check for bad parsers {pull}34561[#34561] (issue: {issue}34351[#34351])

Cross-cluster replication
  • Add fatal_exception field for ccr stats in monitoring mapping {pull}37563[#37563]

  • When removing an AutoFollower also mark it as removed. {pull}37402[#37402] (issue: {issue}36761[#36761])

  • Make shard follow tasks more resilient for restarts {pull}37239[#37239] (issue: {issue}37231[#37231])

  • Resume follow Api should not require a request body {pull}37217[#37217] (issue: {issue}37022[#37022])

  • Report error if auto follower tries auto follow a leader index with soft deletes disabled {pull}36886[#36886] (issue: {issue}33007[#33007])

  • Remote cluster license checker and no license info. {pull}36837[#36837] (issue: {issue}36815[#36815])

  • Make CCR resilient against missing remote cluster connections {pull}36682[#36682] (issues: {issue}36255[#36255], {issue}36667[#36667])

  • Fix follow stats API’s follower index filtering feature {pull}36647[#36647]

  • AutoFollowCoordinator and follower index already created {pull}36540[#36540] (issue: {issue}33007[#33007])

  • AutoFollowCoordinator should tolerate that auto follow patterns may be removed {pull}35945[#35945] (issue: {issue}35937[#35937])

  • Only auto follow indices when all primary shards have started {pull}35814[#35814] (issue: {issue}35480[#35480])

  • Avoid NPE in follower stats when no tasks metadata {pull}35802[#35802]

  • Fix the names of CCR stats endpoints in usage API {pull}35438[#35438]

CRUD
  • Synchronize WriteReplicaResult callbacks {pull}36770[#36770]

  • Fix DeleteRequest validation for nullable or empty id/type {pull}35314[#35314] (issue: {issue}35297[#35297])

  • Fix UpdateRequest.fromXContent {pull}35257[#35257] (issues: {issue}29293[#29293], {issue}34069[#34069])

Distributed
  • Combine the execution of an exclusive replica operation with primary term update {pull}36116[#36116] (issue: {issue}35850[#35850])

  • ActiveShardCount should not fail when closing the index {pull}35936[#35936]

Engine
  • Wrap can_match reader with ElasticsearchDirectoryReader {pull}35857[#35857]

  • Copy checkpoint atomically when rolling generation {pull}35407[#35407]

Features
  • Handle Null in FetchSourceContext#fetchSource {pull}36839[#36839] (issue: {issue}29293[#29293])

Geo
  • More robust handling of ignore_malformed in geoshape parsing {pull}35603[#35603] (issues: {issue}34047[#34047], {issue}34498[#34498])

  • Better handling of malformed geo_points {pull}35554[#35554] (issue: {issue}35419[#35419])

  • Enables coerce support in WKT polygon parser {pull}35414[#35414] (issue: {issue}35059[#35059])

  • Further improve robustness of geo shape parser for malformed shapes {pull}34498[#34498] (issues: {issue}31449[#31449], {issue}34047[#34047])

Index lifecycle management
  • Remove indexing_complete when removing policy {pull}36620[#36620]

Index APIs
  • Fix duplicate phrase in shrink/split error message {pull}36734[#36734] (issue: {issue}36729[#36729])

  • Make XContentBuilder in AliasActions build is_write_index field {pull}35071[#35071]

  • Raise a 404 exception when document source is not found (#33384) {pull}34083[#34083] (issue: {issue}33384[#33384])

Ingest
  • Fix on_failure with Drop processor {pull}36686[#36686] (issue: {issue}36151[#36151])

  • Support default pipelines + bulk upserts {pull}36618[#36618] (issue: {issue}36219[#36219])

  • Support default pipeline through an alias {pull}36231[#36231] (issue: {issue}35817[#35817])

  • Dot_expander_processor prevent null add/append to source document {pull}35106[#35106]

Machine Learning
  • Fix cause of "Sample out of bounds" error message. {ml-pull}335[#335]

  • Fix hang when closing a job or creating a forecast. This problem occurs if you created a forecast for a large job and temporary storage was not cleaned up. {ml-pull}352[#352] (issue: {ml-issue}350[#350])

  • Wait for autodetect to be ready in the datafeed {pull}37349[#37349] (issues: {issue}36810[#36810], {issue}37227[#37227])

  • Stop datafeeds when their jobs are stale {pull}37227[#37227] (issue: {issue}36810[#36810])

  • Order get job stats API response by job id {pull}36841[#36841] (issue: {issue}36683[#36683])

Mapping
  • Make sure to accept empty unnested mappings in create index requests. {pull}37089[#37089]

Monitoring
  • Add missing error type mapping for apm-server {pull}36178[#36178] (issue: {issue}1614[#1614])

Network
  • Do not resolve addresses in remote connection info {pull}36671[#36671] (issue: {issue}35658[#35658])

  • Always compress based on the settings {pull}36522[#36522] (issue: {issue}36399[#36399])

  • http.publish_host should contain CNAME {pull}32806[#32806] (issue: {issue}22029[#22029])

Packaging
  • Fix error message when package install fails due to missing Java {pull}36077[#36077] (issue: {issue}31845[#31845])

  • Add missing entries to conffiles {pull}35810[#35810] (issue: {issue}35691[#35691])

Ranking
  • QueryRescorer should keep the window size when rewriting {pull}36836[#36836]

  • Fix a bug in function_score queries where we use the wrong boost_mode. {pull}35148[#35148] (issue: {issue}35123[#35123])

Recovery
  • Register ResyncTask.Status as a NamedWriteable {pull}36610[#36610]

Rollup
  • Fix Rollup’s metadata parser {pull}36791[#36791] (issue: {issue}36726[#36726])

  • Fix rollup search statistics {pull}36674[#36674]

  • Improve handling of failures on first search {pull}35269[#35269]

  • Proactively resolve index patterns in RollupSearch endoint {pull}34930[#34930] (issue: {issue}34828[#34828])

Scripting
  • Properly support no-offset date formatting {pull}36316[#36316] (issue: {issue}36306[#36306])

  • [Painless] Generate Bridge Methods {pull}36097[#36097]

  • Fix serialization bug in painless execute api request {pull}36075[#36075] (issue: {issue}36050[#36050])

  • Actually add joda time back to whitelist {pull}35965[#35965] (issue: {issue}35915[#35915])

  • Add back joda to whitelist {pull}35915[#35915] (issue: {issue}35913[#35913])

  • [Painless] Partially fixes def boxed types casting {pull}35563[#35563] (issue: {issue}35351[#35351])

  • Add back lookup vars in score script {pull}34833[#34833]

Search
  • Inner hits fail to propagate doc-value format. (#36310) {pull}36355[#36355] (issue: {issue}36310[#36310])

  • Fix custom AUTO issue with Fuzziness#toXContent {pull}35807[#35807] (issue: {issue}33462[#33462])

  • Fix analyzed prefix query in query_string {pull}35756[#35756] (issue: {issue}31702[#31702])

  • Fix problem with MatchNoDocsQuery in disjunction queries {pull}35726[#35726] (issue: {issue}34708[#34708])

  • Fix phrase_slop in query_string query {pull}35533[#35533] (issue: {issue}35125[#35125])

  • Add a More Like This query routing requirement check (#29678) {pull}33974[#33974]

Security
  • Remove license state listeners on closables {pull}36308[#36308] (issues: {issue}33328[#33328], {issue}35627[#35627], {issue}35628[#35628])

Settings
  • Fix setting by time unit {pull}37192[#37192]

  • Fix handling of fractional byte size value settings {pull}37172[#37172]

  • Fix handling of fractional time value settings {pull}37171[#37171]

  • Correctly Identify Noop Updates {pull}36560[#36560] (issue: {issue}36496[#36496])

Snapshot/Restore
  • Improve Resilience SnapshotShardService {pull}36113[#36113] (issue: {issue}32265[#32265])

  • Register Azure max_retries setting {pull}35286[#35286]

  • Restore Should Check Min. Version {pull}34676[#34676] (issue: {issue}34264[#34264])

SQL
  • Fix issue with field names containing "." {pull}37364[#37364] (issue: {issue}37128[#37128])

  • Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) {pull}37254[#37254] (issue: {issue}30285[#30285])

  • Fix COUNT DISTINCT filtering {pull}37176[#37176] (issue: {issue}37086[#37086])

  • Fix issue with wrong NULL optimization {pull}37124[#37124] (issue: {issue}35872[#35872])

  • Count distinct doesn’t recognize that a string field has a keyword version {pull}37176[#37176] (issue: {issue}37087[#37087])

  • Handle the bwc Joda ZonedDateTime scripting class in Painless {pull}37024[#37024] (issue: {issue}37023[#37023])

  • Fix bug regarding histograms usage in scripting {pull}36866[#36866]

  • Fix issue with always false filter involving functions {pull}36830[#36830] (issue: {issue}35980[#35980])

  • Protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {pull}36800[#36800] (issue: {issue}36756[#36756])

  • Fix translation of LIKE/RLIKE keywords {pull}36672[#36672] (issues: {issue}36039[#36039], {issue}36584[#36584])

  • Scripting support for casting functions CAST and CONVERT {pull}36640[#36640] (issue: {issue}36061[#36061])

  • Fix translation to painless for conditionals {pull}36636[#36636] (issue: {issue}36631[#36631])

  • Concat should be always not nullable {pull}36601[#36601] (issue: {issue}36169[#36169])

  • Fix MOD() for long and integer arguments {pull}36599[#36599] (issue: {issue}36364[#36364])

  • Fix issue with complex HAVING and GROUP BY ordinal {pull}36594[#36594] (issue: {issue}36059[#36059])

  • Be lenient for tests involving comparison to H2 but strict for csv spec tests {pull}36498[#36498] (issue: {issue}36483[#36483])

  • Non ISO 8601 versions of DAY_OF_WEEK and WEEK_OF_YEAR functions {pull}36358[#36358] (issue: {issue}36263[#36263])

  • Do not ignore all fields whose names start with underscore {pull}36214[#36214] (issue: {issue}36206[#36206])

  • SUM() and LIKE condition doesn’t work anymore {pull}36672[#36672] (issue: {issue}36161[#36161])

  • Fix issue with wrong data type for scripted Grouping keys {pull}35969[#35969] (issue: {issue}35662[#35662])

  • Fix translation of math functions to painless {pull}35910[#35910] (issue: {issue}35654[#35654])

  • Build: Fix jdbc jar to include deps {pull}35602[#35602]

  • Fix query translation for scripted queries {pull}35408[#35408] (issue: {issue}35232[#35232])

  • Clear the cursor if nested inner hits are enough to fulfill the query required limits {pull}35398[#35398] (issue: {issue}35176[#35176])

  • Fix null handling for AND and OR in SELECT {pull}35277[#35277] (issue: {issue}35240[#35240])

  • Handle null literal for AND and OR in WHERE {pull}35236[#35236] (issue: {issue}35088[#35088])

  • Introduce NotEquals node to simplify expressions {pull}35234[#35234] (issues: {issue}35210[#35210], {issue}35233[#35233])

  • Introduce IsNull node to simplify expressions {pull}35206[#35206] (issues: {issue}34876[#34876], {issue}35171[#35171])

  • Handle wildcard expansion on incorrect fields {pull}35134[#35134] (issue: {issue}35092[#35092])

  • Fix null handling for IN ⇒ painless script {pull}35124[#35124] (issues: {issue}35108[#35108], {issue}35122[#35122])

  • Register missing processors {pull}35121[#35121] (issue: {issue}35119[#35119])

  • Fix NPE thrown if HAVING filter evals to null {pull}35108[#35108] (issue: {issue}35107[#35107])

  • Proper handling of nested fields at the beginning of the columns list {pull}35068[#35068] (issue: {issue}32951[#32951])

  • Fix incorrect AVG data type {pull}34948[#34948] (issue: {issue}33773[#33773])

  • Add CAST and CONVERT to SHOW FUNCTIONS {pull}34940[#34940] (issue: {issue}34939[#34939])

  • Handle aggregation for null group {pull}34916[#34916] (issue: {issue}34896[#34896])

  • Provide null-safe scripts for Not and Neg {pull}34877[#34877] (issue: {issue}34848[#34848])

  • Return error with ORDER BY on non-grouped. {pull}34855[#34855] (issue: {issue}34590[#34590])

  • Fix negation of equals comparison. {pull}34680[#34680] (issue: {issue}34558[#34558])

  • CAST doesn’t work in ORDER BY. {pull}36640[#36640] (issue: {issue}34557[#34557])

  • COUNT(column) takes into account NULLs {pull}37254[#37254] (issue: {issue}34549[#34549])

Watcher
  • Watcher accounts constructed lazily {pull}36656[#36656]

  • Only trigger a watch if new or schedule/changed {pull}35908[#35908]

  • Fix Watcher NotificationService’s secure settings {pull}35610[#35610] (issue: {issue}35378[#35378])

  • Fix integration tests to ensure correct start/stop of Watcher {pull}35271[#35271] (issues: {issue}29877[#29877], {issue}30705[#30705], {issue}33291[#33291], {issue}34448[#34448], {issue}34462[#34462])

Regressions

Scripting
  • Use Number as a return value for BucketAggregationScript {pull}35653[#35653] (issue: {issue}35351[#35351])

Upgrades

Network
  • Upgrade Netty 4.3.32.Final {pull}36102[#36102] (issue: {issue}35360[#35360])

{es} version 6.5.4

Bug fixes

Aggregations
  • Fix MultiValuesSourceFieldConfig toXContent {pull}36525[#36525] (issue: {issue}36474[#36474])

Audit
  • Fix origin.type for connection_* events {pull}36410[#36410]

CCR
  • Fix follow stats API’s follower index filtering feature {pull}36647[#36647]

Machine learning
  • Interrupt Grok in file structure finder timeout {pull}36588[#36588]

  • Prevent stack overflow while copying {ml} jobs and datafeeds {pull}36370[#36370] (issue: {issue}36360[#36360])

  • Fix a problem that could result in shifted model bounds and increased forecast errors with long bucket spans. The solution was to correct query times for model bounds and forecasts in the bucket to match the times assigned to the samples that are added to the model for each bucket. {ml-pull}327[#327]

SQL
  • Fix MOD() for long and integer arguments {pull}36599[#36599] (issue: {issue}36364[#36364])

  • Fix issue with complex HAVING and GROUP BY ordinal {pull}36594[#36594] (issue: {issue}36059[#36059])

Watcher
  • Fix Watcher NotificationService’s secure settings {pull}35610[#35610] (issue: {issue}35378[#35378])

{es} version 6.5.3

Enhancements

Engine
  • Always configure soft-deletes field of IndexWriterConfig {pull}36196[#36196] (issue: {issue}36141[#36141])

Bug fixes

Audit
  • Fix deprecation of audit log settings {pull}36175[#36175] (issue: {issue}36162[#36162])

CCR
  • AutoFollowCoordinator should tolerate that auto follow patterns may be removed {pull}35945[#35945] (issue: {issue}35937[#35937])

Features/Monitoring
  • Add missing error type mapping for apm-server {pull}36178[#36178] (issue: {issue}1614[#1614])

Infra/Packaging
  • Fix error message when package install fails due to missing Java {pull}36077[#36077] (issue: {issue}31845[#31845])

Infra/Scripting
  • Properly support no-offset date formatting {pull}36316[#36316] (issue: {issue}36306[#36306])

  • Fix serialization bug in painless execute api request {pull}36075[#36075] (issue: {issue}36050[#36050])

Security
  • Remove license state listeners on closables {pull}36308[#36308] (issues: {issue}33328[#33328], {issue}35627[#35627], {issue}35628[#35628])

Snapshot/Restore
  • Improve Resilience SnapshotShardService {pull}36113[#36113] (issue: {issue}32265[#32265])

{es} version 6.5.2

{es} version 6.5.1

Enhancements

Authorization
  • Grant .tasks access to kibana_system role {pull}35573[#35573]

Search
  • has_parent builder: exception message/param fix {pull}31182[#31182]

Bug fixes

Aggregations
  • Correct implemented interface of ParsedReverseNested {pull}35455[#35455] (issue: {issue}35449[#35449])

  • Handle IndexOrDocValuesQuery in composite aggregation {pull}35392[#35392]

  • Preserve format when aggregation contains unmapped date fields {pull}35254[#35254] (issue: {issue}31760[#31760])

Infra/Core
  • Upgrade to Joda 2.10.1 {pull}35410[#35410] (issue: {issue}33749[#33749])

Machine Learning
  • Fix find_file_structure NPE with should_trim_fields {pull}35465[#35465] (issue: {issue}35462[#35462])

  • Prevent notifications being created on deletion of a non existent job {pull}35337[#35337] (issues: {issue}34058[#34058], {issue}35336[#35336])

SQL
  • Fix query translation for scripted queries {pull}35408[#35408] (issue: {issue}35232[#35232])

  • Clear the cursor if nested inner hits are enough to fulfill the query required limits {pull}35398[#35398] (issue: {issue}35176[#35176])

  • SQL: Introduce IsNull node to simplify expressions {pull}35206[#35206] (issues: {issue}34876[#34876], {issue}35171[#35171])

Scripting
  • [Painless] Partially fixes def boxed types casting {pull}35563[#35563] (issue: {issue}35351[#35351])

{es} version 6.5.0

Breaking changes

Aggregations
  • Require combine and reduce scripts in scripted metrics aggregation {pull}33452[#33452] (issue: {issue}32804[#32804])

Geo
  • Geo: Don’t flip longitude of envelopes crossing dateline {pull}34535[#34535] (issue: {issue}34418[#34418])

Mapping
  • Disallow "enabled" attribute change for types in mapping update (#33566) {pull}33933[#33933]

Plugins
  • Introduce index store plugins {pull}32375[#32375] (issue: {issue}32267[#32267])

Breaking Java changes

Authorization
  • Use RoleRetrievalResult for better caching {pull}34197[#34197] (issue: {issue}33205[#33205])

Core
  • Use generic AcknowledgedResponse instead of extended classes {pull}32859[#32859]

  • Switch WritePipelineResponse to AcknowledgedResponse {pull}32722[#32722]

Java High Level REST Client
  • HLRC XPack Protocol clean up: Licence, Misc {pull}34469[#34469] (issue: {issue}34451[#34451])

Plugins
  • Fix generics in ScriptPlugin#getContexts() {pull}33426[#33426]

Deprecations

Analysis
  • Deprecates the standard filter {pull}33468[#33468]

Scripting
  • Scripting: Conditionally use java time api in scripting {pull}31441[#31441]

Search
  • Deprecate negative query boost (#34486) {pull}34512[#34512] (issues: {issue}33309[#33309], {issue}34486[#34486])

  • Deprecate negative weight in Function Score Query {pull}33624[#33624] (issue: {issue}31927[#31927])

Suggesters
  • Make Geo Context Mapping Parsing More Strict (6.x version) {pull}32862[#32862] (issue: {issue}32821[#32821])

New features

Aggregations
  • Adds a new auto-interval date histogram {pull}28993[#28993] (issue: {issue}9572[#9572])

Analysis
  • Add script_filter tokenfilter {pull}33431[#33431]

  • Add conditional token filter to elasticsearch {pull}31958[#31958]

Audit
  • Structured audit logging {pull}31931[#31931] and {pull}33894[#33894] (backport) (issue: {issue}31046[#31046]) This introduces a new format for the logfile audit type which is output alongside the previous one. Conforming to the out-of-the-box log4j2.properties configuration, the new and deprecated audit formats are output to the <cluster_name>_access.log and <cluster_name>_audit.log files, respectively. See [audit-log-output] for more details.

Authentication
  • Allow User/Password realms to disable authc {pull}34033[#34033] (issue: {issue}33292[#33292])

Authorization
  • [X-Pack] Beats centralized management: security role + licensing {pull}34305[#34305] (issues: {issue}30493[#30493], {issue}30520[#30520])

  • Add get-user-privileges API {pull}33928[#33928] (issue: {issue}32777[#32777])

  • Add support for "authorization_realms" {pull}33262[#33262]

CCR
  • [CCR] Added auto follow patterns feature {pull}33118[#33118] (issue: {issue}33007[#33007])

Index APIs
  • Add an index setting to control TieredMergePolicy#deletesPctAllowed {pull}32907[#32907]

Java High Level REST Client
  • add start trial API to HLRC {pull}32799[#32799]

Machine Learning
  • Add a file structure determination endpoint {pull}33471[#33471]

  • Partition-wise maximum scores {pull}32748[#32748]

Mapping
  • New Annotated_text field type {pull}30364[#30364] (issue: {issue}29467[#29467])

Monitoring
  • APM server monitoring {pull}32515[#32515]

SQL
  • SQL: Implement CONVERT, an alternative to CAST {pull}34660[#34660] (issue: {issue}34513[#34513])

  • SQL: Implement IN(value1, value2, …​) expression. {pull}34581[#34581] (issue: {issue}32955[#32955])

  • SQL: TRUNCATE and ROUND functions {pull}33779[#33779] (issue: {issue}33494[#33494])

  • SQL: Adds MONTHNAME, DAYNAME and QUARTER functions {pull}33411[#33411] (issue: {issue}33092[#33092])

Search
  • Add max_children limit to nested sort {pull}33587[#33587] (issue: {issue}33592[#33592])

Enhancements

Aggregations
  • Rollup adding support for date field metrics (#34185) {pull}34200[#34200] (issue: {issue}34185[#34185])

  • Add early termination support for min/max aggregations {pull}33375[#33375]

  • Add early termination support to BucketCollector {pull}33279[#33279]

  • Add interval response parameter to AutoDateInterval histogram {pull}33254[#33254]

  • Scripted metric aggregations: add deprecation warning and system property to control legacy params {pull}31597[#31597] (issues: {issue}29328[#29328], {issue}30111[#30111])

Allocation
  • Skip rebalancing when cluster_concurrent_rebalance threshold reached {pull}33329[#33329] (issue: {issue}27628[#27628])

Analysis
  • Allow TokenFilterFactories to rewrite themselves against their preceding chain {pull}33702[#33702] (issue: {issue}33609[#33609])

Audit
  • Add opaque_id to index audit logging {pull}32260[#32260] (issue: {issue}31521[#31521])

Authentication
  • Security: reduce memory usage of DnRoleMapper {pull}34250[#34250] (issue: {issue}34237[#34237])

  • Security: upgrade unboundid ldapsdk to 4.0.8 {pull}34247[#34247] (issue: {issue}33175[#33175])

  • [Kerberos] Add realm name & UPN to user metadata {pull}33338[#33338]

  • [SECURITY] Set Auth-scheme preference {pull}33156[#33156] (issue: {issue}32699[#32699])

  • Token API supports the client_credentials grant {pull}33106[#33106]

Authorization
  • [Authz] Allow update settings action for system user {pull}34030[#34030] (issue: {issue}33119[#33119])

  • Calculate changed roles on roles.yml reload {pull}33525[#33525] (issue: {issue}33205[#33205])

  • [Kerberos] Add authorization realms support to Kerberos realm {pull}32392[#32392]

  • [X-Pack] Beats centralized management: security role + licensing {pull}30520[#30520] (issue: {issue}30493[#30493])

Beats Plugin
  • [Monitoring] Update beats template to include apm-server metrics {pull}33286[#33286]

CRUD
  • Verify primary mode usage with assertions {pull}32667[#32667] (issues: {issue}10708[#10708], {issue}25692[#25692], {issue}32442[#32442])

  • Refactor TransportShardBulkAction to better support retries {pull}31821[#31821]

Circuit Breakers
  • Whitelisting / from Circuit Breaker Exception (#32325) {pull}32666[#32666]

Core
  • Improved IndexNotFoundException’s default error message {pull}34649[#34649] (issue: {issue}34628[#34628])

  • Prevent cause from being null in ShardOperationFailedException {pull}32640[#32640] (issue: {issue}32608[#32608])

  • Enable avoiding mmap bootstrap check {pull}32421[#32421] (issue: {issue}32267[#32267])

Discovery-Plugins
  • [GCE Discovery] Automatically set project-id and zone {pull}33721[#33721] (issue: {issue}13618[#13618])

Distributed
  • Add contains method to LocalCheckpointTracker {pull}33871[#33871] (issue: {issue}33656[#33656])

  • Introduce global checkpoint listeners {pull}32696[#32696] (issue: {issue}32651[#32651])

  • Expose whether or not the global checkpoint updated {pull}32659[#32659] (issue: {issue}32651[#32651])

  • Include translog path in error message when translog is corrupted {pull}32251[#32251] (issue: {issue}24929[#24929])

Docs Infrastructure
  • Docs: Allow snippets to have line continuation {pull}32649[#32649]

Engine
  • Do not alloc full buffer for small change requests {pull}35158[#35158]

  • Fill LocalCheckpointTracker with Lucene commit {pull}34474[#34474] (issues: {issue}0[#0], {issue}2[#2], {issue}33656[#33656])

  • Lock down Engine.Searcher {pull}34363[#34363] (issue: {issue}34357[#34357])

  • Fold EngineSearcher into Engine.Searcher {pull}34082[#34082]

  • Build DocStats from SegmentInfos in ReadOnlyEngine {pull}34079[#34079] (issue: {issue}33903[#33903])

  • Move CompletionStats into the Engine {pull}33847[#33847] (issue: {issue}33835[#33835])

  • Move DocsStats into Engine {pull}33835[#33835]

  • Add read-only Engine {pull}33563[#33563] (issues: {issue}32844[#32844], {issue}32867[#32867])

  • Allow engine to recover from translog upto a seqno {pull}33032[#33032] (issue: {issue}32867[#32867])

Index APIs
  • Introduce index settings version {pull}34429[#34429]

  • Add cluster-wide shard limit warnings {pull}34021[#34021] (issues: {issue}20705[#20705], {issue}32856[#32856])

  • Copy and validate soft-deletes setting on resize {pull}33517[#33517] (issue: {issue}33321[#33321])

  • Introduce mapping version to index metadata {pull}33147[#33147]

  • update rollover to leverage write-alias semantics {pull}32216[#32216]

Ingest
  • ingest: processor stats {pull}34724[#34724] (issue: {issue}34202[#34202])

  • ingest: better support for conditionals with simulate?verbose {pull}34155[#34155]

  • ingest: correctly measure chained pipeline stats {pull}33912[#33912]

  • ingest: support simulate with verbose for pipeline processor {pull}33839[#33839]

  • [ingest] geo-ip performance improvements {pull}33029[#33029]

  • ingest: Introduce the dissect processor {pull}32884[#32884]

  • INGEST: Add Pipeline Processor {pull}32473[#32473] (issue: {issue}31842[#31842])

  • Ingest: Add conditional per processor {pull}32398[#32398] (issue: {issue}21248[#21248])

  • Introduce the dissect library {pull}32297[#32297]

  • INGEST: Enable default pipelines {pull}32286[#32286] (issue: {issue}21101[#21101])

  • INGEST: Implement Drop Processor {pull}32278[#32278] (issue: {issue}23726[#23726])

  • Ingest: Support integer and long hex values in convert {pull}32213[#32213] (issue: {issue}32182[#32182])

  • Add region ISO code to GeoIP Ingest plugin {pull}31669[#31669]

Java High Level REST Client
  • HLRC: Delete role API {pull}34620[#34620]

  • [HLRC] Add support for Delete role mapping API {pull}34531[#34531]

  • [HLRC] Add Start/Stop Watch Service APIs. {pull}34317[#34317]

  • HLRC: ML Add preview datafeed api {pull}34284[#34284] (issue: {issue}29827[#29827])

  • HLRC: ML Adding get datafeed stats API {pull}34271[#34271] (issue: {issue}29827[#29827])

  • HLRC: Deactivate Watch API {pull}34192[#34192] (issues: {issue}29827[#29827], {issue}33988[#33988])

  • Create/Update role mapping API {pull}34171[#34171] (issue: {issue}33745[#33745])

  • HLRC: Get SSL Certificates API {pull}34135[#34135]

  • [ML][HLRC] Replace REST-based ML test cleanup with the ML client {pull}34109[#34109]

  • HLRC: Add activate watch action {pull}33988[#33988] (issue: {issue}29827[#29827])

  • Add support for 'ack watch' to the HLRC. {pull}33962[#33962] (issue: {issue}29827[#29827])

  • HLRC: Add throttling for update & delete-by-query {pull}33951[#33951]

  • HLRC: ML Stop datafeed API {pull}33946[#33946] (issue: {issue}29827[#29827])

  • HLRC: Add get rollup job {pull}33921[#33921]

  • HLRC: ML start data feed API {pull}33898[#33898] (issue: {issue}29827[#29827])

  • HLRC: Add support for reindex rethrottling {pull}33832[#33832]

  • HLRC: Reindex should support requests_per_seconds parameter {pull}33808[#33808]

  • HLRC: Delete ML calendar {pull}33775[#33775] (issue: {issue}29827[#29827])

  • HLRC: Get ML calendars {pull}33760[#33760] (issue: {issue}29827[#29827])

  • [HLRC] Support for role mapper expression dsl {pull}33745[#33745]

  • [HLRC][ML] Add ML get datafeed API to HLRC {pull}33715[#33715] (issue: {issue}29827[#29827])

  • REST client: introduce a strict deprecation mode {pull}33708[#33708] (issue: {issue}33534[#33534])

  • [HLRC][ML] Add ML delete datafeed API to HLRC {pull}33667[#33667] (issue: {issue}29827[#29827])

  • HLRC: Add support for XPack Post Start Basic Licence API {pull}33606[#33606] (issue: {issue}29827[#29827])

  • [HLRC][ML] Add ML put datafeed API to HLRC {pull}33603[#33603] (issue: {issue}29827[#29827])

  • Create a WatchStatus class for the high-level REST client. {pull}33527[#33527]

  • HLRC: ML Delete Forecast API {pull}33526[#33526] (issue: {issue}29827[#29827])

  • Add create rollup job api to high level rest client {pull}33521[#33521] (issues: {issue}29827[#29827], {issue}32703[#32703])

  • HLRC: add change password API support {pull}33509[#33509] (issue: {issue}33481[#33481])

  • HLRC: ML Forecast Job {pull}33506[#33506] (issue: {issue}29827[#29827])

  • HLRC: add enable and disable user API support {pull}33481[#33481] (issue: {issue}29827[#29827])

  • HLRC: Add ML get categories API {pull}33465[#33465] (issue: {issue}29827[#29827])

  • HLRC: ML Post Data {pull}33443[#33443] (issue: {issue}29827[#29827])

  • add start trial API to HLRC {pull}33406[#33406]

  • HLRC: ML Update Job {pull}33392[#33392] (issue: {issue}29827[#29827])

  • HLRC: Add ML get influencers API {pull}33389[#33389] (issue: {issue}29827[#29827])

  • HLRC: ML PUT Calendar {pull}33362[#33362] (issue: {issue}29827[#29827])

  • HLRC: Add ML get overall buckets API {pull}33297[#33297] (issue: {issue}29827[#29827])

  • HLRC: create base timed request class {pull}33216[#33216]

  • HLRC: add client side RefreshPolicy {pull}33209[#33209]

  • HLRC: ML Flush job {pull}33187[#33187] (issue: {issue}29827[#29827])

  • HLRC: Adding ML Job stats {pull}33183[#33183] (issue: {issue}29827[#29827])

  • HLRC: Use Optional in validation logic {pull}33104[#33104]

  • HLRC: Add ML Get Records API {pull}33085[#33085] (issue: {issue}29827[#29827])

  • HLRC: Add ML Get Buckets API {pull}33056[#33056] (issue: {issue}29827[#29827])

  • GraphClient for the high level REST client and associated tests. {pull}33025[#33025] (issue: {issue}29827[#29827])

  • HLRC: Clear ML data after client tests {pull}33023[#33023] (issue: {issue}32993[#32993])

  • HLRC: Add ML Get Job {pull}32960[#32960] (issue: {issue}29827[#29827])

  • HLRC: ML Close Job {pull}32943[#32943] (issue: {issue}29827[#29827])

  • HLRC: Create server agnostic request and response {pull}32912[#32912]

  • Add GetRollupCaps API to high level rest client {pull}32880[#32880] (issues: {issue}29827[#29827], {issue}32703[#32703])

  • HLRC: adding machine learning open job {pull}32860[#32860] (issue: {issue}29827[#29827])

  • HLRC: Refactor WatchStatus {pull}32842[#32842] (issue: {issue}29827[#29827])

  • HLRC: adding machine learning delete job {pull}32820[#32820] (issue: {issue}29827[#29827])

  • HLRC: Refactor WatchStatus and implement activate watch {pull}32802[#32802] (issue: {issue}29827[#29827])

  • REST high-level client: add delete by query API {pull}32782[#32782] (issues: {issue}27205[#27205], {issue}32679[#32679], {issue}32760[#32760])

  • REST high-level client: add update by query API {pull}32760[#32760] (issues: {issue}27205[#27205], {issue}32679[#32679])

  • HLRC: migration get assistance API {pull}32744[#32744] (issue: {issue}29827[#29827])

  • Adding ML HLRC wrapper and put_job API call {pull}32726[#32726]

  • Add create rollup job api to high level rest client {pull}32703[#32703] (issue: {issue}29827[#29827])

  • REST high-level client: add reindex API {pull}32679[#32679] (issue: {issue}27205[#27205])

  • HLRC: Add Delete License API {pull}32586[#32586] (issue: {issue}29827[#29827])

  • Rest HL client: Add get license action {pull}32438[#32438] (issue: {issue}29827[#29827])

  • HLRC: Add delete watch action {pull}32337[#32337] (issue: {issue}29827[#29827])

  • HLRest: add xpack put user API {pull}32332[#32332] (issue: {issue}29827[#29827])

  • Rest HL client: Add put license action {pull}32214[#32214] (issue: {issue}29827[#29827])

  • Add Restore Snapshot High Level REST API {pull}32155[#32155] (issue: {issue}27205[#27205])

  • Add put stored script support to high-level rest client {pull}31323[#31323] (issue: {issue}27205[#27205])

Logging
  • Logging: Make node name consistent in logger {pull}31588[#31588]

Machine Learning
  • Adding support for lazy nodes {pull}34538[#34538] (issue: {issue}29991[#29991])

  • Add an ingest pipeline definition to structure finder {pull}34350[#34350]

  • Add a timeout option to file structure finder {pull}34117[#34117]

  • Allow asynchronous job deletion {pull}34058[#34058] (issue: {issue}32836[#32836])

  • Make certain ML node settings dynamic {pull}33961[#33961] (issue: {issue}33565[#33565])

  • Display integers without .0 in file structure field stats {pull}33947[#33947]

  • Return both Joda and Java formats from structure finder {pull}33900[#33900]

  • Adding node_count to ML Usage {pull}33863[#33863] (issue: {issue}33850[#33850])

  • Delete custom index if the only contained job is deleted {pull}33788[#33788] (issue: {issue}30075[#30075])

  • Allow overrides for some file structure detection decisions {pull}33630[#33630]

  • Minor improvements to categorization Grok pattern creation {pull}33353[#33353]

  • Delete forecast API {pull}33218[#33218] (issue: {issue}31134[#31134])

  • Perform anomaly detection on features derived from multiple bucket values to improve the robustness of detection with respect to misconfigured bucket lengths and to improve the detection of long lasting anomalies. {ml-pull}175[#175]

  • Support decomposing a time series into a piecewise linear trend and with piecewise constant scaling of the periodic components. This extends our decomposition functionality to handle the same types of change points that our modelling capabilities do. {ml-pull}198[#198]

  • Increased independence of anomaly scores across partitions. {ml-pull}182[#182]

  • Avoid potential false positives at model start up when first detecting new components of the time series decomposition. {ml-pull}218[#218]

  • Add a new label (multi_bucket_impact) to record level anomaly results. The value is on a scale of -5 to +5 where -5 means the anomaly is purely single bucket and +5 means the anomaly is purely multi bucket. {pull}34233[#34233], {ml-pull}230[#230]

  • Improve our ability to detect change points in the presence of outliers. {ml-pull}265[#265]

Mapping
  • Preserve the order of nested documents in the Lucene index {pull}34225[#34225] (issue: {issue}33587[#33587])

  • Don’t count metadata fields towards index.mapping.total_fields.limit {pull}33386[#33386] (issue: {issue}24096[#24096])

  • Add expected mapping type to MapperException {pull}31564[#31564] (issue: {issue}31502[#31502])

Monitoring
  • [Monitoring] Add additional necessary mappings for apm-server {pull}34392[#34392]

  • Adding stack_monitoring_agent role {pull}34369[#34369]

  • [Monitoring] Add cluster metadata to cluster_stats docs {pull}33860[#33860] (issue: {issue}33691[#33691])

  • Implement xpack.monitoring.elasticsearch.collection.enabled setting {pull}33474[#33474] (issue: {issue}33290[#33290])

Network
  • Pass the host name on as server_name if proxy mode is on {pull}34559[#34559]

  • Bad regex in CORS settings should throw a nicer error {pull}34035[#34035]

  • Add sni name to SSLEngine in netty transport (#33144) {pull}33513[#33513] (issue: {issue}32517[#32517])

  • Add sni name to SSLEngine in netty transport {pull}33144[#33144] (issue: {issue}32517[#32517])

  • Add proxy support to RemoteClusterConnection {pull}33062[#33062] (issues: {issue}31840[#31840], {issue}32517[#32517])

  • Use a dedicated ConnectionManger for RemoteClusterConnection {pull}32988[#32988] (issue: {issue}31835[#31835])

Packaging
  • Add Ubuntu 18.04 to packaging tests {pull}34139[#34139]

Ranking
  • Add minimal sanity checks to custom/scripted similarities. (backport) {pull}33893[#33893] (issue: {issue}33564[#33564])

  • Use the global doc id to generate random scores {pull}33599[#33599]

Recovery
  • Use soft-deleted docs to resolve strategy for engine operation {pull}35230[#35230] (issues: {issue}0[#0], {issue}1[#1], {issue}33656[#33656], {issue}34474[#34474])

  • Propagate auto_id_timestamp in primary-replica resync {pull}33964[#33964] (issue: {issue}33693[#33693])

  • Restore local history from translog on promotion {pull}33616[#33616] (issues: {issue}32867[#32867], {issue}33473[#33473])

  • Reset replica engine to global checkpoint on promotion {pull}33473[#33473] (issue: {issue}32867[#32867])

  • Bootstrap a new history_uuid when force allocating a stale primary {pull}33432[#33432] (issue: {issue}26712[#26712])

  • Integrates soft-deletes into Elasticsearch {pull}33222[#33222] (issues: {issue}29530[#29530], {issue}30086[#30086], {issue}30120[#30120], {issue}30335[#30335], {issue}30522[#30522], {issue}31106[#31106])

Rollup
  • [Rollup] Add support for date histo format when searching {pull}34537[#34537] (issue: {issue}34391[#34391])

  • [Rollup] Only allow aggregating on multiples of configured interval {pull}32052[#32052]

SQL
  • SQL: Improve CircuitBreaker logic for SqlParser {pull}35300[#35300] (issue: {issue}35299[#35299])

  • SQL: Optimizer rule for folding nullable expressions {pull}35080[#35080] (issue: {issue}34826[#34826])

  • SQL: Improve painless script generated from IN {pull}35055[#35055] (issue: {issue}34750[#34750])

  • SQL: Implement CAST between STRING and IP {pull}34949[#34949] (issue: {issue}34799[#34799])

  • SQL: Fix function args verification and error msgs {pull}34926[#34926] (issues: {issue}33469[#33469], {issue}34752[#34752])

  • SQL: Introduce ODBC mode, similar to JDBC {pull}34825[#34825] (issue: {issue}34720[#34720])

  • SQL: Introduce support for IP fields {pull}34758[#34758] (issue: {issue}32499[#32499])

  • SQL: Implement null handling for IN(v1, v2, …​) {pull}34750[#34750] (issue: {issue}34582[#34582])

  • SQL: handle X-Pack or X-Pack SQL not being available in a more graceful way {pull}34736[#34736] (issue: {issue}30009[#30009])

  • SQL: Support pattern against compatible indices {pull}34718[#34718] (issues: {issue}31611[#31611], {issue}31837[#31837], {issue}33803[#33803])

  • SQL: Allow min/max aggregates on date fields {pull}34699[#34699] (issue: {issue}34477[#34477])

  • SQL: Introduce support for NULL values (#34573) {pull}34640[#34640] (issue: {issue}32079[#32079])

  • SQL: return constants for all matching records in constants-containing SELECTs {pull}34576[#34576] (issue: {issue}31863[#31863])

  • SQL: Functions enhancements (OCTET_LENGTH function, order functions alphabetically, RANDOM function docs) {pull}34101[#34101] (issue: {issue}33477[#33477])

  • SQL: Internal refactoring of operators as functions {pull}34097[#34097] (issue: {issue}33975[#33975])

  • SQL: Remove more ANTLR4 grammar ambiguities {pull}34074[#34074] (issue: {issue}33854[#33854])

  • SQL: Move away internally from JDBCType to SQLType {pull}33913[#33913] (issue: {issue}33904[#33904])

  • SQL: Fix ANTL4 Grammar ambiguities. {pull}33854[#33854] (issue: {issue}31885[#31885])

  • SQL: Better handling of number parsing exceptions {pull}33776[#33776] (issue: {issue}33622[#33622])

  • SQL: Grammar tweak for number declarations {pull}33767[#33767] (issue: {issue}33765[#33765])

  • SQL: Return functions in JDBC driver metadata {pull}33672[#33672] (issue: {issue}33671[#33671])

  • SQL: Make Literal a NamedExpression {pull}33583[#33583] (issue: {issue}33523[#33523])

  • SQL: Improve alias vs index resolution {pull}33393[#33393] (issue: {issue}33363[#33363])

  • SQL: Align SYS TABLE for ODBC SQL_ALL_* args {pull}33364[#33364] (issue: {issue}33312[#33312])

  • SQL: Show/desc commands now support table ids {pull}33363[#33363] (issue: {issue}33294[#33294])

  • SQL: Support multi-index format as table identifier {pull}33278[#33278]

  • SQL: Multiple indices pattern {pull}33162[#33162]

  • SQL: skip uppercasing/lowercasing function tests for AZ locales as well {pull}32910[#32910] (issue: {issue}32589[#32589])

  • SQL: test coverage for JdbcResultSet {pull}32813[#32813] (issue: {issue}32078[#32078])

  • SQL: Added support for string manipulating functions with more than one parameter {pull}32356[#32356] (issue: {issue}31604[#31604])

Scripting
  • Painless: Add Static Methods Shortcut {pull}33440[#33440]

  • Painless: Add Bindings {pull}33042[#33042]

Search
  • Replace version with reader cache key in IndicesRequestCache {pull}34189[#34189] (issues: {issue}27650[#27650], {issue}33473[#33473])

  • Handle terms query when detecting if a query can match nested docs {pull}34072[#34072] (issue: {issue}34067[#34067])

  • Add a limit for graph phrase query expansion {pull}34031[#34031]

  • Clarify RemoteClusterService#groupIndices behaviour {pull}33899[#33899]

  • Add nested and object fields to field capabilities response {pull}33803[#33803] (issue: {issue}33237[#33237])

  • Introduce a search_throttled threadpool {pull}33732[#33732]

  • Upgrade remote cluster settings {pull}33537[#33537] (issues: {issue}33413[#33413], {issue}33536[#33536])

  • Remove unsupported group_shard_failures parameter {pull}33208[#33208] (issue: {issue}32598[#32598])

  • Profiler: Don’t profile NEXTDOC for ConstantScoreQuery. {pull}33196[#33196] (issue: {issue}23430[#23430])

  • Change query field expansion {pull}33020[#33020] (issues: {issue}31655[#31655], {issue}31798[#31798])

  • Expose max_concurrent_shard_requests in _msearch {pull}33016[#33016] (issue: {issue}31877[#31877])

  • Search: Support of wildcard on docvalue_fields {pull}32980[#32980] (issues: {issue}26299[#26299], {issue}26390[#26390])

  • Ignore script fields when size is 0 {pull}31917[#31917] (issue: {issue}31824[#31824])

Security
  • Generate non-encrypted license public key {pull}34626[#34626]

  • Security: don’t call prepare index for reads {pull}34568[#34568] (issues: {issue}33205[#33205], {issue}34246[#34246])

  • Enable security automaton caching {pull}34028[#34028]

  • Add Debug/Trace logging to token service {pull}34022[#34022]

  • Security index expands to a single replica {pull}33131[#33131] (issues: {issue}29712[#29712], {issue}29933[#29933])

  • Introduce fips_mode setting and associated checks {pull}32326[#32326]

Settings
  • Introduce private settings {pull}33327[#33327] (issue: {issue}31286[#31286])

  • Add user-defined cluster metadata {pull}33325[#33325] (issue: {issue}33220[#33220])

  • Add settings updater for 2 affix settings {pull}33050[#33050]

Snapshot/Restore
  • Use more precise does S3 bucket exist method {pull}34123[#34123]

  • Add _source-only snapshot repository {pull}32844[#32844]

  • Increase max chunk size to 256Mb for repo-azure {pull}32101[#32101] (issue: {issue}12448[#12448])

  • Update AWS SDK to 1.11.340 in repository-s3 {pull}30723[#30723] (issues: {issue}22758[#22758], {issue}25552[#25552], {issue}30474[#30474])

Stats
  • Add cluster UUID to Cluster Stats API response {pull}32206[#32206] (issue: {issue}32205[#32205])

Store
  • add elasticsearch-shard tool to 6.x {pull}33848[#33848] (issue: {issue}31389[#31389])

  • drop index.shard.check_on_startup: fix {pull}32279[#32279] (issue: {issue}31389[#31389])

Suggesters
  • Completion types with multi-fields support {pull}34081[#34081] (issue: {issue}15115[#15115])

Watcher
  • Watcher: Reduce script cache churn by checking for mustache tags {pull}33978[#33978] (issue: {issue}29280[#29280])

  • [Watcher] Improved error messages for CronEvalTool {pull}32800[#32800] (issue: {issue}32735[#32735])

  • Watcher: Use Bulkprocessor in HistoryStore/TriggeredWatchStore {pull}32490[#32490]

  • Watcher: migrate PagerDuty v1 events API to v2 API {pull}32285[#32285] (issue: {issue}32243[#32243])

ZenDiscovery
  • Allow excluding folder names when scanning for dangling indices {pull}34349[#34349]

Bug fixes

Aggregations
  • Fix handling of empty keyword in terms aggregation {pull}34457[#34457] (issue: {issue}34434[#34434])

  • Check self references in metric agg after last doc collection (#33593) {pull}34001[#34001]

  • Unmapped aggs should not run pipelines if they delegate reduction {pull}33528[#33528] (issue: {issue}33514[#33514])

  • For filters aggregations, make sure that rewrites preserve other_bucket. {pull}32921[#32921] (issue: {issue}32834[#32834])

Allocation
  • DiskThresholdDecider#canAllocate can report negative free bytes {pull}33641[#33641] (issue: {issue}33596[#33596])

  • Don’t omit default values when updating routing exclusions (#32721) {pull}33638[#33638]

Analysis
  • Check stemmer language setting early {pull}34601[#34601] (issue: {issue}34170[#34170])

Authentication
  • ListenableFuture should preserve ThreadContext {pull}34394[#34394]

  • Allow an AuthenticationResult to return metadata {pull}34382[#34382] (issues: {issue}34290[#34290], {issue}34332[#34332])

  • Preserve thread context during authentication {pull}34290[#34290]

  • [Kerberos] Add debug log statement for exceptions {pull}32663[#32663]

  • [Kerberos] Remove Kerberos bootstrap checks {pull}32451[#32451]

Authorization
  • Handle missing user in user privilege APIs {pull}34575[#34575] (issue: {issue}34567[#34567])

  • Allow query caching by default again {pull}33328[#33328] (issue: {issue}33191[#33191])

  • Fix role query that can match nested documents {pull}32705[#32705]

  • Make get all app privs requires "*" permission {pull}32460[#32460]

CAT APIs
  • Fix potential NPE in _cat/shards/ with partial CommonStats {pull}33858[#33858]

  • Cat apis: Fix index creation time to use strict date format {pull}32510[#32510] (issue: {issue}32466[#32466])

CRUD
  • Fix DeleteRequest validation for nullable or empty id/type {pull}35314[#35314] (issue: {issue}35297[#35297])

  • Fix NOOP bulk updates {pull}32819[#32819] (issues: {issue}31821[#31821], {issue}32808[#32808])

Circuit Breakers
  • Make accounting circuit breaker settings dynamic {pull}34372[#34372] (issue: {issue}34368[#34368])

Core
  • Fix AutoQueueAdjustingExecutorBuilder settings validation {pull}33922[#33922]

  • Fix Javadoc issues in 6.x for JDK11 {pull}33579[#33579]

  • Core: Add java time xcontent serializers {pull}33120[#33120] (issue: {issue}31853[#31853])

  • Protect scheduler engine against throwing listeners {pull}32998[#32998]

  • Fix content type detection with leading whitespace {pull}32632[#32632] (issue: {issue}32357[#32357])

Distributed
  • Only notify ready global checkpoint listeners {pull}33690[#33690]

  • Enable global checkpoint listeners to timeout {pull}33620[#33620] (issue: {issue}32696[#32696])

  • Fix race between replica reset and primary promotion {pull}32442[#32442] (issues: {issue}32118[#32118], {issue}32304[#32304], {issue}32431[#32431])

Engine
  • Acquire seacher on closing engine should throw AlreadyClosedException {pull}33331[#33331] (issue: {issue}33330[#33330])

  • Trim unreferenced translog when the safe commit advanced {pull}32967[#32967] (issues: {issue}28140[#28140], {issue}32089[#32089])

  • All Translog inner closes should happen after tragedy exception is set {pull}32674[#32674] (issue: {issue}32526[#32526])

Geo
  • Fix north pole overflow error in GeoHashUtils.bbox() {pull}32891[#32891] (issue: {issue}32857[#32857])

  • Use the determinant formula for calculating the orientation of a polygon {pull}27967[#27967]

Index APIs
  • Make XContentBuilder in AliasActions build is_write_index field {pull}35071[#35071]

  • Do not update number of replicas on no indices {pull}34481[#34481]

  • [Security] Get Alias API wildcard exclusion with Security {pull}34144[#34144] (issues: {issue}33518[#33518], {issue}33805[#33805])

  • Allow to clear the fielddata cache per field {pull}33807[#33807] (issue: {issue}33798[#33798])

  • CORE: Make Pattern Exclusion Work with Aliases {pull}33518[#33518] (issue: {issue}33395[#33395])

  • Fix IndexMetaData loads after rollover {pull}33394[#33394] (issue: {issue}33316[#33316])

  • Copy missing segment attributes in getSegmentInfo {pull}32396[#32396]

Ingest
  • INGEST: Create Index Before Pipeline Execute {pull}32786[#32786] (issue: {issue}32758[#32758])

Java High Level REST Client
  • HLRC: Fixing bug when getting a missing pipeline {pull}34286[#34286] (issue: {issue}34119[#34119])

  • Aggregations/HL Rest client fix: missing scores {pull}32774[#32774] (issue: {issue}32770[#32770])

  • HLRC: Ban LoggingDeprecationHandler {pull}32756[#32756] (issue: {issue}32151[#32151])

  • HLRC: Move commercial clients from XPackClient {pull}32596[#32596]

  • High-level client: fix clusterAlias parsing in SearchHit {pull}32465[#32465]

  • REST high-level client: parse back _ignored meta field {pull}32362[#32362]

License
  • Address license state update/read thread safety {pull}33396[#33396]

Logging
  • Logging: Configure the node name when we have it {pull}32983[#32983] (issue: {issue}32793[#32793])

Machine Learning
  • Handle pre-6.x time fields {pull}34373[#34373]

  • Fix updating opened jobs scheduled events {pull}32881[#32881] (issue: {issue}31651[#31651])

  • Clear Job#finished_time when it is opened {pull}32755[#32755]

  • Fix thread leak when waiting for job flush {pull}32541[#32541] (issue: {issue}32196[#32196])

Mapping
  • Fix field mapping updates with similarity {pull}33634[#33634] (issue: {issue}33611[#33611])

  • Ensure that _exists queries on keyword fields use norms when they’re available. {pull}33006[#33006]

  • Make sure that field collapsing supports field aliases. {pull}32648[#32648] (issue: {issue}32623[#32623])

  • Improve the error message when an index is incompatible with field aliases. {pull}32482[#32482]

Monitoring
  • Typo in x-pack template for thread_pool.management {pull}34224[#34224]

Network
  • NETWORKING: Add SSL Handler before other Handlers {pull}34636[#34636] (issue: {issue}33998[#33998])

  • Handle null SSLSessions during invalidation {pull}34130[#34130] (issue: {issue}32124[#32124])

  • Support PKCS#11 tokens as keystores and truststores {pull}34063[#34063] (issue: {issue}11[#11])

  • Parse PEM Key files leniantly {pull}33173[#33173] (issue: {issue}33168[#33168])

  • NETWORKING: http.publish_host Should Contain CNAME {pull}32806[#32806] (issue: {issue}22029[#22029])

  • NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS {pull}32764[#32764] (issue: {issue}28858[#28858])

  • Release requests in cors handle {pull}32410[#32410]

  • Release requests in cors handler {pull}32364[#32364]

Packaging
  • Fix use of hostname in Windows service {pull}34193[#34193]

  • Add temporary directory cleanup workarounds {pull}32615[#32615] (issue: {issue}31732[#31732])

Percolator
  • Ignore date ranges containing 'now' when pre-processing a percolator query {pull}35160[#35160]

REST API
  • Core: Fix IndicesSegmentResponse.toXcontent() serialization {pull}33414[#33414] (issue: {issue}29120[#29120])

Recovery
  • Resync fails to notify on unavaiable exceptions {pull}33615[#33615] (issues: {issue}31179[#31179], {issue}33613[#33613])

  • Ensure to generate identical NoOp for the same failure {pull}33141[#33141] (issue: {issue}32986[#32986])

Rollup
  • [Rollup] Proactively resolve index patterns in RollupSearch endoint {pull}34930[#34930] (issue: {issue}34828[#34828])

  • Address BWC bug due to default metrics in (#34764) {pull}34810[#34810] (issue: {issue}34764[#34764])

  • Allowing {index}/_xpack/rollup/data to accept comma delimited list {pull}34115[#34115]

  • [Rollup] Fix Caps Comparator to handle calendar/fixed time {pull}33336[#33336] (issue: {issue}32052[#32052])

  • [Rollup] Better error message when trying to set non-rollup index {pull}32965[#32965]

  • [Rollup] Return empty response when aggs are missing {pull}32796[#32796] (issue: {issue}32256[#32256])

  • [Rollup] Improve ID scheme for rollup documents {pull}32558[#32558] (issue: {issue}32372[#32372])

SQL
  • SQL: Fix null handling for AND and OR in SELECT {pull}35277[#35277] (issue: {issue}35240[#35240])

  • SQL: Handle null literal for AND and OR in WHERE {pull}35236[#35236] (issue: {issue}35088[#35088])

  • SQL: Introduce NotEquals node to simplify expressions {pull}35234[#35234] (issues: {issue}35210[#35210], {issue}35233[#35233])

  • SQL: handle wildcard expansion on incorrect fields {pull}35134[#35134] (issue: {issue}35092[#35092])

  • SQL: Fix null handling for IN ⇒ painless script {pull}35124[#35124] (issues: {issue}35108[#35108], {issue}35122[#35122])

  • SQL: Register missing processors {pull}35121[#35121] (issue: {issue}35119[#35119])

  • SQL: Fix NPE thrown if HAVING filter evals to null {pull}35108[#35108] (issue: {issue}35107[#35107])

  • SQL: Proper handling of nested fields at the beginning of the columns list {pull}35068[#35068] (issue: {issue}32951[#32951])

  • SQL: Fix incorrect AVG data type {pull}34948[#34948] (issue: {issue}33773[#33773])

  • SQL: Add CAST and CONVERT to SHOW FUNCTIONS {pull}34940[#34940] (issue: {issue}34939[#34939])

  • SQL: Handle aggregation for null group {pull}34916[#34916] (issue: {issue}34896[#34896])

  • SQL: Provide null-safe scripts for Not and Neg {pull}34877[#34877] (issue: {issue}34848[#34848])

  • SQL: Return error with ORDER BY on non-grouped. {pull}34855[#34855] (issue: {issue}34590[#34590])

  • SQL: Fix queries with filter resulting in NO_MATCH {pull}34812[#34812] (issue: {issue}34613[#34613])

  • SQL: Fix edge case: <field> IN (null) {pull}34802[#34802] (issue: {issue}34750[#34750])

  • SQL: Verifier allows aliases aggregates for sorting {pull}34773[#34773] (issue: {issue}34607[#34607])

  • SQL: the SSL default configuration shouldn’t override the https protocol if used {pull}34635[#34635] (issue: {issue}33817[#33817])

  • JDBC: Fix artifactId in pom {pull}34478[#34478] (issue: {issue}34399[#34399])

  • SQL: Fix grammar for * in arithm expressions {pull}34176[#34176] (issue: {issue}33957[#33957])

  • SQL: Fix function resolution {pull}34137[#34137] (issue: {issue}34114[#34114])

  • SQL: Fix query translation of GroupBy with Having {pull}34010[#34010] (issue: {issue}33520[#33520])

  • SQL: Prevent StackOverflowError when parsing large statements {pull}33902[#33902] (issue: {issue}32942[#32942])

  • SQL: Fix issue with options for QUERY() and MATCH(). {pull}33828[#33828] (issue: {issue}32602[#32602])

  • SQL: Return correct catalog separator in JDBC {pull}33670[#33670] (issue: {issue}33654[#33654])

  • SQL: Fix result column names for CAST {pull}33604[#33604] (issue: {issue}33571[#33571])

  • SQL: Fix result column names for arithmetic functions {pull}33500[#33500] (issues: {issue}14[#14], {issue}31869[#31869])

  • SQL: Fix bug in REPLACE function. Adds more tests to all string functions {pull}33478[#33478]

  • SQL: handle differently security connection related errors in the CLI {pull}33255[#33255] (issue: {issue}33230[#33230])

  • SQL: prevent duplicate generation for repeated aggs {pull}33252[#33252] (issue: {issue}30287[#30287])

  • SQL: Enable aggregations to create a separate bucket for missing values {pull}32832[#32832] (issue: {issue}32831[#32831])

  • SQL: Bug fix for the optional "start" parameter usage inside LOCATE function {pull}32576[#32576] (issue: {issue}32554[#32554])

  • SQL: Minor fix for javadoc {pull}32573[#32573] (issue: {issue}32553[#32553])

Scripting
  • Scripting: Add back lookup vars in score script {pull}34833[#34833]

  • Scripting: Add back params._source access in scripted metric aggs {pull}34777[#34777] (issue: {issue}33884[#33884])

  • Test: Fix last reference to SearchScript {pull}34731[#34731] (issue: {issue}34683[#34683])

  • Ensure map keys cannot be self referencing {pull}34569[#34569]

  • [Painless] Add a Map for java names to classes for use in the custom classloader {pull}34424[#34424]

  • [Painless] Allow statically imported methods without whitelisted class {pull}34370[#34370]

  • Painless: Remove caching of Painless scripts {pull}34116[#34116]

  • Painless: Fix Bindings Bug {pull}33274[#33274]

  • Painless: Fix Semicolon Regression {pull}33212[#33212] (issue: {issue}33193[#33193])

  • Scripting: Fix painless compiler loader to know about context classes {pull}32385[#32385]

Search
  • Fix inner_hits retrieval when stored fields are disabled {pull}34652[#34652] (issues: {issue}32941[#32941], {issue}33018[#33018])

  • Fix cross fields mode of the query_string query {pull}34216[#34216] (issue: {issue}34215[#34215])

  • Support 'string'-style queries on metadata fields when reasonable. {pull}34089[#34089] (issue: {issue}34062[#34062])

  • Improves doc values format deprecation message {pull}33576[#33576] (issue: {issue}33572[#33572])

  • Fix nested _source retrieval with includes/excludes {pull}33180[#33180] (issues: {issue}33163[#33163], {issue}33170[#33170])

  • Fix quoted exists query {pull}33019[#33019] (issue: {issue}28922[#28922])

  • Fix multi fields empty query {pull}33017[#33017] (issue: {issue}33009[#33009])

  • XContentBuilder to handle BigInteger and BigDecimal {pull}32888[#32888] (issue: {issue}32395[#32395])

  • Do NOT allow termvectors on nested fields {pull}32728[#32728] (issues: {issue}21625[#21625], {issue}32652[#32652])

  • Cross-cluster search: preserve cluster alias in shard failures {pull}32608[#32608]

Security
  • Security: use x-pack config files when present {pull}33688[#33688] (issue: {issue}33464[#33464])

  • Security: use default scroll keepalive {pull}33639[#33639]

  • Enable FIPS140LicenseBootstrapCheck {pull}32903[#32903] (issue: {issue}32437[#32437])

Settings
  • CORE: Validate Type for String Settings {pull}33503[#33503] (issue: {issue}33135[#33135])

  • Fix deprecated setting specializations {pull}33412[#33412]

  • Apply settings filter to get cluster settings API {pull}33247[#33247]

Snapshot/Restore
  • Register Azure max_retries setting {pull}35286[#35286]

  • Do not override named S3 client credentials {pull}33793[#33793] (issue: {issue}33769[#33769])

  • Ensure fully deleted segments are accounted for correctly {pull}33757[#33757] (issues: {issue}32844[#32844], {issue}33689[#33689], {issue}33755[#33755])

Suggesters
  • Fix completion suggester’s score tie-break {pull}34508[#34508] (issue: {issue}34378[#34378])

  • Null completion field should not throw IAE {pull}33268[#33268]

Transport API
  • Fix serialization of empty field capabilities response {pull}33263[#33263]

Watcher
  • watcher: Fix integration tests to ensure correct start/stop of Watcher {pull}35271[#35271] (issues: {issue}29877[#29877], {issue}30705[#30705], {issue}33291[#33291], {issue}34448[#34448], {issue}34462[#34462])

  • Make Watcher validation message copy/pasteable {pull}33497[#33497] (issue: {issue}33369[#33369])

  • Watcher: Reload properly on remote shard change {pull}33167[#33167]

  • Watcher: Fix race condition when reloading watches {pull}33157[#33157]

  • Guard against null in email admin watches {pull}32923[#32923] (issue: {issue}32590[#32590])

  • Watcher: Properly find next valid date in cron expressions {pull}32734[#32734]

ZenDiscovery
  • Fix logging of cluster state update descriptions {pull}34182[#34182] (issue: {issue}28941[#28941])

Regressions

Search
  • Preserve index_uuid when creating QueryShardException {pull}32677[#32677] (issue: {issue}32608[#32608])

Upgrades

Core
  • CORE: Upgrade to Jackson 2.8.11 {pull}32670[#32670] (issue: {issue}30352[#30352])

  • Dependencies: Upgrade to joda time 2.10 {pull}32160[#32160]

Ingest
  • Update geolite2 database in ingest geoip plugin {pull}33840[#33840]

Logging
  • LOGGING: Upgrade to Log4J 2.11.1 (#32616) {pull}32656[#32656] (issues: {issue}27300[#27300], {issue}32537[#32537])

  • LOGGING: Upgrade to Log4J 2.11.1 {pull}32616[#32616] (issues: {issue}27300[#27300], {issue}32537[#32537])

Network
  • NETWORKING: Upgrade Netty to 4.1.30 {pull}34417[#34417] (issue: {issue}34411[#34411])

  • NETWORKING: Upgrade to Netty 4.1.29 {pull}33984[#33984]

Search
  • Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d {pull}32730[#32730]

  • Upgrade to Lucene-7.5.0-snapshot-608f0277b0 {pull}32390[#32390]

Watcher
  • Dependencies: Update javax.mail in watcher to 1.6.2 {pull}33664[#33664]

{es} version 6.4.3

Enhancements

Machine learning
  • Changes linker options on macOS to allow Homebrew installs ({ml-pull}225[#225])

Bug fixes

Aggregations
  • Check self references in metric agg after last doc collection (#33593) {pull}34001[#34001]

Authentication
  • ListenableFuture should preserve ThreadContext (CVE-2018-17244) {pull}34394[#34394]

  • Allow an AuthenticationResult to return metadata {pull}34382[#34382] (issues: {issue}34290[#34290], {issue}34332[#34332])

  • Preserve thread context during authentication {pull}34290[#34290]

Circuit Breakers
  • Make accounting circuit breaker settings dynamic {pull}34372[#34372] (issue: {issue}34368[#34368])

Java High Level REST Client
  • HLRC: Fixing bug when getting a missing pipeline {pull}34286[#34286] (issue: {issue}34119[#34119])

Machine learning
  • Fixes the cause of hard_limit memory errors for jobs with bucket spans greater than one day ({ml-pull}243[#243])

  • Rules that trigger the skip_model_update action should also apply to the anomaly model. This fixes an issue where anomaly scores of results that triggered the rule would decrease if they occurred frequently. {ml-pull}222[#222] (issue: {ml-issue}217[#217])

Network
  • Support PKCS#11 tokens as keystores and truststores {pull}34063[#34063] (issue: {issue}11[#11])

  • Correctly handle PKCS#11 tokens for system keystore {pull}33460[#33460] (issues: {issue}11[#11], {issue}33459[#33459])

Security
  • Security: use x-pack config files when present {pull}33688[#33688] (issue: {issue}33464[#33464])

{es} version 6.4.2

Enhancements

Search
  • Add a limit for graph phrase query expansion {pull}34031[#34031]

Bug fixes

Core
  • Fix AutoQueueAdjustingExecutorBuilder settings validation {pull}33922[#33922]

Index APIs
  • Allow to clear the fielddata cache per field {pull}33807[#33807] (issue: {issue}33798[#33798])

Network
  • Support PKCS#11 tokens as keystores and truststores [OPEN] {pull}34063[#34063] (issue: {issue}11[#11])

  • Correctly handle PKCS#11 tokens for system keystore {pull}33460[#33460] (issues: {issue}11[#11], {issue}33459[#33459])

Security
  • Security: use default scroll keepalive {pull}33639[#33639]

Snapshot/Restore
  • Do not override named S3 client credentials {pull}33793[#33793] (issue: {issue}33769[#33769])

Watcher
  • Watcher: Ensure triggered watch deletion is sync {pull}33799[#33799]

{es} version 6.4.1

Bug Fixes

Rollover

{es} can once again start if any shards on the node have been rolled over. {pull}33394[#33394]

{es} version 6.4.0

Known issues

{es} 6.4.0 fails to start when PEM encoded private key files that have been exported from PKCS#12 keystores are in use. These files can be identified by the existence of lines that start with either Bag Attributes or Key Attributes in the beginning of the key file.

These keys can be converted to a supported format using one of the following methods:

  • Remove the extra lines from the beginning of the file so that the file starts with the line that starts with -----BEGIN

  • Use openssl to convert it, i.e: openssl pkcs8 -in old_format.key -topk8 -nocrypt -out new_format.key assuming the key was not password protected.

Breaking Changes

Plugins
  • Plugins: Remove meta plugins {pull}30670[#30670]

Search
  • Reject regex search if regex string is too long (#28344) {pull}28542[#28542] (issue: {issue}28344[#28344])

Task Management
  • Remove metadata customs that can break serialization {pull}30945[#30945] (issues: {issue}30731[#30731])

Breaking Java changes

Authentication
  • Configurable password hashing algorithm/cost {pull}31234[#31234], {pull}32092[#32092] (issue: {issue}31723[#31723])

Discovery-Plugins
  • Allow multiple unicast host providers {pull}31509[#31509]

Java High Level REST Client
  • Add x-pack-info API {pull}31870[#31870]

Java Low Level REST Client
  • Support host selection {pull}30523[#30523] (issue: {issue}21888[#21888])

Deprecations

Analysis
  • Correct spelling of AnalysisPlugin#requriesAnalysisSettings {pull}32025[#32025]

  • Deprecate nGram and edgeNGram names for ngram filters {pull}30209[#30209]

Index APIs
  • Add deprecation warning for default shards {pull}30587[#30587]

  • Deprecate not copy settings and explicitly disallow {pull}30404[#30404] (issues: {issue}28347[#28347])

Java High Level REST Client
  • Add high-level client methods that accept RequestOptions {pull}31069[#31069]

Java Low Level REST Client
  • Client: Deprecate many argument performRequest {pull}30315[#30315]

Mapping
  • Deprecate unindexed phrases {pull}31072[#31072]

Scripting
  • Deprecate accepting malformed requests in stored script API {pull}28939[#28939] (issue: {issue}27612[#27612])

Search
  • In the field capabilities API, deprecate support for providing fields in the request body. {pull}30157[#30157]

Suggesters
  • Deprecates indexing and querying a context completion field without context {pull}30712[#30712] (issue: {issue}29222[#29222])

New Features

Aggregations
  • Add WeightedAvg metric aggregation {pull}31037[#31037] (issue: {issue}15731[#15731])

  • Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg {pull}29594[#29594]

  • Add missing_bucket option in the composite agg {pull}29465[#29465] (issue: {issue}29380[#29380])

Analysis
  • Expose lucene’s RemoveDuplicatesTokenFilter {pull}31275[#31275]

  • Multiplexing token filter {pull}31208[#31208]

  • Adds a new analysis plugin called analysis_nori that exposes the Lucene Korean analysis module. ({pull}30397[#30397])

  • Adding a char_group tokenizer {pull}24186[#24186]

Authentication
  • Add Kerberos authentication support {pull}32263[#32263] (issue: {issue}30243[#30243])

Authorization
  • Introduce Application Privileges with support for Kibana RBAC {pull}32309[#32309]

Java High Level REST Client
  • Add analyze API to high-level rest client {pull}31577[#31577] (issue: {issue}27205[#27205])

  • Add support for search templates to the high-level REST client. {pull}30473[#30473]

  • Rest High Level client: Add List Tasks {pull}29546[#29546] (issue: {issue}27205[#27205])

Machine learning
  • Detectors now support {ml-docs}/ml-rules.html[custom rules] that enable the user to improve machine learning results by providing some domain-specific knowledge in the form of rule. {ml-pull}119[#119], {pull}31110[#31110], {pull}31294[#31294] (issue: {issue}31110[#31110])

  • Reverse engineer Grok patterns from categorization results {pull}30125[#30125]

Mapping
  • Add support for field aliases. {pull}32172[#32172] (issues: {issue}23714[#23714], {issue}31372[#31372])

  • Add an option to split keyword field on whitespace at query time {pull}30691[#30691] (issue: {issue}30393[#30393])

  • The new _ignored field enables you to know which fields got ignored at index time because of the ignore_malformed option. ({pull}29658[#29658]) (issue: {issue}29494[#29494])

Network
  • Introduce client feature tracking {pull}31020[#31020] (issue: {issue}30731[#30731])

Plugins
  • Reload secure settings for plugins - backport (#31383) {pull}31481[#31481] (issue: {issue}29135[#29135])

SQL
  • SQL: Support for escape sequences {pull}31884[#31884] (issue: {issue}31883[#31883])

Scripting
  • Add more contexts to painless execute api {pull}30511[#30511]

Search
  • Index phrases {pull}30450[#30450]

  • Add a format option to docvalue_fields. {pull}29639[#29639] (issue: {issue}27740[#27740])

Watcher
  • Make watcher settings reloadable {pull}31746[#31746]

Enhancements

Aggregations
  • Fix wrong NaN check in MovingFunctions#stdDev() {pull}31888[#31888]

  • Mitigate date histogram slowdowns with non-fixed timezones. {pull}30534[#30534] (issue: {issue}28727[#28727])

  • Build global ordinals terms bucket from matching ordinals {pull}30166[#30166] (issue: {issue}30117[#30117])

Analysis
  • Add exclusion option to keep_types token filter {pull}32012[#32012] (issue: {issue}29277[#29277])

  • Added lenient flag for synonym token filter {pull}31484[#31484] (issue: {issue}30968[#30968])

  • Consistent encoder names {pull}29492[#29492]

Audit
  • Add opaque_id to audit logging {pull}31878[#31878] (issue: {issue}31521[#31521])

Authentication
  • Support RequestedAuthnContext {pull}31238[#31238] (issue: {issue}29995[#29995])

  • Make native realm usage stats accurate {pull}30824[#30824]

  • Limit user to single concurrent auth per realm {pull}30794[#30794] (issue: {issue}30355[#30355])

  • SAML: Process only signed data {pull}30641[#30641]

CRUD
  • Support for remote path in reindex api {pull}31290[#31290] (issue: {issue}22913[#22913])

  • Don’t swallow exceptions on replication {pull}31179[#31179] (issue: {issue}28571[#28571])

Circuit Breakers
  • Enhance Parent circuit breaker error message {pull}32056[#32056]

  • Split CircuitBreaker-related tests {pull}31659[#31659]

Core
  • Change ObjectParser exception {pull}31030[#31030] (issue: {issue}30605[#30605])

Discovery-Plugins
  • Add support for AWS session tokens {pull}30414[#30414] (issues: {issue}16428[#16428])

Distributed
  • Avoid sending duplicate remote failed shard requests {pull}31313[#31313]

Engine
  • Adjust translog after versionType is removed in 7.0 {pull}32020[#32020] (issue: {issue}31945[#31945])

  • Enable engine factory to be pluggable {pull}31183[#31183]

  • Allow to trim all ops above a certain seq# with a term lower than X {pull}30176[#30176] (issue: {issue}10708[#10708])

  • Do not add noop from local translog to translog again {pull}29637[#29637]

Geo
  • Add support for ignore_unmapped to geo sort {pull}31153[#31153] (issue: {issue}28152[#28152])

Highlighting
  • Bypass highlight query terms extraction on empty fields {pull}32090[#32090]

Index APIs
  • Add Index UUID to /_stats Response {pull}31871[#31871] (issue: {issue}31791[#31791])

  • add support for write index resolution when creating/updating documents {pull}31520[#31520]

  • Allow copying source settings on index resize operations {pull}30255[#30255] (issue: {issue}28347[#28347])

Ingest
  • Extend KV Processor (#31789) {pull}32232[#32232] (issue: {issue}31786[#31786])

  • Make a few Processors callable by Painless {pull}32170[#32170]

  • date_index_name processor template resolution {pull}31841[#31841]

  • Introduction of a bytes processor {pull}31733[#31733]

  • Extend allowed characters for grok field names {pull}31653[#31653], {pull}31722[#31722] (issue: {issue}21745[#21745])

  • Ingest: Add ignore_missing option to RemoveProc {pull}31693[#31693] (issues: {issue}23086[#23086])

  • Enable Templated Fieldnames in Rename {pull}31690[#31690] (issue: {issue}29657[#29657])

  • Add region ISO code to GeoIP Ingest plugin {pull}31669[#31669]

  • Extend allowed characters for grok field names {pull}31653[#31653] (issue: {issue}21745[#21745])

  • Add ingest-attachment support for per document indexed_chars limit {pull}31352[#31352]

Java High Level REST Client
  • Add Snapshots Status API to High Level Rest Client {pull}32295[#32295], {pull}31515[#31515]

  • Add put watch action {pull}32026[#32026], {pull}32191[#32191] (issue: {issue}29827[#29827])

  • Add Get Snapshots High Level REST API {pull}31980[#31980]

  • Add X-Pack usage api {pull}31975[#31975]

  • Check that client methods match API defined in the REST spec {pull}31825[#31825]

  • Clean Up Snapshot Create Rest API {pull}31779[#31779]

  • Add cluster get settings API {pull}31706[#31706] (issue: {issue}27205[#27205])

  • Add get index API {pull}31703[#31703] (issues: {issue}27205[#27205])

  • Turn GetFieldMappingsResponse to ToXContentObject {pull}31544[#31544]

  • Add Get Snapshots High Level REST API {pull}31537[#31537] (issue: {issue}27205[#27205])

  • Add Snapshots Status API to High Level Rest Client {pull}31515[#31515] (issue: {issue}27205[#27205])

  • Add get field mappings to High Level REST API Client {pull}31423[#31423] (issue: {issue}27205[#27205])

  • Add delete snapshot High Level REST API {pull}31393[#31393] (issue: {issue}27205[#27205])

  • Add explain High Level REST API {pull}31387[#31387] (issue: {issue}27205[#27205])

  • Add get stored script and delete stored script to high level REST API {pull}31355[#31355] (issue: {issue}27205[#27205])

  • Add Create Snapshot to High-Level Rest Client {pull}31215[#31215]

  • Add get index templates API {pull}31161[#31161] (issue: {issue}27205[#27205])

  • Add simulate pipeline API {pull}31158[#31158] (issue: {issue}27205[#27205])

  • Add validate query API {pull}31077[#31077] (issue: {issue}27205[#27205])

  • Moved pipeline APIs to ingest namespace {pull}31027[#31027]

  • List tasks failure to not lose nodeId {pull}31001[#31001]

  • Add Verify Repository High Level REST API {pull}30934[#30934] (issue: {issue}27205[#27205])

  • Move list tasks API under tasks namespace {pull}30906[#30906] (issue: {issue}29546[#29546])

  • Add get mappings support to high-level rest client {pull}30889[#30889] (issue: {issue}27205[#27205])

  • Fix AliasMetaData#fromXContent parsing {pull}30866[#30866] (issue: {issue}28799[#28799])

  • Add delete ingest pipeline API {pull}30865[#30865] (issues: {issue}27205[#27205])

  • Add get ingest pipeline API {pull}30847[#30847] (issues: {issue}27205[#27205])

  • Add MultiSearchTemplate support to High Level Rest client {pull}30836[#30836]

  • Add put ingest pipeline API {pull}30793[#30793] (issue: {issue}27205[#27205])

  • Add cancel task API {pull}30745[#30745] (issue: {issue}27205[#27205])

  • Add Delete Repository High Level REST API {pull}30666[#30666] (issue: {issue}27205[#27205])

  • Add synced flush API {pull}30650[#30650] (issues: {issue}27205[#27205])

  • Add PUT Repository High Level REST API {pull}30501[#30501] (issue: {issue}27205[#27205])

  • Allow caller to set per request options {pull}30490[#30490]

  • Add put index template api to high level rest client {pull}30400[#30400] (issue: {issue}27205[#27205])

  • Add GET Repository High Level REST API {pull}30362[#30362] (issue: {issue}27205[#27205])

  • Add support for field capabilities to the high-level REST client. {pull}29664[#29664] (issue: {issue}27205[#27205])

  • Add Cluster Health API {pull}29331[#29331] (issue: {issue}27205[#27205])

  • Add Get Settings API support to java high-level rest client {pull}29229[#29229]

  • Add Get Aliases API to the high-level REST client {pull}28799[#28799] (issue: {issue}27205[#27205])

  • Register ERR metric with NamedXContentRegistry {pull}32320[#32320]

Java Low Level REST Client
  • Node selector per client rather than per request {pull}31471[#31471]

  • NodeSelector for node attributes {pull}31296[#31296]

  • Replace Request#setHeaders with addHeader {pull}30588[#30588]

  • Preserve REST client auth despite 401 response {pull}30558[#30558]

  • Add String flavored setEntity {pull}30447[#30447]

  • Refactor Sniffer and make it testable {pull}29638[#29638] (issues: {issue}25701[#25701], {issue}27697[#27697])

  • Add Request object flavored methods {pull}29623[#29623]

License
  • Reuse expiration date of trial licenses {pull}31033[#31033], {pull}30950[#30950] (issue: {issue}30882[#30882])

Logging
  • Add x-opaque-id to search slow logs {pull}31539[#31539] (issue: {issue}31521[#31521])

Machine learning
  • If a {ml} datafeed is configured to use cross cluster search to retrieve data, the remote clusters must have {xpack} installed and a valid licence for {ml}. If the licence requirements are not met, datafeeds using cross cluster search will not start. {pull}31247[#31247]

  • Use default request durability for .ml-state index {pull}32233[#32233]

  • Return statistics about forecasts as part of the job stats and usage API {pull}31647[#31647] (issue: {issue}31395[#31395])

  • Add description to ML filters {pull}31330[#31330]

  • Clean left behind model state docs {pull}30659[#30659] (issue: {issue}30551[#30551])

  • Hide internal job update options from the REST API {pull}30537[#30537]

  • Provide tmp storage for forecasting and possibly any {ml} native jobs {pull}30399[#30399]

  • Improves and uses periodic boundary condition for seasonal component modeling ({ml-pull}84[#84])

  • Improves robustness with respect to outliers in detection and initialization of seasonal components ({ml-pull}90[#90] (issue: {ml-issue}87[#87]))

  • Improves behavior when there are abrupt changes in the seasonal components present in a time series ({ml-pull}91[#91] (issue: {ml-issue}6[#6]))

  • Adds explicit change point detection and modeling ({ml-pull}92[#92])

  • Improves partition analysis memory usage ({ml-pull}97[#97])

  • Reduces model memory by storing state for periodicity testing in a compressed format ({ml-pull}104[#104], {ml-pull}100[#100])

  • Improves the accuracy of model memory control ({ml-pull}125[#125], {ml-issue}122[#122])

  • Improves adaption of the modeling of cyclic components to very localized features ({ml-pull}138[#138], {ml-pull}134[#134])

  • Reduces the memory consumed by distribution models ({ml-pull}162[#162], {ml-pull}146[#146])

  • Forecasting of large machine learning jobs is now supported by temporarily storing model state on disk ({ml-pull}89[#89])

  • Secures the machine learning processes by preventing system calls such as fork and exec. The Linux implementation uses Seccomp BPF (secure computing with Berkeley Packet Filters) to intercept system calls and is available in kernels since 3.5. On Windows, Job Objects prevent new processes being created and macOS uses the sandbox functionality ({ml-pull}106[#106], {ml-pull}98[#98])

  • Fixes a bug that caused underestimation of the memory used by shared pointers. Also reduces the memory consumed by unnecessary reference counting ({ml-pull}121[#121], {ml-pull}108, {ml-pull}115[#115])

  • Reduces model memory by storing the state for testing predictive calendar features in a compressed format ({ml-pull}137[#137], {ml-pull}127[#127])

  • Always combine duplicate samples when updating population models ({ml-pull}74[#74])

  • Speeds up trend model component prediction ({ml-pull}73[#73])

  • Encodes distribution model weight style by offset in a fixed size weight array ({ml-pull}54[#54])

Mapping
  • Remove RestGetAllMappingsAction {pull}31129[#31129]

  • Add a doc value format to binary fields. {pull}30860[#30860] (issue: {issue}30831[#30831])

Monitoring
  • _cluster/state should always return cluster_uuid {pull}30143[#30143]

Network
  • Backport SSL context names ({pull}32223[#32223], {pull}30953[#30953])

  • Remove client connections from TcpTransport {pull}31886[#31886] (issue: {issue}31835[#31835])

  • Support multiple system store types {pull}31650[#31650]

  • Use remote client in TransportFieldCapsAction {pull}30838[#30838]

  • Replace custom reloadable Key/TrustManager {pull}30509[#30509]

  • Derive max composite buffers from max content len {pull}29448[#29448]

Packaging
  • Set elasticsearch user to have non-existent homedir {pull}29007[#29007] (issue: {issue}14453[#14453])

Plugins
  • Verify signatures on official plugins {pull}30800[#30800]

Ranking
  • Rename ranking evaluation quality_level to metric_score {pull}32168[#32168]

  • Rename ranking evaluation response unknown_docs section {pull}32166[#32166]

  • Add Expected Reciprocal Rank metric {pull}31891[#31891] (issue: {issue}29653[#29653])

  • Add details section for dcg ranking metric {pull}31177[#31177]

  • Move templated _rank_eval tests {pull}30679[#30679] (issue: {issue}30628[#30628])

  • Forbid expensive query parts in ranking evaluation {pull}30151[#30151] (issue: {issue}29674[#29674])

Rollup
  • Rollup now indexes null values, meaning a single "unified" job for heterogeneous data is now the recommended pattern. {pull}31402[#31402]

  • Rollup Search endpoint now supports the terms query. {pull}30973[#30973])

  • Allow rollup job creation only if cluster is X-Pack ready. {pull}30963[#30963]

  • Rollups no longer allow patterns that match its rollup_index, which can lead to strange errors. {pull}30491[#30491]

  • A new API allows getting the rollup capabilities of specific rollup indices, rather than by the target pattern. {pull}30401[#30401]

  • Validation errors thrown while creating a rollup job are now a specialization of the previous ActionRequestValidationException, which makes it easier to catch. The new exception is RollupActionRequestValidationException. {pull}30339[#30339]

  • Validate timezone in range queries to ensure they match the selected job when searching. {pull}30338[#30338]

SQL
  • Allow LEFT and RIGHT as function names {pull}32066[#32066] (issue: {issue}32046[#32046])

  • Add support for single parameter text manipulating functions {pull}31874[#31874] (issue: {issue}31604[#31604])

  • Remove restriction for single column grouping {pull}31818[#31818] (issue: {issue}31793[#31793])

  • Make a single JDBC driver jar {pull}31012[#31012] (issue: {issue}29856[#29856])

  • Remove the last remaining server dependencies from JDBC {pull}30771[#30771] (issue: {issue}29856[#29856])

  • Whitelist SQL utility class for better scripting {pull}30681[#30681] (issue: {issue}29832[#29832])

  • Improve compatibility with MS query {pull}30516[#30516] (issue: {issue}30398[#30398])

  • Reduce number of ranges generated for comparisons {pull}30267[#30267] (issue: {issue}30017[#30017])

  • Teach the CLI to ignore empty commands {pull}30265[#30265] (issue: {issue}30000[#30000])

  • JDBC driver prepared statement set* methods {pull}31494[#31494] (issue: {issue}31493[#31493])

Scripting
  • Handle missing values in painless {pull}[#30975], {pull}31903[#31903] (issue: {issue}29286[#29286])

Search
  • Force execution of fetch tasks {pull}31974[#31974] (issue: {issue}29442[#29442])

  • Add second level of field collapsing {pull}31808[#31808] (issue: {issue}24855[#24855])

  • Remove QueryCachingPolicy#ALWAYS_CACHE {pull}31451[#31451]

  • Cross cluster search: don’t proxy requests for already connected node {pull}31273[#31273]

  • Reject long regex in query_string {pull}31136[#31136] (issue: {issue}28344[#28344])

  • Cross cluster search: do not use dedicated masters as gateways {pull}30926[#30926] (issue: {issue}30687[#30687])

  • Added max_expansion param to span_multi {pull}30913[#30913] (issue: {issue}27432[#27432])

  • Increase the maximum number of filters that may be in the cache. {pull}30655[#30655]

  • Improve explanation in rescore {pull}30629[#30629] (issue: {issue}28725[#28725])

Security
  • Introduce fips_mode setting and associated checks {pull}32326[#32326], {pull}32344[#32344]

  • Tribe: Add error with secure settings copied to tribe {pull}32298[#32298] (issue: {issue}32117[#32117])

  • Only auto-update license signature if all nodes ready {pull}30859[#30859] (issues: {issue}30731[#30731])

  • Limit the scope of BouncyCastle dependency {pull}30358[#30358]

  • Make licensing FIPS-140 compliant {pull}30251[#30251]

Settings
  • Add notion of internal index settings {pull}31286[#31286] (issue: {issue}29823[#29823])

  • Move RestGetSettingsAction to RestToXContentListener {pull}31101[#31101]

  • Harmonize include_defaults tests {pull}30700[#30700]

  • Fold RestGetAllSettingsAction in RestGetSettingsAction {pull}30561[#30561]

Snapshot/Restore
  • ECS Task IAM profile credentials ignored in repository-s3 plugin {pull}31864[#31864] (issues: {issue}26913[#26913])

  • Add write*Blob option to replace existing blob {pull}31729[#31729]

  • Fixture for Minio testing {pull}31688[#31688]

  • Do not check for object existence when deleting repository index files {pull}31680[#31680]

  • Remove extra check for object existence in repository-gcs read object {pull}31661[#31661] time an Azure object is accessed or modified {pull}31617[#31617]

  • Lazy snapshot repository initialization {pull}31606[#31606]

  • Do not check for S3 blob to exist before writing {pull}31128[#31128]

  • Remove extra checks from HdfsBlobContainer {pull}31126[#31126]

  • Allow date math for naming newly-created snapshots {pull}30479[#30479] (issue: {issue}7939[#7939] )

  • Use simpler write-once semantics for HDFS repository {pull}30439[#30439]

  • User proper write-once semantics for GCS repository {pull}30438[#30438]

  • Use stronger write-once semantics for Azure repository {pull}30437[#30437]

  • Use simpler write-once semantics for FS repository {pull}30435[#30435]

  • Do not fail snapshot when deleting a missing snapshotted file {pull}30332[#30332] (issue: {issue}28322[#28322])

  • Repository GCS plugin new client library {pull}30168[#30168] (issue: {issue}29259[#29259])

  • Fail snapshot operations early on repository corruption {pull}30140[#30140] (issues: {issue}29649[#29649])

  • Index name added to snapshot restore exception {pull}29604[#29604] (issue: {issue}27601[#27601])

  • Do not load global state when deleting a snapshot {pull}29278[#29278] (issue: {issue}28934[#28934])

  • Don’t load global state when only restoring indices {pull}29239[#29239] (issue: {issue}28934[#28934])

Stats
  • Add _coordinating_only for nodes resolving in nodes API {pull}30313[#30313] (issue: {issue}28831[#28831])

Store
  • Move caching of the size of a directory to StoreDirectory. {pull}30581[#30581]

Suggesters
  • Ignore empty completion input {pull}30713[#30713] (issue: {issue}23121[#23121])

Task Management
  • Make Persistent Tasks implementations version and feature aware {pull}31045[#31045] (issues: {issue}30731[#30731])

Transport API
  • Implemented XContent serialisation for GetIndexResponse {pull}31675[#31675]

  • Send client headers from TransportClient {pull}30803[#30803]

  • Modify state of VerifyRepositoryResponse for backwards compatibility {pull}30762[#30762]

Watcher
  • Clean up ensureWatchExists use {pull}31926[#31926]

  • Store username on watch execution {pull}31873[#31873] (issue: {issue}31772[#31772])

  • Consolidate setting update registration {pull}31762[#31762]

  • Add secure setting for watcher email password {pull}31620[#31620]

  • Slack message empty text {pull}31596[#31596] (issue: {issue}30071[#30071])

  • Move watcher-history version setting to _meta field {pull}30832[#30832] (issue: {issue}30731[#30731])

  • Only allow x-pack metadata if all nodes are ready {pull}30743[#30743] (issues: {issue}30731[#30731])

  • Configure HttpClient parallel sent requests {pull}30130[#30130]

  • Watcher: Make start/stop cycle more predictable and synchronous {pull}30118[#30118]

ZenDiscovery
  • Preserve response headers on cluster update task {pull}31421[#31421] (issues: {issue}31408[#31408])

  • Treat ack timeout more like a publish timeout {pull}31303[#31303]

  • Use system context for cluster state update tasks {pull}31241[#31241] (issue: {issue}30603[#30603])

Bug Fixes

Aggregations
  • Fix profiling of ordered terms aggs {pull}31814[#31814] (issue: {issue}22123[#22123])

  • Ensure that ip_range aggregations always return bucket keys. {pull}30701[#30701] (issue: {issue}21045[#21045])

  • Fix class cast exception in BucketMetricsPipeline path traversal {pull}30632[#30632] (issue: {issue}30608[#30608])

  • Fix NPE when CumulativeSum agg encounters null value/empty bucket {pull}29641[#29641] (issue: {issue}27544[#27544])

Allocation
  • A replica can be promoted and started in one cluster state update {pull}32042[#32042]

  • Ignore numeric shard count if waiting for ALL {pull}31265[#31265] (issue: {issue}31151[#31151])

  • Move allocation awareness attributes to list setting {pull}30626[#30626] (issue: {issue}30617[#30617])

  • Auto-expand replicas when adding or removing nodes {pull}30423[#30423] (issue: {issue}1873[#1873])

  • Auto-expand replicas only after failing nodes {pull}30553[#30553]

Analysis
  • Call setReferences() on custom referring tokenfilters in _analyze {pull}32157[#32157] (issue: {issue}32154[#32154])

Audit
  • Fix audit index template upgrade loop {pull}30779[#30779]

Authentication
  • [Kerberos] Add debug log statement for exceptions {pull}32663[#32663]

  • Remove Kerberos bootstrap checks {pull}32451[#32451]

  • Fix building AD URL from domain name {pull}31849[#31849]

  • resolveHasher defaults to NOOP {pull}31723[#31723] (issues: {issue}31697[#31697])

  • Check auth scheme case insensitively {pull}31490[#31490] (issue: {issue}31486[#31486])

  • Fix joining cluster with production license {pull}31341[#31341] (issue: {issue}31332[#31332])

  • Fix token backwards compatibility with pre 6.0.0-beta2 {pull}31254[#31254] (issues: {issue}31195[#31195])

  • Compliant SAML Response destination check {pull}31175[#31175]

  • Clean up code in file stores {pull}30348[#30348]

  • Fix TokenMetaData equals and hashcode {pull}30347[#30347]

Authorization
  • Fix role query that can match nested documents {pull}32705[#32705]

  • Make get all application privileges require "*" permission {pull}32460[#32460]

  • Revert to old way of merging automata {pull}32254[#32254]

  • [PKI Realm] Invalidate cache on role mappings change {pull}31510[#31510]

  • Fix dynamic mapping updates with aliases {pull}30787[#30787] (issue: {issue}30597[#30597])

  • Include an empty JSON object in a JSON array when FLS filters out all fields {pull}30709[#30709] (issue: {issue}30624[#30624])

  • Reduce garbage during index resolution {pull}30180[#30180]

CRUD
  • Bulk operation fail to replicate operations when a mapping update times out {pull}30244[#30244]

Core
  • Fix content type detection with leading whitespace {pull}32632[#32632] (issue: {issue}32357[#32357])

  • Disable C2 from using AVX-512 on JDK 10 {pull}32138[#32138] (issue: {issue}31425[#31425])

  • Create default ES_TMPDIR on Windows {pull}30325[#30325]

  • Pick inner most parse exception as root cause {pull}30270[#30270] (issues: issue}30261[#30261])

Distributed
  • Fix race between replica reset and primary promotion {pull}32442[#32442] (issues: {issue}32118[#32118], {issue}32304[#32304], {issue}32431[#32431])

  • ClassCastException when re-throwing "shard not available" exception in TransportShardMultiGetAction {pull}32185[#32185] (issue: {issue}32173[#32173])

Engine
  • Fail shard if IndexShard#storeStats runs into an IOException {pull}32241[#32241] (issue: {issue}29008[#29008])

  • IndexShard should not return null stats {pull}31528[#31528]

Geo
  • Fix handling of points_only with term strategy in geo_shape {pull}31766[#31766] (issue: {issue}31707[#31707])

  • Fix coerce validation_method in GeoBoundingBoxQueryBuilder {pull}31747[#31747] (issue: {issue}31718[#31718])

  • Improve robustness of geo shape parser for malformed shapes {pull}31449[#31449] (issue: {issue}31428[#31428])

  • Fix defaults in GeoShapeFieldMapper output {pull}31302[#31302] (issue: {issue}23206[#23206])

  • Add support for indexed shape routing in geo_shape query {pull}30760[#30760] (issue: {issue}7663[#7663])

  • Add validation that geohashes are not empty and don’t contain unsupported characters {pull}30376[#30376] (issue: {issue}23579[#23579])

Index APIs
  • Copy missing segment attributes in getSegmentInfo {pull}32396[#32396]

  • Add support for is_write_index in put-alias body parsing {pull}31674[#31674]

  • Fix writeIndex evaluation for aliases {pull}31562[#31562]

  • Fix IndexTemplateMetaData parsing from xContent {pull}30917[#30917]

  • Do not ignore request analysis/similarity settings on index resize operations when the source index already contains such settings. {pull}30216[#30216]

  • Do not return all indices if a specific alias is requested via get aliases api. {pull}29538[#29538] (issues: {issue}27763[#27763])

Ingest
  • Fix broken backport of #31578 by adjusting constructor {pull}31587[#31587] (issue: {issue}31578[#31578])

  • Ingest Attachment: Upgrade Tika to 1.18 {pull}31252[#31252]

  • Interrupt the current thread if evaluation grok expressions take too long {pull}31024[#31024] (issue: {issue}28731[#28731])

Java High Level REST Client
  • Ban LoggingDeprecationHandler {pull}32756[#32756] (issue: {issue}32151[#32151])

  • Move commercial clients from XPackClient {pull}32596[#32596]

  • Fix CreateSnapshotRequestTests Failure {pull}31630[#31630] (issue: {issue}31625[#31625])

  • Change bulk’s retry condition to be based on RestStatus {pull}29329[#29329] (issues: {issue}28885[#28885])

Java Low Level REST Client
  • Avoid setting connection request timeout {pull}30384[#30384] (issue: {issue}24069[#24069])

License
  • Do not serialize basic license expiration in X-Pack info {pull}30848[#30848]

Machine learning
  • Move open job failure explanation out of root cause {pull}31925[#31925] (issue: {issue}29950[#29950])

  • Fix calendar and filter updates from non-master nodes {pull}31804[#31804] (issue: {issue}31803[#31803])

  • Don’t treat stale FAILED jobs as OPENING in job allocation {pull}31800[#31800] (issue: {issue}31794[#31794])

  • Rate limit established model memory updates {pull}31768[#31768]

  • Account for gaps in data counts after job is reopened {pull}30294[#30294] (issue: {issue}30080[#30080])

  • Ages seasonal components in proportion to the fraction of values with which they’re updated ({ml-pull}88[#88] (issue: {ml-issue}87[#87]))

  • Fixes persist and restore, which were missing some of the trend model state. ({ml-pull}103[#103], {ml-pull}99[#99])

  • Stops zero variance data from generating a log error in the forecast confidence interval calculation ({ml-pull}120[#120], {ml-pull}107[#107])

  • Fixes corner case which was failing to calculate lgamma values and fixes the corresponding log errors ({ml-pull}131[#131], {ml-pull}126[#126])

  • Fixes influence count per bucket for metric population analyses, which was wrong and lead to incorrect influencer scoring ({ml-pull}153[#153], {ml-pull}150[#150])

  • Fixes a possible SIGSEGV for jobs with multivariate by fields enabled, which caused the jobs to fail ({ml-pull}174[#174], {ml-pull}170[#170])

  • Corrects the model bounds and typical value calculation for time series models which use a multimodal distribution. This issue could cause "Unable to bracket left percentile =…​" errors to appear in the logs. ({ml-pull}178[#178], {ml-pull}176[#176])

Mapping
  • Make sure that field collapsing supports field aliases. {pull}32648[#32648] (issue: {issue}32623[#32623])

  • Improve the error message when an index is incompatible with field aliases. {pull}32482[#32482]

  • Make sure that field aliases count towards the total fields limit. {pull}32222[#32222]

  • Fix range queries on _type field for singe type indices (#31756) {pull}32161[#32161], {pull}31756[#31756] (issues: {issue}31476[#31476])

  • In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. {pull}31514[#31514]

  • Get Mapping API to honour allow_no_indices and ignore_unavailable {pull}31507[#31507] (issue: {issue}31485[#31485])

  • Make sure KeywordFieldMapper#clone preserves split_queries_on_whitespace. {pull}31049[#31049]

  • Delay _uid field data deprecation warning {pull}30651[#30651] (issue: {issue}30625[#30625])

Monitoring
  • Fix _cluster/state to always return cluster_uuid {pull}30656[#30656]

Network
  • Ensure we don’t use a remote profile if cluster name matches {pull}31331[#31331] (issue: {issue}29321[#29321])

  • Transport client: Don’t validate node in handshake (#30737) {pull}31080[#31080] {pull}30737[#30737] (issue: {issue}30141[#30141])

  • Add TRACE, CONNECT, and PATCH http methods {pull}31079[#31079], {pull}31035[#31035] (issue: {issue}31017[#31017])

Packaging
  • Add temporary directory cleanup workarounds {pull}32615[#32615] (issue: {issue}31732[#31732])

  • Add package pre-install check for java binary {pull}31343[#31343] (issue: {issue}29665[#29665])

  • Do not run sysctl for vm.max_map_count when its already set {pull}31285[#31285]

  • Stable filemode for zip distributions {pull}30854[#30854] (issue: {issue}30799[#30799])

  • Force stable file modes for built packages {pull}30823[#30823] (issue: {issue}30799[#30799])

Plugins
  • Template upgrades should happen in a system context {pull}30621[#30621] (issue: {issue}30603[#30603])

REST API
  • Reject forcemerge requests with a body {pull}30792[#30792] (issue: {issue}29584[#29584])

  • Respect accept header on no handler {pull}30383[#30383] (issue: {issue}30329[#30329])

Recovery
  • IndicesClusterStateService should replace an init. replica with an init. primary with the same aId {pull}32374[#32374] (issue: {issue}32308[#32308])

  • Ensure to release translog snapshot in primary-replica resync {pull}32045[#32045] (issue: {issue}32030[#32030])

  • Fix missing historyUUID in peer recovery when rolling upgrade 5.x to 6.3 {pull}31506[#31506] (issue: {issue}31482[#31482])

  • Cancelling a peer recovery on the source can leak a primary permit {pull}30318[#30318]

  • ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled {pull}30316[#30316]

  • Do not log warn shard not-available exception in replication {pull}30205[#30205]

Rollup
  • Move to 128bit document IDs for Rollup. The old IDs were not wide enough and susceptible to hashing collisions. Jobs that are running during cluster upgrade will "self-upgrade" to the new ID scheme, but it is recommended that users fully rebuild Rollup indices from scratch if possible. Any existing collisions are not fixable and so data-loss may affect the rollup index despite the new IDs being used. {pull}32558[#32558] (issue: {issue}32372[#32372])

  • Histo group configurations should support scaled_float {pull}32048[#32048] (issue: {issue}32035[#32035])

  • Fix rollup on date fields that don’t support epoch_millis {pull}31890[#31890]

  • Metric config properly validates itself now {pull}31159[#31159]

SQL
  • HAVING clause should accept only aggregates {pull}31872[#31872] (issue: {issue}31726[#31726])

  • Check timeZone argument in AbstractSqlQueryRequest {pull}31822[#31822]

  • Fix incorrect HAVING equality {pull}31820[#31820] (issue: {issue}31796[#31796])

  • Fix incorrect message for aliases {pull}31792[#31792] (issue: {issue}31611[#31611])

  • Allow long literals {pull}31777[#31777] (issue: {issue}31750[#31750])

  • Fix stackoverflow on getObject and timestamp conversion {pull}31735[#31735] (issue: {issue}31734[#31734])

  • Fix rest endpoint names in node stats {pull}31371[#31371]

  • Preserve scoring in bool queries {pull}30730[#30730] (issue: {issue}29685[#29685])

  • Verify GROUP BY ordering on grouped columns {pull}30585[#30585] (issue: {issue}29900[#29900])

  • SYS TABLES ordered according to *DBC specs {pull}30530[#30530]

  • Fix parsing of dates with milliseconds {pull}30419[#30419] (issue: {issue}30002[#30002])

  • Improve correctness of SYS COLUMNS & TYPES {pull}30418[#30418] (issue: {issue}30386[#30386])

  • Fix bug caused by empty composites {pull}30343[#30343] (issue: {issue}30292[#30292])

  • Correct error message {pull}30138[#30138] (issue: {issue}30016[#30016])

  • Add BinaryMathProcessor to named writeables list {pull}30127[#30127] (issue: {issue}30014[#30014])

Scripting
  • Painless: Fix Bug with Duplicate PainlessClasses {pull}32110[#32110]

  • Painless: Fix bug for static method calls on interfaces {pull}31348[#31348]

  • Deprecate Empty Templates {pull}30194[#30194]

Search
  • Fix multi level nested sort {pull}32204[#32204] (issues: {issue}31554[#31554], {issue}31783[#31783], {issue}32130[#32130])

  • Fix race in clear scroll {pull}31259[#31259]

  • Fix index prefixes to work with span_multi {pull}31066[#31066] (issue: {issue}31056[#31056])

  • Cross Cluster Search: preserve remote status code {pull}30976[#30976] (issue: {issue}27461[#27461])

  • Fix NPE in 'more_like_this' when field has zero tokens {pull}30365[#30365] (issue: {issue}30148[#30148])

  • Fix failure for validate API on a terms query {pull}30319[#30319], {pull}29483[#29483] (issue: {issue}29033[#29033])

  • Fix a bug in FieldCapabilitiesRequest#equals and hashCode. {pull}30181[#30181]

  • Fix TermsSetQueryBuilder.doEquals() method {pull}29629[#29629] (issue: {issue}29620[#29620])

  • Add additional shards routing info in ShardSearchRequest {pull}29533[#29533] (issue: {issue}27550[#27550])

  • Use date format in date_range mapping before fallback to default {pull}29310[#29310] (issue: {issue}29282[#29282])

Security
  • Enable FIPS140LicenseBootstrapCheck {pull}32903[#32903]

  • Detect old trial licenses and mimic behaviour {pull}32209[#32209]

  • Preserve thread context when connecting to remote cluster {pull}31574[#31574] (issues: {issue}31462[#31462])

Snapshot/Restore
  • Fix repository update with the same settings but different type {pull}31458[#31458]

  • Delete temporary blobs before creating index file {pull}30528[#30528] (issues: {issue}30507[#30507])

Store
  • Side-step pending deletes check {pull}30571[#30571] (issues: {issue}30416[#30416], {issue}30503[#30503])

Suggesters
  • Add proper longitude validation in geo_polygon_query {pull}30497[#30497] (issue: {issue}30488[#30488])

  • Fix merging logic of Suggester Options {pull}29514[#29514]

Transport API
  • Fix interoperability with < 6.3 transport clients {pull}30971[#30971] (issue: {issue}30731[#30731])

  • Fix bad version check writing Repository nodes {pull}30846[#30846] (issue: {issue}30807[#30807])

Watcher
  • Guard against null in email admin watches {pull}32923[#32923] (issue: {issue}32590[#32590])

  • Fix null failure in watcher test {pull}31968[#31968] (issue: {issue}31948[#31948])

  • Fix chain input toXcontent serialization {pull}31721[#31721]

  • Add ssl.trust email account setting {pull}31684[#31684]

  • Fix check for currently executed watches {pull}31137[#31137]

  • Prevent duplicate watch triggering during upgrade {pull}30643[#30643]

  • Prevent triggering watch when using activate API {pull}30613[#30613]

  • Ensure trigger service pauses execution {pull}30363[#30363]

  • Fix watch history template for dynamic slack attachments {pull}30172[#30172]

  • Ensure mail message ids are unique per watch action {pull}30112[#30112]

  • Validate xContentType in PutWatchRequest. {pull}31088[#31088] (issue: {issue}30057[#30057])

ZenDiscovery
  • Fsync state file before exposing it {pull}30929[#30929]

  • Use correct cluster state version for node fault detection {pull}30810[#30810]

  • Only ack cluster state updates successfully applied on all nodes {pull}30672[#30672]

Regressions

Engine
  • Give the engine the whole index buffer size on init. {pull}31105[#31105]

Snapshot/Restore
  • S3 repo plugin populate SettingsFilter {pull}30652[#30652]

Upgrades

Core
  • Dependencies: Upgrade to joda time 2.10 {pull}32160[#32160]

Logging
  • LOGGING: Upgrade to Log4J 2.11.1 {pull}32616[#32616], {pull}32668[#32668] (issues: {issue}27300[#27300], {issue}32537[#32537])

Network
  • Upgrade to Netty 4.1.25.Final {pull}31232[#31232] (issues: {issue}31124[#31124], {issue}7463[#7463], {issue}8014[#8014])

  • Revert upgrade to Netty 4.1.25.Final {pull}31282[#31282] (issue: {issue}31232[#31232])

Search
  • Upgrade to Lucene 7.4.0. {pull}31529[#31529]

{es} version 6.3.2

Bug fixes

Allocation
  • A replica can be promoted and started in one cluster state update {pull}32042[#32042]

Authentication
  • Fix building AD URL from domain name {pull}31849[#31849]

Core
  • Disable C2 from using AVX-512 on JDK 10 {pull}32138[#32138] (issue: {issue}31425[#31425])

Machine Learning
  • Fixes calendar and filter updates from non-master nodes {pull}31804[#31804] (issue: {issue}31803[#31803])

  • No longer treats stale FAILED jobs as OPENING in job allocation {pull}31800[#31800] (issue: {issue}31794[#31794])

  • Rate limits established model memory updates {pull}31768[#31768]

  • Fixes SIGSEGV when upgrading a job directly from 5.6.10 to 6.3. It also fixes the upgrade for the trend component model, which caused large prediction errors immediately after upgrading. {ml-pull}143[#143] (issue: {ml-issue}141[#141])

  • Fixes issues upgrading the state, which could cause the autodetect process to crash. {ml-pull}140[#140] (issue: {ml-issue}136[#136])

Recovery
  • X-pack rolling upgrade failures [ISSUE] {pull}31827[#31827]

Rollup
  • Histo group config should support scaled_floats {pull}32048[#32048] (issue: {issue}32035[#32035])

SQL
  • HAVING clause should accept only aggregates {pull}31872[#31872] (issue: {issue}31726[#31726])

  • Fix incorrect HAVING equality {pull}31820[#31820] (issue: {issue}31796[#31796])

  • Fix incorrect message for aliases {pull}31792[#31792] (issue: {issue}31611[#31611])

  • Fix querying an alias having different mappings indices generates an incorrect error message [ISSUE] {pull}31784[#31784]

  • Allow long literals {pull}31777[#31777] (issue: {issue}31750[#31750])

  • Fix stackoverflow on getObject and timestamp conversion {pull}31735[#31735] (issue: {issue}31734[#31734])

Security
  • Detect old trial licenses and mimic behaviour {pull}32209[#32209]

Watcher
  • Add ssl.trust email account setting {pull}31684[#31684]

  • Fix check for currently executed watches {pull}31137[#31137]

  • Increase HttpClient parallel sent requests {pull}31859[#31859] (issue: {issue}30130[#30130])

{es} version 6.3.1

Bug fixes

Authentication
  • Security: fix joining cluster with production license {pull}31341[#31341] (issue: {issue}31332[#31332])

  • Security: fix token bwc with pre 6.0.0-beta2 {pull}31254[#31254] (issues: {issue}30743[#30743], {issue}31195[#31195])

  • Compliant SAML Response destination check {pull}31175[#31175]

Ingest
  • Ingest Attachment: Upgrade Tika to 1.18 {pull}31252[#31252]

Java High Level REST Client
  • Change bulk’s retry condition to be based on RestStatus {pull}29329[#29329] (issues: {issue}28885[#28885], {issue}29254[#29254])

Java Low Level REST Client
  • Avoid setting connection request timeout {pull}30384[#30384] (issue: {issue}24069[#24069])

Machine Learning
  • Fixes a bug introduced in 6.3.0 which may cause the a node to hang and drop out of the cluster if Machine Learning is enabled and has been actively used. The issue can be redressed in 6.3.0 by disabling Machine Learning on all nodes {pull}31691[#31691] (issue: {issue}31683[#31683])

Network
  • Ensure we don’t use a remote profile if cluster name matches {pull}31331[#31331] (issue: {issue}29321[#29321])

Packaging
  • Add package pre-install check for java binary {pull}31343[#31343] (issue: {issue}29665[#29665])

Recovery
  • Fix missing historyUUID in peer recovery when rolling upgrade 5.x to 6.3 {pull}31506[#31506] (issue: {issue}31482[#31482])

Rollup
  • [Rollup] Metric config parser must use builder so validation runs {pull}31159[#31159]

SQL
  • JDBC: Fix stackoverflow on getObject and timestamp conversion {pull}31735[#31735] (issue: {issue}31734[#31734])

  • SQL: Preserve scoring in bool queries {pull}30730[#30730] (issue: {issue}29685[#29685])

Scripting
  • Painless: Fix bug for static method calls on interfaces {pull}31348[#31348]

Search
  • Fix race in clear scroll {pull}31259[#31259]

  • Cross Cluster Search: preserve remote status code {pull}30976[#30976] (issue: {issue}27461[#27461])

Security
  • Preserve thread context when connecting to remote cluster {pull}31574[#31574] (issues: {issue}31241[#31241], {issue}31462[#31462])

Watcher
  • Watcher: Fix put watch action {pull}31524[#31524]

{es} version 6.3.0

Known issues

Upgrades from any 5.x version will fail for indexes which are prepared using the _synced flush API, or were automatically sync-flushed due to inactivity. If upgrading from those versions, please wait for 6.3.1 to be released.

Clusters with a Gold or Platinum license that are upgrading to 6.3 will need to explicitly set xpack.security.enabled: true in elasticsearch.yml to upgrade successfully. If this value is not set, the cluster will be unable to form after upgrade.

Breaking changes

Core
  • Rename the bulk thread pool to write thread pool {pull}29593[#29593]

Monitoring
  • By default when you install {xpack}, monitoring is enabled but data collection is disabled. To enable data collection, use the new xpack.monitoring.collection.enabled setting. You can update this setting by using the Cluster Update Settings API. For more information, see [monitoring-settings].

Packaging
  • Create keystore on package install {pull}28928[#28928]

Plugins
  • Remove silent batch mode from install plugin {pull}29359[#29359]

Security
  • The legacy XPackExtension extension mechanism has been removed and replaced with an SPI based extension mechanism that is installed and built as an elasticsearch plugin.

Breaking Java changes

Aggregations
  • Add a shallow copy method to aggregation builders {pull}28430[#28430] (issue: {issue}27782[#27782])

Deprecations

Analysis
  • Deprecate use of htmlStrip as name for HtmlStripCharFilter {pull}27429[#27429]

Core
  • Deprecate the index thread pool {pull}29540[#29540]

  • Add deprecation message for config prompt {pull}28000[#28000] (issue: {issue}27216[#27216])

Mapping
  • Deprecate the update_all_types option. {pull}28284[#28284]

Network
  • Networking: Deprecate http.enabled setting {pull}29591[#29591] (issue: {issue}12792[#12792])

  • Deprecate large max content length truncation {pull}29339[#29339] (issue: {issue}29337[#29337])

REST API
  • REST : deprecate field_data for Clear Indices Cache API {pull}28943[#28943] (issue: {issue}17804[#17804])

Search
  • Deprecate slicing on _uid. {pull}29353[#29353]

  • Generate deprecation warning for _search request with trailing tokens {pull}29428[#29428] (issue: {issue}28995[#28995])

Stats
  • Deprecate the suggest metrics {pull}29627[#29627] (issue: {issue}29589[#29589])

New features

Aggregations
  • Adds the ability to specify a format on composite date_histogram source {pull}28310[#28310] (issue: {issue}27923[#27923])

  • Calculate sum in Kahan summation algorithm in aggregations (#27807) {pull}27848[#27848] (issue: {issue}27807[#27807])

Geo
  • Add Z value support to geo_point and geo_shape {pull}25738[#25738] (issue: {issue}22917[#22917])

Ingest
  • Add ingest-attachment support for per document indexed_chars limit {pull}28977[#28977] (issue: {issue}28942[#28942])

Java High Level REST Client
  • Add ranking evaluation API to High Level Rest Client {pull}28357[#28357]

  • Add Indices Aliases API to the high level REST client {pull}27876[#27876] (issue: {issue}27205[#27205])

Java Low Level REST Client
  • Client: Wrap synchronous exceptions {pull}28919[#28919] (issue: {issue}28399[#28399])

Network
  • Introduce client feature tracking {pull}31020[#31020] (issue: {issue}30731[#30731])

REST API
  • REST: Include suppressed exceptions on failures {pull}29115[#29115] (issue: {issue}23392[#23392])

Ranking
  • Add indices options support to _rank_eval {pull}29386[#29386]

Scripting
  • Painless execute api {pull}29164[#29164]

Search
  • Search - new flag: allow_partial_search_results {pull}27906[#27906] (issue: {issue}27435[#27435])

Task Management
  • Add new setting to disable persistent tasks allocations {pull}29137[#29137]

Enhancements

Aggregations
  • Build global ordinals terms bucket from matching ordinals {pull}30166[#30166] (issue: {issue}30117[#30117])

  • Reject query if top hits result window exceeds index max result window {pull}29199[#29199] (issue: {issue}29190[#29190])

  • Optimize the composite aggregation for match_all and range queries {pull}28745[#28745] (issue: {issue}28688[#28688])

  • Always return the after_key in composite aggregation response {pull}28358[#28358]

  • Upgrade t-digest to 3.2 {pull}28305[#28305] (issue: {issue}28295[#28295])

CRUD
  • Don’t swallow exceptions on replication {pull}31179[#31179] (issue: {issue}28571[#28571])

Core
  • Implement Iterator#remove for Cache values iter {pull}29633[#29633]

  • Introduce analyze thread pool {pull}29541[#29541]

  • Add useful message when no input from terminal {pull}29369[#29369] (issues: {issue}29359[#29359], {issue}29365[#29365])

  • Improve exception handling on TransportMasterNodeAction {pull}29314[#29314] (issue: {issue}1[#1])

  • Add generic array support to AbstractObjectParser {pull}28552[#28552]

  • Introduce secure security manager to project {pull}28453[#28453]

  • XContent: Factor deprecation handling into callback {pull}28449[#28449] (issue: {issue}27955[#27955])

  • Add settings to control size and count of warning headers in responses {pull}28427[#28427] (issue: {issue}28301[#28301])

  • Trim down usages of ShardOperationFailedException interface {pull}28312[#28312] (issue: {issue}27799[#27799])

  • Enforce that java.io.tmpdir exists on startup {pull}28217[#28217]

  • Add Writeable.Reader support to TransportResponseHandler {pull}28010[#28010] (issue: {issue}26315[#26315])

Discovery-Plugins
  • Add information when master node left to DiscoveryNodes' shortSummary() {pull}28197[#28197] (issue: {issue}28169[#28169])

Distributed
  • Only log warning when actually failing shards {pull}28558[#28558] (issue: {issue}28534[#28534])

  • Allows failing shards without marking as stale {pull}28054[#28054] (issue: {issue}24841[#24841])

Engine
  • Never leave stale delete tombstones in version map {pull}29619[#29619]

  • Avoid side-effect in VersionMap when assertion enabled {pull}29585[#29585]

  • Enforce access to translog via engine {pull}29542[#29542]

  • ElasticsearchMergePolicy should extend from MergePolicyWrapper {pull}29476[#29476]

  • Track Lucene operations in engine explicitly {pull}29357[#29357]

  • Allow _update and upsert to read from the transaction log {pull}29264[#29264] (issue: {issue}26802[#26802])

  • Move trimming unsafe commits from the Engine constructor to Store {pull}29260[#29260] (issue: {issue}28245[#28245])

  • Add primary term to translog header {pull}29227[#29227]

  • Fold EngineDiskUtils into Store, for better lock semantics {pull}29156[#29156] (issue: {issue}28245[#28245])

  • Do not renew sync-id if all shards are sealed {pull}29103[#29103] (issue: {issue}27838[#27838])

  • Prune only gc deletes below the local checkpoint {pull}28790[#28790]

  • Do not optimize append-only operation if normal operation with higher seq# was seen {pull}28787[#28787]

  • Try if tombstone is eligable for pruning before locking on it’s key {pull}28767[#28767]

  • Simplify Engine.Searcher creation {pull}28728[#28728]

  • Revisit deletion policy after release the last snapshot {pull}28627[#28627] (issue: {issue}28140[#28140])

  • Index shard should roll generation via the engine {pull}28537[#28537]

  • Add lower bound for translog flush threshold {pull}28382[#28382] (issues: {issue}23779[#23779], {issue}28350[#28350])

  • Untangle Engine Constructor logic {pull}28245[#28245]

  • Clean up commits when global checkpoint advanced {pull}28140[#28140] (issue: {issue}10708[#10708])

  • Replicate writes only to fully initialized shards {pull}28049[#28049]

  • Track deletes only in the tombstone map instead of maintaining as copy {pull}27868[#27868]

Geo
  • Add null_value support to geo_point type {pull}29451[#29451] (issue: {issue}12998[#12998])

Highlighting
  • Limit analyzed text for highlighting (improvements) {pull}28907[#28907] (issues: {issue}16764[#16764], {issue}27934[#27934])

  • Limit analyzed text for highlighting (improvements) {pull}28808[#28808] (issues: {issue}16764[#16764], {issue}27934[#27934])

Ingest
  • Reduce heap-memory usage of ingest-geoip plugin {pull}28963[#28963] (issue: {issue}28782[#28782])

  • Forbid trappy methods from java.time {pull}28476[#28476]

  • version set in ingest pipeline {pull}27573[#27573] (issue: {issue}27242[#27242])

Java High Level REST Client
  • Remove flatSettings support from request classes {pull}29560[#29560]

  • REST high-level client: add support for Indices Update Settings API [take 2] {pull}29327[#29327] (issue: {issue}27205[#27205])

  • REST high-level client: add force merge API {pull}28896[#28896] (issue: {issue}27205[#27205])

  • REST high-level client: add support for Indices Update Settings API {pull}28892[#28892] (issue: {issue}27205[#27205])

  • REST high-level client: add clear cache API {pull}28866[#28866] (issue: {issue}27205[#27205])

  • REST high-level client: add flush API {pull}28852[#28852] (issue: {issue}27205[#27205])

  • REST high-level client: add support for Rollover Index API {pull}28698[#28698] (issue: {issue}27205[#27205])

  • Add Cluster Put Settings API to the high level REST client {pull}28633[#28633] (issue: {issue}27205[#27205])

  • REST high-level Client: add missing final modifiers {pull}28572[#28572]

  • REST high-level client: add support for split and shrink index API {pull}28425[#28425] (issue: {issue}27205[#27205])

  • Java high-level REST : minor code clean up {pull}28409[#28409]

  • High level rest client : code clean up {pull}28386[#28386]

  • REST high-level client: add support for exists alias {pull}28332[#28332] (issue: {issue}27205[#27205])

  • Added Put Mapping API to high-level Rest client (#27205) {pull}27869[#27869] (issue: {issue}27205[#27205])

  • Add Refresh API for RestHighLevelClient {pull}27799[#27799] (issue: {issue}27205[#27205])

  • Add support for indices exists to REST high level client {pull}27384[#27384]

License
  • Require acknowledgement to start_trial license {pull}30135[#30135] (issue: {issue}30134[#30134])

Logging
  • Fix missing node id prefix in startup logs {pull}29534[#29534]

  • Do not swallow fail to convert exceptions {pull}29043[#29043] (issue: {issue}19573[#19573])

  • Add total hits to the search slow log {pull}29034[#29034] (issue: {issue}20648[#20648])

  • Remove interning from prefix logger {pull}29031[#29031] (issue: {issue}16831[#16831])

  • Log template creation and deletion {pull}29027[#29027] (issue: {issue}10795[#10795])

  • Disallow logger methods with Object parameter {pull}28969[#28969]

Mapping
  • Restrict Document list access in ParseContext {pull}29463[#29463]

  • Check presence of multi-types before validating new mapping {pull}29316[#29316] (issue: {issue}29313[#29313])

  • Validate regular expressions in dynamic templates. {pull}29013[#29013] (issue: {issue}24749[#24749])

Machine Learning
  • Synchronize long and short tests for periodicity {ml-pull}62[#62]

  • Improvements to trend modelling and periodicity testing for forecasting {ml-pull}7[#7] (issue: {ml-issue}5[#5])

  • Clean left behind model state docs {pull}30659[#30659] (issue: {issue}30551[#30551])

  • Hide internal Job update options from the REST API {pull}30537[#30537] (issue: {issue}30512[#30512])

Packaging
  • Configure heap dump path for archive packages {pull}29130[#29130] (issue: {issue}26755[#26755])

  • Configure error file for archive packages {pull}29129[#29129] (issues: {issue}29028[#29028], {issue}29032[#29032])

  • Put JVM crash logs in the default log directory {pull}29028[#29028] (issue: {issue}13982[#13982])

  • Stop sourcing scripts during installation/removal {pull}28918[#28918] (issue: {issue}14630[#14630])

Plugins
  • Ensure that azure stream has socket privileges (#28751) {pull}28773[#28773] (issue: {issue}28662[#28662])

  • Plugins: Remove intermediate "elasticsearch" directory within plugin zips {pull}28589[#28589]

  • Plugins: Store elasticsearch and java versions in PluginInfo {pull}28556[#28556]

  • Plugins: Use one confirmation of all meta plugin permissions {pull}28366[#28366]

  • Replace jvm-example by two plugin examples {pull}28339[#28339]

  • Improve error message when installing an offline plugin {pull}28298[#28298] (issue: {issue}27401[#27401])

REST API
  • REST : Split RestUpgradeAction into two actions {pull}29124[#29124] (issue: {issue}29062[#29062])

  • Change BroadcastResponse from ToXContentFragment to ToXContentObject {pull}28878[#28878] (issues: {issue}27799[#27799], {issue}3889[#3889])

  • Remove AcknowledgedRestListener in favour of RestToXContentListener {pull}28724[#28724] (issue: {issue}3889[#3889])

  • Standardize underscore requirements in parameters {pull}27040[#27040] (issue: {issue}26886[#26886])

Ranking
  • RankEvalRequest should implement IndicesRequest {pull}29188[#29188]

  • Move indices field from RankEvalSpec to RankEvalRequest {pull}28341[#28341]

  • Simplify RankEvalResponse output {pull}28266[#28266]

Recovery
  • Require translogUUID when reading global checkpoint {pull}28587[#28587] (issue: {issue}28435[#28435])

  • Do not ignore shard not-available exceptions in replication {pull}28571[#28571] (issues: {issue}28049[#28049], {issue}28534[#28534])

  • Make primary-replica resync failures less lenient {pull}28534[#28534] (issues: {issue}24841[#24841], {issue}28049[#28049], {issue}28054[#28054])

  • Synced-flush should not seal index of out of sync replicas {pull}28464[#28464] (issue: {issue}10032[#10032])

  • Don’t refresh shard on activation {pull}28013[#28013] (issue: {issue}26055[#26055])

Rollup
  • Allow rollup job creation only if cluster is x-pack ready {pull}30963[#30963] (issue: {issue}30743[#30743])

Scripting
  • Modify Painless grammar to support right brackets as statement delimiters {pull}29566[#29566]

Search
  • Improve explanation in rescore {pull}30629[#30629] (issue: {issue}28725[#28725])

  • Add support to match_phrase query for zero_terms_query. {pull}29598[#29598] (issue: {issue}29344[#29344])

  • Improve similarity integration. {pull}29187[#29187] (issues: {issue}23208[#23208], {issue}29035[#29035])

  • Store offsets in index prefix fields when stored in the parent field {pull}29067[#29067] (issue: {issue}28994[#28994])

  • Add QueryBuilders.matchNoneQuery(), #28679 {pull}28680[#28680]

  • Adds SpanGapQueryBuilder. Feature #27862 {pull}28636[#28636] (issue: {issue}27862[#27862])

  • Provide a better error message for the case when all shards failed {pull}28333[#28333]

  • Add ability to index prefixes on text fields {pull}28290[#28290] (issue: {issue}28222[#28222])

  • Add index_prefix option to text fields {pull}28222[#28222]

Settings
  • Enhance error for out of bounds byte size settings {pull}29338[#29338] (issue: {issue}29337[#29337])

  • Settings: Reimplement keystore format to use FIPS compliant algorithms {pull}28255[#28255]

Snapshot/Restore
  • Do not fail snapshot when deleting a missing snapshotted file {pull}30332[#30332] (issue: {issue}28322[#28322])

  • Update secure settings for the repository azure repository plugin {pull}29319[#29319] (issue: {issue}29135[#29135])

  • Use client settings in repository-gcs {pull}28575[#28575]

Stats
  • Add periodic flush count to flush stats {pull}29360[#29360] (issue: {issue}29125[#29125])

  • Enable selecting adaptive selection stats {pull}28721[#28721]

  • Add translog files age to Translog Stats (#28613) {pull}28613[#28613] (issue: {issue}28189[#28189])

Task Management
  • Make Persistent Tasks implementations version and feature aware {pull}31045[#31045] (issues: {issue}30731[#30731], {issue}31020[#31020])

Transport API
  • Add remote cluster client {pull}29495[#29495]

  • Java api clean-up : consistency for shards_acknowledged getters {pull}27819[#27819] (issue: {issue}27784[#27784])

Watcher
  • Move watcher-history version setting to _meta field {pull}30832[#30832] (issue: {issue}30731[#30731])

  • Only allow x-pack metadata if all nodes are ready {pull}30743[#30743] (issues: {issue}30728[#30728], {issue}30731[#30731])

ZenDiscovery
  • Add support for skippable named writeables {pull}30948[#30948]

Bug fixes

Aggregations
  • Fix date and ip sources in the composite aggregation {pull}29370[#29370]

  • Pass through script params in scripted metric agg {pull}29154[#29154] (issue: {issue}28819[#28819])

  • Force depth_first mode execution for terms aggregation under a nested context {pull}28421[#28421] (issue: {issue}28394[#28394])

  • StringTerms.Bucket.getKeyAsNumber detection type {pull}28118[#28118] (issue: {issue}28012[#28012])

Allocation
  • Move allocation awareness attributes to list setting {pull}30626[#30626] (issue: {issue}30617[#30617])

  • Grammar matters.. {pull}29462[#29462]

  • Don’t break allocation if resize source index is missing {pull}29311[#29311] (issue: {issue}26931[#26931])

  • Add check when trying to reroute a shard to a non-data discovery node {pull}28886[#28886]

Audit
  • Fix audit index template upgrade loop {pull}30779[#30779]

Authentication
  • Security: fix dynamic mapping updates with aliases {pull}30787[#30787] (issue: {issue}30597[#30597])

  • [Security] Include an empty json object in an json array when FLS filters out all fields {pull}30709[#30709] (issue: {issue}30624[#30624])

  • Security: cleanup code in file stores {pull}30348[#30348]

  • Security: fix TokenMetaData equals and hashcode {pull}30347[#30347]

Authorization
  • Security: reduce garbage during index resolution {pull}30180[#30180]

CRUD
  • Bulk operation fail to replicate operations when a mapping update times out {pull}30244[#30244]

Core
  • Create default ES_TMPDIR on Windows {pull}30325[#30325] (issues: {issue}27609[#27609], {issue}28217[#28217])

  • Core: Pick inner most parse exception as root cause {pull}30270[#30270] (issues: {issue}29373[#29373], {issue}30261[#30261])

  • Fix the version ID for v5.6.10 (backport to 6.x). {pull}29571[#29571]

  • Fix the version ID for v5.6.10. {pull}29570[#29570]

  • Fix EsAbortPolicy to conform to API {pull}29075[#29075] (issue: {issue}19508[#19508])

  • Remove special handling for _all in nodes info {pull}28971[#28971] (issue: {issue}28797[#28797])

  • Handle throws on tasks submitted to thread pools {pull}28667[#28667]

  • Fix size blocking queue to not lie about its weight {pull}28557[#28557] (issue: {issue}28547[#28547])

  • Further minor bug fixes found by lgtm.com {pull}27772[#27772]

Engine
  • Add an escape hatch to increase the maximum amount of memory that IndexWriter gets. {pull}31132[#31132] (issue: {issue}31105[#31105])

  • Avoid self-deadlock in the translog {pull}29520[#29520] (issues: {issue}29401[#29401], {issue}29509[#29509])

  • Close translog writer if exception on write channel {pull}29401[#29401] (issue: {issue}29390[#29390])

  • Harden periodically check to avoid endless flush loop {pull}29125[#29125] (issues: {issue}1[#1], {issue}2[#2], {issue}28350[#28350], {issue}29097[#29097], {issue}3[#3])

  • Avoid class cast exception from index writer {pull}28989[#28989]

  • Maybe die before failing engine {pull}28973[#28973] (issues: {issue}27265[#27265], {issue}28967[#28967])

  • Never block on key in LiveVersionMap#pruneTombstones {pull}28736[#28736] (issue: {issue}28714[#28714])

  • Inc store reference before refresh {pull}28656[#28656]

  • Replica recovery could go into an endless flushing loop {pull}28350[#28350]

Geo
  • Fix overflow error in parsing of long geohashes {pull}29418[#29418] (issue: {issue}24616[#24616])

  • Fix bwc in GeoDistanceQuery serialization {pull}29325[#29325] (issues: {issue}22876[#22876], {issue}29301[#29301])

  • Allow using distance measure in the geo context precision {pull}29273[#29273] (issue: {issue}24807[#24807])

  • Fix incorrect geohash for lat 90, lon 180 {pull}29256[#29256] (issue: {issue}22163[#22163])

  • [GEO] Fix points_only indexing failure for GeoShapeFieldMapper {pull}28774[#28774] (issues: {issue}27415[#27415], {issue}28744[#28744])

Index APIs
  • Propagate mapping.single_type setting on shrinked index {pull}29202[#29202]

  • Fix Parsing Bug with Update By Query for Stored Scripts {pull}29039[#29039] (issue: {issue}28002[#28002])

Ingest
  • Don’t allow referencing the pattern bank name in the pattern bank {pull}29295[#29295] (issue: {issue}29257[#29257])

  • Continue registering pipelines after one pipeline parse failure. {pull}28752[#28752] (issue: {issue}28269[#28269])

  • Guard accessDeclaredMembers for Tika on JDK 10 {pull}28603[#28603] (issue: {issue}28602[#28602])

  • Fix for bug that prevents pipelines to load that use stored scripts after a restart {pull}28588[#28588]

Java High Level REST Client
  • Bulk processor#awaitClose to close scheduler {pull}29263[#29263]

  • REST high-level client: encode path parts {pull}28663[#28663] (issue: {issue}28625[#28625])

  • Fix parsing of script fields {pull}28395[#28395] (issue: {issue}28380[#28380])

  • Move to POST when calling API to retrieve which support request body {pull}28342[#28342] (issue: {issue}28326[#28326])

Java Low Level REST Client
  • REST client: hosts marked dead for the first time should not be immediately retried {pull}29230[#29230]

License
  • Do not serialize basic license exp in x-pack info {pull}30848[#30848]

  • Require acknowledgement to start_trial license {pull}30198[#30198] (issue: {issue}30134[#30134])

Machine Learning
  • By-fields should respect model_plot_config.terms {ml-pull}86[#86] (issue: {issue}30004[#30004])

  • Function description for population lat_long results should be lat_long instead of mean {ml-pull}81[#81] (issue: {ml-issue}80[#80])

  • Fix error causing us to overestimate effective history length {ml-pull}66[#66] (issue: {ml-issue}57[#57])

  • Clearing JSON memory allocators {ml-pull}30[#30] (issue: {ml-issue}26[#26])

  • Fix sparse data edge cases for periodicity testing {ml-pull}28[#28] (issue: {ml-issue}20[#20])

  • Impose an absolute cutoff on the minimum variance {ml-pull}8[#8] (issue: {ml-issue}488[#488])

  • Check accesses in bounds when clearing recycled models {ml-pull}79[#79] (issue: {ml-issue}76[#76])

  • Set forecast progress to 100% and status finished in the case of insufficient history (data) {ml-pull}44[#44]

  • Add control message to start background persistence {ml-pull}19[#19]

  • Fail start up if state is missing {ml-pull}4[#4]

  • Do not log incorrect model memory limit {ml-pull}3[#3]

  • Fix background persistence - in 6.1 and 6.2 background persisted models were only visible after jobs were closed

  • Fix anomaly detection for buckets with fewer than two values when using the varp function

  • Fix typical values and model plot bounds when using the multivariate_by_fields option

  • Fix typical values, forecast confidence intervals and model plot bounds for models with multiple modes

Mapping
  • Delay _uid field data deprecation warning {pull}30651[#30651] (issue: {issue}30625[#30625])

  • Ignore null value for range field (#27845) {pull}28116[#28116] (issue: {issue}27845[#27845])

  • Fix a type check that is always false {pull}27726[#27726]

Network
  • Fix handling of bad requests {pull}29249[#29249] (issues: {issue}21974[#21974], {issue}28909[#28909])

Packaging
  • Fix #29057 CWD to ES_HOME does not change drive {pull}29086[#29086]

  • Allow overriding JVM options in Windows service {pull}29044[#29044] (issue: {issue}23484[#23484])

  • CLI: Close subcommands in MultiCommand {pull}28954[#28954]

  • Delay path expansion on Windows {pull}28753[#28753] (issues: {issue}27675[#27675], {issue}28748[#28748])

  • Fix using relative custom config path {pull}28700[#28700] (issue: {issue}27610[#27610])

  • Disable console logging in the Windows service {pull}28618[#28618] (issue: {issue}20422[#20422])

Percolator
  • Fixed bug when non percolator docs end up in the search hits {pull}29447[#29447] (issue: {issue}29429[#29429])

  • Fixed a msm accounting error that can occur during analyzing a percolator query {pull}29415[#29415] (issue: {issue}29393[#29393])

  • Fix more query extraction bugs. {pull}29388[#29388] (issues: {issue}28353[#28353], {issue}29376[#29376])

  • Fix some query extraction bugs. {pull}29283[#29283]

  • Fix percolator query analysis for function_score query {pull}28854[#28854]

  • Improved percolator’s random candidate query duel test {pull}28840[#28840]

  • Do not take duplicate query extractions into account for minimum_should_match attribute {pull}28353[#28353] (issue: {issue}28315[#28315])

Plugins
  • Template upgrades should happen in a system context {pull}30621[#30621] (issue: {issue}30603[#30603])

  • Plugins: Fix native controller confirmation for non-meta plugin {pull}29434[#29434]

  • Plugins: Fix module name conflict check for meta plugins {pull}29146[#29146]

  • Ensure that azure stream has socket privileges {pull}28751[#28751] (issue: {issue}28662[#28662])

  • Fix handling of mandatory meta plugins {pull}28710[#28710] (issue: {issue}28022[#28022])

  • Fix the ability to remove old plugin {pull}28540[#28540] (issue: {issue}28538[#28538])

REST API
  • Respect accept header on no handler {pull}30383[#30383] (issue: {issue}30329[#30329])

  • Protect against NPE in RestNodesAction {pull}29059[#29059]

  • REST api specs : remove unsupported wait_for_merge param {pull}28959[#28959] (issue: {issue}27158[#27158])

  • Rest api specs : remove unsupported parameter parent_node {pull}28841[#28841]

  • Rest api specs : remove a common param from nodes.usage.json {pull}28835[#28835] (issue: {issue}28226[#28226])

  • Missing timeout parameter from the REST API spec JSON files (#28200) {pull}28328[#28328]

Ranking
  • Fix NDCG for empty search results {pull}29267[#29267]

Recovery
  • Cancelling a peer recovery on the source can leak a primary permit {pull}30318[#30318]

  • ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled {pull}30316[#30316]

  • Do not log warn shard not-available exception in replication {pull}30205[#30205] (issues: {issue}28049[#28049], {issue}28571[#28571])

  • Fix outgoing NodeID {pull}28779[#28779] (issue: {issue}28777[#28777])

  • Fsync directory after cleanup {pull}28604[#28604] (issue: {issue}28435[#28435])

Security
  • Reduces the number of object allocations made by {security} when resolving the indices and aliases for a request ({pull}30180[#30180])

  • Respects accept header on requests with no handler ({pull}30383[#30383])

SQL
  • SQL: Verify GROUP BY ordering on grouped columns {pull}30585[#30585] (issue: {issue}29900[#29900])

  • SQL: Fix parsing of dates with milliseconds {pull}30419[#30419] (issue: {issue}30002[#30002])

  • SQL: Fix bug caused by empty composites {pull}30343[#30343] (issue: {issue}30292[#30292])

  • SQL: Correct error message {pull}30138[#30138] (issue: {issue}30016[#30016])

  • SQL: Add BinaryMathProcessor to named writeables list {pull}30127[#30127] (issue: {issue}30014[#30014])

Scripting
  • Correct class to name string conversion {pull}28997[#28997]

  • Painless: Fix For Loop NullPointerException {pull}28506[#28506] (issue: {issue}28501[#28501])

  • Scripts: Fix security for deprecation warning {pull}28485[#28485] (issue: {issue}28408[#28408])

Search
  • Ensure that index_prefixes settings cannot be changed {pull}30967[#30967]

  • Fix TermsSetQueryBuilder.doEquals() method {pull}29629[#29629] (issue: {issue}29620[#29620])

  • Fix binary doc values fetching in _search {pull}29567[#29567] (issues: {issue}26775[#26775], {issue}29565[#29565])

  • Fixes query_string query equals timezone check {pull}29406[#29406] (issue: {issue}29403[#29403])

  • Fixed quote_field_suffix in query_string {pull}29332[#29332] (issue: {issue}29324[#29324])

  • Search: Validate script query is run with a single script {pull}29304[#29304]

  • Propagate ignore_unmapped to inner_hits {pull}29261[#29261] (issue: {issue}29071[#29071])

  • Restore tiebreaker for cross fields query {pull}28935[#28935] (issues: {issue}25115[#25115], {issue}28933[#28933])

  • Fix (simple)_query_string to ignore removed terms {pull}28871[#28871] (issues: {issue}28855[#28855], {issue}28856[#28856])

  • Search option terminate_after does not handle post_filters and aggregations correctly {pull}28459[#28459] (issue: {issue}28411[#28411])

  • Fix AIOOB on indexed geo_shape query {pull}28458[#28458] (issue: {issue}28456[#28456])

Settings
  • Archive unknown or invalid settings on updates {pull}28888[#28888] (issue: {issue}28609[#28609])

  • Settings: Introduce settings updater for a list of settings {pull}28338[#28338] (issue: {issue}28047[#28047])

Snapshot/Restore
  • Delete temporary blobs before creating index file {pull}30528[#30528] (issues: {issue}30332[#30332], {issue}30507[#30507])

  • Fix NPE when using deprecated Azure settings {pull}28769[#28769] (issues: {issue}23518[#23518], {issue}28299[#28299])

Stats
  • Fix AdaptiveSelectionStats serialization bug {pull}28718[#28718] (issue: {issue}28713[#28713])

Suggesters
  • Fix merging logic of Suggester Options {pull}29514[#29514]

Transport API
  • Fix interoperability with < 6.3 transport clients {pull}30971[#30971] (issue: {issue}30731[#30731])

Watcher
  • Watcher: Prevent triggering watch when using activate API {pull}30613[#30613]

ZenDiscovery
  • Fsync state file before exposing it {pull}30929[#30929]

  • Do not return metadata customs by default {pull}30857[#30857] (issue: {issue}30731[#30731])

  • Use correct cluster state version for node fault detection {pull}30810[#30810]

Regressions

Snapshot/Restore
  • S3 repo plugin populate SettingsFilter {pull}30652[#30652]

Upgrades

Network
  • Update Netty to 4.1.16.Final {pull}28345[#28345]

Search
  • Upgrade to lucene-7.3.1 {pull}30729[#30729]

{es} version 6.2.4

Bug Fixes

Engine
  • Harden periodically check to avoid endless flush loop {pull}29125[#29125] (issues: {issue}28350[#28350], {issue}29097[#29097])

Ingest
  • Don’t allow referencing the pattern bank name in the pattern bank {pull}29295[#29295] (issue: {issue}29257[#29257])

Regressions

Fail snapshot operations early when creating or deleting a snapshot on a repository that has been written to by an older Elasticsearch after writing to it with a newer Elasticsearch version. ({pull}30140[#30140])

Java High Level REST Client
  • Bulk processor#awaitClose to close scheduler {pull}29263[#29263] Fix NPE when CumulativeSum agg encounters null value/empty bucket ({pull}29641[#29641]) Do not fail snapshot when deleting a missing snapshotted file ({pull}30332[#30332])

Java Low Level REST Client
  • REST client: hosts marked dead for the first time should not be immediately retried {pull}29230[#29230]

Machine Learning
  • Prevents failed jobs from mistakenly re-opening after node loss recovery.

  • Returns an error when an operation cannot be submitted because the process was killed.

  • Respects the datafeed frequency when it is less or equal to the query_delay setting.

Network
  • Cross-cluster search and default connections can get crossed [OPEN] [ISSUE] {pull}29321[#29321]

Percolator
  • Fixed bug when non percolator docs end up in the search hits {pull}29447[#29447] (issue: {issue}29429[#29429])

  • Fixed a msm accounting error that can occur during analyzing a percolator query {pull}29415[#29415] (issue: {issue}29393[#29393])

  • Fix more query extraction bugs. {pull}29388[#29388] (issues: {issue}28353[#28353], {issue}29376[#29376])

  • Fix some query extraction bugs. {pull}29283[#29283]

Plugins
  • Plugins: Fix native controller confirmation for non-meta plugin {pull}29434[#29434]

Search
  • Propagate ignore_unmapped to inner_hits {pull}29261[#29261] (issue: {issue}29071[#29071])

Security/Authentication
  • Adds missing idp.use_single_logout and populate_user_metadata SAML realm settings. See [ref-saml-settings].

Settings
  • Archive unknown or invalid settings on updates {pull}28888[#28888] (issue: {issue}28609[#28609])

Watcher
  • Re-enables smtp.* account configuration properties in the notification settings. See [email-notification-settings].

  • Ensures starting and stopping {watcher} is properly acknowledged as a master node action.

  • Refrains from appending a question mark to an HTTP request if no parameters are used.

{es} version 6.2.3

Deprecations

Deprecated multi-argument versions of the request methods in the RestClient. Prefer the "Request" object flavored methods. ({pull}30315[#30315])

A new analysis plugin called analysis_nori that exposes the Lucene Korean analysis module. ({pull}30397[#30397])

Enhancements

Highlighting
  • Limit analyzed text for highlighting (improvements) {pull}28808[#28808] (issues: {issue}16764[#16764], {issue}27934[#27934])

Allow copying source settings on index resize operations ({pull}30255[#30255], {pull}30404[#30404])

Added new "Request" object flavored request methods in the RestClient. Prefer these instead of the multi-argument versions. ({pull}29623[#29623])

Recovery
  • Require translogUUID when reading global checkpoint {pull}28587[#28587] (issue: {issue}28435[#28435]) Added setJsonEntity to Request object so it is marginally easier to send JSON. ({pull}30447[#30447]) Watcher HTTP client used in watches now allows more parallel connections to the same endpoint and evicts long running connections. ({pull}30130[#30130])

The cluster state listener to decide if watcher should be stopped/started/paused now runs far less code in an executor but is more synchronous and predictable. Also the trigger engine thread is only started on data nodes. And the Execute Watch API can be triggered regardless is watcher is started or stopped. ({pull}30118[#30118])

Added put index template API to the high level rest client ({pull}30400[#30400])

Add ability to filter coordinating-only nodes when interacting with cluster APIs. ({pull}30313[#30313])

Bug Fixes

Core
  • Remove special handling for _all in nodes info {pull}28971[#28971] (issue: {issue}28797[#28797])

Engine
  • Avoid class cast exception from index writer {pull}28989[#28989]

  • Maybe die before failing engine {pull}28973[#28973] (issues: {issue}27265[#27265], {issue}28967[#28967])

  • Never block on key in LiveVersionMap#pruneTombstones {pull}28736[#28736] (issue: {issue}28714[#28714])

Ingest
  • Continue registering pipelines after one pipeline parse failure. {pull}28752[#28752] (issue: {issue}28269[#28269])

Java High Level REST Client
  • REST high-level client: encode path parts {pull}28663[#28663] (issue: {issue}28625[#28625])

Machine Learning
Monitoring
  • Aligned reporting of index statistics that exist in the current cluster state. This fix avoids subtle race conditions in stats reporting.

Packaging
  • Delay path expansion on Windows {pull}28753[#28753] (issues: {issue}27675[#27675], {issue}28748[#28748])

Percolator
  • Fix percolator query analysis for function_score query {pull}28854[#28854]

  • Improved percolator’s random candidate query duel test {pull}28840[#28840]

Security
  • Fixed handling of comments in XML documents [ESA-2018-07].

  • Fixed auditing such that when you use a local audit index, it maintains the mappings automatically. Maintenance is necessary, for example, when new fields are introduced or document types change.

  • Added and changed settings for the SAML NameID policy. For example, added the nameid.allow_create setting and changed the default value for the SPNameQualifier setting to blank. See [saml-realm].

  • Fixed handling of an Assertion Consumer Service (ACS) URL with existing query parameters. See [saml-realm].

  • Fixed the PKI realm bootstrap check such that it works with secure settings. For more information, see [bootstrap-checks-xpack].

Snapshot/Restore
  • Fix NPE when using deprecated Azure settings {pull}28769[#28769] (issues: {issue}23518[#23518], {issue}28299[#28299])

Stats
  • Fix AdaptiveSelectionStats serialization bug {pull}28718[#28718] (issue: {issue}28713[#28713])

Watcher
  • Fixed the serialization of failed hipchat messages, such that it no longer tries to write the status field twice.

  • Fixed TransformInput toXContent serialization errors. For more information, see [_transforming_chained_input_data].

Allocation

Auto-expand replicas when adding or removing nodes to prevent shard copies from being dropped and resynced when a data node rejoins the cluster ({pull}30423[#30423])

{es} version 6.2.2

Enhancements

Recovery
  • Synced-flush should not seal index of out of sync replicas {pull}28464[#28464] (issue: {issue}10032[#10032])

Bug Fixes

Core
  • Handle throws on tasks submitted to thread pools {pull}28667[#28667]

  • Fix size blocking queue to not lie about its weight {pull}28557[#28557] (issue: {issue}28547[#28547])

Ingest
  • Guard accessDeclaredMembers for Tika on JDK 10 {pull}28603[#28603] (issue: {issue}28602[#28602])

  • Fix for bug that prevents pipelines to load that use stored scripts after a restart {pull}28588[#28588]

Java High Level REST Client
  • Fix parsing of script fields {pull}28395[#28395] (issue: {issue}28380[#28380])

  • Move to POST when calling API to retrieve which support request body {pull}28342[#28342] (issue: {issue}28326[#28326])

Machine Learning
  • Fixed an exception that occurred when a categorization field contained an empty string.

Monitoring
  • Properly registered xpack.monitoring.exporters..headers. settings, which were broken in 6.2.0 and 6.2.1. For more information, see [http-exporter-settings].

Packaging
  • Fix using relative custom config path {pull}28700[#28700] (issue: {issue}27610[#27610])

  • Disable console logging in the Windows service {pull}28618[#28618] (issue: {issue}20422[#20422])

Percolator
  • Do not take duplicate query extractions into account for minimum_should_match attribute {pull}28353[#28353] (issue: {issue}28315[#28315])

Recovery
  • Fsync directory after cleanup {pull}28604[#28604] (issue: {issue}28435[#28435])

Security
  • Added CachingRealm to published artifacts so it can be used in custom realm extensions.

  • If the realm uses native role mappings and the security index health changes, the realm caches are cleared. For example, they are cleared when the index recovers from a red state, when the index is deleted, when the index becomes outdated, and when the index becomes up-to-date.

  • Fixed a bug that could prevent auditing to a remote index if the remote cluster was re-started at the same time as the audited cluster.

  • Removed AuthorityKeyIdentifier’s Issuer and Serial number from certificates generated by certgen and certutil. This improves compatibility with certificate verification in {kib}.

Watcher
  • Proxies now use HTTP by default, which was the default prior to 6.0. This fixes issues with HTTPS requests that tried to access proxies via HTTP.

  • Fixed the HTML sanitizer settings (xpack.notification.email.html.sanitization.*), which were broken in 6.2. For more information, see [notification-settings].

{es} version 6.2.1

The cluster state listener to decide if watcher should be stopped/started/paused now runs far less code in an executor but is more synchronous and predictable. Also the trigger engine thread is only started on data nodes. And the Execute Watch API can be triggered regardless is watcher is started or stopped. ({pull}30118[#30118])

Bug Fixes

Plugin Lang Painless
  • Painless: Fix For Loop NullPointerException {pull}28506[#28506] (issue: {issue}28501[#28501])

Plugins
  • Fix the ability to remove old plugin {pull}28540[#28540] (issue: {issue}28538[#28538])

Security
  • Fixed missing dependencies for x-pack-transport.

  • Fixed saml-metadata env file such that it sources the appropriate environment file.

Machine Learning
  • Account for gaps in data counts after job is reopened ({pull}30294[#30294])

{es} version 6.2.0

Breaking Changes

Aggregations
  • Add a new cluster setting to limit the total number of buckets returned by a request {pull}27581[#27581] (issues: {issue}26012[#26012], {issue}27452[#27452])

Core
  • Forbid granting the all permission in production {pull}27548[#27548]

Highlighting
  • Limit the analyzed text for highlighting {pull}27934[#27934] (issue: {issue}27517[#27517])

Rollover
  • Fail rollover if duplicated alias found in templates {pull}28110[#28110] (issue: {issue}26976[#26976])

Search
  • Introduce limit to the number of terms in Terms Query {pull}27968[#27968] (issue: {issue}18829[#18829])

Breaking Java Changes

Java API
  • Remove operationThreaded from Java API {pull}27836[#27836]

Java High Level REST Client
  • REST high-level client: remove index suffix from indices client method names {pull}28263[#28263]

Deprecations

Analysis
  • Backport delimited payload filter renaming {pull}27535[#27535] (issue: {issue}26625[#26625])

Suggesters
  • deprecating jarowinkler in favor of jaro_winkler {pull}27526[#27526]

  • Deprecating levenstein in favor of levensHtein {pull}27409[#27409] (issue: {issue}27325[#27325])

New Features

Machine Learning
  • Added the ability to identify scheduled events and prevent anomaly detection during these periods. For more information, see {ml-docs}/ml-calendars.html[Calendars and Scheduled Events].

Plugin Ingest GeoIp
  • Enable ASN support for Ingest GeoIP plugin. {pull}27958[#27958] (issue: {issue}27849[#27849])

Plugin Lang Painless
  • Painless: Add spi jar that will be published for extending whitelists {pull}28302[#28302]

  • Painless: Add a simple cache for whitelist methods and fields. {pull}28142[#28142]

Plugins
  • Add the ability to bundle multiple plugins into a meta plugin {pull}28022[#28022] (issue: {issue}27316[#27316])

Rank Evaluation
  • Backport of ranking evaluation API (#27478) {pull}27844[#27844] (issue: {issue}27478[#27478])

Recovery
  • Backport for using lastSyncedGlobalCheckpoint in deletion policy {pull}27866[#27866] (issue: {issue}27826[#27826])

Reindex API
  • Add scroll parameter to _reindex API {pull}28041[#28041] (issue: {issue}27555[#27555])

Security
  • {security} now supports user authentication using SAML Single Sign on. For more information, see [saml-realm].

Watcher

Enhancements

Allocation
  • Fix cluster.routing.allocation.enable and cluster.routing.rebalance.enable case {pull}28037[#28037] (issue: {issue}28007[#28007])

  • Add node id to shard failure message {pull}28024[#28024] (issue: {issue}28018[#28018])

Analysis
  • Limit the analyzed text for highlighting (#27934) {pull}28176[#28176] (issue: {issue}27517[#27517])

  • Allow TrimFilter to be used in custom normalizers {pull}27758[#27758] (issue: {issue}27310[#27310])

Circuit Breakers
  • Add accounting circuit breaker and track segment memory usage {pull}27116[#27116] (issue: {issue}27044[#27044])

Cluster
  • Adds wait_for_no_initializing_shards to cluster health API {pull}27489[#27489] (issue: {issue}25623[#25623])

Core
  • Introduce elasticsearch-core jar {pull}28191[#28191] (issue: {issue}27933[#27933])

  • Rename core module to server {pull}28190[#28190] (issue: {issue}27933[#27933])

  • Rename core module to server {pull}28180[#28180] (issue: {issue}27933[#27933])

  • Introduce elasticsearch-core jar {pull}28178[#28178] (issue: {issue}27933[#27933])

  • Add Writeable.Reader support to TransportResponseHandler {pull}28010[#28010] (issue: {issue}26315[#26315])

  • Simplify rejected execution exception {pull}27664[#27664] (issue: {issue}27663[#27663])

  • Add node name to thread pool executor name {pull}27663[#27663] (issues: {issue}26007[#26007], {issue}26835[#26835])

Discovery
  • Add information when master node left to DiscoveryNodes' shortSummary() {pull}28197[#28197] (issue: {issue}28169[#28169])

Engine
  • Move uid lock into LiveVersionMap {pull}27905[#27905]

  • Optimize version map for append-only indexing {pull}27752[#27752]

Geo
  • [GEO] Add WKT Support to GeoBoundingBoxQueryBuilder {pull}27692[#27692] (issues: {issue}27690[#27690], {issue}9120[#9120])

  • [Geo] Add Well Known Text (WKT) Parsing Support to ShapeBuilders {pull}27417[#27417] (issue: {issue}9120[#9120])

Highlighting
  • Include all sentences smaller than fragment_size in the unified highlighter {pull}28132[#28132] (issue: {issue}28089[#28089])

Ingest
  • Enable convert processor to support Long and Double {pull}27891[#27891] (issues: {issue}23085[#23085], {issue}23423[#23423])

Internal
  • Make KeyedLock reentrant {pull}27920[#27920]

  • Make AbstractQueryBuilder.declareStandardFields to be protected (#27865) {pull}27894[#27894] (issue: {issue}27865[#27865])

  • Tighten the CountedBitSet class {pull}27632[#27632]

  • Avoid doing redundant work when checking for self references. {pull}26927[#26927] (issue: {issue}26907[#26907])

Java API
  • Add missing delegate methods to NodeIndicesStats {pull}28092[#28092]

  • Java api clean-up : consistency for shards_acknowledged getters {pull}27819[#27819] (issue: {issue}27784[#27784])

Java High Level REST Client
  • add toString implementation for UpdateRequest. {pull}27997[#27997] (issue: {issue}27986[#27986])

  • Add Close Index API to the high level REST client {pull}27734[#27734] (issue: {issue}27205[#27205])

  • Add Open Index API to the high level REST client {pull}27574[#27574] (issue: {issue}27205[#27205])

  • Added Create Index support to high-level REST client {pull}27351[#27351] (issue: {issue}27205[#27205])

  • Add multi get api to the high level rest client {pull}27337[#27337] (issue: {issue}27205[#27205])

  • Add msearch api to high level client {pull}27274[#27274]

Machine Learning
  • Increased tokenization flexibility for categorization. Now all {es} analyzer functionality is available, which opens up the possibility of sensibly categorizing non-English log messages. For more information, see {ml-docs}/ml-configuring-categories.html#ml-configuring-analyzer[Customizing the Categorization Analyzer].

  • Improved the sensitivity of the analysis to high variance data with lots of values near zero.

  • Improved the decay rate of the model memory by using a weighted moving average.

  • Machine learning indices created after upgrading to 6.2 have the auto_expand_replicas: 0-1 setting rather than a fixed setting of 1 replica. As a result, {ml} indices created after upgrading to 6.2 can have a green status on single node clusters. There is no impact in multi-node clusters.

  • Changed the credentials that are used by {dfeeds}. When {security} is enabled, a {dfeed} stores the roles of the user who created or updated the {dfeed} at that time. This means that if those roles are updated, the {dfeed} subsequently runs with the new permissions that are associated with the roles. However, if the user’s roles are adjusted after creating or updating the {dfeed} then the {dfeed} continues to run with the permissions that are associated with the original roles. For more information, see {ml-docs}/ml-dfeeds.html[Datafeeds].

  • Added a new scheduled forecast status, which indicates that the forecast has not started yet.

Mapping
  • Allow _doc as a type. {pull}27816[#27816] (issues: {issue}27750[#27750], {issue}27751[#27751])

Monitoring
  • {monitoring} indices (.monitoring) created after upgrading to 6.2 have the auto_expand_replicas: 0-1 setting rather than a fixed setting of 1 replica. As a result, monitoring indices created after upgrading to 6.2 can have a green status on single node clusters. There is no impact in multi-node clusters.

  • Added a cluster alert that triggers whenever a node is added, removed, or restarted.

Network
  • Add NioGroup for use in different transports {pull}27737[#27737] (issue: {issue}27260[#27260])

  • Add read timeouts to http module {pull}27713[#27713]

  • Implement byte array reusage in NioTransport {pull}27696[#27696] (issue: {issue}27563[#27563])

  • Introduce resizable inbound byte buffer {pull}27551[#27551] (issue: {issue}27563[#27563])

  • Decouple nio constructs from the tcp transport {pull}27484[#27484] (issue: {issue}27260[#27260])

Packaging
  • Extend JVM options to support multiple versions {pull}27675[#27675] (issue: {issue}27646[#27646])

  • Add explicit coreutils dependency {pull}27660[#27660] (issue: {issue}27609[#27609])

  • Detect mktemp from coreutils {pull}27659[#27659] (issues: {issue}27609[#27609], {issue}27643[#27643])

  • Enable GC logs by default {pull}27610[#27610]

  • Use private directory for temporary files {pull}27609[#27609] (issues: {issue}14372[#14372], {issue}27144[#27144])

Percolator
  • also extract match_all queries when indexing percolator queries {pull}27585[#27585]

Plugin Lang Painless
  • Painless: Add whitelist extensions {pull}28161[#28161]

  • Painless: Modify Loader to Load Classes Directly from Definition {pull}28088[#28088]

  • Clean Up Painless Cast Object {pull}27794[#27794]

  • Painless: Only allow Painless type names to be the same as the equivalent Java class. {pull}27264[#27264]

Plugins
  • Add client actions to action plugin {pull}28280[#28280] (issue: {issue}27759[#27759])

  • Plugins: Add validation to plugin descriptor parsing {pull}27951[#27951]

  • Plugins: Add plugin extension capabilities {pull}27881[#27881]

  • Add support for filtering mappings fields {pull}27603[#27603]

Rank Evaluation
  • Simplify RankEvalResponse output {pull}28266[#28266]

Recovery
  • Truncate tlog cli should assign global checkpoint {pull}28192[#28192] (issue: {issue}28181[#28181])

  • Replica starts peer recovery with safe commit {pull}28181[#28181] (issue: {issue}10708[#10708])

  • Primary send safe commit in file-based recovery {pull}28038[#28038] (issue: {issue}10708[#10708])

  • Fail resync-failed shards in subsequent writes {pull}28005[#28005]

  • Introduce promoting index shard state {pull}28004[#28004] (issue: {issue}24841[#24841])

  • Non-peer recovery should set the global checkpoint {pull}27965[#27965]

  • Persist global checkpoint when finalizing a peer recovery {pull}27947[#27947] (issue: {issue}27861[#27861])

  • Rollback a primary before recovering from translog {pull}27804[#27804] (issue: {issue}10708[#10708])

Search
  • Use typeName() to check field type in GeoShapeQueryBuilder {pull}27730[#27730]

  • Optimize search_after when sorting in index sort order {pull}26401[#26401]

Security
  • Added the ability to refresh tokens that were created by the token API. The API provides information about a refresh token, which you can use within 24 hours of its creation to extend the life of a token. For more information, see [security-api-tokens].

  • Added principal and role information to access_granted, access_denied, run_as_granted, and run_as_denied audit events. For more information about these events, see [auditing].

  • Added audit event ignore policies, which are a way to tune the verbosity of an audit trail. These policies define rules for ignoring audit events that match specific attribute values. For more information, see [audit-log-ignore-policy].

  • Added a certificates API, which enables you to retrieve information about the X.509 certificates that are used to encrypt communications in your {es} cluster. For more information, see [security-api-ssl].

Sequence IDs
  • Do not keep 5.x commits when having 6.x commits {pull}28188[#28188] (issues: {issue}27606[#27606], {issue}28038[#28038])

  • Use lastSyncedGlobalCheckpoint in deletion policy {pull}27826[#27826] (issue: {issue}27606[#27606])

  • Use CountedBitSet in LocalCheckpointTracker {pull}27793[#27793]

  • Only fsync global checkpoint if needed {pull}27652[#27652]

  • Keep commits and translog up to the global checkpoint {pull}27606[#27606]

  • Adjust CombinedDeletionPolicy for multiple commits {pull}27456[#27456] (issues: {issue}10708[#10708], {issue}27367[#27367])

  • Keeps index commits up to the current global checkpoint {pull}27367[#27367] (issue: {issue}10708[#10708])

  • Dedup translog operations by reading in reverse {pull}27268[#27268] (issue: {issue}10708[#10708])

Settings
  • Add validation of keystore setting names {pull}27626[#27626]

Snapshot/Restore
  • Use AmazonS3.doesObjectExist() method in S3BlobContainer {pull}27723[#27723]

  • Remove XContentType auto detection in BlobStoreRepository {pull}27480[#27480]

  • Include include_global_state in Snapshot status API (#22423) {pull}26853[#26853] (issue: {issue}22423[#22423])

Task Manager
  • Add ability to associate an ID with tasks {pull}27764[#27764] (issue: {issue}23250[#23250])

Translog
  • Simplify MultiSnapshot#SeqNoset {pull}27547[#27547] (issue: {issue}27268[#27268])

  • Enclose CombinedDeletionPolicy in SnapshotDeletionPolicy {pull}27528[#27528] (issues: {issue}27367[#27367], {issue}27456[#27456])

Watcher
  • Added the ability to set the index and doc_type dynamically in an index action. For more information, see [actions-index].

  • Added a refresh index action attribute, which enables you to set the refresh policy of the write request. For more information, see [actions-index].

  • Added support for actions in slack attachments, which enables you to add buttons that can be clicked in slack messages. For more information, see [actions-slack].

  • {watcher} indices (.watch* and triggered_watches) created after upgrading to 6.2 have the auto_expand_replicas: 0-1 setting rather than a fixed setting of 1 replica. As a result, {watcher} indices created after upgrading to 6.2 can have a green status on single node clusters. There is no impact in multi-node clusters.

Bug Fixes

Aggregations
  • Adds metadata to rewritten aggregations {pull}28185[#28185] (issue: {issue}28170[#28170])

  • Fix NPE on composite aggregation with sub-aggregations that need scores {pull}28129[#28129]

  • StringTerms.Bucket.getKeyAsNumber detection type {pull}28118[#28118] (issue: {issue}28012[#28012])

  • Fix incorrect results for aggregations nested under a nested aggregation {pull}27946[#27946] (issue: {issue}27912[#27912])

  • Fix global aggregation that requires breadth first and scores {pull}27942[#27942] (issues: {issue}22321[#22321], {issue}27928[#27928])

  • Fix composite aggregation when after term is missing in the shard {pull}27936[#27936]

  • Fix preserving FiltersAggregationBuilder#keyed field on rewrite {pull}27900[#27900] (issue: {issue}27841[#27841])

  • Using DocValueFormat::parseBytesRef for parsing missing value parameter {pull}27855[#27855] (issue: {issue}27788[#27788])

  • Fix illegal cast of the "low cardinality" optimization of the terms aggregation. {pull}27543[#27543]

  • Always include the _index and _id for nested search hits. {pull}27201[#27201] (issue: {issue}27053[#27053])

Allocation
  • Do not open indices with broken settings {pull}26995[#26995]

Core
  • Fix lock accounting in releasable lock {pull}28202[#28202]

  • Fixes ByteSizeValue to serialise correctly {pull}27702[#27702] (issue: {issue}27568[#27568])

  • Do not set data paths on no local storage required {pull}27587[#27587] (issue: {issue}27572[#27572])

  • Ensure threadcontext is preserved when refresh listeners are invoked {pull}27565[#27565]

  • Ensure logging is configured for CLI commands {pull}27523[#27523] (issue: {issue}27521[#27521])

Engine
  • Replica recovery could go into an endless flushing loop {pull}28350[#28350]

  • Use _refresh to shrink the version map on inactivity {pull}27918[#27918] (issue: {issue}27852[#27852])

  • Allow resize version map under lock even if there are pending operations {pull}27870[#27870] (issue: {issue}27852[#27852])

  • Reset LiveVersionMap on sync commit {pull}27534[#27534] (issue: {issue}27516[#27516])

Geo
  • Correct two equality checks on incomparable types {pull}27688[#27688]

  • Handle case where the hole vertex is south of the containing polygon(s) {pull}27685[#27685] (issue: {issue}25933[#25933])

Highlighting
  • Fix highlighting on a keyword field that defines a normalizer {pull}27604[#27604]

Inner Hits
  • Add version support for inner hits in field collapsing (#27822) {pull}27833[#27833] (issue: {issue}27822[#27822])

Internal
  • Never return null from Strings.tokenizeToStringArray {pull}28224[#28224] (issue: {issue}28213[#28213])

  • Fallback to TransportMasterNodeAction for cluster health retries {pull}28195[#28195] (issue: {issue}28169[#28169])

  • Retain originalIndex info when rewriting FieldCapabilities requests {pull}27761[#27761]

Java REST Client
  • Do not use system properties when building the HttpAsyncClient {pull}27829[#27829] (issue: {issue}27827[#27827])

Machine Learning
  • Improved error reporting for crashes and resource problems on Linux.

  • Improved the detection of seasonal trends in bucket spans longer than 1 hour.

  • Updated the forecast API to wait for validation and return an error if the validation fails.

  • Set the actual bucket value to 0 in model plots for empty buckets for count and sum functions. The count and sum functions treat empty buckets as 0 rather than unknown for anomaly detection, so it was inconsistent not to do the same for model plots. This inconsistency resulted in problems plotting these buckets in {kib}.

Mapping
  • Ignore null value for range field (#27845) {pull}28116[#28116] (issue: {issue}27845[#27845])

  • Pass java.locale.providers=COMPAT to Java 9 onwards {pull}28080[#28080] (issue: {issue}10984[#10984])

  • Allow update of eager_global_ordinals on _parent. {pull}28014[#28014] (issue: {issue}24407[#24407])

  • Fix merging of _meta field {pull}27352[#27352] (issue: {issue}27323[#27323])

Network
  • Only bind loopback addresses when binding to local {pull}28029[#28029] (issue: {issue}1877[#1877])

  • Remove potential nio selector leak {pull}27825[#27825]

  • Fix issue where the incorrect buffers are written {pull}27695[#27695] (issue: {issue}27551[#27551])

  • Throw UOE from compressible bytes stream reset {pull}27564[#27564] (issue: {issue}24927[#24927])

  • Bubble exceptions when closing compressible streams {pull}27542[#27542] (issue: {issue}27540[#27540])

Packaging
  • Allow custom service names when installing on windows {pull}25255[#25255] (issue: {issue}25231[#25231])

Percolator
  • Avoid TooManyClauses exception if number of terms / ranges is exactly equal to 1024 {pull}27519[#27519] (issue: {issue}1[#1])

Plugin Analysis ICU
  • Catch InvalidPathException in IcuCollationTokenFilterFactory {pull}27202[#27202]

Plugin Analysis Phonetic
  • Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter {pull}28225[#28225] (issue: {issue}28211[#28211])

Plugin Lang Painless
  • Painless: Fix variable scoping issue in lambdas {pull}27571[#27571] (issue: {issue}26760[#26760])

  • Painless: Fix errors allowing void to be assigned to def. {pull}27460[#27460] (issue: {issue}27210[#27210])

Plugin Repository HDFS
  • Fix SecurityException when HDFS Repository used against HA Namenodes {pull}27196[#27196]

Plugins
  • Make sure that we don’t detect files as maven coordinate when installing a plugin {pull}28163[#28163]

  • Fix upgrading indices which use a custom similarity plugin. {pull}26985[#26985] (issue: {issue}25350[#25350])

Recovery
  • Open engine should keep only starting commit {pull}28228[#28228] (issues: {issue}27804[#27804], {issue}28181[#28181])

  • Allow shrinking of indices from a previous major {pull}28076[#28076] (issue: {issue}28061[#28061])

  • Set global checkpoint before open engine from store {pull}27972[#27972] (issues: {issue}27965[#27965], {issue}27970[#27970])

  • Check and repair index under the store metadata lock {pull}27768[#27768] (issues: {issue}24481[#24481], {issue}24787[#24787], {issue}27731[#27731])

  • Flush old indices on primary promotion and relocation {pull}27580[#27580] (issue: {issue}27536[#27536])

Rollover
  • Make index rollover action atomic {pull}28039[#28039] (issue: {issue}26976[#26976])

Scripting
  • Ensure we protect Collections obtained from scripts from self-referencing {pull}28335[#28335]

Scroll
  • Reject scroll query if size is 0 (#22552) {pull}27842[#27842] (issue: {issue}22552[#22552])

  • Fix scroll query with a sort that is a prefix of the index sort {pull}27498[#27498]

Search
  • Fix simple_query_string on invalid input {pull}28219[#28219] (issue: {issue}28204[#28204])

  • Use the underlying connection version for CCS connections {pull}28093[#28093]

  • Fix synonym phrase query expansion for cross_fields parsing {pull}28045[#28045]

  • Carry forward weights, etc on rescore rewrite {pull}27981[#27981] (issue: {issue}27979[#27979])

  • Fix routing with leading or trailing whitespace {pull}27712[#27712] (issue: {issue}27708[#27708])

Security
  • Updated the setup-passwords command to generate passwords with characters A-Z, a-z, and 0-9, so that they are safe to use in shell scripts. For more information about this command, see [setup-passwords].

  • Improved the error messages that occur if the x-pack directory is missing when you run users commands.

  • Fixed the ordering of realms in a realm chain, which determines the order in which the realms are consulted. For more information, see [realms].

Sequence IDs
  • Recovery from snapshot may leave seq# gaps {pull}27850[#27850]

  • No longer unidle shard during recovery {pull}27757[#27757] (issue: {issue}26591[#26591])

  • Obey translog durability in global checkpoint sync {pull}27641[#27641]

Settings
  • Settings: Introduce settings updater for a list of settings {pull}28338[#28338] (issue: {issue}28047[#28047])

  • Fix setting notification for complex setting (affixMap settings) that could cause transient settings to be ignored {pull}28317[#28317] (issue: {issue}28316[#28316])

  • Fix environment variable substitutions in list setting {pull}28106[#28106] (issue: {issue}27926[#27926])

  • Allow index settings to be reset by wildcards {pull}27671[#27671] (issue: {issue}27537[#27537])

Snapshot/Restore
  • Consistent updates of IndexShardSnapshotStatus {pull}28130[#28130] (issue: {issue}26480[#26480])

  • Avoid concurrent snapshot finalizations when deleting an INIT snapshot {pull}28078[#28078] (issues: {issue}27214[#27214], {issue}27931[#27931], {issue}27974[#27974])

  • Do not start snapshots that are deleted during initialization {pull}27931[#27931]

  • Do not swallow exception in ChecksumBlobStoreFormat.writeAtomic() {pull}27597[#27597]

  • Consistent update of stage and failure message in IndexShardSnapshotStatus {pull}27557[#27557] (issue: {issue}26480[#26480])

  • Fail restore when the shard allocations max retries count is reached {pull}27493[#27493] (issue: {issue}26865[#26865])

  • Delete shard store files before restoring a snapshot {pull}27476[#27476] (issues: {issue}20220[#20220], {issue}26865[#26865])

Stats
  • Fixes DocStats to properly deal with shards that report -1 index size {pull}27863[#27863]

  • Include internal refreshes in refresh stats {pull}27615[#27615]

Term Vectors
  • Fix term vectors generator with keyword and normalizer {pull}27608[#27608] (issue: {issue}27320[#27320])

Watcher

Upgrades

Core
  • Dependencies: Update joda time to 2.9.9 {pull}28261[#28261]

  • upgrade to lucene 7.2.1 {pull}28218[#28218] (issue: {issue}28044[#28044])

  • Upgrade jna from 4.4.0-1 to 4.5.1 {pull}28183[#28183] (issue: {issue}28172[#28172])

Ingest
  • update ingest-attachment to use Tika 1.17 and newer deps {pull}27824[#27824]

{es} version 6.1.4

Enhancements

Core
  • Fix classes that can exit {pull}27518[#27518]

Bug Fixes

Aggregations
  • StringTerms.Bucket.getKeyAsNumber detection type {pull}28118[#28118] (issue: {issue}28012[#28012])

Core
  • Remove special handling for _all in nodes info {pull}28971[#28971] (issue: {issue}28797[#28797])

Engine
  • Avoid class cast exception from index writer {pull}28989[#28989]

  • Maybe die before failing engine {pull}28973[#28973] (issues: {issue}27265[#27265], {issue}28967[#28967])

Scripting
  • Painless: Fix For Loop NullPointerException {pull}28506[#28506] (issue: {issue}28501[#28501])

{es} version 6.1.3

Bug Fixes

Engine
  • Replica recovery could go into an endless flushing loop {pull}28350[#28350]

Internal
  • Never return null from Strings.tokenizeToStringArray {pull}28224[#28224] (issue: {issue}28213[#28213])

  • Fallback to TransportMasterNodeAction for cluster health retries {pull}28195[#28195] (issue: {issue}28169[#28169])

Mapping
  • Allow update of eager_global_ordinals on _parent. {pull}28014[#28014] (issue: {issue}24407[#24407])

Scripting
  • Ensure we protect Collections obtained from scripts from self-referencing {pull}28335[#28335]

Security
  • Improved cache expiry handling in the token service. Previously, if the token service was idle for more than 60 minutes, the key expired and the service failed to generate user tokens.

Settings
  • Fix setting notification for complex setting (affixMap settings) that could cause transient settings to be ignored {pull}28317[#28317] (issue: {issue}28316[#28316])

  • Fix environment variable substitutions in list setting {pull}28106[#28106] (issue: {issue}27926[#27926])

Snapshot/Restore
  • Avoid concurrent snapshot finalizations when deleting an INIT snapshot {pull}28078[#28078] (issues: {issue}27214[#27214], {issue}27931[#27931], {issue}27974[#27974])

  • Do not start snapshots that are deleted during initialization {pull}27931[#27931]

Watcher

{es} version 6.1.2

Enhancements

Internal
  • Make AbstractQueryBuilder.declareStandardFields to be protected (#27865) {pull}27894[#27894] (issue: {issue}27865[#27865])

Added new "Request" object flavored request methods. Prefer these instead of the multi-argument versions. ({pull}29623[#29623])

Bug Fixes

Aggregations
  • Fix incorrect results for aggregations nested under a nested aggregation {pull}27946[#27946] (issue: {issue}27912[#27912])

  • Fix composite aggregation when after term is missing in the shard {pull}27936[#27936]

  • Fix preserving FiltersAggregationBuilder#keyed field on rewrite {pull}27900[#27900] (issue: {issue}27841[#27841])

Engine
  • Use _refresh to shrink the version map on inactivity {pull}27918[#27918] (issue: {issue}27852[#27852])

  • Allow resize version map under lock even if there are pending operations {pull}27870[#27870] (issue: {issue}27852[#27852])

Machine Learning
  • Fixed the removal of tokens during categorization, where the tokens were incorrectly deemed to be hexadecimal numbers. For more information, see {ml-docs}/ml-configuring-categories.html[Categorizing log messages].

  • Reduced the sensitivity of the analysis to small perturbations in the input data.

  • Disabled the ability to create forecasts for jobs that were created before 6.1.0.

Monitoring
  • Added a cluster_alerts.management.blacklist setting for HTTP Exporters, which you can use to block the creation of specific cluster alerts. For more information, see [monitoring-settings].

Network
  • Only bind loopback addresses when binding to local {pull}28029[#28029]

Recovery
  • Allow shrinking of indices from a previous major {pull}28076[#28076] (issue: {issue}28061[#28061])

Search
  • Use the underlying connection version for CCS connections {pull}28093[#28093]

  • Carry forward weights, etc on rescore rewrite {pull}27981[#27981] (issue: {issue}27979[#27979])

Security
  • Fixed an issue in the Active Directory realm when following referrals that resulted in an increase in the number of connections made to Active Directory.

  • Fixed exception that occurred when using auditing and transport clients. In particular, the problem occurred when the number of processors on the transport client did not match the number of processors on the server.

  • Ensured that TLS is not required to install a license if you are using single-node discovery. For more information, see [single-node-discovery] and [ssl-tls].

  • Fixed the has_privileges API. In particular, the has_all_requested field in the API results was not taking cluster privileges into consideration.

Snapshot/Restore
  • Fail restore when the shard allocations max retries count is reached {pull}27493[#27493] (issue: {issue}26865[#26865])

Translog
  • Only sync translog when global checkpoint increased {pull}27973[#27973] (issues: {issue}27837[#27837], {issue}27970[#27970])

Watcher
  • Fixed encoding of UTF-8 data in the HTTP client.

{es} version 6.1.1

Enhancements

Snapshot/Restore
  • Use AmazonS3.doesObjectExist() method in S3BlobContainer {pull}27723[#27723]

Watcher
  • Ensured the watcher thread pool size is reasonably bound. In particular, the watcher thread pool size is now five times the number of processors until 50 threads are reached. If more than 50 cores exist and 50 threads exist, the watch thread pool size grows to match the number of processors.

Bug Fixes

Inner Hits
  • Add version support for inner hits in field collapsing (#27822) {pull}27833[#27833] (issue: {issue}27822[#27822])

Java REST Client
  • Do not use system properties when building the HttpAsyncClient {pull}27829[#27829] (issue: {issue}27827[#27827])

Monitoring
  • Data collectors now all share the same cluster state that existed at the beginning of data collection. This removes the extremely rare race condition where the cluster state can change between some data collectors, which could cause temporary issues in the Monitoring UI.

Search
  • Fix routing with leading or trailing whitespace {pull}27712[#27712] (issue: {issue}27708[#27708])

Sequence IDs
  • Recovery from snapshot may leave seq# gaps {pull}27850[#27850]

  • No longer unidle shard during recovery {pull}27757[#27757] (issue: {issue}26591[#26591])

Watcher
  • Fixed the pagerduty action to send context data. For more information, see [actions-pagerduty].

Upgrades

Ingest
  • update ingest-attachment to use Tika 1.17 and newer deps {pull}27824[#27824]

{es} version 6.1.0

Breaking Changes

Network
  • Allow only a fixed-size receive predictor {pull}26165[#26165] (issue: {issue}23185[#23185])

REST
  • Standardize underscore requirements in parameters {pull}27414[#27414] (issues: {issue}26886[#26886], {issue}27040[#27040])

Scroll
  • Fail queries with scroll that explicitely set request_cache {pull}27342[#27342]

Search
  • Add a limit to from + size in top_hits and inner hits. {pull}26492[#26492] (issue: {issue}11511[#11511])

Security
  • The certgen command now returns validation errors when it encounters problems reading from an input file (with the -in command option). Previously these errors might have been ignored or caused the command to abort with unclear messages. For more information, see [certgen].

Breaking Java Changes

Aggregations
  • Moves deferring code into its own subclass {pull}26421[#26421]

Core
  • Unify Settings xcontent reading and writing {pull}26739[#26739]

Settings
  • Return List instead of an array from settings {pull}26903[#26903]

  • Remove Settings,put(Map<String,String>) {pull}26785[#26785]

Deprecations

Aggregations
  • Deprecate global_ordinals_hash and global_ordinals_low_cardinality {pull}26173[#26173] (issue: {issue}26014[#26014])

Allocation
  • Add deprecation warning for negative index.unassigned.node_left.delayed_timeout {pull}26832[#26832] (issue: {issue}26828[#26828])

Analysis
  • Add limits for ngram and shingle settings {pull}27411[#27411] (issues: {issue}25887[#25887], {issue}27211[#27211])

Geo
  • [GEO] 6x Deprecate ShapeBuilders and decouple geojson parse logic {pull}27345[#27345]

Mapping
  • Deprecate the index_options parameter for numeric fields {pull}26672[#26672] (issue: {issue}21475[#21475])

Plugin Repository Azure
  • Azure repository: Move to named configurations as we do for S3 repository and secure settings {pull}23405[#23405] (issues: {issue}22762[#22762], {issue}22763[#22763])

Search
  • doc: deprecate _primary and _replica shard option {pull}26792[#26792] (issue: {issue}26335[#26335])

New Features

Aggregations
  • Aggregations: bucket_sort pipeline aggregation {pull}27152[#27152] (issue: {issue}14928[#14928])

  • Add composite aggregator {pull}26800[#26800]

Analysis
  • Added Bengali Analyzer to Elasticsearch with respect to the lucene update {pull}26527[#26527]

Ingest
  • add URL-Decode Processor to Ingest {pull}26045[#26045] (issue: {issue}25837[#25837])

Java High Level REST Client
  • Added Delete Index support to high-level REST client {pull}27019[#27019] (issue: {issue}25847[#25847])

Machine Learning
  • Added the ability to create job forecasts. This feature enables you to use historical behavior to predict the future behavior of your time series. You can create forecasts in {kib} or by using the forecast jobs API.

    Note
    You cannot create forecasts for jobs that were created in previous versions; this functionality is available only for jobs created in 6.1 or later.
  • Added overall buckets, which summarize bucket results for multiple jobs. For more information, see the get overall buckets API.

  • Added job groups, which you can use to manage or retrieve information from multiple jobs at once. Also updated many {ml} APIs to support groups and wildcard expressions in the job identifier.

Nested Docs
  • Multi-level Nested Sort with Filters {pull}26395[#26395]

Query DSL
  • Add terms_set query {pull}27145[#27145] (issue: {issue}26915[#26915])

  • Introduce sorted_after query for sorted index {pull}26377[#26377]

  • Add support for auto_generate_synonyms_phrase_query in match_query, multi_match_query, query_string and simple_query_string {pull}26097[#26097]

Search
  • Expose fuzzy_transpositions parameter in fuzzy queries {pull}26870[#26870] (issue: {issue}18348[#18348])

  • Add upper limit for scroll expiry {pull}26448[#26448] (issues: {issue}11511[#11511], {issue}23268[#23268])

  • Implement adaptive replica selection {pull}26128[#26128] (issue: {issue}24915[#24915])

  • configure distance limit {pull}25731[#25731] (issue: {issue}25528[#25528])

Similarities
  • Add a scripted similarity. {pull}25831[#25831]

Suggesters
  • Expose duplicate removal in the completion suggester {pull}26496[#26496] (issue: {issue}23364[#23364])

  • Support must and should for context query in context suggester {pull}26407[#26407] (issues: {issue}24421[#24421], {issue}24565[#24565])

Enhancements

Aggregations
  • Allow aggregation sorting via nested aggregation {pull}26683[#26683] (issue: {issue}16838[#16838])

Allocation
  • Tie-break shard path decision based on total number of shards on path {pull}27039[#27039] (issue: {issue}26654[#26654])

  • Balance shards for an index more evenly across multiple data paths {pull}26654[#26654] (issue: {issue}16763[#16763])

  • Expand "NO" decision message in NodeVersionAllocationDecider {pull}26542[#26542] (issue: {issue}10403[#10403])

  • _reroute’s retry_failed flag should reset failure counter {pull}25888[#25888] (issue: {issue}25291[#25291])

Analysis
  • Add configurable max_token_length parameter to whitespace tokenizer {pull}26749[#26749] (issue: {issue}26643[#26643])

CRUD
  • Add wait_for_active_shards parameter to index open command {pull}26682[#26682] (issue: {issue}20937[#20937])

Core
  • Fix classes that can exit {pull}27518[#27518]

  • Replace empty index block checks with global block checks in template delete/put actions {pull}27050[#27050] (issue: {issue}10530[#10530])

  • Allow Uid#decodeId to decode from a byte array slice {pull}26987[#26987] (issue: {issue}26931[#26931])

  • Use separate searchers for "search visibility" vs "move indexing buffer to disk {pull}26972[#26972] (issues: {issue}15768[#15768], {issue}26802[#26802], {issue}26912[#26912], {issue}3593[#3593])

  • Add ability to split shards {pull}26931[#26931]

  • Make circuit breaker mutations debuggable {pull}26067[#26067] (issue: {issue}25891[#25891])

Dates
  • DateProcessor Locale {pull}26186[#26186] (issue: {issue}25513[#25513])

Discovery
  • Stop responding to ping requests before master abdication {pull}27329[#27329] (issue: {issue}27328[#27328])

Engine
  • Ensure external refreshes will also refresh internal searcher to minimize segment creation {pull}27253[#27253] (issue: {issue}26972[#26972])

  • Move IndexShard#getWritingBytes() under InternalEngine {pull}27209[#27209] (issue: {issue}26972[#26972])

  • Refactor internal engine {pull}27082[#27082]

Geo
  • Add ignore_malformed to geo_shape fields {pull}24654[#24654] (issue: {issue}23747[#23747])

Ingest
  • add json-processor support for non-map json types {pull}27335[#27335] (issue: {issue}25972[#25972])

  • Introduce templating support to timezone/locale in DateProcessor {pull}27089[#27089] (issue: {issue}24024[#24024])

  • Add support for parsing inline script (#23824) {pull}26846[#26846] (issue: {issue}23824[#23824])

  • Consolidate locale parsing. {pull}26400[#26400]

  • Accept ingest simulate params as ints or strings {pull}23885[#23885] (issue: {issue}23823[#23823])

Internal
  • Avoid uid creation in ParsedDocument {pull}27241[#27241]

  • Upgrade to Lucene 7.1.0 snapshot version {pull}26864[#26864] (issue: {issue}26527[#26527])

  • Remove _index fielddata hack if cluster alias is present {pull}26082[#26082] (issue: {issue}25885[#25885])

Java High Level REST Client
  • Adjust RestHighLevelClient method modifiers {pull}27238[#27238]

  • Decouple BulkProcessor from ThreadPool {pull}26727[#26727] (issue: {issue}26028[#26028])

Logging
  • Add more information on failed_to_convert exception (#21946) {pull}27034[#27034] (issue: {issue}21946[#21946])

  • Improve shard-failed log messages. {pull}26866[#26866]

Machine Learning
  • Improved the way {ml} jobs are allocated to nodes, such that it is primarily determined by the estimated memory requirement of the job. If there is insufficient information about the job’s memory requirements, the allocation decision is based on job counts per node.

  • Increased the default value of the xpack.ml.max_open_jobs setting from 10 to 20. The allocation of jobs to nodes now considers memory usage as well as job counts, so it’s reasonable to permit more small jobs on a single node. For more information, see [ml-settings].

  • Decreased the default model_memory_limit property value to 1 GB for new jobs. If you want to create a job that analyzes high cardinality fields, you can increase this property value. For more information, see [ml-apilimits].

  • Improved analytics related to decay rates when predictions are very accurate.

  • Improved analytics related to detecting non-negative quantities and using this information to constrain analysis, predictions, and confidence intervals.

  • Improved periodic trough or spike detection.

  • Improved the speed of the aggregation of {ml} results.

  • Improved probability calculation performance.

  • Expedited bucket processing time in very large populations by determining when there are nearly duplicate values in a bucket and de-duplicating the samples that are added to the model.

  • Improved handling of periodically missing values.

  • Improved analytics related to diurnal periodicity.

  • Reduced memory usage during population analysis by releasing redundant memory after the bucket results are written.

  • Improved modeling of long periodic components, particularly when there is a long bucket span.

Mapping
  • Allow ip_range to accept CIDR notation {pull}27192[#27192] (issue: {issue}26260[#26260])

  • Deduplicate _field_names. {pull}26550[#26550]

  • Throw a better error message for empty field names {pull}26543[#26543] (issue: {issue}23348[#23348])

  • Stricter validation for min/max values for whole numbers {pull}26137[#26137]

  • Make FieldMapper.copyTo() always non-null. {pull}25994[#25994]

Monitoring
  • Added the new interval_ms field to monitoring documents. This field indicates the current collection interval for {es} or external monitored systems.

Nested Docs
  • Use the primary_term field to identify parent documents {pull}27469[#27469] (issue: {issue}24362[#24362])

  • Prohibit using nested_filter, nested_path and new nested Option at the same time in FieldSortBuilder {pull}26490[#26490] (issue: {issue}17286[#17286])

Network
  • Remove manual tracking of registered channels {pull}27445[#27445] (issue: {issue}27260[#27260])

  • Remove tcp profile from low level nio channel {pull}27441[#27441] (issue: {issue}27260[#27260])

  • Decouple ChannelFactory from Tcp classes {pull}27286[#27286] (issue: {issue}27260[#27260])

Percolator
  • Use Lucene’s CoveringQuery to select percolate candidate matches {pull}27271[#27271] (issues: {issue}26081[#26081], {issue}26307[#26307])

  • Add support to percolate query to percolate multiple documents simultaneously {pull}26418[#26418]

  • Hint what clauses are important in a conjunction query based on fields {pull}26081[#26081]

  • Add support for selecting percolator query candidate matches containing range queries {pull}25647[#25647] (issue: {issue}21040[#21040])

Plugin Discovery EC2
  • update AWS SDK for ECS Task IAM support in discovery-ec2 {pull}26479[#26479] (issue: {issue}23039[#23039])

Plugin Lang Painless
  • Painless: Only allow Painless type names to be the same as the equivalent Java class. {pull}27264[#27264]

  • Allow for the Painless Definition to have multiple instances for white-listing {pull}27096[#27096]

  • Separate Painless Whitelist Loading from the Painless Definition {pull}26540[#26540]

  • Remove Sort enum from Painless Definition {pull}26179[#26179]

Plugin Repository Azure
  • Add azure storage endpoint suffix #26432 {pull}26568[#26568] (issue: {issue}26432[#26432])

  • Support for accessing Azure repositories through a proxy {pull}23518[#23518] (issues: {issue}23506[#23506], {issue}23517[#23517])

Plugin Repository S3
  • Remove S3 output stream {pull}27280[#27280] (issue: {issue}27278[#27278])

  • Update to AWS SDK 1.11.223 {pull}27278[#27278]

Plugins
  • Plugins: Add versionless alias to all security policy codebase properties {pull}26756[#26756] (issue: {issue}26521[#26521])

  • Allow plugins to plug rescore implementations {pull}26368[#26368] (issue: {issue}26208[#26208])

Query DSL
  • Add support for wildcard on _index {pull}27334[#27334] (issue: {issue}25722[#25722])

Reindex API
  • Update by Query is modified to accept short script parameter. {pull}26841[#26841] (issue: {issue}24898[#24898])

  • reindex: automatically choose the number of slices {pull}26030[#26030] (issues: {issue}24547[#24547], {issue}25582[#25582])

Rollover
  • Add size-based condition to the index rollover API {pull}27160[#27160] (issue: {issue}27004[#27004])

  • Add size-based condition to the index rollover API {pull}27115[#27115] (issue: {issue}27004[#27004])

Scripting
  • Script: Convert script query to a dedicated script context {pull}26003[#26003]

Search
  • Make fields optional in multi_match query and rely on index.query.default_field by default {pull}27380[#27380]

  • fix unnecessary logger creation {pull}27349[#27349]

  • ObjectParser : replace IllegalStateException with ParsingException {pull}27302[#27302] (issue: {issue}27147[#27147])

  • Uses norms for exists query if enabled {pull}27237[#27237]

  • Cross Cluster Search: make remote clusters optional {pull}27182[#27182] (issues: {issue}26118[#26118], {issue}27161[#27161])

  • Enhances exists queries to reduce need for _field_names {pull}26930[#26930] (issue: {issue}26770[#26770])

  • Change ParentFieldSubFetchPhase to create doc values iterator once per segment {pull}26815[#26815]

  • Change VersionFetchSubPhase to create doc values iterator once per segment {pull}26809[#26809]

  • Change ScriptFieldsFetchSubPhase to create search scripts once per segment {pull}26808[#26808] (issue: {issue}26775[#26775])

  • Make sure SortBuilders rewrite inner nested sorts {pull}26532[#26532]

  • Extend testing of build method in ScriptSortBuilder {pull}26520[#26520] (issues: {issue}17286[#17286], {issue}26490[#26490])

  • Accept an array of field names and boosts in the index.query.default_field setting {pull}26320[#26320] (issue: {issue}25946[#25946])

  • Reject IPv6-mapped IPv4 addresses when using the CIDR notation. {pull}26254[#26254] (issue: {issue}26078[#26078])

  • Rewrite range queries with open bounds to exists query {pull}26160[#26160] (issue: {issue}22640[#22640])

Security
  • Added the manage_index_templates cluster privilege to the built-in role kibana_system. For more information, see [privileges-list-cluster] and [built-in-roles].

  • Newly created or updated watches execute with the privileges of the user that last modified the watch.

  • Added log messages when a PEM key is found when a PEM certificate was expected (or vice versa) in the xpack.ssl.key or xpack.ssl.certificate settings.

  • Added the new certutil command to simplify the creation of certificates for use with the Elastic stack. For more information, see [certutil].

  • Added automatic detection of support for AES 256 bit TLS ciphers and enabled their use when the JVM supports them.

Sequence IDs
  • Only fsync global checkpoint if needed {pull}27652[#27652]

  • Log primary-replica resync failures {pull}27421[#27421] (issues: {issue}24841[#24841], {issue}27418[#27418])

  • Lazy initialize checkpoint tracker bit sets {pull}27179[#27179] (issue: {issue}10708[#10708])

  • Returns the current primary_term for Get/MultiGet requests {pull}27177[#27177] (issue: {issue}26493[#26493])

Settings
  • Allow affix settings to specify dependencies {pull}27161[#27161]

  • Represent lists as actual lists inside Settings {pull}26878[#26878] (issue: {issue}26723[#26723])

  • Remove Settings#getAsMap() {pull}26845[#26845]

  • Replace group map settings with affix setting {pull}26819[#26819]

  • Throw exception if setting isn’t recognized {pull}26569[#26569] (issue: {issue}25607[#25607])

  • Settings: Move keystore creation to plugin installation {pull}26329[#26329] (issue: {issue}26309[#26309])

Snapshot/Restore
  • Remove XContentType auto detection in BlobStoreRepository {pull}27480[#27480]

  • Snapshot: Migrate TransportRequestHandler to TransportMasterNodeAction {pull}27165[#27165] (issue: {issue}27151[#27151])

  • Fix toString of class SnapshotStatus (#26851) {pull}26852[#26852] (issue: {issue}26851[#26851])

Stats
  • Adds average document size to DocsStats {pull}27117[#27117] (issue: {issue}27004[#27004])

  • Stats to record how often the ClusterState diff mechanism is used successfully {pull}27107[#27107] (issue: {issue}26973[#26973])

  • Expose adaptive replica selection stats in /_nodes/stats API {pull}27090[#27090]

  • Add cgroup memory usage/limit to OS stats on Linux {pull}26166[#26166]

  • Add segment attributes to the _segments API. {pull}26157[#26157] (issue: {issue}26130[#26130])

Suggesters
  • Improve error message for parse failures of completion fields {pull}27297[#27297]

  • Support 'AND' operation for context query in context suggester {pull}24565[#24565] (issue: {issue}24421[#24421])

Watcher
  • Improved error messages when there are no accounts configured for {watcher}.

  • Added thread pool rejection information to execution state, which makes it easier to debug execution failures.

  • Added execution state information to watch status details. It is stored in the status.execution_state field.

  • Enabled the account monitoring url field in the xpack.notification.jira setting to support customized paths. For more information about configuring Jira accounts for use with watches, see [actions-jira].

  • Improved handling of exceptions in {watcher} to make it easier to debug problems.

Bug Fixes

Aggregations
  • Disable the "low cardinality" optimization of terms aggregations. {pull}27545[#27545] (issue: {issue}27543[#27543])

  • scripted_metric _agg parameter disappears if params are provided {pull}27159[#27159] (issues: {issue}19768[#19768], {issue}19863[#19863])

Cluster
  • Properly format IndexGraveyard deletion date as date {pull}27362[#27362]

  • Remove optimisations to reuse objects when applying a new ClusterState {pull}27317[#27317]

Core
  • Do not set data paths on no local storage required {pull}27587[#27587] (issue: {issue}27572[#27572])

  • Ensure threadcontext is preserved when refresh listeners are invoked {pull}27565[#27565]

  • Ensure logging is configured for CLI commands {pull}27523[#27523] (issue: {issue}27521[#27521])

  • Protect shard splitting from illegal target shards {pull}27468[#27468] (issue: {issue}26931[#26931])

  • Avoid NPE when getting build information {pull}27442[#27442]

  • Fix ShardSplittingQuery to respect nested documents. {pull}27398[#27398] (issue: {issue}27378[#27378])

  • When building Settings do not set SecureSettings if empty {pull}26988[#26988] (issue: {issue}316[#316])

Engine
  • Reset LiveVersionMap on sync commit {pull}27534[#27534] (issue: {issue}27516[#27516])

  • Carry over version map size to prevent excessive resizing {pull}27516[#27516] (issue: {issue}20498[#20498])

Geo
  • Correct two equality checks on incomparable types {pull}27688[#27688]

  • [GEO] fix pointsOnly bug for MULTIPOINT {pull}27415[#27415]

Index Templates
  • Prevent constructing an index template without index patterns {pull}27662[#27662]

Ingest
  • Add pipeline support for REST API bulk upsert {pull}27075[#27075] (issue: {issue}25601[#25601])

  • Fixing Grok pattern for Apache 2.4 {pull}26635[#26635]

Inner Hits
  • Return an empty _source for nested inner hit when filtering on a field that doesn’t exist {pull}27531[#27531]

Internal
  • When checking if key exists in ThreadContextStruct:putHeaders() method,should put requestHeaders in map first {pull}26068[#26068]

  • Adding a refresh listener to a recovering shard should be a noop {pull}26055[#26055]

Java High Level REST Client
  • Register ip_range aggregation with the high level client {pull}26383[#26383]

  • add top hits as a parsed aggregation to the rest high level client {pull}26370[#26370]

Machine Learning
  • Improved handling of scenarios where there are insufficient values to interpolate trend components.

  • Improved calculation of confidence intervals.

  • Fixed degrees of freedom calculation that could lead to excessive error logging.

  • Improved trend modeling with long bucket spans.

  • Fixed timing of when model size statistics are written. Previously, if there were multiple partitions, there could be multiple model size stats docs written within the same bucket.

  • Updated the calculation of the model memory to include the memory used by partition, over, by, or influencer fields.

  • Fixed calculation of the frequency property value for {dfeeds} that use aggregations. The value must be a multiple of the histogram interval. For more information, see {ml-docs}/ml-configuring-aggregation.html[Aggregating data for faster performance].

  • Removed unnecessary messages from logs when a job is forcefully closed.

Mapping
  • Fix dynamic mapping update generation. {pull}27467[#27467]

  • Fix merging of _meta field {pull}27352[#27352] (issue: {issue}27323[#27323])

  • Fixed rounding of bounds in scaled float comparison {pull}27207[#27207] (issue: {issue}27189[#27189])

Nested Docs
  • Ensure nested documents have consistent version and seq_ids {pull}27455[#27455]

  • Prevent duplicate fields when mixing parent and root nested includes {pull}27072[#27072] (issue: {issue}26990[#26990])

Network
  • Throw UOE from compressible bytes stream reset {pull}27564[#27564] (issue: {issue}24927[#24927])

  • Bubble exceptions when closing compressible streams {pull}27542[#27542] (issue: {issue}27540[#27540])

  • Do not set SO_LINGER on server channels {pull}26997[#26997]

  • Do not set SO_LINGER to 0 when not shutting down {pull}26871[#26871] (issue: {issue}26764[#26764])

  • Close TcpTransport on RST in some Spots to Prevent Leaking TIME_WAIT Sockets {pull}26764[#26764] (issue: {issue}26701[#26701])

Packaging
  • Removes minimum master nodes default number {pull}26803[#26803]

  • setgid on /etc/elasticearch on package install {pull}26412[#26412] (issue: {issue}26410[#26410])

Percolator
  • Avoid TooManyClauses exception if number of terms / ranges is exactly equal to 1024 {pull}27519[#27519] (issue: {issue}1[#1])

Plugin Analysis ICU
  • Catch InvalidPathException in IcuCollationTokenFilterFactory {pull}27202[#27202]

Plugin Lang Painless
  • Painless: Fix variable scoping issue in lambdas {pull}27571[#27571] (issue: {issue}26760[#26760])

  • Painless: Fix errors allowing void to be assigned to def. {pull}27460[#27460] (issue: {issue}27210[#27210])

Plugin Repository GCS
  • Create new handlers for every new request in GoogleCloudStorageService {pull}27339[#27339] (issue: {issue}27092[#27092])

Recovery
  • Flush old indices on primary promotion and relocation {pull}27580[#27580] (issue: {issue}27536[#27536])

Reindex API
  • Reindex: Fix headers in reindex action {pull}26937[#26937] (issue: {issue}22976[#22976])

Scroll
  • Fix scroll query with a sort that is a prefix of the index sort {pull}27498[#27498]

Search
  • Fix profiling naming issues {pull}27133[#27133]

  • Fix max score tracking with field collapsing {pull}27122[#27122] (issue: {issue}23840[#23840])

  • Apply missing request options to the expand phase {pull}27118[#27118] (issues: {issue}26649[#26649], {issue}27079[#27079])

  • Calculate and cache result when advanceExact is called {pull}26920[#26920] (issue: {issue}26817[#26817])

  • Filter unsupported relation for RangeQueryBuilder {pull}26620[#26620] (issue: {issue}26575[#26575])

  • Handle leniency for phrase query on a field indexed without positions {pull}26388[#26388]

Security
  • Fixed REST requests that required a body but did not validate it, resulting in null pointer exceptions.

Sequence IDs
  • Obey translog durability in global checkpoint sync {pull}27641[#27641]

  • Fix resync request serialization {pull}27418[#27418] (issue: {issue}24841[#24841])

Settings
  • Allow index settings to be reset by wildcards {pull}27671[#27671] (issue: {issue}27537[#27537])

Snapshot/Restore
  • Do not swallow exception in ChecksumBlobStoreFormat.writeAtomic() {pull}27597[#27597]

  • Delete shard store files before restoring a snapshot {pull}27476[#27476] (issues: {issue}20220[#20220], {issue}26865[#26865])

  • Fix snapshot getting stuck in INIT state {pull}27214[#27214] (issue: {issue}27180[#27180])

  • Fix default value of ignore_unavailable for snapshot REST API (#25359) {pull}27056[#27056] (issue: {issue}25359[#25359])

  • Do not create directory on readonly repository (#21495) {pull}26909[#26909] (issue: {issue}21495[#21495])

Stats
  • Include internal refreshes in refresh stats {pull}27615[#27615]

  • Make Segment statistics aware of segments hold by internal readers {pull}27558[#27558]

  • Ensure doc_stats are changing even if refresh is disabled {pull}27505[#27505]

Watcher
  • Fixed handling of watcher templates. Missing watcher templates can be added by any node if that node has a higher version than the master node.

Upgrades

Core
  • Upgrade to Jackson 2.8.10 {pull}27230[#27230]

  • Upgrade to Lucene 7.1 {pull}27225[#27225]

Plugin Discovery EC2
  • Upgrade AWS SDK Jackson Databind to 2.6.7.1 {pull}27361[#27361] (issues: {issue}27278[#27278], {issue}27359[#27359])

Plugin Discovery GCE
  • Update Google SDK to version 1.23.0 {pull}27381[#27381] (issue: {issue}26636[#26636])

Plugin Lang Painless
  • Upgrade Painless from ANTLR 4.5.1-1 to ANTLR 4.5.3. {pull}27153[#27153]

{es} version 6.0.1

Breaking Changes

Scroll
  • Fail queries with scroll that explicitely set request_cache {pull}27342[#27342]

Enhancements

Core
  • Fix classes that can exit {pull}27518[#27518]

Discovery
  • Stop responding to ping requests before master abdication {pull}27329[#27329] (issue: {issue}27328[#27328])

Plugin Repository S3
  • Remove S3 output stream {pull}27280[#27280] (issue: {issue}27278[#27278])

  • Update to AWS SDK 1.11.223 {pull}27278[#27278]

Search
  • fix unnecessary logger creation {pull}27349[#27349]

Sequence IDs
  • Log primary-replica resync failures {pull}27421[#27421] (issues: {issue}24841[#24841], {issue}27418[#27418])

Snapshot/Restore
  • Remove XContentType auto detection in BlobStoreRepository {pull}27480[#27480]

Bug Fixes

Cluster
  • Properly format IndexGraveyard deletion date as date {pull}27362[#27362]

Core
  • Do not set data paths on no local storage required {pull}27587[#27587] (issue: {issue}27572[#27572])

  • Ensure threadcontext is preserved when refresh listeners are invoked {pull}27565[#27565]

  • Avoid NPE when getting build information {pull}27442[#27442]

  • When building Settings do not set SecureSettings if empty {pull}26988[#26988] (issue: {issue}316[#316])

Engine
  • Reset LiveVersionMap on sync commit {pull}27534[#27534] (issue: {issue}27516[#27516])

  • Carry over version map size to prevent excessive resizing {pull}27516[#27516] (issue: {issue}20498[#20498])

Inner Hits
  • Return an empty _source for nested inner hit when filtering on a field that doesn’t exist {pull}27531[#27531]

Machine Learning
  • Fixed analytics problem where sparse data resulted in "update failed" errors in the logs.

Mapping
  • Fix dynamic mapping update generation. {pull}27467[#27467]

  • Fixed rounding of bounds in scaled float comparison {pull}27207[#27207] (issue: {issue}27189[#27189])

Nested Docs
  • Ensure nested documents have consistent version and seq_ids {pull}27455[#27455]

Network
  • Throw UOE from compressible bytes stream reset {pull}27564[#27564] (issue: {issue}24927[#24927])

  • Bubble exceptions when closing compressible streams {pull}27542[#27542] (issue: {issue}27540[#27540])

Plugin Lang Painless
  • Painless: Fix errors allowing void to be assigned to def. {pull}27460[#27460] (issue: {issue}27210[#27210])

Plugin Repository GCS
  • Create new handlers for every new request in GoogleCloudStorageService {pull}27339[#27339] (issue: {issue}27092[#27092])

Recovery
  • Flush old indices on primary promotion and relocation {pull}27580[#27580] (issue: {issue}27536[#27536])

Reindex API
  • Reindex: Fix headers in reindex action {pull}26937[#26937] (issue: {issue}22976[#22976])

Search
  • Fix profiling naming issues {pull}27133[#27133]

Security
  • Fixed error that occurred when attempting to audit system_access_granted events.

  • Fixed the setup-passwords command such that it fails appropriately when invalid URLs are specified in the --url option and when {security} is not enabled.

Sequence IDs
  • Fix resync request serialization {pull}27418[#27418] (issue: {issue}24841[#24841])

Snapshot/Restore
  • Do not swallow exception in ChecksumBlobStoreFormat.writeAtomic() {pull}27597[#27597]

  • Delete shard store files before restoring a snapshot {pull}27476[#27476] (issues: {issue}20220[#20220], {issue}26865[#26865])

  • Fix snapshot getting stuck in INIT state {pull}27214[#27214] (issue: {issue}27180[#27180])

  • Fix default value of ignore_unavailable for snapshot REST API (#25359) {pull}27056[#27056] (issue: {issue}25359[#25359])

  • Do not create directory on readonly repository (#21495) {pull}26909[#26909] (issue: {issue}21495[#21495])

Watcher
  • Fixed handling of Hipchat rooms. For example, room names with spaces did not work as expected. For more information, see [actions-hipchat].

Upgrades

Plugin Discovery EC2
  • Upgrade AWS SDK Jackson Databind to 2.6.7.1 {pull}27361[#27361] (issues: {issue}27278[#27278], {issue}27359[#27359])

Plugin Discovery GCE
  • Update Google SDK to version 1.23.0 {pull}27381[#27381] (issue: {issue}26636[#26636])

{es} version 6.0.0

Breaking Changes

Aggregations
  • Change parsing of numeric to and from parameters in date_range aggregation {pull}25376[#25376] (issue: {issue}17920[#17920])

Aliases
  • Wrong behavior deleting alias {pull}23997[#23997] (issues: {issue}10106[#10106], {issue}23960[#23960])

Allocation
  • Remove cluster.routing.allocation.snapshot.relocation_enabled setting {pull}20994[#20994]

Analysis
  • Do not allow custom analyzers to have the same names as built-in analyzers {pull}22349[#22349] (issue: {issue}22263[#22263])

  • Removing query-string parameters in _analyze API {pull}20704[#20704] (issue: {issue}20246[#20246])

CAT API
  • Write -1 on unbounded queue in cat thread pool {pull}21342[#21342] (issue: {issue}21187[#21187])

CRUD
  • Disallow VersionType.FORCE for GetRequest {pull}21079[#21079] (issue: {issue}20995[#20995])

  • Disallow VersionType.FORCE versioning for 6.x indices {pull}20995[#20995] (issue: {issue}20377[#20377])

  • If the index does not exist, delete document will not auto create it {pull}24518[#24518] (issue: {issue}15425[#15425])

Cluster
  • Disallow : in cluster and index/alias names {pull}26247[#26247] (issue: {issue}23892[#23892])

  • No longer allow cluster name in data path {pull}20433[#20433] (issue: {issue}20391[#20391])

Core
  • Simplify file store {pull}24402[#24402] (issue: {issue}24390[#24390])

  • Make boolean conversion strict {pull}22200[#22200]

  • Remove the default store type. {pull}21616[#21616]

  • Remove store throttling. {pull}21573[#21573]

Geo
  • Remove deprecated geo search features {pull}22876[#22876]

  • Reduce GeoDistance Insanity {pull}19846[#19846]

Highlighting
  • Remove the postings highlighter and make unified the default highlighter choice {pull}25028[#25028]

Index APIs
  • Remove (deprecated) support for '+' in index expressions {pull}25274[#25274] (issue: {issue}24515[#24515])

  • Delete index API to work only against concrete indices {pull}25268[#25268] (issues: {issue}2318[#2318], {issue}23997[#23997])

  • Open/Close index api to allow_no_indices by default {pull}24401[#24401] (issues: {issue}24031[#24031], {issue}24341[#24341])

  • Remove support for controversial ignore_unavailable and allow_no_indices from indices exists api {pull}20712[#20712]

Index Templates
  • Allows multiple patterns to be specified for index templates {pull}21009[#21009] (issue: {issue}20690[#20690])

Indexed Scripts/Templates
  • Scripting: Remove search template actions {pull}25717[#25717]

Ingest
  • update ingest-user-agent regexes.yml {pull}25608[#25608]

  • remove ingest.new_date_format {pull}25583[#25583]

Inner Hits
  • Return the _source of inner hit nested as is without wrapping it into its full path context {pull}26982[#26982] (issues: {issue}26102[#26102], {issue}26944[#26944])

Java API
  • Enforce Content-Type requirement on the rest layer and remove deprecated methods {pull}23146[#23146] (issue: {issue}19388[#19388])

Java REST Client
  • Remove deprecated created and found from index, delete and bulk {pull}25516[#25516] (issues: {issue}19566[#19566], {issue}19630[#19630], {issue}19633[#19633])

Mapping
  • Reject out of range numbers for float, double and half_float {pull}25826[#25826] (issue: {issue}25534[#25534])

  • Enforce at most one type. {pull}24428[#24428] (issue: {issue}24317[#24317])

  • Disallow include_in_all for 6.0+ indices {pull}22970[#22970] (issue: {issue}22923[#22923])

  • Disable _all by default, disallow configuring _all on 6.0+ indices {pull}22144[#22144] (issues: {issue}19784[#19784], {issue}20925[#20925], {issue}21341[#21341])

  • Throw an exception on unrecognized "match_mapping_type" {pull}22090[#22090] (issue: {issue}17285[#17285])

Network
  • Remove unused Netty-related settings {pull}26161[#26161]

  • Remove blocking TCP clients and servers {pull}22639[#22639]

  • Remove modules/transport_netty_3 in favor of netty_4 {pull}21590[#21590]

  • Remove LocalTransport in favor of MockTcpTransport {pull}20695[#20695]

Packaging
  • Configure heap dump path out of the box {pull}26755[#26755] (issue: {issue}26665[#26665])

  • Remove support for ES_INCLUDE {pull}25804[#25804]

  • Setup: Change default heap to 1G {pull}25695[#25695]

  • Use config directory to find jvm.options {pull}25679[#25679] (issue: {issue}23004[#23004])

  • Remove implicit 32-bit support {pull}25435[#25435]

  • Remove default path settings {pull}25408[#25408] (issue: {issue}25357[#25357])

  • Remove path.conf setting {pull}25392[#25392] (issue: {issue}25357[#25357])

  • Honor masking of systemd-sysctl.service {pull}24234[#24234] (issues: {issue}21899[#21899], {issue}806[#806])

  • Rename CONF_DIR to ES_PATH_CONF {pull}26197[#26197] (issue: {issue}26154[#26154])

  • Remove customization of ES_USER and ES_GROUP {pull}23989[#23989] (issue: {issue}23848[#23848])

Percolator
  • Remove deprecated percolate and mpercolate apis {pull}22331[#22331]

Plugin Analysis ICU
  • Upgrade icu4j for the ICU analysis plugin to 59.1 {pull}25243[#25243] (issue: {issue}21425[#21425])

  • Upgrade icu4j to latest version {pull}24821[#24821]

Plugin Delete By Query
  • Require explicit query in _delete_by_query API {pull}23632[#23632] (issue: {issue}23629[#23629])

Plugin Discovery Azure Classic
  • Remove discovery.type BWC layer from the EC2/Azure/GCE plugins {pull}25080[#25080] (issue: {issue}24543[#24543])

Plugin Discovery EC2
  • Ec2 Discovery: Cleanup deprecated settings {pull}24150[#24150]

  • Discovery EC2: Remove region setting {pull}23991[#23991] (issue: {issue}22758[#22758])

  • AWS Plugins: Remove signer type setting {pull}23984[#23984] (issue: {issue}22599[#22599])

Plugin Lang JS
  • Remove lang-python and lang-javascript {pull}20734[#20734] (issue: {issue}20698[#20698])

Plugin Mapper Attachment
  • Remove mapper attachments plugin {pull}20416[#20416] (issue: {issue}18837[#18837])

Plugin Repository Azure
  • Remove global repositories.azure settings {pull}23262[#23262] (issues: {issue}22800[#22800], {issue}22856[#22856])

  • Remove auto creation of container for azure repository {pull}22858[#22858] (issue: {issue}22857[#22857])

Plugin Repository GCS
  • GCS Repository: Remove specifying credential file on disk {pull}24727[#24727]

Plugin Repository S3
  • S3 Repository: Cleanup deprecated settings {pull}24097[#24097]

  • S3 Repository: Remove region setting {pull}22853[#22853] (issue: {issue}22758[#22758])

  • S3 Repository: Remove bucket auto create {pull}22846[#22846] (issue: {issue}22761[#22761])

  • S3 Repository: Remove env var and sysprop credentials support {pull}22842[#22842]

  • Remove deprecated S3 settings {pull}24445[#24445]

Plugins
  • Make plugin loading stricter {pull}25405[#25405]

Query DSL
  • Remove deprecated type and slop field in match query {pull}26720[#26720]

  • Remove several parse field deprecations in query builders {pull}26711[#26711]

  • Remove deprecated parameters from ids_query {pull}26508[#26508]

  • Refactor QueryStringQuery for 6.0 {pull}25646[#25646] (issue: {issue}25574[#25574])

  • Change split_on_whitespace default to false {pull}25570[#25570] (issue: {issue}25470[#25470])

  • Remove deprecated template query {pull}24577[#24577] (issue: {issue}19390[#19390])

  • Throw exception in scroll requests using from {pull}26235[#26235] (issue: {issue}9373[#9373])

  • Remove deprecated minimum_number_should_match in BoolQueryBuilder {pull}22416[#22416]

  • Remove support for empty queries {pull}22092[#22092] (issue: {issue}17624[#17624])

  • Remove deprecated query names: in, geo_bbox, mlt, fuzzy_match and match_fuzzy {pull}21852[#21852]

  • The terms query should always map to a Lucene TermsQuery. {pull}21786[#21786]

  • Be strict when parsing values searching for booleans {pull}21555[#21555] (issue: {issue}21545[#21545])

  • Remove collect payloads parameter {pull}20385[#20385]

REST
  • IndexClosedException to return 400 rather than 403 {pull}25752[#25752]

  • Remove comma-separated feature parsing for GetIndicesAction {pull}24723[#24723] (issue: {issue}24437[#24437])

  • Improve REST error handling when endpoint does not support HTTP verb, add OPTIONS support {pull}24437[#24437] (issues: {issue}0[#0], {issue}15335[#15335], {issue}17916[#17916])

  • Remove ldjson support and document ndjson for bulk/msearch {pull}23049[#23049] (issue: {issue}23025[#23025])

  • Enable strict duplicate checks for all XContent types {pull}22225[#22225] (issues: {issue}19614[#19614], {issue}22073[#22073])

  • Enable strict duplicate checks for JSON content {pull}22073[#22073] (issue: {issue}19614[#19614])

  • Remove lenient stats parsing {pull}21417[#21417] (issues: {issue}20722[#20722], {issue}21410[#21410])

  • Remove allow unquoted JSON {pull}20388[#20388] (issues: {issue}17674[#17674], {issue}17801[#17801])

  • Remove FORCE version_type {pull}20377[#20377] (issue: {issue}19769[#19769])

Scripting
  • remove lang url parameter from stored script requests {pull}25779[#25779] (issue: {issue}22887[#22887])

  • Disallow lang to be used with Stored Scripts {pull}25610[#25610]

  • Remove Deprecated Script Settings {pull}24756[#24756] (issue: {issue}24532[#24532])

  • Scripting: Remove native scripts {pull}24726[#24726] (issue: {issue}19966[#19966])

  • Scripting: Remove file scripts {pull}24627[#24627] (issue: {issue}21798[#21798])

  • Make dates be ReadableDateTimes in scripts {pull}22948[#22948] (issue: {issue}22875[#22875])

  • Remove groovy scripting language {pull}21607[#21607]

  • Remove script access to term statistics {pull}19462[#19462] (issue: {issue}19359[#19359])

Search
  • Make index in TermsLookup mandatory {pull}25753[#25753] (issue: {issue}25750[#25750])

  • Removes FieldStats API {pull}25628[#25628] (issue: {issue}25577[#25577])

  • Remove deprecated fielddata_fields from search request {pull}25566[#25566] (issue: {issue}25537[#25537])

  • Removes deprecated fielddata_fields {pull}25537[#25537] (issue: {issue}19027[#19027])

  • ProfileResult and CollectorResult should print machine readable timing information {pull}22561[#22561]

  • Remove indices query {pull}21837[#21837] (issue: {issue}17710[#17710])

  • Remove ignored type parameter in search_shards api {pull}21688[#21688]

Security
  • Added new security limitations:

    • When a user’s role enables document level security for an index and suggesters are specified, the specified suggesters are ignored. For more information about suggesters, see {ref}/search-suggesters.html[Suggesters].

    • When document level security is enabled, search requests cannot be profiled. For more information about profiling, see the {ref}/search-profile.html[Profile API].

Sequence IDs
  • Change certain replica failures not to fail the replica shard {pull}22874[#22874] (issue: {issue}10708[#10708])

Settings
  • Settings: Remove shared setting property {pull}24728[#24728]

  • Settings: Remove support for yaml and json config files {pull}24664[#24664] (issue: {issue}19391[#19391])

Shadow Replicas
  • Remove shadow replicas {pull}23906[#23906] (issue: {issue}22024[#22024])

Similarities
  • Similarity should accept dynamic settings when possible {pull}20339[#20339] (issue: {issue}6727[#6727])

Breaking Java Changes

Aggregations
  • Remove the unused SignificantTerms.compareTerm() method {pull}24714[#24714]

  • Make SignificantTerms.Bucket an interface rather than an abstract class {pull}24670[#24670] (issue: {issue}24492[#24492])

  • Fix NPE when values is omitted on percentile_ranks agg {pull}26046[#26046]

  • Make Terms.Bucket an interface rather than an abstract class {pull}24492[#24492]

  • Compound order for histogram aggregations {pull}22343[#22343] (issues: {issue}14771[#14771], {issue}20003[#20003], {issue}23613[#23613])

Internal
  • Collapses package structure for some bucket aggs {pull}25579[#25579] (issue: {issue}22868[#22868])

Java API
  • Remove deprecated IdsQueryBuilder ctor {pull}25529[#25529]

  • Removing unneeded getTookInMillis method {pull}23923[#23923]

  • Java api: ActionRequestBuilder#execute to return a PlainActionFuture {pull}24415[#24415] (issues: {issue}24412[#24412], {issue}9201[#9201])

Java High Level REST Client
  • Unify the result interfaces from get and search in Java client {pull}25361[#25361] (issue: {issue}16440[#16440])

  • Allow RestHighLevelClient to use plugins {pull}25024[#25024]

Java REST Client
  • Rename client artifacts {pull}25693[#25693] (issue: {issue}20248[#20248])

Network
  • Simplify TransportAddress {pull}20798[#20798]

Plugin Delete By Query
  • Move DeleteByQuery and Reindex requests into core {pull}24578[#24578]

Plugins
  • Drop name from TokenizerFactory {pull}24869[#24869]

Query DSL
  • Remove QueryParseContext {pull}25486[#25486]

  • Remove QueryParseContext from parsing QueryBuilders {pull}25448[#25448]

REST
  • Return index name and empty map for /{index}/_alias with no aliases {pull}25114[#25114] (issues: {issue}24723[#24723], {issue}25090[#25090])

Deprecations

Index APIs
  • Deprecated use of + in index expressions {pull}24585[#24585] (issue: {issue}24515[#24515])

Index Templates
  • Restore deprecation warning for invalid match_mapping_type values {pull}22304[#22304]

Indexed Scripts/Templates
  • Scripting: Deprecate stored search template apis {pull}25437[#25437] (issue: {issue}24596[#24596])

Internal
  • Deprecate XContentType auto detection methods in XContentFactory {pull}22181[#22181] (issue: {issue}19388[#19388])

Percolator
  • Deprecate percolate query’s document_type parameter. {pull}25199[#25199]

Plugins
  • Plugins: Add backcompat for sha1 checksums {pull}26748[#26748] (issue: {issue}26746[#26746])

Scripting
  • Scripting: Change keys for inline/stored scripts to source/id {pull}25127[#25127]

  • Scripting: Deprecate native scripts {pull}24692[#24692] (issue: {issue}19966[#19966])

  • Scripting: Deprecate index lookup {pull}24691[#24691] (issue: {issue}19359[#19359])

  • Deprecate Fine Grain Settings for Scripts {pull}24573[#24573] (issue: {issue}24532[#24532])

  • Scripting: Deprecate file script settings {pull}24555[#24555] (issue: {issue}21798[#21798])

  • Scripting: Deprecate file scripts {pull}24552[#24552] (issue: {issue}21798[#21798])

Settings
  • Settings: Update settings deprecation from yml to yaml {pull}24663[#24663] (issue: {issue}19391[#19391])

  • Deprecate settings in .yml and .json {pull}24059[#24059] (issue: {issue}19391[#19391])

Tribe Node
  • Deprecate tribe service {pull}24598[#24598] (issue: {issue}24581[#24581])

New Features

Aggregations
  • SignificantText aggregation - like significant_terms, but for text {pull}24432[#24432] (issue: {issue}23674[#23674])

Analysis
  • Expose simplepattern and simplepatternsplit tokenizers {pull}25159[#25159] (issue: {issue}23363[#23363])

  • Parse synonyms with the same analysis chain {pull}8049[#8049] (issue: {issue}7199[#7199])

Core
  • Enable index-time sorting {pull}24055[#24055] (issue: {issue}6720[#6720])

Internal
  • Automatically adjust search threadpool queue_size {pull}23884[#23884] (issue: {issue}3890[#3890])

Mapping
  • Add new ip_range field type {pull}24433[#24433]

Parent/Child
  • Move parent_id query to the parent-join module {pull}25072[#25072] (issue: {issue}20257[#20257])

  • Introduce ParentJoinFieldMapper, a field mapper that creates parent/child relation within documents of the same index {pull}24978[#24978] (issue: {issue}20257[#20257])

Plugin Analysis ICU
  • Add ICUCollationFieldMapper {pull}24126[#24126]

Search
  • Automatically early terminate search query based on index sorting {pull}24864[#24864] (issue: {issue}6720[#6720])

Sequence IDs
  • Add a scheduled translog retention check {pull}25622[#25622] (issues: {issue}10708[#10708], {issue}25294[#25294])

  • Initialize sequence numbers on a shrunken index {pull}25321[#25321] (issue: {issue}10708[#10708])

  • Initialize primary term for shrunk indices {pull}25307[#25307] (issue: {issue}10708[#10708])

  • Introduce translog size and age based retention policies {pull}25147[#25147] (issue: {issue}10708[#10708])

Stats
  • Adds nodes usage API to monitor usages of actions {pull}24169[#24169]

Task Manager
  • Task Management [ISSUE] {pull}15117[#15117]

Upgrade API
  • TemplateUpgraders should be called during rolling restart {pull}25263[#25263] (issues: {issue}24379[#24379], {issue}24680[#24680])

Enhancements

Aggregations
  • Add strict parsing of aggregation ranges {pull}25769[#25769]

  • Adds rewrite phase to aggregations {pull}25495[#25495] (issue: {issue}17676[#17676])

  • Tweak AggregatorBase.addRequestCircuitBreakerBytes {pull}25162[#25162] (issue: {issue}24511[#24511])

  • Add superset size to Significant Term REST response {pull}24865[#24865]

  • Add document count to Matrix Stats aggregation response {pull}24776[#24776]

  • Adds an implementation of LogLogBeta for the cardinality aggregation {pull}22323[#22323] (issue: {issue}22230[#22230])

  • Support distance units in GeoHashGrid aggregation precision {pull}26291[#26291] (issue: {issue}5042[#5042])

  • Reject multiple methods in percentiles aggregation {pull}26163[#26163] (issue: {issue}26095[#26095])

  • Use global_ordinals_hash execution mode when sorting by sub aggregations. {pull}26014[#26014] (issue: {issue}24359[#24359])

  • Add a specialized deferring collector for terms aggregator {pull}25190[#25190]

  • Agg builder accessibility fixes {pull}24323[#24323]

  • Remove support for the include/pattern syntax. {pull}23141[#23141] (issue: {issue}22933[#22933])

  • Promote longs to doubles when a terms agg mixes decimal and non-decimal numbers {pull}22449[#22449] (issue: {issue}22232[#22232])

Allocation
  • Adjust status on bad allocation explain requests {pull}25503[#25503] (issue: {issue}25458[#25458])

  • Promote replica on the highest version node {pull}25277[#25277] (issue: {issue}10708[#10708])

Analysis
  • [Analysis] Support normalizer in request param {pull}24767[#24767] (issue: {issue}23347[#23347])

  • Enforce validation for PathHierarchy tokenizer {pull}23510[#23510]

  • [analysis-icu] Allow setting unicodeSetFilter {pull}20814[#20814] (issue: {issue}20820[#20820])

  • Match- and MultiMatchQueryBuilder should only allow setting analyzer on string values {pull}23684[#23684] (issue: {issue}21665[#21665])

Bulk
  • Simplify bulk request execution {pull}20109[#20109]

CAT API
  • expand /_cat/nodes to return information about hard drive {pull}21775[#21775] (issue: {issue}21679[#21679])

CRUD
  • Added validation for upsert request {pull}24282[#24282] (issue: {issue}16671[#16671])

Circuit Breakers
  • ScriptService: Replace max compilation per minute setting with max compilation rate {pull}26399[#26399]

Cluster
  • Validate a joining node’s version with version of existing cluster nodes {pull}25808[#25808]

  • Switch indices read-only if a node runs out of disk space {pull}25541[#25541] (issue: {issue}24299[#24299])

  • Add a cluster block that allows to delete indices that are read-only {pull}24678[#24678]

  • Separate publishing from applying cluster states {pull}24236[#24236]

  • Adds cluster state size to /_cluster/state response {pull}23440[#23440] (issue: {issue}3415[#3415])

Core
  • Allow InputStreamStreamInput array size validation where applicable {pull}26692[#26692]

  • Refactor bootstrap check results and error messages {pull}26637[#26637]

  • Add BootstrapContext to expose settings and recovered state to bootstrap checks {pull}26628[#26628]

  • Unit testable index creation task on MetaDataCreateIndexService {pull}25961[#25961]

  • Ignore .DS_Store files on macOS {pull}27108[#27108] (issue: {issue}23982[#23982])

  • Add max file size bootstrap check {pull}25974[#25974]

  • Add compatibility versions to main action response {pull}25799[#25799]

  • Index ids in binary form. {pull}25352[#25352] (issues: {issue}18154[#18154], {issue}24615[#24615])

  • Explicitly reject duplicate data paths {pull}25178[#25178]

  • Use SPI in High Level Rest Client to load XContent parsers {pull}25097[#25097]

  • Upgrade to lucene-7.0.0-snapshot-a0aef2f {pull}24775[#24775]

  • Speed up PK lookups at index time. {pull}19856[#19856]

  • Use Java 9 FilePermission model {pull}26302[#26302] (issue: {issue}21534[#21534])

  • Add friendlier message on bad keystore permissions {pull}26284[#26284]

  • Epoch millis and second formats accept float implicitly {pull}26119[#26119] (issue: {issue}14641[#14641])

  • Remove connect SocketPermissions from core {pull}22797[#22797]

  • Add repository-url module and move URLRepository {pull}22752[#22752] (issue: {issue}22116[#22116])

  • Remove accept SocketPermissions from core {pull}22622[#22622] (issue: {issue}22116[#22116])

  • Move IfConfig.logIfNecessary call into bootstrap {pull}22455[#22455] (issue: {issue}22116[#22116])

  • Remove artificial default processors limit {pull}20874[#20874] (issue: {issue}20828[#20828])

  • Simplify write failure handling {pull}19105[#19105] (issue: {issue}20109[#20109])

  • Improve bootstrap checks error messages {pull}24548[#24548]

Discovery
  • Allow plugins to validate cluster-state on join {pull}26595[#26595]

Engine
  • Add refresh stats tracking for realtime get {pull}25052[#25052] (issue: {issue}24806[#24806])

  • Introducing a translog deletion policy {pull}24950[#24950]

  • Fill missing sequence IDs up to max sequence ID when recovering from store {pull}24238[#24238] (issue: {issue}10708[#10708])

  • Use sequence numbers to identify out of order delivery in replicas & recovery {pull}24060[#24060] (issue: {issue}10708[#10708])

  • Add replica ops with version conflict to translog {pull}22626[#22626]

  • Clarify global checkpoint recovery {pull}21934[#21934] (issue: {issue}21254[#21254])

  • Move the IndexDeletionPolicy to be engine internal {pull}24930[#24930] (issue: {issue}10708[#10708])

Exceptions
  • IllegalStateException: Only duplicated jar instead of classpath {pull}24953[#24953]

Highlighting
  • Picks offset source for the unified highlighter directly from the es mapping {pull}25747[#25747] (issue: {issue}25699[#25699])

Index APIs
  • Let primary own its replication group {pull}25692[#25692] (issue: {issue}25485[#25485])

  • Create index request should return the index name {pull}25139[#25139] (issue: {issue}23044[#23044])

Index Templates
  • Fix error message for a put index template request without index_patterns {pull}27102[#27102] (issue: {issue}27100[#27100])

Ingest
  • Add Ingest-Processor specific Rest Endpoints & Add Grok endpoint {pull}25059[#25059] (issue: {issue}24725[#24725])

  • Port support for commercial GeoIP2 databases from Logstash. {pull}24889[#24889]

  • add exclude_keys option to KeyValueProcessor {pull}24876[#24876] (issue: {issue}23856[#23856])

  • Allow removing multiple fields in ingest processor {pull}24750[#24750] (issue: {issue}24622[#24622])

  • Add target_field parameter to ingest processors {pull}24133[#24133] (issues: {issue}23228[#23228], {issue}23682[#23682])

Inner Hits
  • Reuse inner hit query weight {pull}24571[#24571] (issue: {issue}23917[#23917])

Internal
  • TemplateUpgradeService should only run on the master {pull}27294[#27294]

  • Cleanup IndexFieldData visibility {pull}25900[#25900]

  • Bump the min compat version to 5.6.0 {pull}25805[#25805]

  • "shard started" should show index and shard ID {pull}25157[#25157]

  • Break out clear scroll logic from TransportClearScrollAction {pull}25125[#25125] (issue: {issue}25094[#25094])

  • Add helper methods to TransportActionProxy to identify proxy actions and requests {pull}25124[#25124]

  • Add remote cluster infrastructure to fetch discovery nodes. {pull}25123[#25123] (issue: {issue}25094[#25094])

  • Add the ability to set eager_global_ordinals in the new parent-join field {pull}25019[#25019]

  • Disallow multiple parent-join fields per mapping {pull}25002[#25002]

  • Remove the need for _UNRELEASED suffix in versions {pull}24798[#24798] (issue: {issue}24768[#24768])

  • Optimize the order of bytes in uuids for better compression. {pull}24615[#24615] (issue: {issue}18209[#18209])

  • Prevent cluster internal ClusterState.Custom impls to leak to a client {pull}26232[#26232]

  • Use holder pattern for lazy deprecation loggers {pull}26218[#26218] (issue: {issue}26210[#26210])

  • Allow ClusterState.Custom to be created on initial cluster states {pull}26144[#26144]

  • Try to convince the JVM not to lose stacktraces {pull}24426[#24426] (issue: {issue}24376[#24376])

  • Make document write requests immutable {pull}23038[#23038]

  • Add assertions enabled helper {pull}24834[#24834]

Java API
  • Always Accumulate Transport Exceptions {pull}25017[#25017] (issue: {issue}23099[#23099])

Java High Level REST Client
  • [DOCS] restructure java clients docs pages {pull}25517[#25517]

  • Use SPI in High Level Rest Client to load XContent parsers {pull}25098[#25098] (issues: {issue}25024[#25024], {issue}25097[#25097])

  • Add support for clear scroll to high level REST client {pull}25038[#25038]

  • Add search scroll method to high level REST client {pull}24938[#24938] (issue: {issue}23331[#23331])

  • Add search method to high level REST client {pull}24796[#24796] (issues: {issue}24794[#24794], {issue}24795[#24795])

  • Make RestHighLevelClient Closeable and simplify its creation {pull}26180[#26180] (issue: {issue}26086[#26086])

  • Add info method to High Level Rest client {pull}23350[#23350]

  • Add support for named xcontent parsers to high level REST client {pull}23328[#23328]

  • Add BulkRequest support to High Level Rest client {pull}23312[#23312]

  • Add UpdateRequest support to High Level Rest client {pull}23266[#23266]

  • Add delete API to the High Level Rest Client {pull}23187[#23187]

  • Add Index API to High Level Rest Client {pull}23040[#23040]

  • Add get/exists method to RestHighLevelClient {pull}22706[#22706]

  • Add fromxcontent methods to delete response {pull}22680[#22680] (issue: {issue}22229[#22229])

  • Add REST high level client gradle submodule and first simple method {pull}22371[#22371]

  • Add doc_count to ParsedMatrixStats {pull}24952[#24952] (issue: {issue}24776[#24776])

  • Add fromXContent method to ClearScrollResponse {pull}24909[#24909]

  • ClearScrollRequest to implement ToXContentObject {pull}24907[#24907]

  • SearchScrollRequest to implement ToXContentObject {pull}24906[#24906] (issue: {issue}3889[#3889])

  • Add aggs parsers for high level REST Client {pull}24824[#24824] (issues: {issue}23965[#23965], {issue}23973[#23973], {issue}23974[#23974], {issue}24085[#24085], {issue}24160[#24160], {issue}24162[#24162], {issue}24182[#24182], {issue}24183[#24183], {issue}24208[#24208], {issue}24213[#24213], {issue}24239[#24239], {issue}24284[#24284], {issue}24312[#24312], {issue}24330[#24330], {issue}24365[#24365], {issue}24371[#24371], {issue}24442[#24442], {issue}24521[#24521], {issue}24524[#24524], {issue}24564[#24564], {issue}24583[#24583], {issue}24589[#24589], {issue}24648[#24648], {issue}24667[#24667], {issue}24675[#24675], {issue}24682[#24682], {issue}24700[#24700], {issue}24706[#24706], {issue}24717[#24717], {issue}24720[#24720], {issue}24738[#24738], {issue}24746[#24746], {issue}24789[#24789], {issue}24791[#24791], {issue}24794[#24794], {issue}24796[#24796], {issue}24822[#24822])

Java REST Client
  • Shade external dependencies in the rest client jar {pull}25780[#25780] (issue: {issue}25208[#25208])

  • RestClient uses system properties and system default SSLContext {pull}25757[#25757] (issue: {issue}23231[#23231])

  • Wrap rest httpclient with doPrivileged blocks {pull}22603[#22603] (issue: {issue}22116[#22116])

Logging
  • Prevent excessive disk consumption by log files {pull}25660[#25660]

  • Use LRU set to reduce repeat deprecation messages {pull}25474[#25474] (issue: {issue}25457[#25457])

Mapping
  • More efficient encoding of range fields. {pull}26470[#26470] (issue: {issue}26443[#26443])

  • Don’t detect source’s XContentType in DocumentParser.parseDocument() {pull}26880[#26880]

  • Better validation of copy_to. {pull}25983[#25983]

  • Optimize terms queries on ip addresses to use a PointInSetQuery whenever possible. {pull}25669[#25669] (issue: {issue}25667[#25667])

  • Loosen the restrictions on disabling _all in 6.x {pull}26259[#26259]

  • Date detection should not rely on a hardcoded set of characters. {pull}22171[#22171] (issue: {issue}1694[#1694])

  • Identify documents by their _id. {pull}24460[#24460]

Network
  • Add additional low-level logging handler {pull}26887[#26887]

  • Unwrap causes when maybe dying {pull}26884[#26884]

  • Move TransportStats accounting into TcpTransport {pull}25251[#25251]

  • Simplify connection closing and cleanups in TcpTransport {pull}25250[#25250]

  • Disable the Netty recycler in the client {pull}24793[#24793] (issues: {issue}22452[#22452], {issue}24721[#24721])

  • Remove Netty logging hack {pull}24653[#24653] (issues: {issue}24469[#24469], {issue}5624[#5624], {issue}6568[#6568], {issue}6696[#6696])

  • Isolate SocketPermissions to Netty {pull}23057[#23057]

  • Wrap netty accept/connect ops with doPrivileged {pull}22572[#22572] (issue: {issue}22116[#22116])

  • Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions {pull}22287[#22287] (issue: {issue}22116[#22116])

Packaging
  • Remove memlock suggestion from systemd service {pull}25979[#25979]

  • Set address space limit in systemd service file {pull}25975[#25975]

  • Version option should display if snapshot {pull}25970[#25970]

  • Ignore JVM options before checking Java version {pull}25969[#25969]

  • Also skip JAVA_TOOL_OPTIONS on Windows {pull}25968[#25968]

  • Introduce elasticsearch-env for Windows {pull}25958[#25958]

  • Introduce elasticsearch-env {pull}25815[#25815] (issue: {issue}20286[#20286])

  • Stop exporting HOSTNAME from scripts {pull}25807[#25807]

  • Set number of processes in systemd unit file {pull}24970[#24970] (issue: {issue}20874[#20874])

Parent/Child
  • Remove ParentJoinFieldSubFetchPhase {pull}25550[#25550] (issue: {issue}25363[#25363])

  • Support parent id being specified as number in the _source {pull}25547[#25547]

Percolator
  • Store the QueryBuilder’s Writable representation instead of its XContent representation {pull}25456[#25456]

  • Add support for selecting percolator query candidate matches containing wildcard / prefix queries {pull}25351[#25351]

Plugin Discovery EC2
  • Read ec2 discovery address from aws instance tags {pull}22743[#22743] (issue: {issue}22566[#22566])

Plugin Lang Painless
  • Allow Custom Whitelists in Painless {pull}25557[#25557]

  • Update Painless to Allow Augmentation from Any Class {pull}25360[#25360]

  • Add Needs Methods to Painless Script Context Factories {pull}25267[#25267]

  • Support Script Context Stateful Factory in Painless {pull}25233[#25233]

  • Generate Painless Factory for Creating Script Instances {pull}25120[#25120]

  • Update Painless to Use New Script Contexts {pull}25015[#25015]

  • Optimize instance creation in LambdaBootstrap {pull}24618[#24618]

  • Make Painless Compiler Use an Instance Per Context {pull}24972[#24972]

  • Make PainlessScript An Interface {pull}24966[#24966]

Plugin Repository GCS
  • GCS Repository: Add secure storage of credentials {pull}24697[#24697]

Plugin Repository HDFS
  • Add permission checks before reading from HDFS stream {pull}26716[#26716] (issue: {issue}26714[#26714])

  • Add doPrivilege blocks for socket connect ops in repository-hdfs {pull}22793[#22793] (issue: {issue}22116[#22116])

  • Add Kerberos support for Repo HDFS plugin [ISSUE] {pull}21990[#21990]

Plugin Repository S3
  • S3 Repository: Add back repository level credentials {pull}24609[#24609]

Plugins
  • Adjust SHA-512 supported format on plugin install {pull}27093[#27093]

  • Move tribe to a module {pull}25778[#25778]

  • Plugins can register pre-configured char filters {pull}25000[#25000] (issue: {issue}23658[#23658])

  • Add purge option to remove plugin CLI {pull}24981[#24981]

  • Allow plugins to register pre-configured tokenizers {pull}24751[#24751] (issues: {issue}24223[#24223], {issue}24572[#24572])

  • Move ReindexAction class to core {pull}24684[#24684] (issue: {issue}24578[#24578])

  • Make PreConfiguredTokenFilter harder to misuse {pull}24572[#24572] (issue: {issue}23658[#23658])

  • Plugins: Remove leniency for missing plugins dir {pull}24173[#24173]

  • Add doPrivilege blocks for socket connect operations in plugins {pull}22534[#22534] (issue: {issue}22116[#22116])

Query DSL
  • Make slop optional when parsing span_near query {pull}25677[#25677] (issue: {issue}25642[#25642])

  • Require a field when a seed is provided to the random_score function. {pull}25594[#25594] (issue: {issue}25240[#25240])

  • Add support for auto_generate_synonyms_phrase_query in match_query, multi_match_query, query_string and simple_query_string {pull}23147[#23147]

REST
  • Cat shards bytes {pull}26952[#26952]

  • Refactor PathTrie and RestController to use a single trie for all methods {pull}25459[#25459] (issue: {issue}24437[#24437])

  • Make ObjectParser support string to boolean conversion {pull}24668[#24668] (issue: {issue}21802[#21802])

Recovery
  • Introduce a History UUID as a requirement for ops based recovery {pull}26577[#26577] (issue: {issue}10708[#10708])

  • Goodbye, Translog Views {pull}25962[#25962]

  • Disallow multiple concurrent recovery attempts for same target shard {pull}25428[#25428]

  • Live primary-replica resync (no rollback) {pull}24841[#24841] (issue: {issue}10708[#10708])

  • Peer Recovery: remove maxUnsafeAutoIdTimestamp hand off {pull}24243[#24243] (issue: {issue}24149[#24149])

  • Introduce sequence-number-based recovery {pull}22484[#22484] (issue: {issue}10708[#10708])

Scripting
  • Scripting: Rename SearchScript.needsScores to needs_score {pull}25235[#25235]

  • Scripting: Add optional context parameter to put stored script requests {pull}25014[#25014]

  • Add New Security Script Settings {pull}24637[#24637] (issue: {issue}24532[#24532])

  • Add StatefulFactoryType as optional intermediate factory in script contexts {pull}24974[#24974] (issue: {issue}20426[#20426])

  • Make contexts available to ScriptEngine construction {pull}24896[#24896]

  • Make ScriptEngine.compile generic on the script context {pull}24873[#24873]

  • Add instance and compiled classes to script contexts {pull}24868[#24868]

Search
  • Add soft limit on allowed number of script fields in request {pull}26598[#26598] (issue: {issue}26390[#26390])

  • Add a soft limit for the number of requested doc-value fields {pull}26574[#26574] (issue: {issue}26390[#26390])

  • Rewrite search requests on the coordinating nodes {pull}25814[#25814] (issue: {issue}25791[#25791])

  • Ensure query resources are fetched asynchronously during rewrite {pull}25791[#25791]

  • Introduce a new Rewriteable interface to streamline rewriting {pull}25788[#25788]

  • Reduce the scope of QueryRewriteContext {pull}25787[#25787]

  • Reduce the overhead of timeouts and low-level search cancellation. {pull}25776[#25776]

  • Reduce profiling overhead. {pull}25772[#25772] (issue: {issue}24799[#24799])

  • Prevent can_match requests from sending to incompatible nodes {pull}25705[#25705] (issue: {issue}25704[#25704])

  • Add a shard filter search phase to pre-filter shards based on query rewriting {pull}25658[#25658]

  • Ensure we rewrite common queries to match_none if possible {pull}25650[#25650]

  • Limit the number of concurrent shard requests per search request {pull}25632[#25632]

  • Add cluster name validation to RemoteClusterConnection {pull}25568[#25568]

  • Speed up sorted scroll when the index sort matches the search sort {pull}25138[#25138] (issue: {issue}6720[#6720])

  • Leverage scorerSupplier when applicable. {pull}25109[#25109]

  • Add Cross Cluster Search support for scroll searches {pull}25094[#25094]

  • Track EWMA[1] of task execution time in search threadpool executor {pull}24989[#24989] (issue: {issue}24915[#24915])

  • Query range fields by doc values when they are expected to be more efficient than points {pull}24823[#24823] (issue: {issue}24314[#24314])

  • Search: Fairer balancing when routing searches by session ID {pull}24671[#24671] (issue: {issue}24642[#24642])

  • Add parsing from xContent to Suggest {pull}22903[#22903]

  • Add parsing from xContent to ShardSearchFailure {pull}22699[#22699]

  • Eliminate array access in tight loops when profiling is enabled. {pull}24959[#24959]

  • Support Multiple Inner Hits on a Field Collapse Request {pull}24517[#24517]

  • Expand cross cluster search indices for search requests to the concrete index or to it’s aliases {pull}24502[#24502]

Search Templates
  • Add max concurrent searches to multi template search {pull}24255[#24255] (issues: {issue}20912[#20912], {issue}21907[#21907])

Sequence IDs
  • Roll translog generation on primary promotion {pull}27313[#27313]

  • Restoring from snapshot should force generation of a new history uuid {pull}26694[#26694] (issues: {issue}10708[#10708], {issue}26544[#26544], {issue}26557[#26557], {issue}26577[#26577])

  • Add global checkpoint tracking on the primary {pull}26666[#26666] (issue: {issue}26591[#26591])

  • Introduce global checkpoint background sync {pull}26591[#26591] (issues: {issue}26573[#26573], {issue}26630[#26630], {issue}26666[#26666])

  • Move UNASSIGNED_SEQ_NO and NO_OPS_PERFORMED to SequenceNumbers` {pull}26494[#26494] (issue: {issue}10708[#10708])

  • Move primary term from ReplicationRequest to ConcreteShardRequest {pull}25822[#25822]

  • Add reason to global checkpoint updates on replica {pull}25612[#25612] (issue: {issue}10708[#10708])

  • Introduce primary/replica mode for GlobalCheckPointTracker {pull}25468[#25468]

  • Throw back replica local checkpoint on new primary {pull}25452[#25452] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Update global checkpoint when increasing primary term on replica {pull}25422[#25422] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Enable a long translog retention policy by default {pull}25294[#25294] (issues: {issue}10708[#10708], {issue}25147[#25147])

  • Introduce primary context {pull}25122[#25122] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Block older operations on primary term transition {pull}24779[#24779] (issue: {issue}10708[#10708])

  • Block global checkpoint advances when recovering {pull}24404[#24404] (issue: {issue}10708[#10708])

  • Add primary term to doc write response {pull}24171[#24171] (issue: {issue}10708[#10708])

  • Preserve multiple translog generations {pull}24015[#24015] (issue: {issue}10708[#10708])

  • Introduce translog generation rolling {pull}23606[#23606] (issue: {issue}10708[#10708])

  • Replicate write failures {pull}23314[#23314]

  • Introduce sequence-number-aware translog {pull}22822[#22822] (issue: {issue}10708[#10708])

  • Introduce translog no-op {pull}22291[#22291] (issue: {issue}10708[#10708])

  • Tighten sequence numbers recovery {pull}22212[#22212] (issue: {issue}10708[#10708])

  • Add BWC layer to seq no infra and enable BWC tests {pull}22185[#22185] (issue: {issue}21670[#21670])

  • Add internal _primary_term doc values field, fix _seq_no indexing {pull}21637[#21637] (issues: {issue}10708[#10708], {issue}21480[#21480])

  • Add global checkpoint to translog checkpoints {pull}21254[#21254]

  • Sequence numbers commit data for Lucene uses Iterable interface {pull}20793[#20793] (issue: {issue}10708[#10708])

  • Simplify GlobalCheckpointService and properly hook it for cluster state updates {pull}20720[#20720]

  • Fill gaps on primary promotion {pull}24945[#24945] (issue: {issue}10708[#10708])

  • Introduce clean transition on primary promotion {pull}24925[#24925] (issue: {issue}10708[#10708])

  • Guarantee that translog generations are seqNo conflict free {pull}24825[#24825] (issues: {issue}10708[#10708], {issue}24779[#24779])

  • Inline global checkpoints {pull}24513[#24513] (issue: {issue}10708[#10708])

Settings
  • Add disk threshold settings validation {pull}25600[#25600] (issue: {issue}25560[#25560])

  • Enable cross-setting validation {pull}25560[#25560] (issue: {issue}25541[#25541])

  • Validate transport.profiles.* settings {pull}25508[#25508]

  • Cleanup network / transport related settings {pull}25489[#25489]

  • Emit settings deprecation logging at most once {pull}25457[#25457]

  • IndexMetaData: Introduce internal format index setting {pull}25292[#25292]

  • Persist created keystore on startup unless keystore is present {pull}26253[#26253] (issue: {issue}26126[#26126])

  • Settings: Add keystore.seed auto generated secure setting {pull}26149[#26149]

  • Settings: Add keystore creation to add commands {pull}26126[#26126]

Snapshot/Restore
  • Fixed references to Multi Index Syntax {pull}27283[#27283]

  • Improves snapshot logging and snapshot deletion error handling {pull}25264[#25264]

  • Enhances get snapshots API to allow retrieving repository index only {pull}24477[#24477] (issue: {issue}24288[#24288])

Stats
  • Update IndexShard#refreshMetric via a ReferenceManager.RefreshListener {pull}25083[#25083] (issues: {issue}24806[#24806], {issue}25052[#25052])

  • Expose disk usage estimates in nodes stats {pull}22081[#22081] (issue: {issue}8686[#8686])

Store
  • Remote support for lucene versions without checksums {pull}24021[#24021]

Suggesters
  • Remove deprecated _suggest endpoint {pull}22203[#22203] (issue: {issue}20305[#20305])

Task Manager
  • Add descriptions to bulk tasks {pull}22059[#22059] (issue: {issue}21768[#21768])

Translog
  • Translog file recovery should not rely on lucene commits {pull}25005[#25005] (issue: {issue}24950[#24950])

Bug Fixes

Aggregations
  • Do not delegate a null scorer to LeafBucketCollectors {pull}26747[#26747] (issue: {issue}26611[#26611])

  • Create weights lazily in filter and filters aggregation {pull}26983[#26983]

  • Fix IndexOutOfBoundsException in histograms for NaN doubles (#26787) {pull}26856[#26856] (issue: {issue}26787[#26787])

  • Scripted_metric _agg parameter disappears if params are provided {pull}19863[#19863] (issue: {issue}19768[#19768])

  • Fixes array out of bounds for value count agg {pull}26038[#26038] (issue: {issue}17379[#17379])

  • Aggregations bug: Significant_text fails on arrays of text. {pull}25030[#25030] (issue: {issue}25029[#25029])

  • Check bucket metric ages point to a multi bucket agg {pull}26215[#26215] (issue: {issue}25775[#25775])

  • Terms aggregation should remap global ordinal buckets when a sub-aggregator is used to sort the terms {pull}24941[#24941] (issue: {issue}24788[#24788])

  • Correctly set doc_count when MovAvg "predicts" values on existing buckets {pull}24892[#24892] (issue: {issue}24327[#24327])

  • DateHistogram: Fix extended_bounds with offset {pull}23789[#23789] (issue: {issue}23776[#23776])

  • Fix ArrayIndexOutOfBoundsException when no ranges are specified in the query {pull}23241[#23241] (issue: {issue}22881[#22881])

Aliases
  • mget with an alias shouldn’t ignore alias routing {pull}25697[#25697] (issue: {issue}25696[#25696])

  • GET aliases should 404 if aliases are missing {pull}25043[#25043] (issue: {issue}24644[#24644])

Allocation
  • Fix DiskThresholdMonitor flood warning {pull}26204[#26204] (issue: {issue}26201[#26201])

  • Allow wildcards for shard IP filtering {pull}26187[#26187] (issues: {issue}22591[#22591], {issue}26184[#26184])

Analysis
  • Pre-configured shingle filter should disable graph analysis {pull}25853[#25853] (issue: {issue}25555[#25555])

  • PatternAnalyzer should lowercase wildcard queries when lowercase is true. {pull}24967[#24967]

CAT API
  • Fix NPE for /_cat/indices when no primary shard {pull}26953[#26953] (issue: {issue}26942[#26942])

CRUD
  • Serialize and expose timeout of acknowledged requests in REST layer {pull}26189[#26189] (issue: {issue}26213[#26213])

  • Fix silent loss of last command to _bulk and _msearch due to missing newline {pull}25740[#25740] (issue: {issue}7601[#7601])

Cache
  • Reduce the default number of cached queries. {pull}26949[#26949] (issue: {issue}26938[#26938])

  • fix bug of weight computation {pull}24856[#24856]

Circuit Breakers
  • Checks the circuit breaker before allocating bytes for a new big array {pull}25010[#25010] (issue: {issue}24790[#24790])

Cluster
  • Register setting cluster.indices.tombstones.size {pull}26193[#26193] (issue: {issue}26191[#26191])

Core
  • Correctly encode warning headers {pull}27269[#27269] (issue: {issue}27244[#27244])

  • Fix cache compute if absent for expired entries {pull}26516[#26516]

  • Timed runnable should delegate to abstract runnable {pull}27095[#27095] (issue: {issue}27069[#27069])

  • Stop invoking non-existent syscall {pull}27016[#27016] (issue: {issue}20179[#20179])

  • MetaData Builder doesn’t properly prevent an alias with the same name as an index {pull}26804[#26804]

  • Release operation permit on thread-pool rejection {pull}25930[#25930] (issue: {issue}25863[#25863])

  • Node should start up despite of a lingering .es_temp_file {pull}21210[#21210] (issue: {issue}21007[#21007])

  • Fix cache expire after access {pull}24546[#24546]

Dates
  • Fix typo in date format {pull}26503[#26503] (issue: {issue}26500[#26500])

Discovery
  • MasterNodeChangePredicate should use the node instance to detect master change {pull}25877[#25877] (issue: {issue}25471[#25471])

Engine
  • Die with dignity while merging {pull}27265[#27265] (issue: {issue}19272[#19272])

  • Engine - do not index operations with seq# lower than the local checkpoint into lucene {pull}25827[#25827] (issues: {issue}1[#1], {issue}2[#2], {issue}25592[#25592])

Geo
  • Fix typo in GeoUtils#isValidLongitude {pull}25121[#25121]

Highlighting
  • Fix percolator highlight sub fetch phase to not highlight query twice {pull}26622[#26622]

  • FastVectorHighlighter should not cache the field query globally {pull}25197[#25197] (issue: {issue}25171[#25171])

  • Higlighters: Fix MultiPhrasePrefixQuery rewriting {pull}25103[#25103] (issue: {issue}25088[#25088])

  • Fix nested query highlighting {pull}26305[#26305] (issue: {issue}26230[#26230])

Index APIs
  • Shrink API should ignore templates {pull}25380[#25380] (issue: {issue}25035[#25035])

  • Rollover max docs should only count primaries {pull}24977[#24977] (issue: {issue}24217[#24217])

  • Validates updated settings on closed indices {pull}24487[#24487] (issue: {issue}23787[#23787])

Ingest
  • date processor should not fail if timestamp is specified as json number {pull}26986[#26986] (issue: {issue}26967[#26967])

  • date_index_name processor should not fail if timestamp is specified as json number {pull}26910[#26910] (issue: {issue}26890[#26890])

  • Sort Processor does not have proper behavior with targetField {pull}25237[#25237] (issue: {issue}24133[#24133])

  • fix grok’s pattern parsing to validate pattern names in expression {pull}25063[#25063] (issue: {issue}22831[#22831])

  • Remove support for Visio and potm files {pull}22079[#22079] (issue: {issue}22077[#22077])

  • Fix floating-point error when DateProcessor parses UNIX {pull}24947[#24947]

  • add option for _ingest.timestamp to use new ZonedDateTime (5.x backport) {pull}24030[#24030] (issues: {issue}23168[#23168], {issue}23174[#23174])

Inner Hits
  • Do not allow inner hits that fetch _source and have a non nested object field as parent {pull}25749[#25749] (issue: {issue}25315[#25315])

  • When fetching nested inner hits only access stored fields when needed {pull}25864[#25864] (issue: {issue}6[#6])

  • If size / offset are out of bounds just do a plain count {pull}20556[#20556] (issue: {issue}20501[#20501])

  • Fix Source filtering in new field collapsing feature {pull}24068[#24068] (issue: {issue}24063[#24063])

Internal
  • Bump version to 6.0.1 [OPEN] {pull}27386[#27386]

  • IndexShard.routingEntry should only be updated once all internal state is ready {pull}26776[#26776]

  • Catch exceptions and inform handler in RemoteClusterConnection#collectNodes {pull}26725[#26725] (issue: {issue}26700[#26700])

  • Internal: Add versionless alias for rest client codebase in policy files {pull}26521[#26521]

  • Upgrade Lucene to version 7.0.1 {pull}26926[#26926]

  • Fix BytesReferenceStreamInput#skip with offset {pull}25634[#25634]

  • Fix race condition in RemoteClusterConnection node supplier {pull}25432[#25432]

  • Initialise empty lists in BaseTaskResponse constructor {pull}25290[#25290]

  • Extract a common base class for scroll executions {pull}24979[#24979] (issue: {issue}16555[#16555])

  • Obey lock order if working with store to get metadata snapshots {pull}24787[#24787] (issue: {issue}24481[#24481])

  • Fix Version based BWC and set correct minCompatVersion {pull}24732[#24732]

  • Fix _field_caps serialization in order to support cross cluster search {pull}24722[#24722]

  • Avoid race when shutting down controller processes {pull}24579[#24579]

  • Fix handling of document failure exception in InternalEngine {pull}22718[#22718]

  • Ensure remote cluster is connected before fetching _field_caps {pull}24845[#24845] (issue: {issue}24763[#24763])

Java API
  • BulkProcessor flush runnable preserves the thread context from creation time {pull}26718[#26718] (issue: {issue}26596[#26596])

Java High Level REST Client
  • Make RestHighLevelClient’s Request class public {pull}26627[#26627] (issue: {issue}26455[#26455])

  • Forbid direct usage of ContentType.create() methods {pull}26457[#26457] (issues: {issue}22769[#22769], {issue}26438[#26438])

  • Make ShardSearchTarget optional when parsing ShardSearchFailure {pull}27078[#27078] (issue: {issue}27055[#27055])

Java REST Client
  • Better message text for ResponseException {pull}26564[#26564]

  • rest-client-sniffer: configurable threadfactory {pull}26897[#26897]

Logging
  • Allow not configure logging without config {pull}26209[#26209] (issues: {issue}20575[#20575], {issue}24076[#24076])

Machine Learning
  • Fixed a race condition when simultaneous close requests are made for the same job.

Mapping
  • Allow copying from a field to another field that belongs to the same nested object. {pull}26774[#26774] (issue: {issue}26763[#26763])

  • Fixed bug that mapper_parsing_exception is thrown for numeric field with ignore_malformed=true when inserting "NaN" {pull}25967[#25967] (issue: {issue}25289[#25289])

  • Coerce decimal strings for whole number types by truncating the decimal part {pull}25835[#25835] (issue: {issue}25819[#25819])

  • Fix parsing of ip range queries. {pull}25768[#25768] (issue: {issue}25636[#25636])

  • Disable date field mapping changing {pull}25285[#25285] (issue: {issue}25271[#25271])

  • Correctly enable _all for older 5.x indices {pull}25087[#25087] (issue: {issue}25068[#25068])

  • token_count datatype should handle null value {pull}25046[#25046] (issue: {issue}24928[#24928])

  • keep _parent field while updating child type mapping {pull}24407[#24407] (issue: {issue}23381[#23381])

  • ICUCollationKeywordFieldMapper use SortedSetDocValuesField {pull}26267[#26267]

  • Fix serialization of the _all field. {pull}26143[#26143] (issue: {issue}26136[#26136])

More Like This
  • Pass over _routing value with more_like_this items to be retrieved {pull}24679[#24679] (issue: {issue}23699[#23699])

NOT CLASSIFIED
  • DocumentMissingException during Logstash scripted upsert [ISSUE] {pull}27148[#27148]

  • An assertion trips when master opens an index from before 5.x [ISSUE] {pull}24809[#24809]

Nested Docs
  • In case of a single type the _id field should be added to the nested document instead of _uid field {pull}25149[#25149]

  • Inner hits source filtering not working [ISSUE] {pull}23090[#23090]

Network
  • Fixed ByteBuf leaking in org.elasticsearch.http.netty4.Netty4HttpRequestHandler {pull}27222[#27222] (issues: {issue}3[#3], {issue}4[#4], {issue}5[#5], {issue}6[#6])

  • Check for closed connection while opening {pull}26932[#26932]

  • Ensure pending transport handlers are invoked for all channel failures {pull}25150[#25150]

  • Notify onConnectionClosed rather than onNodeDisconnect to prune transport handlers {pull}24639[#24639] (issues: {issue}24557[#24557], {issue}24575[#24575], {issue}24632[#24632])

  • Release pipelined http responses on close {pull}26226[#26226]

  • Fix error message if an incompatible node connects {pull}24884[#24884]

Packaging
  • Fix handling of Windows paths containing parentheses {pull}26916[#26916] (issue: {issue}26454[#26454])

  • Exit Windows scripts promptly on failure {pull}25959[#25959]

  • Pass config path as a system property {pull}25943[#25943]

  • ES_HOME needs to be made absolute before attempt at traversal {pull}25865[#25865]

  • Fix elasticsearch-keystore handling of path.conf {pull}25811[#25811]

  • Stop disabling explicit GC {pull}25759[#25759]

  • Avoid failing install if system-sysctl is masked {pull}25657[#25657] (issue: {issue}24234[#24234])

  • Get short path name for native controllers {pull}25344[#25344]

  • When stopping via systemd only kill the JVM, not its control group {pull}25195[#25195]

  • remove remaining references to scripts directory {pull}24771[#24771]

  • Handle parentheses in batch file path {pull}24731[#24731] (issue: {issue}24712[#24712])

  • Detect modified keystore on package removal {pull}26300[#26300]

  • Create keystore on RPM and Debian package install {pull}26282[#26282]

  • Add safer empty variable checking for Windows {pull}26268[#26268] (issue: {issue}26261[#26261])

  • Export HOSTNAME environment variable {pull}26262[#26262] (issues: {issue}25807[#25807], {issue}26255[#26255])

  • Fix daemonization command status test {pull}26196[#26196] (issue: {issue}26080[#26080])

  • Set RuntimeDirectory in systemd service {pull}23526[#23526]

Parent/Child
  • The default _parent field should not try to load global ordinals {pull}25851[#25851] (issue: {issue}25849[#25849])

Percolator
  • Also support query extraction for queries wrapped inside a ESToParentBlockJoinQuery {pull}26754[#26754]

  • Fix range queries with date range based on current time in percolator queries. {pull}24666[#24666] (issue: {issue}23921[#23921])

Plugin Analysis Kuromoji
  • Fix kuromoji default stoptags {pull}26600[#26600] (issue: {issue}26519[#26519])

Plugin Analysis Phonetic
  • Fix beidermorse phonetic token filter for unspecified languageset {pull}27112[#27112] (issue: {issue}26771[#26771])

Plugin Discovery File
  • Fix discovery-file plugin to use custom config path {pull}26662[#26662] (issue: {issue}26660[#26660])

Plugin Ingest Attachment
  • Add missing mime4j library {pull}22764[#22764] (issue: {issue}22077[#22077])

Plugin Lang Painless
  • Painless: allow doubles to be casted to longs. {pull}25936[#25936]

Plugin Repository Azure
  • Azure snapshots can not be restored anymore {pull}26778[#26778] (issues: {issue}22858[#22858], {issue}26751[#26751], {issue}26777[#26777])

  • Snapshot : azure module - accelerate the listing of files (used in delete snapshot) {pull}25710[#25710] (issue: {issue}25424[#25424])

  • Use Azure upload method instead of our own implementation {pull}26751[#26751]

  • Make calls to CloudBlobContainer#exists privileged {pull}25937[#25937] (issue: {issue}25931[#25931])

Plugin Repository GCS
  • Ensure that gcs client creation is privileged {pull}25938[#25938] (issue: {issue}25932[#25932])

Plugin Repository HDFS
  • Add Log4j to SLF4J binding for repository-hdfs {pull}26514[#26514] (issue: {issue}26512[#26512])

  • Upgrading HDFS Repository Plugin to use HDFS 2.8.1 Client {pull}25497[#25497] (issue: {issue}25450[#25450])

Plugin Repository S3
  • Avoid SecurityException in repository-S3 on DefaultS3OutputStream.flush() {pull}25254[#25254] (issue: {issue}25192[#25192])

  • Wrap getCredentials() in a doPrivileged() block {pull}23297[#23297] (issues: {issue}22534[#22534], {issue}23271[#23271])

Plugins
  • X-Pack plugin download fails on Windows desktop [ISSUE] {pull}24570[#24570]

  • Fix plugin installation permissions {pull}24527[#24527] (issue: {issue}24480[#24480])

Query DSL
  • Fixed incomplete JSON body on count request making org.elasticsearch.rest.action.RestActions#parseTopLevelQueryBuilder go into endless loop {pull}26680[#26680] (issue: {issue}26083[#26083])

  • SpanNearQueryBuilder should return the inner clause when a single clause is provided {pull}25856[#25856] (issue: {issue}25630[#25630])

  • Refactor field expansion for match, multi_match and query_string query {pull}25726[#25726] (issues: {issue}25551[#25551], {issue}25556[#25556])

  • WrapperQueryBuilder should also rewrite the parsed query {pull}25480[#25480]

REST
  • Rest test fixes {pull}27354[#27354]

  • Fix inconsistencies in the rest api specs for cat.snapshots {pull}26996[#26996] (issues: {issue}25737[#25737], {issue}26923[#26923])

  • Fix inconsistencies in the rest api specs for *_script {pull}26971[#26971] (issue: {issue}26923[#26923])

  • exists template needs a template name {pull}25988[#25988]

  • Fix handling of invalid error trace parameter {pull}25785[#25785] (issue: {issue}25774[#25774])

  • Fix handling of exceptions thrown on HEAD requests {pull}25172[#25172] (issue: {issue}21125[#21125])

  • Fixed NPEs caused by requests without content. {pull}23497[#23497] (issue: {issue}24701[#24701])

  • Fix get mappings HEAD requests {pull}23192[#23192] (issue: {issue}21125[#21125])

Recovery
  • Close translog view after primary-replica resync {pull}25862[#25862] (issue: {issue}24841[#24841])

Reindex API
  • Fix update_by_query’s default size parameter {pull}26784[#26784] (issue: {issue}26761[#26761])

  • Reindex: don’t duplicate _source parameter {pull}24629[#24629] (issue: {issue}24628[#24628])

  • Add qa module that tests reindex-from-remote against pre-5.0 versions of Elasticsearch {pull}24561[#24561] (issues: {issue}23828[#23828], {issue}24520[#24520])

Scroll
  • Fix single shard scroll within a cluster with nodes in version >= 5.3 and ⇐ 5.3 {pull}24512[#24512]

Search
  • Fail query when a sort is provided in conjunction with rescorers {pull}26510[#26510]

  • Let search phases override max concurrent requests {pull}26484[#26484] (issue: {issue}26198[#26198])

  • Avoid stack overflow on search phases {pull}27069[#27069] (issue: {issue}27042[#27042])

  • Fix search_after with geo distance sorting {pull}26891[#26891]

  • Fix serialization errors when cross cluster search goes to a single shard {pull}26881[#26881] (issue: {issue}26833[#26833])

  • Early termination with index sorting should not set terminated_early in the response {pull}26597[#26597] (issue: {issue}26408[#26408])

  • Format doc values fields. {pull}22146[#22146]

  • Fix term(s) query for range field {pull}25918[#25918]

  • Caching a MinDocQuery can lead to wrong results. {pull}25909[#25909]

  • Fix random score generation when no seed is provided. {pull}25908[#25908]

  • Merge FunctionScoreQuery and FiltersFunctionScoreQuery {pull}25889[#25889] (issues: {issue}15709[#15709], {issue}23628[#23628])

  • Respect cluster alias in _index aggs and queries {pull}25885[#25885] (issue: {issue}25606[#25606])

  • First increment shard stats before notifying and potentially sending response {pull}25818[#25818]

  • Remove assertion about deviation when casting to a float. {pull}25806[#25806] (issue: {issue}25330[#25330])

  • Prevent skipping shards if a suggest builder is present {pull}25739[#25739] (issue: {issue}25658[#25658])

  • Ensure remote cluster alias is preserved in inner hits aggs {pull}25627[#25627] (issue: {issue}25606[#25606])

  • Do not search locally if remote index pattern resolves to no indices {pull}25436[#25436] (issue: {issue}25426[#25426])

  • Adds check for negative search request size {pull}25397[#25397] (issue: {issue}22530[#22530])

  • Make sure range queries are correctly profiled. {pull}25108[#25108]

  • Fix RangeFieldMapper rangeQuery to properly handle relations {pull}24808[#24808] (issue: {issue}24744[#24744])

  • Fix ExpandSearchPhase when response contains no hits {pull}24688[#24688] (issue: {issue}24672[#24672])

  • Refactor simple_query_string to handle text part like multi_match and query_string {pull}26145[#26145] (issue: {issue}25726[#25726])

  • Fix exists in query_string on empty indices. {pull}25993[#25993] (issue: {issue}25956[#25956])

  • Fix script field sort returning Double.MAX_VALUE for all documents {pull}24942[#24942] (issue: {issue}24940[#24940])

  • Compute the took time of the query after the expand phase of field collapsing {pull}24902[#24902] (issue: {issue}24900[#24900])

Security
  • Prevented 6.0 nodes from joining clusters with un-upgraded version 5 .security indices. For upgrade instructions, see {stack-ref}/upgrading-elastic-stack.html[Upgrading the Elastic Stack].

  • Enabled read-only access to the index audit log by the _xpack internal user. For more information, see [internal-users].

  • Updated the concrete security index such that it is now always named .security-6. In 6.0 beta and RC releases, it was sometimes named .security-v6.

  • Fixed handling of exceptions when retrieving roles from a native roles store. For more information about configuring a native realm, see [native-realm].

Sequence IDs
  • Fire global checkpoint sync under system context {pull}26984[#26984]

  • Fix pre-6.0 response to unknown replication actions {pull}25744[#25744] (issue: {issue}10708[#10708])

  • Track local checkpoint on primary immediately {pull}25434[#25434] (issues: {issue}10708[#10708], {issue}25355[#25355], {issue}25415[#25415])

  • Initialize max unsafe auto ID timestamp on shrink {pull}25356[#25356] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Use correct primary term for replicating NOOPs {pull}25128[#25128]

  • Handle already closed while filling gaps {pull}25021[#25021] (issue: {issue}24925[#24925])

  • TranslogWriter.assertNoSeqNumberConflict failure [ISSUE] {pull}26710[#26710]

  • Avoid losing ops in file-based recovery {pull}22945[#22945] (issue: {issue}22484[#22484])

  • Handle primary failure handling replica response {pull}24926[#24926] (issue: {issue}24935[#24935])

Settings
  • Emit settings deprecation logging on empty update {pull}27017[#27017] (issue: {issue}26419[#26419])

  • Fix filtering for ListSetting {pull}26914[#26914]

  • Fix settings serialization to not serialize secure settings or not take the total size into account {pull}25323[#25323]

  • Keystore CLI should use the AddFileKeyStoreCommand for files {pull}25298[#25298]

  • Allow resetting settings that use an IP validator {pull}24713[#24713] (issue: {issue}24709[#24709])

  • Updating an unrecognized setting should error out with that reason [ISSUE] {pull}25607[#25607]

  • Settings: Fix setting groups to include secure settings {pull}25076[#25076] (issue: {issue}25069[#25069])

Similarities
  • Add boolean similarity to built in similarity types {pull}26613[#26613]

Snapshot/Restore
  • Snapshot/Restore: better handle incorrect chunk_size settings in FS repo {pull}26844[#26844] (issue: {issue}26843[#26843])

  • Snapshot/Restore: Ensure that shard failure reasons are correctly stored in CS {pull}25941[#25941] (issue: {issue}25878[#25878])

  • Output all empty snapshot info fields if in verbose mode {pull}25455[#25455] (issue: {issue}24477[#24477])

  • Remove redundant and broken MD5 checksum from repository-s3 {pull}25270[#25270] (issue: {issue}25269[#25269])

  • Consolidates the logic for cleaning up snapshots on master election {pull}24894[#24894] (issue: {issue}24605[#24605])

  • Removes completed snapshot from cluster state on master change {pull}24605[#24605] (issue: {issue}24452[#24452])

  • Keep snapshot restore state and routing table in sync {pull}20836[#20836] (issue: {issue}19774[#19774])

  • Master failover during snapshotting could leave the snapshot incomplete [OPEN] [ISSUE] {pull}25281[#25281]

  • Fix inefficient (worst case exponential) loading of snapshot repository {pull}24510[#24510] (issue: {issue}24509[#24509])

Stats
  • Fix RestGetAction name typo {pull}27266[#27266]

  • Keep cumulative elapsed scroll time in microseconds {pull}27068[#27068] (issue: {issue}27046[#27046])

  • _nodes/stats should not fail due to concurrent AlreadyClosedException {pull}25016[#25016] (issue: {issue}23099[#23099])

  • Avoid double decrement on current query counter {pull}24922[#24922] (issues: {issue}22996[#22996], {issue}24872[#24872])

  • Adjust available and free bytes to be non-negative on huge FSes {pull}24911[#24911] (issues: {issue}23093[#23093], {issue}24453[#24453])

Suggesters
  • Fix division by zero in phrase suggester that causes assertion to fail {pull}27149[#27149]

  • Context suggester should filter doc values field {pull}25858[#25858] (issue: {issue}25404[#25404])

  • Fix context suggester to read values from keyword type field {pull}24200[#24200] (issue: {issue}24129[#24129])

Templates
  • Tests: Fix FullClusterRestartIT.testSnapshotRestore test failing in 6.x {pull}27218[#27218] (issue: {issue}27213[#27213])

Translog
  • Fix Translog.Delete serialization for sequence numbers {pull}22543[#22543]

Upgrade API
  • Upgrade API: fix excessive logging and unnecessary template updates {pull}26698[#26698] (issue: {issue}26673[#26673])

Regressions

Bulk
  • Only re-parse operation if a mapping update was needed {pull}23832[#23832] (issue: {issue}23665[#23665])

Highlighting
  • Fix Fast Vector Highlighter NPE on match phrase prefix {pull}25116[#25116] (issue: {issue}25088[#25088])

Search
  • Always use DisjunctionMaxQuery to build cross fields disjunction {pull}25115[#25115] (issue: {issue}23966[#23966])

Sequence IDs
  • Indexing performance degradation in 6.0.0-beta1 [ISSUE] {pull}26339[#26339]

Upgrades

Core
  • Upgrade to Lucene 7.0.0 {pull}26744[#26744]

  • Upgrade to lucene-7.0.0-snapshot-d94a5f0. {pull}26441[#26441]

  • Upgrade to lucene-7.0.0-snapshot-a128fcb. {pull}26090[#26090]

  • Upgrade to a Lucene 7 snapshot {pull}24089[#24089] (issues: {issue}23966[#23966], {issue}24086[#24086], {issue}24087[#24087], {issue}24088[#24088])

Logging
  • Upgrade to Log4j 2.9.1 {pull}26750[#26750] (issues: {issue}109[#109], {issue}26464[#26464], {issue}26467[#26467])

  • Upgrade to Log4j 2.9.0 {pull}26450[#26450] (issue: {issue}23798[#23798])

Network
  • Upgrade to Netty 4.1.13.Final {pull}25581[#25581] (issues: {issue}24729[#24729], {issue}6866[#6866])

  • Upgrade to Netty 4.1.11.Final {pull}24652[#24652]

Plugin Ingest Attachment
  • Update to Tika 1.14 {pull}21591[#21591] (issue: {issue}20390[#20390])

Upgrade API
  • Improve stability and logging of TemplateUpgradeServiceIT tests {pull}25386[#25386] (issue: {issue}25382[#25382])

{es} version 6.0.0-rc2

Breaking Changes

Inner Hits
  • Return the _source of inner hit nested as is without wrapping it into its full path context {pull}26982[#26982] (issues: {issue}26102[#26102], {issue}26944[#26944])

Enhancements

Core
  • Ignore .DS_Store files on macOS {pull}27108[#27108] (issue: {issue}23982[#23982])

Index Templates
  • Fix error message for a put index template request without index_patterns {pull}27102[#27102] (issue: {issue}27100[#27100])

Machine Learning
  • Added the xpack.ml.max_model_memory_limit setting, which can be dynamically updated. For more information, see [ml-settings].

  • Added checks and error messages for the ml.enabled and ml.max_open_jobs node attributes. These are reserved for internal use and their values should be set by using xpack.ml.enabled and xpack.ml.max_open_jobs {ml} settings.

Mapping
  • Don’t detect source’s XContentType in DocumentParser.parseDocument() {pull}26880[#26880]

Network
  • Add additional low-level logging handler {pull}26887[#26887]

  • Unwrap causes when maybe dying {pull}26884[#26884]

Plugins
  • Adjust SHA-512 supported format on plugin install {pull}27093[#27093]

REST
  • Cat shards bytes {pull}26952[#26952]

Security
  • Improved the error messages that are returned by the setup-passwords command.

Watcher
  • Added verification that the required templates exist before {watcher} starts. For more information, see [scripts-templates].

  • Added the xpack.watcher.history.cleaner_service.enabled setting. You can use this setting to enable or disable the cleaner service, which removes previous versions of {watcher} indices (for example, .watcher-history*) when it determines that they are old. For more information, see [notification-settings].

Bug Fixes

Aggregations
  • Create weights lazily in filter and filters aggregation {pull}26983[#26983]

  • Fix IndexOutOfBoundsException in histograms for NaN doubles (#26787) {pull}26856[#26856] (issue: {issue}26787[#26787])

  • Scripted_metric _agg parameter disappears if params are provided {pull}19863[#19863] (issue: {issue}19768[#19768])

CAT API
  • Fix NPE for /_cat/indices when no primary shard {pull}26953[#26953] (issue: {issue}26942[#26942])

Cache
  • Reduce the default number of cached queries. {pull}26949[#26949] (issue: {issue}26938[#26938])

Core
  • Timed runnable should delegate to abstract runnable {pull}27095[#27095] (issue: {issue}27069[#27069])

  • Stop invoking non-existent syscall {pull}27016[#27016] (issue: {issue}20179[#20179])

  • MetaData Builder doesn’t properly prevent an alias with the same name as an index {pull}26804[#26804]

Ingest
  • date processor should not fail if timestamp is specified as json number {pull}26986[#26986] (issue: {issue}26967[#26967])

  • date_index_name processor should not fail if timestamp is specified as json number {pull}26910[#26910] (issue: {issue}26890[#26890])

Internal
  • Upgrade Lucene to version 7.0.1 {pull}26926[#26926]

Java High Level REST Client
  • Make ShardSearchTarget optional when parsing ShardSearchFailure {pull}27078[#27078] (issue: {issue}27055[#27055])

Java REST Client
  • rest-client-sniffer: configurable threadfactory {pull}26897[#26897]

Machine Learning
  • Fixed a scenario where models were incorrectly combined. This problem occurred when anomaly detectors were considered to be the same despite having different partition field values.

  • Cleaned up the job closure process for situations where the job was still in the process of opening.

Mapping
  • wrong link target for datatype murmur3 {pull}27143[#27143]

Network
  • Check for closed connection while opening {pull}26932[#26932]

Packaging
  • Fix handling of Windows paths containing parentheses {pull}26916[#26916] (issue: {issue}26454[#26454])

Percolator
  • Also support query extraction for queries wrapped inside a ESToParentBlockJoinQuery {pull}26754[#26754]

Plugin Analysis Phonetic
  • Fix beidermorse phonetic token filter for unspecified languageset {pull}27112[#27112] (issue: {issue}26771[#26771])

Plugin Repository Azure
  • Use Azure upload method instead of our own implementation {pull}26751[#26751]

REST
  • Fix inconsistencies in the rest api specs for cat.snapshots {pull}26996[#26996] (issues: {issue}25737[#25737], {issue}26923[#26923])

  • Fix inconsistencies in the rest api specs for *_script {pull}26971[#26971] (issue: {issue}26923[#26923])

  • exists template needs a template name {pull}25988[#25988]

Reindex API
  • Fix update_by_query’s default size parameter {pull}26784[#26784] (issue: {issue}26761[#26761])

Search
  • Avoid stack overflow on search phases {pull}27069[#27069] (issue: {issue}27042[#27042])

  • Fix search_after with geo distance sorting {pull}26891[#26891]

  • Fix serialization errors when cross cluster search goes to a single shard {pull}26881[#26881] (issue: {issue}26833[#26833])

  • Early termination with index sorting should not set terminated_early in the response {pull}26597[#26597] (issue: {issue}26408[#26408])

  • Format doc values fields. {pull}22146[#22146]

Security
  • Enabled PKI realms to obtain the password for the truststore from either the truststore.secure_password or the truststore.password setting. For more information, see [ref-pki-settings].

  • Fixed document level security such that if your role has authority to access a root document, you also have access to its nested documents.

  • Fixed an issue that caused LDAP authentication requests to be slow and to require multiple binds when authenticating in user search mode.

Sequence IDs
  • Fire global checkpoint sync under system context {pull}26984[#26984]

Settings
  • Emit settings deprecation logging on empty update {pull}27017[#27017] (issue: {issue}26419[#26419])

  • Fix filtering for ListSetting {pull}26914[#26914]

Stats
  • Keep cumulative elapsed scroll time in microseconds {pull}27068[#27068] (issue: {issue}27046[#27046])

Suggesters
  • Fix division by zero in phrase suggester that causes assertion to fail {pull}27149[#27149]

{es} version 6.0.0-rc1

Breaking Changes

Packaging
  • Configure heap dump path out of the box {pull}26755[#26755] (issue: {issue}26665[#26665])

Query DSL
  • Remove deprecated type and slop field in match query {pull}26720[#26720]

  • Remove several parse field deprecations in query builders {pull}26711[#26711]

  • Remove deprecated parameters from ids_query {pull}26508[#26508]

Deprecations

Plugins
  • Plugins: Add backcompat for sha1 checksums {pull}26748[#26748] (issue: {issue}26746[#26746])

Security
  • The xpack.security.authc.token.passphrase is deprecated. If this setting is not used, the cluster automatically generates a key, which is the recommended method. See [security-settings].

Enhancements

Core
  • Allow InputStreamStreamInput array size validation where applicable {pull}26692[#26692]

  • Refactor bootstrap check results and error messages {pull}26637[#26637]

  • Add BootstrapContext to expose settings and recovered state to bootstrap checks {pull}26628[#26628]

  • Unit testable index creation task on MetaDataCreateIndexService {pull}25961[#25961]

Discovery
  • Allow plugins to validate cluster-state on join {pull}26595[#26595]

Mapping
  • More efficient encoding of range fields. {pull}26470[#26470] (issue: {issue}26443[#26443])

Plugin Repository HDFS
  • Add permission checks before reading from HDFS stream {pull}26716[#26716] (issue: {issue}26714[#26714])

Recovery
  • Introduce a History UUID as a requirement for ops based recovery {pull}26577[#26577] (issue: {issue}10708[#10708])

Security
  • Added requirement for TLS/SSL when a cluster with security is running in production. If you try to upgrade to a production license when security is enabled, the upgrade is not successful until you configure TLS. For more information, see [ssl-tls].

  • Added bootstrap check that enforces the use of TLS when security is enabled and you are using a production license.

Scripting
  • ScriptService: Replace max compilation per minute setting with max compilation rate {pull}26399[#26399]

Search
  • Add soft limit on allowed number of script fields in request {pull}26598[#26598] (issue: {issue}26390[#26390])

  • Add a soft limit for the number of requested doc-value fields {pull}26574[#26574] (issue: {issue}26390[#26390])

Sequence IDs
  • Restoring from snapshot should force generation of a new history uuid {pull}26694[#26694] (issues: {issue}10708[#10708], {issue}26544[#26544], {issue}26557[#26557], {issue}26577[#26577])

  • Add global checkpoint tracking on the primary {pull}26666[#26666] (issue: {issue}26591[#26591])

  • Introduce global checkpoint background sync {pull}26591[#26591] (issues: {issue}26573[#26573], {issue}26630[#26630], {issue}26666[#26666])

  • Move UNASSIGNED_SEQ_NO and NO_OPS_PERFORMED to SequenceNumbers` {pull}26494[#26494] (issue: {issue}10708[#10708])

Bug Fixes

Aggregations
  • Do not delegate a null scorer to LeafBucketCollectors {pull}26747[#26747] (issue: {issue}26611[#26611])

Core
  • Fix cache compute if absent for expired entries {pull}26516[#26516]

Dates
  • Fix typo in date format {pull}26503[#26503] (issue: {issue}26500[#26500])

Highlighting
  • Fix percolator highlight sub fetch phase to not highlight query twice {pull}26622[#26622]

Inner Hits
  • Do not allow inner hits that fetch _source and have a non nested object field as parent {pull}25749[#25749] (issue: {issue}25315[#25315])

Internal
  • IndexShard.routingEntry should only be updated once all internal state is ready {pull}26776[#26776]

  • Catch exceptions and inform handler in RemoteClusterConnection#collectNodes {pull}26725[#26725] (issue: {issue}26700[#26700])

  • Internal: Add versionless alias for rest client codebase in policy files {pull}26521[#26521]

Java API
  • BulkProcessor flush runnable preserves the thread context from creation time {pull}26718[#26718] (issue: {issue}26596[#26596])

Java High Level REST Client
  • Make RestHighLevelClient’s Request class public {pull}26627[#26627] (issue: {issue}26455[#26455])

  • Forbid direct usage of ContentType.create() methods {pull}26457[#26457] (issues: {issue}22769[#22769], {issue}26438[#26438])

Java REST Client
  • Better message text for ResponseException {pull}26564[#26564]

Machine Learning
  • Fixed problem with dropped or duplicated data when datafeeds used aggregations.

  • Fixed problems when model plot is enabled and there are sparse metrics.

  • Improved modeling of long-term trends.

  • Fixed a bug in calculation of mean values for seasonal components.

  • Added more accurate adherence to model memory limit.

Mapping
  • Allow copying from a field to another field that belongs to the same nested object. {pull}26774[#26774] (issue: {issue}26763[#26763])

Monitoring
  • Fixed the email message when cluster license expiration issues are resolved.

Plugin Analysis Kuromoji
  • Fix kuromoji default stoptags {pull}26600[#26600] (issue: {issue}26519[#26519])

Plugin Discovery File
  • Fix discovery-file plugin to use custom config path {pull}26662[#26662] (issue: {issue}26660[#26660])

Plugin Repository Azure
  • Azure snapshots can not be restored anymore {pull}26778[#26778] (issues: {issue}22858[#22858], {issue}26751[#26751], {issue}26777[#26777])

  • Snapshot : azure module - accelerate the listing of files (used in delete snapshot) {pull}25710[#25710] (issue: {issue}25424[#25424])

Plugin Repository HDFS
  • Add Log4j to SLF4J binding for repository-hdfs {pull}26514[#26514] (issue: {issue}26512[#26512])

Query DSL
  • Fixed incomplete JSON body on count request making org.elasticsearch.rest.action.RestActions#parseTopLevelQueryBuilder go into endless loop {pull}26680[#26680] (issue: {issue}26083[#26083])

Search
  • Fail query when a sort is provided in conjunction with rescorers {pull}26510[#26510]

  • Let search phases override max concurrent requests {pull}26484[#26484] (issue: {issue}26198[#26198])

Security
  • Added ability infer the keystore type from its pathname when the type is not specified.

  • Added usability improvements for the password bootstrap tool. For more information, see [setup-passwords].

Similarities
  • Add boolean similarity to built in similarity types {pull}26613[#26613]

Upgrade API
  • Upgrade API: fix excessive logging and unnecessary template updates {pull}26698[#26698] (issue: {issue}26673[#26673])

Watcher
  • Fixed {watcher} such that it loads only active watches.

Upgrades

Core
  • Upgrade to Lucene 7.0.0 {pull}26744[#26744]

  • Upgrade to lucene-7.0.0-snapshot-d94a5f0. {pull}26441[#26441]

Logging
  • Upgrade to Log4j 2.9.1 {pull}26750[#26750] (issues: {issue}109[#109], {issue}26464[#26464], {issue}26467[#26467])

  • Upgrade to Log4j 2.9.0 {pull}26450[#26450] (issue: {issue}23798[#23798])

{es} version 6.0.0-beta2

Breaking Changes

Analysis
  • Do not allow custom analyzers to have the same names as built-in analyzers {pull}22349[#22349] (issue: {issue}22263[#22263])

Cluster
  • Disallow : in cluster and index/alias names {pull}26247[#26247] (issue: {issue}23892[#23892])

Inner Hits
  • Unfiltered nested source should keep its full path {pull}26102[#26102] (issues: {issue}18567[#18567], {issue}23090[#23090])

Mapping
  • Reject out of range numbers for float, double and half_float {pull}25826[#25826] (issue: {issue}25534[#25534])

Network
  • Remove unused Netty-related settings {pull}26161[#26161]

Packaging
  • Rename CONF_DIR to ES_PATH_CONF {pull}26197[#26197] (issue: {issue}26154[#26154])

Query DSL
  • Throw exception in scroll requests using from {pull}26235[#26235] (issue: {issue}9373[#9373])

Breaking Java Changes

Aggregations
  • Fix NPE when values is omitted on percentile_ranks agg {pull}26046[#26046]

Deprecations

Machine Learning
  • The max_running_jobs node property is deprecated and is replaced by xpack.ml.max_open_jobs. See [ml-settings].

Enhancements

Aggregations
  • Support distance units in GeoHashGrid aggregation precision {pull}26291[#26291] (issue: {issue}5042[#5042])

  • Reject multiple methods in percentiles aggregation {pull}26163[#26163] (issue: {issue}26095[#26095])

  • Use global_ordinals_hash execution mode when sorting by sub aggregations. {pull}26014[#26014] (issue: {issue}24359[#24359])

  • Add a specialized deferring collector for terms aggregator {pull}25190[#25190]

Core
  • Use Java 9 FilePermission model {pull}26302[#26302] (issue: {issue}21534[#21534])

  • Add friendlier message on bad keystore permissions {pull}26284[#26284]

  • Epoch millis and second formats accept float implicitly {pull}26119[#26119] (issue: {issue}14641[#14641])

Internal
  • Prevent cluster internal ClusterState.Custom impls to leak to a client {pull}26232[#26232]

  • Use holder pattern for lazy deprecation loggers {pull}26218[#26218] (issue: {issue}26210[#26210])

  • Allow ClusterState.Custom to be created on initial cluster states {pull}26144[#26144]

Java High Level REST Client
  • Make RestHighLevelClient Closeable and simplify its creation {pull}26180[#26180] (issue: {issue}26086[#26086])

Machine Learning
  • Added xpack.ml.max_open_jobs as a node attribute. See [ml-settings].

Mapping
  • Loosen the restrictions on disabling _all in 6.x {pull}26259[#26259]

Percolator
  • Store the QueryBuilder’s Writable representation instead of its XContent representation {pull}25456[#25456]

  • Add support for selecting percolator query candidate matches containing wildcard / prefix queries {pull}25351[#25351]

Security
  • Added the keystore.seed setting to create a randomly generated bootstrap password if an actual password is not present.

  • The bootstrap.password secure setting is now managed locally on each node and no longer updates the security index.

  • The xpack.security.authc.token.passphrase setting is no longer mandatory when using the token service. The cluster automatically generates a secure key on startup. See {ref}/security-settings.html[Security Settings in {es}].

  • Added reserved kibana_dashboard_only_user role. For more information, see {kibana-ref}/xpack-dashboard-only-mode.html[Kibana Dashboard Only Mode].

Settings
  • Persist created keystore on startup unless keystore is present {pull}26253[#26253] (issue: {issue}26126[#26126])

  • Settings: Add keystore.seed auto generated secure setting {pull}26149[#26149]

  • Settings: Add keystore creation to add commands {pull}26126[#26126]

Bug Fixes

Aggregations
  • Check bucket metric ages point to a multi bucket agg {pull}26215[#26215] (issue: {issue}25775[#25775])

Allocation
  • Fix DiskThresholdMonitor flood warning {pull}26204[#26204] (issue: {issue}26201[#26201])

  • Allow wildcards for shard IP filtering {pull}26187[#26187] (issues: {issue}22591[#22591], {issue}26184[#26184])

CRUD
  • Serialize and expose timeout of acknowledged requests in REST layer {pull}26189[#26189] (issue: {issue}26213[#26213])

  • Fix silent loss of last command to _bulk and _msearch due to missing newline {pull}25740[#25740] (issue: {issue}7601[#7601])

Cluster
  • Register setting cluster.indices.tombstones.size {pull}26193[#26193] (issue: {issue}26191[#26191])

Highlighting
  • Fix nested query highlighting {pull}26305[#26305] (issue: {issue}26230[#26230])

Logging
  • Allow not configure logging without config {pull}26209[#26209] (issues: {issue}20575[#20575], {issue}24076[#24076])

Machine Learning
  • Fixed calculation of bucket count and empty bucket count statistics.

Mapping
  • ICUCollationKeywordFieldMapper use SortedSetDocValuesField {pull}26267[#26267]

  • Fix serialization of the _all field. {pull}26143[#26143] (issue: {issue}26136[#26136])

Network
  • Release pipelined http responses on close {pull}26226[#26226]

Packaging
  • Detect modified keystore on package removal {pull}26300[#26300]

  • Create keystore on RPM and Debian package install {pull}26282[#26282]

  • Add safer empty variable checking for Windows {pull}26268[#26268] (issue: {issue}26261[#26261])

  • Export HOSTNAME environment variable {pull}26262[#26262] (issues: {issue}25807[#25807], {issue}26255[#26255])

  • Fix daemonization command status test {pull}26196[#26196] (issue: {issue}26080[#26080])

  • Set RuntimeDirectory in systemd service {pull}23526[#23526]

Search
  • Refactor simple_query_string to handle text part like multi_match and query_string {pull}26145[#26145] (issue: {issue}25726[#25726])

  • Fix exists in query_string on empty indices. {pull}25993[#25993] (issue: {issue}25956[#25956])

Security
  • The xpack.security.authc.token.enabled setting now defaults to true when HTTPS is enabled. See [token-service-settings].

  • Improved the safety of file updates in the x-pack/users tool.

  • Bootstrap checks no longer fail when checking secure settings.

  • The setup-password tool no longer fails when using a default elasticsearch.yml configuration file.

  • Fixed validation of the input parameters in the create token API.

Watcher
  • Ensured that a watch can be activated and deactivated during execution.

  • Ensured watch execution always uses the latest watch including its latest status.

Upgrades

Core
  • Upgrade to lucene-7.0.0-snapshot-a128fcb. {pull}26090[#26090]

{es} version 6.0.0-beta1

Breaking Changes

Aggregations
  • Change parsing of numeric to and from parameters in date_range aggregation {pull}25376[#25376] (issue: {issue}17920[#17920])

Aliases
  • Wrong behavior deleting alias {pull}23997[#23997] (issues: {issue}10106[#10106], {issue}23960[#23960])

Highlighting
  • Remove the postings highlighter and make unified the default highlighter choice {pull}25028[#25028]

Index APIs
  • Remove (deprecated) support for '+' in index expressions {pull}25274[#25274] (issue: {issue}24515[#24515])

  • Delete index API to work only against concrete indices {pull}25268[#25268] (issues: {issue}2318[#2318], {issue}23997[#23997])

Indexed Scripts/Templates
  • Scripting: Remove search template actions {pull}25717[#25717]

Ingest
  • update ingest-user-agent regexes.yml {pull}25608[#25608]

  • remove ingest.new_date_format {pull}25583[#25583]

Java REST Client
  • Remove deprecated created and found from index, delete and bulk {pull}25516[#25516] (issues: {issue}19566[#19566], {issue}19630[#19630], {issue}19633[#19633])

Packaging
  • Remove support for ES_INCLUDE {pull}25804[#25804]

  • Setup: Change default heap to 1G {pull}25695[#25695]

  • Use config directory to find jvm.options {pull}25679[#25679] (issue: {issue}23004[#23004])

  • Remove implicit 32-bit support {pull}25435[#25435]

  • Remove default path settings {pull}25408[#25408] (issue: {issue}25357[#25357])

  • Remove path.conf setting {pull}25392[#25392] (issue: {issue}25357[#25357])

  • Honor masking of systemd-sysctl.service {pull}24234[#24234] (issues: {issue}21899[#21899], {issue}806[#806])

Plugin Analysis ICU
  • Upgrade icu4j for the ICU analysis plugin to 59.1 {pull}25243[#25243] (issue: {issue}21425[#21425])

Plugin Discovery Azure Classic
  • Remove discovery.type BWC layer from the EC2/Azure/GCE plugins {pull}25080[#25080] (issue: {issue}24543[#24543])

Plugin Repository GCS
  • GCS Repository: Remove specifying credential file on disk {pull}24727[#24727]

Plugins
  • Make plugin loading stricter {pull}25405[#25405]

Query DSL
  • Refactor QueryStringQuery for 6.0 {pull}25646[#25646] (issue: {issue}25574[#25574])

  • Change split_on_whitespace default to false {pull}25570[#25570] (issue: {issue}25470[#25470])

  • Remove deprecated template query {pull}24577[#24577] (issue: {issue}19390[#19390])

REST
  • IndexClosedException to return 400 rather than 403 {pull}25752[#25752]

  • Remove comma-separated feature parsing for GetIndicesAction {pull}24723[#24723] (issue: {issue}24437[#24437])

  • Improve REST error handling when endpoint does not support HTTP verb, add OPTIONS support {pull}24437[#24437] (issues: {issue}0[#0], {issue}15335[#15335], {issue}17916[#17916])

Scripting
  • remove lang url parameter from stored script requests {pull}25779[#25779] (issue: {issue}22887[#22887])

  • Disallow lang to be used with Stored Scripts {pull}25610[#25610]

  • Remove Deprecated Script Settings {pull}24756[#24756] (issue: {issue}24532[#24532])

  • Scripting: Remove native scripts {pull}24726[#24726] (issue: {issue}19966[#19966])

  • Scripting: Remove file scripts {pull}24627[#24627] (issue: {issue}21798[#21798])

Search
  • Make index in TermsLookup mandatory {pull}25753[#25753] (issue: {issue}25750[#25750])

  • Removes FieldStats API {pull}25628[#25628] (issue: {issue}25577[#25577])

  • Remove deprecated fielddata_fields from search request {pull}25566[#25566] (issue: {issue}25537[#25537])

  • Removes deprecated fielddata_fields {pull}25537[#25537] (issue: {issue}19027[#19027])

Security
  • A new bootstrap check enforces that TLS/SSL is required for inter-node communication when running in production mode. See [encrypting-communications].

  • A new bootstrap check enforces that HTTPS is used by the built-in token service when running in {ref}/bootstrap-checks.html#dev-vs-prod-mode[production mode]. To disable the token service, set xpack.security.authc.token.enabled to false in your elasticsearch.yml. See [token-service-settings].

Settings
  • Settings: Remove shared setting property {pull}24728[#24728]

  • Settings: Remove support for yaml and json config files {pull}24664[#24664] (issue: {issue}19391[#19391])

Similarities
  • Similarity should accept dynamic settings when possible {pull}20339[#20339] (issue: {issue}6727[#6727])

Breaking Java Changes

Aggregations
  • Remove the unused SignificantTerms.compareTerm() method {pull}24714[#24714]

  • Make SignificantTerms.Bucket an interface rather than an abstract class {pull}24670[#24670] (issue: {issue}24492[#24492])

Internal
  • Collapses package structure for some bucket aggs {pull}25579[#25579] (issue: {issue}22868[#22868])

Java API
  • Remove deprecated IdsQueryBuilder ctor {pull}25529[#25529]

  • Removing unneeded getTookInMillis method {pull}23923[#23923]

Java High Level REST Client
  • Unify the result interfaces from get and search in Java client {pull}25361[#25361] (issue: {issue}16440[#16440])

  • Allow RestHighLevelClient to use plugins {pull}25024[#25024]

Java REST Client
  • Rename client artifacts {pull}25693[#25693] (issue: {issue}20248[#20248])

Plugin Delete By Query
  • Move DeleteByQuery and Reindex requests into core {pull}24578[#24578]

Query DSL
  • Remove QueryParseContext {pull}25486[#25486]

  • Remove QueryParseContext from parsing QueryBuilders {pull}25448[#25448]

REST
  • Return index name and empty map for /{index}/_alias with no aliases {pull}25114[#25114] (issues: {issue}24723[#24723], {issue}25090[#25090])

Deprecations

Index APIs
  • Deprecated use of + in index expressions {pull}24585[#24585] (issue: {issue}24515[#24515])

Indexed Scripts/Templates
  • Scripting: Deprecate stored search template apis {pull}25437[#25437] (issue: {issue}24596[#24596])

Percolator
  • Deprecate percolate query’s document_type parameter. {pull}25199[#25199]

Scripting
  • Scripting: Change keys for inline/stored scripts to source/id {pull}25127[#25127]

  • Scripting: Deprecate native scripts {pull}24692[#24692] (issue: {issue}19966[#19966])

  • Scripting: Deprecate index lookup {pull}24691[#24691] (issue: {issue}19359[#19359])

  • Deprecate Fine Grain Settings for Scripts {pull}24573[#24573] (issue: {issue}24532[#24532])

  • Scripting: Deprecate file script settings {pull}24555[#24555] (issue: {issue}21798[#21798])

  • Scripting: Deprecate file scripts {pull}24552[#24552] (issue: {issue}21798[#21798])

Settings
  • Settings: Update settings deprecation from yml to yaml {pull}24663[#24663] (issue: {issue}19391[#19391])

Tribe Node
  • Deprecate tribe service {pull}24598[#24598] (issue: {issue}24581[#24581])

New Features

Analysis
  • Expose simplepattern and simplepatternsplit tokenizers {pull}25159[#25159] (issue: {issue}23363[#23363])

  • Parse synonyms with the same analysis chain {pull}8049[#8049] (issue: {issue}7199[#7199])

Parent/Child
  • Move parent_id query to the parent-join module {pull}25072[#25072] (issue: {issue}20257[#20257])

  • Introduce ParentJoinFieldMapper, a field mapper that creates parent/child relation within documents of the same index {pull}24978[#24978] (issue: {issue}20257[#20257])

Search
  • Automatically early terminate search query based on index sorting {pull}24864[#24864] (issue: {issue}6720[#6720])

Sequence IDs
  • Add a scheduled translog retention check {pull}25622[#25622] (issues: {issue}10708[#10708], {issue}25294[#25294])

  • Initialize sequence numbers on a shrunken index {pull}25321[#25321] (issue: {issue}10708[#10708])

  • Initialize primary term for shrunk indices {pull}25307[#25307] (issue: {issue}10708[#10708])

  • Introduce translog size and age based retention policies {pull}25147[#25147] (issue: {issue}10708[#10708])

Stats
  • Adds nodes usage API to monitor usages of actions {pull}24169[#24169]

Task Manager
  • Task Management {pull}15117[#15117]

Upgrade API
  • TemplateUpgraders should be called during rolling restart {pull}25263[#25263] (issues: {issue}24379[#24379], {issue}24680[#24680])

Enhancements

Aggregations
  • Add strict parsing of aggregation ranges {pull}25769[#25769]

  • Adds rewrite phase to aggregations {pull}25495[#25495] (issue: {issue}17676[#17676])

  • Tweak AggregatorBase.addRequestCircuitBreakerBytes {pull}25162[#25162] (issue: {issue}24511[#24511])

  • Add superset size to Significant Term REST response {pull}24865[#24865]

  • Add document count to Matrix Stats aggregation response {pull}24776[#24776]

  • Adds an implementation of LogLogBeta for the cardinality aggregation {pull}22323[#22323] (issue: {issue}22230[#22230])

Allocation
  • Adjust status on bad allocation explain requests {pull}25503[#25503] (issue: {issue}25458[#25458])

  • Promote replica on the highest version node {pull}25277[#25277] (issue: {issue}10708[#10708])

Analysis
  • [Analysis] Support normalizer in request param {pull}24767[#24767] (issue: {issue}23347[#23347])

  • Enforce validation for PathHierarchy tokenizer {pull}23510[#23510]

  • [analysis-icu] Allow setting unicodeSetFilter {pull}20814[#20814] (issue: {issue}20820[#20820])

CAT API
  • expand /_cat/nodes to return information about hard drive {pull}21775[#21775] (issue: {issue}21679[#21679])

Cluster
  • Validate a joining node’s version with version of existing cluster nodes {pull}25808[#25808]

  • Switch indices read-only if a node runs out of disk space {pull}25541[#25541] (issue: {issue}24299[#24299])

  • Add a cluster block that allows to delete indices that are read-only {pull}24678[#24678]

Core
  • Add max file size bootstrap check {pull}25974[#25974]

  • Add compatibility versions to main action response {pull}25799[#25799]

  • Index ids in binary form. {pull}25352[#25352] (issues: {issue}18154[#18154], {issue}24615[#24615])

  • Explicitly reject duplicate data paths {pull}25178[#25178]

  • Use SPI in High Level Rest Client to load XContent parsers {pull}25097[#25097]

  • Upgrade to lucene-7.0.0-snapshot-a0aef2f {pull}24775[#24775]

  • Speed up PK lookups at index time. {pull}19856[#19856]

Engine
  • Add refresh stats tracking for realtime get {pull}25052[#25052] (issue: {issue}24806[#24806])

  • Introducing a translog deletion policy {pull}24950[#24950]

Exceptions
  • IllegalStateException: Only duplicated jar instead of classpath {pull}24953[#24953]

Highlighting
  • Picks offset source for the unified highlighter directly from the es mapping {pull}25747[#25747] (issue: {issue}25699[#25699])

Index APIs
  • Let primary own its replication group {pull}25692[#25692] (issue: {issue}25485[#25485])

  • Create index request should return the index name {pull}25139[#25139] (issue: {issue}23044[#23044])

Ingest
  • Add Ingest-Processor specific Rest Endpoints & Add Grok endpoint {pull}25059[#25059] (issue: {issue}24725[#24725])

  • Port support for commercial GeoIP2 databases from Logstash. {pull}24889[#24889]

  • add exclude_keys option to KeyValueProcessor {pull}24876[#24876] (issue: {issue}23856[#23856])

  • Allow removing multiple fields in ingest processor {pull}24750[#24750] (issue: {issue}24622[#24622])

  • Add target_field parameter to ingest processors {pull}24133[#24133] (issues: {issue}23228[#23228], {issue}23682[#23682])

Inner Hits
  • Reuse inner hit query weight {pull}24571[#24571] (issue: {issue}23917[#23917])

Internal
  • Cleanup IndexFieldData visibility {pull}25900[#25900]

  • Bump the min compat version to 5.6.0 {pull}25805[#25805]

  • "shard started" should show index and shard ID {pull}25157[#25157]

  • Break out clear scroll logic from TransportClearScrollAction {pull}25125[#25125] (issue: {issue}25094[#25094])

  • Add helper methods to TransportActionProxy to identify proxy actions and requests {pull}25124[#25124]

  • Add remote cluster infrastructure to fetch discovery nodes. {pull}25123[#25123] (issue: {issue}25094[#25094])

  • Add the ability to set eager_global_ordinals in the new parent-join field {pull}25019[#25019]

  • Disallow multiple parent-join fields per mapping {pull}25002[#25002]

  • Remove the need for _UNRELEASED suffix in versions {pull}24798[#24798] (issue: {issue}24768[#24768])

  • Optimize the order of bytes in uuids for better compression. {pull}24615[#24615] (issue: {issue}18209[#18209])

Java API
  • Always Accumulate Transport Exceptions {pull}25017[#25017] (issue: {issue}23099[#23099])

Java High Level REST Client
  • [DOCS] restructure java clients docs pages {pull}25517[#25517]

  • Use SPI in High Level Rest Client to load XContent parsers {pull}25098[#25098] (issues: {issue}25024[#25024], {issue}25097[#25097])

  • Add support for clear scroll to high level REST client {pull}25038[#25038]

  • Add search scroll method to high level REST client {pull}24938[#24938] (issue: {issue}23331[#23331])

  • Add search method to high level REST client {pull}24796[#24796] (issues: {issue}24794[#24794], {issue}24795[#24795])

Java REST Client
  • Shade external dependencies in the rest client jar {pull}25780[#25780] (issue: {issue}25208[#25208])

  • RestClient uses system properties and system default SSLContext {pull}25757[#25757] (issue: {issue}23231[#23231])

Logging
  • Prevent excessive disk consumption by log files {pull}25660[#25660]

  • Use LRU set to reduce repeat deprecation messages {pull}25474[#25474] (issue: {issue}25457[#25457])

Mapping
  • Better validation of copy_to. {pull}25983[#25983]

  • Optimize terms queries on ip addresses to use a PointInSetQuery whenever possible. {pull}25669[#25669] (issue: {issue}25667[#25667])

Network
  • Move TransportStats accounting into TcpTransport {pull}25251[#25251]

  • Simplify connection closing and cleanups in TcpTransport {pull}25250[#25250]

  • Disable the Netty recycler in the client {pull}24793[#24793] (issues: {issue}22452[#22452], {issue}24721[#24721])

  • Remove Netty logging hack {pull}24653[#24653] (issues: {issue}24469[#24469], {issue}5624[#5624], {issue}6568[#6568], {issue}6696[#6696])

Packaging
  • Remove memlock suggestion from systemd service {pull}25979[#25979]

  • Set address space limit in systemd service file {pull}25975[#25975]

  • Version option should display if snapshot {pull}25970[#25970]

  • Ignore JVM options before checking Java version {pull}25969[#25969]

  • Also skip JAVA_TOOL_OPTIONS on Windows {pull}25968[#25968]

  • Introduce elasticsearch-env for Windows {pull}25958[#25958]

  • Introduce elasticsearch-env {pull}25815[#25815] (issue: {issue}20286[#20286])

  • Stop exporting HOSTNAME from scripts {pull}25807[#25807]

Parent/Child
  • Remove ParentJoinFieldSubFetchPhase {pull}25550[#25550] (issue: {issue}25363[#25363])

  • Support parent id being specified as number in the _source {pull}25547[#25547]

Plugin Lang Painless
  • Allow Custom Whitelists in Painless {pull}25557[#25557]

  • Update Painless to Allow Augmentation from Any Class {pull}25360[#25360]

  • Add Needs Methods to Painless Script Context Factories {pull}25267[#25267]

  • Support Script Context Stateful Factory in Painless {pull}25233[#25233]

  • Generate Painless Factory for Creating Script Instances {pull}25120[#25120]

  • Update Painless to Use New Script Contexts {pull}25015[#25015]

  • Optimize instance creation in LambdaBootstrap {pull}24618[#24618]

Plugin Repository GCS
  • GCS Repository: Add secure storage of credentials {pull}24697[#24697]

Plugin Repository S3
  • S3 Repository: Add back repository level credentials {pull}24609[#24609]

Plugins
  • Move tribe to a module {pull}25778[#25778]

  • Plugins can register pre-configured char filters {pull}25000[#25000] (issue: {issue}23658[#23658])

  • Add purge option to remove plugin CLI {pull}24981[#24981]

  • Allow plugins to register pre-configured tokenizers {pull}24751[#24751] (issues: {issue}24223[#24223], {issue}24572[#24572])

  • Move ReindexAction class to core {pull}24684[#24684] (issue: {issue}24578[#24578])

  • Make PreConfiguredTokenFilter harder to misuse {pull}24572[#24572] (issue: {issue}23658[#23658])

Query DSL
  • Make slop optional when parsing span_near query {pull}25677[#25677] (issue: {issue}25642[#25642])

  • Require a field when a seed is provided to the random_score function. {pull}25594[#25594] (issue: {issue}25240[#25240])

REST
  • Refactor PathTrie and RestController to use a single trie for all methods {pull}25459[#25459] (issue: {issue}24437[#24437])

  • Make ObjectParser support string to boolean conversion {pull}24668[#24668] (issue: {issue}21802[#21802])

Recovery
  • Goodbye, Translog Views {pull}25962[#25962]

  • Disallow multiple concurrent recovery attempts for same target shard {pull}25428[#25428]

  • Live primary-replica resync (no rollback) {pull}24841[#24841] (issue: {issue}10708[#10708])

Scripting
  • Scripting: Rename SearchScript.needsScores to needs_score {pull}25235[#25235]

  • Scripting: Add optional context parameter to put stored script requests {pull}25014[#25014]

  • Add New Security Script Settings {pull}24637[#24637] (issue: {issue}24532[#24532])

Search
  • Rewrite search requests on the coordinating nodes {pull}25814[#25814] (issue: {issue}25791[#25791])

  • Ensure query resources are fetched asynchronously during rewrite {pull}25791[#25791]

  • Introduce a new Rewriteable interface to streamline rewriting {pull}25788[#25788]

  • Reduce the scope of QueryRewriteContext {pull}25787[#25787]

  • Reduce the overhead of timeouts and low-level search cancellation. {pull}25776[#25776]

  • Reduce profiling overhead. {pull}25772[#25772] (issue: {issue}24799[#24799])

  • Prevent can_match requests from sending to incompatible nodes {pull}25705[#25705] (issue: {issue}25704[#25704])

  • Add a shard filter search phase to pre-filter shards based on query rewriting {pull}25658[#25658]

  • Ensure we rewrite common queries to match_none if possible {pull}25650[#25650]

  • Limit the number of concurrent shard requests per search request {pull}25632[#25632]

  • Add cluster name validation to RemoteClusterConnection {pull}25568[#25568]

  • Speed up sorted scroll when the index sort matches the search sort {pull}25138[#25138] (issue: {issue}6720[#6720])

  • Leverage scorerSupplier when applicable. {pull}25109[#25109]

  • Add Cross Cluster Search support for scroll searches {pull}25094[#25094]

  • Track EWMA[1] of task execution time in search threadpool executor {pull}24989[#24989] (issue: {issue}24915[#24915])

  • Query range fields by doc values when they are expected to be more efficient than points {pull}24823[#24823] (issue: {issue}24314[#24314])

  • Search: Fairer balancing when routing searches by session ID {pull}24671[#24671] (issue: {issue}24642[#24642])

Sequence IDs
  • Move primary term from ReplicationRequest to ConcreteShardRequest {pull}25822[#25822]

  • Add reason to global checkpoint updates on replica {pull}25612[#25612] (issue: {issue}10708[#10708])

  • Introduce primary/replica mode for GlobalCheckPointTracker {pull}25468[#25468]

  • Throw back replica local checkpoint on new primary {pull}25452[#25452] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Update global checkpoint when increasing primary term on replica {pull}25422[#25422] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Enable a long translog retention policy by default {pull}25294[#25294] (issues: {issue}10708[#10708], {issue}25147[#25147])

  • Introduce primary context {pull}25122[#25122] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Block older operations on primary term transition {pull}24779[#24779] (issue: {issue}10708[#10708])

Settings
  • Add disk threshold settings validation {pull}25600[#25600] (issue: {issue}25560[#25560])

  • Enable cross-setting validation {pull}25560[#25560] (issue: {issue}25541[#25541])

  • Validate transport.profiles.* settings {pull}25508[#25508]

  • Cleanup network / transport related settings {pull}25489[#25489]

  • Emit settings deprecation logging at most once {pull}25457[#25457]

  • IndexMetaData: Introduce internal format index setting {pull}25292[#25292]

Snapshot/Restore
  • Improves snapshot logging and snapshot deletion error handling {pull}25264[#25264]

Stats
  • Update IndexShard#refreshMetric via a ReferenceManager.RefreshListener {pull}25083[#25083] (issues: {issue}24806[#24806], {issue}25052[#25052])

Translog
  • Translog file recovery should not rely on lucene commits {pull}25005[#25005] (issue: {issue}24950[#24950])

Bug Fixes

Aggregations
  • Fixes array out of bounds for value count agg {pull}26038[#26038] (issue: {issue}17379[#17379])

  • Aggregations bug: Significant_text fails on arrays of text. {pull}25030[#25030] (issue: {issue}25029[#25029])

Aliases
  • mget with an alias shouldn’t ignore alias routing {pull}25697[#25697] (issue: {issue}25696[#25696])

  • GET aliases should 404 if aliases are missing {pull}25043[#25043] (issue: {issue}24644[#24644])

Analysis
  • Pre-configured shingle filter should disable graph analysis {pull}25853[#25853] (issue: {issue}25555[#25555])

Circuit Breakers
  • Checks the circuit breaker before allocating bytes for a new big array {pull}25010[#25010] (issue: {issue}24790[#24790])

Core
  • Release operation permit on thread-pool rejection {pull}25930[#25930] (issue: {issue}25863[#25863])

  • Node should start up despite of a lingering .es_temp_file {pull}21210[#21210] (issue: {issue}21007[#21007])

Discovery
  • MasterNodeChangePredicate should use the node instance to detect master change {pull}25877[#25877] (issue: {issue}25471[#25471])

Engine
  • Engine - do not index operations with seq# lower than the local checkpoint into lucene {pull}25827[#25827] (issues: {issue}1[#1], {issue}2[#2], {issue}25592[#25592])

Geo
  • Fix typo in GeoUtils#isValidLongitude {pull}25121[#25121]

Highlighting
  • FastVectorHighlighter should not cache the field query globally {pull}25197[#25197] (issue: {issue}25171[#25171])

  • Higlighters: Fix MultiPhrasePrefixQuery rewriting {pull}25103[#25103] (issue: {issue}25088[#25088])

Index APIs
  • Shrink API should ignore templates {pull}25380[#25380] (issue: {issue}25035[#25035])

  • Rollover max docs should only count primaries {pull}24977[#24977] (issue: {issue}24217[#24217])

Ingest
  • Sort Processor does not have proper behavior with targetField {pull}25237[#25237] (issue: {issue}24133[#24133])

  • fix grok’s pattern parsing to validate pattern names in expression {pull}25063[#25063] (issue: {issue}22831[#22831])

Inner Hits
  • When fetching nested inner hits only access stored fields when needed {pull}25864[#25864] (issue: {issue}6[#6])

Internal
  • Fix BytesReferenceStreamInput#skip with offset {pull}25634[#25634]

  • Fix race condition in RemoteClusterConnection node supplier {pull}25432[#25432]

  • Initialise empty lists in BaseTaskResponse constructor {pull}25290[#25290]

  • Extract a common base class for scroll executions {pull}24979[#24979] (issue: {issue}16555[#16555])

  • Obey lock order if working with store to get metadata snapshots {pull}24787[#24787] (issue: {issue}24481[#24481])

  • Fix Version based BWC and set correct minCompatVersion {pull}24732[#24732]

  • Fix _field_caps serialization in order to support cross cluster search {pull}24722[#24722]

  • Avoid race when shutting down controller processes {pull}24579[#24579]

Mapping
  • Fix parsing of ip range queries. {pull}25768[#25768] (issue: {issue}25636[#25636])

  • Disable date field mapping changing {pull}25285[#25285] (issue: {issue}25271[#25271])

  • Correctly enable _all for older 5.x indices {pull}25087[#25087] (issue: {issue}25068[#25068])

  • token_count datatype should handle null value {pull}25046[#25046] (issue: {issue}24928[#24928])

  • keep _parent field while updating child type mapping {pull}24407[#24407] (issue: {issue}23381[#23381])

More Like This
  • Pass over _routing value with more_like_this items to be retrieved {pull}24679[#24679] (issue: {issue}23699[#23699])

Nested Docs
  • In case of a single type the _id field should be added to the nested document instead of _uid field {pull}25149[#25149]

Network
  • Ensure pending transport handlers are invoked for all channel failures {pull}25150[#25150]

  • Notify onConnectionClosed rather than onNodeDisconnect to prune transport handlers {pull}24639[#24639] (issues: {issue}24557[#24557], {issue}24575[#24575], {issue}24632[#24632])

Packaging
  • Exit Windows scripts promptly on failure {pull}25959[#25959]

  • Pass config path as a system property {pull}25943[#25943]

  • ES_HOME needs to be made absolute before attempt at traversal {pull}25865[#25865]

  • Fix elasticsearch-keystore handling of path.conf {pull}25811[#25811]

  • Stop disabling explicit GC {pull}25759[#25759]

  • Avoid failing install if system-sysctl is masked {pull}25657[#25657] (issue: {issue}24234[#24234])

  • Get short path name for native controllers {pull}25344[#25344]

  • When stopping via systemd only kill the JVM, not its control group {pull}25195[#25195]

  • remove remaining references to scripts directory {pull}24771[#24771]

  • Handle parentheses in batch file path {pull}24731[#24731] (issue: {issue}24712[#24712])

Parent/Child
  • The default _parent field should not try to load global ordinals {pull}25851[#25851] (issue: {issue}25849[#25849])

Percolator
  • Fix range queries with date range based on current time in percolator queries. {pull}24666[#24666] (issue: {issue}23921[#23921])

Plugin Lang Painless
  • Painless: allow doubles to be casted to longs. {pull}25936[#25936]

Plugin Repository Azure
  • Make calls to CloudBlobContainer#exists privileged {pull}25937[#25937] (issue: {issue}25931[#25931])

Plugin Repository GCS
  • Ensure that gcs client creation is privileged {pull}25938[#25938] (issue: {issue}25932[#25932])

Plugin Repository HDFS
  • Upgrading HDFS Repository Plugin to use HDFS 2.8.1 Client {pull}25497[#25497] (issue: {issue}25450[#25450])

Plugin Repository S3
  • Avoid SecurityException in repository-S3 on DefaultS3OutputStream.flush() {pull}25254[#25254] (issue: {issue}25192[#25192])

Plugins
  • X-Pack plugin download fails on Windows desktop {pull}24570[#24570]

Query DSL
  • SpanNearQueryBuilder should return the inner clause when a single clause is provided {pull}25856[#25856] (issue: {issue}25630[#25630])

  • Refactor field expansion for match, multi_match and query_string query {pull}25726[#25726] (issues: {issue}25551[#25551], {issue}25556[#25556])

  • WrapperQueryBuilder should also rewrite the parsed query {pull}25480[#25480]

REST
  • Fix handling of invalid error trace parameter {pull}25785[#25785] (issue: {issue}25774[#25774])

  • Fix handling of exceptions thrown on HEAD requests {pull}25172[#25172] (issue: {issue}21125[#21125])

  • Fixed NPEs caused by requests without content. {pull}23497[#23497] (issue: {issue}24701[#24701])

  • Fix get mappings HEAD requests {pull}23192[#23192] (issue: {issue}21125[#21125])

Recovery
  • Close translog view after primary-replica resync {pull}25862[#25862] (issue: {issue}24841[#24841])

Reindex API
  • Reindex: don’t duplicate _source parameter {pull}24629[#24629] (issue: {issue}24628[#24628])

  • Add qa module that tests reindex-from-remote against pre-5.0 versions of Elasticsearch {pull}24561[#24561] (issues: {issue}23828[#23828], {issue}24520[#24520])

Search
  • Caching a MinDocQuery can lead to wrong results. {pull}25909[#25909]

  • Fix random score generation when no seed is provided. {pull}25908[#25908]

  • Merge FunctionScoreQuery and FiltersFunctionScoreQuery {pull}25889[#25889] (issues: {issue}15709[#15709], {issue}23628[#23628])

  • Respect cluster alias in _index aggs and queries {pull}25885[#25885] (issue: {issue}25606[#25606])

  • First increment shard stats before notifying and potentially sending response {pull}25818[#25818]

  • Remove assertion about deviation when casting to a float. {pull}25806[#25806] (issue: {issue}25330[#25330])

  • Prevent skipping shards if a suggest builder is present {pull}25739[#25739] (issue: {issue}25658[#25658])

  • Ensure remote cluster alias is preserved in inner hits aggs {pull}25627[#25627] (issue: {issue}25606[#25606])

  • Do not search locally if remote index pattern resolves to no indices {pull}25436[#25436] (issue: {issue}25426[#25426])

  • Adds check for negative search request size {pull}25397[#25397] (issue: {issue}22530[#22530])

  • Make sure range queries are correctly profiled. {pull}25108[#25108]

  • Fix RangeFieldMapper rangeQuery to properly handle relations {pull}24808[#24808] (issue: {issue}24744[#24744])

  • Fix ExpandSearchPhase when response contains no hits {pull}24688[#24688] (issue: {issue}24672[#24672])

Sequence IDs
  • Fix pre-6.0 response to unknown replication actions {pull}25744[#25744] (issue: {issue}10708[#10708])

  • Track local checkpoint on primary immediately {pull}25434[#25434] (issues: {issue}10708[#10708], {issue}25355[#25355], {issue}25415[#25415])

  • Initialize max unsafe auto ID timestamp on shrink {pull}25356[#25356] (issues: {issue}10708[#10708], {issue}25355[#25355])

  • Use correct primary term for replicating NOOPs {pull}25128[#25128]

  • Handle already closed while filling gaps {pull}25021[#25021] (issue: {issue}24925[#24925])

Settings
  • Fix settings serialization to not serialize secure settings or not take the total size into account {pull}25323[#25323]

  • Keystore CLI should use the AddFileKeyStoreCommand for files {pull}25298[#25298]

  • Allow resetting settings that use an IP validator {pull}24713[#24713] (issue: {issue}24709[#24709])

Snapshot/Restore
  • Snapshot/Restore: Ensure that shard failure reasons are correctly stored in CS {pull}25941[#25941] (issue: {issue}25878[#25878])

  • Output all empty snapshot info fields if in verbose mode {pull}25455[#25455] (issue: {issue}24477[#24477])

  • Remove redundant and broken MD5 checksum from repository-s3 {pull}25270[#25270] (issue: {issue}25269[#25269])

  • Consolidates the logic for cleaning up snapshots on master election {pull}24894[#24894] (issue: {issue}24605[#24605])

  • Removes completed snapshot from cluster state on master change {pull}24605[#24605] (issue: {issue}24452[#24452])

Stats
  • _nodes/stats should not fail due to concurrent AlreadyClosedException {pull}25016[#25016] (issue: {issue}23099[#23099])

Suggesters
  • Context suggester should filter doc values field {pull}25858[#25858] (issue: {issue}25404[#25404])

Regressions

Highlighting
  • Fix Fast Vector Highlighter NPE on match phrase prefix {pull}25116[#25116] (issue: {issue}25088[#25088])

Search
  • Always use DisjunctionMaxQuery to build cross fields disjunction {pull}25115[#25115] (issue: {issue}23966[#23966])

Upgrades

Network
  • Upgrade to Netty 4.1.13.Final {pull}25581[#25581] (issues: {issue}24729[#24729], {issue}6866[#6866])

  • Upgrade to Netty 4.1.11.Final {pull}24652[#24652]

Upgrade API
  • Improve stability and logging of TemplateUpgradeServiceIT tests {pull}25386[#25386] (issue: {issue}25382[#25382])

{es} version 6.0.0-alpha2

Breaking Changes

CRUD
  • Deleting a document from a non-existing index creates the indexIf the index does not exist, delete document will not auto create it {pull}24518[#24518] (issue: {issue}15425[#15425])

Plugin Analysis ICU
  • Upgrade icu4j to latest version {pull}24821[#24821]

Plugin Repository S3
  • Remove deprecated S3 settings {pull}24445[#24445]

Scripting
  • Remove script access to term statistics {pull}19462[#19462] (issue: {issue}19359[#19359])

Watcher
  • The watch _status field has been renamed to status, as underscores in field names will not be allowed.

Breaking Java Changes

Aggregations
  • Make Terms.Bucket an interface rather than an abstract class {pull}24492[#24492]

  • Compound order for histogram aggregations {pull}22343[#22343] (issues: {issue}14771[#14771], {issue}20003[#20003], {issue}23613[#23613])

Plugins
  • Drop name from TokenizerFactory {pull}24869[#24869]

Deprecations

Settings
  • Deprecate settings in .yml and .json {pull}24059[#24059] (issue: {issue}19391[#19391])

New Features

Aggregations
  • SignificantText aggregation - like significant_terms, but for text {pull}24432[#24432] (issue: {issue}23674[#23674])

Internal
  • Automatically adjust search threadpool queue_size {pull}23884[#23884] (issue: {issue}3890[#3890])

Mapping
  • Add new ip_range field type {pull}24433[#24433]

Plugin Analysis ICU
  • Add ICUCollationFieldMapper {pull}24126[#24126]

Enhancements

Core
  • Improve bootstrap checks error messages {pull}24548[#24548]

Engine
  • Move the IndexDeletionPolicy to be engine internal {pull}24930[#24930] (issue: {issue}10708[#10708])

Internal
  • Add assertions enabled helper {pull}24834[#24834]

Java High Level REST Client
  • Add doc_count to ParsedMatrixStats {pull}24952[#24952] (issue: {issue}24776[#24776])

  • Add fromXContent method to ClearScrollResponse {pull}24909[#24909]

  • ClearScrollRequest to implement ToXContentObject {pull}24907[#24907]

  • SearchScrollRequest to implement ToXContentObject {pull}24906[#24906] (issue: {issue}3889[#3889])

  • Add aggs parsers for high level REST Client {pull}24824[#24824] (issues: {issue}23965[#23965], {issue}23973[#23973], {issue}23974[#23974], {issue}24085[#24085], {issue}24160[#24160], {issue}24162[#24162], {issue}24182[#24182], {issue}24183[#24183], {issue}24208[#24208], {issue}24213[#24213], {issue}24239[#24239], {issue}24284[#24284], {issue}24312[#24312], {issue}24330[#24330], {issue}24365[#24365], {issue}24371[#24371], {issue}24442[#24442], {issue}24521[#24521], {issue}24524[#24524], {issue}24564[#24564], {issue}24583[#24583], {issue}24589[#24589], {issue}24648[#24648], {issue}24667[#24667], {issue}24675[#24675], {issue}24682[#24682], {issue}24700[#24700], {issue}24706[#24706], {issue}24717[#24717], {issue}24720[#24720], {issue}24738[#24738], {issue}24746[#24746], {issue}24789[#24789], {issue}24791[#24791], {issue}24794[#24794], {issue}24796[#24796], {issue}24822[#24822])

Mapping
  • Identify documents by their _id. {pull}24460[#24460]

Packaging
  • Set number of processes in systemd unit file {pull}24970[#24970] (issue: {issue}20874[#20874])

Plugin Lang Painless
  • Make Painless Compiler Use an Instance Per Context {pull}24972[#24972]

  • Make PainlessScript An Interface {pull}24966[#24966]

Recovery
  • Introduce primary context {pull}25031[#25031] (issue: {issue}10708[#10708])

Scripting
  • Add StatefulFactoryType as optional intermediate factory in script contexts {pull}24974[#24974] (issue: {issue}20426[#20426])

  • Make contexts available to ScriptEngine construction {pull}24896[#24896]

  • Make ScriptEngine.compile generic on the script context {pull}24873[#24873]

  • Add instance and compiled classes to script contexts {pull}24868[#24868]

Search
  • Eliminate array access in tight loops when profiling is enabled. {pull}24959[#24959]

  • Support Multiple Inner Hits on a Field Collapse Request {pull}24517[#24517]

  • Expand cross cluster search indices for search requests to the concrete index or to it’s aliases {pull}24502[#24502]

Search Templates
  • Add max concurrent searches to multi template search {pull}24255[#24255] (issues: {issue}20912[#20912], {issue}21907[#21907])

Security
  • Adapted indices resolution to use new ignoreAliases index option.

  • Added the logstash_admin role, which provides access to .logstash-* indices for managing configurations.

Sequence IDs
  • Fill gaps on primary promotion {pull}24945[#24945] (issue: {issue}10708[#10708])

  • Introduce clean transition on primary promotion {pull}24925[#24925] (issue: {issue}10708[#10708])

  • Guarantee that translog generations are seqNo conflict free {pull}24825[#24825] (issues: {issue}10708[#10708], {issue}24779[#24779])

  • Inline global checkpoints {pull}24513[#24513] (issue: {issue}10708[#10708])

Snapshot/Restore
  • Enhances get snapshots API to allow retrieving repository index only {pull}24477[#24477] (issue: {issue}24288[#24288])

Watcher
  • Watcher indices no longer use multiple types.

Bug Fixes

Aggregations
  • Terms aggregation should remap global ordinal buckets when a sub-aggregator is used to sort the terms {pull}24941[#24941] (issue: {issue}24788[#24788])

  • Correctly set doc_count when MovAvg "predicts" values on existing buckets {pull}24892[#24892] (issue: {issue}24327[#24327])

  • DateHistogram: Fix extended_bounds with offset {pull}23789[#23789] (issue: {issue}23776[#23776])

  • Fix ArrayIndexOutOfBoundsException when no ranges are specified in the query {pull}23241[#23241] (issue: {issue}22881[#22881])

Analysis
  • PatternAnalyzer should lowercase wildcard queries when lowercase is true. {pull}24967[#24967]

Cache
  • fix bug of weight computation {pull}24856[#24856]

Core
  • Fix cache expire after access {pull}24546[#24546]

Graph
  • Reinstated _xpack/graph/_explore as the correct graph endpoint. _xpack/_graph/_explore is deprecated and will be removed in v7.0.

Index APIs
  • Validates updated settings on closed indices {pull}24487[#24487] (issue: {issue}23787[#23787])

Ingest
  • Fix floating-point error when DateProcessor parses UNIX {pull}24947[#24947]

  • add option for _ingest.timestamp to use new ZonedDateTime (5.x backport) {pull}24030[#24030] (issues: {issue}23168[#23168], {issue}23174[#23174])

Inner Hits
  • Fix Source filtering in new field collapsing feature {pull}24068[#24068] (issue: {issue}24063[#24063])

Internal
  • Ensure remote cluster is connected before fetching _field_caps {pull}24845[#24845] (issue: {issue}24763[#24763])

Machine Learning
  • If the initial cluster state update to install the Machine Learning Metadata fails, the update is now retried.

Network
  • Fix error message if an incompatible node connects {pull}24884[#24884]

Plugins
  • Fix plugin installation permissions {pull}24527[#24527] (issue: {issue}24480[#24480])

Scroll
  • Fix single shard scroll within a cluster with nodes in version >= 5.3 and ⇐ 5.3 {pull}24512[#24512]

Search
  • Fix script field sort returning Double.MAX_VALUE for all documents {pull}24942[#24942] (issue: {issue}24940[#24940])

  • Compute the took time of the query after the expand phase of field collapsing {pull}24902[#24902] (issue: {issue}24900[#24900])

Sequence IDs
  • Handle primary failure handling replica response {pull}24926[#24926] (issue: {issue}24935[#24935])

Snapshot/Restore
  • Fix inefficient (worst case exponential) loading of snapshot repository {pull}24510[#24510] (issue: {issue}24509[#24509])

Stats
  • Avoid double decrement on current query counter {pull}24922[#24922] (issues: {issue}22996[#22996], {issue}24872[#24872])

  • Adjust available and free bytes to be non-negative on huge FSes {pull}24911[#24911] (issues: {issue}23093[#23093], {issue}24453[#24453])

Suggesters
  • Fix context suggester to read values from keyword type field {pull}24200[#24200] (issue: {issue}24129[#24129])

{es} version 6.0.0-alpha1

Breaking Changes

Allocation
  • Remove cluster.routing.allocation.snapshot.relocation_enabled setting {pull}20994[#20994]

Analysis
  • Removing query-string parameters in _analyze API {pull}20704[#20704] (issue: {issue}20246[#20246])

CAT API
  • Write -1 on unbounded queue in cat thread pool {pull}21342[#21342] (issue: {issue}21187[#21187])

CRUD
  • Disallow VersionType.FORCE for GetRequest {pull}21079[#21079] (issue: {issue}20995[#20995])

  • Disallow VersionType.FORCE versioning for 6.x indices {pull}20995[#20995] (issue: {issue}20377[#20377])

Cluster
  • No longer allow cluster name in data path {pull}20433[#20433] (issue: {issue}20391[#20391])

Core
  • Simplify file store {pull}24402[#24402] (issue: {issue}24390[#24390])

  • Make boolean conversion strict {pull}22200[#22200]

  • Remove the default store type. {pull}21616[#21616]

  • Remove store throttling. {pull}21573[#21573]

Geo
  • Remove deprecated geo search features {pull}22876[#22876]

  • Reduce GeoDistance Insanity {pull}19846[#19846]

Index APIs
  • Open/Close index api to allow_no_indices by default {pull}24401[#24401] (issues: {issue}24031[#24031], {issue}24341[#24341])

  • Remove support for controversial ignore_unavailable and allow_no_indices from indices exists api {pull}20712[#20712]

Index Templates
  • Allows multiple patterns to be specified for index templates {pull}21009[#21009] (issue: {issue}20690[#20690])

Java API
  • Enforce Content-Type requirement on the rest layer and remove deprecated methods {pull}23146[#23146] (issue: {issue}19388[#19388])

Mapping
  • Enforce at most one type. {pull}24428[#24428] (issue: {issue}24317[#24317])

  • Disallow include_in_all for 6.0+ indices {pull}22970[#22970] (issue: {issue}22923[#22923])

  • Disable _all by default, disallow configuring _all on 6.0+ indices {pull}22144[#22144] (issues: {issue}19784[#19784], {issue}20925[#20925], {issue}21341[#21341])

  • Throw an exception on unrecognized "match_mapping_type" {pull}22090[#22090] (issue: {issue}17285[#17285])

Network
  • Remove blocking TCP clients and servers {pull}22639[#22639]

  • Remove modules/transport_netty_3 in favor of netty_4 {pull}21590[#21590]

  • Remove LocalTransport in favor of MockTcpTransport {pull}20695[#20695]

Packaging
  • Remove customization of ES_USER and ES_GROUP {pull}23989[#23989] (issue: {issue}23848[#23848])

Percolator
  • Remove deprecated percolate and mpercolate apis {pull}22331[#22331]

Plugin Delete By Query
  • Require explicit query in _delete_by_query API {pull}23632[#23632] (issue: {issue}23629[#23629])

Plugin Discovery EC2
  • Ec2 Discovery: Cleanup deprecated settings {pull}24150[#24150]

  • Discovery EC2: Remove region setting {pull}23991[#23991] (issue: {issue}22758[#22758])

  • AWS Plugins: Remove signer type setting {pull}23984[#23984] (issue: {issue}22599[#22599])

Plugin Lang JS
  • Remove lang-python and lang-javascript {pull}20734[#20734] (issue: {issue}20698[#20698])

Plugin Mapper Attachment
  • Remove mapper attachments plugin {pull}20416[#20416] (issue: {issue}18837[#18837])

Plugin Repository Azure
  • Remove global repositories.azure settings {pull}23262[#23262] (issues: {issue}22800[#22800], {issue}22856[#22856])

  • Remove auto creation of container for azure repository {pull}22858[#22858] (issue: {issue}22857[#22857])

Plugin Repository S3
  • S3 Repository: Cleanup deprecated settings {pull}24097[#24097]

  • S3 Repository: Remove region setting {pull}22853[#22853] (issue: {issue}22758[#22758])

  • S3 Repository: Remove bucket auto create {pull}22846[#22846] (issue: {issue}22761[#22761])

  • S3 Repository: Remove env var and sysprop credentials support {pull}22842[#22842]

Query DSL
  • Remove deprecated minimum_number_should_match in BoolQueryBuilder {pull}22416[#22416]

  • Remove support for empty queries {pull}22092[#22092] (issue: {issue}17624[#17624])

  • Remove deprecated query names: in, geo_bbox, mlt, fuzzy_match and match_fuzzy {pull}21852[#21852]

  • The terms query should always map to a Lucene TermsQuery. {pull}21786[#21786]

  • Be strict when parsing values searching for booleans {pull}21555[#21555] (issue: {issue}21545[#21545])

  • Remove collect payloads parameter {pull}20385[#20385]

REST
  • Remove ldjson support and document ndjson for bulk/msearch {pull}23049[#23049] (issue: {issue}23025[#23025])

  • Enable strict duplicate checks for all XContent types {pull}22225[#22225] (issues: {issue}19614[#19614], {issue}22073[#22073])

  • Enable strict duplicate checks for JSON content {pull}22073[#22073] (issue: {issue}19614[#19614])

  • Remove lenient stats parsing {pull}21417[#21417] (issues: {issue}20722[#20722], {issue}21410[#21410])

  • Remove allow unquoted JSON {pull}20388[#20388] (issues: {issue}17674[#17674], {issue}17801[#17801])

  • Remove FORCE version_type {pull}20377[#20377] (issue: {issue}19769[#19769])

Scripting
  • Make dates be ReadableDateTimes in scripts {pull}22948[#22948] (issue: {issue}22875[#22875])

  • Remove groovy scripting language {pull}21607[#21607]

Search
  • ProfileResult and CollectorResult should print machine readable timing information {pull}22561[#22561]

  • Remove indices query {pull}21837[#21837] (issue: {issue}17710[#17710])

  • Remove ignored type parameter in search_shards api {pull}21688[#21688]

Security
  • A new bootstrap check enforces that default passwords are disabled for the built-in users when running in {ref}/bootstrap-checks.html#dev-vs-prod-mode[production mode]. You must set xpack.security.authc.accept_default_password to false in your elasticsearch.yml. For more information, see [security-settings] and [setting-up-authentication].

  • A new configuration setting is available to disable support for the default password ("changeme"). For more information, see {ref}/built-in-users.html#disabling-default-password[Disable default password functionality].

Sequence IDs
  • Change certain replica failures not to fail the replica shard {pull}22874[#22874] (issue: {issue}10708[#10708])

Shadow Replicas
  • Remove shadow replicas {pull}23906[#23906] (issue: {issue}22024[#22024])

Watcher
  • The built-in HTTP client used in webhooks, the http input and the http email attachment has been replaced. This results in the need to always escape all parts of an URL.

  • The new built-in HTTP client also enforces a maximum request size, which defaults to 10mb.

Breaking Java Changes

Java API
  • Java api: ActionRequestBuilder#execute to return a PlainActionFuture {pull}24415[#24415] (issues: {issue}24412[#24412], {issue}9201[#9201])

Network
  • Simplify TransportAddress {pull}20798[#20798]

Deprecations

Index Templates
  • Restore deprecation warning for invalid match_mapping_type values {pull}22304[#22304]

Internal
  • Deprecate XContentType auto detection methods in XContentFactory {pull}22181[#22181] (issue: {issue}19388[#19388])

New Features

Core
  • Enable index-time sorting {pull}24055[#24055] (issue: {issue}6720[#6720])

Enhancements

Aggregations
  • Agg builder accessibility fixes {pull}24323[#24323]

  • Remove support for the include/pattern syntax. {pull}23141[#23141] (issue: {issue}22933[#22933])

  • Promote longs to doubles when a terms agg mixes decimal and non-decimal numbers {pull}22449[#22449] (issue: {issue}22232[#22232])

Analysis
  • Match- and MultiMatchQueryBuilder should only allow setting analyzer on string values {pull}23684[#23684] (issue: {issue}21665[#21665])

Bulk
  • Simplify bulk request execution {pull}20109[#20109]

CRUD
  • Added validation for upsert request {pull}24282[#24282] (issue: {issue}16671[#16671])

Cluster
  • Separate publishing from applying cluster states {pull}24236[#24236]

  • Adds cluster state size to /_cluster/state response {pull}23440[#23440] (issue: {issue}3415[#3415])

Core
  • Remove connect SocketPermissions from core {pull}22797[#22797]

  • Add repository-url module and move URLRepository {pull}22752[#22752] (issue: {issue}22116[#22116])

  • Remove accept SocketPermissions from core {pull}22622[#22622] (issue: {issue}22116[#22116])

  • Move IfConfig.logIfNecessary call into bootstrap {pull}22455[#22455] (issue: {issue}22116[#22116])

  • Remove artificial default processors limit {pull}20874[#20874] (issue: {issue}20828[#20828])

  • Simplify write failure handling {pull}19105[#19105] (issue: {issue}20109[#20109])

Engine
  • Fill missing sequence IDs up to max sequence ID when recovering from store {pull}24238[#24238] (issue: {issue}10708[#10708])

  • Use sequence numbers to identify out of order delivery in replicas & recovery {pull}24060[#24060] (issue: {issue}10708[#10708])

  • Add replica ops with version conflict to translog {pull}22626[#22626]

  • Clarify global checkpoint recovery {pull}21934[#21934] (issue: {issue}21254[#21254])

Internal
  • Try to convince the JVM not to lose stacktraces {pull}24426[#24426] (issue: {issue}24376[#24376])

  • Make document write requests immutable {pull}23038[#23038]

Java High Level REST Client
  • Add info method to High Level Rest client {pull}23350[#23350]

  • Add support for named xcontent parsers to high level REST client {pull}23328[#23328]

  • Add BulkRequest support to High Level Rest client {pull}23312[#23312]

  • Add UpdateRequest support to High Level Rest client {pull}23266[#23266]

  • Add delete API to the High Level Rest Client {pull}23187[#23187]

  • Add Index API to High Level Rest Client {pull}23040[#23040]

  • Add get/exists method to RestHighLevelClient {pull}22706[#22706]

  • Add fromxcontent methods to delete response {pull}22680[#22680] (issue: {issue}22229[#22229])

  • Add REST high level client gradle submodule and first simple method {pull}22371[#22371]

Java REST Client
  • Wrap rest httpclient with doPrivileged blocks {pull}22603[#22603] (issue: {issue}22116[#22116])

Mapping
  • Date detection should not rely on a hardcoded set of characters. {pull}22171[#22171] (issue: {issue}1694[#1694])

Network
  • Isolate SocketPermissions to Netty {pull}23057[#23057]

  • Wrap netty accept/connect ops with doPrivileged {pull}22572[#22572] (issue: {issue}22116[#22116])

  • Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions {pull}22287[#22287] (issue: {issue}22116[#22116])

Plugin Discovery EC2
  • Read ec2 discovery address from aws instance tags {pull}22743[#22743] (issue: {issue}22566[#22566])

Plugin Repository HDFS
  • Add doPrivilege blocks for socket connect ops in repository-hdfs {pull}22793[#22793] (issue: {issue}22116[#22116])

Plugins
  • Add doPrivilege blocks for socket connect operations in plugins {pull}22534[#22534] (issue: {issue}22116[#22116])

Recovery
  • Peer Recovery: remove maxUnsafeAutoIdTimestamp hand off {pull}24243[#24243] (issue: {issue}24149[#24149])

  • Introduce sequence-number-based recovery {pull}22484[#22484] (issue: {issue}10708[#10708])

Search
  • Add parsing from xContent to Suggest {pull}22903[#22903]

  • Add parsing from xContent to ShardSearchFailure {pull}22699[#22699]

Sequence IDs
  • Block global checkpoint advances when recovering {pull}24404[#24404] (issue: {issue}10708[#10708])

  • Add primary term to doc write response {pull}24171[#24171] (issue: {issue}10708[#10708])

  • Preserve multiple translog generations {pull}24015[#24015] (issue: {issue}10708[#10708])

  • Introduce translog generation rolling {pull}23606[#23606] (issue: {issue}10708[#10708])

  • Replicate write failures {pull}23314[#23314]

  • Introduce sequence-number-aware translog {pull}22822[#22822] (issue: {issue}10708[#10708])

  • Introduce translog no-op {pull}22291[#22291] (issue: {issue}10708[#10708])

  • Tighten sequence numbers recovery {pull}22212[#22212] (issue: {issue}10708[#10708])

  • Add BWC layer to seq no infra and enable BWC tests {pull}22185[#22185] (issue: {issue}21670[#21670])

  • Add internal _primary_term doc values field, fix _seq_no indexing {pull}21637[#21637] (issues: {issue}10708[#10708], {issue}21480[#21480])

  • Add global checkpoint to translog checkpoints {pull}21254[#21254]

  • Sequence numbers commit data for Lucene uses Iterable interface {pull}20793[#20793] (issue: {issue}10708[#10708])

  • Simplify GlobalCheckpointService and properly hook it for cluster state updates {pull}20720[#20720]

Stats
  • Expose disk usage estimates in nodes stats {pull}22081[#22081] (issue: {issue}8686[#8686])

Store
  • Remote support for lucene versions without checksums {pull}24021[#24021]

Suggesters
  • Remove deprecated _suggest endpoint {pull}22203[#22203] (issue: {issue}20305[#20305])

Task Manager
  • Add descriptions to bulk tasks {pull}22059[#22059] (issue: {issue}21768[#21768])

Bug Fixes

Ingest
  • Remove support for Visio and potm files {pull}22079[#22079] (issue: {issue}22077[#22077])

Inner Hits
  • If size / offset are out of bounds just do a plain count {pull}20556[#20556] (issue: {issue}20501[#20501])

Internal
  • Fix handling of document failure exception in InternalEngine {pull}22718[#22718]

Plugin Ingest Attachment
  • Add missing mime4j library {pull}22764[#22764] (issue: {issue}22077[#22077])

Plugin Repository S3
  • Wrap getCredentials() in a doPrivileged() block {pull}23297[#23297] (issues: {issue}22534[#22534], {issue}23271[#23271])

Sequence IDs
  • Avoid losing ops in file-based recovery {pull}22945[#22945] (issue: {issue}22484[#22484])

Snapshot/Restore
  • Keep snapshot restore state and routing table in sync {pull}20836[#20836] (issue: {issue}19774[#19774])

Translog
  • Fix Translog.Delete serialization for sequence numbers {pull}22543[#22543]

Watcher
  • The HTTP client respects timeouts now and does not get stuck leading to stuck watches.

Regressions

Bulk
  • Only re-parse operation if a mapping update was needed {pull}23832[#23832] (issue: {issue}23665[#23665])

Upgrades

Core
  • Upgrade to a Lucene 7 snapshot {pull}24089[#24089] (issues: {issue}23966[#23966], {issue}24086[#24086], {issue}24087[#24087], {issue}24088[#24088])

Plugin Ingest Attachment
  • Update to Tika 1.14 {pull}21591[#21591] (issue: {issue}20390[#20390])

{es} version 6.0.0-alpha1 (Changes previously released in 5.x)

The changes listed below were first released in the 5.x series. Changes released for the first time in Elasticsearch 6.0.0-alpha1 are listed in {es} version 6.0.0-alpha1.

Breaking Changes

Aliases
  • Validate alias names the same as index names {pull}20771[#20771] (issue: {issue}20748[#20748])

CRUD
  • Fixed naming inconsistency for fields/stored_fields in the APIs {pull}20166[#20166] (issues: {issue}18943[#18943], {issue}20155[#20155])

Core
  • Add system call filter bootstrap check {pull}21940[#21940]

  • Remove ignore system bootstrap checks {pull}20511[#20511]

Internal
  • _flush should block by default {pull}20597[#20597] (issue: {issue}20569[#20569])

Packaging
  • Rename service.bat to elasticsearch-service.bat {pull}20496[#20496] (issue: {issue}17528[#17528])

Plugin Lang Painless
  • Remove all date 'now' methods from Painless {pull}20766[#20766] (issue: {issue}20762[#20762])

Query DSL
  • Fix name of enabled_position_increments {pull}22895[#22895]

REST
  • Change separator for shards preference {pull}20786[#20786] (issues: {issue}20722[#20722], {issue}20769[#20769])

Search
  • Remove DFS_QUERY_AND_FETCH as a search type {pull}22787[#22787]

Settings
  • Remove support for default settings {pull}24093[#24093] (issues: {issue}23981[#23981], {issue}24052[#24052], {issue}24074[#24074])

Breaking Java Changes

Aggregations
  • Move getProperty method out of MultiBucketsAggregation.Bucket interface {pull}23988[#23988]

  • Remove getProperty method from Aggregations interface and impl {pull}23972[#23972]

  • Move getProperty method out of Aggregation interface {pull}23949[#23949]

Allocation
  • Cluster Explain API uses the allocation process to explain shard allocation decisions {pull}22182[#22182] (issues: {issue}20347[#20347], {issue}20634[#20634], {issue}21103[#21103], {issue}21662[#21662], {issue}21691[#21691])

Cluster
  • Remove PROTO-based custom cluster state components {pull}22336[#22336] (issue: {issue}21868[#21868])

Core
  • Remove ability to plug-in TransportService {pull}20505[#20505]

Discovery
  • Remove pluggability of ElectMasterService {pull}21031[#21031]

Exceptions
  • Remove IndexTemplateAlreadyExistsException and IndexShardAlreadyExistsException {pull}21539[#21539] (issue: {issue}21494[#21494])

  • Replace IndexAlreadyExistsException with ResourceAlreadyExistsException {pull}21494[#21494]

Ingest
  • Change type of ingest doc meta-data field 'TIMESTAMP' to Date {pull}22234[#22234] (issue: {issue}22074[#22074])

Internal
  • Replace SearchExtRegistry with namedObject {pull}22492[#22492]

  • Replace Suggesters with namedObject {pull}22491[#22491]

  • Consolidate the last easy parser construction {pull}22095[#22095]

  • Introduce XContentParser#namedObject {pull}22003[#22003]

  • Pass executor name to request interceptor to support async intercept calls {pull}21089[#21089]

  • Remove TransportService#registerRequestHandler leniency {pull}20469[#20469] (issue: {issue}20468[#20468])

Java API
  • Fold InternalSearchHits and friends into their interfaces {pull}23042[#23042]

Network
  • Remove HttpServer and HttpServerAdapter in favor of a simple dispatch method {pull}22636[#22636] (issue: {issue}18482[#18482])

  • Unguice Transport and friends {pull}20526[#20526]

Plugins
  • Deguice rest handlers {pull}22575[#22575]

  • Plugins: Replace Rest filters with RestHandler wrapper {pull}21905[#21905]

  • Plugins: Remove support for onModule {pull}21416[#21416]

  • Cleanup sub fetch phase extension point {pull}20382[#20382]

Query DSL
  • Resolve index names in indices_boost {pull}21393[#21393] (issue: {issue}4756[#4756])

Scripting
  • Refactor ScriptType to be a Top-Level Class {pull}21136[#21136]

Search
  • Remove QUERY_AND_FETCH search type {pull}22996[#22996]

  • Cluster search shards improvements: expose ShardId, adjust visibility of some members {pull}21752[#21752]

Deprecations

Java API
  • Add BulkProcessor methods with XContentType parameter {pull}23078[#23078] (issue: {issue}22691[#22691])

  • Deprecate and remove "minimumNumberShouldMatch" in BoolQueryBuilder {pull}22403[#22403]

Plugin Repository S3
  • S3 Repository: Deprecate remaining repositories.s3.* settings {pull}24144[#24144] (issue: {issue}24143[#24143])

  • Deprecate specifying credentials through env vars, sys props, and remove profile files {pull}22567[#22567] (issues: {issue}21041[#21041], {issue}22479[#22479])

Query DSL
  • Add deprecation logging message for 'fuzzy' query {pull}20993[#20993] (issue: {issue}15760[#15760])

REST
  • Optionally require a valid content type for all rest requests with content {pull}22691[#22691] (issue: {issue}19388[#19388])

Scripting
  • Change Namespace for Stored Script to Only Use Id {pull}22206[#22206]

Shadow Replicas
  • Add a deprecation notice to shadow replicas {pull}22647[#22647] (issue: {issue}22024[#22024])

Stats
  • Deprecate _field_stats endpoint {pull}23914[#23914]

New Features

Aggregations
  • Initial version of an adjacency matrix using the Filters aggregation {pull}22239[#22239] (issue: {issue}22169[#22169])

Analysis
  • Adds pattern keyword marker filter support {pull}23600[#23600] (issue: {issue}4877[#4877])

  • Expose WordDelimiterGraphTokenFilter {pull}23327[#23327] (issue: {issue}23104[#23104])

  • Synonym Graph Support (LUCENE-6664) {pull}21517[#21517]

  • Expose Lucenes Ukrainian analyzer {pull}21176[#21176] (issue: {issue}19433[#19433])

CAT API
  • Provides a cat api endpoint for templates. {pull}20545[#20545] (issue: {issue}20467[#20467])

CRUD
  • Allow an index to be partitioned with custom routing {pull}22274[#22274] (issue: {issue}21585[#21585])

Highlighting
  • Integrate UnifiedHighlighter {pull}21621[#21621] (issue: {issue}21376[#21376])

Index APIs
  • Add FieldCapabilities (_field_caps) API {pull}23007[#23007] (issue: {issue}22438[#22438])

Ingest
  • introduce KV Processor in Ingest Node {pull}22272[#22272] (issue: {issue}22222[#22222])

Mapping
  • Add the ability to set a normalizer on keyword fields. {pull}21919[#21919] (issue: {issue}18064[#18064])

  • Add RangeFieldMapper for numeric and date range types {pull}21002[#21002] (issue: {issue}20999[#20999])

Plugin Discovery File
  • File-based discovery plugin {pull}20394[#20394] (issue: {issue}20323[#20323])

Query DSL
  • Add "all fields" execution mode to simple_query_string query {pull}21341[#21341] (issues: {issue}19784[#19784], {issue}20925[#20925])

  • Add support for quote_field_suffix to simple_query_string. {pull}21060[#21060] (issue: {issue}18641[#18641])

  • Add "all field" execution mode to query_string query {pull}20925[#20925] (issue: {issue}19784[#19784])

Reindex API
  • Add automatic parallelization support to reindex and friends {pull}20767[#20767] (issue: {issue}20624[#20624])

Search
  • Introduce incremental reduction of TopDocs {pull}23946[#23946]

  • Add federated cross cluster search capabilities {pull}22502[#22502] (issue: {issue}21473[#21473])

  • Add field collapsing for search request {pull}22337[#22337] (issue: {issue}21833[#21833])

Settings
  • Add infrastructure for elasticsearch keystore {pull}22335[#22335]

Similarities
  • Adds boolean similarity to Elasticsearch {pull}23637[#23637] (issue: {issue}6731[#6731])

Enhancements

Aggregations
  • Add count to rest output of geo_centroid {pull}24387[#24387] (issue: {issue}24366[#24366])

  • Allow scripted metric agg to access _score {pull}24295[#24295]

  • Add BucketMetricValue interface {pull}24188[#24188]

  • Move aggs CommonFields and TYPED_KEYS_DELIMITER from InternalAggregation to Aggregation {pull}23987[#23987]

  • Use ParseField for aggs CommonFields rather than String {pull}23717[#23717]

  • Share XContent rendering code in terms aggs {pull}23680[#23680]

  • Add unit tests for ParentToChildAggregator {pull}23305[#23305] (issue: {issue}22278[#22278])

  • First step towards incremental reduction of query responses {pull}23253[#23253]

  • value_type is useful regardless of scripting. {pull}22160[#22160] (issue: {issue}20163[#20163])

  • Support for partitioning set of terms {pull}21626[#21626] (issue: {issue}21487[#21487])

  • Rescorer should be applied in the TopHits aggregation {pull}20978[#20978] (issue: {issue}19317[#19317])

Aliases
  • Handle multiple aliases in _cat/aliases api {pull}23698[#23698] (issue: {issue}23661[#23661])

Allocation
  • Trigger replica recovery restarts by master when primary relocation completes {pull}23926[#23926] (issue: {issue}23904[#23904])

  • Makes the same_shard host dynamically updatable {pull}23397[#23397] (issue: {issue}22992[#22992])

  • Include stale replica shard info when explaining an unassigned primary {pull}22826[#22826]

  • Adds setting level to allocation decider explanations {pull}22268[#22268] (issue: {issue}21771[#21771])

  • Improves allocation decider decision explanation messages {pull}21771[#21771]

  • Prepares allocator decision objects for use with the allocation explain API {pull}21691[#21691]

  • Balance step in BalancedShardsAllocator for a single shard {pull}21103[#21103]

  • Process more expensive allocation deciders last {pull}20724[#20724] (issue: {issue}12815[#12815])

  • Separates decision making from decision application in BalancedShardsAllocator {pull}20634[#20634]

Analysis
  • Support Keyword type in Analyze API {pull}23161[#23161]

  • Expose FlattenGraphTokenFilter {pull}22643[#22643]

  • Analyze API Position Length Support {pull}22574[#22574]

  • Remove AnalysisService and reduce it to a simple name to analyzer mapping {pull}20627[#20627] (issues: {issue}19827[#19827], {issue}19828[#19828])

CAT API
  • Adding built-in sorting capability to _cat apis. {pull}20658[#20658] (issue: {issue}16975[#16975])

  • Add health status parameter to cat indices API {pull}20393[#20393]

CRUD
  • Use correct block levels for TRA subclasses {pull}22224[#22224]

  • Make index and delete operation execute as a single bulk item {pull}21964[#21964]

Cache
  • Do not cache term queries. {pull}21566[#21566] (issues: {issue}16031[#16031], {issue}20116[#20116])

  • Parse alias filters on the coordinating node {pull}20916[#20916]

Circuit Breakers
  • Closing a ReleasableBytesStreamOutput closes the underlying BigArray {pull}23941[#23941]

  • Add used memory amount to CircuitBreakingException message (#22521) {pull}22693[#22693] (issue: {issue}22521[#22521])

  • Cluster Settings Updates should not trigger circuit breakers. {pull}20827[#20827]

Cluster
  • Extract a common base class to allow services to listen to remote cluster config updates {pull}24367[#24367]

  • Prevent nodes from joining if newer indices exist in the cluster {pull}23843[#23843]

  • Connect to new nodes concurrently {pull}22984[#22984] (issue: {issue}22828[#22828])

  • Keep NodeConnectionsService in sync with current nodes in the cluster state {pull}22509[#22509]

  • Add a generic way of checking version before serializing custom cluster object {pull}22376[#22376] (issue: {issue}22313[#22313])

  • Add validation for supported index version on node join, restore, upgrade & open index {pull}21830[#21830] (issue: {issue}21670[#21670])

  • Let ClusterStateObserver only hold onto state that’s needed for change detection {pull}21631[#21631] (issue: {issue}21568[#21568])

  • Cache successful shard deletion checks {pull}21438[#21438]

  • Remove mutable status field from cluster state {pull}21379[#21379]

  • Skip shard management code when updating cluster state on client/tribe nodes {pull}20731[#20731]

  • Add clusterUUID to RestMainAction output {pull}20503[#20503]

Core
  • Regex upgrades {pull}24316[#24316] (issue: {issue}24226[#24226])

  • Detect remnants of path.data/default.path.data bug {pull}24099[#24099] (issues: {issue}23981[#23981], {issue}24052[#24052], {issue}24074[#24074], {issue}24093[#24093])

  • Await termination after shutting down executors {pull}23889[#23889]

  • Add early-access check {pull}23743[#23743] (issue: {issue}23668[#23668])

  • Adapter action future should restore interrupts {pull}23618[#23618] (issue: {issue}23617[#23617])

  • Disable bootstrap checks for single-node discovery {pull}23598[#23598] (issues: {issue}23585[#23585], {issue}23595[#23595])

  • Enable explicitly enforcing bootstrap checks {pull}23585[#23585] (issue: {issue}21864[#21864])

  • Add equals/hashcode method to ReplicationResponse {pull}23215[#23215]

  • Simplify ElasticsearchException rendering as a XContent {pull}22611[#22611]

  • Remove setLocalNode from ClusterService and TransportService {pull}22608[#22608]

  • Rename bootstrap.seccomp to bootstrap.system_call_filter {pull}22226[#22226] (issue: {issue}21940[#21940])

  • Cleanup random stats serialization code {pull}22223[#22223]

  • Avoid corruption when deserializing booleans {pull}22152[#22152]

  • Reduce memory pressure when sending large terms queries. {pull}21776[#21776]

  • Install a security manager on startup {pull}21716[#21716]

  • Log node ID on startup {pull}21673[#21673]

  • Ensure source filtering automatons are only compiled once {pull}20857[#20857] (issue: {issue}20839[#20839])

  • Improve scheduling fairness when batching cluster state changes with equal priority {pull}20775[#20775] (issue: {issue}20768[#20768])

  • Add production warning for pre-release builds {pull}20674[#20674]

  • Add serial collector bootstrap check {pull}20558[#20558]

  • Do not log full bootstrap checks exception {pull}19989[#19989]

Dates
  • Improve error handling for epoch format parser with time zone (#22621) {pull}23689[#23689]

Discovery
  • Introduce single-node discovery {pull}23595[#23595]

  • UnicastZenPing shouldn’t ping the address of the local node {pull}23567[#23567]

  • MasterFaultDetection can start after the initial cluster state has been processed {pull}23037[#23037] (issue: {issue}22828[#22828])

  • Simplify Unicast Zen Ping {pull}22277[#22277] (issues: {issue}19370[#19370], {issue}21739[#21739], {issue}22120[#22120], {issue}22194[#22194])

  • Prefer joining node with conflicting transport address when becoming master {pull}22134[#22134] (issues: {issue}22049[#22049], {issue}22120[#22120])

Engine
  • Engine: store maxUnsafeAutoIdTimestamp in commit {pull}24149[#24149]

  • Replace EngineClosedException with AlreadyClosedExcpetion {pull}22631[#22631]

Exceptions
  • Add BWC layer for Exceptions {pull}21694[#21694] (issue: {issue}21656[#21656])

Geo
  • Optimize geo-distance sorting. {pull}20596[#20596] (issue: {issue}20450[#20450])

Highlighting
  • Add support for fragment_length in the unified highlighter {pull}23431[#23431]

  • Add BreakIteratorBoundaryScanner support {pull}23248[#23248]

Index APIs
  • Open and close index to honour allow_no_indices option {pull}24222[#24222] (issue: {issue}24031[#24031])

  • Wildcard cluster names for cross cluster search {pull}23985[#23985] (issue: {issue}23893[#23893])

  • Indexing: Add shard id to indexing operation listener {pull}22606[#22606]

  • Better error when can’t auto create index {pull}22488[#22488] (issues: {issue}21448[#21448], {issue}22435[#22435])

  • Add date-math support to _rollover {pull}20709[#20709]

Ingest
  • Lazy load the geoip databases {pull}23337[#23337]

  • add ignore_missing flag to ingest plugins {pull}22273[#22273]

  • Added ability to remove pipelines via wildcards (#22149) {pull}22191[#22191] (issue: {issue}22149[#22149])

  • Enables the ability to inject serialized json fields into root of document {pull}22179[#22179] (issue: {issue}21898[#21898])

  • compile ScriptProcessor inline scripts when creating ingest pipelines {pull}21858[#21858] (issue: {issue}21842[#21842])

  • add ignore_missing option to SplitProcessor {pull}20982[#20982] (issues: {issue}19995[#19995], {issue}20840[#20840])

  • add ignore_missing option to convert,trim,lowercase,uppercase,grok,rename {pull}20194[#20194] (issue: {issue}19995[#19995])

  • introduce the JSON Processor {pull}20128[#20128] (issue: {issue}20052[#20052])

Internal
  • Add cross cluster support to _field_caps {pull}24463[#24463] (issue: {issue}24334[#24334])

  • Log JVM arguments on startup {pull}24451[#24451]

  • Preserve cluster alias throughout search execution to lookup nodes by cluster and ID {pull}24438[#24438]

  • Move RemoteClusterService into TransportService {pull}24424[#24424]

  • Enum related performance additions. {pull}24274[#24274] (issue: {issue}24226[#24226])

  • Add a dedicated TransportRemoteInfoAction for consistency {pull}24040[#24040] (issue: {issue}23969[#23969])

  • Simplify sorted top docs merging in SearchPhaseController {pull}23881[#23881]

  • Synchronized CollapseTopFieldDocs with lucenes relatives {pull}23854[#23854]

  • Cleanup SearchPhaseController interface {pull}23844[#23844]

  • Do not create String instances in 'Strings' methods accepting StringBuilder {pull}22907[#22907]

  • Improve connection closing in RemoteClusterConnection {pull}22804[#22804] (issue: {issue}22803[#22803])

  • Remove some more usages of ParseFieldMatcher {pull}22437[#22437] (issues: {issue}19552[#19552], {issue}22130[#22130])

  • Remove some more usages of ParseFieldMatcher {pull}22398[#22398] (issues: {issue}19552[#19552], {issue}22130[#22130])

  • Remove some more usages of ParseFieldMatcher {pull}22395[#22395] (issues: {issue}19552[#19552], {issue}22130[#22130])

  • Remove some ParseFieldMatcher usages {pull}22389[#22389] (issues: {issue}19552[#19552], {issue}22130[#22130])

  • Introduce ToXContentObject interface {pull}22387[#22387] (issue: {issue}16347[#16347])

  • Add infrastructure to manage network connections outside of Transport/TransportService {pull}22194[#22194]

  • Replace strict parsing mode with response headers assertions {pull}22130[#22130] (issues: {issue}11859[#11859], {issue}19552[#19552], {issue}20993[#20993])

  • Start using ObjectParser for aggs. {pull}22048[#22048] (issue: {issue}22009[#22009])

  • Don’t output null source node in RecoveryFailedException {pull}21963[#21963]

  • ClusterService should expose "applied" cluster states (i.e., remove ClusterStateStatus) {pull}21817[#21817]

  • Rename ClusterState#lookupPrototypeSafe to lookupPrototype and remove "unsafe" unused variant {pull}21686[#21686]

  • ShardActiveResponseHandler shouldn’t hold to an entire cluster state {pull}21470[#21470] (issue: {issue}21394[#21394])

  • Remove unused ClusterService dependency from SearchPhaseController {pull}21421[#21421]

  • Remove special case in case no action filters are registered {pull}21251[#21251]

  • Use TimveValue instead of long for CacheBuilder methods {pull}20887[#20887]

  • Remove SearchContext#current and all it’s threadlocals {pull}20778[#20778] (issue: {issue}19341[#19341])

  • Remove poor-mans compression in InternalSearchHit and friends {pull}20472[#20472]

Java API
  • Added types options to DeleteByQueryRequest {pull}23265[#23265] (issue: {issue}21984[#21984])

  • prevent NPE when trying to uncompress a null BytesReference {pull}22386[#22386]

Java High Level REST Client
  • Add utility method to parse named XContent objects with typed prefix {pull}24240[#24240] (issue: {issue}22965[#22965])

  • Convert suggestion response parsing to use NamedXContentRegistry {pull}23355[#23355]

  • UpdateRequest implements ToXContent {pull}23289[#23289]

  • Add javadoc for DocWriteResponse.Builders {pull}23267[#23267]

  • Expose WriteRequest.RefreshPolicy string representation {pull}23106[#23106]

  • Use typed_keys parameter to prefix suggester names by type in search responses {pull}23080[#23080] (issue: {issue}22965[#22965])

  • Add parsing from xContent to MainResponse {pull}22934[#22934]

  • Parse elasticsearch exception’s root causes {pull}22924[#22924]

  • Add parsing method to BytesRestResponse’s error {pull}22873[#22873]

  • Add parsing methods to BulkItemResponse {pull}22859[#22859]

  • Add parsing method for ElasticsearchException.generateFailureXContent() {pull}22815[#22815]

  • Add parsing method for ElasticsearchException.generateThrowableXContent() {pull}22783[#22783]

  • Add parsing methods for UpdateResponse {pull}22586[#22586]

  • Add parsing from xContent to InternalSearchHit and InternalSearchHits {pull}22429[#22429]

  • Add fromxcontent methods to index response {pull}22229[#22229]

  • Add fromXContent() methods for ReplicationResponse {pull}22196[#22196] (issue: {issue}22082[#22082])

  • Add parsing method for ElasticsearchException {pull}22143[#22143]

  • Add fromXContent method to GetResponse {pull}22082[#22082]

Java REST Client
  • move ignore parameter support from yaml test client to low level rest client {pull}22637[#22637]

  • Warn log deprecation warnings received from server {pull}21895[#21895]

  • Support Preemptive Authentication with RestClient {pull}21336[#21336]

  • Provide error message when rest request path is null {pull}21233[#21233] (issue: {issue}21232[#21232])

Logging
  • Log deleting indices at info level {pull}22627[#22627] (issue: {issue}22605[#22605])

  • Expose logs base path {pull}22625[#22625]

  • Log failure to connect to node at info instead of debug {pull}21809[#21809] (issue: {issue}6468[#6468])

  • Truncate log messages from the end {pull}21609[#21609] (issue: {issue}21602[#21602])

  • Ensure logging is initialized in CLI tools {pull}20575[#20575]

  • Give useful error message if log config is missing {pull}20493[#20493]

  • Complete Elasticsearch logger names {pull}20457[#20457] (issue: {issue}20326[#20326])

  • Logging shutdown hack {pull}20389[#20389] (issue: {issue}20304[#20304])

  • Disable console logging {pull}20387[#20387]

  • Warn on not enough masters during election {pull}20063[#20063] (issue: {issue}8362[#8362])

Mapping
  • Do not index _type when there is at most one type. {pull}24363[#24363]

  • Only allow one type on 6.0 indices {pull}24317[#24317] (issue: {issue}15613[#15613])

  • token_count type : add an option to count tokens (fix #23227) {pull}24175[#24175] (issue: {issue}23227[#23227])

  • Atomic mapping updates across types {pull}22220[#22220]

  • Only update DocumentMapper if field type changes {pull}22165[#22165]

  • Better error message when _parent isn’t an object {pull}21987[#21987]

  • Create the QueryShardContext lazily in DocumentMapperParser. {pull}21287[#21287]

Nested Docs
  • Avoid adding unnecessary nested filters when ranges are used. {pull}23427[#23427]

Network
  • Set available processors for Netty {pull}24420[#24420] (issue: {issue}6224[#6224])

  • Adjust default Netty receive predictor size to 64k {pull}23542[#23542] (issue: {issue}23185[#23185])

  • Keep the pipeline handler queue small initially {pull}23335[#23335]

  • Set network receive predictor size to 32kb {pull}23284[#23284] (issue: {issue}23185[#23185])

  • TransportService.connectToNode should validate remote node ID {pull}22828[#22828] (issue: {issue}22194[#22194])

  • Disable the Netty recycler {pull}22452[#22452] (issues: {issue}22189[#22189], {issue}22360[#22360], {issue}22406[#22406], {issue}5904[#5904])

  • Tell Netty not to be unsafe in transport client {pull}22284[#22284]

  • Introduce a low level protocol handshake {pull}22094[#22094]

  • Detach handshake from connect to node {pull}22037[#22037]

  • Reduce number of connections per node depending on the nodes role {pull}21849[#21849]

  • Add a connect timeout to the ConnectionProfile to allow per node connect timeouts {pull}21847[#21847] (issue: {issue}19719[#19719])

  • Grant Netty permission to read system somaxconn {pull}21840[#21840]

  • Remove connectToNodeLight and replace it with a connection profile {pull}21799[#21799]

  • Lazy resolve unicast hosts {pull}21630[#21630] (issues: {issue}14441[#14441], {issue}16412[#16412])

  • Fix handler name on message not fully read {pull}21478[#21478]

  • Handle rejected pings on shutdown gracefully {pull}20842[#20842]

  • Network: Allow to listen on virtual interfaces. {pull}19568[#19568] (issues: {issue}17473[#17473], {issue}19537[#19537])

Packaging
  • Introduce Java version check {pull}23194[#23194] (issue: {issue}21102[#21102])

  • Improve the out-of-the-box experience {pull}21920[#21920] (issues: {issue}18317[#18317], {issue}21783[#21783])

  • Add empty plugins dir for archive distributions {pull}21204[#21204] (issue: {issue}20342[#20342])

  • Make explicit missing settings for Windows service {pull}21200[#21200] (issue: {issue}18317[#18317])

  • Change permissions on config files {pull}20966[#20966]

  • Add quiet option to disable console logging {pull}20422[#20422] (issues: {issue}15315[#15315], {issue}16159[#16159], {issue}17220[#17220])

Percolator
  • Allowing range queries with now ranges inside percolator queries {pull}23921[#23921] (issue: {issue}23859[#23859])

  • Add term extraction support for MultiPhraseQuery {pull}23176[#23176]

Plugin Discovery EC2
  • Settings: Migrate ec2 discovery sensitive settings to elasticsearch keystore {pull}23961[#23961] (issue: {issue}22475[#22475])

  • Add support for ca-central-1 region to EC2 and S3 plugins {pull}22458[#22458] (issue: {issue}22454[#22454])

  • Support for eu-west-2 (London) cloud-aws plugin {pull}22308[#22308] (issue: {issue}22306[#22306])

  • Add us-east-2 AWS region {pull}21961[#21961] (issue: {issue}21881[#21881])

  • Add setting to set read timeout for EC2 discovery and S3 repository plugins {pull}21956[#21956] (issue: {issue}19078[#19078])

Plugin Ingest GeoIp
  • Cache results of geoip lookups {pull}22231[#22231] (issue: {issue}22074[#22074])

Plugin Lang Painless
  • Allow painless to load stored fields {pull}24290[#24290]

  • Start on custom whitelists for Painless {pull}23563[#23563]

  • Fix Painless’s implementation of interfaces returning primitives {pull}23298[#23298] (issue: {issue}22983[#22983])

  • Allow painless to implement more interfaces {pull}22983[#22983]

  • Generate reference links for painless API {pull}22775[#22775]

  • Painless: Add augmentation to String for base 64 {pull}22665[#22665] (issue: {issue}22648[#22648])

  • Improve painless’s ScriptException generation {pull}21762[#21762] (issue: {issue}21733[#21733])

  • Add Debug.explain to painless {pull}21723[#21723] (issue: {issue}20263[#20263])

  • Implement the ?: operator in painless {pull}21506[#21506]

  • In painless suggest a long constant if int won’t do {pull}21415[#21415] (issue: {issue}21313[#21313])

  • Support decimal constants with trailing [dD] in painless {pull}21412[#21412] (issue: {issue}21116[#21116])

  • Implement reading from null safe dereferences {pull}21239[#21239]

  • Painless negative offsets {pull}21080[#21080] (issue: {issue}20870[#20870])

  • Remove more equivalents of the now method from the Painless whitelist. {pull}21047[#21047]

  • Disable regexes by default in painless {pull}20427[#20427] (issue: {issue}20397[#20397])

Plugin Repository Azure
  • Add Backoff policy to azure repository {pull}23387[#23387] (issue: {issue}22728[#22728])

Plugin Repository S3
  • Removes the retry mechanism from the S3 blob store {pull}23952[#23952] (issue: {issue}22845[#22845])

  • S3 Repository: Eagerly load static settings {pull}23910[#23910]

  • S3 repository: Add named configurations {pull}22762[#22762] (issues: {issue}22479[#22479], {issue}22520[#22520])

  • Make the default S3 buffer size depend on the available memory. {pull}21299[#21299]

Plugins
  • Plugins: Add support for platform specific plugins {pull}24265[#24265]

  • Plugins: Remove leniency for missing plugins dir {pull}24173[#24173]

  • Modify permissions dialog for plugins {pull}23742[#23742]

  • Plugins: Add plugin cli specific exit codes {pull}23599[#23599] (issue: {issue}15295[#15295])

  • Plugins: Output better error message when existing plugin is incompatible {pull}23562[#23562] (issue: {issue}20691[#20691])

  • Add the ability to define search response listeners in search plugin {pull}22682[#22682]

  • Pass ThreadContext to transport interceptors to allow header modification {pull}22618[#22618] (issue: {issue}22585[#22585])

  • Provide helpful error message if a plugin exists {pull}22305[#22305] (issue: {issue}22084[#22084])

  • Add shutdown hook for closing CLI commands {pull}22126[#22126] (issue: {issue}22111[#22111])

  • Allow plugins to install bootstrap checks {pull}22110[#22110]

  • Clarify that plugins can be closed {pull}21669[#21669]

  • Plugins: Convert custom discovery to pull based plugin {pull}21398[#21398]

  • Removing plugin that isn’t installed shouldn’t trigger usage information {pull}21272[#21272] (issue: {issue}21250[#21250])

  • Remove pluggability of ZenPing {pull}21049[#21049]

  • Make UnicastHostsProvider extension pull based {pull}21036[#21036]

  • Revert "Display plugins versions" {pull}20807[#20807] (issues: {issue}18683[#18683], {issue}20668[#20668])

  • Provide error message when plugin id is missing {pull}20660[#20660]

Query DSL
  • Make it possible to validate a query on all shards instead of a single random shard {pull}23697[#23697] (issue: {issue}18254[#18254])

  • QueryString and SimpleQueryString Graph Support {pull}22541[#22541]

  • Additional Graph Support in Match Query {pull}22503[#22503] (issue: {issue}22490[#22490])

  • RangeQuery WITHIN case now normalises query {pull}22431[#22431] (issue: {issue}22412[#22412])

  • Un-deprecate fuzzy query {pull}22088[#22088] (issue: {issue}15760[#15760])

  • support numeric bounds with decimal parts for long/integer/short/byte datatypes {pull}21972[#21972] (issue: {issue}21600[#21600])

  • Using ObjectParser in MatchAllQueryBuilder and IdsQueryBuilder {pull}21273[#21273]

  • Expose splitOnWhitespace in Query String Query {pull}20965[#20965] (issue: {issue}20841[#20841])

  • Throw error if query element doesn’t end with END_OBJECT {pull}20528[#20528] (issue: {issue}20515[#20515])

  • Remove lowercase_expanded_terms and locale from query-parser options. {pull}20208[#20208] (issue: {issue}9978[#9978])

REST
  • Allow passing single scrollID in clear scroll API body {pull}24242[#24242] (issue: {issue}24233[#24233])

  • Validate top-level keys when parsing mget requests {pull}23746[#23746] (issue: {issue}23720[#23720])

  • Cluster stats should not render empty http/transport types {pull}23735[#23735]

  • Add parameter to prefix aggs name with type in search responses {pull}22965[#22965]

  • Add a REST spec for the create API {pull}20924[#20924]

  • Add response params to REST params did you mean {pull}20753[#20753] (issues: {issue}20722[#20722], {issue}20747[#20747])

  • Add did you mean to strict REST params {pull}20747[#20747] (issue: {issue}20722[#20722])

Reindex API
  • Increase visibility of doExecute so it can be used directly {pull}22614[#22614]

  • Improve error message when reindex-from-remote gets bad json {pull}22536[#22536] (issue: {issue}22330[#22330])

  • Reindex: Better error message for pipeline in wrong place {pull}21985[#21985]

  • Timeout improvements for rest client and reindex {pull}21741[#21741] (issue: {issue}21707[#21707])

  • Add "simple match" support for reindex-from-remote whitelist {pull}21004[#21004]

  • Make reindex-from-remote ignore unknown fields {pull}20591[#20591] (issue: {issue}20504[#20504])

Scripting
  • Expose multi-valued dates to scripts and document painless’s date functions {pull}22875[#22875] (issue: {issue}22162[#22162])

  • Wrap VerifyError in ScriptException {pull}21769[#21769]

  • Log ScriptException’s xcontent if file script compilation fails {pull}21767[#21767] (issue: {issue}21733[#21733])

  • Support binary field type in script values {pull}21484[#21484] (issue: {issue}14469[#14469])

  • Mustache: Add {{#url}}{{/url}} function to URL encode strings {pull}20838[#20838]

  • Expose ctx._now in update scripts {pull}20835[#20835] (issue: {issue}17895[#17895])

Search
  • Remove leniency when merging fetched hits in a search response phase {pull}24158[#24158]

  • Set shard count limit to unlimited {pull}24012[#24012]

  • Streamline shard index availability in all SearchPhaseResults {pull}23788[#23788]

  • Search took time should use a relative clock {pull}23662[#23662]

  • Prevent negative from parameter in SearchSourceBuilder {pull}23358[#23358] (issue: {issue}23324[#23324])

  • Remove unnecessary result sorting in SearchPhaseController {pull}23321[#23321]

  • Expose batched_reduce_size via _search {pull}23288[#23288] (issue: {issue}23253[#23253])

  • Adding fromXContent to Suggest and Suggestion class {pull}23226[#23226] (issue: {issue}23202[#23202])

  • Adding fromXContent to Suggestion.Entry and subclasses {pull}23202[#23202]

  • Add CollapseSearchPhase as a successor for the FetchSearchPhase {pull}23165[#23165]

  • Integrate IndexOrDocValuesQuery. {pull}23119[#23119]

  • Detach SearchPhases from AbstractSearchAsyncAction {pull}23118[#23118]

  • Fix GraphQuery expectation after Lucene upgrade to 6.5 {pull}23117[#23117] (issue: {issue}23102[#23102])

  • Nested queries should avoid adding unnecessary filters when possible. {pull}23079[#23079] (issue: {issue}20797[#20797])

  • Add xcontent parsing to completion suggestion option {pull}23071[#23071]

  • Add xcontent parsing to suggestion options {pull}23018[#23018]

  • Separate reduce (aggs, suggest and profile) from merging fetched hits {pull}23017[#23017]

  • Add a setting to disable remote cluster connections on a node {pull}23005[#23005]

  • First step towards separating individual search phases {pull}22802[#22802]

  • Add parsing from xContent to SearchProfileShardResults and nested classes {pull}22649[#22649]

  • Move SearchTransportService and SearchPhaseController creation outside of TransportSearchAction constructor {pull}21754[#21754]

  • Don’t carry ShardRouting around when not needed in AbstractSearchAsyncAction {pull}21753[#21753]

  • ShardSearchRequest to take ShardId constructor argument rather than the whole ShardRouting {pull}21750[#21750]

  • Use index uuid as key in the alias filter map rather than the index name {pull}21749[#21749]

  • Add indices and filter information to search shards api output {pull}21738[#21738] (issue: {issue}20916[#20916])

  • remove pointless catch exception in TransportSearchAction {pull}21689[#21689]

  • Optimize query with types filter in the URL (t/t/_search) {pull}20979[#20979]

  • Makes search action cancelable by task management API {pull}20405[#20405]

Search Templates
  • Add profile and explain parameters to template API {pull}20451[#20451]

Settings
  • Add secure file setting to keystore {pull}24001[#24001]

  • Add a property to mark setting as final {pull}23872[#23872]

  • Remove obsolete index setting index.version.minimum_compatible. {pull}23593[#23593]

  • Provide a method to retrieve a closeable char[] from a SecureString {pull}23389[#23389]

  • Update indices settings api to support CBOR and SMILE format {pull}23309[#23309] (issues: {issue}23242[#23242], {issue}23245[#23245])

  • Improve setting deprecation message {pull}23156[#23156] (issue: {issue}22849[#22849])

  • Add secure settings validation on startup {pull}22894[#22894]

  • Allow comma delimited array settings to have a space after each entry {pull}22591[#22591] (issue: {issue}22297[#22297])

  • Allow affix settings to be dynamic / updatable {pull}22526[#22526]

  • Allow affix settings to delegate to actual settings {pull}22523[#22523]

  • Make s3 repository sensitive settings use secure settings {pull}22479[#22479]

  • Speed up filter and prefix settings operations {pull}22249[#22249]

  • Add precise logging on unknown or invalid settings {pull}20951[#20951] (issue: {issue}20946[#20946])

Snapshot/Restore
  • Ensure every repository has an incompatible-snapshots blob {pull}24403[#24403] (issue: {issue}22267[#22267])

  • Change snapshot status error to use generic SnapshotException {pull}24355[#24355] (issue: {issue}24225[#24225])

  • Duplicate snapshot name throws InvalidSnapshotNameException {pull}22921[#22921] (issue: {issue}18228[#18228])

  • Fixes retrieval of the latest snapshot index blob {pull}22700[#22700]

  • Use general cluster state batching mechanism for snapshot state updates {pull}22528[#22528] (issue: {issue}14899[#14899])

  • Synchronize snapshot deletions on the cluster state {pull}22313[#22313] (issue: {issue}19957[#19957])

  • Abort snapshots on a node that leaves the cluster {pull}21084[#21084] (issue: {issue}20876[#20876])

Stats
  • Show JVM arguments {pull}24450[#24450]

  • Add cross-cluster search remote cluster info API {pull}23969[#23969] (issue: {issue}23925[#23925])

  • Add geo_point to FieldStats {pull}21947[#21947] (issue: {issue}20707[#20707])

  • Include unindexed field in FieldStats response {pull}21821[#21821] (issue: {issue}21952[#21952])

  • Remove load average leniency {pull}21380[#21380]

  • Strengthen handling of unavailable cgroup stats {pull}21094[#21094] (issue: {issue}21029[#21029])

  • Add basic cgroup CPU metrics {pull}21029[#21029]

Suggesters
  • Provide informative error message in case of unknown suggestion context. {pull}24241[#24241]

  • Allow different data types for category in Context suggester {pull}23491[#23491] (issue: {issue}22358[#22358])

Task Manager
  • Limit IndexRequest toString() length {pull}22832[#22832]

  • Improve the error message if task and node isn’t found {pull}22062[#22062] (issue: {issue}22027[#22027])

  • Add descriptions to create snapshot and restore snapshot tasks. {pull}21901[#21901] (issue: {issue}21768[#21768])

  • Add proper descriptions to reindex, update-by-query and delete-by-query tasks. {pull}21841[#21841] (issue: {issue}21768[#21768])

  • Add search task descriptions {pull}21740[#21740]

Tribe Node
  • Add support for merging custom meta data in tribe node {pull}21552[#21552] (issues: {issue}20544[#20544], {issue}20791[#20791], {issue}9372[#9372])

Upgrade API
  • Allow plugins to upgrade templates and index metadata on startup {pull}24379[#24379]

Bug Fixes

Aggregations
  • InternalPercentilesBucket should not rely on ordered percents array {pull}24336[#24336] (issue: {issue}24331[#24331])

  • Align behavior HDR percentiles iterator with percentile() method {pull}24206[#24206]

  • The filter and significant_terms aggregations should parse the filter as a filter, not a query. {pull}23797[#23797]

  • Completion suggestion should also consider text if prefix/regex is missing {pull}23451[#23451] (issue: {issue}23340[#23340])

  • Fixes the per term error in the terms aggregation {pull}23399[#23399]

  • Fixes terms error count for multiple reduce phases {pull}23291[#23291] (issue: {issue}23286[#23286])

  • Fix scaled_float numeric type in aggregations {pull}22351[#22351] (issue: {issue}22350[#22350])

  • Allow terms aggregations on pure boolean scripts. {pull}22201[#22201] (issue: {issue}20941[#20941])

  • Fix numeric terms aggregations with includes/excludes and minDocCount=0 {pull}22141[#22141] (issue: {issue}22140[#22140])

  • Fix missing on aggs on boolean fields. {pull}22135[#22135] (issue: {issue}22009[#22009])

  • IP range masks exclude the maximum address of the range. {pull}22018[#22018] (issue: {issue}22005[#22005])

  • Fix other_bucket on the filters agg to be enabled if a key is set. {pull}21994[#21994] (issue: {issue}21951[#21951])

  • Rewrite Queries/Filter in FilterAggregationBuilder and ensure client usage marks query as non-cachable {pull}21303[#21303] (issue: {issue}21301[#21301])

  • Percentiles bucket fails for 100th percentile {pull}21218[#21218]

  • Thread safety for scripted significance heuristics {pull}21113[#21113] (issue: {issue}18120[#18120])

  • ip_range aggregation should accept null bounds. {pull}21043[#21043] (issue: {issue}21006[#21006])

  • Fixes bug preventing script sort working on top_hits aggregation {pull}21023[#21023] (issue: {issue}21022[#21022])

  • Fixed writeable name from range to geo_distance {pull}20860[#20860]

  • Fix date_range aggregation to not cache if now is used {pull}20740[#20740]

  • The top_hits aggregation should compile scripts only once. {pull}20738[#20738]

Allocation
  • Discard stale node responses from async shard fetching {pull}24434[#24434] (issue: {issue}24007[#24007])

  • Cannot force allocate primary to a node where the shard already exists {pull}22031[#22031] (issue: {issue}22021[#22021])

  • Promote shadow replica to primary when initializing primary fails {pull}22021[#22021]

  • Trim in-sync allocations set only when it grows {pull}21976[#21976] (issue: {issue}21719[#21719])

  • Allow master to assign primary shard to node that has shard store locked during shard state fetching {pull}21656[#21656] (issue: {issue}19416[#19416])

  • Keep a shadow replicas' allocation id when it is promoted to primary {pull}20863[#20863] (issue: {issue}20650[#20650])

  • IndicesClusterStateService should clean local started when re-assigns an initializing shard with the same aid {pull}20687[#20687]

  • IndexRoutingTable.initializeEmpty shouldn’t override supplied primary RecoverySource {pull}20638[#20638] (issue: {issue}20637[#20637])

  • Update incoming recoveries stats when shadow replica is reinitialized {pull}20612[#20612]

  • index.routing.allocation.initial_recovery limits replica allocation {pull}20589[#20589]

Analysis
  • AsciiFoldingFilter’s multi-term component should never preserve the original token. {pull}21982[#21982]

  • Pre-built analysis factories do not implement MultiTermAware correctly. {pull}21981[#21981]

  • Can load non-PreBuiltTokenFilter in Analyze API {pull}20396[#20396]

  • Named analyzer should close the analyzer that it wraps {pull}20197[#20197]

Bulk
  • Reject empty IDs {pull}24118[#24118] (issue: {issue}24116[#24116])

CAT API
  • Consume full_id request parameter early {pull}21270[#21270] (issue: {issue}21266[#21266])

CRUD
  • Reject external versioning and explicit version numbers on create {pull}21998[#21998]

  • MultiGet should not fail entirely if alias resolves to many indices {pull}20858[#20858] (issue: {issue}20845[#20845])

  • Fixed date math expression support in multi get requests. {pull}20659[#20659] (issue: {issue}17957[#17957])

Cache
  • Invalidate cached query results if query timed out {pull}22807[#22807] (issue: {issue}22789[#22789])

  • Fix the request cache keys to not hold references to the SearchContext. {pull}21284[#21284]

  • Prevent requests that use scripts or now() from being cached {pull}20750[#20750] (issue: {issue}20645[#20645])

Circuit Breakers
  • ClusterState publishing shouldn’t trigger circuit breakers {pull}20986[#20986] (issues: {issue}20827[#20827], {issue}20960[#20960])

Cluster
  • Don’t set local node on cluster state used for node join validation {pull}23311[#23311] (issues: {issue}21830[#21830], {issue}3[#3], {issue}4[#4], {issue}6[#6], {issue}9[#9])

  • Allow a cluster state applier to create an observer and wait for a better state {pull}23132[#23132] (issue: {issue}21817[#21817])

  • Cluster allocation explain to never return empty response body {pull}23054[#23054]

  • IndicesService handles all exceptions during index deletion {pull}22433[#22433]

  • Remove cluster update task when task times out {pull}21578[#21578] (issue: {issue}21568[#21568])

Core
  • Check for default.path.data included in path.data {pull}24285[#24285] (issue: {issue}24283[#24283])

  • Improve performance of extracting warning value {pull}24114[#24114] (issue: {issue}24018[#24018])

  • Reject duplicate settings on the command line {pull}24053[#24053]

  • Restrict build info loading to ES jar, not any jar {pull}24049[#24049] (issue: {issue}21955[#21955])

  • Streamline foreign stored context restore and allow to perserve response headers {pull}22677[#22677] (issue: {issue}22647[#22647])

  • Support negative numbers in readVLong {pull}22314[#22314]

  • Add a StreamInput#readArraySize method that ensures sane array sizes {pull}21697[#21697]

  • Use a buffer to do character to byte conversion in StreamOutput#writeString {pull}21680[#21680] (issue: {issue}21660[#21660])

  • Fix ShardInfo#toString {pull}21319[#21319]

  • Protect BytesStreamOutput against overflows of the current number of written bytes. {pull}21174[#21174] (issue: {issue}21159[#21159])

  • Return target index name even if _rollover conditions are not met {pull}21138[#21138]

  • .es_temp_file remains after system crash, causing it not to start again {pull}21007[#21007] (issue: {issue}20992[#20992])

  • StoreStatsCache should also ignore AccessDeniedException when checking file size {pull}20790[#20790] (issue: {issue}17580[#17580])

Dates
  • Fix time zone rounding edge case for DST overlaps {pull}21550[#21550] (issue: {issue}20833[#20833])

Discovery
  • ZenDiscovery - only validate min_master_nodes values if local node is master {pull}23915[#23915] (issue: {issue}23695[#23695])

  • Close InputStream when receiving cluster state in PublishClusterStateAction {pull}22711[#22711]

  • Do not reply to pings from another cluster {pull}21894[#21894] (issue: {issue}21874[#21874])

  • Add current cluster state version to zen pings and use them in master election {pull}20384[#20384] (issue: {issue}20348[#20348])

Engine
  • Close and flush refresh listeners on shard close {pull}22342[#22342]

  • Die with dignity on the Lucene layer {pull}21721[#21721] (issue: {issue}19272[#19272])

  • Fix InternalEngine#isThrottled to not always return false. {pull}21592[#21592]

  • Retrying replication requests on replica doesn’t call onRetry {pull}21189[#21189] (issue: {issue}20211[#20211])

  • Take refresh IOExceptions into account when catching ACE in InternalEngine {pull}20546[#20546] (issue: {issue}19975[#19975])

Exceptions
  • Stop returning "es." internal exception headers as http response headers {pull}22703[#22703] (issue: {issue}17593[#17593])

  • Fixing shard recovery error message to report the number of docs correctly for each node {pull}22515[#22515] (issue: {issue}21893[#21893])

Highlighting
  • Fix FiltersFunctionScoreQuery highlighting {pull}21827[#21827]

  • Fix highlighting on a stored keyword field {pull}21645[#21645] (issue: {issue}21636[#21636])

  • Fix highlighting of MultiTermQuery within a FunctionScoreQuery {pull}20400[#20400] (issue: {issue}20392[#20392])

Index APIs
  • Fixes restore of a shrunken index when initial recovery node is gone {pull}24322[#24322] (issue: {issue}24257[#24257])

  • Honor update request timeout {pull}23825[#23825]

  • Ensure shrunk indices carry over version information from its source {pull}22469[#22469] (issue: {issue}22373[#22373])

  • Validate the _rollover target index name early to also fail if dry_run=true {pull}21330[#21330] (issue: {issue}21149[#21149])

  • Only negate index expression on all indices with preceding wildcard {pull}20898[#20898] (issues: {issue}19800[#19800], {issue}20033[#20033])

  • Fix IndexNotFoundException in multi index search request. {pull}20188[#20188] (issue: {issue}3839[#3839])

Index Templates
  • Fix integer overflows when dealing with templates. {pull}21628[#21628] (issue: {issue}21622[#21622])

Ingest
  • Improve missing ingest processor error {pull}23379[#23379] (issue: {issue}23392[#23392])

  • update _ingest.timestamp to use new ZonedDateTime {pull}23174[#23174] (issue: {issue}23168[#23168])

  • fix date-processor to a new default year for every new pipeline execution {pull}22601[#22601] (issue: {issue}22547[#22547])

  • fix index out of bounds error in KV Processor {pull}22288[#22288] (issue: {issue}22272[#22272])

  • Fixes GrokProcessor’s ignorance of named-captures with same name. {pull}22131[#22131] (issue: {issue}22117[#22117])

  • fix trace_match behavior for when there is only one grok pattern {pull}21413[#21413] (issue: {issue}21371[#21371])

  • Stored scripts and ingest node configurations should be included into a snapshot {pull}21227[#21227] (issue: {issue}21184[#21184])

  • make painless the default scripting language for ScriptProcessor {pull}20981[#20981] (issue: {issue}20943[#20943])

  • no null values in ingest configuration error messages {pull}20616[#20616]

  • JSON Processor was not properly added {pull}20613[#20613]

Inner Hits
  • Replace NestedChildrenQuery with ParentChildrenBlockJoinQuery {pull}24016[#24016] (issue: {issue}24009[#24009])

  • Changed DisMaxQueryBuilder to extract inner hits from leaf queries {pull}23512[#23512] (issue: {issue}23482[#23482])

  • Inner hits and ignore unmapped {pull}21693[#21693] (issue: {issue}21620[#21620])

  • Skip adding a parent field to nested documents. {pull}21522[#21522] (issue: {issue}21503[#21503])

Internal
  • Fix NPE if field caps request has a field that exists not in all indices {pull}24504[#24504]

  • Add infrastructure to mark contexts as system contexts {pull}23830[#23830]

  • Always restore the ThreadContext for operations delayed due to a block {pull}23349[#23349]

  • Index creation and setting update may not return deprecation logging {pull}22702[#22702]

  • Rethrow ExecutionException from the loader to concurrent callers of Cache#computeIfAbsent {pull}21549[#21549]

  • Restore thread’s original context before returning to the ThreadPool {pull}21411[#21411]

  • Fix NPE in SearchContext.toString() {pull}21069[#21069]

  • Prevent AbstractArrays from release bytes more than once {pull}20819[#20819]

  • Source filtering should treat dots in field names as sub objects. {pull}20736[#20736] (issue: {issue}20719[#20719])

  • IndicesAliasesRequest should not implement CompositeIndicesRequest {pull}20726[#20726]

  • Ensure elasticsearch doesn’t start with unuspported indices {pull}20514[#20514] (issue: {issue}20512[#20512])

Java API
  • Don’t output empty ext object in SearchSourceBuilder#toXContent {pull}22093[#22093] (issue: {issue}20969[#20969])

  • Transport client: Fix remove address to actually work {pull}21743[#21743]

  • Add a HostFailureListener to notify client code if a node got disconnected {pull}21709[#21709] (issue: {issue}21424[#21424])

  • Fix InternalSearchHit#hasSource to return the proper boolean value {pull}21441[#21441] (issue: {issue}21419[#21419])

  • Null checked for source when calling sourceRef {pull}21431[#21431] (issue: {issue}19279[#19279])

  • ClusterAdminClient.prepareDeletePipeline method should accept pipeline id to delete {pull}21228[#21228]

  • fix IndexResponse#toString to print out shards info {pull}20562[#20562]

Java High Level REST Client
  • Correctly parse BulkItemResponse.Failure’s status {pull}23432[#23432]

Java REST Client
  • Make buffer limit configurable in HeapBufferedConsumerFactory {pull}23970[#23970] (issue: {issue}23958[#23958])

  • RestClient asynchronous execution should not throw exceptions {pull}23307[#23307]

  • Don’t use null charset in RequestLogger {pull}22197[#22197] (issue: {issue}22190[#22190])

  • Rest client: don’t reuse the same HttpAsyncResponseConsumer across multiple retries {pull}21378[#21378]

Logging
  • Do not prematurely shutdown Log4j {pull}21519[#21519] (issue: {issue}21514[#21514])

  • Assert status logger does not warn on Log4j usage {pull}21339[#21339]

  • Fix logger names for Netty {pull}21223[#21223] (issue: {issue}20457[#20457])

  • Fix logger when you can not create an azure storage client {pull}20670[#20670] (issues: {issue}20633[#20633], {issue}20669[#20669])

  • Avoid unnecessary creation of prefix loggers {pull}20571[#20571] (issue: {issue}20570[#20570])

  • Fix logging hierarchy configs {pull}20463[#20463]

  • Fix prefix logging {pull}20429[#20429]

Mapping
  • Preserve response headers when creating an index {pull}23950[#23950] (issue: {issue}23947[#23947])

  • Improves disabled fielddata error message {pull}23841[#23841] (issue: {issue}22768[#22768])

  • Fix MapperService StackOverflowError {pull}23605[#23605] (issue: {issue}23604[#23604])

  • Fix NPE with scaled floats stats when field is not indexed {pull}23528[#23528] (issue: {issue}23487[#23487])

  • Range types causing GetFieldMappingsIndexRequest to fail due to NullPointerException in RangeFieldMapper.doXContentBody when include_defaults=true is on the query string {pull}22925[#22925]

  • Disallow introducing illegal object mappings (double '..') {pull}22891[#22891] (issue: {issue}22794[#22794])

  • The _all default mapper is not completely configured. {pull}22236[#22236]

  • Fix MapperService.allEnabled(). {pull}22227[#22227]

  • Dynamic date fields should use the format that was used to detect it is a date. {pull}22174[#22174] (issue: {issue}9410[#9410])

  • Sub-fields should not accept include_in_all parameter {pull}21971[#21971] (issue: {issue}21710[#21710])

  • Mappings: Fix get mapping when no indexes exist to not fail in response generation {pull}21924[#21924] (issue: {issue}21916[#21916])

  • Fail to index fields with dots in field names when one of the intermediate objects is nested. {pull}21787[#21787] (issue: {issue}21726[#21726])

  • Uncommitted mapping updates should not efect existing indices {pull}21306[#21306] (issue: {issue}21189[#21189])

Nested Docs
  • Fix bug in query builder rewrite that ignores the ignore_unmapped option {pull}22456[#22456]

Network
  • Respect promises on pipelined responses {pull}23317[#23317] (issues: {issue}23310[#23310], {issue}23322[#23322])

  • Ensure that releasing listener is called {pull}23310[#23310]

  • Pass forceExecution flag to transport interceptor {pull}22739[#22739]

  • Ensure new connections won’t be opened if transport is closed or closing {pull}22589[#22589] (issue: {issue}22554[#22554])

  • Prevent open channel leaks if handshake times out or is interrupted {pull}22554[#22554]

  • Execute low level handshake in #openConnection {pull}22440[#22440]

  • Handle connection close / reset events gracefully during handshake {pull}22178[#22178]

  • Do not lose host information when pinging {pull}21939[#21939] (issue: {issue}21828[#21828])

  • DiscoveryNode and TransportAddress should preserve host information {pull}21828[#21828]

  • Die with dignity on the network layer {pull}21720[#21720] (issue: {issue}19272[#19272])

  • Fix connection close header handling {pull}20956[#20956] (issue: {issue}20938[#20938])

  • Ensure port range is readable in the exception message {pull}20893[#20893]

  • Prevent double release in TcpTransport if send listener throws an exception {pull}20880[#20880]

Packaging
  • Fall back to non-atomic move when removing plugins {pull}23548[#23548] (issue: {issue}35[#35])

  • Another fix for handling of paths on Windows {pull}22132[#22132] (issue: {issue}21921[#21921])

  • Fix handling of spaces in Windows paths {pull}21921[#21921] (issues: {issue}20809[#20809], {issue}21525[#21525])

  • Add option to skip kernel parameters on install {pull}21899[#21899] (issue: {issue}21877[#21877])

  • Set vm.max_map_count on systemd package install {pull}21507[#21507]

  • Export ES_JVM_OPTIONS for SysV init {pull}21445[#21445] (issue: {issue}21255[#21255])

  • Debian: configure start-stop-daemon to not go into background {pull}21343[#21343] (issues: {issue}12716[#12716], {issue}21300[#21300])

  • Generate POM files with non-wildcard excludes {pull}21234[#21234] (issue: {issue}21170[#21170])

  • [Packaging] Do not remove scripts directory on upgrade {pull}20452[#20452]

  • [Package] Remove bin/lib/modules directories on RPM uninstall/upgrade {pull}20448[#20448]

Parent/Child
  • Add null check in case of orphan child document {pull}22772[#22772] (issue: {issue}22770[#22770])

Percolator
  • Fix memory leak when percolator uses bitset or field data cache {pull}24115[#24115] (issue: {issue}24108[#24108])

  • Fix NPE in percolator’s 'now' range check for percolator queries with range queries {pull}22356[#22356] (issue: {issue}22355[#22355])

Plugin Analysis Stempel
  • Fix thread safety of Stempel’s token filter factory {pull}22610[#22610] (issue: {issue}21911[#21911])

Plugin Discovery EC2
  • Fix ec2 discovery when used with IAM profiles. {pull}21048[#21048] (issue: {issue}21039[#21039])

Plugin Ingest GeoIp
  • [ingest-geoip] update geoip to not include null-valued results from {pull}20455[#20455]

Plugin Lang Painless
  • painless: Fix method references to ctor with the new LambdaBootstrap and cleanup code {pull}24406[#24406]

  • Fix Painless Lambdas for Java 9 {pull}24070[#24070] (issue: {issue}23473[#23473])

  • Fix painless’s regex lexer and error messages {pull}23634[#23634]

  • Replace Painless’s Cast with casting strategies {pull}23369[#23369]

  • Fix Bad Casts In Painless {pull}23282[#23282] (issue: {issue}23238[#23238])

  • Don’t allow casting from void to def in painless {pull}22969[#22969] (issue: {issue}22908[#22908])

  • Fix def invoked qualified method refs {pull}22918[#22918]

  • Whitelist some ScriptDocValues in painless {pull}22600[#22600] (issue: {issue}22584[#22584])

  • Update Painless Loop Counter to be Higher {pull}22560[#22560] (issue: {issue}22508[#22508])

  • Fix some issues with painless’s strings {pull}22393[#22393] (issue: {issue}22372[#22372])

  • Test fix for def equals in Painless {pull}21945[#21945] (issue: {issue}21801[#21801])

  • Fix a VerifyError bug in Painless {pull}21765[#21765]

  • Fix Lambdas in Painless to be Able to Use Top-Level Variables Such as params and doc {pull}21635[#21635] (issues: {issue}20869[#20869], {issue}21479[#21479])

  • Fix String Concatenation Bug In Painless {pull}20623[#20623]

Plugin Repository Azure
  • Azure blob store’s readBlob() method first checks if the blob exists {pull}23483[#23483] (issue: {issue}23480[#23480])

  • Fixes default chunk size for Azure repositories {pull}22577[#22577] (issue: {issue}22513[#22513])

  • readonly on azure repository must be taken into account {pull}22055[#22055] (issues: {issue}22007[#22007], {issue}22053[#22053])

Plugin Repository HDFS
  • Fixing permission errors for KERBEROS security mode for HDFS Repository {pull}23439[#23439] (issue: {issue}22156[#22156])

Plugin Repository S3
  • Handle BlobPath’s trailing separator case. Add test cases to BlobPathTests.java {pull}23091[#23091]

  • Fixes leading forward slash in S3 repository base_path {pull}20861[#20861]

Plugins
  • Fix delete of plugin directory on remove plugin {pull}24266[#24266] (issue: {issue}24252[#24252])

  • Use a marker file when removing a plugin {pull}24252[#24252] (issue: {issue}24231[#24231])

  • Remove hidden file leniency from plugin service {pull}23982[#23982] (issue: {issue}12465[#12465])

  • Add check for null pluginName in remove command {pull}22930[#22930] (issue: {issue}22922[#22922])

  • Use sysprop like with es.path.home to pass conf dir {pull}18870[#18870] (issue: {issue}18689[#18689])

Query DSL
  • FuzzyQueryBuilder should error when parsing array of values {pull}23762[#23762] (issue: {issue}23759[#23759])

  • Fix parsing for max_determinized_states {pull}22749[#22749] (issue: {issue}22722[#22722])

  • Fix script score function that combines _score and weight {pull}22713[#22713] (issue: {issue}21483[#21483])

  • Fixes date range query using epoch with timezone {pull}21542[#21542] (issue: {issue}21501[#21501])

  • Allow overriding all-field leniency when lenient option is specified {pull}21504[#21504] (issues: {issue}20925[#20925], {issue}21341[#21341])

  • Max score should be updated when a rescorer is used {pull}20977[#20977] (issue: {issue}20651[#20651])

  • Fixes MultiMatchQuery so that it doesn’t provide a null context {pull}20882[#20882]

  • Fix silently accepting malformed queries {pull}20515[#20515] (issue: {issue}20500[#20500])

  • Fix match_phrase_prefix query with single term on _all field {pull}20471[#20471] (issue: {issue}20470[#20470])

REST
  • [API] change wait_for_completion default according to docs {pull}23672[#23672]

  • Deprecate request_cache for clear-cache {pull}23638[#23638] (issue: {issue}22748[#22748])

  • HTTP transport stashes the ThreadContext instead of the RestController {pull}23456[#23456]

  • Fix date format in warning headers {pull}23418[#23418] (issue: {issue}23275[#23275])

  • Align REST specs for HEAD requests {pull}23313[#23313] (issue: {issue}21125[#21125])

  • Correct warning header to be compliant {pull}23275[#23275] (issue: {issue}22986[#22986])

  • Fix get HEAD requests {pull}23186[#23186] (issue: {issue}21125[#21125])

  • Handle bad HTTP requests {pull}23153[#23153] (issue: {issue}23034[#23034])

  • Fix get source HEAD requests {pull}23151[#23151] (issue: {issue}21125[#21125])

  • Properly encode location header {pull}23133[#23133] (issues: {issue}21057[#21057], {issue}23115[#23115])

  • Fix template HEAD requests {pull}23130[#23130] (issue: {issue}21125[#21125])

  • Fix index HEAD requests {pull}23112[#23112] (issue: {issue}21125[#21125])

  • Fix alias HEAD requests {pull}23094[#23094] (issue: {issue}21125[#21125])

  • Strict level parsing for indices stats {pull}21577[#21577] (issue: {issue}21024[#21024])

  • The routing query string param is supported by mget but was missing from the rest spec {pull}21357[#21357]

  • fix thread_pool_patterns path variable definition {pull}21332[#21332]

  • Read indices options in indices upgrade API {pull}21281[#21281] (issue: {issue}21099[#21099])

  • ensure the XContentBuilder is always closed in RestBuilderListener {pull}21124[#21124]

  • Add correct Content-Length on HEAD requests {pull}21123[#21123] (issue: {issue}21077[#21077])

  • Make sure HEAD / has 0 Content-Length {pull}21077[#21077] (issue: {issue}21075[#21075])

  • Adds percent-encoding for Location headers {pull}21057[#21057] (issue: {issue}21016[#21016])

  • Whitelist node stats indices level parameter {pull}21024[#21024] (issue: {issue}20722[#20722])

  • Remove lenient URL parameter parsing {pull}20722[#20722] (issue: {issue}14719[#14719])

  • XContentBuilder: Avoid building self-referencing objects {pull}20550[#20550] (issues: {issue}19475[#19475], {issue}20540[#20540])

Recovery
  • Provide target allocation id as part of start recovery request {pull}24333[#24333] (issue: {issue}24167[#24167])

  • Fix primary relocation for shadow replicas {pull}22474[#22474] (issue: {issue}20300[#20300])

  • Don’t close store under CancellableThreads {pull}22434[#22434] (issue: {issue}22325[#22325])

  • Use a fresh recovery id when retrying recoveries {pull}22325[#22325] (issue: {issue}22043[#22043])

  • Allow flush/force_merge/upgrade on shard marked as relocated {pull}22078[#22078] (issue: {issue}22043[#22043])

  • Fix concurrency issues between cancelling a relocation and marking shard as relocated {pull}20443[#20443]

Reindex API
  • Fix throttled reindex_from_remote {pull}23953[#23953] (issues: {issue}23828[#23828], {issue}23945[#23945])

  • Fix reindex with a remote source on a version before 2.0.0 {pull}23805[#23805]

  • Make reindex wait for cleanup before responding {pull}23677[#23677] (issue: {issue}23653[#23653])

  • Reindex: do not log when can’t clear old scroll {pull}22942[#22942] (issue: {issue}22937[#22937])

  • Fix reindex-from-remote from <2.0 {pull}22931[#22931] (issue: {issue}22893[#22893])

  • Fix reindex from remote clearing scroll {pull}22525[#22525] (issue: {issue}22514[#22514])

  • Fix source filtering in reindex-from-remote {pull}22514[#22514] (issue: {issue}22507[#22507])

  • Remove content type detection from reindex-from-remote {pull}22504[#22504] (issue: {issue}22329[#22329])

  • Don’t close rest client from its callback {pull}22061[#22061] (issue: {issue}22027[#22027])

  • Keep context during reindex’s retries {pull}21941[#21941]

  • Ignore IllegalArgumentException with assertVersionSerializable {pull}21409[#21409] (issues: {issue}20767[#20767], {issue}21350[#21350])

  • Bump reindex-from-remote’s buffer to 200mb {pull}21222[#21222] (issue: {issue}21185[#21185])

  • Fix reindex-from-remote for parent/child from <2.0 {pull}21070[#21070] (issue: {issue}21044[#21044])

Scripting
  • Convert script/template objects to json format internally {pull}23308[#23308] (issue: {issue}23245[#23245])

  • Script: Fix value of ctx._now to be current epoch time in milliseconds {pull}23175[#23175] (issue: {issue}23169[#23169])

  • Expose ip fields as strings in scripts. {pull}21997[#21997] (issue: {issue}21977[#21977])

  • Add support for booleans in scripts {pull}20950[#20950] (issue: {issue}20949[#20949])

  • Native scripts should be created once per index, not per segment. {pull}20609[#20609]

Search
  • Include all aliases including non-filtering in _search_shards response {pull}24489[#24489]

  • Cross Cluster Search: propagate original indices per cluster {pull}24328[#24328]

  • Query string default field {pull}24214[#24214]

  • Speed up parsing of large terms queries. {pull}24210[#24210]

  • IndicesQueryCache should delegate the scorerSupplier method. {pull}24209[#24209]

  • Disable graph analysis at query time for shingle and cjk filters producing tokens of different size {pull}23920[#23920] (issue: {issue}23918[#23918])

  • Fix cross-cluster remote node gateway attributes {pull}23863[#23863]

  • Use a fixed seed for computing term hashCode in TermsSliceQuery {pull}23795[#23795]

  • Honor max concurrent searches in multi-search {pull}23538[#23538] (issue: {issue}23527[#23527])

  • Avoid stack overflow in multi-search {pull}23527[#23527] (issue: {issue}23523[#23523])

  • Fix query_string_query to transform "foo:*" in an exists query on the field name {pull}23433[#23433] (issue: {issue}23356[#23356])

  • Factor out filling of TopDocs in SearchPhaseController {pull}23380[#23380] (issues: {issue}19356[#19356], {issue}23357[#23357])

  • Replace blocking calls in ExpandCollapseSearchResponseListener by asynchronous requests {pull}23053[#23053] (issue: {issue}23048[#23048])

  • Ensure fixed serialization order of InnerHitBuilder {pull}22820[#22820] (issue: {issue}22808[#22808])

  • Improve concurrency of ShardCoreKeyMap. {pull}22316[#22316]

  • Make -0 compare less than +0 consistently. {pull}22173[#22173] (issue: {issue}22167[#22167])

  • Fix boost_mode propagation when the function score query builder is rewritten {pull}22172[#22172] (issue: {issue}22138[#22138])

  • FiltersAggregationBuilder: rewriting filter queries, the same way as in FilterAggregationBuilder {pull}22076[#22076]

  • Fix cross_fields type on multi_match query with synonyms {pull}21638[#21638] (issue: {issue}21633[#21633])

  • Fix match_phrase_prefix on boosted fields {pull}21623[#21623] (issue: {issue}21613[#21613])

  • Respect default search timeout {pull}21599[#21599] (issues: {issue}12211[#12211], {issue}21595[#21595])

  • Remove LateParsingQuery to prevent timestamp access after context is frozen {pull}21328[#21328] (issue: {issue}21295[#21295])

  • Make range queries round up upper bounds again. {pull}20582[#20582] (issues: {issue}20579[#20579], {issue}8889[#8889])

  • Throw error when trying to fetch fields from source and source is disabled {pull}20424[#20424] (issues: {issue}20093[#20093], {issue}20408[#20408])

Search Templates
  • No longer add illegal content type option to stored search templates {pull}24251[#24251] (issue: {issue}24227[#24227])

  • SearchTemplateRequest to implement CompositeIndicesRequest {pull}21865[#21865] (issue: {issue}21747[#21747])

Settings
  • Do not set path.data in environment if not set {pull}24132[#24132] (issue: {issue}24099[#24099])

  • Correct handling of default and array settings {pull}24074[#24074] (issues: {issue}23981[#23981], {issue}24052[#24052])

  • Fix merge scheduler config settings {pull}23391[#23391]

  • Settings: Fix keystore cli prompting for yes/no to handle console returning null {pull}23320[#23320]

  • Expose search.highlight.term_vector_multi_value as a node level setting {pull}22999[#22999]

  • NPE when no setting name passed to elasticsearch-keystore {pull}22609[#22609]

  • Handle spaces in action.auto_create_index gracefully {pull}21790[#21790] (issue: {issue}21449[#21449])

  • Fix settings diff generation for affix and group settings {pull}21788[#21788]

  • Don’t reset non-dynamic settings unless explicitly requested {pull}21646[#21646] (issue: {issue}21593[#21593])

  • Fix Setting.timeValue() method {pull}20696[#20696] (issue: {issue}20662[#20662])

  • Add a hard limit for index.number_of_shard {pull}20682[#20682]

  • Include complex settings in settings requests {pull}20622[#20622]

Snapshot/Restore
  • Fixes maintaining the shards a snapshot is waiting on {pull}24289[#24289]

  • Fixes snapshot status on failed snapshots {pull}23833[#23833] (issue: {issue}23716[#23716])

  • Fixes snapshot deletion handling on in-progress snapshot failure {pull}23703[#23703] (issue: {issue}23663[#23663])

  • Prioritize listing index-N blobs over index.latest in reading snapshots {pull}23333[#23333]

  • Gracefully handles pre 2.x compressed snapshots {pull}22267[#22267]

  • URLRepository should throw NoSuchFileException to correctly adhere to readBlob contract {pull}22069[#22069] (issue: {issue}22004[#22004])

  • Fixes shard level snapshot metadata loading when index-N file is missing {pull}21813[#21813]

  • Ensures cleanup of temporary index-* generational blobs during snapshotting {pull}21469[#21469] (issue: {issue}21462[#21462])

  • Fixes get snapshot duplicates when asking for _all {pull}21340[#21340] (issue: {issue}21335[#21335])

Stats
  • Avoid overflow when computing total FS stats {pull}23641[#23641]

  • Handle existence of cgroup version 2 hierarchy {pull}23493[#23493] (issue: {issue}23486[#23486])

  • Handle long overflow when adding paths' totals {pull}23293[#23293] (issue: {issue}23093[#23093])

  • Fix control group pattern {pull}23219[#23219] (issue: {issue}23218[#23218])

  • Fix total disk bytes returning negative value {pull}23093[#23093]

  • Implement stats for geo_point and geo_shape field {pull}22391[#22391] (issue: {issue}22384[#22384])

  • Use reader for doc stats {pull}22317[#22317] (issue: {issue}22285[#22285])

  • Avoid NPE in NodeService#stats if HTTP is disabled {pull}22060[#22060] (issue: {issue}22058[#22058])

  • Add support for "include_segment_file_sizes" in indices stats REST handler {pull}21879[#21879] (issue: {issue}21878[#21878])

  • Remove output_uuid parameter from cluster stats {pull}21020[#21020] (issue: {issue}20722[#20722])

  • Fix FieldStats deserialization of ip field {pull}20522[#20522] (issue: {issue}20516[#20516])

Task Manager
  • Task Management: Make TaskInfo parsing forwards compatible {pull}24073[#24073] (issue: {issue}23250[#23250])

  • Fix hanging cancelling task with no children {pull}22796[#22796]

  • Fix broken TaskInfo.toString() {pull}22698[#22698] (issue: {issue}22387[#22387])

  • Task cancellation command should wait for all child nodes to receive cancellation request before returning {pull}21397[#21397] (issue: {issue}21126[#21126])

Term Vectors
  • Fix _termvectors with preference to not hit NPE {pull}21959[#21959]

  • Return correct term statistics when a field is not found in a shard {pull}21922[#21922] (issue: {issue}21906[#21906])

Tribe Node
  • Add socket permissions for tribe nodes {pull}21546[#21546] (issues: {issue}16392[#16392], {issue}21122[#21122])

Regressions

Bulk
  • Fix _bulk response when it can’t create an index {pull}24048[#24048] (issues: {issue}22488[#22488], {issue}24028[#24028])

Core
  • Source filtering: only accept array items if the previous include pattern matches {pull}22593[#22593] (issue: {issue}22557[#22557])

Highlighting
  • Handle SynonymQuery extraction for the FastVectorHighlighter {pull}20829[#20829] (issue: {issue}20781[#20781])

Logging
  • Restores the original default format of search slow log {pull}21770[#21770] (issue: {issue}21711[#21711])

Network
  • You had one job Netty logging guard {pull}24469[#24469] (issues: {issue}5624[#5624], {issue}6568[#6568])

Plugin Discovery EC2
  • Fix ec2 discovery when used with IAM profiles. {pull}21042[#21042] (issue: {issue}21039[#21039])

Plugin Repository S3
  • Fix s3 repository when used with IAM profiles {pull}21058[#21058] (issue: {issue}21048[#21048])

Plugins
  • Plugins: Add back user agent when downloading plugins {pull}20872[#20872]

Search
  • Handle specialized term queries in MappedFieldType.extractTerm(Query) {pull}21889[#21889] (issue: {issue}21882[#21882])

Upgrades

Aggregations
  • Upgrade HDRHistogram to 2.1.9 {pull}23254[#23254] (issue: {issue}23239[#23239])

Core
  • Upgrade to Lucene 6.5.0 {pull}23750[#23750]

  • Upgrade from JNA 4.2.2 to JNA 4.4.0 {pull}23636[#23636]

  • Upgrade to lucene-6.5.0-snapshot-d00c5ca {pull}23385[#23385]

  • Upgrade to lucene-6.5.0-snapshot-f919485. {pull}23087[#23087]

  • Upgrade to Lucene 6.4.0 {pull}22724[#22724]

  • Update Jackson to 2.8.6 {pull}22596[#22596] (issue: {issue}22266[#22266])

  • Upgrade to lucene-6.4.0-snapshot-084f7a0. {pull}22413[#22413]

  • Upgrade to lucene-6.4.0-snapshot-ec38570 {pull}21853[#21853]

  • Upgrade to lucene-6.3.0. {pull}21464[#21464]

Dates
  • Update Joda Time to version 2.9.5 {pull}21468[#21468] (issues: {issue}20911[#20911], {issue}332[#332], {issue}373[#373], {issue}378[#378], {issue}379[#379], {issue}386[#386], {issue}394[#394], {issue}396[#396], {issue}397[#397], {issue}404[#404], {issue}69[#69])

Internal
  • Upgrade to Lucene 6.4.1. {pull}22978[#22978]

Logging
  • Upgrade to Log4j 2.8.2 {pull}23995[#23995]

  • Upgrade Log4j 2 to version 2.7 {pull}20805[#20805] (issue: {issue}20304[#20304])

Network
  • Upgrade Netty to 4.1.10.Final {pull}24414[#24414]

  • Upgrade to Netty 4.1.9 {pull}23540[#23540] (issues: {issue}23172[#23172], {issue}6308[#6308], {issue}6374[#6374])

  • Upgrade to Netty 4.1.8 {pull}23055[#23055]

  • Upgrade to Netty 4.1.7 {pull}22587[#22587]

  • Upgrade to Netty 4.1.6 {pull}21051[#21051]

Plugin Repository Azure
  • Update to Azure Storage 5.0.0 {pull}23517[#23517] (issue: {issue}23448[#23448])