"Fossies" - the Fresh Open Source Software Archive

Member "solr-8.4.1/lucene/CHANGES.txt" (10 Jan 2020, 742863 Bytes) of package /linux/www/solr-8.4.1-src.tgz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "CHANGES.txt" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 8.4.0_vs_8.4.1.

    1 Lucene Change Log
    2 
    3 For more information on past and future Lucene versions, please see:
    4 http://s.apache.org/luceneversions
    5 
    6 ======================= Lucene 8.4.1 =======================
    7 
    8 Bug Fixes
    9 ---------------------
   10 (No changes)
   11 
   12 ======================= Lucene 8.4.0 =======================
   13 
   14 API Changes
   15 
   16 * LUCENE-9029: Deprecate SloppyMath toRadians/toDegrees in favor of Java Math.
   17   (Jack Conradson via Adrien Grand)
   18 
   19 New Features
   20 
   21 * LUCENE-8620: Add CONTAINS support for LatLonShape and XYShape. (Ignacio Vera)
   22 
   23 Improvements
   24 
   25 * LUCENE-9002: Skip costly caching clause in LRUQueryCache if it makes the query
   26   many times slower. (Guoqiang Jiang)
   27 
   28 * LUCENE-9006: WordDelimiterGraphFilter's catenateAll token is now ordered before any token parts, like WDF did.
   29   (David Smiley)
   30 
   31 * LUCENE-9028: introducing Intervals.multiterm() (Mikhail Khludnev)
   32 
   33 * LUCENE-9018: ConcatenateGraphFilter now has a configurable separator. (Stanislav Mikulchik, David Smiley)
   34 
   35 * LUCENE-9036: ExitableDirectoryReader may interupt scaning over DocValues (Mikhail Khludnev)
   36 
   37 * LUCENE-9062: QueryVisitor now has a consumeTermsMatching() method, allowing queries
   38   that match a class of terms to pass a ByteRunAutomaton matching those that class
   39   back to the visitor. (Alan Woodward, David Smiley)
   40 
   41 * LUCENE-9073: IntervalQuery to respond field on toString() and explain() (Mikhail Khludnev)
   42 
   43 Optimizations
   44 
   45 * LUCENE-8928: When building a kd-tree for dimensions n > 2, compute exact bounds for an inner node every N splits
   46   to improve the quality of the tree. N is defined by SPLITS_BEFORE_EXACT_BOUNDS which is set to 4.
   47   (Ignacio Vera, Adrien Grand)
   48 
   49 * BaseDirectoryReader no longer sums up the `LeafReader#numDocs` of its leaves
   50   eagerly. This especially helps when creating views of readers that hide
   51   documents, since computing the number of live documents is an expensive
   52   operation. (Adrien Grand)
   53 
   54 * LUCENE-8992: TopFieldCollector and TopScoreDocCollector can now share minimum scores across leaves
   55   concurrently. (Adrien Grand, Atri Sharma, Jim Ferenczi)
   56 
   57 * LUCENE-8932: BKDReader's index is now stored off-heap when the IndexInput is
   58   an instance of ByteBufferIndexInput. (Jack Conradson via Adrien Grand)
   59 
   60 * LUCENE-9024: IntroSelector now falls back to the median of medians algorithm
   61   instead of sorting when the maximum recursion level is exceeded, providing
   62   better worst-case runtime. (Paul Sanwald via Adrien Grand)
   63 
   64 * LUCENE-8920: The denser arcs of FST now index labels with a bitset in order
   65   to provide near constant time access. (Bruno Roustant, Mike Sokolov via Adrien Grand)
   66 
   67 * LUCENE-9027: Use SIMD instructions to decode postings. (Adrien Grand)
   68 
   69 * LUCENE-9049: Remove FST cached root arcs now redundant with labels indexed by bitset.
   70   This frees some on-heap FST space. (Jack Conradson via Bruno Roustant)
   71 
   72 * LUCENE-9045: Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat. (Bruno Roustant)
   73 
   74 Bug Fixes
   75 
   76 * LUCENE-9001: Fix race condition in SetOnce. (Przemko Robakowski)
   77 
   78 * LUCENE-9030: Fix WordnetSynonymParser behaviour so it behaves similar to
   79   SolrSynonymParser. (Christoph Buescher via Alan Woodward)
   80 
   81 * LUCENE-9054: Fix reproduceJenkinsFailures.py to not overwrite junit XML files when retrying (hossman)
   82 
   83 * LUCENE-9031: UnsupportedOperationException on MatchesIterator.getQuery() (Alan Woodward, Mikhail Khludnev)
   84 
   85 * LUCENE-8996: maxScore was sometimes missing from distributed grouped responses.
   86   (Julien Massenet, Diego Ceccarelli, Munendra S N, Christine Poerschke)
   87 
   88 * LUCENE-9055: Fix the detection of lines crossing triangles through edge points.
   89   (Ignacio Vera)
   90 
   91 * LUCENE-9103: Disjunctions can miss some hits in some rare conditions. (Adrien Grand)
   92 
   93 Other
   94 
   95 * LUCENE-8979: Code Cleanup: Use entryset for map iteration wherever possible. - Part 2 (Koen De Groote)
   96 
   97 * LUCENE-8746: Refactor EdgeTree - Introduce a Component tree that represents the tree of components (e.g polygons).
   98   Edge tree is now just a tree of edges. (Ignacio Vera)
   99 
  100 * LUCENE-8994: Code Cleanup - Pass values to list constructor instead of empty constructor followed by addAll(). (Koen De Groote)
  101 
  102 * LUCENE-9046: Fix wrong example in Javadoc of TermInSetQuery (Namgyu Kim)
  103 
  104 * LUCENE-8983: Add sandbox PhraseWildcardQuery to control multi-terms expansions in a phrase. (Bruno Roustant)
  105 
  106 * LUCENE-9067: Polygon2D#contains() is now thread safe. (Ignacio Vera)
  107 
  108 Build
  109 
  110 * Upgrade forbiddenapis to version 2.7; upgrade Groovy to 2.4.17.  (Uwe Schindler)
  111 
  112 * LUCENE-9041: Upgrade ecj to 3.19.0 to fix sporadic precommit javadoc issues (Kevin Risden)
  113 
  114 ======================= Lucene 8.3.1 =======================
  115 
  116 Bug Fixes
  117 
  118 * LUCENE-9050: MultiTermIntervalsSource.visit() was not calling back to its
  119   visitor. (Alan Woodward)
  120 
  121 ======================= Lucene 8.3.0 =======================
  122 
  123 API Changes
  124 
  125 * LUCENE-8909: IndexWriter#getFieldNames() method is used to get fields present in index. After LUCENE-8316, this
  126   method is no longer required. Hence, deprecate IndexWriter#getFieldNames() method. (Adrien Grand, Munendra S N)
  127 
  128 * LUCENE-8755: SpatialPrefixTreeFactory now consumes the "version" parsed with Lucene's Version class.  The quad
  129   and packed quad prefix trees are sensitive to this.  It's recommended to pass the version like you
  130   should do likewise for analysis components for tokenized text, or else changes to the encoding in future versions
  131   may be incompatible with older indexes.  (Chongchen Chen, David Smiley)
  132 
  133 * LUCENE-8956: QueryRescorer now only sorts the first topN hits instead of all
  134   initial hits. (Paul Sanwald via Adrien Grand)
  135 
  136 * LUCENE-8921: IndexSearcher.termStatistics() no longer takes a TermStates; it takes the docFreq and totalTermFreq.
  137   And don't call if docFreq <= 0.  The previous implementation survives as deprecated and final.  It's removed in 9.0.
  138   (Bruno Roustant, David Smiley, Alan Woodward)
  139 
  140 * LUCENE-8990: PointValues#estimateDocCount(visitor) estimates the number of documents that would be matched by
  141   the given IntersectVisitor. THe method is used to compute the cost() of ScorerSuppliers instead of
  142   PointValues#estimatePointCount(visitor). (Ignacio Vera, Adrien Grand)
  143 
  144 New Features
  145 
  146 * LUCENE-8936: Add SpanishMinimalStemFilter (vinod kumar via Tomoko Uchida)
  147 
  148 * LUCENE-8764 LUCENE-8945: Add "export all terms and doc freqs" feature to Luke with delimiters. (Leonardo Menezes, Amish Shah via Tomoko Uchida)
  149 
  150 * LUCENE-8747: Composite Matches from multiple subqueries now allow access to
  151   their submatches, and a new NamedMatches API allows marking of subqueries
  152   and a simple way to find which subqueries have matched on a given document
  153   (Alan Woodward, Jim Ferenczi)
  154 
  155 * LUCENE-8769: Introduce Range Query For Multiple Connected Ranges (Atri Sharma)
  156 
  157 * LUCENE-8960: Introduce LatLonDocValuesPointInPolygonQuery for LatLonDocValuesField (Ignacio Vera)
  158 
  159 * LUCENE-8753: New UniformSplitPostingsFormat (name "UniformSplit") primarily benefiting in simplicity and
  160   extensibility.  New STUniformSplitPostingsFormat (name "SharedTermsUniformSplit") that shares a single internal
  161   term dictionary across fields.  (Bruno Roustant, Juan Rodriguez, David Smiley)
  162 
  163 Improvements
  164 
  165 * LUCENE-8874: Show SPI names instead of class names in Luke Analysis tab. (Tomoko Uchida)
  166 
  167 * LUCENE-8894: Add APIs to find SPI names for Tokenizer/CharFilter/TokenFilter factory classes. (Tomoko Uchida)
  168 
  169 * LUCENE-8914: move the logic for discarding inner modes in FloatPointNearestNeighbor to the IntersectVisitor
  170   so we take advantage of the change introduced in LUCENE-7862. (Ignacio Vera)
  171 
  172 * LUCENE-8955: move the logic for discarding inner modes in LatLonPoint NearestNeighbor to the IntersectVisitor
  173   so we take advantage of the change introduced in LUCENE-7862. (Ignacio Vera)
  174 
  175 * LUCENE-8918: PhraseQuery throws exceptions at construction time if it is passed
  176   null arguments. (Alan Woodward)
  177 
  178 * LUCENE-8916: GraphTokenStreamFiniteStrings preserves all Token attributes
  179   through its finite strings TokenStreams (Alan Woodward)
  180 
  181 * LUCENE-8933: Check kuromoji user dictionary beforehand to avoid unexpected runtime exceptions. (Tomoko Uchida
  182 
  183 * LUCENE-8906: Expose Lucene50PostingsFormat.IntBlockTermState as public so that other postings formats can re-use it.
  184   (Bruno Roustant)
  185 
  186 * LUCENE-8942: Remove redundant parameters and improve visibility strictness in
  187   LRUQueryCache (Atri Sharma)
  188 
  189 * SOLR-13663: Introduce <SpanPositionRange> into XML Query Parser (Alessandro Benedetti via Mikhail Khludnev)
  190 
  191 * LUCENE-8952: Use a sort key instead of true distance in NearestNeighbor (Julie Tibshirani).
  192 
  193 * LUCENE-8620: Tessellator labels the edges of the generated triangles whether they belong to
  194   the original polygon. This information is added to the triangle encoding. (Ignacio Vera)
  195 
  196 * LUCENE-8964: Fix geojson shape parsing on string arrays in properties
  197    (Alexander Reelsen)
  198 
  199 * LUCENE-8976: Use exact distance between point and bounding rectangle in FloatPointNearestNeighbor. (Ignacio Vera)
  200 
  201 * LUCENE-8966: The Korean analyzer now splits tokens on boundaries between digits and alphabetic characters. (Jim Ferenczi)
  202 
  203 * LUCENE-8984: MoreLikeThis MLT is biased for uncommon fields (Andy Hind via Anshum Gupta)
  204 
  205 Optimizations
  206 
  207 * LUCENE-8922: DisjunctionMaxQuery more efficiently leverages impacts to skip
  208   non-competitive hits. (Adrien Grand)
  209 
  210 * LUCENE-8935: BooleanQuery with no scoring clause can now early terminate the query when
  211   the total hits is not requested. (Jim Ferenczi)
  212 
  213 * LUCENE-8941: Matches on wildcard queries will defer building their full
  214   disjunction until a MatchesIterator is pulled (Alan Woodward)
  215 
  216 * LUCENE-8755: spatial-extras quad and packed quad prefix trees now index points faster.
  217   (Chongchen Chen, David Smiley)
  218 
  219 * LUCENE-8860: add additional leaf node level optimizations in LatLonShapeBoundingBoxQuery.
  220   (Igor Motov via Ignacio Vera)
  221   
  222 * LUCENE-8968: Improve performance of WITHIN and DISJOINT queries for Shape queries by
  223   doing just one pass whenever possible. (Ignacio Vera)
  224 
  225 * LUCENE-8939: Introduce shared count based early termination across multiple slices
  226   (Atri Sharma)
  227 
  228 * LUCENE-8980: Blocktree's seekExact now short-circuits false if the term isn't in the min-max range of the segment.
  229   Large perf gain for ID/time like data when populated sequentially.  (Guoqiang Jiang)
  230 
  231 Bug Fixes
  232 
  233 * LUCENE-8755: spatial-extras quad and packed quad prefix trees could throw a
  234   NullPointerException for certain cell edge coordinates (Chongchen Chen, David Smiley)
  235 
  236 * LUCENE-9005: BooleanQuery.visit() would pull subVisitors from its parent visitor, rather
  237   than from a visitor for its own specific query.  This could cause problems when BQ was
  238   nested under another BQ. Instead, we now pull a MUST subvisitor, pass it to any MUST
  239   subclauses, and then pull SHOULD, MUST_NOT and FILTER visitors from it rather than from
  240   the parent.  (Alan Woodward)
  241 
  242 Other
  243 
  244 * LUCENE-8778 LUCENE-8911 LUCENE-8957: Define analyzer SPI names as static final fields and document the names in Javadocs.
  245   (Tomoko Uchida, Uwe Schindler)
  246 
  247 * LUCENE-8758: QuadPrefixTree: removed levelS and levelN fields which weren't used. (Amish Shah)
  248 
  249 * LUCENE-8975: Code Cleanup: Use entryset for map iteration wherever possible. (Koen De Groote)
  250 
  251 * LUCENE-8993, LUCENE-8807: Changed all repository and download references in build files
  252   to HTTPS. (Uwe Schindler)
  253 
  254 * LUCENE-8998: Fix OverviewImplTest.testIsOptimized reproducible failure. (Tomoko Uchida)
  255 
  256 * LUCENE-8999: LuceneTestCase.expectThrows now propogates assert/assumption failures up to the test
  257   w/o wrapping in a new assertion failure unless the caller has explicitly expected them (hossman)
  258 
  259 * LUCENE-8062: GlobalOrdinalsWithScoreQuery is no longer eligible for query caching. (Jim Ferenczi)
  260 
  261 
  262 ======================= Lucene 8.2.0 =======================
  263 
  264 API Changes
  265 
  266 * LUCENE-8865: IndexSearcher now uses Executor instead of ExecutorSerivce.
  267   This change is fully backwards compatible since ExecutorService directly
  268   implements Executor. (Simon Willnauer)
  269 
  270 * LUCENE-8856: Intervals queries have moved from the sandbox to the queries
  271   module. (Alan Woodward)
  272 
  273 * LUCENE-8893: Intervals.wildcard() and Intervals.prefix() methods now take
  274   BytesRef rather than String. (Alan Woodward)
  275 
  276 New Features
  277 
  278 * LUCENE-8632: New XYShape Field and Queries for indexing and searching general cartesian
  279   geometries. (Nick Knize)
  280 
  281 * LUCENE-8891: Snowball stemmer/analyzer for the Estonian language.
  282  (Gert Morten Paimla via Tomoko Uchida)
  283 
  284 * LUCENE-8815: Provide a DoubleValues implementation for retrieving the value of features without
  285   requiring a separate numeric field. Note that as feature values are stored with only 8 bits of 
  286   mantissa the values returned may have a delta from the original values indexed.
  287   (Colin Goodheart-Smithe via Adrien Grand)
  288 
  289 * LUCENE-8803: Provide a FeatureSortfield to allow sorting search hits by descending value of a
  290   feature. This is exposed via the factory method FeatureField#newFeatureSort.
  291   (Colin Goodheart-Smithe via Adrien Grand)
  292 
  293 * LUCENE-8784: The KoreanTokenizer now preserves punctuations if discardPunctuation is set
  294   to false (defaults to true).
  295   (Namgyu Kim via Jim Ferenczi)
  296 
  297 * LUCENE-8812: Add new KoreanNumberFilter that can change Hangul character to number
  298   and process decimal point. It is similar to the JapaneseNumberFilter.
  299   (Namgyu Kim)
  300 
  301 * LUCENE-8362: Add doc-value support to range fields. (Atri Sharma via Adrien Grand)
  302 
  303 * LUCENE-8766: Add monitor subproject (previously Luwak monitoring library). This
  304   allows a stream of documents to be matched against a set of registered queries
  305   in an efficient manner, for use as a monitoring or classification tool.
  306   (Alan Woodward)
  307 
  308 * LUCENE-7714: Add a numeric range query in sandbox that takes advantage of index sorting.
  309   (Julie Tibshirani via Jim Ferenczi)
  310 
  311 * LUCENE-8859: The completion suggester's postings format now have an option to
  312   load its internal FST off-heap. (Jim Ferenczi)
  313 
  314 Bug Fixes
  315 
  316 * LUCENE-8831: Fixed LatLonShapeBoundingBoxQuery .hashCode methods. (Ignacio Vera)
  317 
  318 * LUCENE-8775: Improve tessellator to handle better cases where a hole share a vertex
  319   with the polygon. (Ignacio Vera)
  320 
  321 * LUCENE-8785: Ensure new threadstates are locked before retrieving the number of active threadstates.
  322   This causes assertion errors and potentially broken field attributes in the IndexWriter when
  323   IndexWriter#deleteAll is called while actively indexing. (Simon Willnauer)
  324 
  325 * LUCENE-8804: Forbid calls to putAttribute on frozen FieldType instances.
  326   (Vamshi Vijay Nakkirtha via Adrien Grand)
  327 
  328 * LUCENE-8828: Removes the buggy 'disallow overlaps' boolean from Intervals.unordered(),
  329   and replaces it with a new Intervals.unorderedNoOverlaps() method (Alan Woodward)
  330 
  331 * LUCENE-8843: Don't ignore exceptions that are thrown when trying to open a
  332   file in IOUtils#fsync. (Jason Tedor via Adrien Grand)
  333 
  334 * LUCENE-8835: FileSwitchDirectory now respects the file extension when listing directory
  335   contents to ensure we don't expose pending deletes if both directory point to the same
  336   underlying filesystem directory. (Simon Willnauer)
  337 
  338 * LUCENE-8853: FileSwitchDirectory now applies best effort to place tmp files in the same
  339   directory as the target files. (Simon Willnauer)
  340 
  341 * LUCENE-8892: Add missing closing parentheses in MultiBoolFunction's description() (Florian Diebold, Munendra S N)
  342 
  343 Improvements
  344 
  345 * LUCENE-7840: Non-scoring BooleanQuery now removes SHOULD clauses before building the scorer supplier
  346   as opposed to eliminating them during scoring construction. (Atri Sharma via Jim Ferenczi)
  347 
  348 * LUCENE-8770: BlockMaxConjunctionScorer now leverages two-phase iterators in order to avoid
  349   executing the second phase when scorers don't intersect. (Adrien Grand, Jim Ferenczi)
  350 
  351 * LUCENE-8781: FST lookup performance has been improved in many cases by
  352   encoding Arcs using full-sized arrays with gaps. The new encoding is
  353   enabled for postings in the default codec and for suggesters. (Mike Sokolov)
  354 
  355 * LUCENE-8818: Fix smokeTestRelease.py encoding bug (janhoy)
  356 
  357 * LUCENE-8845: Allow Intervals.prefix() and Intervals.wildcard() to specify
  358   their maximum allowed expansions (Alan Woodward)
  359 
  360 * LUCENE-8875: Introduce a Collector optimized for use cases when large
  361   number of hits are requested (Atri Sharma)
  362 
  363 * LUCENE-8848 LUCENE-7757 LUCENE-8492: The UnifiedHighlighter now detects that parts of the query are not understood by
  364   it, and thus it should not make optimizations that result in no highlights or slow highlighting.  This generally works
  365   best for WEIGHT_MATCHES mode.  Consequently queries produced by ComplexPhraseQueryParser and the surround QueryParser
  366   will now highlight correctly. (David Smiley)
  367 
  368 * LUCENE-8793: Luke enhanced UI for CustomAnalyzer: show detailed analysis steps. (Jun Ohtani via Tomoko Uchida)
  369 
  370 * LUCENE-8855: Add Accountable to some Query implementations (ab, Adrien Grand)
  371 
  372 Optimizations
  373 
  374 * LUCENE-8796: Use exponential search instead of binary search in
  375   IntArrayDocIdSet#advance method (Luca Cavanna via Adrien Grand)
  376 
  377 * LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor.
  378   Now caller threads execute at least one search on an index even if there is
  379   an executor provided to minimize thread context switching. (Simon Willnauer)
  380 
  381 * LUCENE-8868: New storing strategy for BKD tree leaves with low cardinality.
  382   It stores the distinct values once with the cardinality value reducing the
  383   storage cost. (Ignacio Vera)
  384 
  385 * LUCENE-8885: Optimise BKD reader by exploiting cardinality information stored
  386   on leaves. (Ignacio Vera)
  387 
  388 * LUCENE-8896: Override default implementation of IntersectVisitor#visit(DocIDSetBuilder, byte[])
  389   for several queries. (Ignacio Vera)
  390 
  391 * LUCENE-8901: Load frequencies lazily only when needed in BlockDocsEnum and
  392   BlockImpactsEverythingEnum (Mayya Sharipova).
  393 
  394 * LUCENE-8888: Optimize distribution of points with data dimensions in
  395   BKD tree leaves. (Ignacio Vera)
  396 
  397 * LUCENE-8311: Phrase queries now leverage impacts. (Adrien Grand)
  398 
  399 Test Framework
  400 
  401 * LUCENE-8825: CheckHits now display the shard index in case of mismatch
  402   between top hits. (Atri Sharma via Adrien Grand)
  403 
  404 Other
  405 
  406 * LUCENE-8847: Code Cleanup: Remove StringBuilder.append with concatenated
  407   strings. (Koen De Groote via Uwe Schindler)
  408 
  409 * LUCENE-8861: Script to find open Github PRs that needs attention (janhoy)
  410 
  411 * LUCENE-8852: ReleaseWizard tool for release managers (janhoy)
  412 
  413 * LUCENE-8838: Remove support for Steiner points on Tessellator. (Ignacio Vera)
  414 
  415 * LUCENE-8879: Improve BKDRadixSelector tests. (Ignacio Vera)
  416 
  417 * LUCENE-8886: Fix TestMutablePointsReaderUtils tests. (Ignacio Vera)
  418 
  419 ======================= Lucene 8.1.1 =======================
  420 (No Changes)
  421 
  422 ======================= Lucene 8.1.0 =======================
  423 
  424 API Changes
  425 
  426 * LUCENE-3041: A query introspection API has been added.  Queries should
  427   implement a visit() method, taking a QueryVisitor, and either pass the
  428   visitor down to any child queries, or call a visitX() or consumeX() method
  429   on it.  All locations in the code that called Weight.extractTerms() 
  430   have been changed to use this API, and the extractTerms() method has 
  431   been deprecated. (Alan Woodward, Simon Willnauer, David Smiley, Luca
  432   Cavanna)
  433 
  434 * LUCENE-8735: Directory.getPendingDeletions is now abstract to ensure
  435   subclasses override it. FilterDirectory now delegates the call, ensuring
  436   correct default behaviour for subclasses. (Henning Andersen)
  437 
  438 New Features
  439 
  440 * LUCENE-2562: The well-known graphical user interface for inspecting Lucene
  441   indexes "Luke" was added as a Lucene module. It can be started from the
  442   binary distribution by calling the shell scripts in the module folder
  443   or from the source checkout by using `ant -f lucene/luke/build.xml run`.
  444   Luke provides a Swing-based user interface and can be used to open
  445   Lucene or Solr (or Elasticsearch) indexes, inspect documents, check index
  446   commits and segments, or test (custom) analyzers. It also has maintenance
  447   functions to check index structures and force merge indexes for archival.
  448   Luke was originally developed by Andrzej Bialecki, later maintained by
  449   Dmitry Kan and finally rewritten by Tomoko Uchida to use the ASF licensing
  450   compatible Swing framework (as shipped with JDKs).
  451   (Tomoko Uchida, Uwe Schindler)
  452 
  453 Bug fixes
  454 
  455 * LUCENE-8736: LatLonShapePolygonQuery returns incorrect WITHIN results
  456   with shared boundaries. Point in Polygon now correctly includes boundary
  457   points. Box and Polygon relations with triangles have also been improved to
  458   correctly include boundary points. (Nick Knize)
  459 
  460 * LUCENE-8712: Polygon2D does not detect crossings through segment edges.
  461   (Ignacio Vera)
  462 
  463 * LUCENE-8720: NameIntCacheLRU (in the facets module) had an int
  464   overflow bug that disabled cleaning of the cache (Russell A Brown)
  465 
  466 * LUCENE-8726: ValueSource.asDoubleValuesSource() could leak a reference to
  467   IndexSearcher (Alan Woodward, Yury Pakhomov)
  468 
  469 * LUCENE-8719: FixedShingleFilter can miss shingles at the end of a token stream if
  470   there are multiple paths with different lengths. (Alan Woodward)
  471 
  472 * LUCENE-8688: TieredMergePolicy#findForcedMerges now tries to create the
  473   cheapest merges that allow the index to go down to `maxSegmentCount` segments
  474   or less. (Armin Braun via Adrien Grand)
  475 
  476 * LUCENE-8477: Interval disjunctions could miss valid hits if some of the
  477   clauses of the disjunction are minimized away.  We now rewrite intervals
  478   if a source contains a disjunction and the internal gaps matter for
  479   matching.  This behaviour can be disabled if users are more interested
  480   in speed rather than accuracy of matching. (Alan Woodward, Jim Ferenczi)
  481 
  482 * LUCENE-8741: ValueSource.fromDoubleValuesSource() was casting to
  483   Scorer instead of Scorable, leading to ClassCastExceptions (Markus Jelsma,
  484   Alan Woodward)
  485 
  486 * LUCENE-8754: Fix ConcurrentModificationException in SegmentInfo if
  487   attributes are accessed in MergePolicy while the merge is running (Simon Willnauer)
  488 
  489 * LUCENE-8765: Fixed validation of the number of added points in KD trees.
  490   (Zhao Yang via Adrien Grand)
  491 
  492 Improvements
  493 
  494 * LUCENE-8673: Use radix partitioning when merging dimensional points instead
  495   of sorting all dimensions before hand. (Ignacio Vera, Adrien Grand)
  496 
  497 * LUCENE-8687: Optimise radix partitioning for points on heap. (Ignacio Vera)
  498 
  499 * LUCENE-8699: Change HeapPointWriter to use a single byte array instead to a list
  500   of byte arrays. In addition a new interface PointValue is added to abstract out
  501   the different formats between offline and on-heap writers. (Ignacio Vera)
  502 
  503 * LUCENE-8703: Build point writers in the BKD tree only when they are needed.
  504   (Ignacio Vera)
  505 
  506 * LUCENE-8652: SynonymQuery can now deboost the document frequency of each term when
  507   blending the score of the synonym. (Jim Ferenczi)
  508 
  509 * LUCENE-8631: The Korean's user dictionary now picks the longest-matching word and discards
  510   the other matches. (Yeongsu Kim via Jim Ferenczi)
  511 
  512 * LUCENE-8732: ConstantScoreQuery can now early terminate the query if the minimum score is
  513   greater than the constant score and total hits are not requested. (Jim Ferenczi)
  514 
  515 * LUCENE-8750: Implements setMissingValue() on sort fields produced from 
  516   DoubleValuesSource and LongValuesSource (Mike Sokolov via Alan Woodward)
  517   
  518 * LUCENE-8701: ToParentBlockJoinQuery now creates a child scorer that disallows skipping over
  519   non-competitive documents if the score of a parent depends on the score of multiple
  520   children (avg, max, min). Additionally the score mode `none` that assigns a constant score to
  521   each parent can early terminate top scores's collection. (Jim Ferenczi)
  522 
  523 * LUCENE-8751: Weight#matches now use the ScorerSupplier to build scorers with a lead cost of 1
  524   (single document). (Jim Ferenczi)
  525 
  526 * LUCENE-8752: Japanese new era name '令和' (Reiwa) is added to the dictionary used in
  527   JapaneseTokenizer so that the analyzer handles the era name correctly.
  528   Reiwa is set to replace the Heisei Era on May 1, 2019. (Tomoko Uchida)
  529 
  530 * LUCENE-8671: Introduced reader attributes allows a per IndexReader configuration
  531   of codec internals. This enables a per reader configuration if FSTs are on- or off-heap on a
  532   per field basis (Simon Willnauer)
  533 
  534 * LUCENE-8787: spatial-extras DateRangePrefixTree used to only parse ISO-8601 timestamps with 0 or 3
  535   digits of milliseconds precision but now parses other lengths (although > 3 not used).
  536   (Thomas Lemmé via David Smiley)
  537 
  538 Changes in Runtime Behavior
  539 
  540 * LUCENE-8671: Load FST off-heap also for ID-like fields if reader is not opened
  541   from an IndexWriter. (Simon Willnauer)
  542 
  543 * LUCENE-8730: WordDelimiterGraphFilter always emits its original token first.  This
  544   brings its behaviour into line with the deprecated WordDelimiterFilter, so that
  545   the only difference in output between the two is in the position length
  546   attribute.  (Alan Woodward, Jim Ferenczi)
  547 
  548 * LUCENE-7386: Disjunctions nested in disjunctions are now flattened. This might
  549   trigger changes in the produced scores due to changes to the order in which
  550   scores of sub clauses are summed up. (Adrien Grand)
  551 
  552 * LUCENE-8756: MoreLikeThisQuery now respects custom term frequencies
  553   (TermFrequencyAttribute) at search time (Olli Kuonanoja)
  554 
  555 Other
  556 
  557 * LUCENE-8680: Refactor EdgeTree#relateTriangle method. (Ignacio Vera)
  558 
  559 * LUCENE-8685: Refactor LatLonShape tests. (Ignacio Vera)
  560 
  561 * LUCENE-8713: Add Line2D tests. (Ignacio Vera)
  562 
  563 * LUCENE-8729: Workaround: Disable accessibility doclints (Java 13+),
  564   so compilation with recent JDK succeeds.  (Uwe Schindler)
  565 
  566 * LUCENE-8725: Make TermsQuery.SeekingTermSetTermsEnum a top level class and public (noble)
  567 
  568 ======================= Lucene 8.0.0 =======================
  569 
  570 API Changes
  571 
  572 * LUCENE-8662: TermsEnum.seekExact(BytesRef) to abstract and delegate seekExact(BytesRef)
  573   in FilterLeafReader.FilterTermsEnum. (Jeffery Yuan via Tomás Fernández Löbbe, Simon Willnauer)
  574 
  575 * LUCENE-8469: Deprecated StringHelper.compare has been removed. (Dawid Weiss)
  576 
  577 * LUCENE-8039: Introduce a "delta distance" method set to GeoDistance.  This
  578   allows distance calculations, especially for paths, to take into account an
  579   "excursion" to include the specified point.
  580 
  581 * LUCENE-8007: Index statistics Terms.getSumDocFreq(), Terms.getDocCount() are
  582   now required to be stored by codecs. Additionally, TermsEnum.totalTermFreq()
  583   and Terms.getSumTotalTermFreq() are now required: if frequencies are not
  584   stored they are equal to TermsEnum.docFreq() and Terms.getSumDocFreq(),
  585   respectively, because all freq() values equal 1. (Adrien Grand, Robert Muir)
  586 
  587 * LUCENE-8038: Deprecated PayloadScoreQuery constructors have been removed (Alan
  588   Woodward)
  589 
  590 * LUCENE-8014: Similarity.computeSlopFactor() and
  591   Similarity.computePayloadFactor() have been removed (Alan Woodward)
  592 
  593 * LUCENE-7996: Queries are now required to produce positive scores.
  594   (Adrien Grand)
  595 
  596 * LUCENE-8099: CustomScoreQuery, BoostedQuery and BoostingQuery have been
  597   removed (Alan Woodward)
  598 
  599 * LUCENE-8012: Explanation now takes Number rather than float (Alan Woodward,
  600   Robert Muir)
  601 
  602 * LUCENE-8116: SimScorer now only takes a frequency and a norm as per-document
  603   scoring factors. (Adrien Grand)
  604 
  605 * LUCENE-8113: TermContext has been renamed to TermStates, and can now be
  606   constructed lazily if term statistics are not required (Alan Woodward)
  607 
  608 * LUCENE-8242: Deprecated method IndexSearcher#createNormalizedWeight() has
  609   been removed (Alan Woodward)
  610 
  611 * LUCENE-8267: Memory codecs removed from the codebase (MemoryPostings,
  612   MemoryDocValues). (Dawid Weiss)
  613 
  614 * LUCENE-8144: Moved QueryCachingPolicy.ALWAYS_CACHE to the test framework.
  615   (Nhat Nguyen via Adrien Grand)
  616 
  617 * LUCENE-8356: StandardFilter and StandardFilterFactory have been removed
  618   (Alan Woodward)
  619 
  620 * LUCENE-8373: StandardAnalyzer.ENGLISH_STOP_WORD_SET has been removed
  621   (Alan Woodward)
  622 
  623 * LUCENE-8388: Unused PostingsEnum#attributes() method has been removed 
  624   (Alan Woodward)
  625 
  626 * LUCENE-8405: TopDocs.maxScore is removed. IndexSearcher and TopFieldCollector
  627   no longer have an option to compute the maximum score when sorting by field.
  628   (Adrien Grand)
  629 
  630 * LUCENE-8411: TopFieldCollector no longer takes a fillFields option, it now
  631   always fills fields. (Adrien Grand)
  632 
  633 * LUCENE-8412: TopFieldCollector no longer takes a trackDocScores option. Scores
  634   need to be set on top hits via TopFieldCollector#populateScores instead.
  635   (Adrien Grand)
  636 
  637 * LUCENE-6228: A new Scorable abstract class has been added, containing only those
  638   methods from Scorer that should be called from Collectors.  LeafCollector.setScorer()
  639   now takes a Scorable rather than a Scorer. (Alan Woodward, Adrien Grand)
  640 
  641 * LUCENE-8475: Deprecated constants have been removed from RamUsageEstimator.
  642   (Dimitrios Athanasiou)
  643 
  644 * LUCENE-8483: Scorers may no longer take null as a Weight (Alan Woodward)
  645 
  646 * LUCENE-8352: TokenStreamComponents is now final, and can take a Consumer<Reader>
  647   in its constructor (Mark Harwood, Alan Woodward, Adrien Grand)
  648 
  649 * LUCENE-8498: LowerCaseTokenizer has been removed, and CharTokenizer no longer
  650   takes a normalizer function. (Alan Woodward)
  651 
  652 * LUCENE-7875: Moved MultiFields static methods out of the class.  getLiveDocs is now
  653   in MultiBits which is now public.  getMergedFieldInfos and getIndexedFields are now in
  654   FieldInfos.  getTerms is now in MultiTerms.  getTermPositionsEnum and getTermDocsEnum
  655   were collapsed and renamed to just getTermPostingsEnum and moved to MultiTerms.
  656   (David Smiley)
  657 
  658 * LUCENE-8513: MultiFields.getFields is now removed.  Please avoid this class,
  659   and Fields in general, when possible. (David Smiley)
  660 
  661 * LUCENE-8497: MultiTermAwareComponent has been removed, and in its place
  662   TokenFilterFactory and CharFilterFactory now expose type-safe normalize()
  663   methods.  This decouples normalization from tokenization entirely.
  664   (Mayya Sharipova, Alan Woodward)
  665 
  666 * LUCENE-8597: IntervalIterator now exposes a gaps() method that reports the
  667   number of gaps between its component sub-intervals.  This can be used in a 
  668   new filter available via Intervals.maxgaps().  (Alan Woodward)
  669 
  670 * LUCENE-8609: Remove IndexWriter#numDocs() and IndexWriter#maxDoc() in favor
  671   of IndexWriter#getDocStats(). (Simon Willnauer)
  672 
  673 * LUCENE-8292: Make TermsEnum fully abstract. (Simon Willnauer)
  674 
  675 Changes in Runtime Behavior
  676 
  677 * LUCENE-8333: Switch MoreLikeThis.setMaxDocFreqPct to use maxDoc instead of
  678   numDocs. (Robert Muir, Dawid Weiss).
  679 
  680 * LUCENE-7837: Indices that were created before the previous major version
  681   will now fail to open even if they have been merged with the previous major
  682   version. (Adrien Grand)
  683 
  684 * LUCENE-8020: Similarities are no longer passed terms that don't exist by
  685   queries such as SpanOrQuery, so scoring formulas no longer require
  686   divide-by-zero hacks.  IndexSearcher.termStatistics/collectionStatistics return null
  687   instead of returning bogus values for a non-existent term or field. (Robert Muir)
  688 
  689 * LUCENE-7996: FunctionQuery and FunctionScoreQuery now return a score of 0
  690   when the function produces a negative value. (Adrien Grand)
  691 
  692 * LUCENE-8116: Similarities now score fields that omit norms as if the norm was
  693   1. This might change score values on fields that omit norms. (Adrien Grand)
  694 
  695 * LUCENE-8134: Index options are no longer automatically downgraded.
  696   (Adrien Grand)
  697 
  698 * LUCENE-8031: Length normalization correctly reflects omission of term frequencies.
  699   (Robert Muir, Adrien Grand)
  700 
  701 * LUCENE-7444: StandardAnalyzer no longer defaults to removing English stopwords
  702   (Alan Woodward)
  703 
  704 * LUCENE-8060: IndexSearcher's search and searchAfter methods now only compute
  705   total hit counts accurately up to 1,000 in order to enable top-hits
  706   optimizations such as block-max WAND (LUCENE-8135). (Adrien Grand)
  707 
  708 * LUCENE-8505: IndexWriter#addIndices will now fail if the target index is sorted but
  709   the candidate is not. (Jim Ferenczi)
  710 
  711 * LUCENE-8535: Highlighter and FVH doesn't support ToParent and ToChildBlockJoinQuery out of the
  712   box anymore. In order to highlight on Block-Join Queries a custom WeightedSpanTermExtractor / FieldQuery
  713   should be used. (Simon Willnauer, Jim Ferenczi, Julie Tibshirani)
  714 
  715 * LUCENE-8563: BM25 scores don't include the (k1+1) factor in their numerator
  716   anymore. This doesn't affect ordering as this is a constant factor which is
  717   the same for every document. (Luca Cavanna via Adrien Grand)
  718 
  719 * LUCENE-8509: WordDelimiterGraphFilter will no longer set the offsets of internal
  720   tokens by default, preventing a number of bugs when the filter is chained with
  721   tokenfilters that change the length of their tokens (Alan Woodward)
  722 
  723 * LUCENE-8633: IntervalQuery scores do not use term weighting any more, the score
  724   is instead calculated as a function of the sloppy frequency of the matching
  725   intervals. (Alan Woodward, Jim Ferenczi)
  726 
  727 * LUCENE-8635: FSTs can now remain off-heap, accessed via
  728   IndexInput, and the default codec's term dictionary
  729   (BlockTreeTermsReader) will now leave the FST for the terms index
  730   off-heap for non-primary-key fields using MMapDirectory, reducing
  731   heap usage for such fields. (Ankit Jain)
  732 
  733 New Features
  734 
  735 * LUCENE-8340: LongPoint#newDistanceFeatureQuery may be used to boost scores based on
  736   how close a value of a long field is from an configurable origin. This is
  737   typically useful to boost by recency. (Adrien Grand)
  738 
  739 * LUCENE-8482: LatLonPoint#newDistanceFeatureQuery may be used to boost scores
  740   based on the haversine distance of a LatLonPoint field to a provided point. This is
  741   typically useful to boost by distance. (Ignacio Vera)
  742 
  743 * LUCENE-8216: Added a new BM25FQuery in sandbox to blend statistics across several fields
  744   using the BM25F formula. (Adrien Grand, Jim Ferenczi)
  745 
  746 * LUCENE-8564: GraphTokenFilter is an abstract class useful for token filters that need
  747   to read-ahead in the token stream and take into account graph structures.  This
  748   also changes FixedShingleFilter to extend GraphTokenFilter (Alan Woodward)
  749 
  750 * LUCENE-8612: Intervals.extend() treats an interval as if it covered a wider
  751   span than it actually does, allowing users to force minimum gaps between
  752   intervals in a phrase. (Alan Woodward)
  753 
  754 * LUCENE-8629: New interval functions: Intervals.before(), Intervals.after(),
  755   Intervals.within() and Intervals.overlapping(). (Alan Woodward)
  756 
  757 * LUCENE-8622: Adds a minimum-should-match interval function that produces intervals
  758   spanning a subset of a set of sources. (Alan Woodward)
  759 
  760 * LUCENE-8645: Intervals.fixField() allows you to report intervals from one field
  761   as if they came from another. (Alan Woodward)
  762 
  763 * LUCENE-8646: New interval functions: Intervals.prefix() and Intervals.wildcard()
  764   (Alan Woodward)
  765 
  766 * LUCENE-8655: Add a getter in FunctionScoreQuery class in order to access to the 
  767   underlying DoubleValuesSource. (Gérald Quaire via Alan Woodward)
  768 
  769 * LUCENE-8697: GraphTokenStreamFiniteStrings correctly handles side paths 
  770   containing gaps (Alan Woodward)
  771 
  772 * LUCENE-8702: Simplify intervals returned from vararg Intervals factory methods
  773   (Alan Woodward)
  774 
  775 Improvements
  776 
  777 * LUCENE-7997: Add BaseSimilarityTestCase to sanity check similarities.
  778   SimilarityBase switches to 64-bit doubles internally to help avoid common numeric issues.
  779   Add missing range checks for similarity parameters.
  780   Improve BM25 and ClassicSimilarity's explanations. (Robert Muir)
  781 
  782 * LUCENE-8011: Improved similarity explanations.
  783   (Mayya Sharipova via Adrien Grand)
  784 
  785 * LUCENE-4198: Codecs now have the ability to index score impacts.
  786   (Adrien Grand)
  787 
  788 * LUCENE-8135: Boolean queries now implement the block-max WAND algorithm in
  789   order to speed up selection of top scored documents. (Adrien Grand)
  790 
  791 * LUCENE-8279: CheckIndex now cross-checks terms with norms. (Adrien Grand)
  792 
  793 * LUCENE-8660: TopDocsCollectors now return an accurate count (instead of a lower bound)
  794   if the total hit count is equal to the provided threshold. (Adrien Grand, Jim Ferenczi)
  795 
  796 Optimizations
  797 
  798 * LUCENE-8040: Optimize IndexSearcher.collectionStatistics, avoiding MultiFields/MultiTerms
  799   (David Smiley, Robert Muir)
  800 
  801 * LUCENE-4100: Disjunctions now support faster collection of top hits when the
  802   total hit count is not required. (Stefan Pohl, Adrien Grand, Robert Muir)
  803 
  804 * LUCENE-7993: Phrase queries are now faster if total hit counts are not
  805   required. (Adrien Grand)
  806 
  807 * LUCENE-8109: Boolean queries propagate information about the minimum
  808   competitive score in order to make collection faster if there are disjunctions
  809   or phrase queries as sub queries, which know how to leverage this information
  810   to run faster. (Adrien Grand)
  811 
  812 * LUCENE-8439: Disjunction max queries can skip blocks to select the top documents
  813   if the total hit count is not required. (Jim Ferenczi, Adrien Grand)
  814 
  815 * LUCENE-8204: Boolean queries with a mix of required and optional clauses are
  816   now faster if the total hit count is not required. (Jim Ferenczi, Adrien Grand)
  817 
  818 * LUCENE-8448: Boolean queries now propagates the mininum score to their sub-scorers.
  819   (Jim Ferenczi, Adrien Grand)
  820 
  821 * LUCENE-8511: MultiFields.getIndexedFields is now optimized; does not call getMergedFieldInfos
  822   (David Smiley)
  823 
  824 * LUCENE-8507: TopFieldCollector can now update the minimum competitive score if the primary sort
  825   is by relevancy and the total hit count is not required. (Jim Ferenczi)
  826 
  827 * LUCENE-8464: ConstantScoreScorer now implements setMinCompetitveScore in order
  828   to early terminate the iterator if the minimum score is greater than the constant
  829   score. (Christophe Bismuth via Jim Ferenczi)
  830 
  831 * LUCENE-8607: MatchAllDocsQuery can shortcut when total hit count is not
  832   required (Alan Woodward, Adrien Grand)
  833 
  834 * LUCENE-8585: Index-time jump-tables for DocValues, for O(1) advance when retrieving doc values.
  835   (Toke Eskildsen, Adrien Grand)
  836 
  837 ======================= Lucene 7.7.2 =======================
  838 
  839 Bug fixes
  840 
  841 * LUCENE-8726: ValueSource.asDoubleValuesSource() could leak a reference to
  842   IndexSearcher (Alan Woodward, Yury Pakhomov)
  843 
  844 * LUCENE-8735: FilterDirectory.getPendingDeletions now forwards to the delegate
  845   even the method is not abstract in the super class. This prevents issues
  846   where our best effort in carrying on generations in the IndexWriter since pending
  847   deletions are swallowed by the FilterDirectory. (Henning Andersen, Simon Willnauer)
  848 
  849 * LUCENE-8688: TieredMergePolicy#findForcedMerges now tries to create the
  850   cheapest merges that allow the index to go down to `maxSegmentCount` segments
  851   or less. (Armin Braun via Adrien Grand)
  852 
  853 * LUCENE-8785: Ensure new threadstates are locked before retrieving the number of active threadstates.
  854   This causes assertion errors and potentially broken field attributes in the IndexWriter when
  855   IndexWriter#deleteAll is called while actively indexing. (Simon Willnauer)
  856 
  857 * LUCENE-8720: NameIntCacheLRU (in the facets module) had an int
  858   overflow bug that disabled cleaning of the cache (Russell A Brown)
  859 
  860 * LUCENE-8809: Refresh and rollback concurrently can leave segment states unclosed (Nhat Nguyen)
  861 
  862 ======================= Lucene 7.7.1 =======================
  863 (No Changes)
  864 
  865 ======================= Lucene 7.7.0 =======================
  866 
  867 Changes in Runtime Behavior
  868 
  869 * LUCENE-8527: StandardTokenizer and UAX29URLEmailTokenizer now support Unicode 9.0,
  870   and provide Unicode UTS#51 v11.0 Emoji tokenization with the "<EMOJI>" token type. 
  871 
  872 Build
  873 
  874 * LUCENE-8611: Update randomizedtesting to 2.7.2, JUnit to 4.12, add hamcrest-core 
  875   dependency. (Dawid Weiss)
  876 
  877 * LUCENE-8537: ant test command fails under lucene/tools (Peter Somogyi)
  878 
  879 Bug fixes:
  880 
  881 * LUCENE-8669: Fix LatLonShape WITHIN queries that fail with Multiple search Polygons
  882   that share the dateline. (Nick Knize)
  883 
  884 * LUCENE-8603: Fix the inversion of right ids for additional nouns in the Korean user dictionary.
  885   (Yoo Jeongin via Jim Ferenczi)
  886 
  887 * LUCENE-8624: int overflow in ByteBuffersDataOutput.size(). (Mulugeta Mammo, 
  888   Dawid Weiss)
  889 
  890 * LUCENE-8625: int overflow in ByteBuffersDataInput.sliceBufferList. (Mulugeta Mammo,
  891   Dawid Weiss)
  892 
  893 * LUCENE-8639: Newly created threadstates while flushing / refreshing can cause duplicated
  894   sequence IDs on IndexWriter. (Simon Willnauer)
  895 
  896 * LUCENE-8649: LatLonShape's within and disjoint queries can return false positives with
  897   indexed multi-shapes. (Ignacio Vera)
  898 
  899 * LUCENE-8654: Polygon2D#relateTriangle returns the wrong answer if polygon is inside
  900   the triangle. (Ignacio Vera)
  901 
  902 * LUCENE-8650: ConcatenatingTokenStream did not correctly clear its state in reset(), and
  903   was not propagating final position increments from its child streams correctly.
  904   (Dan Meehl, Alan Woodward)
  905 
  906 * LUCENE-8676: The Korean tokenizer does not update the last position if the backtrace is caused
  907   by a big buffer (1024 chars). (Jim Ferenczi)
  908 
  909 New Features
  910 
  911 * LUCENE-8026: ExitableDirectoryReader may now time out queries that run on
  912   points such as range queries or geo queries.
  913   (Christophe Bismuth via Adrien Grand)
  914 
  915 * LUCENE-8508: IndexWriter can now set the created version via
  916   IndexWriterConfig#setIndexCreatedVersionMajor. This is an expert feature.
  917   (Adrien Grand)
  918 
  919 * LUCENE-8601: Attributes set in the IndexableFieldType for each field during indexing will
  920   now be recorded into the corresponding FieldInfo's attributes, accessible at search
  921   time (Murali Krishna P)
  922 
  923 Improvements
  924 
  925 * LUCENE-8463: TopFieldCollector can now early-terminates queries when sorting by SortField.DOC.
  926   (Christophe Bismuth via Jim Ferenczi)
  927 
  928 * LUCENE-8562: Speed up merging segments of points with data dimensions by only sorting on the indexed
  929   dimensions. (Ignacio Vera)
  930 
  931 * LUCENE-8529: TopSuggestDocsCollector will now use the completion key to tiebreak completion
  932   suggestion with identical scores. (Jim Ferenczi)
  933 
  934 * LUCENE-8575: SegmentInfos#toString now includes attributes and diagnostics.
  935   (Namgyu Kim via Adrien Grand)
  936 
  937 * LUCENE-8548: The KoreanTokenizer no longer splits unknown words on combining diacritics and
  938   detects script boundaries more accurately with Character#UnicodeScript#of.
  939   (Christophe Bismuth, Jim Ferenczi)
  940 
  941 * LUCENE-8581: Change LatLonShape encoding to use 4 bytes Per Dimension.
  942   (Ignacio Vera, Nick Knize, Adrien Grand)
  943   
  944 * LUCENE-8527: Upgrade JFlex dependency to 1.7.0; in StandardTokenizer and UAX29URLEmailTokenizer,
  945   increase supported Unicode version from 6.3 to 9.0, and support Unicode UTS#51 v11.0 Emoji tokenization.
  946 
  947 * LUCENE-8640: Date Range format validation (Lucky Sharma, David Smiley via Mikhail Khludnev)
  948 
  949 Optimizations
  950 
  951 * LUCENE-8552: FieldInfos.getMergedFieldInfos no longer does any merging if there is <= 1 segment.
  952   (Christophe Bismuth via David Smiley)
  953 
  954 * LUCENE-8590: BufferedUpdates now uses an optimized storage for buffering docvalues updates that
  955   can safe up to 80% of the heap used compared to the previous implementation and uses non-object
  956   based datastructures. (Simon Willnauer, Mike McCandless, Shai Erera, Adrien Grand)
  957 
  958 * LUCENE-8598: Moved to the default accepted overhead ratio for packet ints in DocValuesFieldUpdats
  959   yields an up-to 4x performance improvement when applying doc values updates. (Simon Willnauer, Adrien Grand)
  960 
  961 * LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates. 
  962   (Simon Willnauer, Adrien Grand)
  963 
  964 * LUCENE-8600: Doc-value updates get applied faster by sorting with quicksort,
  965   rather than an in-place mergesort, which needs to perform fewer swaps.
  966   (Adrien Grand)
  967 
  968 * LUCENE-8623: Decrease I/O pressure when merging high dimensional points. (Ignacio Vera)
  969 
  970 Test Framework
  971 
  972 * LUCENE-8604: TestRuleLimitSysouts now has an optional "hard limit" of bytes that can be written
  973   to stderr and stdout (anything beyond the hard limit is ignored). The default hard limit is 2 GB of 
  974   logs per test class. (Dawid Weiss)
  975 
  976 Other
  977 
  978 * LUCENE-8573: BKDWriter now uses FutureArrays#mismatch to compute shared prefixes.
  979   (Christoph Büscher via Adrien Grand)
  980 
  981 * LUCENE-8605: Separate bounding box spatial logic from query logic on LatLonShapeBoundingBoxQuery.
  982   (Ignacio Vera)
  983 
  984 * LUCENE-8609: Deprecated IndexWriter#numDocs() and IndexWriter#maxDoc() in favor of IndexWriter#getDocStats()
  985   that allows to get consistent numDocs and maxDoc stats that are not subject to concurrent changes.
  986   (Simon Willnauer, Nhat Nguyen)
  987 
  988 ======================= Lucene 7.6.0 =======================
  989 
  990 Build
  991 
  992 * LUCENE-8504: Upgrade forbiddenapis to version 2.6.  (Uwe Schindler)
  993 
  994 * LUCENE-8493: Stop publishing insecure .sha1 files with releases (janhoy)
  995 
  996 Bug fixes
  997 
  998 * LUCENE-8479: QueryBuilder#analyzeGraphPhrase now throws TooManyClause exception
  999   if the number of expanded path reaches the BooleanQuery#maxClause limit. (Jim Ferenczi)
 1000 
 1001 * LUCENE-8522: throw InvalidShapeException when constructing a polygon and
 1002   all points are coplanar. (Ignacio Vera)
 1003 
 1004 * LUCENE-8531: QueryBuilder#analyzeGraphPhrase now creates one phrase query per finite strings
 1005   in the graph if the slop is greater than 0. Span queries cannot be used in this case because
 1006   they don't handle slop the same way than phrase queries. (Steve Rowe, Uwe Schindler, Jim Ferenczi)
 1007 
 1008 * LUCENE-8524: Add the Hangul Letter Araea (interpunct) as a separator in Nori's tokenizer.
 1009   This change also removes empty terms and trim surface form in Nori's Korean dictionary. (Trey Jones, Jim Ferenczi)
 1010 
 1011 * LUCENE-8550: Fix filtering of coplanar points when creating linked list on
 1012   polygon tesselator. (Ignacio Vera)
 1013 
 1014 * LUCENE-8549: Polygon tessellator throws an error if some parts of the shape
 1015    could not be processed. (Ignacio Vera)
 1016 
 1017 * LUCENE-8540: Better handling of min/max values for Geo3d encoding. (Ignacio Vera)
 1018 
 1019 * LUCENE-8534: Fix incorrect computation for triangles intersecting polygon edges in
 1020   shape tessellation. (Ignacio Vera)
 1021 
 1022 * LUCENE-8559: Fix bug where polygon edges were skipped when checking for intersections.
 1023   (Ignacio Vera)
 1024 
 1025 * LUCENE-8556: Use latitude and longitude instead of encoding values to check if triangle is ear
 1026   when using morton optimisation. (Ignacio Vera)
 1027 
 1028 * LUCENE-8586: Intervals.or() could get stuck in an infinite loop on certain indexes
 1029   (Alan Woodward)
 1030 
 1031 * LUCENE-8595: Fix interleaved DV update and reset. Interleaved update and reset value
 1032   to the same doc in the same updates package looses an update if the reset comes before
 1033   the update as well as loosing the reset if the update comes frist. (Simon Willnauer, Adrien Grand)
 1034 
 1035 * LUCENE-8592: Fix index sorting corruption due to numeric overflow. The merge of sorted segments
 1036   can produce an invalid sort if the sort field is an Integer/Long that uses reverse order and contains
 1037   values equal to Integer/Long#MIN_VALUE. These values are always sorted first during a merge
 1038   (instead of last because of the reverse order) due to this bug. Indices affected by the bug can be
 1039   detected by running the CheckIndex command on a distribution that contains the fix (7.6+).
 1040   (Jim Ferenczi, Adrien Grand, Mike McCandless, Simon Willnauer)
 1041 
 1042 New Features
 1043 
 1044 * LUCENE-8496: Selective indexing - modify BKDReader/BKDWriter to allow users
 1045   to select a fewer number of dimensions to be used for creating the index than
 1046   the total number of dimensions used for field encoding. i.e., dimensions 0 to N
 1047   may be used to determine how to split the inner nodes, and dimensions N+1 to D
 1048   are ignored and stored as data dimensions at the leaves. (Nick Knize)
 1049 
 1050 * LUCENE-8538: Add a Simple WKT Shape Parser for creating Lucene Geometries (Polygon, Line,
 1051   Rectangle) from WKT format. (Nick Knize)
 1052 
 1053 * LUCENE-8462: Adds an Arabic snowball stemmer based on
 1054   https://github.com/snowballstem/snowball/blob/master/algorithms/arabic.sbl 
 1055   (Ryadh Dahimene via Jim Ferenczi)
 1056 
 1057 * LUCENE-8554: Add new LatLonShapeLineQuery that queries indexed LatLonShape fields
 1058   by arbitrary lines. (Nick Knize)
 1059 
 1060 * LUCENE-8555: Add dateline crossing support to LatLonShapeBoundingBoxQuery. (Nick Knize)
 1061 
 1062 Improvements
 1063 
 1064 * LUCENE-8521: Change LatLonShape encoding to 7 dimensions instead of 6; where the
 1065   first 4 are index dimensions defining the bounding box of the Triangle and the
 1066   remaining 3 data dimensions define the vertices of the triangle. (Nick Knize)
 1067 
 1068 * LUCENE-8557: LeafReader.getFieldInfos is now documented and tested that it ought to return
 1069   the same cached instance.  MemoryIndex's impl now pre-creates the FieldInfos instead of
 1070   re-calculating a new instance each time.  (Tim Underwood, David Smiley)
 1071 
 1072 * LUCENE-8558: Replace O(N) lookup with O(1) lookup in PerFieldMergeState#FilterFieldInfos.
 1073   (Kranthi via Simon Willnauer)
 1074 
 1075 Other
 1076 
 1077 * LUCENE-8523: Correct typo in JapaneseNumberFilterFactory javadocs (Ankush Jhalani
 1078   via Alan Woodward)
 1079 
 1080 * LUCENE-8533: Fix Javadocs of DataInput#readVInt(): Negative numbers are
 1081   supported, but should be avoided. (Vladimir Dolzhenko via Uwe Schindler)
 1082 
 1083 ======================= Lucene 7.5.1 =======================
 1084 
 1085 Bug Fixes
 1086 
 1087 * LUCENE-8454: Fix incorrect vertex indexing and other computation errors in
 1088   shape tessellation that would sometimes cause an infinite loop. (Nick Knize)
 1089 
 1090 ======================= Lucene 7.5.0 =======================
 1091 
 1092 API Changes
 1093 
 1094 * LUCENE-8467: RAMDirectory, RAMFile, RAMInputStream, RAMOutputStream are deprecated
 1095   (Dawid Weiss)
 1096 
 1097 * LUCENE-8356: StandardFilter is deprecated (Alan Woodward)
 1098 
 1099 * LUCENE-8373: ENGLISH_STOP_WORD_SET on StandardAnalyzer is deprecated.  Instead
 1100   use EnglishAnalyzer.ENGLISH_STOP_WORD_SET.  The default constructor for
 1101   StopAnalyzer is also deprecated, and a stop word set should be explicitly
 1102   passed to the constructor.  (Alan Woodward)
 1103 
 1104 * LUCENE-8378: Add DocIdSetIterator.range static method to return an iterator
 1105   matching a range of docids (Mike McCandless)
 1106 
 1107 * LUCENE-8379: Add experimental TermQuery.getTermStates method (Mike McCandless)
 1108 
 1109 * LUCENE-8407: Add experimental SpanTermQuery.getTermStates method (David Smiley)
 1110 
 1111 * LUCENE-8390: MatchesIteratorSupplier replaced by IOSupplier (Alan Woodward,
 1112   David Smiley)
 1113 
 1114 * LUCENE-8397: Add DirectoryTaxonomyWriter.getCache (Mike McCandless)
 1115 
 1116 * LUCENE-8387: Add experimental IndexSearcher.getSlices API to see which slices
 1117   IndexSearcher is searching concurrently when it's created with an ExecutorService
 1118   (Mike McCandless)
 1119 
 1120 * LUCENE-8263: TieredMergePolicy's reclaimDeletesWeight has been replaced with a
 1121   new deletesPctAllowed setting to control how aggressively deletes should be
 1122   reclaimed. (Erick Erickson, Adrien Grand)
 1123 
 1124 * LUCENE-7314: Graduate LatLonPoint and query classes to core (Nick Knize)
 1125 
 1126 * LUCENE-8428: The way that oal.util.PriorityQueue creates sentinel objects has
 1127   been changed from a protected method to a java.util.function.Supplier as a
 1128   constructor argument. (Adrien Grand)
 1129 
 1130 * LUCENE-8437: CheckIndex.Status.cantOpenSegments and missingSegmentVersion
 1131   have been removed as they were not computed correctly. (Adrien Grand)
 1132 
 1133 * LUCENE-8286: The UnifiedHighlighter has a new HighlightFlag.WEIGHT_MATCHES flag that
 1134   will tell this highlighter to use the new MatchesIterator API as the underlying
 1135   approach to navigate matching hits for a query.  This mode will highlight more
 1136   accurately than any other highlighter, and can mark up phrases as one span instead of
 1137   word-by-word.  The UH's public internal APIs changed a bit in the process.
 1138   (David Smiley)
 1139 
 1140 * LUCENE-8471: IndexWriter.getFlushingBytes() returns how many bytes are currently
 1141   being flushed to disk. (Alan Woodward)
 1142 
 1143 * LUCENE-8422: Static helper functions for Matches and MatchesIterator implementations
 1144   have been moved from Matches to MatchesUtils (Alan Woodward)
 1145 
 1146 * LUCENE-8343: Suggesters now require Long (versus long, previously) from weight() method
 1147   while indexing, and provide double (versus long, previously) scores at lookup time
 1148   (Alessandro Benedetti)
 1149 
 1150 * LUCENE-8459: SearcherTaxonomyManager now has a constructor taking already opened
 1151   IndexReaders, allowing the caller to pass a FilterDirectoryReader, for example.
 1152   (Mike McCandless)
 1153 
 1154 Bug Fixes
 1155 
 1156 * LUCENE-8445: Tighten condition when two planes are identical to prevent constructing
 1157   bogus tiles when building GeoPolygons. (Ignacio Vera)
 1158 
 1159 * LUCENE-8444: Prevent building functionally identical plane bounds when constructing
 1160   DualCrossingEdgeIterator . (Ignacio Vera)
 1161 
 1162 * LUCENE-8380: UTF8TaxonomyWriterCache inconsistency. (Ruslan Torobaev, Dawid Weiss)
 1163 
 1164 * LUCENE-8164: IndexWriter silently accepts broken payload. This has been fixed
 1165   via LUCENE-8165 since we are now checking for offset+length going out of bounds.
 1166   (Robert Muir, Nhat Nyugen, Simon Willnauer)
 1167 
 1168 * LUCENE-8370: Reproducing 
 1169   TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()
 1170   failures (Erick Erickson)
 1171 
 1172 * LUCENE-8376, LUCENE-8371: ConditionalTokenFilter.end() would not propagate correctly
 1173   if the last token in the stream was subsequently dropped; FixedShingleFilter did
 1174   not set position increment in end() (Alan Woodward)
 1175 
 1176 * LUCENE-8395: WordDelimiterGraphFilter would incorrectly insert a hole into a
 1177   TokenStream if a token consisting entirely of delimiter characters was 
 1178   encountered, but preserve_original was set. (Alan Woodward)
 1179 
 1180 * LUCENE-8398: TieredMergePolicy.getMaxMergedSegmentMB has rounding error (Erick Erickson)
 1181 
 1182 * LUCENE-8429: DaciukMihovAutomatonBuilder is no longer prone to stack
 1183   overflows by enforcing a maximum term length. (Adrien Grand)
 1184 
 1185 * LUCENE-8441: IndexWriter now checks doc value type for index sort fields
 1186   and fails the document if they are not compatible. (Jim Ferenczi, Mike McCandless)
 1187 
 1188 * LUCENE-8458: Adjust initialization condition of PendingSoftDeletes and ensures
 1189   it is initialized before accepting deletes (Simon Willnauer, Nhat Nguyen)
 1190 
 1191 * LUCENE-8466: IndexWriter.deleteDocs(Query... query) incorrectly applies deletes on flush
 1192   if the index is sorted. (Adrien Grand, Jim Ferenczi, Vish Ramachandran)
 1193 
 1194 * LUCENE-8502: Allow access to delegate in FilterCodecReader. FilterCodecReader didn't 
 1195   allow access to it's delegate like other filter readers. This adds a new #getDelegate method
 1196   to access the wrapped reader. (Simon Willnauer)
 1197 
 1198 Changes in Runtime Behavior
 1199 
 1200 * LUCENE-7976: TieredMergePolicy now respects maxSegmentSizeMB by default when executing
 1201   findForcedMerges and findForcedDeletesMerges (Erick Erickson)
 1202 
 1203 * LUCENE-8263: TieredMergePolicy now reclaims deleted documents more
 1204   aggressively by default ensuring that no more than ~1/3 of the index size is
 1205   used by deleted documents. (Adrien Grand)
 1206 
 1207 * LUCENE-8503: Call #getDelegate instead of direct member access during unwrap. 
 1208   Filter*Reader instances access the member or the delegate directly instead of 
 1209   calling getDelegate(). In order to track access of the delegate these methods
 1210   should call #getDelegate() (Simon Willnauer)
 1211 
 1212 Improvements
 1213 
 1214 * LUCENE-8468: A ByteBuffer based Directory implementation. (Dawid Weiss)
 1215 
 1216 * LUCENE-8447: Add DISJOINT and WITHIN support to LatLonShape queries. (Nick Knize)
 1217 
 1218 * LUCENE-8440: Add support for indexing and searching Line and Point shapes using LatLonShape encoding (Nick Knize)
 1219 
 1220 * LUCENE-8435: Add new LatLonShapePolygonQuery for querying indexed LatLonShape fields by arbitrary polygons (Nick Knize)
 1221 
 1222 * LUCENE-8367: Make per-dimension drill down optional for each facet dimension (Mike McCandless)
 1223 
 1224 * LUCENE-8396: Add Points Based Shape Indexing and Search that decomposes shapes
 1225   into a triangular mesh and indexes individual triangles as a 6 dimension point (Nick Knize)
 1226 
 1227 * LUCENE-8345, GitHub PR #392: Remove instantiation of redundant wrapper classes for primitives;
 1228   add wrapper class constructors to forbiddenapis.  (Michael Braun via Uwe Schindler)
 1229 
 1230 * LUCENE-8415: Clean up Directory contracts and JavaDoc comments. (Dawid Weiss)
 1231 
 1232 * LUCENE-8414: Make segmentInfos private in IndexWriter (Simon Willnauer, Nhat Nguyen)
 1233 
 1234 * LUCENE-8446: The UnifiedHighlighter's DefaultPassageFormatter now treats overlapping matches in
 1235   the passage as merged (as if one larger match).  (David Smiley)
 1236 
 1237 * LUCENE-8460: Better argument validation in StoredField. (Namgyu Kim)
 1238 
 1239 * LUCENE-8432: TopFieldComparator stops comparing documents if the index is
 1240   sorted, even if hits still need to be visited to compute the hit count.
 1241   (Nikolay Khitrin)
 1242 
 1243 * LUCENE-8422: IntervalQuery now returns useful Matches (Alan Woodward)
 1244 
 1245 * LUCENE-7862: Store the real bounds of the leaf cells in the BKD index when the
 1246   number of dimensions is bigger than 1. It improves performance when there is
 1247   correlation between the dimensions, for example ranges. (Ignacio Vera, Adrien Grand)
 1248 
 1249 Build
 1250 
 1251 * LUCENE-5143: Stop publishing KEYS file with each version, use topmost lucene/KEYS file only. 
 1252   The buildAndPushRelease.py script validates that RM's PGP key is in the KEYS file.
 1253   Remove unused 'copy-to-stage' and '-dist-keys' targets from ant build. (janhoy)
 1254 
 1255 Other
 1256 
 1257 * LUCENE-8485: Update randomizedtesting to version 2.6.4. (Dawid Weiss)
 1258 
 1259 * LUCENE-8366: Upgrade to ICU 62.1. Emoji handling now uses Unicode 11's
 1260   Extended_Pictographic property. (Robert Muir)
 1261 
 1262 * LUCENE-8408: original Highlighter:  Remove obsolete static AttributeFactory instance
 1263   in TokenStreamFromTermVector.  (Michael Braun, David Smiley)
 1264 
 1265 * LUCENE-8420: Upgrade OpenNLP to 1.9.0 so OpenNLP tool can read the new model format which 1.8.x
 1266   cannot read. 1.9.0 can read the old format. (Koji Sekiguchi)
 1267 
 1268 * LUCENE-8453: Add documentation to analysis factories of Korean (Nori) analyzer
 1269   module.  (Tomoko Uchida via Uwe Schindler)
 1270 
 1271 * LUCENE-8455: Upgrade ECJ compiler to 4.6.1 in lucene/common-build.xml (Erick Erickson)
 1272 
 1273 * LUCENE-8456: Upgrade Apache Commons Compress to v1.18 (Steve Rowe)
 1274 
 1275 * LUCENE-765: Improved org.apache.lucene.index javadocs. (Mike Sokolov)
 1276 
 1277 * LUCENE-8476: Remove redundant nullity check and switch to optimized List.sort in the
 1278   Korean's user dictionary. (Namgyu Kim)
 1279 
 1280 ======================= Lucene 7.4.1 =======================
 1281 
 1282 Bug Fixes
 1283  
 1284  * LUCENE-8365: Fix ArrayIndexOutOfBoundsException in UnifiedHighlighter. This fixes 
 1285    a "off by one" error in the UnifiedHighlighter's code that is only triggered when
 1286    two nested SpanNearQueries contain the same term. (Marc-Andre Morissette via Simon Willnauer)
 1287 
 1288  * LUCENE-8381: Fix IndexWriter incorrectly interprets hard-deletes as soft-deletes
 1289    while wrapping reader for merges. (Simon Willnauer, Nhat Nguyen)
 1290 
 1291  * LUCENE-8384: Fix missing advance docValues generation while handling docValues
 1292    update in PendingSoftDeletes. (Simon Willnauer, Nhat Nguyen)
 1293 
 1294  * LUCENE-8472: Always rewrite the soft-deletes merge retention query. (Adrien Grand, Nhat Nguyen)
 1295 
 1296 ======================= Lucene 7.4.0 =======================
 1297 
 1298 Upgrading
 1299 
 1300 * LUCENE-8344: If you are using the AnalyzingSuggester or FuzzySuggester subclass, and if you
 1301   explicitly use the preservePositionIncrements=false setting (not the default), then you ought
 1302   to rebuild your suggester index. If you don't, queries or indexed data with trailing position
 1303   gaps (e.g. stop words) may not work correctly. (David Smiley, Jim Ferenczi)
 1304 
 1305 API Changes
 1306 
 1307 * LUCENE-8242: IndexSearcher.createNormalizedWeight() has been deprecated.
 1308   Instead use IndexSearcher.createWeight(), rewriting the query first.
 1309   (Alan Woodward)
 1310 
 1311 * LUCENE-8248: MergePolicyWrapper is renamed to FilterMergePolicy and now
 1312   also overrides getMaxCFSSegmentSizeMB (Mike Sokolov via Mike McCandless)
 1313 
 1314 * LUCENE-8303: LiveDocsFormat is now only responsible for (de)serialization of
 1315   live docs. (Adrien Grand)
 1316 
 1317 Changes in Runtime Behavior
 1318 
 1319 * LUCENE-8309: Live docs are no longer backed by a FixedBitSet. (Adrien Grand)
 1320 
 1321 * LUCENE-8330: Detach IndexWriter from MergePolicy. MergePolicy now instead of 
 1322   requiring IndexWriter as a hard dependency expects a MergeContext which
 1323   IndexWriter implements. (Simon Willnauer, Robert Muir, Dawid Weiss, Mike McCandless)
 1324 
 1325 New Features
 1326 
 1327 * LUCENE-8200: Allow doc-values to be updated atomically together
 1328   with a document. Doc-Values updates now can be used as a soft-delete
 1329   mechanism to all keeping several version of a document or already 
 1330   deleted documents around for later reuse. See "IW.softUpdateDocument(...)"
 1331   for reference. (Simon Willnauer)
 1332 
 1333 * LUCENE-8197: A new FeatureField makes it easy and efficient to integrate
 1334   static relevance signals into the final score. (Adrien Grand, Robert Muir)
 1335 
 1336 * LUCENE-8202: Add a FixedShingleFilter (Alan Woodward, Adrien Grand, Jim
 1337   Ferenczi)
 1338 
 1339 * LUCENE-8125: ICUTokenizer support for emoji/emoji sequence tokens. (Robert Muir)
 1340 
 1341 * LUCENE-8196, LUCENE-8300: A new IntervalQuery in the sandbox allows efficient proximity
 1342   searches based on minimum-interval semantics. (Alan Woodward, Adrien Grand,
 1343   Jim Ferenczi, Simon Willnauer, Matt Weber)
 1344 
 1345 * LUCENE-8233: Add support for soft deletes to IndexWriter delete accounting. 
 1346   Soft deletes are accounted for inside the index writer and therefor also
 1347   by merge policies. A SoftDeletesRetentionMergePolicy is added that allows
 1348   to selectively carry over soft_deleted document across merges for retention
 1349   policies (Simon Willnauer, Mike McCandless, Robert Muir)
 1350 
 1351 * LUCENE-8237: Add a SoftDeletesDirectoryReaderWrapper that allows to respect
 1352   soft deletes if the reader is opened form a directory. (Simon Willnauer,
 1353   Mike McCandless, Uwe Schindler, Adrien Grand)
 1354 
 1355 * LUCENE-8229, LUCENE-8270: Add a method Weight.matches(LeafReaderContext, doc) 
 1356   that returns an iterator over matching positions for a given query and document.
 1357   This allows exact hit extraction and will enable implementation of accurate 
 1358   highlighters. (Alan Woodward, Adrien Grand, David Smiley)
 1359 
 1360 * LUCENE-8249: Implement Matches API for phrase queries (Alan Woodward, Adrien 
 1361   Grand)
 1362 
 1363 * LUCENE-8246: Allow to customize the number of deletes a merge claims. This
 1364   helps merge policies in the soft-delete case to correctly implement retention
 1365   policies without triggering uncessary merges. (Simon Willnauer, Mike McCandless)
 1366 
 1367 * LUCENE-8231: A new analysis module (nori) similar to Kuromoji
 1368   but to handle Korean using mecab-ko-dic and morphological analysis.
 1369   (Robert Muir, Jim Ferenczi)
 1370 
 1371 * LUCENE-8265: WordDelimter/GraphFilter now have an option to skip tokens
 1372   marked with KeywordAttribute (Mike Sokolov via Mike McCandless)
 1373 
 1374 * LUCENE-8297: Add IW#tryUpdateDocValues(Reader, int, Fields...) IndexWriter can
 1375   update doc values for a specific term but this might affect all documents 
 1376   containing the term. With tryUpdateDocValues users can update doc-values 
 1377   fields for individual documents. This allows for instance to soft-delete
 1378   individual documents. (Simon Willnauer)
 1379 
 1380 * LUCENE-8298: Allow DocValues updates to reset a value. Passing a DV field with a null
 1381   value to IW#updateDocValues or IW#tryUpdateDocValues will now remove the value from the 
 1382   provided document. This allows to undelete a soft-deleted document unless it's been claimed
 1383   by a merge. (Simon Willnauer) 
 1384 
 1385 * LUCENE-8273: ConditionalTokenFilter allows analysis chains to skip particular token
 1386   filters based on the attributes of the current token. This generalises the keyword
 1387   token logic currently used for stemmers and WDF.  It is integrated into 
 1388   CustomAnalyzer by using the `when` and `whenTerm` builder methods, and a new
 1389   ProtectedTermFilter is added as an example.  (Alan Woodward, Robert Muir,
 1390   David Smiley, Steve Rowe, Mike Sokolov)
 1391 
 1392 * LUCENE-8310: Ensure IndexFileDeleter accounts for pending deletes. Today we fail
 1393   creating the IndexWriter when the directory has a pending delete. Yet, this
 1394   is mainly done to prevent writing still existing files more than once.
 1395   IndexFileDeleter already accounts for that for existing files which we can
 1396   now use to also take pending deletes into account which ensures that all file
 1397   generations per segment always go forward. (Simon Willnauer)
 1398   
 1399 * LUCENE-7960: Add preserveOriginal option to the NGram and EdgeNGram filters.
 1400   (Ingomar Wesp, Shawn Heisey via Robert Muir)
 1401 
 1402 * LUCENE-8335: Enforce soft-deletes field up-front. Soft deletes field must be marked
 1403   as such once it's introduced and can't be changed after the fact.
 1404   (Nhat Nguyen via Simon Willnauer)
 1405 
 1406 * LUCENE-8332: New ConcatenateGraphFilter for concatenating all tokens into one (or more
 1407   in the event of a graph input).  This is useful for fast analyzed exact-match lookup,
 1408   suggesters, and as a component of a named entity recognition system.  This was excised
 1409   out of CompletionTokenStream in the NRT doc suggester.  (David Smiley, Jim Ferenczi)
 1410 
 1411 Bug Fixes
 1412 
 1413 * LUCENE-8221: MoreLikeThis.setMaxDocFreqPct can easily int-overflow on larger
 1414   indexes.
 1415 
 1416 * LUCENE-8266: Detect bogus tiles when creating a standard polygon and
 1417   throw a TileException. (Ignacio Vera)
 1418 
 1419 * LUCENE-8234: Fixed bug in how spatial relationship is computed for
 1420   GeoStandardCircle when it covers the whole world. (Ignacio Vera)
 1421 
 1422 * LUCENE-8236: Filter duplicated points when creating GeoPath shapes to
 1423   avoid creation of bogus planes. (Ignacio Vera)
 1424 
 1425 * LUCENE-8243: IndexWriter.addIndexes(Directory[]) did not properly preserve
 1426   index file names for updated doc values fields (Simon Willnauer,
 1427   Michael McCandless, Nhat Nguyen)
 1428 
 1429 * LUCENE-8275: Push up #checkPendingDeletes to Directory to ensure IW fails if 
 1430   the directory has pending deletes files even if the directory is filtered or 
 1431   a FileSwitchDirectory (Simon Willnauer, Robert Muir)
 1432 
 1433 * LUCENE-8244: Do not leak open file descriptors in SearcherTaxonomyManager's
 1434   refresh on exception (Mike McCandless)
 1435 
 1436 * LUCENE-8305: ComplexPhraseQuery.rewrite now handles an embedded MultiTermQuery
 1437   that rewrites to a MatchNoDocsQuery instead of throwing an exception.
 1438   (Bjarke Mortensen, Andy Tran via David Smiley)
 1439 
 1440 * LUCENE-8287: Ensure that empty regex completion queries always return no results.
 1441   (Julie Tibshirani via Jim Ferenczi)
 1442 
 1443 * LUCENE-8317: Prevent concurrent deletes from being applied during full flush.
 1444   Future deletes could potentially be exposed to flushes/commits/refreshes if the 
 1445   amount of RAM used by deletes is greater than half of the IW RAM buffer. (Simon Willnauer)
 1446 
 1447 * LUCENE-8320: Fix WindowsFS to correctly account for rename and hardlinks. 
 1448   (Simon Willnauer, Nhat Nguyen)
 1449 
 1450 * LUCENE-8328: Ensure ReadersAndUpdates consistently executes under lock.
 1451   (Nhat Nguyen via Simon Willnauer)
 1452 
 1453 * LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs.
 1454   (chengpohi via Jim Ferenczi)
 1455 
 1456 * LUCENE-8186: LowerCaseTokenizerFactory now lowercases text in multi-term
 1457   queries. (Tim Allison via Adrien Grand)
 1458 
 1459 * LUCENE-8278: Some end-of-input no-scheme domain-only URL tokens are typed as
 1460   <ALPHANUM> rather than <URL>.  (Junte Zhang, Steve Rowe)
 1461 
 1462 * LUCENE-8355: Prevent IW from opening an already dropped segment while DV updates
 1463   are written. (Nhat Nguyen via Simon Willnauer)
 1464 
 1465 * LUCENE-8344: TokenStreamToAutomaton (used by some suggesters) was not ignoring a trailing
 1466   position increment when the preservePositionIncrement setting is false.
 1467   (David Smiley, Jim Ferenczi)
 1468 
 1469 * LUCENE-8357: FunctionScoreQuery.boostByQuery() and boostByValue() were
 1470   producing truncated Explanations (Markus Jelsma, Alan Woodward)
 1471 
 1472 * LUCENE-8360: NGramTokenFilter and EdgeNGramTokenFilter did not correctly
 1473   set position increments in end() (Alan Woodward)
 1474 
 1475 Other
 1476 
 1477 * LUCENE-8301: Update randomizedtesting to 2.6.0. (Dawid Weiss)
 1478 
 1479 * LUCENE-8299: Geo3D wrapper uses new polygon method factory that gives better
 1480   support for polygons with many points (>100). (Ignacio vera)
 1481 
 1482 * LUCENE-8261: InterpolatedProperties.interpolate and recursive property
 1483   references. (Steve Rowe, Dawid Weiss)
 1484 
 1485 * LUCENE-8228: removed obsolete IndexDeletionPolicy clone() requirements from
 1486   the javadoc. (Dawid Weiss)
 1487 
 1488 * LUCENE-8219: Use a realistic estimate of the number of nodes and links in
 1489    LevensteinAutomaton.java, to save reallocation of arrays.
 1490    (Christian Ziech)
 1491 
 1492 * LUCENE-8214: Improve selection of testPoint for GeoComplexPolygon.
 1493   (Ignacio Vera)
 1494   
 1495 * SOLR-10912: Add automatic patch validation. (Mano Kovacs, Steve Rowe)
 1496 
 1497 * LUCENE-8122, LUCENE-8175: Upgrade analysis/icu to ICU 61.1.
 1498   (Robert Muir, Adrien Grand, Uwe Schindler)
 1499 
 1500 * LUCENE-8291: Remove QueryTemplateManager utility class from XML queryparser.
 1501   This class is just a general XML transforming tool (using property files and
 1502   XSLT) and has nothing to do with query parsing. It can easily be implemented
 1503   using more sophisticated libraries or using XSL transformers from the JDK.
 1504   This change also removes the Lucene demo webapp to prevent XSS issues in
 1505   untested/unmaintained code. (Uwe Schindler)
 1506 
 1507 Build
 1508 
 1509 * LUCENE-7935: Publish .sha512 hash files with the release artifacts and stop
 1510   publishing .md5 hashes since the algorithm is broken (janhoy)
 1511 
 1512 * LUCENE-8230: Upgrade forbiddenapis to version 2.5.  (Uwe Schindler)
 1513 
 1514 Documentation
 1515 
 1516 * LUCENE-8238: Improve WordDelimiterFilter and WordDelimiterGraphFilter javadocs
 1517   (Mike Sokolov via Mike McCandless)
 1518 
 1519 ======================= Lucene 7.3.1 =======================
 1520 
 1521 Bug fixes
 1522 
 1523 * LUCENE-8254: LRUQueryCache could cause IndexReader to hang on close, when
 1524   shared with another reader with no CacheHelper (Alan Woodward, Simon Willnauer,
 1525   Adrien Grand)
 1526 
 1527 ======================= Lucene 7.3.0 =======================
 1528 
 1529 API Changes
 1530 
 1531 * LUCENE-8051: LevensteinDistance renamed to LevenshteinDistance.
 1532   (Pulak Ghosh via Adrien Grand)
 1533 
 1534 * LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery and BoostingQuery.
 1535   Users should instead use FunctionScoreQuery, possibly combined with
 1536   a lucene expression (Alan Woodward)
 1537 
 1538 * LUCENE-8104: Remove facets module compile-time dependency on queries
 1539   (Alan Woodward)
 1540 
 1541 * LUCENE-8145: UnifiedHighlighter now uses a unitary OffsetsEnum rather
 1542   than a list of enums (Alan Woodward, David Smiley, Jim Ferenczi, Timothy
 1543   Rodriguez)
 1544 
 1545 New Features
 1546 
 1547 * LUCENE-2899: Add new module analysis/opennlp, with analysis components
 1548   to perform tokenization, part-of-speech tagging, lemmatization and phrase
 1549   chunking by invoking the corresponding OpenNLP tools. Named entity
 1550   recognition is also provided as a Solr update request processor.
 1551   (Lance Norskog, Grant Ingersoll, Joern Kottmann, Em, Kai Gülzau,
 1552   Rene Nederhand, Robert Muir, Steven Bower, Steve Rowe)
 1553 
 1554 * LUCENE-8126: Add new spatial prefix tree (SPT) based on google S2 geometry.
 1555   It can only be used currently with Geo3D spatial context and it provides
 1556   improvements on indexing time for non-points shapes and on query performance.
 1557   (Ignacio Vera, David Smiley).
 1558 
 1559 Improvements
 1560 
 1561 * LUCENE-8081: Allow IndexWriter to opt out of flushing on indexing threads
 1562   Index/Update Threads try to help out flushing pending document buffers to
 1563   disk. This change adds an expert setting to opt ouf of this behavior unless
 1564   flusing is falling behind. (Simon Willnauer)
 1565 
 1566 * LUCENE-8086: spatial-extras Geo3dFactory: Use GeoExactCircle with
 1567   configurable precision for non-spherical planet models.
 1568   (Ignacio Vera via David Smiley)
 1569 
 1570 * LUCENE-8093: TrimFilterFactory implements MultiTermAwareComponent (Alan Woodward)
 1571 
 1572 * LUCENE-8094: TermInSetQuery.toString now returns "field:(A B C)" (Mike McCandless)
 1573 
 1574 * LUCENE-8121: UnifiedHighlighter passage relevancy is improved for terms that are
 1575   position sensitive (e.g. part of a phrase) by having an accurate freq.
 1576   (David Smiley)
 1577 
 1578 * LUCENE-8129: A Unicode set filter can now be specified when using ICUFoldingFilter.
 1579   (Ere Maijala)
 1580 
 1581 * LUCENE-7966: Build Multi-Release JARs to enable usage of optimized intrinsic methods
 1582   from Java 9 for index bounds checking and array comparison/mismatch. This change
 1583   introduces Java 8 replacements for those Java 9 methods and patches the compiled
 1584   classes to use the optimized variants through the MR-JAR mechanism.
 1585   (Uwe Schindler, Robert Muir, Adrien Grand, Mike McCandless)
 1586 
 1587 * LUCENE-8127: Speed up rewriteNoScoring when there are no MUST clauses.
 1588   (Michael Braun via Adrien Grand)
 1589 
 1590 * LUCENE-8152: Improve consumption of doc-value iterators. (Horatiu Lazu via
 1591   Adrien Grand)
 1592 
 1593 * LUCENE-8033: FieldInfos now always use a dense encoding. (Mayya Sharipova
 1594   via Adrien Grand)
 1595 
 1596 * LUCENE-8190: Specialized cell interface to allow any spatial prefix tree to
 1597   benefit from the setting setPruneLeafyBranches on RecursivePrefixTreeStrategy.
 1598   (Ignacio Vera)
 1599 
 1600 Bug Fixes
 1601 
 1602 * LUCENE-8077: Fixed bug in how CheckIndex verifies doc-value iterators.
 1603   (Xiaoshan Sun via Adrien Grand)
 1604 
 1605 * SOLR-11758: Fixed FloatDocValues.boolVal to correctly return true for all values != 0.0F
 1606   (Munendra S N via hossman)
 1607 
 1608 * LUCENE-8121: The UnifiedHighlighter would highlight some terms within some nested
 1609   SpanNearQueries at positions where it should not have.  It's fixed in the UH by
 1610   switching to the SpanCollector API.  The original Highlighter still has this
 1611   problem (LUCENE-2287, LUCENE-5455, LUCENE-6796).  Some public but internal parts of
 1612   the UH were refactored. (David Smiley, Steve Davids)
 1613 
 1614 * LUCENE-8120: Fix LatLonBoundingBox's toString() method (Martijn van Groningen, Adrien Grand)
 1615 
 1616 * LUCENE-8130: Fix NullPointerException from TermStates.toString() (Mike McCandless)
 1617 
 1618 * LUCENE-8124: Fixed HyphenationCompoundWordTokenFilter to handle correctly
 1619   hyphenation patterns with indicator >= 7. (Holger Bruch via Adrien Grand)
 1620 
 1621 * LUCENE-8163: BaseDirectoryTestCase could produce random filenames that fail
 1622   on Windows (Alan Woodward)
 1623 
 1624 * LUCENE-8174: Fixed {Float,Double,Int,Long}Range.toString(). (Oliver Kaleske
 1625   via Adrien Grand)
 1626 
 1627 * LUCENE-8182: Fixed BoostingQuery to apply the context boost instead of the parent query
 1628   boost (Jim Ferenczi)
 1629 
 1630 * LUCENE-8188: Fixed bugs in OpenNLPOpsFactory that were causing InputStreams fetched from the
 1631   ResourceLoader to be leaked (hossman)
 1632 
 1633 
 1634 Other
 1635 
 1636 * LUCENE-8111: IndexOrDocValuesQuery Javadoc references outdated method name.
 1637   (Kai Chan via Adrien Grand)
 1638 
 1639 * LUCENE-8106: Add script (reproduceJenkinsFailures.py) to attempt to reproduce
 1640   failing tests from a Jenkins log. (Steve Rowe)
 1641 
 1642 * LUCENE-8075: Removed unnecessary null check in IntersectTermsEnum.
 1643   (Pulak Ghosh via Adrien Grand)
 1644 
 1645 * LUCENE-8156: Require users to not have ASM on the Ant classpath during build.
 1646   This is required by LUCENE-7966. (Adrien Grand, Uwe Schindler)
 1647 
 1648 * LUCENE-8161: spatial-extras: the Spatial4j dependency has been updated from 0.6 to 0.7,
 1649   which is drop-in compatible (Lucene doesn't expressly use any of the few API differences).
 1650   Spatial4j 0.7 is compatible with JTS 1.15.0 and not any prior version.  JTS 1.15.0 is
 1651   dual-licensed to include BSD; prior versions were LGPL.  (David Smiley)
 1652 
 1653 * LUCENE-8155: Add back support in smoke tester to run against later Java versions.
 1654   (Uwe Schindler)
 1655 
 1656 * LUCENE-8169: Migrated build to use OpenClover 4.2.1 for checking code coverage.
 1657   (Uwe Schindler)
 1658 
 1659 * LUCENE-8170: Improve OpenClover reports (separate test from production code);
 1660   enable coverage reports inside test-frameworks.  (Uwe Schindler)
 1661 
 1662 Build
 1663 
 1664 * LUCENE-8168: Moved Groovy scripts in build files to separate files.
 1665   Update Groovy to 2.4.13.  (Uwe Schindler)
 1666 
 1667 * LUCENE-8176: HttpReplicatorTest awaits more than a minute for stopping Jetty threads
 1668   (Mikhail Khludnev)
 1669 
 1670 ======================= Lucene 7.2.1 =======================
 1671 
 1672 Bug Fixes
 1673 
 1674 * LUCENE-8117: Fix advanceExact on SortedNumericDocValues produced by Lucene54DocValues. (Jim Ferenczi).
 1675 
 1676 ======================= Lucene 7.2.0 =======================
 1677 
 1678 API Changes
 1679 
 1680 * LUCENE-8017, LUCENE-8042: Weight, DoubleValuesSource and related objects
 1681   now implement a SegmentCacheable interface, with a single method
 1682   isCacheable(LeafReaderContext) determining whether or not the object may
 1683   be cached against a LeafReader. (Alan Woodward, Robert Muir)
 1684 
 1685 * LUCENE-8038: Payload factors for scoring in PayloadScoreQuery are now
 1686   calculated by a PayloadDecoder, instead of delegating to the Similarity.
 1687   (Alan Woodward)
 1688 
 1689 * LUCENE-8014: Similarity.computeSlopFactor() and
 1690   Similarity.computePayloadFactor() have been deprecated. (Alan Woodward)
 1691 
 1692 * LUCENE-6278: Scorer.freq() has been removed (Alan Woodward)
 1693 
 1694 * LUCENE-7736: DoubleValuesSource and LongValuesSource now expose a
 1695   rewrite(IndexSearcher) function. (Alan Woodward)
 1696 
 1697 * LUCENE-7998: DoubleValuesSource.fromQuery() allows you to use the scores
 1698   from a Query as a DoubleValuesSource. (Alan Woodward)
 1699 
 1700 * LUCENE-8049: IndexWriter.getMergingSegments()'s return type was changed from
 1701   Collection to Set to more accurately reflect it's nature. (David Smiley)
 1702 
 1703 * LUCENE-8059: TopFieldDocCollector can now early terminate collection when
 1704   the sort order is compatible with the index order. As a consequence,
 1705   EarlyTerminatingSortingCollector is now deprecated. (Adrien Grand)
 1706 
 1707 New Features
 1708 
 1709 * LUCENE-8061: Add convenience factory methods to create BBoxes and XYZSolids
 1710   directly from bounds objects.
 1711   
 1712 * LUCENE-7736: IndexReaderFunctions expose various IndexReader statistics as
 1713   DoubleValuesSources. (Alan Woodward)
 1714 
 1715 * LUCENE-8068: Allow IndexWriter to write a single DWPT to disk Adds a
 1716   flushNextBuffer method to IndexWriter that allows the caller to
 1717   synchronously move the next pending or the biggest non-pending index buffer to
 1718   disk. This enables flushing selected buffer to disk without highjacking an
 1719   indexing thread. This is for instance useful if more than one IW (shards) must
 1720   be maintained in a single JVM / system. (Simon Willnauer)
 1721 
 1722 Bug Fixes
 1723 
 1724 * LUCENE-8076: Normalize Vincenti distance calculation for planet models that aren't normalized.
 1725   (Ignacio Vera)
 1726 
 1727 * LUCENE-8057: Exact circle bounds computation was incorrect.
 1728   (Ignacio Vera)
 1729 
 1730 * LUCENE-8056: Exact circle segment bounding suffered from precision errors.
 1731   (Karl Wright)
 1732 
 1733 * LUCENE-8054: Fix the exact circle case where relationships fail when the
 1734   planet model has c <= ab, because the planes are constructed incorrectly.
 1735   (Ignacio Vera)
 1736 
 1737 * LUCENE-7991: KNearestNeighborDocumentClassifier.knnSearch no longer applies
 1738   a previous boosted field's factor to subsequent unboosted fields.
 1739   (Christine Poerschke)
 1740 
 1741 * LUCENE-7999: Switch from int to long to track the name for the next
 1742   segment to write, so that very long lived indices with very frequent
 1743   refreshes or commits, and high indexing thread counts, do not
 1744   overflow an int (Mykhailo Demianenko via Mike McCandless)
 1745 
 1746 * LUCENE-8025: Use sumTotalTermFreq=sumDocFreq when scoring DOCS_ONLY fields
 1747   that omit term frequency information, as it is equivalent in that case.
 1748   Previously bogus numbers were used, and many similarities would
 1749   completely degrade. (Robert Muir, Adrien Grand)
 1750 
 1751 * LUCENE-8045: ParallelLeafReader did not correctly report FieldInfo.dvGen
 1752   (Alan Woodward)
 1753 
 1754 * LUCENE-8034: Use subtraction instead of addition to sidestep int
 1755   overflow in SpanNotQuery.  (Hari Menon via Mike McCandless)
 1756 
 1757 * LUCENE-8078: The query cache should not cache instances of
 1758   MatchNoDocsQuery. (Jon Harper via Adrien Grand)
 1759 
 1760 * LUCENE-8048: Filesystems do not guarantee order of directories updates
 1761   (Nikolay Martynov, Simon Willnauer, Erick Erickson)
 1762   
 1763 Optimizations
 1764 
 1765 * LUCENE-8018: Smaller FieldInfos memory footprint by not retaining unnecessary
 1766   references to TreeMap entries. (Julian Vassev via Adrien Grand)
 1767 
 1768 * LUCENE-7994: Use int/int scatter map to gather facet counts when the
 1769   number of hits is small relative to the number of unique facet labels
 1770   (Dawid Weiss, Robert Muir, Mike McCandless)
 1771 
 1772 * LUCENE-8062: GlobalOrdinalsQuery is no longer eligible for caching. (Jim Ferenczi)
 1773 
 1774 * LUCENE-8058: Large instances of TermInSetQuery are no longer eligible for
 1775   caching as they could break memory accounting of the query cache.
 1776   (Adrien Grand)
 1777 
 1778 * LUCENE-8055: MemoryIndex.MemoryDocValuesIterator returns 2 documents
 1779   instead of 1. (Simon Willnauer)
 1780 
 1781 * LUCENE-8043: Fix document accounting in IndexWriter to prevent writing too many
 1782   documents. Once this happens, Lucene refuses to open the index and throws a
 1783   CorruptIndexException. (Simon Willnauer, Yonik Seeley, Mike McCandless)
 1784 
 1785 Tests
 1786 
 1787 * LUCENE-8035: Run tests with JDK-specific options: --illegal-access=deny
 1788   on Java 9+.  (Uwe Schindler)
 1789 
 1790 Build
 1791 
 1792 * LUCENE-6144: Upgrade Ivy to 2.4.0; 'ant ivy-bootstrap' now removes old Ivy
 1793   jars in ~/.ant/lib/.  (Shawn Heisey, Steve Rowe)
 1794 
 1795 
 1796 ======================= Lucene 7.1.0 =======================
 1797 
 1798 Changes in Runtime Behavior
 1799 
 1800 * Resolving of external entities in queryparser/xml/CoreParser is disallowed
 1801   by default. See SOLR-11477 for details.
 1802 
 1803 New Features
 1804 
 1805 * LUCENE-7970: Add a shape to Geo3D that consists of multiple planes that
 1806   approximate a true circle, rather than an ellipse, for non-spherical planet models.
 1807   (Karl Wright, Ignacio Vera)
 1808   
 1809 * LUCENE-7955: Add support for the concept of "nearest distance" to Geo3D's
 1810   GeoPath abstraction, which is the distance along the path to the point that is
 1811   closest to the provided point. (Karl Wright)
 1812   
 1813 * LUCENE-7906: Add spatial relationships between all currently-defined Geo shapes.
 1814   (Ignacio Vera)
 1815 
 1816 * LUCENE-7955: Add support for zero-width paths. (Karl Wright)
 1817 
 1818 * LUCENE-7936: Add serialization and deserialization support to Geo3D. (Karl Wright,
 1819   Ignacio Vera)
 1820 
 1821 * LUCENE-7942: Distance computations now have the ability to accurately aggregate
 1822   distances, rather than just doing sums. (Karl Wright)
 1823 
 1824 * LUCENE-7934: Add a planet model interface. (Karl Wright)
 1825 
 1826 * LUCENE-7918: Revamp the API for composites so that it's generic and can be used
 1827   for many kinds of shapes. (Ignacio Vera)
 1828 
 1829 * LUCENE-7621: Add CoveringQuery, a query whose required number of matching
 1830   clauses can be defined per document. (Adrien Grand)
 1831 
 1832 * LUCENE-7927: Add LongValueFacetCounts, to compute facet counts for individual
 1833   numeric values (Mike McCandless)
 1834 
 1835 * LUCENE-7940: Add BengaliAnalyzer. (Md. Abdulla-Al-Sun via Robert Muir)
 1836 
 1837 * LUCENE-7392: Add point based LatLonBoundingBox as new RangeField Type.
 1838   (Nick Knize)
 1839 
 1840 * LUCENE-7951: Spatial-extras has much better Geo3d support by implementing Spatial4j
 1841   abstractions: SpatialContextFactory, ShapeFactory, BinaryCodec, DistanceCalculator.
 1842   (Ignacio Vera, David Smiley)
 1843 
 1844 * LUCENE-7973: Update dictionary version for Ukrainian analyzer to 3.9.0 (Andriy
 1845   Rysin via Dawid Weiss)
 1846   
 1847 * LUCENE-7974: Add FloatPointNearestNeighbor, an N-dimensional FloatPoint
 1848   K-nearest-neighbor search implementation.  (Steve Rowe)
 1849 
 1850 * LUCENE-7975: Change the default taxonomy facets cache to a faster
 1851   byte[] (UTF-8) based cache.  (Mike McCandless)
 1852 
 1853 * LUCENE-7972: DirectoryTaxonomyReader, in Lucene's facet module, now
 1854   implements Accountable, so you can more easily track how much heap
 1855   it's using.  (Mike McCandless)
 1856 
 1857 * LUCENE-7982: A new NormsFieldExistsQuery matches documents that have
 1858   norms in a specified field (Colin Goodheart-Smithe via Mike McCandless)
 1859 
 1860 Optimizations
 1861 
 1862 * LUCENE-7905: Optimize how OrdinalMap (used by
 1863   SortedSetDocValuesFacetCounts and others) builds its map (Robert
 1864   Muir, Adrien Grand, Mike McCandless)
 1865 
 1866 * LUCENE-7655: Speed up geo-distance queries in case of dense single-valued
 1867   fields when most documents match. (Maciej Zasada via Adrien Grand)
 1868 
 1869 * LUCENE-7897: IndexOrDocValuesQuery now requires the range cost to be more
 1870   than 8x greater than the cost of the lead iterator in order to use doc values.
 1871   (Murali Krishna P via Adrien Grand)
 1872 
 1873 * LUCENE-7925: Collapse duplicate SHOULD or MUST clauses by summing up their
 1874   boosts. (Adrien Grand)
 1875 
 1876 * LUCENE-7939: MinShouldMatchSumScorer now leverages two-phase iteration in
 1877   order to be faster when used in conjunctions. (Adrien Grand)
 1878 
 1879 * LUCENE-7827: AnalyzingInfixSuggester doesn't create "textgrams" 
 1880   when minPrefixChar=0 (Mikhail Khludnev)
 1881 
 1882 Bug Fixes
 1883 
 1884 * LUCENE-8066: It was still possible to construct a concave GeoExactCircle, so use
 1885    a sector approach to prevent that. (Ignacio Vera)
 1886 
 1887 * LUCENE-7967: The GeoDegeneratePoint isWithin() method needed allowance for
 1888    numerical precision. (Karl Wright)
 1889 
 1890 * LUCENE-7965: GeoBBoxFactory was constructing the wrong shape at the poles
 1891   if the longitude span was greater than 180 degrees. (Karl Wright)
 1892 
 1893 * LUCENE-7916: Prevent ArrayIndexOutOfBoundsException if ICUTokenizer is used
 1894   with a different ICU JAR version than it is compiled against. Note, this is
 1895   not recommended, lucene-analyzers-icu contains binary data structures
 1896   specific to ICU/Unicode versions it is built against. (Chris Koenig, Robert Muir)
 1897 
 1898 * LUCENE-7891: Lucene's taxonomy facets now uses a non-buggy LRU cache
 1899   by default.  (Jan-Willem van den Broek via Mike McCandless)
 1900 
 1901 * LUCENE-7959: Improve NativeFSLockFactory's exception message if it cannot create
 1902   write.lock for an empty index due to bad permissions/read-only filesystem/etc.
 1903   (Erick Erickson, Shawn Heisey, Robert Muir)
 1904 
 1905 * LUCENE-7968: AnalyzingSuggester would sometimes order suggestions incorrectly,
 1906   it did not properly break ties on the surface forms when both the weights and
 1907   the analyzed forms were equal. (Robert Muir)
 1908 
 1909 * LUCENE-7957: ConjunctionScorer.getChildren was failing to return all
 1910   child scorers (Adrien Grand, Mike McCandless)
 1911 
 1912 * SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser
 1913   by default. (Michael Stepankin, Olga Barinova, Uwe Schindler, Christine Poerschke)
 1914 
 1915 Build
 1916 
 1917 * SOLR-11181: Switch order of maven artifact publishing procedure: deploy first
 1918   instead of locally installing first, to workaround a double repository push of
 1919   *-sources.jar and *-javadoc.jar files.  (Lynn Monson via Steve Rowe)
 1920   
 1921 * LUCENE-6673: Maven build fails for target javadoc:jar.  
 1922   (Ramkumar Aiyengar, Daniel Collins via Steve Rowe)
 1923 
 1924 * LUCENE-7985: Upgrade forbiddenapis to 2.4.1.  (Uwe Schindler)
 1925 
 1926 Other
 1927 
 1928 * LUCENE-7948, LUCENE-7937: Upgrade randomizedtesting to 2.5.3 (minor fixes 
 1929   in test filtering for IDEs). (Mike Sokolov, Dawid Weiss)
 1930 
 1931 * LUCENE-7933: LongBitSet now validates the numBits parameter (Won
 1932   Jonghoon, Mike McCandless)
 1933 
 1934 * LUCENE-7978: Add some more documentation about setting up build
 1935   environment.  (Anton R. Yuste via Uwe Schindler)
 1936 
 1937 * LUCENE-7983: IndexWriter.IndexReaderWarmer is now a functional interface
 1938   instead of an abstract class with a single method (Dawid Weiss)
 1939   
 1940 * LUCENE-5753: Update TLDs recognized by UAX29URLEmailTokenizer. (Steve Rowe)
 1941 
 1942 
 1943 ======================= Lucene 7.0.1 =======================
 1944 
 1945 Bug Fixes
 1946 
 1947 * LUCENE-7957: ConjunctionScorer.getChildren was failing to return all
 1948   child scorers (Adrien Grand, Mike McCandless)
 1949 
 1950 ======================= Lucene 7.0.0 =======================
 1951 
 1952 New Features
 1953 
 1954 * LUCENE-7703: SegmentInfos now record the major Lucene version at index
 1955   creation time. (Adrien Grand)
 1956 
 1957 * LUCENE-7756: LeafReader.getMetaData now exposes the index created version as
 1958   well as the oldest Lucene version that contributed to the segment.
 1959   (Adrien Grand)
 1960 
 1961 * LUCENE-7854: The new TermFrequencyAttribute used during analysis
 1962   with a custom token stream allows indexing custom term frequencies
 1963   (Mike McCandless)
 1964 
 1965 * LUCENE-7866: Add a new DelimitedTermFrequencyTokenFilter that allows to
 1966   mark tokens with a custom term frequency (LUCENE-7854). It parses a numeric
 1967   value after a separator char ('|') at the end of each token and changes
 1968   the term frequency to this value.  (Uwe Schindler, Robert Muir, Mike
 1969   McCandless)
 1970 
 1971 * LUCENE-7868: Multiple threads can now resolve deletes and doc values
 1972   updates concurrently, giving sizable speedups in update-heavy
 1973   indexing use cases (Simon Willnauer, Mike McCandless)
 1974 
 1975 * LUCENE-7823: Pure query based naive bayes classifier using BM25 scores (Tommaso Teofili)
 1976 
 1977 * LUCENE-7838: Knn classifier based on fuzzified term queries (Tommaso Teofili)
 1978 
 1979 * LUCENE-7855: Added advanced options of the Wikipedia tokenizer to its factory.
 1980   (Juan Pedro via Adrien Grand)
 1981 
 1982 API Changes
 1983 
 1984 * LUCENE-2605: Classic QueryParser no longer splits on whitespace by default.
 1985   Use setSplitOnWhitespace(true) to get the old behavior.  (Steve Rowe)
 1986 
 1987 * LUCENE-7369: Similarity.coord and BooleanQuery.disableCoord are removed.
 1988   (Adrien Grand)
 1989 
 1990 * LUCENE-7368: Removed query normalization. (Adrien Grand)
 1991 
 1992 * LUCENE-7355: AnalyzingQueryParser has been removed as its functionality has
 1993   been folded into the classic QueryParser. (Adrien Grand)
 1994 
 1995 * LUCENE-7407: Doc values APIs have been switched from random access
 1996   to iterators, enabling future codec compression improvements. (Mike
 1997   McCandless)
 1998 
 1999 * LUCENE-7475: Norms now support sparsity, allowing to pay for what is
 2000   actually used. (Adrien Grand)
 2001 
 2002 * LUCENE-7494: Points now have a per-field API, like doc values. (Adrien Grand)
 2003 
 2004 * LUCENE-7410: Cache keys and close listeners have been refactored in order
 2005   to be less trappy. See IndexReader.getReaderCacheHelper and
 2006   LeafReader.getCoreCacheHelper. (Adrien Grand)
 2007 
 2008 * LUCENE-6819: Index-time boosts are not supported anymore. As a replacement,
 2009   index-time scoring factors should be indexed into a doc value field and
 2010   combined at query time using eg. FunctionScoreQuery. (Adrien Grand)
 2011 
 2012 * LUCENE-7734: FieldType's copy constructor was widened to accept any IndexableFieldType.
 2013   (David Smiley)
 2014 
 2015 * LUCENE-7701: Grouping collectors have been refactored, such that groups are
 2016   now defined by a GroupSelector implementation. (Alan Woodward)
 2017 
 2018 * LUCENE-7741: DoubleValuesSource now has an explain() method (Alan Woodward,
 2019   Adrien Grand)
 2020 
 2021 * LUCENE-7815: Removed the PostingsHighlighter; you should use the UnifiedHighlighter
 2022   instead, which derived from the UH.  WholeBreakIterator and
 2023   CustomSeparatorBreakIterator were moved to UH's package. (David Smiley)
 2024 
 2025 * LUCENE-7850: Removed support for legacy numerics. (Adrien Grand)
 2026 
 2027 * LUCENE-7500: Removed abstract LeafReader.fields(); instead terms(fieldName)
 2028   has been made abstract, fomerly was final.  Also, MultiFields.getTerms
 2029   was optimized to work directly instead of being implemented on getFields.
 2030   (David Smiley)
 2031 
 2032 * LUCENE-7872: TopDocs.totalHits is now a long. (Adrien Grand, hossman)
 2033 
 2034 * LUCENE-7868: IndexWriterConfig.setMaxBufferedDeleteTerms is
 2035   removed. (Simon Willnauer, Mike McCandless)
 2036 
 2037 * LUCENE-7877: PrefixAwareTokenStream is replaced with ConcatenatingTokenStream
 2038   (Alan Woodward, Uwe Schindler, Adrien Grand)
 2039 
 2040 * LUCENE-7867: The deprecated Token class is now only available in the test
 2041   framework (Alan Woodward, Adrien Grand)
 2042 
 2043 * LUCENE-7723: DoubleValuesSource enforces implementation of equals() and
 2044   hashCode() (Alan Woodward)
 2045 
 2046 * LUCENE-7737: The spatial-extras module no longer has a dependency on the
 2047   queries module.  All uses of ValueSource are either replaced with core
 2048   DoubleValuesSource extensions, or with the new ShapeValuesSource and
 2049   ShapeValuesPredicate classes (Alan Woodward, David Smiley)
 2050 
 2051 * LUCENE-7892: Doc-values query factory methods have been renamed so that their
 2052   name contains "slow" in order to cleary indicate that they would usually be a
 2053   bad choice. (Adrien Grand)
 2054 
 2055 * LUCENE-7899: FieldValueQuery is renamed to DocValuesFieldExistsQuery
 2056   (Adrien Grand, Mike McCandless)
 2057 
 2058 Bug Fixes
 2059 
 2060 * LUCENE-7626: IndexWriter will no longer accept broken token offsets
 2061   (Mike McCandless)
 2062 
 2063 * LUCENE-7859: Spatial-extras PackedQuadPrefixTree bug that only revealed itself
 2064   with the new pointsOnly optimizations in LUCENE-7845. (David Smiley)
 2065 
 2066 * LUCENE-7871: fix false positive match in BlockJoinSelector when children have no value, introducing 
 2067   wrap methods accepting children as DISI. Extracting ToParentDocValues (Mikhail Khludnev)
 2068 
 2069 * LUCENE-7914: Add a maximum recursion level in automaton recursive
 2070   functions (Operations.isFinite and Operations.topsortState) to prevent
 2071   large automaton to overflow the stack (Robert Muir, Adrien Grand, Jim Ferenczi)
 2072 
 2073 * LUCENE-7864: IndexMergeTool is not using intermediate hard links (even
 2074   if possible). (Dawid Weiss)
 2075 
 2076 * LUCENE-7956: Fixed potential stack overflow error in ICUNormalizer2CharFilter.
 2077   (Adrien Grand)
 2078 
 2079 * LUCENE-7963: Remove useless getAttribute() in DefaultIndexingChain that
 2080   causes performance drop, introduced by LUCENE-7626.  (Daniel Mitterdorfer
 2081   via Uwe Schindler)
 2082 
 2083 Improvements
 2084 
 2085 * LUCENE-7489: Better storage of sparse doc-values fields with the default
 2086   codec. (Adrien Grand)
 2087 
 2088 * LUCENE-7730: More accurate encoding of the length normalization factor
 2089   thanks to the removal of index-time boosts. (Adrien Grand)
 2090 
 2091 * LUCENE-7901: Original Highlighter now eagerly throws an exception if you
 2092   provide components that are null. (Jason Gerlowski, David Smiley)
 2093 
 2094 * LUCENE-7841: Normalize ґ to г in Ukrainian analyzer. (Andriy Rysin via Dawid Weiss)
 2095 
 2096 Optimizations
 2097 
 2098 * LUCENE-7416: BooleanQuery optimizes queries that have queries that occur both
 2099   in the sets of SHOULD and FILTER clauses, or both in MUST/FILTER and MUST_NOT
 2100   clauses. (Spyros Kapnissis via Adrien Grand, Uwe Schindler)
 2101 
 2102 * LUCENE-7506: FastTaxonomyFacetCounts should use CPU in proportion to
 2103   the size of the intersected set of hits from the query and documents
 2104   that have a facet value, so sparse faceting works as expected
 2105   (Adrien Grand via Mike McCandless)
 2106 
 2107 * LUCENE-7519: Add optimized APIs to compute browse-only top level
 2108   facets (Mike McCandless)
 2109 
 2110 * LUCENE-7589: Numeric doc values now have the ability to encode blocks of
 2111   values using different numbers of bits per value if this proves to save
 2112   storage. (Adrien Grand)
 2113 
 2114 * LUCENE-7845: Enhance spatial-extras RecursivePrefixTreeStrategy queries when the
 2115   query is a point (for 2D) or a is a simple date interval (e.g. 1 month).  When
 2116   the strategy is marked as pointsOnly, the results is a TermQuery. (David Smiley)
 2117 
 2118 * LUCENE-7874: DisjunctionMaxQuery rewrites to a BooleanQuery when tiebreaker is set to 1. (Jim Ferenczi)
 2119 
 2120 * LUCENE-7828: Speed up range queries on range fields by improving how we
 2121   compute the relation between the query and inner nodes of the BKD tree.
 2122   (Adrien Grand)
 2123 
 2124 Other
 2125 
 2126 * LUCENE-7923: Removed FST.Arc.node field (unused). (Dawid Weiss)
 2127 
 2128 * LUCENE-7328: Remove LegacyNumericEncoding from GeoPointField. (Nick Knize)
 2129 
 2130 * LUCENE-7360: Remove Explanation.toHtml() (Alan Woodward)
 2131 
 2132 * LUCENE-7681: MemoryIndex uses new DocValues API (Alan Woodward)
 2133 
 2134 * LUCENE-7753: Make fields static when possible.
 2135   (Daniel Jelinski via Adrien Grand)
 2136 
 2137 * LUCENE-7540: Upgrade ICU to 59.1 (Mike McCandless, Jim Ferenczi)
 2138 
 2139 * LUCENE-7852: Correct copyright year(s) in lucene/LICENSE.txt file.
 2140   (Christine Poerschke, Steve Rowe)
 2141 
 2142 * LUCENE-7719: Generalized the UnifiedHighlighter's support for AutomatonQuery
 2143   for character & binary automata. Added AutomatonQuery.isBinary. (David Smiley)
 2144 
 2145 * LUCENE-7873: Due to serious problems with context class loaders in several
 2146   frameworks (OSGI, Java 9 Jigsaw), the lookup of Codecs, PostingsFormats,
 2147   DocValuesFormats and all analysis factories was changed to only inspect the
 2148   current classloader that defined the interface class (lucene-core.jar).
 2149   See MIGRATE.txt for more information!  (Uwe Schindler, Dawid Weiss)
 2150 
 2151 * LUCENE-7883: Lucene no longer uses the context class loader when resolving
 2152   resources in CustomAnalyzer or ClassPathResourceLoader. Resources are only
 2153   resolved against Lucene's class loader by default. Please use another builder
 2154   method to change to a custom classloader.  (Uwe Schindler)
 2155 
 2156 * LUCENE-5822: Convert README to Markdown (Jason Gerlowski via Mike Drob)
 2157 
 2158 * LUCENE-7773: Remove unused/deprecated token types from StandardTokenizer.
 2159   (Ahmet Arslan via Steve Rowe)
 2160 
 2161 * LUCENE-7800: Remove code that potentially rethrows checked exceptions
 2162   from methods that don't declare them ("sneaky throw" hack). (Robert Muir,
 2163   Uwe Schindler, Dawid Weiss)
 2164 
 2165 * LUCENE-7876: Avoid calls to LeafReader.fields() and MultiFields.getFields()
 2166   that are trivially replaced by LeafReader.terms() and MultiFields.getTerms()
 2167   (David Smiley)
 2168 
 2169 ======================= Lucene 6.6.5 =======================
 2170 (No Changes)
 2171 
 2172 ======================= Lucene 6.6.4 =======================
 2173 (No Changes)
 2174 
 2175 ======================= Lucene 6.6.3 =======================
 2176 
 2177 Build
 2178 
 2179 * LUCENE-6144: Upgrade Ivy to 2.4.0; 'ant ivy-bootstrap' now removes old Ivy
 2180   jars in ~/.ant/lib/.  (Shawn Heisey, Steve Rowe)
 2181 
 2182 ======================= Lucene 6.6.2 =======================
 2183 
 2184 Changes in Runtime Behavior
 2185 
 2186 * Resolving of external entities in queryparser/xml/CoreParser is disallowed
 2187   by default. See SOLR-11477 for details.
 2188 
 2189 Bug Fixes
 2190 
 2191 * SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser
 2192   by default. (Michael Stepankin, Olga Barinova, Uwe Schindler, Christine Poerschke)
 2193 
 2194 ======================= Lucene 6.6.1 =======================
 2195 
 2196 Bug Fixes
 2197 
 2198 * LUCENE-7869: Changed MemoryIndex to sort 1d points. In case of 1d points, the PointInSetQuery.MergePointVisitor expects
 2199   that these points are visited in ascending order. The memory index doesn't do this and this can result in document
 2200   with multiple points that should match to not match. (Martijn van Groningen)
 2201 
 2202 * LUCENE-7878: Fix query builder to keep the SHOULD clause that wraps multi-word synonyms. (Jim Ferenczi)
 2203 
 2204 ======================= Lucene 6.6.0 =======================
 2205 
 2206 New Features
 2207 
 2208 * LUCENE-7811: Add a concurrent SortedSet facets implementation.
 2209   (Mike McCandless)
 2210 
 2211 Bug Fixes
 2212 
 2213 * LUCENE-7777: ByteBlockPool.readBytes sometimes throws
 2214   ArrayIndexOutOfBoundsException when byte blocks larger than 32 KB
 2215   were added (Mike McCandless)
 2216 
 2217 * LUCENE-7797: The static FSDirectory.listAll(Path) method was always
 2218   returning an empty array.  (Atkins Chang via Mike McCandless)
 2219 
 2220 * LUCENE-7481: Fixed missing rewrite methods for SpanPayloadCheckQuery
 2221   and PayloadScoreQuery. (Erik Hatcher)
 2222 
 2223 * LUCENE-7808: Fixed PayloadScoreQuery and SpanPayloadCheckQuery
 2224   .equals and .hashCode methods.  (Erik Hatcher)
 2225 
 2226 * LUCENE-7798: Add .equals and .hashCode to ToParentBlockJoinSortField 
 2227   (Mikhail Khludnev)
 2228 
 2229 * LUCENE-7814: DateRangePrefixTree (in spatial-extras) had edge-case bugs for
 2230   years >= 292,000,000. (David Smiley)
 2231 
 2232 * LUCENE-5365, LUCENE-7818: Fix incorrect condition in queryparser's
 2233   QueryNodeOperation#logicalAnd().  (Olivier Binda, Amrit Sarkar,
 2234   AppChecker via Uwe Schindler)
 2235   
 2236 * LUCENE-7821: The classic and flexible query parsers, as well as Solr's
 2237  "lucene"/standard query parser, should require " TO " in range queries,
 2238   and accept "TO" as endpoints in range queries. (hossman, Steve Rowe)
 2239 
 2240 * LUCENE-7824: Fix graph query analysis for multi-word synonym rules with common terms (eg. new york, new york city).
 2241   (Jim Ferenczi)
 2242 
 2243 * LUCENE-7817: Pass cached query to onQueryCache instead of null.
 2244   (Christoph Kaser via Adrien Grand)
 2245 
 2246 * LUCENE-7831: CodecUtil should not seek to negative offsets. (Adrien Grand)
 2247 
 2248 * LUCENE-7833: ToParentBlockJoinQuery computed the min score instead of the max
 2249   score with ScoreMode.MAX. (Adrien Grand)
 2250 
 2251 * LUCENE-7847: Fixed all-docs-match optimization of range queries on range
 2252   fields. (Adrien Grand)
 2253 
 2254 * LUCENE-7810: Fix equals() and hashCode() methods of several join queries.
 2255   (Hossman, Adrien Grand, Martijn van Groningen)
 2256 
 2257 Improvements
 2258 
 2259 * LUCENE-7782: OfflineSorter now passes the total number of items it
 2260   will write to getWriter (Mike McCandless)
 2261 
 2262 * LUCENE-7785: Move dictionary for Ukrainian analyzer to external dependency.
 2263   (Andriy Rysin via Steve Rowe, Dawid Weiss)
 2264 
 2265 * LUCENE-7801: SortedSetDocValuesReaderState now implements
 2266   Accountable so you can see how much RAM it's using (Robert Muir,
 2267   Mike McCandless)
 2268 
 2269 * LUCENE-7792: OfflineSorter can now run concurrently if you pass it
 2270   an optional ExecutorService (Dawid Weiss, Mike McCandless)
 2271 
 2272 * LUCENE-7811: Sorted set facets now use sparse storage when
 2273   collecting hits, when appropriate.  (Mike McCandless)
 2274 
 2275 Optimizations
 2276 
 2277 * LUCENE-7787: spatial-extras HeatmapFacetCounter will now short-circuit it's
 2278   work when Bits.MatchNoBits is passed. (David Smiley)
 2279 
 2280 Other
 2281 
 2282 * LUCENE-7796: Make IOUtils.reThrow idiom declare Error return type so 
 2283   callers may use it in a way that compiler knows subsequent code is 
 2284   unreachable. reThrow is now deprecated in favor of IOUtils.rethrowAlways
 2285   with a slightly different semantics (see javadoc). (Hossman, Robert Muir, 
 2286   Dawid Weiss)
 2287 
 2288 * LUCENE-7754: Inner classes should be static whenever possible.
 2289   (Daniel Jelinski via Adrien Grand)
 2290 
 2291 * LUCENE-7751: Avoid boxing primitives only to call compareTo.
 2292   (Daniel Jelinski via Adrien Grand)
 2293 
 2294 * LUCENE-7743: Never call new String(String).
 2295   (Daniel Jelinski via Adrien Grand)
 2296 
 2297 * LUCENE-7761: Fixed comment in ReqExclScorer.
 2298   (Pablo Pita Leira via Adrien Grand)
 2299 
 2300 ======================= Lucene 6.5.1 =======================
 2301 
 2302 Bug Fixes
 2303 
 2304 * LUCENE-7755: Fixed join queries to not reference IndexReaders, as it could
 2305   cause leaks if they are cached. (Adrien Grand)
 2306 
 2307 * LUCENE-7749: Made LRUQueryCache delegate the scoreSupplier method.
 2308   (Martin Amirault via Adrien Grand)
 2309 
 2310 * LUCENE-7769: The UnifiedHighligter wasn't highlighting portions of the query
 2311   wrapped in BoostQuery or SpanBoostQuery. (David Smiley, Dmitry Malinin)
 2312 
 2313 Other
 2314 
 2315 * LUCENE-7763: Remove outdated comment in IndexWriterConfig.setIndexSort javadocs.
 2316   (马可阳 via Christine Poerschke)
 2317 
 2318 ======================= Lucene 6.5.0 =======================
 2319 
 2320 API Changes
 2321 
 2322 * LUCENE-7740: Refactor Range Fields to remove Field suffix (e.g., DoubleRange),
 2323   move InetAddressRange and InetAddressPoint from sandbox to misc module, and
 2324   refactor all other range fields from sandbox to core. (Nick Knize)
 2325 
 2326 * LUCENE-7624: TermsQuery has been renamed as TermInSetQuery and moved to core.
 2327   (Alan Woodward)
 2328 
 2329 * LUCENE-7637: TermInSetQuery requires that all terms come from the same field.
 2330   (Adrien Grand)
 2331 
 2332 * LUCENE-7644: FieldComparatorSource.newComparator() and
 2333   SortField.getComparator() no longer throw IOException (Alan Woodward)
 2334 
 2335 * LUCENE-7643: Replaced doc-values queries in lucene/sandbox with factory
 2336   methods on the *DocValuesField classes. (Adrien Grand)
 2337 
 2338 * LUCENE-7659: Added a IndexWriter#getFieldNames() method (experimental) to return
 2339   all field names as visible from the IndexWriter. This would be useful for
 2340   IndexWriter#updateDocValues() calls, to prevent calling with non-existent
 2341   docValues fields (Ishan Chattopadhyaya, Adrien Grand, Mike McCandless)
 2342 
 2343 * LUCENE-6959: Removed ToParentBlockJoinCollector in favour of
 2344   ParentChildrenBlockJoinQuery, that can return the matching children documents per
 2345   parent document. This query should be executed for each matching parent document
 2346   after the main query has been executed. (Adrien Grand, Martijn van Groningen,
 2347   Mike McCandless)
 2348 
 2349 * LUCENE-7628: Scorer.getChildren() now only returns Scorers that are
 2350   positioned on the current document, and can throw an IOException.
 2351   AssertingScorer checks that getChildren() is not called on an unpositioned
 2352   Scorer.  (Alan Woodward, Adrien Grand)
 2353 
 2354 * LUCENE-7702: Removed GraphQuery in favour of simple boolean query. (Matt Webber via Jim Ferenczi)
 2355 
 2356 * LUCENE-7707: TopDocs.merge now takes a boolean option telling it
 2357   when to use the incoming shard index versus when to assign the shard
 2358   index itself, allowing users to merge shard responses incrementally
 2359   instead of once all shard responses are present. (Simon Willnauer,
 2360   Mike McCandless)
 2361 
 2362 * LUCENE-7700: A cleanup of merge throughput control logic. Refactored all the
 2363   code previously scattered throughout the IndexWriter and 
 2364   ConcurrentMergeScheduler into a more accessible set of public methods (see 
 2365   MergePolicy.OneMergeProgress, MergeScheduler.wrapForMerge and 
 2366   OneMerge.mergeInit). (Dawid Weiss, Mike McCandless).
 2367 
 2368 * LUCENE-7734: FieldType's copy constructor was widened to accept any IndexableFieldType.
 2369   (David Smiley)
 2370 
 2371 New Features
 2372 
 2373 * LUCENE-7738: Add new InetAddressRange for indexing and querying InetAddress
 2374   ranges. (Nick Knize)
 2375 
 2376 * LUCENE-7449: Add CROSSES relation support to RangeFieldQuery. (Nick Knize)
 2377 
 2378 * LUCENE-7623: Add FunctionScoreQuery and FunctionMatchQuery (Alan Woodward,
 2379   Adrien Grand, David Smiley)
 2380 
 2381 * LUCENE-7619: Add WordDelimiterGraphFilter, just like
 2382   WordDelimiterFilter except it produces correct token graphs so that
 2383   proximity queries at search time will produce correct results (Mike
 2384   McCandless)
 2385 
 2386 * LUCENE-7656: Added the LatLonDocValuesField.new(Box/Distance)Query() factory
 2387   methods that are the equivalent of factory methods on LatLonPoint but operate
 2388   on doc values. These new methods should be wrapped in an IndexOrDocValuesQuery
 2389   for best performance. (Adrien Grand)
 2390 
 2391 * LUCENE-7673: Added MultiValued[Int/Long/Float/Double]FieldSource that given a
 2392   SortedNumericSelector.Type can give a ValueSource view of a 
 2393   SortedNumericDocValues field. (Tomás Fernández Löbbe)
 2394 
 2395 * LUCENE-7465: Add SimplePatternTokenizer and
 2396   SimplePatternSplitTokenizer, using Lucene's regexp/automaton
 2397   implementation for analysis/tokenization (Clinton Gormley, Mike
 2398   McCandless)
 2399 
 2400 * LUCENE-7688: Add OneMergeWrappingMergePolicy class.
 2401   (Keith Laban, Christine Poerschke)
 2402 
 2403 * LUCENE-7686: The near-real-time document suggester can now
 2404   efficiently filter out duplicate suggestions (Uwe Schindler, Mike
 2405   McCandless)
 2406 
 2407 * LUCENE-7712: SimpleQueryParser now supports default fuzziness
 2408   syntax, mapping foo~ to a FuzzyQuery with edit distance 2.  (Lee
 2409   Hinman, David Pilato via Mike McCandless)
 2410 
 2411 Bug Fixes
 2412 
 2413 * LUCENE-7630: Fix (Edge)NGramTokenFilter to no longer drop payloads
 2414   and preserve all attributes. (Nathan Gass via Uwe Schindler)
 2415 
 2416 * LUCENE-7679: MemoryIndex was ignoring omitNorms settings on passed-in
 2417   IndexableFields. (Alan Woodward)
 2418 
 2419 * LUCENE-7692: PatternReplaceCharFilterFactory now implements MultiTermAware.
 2420   (Adrien Grand)
 2421 
 2422 * LUCENE-7685: ToParentBlockJoinQuery and ToChildBlockJoinQuery now use the
 2423   rewritten child query in their equals and hashCode implementations.
 2424   (Adrien Grand)
 2425 
 2426 * LUCENE-7698: CommonGramsQueryFilter was producing a disconnected
 2427   token graph, messing up phrase queries when it was used during query
 2428   parsing (Ere Maijala via Mike McCandless)
 2429 
 2430 * LUCENE-7708: ShingleFilter without unigram was producing a disconnected
 2431   token graph, messing up queries when it was used during query
 2432   parsing (Jim Ferenczi)
 2433 
 2434 Improvements
 2435 
 2436 * LUCENE-7055: Added Weight#scorerSupplier, which allows to estimate the cost
 2437   of a Scorer before actually building it, in order to optimize how the query
 2438   should be run, eg. using points or doc values depending on costs of other
 2439   parts of the query. (Adrien Grand)
 2440 
 2441 * LUCENE-7643: IndexOrDocValuesQuery allows to execute range queries using
 2442   either points or doc values depending on which one is more efficient.
 2443   (Adrien Grand)
 2444 
 2445 * LUCENE-7662: If index files are missing, throw CorruptIndexException instead
 2446   of the less descriptive FileNotFound or NoSuchFileException (Mike Drob via 
 2447   Mike McCandless, Erick Erickson)
 2448 
 2449 * LUCENE-7680: UsageTrackingQueryCachingPolicy never caches term filters anymore
 2450   since they are plenty fast. This also has the side-effect of leaving more
 2451   space in the history for costly filters. (Adrien Grand)
 2452 
 2453 * LUCENE-7677: UsageTrackingQueryCachingPolicy now caches compound queries a bit
 2454   earlier than regular queries in order to improve cache efficiency.
 2455   (Adrien Grand)
 2456 
 2457 * LUCENE-7710: BlockPackedReader throws CorruptIndexException and includes
 2458   IndexInput description instead of plain IOException (Mike Drob via
 2459   Mike McCandless)
 2460 
 2461 * LUCENE-7695: ComplexPhraseQueryParser to support query time synonyms (Markus Jelsma
 2462   via Mikhail Khludnev) 
 2463 
 2464 * LUCENE-7747: QueryBuilder now iterates lazily over the possible paths when building a graph query
 2465   (Jim Ferenczi)
 2466 
 2467 Optimizations
 2468 
 2469 * LUCENE-7641: Optimized point range queries to compute documents that do not
 2470   match the range on single-valued fields when more than half the documents in
 2471   the index would match. (Adrien Grand)
 2472 
 2473 * LUCENE-7656: Speed up for LatLonPointDistanceQuery by computing distances even
 2474   less often. (Adrien Grand)
 2475 
 2476 * LUCENE-7661: Speed up for LatLonPointInPolygonQuery by pre-computing the
 2477   relation of the polygon with a grid. (Adrien Grand)
 2478 
 2479 * LUCENE-7660: Speed up LatLonPointDistanceQuery by improving the detection of
 2480   whether BKD cells are entirely within the distance close to the dateline.
 2481   (Adrien Grand)
 2482 
 2483 * LUCENE-7654: ToParentBlockJoinQuery now implements two-phase iteration and
 2484   computes scores lazily in order to be faster when used in conjunctions.
 2485   (Adrien Grand)
 2486 
 2487 * LUCENE-7667: BKDReader now calls `IntersectVisitor.grow()` on larger
 2488   increments. (Adrien Grand)
 2489 
 2490 * LUCENE-7638: Query parsers now analyze the token graph for articulation
 2491   points (or cut vertices) in order to create more efficient queries for
 2492   multi-token synonyms. (Jim Ferenczi)
 2493 
 2494 * LUCENE-7699: Query parsers now use span queries to produce more efficient
 2495   phrase queries for multi-token synonyms. (Matt Webber via Jim Ferenczi)
 2496 
 2497 * LUCENE-7742: Fix places where we were unboxing and then re-boxing
 2498   according to FindBugs (Daniel Jelinski via Mike McCandless)
 2499 
 2500 * LUCENE-7739: Fix places where we unnecessarily boxed while parsing
 2501   a numeric value according to FindBugs (Daniel Jelinski via Mike
 2502   McCandless)
 2503 
 2504 Build
 2505 
 2506 * LUCENE-7653: Update randomizedtesting to version 2.5.0. (Dawid Weiss)
 2507 
 2508 * LUCENE-7665: Remove grouping dependency from the join module.
 2509   (Martijn van Groningen)
 2510   
 2511 * SOLR-10023: Add non-recursive 'test-nocompile' target: Only runs unit tests.
 2512   Jars are not downloaded; compilation is not updated; and Clover is not enabled.
 2513   (Steve Rowe)
 2514 
 2515 * LUCENE-7694: Update forbiddenapis to version 2.3. (Uwe Schindler)
 2516 
 2517 * LUCENE-7693: Replace "org.apache." logic in GetMavenDependenciesTask.
 2518   (Daniel Collins, Christine Poerschke)
 2519 
 2520 * LUCENE-7726: Fix HTML entity bugs in Javadocs to be able to build with
 2521   Java 9. (Uwe Schindler, Hossman)
 2522 
 2523 * LUCENE-7727: Replace end-of-life Markdown parser "Pegdown" by "Flexmark"
 2524   for compatibility with Java 9. (Uwe Schindler)
 2525 
 2526 Other
 2527 
 2528 * LUCENE-7666: Fix typos in lucene-join package info javadoc.
 2529   (Tom Saleeba via Christine Poerschke)
 2530 
 2531 * LUCENE-7658: queryparser/xml CoreParser now implements SpanQueryBuilder interface.
 2532   (Daniel Collins, Christine Poerschke)
 2533 
 2534 * LUCENE-7715: NearSpansUnordered simplifications.
 2535   (Paul Elschot via Adrien Grand)
 2536 
 2537 ======================= Lucene 6.4.2 =======================
 2538 
 2539 Bug Fixes
 2540 
 2541 * LUCENE-7676: Fixed FilterCodecReader to override more super-class methods.
 2542   Also added TestFilterCodecReader class. (Christine Poerschke)
 2543 
 2544 * LUCENE-7717: The UnifiedHighlighter and PostingsHighlighter were not highlighting
 2545   prefix queries with multi-byte characters. TermRangeQuery is affected too.
 2546   (Dmitry Malinin, David Smiley)
 2547 
 2548 ======================= Lucene 6.4.1 =======================
 2549 
 2550 Build
 2551 
 2552 * LUCENE-7651: Fix Javadocs build for Java 8u121 by injecting "Google Code
 2553   Prettify" without adding Javascript to Javadocs's -bottom parameter.
 2554   Also update Prettify to latest version to fix Google Chrome issue.
 2555   (Uwe Schindler)
 2556 
 2557 Bug Fixes
 2558 
 2559 * LUCENE-7657: Fixed potential memory leak in the case that a (Span)TermQuery
 2560   with a TermContext is cached. (Adrien Grand)
 2561 
 2562 * LUCENE-7647: Made stored fields reclaim native memory more aggressively when
 2563   configured with BEST_COMPRESSION. This could otherwise result in out-of-memory
 2564   issues. (Adrien Grand)
 2565 
 2566 * LUCENE-7670: AnalyzingInfixSuggester should not immediately open an
 2567   IndexWriter over an already-built index. (Steve Rowe)
 2568 
 2569 ======================= Lucene 6.4.0 =======================
 2570 
 2571 API Changes
 2572 
 2573 * LUCENE-7533: Classic query parser no longer allows autoGeneratePhraseQueries
 2574   to be set to true when splitOnWhitespace is false (and vice-versa).
 2575 
 2576 * LUCENE-7607: LeafFieldComparator.setScorer and SimpleFieldComparator.setScorer
 2577   are declared as throwing IOException (Alan Woodward)
 2578 
 2579 * LUCENE-7617: Collector construction for two-pass grouping queries is
 2580   abstracted into a new Grouper class, which can be passed as a constructor
 2581   parameter to GroupingSearch.  The abstract base classes for the different
 2582   grouping Collectors are renamed to remove the Abstract* prefix.
 2583   (Alan Woodward, Martijn van Groningen)
 2584 
 2585 * LUCENE-7609: The expressions module now uses the DoubleValuesSource API, and 
 2586   no longer depends on the queries module.  Expression#getValueSource() is
 2587   replaced with Expression#getDoubleValuesSource(). (Alan Woodward, Adrien
 2588   Grand)
 2589 
 2590 * LUCENE-7610: The facets module now uses the DoubleValuesSource API, and
 2591   methods that take ValueSource parameters are deprecated (Alan Woodward)
 2592 
 2593 * LUCENE-7611: DocumentValueSourceDictionary now takes a LongValuesSource
 2594   as a parameter, and the ValueSource equivalent is deprecated (Alan Woodward)
 2595 
 2596 New features
 2597 
 2598 * LUCENE-5867: Added BooleanSimilarity. (Robert Muir, Adrien Grand)
 2599 
 2600 * LUCENE-7466: Added AxiomaticSimilarity. (Peilin Yang via Tommaso Teofili)
 2601 
 2602 * LUCENE-7590: Added DocValuesStatsCollector to compute statistics on DocValues
 2603   fields. (Shai Erera)
 2604 
 2605 * LUCENE-7587: The new FacetQuery and MultiFacetQuery helper classes
 2606   make it simpler to execute drill down when drill sideways counts are
 2607   not needed (Emmanuel Keller via Mike McCandless)
 2608 
 2609 * LUCENE-6664: A new SynonymGraphFilter outputs a correct graph
 2610   structure for multi-token synonyms, separating out a
 2611   FlattenGraphFilter that is hardwired into the current
 2612   SynonymFilter.  This finally makes it possible to implement
 2613   correct multi-token synonyms at search time.  See
 2614   http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html
 2615   for details. (Mike McCandless)
 2616 
 2617 * LUCENE-5325: Added LongValuesSource and DoubleValuesSource, intended as
 2618   type-safe replacements for ValueSource in the queries module.  These
 2619   expose per-segment LongValues or DoubleValues iterators. (Alan Woodward, Adrien Grand)
 2620 
 2621 * LUCENE-7603: Graph token streams are now handled accurately by query
 2622   parsers, by enumerating all paths and creating the corresponding
 2623   query/ies as sub-clauses (Matt Weber via Mike McCandless)
 2624 
 2625 * LUCENE-7588: DrillSideways can now run queries concurrently, and
 2626   supports an IndexSearcher using an executor service to run each query
 2627   concurrently across all segments in the index (Emmanuel Keller via
 2628   Mike McCandless)
 2629 
 2630 * LUCENE-7627: Added .intersect methods to SortedDocValues and 
 2631   SortedSetDocValues to allow filtering their TermsEnums with a
 2632   CompiledAutomaton (Alan Woodward, Mike McCandless)
 2633 
 2634 Bug Fixes
 2635 
 2636 * LUCENE-7547: JapaneseTokenizerFactory was failing to close the
 2637   dictionary file it opened (Markus via Mike McCandless)
 2638 
 2639 * LUCENE-7562: CompletionFieldsConsumer sometimes throws
 2640   NullPointerException on ghost fields (Oliver Eilhard via Mike McCandless)
 2641   
 2642 * LUCENE-7533: Classic query parser: disallow autoGeneratePhraseQueries=true
 2643   when splitOnWhitespace=false (and vice-versa). (Steve Rowe)
 2644 
 2645 * LUCENE-7536: ASCIIFoldingFilterFactory used to return an illegal multi-term
 2646   component when preserveOriginal was set to true. (Adrien Grand)
 2647 
 2648 * LUCENE-7576: Fix Terms.intersect in the default codec to detect when
 2649   the incoming automaton is a special case and throw a clearer
 2650   exception than NullPointerException (Tom Mortimer via Mike McCandless)
 2651 
 2652 * LUCENE-6989: Fix Exception handling in MMapDirectory's unmap hack
 2653   support code to work with Java 9's new InaccessibleObjectException
 2654   that does not extend ReflectiveAccessException in Java 9.
 2655   (Uwe Schindler)
 2656 
 2657 * LUCENE-7581: Lucene now prevents updating a doc values field that is used
 2658   in the index sort, since this would lead to corruption.  (Jim
 2659   Ferenczi via Mike McCandless)
 2660 
 2661 * LUCENE-7570: IndexWriter may deadlock if a commit is running while
 2662   there are too many merges running and one of the merges hits a
 2663   tragic exception (Joey Echeverria via Mike McCandless)
 2664 
 2665 * LUCENE-7594: Fixed point range queries on floating-point types to recommend
 2666   using helpers for exclusive bounds that are consistent with Double.compare.
 2667   (Adrien Grand, Dawid Weiss)
 2668 
 2669 * LUCENE-7606: Normalization with CustomAnalyzer would only apply the last
 2670   token filter. (Adrien Grand)
 2671 
 2672 * LUCENE-7612: Removed an unused dependency from the suggester to the misc
 2673   module. (Alan Woodward)
 2674 
 2675 Improvements
 2676 
 2677 * LUCENE-7532: Add back lost codec file format documentation
 2678   (Shinichiro Abe via Mike McCandless)
 2679 
 2680 * LUCENE-6824: TermAutomatonQuery now rewrites to TermQuery,
 2681   PhraseQuery or MultiPhraseQuery when the word automaton is simple
 2682   (Mike McCandless)
 2683 
 2684 * LUCENE-7431: Allow a certain amount of overlap to be specified between the include
 2685   and exclude arguments of SpanNotQuery via negative pre and/or post arguments.
 2686   (Marc Morissette via David Smiley)
 2687 
 2688 * LUCENE-7544: UnifiedHighlighter: add extension points for handling custom queries.
 2689   (Michael Braun, David Smiley)
 2690 
 2691 * LUCENE-7538: Asking IndexWriter to store a too-massive text field
 2692   now throws IllegalArgumentException instead of a cryptic exception
 2693   that closes your IndexWriter (Steve Chen via Mike McCandless)
 2694 
 2695 * LUCENE-7524: Added more detailed explanation of how IDF is computed in
 2696   ClassicSimilarity and BM25Similarity. (Adrien Grand)
 2697   
 2698 * LUCENE-7564: AnalyzingInfixSuggester should close its IndexWriter by default
 2699   at the end of build(). (Steve Rowe)
 2700 
 2701 * LUCENE-7526: Enhanced UnifiedHighlighter's passage relevancy for queries with
 2702   wildcards and sometimes just terms. Added shouldPreferPassageRelevancyOverSpeed()
 2703   which can be overridden to return false to eek out more speed in some cases.
 2704   (Timothy M. Rodriguez, David Smiley)
 2705 
 2706 * LUCENE-7560: QueryBuilder.createFieldQuery is no longer final,
 2707   giving custom query parsers subclassing QueryBuilder more freedom to
 2708   control how text is analyzed and converted into a query (Matt Weber
 2709   via Mike McCandless)
 2710 
 2711 * LUCENE-7537: Index time sorting now supports multi-valued sorts
 2712   using selectors (MIN, MAX, etc.) (Jim Ferenczi via Mike McCandless)
 2713 
 2714 * LUCENE-7575: UnifiedHighlighter can now highlight fields with queries that don't
 2715   necessarily refer to that field (AKA requireFieldMatch==false). Disabled by default.
 2716   See UH get/setFieldMatcher. (Jim Ferenczi via David Smiley)
 2717 
 2718 * LUCENE-7592: If the segments file is truncated, we now throw
 2719   CorruptIndexException instead of the more confusing EOFException
 2720   (Mike Drob via Mike McCandless)
 2721 
 2722 * LUCENE-6989: Make MMapDirectory's unmap hack work with Java 9 EA (b150+):
 2723   Unmapping uses new sun.misc.Unsafe#invokeCleaner(ByteBuffer).
 2724   Java 9 now needs same permissions like Java 8;
 2725   RuntimePermission("accessClassInPackage.jdk.internal.ref")
 2726   is no longer needed. Support for older Java 9 builds was removed.
 2727   (Uwe Schindler)
 2728 
 2729 * LUCENE-7401: Changed the way BKD trees pick the split dimension in order to
 2730   ensure all dimensions are indexed. (Adrien Grand)
 2731 
 2732 * LUCENE-7614: Complex Phrase Query parser ignores double quotes around single token 
 2733   prefix, wildcard, range queries (Mikhail Khludnev)
 2734 
 2735 * LUCENE-7620: Added LengthGoalBreakIterator, a wrapper around another B.I. to skip breaks
 2736   that would create Passages that are too short.  Only for use with the UnifiedHighlighter
 2737   (and probably PostingsHighlighter).  (David Smiley)
 2738 
 2739 Optimizations
 2740 
 2741 * LUCENE-7568: Optimize merging when index sorting is used but the
 2742   index is already sorted (Jim Ferenczi via Mike McCandless)
 2743 
 2744 * LUCENE-7563: The BKD in-memory index for dimensional points now uses
 2745   a compressed format, using substantially less RAM in some cases
 2746   (Adrien Grand, Mike McCandless)
 2747 
 2748 * LUCENE-7583: BKD writing now buffers each leaf block in heap before
 2749   writing to disk, giving a small speedup in points-heavy use cases.
 2750   (Mike McCandless)
 2751 
 2752 * LUCENE-7572: Doc values queries now cache their hash code. (Adrien Grand)
 2753 
 2754 Other
 2755 
 2756 * LUCENE-7546: Fixed references to benchmark wikipedia data and the Jenkins line-docs file
 2757   (David Smiley)
 2758 
 2759 * LUCENE-7534: fix smokeTestRelease.py to run on Cygwin (Mikhail Khludnev)
 2760 
 2761 * LUCENE-7559: UnifiedHighlighter: Make Passage and OffsetsEnum more exposed to allow
 2762   passage creation to be customized. (David Smiley)
 2763 
 2764 * LUCENE-7599: Simplify TestRandomChains using Java's built-in Predicate and
 2765   Function interfaces. (Ahmet Arslan via Adrien Grand)
 2766 
 2767 * LUCENE-7595: Improve RAMUsageTester in test-framework to estimate memory usage of
 2768   runtime classes and work with Java 9 EA (b148+). Disable static field heap usage
 2769   checker in LuceneTestCase.  (Uwe Schindler, Dawid Weiss)
 2770 
 2771 Build
 2772 
 2773 * LUCENE-7387: fix defaultCodec in build.xml to account for the line ending (hossman)
 2774 
 2775 * LUCENE-7543: Make changes-to-html target an offline operation, by moving the
 2776   Lucene and Solr DOAP RDF files into the Git source repository under
 2777   dev-tools/doap/ and then pulling release dates from those files, rather than
 2778   from JIRA. (Mano Kovacs, hossman, Steve Rowe)
 2779   
 2780 * LUCENE-7596: Update Groovy to version 2.4.8 to allow building with Java 9
 2781   build 148+. Also update JGit version for working-copy checks. (Uwe Schindler)
 2782 
 2783 ======================= Lucene 6.3.0 =======================
 2784 
 2785 API Changes
 2786 
 2787 New Features
 2788 
 2789 * LUCENE-7438: New "UnifiedHighlighter" derivative of the PostingsHighlighter that
 2790   can consume offsets from postings, term vectors, or analysis.  It can highlight phrases
 2791   as accurately as the standard Highlighter. Light term vectors can be used with offsets
 2792   in postings for fast wildcard (MultiTermQuery) highlighting.
 2793   (David Smiley, Timothy Rodriguez)
 2794 
 2795 * LUCENE-7490: SimpleQueryParser now parses '*' to MatchAllDocsQuery
 2796   (Lee Hinman via Mike McCandless)
 2797 
 2798 Bug Fixes
 2799 
 2800 * LUCENE-7507: Upgrade morfologik-stemming to version 2.1.1 (fixes security
 2801   manager issue with Polish dictionary lookup). (Dawid Weiss)
 2802 
 2803 * LUCENE-7472: MultiFieldQueryParser.getFieldQuery() drops queries that are
 2804   neither BooleanQuery nor TermQuery.  (Steve Rowe)
 2805 
 2806 * LUCENE-7456: PerFieldPostings/DocValues was failing to delegate the
 2807   merge method (Julien MASSENET via Mike McCandless)
 2808 
 2809 * LUCENE-7468: ASCIIFoldingFilter should not emit duplicated tokens when
 2810   preserve original is on. (David Causse via Adrien Grand)
 2811 
 2812 * LUCENE-7484: FastVectorHighlighter failed to highlight SynonymQuery
 2813   (Jim Ferenczi via Mike McCandless)
 2814 
 2815 * LUCENE-7476: JapaneseNumberFilter should not invoke incrementToken
 2816   on its input after it's exhausted (Andy Hind via Mike McCandless)
 2817 
 2818 * LUCENE-7486: DisjunctionMaxQuery does not work correctly with queries that
 2819   return negative scores.  (Ivan Provalov, Uwe Schindler, Adrien Grand)
 2820 
 2821 * LUCENE-7491: Suddenly turning on dimensional points for some fields
 2822   that already exist in an index but didn't previously index
 2823   dimensional points could cause unexpected merge exceptions (Hans
 2824   Lund, Mike McCandless)
 2825 
 2826 * LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points.
 2827   (Hossman)
 2828 
 2829 * LUCENE-7493: FacetCollector.search threw an unexpected exception if
 2830   you asked for zero hits but wanted facets (Mahesh via Mike McCandless)
 2831 
 2832 * LUCENE-7505: AnalyzingInfixSuggester returned invalid results when
 2833   allTermsRequired is false and context filters are specified (Mike
 2834   McCandless)
 2835 
 2836 * LUCENE-7429: AnalyzerWrapper can now modify the normalization chain too and
 2837   DelegatingAnalyzerWrapper does the right thing automatically. (Adrien Grand)
 2838 
 2839 * LUCENE-7135: Lucene's check for 32 or 64 bit JVM now works around security
 2840   manager blocking access to some properties (Aaron Madlon-Kay via
 2841   Mike McCandless)
 2842 
 2843 Improvements
 2844 
 2845 * LUCENE-7439: FuzzyQuery now matches all terms within the specified
 2846   edit distance, even if they are short terms (Mike McCandless)
 2847 
 2848 * LUCENE-7496: Better toString for SweetSpotSimilarity (janhoy)
 2849 
 2850 * LUCENE-7520: Highlighter's WeightedSpanTermExtractor shouldn't attempt to expand a MultiTermQuery
 2851   when its field doesn't match the field the extraction is scoped to.
 2852   (Cao Manh Dat via David Smiley)
 2853 
 2854 Optimizations
 2855 
 2856 * LUCENE-7501: BKDReader should not store the split dimension explicitly in the
 2857   1D case. (Adrien Grand)
 2858 
 2859 Other
 2860 
 2861 * LUCENE-7513: Upgrade randomizedtesting to 2.4.0. (Dawid Weiss)
 2862 
 2863 * LUCENE-7452: Block join query exception suggests how to find a doc, which 
 2864  violates orthogonality requirement. (Mikhail Khludnev)
 2865 
 2866 * LUCENE-7438: Renovate the Benchmark module's support for benchmarking highlighting. All
 2867   highlighters are supported via SearchTravRetHighlight. (David Smiley)
 2868 
 2869 Build
 2870 
 2871 * LUCENE-7292: Fix build to use "--release 8" instead of "-release 8" on
 2872   Java 9 (this changed with recent EA build b135).  (Uwe Schindler)
 2873 
 2874 ======================= Lucene 6.2.1 =======================
 2875 
 2876 API Changes
 2877 
 2878 * LUCENE-7436: MinHashFilter's constructor, and some of its default
 2879   settings, should be public.  (Doug Turnbull via Mike McCandless)
 2880 
 2881 Bug Fixes
 2882 
 2883 * LUCENE-7417: The standard Highlighter could throw an IllegalArgumentException when
 2884   trying to highlight a query containing a degenerate case of a MultiPhraseQuery with one
 2885   term.  (Thomas Kappler via David Smiley)
 2886 
 2887 * LUCENE-7440: Document id skipping (PostingsEnum.advance) could throw an
 2888   ArrayIndexOutOfBoundsException exception on large index segments (>1.8B docs)
 2889   with large skips. (yonik)
 2890 
 2891 * LUCENE-7442: MinHashFilter's ctor should validate its args.
 2892   (Cao Manh Dat via Steve Rowe)
 2893 
 2894 * LUCENE-7318: Fix backwards compatibility issues around StandardAnalyzer
 2895   and its components, introduced with Lucene 6.2.0. The moved classes
 2896   were restored in their original packages: LowercaseFilter and StopFilter,
 2897   as well as several utility classes.  (Uwe Schindler, Mike McCandless)
 2898 
 2899 ======================= Lucene 6.2.0 =======================
 2900 
 2901 API Changes
 2902 
 2903 * ScoringWrapperSpans was removed since it had no purpose or effect as of Lucene 5.5.
 2904 
 2905 New Features
 2906 
 2907 * LUCENE-7388: Add point based IntRangeField, FloatRangeField, LongRangeField along with
 2908   supporting queries and tests (Nick Knize)
 2909 
 2910 * LUCENE-7381: Add point based DoubleRangeField and RangeFieldQuery for
 2911   indexing and querying on Ranges up to 4 dimensions (Nick Knize)
 2912 
 2913 * LUCENE-6968: LSH Filter (Tommaso Teofili, Andy Hind, Cao Manh Dat)
 2914 
 2915 * LUCENE-7302: IndexWriter methods that change the index now return a
 2916   long "sequence number" indicating the effective equivalent
 2917   single-threaded execution order (Mike McCandless)
 2918 
 2919 * LUCENE-7335: IndexWriter's commit data is now late binding,
 2920   recording key/values from a provided iterable based on when the
 2921   commit actually takes place (Mike McCandless)
 2922 
 2923 * LUCENE-7287: UkrainianMorfologikAnalyzer is a new dictionary-based
 2924   analyzer for the Ukrainian language (Andriy Rysin via Mike
 2925   McCandless)
 2926 
 2927 * LUCENE-7373: Directory.renameFile, which did both renaming and fsync
 2928   of the directory metadata, has been deprecated; use the new separate
 2929   methods Directory.rename and Directory.syncMetaData instead (Robert Muir,
 2930   Uwe Schindler, Mike McCandless)
 2931 
 2932 * LUCENE-7355: Added Analyzer#normalize(), which only applies normalization to
 2933   an input string. (Adrien Grand)
 2934 
 2935 * LUCENE-7380: Add Polygon.fromGeoJSON for more easily creating
 2936   Polygon instances from a standard GeoJSON string (Robert Muir, Mike
 2937   McCandless)
 2938 
 2939 * LUCENE-7395: PerFieldSimilarityWrapper requires a default similarity
 2940   for calculating query norm and coordination factor in Lucene 6.x.
 2941   Lucene 7 will no longer have those factors.  (Uwe Schindler, Sascha Markus)
 2942 
 2943 * SOLR-9279: Queries module: new ComparisonBoolFunction base class
 2944   (Doug Turnbull via David Smiley)
 2945 
 2946 Bug Fixes
 2947 
 2948 * LUCENE-6662: Fixed potential resource leaks. (Rishabh Patel via Adrien Grand)
 2949 
 2950 * LUCENE-7340: MemoryIndex.toString() could throw NPE; fixed. Renamed to toStringDebug().
 2951   (Daniel Collins, David Smiley)
 2952 
 2953 * LUCENE-7382: Fix bug introduced by LUCENE-7355 that used the
 2954   wrong default AttributeFactory for new Tokenizers.
 2955   (Terry Smith, Uwe Schindler)
 2956 
 2957 * LUCENE-7389: Fix FieldType.setDimensions(...) validation for the dimensionNumBytes
 2958   parameter. (Martijn van Groningen)
 2959 
 2960 * LUCENE-7391: Fix performance regression in MemoryIndex's fields() introduced
 2961   in Lucene 6. (Steve Mason via David Smiley)
 2962 
 2963 * LUCENE-7395, SOLR-9315: Fix PerFieldSimilarityWrapper to also delegate query
 2964   norm and coordination factor using a default similarity added as ctor param.
 2965   (Uwe Schindler, Sascha Markus)
 2966 
 2967 * SOLR-9413: Fix analysis/kuromoji's CSVUtil.quoteEscape logic, add TestCSVUtil test.
 2968   (AppChecker, Christine Poerschke)
 2969 
 2970 * LUCENE-7419: Fix performance bug with TokenStream.end(), where it would lookup
 2971   PositionIncrementAttribute every time. (Mike McCandless, Robert Muir)
 2972 
 2973 Improvements
 2974 
 2975 * LUCENE-7323: Compound file writing now verifies the incoming
 2976   sub-files' checkums and segment IDs, to catch hardware issues or
 2977   filesytem bugs earlier (Robert Muir, Mike McCandless)
 2978 
 2979 * LUCENE-6766: Index time sorting has graduated from the misc module
 2980   to core, is much simpler to use, via
 2981   IndexWriter.setIndexSort, and now works with dimensional points.
 2982   (Adrien Grand, Mike McCandless)
 2983 
 2984 * LUCENE-5931: Detect when an application tries to reopen an
 2985   IndexReader after (illegally) removing the old index and
 2986   reindexing (Vitaly Funstein, Robert Muir, Mike McCandless)
 2987 
 2988 * LUCENE-6171: Lucene now passes the StandardOpenOption.CREATE_NEW
 2989   option when writing new files so the filesystem enforces our
 2990   write-once architecture, possibly catching externally caused
 2991   issues sooner (Robert Muir, Mike McCandless)
 2992 
 2993 * LUCENE-7318: StandardAnalyzer has been moved from the analysis
 2994   module into core and is now the default analyzer in
 2995   IndexWriterConfig (Robert Muir, Mike McCandless)
 2996 
 2997 * LUCENE-7345: RAMDirectory now enforces write-once files as well
 2998   (Robert Muir, Mike McCandless)
 2999 
 3000 * LUCENE-7337: MatchNoDocsQuery now scores with 0 normalization factor
 3001   and empty boolean queries now rewrite to MatchNoDocsQuery instead of
 3002   vice/versa (Jim Ferenczi via Mike McCandless)
 3003 
 3004 * LUCENE-7359: Add equals() and hashCode() to Explanation (Alan Woodward)
 3005 
 3006 * LUCENE-7353: ScandinavianFoldingFilterFactory and
 3007   ScandinavianNormalizationFilterFactory now implement MultiTermAwareComponent.
 3008   (Adrien Grand)
 3009 
 3010 * LUCENE-2605: Add classic QueryParser option setSplitOnWhitespace() to
 3011   control whether to split on whitespace prior to text analysis.  Default
 3012   behavior remains unchanged: split-on-whitespace=true. (Steve Rowe)
 3013 
 3014 * LUCENE-7276: MatchNoDocsQuery now includes an optional reason for
 3015   why it was used (Jim Ferenczi via Mike McCandless)
 3016 
 3017 * LUCENE-7355: AnalyzingQueryParser now only applies the subset of the analysis
 3018   chain that is about normalization for range/fuzzy/wildcard queries.
 3019   (Adrien Grand)
 3020 
 3021 * LUCENE-7376: Add support for ToParentBlockJoinQuery to fast vector highlighter's
 3022   FieldQuery. (Martijn van Groningen)
 3023 
 3024 * LUCENE-7385: Improve/fix assert messages in SpanScorer. (David Smiley)
 3025 
 3026 * LUCENE-7393: Add ICUTokenizer option to parse Myanmar text as syllables instead of words,
 3027   because the ICU word-breaking algorithm has some issues. This allows for the previous 
 3028   tokenization used before Lucene 5. (AM, Robert Muir)
 3029 
 3030 * LUCENE-7409: Changed MMapDirectory's unmapping to work safer, but still with
 3031   no guarantees. This uses a store-store barrier and yields the current thread
 3032   before unmapping to allow in-flight requests to finish. The new code no longer
 3033   uses WeakIdentityMap as it delegates all ByteBuffer reads throgh a new
 3034   ByteBufferGuard wrapper that is shared between all ByteBufferIndexInput clones.
 3035   (Robert Muir, Uwe Schindler)
 3036 
 3037 Optimizations
 3038 
 3039 * LUCENE-7330, LUCENE-7339: Speed up conjunction queries. (Adrien Grand)
 3040 
 3041 * LUCENE-7356: SearchGroup tweaks. (Christine Poerschke)
 3042 
 3043 * LUCENE-7351: Doc id compression for points. (Adrien Grand)
 3044 
 3045 * LUCENE-7371: Point values are now better compressed using run-length
 3046   encoding. (Adrien Grand)
 3047 
 3048 * LUCENE-7311: Cached term queries do not seek the terms dictionary anymore.
 3049   (Adrien Grand)
 3050 
 3051 * LUCENE-7396, LUCENE-7399: Faster flush of points.
 3052   (Adrien Grand, Mike McCandless)
 3053 
 3054 * LUCENE-7406: Automaton and PrefixQuery tweaks (fewer object (re)allocations).
 3055   (Christine Poerschke)
 3056 
 3057 Other
 3058 
 3059 * LUCENE-4787: Fixed some highlighting javadocs. (Michael Dodsworth via Adrien
 3060   Grand)
 3061 
 3062 * LUCENE-7334: Update ASM dependency to 5.1.  (Uwe Schindler)
 3063 
 3064 * LUCENE-7346: Update forbiddenapis to version 2.2.
 3065   (Uwe Schindler)
 3066 
 3067 * LUCENE-7360: Explanation.toHtml() is deprecated. (Alan Woodward)
 3068   
 3069 * LUCENE-7372: Factor out an org.apache.lucene.search.FilterWeight class.
 3070   (Christine Poerschke, Adrien Grand, David Smiley)
 3071 
 3072 * LUCENE-7384: Removed ScoringWrapperSpans. And tweaked SpanWeight.buildSimWeight() to
 3073   reuse the existing Similarity instead of creating a new one. (David Smiley)
 3074 
 3075 ======================= Lucene 6.1.0 =======================
 3076 
 3077 New Features
 3078 
 3079 * LUCENE-7099: Add LatLonDocValuesField.newDistanceSort to the sandbox.
 3080   (Robert Muir)
 3081 
 3082 * LUCENE-7140: Add PlanetModel.bisection to spatial3d (Karl Wright via
 3083   Mike McCandless)
 3084 
 3085 * LUCENE-7069: Add LatLonPoint.nearest, to find nearest N points to a
 3086   provided query point (Mike McCandless)
 3087 
 3088 * LUCENE-7234: Added InetAddressPoint.nextDown/nextUp to easily generate range
 3089   queries with excluded bounds. (Adrien Grand)
 3090 
 3091 * LUCENE-7300: The misc module now has a directory wrapper that uses hard-links if
 3092   applicable and supported when copying files from another FSDirectory in 
 3093   Directory#copyFrom. (Simon Willnauer)
 3094 
 3095 API Changes
 3096 
 3097 * LUCENE-7184: Refactor LatLonPoint encoding methods to new GeoEncodingUtils
 3098   helper class in core geo package. Also refactors LatLonPointTests to
 3099   TestGeoEncodingUtils (Nick Knize)
 3100 
 3101 * LUCENE-7163: refactor GeoRect, Polygon, and GeoUtils tests to geo
 3102   package in core (Nick Knize)
 3103 
 3104 * LUCENE-7152: Refactor GeoUtils from lucene-spatial package to
 3105   core (Nick Knize)
 3106 
 3107 * LUCENE-7141: Switch OfflineSorter's ByteSequencesReader to
 3108   BytesRefIterator (Mike McCandless)
 3109 
 3110 * LUCENE-7150: Spatial3d gets useful APIs to create common shape
 3111   queries, matching LatLonPoint.  (Karl Wright via Mike McCandless)
 3112 
 3113 * LUCENE-7243: Removed the LeafReaderContext parameter from
 3114   QueryCachingPolicy#shouldCache. (Adrien Grand)
 3115 
 3116 Optimizations
 3117 
 3118 * LUCENE-7071: Reduce bytes copying in OfflineSorter, giving ~10%
 3119   speedup on merging 2D LatLonPoint values (Mike McCandless)
 3120 
 3121 * LUCENE-7105, LUCENE-7215: Optimize LatLonPoint's newDistanceQuery. 
 3122   (Robert Muir)
 3123 
 3124 * LUCENE-7097: IntroSorter now recurses to 2 * log_2(count) quicksort
 3125   stack depth before switching to heapsort (Adrien Grand, Mike McCandless)
 3126 
 3127 * LUCENE-7115: Speed up FieldCache.CacheEntry toString by setting initial
 3128   StringBuilder capacity (Gregory Chanan)
 3129 
 3130 * LUCENE-7147: Improve disjoint check for geo distance query traversal
 3131   (Ryan Ernst, Robert Muir, Mike McCandless)
 3132 
 3133 * LUCENE-7153: GeoPointField and LatLonPoint polygon queries now support
 3134   multiple polygons and holes, with memory usage independent of
 3135   polygon complexity. (Karl Wright, Mike McCandless, Robert Muir)
 3136 
 3137 * LUCENE-7159: Speed up LatLonPoint polygon performance. (Robert Muir, Ryan Ernst)
 3138 
 3139 * LUCENE-7211: Reduce memory & GC for spatial RPT Intersects when the number of
 3140   matching docs is small. (Jeff Wartes, David Smiley)
 3141 
 3142 * LUCENE-7235: LRUQueryCache should not take a lock for segments that it will
 3143   not cache on anyway. (Adrien Grand)
 3144 
 3145 * LUCENE-7238: Explicitly disable the query cache in MemoryIndex#createSearcher.
 3146   (Adrien Grand)
 3147 
 3148 * LUCENE-7237: LRUQueryCache now prefers returning an uncached Scorer than
 3149   waiting on a lock. (Adrien Grand)
 3150 
 3151 * LUCENE-7261, LUCENE-7262, LUCENE-7264, LUCENE-7258: Speed up DocIdSetBuilder
 3152   (which is used by TermsQuery, multi-term queries and several point queries).
 3153   (Adrien Grand, Jeff Wartes, David Smiley)
 3154 
 3155 * LUCENE-7299: Speed up BytesRefHash.sort() using radix sort. (Adrien Grand)
 3156 
 3157 * LUCENE-7306: Speed up points indexing and merging using radix sort.
 3158   (Adrien Grand)
 3159 
 3160 Bug Fixes
 3161 
 3162 * LUCENE-7127: Fix corner case bugs in GeoPointDistanceQuery. (Robert Muir)
 3163 
 3164 * LUCENE-7166: Fix corner case bugs in LatLonPoint/GeoPointField bounding box
 3165   queries. (Robert Muir)
 3166 
 3167 * LUCENE-7168: Switch to stable encode for geo3d, remove quantization
 3168   test leniency, remove dead code (Mike McCandless)
 3169 
 3170 * LUCENE-7301: Multiple doc values updates to the same document within
 3171   one update batch could be applied in the wrong order resulting in
 3172   the wrong updated value (Ishan Chattopadhyaya, hossman, Mike McCandless)
 3173 
 3174 * LUCENE-7312: Fix geo3d's x/y/z double to int encoding to ensure it always
 3175   rounds down (Karl Wright, Mike McCandless)
 3176 
 3177 * LUCENE-7132: BooleanQuery sometimes assigned too-low scores in cases
 3178   where ranges of documents had only a single clause matching while
 3179   other ranges had more than one clause matching (Ahmet Arslan,
 3180   hossman, Mike McCandless)
 3181 
 3182 * LUCENE-7286: Added support for highlighting SynonymQuery. (Adrien Grand)
 3183 
 3184 * LUCENE-7291: Spatial heatmap faceting could mis-count when the heatmap crosses the
 3185   dateline and indexed non-point shapes are much bigger than the heatmap region.
 3186   (David Smiley)
 3187 
 3188 * LUCENE-7333: Fix test bug where randomSimpleString() generated a filename
 3189   that is a reserved device name on Windows.  (Uwe Schindler, Mike McCandless)
 3190 
 3191 Other
 3192 
 3193 * LUCENE-7295: TermAutomatonQuery.hashCode calculates Automaton.toDot().hash,
 3194   equivalence relationship replaced with object identity. (Dawid Weiss)
 3195 
 3196 * LUCENE-7277: Make Query.hashCode and Query.equals abstract. (Paul Elschot, 
 3197   Dawid Weiss)
 3198 
 3199 * LUCENE-7174: Upgrade randomizedtesting to 2.3.4. (Uwe Schindler, Dawid Weiss)
 3200 
 3201 * LUCENE-7205: Remove repeated nl.getLength() calls in
 3202   (Boolean|DisjunctionMax|FuzzyLikeThis)QueryBuilder. (Christine Poerschke)
 3203 
 3204 * LUCENE-7210: Make TestCore*Parser's analyzer choice override-able
 3205   (Christine Poerschke, Daniel Collins)
 3206 
 3207 * LUCENE-7263: Make queryparser/xml/CoreParser's SpanQueryBuilderFactory
 3208   accessible to deriving classes. (Daniel Collins via Christine Poerschke)
 3209 
 3210 * SOLR-9109/SOLR-9121: Allow specification of a custom Ivy settings file via system
 3211   property "ivysettings.xml". (Misha Dmitriev, Christine Poerschke, Uwe Schindler, Steve Rowe)
 3212 
 3213 * LUCENE-7206: Improve the ToParentBlockJoinQuery's explain by including the explain
 3214   of the best matching child doc. (Ilya Kasnacheev, Jeff Evans via Martijn van Groningen)
 3215 
 3216 * LUCENE-7307: Add getters to the PointInSetQuery and PointRangeQuery queries.
 3217   (Martijn van Groningen, Adrien Grand)
 3218 
 3219 Build
 3220 
 3221 * LUCENE-7292: Use '-release' instead of '-source/-target' during
 3222   compilation on Java 9+ to ensure real cross-compilation.
 3223   (Uwe Schindler)
 3224   
 3225 * LUCENE-7296: Update forbiddenapis to version 2.1.
 3226   (Uwe Schindler)
 3227   
 3228 ======================= Lucene 6.0.1 =======================
 3229 
 3230 New Features
 3231 
 3232 * LUCENE-7278: Spatial-extras DateRangePrefixTree's Calendar is now configurable, to
 3233   e.g. clear the Gregorian Change Date.  Also, toString(cal) is now identical to
 3234   DateTimeFormatter.ISO_INSTANT. (David Smiley)
 3235 
 3236 Bug Fixes
 3237 
 3238 * LUCENE-7187: Block join queries' Weight#extractTerms(...) implementations
 3239   should delegate to the wrapped weight. (Martijn van Groningen)
 3240 
 3241 * LUCENE-7209: Fixed explanations of FunctionScoreQuery. (Adrien Grand)
 3242 
 3243 * LUCENE-7232: Fixed InetAddressPoint.newPrefixQuery, which was generating an
 3244   incorrect query when the prefix length was not a multiple of 8. (Adrien Grand)
 3245 
 3246 * LUCENE-7279: JapaneseTokenizer throws ArrayIndexOutOfBoundsException
 3247   on some valid inputs (Mike McCandless)
 3248 
 3249 * LUCENE-7188: remove incorrect sanity check in NRTCachingDirectory.listAll()
 3250   that led to IllegalStateException being thrown when nothing was wrong.
 3251   (David Smiley, yonik)
 3252 
 3253 * LUCENE-7219: Make queryparser/xml (Point|LegacyNumeric)RangeQuery builders
 3254   match the underlying queries' (lower|upper)Term optionality logic.
 3255   (Kaneshanathan Srivisagan, Christine Poerschke)
 3256 
 3257 * LUCENE-7257: Fixed PointValues#size(IndexReader, String), docCount,
 3258   minPackedValue and maxPackedValue to skip leaves that do not have points
 3259   rather than raising an IllegalStateException. (Adrien Grand)
 3260 
 3261 * LUCENE-7284: GapSpans needs to implement positionsCost(). (Daniel Bigham, Alan
 3262   Woodward)
 3263 
 3264 * LUCENE-7231: WeightedSpanTermExtractor didn't deal correctly with single-term
 3265   phrase queries. (Eva Popenda, Alan Woodward)
 3266 
 3267 * LUCENE-7293: Don't try to highlight GeoPoint queries (Britta Weber,
 3268   Nick Knize, Mike McCandless, Uwe Schindler)
 3269 
 3270 Documentation
 3271 
 3272 * LUCENE-7223: Improve XXXPoint javadocs to make it clear that you
 3273   should separately add StoredField if you want to retrieve these
 3274   field values at search time (Greg Huber, Robert Muir, Mike McCandless)
 3275 
 3276 ======================= Lucene 6.0.0 =======================
 3277 
 3278 System Requirements
 3279 
 3280 * LUCENE-5950: Move to Java 8 as minimum Java version.
 3281   (Ryan Ernst, Uwe Schindler)
 3282 
 3283 * LUCENE-6069: Lucene Core now gets compiled with Java 8 "compact1" profile,
 3284   all other modules with "compact2".  (Robert Muir, Uwe Schindler)
 3285 
 3286 New Features
 3287 
 3288 * LUCENE-6631: Lucene Document classification (Tommaso Teofili, Alessandro Benedetti)
 3289 
 3290 * LUCENE-6747: FingerprintFilter is a TokenFilter that outputs a single
 3291   token which is a concatenation of the sorted and de-duplicated set of 
 3292   input tokens. Useful for normalizing short text in clustering/linking 
 3293   tasks. (Mark Harwood, Adrien Grand)
 3294 
 3295 * LUCENE-5735: NumberRangePrefixTreeStrategy now includes interval/range faceting
 3296   for counting ranges that align with the underlying terms as defined by the
 3297   NumberRangePrefixTree (e.g. familiar date units like days).  (David Smiley)
 3298 
 3299 * LUCENE-6711: Use CollectionStatistics.docCount() for IDF and average field
 3300   length computations, to avoid skew from documents that don't have the field.
 3301   (Ahmet Arslan via Robert Muir)
 3302 
 3303 * LUCENE-6758: Use docCount+1 for DefaultSimilarity's IDF, so that queries 
 3304   containing nonexistent fields won't screw up querynorm. (Terry Smith, Robert Muir)
 3305 
 3306 * SOLR-7876: The QueryTimeout interface now has a isTimeoutEnabled method
 3307   that can return false to exit from ExitableDirectoryReader wrapping at
 3308   the point fields() is called. (yonik)
 3309 
 3310 * LUCENE-6825: Add low-level support for block-KD trees (Mike McCandless)
 3311 
 3312 * LUCENE-6852, LUCENE-6975: Add support for points (dimensionally
 3313   indexed values) to index, document and codec APIs, including a
 3314   simple text implementation.  (Mike McCandless)
 3315 
 3316 * LUCENE-6861: Create Lucene60Codec, supporting points.
 3317   (Mike McCandless)
 3318 
 3319 * LUCENE-6879: Allow to define custom CharTokenizer instances without
 3320   subclassing using Java 8 lambdas or method references. (Uwe Schindler)
 3321 
 3322 * LUCENE-6881: Cutover all BKD implementations to points
 3323   (Mike McCandless)
 3324 
 3325 * LUCENE-6837: Add N-best output support to JapaneseTokenizer.
 3326   (Hiroharu Konno via Christian Moen)
 3327 
 3328 * LUCENE-6962: Add per-dimension min/max to points
 3329   (Mike McCandless)
 3330 
 3331 * LUCENE-6975: Add ExactPointQuery, to match a single N-dimensional
 3332   point (Robert Muir, Mike McCandless)
 3333 
 3334 * LUCENE-6989: Add preliminary support for MMapDirectory unmapping in Java 9.
 3335   (Uwe Schindler, Chris Hegarty, Peter Levart)
 3336 
 3337 * LUCENE-7040: Upgrade morfologik-stemming to version 2.1.0.
 3338   (Dawid Weiss)
 3339 
 3340 * LUCENE-7048: Add XXXPoint.newSetQuery, to create a query that
 3341   efficiently matches all documents containing any of the specified
 3342   point values.  This is the analog of TermsQuery, but for points
 3343   instead.   (Adrien Grand, Robert Muir, Mike McCandless)
 3344 
 3345 API Changes
 3346 
 3347 * LUCENE-7094: BBoxStrategy and PointVectorStrategy now support
 3348   PointValues (in addition to legacy numeric trie).  Their APIs
 3349   were changed a little and also made more consistent.  PointValues/Trie
 3350   is optional, DocValues is optional, stored value is optional.
 3351   (Nick Knize, David Smiley)
 3352 
 3353 * LUCENE-6067: Accountable.getChildResources has a default
 3354   implementation returning the empty list.  (Robert Muir)
 3355 
 3356 * LUCENE-6583: FilteredQuery has been removed. Instead, you can construct a
 3357   BooleanQuery with one MUST clause for the query, and one FILTER clause for
 3358   the filter. (Adrien Grand)
 3359 
 3360 * LUCENE-6651: AttributeImpl#reflectWith(AttributeReflector) was made
 3361   abstract and has no reflection-based default implementation anymore.
 3362   (Uwe Schindler)
 3363 
 3364 * LUCENE-6706: PayloadTermQuery and PayloadNearQuery have been removed.
 3365   Instead, use PayloadScoreQuery to wrap any SpanQuery. (Alan Woodward)
 3366 
 3367 * LUCENE-6829: OfflineSorter, and the classes that use it (suggesters,
 3368   hunspell) now do all temporary file IO via Directory instead of
 3369   directly through java's temp dir.  Directory.createTempOutput
 3370   creates a uniquely named IndexOutput, and the new
 3371   IndexOutput.getName returns its name (Dawid Weiss, Robert Muir, Mike
 3372   McCandless)
 3373 
 3374 * LUCENE-6917: Deprecate and rename NumericXXX classes to
 3375   LegacyNumericXXX in favor of points (Mike McCandless)
 3376 
 3377 * LUCENE-6947: SortField.missingValue is now protected. You can read its
 3378   value using the new SortField.getMissingValue getter. (Adrien Grand)
 3379 
 3380 * LUCENE-7028: Remove duplicate method in LegacyNumericUtils.
 3381   (Uwe Schindler)
 3382 
 3383 * LUCENE-7052, LUCENE-7053: Remove custom comparators from BytesRef
 3384   class and solely use natural byte[] comparator throughout codebase.
 3385   This also simplifies API of BytesRefHash. It also replaces the natural
 3386   comparator in ArrayUtil by Java 8's Comparator#naturalOrder().
 3387   (Mike McCandless, Uwe Schindler, Robert Muir)
 3388 
 3389 * LUCENE-7060: Update Spatial4j to 0.6.  The package com.spatial4j.core
 3390   is now org.locationtech.spatial4j. (David Smiley)
 3391 
 3392 * LUCENE-7058: Add getters to various Query implementations (Guillaume Smet via
 3393   Alan Woodward)
 3394 
 3395 * LUCENE-7064: MultiPhraseQuery is now immutable and should be constructed
 3396   with MultiPhraseQuery.Builder. (Luc Vanlerberghe via Adrien Grand)
 3397 
 3398 * LUCENE-7072: Geo3DPoint always uses WGS84 planet model.
 3399   (Robert Muir, Mike McCandless)
 3400 
 3401 * LUCENE-7056: Geo3D classes are in different packages now. (David Smiley)
 3402 
 3403 * LUCENE-6952: These classes are now abstract: FilterCodecReader, FilterLeafReader,
 3404   FilterCollector, FilterDirectory.  And some Filter* classes in
 3405   lucene-test-framework too. (David Smiley)
 3406 
 3407 * SOLR-8867: FunctionValues.getRangeScorer now takes a LeafReaderContext instead
 3408   of an IndexReader, and avoids matching documents without a value in the field
 3409   for numeric fields. (yonik)
 3410 
 3411 Optimizations
 3412 
 3413 * LUCENE-6891: Use prefix coding when writing points in 
 3414   each leaf block in the default codec, to reduce the index
 3415   size (Mike McCandless)
 3416 
 3417 * LUCENE-6901: Optimize points indexing: use faster
 3418   IntroSorter instead of InPlaceMergeSorter, and specialize 1D
 3419   merging to merge sort the already sorted segments instead of
 3420   re-indexing (Mike McCandless)
 3421 
 3422 * LUCENE-6793: LegacyNumericRangeQuery.hashCode() is now less subject to hash
 3423   collisions. (J.B. Langston via Adrien Grand)
 3424 
 3425 * LUCENE-7050: TermsQuery is now cached more aggressively by the default
 3426   query caching policy. (Adrien Grand)
 3427 
 3428 * LUCENE-7066: PointRangeQuery got optimized for the case that all documents
 3429   have a value and all points from the segment match. (Adrien Grand)
 3430 
 3431 Changes in Runtime Behavior
 3432 
 3433 * LUCENE-6789: IndexSearcher's default Similarity is changed to BM25Similarity.
 3434   Use ClassicSimilarity to get the old vector space DefaultSimilarity. (Robert Muir)
 3435 
 3436 * LUCENE-6886: Reserve the .tmp file name extension for temp files,
 3437   and codec components are no longer allowed to use this extension
 3438   (Robert Muir, Mike McCandless)
 3439 
 3440 * LUCENE-6835: Directory.listAll now returns entries in sorted order,
 3441   to not leak platform-specific behavior, and "retrying file deletion"
 3442   is now the responsibility of Directory.deleteFile, not the caller.
 3443   (Robert Muir, Mike McCandless)
 3444 
 3445 Tests
 3446 
 3447 * LUCENE-7009: Add expectThrows utility to LuceneTestCase. This uses a lambda
 3448   expression to encapsulate a statement that is expected to throw an exception.
 3449   (Ryan Ernst)
 3450 
 3451 Bug Fixes
 3452 
 3453 * LUCENE-7065: Fix the explain for the global ordinals join query. Before the
 3454   explain would also indicate that non matching documents would match.
 3455   On top of that with score mode average, the explain would fail with a NPE.
 3456   (Martijn van Groningen)
 3457 
 3458 * LUCENE-7101: OfflineSorter had O(N^2) merge cost, and used too many
 3459   temporary file descriptors, for large sorts (Mike McCandless)
 3460 
 3461 * LUCENE-7111: DocValuesRangeQuery.newLongRange behaves incorrectly for
 3462   Long.MAX_VALUE and Long.MIN_VALUE (Ishan Chattopadhyaya via Steve Rowe)
 3463 
 3464 * LUCENE-7139: Fix bugs in geo3d's Vincenty surface distance
 3465   implementation (Karl Wright via Mike McCandless)
 3466 
 3467 * LUCENE-7112: WeightedSpanTermExtractor.extractUnknownQuery is only called
 3468   on queries that could not be extracted. (Adrien Grand)
 3469 
 3470 * LUCENE-7126: Remove GeoPointDistanceRangeQuery. This query was implemented
 3471   with boolean NOT, and incorrect for multi-valued documents. (Robert Muir)
 3472 
 3473 * LUCENE-7158: Consistently use earth's WGS84 mean radius wherever our
 3474   geo search implementations approximate the earth as a sphere (Karl
 3475   Wright via Mike McCandless)
 3476 
 3477 Other
 3478 
 3479 * LUCENE-7035: Upgrade icu4j to 56.1/unicode 8. (Robert Muir)
 3480 
 3481 * LUCENE-7087: Let MemoryIndex#fromDocument(...) accept 'Iterable<? extends IndexableField>'
 3482   as document instead of 'Document'. (Martijn van Groningen)
 3483 
 3484 * LUCENE-7091: Add doc values support to MemoryIndex
 3485   (Martijn van Groningen, David Smiley)
 3486 
 3487 * LUCENE-7093: Add point values support to MemoryIndex
 3488   (Martijn van Groningen, Mike McCandless)
 3489 
 3490 * LUCENE-7095: Add point values support to the numeric field query time join.
 3491   (Martijn van Groningen, Mike McCandless)
 3492 
 3493 ======================= Lucene 5.5.5 =======================
 3494 
 3495 Changes in Runtime Behavior
 3496 
 3497 * Resolving of external entities in queryparser/xml/CoreParser is disallowed
 3498   by default. See SOLR-11477 for details.
 3499 
 3500 Bug Fixes
 3501 
 3502 * LUCENE-7419: Fix performance bug with TokenStream.end(), where it would lookup
 3503   PositionIncrementAttribute every time. (Mike McCandless, Robert Muir)
 3504 
 3505 * SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser
 3506   by default. (Michael Stepankin, Olga Barinova, Uwe Schindler, Christine Poerschke)
 3507 
 3508 ======================= Lucene 5.5.4 =======================
 3509 
 3510 Bug Fixes
 3511 
 3512 * LUCENE-7417: The standard Highlighter could throw an IllegalArgumentException when
 3513   trying to highlight a query containing a degenerate case of a MultiPhraseQuery with one
 3514   term.  (Thomas Kappler via David Smiley)
 3515 
 3516 * LUCENE-7657: Fixed potential memory leak in the case that a (Span)TermQuery
 3517   with a TermContext is cached. (Adrien Grand)
 3518 
 3519 * LUCENE-7647: Made stored fields reclaim native memory more aggressively when
 3520   configured with BEST_COMPRESSION. This could otherwise result in out-of-memory
 3521   issues. (Adrien Grand)
 3522 
 3523 * LUCENE-7562: CompletionFieldsConsumer sometimes throws
 3524   NullPointerException on ghost fields (Oliver Eilhard via Mike McCandless)
 3525 
 3526 * LUCENE-7547: JapaneseTokenizerFactory was failing to close the
 3527   dictionary file it opened (Markus via Mike McCandless)
 3528 
 3529 * LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points.
 3530   (Hossman)
 3531 
 3532 * LUCENE-7440: Document id skipping (PostingsEnum.advance) could throw an
 3533   ArrayIndexOutOfBoundsException exception on large index segments (>1.8B docs)
 3534   with large skips. (yonik)
 3535 
 3536 * LUCENE-7570: IndexWriter may deadlock if a commit is running while
 3537   there are too many merges running and one of the merges hits a
 3538   tragic exception (Joey Echeverria via Mike McCandless)
 3539 
 3540 Other
 3541 
 3542 * LUCENE-6989: Backport MMapDirectory's unmapping code from Lucene 6.4 to use
 3543   MethodHandles. This allows it to work with Java 9 (EA build 150 and later).
 3544   (Uwe Schindler)
 3545 
 3546 Build
 3547 
 3548 * LUCENE-7543: Make changes-to-html target an offline operation, by moving the
 3549   Lucene and Solr DOAP RDF files into the Git source repository under
 3550   dev-tools/doap/ and then pulling release dates from those files, rather than
 3551   from JIRA. (Mano Kovacs, hossman, Steve Rowe)
 3552 
 3553 * LUCENE-7596: Update Groovy to version 2.4.8 to allow building with Java 9
 3554   build 148+. Also update JGit version for working-copy checks. This does not
 3555   fix all issues with Java 9, but allows to build the distribution.
 3556   (Uwe Schindler)
 3557 
 3558 * LUCENE-7651: Backport (Lucene 6.4.1) fix for Java 8u121 to allow documentation
 3559   build to inject "Google Code Prettify" without adding Javascript to Javadocs's
 3560   -bottom parameter. Unfortunately, this fix disables Prettify if Javadocs are
 3561   built with Java 7, as there is no generic way in Java 7 to inject Javascript
 3562   without breaking Java 8 (and possible paid Java 7 security updates). This
 3563   fix also updates Prettify to latest version to work around a Google Chrome
 3564   issue. (Uwe Schindler)
 3565 
 3566 ======================= Lucene 5.5.3 =======================
 3567 (No Changes)
 3568 
 3569 ======================= Lucene 5.5.2 =======================
 3570 
 3571 Bug Fixes
 3572 
 3573 * LUCENE-7065: Fix the explain for the global ordinals join query. Before the
 3574   explain would also indicate that non matching documents would match.
 3575   On top of that with score mode average, the explain would fail with a NPE.
 3576   (Martijn van Groningen)
 3577 
 3578 * LUCENE-7111: DocValuesRangeQuery.newLongRange behaves incorrectly for
 3579   Long.MAX_VALUE and Long.MIN_VALUE (Ishan Chattopadhyaya via Steve Rowe)
 3580 
 3581 * LUCENE-7139: Fix bugs in geo3d's Vincenty surface distance
 3582   implementation (Karl Wright via Mike McCandless)
 3583 
 3584 * LUCENE-7187: Block join queries' Weight#extractTerms(...) implementations
 3585   should delegate to the wrapped weight. (Martijn van Groningen)
 3586 
 3587 * LUCENE-7279: JapaneseTokenizer throws ArrayIndexOutOfBoundsException
 3588   on some valid inputs (Mike McCandless)
 3589 
 3590 * LUCENE-7219: Make queryparser/xml (Point|LegacyNumeric)RangeQuery builders
 3591   match the underlying queries' (lower|upper)Term optionality logic.
 3592   (Kaneshanathan Srivisagan, Christine Poerschke)
 3593 
 3594 * LUCENE-7284: GapSpans needs to implement positionsCost(). (Daniel Bigham, Alan
 3595   Woodward)
 3596 
 3597 * LUCENE-7231: WeightedSpanTermExtractor didn't deal correctly with single-term
 3598   phrase queries. (Eva Popenda, Alan Woodward)
 3599 
 3600 * LUCENE-7301: Multiple doc values updates to the same document within
 3601   one update batch could be applied in the wrong order resulting in
 3602   the wrong updated value (Ishan Chattopadhyaya, hossman, Mike McCandless)
 3603 
 3604 * LUCENE-7132: BooleanQuery sometimes assigned too-low scores in cases
 3605   where ranges of documents had only a single clause matching while
 3606   other ranges had more than one clause matching (Ahmet Arslan,
 3607   hossman, Mike McCandless)
 3608 
 3609 * LUCENE-7291: Spatial heatmap faceting could mis-count when the heatmap crosses the
 3610   dateline and indexed non-point shapes are much bigger than the heatmap region.
 3611   (David Smiley)
 3612 
 3613 ======================= Lucene 5.5.1 =======================
 3614 
 3615 Bug fixes
 3616 
 3617 * LUCENE-7112: WeightedSpanTermExtractor.extractUnknownQuery is only called
 3618   on queries that could not be extracted. (Adrien Grand)
 3619 
 3620 * LUCENE-7188: remove incorrect sanity check in NRTCachingDirectory.listAll()
 3621   that led to IllegalStateException being thrown when nothing was wrong.
 3622   (David Smiley, yonik)
 3623 
 3624 * LUCENE-7209: Fixed explanations of FunctionScoreQuery. (Adrien Grand)
 3625 
 3626 ======================= Lucene 5.5.0 =======================
 3627 
 3628 New Features
 3629 
 3630 * LUCENE-5868: JoinUtil.createJoinQuery(..,NumericType,..) query-time join 
 3631   for LONG and INT fields with NUMERIC and SORTED_NUMERIC doc values.
 3632   (Alexey Zelin via Mikhail Khludnev) 
 3633 
 3634 * LUCENE-6939: Add exponential reciprocal scoring to
 3635   BlendedInfixSuggester, to even more strongly favor suggestions that
 3636   match closer to the beginning (Arcadius Ahouansou via Mike McCandless)
 3637 
 3638 * LUCENE-6958: Improved CustomAnalyzer to take class references to factories
 3639   as alternative to their SPI name. This enables compile-time safety when
 3640   defining analyzer's components.  (Uwe Schindler, Shai Erera)
 3641 
 3642 * LUCENE-6818, LUCENE-6986: Add DFISimilarity implementing the divergence 
 3643   from independence model. (Ahmet Arslan via Robert Muir)
 3644 
 3645 * SOLR-4619: Added removeAllAttributes() to AttributeSource, which removes
 3646   all previously added attributes.
 3647   
 3648 * LUCENE-7010: Added MergePolicyWrapper to allow easy wrapping of other policies.
 3649   (Shai Erera)
 3650 
 3651 API Changes
 3652 
 3653 * LUCENE-6997: refactor sandboxed GeoPointField and query classes to lucene-spatial 
 3654   module under new lucene.spatial.geopoint package (Nick Knize)
 3655 
 3656 * LUCENE-6908: GeoUtils static relational methods have been refactored to new 
 3657   GeoRelationUtils and now correctly handle large irregular rectangles, and
 3658   pole crossing distance queries. (Nick Knize)
 3659 
 3660 * LUCENE-6900: Grouping sortWithinGroup variables used to allow null to mean
 3661   Sort.RELEVANCE.  Null is no longer permitted.  (David Smiley)
 3662 
 3663 * LUCENE-6919: The Scorer class has been refactored to expose an iterator
 3664   instead of extending DocIdSetIterator. asTwoPhaseIterator() has been renamed
 3665   to twoPhaseIterator() for consistency. (Adrien Grand)
 3666 
 3667 * LUCENE-6973: TeeSinkTokenFilter no longer accepts a SinkFilter (the latter
 3668   has been removed). If you wish to filter the sinks, you can wrap them with
 3669   any other TokenFilter (e.g. a FilteringTokenFilter). Also, you can no longer
 3670   add a SinkTokenStream to an existing TeeSinkTokenFilter. If you need to
 3671   share multiple streams with a single sink, chain them with multiple
 3672   TeeSinkTokenFilters.
 3673   DateRecognizerSinkFilter was renamed to DateRecognizerFilter and moved under
 3674   analysis/common. TokenTypeSinkFilter was removed (use TypeTokenFilter instead).
 3675   TokenRangeSinkFilter was removed. (Shai Erera, Uwe Schindler)
 3676 
 3677 * LUCENE-6980: Default applyAllDeletes to true when opening
 3678   near-real-time readers (Mike McCandless)
 3679 
 3680 * LUCENE-6981: SpanQuery.getTermContexts() helper methods are now public, and
 3681   SpanScorer has a public getSpans() method. (Alan Woodward)
 3682 
 3683 * LUCENE-6932: IndexInput.seek implementations now throw EOFException
 3684   if you seek beyond the end of the file (Adrien Grand, Mike McCandless)
 3685 
 3686 * LUCENE-6988: IndexableField.tokenStream() no longer throws IOException
 3687   (Alan Woodward)
 3688 
 3689 * LUCENE-7028: Deprecate a duplicate method in NumericUtils.
 3690   (Uwe Schindler)
 3691 
 3692 Optimizations
 3693 
 3694 * LUCENE-6930: Decouple GeoPointField from NumericType by using a custom
 3695   and efficient GeoPointTokenStream and TermEnum designed for GeoPoint prefix
 3696   terms. (Nick Knize)
 3697 
 3698 * LUCENE-6951: Improve GeoPointInPolygonQuery using point orientation based
 3699   line crossing algorithm, and adding result for multi-value docs when least
 3700   1 point satisfies polygon criteria. (Nick Knize)
 3701 
 3702 * LUCENE-6889: BooleanQuery.rewrite now performs some query optimization, in
 3703   particular to rewrite queries that look like: "+*:* #filter" to a
 3704   "ConstantScore(filter)". (Adrien Grand)
 3705 
 3706 * LUCENE-6912: Grouping's Collectors now calculate a response to needsScores()
 3707   instead of always 'true'. (David Smiley)
 3708 
 3709 * LUCENE-6815: DisjunctionScorer now advances two-phased iterators lazily,
 3710   stopping to evaluate them as soon as a single one matches. The other iterators
 3711   will be confirmed lazily when computing score() or freq(). (Adrien Grand)
 3712 
 3713 * LUCENE-6926: MUST_NOT clauses now use the match cost API to run the slow bits
 3714   last whenever possible. (Adrien Grand)
 3715 
 3716 * LUCENE-6944: BooleanWeight no longer creates sub-scorers if BS1 is not
 3717   applicable. (Adrien Grand)
 3718 
 3719 * LUCENE-6940: MUST_NOT clauses execute faster, especially when they are sparse.
 3720   (Adrien Grand)
 3721 
 3722 * LUCENE-6470: Improve efficiency of TermsQuery constructors. (Robert Muir)
 3723 
 3724 Bug Fixes
 3725 
 3726 * LUCENE-6976: BytesRefTermAttributeImpl.copyTo NPE'ed if BytesRef was null.
 3727   Added equals & hashCode, and a new test for these things. (David Smiley)
 3728 
 3729 * LUCENE-6932: RAMDirectory's IndexInput was failing to throw
 3730   EOFException in some cases (Stéphane Campinas, Adrien Grand via Mike
 3731   McCandless)
 3732 
 3733 * LUCENE-6896: Don't treat the smallest possible norm value as an infinitely
 3734   long document in SimilarityBase or BM25Similarity. Add more warnings to sims
 3735   that will not work well with extreme tf values. (Ahmet Arslan, Robert Muir)
 3736 
 3737 * LUCENE-6984: SpanMultiTermQueryWrapper no longer modifies its wrapped query.
 3738   (Alan Woodward, Adrien Grand)
 3739 
 3740 * LUCENE-6998: Fix a couple places to better detect truncated index files
 3741   as corruption.  (Robert Muir, Mike McCandless)
 3742 
 3743 * LUCENE-7002: Fixed MultiCollector to not throw a NPE if setScorer is called
 3744   after one of the sub collectors is done collecting. (John Wang, Adrien Grand)
 3745 
 3746 * LUCENE-7027: Fixed NumericTermAttribute to not throw IllegalArgumentException
 3747   after NumericTokenStream was exhausted.  (Uwe Schindler, Lee Hinman,
 3748   Mike McCandless)
 3749 
 3750 * LUCENE-7018: Fix GeoPointTermQueryConstantScoreWrapper to add document on
 3751   first GeoPointField match. (Nick Knize)
 3752 
 3753 * LUCENE-7019: Add two-phase iteration to GeoPointTermQueryConstantScoreWrapper.
 3754   (Robert Muir via Nick Knize)
 3755 
 3756 * LUCENE-6989: Improve MMapDirectory's unmapping checks to catch more non-working
 3757   cases. The unmap-hack does not yet work with recent Java 9. Official support
 3758   will come with Lucene 6.  (Uwe Schindler)
 3759 
 3760 Other
 3761 
 3762 * LUCENE-6924: Upgrade randomizedtesting to 2.3.2. (Dawid Weiss)
 3763 
 3764 * LUCENE-6920: Improve custom function checks in expressions module
 3765   to use MethodHandles and work without extra security privileges.
 3766   (Uwe Schindler, Robert Muir)
 3767 
 3768 * LUCENE-6921: Fix SPIClassIterator#isParentClassLoader to don't
 3769   require extra permissions.  (Uwe Schindler)
 3770 
 3771 * LUCENE-6923: Fix RamUsageEstimator to access private fields inside
 3772   AccessController block for computing size. (Robert Muir)
 3773 
 3774 * LUCENE-6907: make TestParser extendable, rename test/.../xml/
 3775   NumericRangeQueryQuery.xml to NumericRangeQuery.xml
 3776   (Christine Poerschke)
 3777 
 3778 * LUCENE-6925: add ForceMergePolicy class in test-framework
 3779   (Christine Poerschke)
 3780 
 3781 * LUCENE-6945: factor out TestCorePlus(Queries|Extensions)Parser from
 3782   TestParser, rename TestParser to TestCoreParser (Christine Poerschke)
 3783 
 3784 * LUCENE-6949: fix (potential) resource leak in SynonymFilterFactory
 3785   (https://scan.coverity.com/projects/5620 CID 120656)
 3786   (Christine Poerschke, Coverity Scan (via Rishabh Patel))
 3787 
 3788 * LUCENE-6961: Improve Exception handling in AnalysisFactories /
 3789   AnalysisSPILoader: Don't wrap exceptions occuring in factory's
 3790   ctor inside InvocationTargetException.  (Uwe Schindler)
 3791 
 3792 * LUCENE-6965: Expression's JavascriptCompiler now throw ParseException 
 3793   with bad function names or bad arity instead of IllegalArgumentException.
 3794   (Tomás Fernández Löbbe, Uwe Schindler, Ryan Ernst)
 3795 
 3796 * LUCENE-6964: String-based signatures in JavascriptCompiler replaced
 3797   with better compile-time-checked MethodType; generated class files
 3798   are no longer marked as synthetic.  (Uwe Schindler)
 3799 
 3800 * LUCENE-6978: Refactor several code places that lookup locales
 3801   by string name to use BCP47 locale tag instead. LuceneTestCase
 3802   now also prints locales on failing tests this way.
 3803   Locale#forLanguageTag() and Locale#toString() were placed on list
 3804   of forbidden signatures.  (Uwe Schindler, Robert Muir)
 3805 
 3806 * LUCENE-6988: You can now add IndexableFields directly to a MemoryIndex,
 3807   and create a MemoryIndex from a lucene Document.  (Alan Woodward)
 3808 
 3809 * LUCENE-7005: TieredMergePolicy tweaks (>= vs. >, @see get vs. set)
 3810   (Christine Poerschke)
 3811 
 3812 * LUCENE-7006: increase BaseMergePolicyTestCase use (TestNoMergePolicy and
 3813   TestSortingMergePolicy now extend it, TestUpgradeIndexMergePolicy added)
 3814   (Christine Poerschke)
 3815 
 3816 ======================= Lucene 5.4.1 =======================
 3817 
 3818 Bug Fixes
 3819 
 3820 * LUCENE-6910: fix 'if ... > Integer.MAX_VALUE' check in
 3821   (Binary|Numeric)DocValuesFieldUpdates.merge
 3822   (https://scan.coverity.com/projects/5620 CID 119973 and CID 120081)
 3823   (Christine Poerschke, Coverity Scan (via Rishabh Patel))
 3824 
 3825 * LUCENE-6946: SortField.equals now takes the missingValue parameter into
 3826   account. (Adrien Grand)
 3827 
 3828 * LUCENE-6918: LRUQueryCache.onDocIdSetEviction is only called when at least
 3829   one DocIdSet is being evicted. (Adrien Grand)
 3830 
 3831 * LUCENE-6929: Fix SpanNotQuery rewriting to not drop the pre/post parameters.
 3832   (Tim Allison via Adrien Grand)
 3833 
 3834 * LUCENE-6950: Fix FieldInfos handling of UninvertingReader, e.g. do not
 3835   hide the true docvalues update generation or other properties.
 3836   (Ishan Chattopadhyaya via Robert Muir)
 3837 
 3838 * LUCENE-6948: Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill
 3839   by removing an unnecessary long-to-int cast.
 3840   (Michael Lawley via Christine Poerschke)
 3841 
 3842 * SOLR-7865: BlendedInfixSuggester was returning too many results
 3843   (Arcadius Ahouansou via Mike McCandless)
 3844 
 3845 * LUCENE-6970: Fixed off-by-one error in Lucene54DocValuesProducer that could
 3846   potentially corrupt doc values. (Adrien Grand)
 3847 
 3848 * LUCENE-2229: Fix Highlighter's SimpleSpanFragmenter when multiple adjacent
 3849   stop words following a span can unduly make the fragment way too long.
 3850   (Elmer Garduno, Lukhnos Liu via David Smiley)
 3851 
 3852 ======================= Lucene 5.4.0 =======================
 3853 
 3854 New Features
 3855 
 3856 * LUCENE-6875: New Serbian Filter. (Nikola Smolenski via Robert Muir, 
 3857   Dawid Weiss)
 3858 
 3859 * LUCENE-6720: New FunctionRangeQuery wrapper around ValueSourceScorer
 3860   (returned from ValueSource/FunctionValues.getRangeScorer()). (David Smiley)
 3861 
 3862 * LUCENE-6724: Add utility APIs to GeoHashUtils to compute neighbor
 3863   geohash cells (Nick Knize via Mike McCandless).
 3864 
 3865 * LUCENE-6737: Add DecimalDigitFilter which folds unicode digits to basic latin.
 3866   (Robert Muir)
 3867 
 3868 * LUCENE-6699: Add integration of BKD tree and geo3d APIs to give
 3869   fast, very accurate query to find all indexed points within an
 3870   earth-surface shape (Karl Wright, Mike McCandless)
 3871 
 3872 * LUCENE-6838: Added IndexSearcher#getQueryCache and #getQueryCachingPolicy.
 3873   (Adrien Grand)
 3874 
 3875 * LUCENE-6844: PayloadScoreQuery can include or exclude underlying span scores
 3876   from its score calculations (Bill Bell, Alan Woodward)
 3877 
 3878 * LUCENE-6778: Add GeoPointDistanceRangeQuery, to search for points
 3879   within a "ring" (beyond a minimum distance and below a maximum
 3880   distance) (Nick Knize via Mike McCandless)
 3881 
 3882 * LUCENE-6874: Add a new UnicodeWhitespaceTokenizer to analysis/common
 3883   that uses Unicode character properties extracted from ICU4J to tokenize
 3884   text on whitespace. This tokenizer will split on non-breaking
 3885   space (NBSP), too.  (David Smiley, Uwe Schindler, Steve Rowe)
 3886 
 3887 API Changes
 3888 
 3889 * LUCENE-6590: Query.setBoost(), Query.getBoost() and Query.clone() are gone.
 3890   In order to apply boosts, you now need to wrap queries in a BoostQuery.
 3891   (Adrien Grand)
 3892 
 3893 * LUCENE-6716: SpanPayloadCheckQuery now takes a List<BytesRef> rather than
 3894   a Collection<byte[]>. (Alan Woodward)
 3895 
 3896 * LUCENE-6489: The various span payload queries have been moved to the queries
 3897   submodule, and PayloadSpanUtil is now in sandbox. (Alan Woodward)
 3898 
 3899 * LUCENE-6650: The spatial module no longer uses Filter in any way.  All
 3900   spatial Filters are now subclass Query.  The spatial heatmap/facet API
 3901   now accepts a Bits parameter to filter counts. (David Smiley, Adrien Grand)
 3902 
 3903 * LUCENE-6803: Deprecate sandbox Regexp Query. (Uwe Schindler)
 3904 
 3905 * LUCENE-6301: org.apache.lucene.search.Filter is now deprecated. You should use
 3906   Query objects instead of Filters, and the BooleanClause.Occur.FILTER clause in
 3907   order to let Lucene know that a Query should be used for filtering but not
 3908   scoring.
 3909 
 3910 * LUCENE-6939: SpanOrQuery.addClause is now deprecated, clauses should all be
 3911   provided at construction time. (Paul Elschot via Adrien Grand)
 3912 
 3913 * LUCENE-6855: CachingWrapperQuery is deprecated and will be removed in 6.0.
 3914   (Adrien Grand)
 3915 
 3916 * LUCENE-6870: DisjunctionMaxQuery#add is now deprecated, clauses should all be
 3917   provided at construction time. (Adrien Grand) 
 3918 
 3919 * LUCENE-6884: Analyzer.tokenStream() and Tokenizer.setReader() are no longer
 3920   declared as throwing IOException. (Alan Woodward)
 3921 
 3922 * LUCENE-6849: Expose IndexWriter.flush() method, to move all
 3923   in-memory segments to disk without opening a near-real-time reader
 3924   nor calling fsync (Robert Muir, Simon Willnauer, Mike McCandless)
 3925 
 3926 * LUCENE-6911: Add correct StandardQueryParser.getMultiFields() method,
 3927   deprecate no-op StandardQueryParser.getMultiFields(CharSequence[]) method.
 3928   (Christine Poerschke, Mikhail Khludnev, Coverity Scan (via Rishabh Patel))
 3929 
 3930 Optimizations
 3931 
 3932 * LUCENE-6708: TopFieldCollector does not compute the score several times on the
 3933   same document anymore. (Adrien Grand)
 3934 
 3935 * LUCENE-6720: ValueSourceScorer, returned from
 3936   FunctionValues.getRangeScorer(), now uses TwoPhaseIterator. (David Smiley)
 3937 
 3938 * LUCENE-6756: MatchAllDocsQuery now has a dedicated BulkScorer for better
 3939   performance when used as a top-level query. (Adrien Grand)
 3940 
 3941 * LUCENE-6746: DisjunctionMaxQuery, BoostingQuery and BoostedQuery now create
 3942   sub weights through IndexSearcher so that they can be cached. (Adrien Grand)
 3943 
 3944 * LUCENE-6754: Optimized IndexSearcher.count for the cases when it can use
 3945   index statistics instead of collecting all matches. (Adrien Grand)
 3946 
 3947 * LUCENE-6773: Nested conjunctions now iterate over documents as if clauses
 3948   were all at the same level. (Adrien Grand)
 3949 
 3950 * LUCENE-6777: Reuse BytesRef when visiting term ranges in
 3951   GeoPointTermsEnum to reduce GC pressure (Nick Knize via Mike
 3952   McCandless)
 3953 
 3954 * LUCENE-6779: Reduce memory allocated by CompressingStoredFieldsWriter to write
 3955   strings larger than 64kb by an amount equal to string's utf8 size.
 3956   (Dawid Weiss, Robert Muir, shalin)
 3957 
 3958 * LUCENE-6850: Optimize BooleanScorer for sparse clauses. (Adrien Grand)
 3959 
 3960 * LUCENE-6840: Ordinal indexes for SORTED_SET/SORTED_NUMERIC fields and
 3961   addresses for BINARY fields are now stored on disk instead of in memory.
 3962   (Adrien Grand)
 3963 
 3964 * LUCENE-6878: Speed up TopDocs.merge. (Daniel Jelinski via Adrien Grand)
 3965 
 3966 * LUCENE-6885: StandardDirectoryReader (initialCapacity) tweaks
 3967   (Christine Poerschke)
 3968 
 3969 * LUCENE-6863: Optimized storage requirements of doc values fields when less
 3970   than 1% of documents have a value. (Adrien Grand)
 3971 
 3972 * LUCENE-6892: various lucene.index initialCapacity tweaks
 3973   (Christine Poerschke)
 3974 
 3975 * LUCENE-6276: Added TwoPhaseIterator.matchCost() which allows to confirm the
 3976   least costly TwoPhaseIterators first. (Paul Elschot via Adrien Grand)
 3977 
 3978 * LUCENE-6898: In the default codec, the last stored field value will not
 3979   be fully read from disk if the supplied StoredFieldVisitor doesn't want it.
 3980   So put your largest text field value last to benefit. (David Smiley)
 3981 
 3982 * LUCENE-6909: Remove unnecessary synchronized from
 3983   FacetsConfig.getDimConfig for better concurrency (Sanne Grinovero
 3984   via Mike McCandless)
 3985   
 3986 * SOLR-7730: Speed up SlowCompositeReaderWrapper.getSortedSetDocValues() by
 3987   avoiding merging FieldInfos just to check doc value type.
 3988   (Paul Vasilyev, Yuriy Pakhomov, Mikhail Khludnev, yonik)
 3989 
 3990 Bug Fixes
 3991 
 3992 * LUCENE-6905: Unwrap center longitude for dateline crossing 
 3993   GeoPointDistanceQuery. (Nick Knize)
 3994 
 3995 * LUCENE-6817: ComplexPhraseQueryParser.ComplexPhraseQuery does not display 
 3996   slop in toString(). (Ahmet Arslan via Dawid Weiss)
 3997 
 3998 * LUCENE-6730: Hyper-parameter c is ignored in term frequency NormalizationH1.
 3999   (Ahmet Arslan via Robert Muir)
 4000 
 4001 * LUCENE-6742: Lovins & Finnish implementation of SnowballFilter was
 4002   fixed to behave exactly as specified. A bug in the snowball compiler
 4003   caused differences in output of the filter in comparison to the original
 4004   test data.  In addition, the performance of those filters was improved
 4005   significantly.  (Uwe Schindler, Robert Muir)
 4006 
 4007 * LUCENE-6783: Removed side effects from FuzzyLikeThisQuery.rewrite.
 4008   (Adrien Grand)
 4009 
 4010 * LUCENE-6776: Fix geo3d math to handle randomly squashed planet
 4011   models (Karl Wright via Mike McCandless) 
 4012 
 4013 * LUCENE-6792: Fix TermsQuery.toString() to work with binary terms.
 4014   (Ruslan Muzhikov, Robert Muir)
 4015 
 4016 * LUCENE-5503: When Highlighter's WeightedSpanTermExtractor converts a
 4017   PhraseQuery to an equivalent SpanQuery, it would sometimes use a slop that is
 4018   too low (no highlight) or determine inOrder wrong.
 4019   (Tim Allison via David Smiley)
 4020 
 4021 * LUCENE-6790: Fix IndexWriter thread safety when one thread is
 4022   handling a tragic exception but another is still committing (Mike
 4023   McCandless)
 4024 
 4025 * LUCENE-6810: Upgrade to Spatial4j 0.5 -- fixes some edge-case bugs in the
 4026   spatial module. See https://github.com/locationtech/spatial4j/blob/master/CHANGES.md
 4027   (David Smiley)
 4028 
 4029 * LUCENE-6813: OfflineSorter no longer removes its output Path up
 4030   front, and instead opens it for write with the
 4031   StandardCopyOption.REPLACE_EXISTING to overwrite any prior file, so
 4032   that callers can safely use Files.createTempFile for the output.
 4033   This change also fixes OfflineSorter's default temp directory when
 4034   running tests to use mock filesystems so e.g. we detect file handle
 4035   leaks (Dawid Weiss, Robert Muir, Mike McCandless)
 4036 
 4037 * LUCENE-6813: RangeTreeWriter was failing to close all file handles
 4038   it opened, leading to intermittent failures on Windows (Dawid Weiss,
 4039   Robert Muir, Mike McCandless)
 4040 
 4041 * LUCENE-6826: Fix ClassCastException when merging a field that has no
 4042   terms because they were filtered out by e.g. a FilterCodecReader
 4043   (Trejkaz via Mike McCandless)
 4044 
 4045 * LUCENE-6823: LocalReplicator should use System.nanoTime as its clock
 4046   source for checking for expiration (Ishan Chattopadhyaya via Mike
 4047   McCandless)
 4048 
 4049 * LUCENE-6856: The Weight wrapper used by LRUQueryCache now delegates to the
 4050   original Weight's BulkScorer when applicable. (Adrien Grand)
 4051 
 4052 * LUCENE-6858: Fix ContextSuggestField to correctly wrap token stream
 4053   when using CompletionAnalyzer. (Areek Zillur)
 4054 
 4055 * LUCENE-6872: IndexWriter handles any VirtualMachineError, not just OOM,
 4056   as tragic. (Robert Muir)
 4057 
 4058 * LUCENE-6814: PatternTokenizer no longer hangs onto heap sized to the
 4059   maximum input string it's ever seen, which can be a large memory
 4060   "leak" if you tokenize large strings with many threads across many
 4061   indices (Alex Chow via Mike McCandless)
 4062 
 4063 * LUCENE-6888: Explain output of map() function now also prints default value (janhoy)
 4064 
 4065 Other
 4066 
 4067 * LUCENE-6899: Upgrade randomizedtesting to 2.3.1. (Dawid Weiss)
 4068 
 4069 * LUCENE-6478: Test execution can hang with java.security.debug. (Dawid Weiss)
 4070 
 4071 * LUCENE-6862: Upgrade of RandomizedRunner to version 2.2.0. (Dawid Weiss)
 4072 
 4073 * LUCENE-6857: Validate StandardQueryParser with NOT operator 
 4074   with-in parantheses. (Jigar Shah via Dawid Weiss)
 4075 
 4076 * LUCENE-6827: Use explicit capacity ArrayList instead of a LinkedList 
 4077   in MultiFieldQueryNodeProcessor. (Dawid Weiss).
 4078 
 4079 * LUCENE-6812: Upgrade RandomizedTesting to 2.1.17. (Dawid Weiss)
 4080 
 4081 * LUCENE-6174: Improve "ant eclipse" to select right JRE for building.
 4082   (Uwe Schindler, Dawid Weiss)
 4083 
 4084 * LUCENE-6417, LUCENE-6830: Upgrade ANTLR used in expressions module
 4085   to version 4.5.1-1.  (Jack Conradson, Uwe Schindler)
 4086 
 4087 * LUCENE-6729: Upgrade ASM used in expressions module to version 5.0.4.
 4088   (Uwe Schindler)
 4089 
 4090 * LUCENE-6738: remove IndexWriterConfig.[gs]etIndexingChain
 4091   (Christine Poerschke)
 4092 
 4093 * LUCENE-6755: more tests of ToChildBlockJoinScorer.advance (hossman)
 4094 
 4095 * LUCENE-6571: fix some private access level javadoc errors and warnings
 4096   (Cao Manh Dat, Christine Poerschke)
 4097 
 4098 * LUCENE-6768: AbstractFirstPassGroupingCollector.groupSort private member
 4099   is not needed. (Christine Poerschke)
 4100 
 4101 * LUCENE-6761: MatchAllDocsQuery's Scorers do not expose approximations
 4102   anymore. (Adrien Grand)
 4103 
 4104 * LUCENE-6775, LUCENE-6833: Improved MorfologikFilterFactory to allow 
 4105   loading of custom dictionaries from ResourceLoader. Upgraded 
 4106   Morfologik to version 2.0.1. The 'dictionary' attribute has been
 4107   reverted back and now points at the dictionary resource to be 
 4108   loaded instead of the default Polish dictionary.
 4109   (Uwe Schindler, Dawid Weiss)
 4110 
 4111 * LUCENE-6797: Make GeoCircle an interface and use a factory to create
 4112   it, to eventually handle degenerate cases (Karl Wright via Mike
 4113   McCandless)
 4114 
 4115 * LUCENE-6800: Use XYZSolidFactory to create XYZSolids (Karl Wright
 4116   via Mike McCandless)
 4117 
 4118 * LUCENE-6798: Geo3d now models degenerate (too tiny) circles as a
 4119   single point (Karl Wright via Mike McCandless)
 4120 
 4121 * LUCENE-6770: Add javadocs that FSDirectory canonicalizes the path.
 4122   (Uwe Schindler, Vladimir Kuzmin)
 4123 
 4124 * LUCENE-6795: Fix various places where code used
 4125   AccessibleObject#setAccessible() without a privileged block. Code
 4126   without a hard requirement to do reflection were rewritten. This
 4127   makes Lucene and Solr ready for Java 9 Jigsaw's module system, where
 4128   reflection on Java's runtime classes is very restricted.
 4129   (Robert Muir, Uwe Schindler)
 4130 
 4131 * LUCENE-6467: Simplify Query.equals. (Paul Elschot via Adrien Grand)
 4132 
 4133 * LUCENE-6845: SpanScorer is now merged into Spans (Alan Woodward, David Smiley)
 4134 
 4135 * LUCENE-6887: DefaultSimilarity is deprecated, use ClassicSimilarity for equivalent behavior,
 4136   or consider switching to BM25Similarity which will become the new default in Lucene 6.0 (hossman)
 4137 
 4138 * LUCENE-6893: factor out CorePlusQueriesParser from CorePlusExtensionsParser
 4139   (Christine Poerschke)
 4140 
 4141 * LUCENE-6902: Don't retry to fsync files / directories; fail
 4142   immediately. (Daniel Mitterdorfer, Uwe Schindler)
 4143 
 4144 * LUCENE-6801: Clarify JavaDocs of PhraseQuery that it in fact supports terms
 4145   at the same position (as does MultiPhraseQuery), treated like a conjunction.
 4146   Added test. (David Smiley, Adrien Grand)
 4147 
 4148 Build
 4149 
 4150 * LUCENE-6732: Improve checker for invalid source patterns to also
 4151   detect javadoc-style license headers. Use Groovy to implement the
 4152   checks instead of plain Ant.  (Uwe Schindler)
 4153 
 4154 * LUCENE-6594: Update forbiddenapis to 2.0.  (Uwe Schindler)
 4155 
 4156 Tests
 4157 
 4158 * LUCENE-6752: Add Math#random() to forbiddenapis.  (Uwe Schindler,
 4159   Mikhail Khludnev, Andrei Beliakov)
 4160 
 4161 Changes in Backwards Compatibility Policy
 4162 
 4163 * LUCENE-6742: The Lovins & Finnish implementation of SnowballFilter
 4164   were fixed to now behave exactly like the original Snowball stemmer.
 4165   If you have indexed text using those stemmers you may need to reindex.
 4166   (Uwe Schindler, Robert Muir)
 4167 
 4168 Changes in Runtime Behavior
 4169 
 4170 * LUCENE-6772: MultiCollector now catches CollectionTerminatedException and
 4171   removes the collector that threw this exception from the list of sub
 4172   collectors to collect. (Adrien Grand)
 4173 
 4174 * LUCENE-6784: IndexSearcher's query caching is enabled by default. Run
 4175   indexSearcher.setQueryCache(null) to disable. (Adrien Grand)
 4176 
 4177 * LUCENE-6305: BooleanQuery.equals and hashcode do not depend on the order of
 4178   clauses anymore. (Adrien Grand)
 4179 
 4180 ======================= Lucene 5.3.2 =======================
 4181 
 4182 Bug Fixes
 4183 
 4184 * SOLR-7865: BlendedInfixSuggester was returning too many results
 4185   (Arcadius Ahouansou via Mike McCandless)
 4186 
 4187 ======================= Lucene 5.3.1 =======================
 4188 
 4189 Bug Fixes
 4190 
 4191 * LUCENE-6774: Remove classloader hack in MorfologikFilter. (Robert Muir,
 4192   Uwe Schindler)
 4193 
 4194 * LUCENE-6748: UsageTrackingQueryCachingPolicy no longer caches trivial queries
 4195   like MatchAllDocsQuery. (Adrien Grand)
 4196 
 4197 * LUCENE-6781: Fixed BoostingQuery to rewrite wrapped queries. (Adrien Grand)
 4198 
 4199 Tests
 4200 
 4201 * LUCENE-6760, SOLR-7958: Move TestUtil#randomWhitespace to the only
 4202   Solr test that is using it. The method is not useful for Lucene tests
 4203   (and easily breaks, e.g., in Java 9 caused by Unicode version updates).
 4204   (Uwe Schindler)
 4205 
 4206 
 4207 ======================= Lucene 5.3.0 =======================
 4208 
 4209 New Features
 4210 
 4211 * LUCENE-6485: Add CustomSeparatorBreakIterator to postings
 4212   highlighter which splits on any character. For example, it 
 4213   can be used with getMultiValueSeparator render whole field
 4214   values.  (Luca Cavanna via Robert Muir)
 4215 
 4216 * LUCENE-6459: Add common suggest API that mirrors Lucene's
 4217   Query/IndexSearcher APIs for Document based suggester.
 4218   Adds PrefixCompletionQuery, RegexCompletionQuery,
 4219   FuzzyCompletionQuery and ContextQuery.
 4220   (Areek Zillur via Mike McCandless)
 4221 
 4222 * LUCENE-6487: Spatial Geo3D API now has a WGS84 ellipsoid world model option.
 4223   (Karl Wright via David Smiley)
 4224 
 4225 * LUCENE-6477: Add experimental BKD geospatial tree doc values format
 4226   and queries, for fast "bbox/polygon contains lat/lon points" (Mike
 4227   McCandless)
 4228 
 4229 * LUCENE-6526: Asserting(Query|Weight|Scorer) now ensure scores are not computed
 4230   if they are not needed. (Adrien Grand)
 4231 
 4232 * LUCENE-6481: Add GeoPointField, GeoPointInBBoxQuery,
 4233   GeoPointInPolygonQuery for simple "indexed lat/lon point in
 4234   bbox/shape" searching.  (Nick Knize via Mike McCandless)
 4235 
 4236 * LUCENE-5954: The segments_N commit point now stores the Lucene
 4237   version that wrote the commit as well as the lucene version that
 4238   wrote the oldest segment in the index, for faster checking of "too
 4239   old" indices (Ryan Ernst, Robert Muir, Mike McCandless)
 4240 
 4241 * LUCENE-6519: BKDPointInPolygonQuery is much faster by avoiding
 4242   the per-hit polygon check when a leaf cell is fully contained by the
 4243   polygon.  (Nick Knize, Mike McCandless)
 4244 
 4245 * LUCENE-6549: Add preload option to MMapDirectory. (Robert Muir)
 4246 
 4247 * LUCENE-6504: Add Lucene53Codec, with norms implemented directly
 4248   via the Directory's RandomAccessInput api. (Robert Muir)
 4249 
 4250 * LUCENE-6539: Add new DocValuesNumbersQuery, to match any document
 4251   containing one of the specified long values.  This change also
 4252   moves the existing DocValuesTermsQuery and DocValuesRangeQuery
 4253   to Lucene's sandbox module, since in general these queries are
 4254   quite slow and are only fast in specific cases.  (Adrien Grand,
 4255   Robert Muir, Mike McCandless)
 4256 
 4257 * LUCENE-6577: Give earlier and better error message for invalid CRC.
 4258   (Robert Muir)
 4259 
 4260 * LUCENE-6544: Geo3D: (1) Regularize path & polygon construction, (2) add
 4261   PlanetModel.surfaceDistance() (ellipsoidal calculation), (3) cache lat & lon
 4262   in GeoPoint, (4) add thread-safety where missing -- Geo3dShape. (Karl Wright,
 4263   David Smiley)
 4264 
 4265 * LUCENE-6606: SegmentInfo.toString now confesses how the documents
 4266   were sorted, when SortingMergePolicy was used (Christine Poerschke
 4267   via Mike McCandless)
 4268 
 4269 * LUCENE-6524: IndexWriter can now be initialized from an already open
 4270   near-real-time or non-NRT reader.  (Boaz Leskes, Robert Muir, Mike
 4271   McCandless)
 4272 
 4273 * LUCENE-6578: Geo3D can now compute the distance from a point to a shape, both
 4274   inner distance and to an outside edge. Multiple distance algorithms are
 4275   available.  (Karl Wright, David Smiley)
 4276 
 4277 * LUCENE-6632: Geo3D: Compute circle planes more accurately.
 4278   (Karl Wright via David Smiley)
 4279 
 4280 * LUCENE-6653: Added general purpose BytesTermAttribute to basic token
 4281   attributes package that can be used for TokenStreams that solely produce
 4282   binary terms.  (Uwe Schindler)
 4283 
 4284 * LUCENE-6365: Add Operations.topoSort, to run topological sort of the
 4285   states in an Automaton (Markus Heiden via Mike McCandless)
 4286 
 4287 * LUCENE-6365: Replace Operations.getFiniteStrings with a
 4288   more scalable iterator API (FiniteStringsIterator) (Markus Heiden
 4289   via Mike McCandless)
 4290 
 4291 * LUCENE-6589: Add a new org.apache.lucene.search.join.CheckJoinIndex class
 4292   that can be used to validate that an index has an appropriate structure to
 4293   run join queries. (Adrien Grand)
 4294 
 4295 * LUCENE-6659: Remove IndexWriter's unnecessary hard limit on max concurrency
 4296   (Robert Muir, Mike McCandless)
 4297 
 4298 * LUCENE-6547: Add GeoPointDistanceQuery, matching all points within
 4299   the specified distance from the center point.  Fix
 4300   GeoPointInBBoxQuery to handle dateline crossing.
 4301 
 4302 * LUCENE-6694: Add LithuanianAnalyzer and LithuanianStemmer.
 4303   (Dainius Jocas via Robert Muir)
 4304 
 4305 * LUCENE-6695: Added a new BlendedTermQuery to blend statistics across several
 4306   terms. (Simon Willnauer, Adrien Grand)
 4307 
 4308 * LUCENE-6706: Added a new PayloadScoreQuery that generalises the behaviour of
 4309   PayloadTermQuery and PayloadNearQuery to all Span queries. (Alan Woodward)
 4310 
 4311 * LUCENE-6697: Add experimental range tree doc values format and
 4312   queries, based on a 1D version of the spatial BKD tree, for a faster
 4313   and smaller alternative to postings-based numeric and binary term
 4314   filtering.  Range trees can also handle values larger than 64 bits.
 4315   (Adrien Grand, Mike McCandless)
 4316 
 4317 * LUCENE-6647: Add GeoHash string utility APIs (Nick Knize via Mike
 4318   McCandless).
 4319 
 4320 * LUCENE-6710: GeoPointField now uses full 64 bits (up from 62) to encode
 4321   lat/lon (Nick Knize via Mike McCandless).
 4322 
 4323 * LUCENE-6580: SpanNearQuery now allows defined-width gaps in its subqueries
 4324   (Alan Woodward, Adrien Grand).
 4325 
 4326 * LUCENE-6712: Use doc values to post-filter GeoPointField hits that
 4327   fall in boundary cells, resulting in smaller index, faster searches
 4328   and less heap used for each query (Nick Knize via Mike McCandless).
 4329 
 4330 API Changes
 4331 
 4332 * LUCENE-6508: Simplify Lock api, there is now just 
 4333   Directory.obtainLock() which returns a Lock that can be 
 4334   released (or fails with exception). Add lock verification 
 4335   to IndexWriter. Improve exception messages when locking fails.
 4336   (Uwe Schindler, Mike McCandless, Robert Muir)
 4337 
 4338 * LUCENE-6371, LUCENE-6490: Payload collection from Spans is moved to a more generic
 4339   SpanCollector framework.  Spans no longer implements .hasPayload() and
 4340   .getPayload() methods, and instead exposes a collect() method that allows
 4341   the collection of arbitrary postings information. SpanPayloadCheckQuery and
 4342   SpanPayloadNearCheckQuery have moved from the .spans package to the .payloads
 4343   package. (Alan Woodward, David Smiley, Paul Elschot, Robert Muir)
 4344 
 4345 * LUCENE-6529: Removed an optimization in UninvertingReader that was causing
 4346   incorrect results for Numeric fields using precisionStep
 4347   (hossman, Robert Muir)
 4348 
 4349 * LUCENE-6551: Add missing ConcurrentMergeScheduler.getAutoIOThrottle
 4350   getter (Simon Willnauer, Mike McCandless)
 4351 
 4352 * LUCENE-6552: Add MergePolicy.OneMerge.getMergeInfo and rename
 4353   setInfo to setMergeInfo (Simon Willnauer, Mike McCandless)
 4354 
 4355 * LUCENE-6525: Deprecate IndexWriterConfig's writeLockTimeout.
 4356   (Robert Muir)
 4357 
 4358 * LUCENE-6583: FilteredQuery is deprecated and will be removed in 6.0. It should
 4359   be replaced with a BooleanQuery which handle the query as a MUST clause and
 4360   the filter as a FILTER clause. (Adrien Grand)
 4361 
 4362 * LUCENE-6553: The postings, spans and scorer APIs no longer take an acceptDocs
 4363   parameter. Live docs are now always checked on top of these APIs.
 4364   (Adrien Grand)
 4365 
 4366 * LUCENE-6634: PKIndexSplitter now takes a Query instead of a Filter to decide
 4367   how to split an index. (Adrien Grand)
 4368 
 4369 * LUCENE-6643: GroupingSearch from lucene/grouping was changed to take a Query
 4370   object to define groups instead of a Filter. (Adrien Grand)
 4371 
 4372 * LUCENE-6554: ToParentBlockJoinFieldComparator was removed because of a bug
 4373   with missing values that could not be fixed. ToParentBlockJoinSortField now
 4374   works with string or numeric doc values selectors. Sorting on anything else
 4375   than a string or numeric field would require to implement a custom selector.
 4376   (Adrien Grand)
 4377 
 4378 * LUCENE-6648: All lucene/facet APIs now take Query objects where they used to
 4379   take Filter objects. (Adrien Grand)
 4380 
 4381 * LUCENE-6640: Suggesters now take a BitsProducer object instead of a Filter
 4382   object to reduce the scope of doc IDs that may be returned, emphasizing the
 4383   fact that these objects need to support random-access. (Adrien Grand)
 4384 
 4385 * LUCENE-6646: Make EarlyTerminatingCollector take a Sort object directly
 4386   instead of a SortingMergePolicy. (Christine Poerschke via Adrien Grand)
 4387 
 4388 * LUCENE-6649: BitDocIdSetFilter and BitDocIdSetCachingWrapperFilter are now
 4389   deprecated in favour of BitSetProducer and QueryBitSetProducer, which do not
 4390   extend oal.search.Filter. (Adrien Grand)
 4391 
 4392 * LUCENE-6607: Factor out geo3d into its own spatial3d module.  (Karl
 4393   Wright, Nick Knize, David Smiley, Mike McCandless)
 4394 
 4395 * LUCENE-6531: PhraseQuery is now immutable and can be built using the
 4396   PhraseQuery.Builder class. (Adrien Grand)
 4397 
 4398 * LUCENE-6570: BooleanQuery is now immutable and can be built using the
 4399   BooleanQuery.Builder class. (Adrien Grand)
 4400 
 4401 * LUCENE-6702: NRTSuggester: Add a method to inject context values at index time
 4402   in ContextSuggestField. Simplify ContextQuery logic for extracting contexts and
 4403   add dedicated method to consider all context values at query time.
 4404   (Areek Zillur, Mike McCandless)
 4405 
 4406 * LUCENE-6719: NumericUtils getMinInt, getMaxInt, getMinLong, getMaxLong now
 4407   return null if there are no terms for the specified field, previously these
 4408   methods returned primitive values and raised an undocumented NullPointerException
 4409   if there were no terms for the field. (hossman, Timothy Potter)
 4410 
 4411 Bug fixes
 4412 
 4413 * LUCENE-6500: ParallelCompositeReader did not always call
 4414   closed listeners. This was fixed by LUCENE-6501.
 4415   (Adrien Grand, Uwe Schindler)
 4416 
 4417 * LUCENE-6520: Geo3D GeoPath.done() would throw an NPE if adjacent path
 4418   segments were co-linear. (Karl Wright via David Smiley)
 4419 
 4420 * LUCENE-5805: QueryNodeImpl.removeFromParent was doing nothing in a
 4421   costly manner (Christoph Kaser, Cao Manh Dat via Mike McCAndless)
 4422 
 4423 * LUCENE-6533: SlowCompositeReaderWrapper no longer caches its live docs
 4424   instance since this can prevent future improvements like a
 4425   disk-backed live docs (Adrien Grand, Mike McCandless)
 4426 
 4427 * LUCENE-6558: Highlighters now work with CustomScoreQuery (Cao Manh
 4428   Dat via Mike McCandless)
 4429 
 4430 * LUCENE-6560: BKDPointInBBoxQuery now handles "dateline crossing"
 4431   correctly (Nick Knize, Mike McCandless)
 4432 
 4433 * LUCENE-6564: Change PrintStreamInfoStream to use thread safe Java 8
 4434   ISO-8601 date formatting (in Lucene 5.x use Java 7 FileTime#toString
 4435   as workaround); fix output of tests to use same format.  (Uwe Schindler,
 4436   Ramkumar Aiyengar)
 4437 
 4438 * LUCENE-6593: Fixed ToChildBlockJoinQuery's scorer to not refuse to advance
 4439   to a document that belongs to the parent space. (Adrien Grand)
 4440 
 4441 * LUCENE-6591: Never write a negative vLong (Robert Muir, Ryan Ernst,
 4442   Adrien Grand, Mike McCandless)
 4443 
 4444 * LUCENE-6588: Fix how ToChildBlockJoinQuery deals with acceptDocs.
 4445   (Christoph Kaser via Adrien Grand)
 4446 
 4447 * LUCENE-6597: Geo3D's GeoCircle now supports a world-globe diameter.
 4448   (Karl Wright via David Smiley)
 4449 
 4450 * LUCENE-6608: Fix potential resource leak in BigramDictionary.
 4451   (Rishabh Patel via Uwe Schindler)
 4452 
 4453 * LUCENE-6614: Improve partition detection in IOUtils#spins() so it
 4454   works with NVMe drives.  (Uwe Schindler, Mike McCandless)
 4455 
 4456 * LUCENE-6586: Fix typo in GermanStemmer, causing possible wrong value
 4457   for substCount.  (Christoph Kaser via Mike McCandless)
 4458 
 4459 * LUCENE-6658: Fix IndexUpgrader to also upgrade indexes without any
 4460   segments.  (Trejkaz, Uwe Schindler)
 4461 
 4462 * LUCENE-6677: QueryParserBase fails to enforce maxDeterminizedStates when
 4463   creating a WildcardQuery (David Causse via Mike McCandless)
 4464 
 4465 * LUCENE-6680: Preserve two suggestions that have same key and weight but
 4466   different payloads (Arcadius Ahouansou via Mike McCandless)
 4467 
 4468 * LUCENE-6681: SortingMergePolicy must override MergePolicy.size(...).
 4469   (Christine Poerschke via Adrien Grand)
 4470   
 4471 * LUCENE-6682: StandardTokenizer performance bug: scanner buffer is
 4472   unnecessarily copied when maxTokenLength doesn't change.  Also stop silently
 4473   maxing out buffer size (and effectively also max token length) at 1M chars,
 4474   but instead throw an exception from setMaxTokenLength() when the given
 4475   length is greater than 1M chars.  (Piotr Idzikowski, Steve Rowe)
 4476 
 4477 * LUCENE-6696: Fix FilterDirectoryReader.close() to never close the
 4478   underlying reader several times. (Adrien Grand)
 4479 
 4480 * LUCENE-6334: FastVectorHighlighter failed to highlight phrases across
 4481   more than one value in a multi-valued field. (Chris Earle, Nik Everett
 4482   via Mike McCandless)
 4483 
 4484 * LUCENE-6704: GeoPointDistanceQuery was visiting too many term ranges,
 4485   consuming too much heap for a large radius (Nick Knize via Mike McCandless)
 4486   
 4487 * SOLR-5882: fix ScoreMode.Min at ToParentBlockJoinQuery (Mikhail Khludnev)
 4488 
 4489 * LUCENE-6718: JoinUtil.createJoinQuery failed to rewrite queries before
 4490   creating a Weight. (Adrien Grand)
 4491 
 4492 * LUCENE-6713: TooComplexToDeterminizeException claims to be serializable
 4493   but wasn't (Simon Willnauer, Mike McCandless)
 4494 
 4495 * LUCENE-6723: Fix date parsing problems in Java 9 with date formats using
 4496   English weekday/month names.  (Uwe Schindler)
 4497 
 4498 * LUCENE-6618: Properly set MMapDirectory.UNMAP_SUPPORTED when it is now allowed
 4499   by security policy. (Robert Muir)
 4500 
 4501 Changes in Runtime Behavior
 4502 
 4503 * LUCENE-6501: The subreader structure in ParallelCompositeReader
 4504   was flattened, because the current implementation had too many
 4505   hidden bugs regarding refounting and close listeners.
 4506   If you create a new ParallelCompositeReader, it will just take
 4507   all leaves of the passed readers and form a flat structure of
 4508   ParallelLeafReaders instead of trying to assemble the original
 4509   structure of composite and leaf readers.  (Adrien Grand,
 4510   Uwe Schindler)
 4511 
 4512 * LUCENE-6537: NearSpansOrdered no longer tries to minimize its
 4513   Span matches.  This means that the matching algorithm is entirely
 4514   lazy.  All spans returned by the previous implementation are still 
 4515   reported, but matching documents may now also return additional 
 4516   spans that were previously discarded in preference to shorter 
 4517   overlapping ones. (Alan Woodward, Adrien Grand, Paul Elschot)
 4518 
 4519 * LUCENE-6538: Also include java.vm.version and java.runtime.version
 4520   in per-segment diagnostics (Robert Muir, Mike McCandless)
 4521 
 4522 * LUCENE-6569: Optimize MultiFunction.anyExists and allExists to eliminate
 4523   excessive array creation in common 2 argument usage (Jacob Graves, hossman)
 4524 
 4525 * LUCENE-2880: Span queries now score more consistently with regular queries.
 4526   (Robert Muir, Adrien Grand)
 4527 
 4528 * LUCENE-6601: FilteredQuery now always rewrites to a BooleanQuery which handles
 4529   the query as a MUST clause and the filter as a FILTER clause.
 4530   LEAP_FROG_QUERY_FIRST_STRATEGY and LEAP_FROG_FILTER_FIRST_STRATEGY do not
 4531   guarantee anymore which iterator will be advanced first, it will depend on the
 4532   respective costs of the iterators. QUERY_FIRST_FILTER_STRATEGY and
 4533   RANDOM_ACCESS_FILTER_STRATEGY still consume the filter using its random-access
 4534   API, however the returned bits may be called on different documents compared
 4535   to before. (Adrien Grand)
 4536 
 4537 * LUCENE-6542: FSDirectory's ctor now works with security policies or file systems
 4538   that restrict write access.  (Trejkaz, hossman, Uwe Schindler)
 4539 
 4540 * LUCENE-6651: The default implementation of AttributeImpl#reflectWith(AttributeReflector)
 4541   now uses AccessControler#doPrivileged() to do the reflection. Please consider
 4542   implementing this method in all your custom attributes, because the method will be
 4543   made abstract in Lucene 6.  (Uwe Schindler)
 4544 
 4545 * LUCENE-6639: LRUQueryCache and CachingWrapperQuery now consider a query as
 4546   "used" when the first Scorer is pulled instead of when a Scorer is pulled on
 4547   the first segment on an index. (Terry Smith, Adrien Grand)
 4548 
 4549 * LUCENE-6579: IndexWriter now sacrifices (closes) itself to protect the index
 4550   when an unexpected, tragic exception strikes while merging. (Robert
 4551   Muir, Mike McCandless) 
 4552 
 4553 * LUCENE-6691: SortingMergePolicy.isSorted now considers FilterLeafReader instances.
 4554   EarlyTerminatingSortingCollector.terminatedEarly accessor added.
 4555   TestEarlyTerminatingSortingCollector.testTerminatedEarly test added.
 4556   (Christine Poerschke)
 4557 
 4558 * LUCENE-6609: Add getSortField impls to many subclasses of FieldCacheSource which return
 4559   the most direct SortField implementation.  In many trivial sort by ValueSource usages, this
 4560   will result in less RAM, and more precise sorting of extreme values due to no longer
 4561   converting to double. (hossman)
 4562 
 4563 Optimizations
 4564 
 4565 * LUCENE-6548: Some optimizations for BlockTree's intersect with very
 4566   finite automata (Mike McCandless)
 4567 
 4568 * LUCENE-6585: Flatten conjunctions and conjunction approximations into
 4569   parent conjunctions. For example a sloppy phrase query of "foo bar"~5
 4570   with a filter of "baz" will internally leapfrog foo,bar,baz as one
 4571   conjunction. (Ryan Ernst, Robert Muir, Adrien Grand)
 4572 
 4573 * LUCENE-6325: Reduce RAM usage of FieldInfos, and speed up lookup by
 4574   number, by using an array instead of TreeMap except in very sparse
 4575   cases (Robert Muir, Mike McCandless)
 4576 
 4577 * LUCENE-6617: Reduce heap usage for small FSTs (Mike McCandless)
 4578 
 4579 * LUCENE-6616: IndexWriter now lists the files in the index directory
 4580   only once on init, and IndexFileDeleter no longer suppresses
 4581   FileNotFoundException and NoSuchFileException.  This also improves
 4582   IndexFileDeleter to delete segments_N files last, so that in the
 4583   presence of a virus checker, the index is never left in a state
 4584   where an expired segments_N references non-existing files (Robert
 4585   Muir, Mike McCandless)
 4586 
 4587 * LUCENE-6645: Optimized the way we merge postings lists in multi-term queries
 4588   and TermsQuery. This should especially help when there are lots of small
 4589   postings lists. (Adrien Grand, Mike McCandless)
 4590 
 4591 * LUCENE-6668: Optimized storage for sorted set and sorted numeric doc values
 4592   in the case that there are few unique sets of values.
 4593   (Adrien Grand, Robert Muir)
 4594 
 4595 * LUCENE-6690: Sped up MultiTermsEnum.next() on high-cardinality fields.
 4596   (Adrien Grand)
 4597 
 4598 * LUCENE-6621: Removed two unused variables in analysis/stempel/src/java/org/
 4599   egothor/stemmer/Compile.java
 4600   (Rishabh Patel via Christine Poerschke)
 4601     
 4602 Build
 4603 
 4604 * LUCENE-6518: Don't report false thread leaks from IBM J9
 4605   ClassCache Reaper in test framework. (Dawid Weiss)
 4606 
 4607 * LUCENE-6567: Simplify payload checking in SpanPayloadCheckQuery (Alan
 4608   Woodward)
 4609 
 4610 * LUCENE-6568: Make rat invocation depend on ivy configuration being set up
 4611   (Ramkumar Aiyengar)
 4612   
 4613 * LUCENE-6683: ivy-fail goal directs people to non-existent page
 4614   (Mike Drob via Steve Rowe)
 4615 
 4616 * LUCENE-6693: Updated Groovy to 2.4.4, Pegdown to 1.5, Svnkit to 1.8.10.
 4617   Also fixed some PermGen errors while running full build caused by
 4618   these updates: Tasks are now installed from root's build.xml.
 4619   (Uwe Schindler)
 4620 
 4621 * LUCENE-6741: Fix jflex files to regenerate the java files correctly.
 4622   (Uwe Schindler)
 4623 
 4624 Test Framework
 4625 
 4626 * LUCENE-6637: Fix FSTTester to not violate file permissions
 4627   on -Dtests.verbose=true.  (Mesbah M. Alam, Uwe Schindler)
 4628 
 4629 * LUCENE-6542: LuceneTestCase now has runWithRestrictedPermissions() to run
 4630   an action with reduced permissions. This can be used to simulate special
 4631   environments (e.g., read-only dirs). If tests are running without a security
 4632   manager, an assume cancels test execution automatically.  (Uwe Schindler)
 4633 
 4634 * LUCENE-6652: Removed lots of useless Byte(s)TermAttributes all over test
 4635   infrastructure.  (Uwe Schindler)
 4636 
 4637 * LUCENE-6563: Improve MockFileSystemTestCase.testURI to check if a path
 4638   can be encoded according to local filesystem requirements. Otherwise
 4639   stop test execution.  (Christine Poerschke via Uwe Schindler)
 4640 
 4641 Changes in Backwards Compatibility Policy
 4642 
 4643 * LUCENE-6553: The iterator returned by the LeafReader.postings method now
 4644   always includes deleted docs, so you have to check for deleted documents on
 4645   top of the iterator. (Adrien Grand)
 4646 
 4647 * LUCENE-6633: DuplicateFilter has been deprecated and will be removed in 6.0.
 4648   DiversifiedTopDocsCollector can be used instead with a maximum number of hits
 4649   per key equal to 1. (Adrien Grand)
 4650 
 4651 * LUCENE-6653: The workflow for consuming the TermToBytesRefAttribute was changed:
 4652   getBytesRef() now does all work and is called on each token, fillBytesRef()
 4653   was removed. The implementation is free to reuse the internal BytesRef
 4654   or return a new one on each call.  (Uwe Schindler)
 4655 
 4656 * LUCENE-6682: StandardTokenizer.setMaxTokenLength() now throws an exception if
 4657   a length greater than 1M chars is given.  Previously the effective max token
 4658   length (the scanner's buffer) was capped at 1M chars, but getMaxTokenLength()
 4659   incorrectly returned the previously requested length, even when it exceeded 1M.
 4660   (Piotr Idzikowski, Steve Rowe)
 4661 
 4662 
 4663 ======================= Lucene 5.2.1 =======================
 4664 
 4665 Bug Fixes
 4666 
 4667 * LUCENE-6482: Fix class loading deadlock relating to Codec initialization,
 4668   default codec and SPI discovery.  (Shikhar Bhushan, Uwe Schindler)
 4669 
 4670 * LUCENE-6523: NRT readers now reflect a new commit even if there is
 4671   no change to the commit user data (Mike McCandless)
 4672 
 4673 * LUCENE-6527: Queries now get a dummy Similarity when scores are not needed
 4674   in order to not load unnecessary information like norms. (Adrien Grand)
 4675 
 4676 * LUCENE-6559: TimeLimitingCollector now also checks for timeout when a new
 4677   leaf reader is pulled ie. if we move from one segment to another even without
 4678   collecting a hit. (Simon Willnauer)
 4679 
 4680 ======================= Lucene 5.2.0 =======================
 4681 
 4682 New Features
 4683 
 4684 * LUCENE-6308, LUCENE-6385, LUCENE-6391: Span queries now share 
 4685   document conjunction/intersection
 4686   code with boolean queries, and use two-phased iterators for
 4687   faster intersection by avoiding loading positions in certain cases.
 4688   (Paul Elschot, Terry Smith, Robert Muir via Mike McCandless)
 4689   
 4690 * LUCENE-6393: Add two-phase support to SpanPositionCheckQuery
 4691   and its subclasses: SpanPositionRangeQuery, SpanPayloadCheckQuery, 
 4692   SpanNearPayloadCheckQuery, SpanFirstQuery. (Paul Elschot, Robert Muir)
 4693 
 4694 * LUCENE-6394: Add two-phase support to SpanNotQuery and refactor
 4695   FilterSpans to just have an accept(Spans candidate) method for
 4696   subclasses. (Robert Muir)
 4697 
 4698 * LUCENE-6373: SpanOrQuery shares disjunction logic with boolean
 4699   queries, and supports two-phased iterators to avoid loading
 4700   positions when possible. (Paul Elschot via Robert Muir)
 4701 
 4702 * LUCENE-6352, LUCENE-6472: Added a new query time join to the join module
 4703   that uses global ordinals, which is faster for subsequent joins between
 4704   reopens. (Martijn van Groningen, Adrien Grand)
 4705 
 4706 * LUCENE-5879: Added experimental auto-prefix terms to BlockTree terms
 4707   dictionary, exposed as AutoPrefixPostingsFormat (Adrien Grand,
 4708   Uwe Schindler, Robert Muir, Mike McCandless)
 4709 
 4710 * LUCENE-5579: New CompositeSpatialStrategy combines speed of RPT with
 4711   accuracy of SDV. Includes optimized Intersect predicate to avoid many
 4712   geometry checks. Uses TwoPhaseIterator. (David Smiley)
 4713 
 4714 * LUCENE-5989: Allow passing BytesRef to StringField to make it easier
 4715   to index arbitrary binary tokens, and change the experimental
 4716   StoredFieldVisitor.stringField API to take UTF-8 byte[] instead of
 4717   String (Mike McCandless)
 4718 
 4719 * LUCENE-6389: Added ScoreMode.Min that aggregates the lowest child score
 4720   to the parent hit. (Martijn van Groningen, Adrien Grand)
 4721 
 4722 * LUCENE-6423: New LimitTokenOffsetFilter that limits tokens to those before
 4723   a configured maximum start offset. (David Smiley)
 4724 
 4725 * LUCENE-6422: New spatial PackedQuadPrefixTree, a generally more efficient
 4726   choice than QuadPrefixTree, especially for high precision shapes.
 4727   When used, you should typically disable RPT's pruneLeafyBranches option.
 4728   (Nick Knize, David Smiley)
 4729 
 4730 * LUCENE-6451: Expressions now support bindings keys that look like
 4731   zero arg functions (Jack Conradson via Ryan Ernst)
 4732 
 4733 * LUCENE-6083: Add SpanWithinQuery and SpanContainingQuery that return
 4734   spans inside of / containing another spans. (Paul Elschot via Robert Muir)
 4735 
 4736 * LUCENE-6454: Added distinction between member variable and method in
 4737   expression helper VariableContext
 4738   (Jack Conradson via Ryan Ernst)
 4739 
 4740 * LUCENE-6196: New Spatial "Geo3d" API with partial Spatial4j integration.
 4741   It is a set of shapes implemented using 3D planar geometry for calculating
 4742   spatial relations on the surface of a sphere. Shapes include Point, BBox,
 4743   Circle, Path (buffered line string), and Polygon.
 4744   (Karl Wright via David Smiley)
 4745 
 4746 * LUCENE-6464: Add a new expert lookup method to
 4747   AnalyzingInfixSuggester to accept an arbitrary BooleanQuery to
 4748   express how contexts should be filtered. (Arcadius Ahouansou via
 4749   Mike McCandless)
 4750 
 4751 Optimizations
 4752 
 4753 * LUCENE-6379: IndexWriter.deleteDocuments(Query...) now detects if
 4754   one of the queries is MatchAllDocsQuery and just invokes the much
 4755   faster IndexWriter.deleteAll in that case (Robert Muir, Adrien
 4756   Grand, Mike McCandless)
 4757 
 4758 * LUCENE-6388: Optimize SpanNearQuery when payloads are not present.
 4759   (Robert Muir)
 4760 
 4761 * LUCENE-6421: Defer reading of positions in MultiPhraseQuery until
 4762   they are needed. (Robert Muir)
 4763 
 4764 * LUCENE-6392: Highligher- reduce memory of tokens in
 4765   TokenStreamFromTermVector, and add maxStartOffset limit. (David Smiley)
 4766 
 4767 * LUCENE-6456: Queries that generate doc id sets that are too large for the
 4768   query cache are not cached instead of evicting everything. (Adrien Grand)
 4769 
 4770 * LUCENE-6455: Require a minimum index size to enable query caching in order
 4771   not to cache eg. on MemoryIndex. (Adrien Grand)
 4772 
 4773 * LUCENE-6330: BooleanScorer (used for top-level disjunctions) does not decode
 4774   norms when not necessary anymore. (Adrien Grand)
 4775 
 4776 * LUCENE-6350: TermsQuery is now compressed with PrefixCodedTerms.
 4777   (Robert Muir, Mike McCandless, Adrien Grand)
 4778 
 4779 * LUCENE-6458: Multi-term queries matching few terms per segment now execute
 4780   like a disjunction. (Adrien Grand)
 4781 
 4782 * LUCENE-6360: TermsQuery rewrites to a disjunction when there are 16 matching
 4783   terms or less. (Adrien Grand)
 4784 
 4785 Bug Fixes
 4786 
 4787 * LUCENE-329: Fix FuzzyQuery defaults to rank exact matches highest.
 4788   (Mark Harwood, Adrien Grand)
 4789 
 4790 * LUCENE-6378: Fix all RuntimeExceptions to throw the underlying root cause.
 4791   (Varun Thacker, Adrien Grand, Mike McCandless)
 4792 
 4793 * LUCENE-6415: TermsQuery.extractTerms is a no-op (used to throw an
 4794   UnsupportedOperationException). (Adrien Grand)
 4795 
 4796 * LUCENE-6416: BooleanQuery.extractTerms now only extracts terms from scoring
 4797   clauses. (Adrien Grand)
 4798 
 4799 * LUCENE-6409: Fixed integer overflow in LongBitSet.ensureCapacity.
 4800   (Luc Vanlerberghe via Adrien Grand)
 4801 
 4802 * LUCENE-6424, LUCENE-6430: Fix many bugs with mockfs filesystems in the 
 4803   test-framework: always consistently wrap Path, fix buggy behavior for 
 4804   globs, implement equals/hashcode for filtered Paths, etc.  
 4805   (Ryan Ernst, Simon Willnauer, Robert Muir)
 4806 
 4807 * LUCENE-6426: Fix FieldType's copy constructor to also copy over the numeric
 4808   precision step. (Adrien Grand)
 4809 
 4810 * LUCENE-6345: Null check terms/fields in Lucene queries (Lee
 4811   Hinman via Mike McCandless)
 4812 
 4813 * LUCENE-6400: SolrSynonymParser should preserve original token instead
 4814   of replacing it with a synonym, when expand=true and there is no
 4815   explicit mapping (Ian Ribas, Robert Muir, Mike McCandless)
 4816 
 4817 * LUCENE-6449: Don't throw NullPointerException if some segments are
 4818   missing the field being highlighted, in PostingsHighlighter (Roman
 4819   Khmelichek via Mike McCandless)
 4820 
 4821 * LUCENE-6427: Added assertion about the presence of ghost bits in
 4822   (Fixed|Long)BitSet. (Luc Vanlerberghe via Adrien Grand)
 4823 
 4824 * LUCENE-6468: Fixed NPE with empty Kuromoji user dictionary.
 4825   (Jun Ohtani via Christian Moen)
 4826 
 4827 * LUCENE-6483: Ensure core closed listeners are called on the same cache key as
 4828   the reader which has been used to register the listener. (Adrien Grand)
 4829 
 4830 * LUCENE-6486 DocumentDictionary iterator no longer skips
 4831   documents with no payloads and now returns an empty BytesRef instead
 4832   (Marius Grama via Michael McCandless)
 4833 
 4834 * LUCENE-6505: NRT readers now reflect segments_N filename and commit
 4835   user data from previous commits (Mike McCandless)
 4836 
 4837 * LUCENE-6507: Don't let NativeFSLock.close() release other locks
 4838   (Simon Willnauer, Robert Muir, Uwe Schindler, Mike McCandless)
 4839 
 4840 API Changes
 4841 
 4842 * LUCENE-6377: SearcherFactory#newSearcher now accepts the previous reader
 4843   to simplify warming logic during opening new searchers. (Simon Willnauer)
 4844 
 4845 * LUCENE-6410: Removed unused "reuse" parameter to
 4846   Terms.iterator. (Robert Muir, Mike McCandless)
 4847 
 4848 * LUCENE-6425: Replaced Query.extractTerms with Weight.extractTerms.
 4849   (Adrien Grand)
 4850 
 4851 * LUCENE-6446: Simplified Explanation API. (Adrien Grand)
 4852 
 4853 * LUCENE-6445: Two new methods in Highlighter's TokenSources; the existing
 4854   methods are now marked deprecated. (David Smiley)
 4855 
 4856 * LUCENE-6484: Removed EliasFanoDocIdSet, which was unused.
 4857   (Paul Elschot via Adrien Grand)
 4858 
 4859 * LUCENE-6466: Moved SpanQuery.getSpans() and .extractTerms() to SpanWeight
 4860   (Alan Woodward, Robert Muir)
 4861 
 4862 * LUCENE-6497: Allow subclasses of FieldType to check frozen state
 4863   (Ryan Ernst)
 4864 
 4865 Other
 4866 
 4867 * LUCENE-6413: Test runner should report the number of suites completed/ 
 4868   remaining. (Dawid Weiss)
 4869 
 4870 * LUCENE-5439: Add 'ant jacoco' build target. (Robert Muir)
 4871 
 4872 * LUCENE-6315: Simplify the private iterator Lucene uses internally
 4873   when resolving deleted terms to matched docids. (Robert Muir, Adrien
 4874   Grand, Mike McCandless)
 4875 
 4876 * LUCENE-6399: Benchmark module's QueryMaker.resetInputs should call setConfig
 4877   so queries can react to property changes in new rounds. (David Smiley)
 4878 
 4879 * LUCENE-6382: Lucene now enforces that positions never exceed the
 4880   maximum value IndexWriter.MAX_POSITION.  (Robert Muir, Mike McCandless)
 4881 
 4882 * LUCENE-6372: Simplified and improved equals/hashcode of span queries.
 4883   (Paul Elschot via Adrien Grand)
 4884 
 4885 Build
 4886 
 4887 * LUCENE-6420: Update forbiddenapis to v1.8  (Uwe Schindler)
 4888 
 4889 Test Framework
 4890 
 4891 * LUCENE-6419: Added two-phase iteration assertions to AssertingQuery.
 4892   (Adrien Grand)
 4893 
 4894 * LUCENE-6437: Randomly set CPU core count and spins, derived from
 4895   test's master seed, used by ConcurrentMergeScheduler to set dynamic
 4896   defaults, for better test randomization and to help tests reproduce
 4897   (Robert Muir, Mike McCandless)
 4898 
 4899 ======================= Lucene 5.1.0 =======================
 4900 
 4901 New Features
 4902 
 4903 * LUCENE-6066: Added DiversifiedTopDocsCollector to misc for collecting no more 
 4904   than a given number of results under a choice of key. Introduces new remove 
 4905   method to core's PriorityQueue. (Mark Harwood)
 4906 
 4907 * LUCENE-6191: New spatial 2D heatmap faceting for PrefixTreeStrategy. (David Smiley)
 4908 
 4909 * LUCENE-6227: Added BooleanClause.Occur.FILTER to filter documents without
 4910   participating in scoring (on the contrary to MUST). (Adrien Grand)
 4911 
 4912 * LUCENE-6294: Added oal.search.CollectorManager to allow for parallelization
 4913   of the document collection process on IndexSearcher. (Adrien Grand)
 4914 
 4915 * LUCENE-6303: Added filter caching baked into IndexSearcher, disabled by
 4916   default. (Adrien Grand)
 4917 
 4918 * LUCENE-6304: Added a new MatchNoDocsQuery that matches no documents.
 4919   (Lee Hinman via Adrien Grand)
 4920 
 4921 * LUCENE-6341: Add a -fast option to CheckIndex. (Robert Muir)
 4922 
 4923 * LUCENE-6355: IndexWriter's infoStream now also logs time to write FieldInfos
 4924   during merge (Lee Hinman via Mike McCandless)
 4925 
 4926 * LUCENE-6339: Added Near-real time Document Suggester via custom postings format
 4927   (Areek Zillur, Mike McCandless, Simon Willnauer)
 4928 
 4929 Bug Fixes
 4930 
 4931 * LUCENE-6368: FST.save can truncate output (BufferedOutputStream may be closed 
 4932   after the underlying stream). (Ippei Matsushima via Dawid Weiss)
 4933 
 4934 * LUCENE-6249: StandardQueryParser doesn't support pure negative clauses. 
 4935   (Dawid Weiss)
 4936   
 4937 * LUCENE-6190: Spatial pointsOnly flag on PrefixTreeStrategy shouldn't switch all predicates to
 4938   Intersects. (David Smiley)
 4939 
 4940 * LUCENE-6242: Ram usage estimation was incorrect for SparseFixedBitSet when
 4941   object alignment was different from 8. (Uwe Schindler, Adrien Grand)
 4942 
 4943 * LUCENE-6293: Fixed TimSorter bug. (Adrien Grand)
 4944 
 4945 * LUCENE-6001: DrillSideways hits NullPointerException for certain
 4946   BooleanQuery searches.  (Dragan Jotannovic, jane chang via Mike
 4947   McCandless)
 4948 
 4949 * LUCENE-6311: Fix NIOFSDirectory and SimpleFSDirectory so that the
 4950   toString method of IndexInputs confess when they are from a compound
 4951   file. (Robert Muir, Mike McCandless)
 4952 
 4953 * LUCENE-6381: Add defensive wait time limit in
 4954   DocumentsWriterStallControl to prevent hangs during indexing if we
 4955   miss a .notify/All somewhere (Mike McCandless)
 4956 
 4957 * LUCENE-6386: Correct IndexWriter.forceMerge documentation to state
 4958   that up to 3X (X = current index size) spare disk space may be needed
 4959   to complete forceMerge(1).  (Robert Muir, Shai Erera, Mike McCandless)
 4960 
 4961 * LUCENE-6395: Seeking by term ordinal was failing to set the term's
 4962   bytes in MemoryIndex (Mike McCandless)
 4963 
 4964 * LUCENE-6429: Removed the TermQuery(Term,int) constructor which could lead to
 4965   inconsistent term statistics. (Adrien Grand, Robert Muir)
 4966 
 4967 Optimizations
 4968 
 4969 * LUCENE-6183, LUCENE-5647: Avoid recompressing stored fields
 4970   and term vectors when merging segments without deletions. 
 4971   Lucene50Codec's BEST_COMPRESSION mode uses a higher deflate 
 4972   level for more compact storage.  (Robert Muir)
 4973 
 4974 * LUCENE-6184: Make BooleanScorer only score windows that contain
 4975   matches. (Adrien Grand)
 4976 
 4977 * LUCENE-6161: Speed up resolving of deleted terms to docIDs by doing
 4978   a combined merge sort between deleted terms and segment terms
 4979   instead of a separate merge sort for each segment.  In delete-heavy
 4980   use cases this can be a sizable speedup. (Mike McCandless)
 4981 
 4982 * LUCENE-6201: BooleanScorer can now deal with values of minShouldMatch that
 4983   are greater than one and is used when queries produce dense result sets.
 4984   (Adrien Grand)
 4985 
 4986 * LUCENE-6218: Don't decode frequencies or match all positions when scoring
 4987   is not needed. (Robert Muir)
 4988 
 4989 * LUCENE-6233 Speed up CheckIndex when the index has term vectors
 4990   (Robert Muir, Mike McCandless)
 4991 
 4992 * LUCENE-6198: Added the TwoPhaseIterator API, exposed on scorers which
 4993   is for now only used on phrase queries and conjunctions in order to check
 4994   positions lazily if the phrase query is in a conjunction with other queries.
 4995   (Robert Muir, Adrien Grand, David Smiley)
 4996 
 4997 * LUCENE-6244, LUCENE-6251: All boolean queries but those that have a
 4998   minShouldMatch > 1 now either propagate or take advantage of the two-phase
 4999   iteration capabilities added in LUCENE-6198. (Adrien Grand, Robert Muir)
 5000 
 5001 * LUCENE-6241: FSDirectory.listAll() doesnt filter out subdirectories anymore,
 5002   for faster performance. Subdirectories don't matter to Lucene. If you need to
 5003   filter out non-index files with some custom usage, you may want to look at 
 5004   the IndexFileNames class. (Robert Muir)
 5005 
 5006 * LUCENE-6262: ConstantScoreQuery does not wrap the inner weight anymore when
 5007   scores are not required. (Adrien Grand)
 5008 
 5009 * LUCENE-6263: MultiCollector automatically caches scores when several
 5010   collectors need them. (Adrien Grand)
 5011 
 5012 * LUCENE-6275: SloppyPhraseScorer now uses the same logic as ConjunctionScorer
 5013   in order to advance doc IDs, which takes advantage of the cost() API.
 5014   (Adrien Grand)
 5015 
 5016 * LUCENE-6290: QueryWrapperFilter propagates approximations and FilteredQuery
 5017   rewrites to a BooleanQuery when the filter is a QueryWrapperFilter in order
 5018   to leverage approximations. (Adrien Grand)
 5019 
 5020 * LUCENE-6318: Reduce RAM usage of FieldInfos when there are many fields.
 5021   (Mike McCandless, Robert Muir)
 5022 
 5023 * LUCENE-6320: Speed up CheckIndex. (Robert Muir)
 5024 
 5025 * LUCENE-4942: Optimized the encoding of PrefixTreeStrategy indexes for
 5026   non-point data: 33% smaller index, 68% faster indexing, and 44% faster
 5027   searching. YMMV (David Smiley)
 5028 
 5029 API Changes
 5030 
 5031 * LUCENE-6204, LUCENE-6208: Simplify CompoundFormat: remove files()
 5032   and remove files parameter to write(). (Robert Muir)
 5033 
 5034 * LUCENE-6217: Add IndexWriter.isOpen and getTragicException.  (Simon
 5035   Willnauer, Mike McCandless)
 5036 
 5037 * LUCENE-6218, LUCENE-6220: Add Collector.needsScores() and needsScores
 5038   parameter to Query.createWeight(). (Robert Muir, Adrien Grand)
 5039 
 5040 * LUCENE-4524, LUCENE-6246, LUCENE-6256, LUCENE-6271: Merge DocsEnum and DocsAndPositionsEnum
 5041   into a single PostingsEnum iterator.  TermsEnum.docs() and TermsEnum.docsAndPositions()
 5042   are replaced by TermsEnum.postings(). 
 5043   (Alan Woodward, Simon Willnauer, Robert Muir, Ryan Ernst)
 5044 
 5045 * LUCENE-6222: Removed TermFilter, use a QueryWrapperFilter(TermQuery)
 5046   instead. This will be as efficient now that queries can opt out from
 5047   scoring. (Adrien Grand)
 5048 
 5049 * LUCENE-6269: Removed BooleanFilter, use a QueryWrapperFilter(BooleanQuery)
 5050   instead. (Adrien Grand)
 5051 
 5052 * LUCENE-6270: Replaced TermsFilter with TermsQuery, use a
 5053   QueryWrapperFilter(TermsQuery) instead. (Adrien Grand)
 5054 
 5055 * LUCENE-6223: Move BooleanQuery.BooleanWeight to BooleanWeight.
 5056   (Robert Muir)
 5057 
 5058 * LUCENE-1518: Make Filter extend Query and return 0 as score.
 5059   (Uwe Schindler, Adrien Grand)
 5060 
 5061 * LUCENE-6245: Force Filter subclasses to implement toString API from Query.
 5062   (Ryan Ernst)
 5063 
 5064 * LUCENE-6268: Replace FieldValueFilter and DocValuesRangeFilter with equivalent
 5065   queries that support approximations. (Adrien Grand)
 5066 
 5067 * LUCENE-6289: Replace DocValuesRangeFilter with DocValuesRangeQuery which
 5068   supports approximations. (Adrien Grand)
 5069 
 5070 * LUCENE-6266: Remove unnecessary Directory params from SegmentInfo.toString, 
 5071   SegmentInfos.files/toString, and SegmentCommitInfo.toString. (Robert Muir)
 5072 
 5073 * LUCENE-6272: Scorer extends DocSetIdIterator rather than DocsEnum (Alan
 5074   Woodward)
 5075 
 5076 * LUCENE-6281: Removed support for slow collations from lucene/sandbox. Better
 5077   performance would be achieved through CollationKeyAnalyzer or
 5078   ICUCollationKeyAnalyzer. (Adrien Grand)
 5079 
 5080 * LUCENE-6286: Removed IndexSearcher methods that take a Filter object.
 5081   A BooleanQuery with a filter clause must be used instead. (Adrien Grand)
 5082 
 5083 * LUCENE-6300: PrefixFilter, TermRangeFilter and NumericRangeFilter have been
 5084   removed. Use PrefixQuery, TermRangeQuery and NumericRangeQuery instead.
 5085   (Adrien Grand)
 5086 
 5087 * LUCENE-6303: Replaced FilterCache with QueryCache and CachingWrapperFilter
 5088   with CachingWrapperQuery. (Adrien Grand)
 5089 
 5090 * LUCENE-6317: Deprecate DataOutput.writeStringSet and writeStringStringMap.
 5091   Use writeSetOfStrings/Maps instead. (Mike McCandless, Robert Muir)
 5092 
 5093 * LUCENE-6307: Rename SegmentInfo.getDocCount -> .maxDoc,
 5094   SegmentInfos.totalDocCount -> .totalMaxDoc, MergeInfo.totalDocCount
 5095   -> .totalMaxDoc and MergePolicy.OneMerge.totalDocCount ->
 5096   .totalMaxDoc (Adrien Grand, Robert Muir, Mike McCandless)
 5097 
 5098 * LUCENE-6367: PrefixQuery now subclasses AutomatonQuery, removing the
 5099   specialized PrefixTermsEnum.  (Robert Muir, Mike McCandless)
 5100 
 5101 Other
 5102 
 5103 * LUCENE-6248: Remove unused odd constants from StandardSyntaxParser.jj
 5104   (Dawid Weiss)
 5105 
 5106 * LUCENE-6193: Collapse identical catch branches in try-catch statements.
 5107   (shalin)
 5108 
 5109 * LUCENE-6239: Removed RAMUsageEstimator's sun.misc.Unsafe calls.
 5110   (Robert Muir, Dawid Weiss, Uwe Schindler)
 5111 
 5112 * LUCENE-6292: Seed StringHelper better. (Robert Muir)
 5113 
 5114 * LUCENE-6333: Refactored queries to delegate their equals and hashcode
 5115   impls to the super class. (Lee Hinman via Adrien Grand)
 5116 
 5117 * LUCENE-6343: DefaultSimilarity javadocs had the wrong float value to
 5118   demonstrate precision of encoded norms (András Péteri via Mike McCandless)
 5119 
 5120 Changes in Runtime Behavior
 5121 
 5122 * LUCENE-6255: PhraseQuery now ignores leading holes and requires that
 5123   positions are positive and added in order. (Adrien Grand)
 5124 
 5125 * LUCENE-6298: SimpleQueryParser returns an empty query rather than
 5126   null, if e.g. the terms were all stopwords. (Lee Hinman via Robert Muir)
 5127 
 5128 ======================= Lucene 5.0.0 =======================
 5129 
 5130 New Features
 5131 
 5132 * LUCENE-5945: All file handling converted to NIO.2 apis. (Robert Muir)
 5133 
 5134 * LUCENE-5946: SimpleFSDirectory now uses Files.newByteChannel, for 
 5135   portability with custom FileSystemProviders. If you want the old
 5136   non-interruptible behavior of RandomAccessFile, use RAFDirectory
 5137   in the misc/ module. (Uwe Schindler, Robert Muir)
 5138 
 5139 * SOLR-3359: Added analyzer attribute/property to SynonymFilterFactory.
 5140   (Ryo Onodera via Koji Sekiguchi)
 5141 
 5142 * LUCENE-5648: Index and search date ranges, particularly multi-valued ones. It's
 5143   implemented in the spatial module as DateRangePrefixTree used with
 5144   NumberRangePrefixTreeStrategy. (David Smiley)
 5145 
 5146 * LUCENE-5895: Lucene now stores a unique id per-segment and per-commit to aid
 5147   in accurate replication of index files (Robert Muir, Mike McCandless)
 5148 
 5149 * LUCENE-5889: Add commit method to AnalyzingInfixSuggester, and allow just using .add
 5150   to build up the suggester.  (Varun Thacker via Mike McCandless)
 5151 
 5152 * LUCENE-5123: Add a "pull" option to the postings writing API, so
 5153   that a PostingsFormat now receives a Fields instance and it is
 5154   responsible for iterating through all fields, terms, documents and
 5155   positions.  (Robert Muir, Mike McCandless)
 5156 
 5157 * LUCENE-5268: Full cutover of all postings formats to the "pull"
 5158   FieldsConsumer API, removing PushFieldsConsumer.  Added new
 5159   PushPostingsWriterBase for single-pass push of docs/positions to the
 5160   postings format.  (Mike McCandless)
 5161 
 5162 * LUCENE-5906: Use Files.delete everywhere instead of File.delete, so that
 5163   when things go wrong, you get a real exception message why.
 5164   (Uwe Schindler, Robert Muir)
 5165 
 5166 * LUCENE-5933: Added FilterSpans for easier wrapping of Spans instance. (Shai Erera)
 5167 
 5168 * LUCENE-5925: Remove fallback logic from opening commits, instead use
 5169   Directory.renameFile so that in-progress commits are never visible.
 5170   (Robert Muir)
 5171 
 5172 * LUCENE-5820: SuggestStopFilter should have a factory.
 5173   (Varun Thacker via Steve Rowe)
 5174 
 5175 * LUCENE-5949: Add Accountable.getChildResources(). (Robert Muir)
 5176 
 5177 * SOLR-5986: Added ExitableDirectoryReader that extends FilterDirectoryReader and enables
 5178   exiting requests that take too long to enumerate over terms. (Anshum Gupta, Steve Rowe,
 5179   Robert Muir)
 5180 
 5181 * LUCENE-5911: Add MemoryIndex.freeze() to allow thread-safe searching over a 
 5182   MemoryIndex. (Alan Woodward, David Smiley, Robert Muir)
 5183 
 5184 * LUCENE-5969: Lucene 5.0 has a new index format with mismatched file detection,
 5185   improved exception handling, and indirect norms encoding for sparse fields.
 5186   (Mike McCandless, Ryan Ernst, Robert Muir)
 5187 
 5188 * LUCENE-6053: Add Serbian analyzer.  (Nikola Smolenski via Robert Muir, Mike McCandless)
 5189 
 5190 * LUCENE-4400: Add support for new NYSIIS Apache commons phonetic
 5191   codec (Thomas Neidhart via Mike McCandless)
 5192 
 5193 * LUCENE-6059: Add Daitch-Mokotoff Soundex phonetic Apache commons
 5194   phonetic codec, and upgrade to Apache commons codec 1.10. (Thomas
 5195   Neidhart via Mike McCandless)
 5196 
 5197 * LUCENE-6058: With the upgrade to Apache commons codec 1.10, the
 5198   experimental BeiderMorseFilter has changed its behavior, so any
 5199   index using it will need to be rebuilt.  (Thomas
 5200   Neidhart via Mike McCandless)
 5201 
 5202 * LUCENE-6050: Accept MUST and MUST_NOT (in addition to SHOULD) for
 5203   each context passed to Analyzing/BlendedInfixSuggester (Arcadius
 5204   Ahouansou, jane chang via Mike McCandless)
 5205 
 5206 * LUCENE-5929: Also extract terms to highlight from block join
 5207   queries. (Julie Tibshirani via Mike McCandless)
 5208 
 5209 * LUCENE-6063: Allow overriding whether/how ConcurrentMergeScheduler
 5210   stalls incoming threads when merges are falling behind (Mike
 5211   McCandless)
 5212 
 5213 * LUCENE-5833: DocumentDictionary now enumerates each value separately
 5214   in a multi-valued field (not just the first value), so you can build
 5215   suggesters from multi-valued fields.  (Varun Thacker via Mike
 5216   McCandless)
 5217 
 5218 * LUCENE-6077: Added a filter cache. (Adrien Grand, Robert Muir)
 5219 
 5220 * LUCENE-6088: TermsFilter implements Accountable. (Adrien Grand)
 5221 
 5222 * LUCENE-6034: The default highlighter when used with QueryScorer will highlight payload-sensitive
 5223   queries provided that term vectors with positions, offsets, and payloads are present. This is the
 5224   only highlighter that can highlight such queries accurately. (David Smiley)
 5225 
 5226 * LUCENE-5914: Add an option to Lucene50Codec to support either BEST_SPEED
 5227   or BEST_COMPRESSION for stored fields. (Adrien Grand, Robert Muir)
 5228 
 5229 * LUCENE-6119: Add auto-IO-throttling to ConcurrentMergeScheduler, to
 5230   rate limit IO writes for each merge depending on incoming merge
 5231   rate.  (Mike McCandless)
 5232 
 5233 * LUCENE-6155: Add payload support to MemoryIndex. The default highlighter's
 5234   QueryScorer and WeighedSpanTermExtractor now have setUsePayloads(bool).
 5235   (David Smiley)
 5236 
 5237 * LUCENE-6166: Deletions (alone) can now trigger new merges.  (Mike McCandless)
 5238 
 5239 * LUCENE-6177: Add CustomAnalyzer that allows to configure analyzers
 5240   like you do in Solr's index schema. This class has a builder API to configure
 5241   Tokenizers, TokenFilters, and CharFilters based on their SPI names
 5242   and parameters as documented by the corresponding factories.
 5243   (Uwe Schindler)
 5244 
 5245 Optimizations
 5246 
 5247 * LUCENE-5960: Use a more efficient bitset, not a Set<Integer>, to
 5248   track visited states.  (Markus Heiden via Mike McCandless)
 5249 
 5250 * LUCENE-5959: Don't allocate excess memory when building automaton in
 5251   finish. (Markus Heiden via Mike McCandless)
 5252 
 5253 * LUCENE-5963: Reduce memory allocations in
 5254   AnalyzingSuggester. (Markus Heiden via Mike McCandless)
 5255 
 5256 * LUCENE-5938: MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE is now faster on
 5257   queries that match few documents by using a sparse bit set implementation.
 5258   (Adrien Grand)
 5259 
 5260 * LUCENE-5969: Refactor merging to be more efficient, checksum calculation is
 5261   per-segment/per-producer, and norms and doc values merging no longer cause 
 5262   RAM spikes for latent fields. (Mike McCandless, Robert Muir)
 5263 
 5264 * LUCENE-5983: CachingWrapperFilter now uses a new DocIdSet implementation
 5265   called RoaringDocIdSet instead of WAH8DocIdSet. (Adrien Grand)
 5266 
 5267 * LUCENE-6022: DocValuesDocIdSet checks live docs before doc values.
 5268   (Adrien Grand)
 5269 
 5270 * LUCENE-6030: Add norms patched compression for a small number of common values
 5271   (Ryan Ernst)
 5272 
 5273 * LUCENE-6040: Speed up EliasFanoDocIdSet through broadword bit selection.
 5274   (Paul Elschot)
 5275 
 5276 * LUCENE-6033: CachingTokenFilter now uses ArrayList not LinkedList, and has new
 5277   isCached() method. (David Smiley)
 5278 
 5279 * LUCENE-6031: TokenSources (in the default highlighter) converts term vectors into a
 5280   TokenStream much faster in linear time (not N*log(N) using less memory, and with reset()
 5281   implemented.  Only one of offsets or positions are required of the term vector.
 5282   (David Smiley)
 5283 
 5284 * LUCENE-6089, LUCENE-6090: Tune CompressionMode.HIGH_COMPRESSION for 
 5285   better compression and less cpu usage. (Adrien Grand, Robert Muir)
 5286 
 5287 * LUCENE-6034: QueryScorer, used by the default highlighter, needn't re-index the provided
 5288   TokenStream with MemoryIndex when it comes from TokenSources (term vectors) with offsets and
 5289   positions. (David Smiley)
 5290 
 5291 * LUCENE-5951: ConcurrentMergeScheduler detects whether the index is on SSD or not
 5292   and does a better job defaulting its settings.  This only works on Linux for now;
 5293   other OS's will continue to use the previous defaults (tuned for spinning disks).
 5294   (Robert Muir, Uwe Schindler, hossman, Mike McCandless)
 5295 
 5296 * LUCENE-6131: Optimize SortingMergePolicy. (Robert Muir)
 5297 
 5298 * LUCENE-6133: Improve default StoredFieldsWriter.merge() to be more efficient. 
 5299   (Robert Muir)
 5300 
 5301 * LUCENE-6145: Make EarlyTerminatingSortingCollector able to early-terminate
 5302   when the sort order is a prefix of the index-time order. (Adrien Grand)
 5303 
 5304 * LUCENE-6178: Score boolean queries containing MUST_NOT clauses with BooleanScorer2,
 5305   to use skip list data and avoid unnecessary scoring. (Adrien Grand, Robert Muir)
 5306 
 5307 API Changes
 5308 
 5309 * LUCENE-5900: Deprecated more constructors taking Version in *InfixSuggester and
 5310   ICUCollationKeyAnalyzer, and removed TEST_VERSION_CURRENT from the test framework.
 5311   (Ryan Ernst)
 5312 
 5313 * LUCENE-4535: oal.util.FilterIterator is now an internal API.
 5314   (Adrien Grand)
 5315 
 5316 * LUCENE-4924: DocIdSetIterator.docID() must now return -1 when the iterator is
 5317   not positioned. This change affects all classes that inherit from
 5318   DocIdSetIterator, including DocsEnum and DocsAndPositionsEnum. (Adrien Grand)
 5319 
 5320 * LUCENE-5127: Reduce RAM usage of FixedGapTermsIndex. Remove 
 5321   IndexWriterConfig.setTermIndexInterval, IndexWriterConfig.setReaderTermsIndexDivisor,
 5322   and termsIndexDivisor from StandardDirectoryReader. These options have been no-ops
 5323   with the default codec since Lucene 4.0. If you want to configure the interval for
 5324   this term index, pass it directly in your codec, where it can also be configured
 5325   per-field. (Robert Muir)
 5326 
 5327 * LUCENE-5388: Remove Reader from Tokenizer's constructor and from
 5328   Analyzer's createComponents. TokenStreams now always get their input
 5329   via setReader.  
 5330   (Benson Margulies via Robert Muir - pull request #16)
 5331 
 5332 * LUCENE-5527: The Collector API has been refactored to use a dedicated Collector
 5333   per leaf. (Shikhar Bhushan, Adrien Grand)
 5334 
 5335 * LUCENE-5702: The FieldComparator API has been refactor to a per-leaf API, just
 5336   like Collectors. (Adrien Grand)
 5337 
 5338 * LUCENE-4246: IndexWriter.close now always closes, even if it throws
 5339   an exception.  The new IndexWriterConfig.setCommitOnClose (default
 5340   true) determines whether close() should commit before closing.
 5341 
 5342 * LUCENE-5608, LUCENE-5565: Refactor SpatialPrefixTree/Cell API. Doesn't use Strings
 5343   as tokens anymore, and now iterates cells on-demand during indexing instead of
 5344   building a collection.  RPT now has more setters. (David Smiley)
 5345 
 5346 * LUCENE-5666: Change uninverted access (sorting, faceting, grouping, etc)
 5347   to use the DocValues API instead of FieldCache. For FieldCache functionality,
 5348   use UninvertingReader in lucene/misc (or implement your own FilterReader).
 5349   UninvertingReader is more efficient: supports multi-valued numeric fields,
 5350   detects when a multi-valued field is single-valued, reuses caches
 5351   of compatible types (e.g. SORTED also supports BINARY and SORTED_SET access
 5352   without insanity).  "Insanity" is no longer possible unless you explicitly want it. 
 5353   Rename FieldCache* and DocTermOrds* classes in the search package to DocValues*. 
 5354   Move SortedSetSortField to core and add SortedSetFieldSource to queries/, which
 5355   takes the same selectors. Add helper methods to DocValues.java that are better 
 5356   suited for search code (never return null, etc).  (Mike McCandless, Robert Muir)
 5357 
 5358 * LUCENE-5871: Remove Version from IndexWriterConfig. Use
 5359   IndexWriterConfig.setCommitOnClose to change the behavior of IndexWriter.close().
 5360   The default has been changed to match that of 4.x.
 5361   (Ryan Ernst, Mike McCandless)
 5362 
 5363 * LUCENE-5965: CorruptIndexException requires a String or DataInput resource.
 5364   (Robert Muir)
 5365 
 5366 * LUCENE-5972: IndexFormatTooOldException and IndexFormatTooNewException now
 5367                extend from IOException.
 5368   (Ryan Ernst, Robert Muir)
 5369 
 5370 * LUCENE-5569: *AtomicReader/AtomicReaderContext have been renamed to *LeafReader/LeafReaderContext.
 5371   (Ryan Ernst)
 5372 
 5373 * LUCENE-5938: Removed MultiTermQuery.ConstantScoreAutoRewrite as
 5374   MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE is usually better. (Adrien Grand)
 5375 
 5376 * LUCENE-5924: Rename CheckIndex -fix option to -exorcise. This option does not
 5377   actually fix the index, it just drops data.  (Robert Muir)
 5378 
 5379 * LUCENE-5969: Add Codec.compoundFormat, which handles the encoding of compound 
 5380   files. Add getMergeInstance() to codec producer APIs, which can be overridden
 5381   to return an instance optimized for merging instead of searching. Add 
 5382   Terms.getStats() which can return additional codec-specific statistics about a field.
 5383   Change instance method SegmentInfos.read() to two static methods: SegmentInfos.readCommit()
 5384   and SegmentInfos.readLatestCommit().
 5385   (Mike McCandless, Robert Muir)
 5386 
 5387 * LUCENE-5992: Remove FieldInfos from SegmentInfosWriter.write API. (Robert Muir, Mike McCandless)
 5388 
 5389 * LUCENE-5998: Simplify Field/SegmentInfoFormat to read+write methods.
 5390   (Robert Muir)
 5391 
 5392 * LUCENE-6000: Removed StandardTokenizerInterface.  Tokenizers now use
 5393   their jflex impl directly.
 5394   (Ryan Ernst)
 5395 
 5396 * LUCENE-6006: Removed FieldInfo.normType since it's redundant: it
 5397   will be DocValuesType.NUMERIC if the field indexed and does not omit
 5398   norms, else null.  (Robert Muir, Mike McCandless)
 5399 
 5400 * LUCENE-6013: Removed indexed boolean from IndexableFieldType and
 5401   FieldInfo, since it's redundant with IndexOptions != null. (Robert
 5402   Muir, Mike McCandless)
 5403 
 5404 * LUCENE-6021: FixedBitSet.nextSetBit now returns DocIdSetIterator.NO_MORE_DOCS
 5405   instead of -1 when there are no more bits which are set. (Adrien Grand)
 5406 
 5407 * LUCENE-5953: Directory and LockFactory APIs were restructured: Locking is
 5408   now under the responsibility of the Directory implementation. LockFactory is
 5409   only used by subclasses of BaseDirectory to delegate locking to an impl
 5410   class. LockFactories are now singletons and are responsible to create a Lock
 5411   instance based on a Directory implementation passed to the factory method.
 5412   See MIGRATE.txt for more details.  (Uwe Schindler, Robert Muir)
 5413 
 5414 * LUCENE-6062: Throw exception instead of silently doing nothing if you try to 
 5415   sort/group/etc on a misconfigured field (e.g. no docvalues, no UninvertingReader, etc).  
 5416   (Robert Muir)
 5417 
 5418 * LUCENE-6068: LeafReader.fields() never returns null. (Robert Muir)
 5419 
 5420 * LUCENE-6082: Remove abort() from codec apis. (Robert Muir)
 5421 
 5422 * LUCENE-6084: IndexOutput's constructor now requires a String
 5423   resourceDescription so its toString is sane (Robert Muir, Mike
 5424   McCandless)
 5425 
 5426 * LUCENE-6087: Allow passing custom DirectoryReader to SearcherManager
 5427   (Mike McCandless)
 5428 
 5429 * LUCENE-6085: Undeprecate SegmentInfo attributes, but add safety so they
 5430   won't be trappy if codec tries to use them during docvalues updates.
 5431   (Robert Muir)
 5432 
 5433 * LUCENE-6097: Remove dangerous / overly expert
 5434   IndexWriter.abortMerges and waitForMerges methods.  (Robert Muir,
 5435   Mike McCandless)
 5436 
 5437 * LUCENE-6099: Add FilterDirectory.unwrap and
 5438   FilterDirectoryReader.unwrap (Simon Willnauer, Mike McCandless)
 5439 
 5440 * LUCENE-6121: CachingTokenFilter.reset() now propagates to its input if called before
 5441   incrementToken().  You must call reset() now on this filter instead of doing it a-priori on the
 5442   input(), which previously didn't work.  (David Smiley, Robert Muir)
 5443 
 5444 * LUCENE-6147: Make the core Accountables.namedAccountable function public
 5445   (Ryan Ernst)
 5446 
 5447 * LUCENE-6150: Remove staleFiles set and onIndexOutputClosed() from FSDirectory.
 5448   (Uwe Schindler, Robert Muir, Mike McCandless)
 5449 
 5450 * LUCENE-6146: Replaced Directory.copy() with Directory.copyFrom().
 5451   (Robert Muir)
 5452 
 5453 * LUCENE-6149: Infix suggesters' highlighting and allTermsRequired can
 5454   be set at the constructor for non-contextual lookup.
 5455   (Boon Low, Tomás Fernández Löbbe)
 5456 
 5457 * LUCENE-6158, LUCENE-6165: IndexWriter.addIndexes(IndexReader...) changed to
 5458   addIndexes(CodecReader...) (Robert Muir)
 5459 
 5460 * LUCENE-6179: Out-of-order scoring is not allowed anymore, so
 5461   Weight.scoresDocsOutOfOrder and LeafCollector.acceptsDocsOutOfOrder have been
 5462   removed and boolean queries now always score in order.
 5463 
 5464 * LUCENE-6212: IndexWriter no longer accepts per-document Analyzer to
 5465   add/updateDocument.  These methods were trappy as they made it
 5466   easy to accidentally index tokens that were not easily
 5467   searchable. (Mike McCandless)
 5468 
 5469 Bug Fixes
 5470 
 5471 * LUCENE-5650: Enforce read-only access to any path outside the temporary
 5472   folder via security manager, and make test temp dirs absolute.
 5473   (Ryan Ernst, Dawid Weiss)
 5474 
 5475 * LUCENE-5948: RateLimiter now fully inits itself on init.  (Varun
 5476   Thacker via Mike McCandless)
 5477 
 5478 * LUCENE-5981: CheckIndex obtains write.lock, since with some parameters it
 5479   may modify the index, and to prevent false corruption reports, as it does
 5480   not have the regular "spinlock" of DirectoryReader.open. It now implements
 5481   Closeable and you must close it to release the lock.  (Mike McCandless, Robert Muir)
 5482 
 5483 * LUCENE-6004: Don't highlight the LookupResult.key returned from
 5484   AnalyzingInfixSuggester (Christian Reuschling, jane chang via Mike McCandless)
 5485 
 5486 * LUCENE-5980: Don't let document length overflow. (Robert Muir)
 5487 
 5488 * LUCENE-5961: Fix the exists() method for FunctionValues returned by many ValueSources to
 5489   behave properly when wrapping other ValueSources which do not exist for the specified document
 5490   (hossman)
 5491 
 5492 * LUCENE-6039: Add IndexOptions.NONE and DocValuesType.NONE instead of
 5493   using null to mean not index and no doc values, renamed
 5494   IndexOptions.DOCS_ONLY to DOCS, and pulled IndexOptions and
 5495   DocValues out of FieldInfo into their own classes in
 5496   org.apache.lucene.index (Simon Willnauer, Robert Muir, Mike
 5497   McCandless)
 5498 
 5499 * LUCENE-6041: Remove sugar methods FieldInfo.isIndexed and
 5500   FieldInfo.hasDocValues.  (Robert Muir, Mike McCandless)
 5501 
 5502 * LUCENE-6044: Fix backcompat support for token filters with enablePositionIncrements=false.
 5503   Also fixed backcompat for TrimFilter with updateOffsets=true.  These options
 5504   are supported with a match version before 4.4, and no longer valid at all with 5.0.
 5505   (Ryan Ernst) 
 5506 
 5507 * LUCENE-6042: CustomScoreQuery explain was incorrect in some cases,
 5508   such as when nested inside a boolean query. (Denis Lantsman via Robert Muir)
 5509 
 5510 * LUCENE-6046: Add maxDeterminizedStates safety to determinize (which has
 5511   an exponential worst case) so that if it would create too many states, it
 5512   now throws an exception instead of exhausting CPU/RAM.  (Nik
 5513   Everett via Mike McCandless)
 5514 
 5515 * LUCENE-6054: Allow repeating the empty automaton (Nik Everett via
 5516   Mike McCandless)
 5517 
 5518 * LUCENE-6049: Don't throw cryptic exception writing a segment when
 5519   the only docs in it had fields that hit non-aborting exceptions
 5520   during indexing but also had doc values.  (Mike McCandless)
 5521 
 5522 * LUCENE-6055: PayloadAttribute.clone() now does a deep clone of the underlying
 5523   bytes. (Shai Erera)
 5524 
 5525 * LUCENE-6060: Remove dangerous IndexWriter.unlock method (Simon
 5526   Willnauer, Mike McCandless)
 5527 
 5528 * LUCENE-6062: Pass correct fieldinfos to docvalues producer when the
 5529   segment has updates. (Mike McCandless, Shai Erera, Robert Muir)
 5530 
 5531 * LUCENE-6075: Don't overflow int in SimpleRateLimiter (Boaz Leskes
 5532   via Mike McCandless)
 5533 
 5534 * LUCENE-5987: IndexWriter will now forcefully close itself on
 5535   aborting exception (an exception that would otherwise cause silent
 5536   data loss).  (Robert Muir, Mike McCandless)
 5537 
 5538 * LUCENE-6094: Allow IW.rollback to stop ConcurrentMergeScheduler even
 5539   when it's stalling because there are too many merges. (Mike McCandless)
 5540 
 5541 * LUCENE-6105: Don't cache FST root arcs if the number of root arcs is
 5542   small, or if the cache would be > 20% of the size of the FST.
 5543   (Robert Muir, Mike McCandless)
 5544 
 5545 * LUCENE-6124: Fix double-close() problems in codec and store APIs.
 5546   (Robert Muir)
 5547   
 5548 * LUCENE-6152: Fix double close problems in OutputStreamIndexOutput.
 5549   (Uwe Schindler)
 5550 
 5551 * LUCENE-6139: Highlighter: TokenGroup start & end offset getters should have
 5552   been returning the offsets of just the matching tokens in the group when
 5553   there's a distinction. (David Smiley)
 5554   
 5555 * LUCENE-6173: NumericTermAttribute and spatial/CellTokenStream do not clone
 5556   their BytesRef(Builder)s. Also equals/hashCode was missing.  (Uwe Schindler)
 5557 
 5558 * LUCENE-6205: Fixed intermittent concurrency issue that could cause
 5559   FileNotFoundException when writing doc values updates at the same
 5560   time that a merge kicks off.  (Mike McCandless)
 5561   
 5562 * LUCENE-6192: Fix int overflow corruption case in skip data for
 5563   high frequency terms in extremely large indices (Robert Muir, Mike
 5564   McCandless)
 5565 
 5566 * LUCENE-6093: Don't throw NullPointerException from
 5567   BlendedInfixSuggester for lookups that do not end in a prefix
 5568   token.  (jane chang via Mike McCandless)
 5569 
 5570 * LUCENE-6214: Fixed IndexWriter deadlock when one thread is
 5571   committing while another opens a near-real-time reader and an
 5572   unrecoverable (tragic) exception is hit.  (Simon Willnauer, Mike
 5573   McCandless)
 5574 
 5575 Documentation
 5576 
 5577 * LUCENE-5392: Add/improve analysis package documentation to reflect
 5578   analysis API changes.  (Benson Margulies via Robert Muir - pull request #17)
 5579 
 5580 * LUCENE-6057: Improve Sort(SortField) docs (Martin Braun via Mike McCandless)
 5581 
 5582 * LUCENE-6112: Fix compile error in FST package example code
 5583   (Tomoko Uchida via Koji Sekiguchi)
 5584 
 5585 Tests
 5586 
 5587 * LUCENE-5957: Add option for tests to not randomize codec
 5588   (Ryan Ernst)
 5589 
 5590 * LUCENE-5974: Add check that backcompat indexes use default codecs
 5591   (Ryan Ernst)
 5592 
 5593 * LUCENE-5971: Create addBackcompatIndexes.py script to build and add 
 5594   backcompat test indexes for a given lucene version. Also renamed backcompat
 5595   index files to use Version.toString() in filename.
 5596   (Ryan Ernst)
 5597 
 5598 * LUCENE-6002: Monster tests no longer fail.  Most of them now have an 80 hour
 5599   timeout, effectively removing the timeout.  The tests that operate near the 2
 5600   billion limit now use IndexWriter.MAX_DOCS instead of Integer.MAX_VALUE.
 5601   Some of the slow Monster tests now explicitly choose the default codec.
 5602   (Mike McCandless, Shawn Heisey)
 5603 
 5604 * LUCENE-5968: Improve error message when 'ant beast' is run on top-level
 5605   modules.  (Ramkumar Aiyengar, Uwe Schindler)
 5606 
 5607 * LUCENE-6120: Fix MockDirectoryWrapper's close() handling.
 5608   (Mike McCandless, Robert Muir)
 5609 
 5610 Build
 5611 
 5612 * LUCENE-5909: Smoke tester now has better command line parsing and
 5613   optionally also runs on Java 8.  (Ryan Ernst, Uwe Schindler)
 5614 
 5615 * LUCENE-5902: Add bumpVersion.py script to manage version increase after release branch is cut.
 5616 
 5617 * LUCENE-5962: Rename diffSources.py to createPatch.py and make it work with all text file types.
 5618   (Ryan Ernst)
 5619 
 5620 * LUCENE-5995: Upgrade ICU to 54.1 (Robert Muir)
 5621 
 5622 * LUCENE-6070: Upgrade forbidden-apis to 1.7 (Uwe Schindler)
 5623 
 5624 Other
 5625 
 5626 * LUCENE-5563: Removed sep layout: which has fallen behind on features and doesn't
 5627   perform as well as other options.  (Robert Muir)
 5628 
 5629 * LUCENE-4086: Removed support for Lucene 3.x indexes. See migration guide for
 5630   more information.  (Robert Muir)
 5631 
 5632 * LUCENE-5858: Moved Lucene 4 compatibility codecs to 'lucene-backward-codecs.jar'. 
 5633   (Adrien Grand, Robert Muir)
 5634 
 5635 * LUCENE-5915: Remove Pulsing postings format. (Robert Muir)
 5636 
 5637 * LUCENE-6213: Add useful exception message when commit contains segments from legacy codecs.
 5638   (Ryan Ernst)
 5639 
 5640 ======================= Lucene 4.10.4 ======================
 5641 
 5642 Bug fixes
 5643 
 5644 * LUCENE-6019, LUCENE-6117: Remove -Dtests.assert to make IndexWriter
 5645   infoStream sane.  (Robert Muir, Mike McCandless)
 5646 
 5647 * LUCENE-6161: Resolving deletes was failing to reuse DocsEnum likely
 5648   causing substantial performance cost for use cases that frequently
 5649   delete old documents (Mike McCandless)
 5650 
 5651 * LUCENE-6192: Fix int overflow corruption case in skip data for
 5652   high frequency terms in extremely large indices (Robert Muir, Mike
 5653   McCandless)
 5654 
 5655 * LUCENE-6207: Fixed consumption of several terms enums on the same
 5656   sorted (set) doc values instance at the same time.
 5657   (Tom Shally, Robert Muir, Adrien Grand)
 5658 
 5659 * LUCENE-6093: Don't throw NullPointerException from
 5660   BlendedInfixSuggester for lookups that do not end in a prefix
 5661   token.  (jane chang via Mike McCandless)
 5662 
 5663 * LUCENE-6279: Don't let an abusive leftover _N_upgraded.si in the
 5664   index directory cause index corruption on upgrade (Robert Muir, Mike
 5665   McCandless)
 5666 
 5667 * LUCENE-6287: Fix concurrency bug in IndexWriter that could cause
 5668   index corruption (missing _N.si files) the first time 4.x kisses a
 5669   3.x index if merges are also running.  (Simon Willnauer, Mike
 5670   McCandless)
 5671 
 5672 * LUCENE-6205: Fixed intermittent concurrency issue that could cause
 5673   FileNotFoundException when writing doc values updates at the same
 5674   time that a merge kicks off.  (Mike McCandless)
 5675 
 5676 * LUCENE-6214: Fixed IndexWriter deadlock when one thread is
 5677   committing while another opens a near-real-time reader and an
 5678   unrecoverable (tragic) exception is hit.  (Simon Willnauer, Mike
 5679   McCandless)
 5680 
 5681 * LUCENE-6105: Don't cache FST root arcs if the number of root arcs is
 5682   small, or if the cache would be > 20% of the size of the FST.
 5683   (Robert Muir, Mike McCandless)
 5684 
 5685 * LUCENE-6001: DrillSideways hits NullPointerException for certain
 5686   BooleanQuery searches.  (Dragan Jotannovic, jane chang via Mike
 5687   McCandless)
 5688   
 5689 * LUCENE-6306: Merging of doc values and norms now checks whether the
 5690   merge was aborted so IndexWriter.rollback can more promptly abort a
 5691   running merge. (Robert Muir, Mike McCandless)
 5692 
 5693 API Changes
 5694 
 5695 * LUCENE-6212: Deprecate IndexWriter APIs that accept per-document Analyzer.
 5696   These methods were trappy as they made it easy to accidentally index
 5697   tokens that were not easily searchable and will be removed in 5.0.0.
 5698   (Mike McCandless)
 5699 
 5700 ======================= Lucene 4.10.3 ======================
 5701 
 5702 Bug fixes
 5703 
 5704 * LUCENE-6046: Add maxDeterminizedStates safety to determinize (which has
 5705   an exponential worst case) so that if it would create too many states, it
 5706   now throws an exception instead of exhausting CPU/RAM.  (Nik
 5707   Everett via Mike McCandless)
 5708 
 5709 * LUCENE-6054: Allow repeating the empty automaton (Nik Everett via
 5710   Mike McCandless)
 5711 
 5712 * LUCENE-6049: Don't throw cryptic exception writing a segment when
 5713   the only docs in it had fields that hit non-aborting exceptions
 5714   during indexing but also had doc values.  (Mike McCandless)
 5715 
 5716 * LUCENE-6060: Deprecate IndexWriter.unlock (Simon Willnauer, Mike
 5717   McCandless)
 5718 
 5719 * LUCENE-3229: Overlapping ordered SpanNearQuery spans should not match.
 5720   (Ludovic Boutros, Paul Elschot, Greg Dearing, ehatcher)
 5721 
 5722 * LUCENE-6004: Don't highlight the LookupResult.key returned from
 5723   AnalyzingInfixSuggester (Christian Reuschling, jane chang via Mike McCandless)
 5724 
 5725 * LUCENE-6075: Don't overflow int in SimpleRateLimiter (Boaz Leskes
 5726   via Mike McCandless)
 5727   
 5728 * LUCENE-5980: Don't let document length overflow. (Robert Muir)
 5729 
 5730 * LUCENE-6042: CustomScoreQuery explain was incorrect in some cases,
 5731   such as when nested inside a boolean query. (Denis Lantsman via Robert Muir)
 5732 
 5733 * LUCENE-5948: RateLimiter now fully inits itself on init.  (Varun
 5734   Thacker via Mike McCandless)
 5735 
 5736 * LUCENE-6055: PayloadAttribute.clone() now does a deep clone of the underlying
 5737   bytes. (Shai Erera)
 5738 
 5739 * LUCENE-6094: Allow IW.rollback to stop ConcurrentMergeScheduler even
 5740   when it's stalling because there are too many merges. (Mike McCandless)
 5741   
 5742 Documentation
 5743 
 5744 * LUCENE-6057: Improve Sort(SortField) docs (Martin Braun via Mike McCandless)
 5745 
 5746 ======================= Lucene 4.10.2 ======================
 5747 
 5748 Bug fixes
 5749 
 5750 * LUCENE-5977: Fix tokenstream safety checks in IndexWriter to properly
 5751   work across multi-valued fields. Previously some cases across multi-valued
 5752   fields would happily create a corrupt index. (Dawid Weiss, Robert Muir)
 5753 
 5754 * LUCENE-6019: Detect when DocValuesType illegally changes for the
 5755   same field name.  Also added -Dtests.asserts=true|false so we can
 5756   run tests with and without assertions. (Simon Willnauer, Robert
 5757   Muir, Mike McCandless).
 5758 
 5759 ======================= Lucene 4.10.1 ======================
 5760 
 5761 Bug fixes
 5762 
 5763 * LUCENE-5934: Fix backwards compatibility for 4.0 indexes.
 5764   (Ian Lea, Uwe Schindler, Robert Muir, Ryan Ernst)
 5765 
 5766 * LUCENE-5939: Regenerate old backcompat indexes to ensure they were built with
 5767   the exact release
 5768   (Ryan Ernst, Uwe Schindler)
 5769 
 5770 * LUCENE-5952: Improve error messages when version cannot be parsed;
 5771   don't check for too old or too new major version (it's too low level
 5772   to enforce here); use simple string tokenizer.  (Ryan Ernst, Uwe Schindler,
 5773   Robert Muir, Mike McCandless)
 5774 
 5775 * LUCENE-5958: Don't let exceptions during checkpoint corrupt the index. 
 5776   Refactor existing OOM handling too, so you don't need to handle OOM special
 5777   for every IndexWriter method: instead such disasters will cause IW to close itself
 5778   defensively. (Robert Muir, Mike McCandless)
 5779 
 5780 * LUCENE-5904: Fixed a corruption case that can happen when 1)
 5781   IndexWriter is uncleanly shut-down (OS crash, power loss, etc.), 2)
 5782   on startup, when a new IndexWriter is created, a virus checker is
 5783   holding some of the previously written but unused files open and
 5784   preventing deletion, 3) IndexWriter writes these files again during
 5785   the course of indexing, then the files can later be deleted, causing
 5786   corruption.  This case was detected by adding evilness to
 5787   MockDirectoryWrapper to have it simulate a virus checker holding a
 5788   file open and preventing deletion (Robert Muir, Mike McCandless)
 5789 
 5790 * LUCENE-5916: Static scope test components should be consistent between
 5791   tests (and test iterations). Fix for FaultyIndexInput in particular.
 5792   (Dawid Weiss)
 5793 
 5794 * LUCENE-5975: Fix reading of 3.0-3.3 indexes, where bugs in these old
 5795   index formats would result in CorruptIndexException "did not read all
 5796   bytes from file" when reading the deleted docs file. (Patrick Mi, Robert MUir)
 5797 
 5798 Tests
 5799 
 5800 * LUCENE-5936: Add backcompat checks to verify what is tested matches known versions
 5801   (Ryan Ernst)
 5802 
 5803 ======================= Lucene 4.10.0 ======================
 5804 
 5805 New Features
 5806 
 5807 * LUCENE-5778: Support hunspell morphological description fields/aliases.
 5808   (Robert Muir)
 5809 
 5810 * LUCENE-5801: Added (back) OrdinalMappingAtomicReader for merging search
 5811   indexes that contain category ordinals from separate taxonomy indexes.
 5812   (Nicola Buso via Shai Erera)
 5813 
 5814 * LUCENE-4175, LUCENE-5714, LUCENE-5779: Index and search rectangles with spatial
 5815   BBoxSpatialStrategy using most predicates.  Sort documents by relative overlap
 5816   of query areas or just by indexed shape area. (Ryan McKinley, David Smiley)
 5817 
 5818 * LUCENE-5806: Extend expressions grammar to support array access in variables.
 5819   Added helper class VariableContext to parse complex variable into pieces.
 5820   (Ryan Ernst)
 5821 
 5822 * LUCENE-5826: Support proper hunspell case handling, LANG, KEEPCASE, NEEDAFFIX,
 5823   and ONLYINCOMPOUND flags.  (Robert Muir)
 5824 
 5825 * LUCENE-5815: Add TermAutomatonQuery, a proximity query allowing you
 5826   to create an arbitrary automaton, using terms on the transitions,
 5827   expressing which sequence of sequential terms (including a special
 5828   "any" term) are allowed.  This is a generalization of
 5829   MultiPhraseQuery and span queries, and enables "correct" (including
 5830   position) length search-time graph synonyms.  (Mike McCandless)
 5831 
 5832 * LUCENE-5819: Add OrdsLucene41 block tree terms dict and postings
 5833   format, to include term ordinals in the index so the optional
 5834   TermsEnum.ord() and TermsEnum.seekExact(long ord) APIs work.  (Mike
 5835   McCandless)
 5836 
 5837 * LUCENE-5835: TermValComparator can sort missing values last. (Adrien Grand)
 5838 
 5839 * LUCENE-5825: Benchmark module can use custom postings format, e.g.:
 5840  codec.postingsFormat=Memory (Varun Shenoy, David Smiley)
 5841 
 5842 * LUCENE-5842: When opening large files (where it's too expensive to compare
 5843   checksum against all the bytes), retrieve checksum to validate structure
 5844   of footer, this can detect some forms of corruption such as truncation.
 5845   (Robert Muir)
 5846 
 5847 * LUCENE-5739: Added DataInput.readZ(Int|Long) and DataOutput.writeZ(Int|Long)
 5848   to read and write small signed integers. (Adrien Grand)
 5849 
 5850 API Changes
 5851 
 5852 * LUCENE-5752: Simplified Automaton API to be immutable. (Mike McCandless)
 5853 
 5854 * LUCENE-5793: Add equals/hashCode to FieldType. (Shay Banon, Robert Muir)
 5855 
 5856 * LUCENE-5692: DisjointSpatialFilter is deprecated (used by RecursivePrefixTreeStrategy)
 5857   (David Smiley)
 5858 
 5859 * LUCENE-5771: SpatialOperation's predicate names are now aliased to OGC standard names.
 5860   Thus you can use: Disjoint, Equals, Intersects, Overlaps, Within, Contains, Covers,
 5861   CoveredBy. The area requirement on the predicates was removed, and Overlaps' definition
 5862   was fixed. (David Smiley)
 5863 
 5864 * LUCENE-5850: Made Version handling more robust and extensible. Deprecated
 5865   Constants.LUCENE_MAIN_VERSION, Constants.LUCENE_VERSION and current Version
 5866   constants of the form LUCENE_X_Y. Added version constants that include bugfix
 5867   number of form LUCENE_X_Y_Z.  Changed Version.LUCENE_CURRENT to Version.LATEST.
 5868   CheckIndex now prints the Lucene version used to write each segment.
 5869   (Ryan Ernst, Uwe Schindler, Robert Muir, Mike McCandless)
 5870 
 5871 * LUCENE-5836: BytesRef has been splitted into BytesRef, whose intended usage is
 5872   to be just a reference to a section of a larger byte[] and BytesRefBuilder
 5873   which is a StringBuilder-like class for BytesRef instances. (Adrien Grand)
 5874 
 5875 * LUCENE-5883: You can now change the MergePolicy instance on a live IndexWriter,
 5876   without first closing and reopening the writer. This allows to e.g. run a special
 5877   merge with UpgradeIndexMergePolicy without reopening the writer. Also, MergePolicy
 5878   no longer implements Closeable; if you need to release your custom MergePolicy's
 5879   resources, you need to implement close() and call it explicitly. (Shai Erera)
 5880 
 5881 * LUCENE-5859: Deprecate Analyzer constructors taking Version.  Use Analyzer.setVersion()
 5882   to set the version an analyzer to replicate behavior from a specific release.
 5883   (Ryan Ernst, Robert Muir)
 5884 
 5885   
 5886 Optimizations
 5887 
 5888 * LUCENE-5780: Make OrdinalMap more memory-efficient, especially in case the
 5889   first segment has all values. (Adrien Grand, Robert Muir)
 5890 
 5891 * LUCENE-5782: OrdinalMap now sorts enums before being built in order to
 5892   improve compression. (Adrien Grand)
 5893 
 5894 * LUCENE-5798: Optimize MultiDocsEnum reuse. (Robert Muir)
 5895 
 5896 * LUCENE-5799: Optimize numeric docvalues merging. (Robert Muir)
 5897 
 5898 * LUCENE-5797: Optimize norms merging (Adrien Grand, Robert Muir)
 5899 
 5900 * LUCENE-5803: Add DelegatingAnalyzerWrapper, an optimized variant
 5901   of AnalyzerWrapper that doesn't allow to wrap components or readers.
 5902   This wrapper class is the base class of all analyzers that just delegate
 5903   to another analyzer, e.g. per field name: PerFieldAnalyzerWrapper and
 5904   Solr's schema support.  (Shay Banon, Uwe Schindler, Robert Muir)
 5905 
 5906 * LUCENE-5795: MoreLikeThisQuery now only collects the top N terms instead
 5907   of collecting all terms from the like text when building the query. 
 5908   (Alex Ksikes, Simon Willnauer)
 5909 
 5910 * LUCENE-5681: Fix RAMDirectory's IndexInput to not do double buffering
 5911   on slices (causes useless data copying, especially on random access slices).
 5912   This also improves slices of NRTCachingDirectory, because the cache
 5913   is based on RAMDirectory. BufferedIndexInput.wrap() was marked with a
 5914   warning in javadocs. It is almost always a better idea to implement
 5915   slicing on your own!  (Uwe Schindler, Robert Muir)
 5916 
 5917 * LUCENE-5834: Empty sorted set and numeric doc values are now singletons.
 5918   (Adrien Grand)
 5919 
 5920 * LUCENE-5841: Improve performance of block tree terms dictionary when
 5921   assigning terms to blocks.  (Mike McCandless)
 5922 
 5923 * LUCENE-5856: Optimize Fixed/Open/LongBitSet to remove unnecessary AND. 
 5924   (Robert Muir)
 5925 
 5926 * LUCENE-5884: Optimize FST.ramBytesUsed.  (Adrien Grand, Robert Muir,
 5927   Mike McCandless)
 5928 
 5929 * LUCENE-5882: Add Lucene410DocValuesFormat, with faster term lookups
 5930   for SORTED/SORTED_SET fields.  (Robert Muir)
 5931 
 5932 * LUCENE-5887: Remove WeakIdentityMap caching in AttributeFactory,
 5933   AttributeSource, and VirtualMethod in favour of Java 7's ClassValue.
 5934   Always use MethodHandles to create AttributeImpl classes.
 5935   (Uwe Schindler)
 5936 
 5937 Bug Fixes
 5938 
 5939 * LUCENE-5796: Fixes the Scorer.getChildren() method for two combinations 
 5940   of BooleanQuery. (Terry Smith via Robert Muir)
 5941 
 5942 * LUCENE-5790: Fix compareTo in MutableValueDouble and MutableValueBool, this caused 
 5943   incorrect results when grouping on fields with missing values. 
 5944   (海老澤 志信, hossman)
 5945 
 5946 * LUCENE-5817: Fix hunspell zero-affix handling: previously only zero-strips worked
 5947   correctly.  (Robert Muir)
 5948 
 5949 * LUCENE-5818, LUCENE-5823: Fix hunspell overgeneration for short strings that also 
 5950   match affixes, words are only stripped to a zero-length string if FULLSTRIP option
 5951   is specified in the dictionary.  (Robert Muir)
 5952 
 5953 * LUCENE-5824: Fix hunspell 'long' flag handling. (Robert Muir)
 5954 
 5955 * LUCENE-5838: Fix hunspell when the .aff file has over 64k affixes. (Robert Muir)
 5956 
 5957 * LUCENE-5869: Added restriction to positive values for maxExpansions in
 5958   FuzzyQuery.  (Ryan Ernst)
 5959 
 5960 * LUCENE-5672: IndexWriter.addIndexes() calls maybeMerge(), to ensure the index stays
 5961   healthy. If you don't want merging use NoMergePolicy instead. (Robert Muir)
 5962   
 5963 * LUCENE-5908: Fix Lucene43NGramTokenizer to be final
 5964   
 5965 Test Framework
 5966 
 5967 * LUCENE-5786: Unflushed/ truncated events file (hung testing subprocess).
 5968   (Dawid Weiss)
 5969 
 5970 * LUCENE-5881: Add "beasting" of tests: repeats the whole "test" Ant target
 5971   N times with "ant beast -Dbeast.iters=N".  (Uwe Schindler, Robert Muir,
 5972   Ryan Ernst, Dawid Weiss)
 5973 
 5974 Build
 5975  
 5976 * LUCENE-5770: Upgrade to JFlex 1.6, which has direct support for
 5977   supplementary code points - as a result, ICU4J is no longer used
 5978   to generate surrogate pairs to augment JFlex scanner specifications.
 5979   (Steve Rowe)
 5980 
 5981 * SOLR-6358: Remove VcsDirectoryMappings from idea configuration 
 5982   vcs.xml (Ramkumar Aiyengar via Steve Rowe)
 5983 
 5984 ======================= Lucene 4.9.1 ======================
 5985 
 5986 Bug fixes
 5987 
 5988 * LUCENE-5907: Fix corruption case when opening a pre-4.x index with
 5989   IndexWriter, then opening an NRT reader from that writer, then
 5990   calling commit from the writer, then closing the NRT reader.  This
 5991   case would remove the wrong files from the index leading to a
 5992   corrupt index.  (Mike McCandless)
 5993 
 5994 * LUCENE-5919: Fix exception handling inside IndexWriter when
 5995   deleteFile throws an exception, to not over-decRef index files,
 5996   possibly deleting a file that's still in use in the index, leading
 5997   to corruption.  (Mike McCandless)
 5998 
 5999 * LUCENE-5922: DocValuesDocIdSet on 5.x and FieldCacheDocIdSet on 4.x
 6000   are not cacheable. (Adrien Grand)
 6001 
 6002 * LUCENE-5843: Added IndexWriter.MAX_DOCS which is the maximum number
 6003   of documents allowed in a single index, and any operations that add
 6004   documents will now throw IllegalStateException if the max count
 6005   would be exceeded, instead of silently creating an unusable
 6006   index.  (Mike McCandless)
 6007 
 6008 * LUCENE-5844: ArrayUtil.grow/oversize now returns a maximum of
 6009   Integer.MAX_VALUE - 8 for the maximum array size.  (Robert Muir,
 6010   Mike McCandless)
 6011 
 6012 * LUCENE-5827: Make all Directory implementations correctly fail with
 6013   IllegalArgumentException if slices are out of bounds.  (Uwe Schindler)
 6014 
 6015 * LUCENE-5897, LUCENE-5400: JFlex-based tokenizers StandardTokenizer and
 6016   UAX29URLEmailTokenizer tokenize extremely slowly over long sequences of
 6017   text partially matching certain grammar rules.  The scanner default
 6018   buffer size was reduced, and scanner buffer growth was disabled, resulting
 6019   in much, much faster tokenization for these text sequences.  
 6020   (Chris Geeringh, Robert Muir, Steve Rowe)
 6021 
 6022 ======================= Lucene 4.9.0 =======================
 6023 
 6024 Changes in Runtime Behavior
 6025 
 6026 * LUCENE-5611: Changing the term vector options for multiple field
 6027   instances by the same name in one document is not longer accepted;
 6028   IndexWriter will now throw IllegalArgumentException.  (Robert Muir,
 6029   Mike McCandless)
 6030 
 6031 * LUCENE-5646: Remove rare/undertested bulk merge algorithm in 
 6032   CompressingStoredFieldsWriter. (Robert Muir, Adrien Grand)
 6033 
 6034 New Features
 6035 
 6036 * LUCENE-5610: Add Terms.getMin and Terms.getMax to get the lowest and
 6037   highest terms, and NumericUtils.get{Min/Max}{Int/Long} to get the
 6038   minimum numeric values from the provided Terms.  (Robert Muir, Mike
 6039   McCandless)
 6040 
 6041 * LUCENE-5675: Add IDVersionPostingsFormat, a postings format
 6042   optimized for primary-key (ID) fields that also record a version
 6043   (long) for each ID.  (Robert Muir, Mike McCandless)
 6044   
 6045 * LUCENE-5680: Add ability to atomically update a set of DocValues
 6046   fields. (Shai Erera)
 6047 
 6048 * LUCENE-5717: Add support for multiterm queries nested inside
 6049   filtered and constant-score queries to postings highlighter.
 6050   (Luca Cavanna via Robert Muir)
 6051 
 6052 * LUCENE-5731, LUCENE-5760: Add RandomAccessInput, a random access API for directory.
 6053   Add DirectReader/Writer, optimized for reading packed integers directly 
 6054   from Directory. Add Lucene49Codec and Lucene49DocValuesFormat that make
 6055   use of these.  (Robert Muir)
 6056 
 6057 * LUCENE-5743: Add Lucene49NormsFormat, which can compress in some cases
 6058   such as very short fields.  (Ryan Ernst, Adrien Grand, Robert Muir)
 6059 
 6060 * LUCENE-5748: Add SORTED_NUMERIC docvalues type, which is efficient
 6061   for processing numeric fields with multiple values.  (Robert Muir)
 6062 
 6063 * LUCENE-5754: Allow "$" as part of variable and function names in
 6064   expressions module.  (Uwe Schindler)
 6065 
 6066 Changes in Backwards Compatibility Policy
 6067 
 6068 * LUCENE-5634: Add reuse argument to IndexableField.tokenStream. This
 6069   can be used by custom fieldtypes, which don't use the Analyzer, but
 6070   implement their own TokenStream.  (Uwe Schindler, Robert Muir)
 6071 
 6072 * LUCENE-5640: AttributeSource.AttributeFactory was moved to a
 6073   top-level class: org.apache.lucene.util.AttributeFactory
 6074   (Uwe Schindler, Robert Muir)
 6075 
 6076 * LUCENE-4371: Removed IndexInputSlicer and Directory.createSlicer() and replaced
 6077   with IndexInput.slice(). (Robert Muir)
 6078 
 6079 * LUCENE-5727, LUCENE-5678: Remove IndexOutput.seek, IndexOutput.setLength().
 6080   (Robert Muir, Uwe Schindler)
 6081 
 6082 API Changes
 6083 
 6084 * LUCENE-5756: IndexWriter now implements Accountable and IW#ramSizeInBytes()
 6085   has been deprecated in favor of IW#ramBytesUsed() (Simon Willnauer)
 6086 
 6087 * LUCENE-5725: MoreLikeThis#like now accepts multiple values per field.
 6088   The pre-existing method has been deprecated in favor of a variable arguments
 6089   for the like text. (Alex Ksikes via Simon Willnauer)
 6090 
 6091 * LUCENE-5711: MergePolicy accepts an IndexWriter instance
 6092   on each method rather than holding state against a single
 6093   IndexWriter instance. (Simon Willnauer)
 6094 
 6095 * LUCENE-5582: Deprecate IndexOutput.length (just use
 6096   IndexOutput.getFilePointer instead) and IndexOutput.setLength.
 6097   (Mike McCandless)
 6098 
 6099 * LUCENE-5621: Deprecate IndexOutput.flush: this is not used by Lucene.
 6100   (Robert Muir)
 6101 
 6102 * LUCENE-5611: Simplified Lucene's default indexing chain / APIs.
 6103   AttributeSource/TokenStream.getAttribute now returns null if the
 6104   attribute is not present (previously it threw
 6105   IllegalArgumentException).  StoredFieldsWriter.startDocument no
 6106   longer receives the number of fields that will be added (Robert
 6107   Muir, Mike McCandless)
 6108 
 6109 * LUCENE-5632: In preparation for coming Lucene versions, the Version
 6110   enum constants were renamed to make them better readable. The constant
 6111   for Lucene 4.9 is now "LUCENE_4_9". Version.parseLeniently() is still
 6112   able to parse the old strings ("LUCENE_49"). The old identifiers got
 6113   deprecated and will be removed in Lucene 5.0.  (Uwe Schindler,
 6114   Robert Muir)
 6115 
 6116 * LUCENE-5633: Change NoMergePolicy to a singleton with no distinction between
 6117   compound and non-compound types. (Shai Erera)
 6118 
 6119 * LUCENE-5640: The Token class was deprecated. Since Lucene 2.9, TokenStreams
 6120   are using Attributes, Token is no longer used.  (Uwe Schindler, Robert Muir)
 6121 
 6122 * LUCENE-5679: Consolidated IndexWriter.deleteDocuments(Term) and 
 6123   IndexWriter.deleteDocuments(Query) with their varargs counterparts.
 6124   (Shai Erera)
 6125 
 6126 * LUCENE-5701: Core closed listeners are now available in the AtomicReader API,
 6127   they used to sit only in SegmentReader. (Adrien Grand, Robert Muir)
 6128 
 6129 * LUCENE-5706: Removed the option to unset a DocValues field through DocValues
 6130   updates. (Shai Erera)
 6131 
 6132 * LUCENE-5700: Added oal.util.Accountable that is now implemented by all
 6133   classes whose memory usage can be estimated. (Robert Muir, Adrien Grand)
 6134 
 6135 * LUCENE-5708: Remove IndexWriterConfig.clone, so now IndexWriter
 6136   simply uses the IndexWriterConfig you pass it, and you must create a
 6137   new IndexWriterConfig for each IndexWriter.  (Mike McCandless)
 6138   
 6139 * LUCENE-5678: IndexOutput no longer allows seeking, so it is no longer required
 6140   to use RandomAccessFile to write Indexes. Lucene now uses standard FileOutputStream
 6141   wrapped with OutputStreamIndexOutput to write index data. BufferedIndexOutput was
 6142   removed, because buffering and checksumming is provided by FilterOutputStreams,
 6143   provided by the JDK.  (Uwe Schindler, Mike McCandless)
 6144 
 6145 * LUCENE-5703: BinaryDocValues API changed to work like TermsEnum and not allocate/
 6146   copy bytes on each access, you are responsible for cloning if you want to keep
 6147   data around. (Adrien Grand)
 6148 
 6149 * LUCENE-5695: DocIdSet implements Accountable. (Adrien Grand)
 6150 
 6151 * LUCENE-5757: Moved RamUsageEstimator's reflection-based processing to RamUsageTester
 6152   in the test-framework module. (Robert Muir)
 6153 
 6154 * LUCENE-5761: Removed DiskDocValuesFormat, it was very inefficient and saved very little
 6155   RAM over the default codec. (Robert Muir)
 6156 
 6157 * LUCENE-5775: Deprecate JaspellLookup. (Mike McCandless)
 6158 
 6159 Optimizations
 6160 
 6161 * LUCENE-5603: hunspell stemmer more efficiently strips prefixes
 6162   and suffixes.  (Robert Muir)
 6163   
 6164 * LUCENE-5599: HttpReplicator did not properly delegate bulk read() to wrapped
 6165   InputStream. (Christoph Kaser via Shai Erera)
 6166   
 6167 * LUCENE-5591: pass an IOContext with estimated flush size when applying DV
 6168   updates. (Shai Erera)
 6169 
 6170 * LUCENE-5634: IndexWriter reuses TokenStream instances for String and Numeric
 6171   fields by default. (Uwe Schindler, Shay Banon, Mike McCandless, Robert Muir)
 6172 
 6173 * LUCENE-5638, LUCENE-5640: TokenStream uses a more performant AttributeFactory
 6174   by default, that packs the core attributes into one implementation
 6175   (PackedTokenAttributeImpl), for faster clearAttributes(), saveState(), and
 6176   restoreState(). In addition, AttributeFactory uses Java 7 MethodHandles for
 6177   instantiating Attribute implementations.  (Uwe Schindler, Robert Muir)
 6178 
 6179 * LUCENE-5609: Changed the default NumericField precisionStep from 4
 6180   to 8 (for int/float) and 16 (for long/double), for faster indexing
 6181   time and smaller indices. (Robert Muir, Uwe Schindler, Mike McCandless)
 6182 
 6183 * LUCENE-5670: Add skip/FinalOutput to FST Outputs.  (Christian
 6184   Ziech via Mike McCandless).
 6185 
 6186 * LUCENE-4236: Optimize BooleanQuery's in-order scoring. This speeds up
 6187   some types of boolean queries.  (Robert Muir)
 6188 
 6189 * LUCENE-5694: Don't score() subscorers in DisjunctionSumScorer or 
 6190   DisjunctionMaxScorer unless score() is called.  (Robert Muir)
 6191 
 6192 * LUCENE-5720: Optimize DirectPackedReader's decompression. (Robert Muir)
 6193 
 6194 * LUCENE-5722: Optimize ByteBufferIndexInput#seek() by specializing
 6195   implementations. This improves random access as used by docvalues codecs
 6196   if used with MMapDirectory.  (Robert Muir, Uwe Schindler)
 6197 
 6198 * LUCENE-5730: FSDirectory.open returns MMapDirectory for 64-bit operating
 6199   systems, not just Linux and Windows. (Robert Muir)
 6200 
 6201 * LUCENE-5703: BinaryDocValues producers don't allocate or copy bytes on 
 6202   each access anymore.  (Adrien Grand)
 6203 
 6204 * LUCENE-5721: Monotonic compression doesn't use zig-zag encoding anymore.
 6205   (Robert Muir, Adrien Grand)
 6206 
 6207 * LUCENE-5750: Speed up monotonic addressing for BINARY and SORTED_SET 
 6208   docvalues. (Robert Muir)
 6209 
 6210 * LUCENE-5751: Speed up MemoryDocValues. (Adrien Grand, Robert Muir)
 6211 
 6212 * LUCENE-5767: OrdinalMap optimizations, that mostly help on low cardinalities.
 6213   (Martijn van Groningen, Adrien Grand) 
 6214 
 6215 * LUCENE-5769: SingletonSortedSetDocValues now supports random access ordinals.
 6216   (Robert Muir)
 6217 
 6218 Bug fixes
 6219 
 6220 * LUCENE-5738: Ensure NativeFSLock prevents opening the file channel for the
 6221   lock if the lock is already obtained by the JVM. Trying to obtain an already
 6222   obtained lock in the same JVM can unlock the file might allow other processes
 6223   to lock the file even without explicitly unlocking the FileLock. This behavior
 6224   is operating system dependent. (Simon Willnauer) 
 6225 
 6226 * LUCENE-5673: MMapDirectory: Work around a "bug" in the JDK that throws
 6227   a confusing OutOfMemoryError wrapped inside IOException if the FileChannel
 6228   mapping failed because of lack of virtual address space. The IOException is
 6229   rethrown with more useful information about the problem, omitting the
 6230   incorrect OutOfMemoryError.  (Robert Muir, Uwe Schindler)
 6231 
 6232 * LUCENE-5682: NPE in QueryRescorer when Scorer is null
 6233   (Joel Bernstein, Mike McCandless)
 6234 
 6235 * LUCENE-5691: DocTermOrds lookupTerm(BytesRef) would return incorrect results
 6236   if the underlying TermsEnum supports ord() and the insertion point would
 6237   be at the end. (Robert Muir)
 6238 
 6239 * LUCENE-5618, LUCENE-5636: SegmentReader referenced unneeded files following 
 6240   doc-values updates. Now doc-values field updates are written in separate file
 6241   per field. (Shai Erera, Robert Muir)
 6242 
 6243 * LUCENE-5684: Make best effort to detect invalid usage of Lucene,
 6244   when IndexReader is reopened after all files in its index were
 6245   removed and recreated by the application (the proper way to do
 6246   this is IndexWriter.deleteAll, or opening an IndexWriter with
 6247   OpenMode.CREATE)  (Mike McCandless)
 6248 
 6249 * LUCENE-5704: Fix compilation error with Java 8u20.  (Uwe Schindler)
 6250 
 6251 * LUCENE-5710: Include the inner exception as the cause and in the
 6252   exception message when an immense term is hit during indexing (Lee
 6253   Hinman via Mike McCandless)
 6254 
 6255 * LUCENE-5724: CompoundFileWriter was failing to pass through the
 6256   IOContext in some cases, causing NRTCachingDirectory to cache
 6257   compound files when it shouldn't, then causing OOMEs.  (Mike
 6258   McCandless)
 6259 
 6260 * LUCENE-5747: Project-specific settings for the eclipse development
 6261   environment will prevent automatic code reformatting. (Shawn Heisey)
 6262 
 6263 * LUCENE-5768, LUCENE-5777: Hunspell condition checks containing character classes
 6264   were buggy. (Clinton Gormley, Robert Muir)
 6265 
 6266 Test Framework
 6267 
 6268 * LUCENE-5622: Fail tests if they print over the given limit of bytes to 
 6269   System.out or System.err. (Robert Muir, Dawid Weiss)
 6270   
 6271 * LUCENE-5619: Added backwards compatibility tests to ensure we can update existing
 6272   indexes with doc-values updates. (Shai Erera, Robert Muir)
 6273   
 6274 Build
 6275   
 6276 * LUCENE-5442: The Ant check-lib-versions target now runs Ivy resolution
 6277   transitively, then fails the build when it finds a version conflict: when a
 6278   transitive dependency's version is more recent than the direct dependency's
 6279   version specified in lucene/ivy-versions.properties.  Exceptions are
 6280   specifiable in lucene/ivy-ignore-conflicts.properties.
 6281   (Steve Rowe)
 6282   
 6283 * LUCENE-5715: Upgrade direct dependencies known to be older than transitive
 6284   dependencies: com.sun.jersey.version:1.8->1.9; com.sun.xml.bind:jaxb-impl:2.2.2->2.2.3-1;
 6285   commons-beanutils:commons-beanutils:1.7.0->1.8.3; commons-digester:commons-digester:2.0->2.1;
 6286   commons-io:commons-io:2.1->2.3; commons-logging:commons-logging:1.1.1->1.1.3;
 6287   io.netty:netty:3.6.2.Final->3.7.0.Final; javax.activation:activation:1.1->1.1.1;
 6288   javax.mail:mail:1.4.1->1.4.3; log4j:log4j:1.2.16->1.2.17; org.apache.avro:avro:1.7.4->1.7.5;
 6289   org.tukaani:xz:1.2->1.4; org.xerial.snappy:snappy-java:1.0.4.1->1.0.5 (Steve Rowe)
 6290 
 6291 ======================= Lucene 4.8.1 =======================
 6292 
 6293 Bug fixes
 6294 
 6295 * LUCENE-5639: Fix PositionLengthAttribute implementation in Token class.
 6296   (Uwe Schindler, Robert Muir)
 6297 
 6298 * LUCENE-5635: IndexWriter didn't properly handle IOException on TokenStream.reset(),
 6299   which could leave the analyzer in an inconsistent state.  (Robert Muir)
 6300 
 6301 * LUCENE-5599: HttpReplicator did not properly delegate bulk read() to wrapped
 6302   InputStream. (Christoph Kaser via Shai Erera)
 6303   
 6304 * LUCENE-5600: HttpClientBase did not properly consume a connection if a server
 6305   error occurred. (Christoph Kaser via Shai Erera)
 6306 
 6307 * LUCENE-5628: Change getFiniteStrings to iterative not recursive
 6308   implementation, so that building suggesters on a long suggestion
 6309   doesn't risk overflowing the stack; previously it consumed one Java
 6310   stack frame per character in the expanded suggestion.  If you are building
 6311   a suggester this is a nasty trap. (Robert Muir, Simon Willnauer,
 6312   Mike McCandless).
 6313 
 6314 * LUCENE-5559: Add additional argument validation for CapitalizationFilter
 6315   and CodepointCountFilter. (Ahmet Arslan via Robert Muir)
 6316 
 6317 * LUCENE-5641: SimpleRateLimiter would silently rate limit at 8 MB/sec
 6318   even if you asked for higher rates.  (Mike McCandless)
 6319 
 6320 * LUCENE-5644: IndexWriter clears which threads use which internal
 6321   thread states on flush, so that if an application reduces how many
 6322   threads it uses for indexing, that results in a reduction of how
 6323   many segments are flushed on a full-flush (e.g. to obtain a
 6324   near-real-time reader).  (Simon Willnauer, Mike McCandless)
 6325 
 6326 * LUCENE-5653: JoinUtil with ScoreMode.Avg on a multi-valued field
 6327   with more than 256 values would throw exception.
 6328   (Mikhail Khludnev via Robert Muir)
 6329 
 6330 * LUCENE-5654: Fix various close() methods that could suppress 
 6331   throwables such as OutOfMemoryError, instead returning scary messages
 6332   that look like index corruption.  (Mike McCandless, Robert Muir)
 6333 
 6334 * LUCENE-5656: Fix rare fd leak in SegmentReader when multiple docvalues
 6335   fields have been updated with IndexWriter.updateXXXDocValue and one
 6336   hits exception. (Shai Erera, Robert Muir)
 6337 
 6338 * LUCENE-5660: AnalyzingSuggester.build will now throw IllegalArgumentException if
 6339   you give it a longer suggestion than it can handle (Robert Muir, Mike McCandless)
 6340 
 6341 * LUCENE-5662: Add missing checks to Field to prevent IndexWriter.abort
 6342   if a stored value is null. (Robert Muir)
 6343 
 6344 * LUCENE-5668: Fix off-by-one in TieredMergePolicy (Mike McCandless)
 6345 
 6346 * LUCENE-5671: Upgrade ICU version to fix an ICU concurrency problem that
 6347   could cause exceptions when indexing. (feedly team, Robert Muir)
 6348 
 6349 ======================= Lucene 4.8.0 =======================
 6350 
 6351 System Requirements
 6352 
 6353 * LUCENE-4747, LUCENE-5514: Move to Java 7 as minimum Java version.
 6354   (Robert Muir, Uwe Schindler)
 6355 
 6356 Changes in Runtime Behavior
 6357 
 6358 * LUCENE-5472: IndexWriter.addDocument will now throw an IllegalArgumentException
 6359   if a Term to be indexed exceeds IndexWriter.MAX_TERM_LENGTH.  To recreate previous
 6360   behavior of silently ignoring these terms, use LengthFilter in your Analyzer.
 6361   (hossman, Mike McCandless, Varun Thacker)
 6362 
 6363 New Features
 6364 
 6365 * LUCENE-5356: Morfologik filter can accept custom dictionary resources.
 6366   (Michal Hlavac, Dawid Weiss)
 6367 
 6368 * LUCENE-5454: Add SortedSetSortField to lucene/sandbox, to allow sorting
 6369   on multi-valued field. (Robert Muir)
 6370 
 6371 * LUCENE-5478: CommonTermsQuery now allows to create custom term queries
 6372   similar to the query parser by overriding a newTermQuery method.
 6373   (Simon Willnauer)
 6374 
 6375 * LUCENE-5477: AnalyzingInfixSuggester now supports near-real-time
 6376   additions and updates (to change weight or payload of an existing
 6377   suggestion).  (Mike McCandless)
 6378 
 6379 * LUCENE-5482: Improve default TurkishAnalyzer by adding apostrophe
 6380   handling suitable for Turkish.  (Ahmet Arslan via Robert Muir)
 6381 
 6382 * LUCENE-5479: FacetsConfig subclass can now customize the default
 6383   per-dim facets configuration.  (Rob Audenaerde via Mike McCandless)
 6384 
 6385 * LUCENE-5485: Add circumfix support to HunspellStemFilter. (Robert Muir)
 6386 
 6387 * LUCENE-5224: Add iconv, oconv, and ignore support to HunspellStemFilter.
 6388   (Robert Muir)
 6389 
 6390 * LUCENE-5493: SortingMergePolicy, and EarlyTerminatingSortingCollector
 6391   support arbitrary Sort specifications.  
 6392   (Robert Muir, Mike McCandless, Adrien Grand)
 6393   
 6394 * LUCENE-3758: Allow the ComplexPhraseQueryParser to search order or 
 6395   un-order proximity queries. (Ahmet Arslan via Erick Erickson)
 6396   
 6397 * LUCENE-5530: ComplexPhraseQueryParser throws ParseException for fielded queries.
 6398   (Erick Erickson via Tomas Fernandez Lobbe and Ahmet Arslan)
 6399 
 6400 * LUCENE-5513: Add IndexWriter.updateBinaryDocValue which lets
 6401   you update the value of a BinaryDocValuesField without reindexing the
 6402   document(s). (Shai Erera)
 6403 
 6404 * LUCENE-4072: Add ICUNormalizer2CharFilter, which lets you do unicode normalization
 6405   with offset correction before the tokenizer. (David Goldfarb, Ippei UKAI via Robert Muir)
 6406 
 6407 * LUCENE-5476: Add RandomSamplingFacetsCollector for computing facets on a sampled
 6408   set of matching hits, in cases where there are millions of hits.
 6409   (Rob Audenaerde, Gilad Barkai, Shai Erera)
 6410 
 6411 * LUCENE-4984: Add SegmentingTokenizerBase, abstract class for tokenizers
 6412   that want to do two-pass tokenization such as by sentence and then by word.
 6413   (Robert Muir)
 6414 
 6415 * LUCENE-5489: Add Rescorer/QueryRescorer, to resort the hits from a
 6416   first pass search using scores from a more costly second pass
 6417   search. (Simon Willnauer, Robert Muir, Mike McCandless)
 6418 
 6419 * LUCENE-5528: Add context to suggesters (InputIterator and Lookup
 6420   classes), and fix AnalyzingInfixSuggester to handle contexts.
 6421   Suggester contexts allow you to filter suggestions.  (Areek Zillur,
 6422   Mike McCandless)
 6423 
 6424 * LUCENE-5545: Add SortRescorer and Expression.getRescorer, to
 6425   resort the hits from a first pass search using a Sort or an
 6426   Expression. (Simon Willnauer, Robert Muir, Mike McCandless)
 6427 
 6428 * LUCENE-5558: Add TruncateTokenFilter which truncates terms to
 6429   the specified length.  (Ahmet Arslan via Robert Muir)
 6430 
 6431 * LUCENE-2446: Added checksums to lucene index files. As of 4.8, the last 8 
 6432   bytes of each file contain a zlib-crc32 checksum. Small metadata files are
 6433   verified on load. Larger files can be checked on demand via 
 6434   AtomicReader.checkIntegrity. You can configure this to happen automatically
 6435   before merges by enabling IndexWriterConfig.setCheckIntegrityAtMerge.
 6436   (Robert Muir)
 6437 
 6438 * LUCENE-5580: Checksums are automatically verified on the default stored
 6439   fields format when performing a bulk merge. (Adrien Grand)
 6440 
 6441 * LUCENE-5602: Checksums are automatically verified on the default term
 6442   vectors format when performing a bulk merge. (Adrien Grand, Robert Muir)
 6443 
 6444 * LUCENE-5583: Added DataInput.skipBytes. ChecksumIndexInput can now seek, but
 6445   only forward. (Adrien Grand, Mike McCandless, Simon Willnauer, Uwe Schindler)
 6446 
 6447 * LUCENE-5588: Lucene now calls fsync() on the index directory, ensuring
 6448   that all file metadata is persisted on disk in case of power failure.
 6449   This does not work on all file systems and operating systems, but Linux
 6450   and MacOSX are known to work. On Windows, fsyncing a directory is not
 6451   possible with Java APIs.  (Mike McCandless, Uwe Schindler)
 6452 
 6453 API Changes
 6454 
 6455 * LUCENE-5454: Add RandomAccessOrds, an optional extension of SortedSetDocValues
 6456   that supports random access to the ordinals in a document. (Robert Muir)
 6457 
 6458 * LUCENE-5468: Move offline Sort (from suggest module) to OfflineSort. (Robert Muir)
 6459 
 6460 * LUCENE-5493: SortingMergePolicy and EarlyTerminatingSortingCollector take
 6461   Sort instead of Sorter. BlockJoinSorter is removed, replaced with 
 6462   BlockJoinComparatorSource, which can take a Sort for ordering of parents
 6463   and a separate Sort for ordering of children within a block. 
 6464   (Robert Muir, Mike McCandless, Adrien Grand)
 6465 
 6466 * LUCENE-5516: MergeScheduler#merge() now accepts a MergeTrigger as well as
 6467   a boolean that indicates if a new merge was found in the caller thread before
 6468   the scheduler was called. (Simon Willnauer)
 6469 
 6470 * LUCENE-5487: Separated bulk scorer (new Weight.bulkScorer method) from
 6471   normal scoring (Weight.scorer) for those queries that can do bulk
 6472   scoring more efficiently, e.g. BooleanQuery in some cases.  This
 6473   also simplified the Weight.scorer API by removing the two confusing
 6474   booleans.  (Robert Muir, Uwe Schindler, Mike McCandless)
 6475 
 6476 * LUCENE-5519: TopNSearcher now allows to retrieve incomplete results if the max
 6477   size of the candidate queue is unknown. The queue can still be bound in order
 6478   to apply pruning while retrieving the top N but will not throw an exception if
 6479   too many results are rejected to guarantee an absolutely correct top N result.
 6480   The TopNSearcher now returns a struct like class that indicates if the result
 6481   is complete in the sense of the top N or not. Consumers of this API should assert
 6482   on the completeness if the bounded queue size is know ahead of time. (Simon Willnauer)
 6483 
 6484 * LUCENE-4984: Deprecate ThaiWordFilter and smartcn SentenceTokenizer and WordTokenFilter.
 6485   These filters would not work correctly with CharFilters and could not be safely placed
 6486   at an arbitrary position in the analysis chain. Use ThaiTokenizer and HMMChineseTokenizer
 6487   instead. (Robert Muir)
 6488 
 6489 * LUCENE-5543: Remove/deprecate Directory.fileExists (Mike McCandless)
 6490 
 6491 * LUCENE-5573: Move docvalues constants and helper methods to o.a.l.index.DocValues.
 6492   (Dawid Weiss, Robert Muir)
 6493 
 6494 * LUCENE-5604: Switched BytesRef.hashCode to MurmurHash3 (32 bit).
 6495   TermToBytesRefAttribute.fillBytesRef no longer returns the hash
 6496   code.  BytesRefHash now uses MurmurHash3 for its hashing.  (Robert
 6497   Muir, Mike McCandless)
 6498 
 6499 Optimizations
 6500 
 6501 * LUCENE-5468: HunspellStemFilter uses 10 to 100x less RAM. It also loads
 6502   all known openoffice dictionaries without error, and supports an additional 
 6503   longestOnly option for a less aggressive approach.  (Robert Muir)
 6504 
 6505 * LUCENE-4848: Use Java 7 NIO2-FileChannel instead of RandomAccessFile
 6506   for NIOFSDirectory and MMapDirectory. This allows to delete open files
 6507   on Windows if NIOFSDirectory is used, mmapped files are still locked.
 6508   (Michael Poindexter, Robert Muir, Uwe Schindler)
 6509 
 6510 * LUCENE-5515: Improved TopDocs#merge to create a merged ScoreDoc
 6511   array with length of at most equal to the specified size instead of length
 6512   equal to at most from + size as was before. (Martijn van Groningen)
 6513   
 6514 * LUCENE-5529: Spatial search of non-point indexed shapes should be a little
 6515   faster due to skipping intersection tests on redundant cells. (David Smiley)
 6516 
 6517 Bug fixes
 6518 
 6519 * LUCENE-5483: Fix inaccuracies in HunspellStemFilter. Multi-stage affix-stripping,
 6520   prefix-suffix dependencies, and COMPLEXPREFIXES now work correctly according
 6521   to the hunspell algorithm. Removed recursionCap parameter, as it's no longer needed, rules for
 6522   recursive affix application are driven correctly by continuation classes in the affix file.
 6523   (Robert Muir)
 6524 
 6525 * LUCENE-5497: HunspellStemFilter properly handles escaped terms and affixes without conditions.
 6526   (Robert Muir)
 6527 
 6528 * LUCENE-5505: HunspellStemFilter ignores BOM markers in dictionaries and handles varying
 6529   types of whitespace in SET/FLAG commands. (Robert Muir)
 6530 
 6531 * LUCENE-5507: Fix HunspellStemFilter loading of dictionaries with large amounts of aliases
 6532   etc before the encoding declaration.  (Robert Muir)
 6533 
 6534 * LUCENE-5111: Fix WordDelimiterFilter to return offsets in correct order.  (Robert Muir)
 6535 
 6536 * LUCENE-5555: Fix SortedInputIterator to correctly encode/decode contexts in presence of payload (Areek Zillur)
 6537 
 6538 * LUCENE-5559: Add missing argument checks to tokenfilters taking
 6539   numeric arguments.  (Ahmet Arslan via Robert Muir)
 6540 
 6541 * LUCENE-5568: Benchmark module's "default.codec" option didn't work. (David Smiley)
 6542 
 6543 * SOLR-5983: HTMLStripCharFilter is treating CDATA sections incorrectly.
 6544   (Dan Funk, Steve Rowe)
 6545 
 6546 * LUCENE-5615: Validate per-segment delete counts at write time, to
 6547   help catch bugs that might otherwise cause corruption (Mike McCandless)
 6548 
 6549 * LUCENE-5612: NativeFSLockFactory no longer deletes its lock file. This cannot be done
 6550   safely without the risk of deleting someone else's lock file. If you use NativeFSLockFactory,
 6551   you may see write.lock hanging around from time to time: it's harmless.  
 6552   (Uwe Schindler, Mike McCandless, Robert Muir)
 6553 
 6554 * LUCENE-5624: Ensure NativeFSLockFactory does not leak file handles if it is unable
 6555   to obtain the lock. (Uwe Schindler, Robert Muir)
 6556 
 6557 * LUCENE-5626: Fix bug in SimpleFSLockFactory's obtain() that sometimes throwed
 6558   IOException (ERROR_ACCESS_DENIED) on Windows if the lock file was created
 6559   concurrently. This error is now handled the same way like in NativeFSLockFactory
 6560   by returning false.  (Uwe Schindler, Robert Muir, Dawid Weiss)
 6561 
 6562 * LUCENE-5630: Add missing META-INF entry for UpperCaseFilterFactory.
 6563   (Robert Muir)
 6564 
 6565 Tests
 6566   
 6567 * LUCENE-5630: Fix TestAllAnalyzersHaveFactories to correctly check for existence
 6568   of class and corresponding Map<String,String> ctor.  (Uwe Schindler, Robert Muir)
 6569 
 6570 Test Framework
 6571 
 6572 * LUCENE-5592: Incorrectly reported uncloseable files. (Dawid Weiss)
 6573 
 6574 * LUCENE-5577: Temporary folder and file management (and cleanup facilities)
 6575   (Mark Miller, Uwe Schindler, Dawid Weiss)
 6576 
 6577 * LUCENE-5567: When a suite fails with zombie threads failure marker and count 
 6578   is not propagated properly. (Dawid Weiss)
 6579 
 6580 * LUCENE-5449: Rename _TestUtil and _TestHelper to remove the leading _.
 6581 
 6582 * LUCENE-5501: Added random out-of-order collection testing (when the collector
 6583   supports it) to AssertingIndexSearcher. (Adrien Grand)
 6584 
 6585 Build
 6586 
 6587 * LUCENE-5463: RamUsageEstimator.(human)sizeOf(Object) is now a forbidden API.
 6588   (Adrien Grand, Robert Muir)
 6589 
 6590 * LUCENE-5512: Remove redundant typing (use diamond operator) throughout
 6591   the codebase.  (Furkan KAMACI via Robert Muir)
 6592 
 6593 * LUCENE-5614: Enable building on Java 8 using Apache Ant 1.8.3 or 1.8.4
 6594   by adding a workaround for the Ant bug.  (Uwe Schindler)
 6595 
 6596 * LUCENE-5612: Add a new Ant target in lucene/core to test LockFactory
 6597   implementations: "ant test-lock-factory".  (Uwe Schindler, Mike McCandless,
 6598   Robert Muir)
 6599 
 6600 Documentation
 6601 
 6602 * LUCENE-5534: Add javadocs to GreekStemmer methods. 
 6603   (Stamatis Pitsios via Robert Muir)
 6604 
 6605 ======================= Lucene 4.7.2 =======================
 6606 
 6607 Bug Fixes
 6608 
 6609 * LUCENE-5574: Closing a near-real-time reader no longer attempts to
 6610   delete unreferenced files if the original writer has been closed;
 6611   this could cause index corruption in certain cases where index files
 6612   were directly changed (deleted, overwritten, etc.) in the index
 6613   directory outside of Lucene.  (Simon Willnauer, Shai Erera, Robert
 6614   Muir, Mike McCandless)
 6615 
 6616 * LUCENE-5570: Don't let FSDirectory.sync() create new zero-byte files, instead throw
 6617   exception if a file is missing.  (Uwe Schindler, Mike McCandless, Robert Muir)
 6618 
 6619 ======================= Lucene 4.7.1 =======================
 6620 
 6621 Changes in Runtime Behavior
 6622 
 6623 * LUCENE-5532: AutomatonQuery.equals is no longer implemented as "accepts same language".
 6624   This was inconsistent with hashCode, and unnecessary for any subclasses in Lucene.
 6625   If you desire this in a custom subclass, minimize the automaton.  (Robert Muir)
 6626 
 6627 Bug Fixes
 6628 
 6629 * LUCENE-5450: Fix getField() NPE issues with SpanOr/SpanNear when they have an 
 6630   empty list of clauses. This can happen for example,  when a wildcard matches 
 6631   no terms.  (Tim Allison via Robert Muir)
 6632 
 6633 * LUCENE-5473: Throw IllegalArgumentException, not
 6634   NullPointerException, if the synonym map is empty when creating
 6635   SynonymFilter (帅广应 via Mike McCandless)
 6636 
 6637 * LUCENE-5432: EliasFanoDocIdSet: Fix number of index entry bits when the maximum
 6638   entry is a power of 2. (Paul Elschot via Adrien Grand)
 6639 
 6640 * LUCENE-5466: query is always null in countDocsWithClass() of SimpleNaiveBayesClassifier.
 6641   (Koji Sekiguchi)
 6642 
 6643 * LUCENE-5502: Fixed TermsFilter.equals that could return true for different
 6644   filters. (Igor Motov via Adrien Grand)
 6645   
 6646 * LUCENE-5522: FacetsConfig didn't add drill-down terms for association facet 
 6647   fields labels. (Shai Erera)
 6648 
 6649 * LUCENE-5520: ToChildBlockJoinQuery would hit
 6650   ArrayIndexOutOfBoundsException if a parent document had no children
 6651   (Sally Ang via Mike McCandless)
 6652 
 6653 * LUCENE-5532: AutomatonQuery.hashCode was not thread-safe. (Robert Muir)
 6654 
 6655 * LUCENE-5525: Implement MultiFacets.getAllDims, so you can do sparse
 6656   facets through DrillSideways, for example.  (Jose Peleteiro, Mike
 6657   McCandless)
 6658 
 6659 * LUCENE-5481: IndexWriter.forceMerge used to run a merge even if there was a
 6660   single segment in the index. (Adrien Grand, Mike McCandless)
 6661 
 6662 * LUCENE-5538: Fix FastVectorHighlighter bug with index-time synonyms when the
 6663   query is more complex than a single phrase.  (Robert Muir)
 6664 
 6665 * LUCENE-5544: Exceptions during IndexWriter.rollback could leak file handles
 6666   and the write lock. (Robert Muir)
 6667 
 6668 * LUCENE-4978: Spatial RecursivePrefixTree queries could result in false-negatives for
 6669   indexed shapes within 1/2 maxDistErr from the edge of the query shape.  This meant
 6670   searching for a point by the same point as a query rarely worked.  (David Smiley)
 6671 
 6672 * LUCENE-5553: IndexReader#ReaderClosedListener is not always invoked when 
 6673   IndexReader#close() is called or if refCount is 0. If an exception is 
 6674   thrown during internal close or on any of the close listeners some or all
 6675   listeners might be missed. This can cause memory leaks if the core listeners
 6676   are used to clear caches. (Simon Willnauer)
 6677 
 6678 Build
 6679 
 6680 * LUCENE-5511: "ant precommit" / "ant check-svn-working-copy" now work again
 6681   with any working copy format (thanks to svnkit 1.8.4).  (Uwe Schindler)
 6682 
 6683 ======================= Lucene 4.7.0 =======================
 6684 
 6685 New Features
 6686 
 6687 * LUCENE-5336: Add SimpleQueryParser: parser for human-entered queries.
 6688   (Jack Conradson via Robert Muir)
 6689   
 6690 * LUCENE-5337: Add Payload support to FileDictionary (Suggest) and make it more
 6691   configurable (Areek Zillur via Erick Erickson)
 6692 
 6693 * LUCENE-5329: suggest: DocumentDictionary and
 6694   DocumentExpressionDictionary are now lenient for dirty documents
 6695   (missing the term, weight or payload).  (Areek Zillur via
 6696   Mike McCandless)
 6697 
 6698 * LUCENE-5404: Add .getCount method to all suggesters (Lookup); persist count 
 6699   metadata on .store(); Dictionary returns InputIterator; Dictionary.getWordIterator
 6700   renamed to .getEntryIterator. (Areek Zillur)
 6701 
 6702 * SOLR-1871: The RangeMapFloatFunction accepts an arbitrary ValueSource
 6703   as target and default values. (Chris Harris, shalin)
 6704 
 6705 * LUCENE-5371: Speed up Lucene range faceting from O(N) per hit to
 6706   O(log(N)) per hit using segment trees; this only really starts to
 6707   matter in practice if the number of ranges is over 10 or so.  (Mike
 6708   McCandless)
 6709 
 6710 * LUCENE-5379: Add Analyzer for Kurdish.  (Robert Muir)
 6711 
 6712 * LUCENE-5369: Added an UpperCaseFilter to make UPPERCASE tokens. (ryan)
 6713 
 6714 * LUCENE-5345: Add a new BlendedInfixSuggester, which is like
 6715   AnalyzingInfixSuggester but boosts suggestions that matched tokens
 6716   with lower positions.  (Remi Melisson via Mike McCandless)
 6717 
 6718 * LUCENE-5399: When sorting by String (SortField.STRING), you can now
 6719   specify whether missing values should be sorted first (the default),
 6720   using SortField.setMissingValue(SortField.STRING_FIRST), or last,
 6721   using SortField.setMissingValue(SortField.STRING_LAST). (Rob Muir,
 6722   Mike McCandless)
 6723 
 6724 * LUCENE-5099: QueryNode should have the ability to detach from its node
 6725   parent. Added QueryNode.removeFromParent() that allows nodes to be 
 6726   detached from its parent node. (Adriano Crestani)
 6727 
 6728 * LUCENE-5395 LUCENE-5451: Upgrade to Spatial4j 0.4.1: Parses WKT (including
 6729   ENVELOPE) with extension "BUFFER"; buffering a point results in a Circle.
 6730   JTS isn't needed for WKT any more but remains required for Polygons. New
 6731   Shapes: ShapeCollection and BufferedLineString. Various other improvements and
 6732   bug fixes too. More info:
 6733   https://github.com/spatial4j/spatial4j/blob/master/CHANGES.md  (David Smiley)
 6734 
 6735 * LUCENE-5415: Add multitermquery (wildcards,prefix,etc) to PostingsHighlighter.
 6736   (Mike McCandless, Robert Muir)
 6737 
 6738 * LUCENE-3069: Add two memory resident dictionaries (FST terms dictionary and 
 6739   FSTOrd terms dictionary) to improve primary key lookups. The PostingsBaseFormat 
 6740   API is also changed so that term dictionaries get the ability to block 
 6741   encode term metadata, and all dictionary implementations can now plug in any 
 6742   PostingsBaseFormat. (Han Jiang, Mike McCandless)
 6743 
 6744 * LUCENE-5353: ShingleFilter's filler token should be configurable.
 6745   (Ahmet Arslan, Simon Willnauer, Steve Rowe)
 6746 
 6747 * LUCENE-5320: Add SearcherTaxonomyManager over search and taxonomy index
 6748   directories (i.e. not only NRT). (Shai Erera)
 6749 
 6750 * LUCENE-5410: Add fuzzy and near support via '~' operator to SimpleQueryParser.
 6751   (Lee Hinman via Robert Muir)
 6752 
 6753 * LUCENE-5426: Make SortedSetDocValuesReaderState abstract to allow
 6754   custom implementations for Lucene doc values faceting (John Wang via
 6755   Mike McCandless)
 6756 
 6757 * LUCENE-5434: NRT support for file systems that do no have delete on last
 6758   close or cannot delete while referenced semantics. 
 6759   (Mark Miller, Mike McCandless)
 6760 
 6761 * LUCENE-5418: Drilling down or sideways on a Lucene facet range
 6762   (using Range.getFilter()) is now faster for costly filters (uses
 6763   random access, not iteration); range facet counts now accept a
 6764   fast-match filter to avoid computing the value for documents that
 6765   are out of bounds, e.g. using a bounding box filter with distance
 6766   range faceting.  (Mike McCandless)
 6767 
 6768 * LUCENE-5440: Add LongBitSet for managing more than 2.1B bits (otherwise use
 6769   FixedBitSet). (Shai Erera)
 6770 
 6771 * LUCENE-5437: ASCIIFoldingFilter now has an option to preserve the original token
 6772   and emit it on the same position as the folded token only if the actual token was
 6773   folded. (Simon Willnauer, Nik Everett)
 6774 
 6775 * LUCENE-5408: Add spatial SerializedDVStrategy that serializes a binary
 6776   representations of a shape into BinaryDocValues. It supports exact geometry
 6777   relationship calculations. (David Smiley)
 6778 
 6779 * LUCENE-5457: Add SloppyMath.earthDiameter(double latitude) that returns an
 6780   approximate value of the diameter of the earth at the given latitude.
 6781   (Adrien Grand)
 6782 
 6783 * LUCENE-5979: FilteredQuery uses the cost API to decide on whether to use
 6784   random-access or leap-frog to intersect the filter with the query.
 6785   (Adrien Grand)
 6786 
 6787 Build
 6788 
 6789 * LUCENE-5217,LUCENE-5420: Maven config: get dependencies from Ant+Ivy config;
 6790   disable transitive dependency resolution for all depended-on artifacts by
 6791   putting an exclusion for each transitive dependency in the
 6792   <dependencyManagement> section of the grandparent POM. (Steve Rowe)
 6793 
 6794 * LUCENE-5322: Clean up / simplify Maven-related Ant targets.
 6795   (Steve Rowe)
 6796 
 6797 * LUCENE-5347: Upgrade forbidden-apis checker to version 1.4.
 6798   (Uwe Schindler)
 6799 
 6800 * LUCENE-4381: Upgrade analysis/icu to 52.1. (Robert Muir)
 6801 
 6802 * LUCENE-5357: Upgrade StandardTokenizer and UAX29URLEmailTokenizer to
 6803   Unicode 6.3; update UAX29URLEmailTokenizer's recognized top level
 6804   domains in URLs and Emails from the IANA Root Zone Database.
 6805   (Steve Rowe)
 6806   
 6807 * LUCENE-5360: Add support for developing in Netbeans IDE.
 6808   (Michal Hlavac, Uwe Schindler, Steve Rowe)
 6809 
 6810 * SOLR-5590: Upgrade HttpClient/HttpComponents to 4.3.x.
 6811   (Karl Wright via Shawn Heisey)
 6812 
 6813 * LUCENE-5385: "ant precommit" / "ant check-svn-working-copy" now work
 6814   for SVN 1.8 or GIT checkouts. The ANT target prints a warning instead
 6815   of failing. It also instructs the user, how to run on SVN 1.8 working
 6816   copies.  (Robert Muir, Uwe Schindler)
 6817 
 6818 * LUCENE-5383: fix changes2html to link pull requests (Steve Rowe)
 6819 
 6820 * LUCENE-5411: Upgrade to released JFlex 1.5.0; stop requiring
 6821   a locally built JFlex snapshot jar. (Steve Rowe)
 6822 
 6823 * LUCENE-5465: Solr Contrib "map-reduce" breaks Manifest of all other
 6824   JAR files by adding a broken Main-Class attribute.
 6825   (Uwe Schindler, Steve Rowe)
 6826 
 6827 Bug fixes
 6828 
 6829 * LUCENE-5285: Improved highlighting of multi-valued fields with
 6830   FastVectorHighlighter. (Nik Everett via Adrien Grand)
 6831 
 6832 * LUCENE-5391: UAX29URLEmailTokenizer should not tokenize no-scheme
 6833   domain-only URLs that are followed by an alphanumeric character.
 6834   (Chris Geeringh, Steve Rowe)
 6835 
 6836 * LUCENE-5405: If an analysis component throws an exception, Lucene
 6837   logs the field name to the info stream to assist in
 6838   diagnosis. (Benson Margulies)
 6839 
 6840 * SOLR-5661: PriorityQueue now refuses to allocate itself if the
 6841   incoming maxSize is too large (Raintung Li via Mike McCandless)
 6842 
 6843 * LUCENE-5228: IndexWriter.addIndexes(Directory[]) now acquires a
 6844   write lock in each Directory, to ensure that no open IndexWriter is
 6845   changing the incoming indices.  This also means that you cannot pass
 6846   the same Directory to multiple concurrent addIndexes calls (which is
 6847   anyways unusual).  (Robert Muir, Mike McCandless)
 6848 
 6849 * LUCENE-5415: SpanMultiTermQueryWrapper didn't handle its boost in
 6850   hashcode/equals/tostring/rewrite.  (Robert Muir)
 6851 
 6852 * LUCENE-5409: ToParentBlockJoinCollector.getTopGroups would fail to
 6853   return any groups when the joined query required more than one
 6854   rewrite step (Peng Cheng via Mike McCandless)
 6855 
 6856 * LUCENE-5398: NormValueSource was incorrectly casting the long value
 6857   to byte, before calling Similarity.decodeNormValue.  (Peng Cheng via
 6858   Mike McCandless)
 6859 
 6860 * LUCENE-5436: ReferenceManager#accquire can result in infinite loop if
 6861   managed resource is abused outside of the ReferenceManager. Decrementing
 6862   the reference without a corresponding incRef() call can cause an infinite
 6863   loop. ReferenceManager now throws IllegalStateException if currently managed
 6864   resources ref count is 0. (Simon Willnauer)
 6865 
 6866 * LUCENE-5443: Lucene45DocValuesProducer.ramBytesUsed() may throw
 6867   ConcurrentModificationException. (Shai Erera, Simon Willnauer)
 6868 
 6869 * LUCENE-5444: MemoryIndex didn't respect the analyzers offset gap and
 6870   offsets were corrupted if multiple fields with the same name were
 6871   added to the memory index. (Britta Weber, Simon Willnauer)
 6872 
 6873 * LUCENE-5447: StandardTokenizer should break at consecutive chars matching
 6874   Word_Break = MidLetter, MidNum and/or MidNumLet (Steve Rowe)
 6875 
 6876 * LUCENE-5462: RamUsageEstimator.sizeOf(Object) is not used anymore to
 6877   estimate memory usage of segments. This used to make
 6878   SegmentReader.ramBytesUsed very CPU-intensive. (Adrien Grand)
 6879 
 6880 * LUCENE-5461: ControlledRealTimeReopenThread would sometimes wait too
 6881   long (up to targetMaxStaleSec) when a searcher is waiting for a
 6882   specific generation, when it should have waited for at most
 6883   targetMinStaleSec. (Hans Lund via Mike McCandless)
 6884   
 6885 API Changes
 6886 
 6887 * LUCENE-5339: The facet module was simplified/reworked to make the
 6888   APIs more approachable to new users. Note: when migrating to the new 
 6889   API, you must pass the Document that is returned from FacetConfig.build() 
 6890   to IndexWriter.addDocument(). (Shai Erera, Gilad Barkai, Rob
 6891   Muir, Mike McCandless)
 6892 
 6893 * LUCENE-5405: Make ShingleAnalyzerWrapper.getWrappedAnalyzer() public final (gsingers)
 6894 
 6895 * LUCENE-5395: The SpatialArgsParser now only reads WKT, no more "lat, lon"
 6896   etc. but it's easy to override the parseShape method if you wish. (David
 6897   Smiley)
 6898 
 6899 * LUCENE-5414: DocumentExpressionDictionary was renamed to 
 6900   DocumentValueSourceDictionary and all dependencies to the lucene-expression
 6901   module were removed from lucene-suggest. DocumentValueSourceDictionary now
 6902   only accepts a ValueSource instead of a convenience ctor for an expression
 6903   string. (Simon Willnauer)
 6904 
 6905 * LUCENE-3069: PostingsWriterBase and PostingsReaderBase are no longer
 6906   responsible for encoding/decoding a block of terms.  Instead, they
 6907   should encode/decode each term to/from a long[] and byte[].  (Han
 6908   Jiang, Mike McCandless)
 6909   
 6910 * LUCENE-5425: FacetsCollector and MatchingDocs use a general DocIdSet, 
 6911   allowing for custom implementations to be used when faceting.
 6912   (John Wang, Lei Wang, Shai Erera) 
 6913 
 6914 Optimizations
 6915 
 6916 * LUCENE-5372: Replace StringBuffer by StringBuilder, where possible.
 6917   (Joshua Hartman via Uwe Schindler, Dawid Weiss, Mike McCandless)
 6918 
 6919 * LUCENE-5271: A slightly more accurate SloppyMath distance.
 6920   (Gilad Barkai via Ryan Ernst)
 6921 
 6922 * LUCENE-5399: Deep paging using IndexSearcher.searchAfter when
 6923   sorting by fields is faster (Rob Muir, Mike McCandless)
 6924 
 6925 Changes in Runtime Behavior
 6926 
 6927 * LUCENE-5362: IndexReader and SegmentCoreReaders now throw 
 6928   AlreadyClosedException if the refCount in incremented but
 6929   is less that 1. (Simon Willnauer)
 6930 
 6931 Documentation
 6932 
 6933 * LUCENE-5384: Add some tips for making tokenfilters and tokenizers 
 6934   to the analysis package overview.  
 6935   (Benson Margulies via Robert Muir - pull request #12)
 6936 
 6937 * LUCENE-5389: Add more guidance in the analysis documentation 
 6938   package overview.
 6939   (Benson Margulies via Robert Muir - pull request #14)
 6940 
 6941 ======================= Lucene 4.6.1 =======================
 6942 
 6943 Bug fixes
 6944 
 6945 * LUCENE-5373: Memory usage of
 6946   [Lucene40/Lucene42/Memory/Direct]DocValuesFormat was over-estimated.
 6947   (Shay Banon, Adrien Grand, Robert Muir)
 6948 
 6949 * LUCENE-5361: Fixed handling of query boosts in FastVectorHighlighter.
 6950   (Nik Everett via Adrien Grand)
 6951 
 6952 * LUCENE-5374: IndexWriter processes internal events after the it
 6953   closed itself internally. This rare condition can happen if an
 6954   IndexWriter has internal changes that were not fully applied yet
 6955   like when index / flush requests happen concurrently to the close or
 6956   rollback call. (Simon Willnauer)
 6957 
 6958 * LUCENE-5394: Fix TokenSources.getTokenStream to return payloads if
 6959   they were indexed with the term vectors. (Mike McCandless)
 6960 
 6961 * LUCENE-5344: Flexible StandardQueryParser behaves differently than
 6962   ClassicQueryParser. (Adriano Crestani)
 6963 
 6964 * LUCENE-5375: ToChildBlockJoinQuery works harder to detect mis-use,
 6965   when the parent query incorrectly returns child documents, and throw
 6966   a clear exception saying so. (Dr. Oleg Savrasov via Mike McCandless)
 6967 
 6968 * LUCENE-5401: Field.StringTokenStream#end() calls super.end() now,
 6969   preventing wrong term positions for fields that use
 6970   StringTokenStream. (Michael Busch)
 6971 
 6972 * LUCENE-5377: IndexWriter.addIndexes(Directory[]) would cause corruption
 6973   on Lucene 4.6 if any index segments were Lucene 4.0-4.5.
 6974   (Littlestar, Mike McCandless, Shai Erera, Robert Muir)
 6975 
 6976 ======================= Lucene 4.6.0 =======================
 6977 
 6978 New Features
 6979 
 6980 * LUCENE-4906: PostingsHighlighter can now render to custom Object,
 6981   for advanced use cases where String is too restrictive (Luca
 6982   Cavanna, Robert Muir, Mike McCandless)
 6983 
 6984 * LUCENE-5133: Changed AnalyzingInfixSuggester.highlight to return
 6985   Object instead of String, to allow for advanced use cases where
 6986   String is too restrictive (Robert Muir, Shai Erera, Mike
 6987   McCandless)
 6988 
 6989 * LUCENE-5207, LUCENE-5334: Added expressions module for customizing ranking
 6990   with script-like syntax. 
 6991   (Jack Conradson, Ryan Ernst, Uwe Schindler via Robert Muir)
 6992 
 6993 * LUCENE-5180: ShingleFilter now creates shingles with trailing holes,
 6994   for example if a StopFilter had removed the last token.  (Mike
 6995   McCandless)
 6996 
 6997 * LUCENE-5219: Add support to SynonymFilterFactory for custom
 6998   parsers.  (Ryan Ernst via Robert Muir)
 6999 
 7000 * LUCENE-5235: Tokenizers now throw an IllegalStateException if the
 7001   consumer does not call reset() before consuming the stream. Previous
 7002   versions throwed NullPointerException or ArrayIndexOutOfBoundsException
 7003   on best effort which was not user-friendly.
 7004   (Uwe Schindler, Robert Muir)
 7005 
 7006 * LUCENE-5240: Tokenizers now throw an IllegalStateException if the
 7007   consumer neglects to call close() on the previous stream before consuming
 7008   the next one. (Uwe Schindler, Robert Muir)
 7009 
 7010 * LUCENE-5214: Add new FreeTextSuggester, to predict the next word
 7011   using a simple ngram language model.  This is useful for the "long
 7012   tail" suggestions, when a primary suggester fails to find a
 7013   suggestion.  (Mike McCandless)
 7014 
 7015 * LUCENE-5251: New DocumentDictionary allows building suggesters via
 7016   contents of existing field, weight and optionally payload stored
 7017   fields in an index (Areek Zillur via Mike McCandless)
 7018 
 7019 * LUCENE-5261: Add QueryBuilder, a simple API to build queries from
 7020   the analysis chain directly, or to make it easier to implement
 7021   query parsers.  (Robert Muir, Uwe Schindler)
 7022 
 7023 * LUCENE-5270: Add Terms.hasFreqs, to determine whether a given field
 7024   indexed per-doc term frequencies.  (Mike McCandless)
 7025 
 7026 * LUCENE-5269: Add CodepointCountFilter. (Robert Muir)
 7027 
 7028 * LUCENE-5294: Suggest module: add DocumentExpressionDictionary to
 7029   compute each suggestion's weight using a javascript expression.
 7030   (Areek Zillur via Mike McCandless)
 7031 
 7032 * LUCENE-5274: FastVectorHighlighter now supports highlighting against several
 7033   indexed fields. (Nik Everett via Adrien Grand)
 7034 
 7035 * LUCENE-5304: SingletonSortedSetDocValues can now return the wrapped
 7036   SortedDocValues (Robert Muir, Adrien Grand)
 7037 
 7038 * LUCENE-2844: The benchmark module can now test the spatial module. See
 7039   spatial.alg  (David Smiley, Liviy Ambrose)
 7040 
 7041 * LUCENE-5302: Make StemmerOverrideMap's methods public (Alan Woodward)
 7042 
 7043 * LUCENE-5296: Add DirectDocValuesFormat, which holds all doc values
 7044   in heap as uncompressed java native arrays.  (Mike McCandless)
 7045 
 7046 * LUCENE-5189: Add IndexWriter.updateNumericDocValues, to update
 7047   numeric DocValues fields of documents, without re-indexing them.
 7048   (Shai Erera, Mike McCandless, Robert Muir)
 7049   
 7050 * LUCENE-5298: Add SumValueSourceFacetRequest for aggregating facets by
 7051   a ValueSource, such as a NumericDocValuesField or an expression.
 7052   (Shai Erera)
 7053 
 7054 * LUCENE-5323: Add .sizeInBytes method to all suggesters (Lookup).
 7055   (Areek Zillur via Mike McCandless)
 7056   
 7057 * LUCENE-5312: Add BlockJoinSorter, a new Sorter implementation that makes sure
 7058   to never split up blocks of documents indexed with IndexWriter.addDocuments.
 7059   (Adrien Grand)
 7060   
 7061 * LUCENE-5297: Allow to range-facet on any ValueSource, not just
 7062   NumericDocValues fields. (Shai Erera)
 7063   
 7064 Bug Fixes
 7065 
 7066 * LUCENE-5272: OpenBitSet.ensureCapacity did not modify numBits, causing 
 7067   false assertion errors in fastSet. (Shai Erera)
 7068 
 7069 * LUCENE-5303: OrdinalsCache did not use coreCacheKey, resulting in
 7070   over caching across multiple threads. (Mike McCandless, Shai Erera)
 7071 
 7072 * LUCENE-5307: Fix topScorer inconsistency in handling QueryWrapperFilter
 7073   inside ConstantScoreQuery, which now rewrites to a query removing the
 7074   obsolete QueryWrapperFilter.  (Adrien Grand, Uwe Schindler)
 7075 
 7076 * LUCENE-5330: IndexWriter didn't process all internal events on 
 7077   #getReader(), #close() and #rollback() which causes files to be
 7078   deleted at a later point in time. This could cause short-term disk
 7079   pollution or OOM if in-memory directories are used. (Simon Willnauer)
 7080 
 7081 * LUCENE-5342: Fixed bulk-merge issue in CompressingStoredFieldsFormat which
 7082   created corrupted segments when mixing chunk sizes.
 7083   Lucene41StoredFieldsFormat is not impacted. (Adrien Grand, Robert Muir)
 7084 
 7085 API Changes
 7086 
 7087 * LUCENE-5222: Add SortField.needsScores(). Previously it was not possible
 7088   for a custom Sort that makes use of the relevance score to work correctly
 7089   with IndexSearcher when an ExecutorService is specified.
 7090   (Ryan Ernst, Mike McCandless, Robert Muir)
 7091 
 7092 * LUCENE-5275: Change AttributeSource.toString() to display the current
 7093   state of attributes. (Robert Muir)
 7094 
 7095 * LUCENE-5277: Modify FixedBitSet copy constructor to take an additional
 7096   numBits parameter to allow growing/shrinking the copied bitset. You can
 7097   use FixedBitSet.clone() if you only need to clone the bitset. (Shai Erera)
 7098 
 7099 * LUCENE-5260: Use TermFreqPayloadIterator for all suggesters; those
 7100   suggesters that can't support payloads will throw an exception if
 7101   hasPayloads() is true.  (Areek Zillur via Mike McCandless)
 7102 
 7103 * LUCENE-5280: Rename TermFreqPayloadIterator -> InputIterator, along
 7104   with associated suggest/spell classes.  (Areek Zillur via Mike
 7105   McCandless)
 7106 
 7107 * LUCENE-5157: Rename OrdinalMap methods to clarify API and internal structure.
 7108   (Boaz Leskes via Adrien Grand)
 7109 
 7110 * LUCENE-5313: Move preservePositionIncrements from setter to ctor in
 7111   Analyzing/FuzzySuggester.  (Areek Zillur via Mike McCandless)
 7112 
 7113 * LUCENE-5321: Remove Facet42DocValuesFormat. Use DirectDocValuesFormat if you
 7114   want to load the category list into memory. (Shai Erera, Mike McCandless)
 7115 
 7116 * LUCENE-5324: AnalyzerWrapper.getPositionIncrementGap and getOffsetGap can now
 7117   be overridden. (Adrien Grand)
 7118 
 7119 Optimizations
 7120 
 7121 * LUCENE-5225: The ToParentBlockJoinQuery only keeps tracks of the the child
 7122   doc ids and child scores if the ToParentBlockJoinCollector is used.
 7123   (Martijn van Groningen)
 7124 
 7125 * LUCENE-5236: EliasFanoDocIdSet now has an index and uses broadword bit
 7126   selection to speed-up advance(). (Paul Elschot via Adrien Grand)
 7127 
 7128 * LUCENE-5266: Improved number of read calls and branches in DirectPackedReader. (Ryan Ernst)
 7129 
 7130 * LUCENE-5300: Optimized SORTED_SET storage for fields which are single-valued.
 7131   (Adrien Grand)
 7132 
 7133 Documentation
 7134 
 7135 * LUCENE-5211: Better javadocs and error checking of 'format' option in 
 7136   StopFilterFactory, as well as comments in all snowball formatted files
 7137   about specifying format option.  (hossman)
 7138 
 7139 Changes in backwards compatibility policy
 7140 
 7141 * LUCENE-5235: Sub classes of Tokenizer have to call super.reset()
 7142   when implementing reset(). Otherwise the consumer will get an
 7143   IllegalStateException because the Reader is not correctly assigned.
 7144   It is important to never change the "input" field on Tokenizer
 7145   without using setReader(). The "input" field must not be used
 7146   outside reset(), incrementToken(), or end() - especially not in
 7147   the constructor.  (Uwe Schindler, Robert Muir)
 7148 
 7149 * LUCENE-5204: Directory doesn't have default implementations for
 7150   LockFactory-related methods, which have been moved to BaseDirectory. If you
 7151   had a custom Directory implementation that extended Directory, you need to
 7152   extend BaseDirectory instead. (Adrien Grand)
 7153 
 7154 Build
 7155 
 7156 * LUCENE-5283: Fail the build if ant test didn't execute any tests 
 7157   (everything filtered out). (Dawid Weiss, Uwe Schindler)
 7158 
 7159 * LUCENE-5249, LUCENE-5257: All Lucene/Solr modules should use the same
 7160   dependency versions. (Steve Rowe)
 7161   
 7162 * LUCENE-5273: Binary artifacts in Lucene and Solr convenience binary
 7163   distributions accompanying a release, including on Maven Central,
 7164   should be identical across all distributions. (Steve Rowe, Uwe Schindler,
 7165   Shalin Shekhar Mangar)
 7166 
 7167 * LUCENE-4753: Run forbidden-apis Ant task per module. This allows more
 7168   improvements and prevents OOMs after the number of class files
 7169   raised recently.  (Uwe Schindler)
 7170 
 7171 Tests
 7172 
 7173 * LUCENE-5278: Fix MockTokenizer to work better with more regular expression
 7174   patterns. Previously it could only behave like CharTokenizer (where a character
 7175   is either a "word" character or not), but now it gives a general longest-match
 7176   behavior.  (Nik Everett via Robert Muir)
 7177 
 7178 ======================= Lucene 4.5.1 =======================
 7179 
 7180 Bug Fixes
 7181 
 7182 * LUCENE-4998: Fixed a few places to pass IOContext.READONCE instead
 7183   of IOContext.READ (Shikhar Bhushan via Mike McCandless)
 7184 
 7185 * LUCENE-5242: DirectoryTaxonomyWriter.replaceTaxonomy did not fully reset
 7186   its state, which could result in exceptions being thrown, as well as
 7187   incorrect ordinals returned from getParent. (Shai Erera)
 7188 
 7189 * LUCENE-5254: Fixed bounded memory leak, where objects like live
 7190   docs bitset were not freed from an starting reader after reopening
 7191   to a new reader and closing the original one.  (Shai Erera, Mike
 7192   McCandless)
 7193 
 7194 * LUCENE-5262: Fixed file handle leaks when multiple attempts to open an
 7195   NRT reader hit exceptions. (Shai Erera)
 7196 
 7197 * LUCENE-5263: Transient IOExceptions, e.g. due to disk full or file
 7198   descriptor exhaustion, hit at unlucky times inside IndexWriter could
 7199   lead to silently losing deletions. (Shai Erera, Mike McCandless)
 7200 
 7201 * LUCENE-5264: CommonTermsQuery ignored minMustMatch if only high-frequent
 7202   terms were present in the query and the high-frequent operator was set
 7203   to SHOULD. (Simon Willnauer)
 7204 
 7205 * LUCENE-5269: Fix bug in NGramTokenFilter where it would sometimes count
 7206   unicode characters incorrectly. (Mike McCandless, Robert Muir)
 7207 
 7208 * LUCENE-5289: IndexWriter.hasUncommittedChanges was returning false
 7209   when there were buffered delete-by-Term.  (Shalin Shekhar Mangar,
 7210   Mike McCandless)
 7211 
 7212 ======================= Lucene 4.5.0 =======================
 7213 
 7214 New features
 7215 
 7216 * LUCENE-5084: Added new Elias-Fano encoder, decoder and DocIdSet
 7217   implementations. (Paul Elschot via Adrien Grand)
 7218 
 7219 * LUCENE-5081: Added WAH8DocIdSet, an in-memory doc id set implementation based
 7220   on word-aligned hybrid encoding. (Adrien Grand)
 7221 
 7222 * LUCENE-5098: New broadword utility methods in oal.util.BroadWord.
 7223   (Paul Elschot via Adrien Grand, Dawid Weiss)
 7224 
 7225 * LUCENE-5030: FuzzySuggester now supports optional unicodeAware
 7226   (default is false).  If true then edits are measured in Unicode code
 7227   points instead of UTF8 bytes.  (Artem Lukanin via Mike McCandless)
 7228 
 7229 * LUCENE-5118: SpatialStrategy.makeDistanceValueSource() now has an optional
 7230   multiplier for scaling degrees to another unit. (David Smiley)
 7231 
 7232 * LUCENE-5091: SpanNotQuery can now be configured with pre and post slop to act
 7233   as a hypothetical SpanNotNearQuery. (Tim Allison via David Smiley)
 7234 
 7235 * LUCENE-4985: FacetsAccumulator.create() is now able to create a 
 7236   MultiFacetsAccumulator over a mixed set of facet requests. MultiFacetsAccumulator
 7237   allows wrapping multiple FacetsAccumulators, allowing to easily mix
 7238   existing and custom ones. TaxonomyFacetsAccumulator supports any
 7239   FacetRequest which implements createFacetsAggregator and was indexed
 7240   using the taxonomy index. (Shai Erera)
 7241 
 7242 * LUCENE-5153: AnalyzerWrapper.wrapReader allows wrapping the Reader given to 
 7243   inputReader. (Shai Erera)
 7244 
 7245 * LUCENE-5155: FacetRequest.getValueOf and .getFacetArraysSource replaced by
 7246   FacetsAggregator.createOrdinalValueResolver. This gives better options for
 7247   resolving an ordinal's value by FacetAggregators. (Shai Erera)
 7248 
 7249 * LUCENE-5165: Add SuggestStopFilter, to be used with analyzing
 7250   suggesters, so that a stop word at the very end of the lookup query,
 7251   and without any trailing token characters, will be preserved.  This
 7252   enables query "a" to suggest apple; see 
 7253   http://blog.mikemccandless.com/2013/08/suggeststopfilter-carefully-removes.html
 7254   for details.
 7255 
 7256 * LUCENE-5178: Added support for missing values to DocValues fields.
 7257   AtomicReader.getDocsWithField returns a Bits of documents with a value,
 7258   and FieldCache.getDocsWithField forwards to that for DocValues fields. Things like 
 7259   SortField.setMissingValue, FunctionValues.exists, and FieldValueFilter now
 7260   work with DocValues fields.  (Robert Muir)
 7261 
 7262 * LUCENE-5124: Lucene 4.5 has a new Lucene45Codec with Lucene45DocValues,
 7263   supporting missing values and with most datastructures residing off-heap.
 7264   Added "Memory" docvalues format that works entirely in heap, and "Disk"
 7265   loads no datastructures into RAM. Both of these also support missing values.
 7266   Added DiskNormsFormat (in case you want norms entirely on disk).  (Robert Muir)
 7267 
 7268 * LUCENE-2750: Added PForDeltaDocIdSet, an in-memory doc id set implementation
 7269   based on the PFOR encoding. (Adrien Grand)
 7270 
 7271 * LUCENE-5186: Added CachingWrapperFilter.getFilter in order to be able to get
 7272   the wrapped filter. (Trejkaz via Adrien Grand)
 7273 
 7274 * LUCENE-5197: Added SegmentReader.ramBytesUsed to return approximate heap RAM
 7275   used by index datastructures. (Areek Zillur via Robert Muir)
 7276 
 7277 Bug Fixes
 7278 
 7279 * LUCENE-5116: IndexWriter.addIndexes(IndexReader...) should drop empty (or all
 7280   deleted) segments. (Robert Muir, Shai Erera)
 7281 
 7282 * LUCENE-5132: Spatial RecursivePrefixTree Contains predicate will throw an NPE
 7283   when there's no indexed data and maybe in other circumstances too. (David Smiley)
 7284 
 7285 * LUCENE-5146: AnalyzingSuggester sort comparator read part of the input key as the
 7286   weight that caused the sorter to never sort by weight first since the weight is only
 7287   considered if the input is equal causing the malformed weight to be identical as well.
 7288   (Simon Willnauer)
 7289 
 7290 * LUCENE-5151: Associations FacetsAggregators could enter an infinite loop when
 7291   some result documents were missing category associations. (Shai Erera)
 7292 
 7293 * LUCENE-5152: Fix MemoryPostingsFormat to not modify borrowed BytesRef from FSTEnum
 7294   seek/lookup which can cause side effects if done on a cached FST root arc.
 7295   (Simon Willnauer)
 7296 
 7297 * LUCENE-5160: Handle the case where reading from a file or FileChannel returns -1,
 7298   which could happen in rare cases where something happens to the file between the
 7299   time we start the read loop (where we check the length) and when we actually do
 7300   the read. (gsingers, yonik, Robert Muir, Uwe Schindler)
 7301 
 7302 * LUCENE-5166: PostingsHighlighter would throw IOOBE if a term spanned the maxLength
 7303   boundary, made it into the top-N and went to the formatter.
 7304   (Manuel Amoabeng, Michael McCandless, Robert Muir)
 7305 
 7306 * LUCENE-4583: Indexing core no longer enforces a limit on maximum
 7307   length binary doc values fields, but individual codecs (including
 7308   the default one) have their own limits (David Smiley, Robert Muir,
 7309   Mike McCandless)
 7310 
 7311 * LUCENE-3849: TokenStreams now set the position increment in end(),
 7312   so we can handle trailing holes.  If you have a custom TokenStream
 7313   implementing end() then be sure it calls super.end().  (Robert Muir,
 7314   Mike McCandless)
 7315 
 7316 * LUCENE-5192: IndexWriter could allow adding same field name with different
 7317   DocValueTypes under some circumstances. (Shai Erera)
 7318 
 7319 * LUCENE-5191: SimpleHTMLEncoder in Highlighter module broke Unicode
 7320   outside BMP because it encoded UTF-16 chars instead of codepoints.
 7321   The escaping of codepoints > 127 was removed (not needed for valid HTML)
 7322   and missing escaping for ' and / was added.  (Uwe Schindler)
 7323 
 7324 * LUCENE-5201: Fixed compression bug in LZ4.compressHC when the input is highly
 7325   compressible and the start offset of the array to compress is > 0.
 7326   (Adrien Grand)
 7327 
 7328 * LUCENE-5221: SimilarityBase did not write norms the same way as DefaultSimilarity
 7329   if discountOverlaps == false and index-time boosts are present for the field.
 7330   (Yubin Kim via Robert Muir)
 7331 
 7332 * LUCENE-5223: Fixed IndexUpgrader command line parsing: -verbose is not required
 7333   and -dir-impl option now works correctly.  (hossman)
 7334 
 7335 * LUCENE-5245: Fix MultiTermQuery's constant score rewrites to always
 7336   return a ConstantScoreQuery to make scoring consistent. Previously it
 7337   returned an empty unwrapped BooleanQuery, if no terms were available,
 7338   which has a different query norm.  (Nik Everett, Uwe Schindler)
 7339 
 7340 * LUCENE-5218: In some cases, trying to retrieve or merge a 0-length
 7341   binary doc value would hit an ArrayIndexOutOfBoundsException.
 7342   (Littlestar via Mike McCandless)
 7343   
 7344 API Changes
 7345 
 7346 * LUCENE-5094: Add ramBytesUsed() to MultiDocValues.OrdinalMap.
 7347   (Robert Muir)
 7348 
 7349 * LUCENE-5114: Remove unused boolean useCache parameter from
 7350   TermsEnum.seekCeil and .seekExact (Mike McCandless)
 7351 
 7352 * LUCENE-5128: IndexSearcher.searchAfter throws IllegalArgumentException if 
 7353   searchAfter exceeds the number of documents in the reader. 
 7354   (Crocket via Shai Erera)
 7355 
 7356 * LUCENE-5129: CategoryAssociationsContainer no longer supports null
 7357   association values for categories. If you want to index categories without
 7358   associations, you should add them using FacetFields. (Shai Erera)
 7359 
 7360 * LUCENE-4876: IndexWriter no longer clones the given IndexWriterConfig. If you
 7361   need to use the same config more than once, e.g. when sharing between multiple 
 7362   writers, make sure to clone it before passing to each writer.
 7363   (Shai Erera, Mike McCandless)
 7364 
 7365 * LUCENE-5144: StandardFacetsAccumulator renamed to OldFacetsAccumulator, and all
 7366   associated classes were moved under o.a.l.facet.old. The intention to remove it
 7367   one day, when the features it covers (complements, partitions, sampling) will be
 7368   migrated to the new FacetsAggregator and FacetsAccumulator API. Also,
 7369   FacetRequest.createAggregator was replaced by OldFacetsAccumulator.createAggregator.
 7370   (Shai Erera)
 7371 
 7372 * LUCENE-5149: CommonTermsQuery now allows to set the minimum number of terms that 
 7373   should match for its high and low frequent sub-queries. Previously this was only
 7374   supported on the low frequent terms query. (Simon Willnauer)  
 7375 
 7376 * LUCENE-5156: CompressingTermVectors TermsEnum no longer supports ord().
 7377   (Robert Muir)
 7378 
 7379 * LUCENE-5161, LUCENE-5164: Fix default chunk sizes in FSDirectory to not be
 7380   unnecessarily large (now 8192 bytes); also use chunking when writing to index
 7381   files. FSDirectory#setReadChunkSize() is now deprecated and will be removed
 7382   in Lucene 5.0.  (Uwe Schindler, Robert Muir, gsingers)
 7383 
 7384 * LUCENE-5170: Analyzer.ReuseStrategy instances are now stateless and can
 7385   be reused in other Analyzer instances, which was not possible before.
 7386   Lucene ships now with stateless singletons for per field and global reuse.
 7387   Legacy code can still instantiate the deprecated implementation classes,
 7388   but new code should use the constants. Implementors of custom strategies
 7389   have to take care of new method signatures. AnalyzerWrapper can now be
 7390   configured to use a custom strategy, too, ideally the one from the wrapped
 7391   Analyzer. Analyzer adds a getter to retrieve the strategy for this use-case.
 7392   (Uwe Schindler, Robert Muir, Shay Banon)
 7393 
 7394 * LUCENE-5173: Lucene never writes segments with 0 documents anymore.
 7395   (Shai Erera, Uwe Schindler, Robert Muir)
 7396 
 7397 * LUCENE-5178: SortedDocValues always returns -1 ord when a document is missing
 7398   a value for the field. Previously it only did this if the SortedDocValues
 7399   was produced by uninversion on the FieldCache.  (Robert Muir)
 7400 
 7401 * LUCENE-5183: remove BinaryDocValues.MISSING. In order to determine a document
 7402   is missing a field, use getDocsWithField instead.  (Robert Muir)
 7403 
 7404 Changes in Runtime Behavior
 7405 
 7406 * LUCENE-5178: DocValues codec consumer APIs (iterables) return null values 
 7407   when the document has no value for the field. (Robert Muir)
 7408 
 7409 * LUCENE-5200: The HighFreqTerms command-line tool returns the true top-N
 7410   by totalTermFreq when using the -t option, it uses the term statistics (faster)
 7411   and now always shows totalTermFreq in the output.  (Robert Muir)
 7412 
 7413 Optimizations
 7414 
 7415 * LUCENE-5088: Added TermFilter to filter docs by a specific term.
 7416   (Martijn van Groningen)
 7417 
 7418 * LUCENE-5119: DiskDV keeps the document-to-ordinal mapping on disk for 
 7419   SortedDocValues.  (Robert Muir)
 7420 
 7421 * LUCENE-5145: New AppendingPackedLongBuffer, a new variant of the former
 7422   AppendingLongBuffer which assumes values are 0-based.
 7423   (Boaz Leskes via Adrien Grand)
 7424 
 7425 * LUCENE-5145: All Appending*Buffer now support bulk get.
 7426   (Boaz Leskes via Adrien Grand)
 7427 
 7428 * LUCENE-5140: Fixed a performance regression of span queries caused by
 7429   LUCENE-4946. (Alan Woodward, Adrien Grand)
 7430 
 7431 * LUCENE-5150: Make WAH8DocIdSet able to inverse its encoding in order to
 7432   compress dense sets efficiently as well. (Adrien Grand)
 7433 
 7434 * LUCENE-5159: Prefix-code the sorted/sortedset value dictionaries in DiskDV.
 7435   (Robert Muir)
 7436 
 7437 * LUCENE-5170: Fixed several wrapper analyzers to inherit the reuse strategy
 7438   of the wrapped Analyzer.  (Uwe Schindler, Robert Muir, Shay Banon)
 7439 
 7440 * LUCENE-5006: Simplified DocumentsWriter and DocumentsWriterPerThread
 7441   synchronization and concurrent interaction with IndexWriter. DWPT is now
 7442   only setup once and has no reset logic. All segment publishing and state
 7443   transition from DWPT into IndexWriter is now done via an Event-Queue
 7444   processed from within the IndexWriter in order to prevent situations
 7445   where DWPT or DW calling int IW causing deadlocks. (Simon Willnauer)
 7446 
 7447 * LUCENE-5182: Terminate phrase searches early if max phrase window is 
 7448   exceeded in FastVectorHighlighter to prevent very long running phrase
 7449   extraction if phrase terms are high frequent. (Simon Willnauer)
 7450 
 7451 * LUCENE-5188: CompressingStoredFieldsFormat now slices chunks containing big
 7452   documents into fixed-size blocks so that requesting a single field does not
 7453   necessarily force to decompress the whole chunk. (Adrien Grand)
 7454 
 7455 * LUCENE-5101: CachingWrapper makes it easier to plug-in a custom cacheable
 7456   DocIdSet implementation and uses WAH8DocIdSet by default, which should be
 7457   more memory efficient than FixedBitSet on average as well as faster on small
 7458   sets. (Robert Muir)
 7459   
 7460 Documentation
 7461 
 7462 * LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
 7463   package's documentation and classes javadocs. (Shai Erera)
 7464 
 7465 * LUCENE-5206: Clarify FuzzyQuery's unexpected behavior on short
 7466   terms. (Tim Allison via Mike McCandless)
 7467 
 7468 Changes in backwards compatibility policy
 7469 
 7470 * LUCENE-5141: CheckIndex.fixIndex(Status,Codec) is now
 7471   CheckIndex.fixIndex(Status). If you used to pass a codec to this method, just
 7472   remove it from the arguments. (Adrien Grand)
 7473 
 7474 * LUCENE-5089, SOLR-5126: Update to Morfologik 1.7.1. MorfologikAnalyzer and MorfologikFilter 
 7475   no longer support multiple "dictionaries" as there is only one dictionary available.
 7476   (Dawid Weiss)
 7477 
 7478 * LUCENE-5170: Changed method signatures of Analyzer.ReuseStrategy to take
 7479   Analyzer. Closeable interface was removed because the class was changed to
 7480   be stateless.  (Uwe Schindler, Robert Muir, Shay Banon)
 7481 
 7482 * LUCENE-5187: SlowCompositeReaderWrapper constructor is now private,
 7483   SlowCompositeReaderWrapper.wrap should be used instead. (Adrien Grand)
 7484 
 7485 * LUCENE-5101: CachingWrapperFilter doesn't always return FixedBitSet instances
 7486   anymore. Users of the join module can use
 7487   oal.search.join.FixedBitSetCachingWrapperFilter instead. (Adrien Grand)
 7488 
 7489 Build
 7490 
 7491 * SOLR-5159: Manifest includes non-parsed maven variables. 
 7492   (Artem Karpenko via Steve Rowe)
 7493 
 7494 * LUCENE-5193: Add jar-src as top-level target to generate all Lucene and Solr
 7495   *-src.jar. (Steve Rowe, Shai Erera)
 7496 
 7497 ======================= Lucene 4.4.0 =======================
 7498 
 7499 Changes in backwards compatibility policy
 7500 
 7501 * LUCENE-5085: MorfologikFilter will no longer stem words marked as keywords
 7502   (Dawid Weiss, Grzegorz Sobczyk)
 7503 
 7504 * LUCENE-4955: NGramTokenFilter now emits all n-grams for the same token at the
 7505   same position and preserves the position length and the offsets of the
 7506   original token. (Simon Willnauer, Adrien Grand)
 7507 
 7508 * LUCENE-4955: NGramTokenizer now emits n-grams in a different order
 7509   (a, ab, b, bc, c) instead of (a, b, c, ab, bc) and doesn't trim trailing
 7510   whitespaces. (Adrien Grand)
 7511 
 7512 * LUCENE-5042: The n-gram and edge n-gram tokenizers and filters now correctly
 7513   handle supplementary characters, and the tokenizers have the ability to
 7514   pre-tokenize the input stream similarly to CharTokenizer. (Adrien Grand)
 7515 
 7516 * LUCENE-4967: NRTManager is replaced by
 7517   ControlledRealTimeReopenThread, for controlling which requests must
 7518   see which indexing changes, so that it can work with any
 7519   ReferenceManager (Mike McCandless)
 7520 
 7521 * LUCENE-4973: SnapshotDeletionPolicy no longer requires a unique
 7522   String id (Mike McCandless, Shai Erera)
 7523 
 7524 * LUCENE-4946: The internal sorting API (SorterTemplate, now Sorter) has been
 7525   completely refactored to allow for a better implementation of TimSort.
 7526   (Adrien Grand, Uwe Schindler, Dawid Weiss)
 7527 
 7528 * LUCENE-4963: Some TokenFilter options that generate broken TokenStreams have
 7529   been deprecated: updateOffsets=true on TrimFilter and
 7530   enablePositionIncrements=false on all classes that inherit from
 7531   FilteringTokenFilter: JapanesePartOfSpeechStopFilter, KeepWordFilter,
 7532   LengthFilter, StopFilter and TypeTokenFilter. (Adrien Grand)
 7533 
 7534 * LUCENE-4963: In order not to take position increments into account in
 7535   suggesters, you now need to call setPreservePositionIncrements(false) instead
 7536   of configuring the token filters to not increment positions. (Adrien Grand)
 7537 
 7538 * LUCENE-3907: EdgeNGramTokenizer now supports maxGramSize > 1024, doesn't trim
 7539   the input, sets position increment = 1 for all tokens and doesn't support
 7540   backward grams anymore. (Adrien Grand)
 7541 
 7542 * LUCENE-3907: EdgeNGramTokenFilter does not support backward grams and does
 7543   not update offsets anymore. (Adrien Grand)
 7544 
 7545 * LUCENE-4981: PositionFilter is now deprecated as it can corrupt token stream
 7546   graphs. Since it main use-case was to make query parsers generate boolean
 7547   queries instead of phrase queries, it is now advised to use
 7548   QueryParser.setAutoGeneratePhraseQueries(false) (for simple cases) or to
 7549   override QueryParser.newFieldQuery. (Adrien Grand, Steve Rowe)
 7550 
 7551 * LUCENE-5018: CompoundWordTokenFilterBase and its children
 7552   DictionaryCompoundWordTokenFilter and HyphenationCompoundWordTokenFilter don't
 7553   update offsets anymore. (Adrien Grand)
 7554 
 7555 * LUCENE-5015: SamplingAccumulator no longer corrects the counts of the sampled 
 7556   categories. You should set TakmiSampleFixer on SamplingParams if required (but 
 7557   notice that this means slower search). (Rob Audenaerde, Gilad Barkai, Shai Erera)
 7558 
 7559 * LUCENE-4933: Replace ExactSimScorer/SloppySimScorer with just SimScorer. Previously
 7560   there were 2 implementations as a performance hack to support tableization of
 7561   sqrt(), but this caching is removed, as sqrt is implemented in hardware with modern 
 7562   jvms and it's faster not to cache.  (Robert Muir)
 7563 
 7564 * LUCENE-5038: MergePolicy now has a default implementation for useCompoundFile based
 7565   on segment size and noCFSRatio. The default implementation was pulled up from
 7566   TieredMergePolicy. (Simon Willnauer)
 7567 
 7568 * LUCENE-5063: FieldCache.get(Bytes|Shorts), SortField.Type.(BYTE|SHORT) and
 7569   FieldCache.DEFAULT_(BYTE|SHORT|INT|LONG|FLOAT|DOUBLE)_PARSER are now
 7570   deprecated. These methods/types assume that data is stored as strings although
 7571   Lucene has much better support for numeric data through (Int|Long)Field,
 7572   NumericRangeQuery and FieldCache.get(Int|Long)s. (Adrien Grand)
 7573 
 7574 * LUCENE-5078: TfIDFSimilarity lets you encode the norm value as any arbitrary long.
 7575   As a result, encode/decodeNormValue were made abstract with their signatures changed.
 7576   The default implementation was moved to DefaultSimilarity, which encodes the norm as
 7577   a single-byte value. (Shai Erera)
 7578 
 7579 Bug Fixes
 7580 
 7581 * LUCENE-4890: QueryTreeBuilder.getBuilder() only finds interfaces on the 
 7582   most derived class. (Adriano Crestani)
 7583 
 7584 * LUCENE-4997: Internal test framework's tests are sensitive to previous 
 7585   test failures and tests.failfast. (Dawid Weiss, Shai Erera)
 7586 
 7587 * LUCENE-4955: NGramTokenizer now supports inputs larger than 1024 chars.
 7588   (Adrien Grand)
 7589 
 7590 * LUCENE-4959: Fix incorrect return value in
 7591   SimpleNaiveBayesClassifier.assignClass. (Alexey Kutin via Adrien Grand)
 7592 
 7593 * LUCENE-4972: DirectoryTaxonomyWriter created empty commits even if no changes
 7594   were made. (Shai Erera, Michael McCandless)
 7595   
 7596 * LUCENE-949: AnalyzingQueryParser can't work with leading wildcards.
 7597   (Tim Allison, Robert Muir, Steve Rowe)
 7598 
 7599 * LUCENE-4980: Fix issues preventing mixing of RangeFacetRequest and
 7600   non-RangeFacetRequest when using DrillSideways.  (Mike McCandless,
 7601   Shai Erera)
 7602 
 7603 * LUCENE-4996: Ensure DocInverterPerField always includes field name
 7604   in exception messages.  (Markus Jelsma via Robert Muir)
 7605 
 7606 * LUCENE-4992: Fix constructor of CustomScoreQuery to take FunctionQuery
 7607   for scoringQueries. Instead use QueryValueSource to safely wrap arbitrary 
 7608   queries and use them with CustomScoreQuery.  (John Wang, Robert Muir)
 7609 
 7610 * LUCENE-5016: SamplingAccumulator returned inconsistent label if asked to
 7611   aggregate a non-existing category. Also fixed a bug in RangeAccumulator if
 7612   some readers did not have the requested numeric DV field.
 7613   (Rob Audenaerde, Shai Erera)
 7614 
 7615 * LUCENE-5028: Remove pointless and confusing doShare option in FST's
 7616   PositiveIntOutputs (Han Jiang via Mike McCandless)
 7617 
 7618 * LUCENE-5032: Fix IndexOutOfBoundsExc in PostingsHighlighter when
 7619   multi-valued fields exceed maxLength (Tomás Fernández Löbbe
 7620   via Mike McCandless)
 7621 
 7622 * LUCENE-4933: SweetSpotSimilarity didn't apply its tf function to some
 7623   queries (SloppyPhraseQuery, SpanQueries).  (Robert Muir)
 7624 
 7625 * LUCENE-5033: SlowFuzzyQuery was accepting too many terms (documents) when
 7626   provided minSimilarity is an int > 1 (Tim Allison via Mike McCandless)
 7627 
 7628 * LUCENE-5045: DrillSideways.search did not work on an empty index. (Shai Erera)
 7629 
 7630 * LUCENE-4995: CompressingStoredFieldsReader now only reuses an internal buffer
 7631   when there is no more than 32kb to decompress. This prevents from running
 7632   into out-of-memory errors when working with large stored fields.
 7633   (Adrien Grand)
 7634 
 7635 * LUCENE-5062: If the spatial data for a document was comprised of multiple
 7636   overlapping or adjacent parts then a CONTAINS predicate query might not match
 7637   when the sum of those shapes contain the query shape but none do individually.
 7638   A flag was added to use the original faster algorithm. (David Smiley)
 7639 
 7640 * LUCENE-4971: Fixed NPE in AnalyzingSuggester when there are too many
 7641   graph expansions.  (Alexey Kudinov via Mike McCandless)
 7642 
 7643 * LUCENE-5080: Combined setMaxMergeCount and setMaxThreadCount into one
 7644   setter in ConcurrentMergePolicy: setMaxMergesAndThreads.  Previously these
 7645   setters would not work unless you invoked them very carefully.
 7646   (Robert Muir, Shai Erera)
 7647   
 7648 * LUCENE-5068: QueryParserUtil.escape() does not escape forward slash.
 7649   (Matias Holte via Steve Rowe)
 7650 
 7651 * LUCENE-5103: A join on A single-valued field with deleted docs scored too few
 7652   docs. (David Smiley)
 7653 
 7654 * LUCENE-5090: Detect mismatched readers passed to
 7655   SortedSetDocValuesReaderState and SortedSetDocValuesAccumulator.
 7656   (Robert Muir, Mike McCandless)
 7657 
 7658 * LUCENE-5120: AnalyzingSuggester modified its FST's cached root arc if payloads
 7659   are used and the entire output resided on the root arc on the first access. This
 7660   caused subsequent suggest calls to fail. (Simon Willnauer)
 7661 
 7662 Optimizations
 7663 
 7664 * LUCENE-4936: Improve numeric doc values compression in case all values share
 7665   a common divisor. In particular, this improves the compression ratio of dates
 7666   without time when they are encoded as milliseconds since Epoch. Also support
 7667   TABLE compressed numerics in the Disk codec.  (Robert Muir, Adrien Grand)
 7668 
 7669 * LUCENE-4951: DrillSideways uses the new Scorer.cost() method to make
 7670   better decisions about which scorer to use internally.  (Mike McCandless)
 7671   
 7672 * LUCENE-4976: PersistentSnapshotDeletionPolicy writes its state to a
 7673   single snapshots_N file, and no longer requires closing (Mike
 7674   McCandless, Shai Erera)
 7675 
 7676 * LUCENE-5035: Compress addresses in FieldCacheImpl.SortedDocValuesImpl more
 7677   efficiently. (Adrien Grand, Robert Muir)
 7678 
 7679 * LUCENE-4941: Sort "from" terms only once when using JoinUtil.
 7680   (Martijn van Groningen)
 7681 
 7682 * LUCENE-5050: Close the stored fields and term vectors index files as soon as
 7683   the index has been loaded into memory to save file descriptors. (Adrien Grand)
 7684 
 7685 * LUCENE-5086: RamUsageEstimator now uses official Java 7 API or a proprietary
 7686   Oracle Java 6 API to get Hotspot MX bean, preventing AWT classes to be
 7687   loaded on MacOSX.  (Shay Banon, Dawid Weiss, Uwe Schindler)
 7688 
 7689 New Features
 7690 
 7691 * LUCENE-5085: MorfologikFilter will no longer stem words marked as keywords
 7692   (Dawid Weiss, Grzegorz Sobczyk)
 7693 
 7694 * LUCENE-5064: Added PagedMutable (internal), a paged extension of
 7695   PackedInts.Mutable which allows for storing more than 2B values. (Adrien Grand)
 7696 
 7697 * LUCENE-4766: Added a PatternCaptureGroupTokenFilter that uses Java regexes to 
 7698   emit multiple tokens one for each capture group in one or more patterns.
 7699   (Simon Willnauer, Clinton Gormley)
 7700 
 7701 * LUCENE-4952: Expose control (protected method) in DrillSideways to
 7702   force all sub-scorers to be on the same document being collected.
 7703   This is necessary when using collectors like
 7704   ToParentBlockJoinCollector with DrillSideways.  (Mike McCandless)
 7705 
 7706 * SOLR-4761: Add SimpleMergedSegmentWarmer, which just initializes terms,
 7707   norms, docvalues, and so on. (Mark Miller, Mike McCandless, Robert Muir)
 7708 
 7709 * LUCENE-4964: Allow arbitrary Query for per-dimension drill-down to
 7710   DrillDownQuery and DrillSideways, to support future dynamic faceting
 7711   methods (Mike McCandless)
 7712 
 7713 * LUCENE-4966: Add CachingWrapperFilter.sizeInBytes() (Mike McCandless)
 7714 
 7715 * LUCENE-4965: Add dynamic (no taxonomy index used) numeric range
 7716   faceting to Lucene's facet module (Mike McCandless, Shai Erera)
 7717 
 7718 * LUCENE-4979: LiveFieldFields can work with any ReferenceManager, not
 7719   just ReferenceManager<IndexSearcher> (Mike McCandless).
 7720 
 7721 * LUCENE-4975: Added a new Replicator module which can replicate index 
 7722   revisions between server and client. (Shai Erera, Mike McCandless)
 7723 
 7724 * LUCENE-5022: Added FacetResult.mergeHierarchies to merge multiple
 7725   FacetResult of the same dimension into a single one with the reconstructed
 7726   hierarchy. (Shai Erera)
 7727 
 7728 * LUCENE-5026: Added PagedGrowableWriter, a new internal packed-ints structure
 7729   that grows the number of bits per value on demand, can store more than 2B
 7730   values and supports random write and read access. (Adrien Grand)
 7731 
 7732 * LUCENE-5025: FST's Builder can now handle more than 2.1 billion
 7733   "tail nodes" while building a minimal FST.  (Aaron Binns, Adrien
 7734   Grand, Mike McCandless)
 7735 
 7736 * LUCENE-5063: FieldCache.DEFAULT.get(Ints|Longs) now uses bit-packing to save
 7737   memory. (Adrien Grand)
 7738 
 7739 * LUCENE-5079: IndexWriter.hasUncommittedChanges() returns true if there are
 7740   changes that have not been committed. (yonik, Mike McCandless, Uwe Schindler)
 7741 
 7742 * SOLR-4565: Extend NorwegianLightStemFilter and NorwegianMinimalStemFilter 
 7743   to handle "nynorsk" (Erlend Garåsen, janhoy via Robert Muir)
 7744 
 7745 * LUCENE-5087: Add getMultiValuedSeparator to PostingsHighlighter, for cases
 7746   where you want a different logical separator between field values. This can
 7747   be set to e.g. U+2029 PARAGRAPH SEPARATOR if you never want passes to span
 7748   values. (Mike McCandless, Robert Muir)
 7749 
 7750 * LUCENE-5013: Added ScandinavianFoldingFilterFactory and
 7751   ScandinavianNormalizationFilterFactory (Karl Wettin via janhoy)
 7752 
 7753 * LUCENE-4845: AnalyzingInfixSuggester finds suggestions based on
 7754   matches to any tokens in the suggestion, not just based on pure
 7755   prefix matching.  (Mike McCandless, Robert Muir)
 7756 
 7757 API Changes
 7758 
 7759 * LUCENE-5077: Make it easier to use compressed norms. Lucene42NormsFormat takes
 7760   an overhead parameter, so you can easily pass a different value other than
 7761   PackedInts.FASTEST from your own codec.  (Robert Muir)
 7762   
 7763 * LUCENE-5097: Analyzer now has an additional tokenStream(String fieldName,
 7764   String text) method, so wrapping by StringReader for common use is no
 7765   longer needed. This method uses an internal reusable reader, which was
 7766   previously only used by the Field class.  (Uwe Schindler, Robert Muir)
 7767   
 7768 * LUCENE-4542: HunspellStemFilter's maximum recursion level is now configurable.
 7769   (Piotr, Rafał Kuć via Adrien Grand)
 7770   
 7771 Build
 7772 
 7773 * LUCENE-4987: Upgrade randomized testing to version 2.0.10: 
 7774   Test framework may fail internally due to overly aggressive J9 optimizations. 
 7775   (Dawid Weiss, Shai Erera)
 7776 
 7777 * LUCENE-5043: The eclipse target now uses the containing directory for the
 7778   project name.  This also enforces UTF-8 encoding when files are copied with
 7779   filtering.
 7780 
 7781 * LUCENE-5055: "rat-sources" target now checks also build.xml, ivy.xml,
 7782   forbidden-api signatures, and parts of resources folders.  (Ryan Ernst,
 7783   Uwe Schindler)
 7784 
 7785 * LUCENE-5072: Automatically patch javadocs generated by JDK versions
 7786   before 7u25 to work around the frame injection vulnerability (CVE-2013-1571,
 7787   VU#225657).  (Uwe Schindler)
 7788 
 7789 Tests
 7790 
 7791 * LUCENE-4901: TestIndexWriterOnJRECrash should work on any 
 7792   JRE vendor via Runtime.halt().
 7793   (Mike McCandless, Robert Muir, Uwe Schindler, Rodrigo Trujillo, Dawid Weiss)
 7794 
 7795 Changes in runtime behavior
 7796 
 7797 * LUCENE-5038: New segments written by IndexWriter are now wrapped into CFS
 7798   by default. DocumentsWriterPerThread doesn't consult MergePolicy anymore 
 7799   to decide if a CFS must be written, instead IndexWriterConfig now has a
 7800   property to enable / disable CFS for newly created segments. (Simon Willnauer)
 7801 
 7802 * LUCENE-5107: Properties files by Lucene are now written in UTF-8 encoding,
 7803   Unicode is no longer escaped. Reading of legacy properties files with
 7804   \u escapes is still possible.  (Uwe Schindler, Robert Muir)
 7805 
 7806 ======================= Lucene 4.3.1 =======================
 7807 
 7808 Bug Fixes
 7809 
 7810 * SOLR-4813: Fix SynonymFilterFactory to allow init parameters for
 7811   tokenizer factory used when parsing synonyms file.  (Shingo Sasaki, hossman)
 7812 
 7813 * LUCENE-4935: CustomScoreQuery wrongly applied its query boost twice
 7814   (boost^2).  (Robert Muir)
 7815 
 7816 * LUCENE-4948: Fixed ArrayIndexOutOfBoundsException in PostingsHighlighter
 7817   if you had a 64-bit JVM without compressed OOPS: IBM J9, or Oracle with
 7818   large heap/explicitly disabled.  (Mike McCandless, Uwe Schindler, Robert Muir)
 7819 
 7820 * LUCENE-4953: Fixed ParallelCompositeReader to inform ReaderClosedListeners of
 7821   its synthetic subreaders. FieldCaches keyed on the atomic children will be purged
 7822   earlier and FC insanity prevented.  In addition, ParallelCompositeReader's
 7823   toString() was changed to better reflect the reader structure.
 7824   (Mike McCandless, Uwe Schindler)
 7825 
 7826 * LUCENE-4968: Fixed ToParentBlockJoinQuery/Collector: correctly handle parent
 7827   hits that had no child matches, don't throw IllegalArgumentEx when
 7828   the child query has no hits, more aggressively catch cases where childQuery
 7829   incorrectly matches parent documents (Mike McCandless)
 7830 
 7831 * LUCENE-4970: Fix boost value of rewritten NGramPhraseQuery.
 7832   (Shingo Sasaki via Adrien Grand)
 7833 
 7834 * LUCENE-4974: CommitIndexTask was broken if no params were set. (Shai Erera)
 7835 
 7836 * LUCENE-4986: Fixed case where a newly opened near-real-time reader
 7837   fails to reflect a delete from IndexWriter.tryDeleteDocument (Reg,
 7838   Mike McCandless)
 7839 
 7840 * LUCENE-4994: Fix PatternKeywordMarkerFilter to have public constructor.
 7841   (Uwe Schindler)
 7842 
 7843 * LUCENE-4993: Fix BeiderMorseFilter to preserve custom attributes when
 7844   inserting tokens with position increment 0.  (Uwe Schindler)
 7845 
 7846 * LUCENE-4991: Fix handling of synonyms in classic QueryParser.getFieldQuery for
 7847   terms not separated by whitespace. PositionIncrementAttribute was ignored, so with
 7848   default AND synonyms wrongly became mandatory clauses, and with OR, the
 7849   coordination factor was wrong.  (李威, Robert Muir)
 7850 
 7851 * LUCENE-5002: IndexWriter#deleteAll() caused a deadlock in DWPT / DWSC if a
 7852   DwPT was flushing concurrently while deleteAll() aborted all DWPT. The IW
 7853   should never wait on DWPT via the flush control while holding on to the IW
 7854   Lock. (Simon Willnauer)
 7855 
 7856 Optimizations
 7857 
 7858 * LUCENE-4938: Don't use an unnecessarily large priority queue in IndexSearcher
 7859   methods that take top-N.  (Uwe Schindler, Mike McCandless, Robert Muir)
 7860 
 7861 
 7862 ======================= Lucene 4.3.0 =======================
 7863 
 7864 Changes in backwards compatibility policy
 7865 
 7866 * LUCENE-4810: EdgeNGramTokenFilter no longer increments position for
 7867   multiple ngrams derived from the same input token. (Walter Underwood
 7868   via Mike McCandless)
 7869 
 7870 * LUCENE-4822: KeywordTokenFilter is now an abstract class. Subclasses
 7871   need to implement #isKeyword() in order to mark terms as keywords. 
 7872   The existing functionality has been factored out into a new 
 7873   SetKeywordTokenFilter class. (Simon Willnauer, Uwe Schindler)
 7874   
 7875 * LUCENE-4642: Remove Tokenizer's and subclasses' ctors taking
 7876   AttributeSource. (Renaud Delbru, Uwe Schindler, Steve Rowe)
 7877 
 7878 * LUCENE-4833: IndexWriterConfig used to use LogByteSizeMergePolicy when
 7879   calling setMergePolicy(null) although the default merge policy is
 7880   TieredMergePolicy. IndexWriterConfig setters now throw an exception when
 7881   passed null if null is not a valid value. (Adrien Grand)
 7882 
 7883 * LUCENE-4849: Made ParallelTaxonomyArrays abstract with a concrete
 7884   implementation for DirectoryTaxonomyWriter/Reader. Also moved it under
 7885   o.a.l.facet.taxonomy. (Shai Erera)
 7886 
 7887 * LUCENE-4876: IndexDeletionPolicy is now an abstract class instead of an
 7888   interface. IndexDeletionPolicy, MergeScheduler and InfoStream now implement
 7889   Cloneable. (Adrien Grand)
 7890 
 7891 * LUCENE-4874: FilterAtomicReader and related classes (FilterTerms,
 7892   FilterDocsEnum, ...) don't forward anymore to the filtered instance when the
 7893   method has a default implementation through other abstract methods.
 7894   (Adrien Grand, Robert Muir)
 7895 
 7896 * LUCENE-4642, LUCENE-4877: Implementors of TokenizerFactory, TokenFilterFactory,
 7897   and CharFilterFactory now need to provide at least one constructor taking
 7898   Map<String,String> to be able to be loaded by the SPI framework (e.g., from Solr).
 7899   In addition, TokenizerFactory needs to implement the abstract
 7900   create(AttributeFactory,Reader) method.  (Renaud Delbru, Uwe Schindler,
 7901   Steve Rowe, Robert Muir)
 7902 
 7903 API Changes
 7904 
 7905 * LUCENE-4896: Made PassageFormatter abstract in PostingsHighlighter, made
 7906   members of DefaultPassageFormatter protected.  (Luca Cavanna via Robert Muir)
 7907 
 7908 * LUCENE-4844: removed TaxonomyReader.getParent(), you should use
 7909   TaxonomyReader.getParallelArrays().parents() instead. (Shai Erera)
 7910 
 7911 * LUCENE-4742: Renamed spatial 'Node' to 'Cell', along with any method names
 7912   and variables using this terminology. (David Smiley)
 7913 
 7914 New Features
 7915 
 7916 * LUCENE-4815: DrillSideways now allows more than one FacetRequest per
 7917   dimension (Mike McCandless)
 7918   
 7919 * LUCENE-3918: IndexSorter has been ported to 4.3 API and now supports
 7920   sorting documents by a numeric DocValues field, or reverse the order of
 7921   the documents in the index. Additionally, apps can implement their own
 7922   sort criteria. (Anat Hashavit, Shai Erera)
 7923 
 7924 * LUCENE-4817: Added KeywordRepeatFilter that allows to emit a token twice
 7925   once as a keyword and once as an ordinary token allow stemmers to emit
 7926   a stemmed version along with the un-stemmed version. (Simon Willnauer)
 7927 
 7928 * LUCENE-4822: PatternKeywordTokenFilter can mark tokens as keywords based
 7929   on regular expressions. (Simon Willnauer, Uwe Schindler)
 7930 
 7931 * LUCENE-4821: AnalyzingSuggester now uses the ending offset to
 7932   determine whether the last token was finished or not, so that a
 7933   query "i " will no longer suggest "Isla de Muerta" for example.
 7934   (Mike McCandless)
 7935   
 7936 * LUCENE-4642: Add create(AttributeFactory) to TokenizerFactory and
 7937   subclasses with ctors taking AttributeFactory. 
 7938   (Renaud Delbru, Uwe Schindler, Steve Rowe)
 7939 
 7940 * LUCENE-4820: Add payloads to Analyzing/FuzzySuggester, to record an
 7941   arbitrary byte[] per suggestion (Mike McCandless)
 7942 
 7943 * LUCENE-4816: Add WholeBreakIterator to PostingsHighlighter
 7944   for treating the entire content as a single Passage.  (Robert
 7945   Muir, Mike McCandless)
 7946 
 7947 * LUCENE-4827: Add additional ctor to PostingsHighlighter PassageScorer
 7948   to provide bm25 k1,b,avgdl parameters. (Robert Muir)
 7949 
 7950 * LUCENE-4607: Add DocIDSetIterator.cost() and Spans.cost() for optimizing
 7951   scoring.  (Simon Willnauer, Robert Muir)
 7952 
 7953 * LUCENE-4795: Add SortedSetDocValuesFacetFields and
 7954   SortedSetDocValuesAccumulator, to compute topK facet counts from a
 7955   field's SortedSetDocValues.  This method only supports flat
 7956   (dim/label) facets, is a bit (~25%) slower, has added cost
 7957   per-IndexReader-open to compute its ordinal map, but it requires no
 7958   taxonomy index and it tie-breaks facet labels in an understandable
 7959   (by Unicode sort order) way.  (Robert Muir, Mike McCandless)
 7960   
 7961 * LUCENE-4843: Add LimitTokenPositionFilter: don't emit tokens with 
 7962   positions that exceed the configured limit.  (Steve Rowe)
 7963 
 7964 * LUCENE-4832: Add ToParentBlockJoinCollector.getTopGroupsWithAllChildDocs, to retrieve
 7965   all children in each group.  (Aleksey Aleev via Mike McCandless)
 7966 
 7967 * LUCENE-4846: PostingsHighlighter subclasses can override where the
 7968   String values come from (it still defaults to pulling from stored
 7969   fields).  (Robert Muir, Mike McCandless)
 7970 
 7971 * LUCENE-4853: Add PostingsHighlighter.highlightFields method that
 7972   takes int[] docIDs instead of TopDocs.  (Robert Muir, Mike
 7973   McCandless)
 7974 
 7975 * LUCENE-4856: If there are no matches for a given field, return the
 7976   first maxPassages sentences (Robert Muir, Mike McCandless)
 7977 
 7978 * LUCENE-4859: IndexReader now exposes Terms statistics: getDocCount,
 7979   getSumDocFreq, getSumTotalTermFreq. (Shai Erera)
 7980 
 7981 * LUCENE-4862: It is now possible to terminate collection of a single
 7982   IndexReader leaf by throwing a CollectionTerminatedException in
 7983   Collector.collect. (Adrien Grand, Shai Erera)
 7984 
 7985 * LUCENE-4752: New SortingMergePolicy (in lucene/misc) that sorts documents
 7986   before merging segments. (Adrien Grand, Shai Erera, David Smiley)
 7987 
 7988 * LUCENE-4860: Customize scoring and formatting per-field in
 7989   PostingsHighlighter by subclassing and overriding the getFormatter
 7990   and/or getScorer methods.  This also changes Passage.getMatchTerms()
 7991   to return BytesRef[] instead of Term[].  (Robert Muir, Mike
 7992   McCandless)
 7993 
 7994 * LUCENE-4839: Added SorterTemplate.timSort, a O(n log n) stable sort algorithm
 7995   that performs well on partially sorted data. (Adrien Grand)
 7996 
 7997 * LUCENE-4644: Added support for the "IsWithin" spatial predicate for
 7998   RecursivePrefixTreeStrategy. It's for matching non-point indexed shapes; if
 7999   you only have points (1/doc) then "Intersects" is equivalent and faster.
 8000   See the javadocs.  (David Smiley)
 8001 
 8002 * LUCENE-4861: Make BreakIterator per-field in PostingsHighlighter. This means
 8003   you can override getBreakIterator(String field) to use different mechanisms
 8004   for e.g. title vs. body fields.  (Mike McCandless, Robert Muir)
 8005 
 8006 * LUCENE-4645: Added support for the "Contains" spatial predicate for
 8007   RecursivePrefixTreeStrategy.  (David Smiley)
 8008 
 8009 * LUCENE-4898: DirectoryReader.openIfChanged now allows opening a reader
 8010   on an IndexCommit starting from a near-real-time reader (previously
 8011   this would throw IllegalArgumentException).  (Mike McCandless)
 8012 
 8013 * LUCENE-4905: Made the maxPassages parameter per-field in PostingsHighlighter.
 8014   (Robert Muir)
 8015 
 8016 * LUCENE-4897: Added TaxonomyReader.getChildren for traversing a category's 
 8017   children. (Shai Erera)
 8018 
 8019 * LUCENE-4902: Added FilterDirectoryReader to allow easy filtering of a
 8020   DirectoryReader's subreaders. (Alan Woodward, Adrien Grand, Uwe Schindler)
 8021 
 8022 * LUCENE-4858: Added EarlyTerminatingSortingCollector to be used in conjunction
 8023   with SortingMergePolicy, which allows to early terminate queries on sorted
 8024   indexes, when the sort order matches the index order. (Adrien Grand, Shai Erera)
 8025 
 8026 * LUCENE-4904: Added descending sort order to NumericDocValuesSorter. (Shai Erera)
 8027 
 8028 * LUCENE-3786: Added SearcherTaxonomyManager, to manage access to both
 8029   IndexSearcher and DirectoryTaxonomyReader for near-real-time
 8030   faceting.  (Shai Erera, Mike McCandless)
 8031 
 8032 * LUCENE-4915: DrillSideways now allows drilling down on fields that
 8033   are not faceted. (Mike McCandless)
 8034 
 8035 * LUCENE-4895: Added support for the "IsDisjointTo" spatial predicate for
 8036   RecursivePrefixTreeStrategy.  (David Smiley)
 8037 
 8038 * LUCENE-4774: Added FieldComparator that allows sorting parent documents based on
 8039   fields on the child / nested document level. (Martijn van Groningen)
 8040 
 8041 Optimizations
 8042 
 8043 * LUCENE-4839: SorterTemplate.merge can now be overridden in order to replace
 8044   the default implementation which merges in-place by a faster implementation
 8045   that could require fewer swaps at the expense of some extra memory.
 8046   ArrayUtil and CollectionUtil override it so that their mergeSort and timSort
 8047   methods are faster but only require up to 1% of extra memory. (Adrien Grand)
 8048 
 8049 * LUCENE-4571: Speed up BooleanQuerys with minNrShouldMatch to use
 8050   skipping.  (Stefan Pohl via Robert Muir)
 8051 
 8052 * LUCENE-4863: StemmerOverrideFilter now uses a FST to represent its overrides
 8053   in memory. (Simon Willnauer)
 8054 
 8055 * LUCENE-4889: UnicodeUtil.codePointCount implementation replaced with a
 8056   non-array-lookup version. (Dawid Weiss)
 8057 
 8058 * LUCENE-4923: Speed up BooleanQuerys processing of in-order disjunctions.
 8059   (Robert Muir)
 8060 
 8061 * LUCENE-4926: Speed up DisjunctionMatchQuery.  (Robert Muir)
 8062 
 8063 * LUCENE-4930: Reduce contention in older/buggy JVMs when using
 8064   AttributeSource#addAttribute() because java.lang.ref.ReferenceQueue#poll()
 8065   is implemented using synchronization.  (Christian Ziech, Karl Wright,
 8066   Uwe Schindler)
 8067 
 8068 Bug Fixes
 8069 
 8070 * LUCENE-4868: SumScoreFacetsAggregator used an incorrect index into
 8071   the scores array. (Shai Erera)
 8072 
 8073 * LUCENE-4882: FacetsAccumulator did not allow to count ROOT category (i.e.
 8074   count dimensions). (Shai Erera)
 8075 
 8076 * LUCENE-4876: IndexWriterConfig.clone() now clones its MergeScheduler,
 8077   IndexDeletionPolicy and InfoStream in order to make an IndexWriterConfig and
 8078   its clone fully independent. (Adrien Grand)
 8079 
 8080 * LUCENE-4893: Facet counts were multiplied as many times as 
 8081   FacetsCollector.getFacetResults() is called. (Shai Erera)
 8082 
 8083 * LUCENE-4888: Fixed SloppyPhraseScorer, MultiDocs(AndPositions)Enum and
 8084   MultiSpansWrapper which happened to sometimes call DocIdSetIterator.advance
 8085   with target<=current (in this case the behavior of advance is undefined).
 8086   (Adrien Grand)
 8087 
 8088 * LUCENE-4899: FastVectorHighlighter failed with StringIndexOutOfBoundsException
 8089   if a single highlight phrase or term was greater than the fragCharSize producing
 8090   negative string offsets. (Simon Willnauer)
 8091 
 8092 * LUCENE-4877: Throw exception for invalid arguments in analysis factories.
 8093   (Steve Rowe, Uwe Schindler, Robert Muir)
 8094 
 8095 * LUCENE-4914: SpatialPrefixTree's Node/Cell.reset() forgot to reset the 'leaf'
 8096   flag.  It affects SpatialRecursivePrefixTreeStrategy on non-point indexed
 8097   shapes, as of Lucene 4.2. (David Smiley)
 8098 
 8099 * LUCENE-4913: FacetResultNode.ordinal was always 0 when all children
 8100   are returned. (Mike McCandless)
 8101 
 8102 * LUCENE-4918: Highlighter closes the given IndexReader if QueryScorer
 8103   is used with an external IndexReader. (Simon Willnauer, Sirvan Yahyaei)
 8104 
 8105 * LUCENE-4880: Fix MemoryIndex to consume empty terms from the tokenstream consistent
 8106   with IndexWriter. Previously it discarded them.  (Timothy Allison via Robert Muir)
 8107 
 8108 * LUCENE-4885: FacetsAccumulator did not set the correct value for
 8109   FacetResult.numValidDescendants. (Mike McCandless, Shai Erera)
 8110 
 8111 * LUCENE-4925: Fixed IndexSearcher.search when the argument list contains a Sort
 8112   and one of the sort fields is the relevance score. Only IndexSearchers created
 8113   with an ExecutorService are concerned. (Adrien Grand)
 8114 
 8115 * LUCENE-4738, LUCENE-2727, LUCENE-2812: Simplified
 8116   DirectoryReader.indexExists so that it's more robust to transient
 8117   IOExceptions (e.g. due to issues like file descriptor exhaustion),
 8118   but this will also cause it to err towards returning true for
 8119   example if the directory contains a corrupted index or an incomplete
 8120   initial commit.  In addition, IndexWriter with OpenMode.CREATE will
 8121   now succeed even if the directory contains a corrupted index (Billow
 8122   Gao, Robert Muir, Mike McCandless)
 8123 
 8124 * LUCENE-4928: Stored fields and term vectors could become super slow in case
 8125   of tiny documents (a few bytes). This is especially problematic when switching
 8126   codecs since bulk-merge strategies can't be applied and the same chunk of
 8127   documents can end up being decompressed thousands of times. A hard limit on
 8128   the number of documents per chunk has been added to fix this issue.
 8129   (Robert Muir, Adrien Grand)
 8130 
 8131 * LUCENE-4934: Fix minor equals/hashcode problems in facet/DrillDownQuery, 
 8132   BoostingQuery, MoreLikeThisQuery, FuzzyLikeThisQuery, and block join queries.
 8133   (Robert Muir, Uwe Schindler)
 8134 
 8135 * LUCENE-4504: Fix broken sort comparator in ValueSource.getSortField,
 8136   used when sorting by a function query.  (Tom Shally via Robert Muir)
 8137 
 8138 * LUCENE-4937: Fix incorrect sorting of float/double values (+/-0, NaN).
 8139   (Robert Muir, Uwe Schindler)
 8140 
 8141 Documentation
 8142 
 8143 * LUCENE-4841: Added example SimpleSortedSetFacetsExample to show how
 8144   to use the new SortedSetDocValues backed facet implementation.
 8145   (Shai Erera, Mike McCandless)
 8146 
 8147 Build
 8148 
 8149 * LUCENE-4879: Upgrade randomized testing to version 2.0.9: 
 8150   Filter stack traces on console output. (Dawid Weiss, Robert Muir)
 8151 
 8152 
 8153 ======================= Lucene 4.2.1 =======================
 8154 
 8155 Bug Fixes
 8156 
 8157 * LUCENE-4713: The SPI components used to load custom codecs or analysis
 8158   components were fixed to also scan the Lucene ClassLoader in addition
 8159   to the context ClassLoader, so Lucene is always able to find its own
 8160   codecs. The special case of a null context ClassLoader is now also
 8161   supported.  (Christian Kohlschütter, Uwe Schindler)
 8162 
 8163 * LUCENE-4819: seekExact(BytesRef, boolean) did not work correctly with
 8164   Sorted[Set]DocValuesTermsEnum.  (Robert Muir)
 8165 
 8166 * LUCENE-4826: PostingsHighlighter was not returning the top N best
 8167   scoring passages. (Robert Muir, Mike McCandless)
 8168 
 8169 * LUCENE-4854: Fix DocTermOrds.getOrdTermsEnum() to not return negative
 8170   ord on initial next().  (Robert Muir)
 8171 
 8172 * LUCENE-4836: Fix SimpleRateLimiter#pause to return the actual time spent
 8173   sleeping instead of the wakeup timestamp in nano seconds. (Simon Willnauer)
 8174 
 8175 * LUCENE-4828: BooleanQuery no longer extracts terms from its MUST_NOT
 8176   clauses.  (Mike McCandless)
 8177 
 8178 * SOLR-4589: Fixed CPU spikes and poor performance in lazy field loading 
 8179   of multivalued fields. (hossman)
 8180 
 8181 * LUCENE-4870: Fix bug where an entire index might be deleted by the IndexWriter
 8182   due to false detection if an index exists in the directory when
 8183   OpenMode.CREATE_OR_APPEND is used. This might also affect application that set
 8184   the open mode manually using DirectoryReader#indexExists. (Simon Willnauer)
 8185 
 8186 * LUCENE-4878: Override getRegexpQuery in MultiFieldQueryParser to prevent
 8187   NullPointerException when regular expression syntax is used with
 8188   MultiFieldQueryParser. (Simon Willnauer, Adam Rauch)
 8189 
 8190 Optimizations
 8191 
 8192 * LUCENE-4819: Added Sorted[Set]DocValues.termsEnum(), and optimized the
 8193   default codec for improved enumeration performance.  (Robert Muir)
 8194 
 8195 * LUCENE-4854: Speed up TermsEnum of FieldCache.getDocTermOrds.
 8196   (Robert Muir)
 8197 
 8198 * LUCENE-4857: Don't unnecessarily copy stem override map in
 8199   StemmerOverrideFilter. (Simon Willnauer)
 8200 
 8201 ======================= Lucene 4.2.0 =======================
 8202 
 8203 Changes in backwards compatibility policy
 8204 
 8205 * LUCENE-4602: FacetFields now stores facet ordinals in a DocValues field, 
 8206   rather than a payload. This forces rebuilding existing indexes, or do a
 8207   one time migration using FacetsPayloadMigratingReader. Since DocValues 
 8208   support in-memory caching, CategoryListCache was removed too.
 8209   (Shai Erera, Michael McCandless)
 8210 
 8211 * LUCENE-4697: FacetResultNode is now a concrete class with public members
 8212   (instead of getter methods). (Shai Erera)
 8213   
 8214 * LUCENE-4600: FacetsCollector is now an abstract class with two
 8215   implementations: StandardFacetsCollector (the old version of 
 8216   FacetsCollector) and CountingFacetsCollector. FacetsCollector.create()
 8217   returns the most optimized collector for the given parameters.
 8218   (Shai Erera, Michael McCandless)
 8219   
 8220 * LUCENE-4700: OrdinalPolicy is now per CategoryListParams, and is no longer
 8221   an interface, but rather an enum with values NO_PARENTS and ALL_PARENTS.
 8222   PathPolicy was removed, you should extend FacetFields and DrillDownStream
 8223   to control which categories are added as drill-down terms. (Shai Erera)
 8224 
 8225 * LUCENE-4547: DocValues improvements:
 8226   - Simplified codec API: codecs are now only responsible for encoding and
 8227     decoding docvalues, they do not need to do buffering or RAM accounting.
 8228   - Per-Field support: added PerFieldDocValuesFormat, which allows you to
 8229     use a different DocValuesFormat per field (like postings).
 8230   - Unified with FieldCache api: DocValues can be accessed via FieldCache API,
 8231     so it works automatically with grouping/join/sort/function queries, etc.
 8232   - Simplified types: There are only 3 types (NUMERIC, BINARY, SORTED), so it's
 8233     not necessary to specify for example that all of your binary values have
 8234     the same length. Instead it's easy for the Codec API to optimize encoding
 8235     based on any properties of the content.
 8236   (Simon Willnauer, Adrien Grand, Mike McCandless, Robert Muir)
 8237 
 8238 * LUCENE-4757: Cleanup and refactoring of FacetsAccumulator, FacetRequest,
 8239   FacetsAggregator and FacetResultsHandler API. If your application did 
 8240   FacetsCollector.create(), you should not be affected, but if you wrote
 8241   an Aggregator, then you should migrate it to the per-segment
 8242   FacetsAggregator. You can still use StandardFacetsAccumulator, which works
 8243   with the old API (for now). (Shai Erera) 
 8244 
 8245 * LUCENE-4761: Facet packages reorganized. Should be easy to fix your import
 8246   statements, if you use an IDE such as Eclipse. (Shai Erera)
 8247 
 8248 * LUCENE-4750: Convert DrillDown to DrillDownQuery, so you can initialize it
 8249   and add drill-down categories to it. (Michael McCandless, Shai Erera)
 8250 
 8251 * LUCENE-4759: remove FacetRequest.SortBy; result categories are always 
 8252   sorted by value, while ties are broken by category ordinal. (Shai Erera)
 8253 
 8254 * LUCENE-4772: Facet associations moved to new FacetsAggregator API. You
 8255   should override FacetsAccumulator and return the relevant aggregator,
 8256   for aggregating the association values. (Shai Erera)
 8257   
 8258 * LUCENE-4748: A FacetRequest on a non-existent field now returns an
 8259   empty FacetResult instead of skipping it.  (Shai Erera, Mike McCandless)
 8260 
 8261 * LUCENE-4806: The default category delimiter character was changed
 8262   from U+F749 to U+001F, since the latter uses 1 byte vs 3 bytes for
 8263   the former.  Existing facet indices must be reindexed.  (Robert
 8264   Muir, Shai Erera, Mike McCandless)
 8265 
 8266 Optimizations
 8267 
 8268 * LUCENE-4687: BloomFilterPostingsFormat now lazily initializes delegate
 8269   TermsEnum only if needed to do a seek or get a DocsEnum. (Simon Willnauer) 
 8270 
 8271 * LUCENE-4677, LUCENE-4682: unpacked FSTs now use vInt to encode the node target,
 8272   to reduce their size (Mike McCandless)
 8273 
 8274 * LUCENE-4678: FST now uses a paged byte[] structure instead of a
 8275   single byte[] internally, to avoid large memory spikes during
 8276   building (James Dyer, Mike McCandless)
 8277 
 8278 * LUCENE-3298: FST can now be larger than 2.1 GB / 2.1 B nodes.
 8279   (James Dyer, Mike McCandless)
 8280 
 8281 * LUCENE-4690: Performance improvements and non-hashing versions
 8282   of NumericUtils.*ToPrefixCoded() (yonik)
 8283 
 8284 * LUCENE-4715: CategoryListParams.getOrdinalPolicy now allows to return a
 8285   different OrdinalPolicy per dimension, to better tune how you index
 8286   facets. Also added OrdinalPolicy.ALL_BUT_DIMENSION. 
 8287   (Shai Erera, Michael McCandless)
 8288 
 8289 * LUCENE-4740: Don't track clones of MMapIndexInput if unmapping
 8290   is disabled. This reduces GC overhead. (Kristofer Karlsson, Uwe Schindler)
 8291 
 8292 * LUCENE-4733: The default Lucene 4.2 codec now uses a more compact
 8293   TermVectorsFormat (Lucene42TermVectorsFormat) based on
 8294   CompressingTermVectorsFormat. (Adrien Grand)
 8295 
 8296 * LUCENE-3729: The default Lucene 4.2 codec now uses a more compact
 8297   DocValuesFormat (Lucene42DocValuesFormat). Sorted values are stored in an
 8298   FST, Numerics and Ordinals use a number of strategies (delta-compression, 
 8299   table-compression, etc), and memory addresses use MonotonicBlockPackedWriter.
 8300   (Simon Willnauer, Adrien Grand, Mike McCandless, Robert Muir)
 8301 
 8302 * LUCENE-4792: Reduction of the memory required to build the doc ID maps used
 8303   when merging segments. (Adrien Grand)
 8304 
 8305 * LUCENE-4794: Spatial RecursivePrefixTreeStrategy's search filter: Skip calls
 8306   to termsEnum.seek() when the next term is known to follow the current cell.
 8307   (David Smiley)
 8308 
 8309 New Features
 8310 
 8311 * LUCENE-4686: New specialized DGapVInt8IntEncoder for facets (now the 
 8312   default). (Shai Erera)
 8313 
 8314 * LUCENE-4703: Add simple PrintTaxonomyStats tool to see summary
 8315   information about the facets taxonomy index.  (Mike McCandless)
 8316 
 8317 * LUCENE-4599: New oal.codecs.compressing.CompressingTermVectorsFormat which
 8318   compresses term vectors into chunks of documents similarly to
 8319   CompressingStoredFieldsFormat. (Adrien Grand)
 8320 
 8321 * LUCENE-4695: Added LiveFieldValues utility class, for getting the
 8322   current (live, real-time) value for any indexed doc/field.  The
 8323   class buffers recently indexed doc/field values until a new
 8324   near-real-time reader is opened that contains those changes.
 8325   (Robert Muir, Mike McCandless)
 8326 
 8327 * LUCENE-4723: Add AnalyzerFactoryTask to benchmark, and enable analyzer
 8328   creation via the resulting factories using NewAnalyzerTask.  (Steve Rowe)
 8329 
 8330 * LUCENE-4728: Unknown and not explicitly mapped queries are now rewritten
 8331   against the highlighting IndexReader to obtain primitive queries before 
 8332   discarding the query entirely. WeightedSpanTermExtractor now builds a
 8333   MemoryIndex only once even if multiple fields are highlighted.
 8334   (Simon Willnauer)
 8335 
 8336 * LUCENE-4035: Added ICUCollationDocValuesField, more efficient
 8337   support for Locale-sensitive sort and range queries for
 8338   single-valued fields.  (Robert Muir)
 8339 
 8340 * LUCENE-4547: Added MonotonicBlockPacked(Reader/Writer), which provide 
 8341   efficient random access to large amounts of monotonically increasing 
 8342   positive values (e.g. file offsets). Each block stores the minimum value
 8343   and the average gap, and values are encoded as signed deviations from
 8344   the expected value.  (Adrien Grand)
 8345   
 8346 * LUCENE-4547: Added AppendingLongBuffer, an append-only buffer that packs
 8347   signed long values in memory and provides an efficient iterator API.
 8348   (Adrien Grand) 
 8349 
 8350 * LUCENE-4540: It is now possible for a codec to represent norms with
 8351   less than 8 bits per value. For performance reasons this is not done
 8352   by default, but you can customize your codec (e.g. pass PackedInts.DEFAULT
 8353   to Lucene42DocValuesConsumer) if you want to make this tradeoff.
 8354   (Adrien Grand, Robert Muir)
 8355 
 8356 * LUCENE-4764: A new Facet42Codec and Facet42DocValuesFormat provide
 8357   faster but more RAM-consuming facet performance.  (Shai Erera, Mike
 8358   McCandless) 
 8359 
 8360 * LUCENE-4769: Added OrdinalsCache and CachedOrdsCountingFacetsAggregator
 8361   which uses the cache to obtain a document's ordinals. This aggregator
 8362   is faster than others, however consumes much more RAM.
 8363   (Michael McCandless, Shai Erera)
 8364   
 8365 * LUCENE-4778: Add a getter for the delegate in RateLimitedDirectoryWrapper.
 8366   (Mark Miller)
 8367 
 8368 * LUCENE-4765: Add a multi-valued docvalues type (SORTED_SET). This is equivalent
 8369   to building a FieldCache.getDocTermOrds at index-time.  (Robert Muir)
 8370 
 8371 * LUCENE-4780: Add MonotonicAppendingLongBuffer: an append-only buffer for
 8372   monotonically increasing values.  (Adrien Grand)
 8373 
 8374 * LUCENE-4748: Added DrillSideways utility class for computing both
 8375   drill-down and drill-sideways counts for a DrillDownQuery.  (Mike
 8376   McCandless)
 8377  
 8378 API Changes
 8379 
 8380 * LUCENE-4709: FacetResultNode no longer has a residue field. (Shai Erera)
 8381 
 8382 * LUCENE-4716: DrillDown.query now takes Occur, allowing to specify if
 8383   categories should be OR'ed or AND'ed. (Shai Erera)
 8384 
 8385 * LUCENE-4695: ReferenceManager.RefreshListener.afterRefresh now takes
 8386   a boolean indicating whether a new reference was in fact opened, and
 8387   a new beforeRefresh method notifies you when a refresh attempt is
 8388   starting.  (Robert Muir, Mike McCandless)
 8389 
 8390 * LUCENE-4794: Spatial RecursivePrefixTreeFilter replaced by
 8391   IntersectsPrefixTreeFilter and some extensible base classes. (David Smiley)
 8392   
 8393 Bug Fixes
 8394 
 8395 * LUCENE-4705: Pass on FilterStrategy in FilteredQuery if the filtered query is 
 8396   rewritten. (Simon Willnauer)
 8397 
 8398 * LUCENE-4712: MemoryIndex#normValues() throws NPE if field doesn't exist. 
 8399   (Simon Willnauer, Ricky Pritchett)
 8400 
 8401 * LUCENE-4550: Shapes wider than 180 degrees would use too much accuracy for the
 8402   PrefixTree based SpatialStrategy. For a pathological case of nearly 360
 8403   degrees and barely any height, it would generate so many indexed terms
 8404   (> 500k) that it could even cause an OutOfMemoryError. Fixed. (David Smiley)
 8405 
 8406 * LUCENE-4704: Make join queries override hashcode and equals methods.
 8407   (Martijn van Groningen)
 8408 
 8409 * LUCENE-4724: Fix bug in CategoryPath which allowed passing null or empty
 8410   string components. This is forbidden now (throws an exception). Note that if
 8411   you have a taxonomy index created with such strings, you should rebuild it.
 8412   (Michael McCandless, Shai Erera)
 8413 
 8414 * LUCENE-4732: Fixed TermsEnum.seekCeil/seekExact on term vectors.
 8415   (Adrien Grand, Robert Muir)
 8416 
 8417 * LUCENE-4739: Fixed bugs that prevented FSTs more than ~1.1GB from
 8418   being saved and loaded (Adrien Grand, Mike McCandless)
 8419 
 8420 * LUCENE-4717: Fixed bug where Lucene40DocValuesFormat would sometimes write 
 8421   an extra unused ordinal for sorted types. The bug is detected and corrected
 8422   on-the-fly for old indexes.  (Robert Muir)
 8423 
 8424 * LUCENE-4547: Fixed bug where Lucene40DocValuesFormat was unable to encode
 8425   segments that would exceed 2GB total data. This could happen in some surprising
 8426   cases, for example if you had an index with more than 260M documents and a 
 8427   VAR_INT field.  (Simon Willnauer, Adrien Grand, Mike McCandless, Robert Muir)
 8428 
 8429 * LUCENE-4775: Remove SegmentInfo.sizeInBytes() and make
 8430   MergePolicy.OneMerge.totalBytesSize thread safe (Josh Bronson via
 8431   Robert Muir, Mike McCandless)
 8432 
 8433 * LUCENE-4770: If spatial's TermQueryPrefixTreeStrategy was used to search
 8434   indexed non-point shapes, then there was an edge case where a query should
 8435   find a shape but it didn't. The fix is the removal of an optimization that
 8436   simplifies some leaf cells into a parent. The index data for such a field is
 8437   now ~20% larger. This optimization is still done for the query shape, and for
 8438   indexed data for RecursivePrefixTreeStrategy. Furthermore, this optimization
 8439   is enhanced to roll up beyond the bottom cell level. (David Smiley,
 8440   Florian Schilling)
 8441 
 8442 * LUCENE-4790: Fix FieldCacheImpl.getDocTermOrds to not bake deletes into the
 8443   cached datastructure. Otherwise this can cause inconsistencies with readers
 8444   at different points in time.  (Robert Muir)
 8445 
 8446 * LUCENE-4791: A conjunction of terms (ConjunctionTermScorer) scanned on
 8447   the lowest frequency term instead of skipping, leading to potentially
 8448   large performance impacts for many non-random or non-uniform
 8449   term distributions.  (John Wang, yonik)
 8450 
 8451 * LUCENE-4798: PostingsHighlighter's formatter sometimes didn't highlight 
 8452   matched terms.  (Robert Muir)
 8453 
 8454 * LUCENE-4796, SOLR-4373: Fix concurrency issue in NamedSPILoader and
 8455   AnalysisSPILoader when doing reload (e.g. from Solr).
 8456   (Uwe Schindler, Hossman)
 8457 
 8458 * LUCENE-4802: Don't compute norms for drill-down facet fields. (Mike McCandless)
 8459 
 8460 * LUCENE-4804: PostingsHighlighter sometimes applied terms to the wrong passage,
 8461   if they started exactly on a passage boundary.  (Robert Muir)
 8462 
 8463 Documentation
 8464 
 8465 * LUCENE-4718: Fixed documentation of oal.queryparser.classic.
 8466   (Hayden Muhl via Adrien Grand)
 8467 
 8468 * LUCENE-4784, LUCENE-4785, LUCENE-4786: Fixed references to deprecated classes
 8469   SinkTokenizer, ValueSourceQuery and RangeQuery. (Hao Zhong via Adrien Grand) 
 8470 
 8471 Build
 8472 
 8473 * LUCENE-4654: Test duration statistics from multiple test runs should be 
 8474   reused. (Dawid Weiss)
 8475 
 8476 * LUCENE-4636: Upgrade ivy to 2.3.0 (Shawn Heisey via Robert Muir)
 8477 
 8478 * LUCENE-4570: Use the Policeman Forbidden API checker, released separately
 8479   from Lucene and downloaded via Ivy.  (Uwe Schindler, Robert Muir)
 8480 
 8481 * LUCENE-4758: 'ant jar', 'ant compile', and 'ant compile-test' should
 8482   recurse.  (Steve Rowe)
 8483 
 8484 ======================= Lucene 4.1.0 =======================
 8485 
 8486 Changes in backwards compatibility policy
 8487 
 8488 * LUCENE-4514: Scorer's freq() method returns an integer value indicating
 8489   the number of times the scorer matches the current document. Previously
 8490   this was only sometimes the case, in some cases it returned a (meaningless)
 8491   floating point value.  Scorer now extends DocsEnum so it has attributes().
 8492   (Robert Muir)
 8493 
 8494 * LUCENE-4543: TFIDFSimilarity's index-time computeNorm is now final to
 8495   match the fact that its query-time norm usage requires a FIXED_8 encoding. 
 8496   Override lengthNorm and/or encode/decodeNormValue to change the specifics,
 8497   like Lucene 3.x. (Robert Muir)
 8498 
 8499 * LUCENE-3441: The facet module now supports NRT. As a result, the following
 8500   changes were made:
 8501   - DirectoryTaxonomyReader has a new constructor which takes a 
 8502     DirectoryTaxonomyWriter. You should use that constructor in order to get
 8503     the NRT support (or the old one for non-NRT).
 8504   - TaxonomyReader.refresh() removed in exchange for TaxonomyReader.openIfChanged
 8505     static method. Similar to DirectoryReader, the method either returns null
 8506     if no changes were made to the taxonomy, or a new TR instance otherwise.
 8507     Instead of calling refresh(), you should write similar code to how you reopen
 8508     a regular DirectoryReader.
 8509   - TaxonomyReader.openIfChanged (previously refresh()) no longer throws
 8510     InconsistentTaxonomyException, and supports recreate. InconsistentTaxoEx
 8511     was removed.
 8512   - ChildrenArrays was pulled out of TaxonomyReader into a top-level class.
 8513   - TaxonomyReader was made an abstract class (instead of an interface), with
 8514     methods such as close() and reference counting management pulled from
 8515     DirectoryTaxonomyReader, and made final. The rest of the methods, remained
 8516     abstract.
 8517   (Shai Erera, Gilad Barkai)
 8518 
 8519 * LUCENE-4576: Remove CachingWrapperFilter(Filter, boolean). This recacheDeletes
 8520   option gave less than 1% speedup at the expense of cache churn (filters were
 8521   invalidated on reopen if even a single delete was posted against the segment).
 8522   (Robert Muir)
 8523 
 8524 * LUCENE-4575: Replace IndexWriter's commit/prepareCommit versions that take
 8525   commitData with setCommitData(). That allows committing changes to IndexWriter
 8526   even if the commitData is the only thing that changes.
 8527   (Shai Erera, Michael McCandless)
 8528 
 8529 * LUCENE-4565: TaxonomyReader.getParentArray and .getChildrenArrays consolidated
 8530   into one getParallelTaxonomyArrays(). You can obtain the 3 arrays that the
 8531   previous two methods returned by calling parents(), children() or siblings()
 8532   on the returned ParallelTaxonomyArrays. (Shai Erera)
 8533   
 8534 * LUCENE-4585: Spatial PrefixTree based Strategies (either TermQuery or
 8535   RecursivePrefix based) MAY want to re-index if used for point data. If a
 8536   re-index is not done, then an indexed point is ~1/2 the smallest grid cell
 8537   larger and as such is slightly more likely to match a query shape.
 8538   (David Smiley)
 8539 
 8540 * LUCENE-4604: DefaultOrdinalPolicy removed in favor of OrdinalPolicy.ALL_PARENTS.
 8541   Same for DefaultPathPolicy (now PathPolicy.ALL_CATEGORIES). In addition, you
 8542   can use OrdinalPolicy.NO_PARENTS to never write any parent category ordinal
 8543   to the fulltree posting payload (but note that you need a special 
 8544   FacetsAccumulator - see javadocs). (Shai Erera)
 8545 
 8546 * LUCENE-4594: Spatial PrefixTreeStrategy no longer indexes center points of
 8547   non-point shapes.  If you want to call makeDistanceValueSource() based on
 8548   shape centers, you need to do this yourself in another spatial field.
 8549   (David Smiley)
 8550 
 8551 * LUCENE-4615: Replace IntArrayAllocator and FloatArrayAllocator by ArraysPool.
 8552   FacetArrays no longer takes those allocators; if you need to reuse the arrays,
 8553   you should use ReusingFacetArrays. (Shai Erera, Gilad Barkai)
 8554 
 8555 * LUCENE-4621: FacetIndexingParams is now a concrete class (instead of DefaultFIP).
 8556   Also, the entire IndexingParams chain is now immutable. If you need to override
 8557   a setting, you should extend the relevant class.
 8558   Additionally, FacetSearchParams is now immutable, and requires all FacetRequests
 8559   to specified at initialization time. (Shai Erera)
 8560 
 8561 * LUCENE-4647: CategoryDocumentBuilder and EnhancementsDocumentBuilder are replaced
 8562   by FacetFields and AssociationsFacetFields respectively. CategoryEnhancement and
 8563   AssociationEnhancement were removed in favor of a simplified CategoryAssociation
 8564   interface, with CategoryIntAssociation and CategoryFloatAssociation 
 8565   implementations.
 8566   NOTE: indexes that contain category enhancements/associations are not supported
 8567   by the new code and should be recreated. (Shai Erera)
 8568 
 8569 * LUCENE-4659: Massive cleanup to CategoryPath API. Additionally, CategoryPath is 
 8570   now immutable, so you don't need to clone() it. (Shai Erera)
 8571 
 8572 * LUCENE-4670: StoredFieldsWriter and TermVectorsWriter have new finish* callbacks
 8573   which are called after a doc/field/term has been completely added.
 8574   (Adrien Grand, Robert Muir)
 8575 
 8576 * LUCENE-4620: IntEncoder/Decoder were changed to do bulk encoding/decoding. As a
 8577   result, few other classes such as Aggregator and CategoryListIterator were
 8578   changed to handle bulk category ordinals. (Shai Erera)
 8579 
 8580 * LUCENE-4683: CategoryListIterator and Aggregator are now per-segment. As such
 8581   their implementations no longer take a top-level IndexReader in the constructor
 8582   but rather implement a setNextReader. (Shai Erera)
 8583   
 8584 New Features
 8585 
 8586 * LUCENE-4226: New experimental StoredFieldsFormat that compresses chunks of
 8587   documents together in order to improve the compression ratio. (Adrien Grand)
 8588 
 8589 * LUCENE-4426: New ValueSource implementations (in lucene/queries) for
 8590   DocValues fields. (Adrien Grand)
 8591 
 8592 * LUCENE-4410: FilteredQuery now exposes a FilterStrategy that exposes
 8593   how filters are applied during query execution. (Simon Willnauer) 
 8594 
 8595 * LUCENE-4404: New ListOfOutputs (in lucene/misc) for FSTs wraps
 8596   another Outputs implementation, allowing you to store more than one
 8597   output for a single input.  UpToTwoPositiveIntsOutputs was moved
 8598   from lucene/core to lucene/misc.  (Mike McCandless)
 8599 
 8600 * LUCENE-3842: New AnalyzingSuggester, for doing auto-suggest
 8601   using an analyzer.  This can create powerful suggesters: if the analyzer
 8602   remove stop words then "ghost chr..." could suggest "The Ghost of
 8603   Christmas Past"; if SynonymFilter is used to map wifi and wireless
 8604   network to hotspot, then "wirele..." could suggest "wifi router";
 8605   token normalization likes stemmers, accent removal, etc. would allow
 8606   the suggester to ignore such variations. (Robert Muir, Sudarshan
 8607   Gaikaiwari, Mike McCandless)
 8608 
 8609 * LUCENE-4446: Lucene 4.1 has a new default index format (Lucene41Codec)
 8610   that incorporates the previously experimental "Block" postings format
 8611   for better search performance. 
 8612   (Han Jiang, Adrien Grand, Robert Muir, Mike McCandless)
 8613 
 8614 * LUCENE-3846: New FuzzySuggester, like AnalyzingSuggester except it
 8615   also finds completions allowing for fuzzy edits in the input string.
 8616   (Robert Muir, Simon Willnauer, Mike McCandless)
 8617 
 8618 * LUCENE-4515: MemoryIndex now supports adding the same field multiple
 8619   times. (Simon Willnauer)
 8620 
 8621 * LUCENE-4489: Added consumeAllTokens option to LimitTokenCountFilter
 8622   (hossman, Robert Muir)
 8623 
 8624 * LUCENE-4566: Add NRT/SearcherManager.RefreshListener/addListener to
 8625   be notified whenever a new searcher was opened. (selckin via Shai
 8626   Erera, Mike McCandless)
 8627 
 8628 * SOLR-4123: Add per-script customizability to ICUTokenizerFactory via
 8629   rule files in the ICU RuleBasedBreakIterator format.
 8630   (Shawn Heisey, Robert Muir, Steve Rowe)
 8631 
 8632 * LUCENE-4590: Added WriteEnwikiLineDocTask - a benchmark task for writing
 8633   Wikipedia category pages and non-category pages into separate line files.
 8634   extractWikipedia.alg was changed to use this task, so now it creates two
 8635   files. (Doron Cohen)
 8636 
 8637 * LUCENE-4290: Added PostingsHighlighter to the highlighter module. It uses
 8638   offsets from the postings lists to highlight documents. (Robert Muir)
 8639 
 8640 * LUCENE-4628: Added CommonTermsQuery that executes high-frequency terms
 8641   in a optional sub-query to prevent slow queries due to "common" terms 
 8642   like stopwords. (Simon Willnauer)
 8643 
 8644 API Changes
 8645 
 8646 * LUCENE-4399: Deprecated AppendingCodec. Lucene's term dictionaries
 8647   no longer seek when writing.  (Adrien Grand, Robert Muir)
 8648 
 8649 * LUCENE-4479: Rename TokenStream.getTokenStream(IndexReader, int, String)
 8650   to TokenStream.getTokenStreamWithOffsets, and return null on failure
 8651   rather than throwing IllegalArgumentException.  (Alan Woodward)
 8652 
 8653 * LUCENE-4472: MergePolicy now accepts a MergeTrigger that provides 
 8654   information about the trigger of the merge ie. merge triggered due
 8655   to a segment merge or a full flush etc. (Simon Willnauer)
 8656 
 8657 * LUCENE-4415: TermsFilter is now immutable. All terms need to be provided
 8658   as constructor argument. (Simon Willnauer)
 8659 
 8660 * LUCENE-4520: ValueSource.getSortField no longer throws IOExceptions
 8661   (Alan Woodward)
 8662   
 8663 * LUCENE-4537: RateLimiter is now separated from FSDirectory and exposed via
 8664   RateLimitingDirectoryWrapper. Any Directory can now be rate-limited.
 8665   (Simon Willnauer)  
 8666 
 8667 * LUCENE-4591: CompressingStoredFields{Writer,Reader} now accept a segment
 8668   suffix as a constructor parameter. (Renaud Delbru via Adrien Grand)
 8669   
 8670 * LUCENE-4605: Added DocsEnum.FLAG_NONE which can be passed instead of 0 as
 8671   the flag to .docs() and .docsAndPositions(). (Shai Erera)
 8672 
 8673 * LUCENE-4617: Remove FST.pack() method. Previously to make a packed FST,
 8674   you had to make a Builder with willPackFST=true (telling it you will later pack it),
 8675   create your fst with finish(), and then call pack() to get another FST.
 8676   Instead just pass true for doPackFST to Builder and finish() returns a packed FST.
 8677   (Robert Muir)
 8678 
 8679 * LUCENE-4663: Deprecate IndexSearcher.document(int, Set). This was not intended
 8680   to be final, nor named document(). Use IndexSearcher.doc(int, Set) instead.
 8681   (Robert Muir)
 8682 
 8683 * LUCENE-4684: Made DirectSpellChecker extendable.
 8684   (Martijn van Groningen)
 8685   
 8686 Bug Fixes
 8687 
 8688 * LUCENE-1822: BaseFragListBuilder hard-coded 6 char margin is too naive.
 8689   (Alex Vigdor, Arcadius Ahouansou, Koji Sekiguchi)
 8690 
 8691 * LUCENE-4468: Fix rareish integer overflows in Lucene41 postings 
 8692   format. (Robert Muir)
 8693   
 8694 * LUCENE-4486: Add support for ConstantScoreQuery in Highlighter.
 8695  (Simon Willnauer)
 8696 
 8697 * LUCENE-4485: When CheckIndex terms, terms/docs pairs and tokens,
 8698   these counts now all exclude deleted documents.  (Mike McCandless)
 8699 
 8700 * LUCENE-4479: Highlighter works correctly for fields with term vector
 8701   positions, but no offsets.  (Alan Woodward)
 8702 
 8703 * SOLR-3906: JapaneseReadingFormFilter in romaji mode will return
 8704   romaji even for out-of-vocabulary kana cases (e.g. half-width forms).
 8705   (Robert Muir)
 8706 
 8707 * LUCENE-4511: TermsFilter might return wrong results if a field is not 
 8708   indexed or doesn't exist in the index. (Simon Willnauer)
 8709 
 8710 * LUCENE-4521: IndexWriter.tryDeleteDocument could return true
 8711   (successfully deleting the document) but then on IndexWriter
 8712   close/commit fail to write the new deletions, if no other changes
 8713   happened in the IndexWriter instance.  (Ivan Vasilev via Mike
 8714   McCandless)
 8715 
 8716 * LUCENE-4513: Fixed that deleted nested docs are scored into the
 8717   parent doc when using ToParentBlockJoinQuery. (Martijn van Groningen)
 8718 
 8719 * LUCENE-4534: Fixed WFSTCompletionLookup and Analyzing/FuzzySuggester
 8720   to allow 0 byte values in the lookup keys.  (Mike McCandless)
 8721 
 8722 * LUCENE-4532: DirectoryTaxonomyWriter use a timestamp to denote taxonomy
 8723   index re-creation, which could cause a bug in case machine clocks were
 8724   not synced. Instead, it now tracks an 'epoch' version, which is incremented
 8725   whenever the taxonomy is re-created, or replaced. (Shai Erera)
 8726 
 8727 * LUCENE-4544: Fixed off-by-1 in ConcurrentMergeScheduler that would
 8728   allow 1+maxMergeCount merges threads to be created, instead of just
 8729   maxMergeCount (Radim Kolar, Mike McCandless)
 8730 
 8731 * LUCENE-4567: Fixed NullPointerException in analyzing, fuzzy, and
 8732   WFST suggesters when no suggestions were added (selckin via Mike
 8733   McCandless)
 8734 
 8735 * LUCENE-4568: Fixed integer overflow in
 8736   PagedBytes.PagedBytesData{In,Out}put.getPosition. (Adrien Grand)
 8737 
 8738 * LUCENE-4581: GroupingSearch.setAllGroups(true) was failing to
 8739   actually compute allMatchingGroups (dizh@neusoft.com via Mike
 8740   McCandless)
 8741 
 8742 * LUCENE-4009: Improve TermsFilter.toString (Tim Costermans via Chris
 8743   Male, Mike McCandless)
 8744 
 8745 * LUCENE-4588: Benchmark's EnwikiContentSource was discarding last wiki 
 8746   document and had leaking threads in 'forever' mode. (Doron Cohen)   
 8747 
 8748 * LUCENE-4585: Spatial RecursivePrefixTreeFilter had some bugs that only
 8749   occurred when shapes were indexed.  In what appears to be rare circumstances,
 8750   documents with shapes near a query shape were erroneously considered a match.
 8751   In addition, it wasn't possible to index a shape representing the entire
 8752   globe.
 8753 
 8754 * LUCENE-4595: EnwikiContentSource had a thread safety problem (NPE) in 
 8755   'forever' mode (Doron Cohen)
 8756 
 8757 * LUCENE-4587: fix WordBreakSpellChecker to not throw AIOOBE when presented
 8758   with 2-char codepoints, and to correctly break/combine terms containing
 8759   non-latin characters. (James Dyer, Andreas Hubold)
 8760 
 8761 * LUCENE-4596: fix a concurrency bug in DirectoryTaxonomyWriter.
 8762   (Shai Erera)
 8763 
 8764 * LUCENE-4594: Spatial PrefixTreeStrategy would index center-points in addition
 8765   to the shape to index if it was non-point, in the same field.  But sometimes
 8766   the center-point isn't actually in the shape (consider a LineString), and for
 8767   highly precise shapes it could cause makeDistanceValueSource's cache to load
 8768   parts of the shape's boundary erroneously too.  So center points aren't
 8769   indexed any more; you should use another spatial field. (David Smiley)
 8770 
 8771 * LUCENE-4629: IndexWriter misses to delete documents if a document block is
 8772   indexed and the Iterator throws an exception. Documents were only rolled back
 8773   if the actual indexing process failed. (Simon Willnauer)
 8774 
 8775 * LUCENE-4608: Handle large number of requested fragments better.
 8776   (Martijn van Groningen)
 8777 
 8778 * LUCENE-4633: DirectoryTaxonomyWriter.replaceTaxonomy did not refresh its
 8779   internal reader, which could cause an existing category to be added twice.
 8780   (Shai Erera)
 8781 
 8782 * LUCENE-4461: If you added the same FacetRequest more than once, you would get
 8783   inconsistent results. (Gilad Barkai via Shai Erera)
 8784 
 8785 * LUCENE-4656: Fix regression in IndexWriter to work with empty TokenStreams
 8786   that have no TermToBytesRefAttribute (commonly provided by CharTermAttribute),
 8787   e.g., oal.analysis.miscellaneous.EmptyTokenStream.
 8788   (Uwe Schindler, Adrien Grand, Robert Muir)
 8789 
 8790 * LUCENE-4660: ConcurrentMergeScheduler was taking too long to
 8791   un-pause incoming threads it had paused when too many merges were
 8792   queued up. (Mike McCandless)
 8793 
 8794 * LUCENE-4662: Add missing elided articles and prepositions to FrenchAnalyzer's
 8795   DEFAULT_ARTICLES list passed to ElisionFilter.  (David Leunen via Steve Rowe)
 8796 
 8797 * LUCENE-4671: Fix CharsRef.subSequence method.  (Tim Smith via Robert Muir)
 8798 
 8799 * LUCENE-4465: Let ConstantScoreQuery's Scorer return its child scorer.
 8800   (selckin via Uwe Schindler)
 8801     
 8802 Changes in Runtime Behavior
 8803   
 8804 * LUCENE-4586: Change default ResultMode of FacetRequest to PER_NODE_IN_TREE.
 8805   This only affects requests with depth>1. If you execute such requests and
 8806   rely on the facet results being returned flat (i.e. no hierarchy), you should
 8807   set the ResultMode to GLOBAL_FLAT. (Shai Erera, Gilad Barkai) 
 8808 
 8809 * LUCENE-1822: Improves the text window selection by recalculating the starting margin
 8810   once all phrases in the fragment have been identified in FastVectorHighlighter. This
 8811   way if a single word is matched in a fragment, it will appear in the middle of the highlight,
 8812   instead of 6 characters from the beginning. This way one can also guarantee that
 8813   the entirety of short texts are represented in a fragment by specifying a large
 8814   enough fragCharSize.
 8815   
 8816 Optimizations
 8817 
 8818 * LUCENE-2221: oal.util.BitUtil was modified to use Long.bitCount and
 8819   Long.numberOfTrailingZeros (which are intrinsics since Java 6u18) instead of
 8820   pure java bit twiddling routines in order to improve performance on modern
 8821   JVMs/hardware. (Dawid Weiss, Adrien Grand)
 8822 
 8823 * LUCENE-4509: Enable stored fields compression by default in the Lucene 4.1
 8824   default codec. (Adrien Grand)
 8825 
 8826 * LUCENE-4536: PackedInts on-disk format is now byte-aligned (it used to be
 8827   long-aligned), saving up to 7 bytes per array of values.
 8828   (Adrien Grand, Mike McCandless)
 8829 
 8830 * LUCENE-4512: Additional memory savings for CompressingStoredFieldsFormat.
 8831   (Adrien Grand, Robert Muir)
 8832 
 8833 * LUCENE-4443: Lucene41PostingsFormat no longer writes unnecessary offsets 
 8834   into the skipdata. (Robert Muir)
 8835 
 8836 * LUCENE-4459: Improve WeakIdentityMap.keyIterator() to remove GCed keys
 8837   from backing map early instead of waiting for reap(). This makes test
 8838   failures in TestWeakIdentityMap disappear, too.
 8839   (Uwe Schindler, Mike McCandless, Robert Muir)
 8840 
 8841 * LUCENE-4473: Lucene41PostingsFormat encodes offsets more efficiently
 8842   for low frequency terms (< 128 occurrences).  (Robert Muir)
 8843 
 8844 * LUCENE-4462: DocumentsWriter now flushes deletes, segment infos and builds
 8845   CFS files if necessary during segment flush and not during publishing. The latter
 8846   was a single threaded process while now all IO and CPU heavy computation is done
 8847   concurrently in DocumentsWriterPerThread. (Simon Willnauer)
 8848 
 8849 * LUCENE-4496: Optimize Lucene41PostingsFormat when requesting a subset of
 8850   the postings data (via flags to TermsEnum.docs/docsAndPositions) to use
 8851   ForUtil.skipBlock.  (Robert Muir)
 8852 
 8853 * LUCENE-4497: Don't write PosVIntCount to the positions file in 
 8854   Lucene41PostingsFormat, as it's always totalTermFreq % BLOCK_SIZE. (Robert Muir)
 8855 
 8856 * LUCENE-4498: In Lucene41PostingsFormat, when a term appears in only one document, 
 8857   Instead of writing a file pointer to a VIntBlock containing the doc id, just 
 8858   write the doc id.  (Mike McCandless, Robert Muir)
 8859 
 8860 * LUCENE-4515: MemoryIndex now uses Byte/IntBlockPool internally to hold terms and 
 8861   posting lists. All index data is represented as consecutive byte/int arrays to 
 8862   reduce GC cost and memory overhead. (Simon Willnauer) 
 8863 
 8864 * LUCENE-4538: DocValues now caches direct sources in a ThreadLocal exposed via SourceCache. 
 8865   Users of this API can now simply obtain an instance via DocValues#getDirectSource per thread.
 8866   (Simon Willnauer)
 8867   
 8868 * LUCENE-4580: DrillDown.query variants return a ConstantScoreQuery with boost set to 0.0f
 8869   so that documents scores are not affected by running a drill-down query. (Shai Erera)  
 8870 
 8871 * LUCENE-4598: PayloadIterator no longer uses top-level IndexReader to iterate on the
 8872   posting's payload. (Shai Erera, Michael McCandless)
 8873 
 8874 * LUCENE-4661: Drop default maxThreadCount to 1 and maxMergeCount to 2
 8875   in ConcurrentMergeScheduler, for faster merge performance on
 8876   spinning-magnet drives (Mike McCandless)
 8877   
 8878 Documentation
 8879 
 8880 * LUCENE-4483: Refer to BytesRef.deepCopyOf in Term's constructor that takes BytesRef.
 8881   (Paul Elschot via Robert Muir)
 8882 
 8883 Build
 8884 
 8885 * LUCENE-4650: Upgrade randomized testing to version 2.0.8: make the
 8886   test framework more robust under low memory conditions. (Dawid Weiss)
 8887 
 8888 * LUCENE-4603: Upgrade randomized testing to version 2.0.5: print forked
 8889   JVM PIDs on heartbeat from hung tests (Dawid Weiss)
 8890 
 8891 * Upgrade randomized testing to version 2.0.4: avoid hangs on shutdown
 8892   hooks hanging forever by calling Runtime.halt() in addition to 
 8893   Runtime.exit() after a short delay to allow graceful shutdown (Dawid Weiss)
 8894 
 8895 * LUCENE-4451: Memory leak per unique thread caused by 
 8896   RandomizedContext.contexts static map. Upgrade randomized testing
 8897   to version 2.0.2 (Mike McCandless, Dawid Weiss)
 8898 
 8899 * LUCENE-4589: Upgraded benchmark module's Nekohtml dependency to version
 8900   1.9.17, removing the workaround in Lucene's HTML parser for the
 8901   Turkish locale.  (Uwe Schindler)
 8902 
 8903 * LUCENE-4601: Fix ivy availability check to use typefound, so it works
 8904   if called from another build file.  (Ryan Ernst via Robert Muir)
 8905   
 8906   
 8907 ======================= Lucene 4.0.0 =======================
 8908 
 8909 Changes in backwards compatibility policy
 8910 
 8911 * LUCENE-4392: Class org.apache.lucene.util.SortedVIntList has been removed.
 8912   (Adrien Grand)
 8913 
 8914 * LUCENE-4393: RollingCharBuffer has been moved to the o.a.l.analysis.util
 8915   package of lucene-analysis-common. (Adrien Grand)
 8916 
 8917 New Features
 8918 
 8919 * LUCENE-1888: Added the option to store payloads in the term
 8920   vectors (IndexableFieldType.storeTermVectorPayloads()). Note 
 8921   that you must store term vector positions to store payloads.
 8922   (Robert Muir)
 8923 
 8924 * LUCENE-3892: Add a new BlockPostingsFormat that bulk-encodes docs,
 8925   freqs and positions in large (size 128) packed-int blocks for faster
 8926   search performance.  This was from Han Jiang's 2012 Google Summer of
 8927   Code project (Han Jiang, Adrien Grand, Robert Muir, Mike McCandless)
 8928 
 8929 * LUCENE-4323: Added support for an absolute maximum CFS segment size
 8930   (in MiB) to LogMergePolicy and TieredMergePolicy.
 8931   (Alexey Lef via Uwe Schindler)
 8932   
 8933 * LUCENE-4339: Allow deletes against 3.x segments for easier upgrading.
 8934   Lucene3x Codec is still otherwise read-only, you should not set it
 8935   as the default Codec on IndexWriter, because it cannot write new segments.
 8936   (Mike McCandless, Robert Muir)
 8937 
 8938 * SOLR-3441: ElisionFilterFactory is now MultiTermAware
 8939   (Jack Krupansky via hossman)
 8940   
 8941 API Changes
 8942 
 8943 * LUCENE-4391, LUCENE-4440: All methods of Lucene40Codec but
 8944   getPostingsFormatForField are now final. To reuse functionality 
 8945   of Lucene40, you should extend FilterCodec and delegate to Lucene40
 8946   instead of extending Lucene40Codec.  (Adrien Grand, Shai Erera,
 8947   Robert Muir, Uwe Schindler)
 8948 
 8949 * LUCENE-4299: Added Terms.hasPositions() and Terms.hasOffsets().
 8950   Previously you had no real way to know that a term vector field
 8951   had positions or offsets, since this can be configured on a 
 8952   per-field-per-document basis. (Robert Muir)
 8953 
 8954 * Removed DocsAndPositionsEnum.hasPayload() and simplified the
 8955   contract of getPayload(). It returns null if there is no payload,
 8956   otherwise returns the current payload. You can now call it multiple
 8957   times per position if you want. (Robert Muir)
 8958 
 8959 * Removed FieldsEnum. Fields API instead implements Iterable<String>
 8960   and exposes Iterator, so you can iterate over field names with
 8961   for (String field : fields) instead.  (Robert Muir)
 8962 
 8963 * LUCENE-4152: added IndexReader.leaves(), which lets you enumerate
 8964   the leaf atomic reader contexts for all readers in the tree.
 8965   (Uwe Schindler, Robert Muir)
 8966 
 8967 * LUCENE-4304: removed PayloadProcessorProvider. If you want to change
 8968   payloads (or other things) when merging indexes, it's recommended
 8969   to just use a FilterAtomicReader + IndexWriter.addIndexes. See the
 8970   OrdinalMappingAtomicReader and TaxonomyMergeUtils in the facets
 8971   module if you want an example of this.
 8972   (Mike McCandless, Uwe Schindler, Shai Erera, Robert Muir)
 8973 
 8974 * LUCENE-4304: Make CompositeReader.getSequentialSubReaders()
 8975   protected. To get atomic leaves of any IndexReader use the new method
 8976   leaves() (LUCENE-4152), which lists AtomicReaderContexts including
 8977   the doc base of each leaf.  (Uwe Schindler, Robert Muir)
 8978 
 8979 * LUCENE-4307: Renamed IndexReader.getTopReaderContext to 
 8980   IndexReader.getContext.  (Robert Muir)
 8981 
 8982 * LUCENE-4316: Deprecate Fields.getUniqueTermCount and remove it from
 8983   AtomicReader. If you really want the unique term count across all
 8984   fields, just sum up Terms.size() across those fields. This method
 8985   only exists so that this statistic can be accessed for Lucene 3.x 
 8986   segments, which don't support Terms.size().  (Uwe Schindler, Robert Muir)
 8987 
 8988 * LUCENE-4321: Change CharFilter to extend Reader directly, as FilterReader
 8989   overdelegates (read(), read(char[], int, int), skip, etc). This made it
 8990   hard to implement CharFilters that were correct. Instead only close() is
 8991   delegated by default: read(char[], int, int) and correct(int) are abstract
 8992   so that it's obvious which methods you should implement.  The protected 
 8993   inner Reader is 'input' like CharFilter in the 3.x series, instead of 'in'.  
 8994   (Dawid Weiss, Uwe Schindler, Robert Muir)
 8995 
 8996 * LUCENE-3309: The expert FieldSelector API, used to load only certain
 8997   fields in a stored document, has been replaced with the simpler
 8998   StoredFieldVisitor API.  (Mike McCandless)
 8999 
 9000 * LUCENE-4343: Made Tokenizer.setReader final. This is a setter that should
 9001   not be overridden by subclasses: per-stream initialization should happen
 9002   in reset().  (Robert Muir)
 9003 
 9004 * LUCENE-4377: Remove IndexInput.copyBytes(IndexOutput, long). 
 9005   Use DataOutput.copyBytes(DataInput, long) instead.
 9006   (Mike McCandless, Robert Muir)
 9007 
 9008 * LUCENE-4355: Simplify AtomicReader's sugar methods such as termDocsEnum,
 9009   termPositionsEnum, docFreq, and totalTermFreq to only take Term as a
 9010   parameter. If you want to do expert things such as pass a different
 9011   Bits as liveDocs, then use the flex apis (fields(), terms(), etc) directly.
 9012   (Mike McCandless, Robert Muir)
 9013 
 9014 * LUCENE-4425: clarify documentation of StoredFieldVisitor.binaryValue
 9015   and simplify the api to binaryField(FieldInfo, byte[]).
 9016   (Adrien Grand, Robert Muir)
 9017 
 9018 Bug Fixes
 9019 
 9020 * LUCENE-4423: DocumentStoredFieldVisitor.binaryField ignored offset and
 9021   length. (Adrien Grand)
 9022 
 9023 * LUCENE-4297: BooleanScorer2 would multiply the coord() factor
 9024   twice for conjunctions: for most users this is no problem, but
 9025   if you had a customized Similarity that returned something other
 9026   than 1 when overlap == maxOverlap (always the case for conjunctions),
 9027   then the score would be incorrect.  (Pascal Chollet, Robert Muir)
 9028 
 9029 * LUCENE-4298: MultiFields.getTermDocsEnum(IndexReader, Bits, String, BytesRef)
 9030   did not work at all, it would infinitely recurse.
 9031   (Alberto Paro via Robert Muir)
 9032 
 9033 * LUCENE-4300: BooleanQuery's rewrite was not always safe: if you
 9034   had a custom Similarity where coord(1,1) != 1F, then the rewritten
 9035   query would be scored differently.  (Robert Muir)
 9036 
 9037 * Don't allow negatives in the positions file. If you have an index
 9038   from 2.4.0 or earlier with such negative positions, and you already 
 9039   upgraded to 3.x, then to Lucene 4.0-ALPHA or -BETA, you should run 
 9040   CheckIndex. If it fails, then you need to upgrade again to 4.0  (Robert Muir)
 9041 
 9042 * LUCENE-4303: PhoneticFilterFactory and SnowballPorterFilterFactory load their
 9043   encoders / stemmers via the ResourceLoader now instead of Class.forName().
 9044   Solr users should now no longer have to embed these in its war. (David Smiley)
 9045 
 9046 * SOLR-3737: StempelPolishStemFilterFactory loaded its stemmer table incorrectly.
 9047   Also, ensure immutability and use only one instance of this table in RAM (lazy
 9048   loaded) since it's quite large. (sausarkar, Steven Rowe, Robert Muir)
 9049 
 9050 * LUCENE-4310: MappingCharFilter was failing to match input strings
 9051   containing non-BMP Unicode characters.  (Dawid Weiss, Robert Muir,
 9052   Mike McCandless)
 9053 
 9054 * LUCENE-4224: Add in-order scorer to query time joining and the
 9055   out-of-order scorer throws an UOE. (Martijn van Groningen, Robert Muir)
 9056 
 9057 * LUCENE-4333: Fixed NPE in TermGroupFacetCollector when faceting on mv fields.
 9058   (Jesse MacVicar, Martijn van Groningen)
 9059 
 9060 * LUCENE-4218: Document.get(String) and Field.stringValue() again return
 9061   values for numeric fields, like Lucene 3.x and consistent with the documentation.
 9062   (Jamie, Uwe Schindler, Robert Muir)
 9063 
 9064 * NRTCachingDirectory was always caching a newly flushed segment in
 9065   RAM, instead of checking the estimated size of the segment
 9066   to decide whether to cache it. (Mike McCandless)
 9067 
 9068 * LUCENE-3720: fix memory-consumption issues with BeiderMorseFilter.
 9069   (Thomas Neidhart via Robert Muir)
 9070 
 9071 * LUCENE-4401: Fix bug where DisjunctionSumScorer would sometimes call score()
 9072   on a subscorer that had already returned NO_MORE_DOCS.  (Liu Chao, Robert Muir)
 9073 
 9074 * LUCENE-4411: when sampling is enabled for a FacetRequest, its depth
 9075   parameter is reset to the default (1), even if set otherwise.
 9076   (Gilad Barkai via Shai Erera)
 9077 
 9078 * LUCENE-4455: Fix bug in SegmentInfoPerCommit.sizeInBytes() that was
 9079   returning 2X the true size, inefficiently.  Also fixed bug in
 9080   CheckIndex that would report no deletions when a segment has
 9081   deletions, and vice/versa.  (Uwe Schindler, Robert Muir, Mike McCandless)
 9082 
 9083 * LUCENE-4456: Fixed double-counting sizeInBytes for a segment
 9084   (affects how merge policies pick merges); fixed CheckIndex's
 9085   incorrect reporting of whether a segment has deletions; fixed case
 9086   where on abort Lucene could remove files it didn't create; fixed
 9087   many cases where IndexWriter could leave leftover files (on
 9088   exception in various places, on reuse of a segment name after crash
 9089   and recovery.  (Uwe Schindler, Robert Muir, Mike McCandless)
 9090   
 9091 Optimizations
 9092 
 9093 * LUCENE-4322: Decrease lucene-core JAR size. The core JAR size had increased a
 9094   lot because of generated code introduced in LUCENE-4161 and LUCENE-3892.
 9095   (Adrien Grand)
 9096 
 9097 * LUCENE-4317: Improve reuse of internal TokenStreams and StringReader
 9098   in oal.document.Field.  (Uwe Schindler, Chris Male, Robert Muir)
 9099 
 9100 * LUCENE-4327: Support out-of-order scoring in FilteredQuery for higher
 9101   performance.  (Mike McCandless, Robert Muir)
 9102 
 9103 * LUCENE-4364: Optimize MMapDirectory to not make a mapping per-cfs-slice,
 9104   instead one map per .cfs file. This reduces the total number of maps.
 9105   Additionally factor out a (package-private) generic 
 9106   ByteBufferIndexInput from MMapDirectory.  (Uwe Schindler, Robert Muir)
 9107 
 9108 Build
 9109 
 9110 * LUCENE-4406, LUCENE-4407: Upgrade to randomizedtesting 2.0.1.
 9111   Workaround for broken test output XMLs due to non-XML text unicode
 9112   chars in strings. Added printing of failed tests at the end of a
 9113   test run (Dawid Weiss)
 9114 
 9115 * LUCENE-4252: Detect/Fail tests when they leak RAM in static fields
 9116   (Robert Muir, Dawid Weiss)
 9117 
 9118 * LUCENE-4360: Support running the same test suite multiple times in 
 9119   parallel (Dawid Weiss)
 9120 
 9121 * LUCENE-3985: Upgrade to randomizedtesting 2.0.0. Added support for 
 9122   thread leak detection. Added support for suite timeouts. (Dawid Weiss)
 9123 
 9124 * LUCENE-4354: Corrected maven dependencies to be consistent with
 9125   the licenses/ folder and the binary release. Some had different
 9126   versions or additional unnecessary dependencies. (selckin via Robert Muir)
 9127 
 9128 * LUCENE-4340: Move all non-default codec, postings format and terms
 9129   dictionary implementations to lucene/codecs. (Adrien Grand)
 9130 
 9131 Documentation
 9132 
 9133 * LUCENE-4302: Fix facet userguide to have HTML loose doctype like
 9134   all other javadocs.  (Karl Nicholas via Uwe Schindler)
 9135 
 9136 ======================= Lucene 4.0.0-BETA =======================
 9137 
 9138 New features
 9139 
 9140 * LUCENE-4249: Changed the explanation of the PayloadTermWeight to use the
 9141   underlying PayloadFunction's explanation as the explanation
 9142   for the payload score. (Scott Smerchek via Robert Muir)
 9143   
 9144 * LUCENE-4069: Added BloomFilteringPostingsFormat for use with low-frequency terms
 9145   such as primary keys (Mark Harwood, Mike McCandless) 
 9146   
 9147 * LUCENE-4201: Added JapaneseIterationMarkCharFilter to normalize Japanese
 9148   iteration marks. (Robert Muir, Christian Moen)
 9149 
 9150 * LUCENE-3832: Added BasicAutomata.makeStringUnion method to efficiently
 9151   create automata from a fixed collection of UTF-8 encoded BytesRef
 9152   (Dawid Weiss, Robert Muir)
 9153 
 9154 * LUCENE-4153: Added option to fast vector highlighting via BaseFragmentsBuilder to
 9155   respect field boundaries in the case of highlighting for multivalued fields.
 9156   (Martijn van Groningen)
 9157 
 9158 * LUCENE-4227: Added DirectPostingsFormat, to hold all postings in
 9159   memory as uncompressed simple arrays.  This uses a tremendous amount
 9160   of RAM but gives good search performance gains.  (Mike McCandless)
 9161 
 9162 * LUCENE-2510, LUCENE-4044: Migrated Solr's Tokenizer-, TokenFilter-, and
 9163   CharFilterFactories to the lucene-analysis module. The API is still
 9164   experimental.  (Chris Male, Robert Muir, Uwe Schindler)
 9165 
 9166 * LUCENE-4230: When pulling a DocsAndPositionsEnum you can now
 9167   specify whether or not you require payloads (in addition to
 9168   offsets); turning one or both off may allow some codec
 9169   implementations to optimize the enum implementation.  (Robert Muir,
 9170   Mike McCandless)
 9171 
 9172 * LUCENE-4203: Add IndexWriter.tryDeleteDocument(AtomicReader reader,
 9173   int docID), to attempt deletion by docID as long as the provided
 9174   reader is an NRT reader, and the segment has not yet been merged
 9175   away (Mike McCandless).
 9176   
 9177 * LUCENE-4286: Added option to CJKBigramFilter to always also output
 9178   unigrams. This can be used for a unigram+bigram approach, or at 
 9179   index-time only for better support of short queries.
 9180   (Tom Burton-West, Robert Muir)
 9181 
 9182 API Changes
 9183 
 9184 * LUCENE-4138: update of morfologik (Polish morphological analyzer) to 1.5.3.
 9185   The tag attribute class has been renamed to MorphosyntacticTagsAttribute and
 9186   has a different API (carries a list of tags instead of a compound tag). Upgrade
 9187   of embedded morfologik dictionaries to version 1.9. (Dawid Weiss)
 9188 
 9189 * LUCENE-4178: set 'tokenized' to true on FieldType by default, so that if you
 9190   make a custom FieldType and set indexed = true, it's analyzed by the analyzer.
 9191   (Robert Muir)
 9192 
 9193 * LUCENE-4220: Removed the buggy JavaCC-based HTML parser in the benchmark
 9194   module and replaced by NekoHTML. HTMLParser interface was cleaned up while
 9195   changing method signatures.  (Uwe Schindler, Robert Muir)
 9196 
 9197 * LUCENE-2191: Rename Tokenizer.reset(Reader) to Tokenizer.setReader(Reader).
 9198   The purpose of this method was always to set a new Reader on the Tokenizer,
 9199   reusing the object. But the name was often confused with TokenStream.reset().
 9200   (Robert Muir)
 9201 
 9202 * LUCENE-4228: Refactored CharFilter to extend java.io.FilterReader. CharFilters
 9203   filter another reader and you override correct() for offset correction.
 9204   (Robert Muir)
 9205 
 9206 * LUCENE-4240: Analyzer api now just takes fieldName for getOffsetGap. If the
 9207   field is not analyzed (e.g. StringField), then the analyzer is not invoked
 9208   at all. If you want to tweak things like positionIncrementGap and offsetGap,
 9209   analyze the field with KeywordTokenizer instead.  (Grant Ingersoll, Robert Muir)
 9210 
 9211 * LUCENE-4250: Pass fieldName to the PayloadFunction explain method, so it
 9212   parallels with docScore and the default implementation is correct.
 9213   (Robert Muir)
 9214 
 9215 * LUCENE-3747: Support Unicode 6.1.0. (Steve Rowe)
 9216 
 9217 * LUCENE-3884: Moved ElisionFilter out of org.apache.lucene.analysis.fr
 9218   package into org.apache.lucene.analysis.util.  (Robert Muir)
 9219 
 9220 * LUCENE-4230: When pulling a DocsAndPositionsEnum you now pass an int
 9221   flags instead of the previous boolean needOffsets.  Currently
 9222   recognized flags are DocsAndPositionsEnum.FLAG_PAYLOADS and
 9223   DocsAndPositionsEnum.FLAG_OFFSETS (Robert Muir, Mike McCandless)
 9224 
 9225 * LUCENE-4273: When pulling a DocsEnum, you can pass an int flags
 9226   instead of the previous boolean needsFlags; consistent with the changes
 9227   for DocsAndPositionsEnum in LUCENE-4230. Currently the only flag
 9228   is DocsEnum.FLAG_FREQS. (Robert Muir, Mike McCandless)
 9229   
 9230 * LUCENE-3616: TextField(String, Reader, Store) was reduced to TextField(String, Reader),
 9231   as the Store parameter didn't make sense: if you supplied Store.YES, you would only 
 9232   receive an exception anyway. (Robert Muir)
 9233 
 9234 Optimizations
 9235 
 9236 * LUCENE-4171: Performance improvements to Packed64.
 9237   (Toke Eskildsen via Adrien Grand)
 9238 
 9239 * LUCENE-4184: Performance improvements to the aligned packed bits impl.
 9240   (Toke Eskildsen, Adrien Grand)
 9241 
 9242 * LUCENE-4235: Remove enforcing of Filter rewrite for NRQ queries.
 9243   (Uwe Schindler)
 9244 
 9245 * LUCENE-4279: Regenerated snowball Stemmers from snowball r554,
 9246   making them substantially more lightweight. Behavior is unchanged. 
 9247   (Robert Muir)
 9248 
 9249 * LUCENE-4291: Reduced internal buffer size for Jflex-based tokenizers 
 9250   such as StandardTokenizer from 32kb to 8kb.  
 9251   (Raintung Li, Steven Rowe, Robert Muir)
 9252 
 9253 Bug Fixes
 9254 
 9255 * LUCENE-4109: BooleanQueries are not parsed correctly with the 
 9256   flexible query parser. (Karsten Rauch via Robert Muir)
 9257 
 9258 * LUCENE-4176: Fix AnalyzingQueryParser to analyze range endpoints as bytes,
 9259   so that it works correctly with Analyzers that produce binary non-UTF-8 terms
 9260   such as CollationAnalyzer. (Nattapong Sirilappanich via Robert Muir) 
 9261 
 9262 * LUCENE-4209: Fix FSTCompletionLookup to close its sorter, so that it won't
 9263   leave temp files behind in /tmp. Fix SortedTermFreqIteratorWrapper to not
 9264   leave temp files behind in /tmp on Windows. Fix Sort to not leave
 9265   temp files behind when /tmp is a separate volume. (Uwe Schindler, Robert Muir)
 9266 
 9267 * LUCENE-4221: Fix overeager CheckIndex validation for term vector offsets.
 9268   (Robert Muir)
 9269 
 9270 * LUCENE-4222: TieredMergePolicy.getFloorSegmentMB was returning the
 9271   size in bytes not MB (Chris Fuller via Mike McCandless)
 9272 
 9273 * LUCENE-3505: Fix bug (Lucene 4.0alpha only) where boolean conjunctions
 9274   were sometimes scored incorrectly. Conjunctions of only termqueries where
 9275   at least one term omitted term frequencies (IndexOptions.DOCS_ONLY) would 
 9276   be scored as if all terms omitted term frequencies.  (Robert Muir)
 9277 
 9278 * LUCENE-2686, LUCENE-3505: Fixed BooleanQuery scorers to return correct 
 9279   freq().  Added support for scorer navigation API (Scorer.getChildren) to 
 9280   all queries.  Made Scorer.freq() abstract. 
 9281   (Koji Sekiguchi, Mike McCandless, Robert Muir)
 9282 
 9283 * LUCENE-4234: Exception when FacetsCollector is used with ScoreFacetRequest, 
 9284   and the number of matching documents is too large. (Gilad Barkai via Shai Erera)
 9285 
 9286 * LUCENE-4245: Make IndexWriter#close() and MergeScheduler#close()
 9287   non-interruptible.  (Mark Miller, Uwe Schindler)
 9288 
 9289 * LUCENE-4190: restrict allowed filenames that a codec may create to
 9290   the patterns recognized by IndexFileNames.  This also fixes
 9291   IndexWriter to only delete files matching this pattern from an index
 9292   directory, to reduce risk when the wrong index path is accidentally
 9293   passed to IndexWriter (Robert Muir, Mike McCandless)
 9294   
 9295 * LUCENE-4277: Fix IndexWriter deadlock during rollback if flushable DWPT
 9296   instance are already checked out and queued up but not yet flushed. 
 9297   (Simon Willnauer)
 9298 
 9299 * LUCENE-4282: Automaton FuzzyQuery didn't always deliver all results.
 9300   (Johannes Christen, Uwe Schindler, Robert Muir)
 9301 
 9302 * LUCENE-4289: Fix minor idf inconsistencies/inefficiencies in highlighter.
 9303   (Robert Muir)
 9304 
 9305 Changes in Runtime Behavior
 9306 
 9307 * LUCENE-4109: Enable position increments in the flexible queryparser by default.
 9308   (Karsten Rauch via Robert Muir)
 9309   
 9310 * LUCENE-3616: Field throws exception if you try to set a boost on an 
 9311   unindexed field or one that omits norms. (Robert Muir)
 9312 
 9313 Build
 9314 
 9315 * LUCENE-4094: Support overriding file.encoding on forked test JVMs
 9316   (force via -Drandomized.file.encoding=XXX). (Dawid Weiss)
 9317 
 9318 * LUCENE-4189: Test output should include timestamps (start/end for each 
 9319   test/ suite). Added -Dtests.timestamps=[off by default]. (Dawid Weiss)
 9320 
 9321 * LUCENE-4110: Report long periods of forked jvm inactivity (hung tests/ suites).
 9322   Added -Dtests.heartbeat=[seconds] with the default of 60 seconds.
 9323   (Dawid Weiss)
 9324 
 9325 * LUCENE-4160: Added a property to quit the tests after a given
 9326   number of failures has occurred. This is useful in combination
 9327   with -Dtests.iters=N (you can start N iterations and wait for M
 9328   failures, in particular M = 1). -Dtests.maxfailures=M. Alternatively,
 9329   specify -Dtests.failfast=true to skip all tests after the first failure.
 9330   (Dawid Weiss)
 9331 
 9332 * LUCENE-4115: JAR resolution/ cleanup should be done automatically for ant 
 9333   clean/ eclipse/ resolve (Dawid Weiss)
 9334 
 9335 * LUCENE-4199, LUCENE-4202, LUCENE-4206: Add a new target "check-forbidden-apis"
 9336   that parses all generated .class files for use of APIs that use default
 9337   charset, default locale, or default timezone and fail build if violations
 9338   found. This ensures, that Lucene / Solr is independent on local configuration
 9339   options.  (Uwe Schindler, Robert Muir, Dawid Weiss)
 9340   
 9341 * LUCENE-4217: Add the possibility to run tests with Atlassian Clover
 9342   loaded from IVY. A development License solely for Apache code was added in
 9343   the tools/ folder, but is not included in releases.  (Uwe Schindler)
 9344 
 9345 Documentation
 9346 
 9347 * LUCENE-4195: Added package documentation and examples for 
 9348   org.apache.lucene.codecs (Alan Woodward via Robert Muir)
 9349 
 9350 ======================= Lucene 4.0.0-ALPHA =======================
 9351 
 9352 More information about this release, including any errata related to the 
 9353 release notes, upgrade instructions, or other changes may be found online at:
 9354    https://wiki.apache.org/lucene-java/Lucene4.0
 9355 
 9356 For "contrib" changes prior to 4.0, please see:
 9357 http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_3_6_0/lucene/contrib/CHANGES.txt
 9358 
 9359 Changes in backwards compatibility policy
 9360 
 9361 * LUCENE-1458, LUCENE-2111, LUCENE-2354: Changes from flexible indexing:
 9362 
 9363   - On upgrading to 4.0, if you do not fully reindex your documents,
 9364     Lucene will emulate the new flex API on top of the old index,
 9365     incurring some performance cost (up to ~10% slowdown, typically).
 9366     To prevent this slowdown, use oal.index.IndexUpgrader
 9367     to upgrade your indexes to latest file format (LUCENE-3082).
 9368 
 9369     Mixed flex/pre-flex indexes are perfectly fine -- the two
 9370     emulation layers (flex API on pre-flex index, and pre-flex API on
 9371     flex index) will remap the access as required.  So on upgrading to
 9372     4.0 you can start indexing new documents into an existing index.
 9373     To get optimal performance, use oal.index.IndexUpgrader
 9374     to upgrade your indexes to latest file format (LUCENE-3082).
 9375 
 9376   - The postings APIs (TermEnum, TermDocsEnum, TermPositionsEnum)
 9377     have been removed in favor of the new flexible
 9378     indexing (flex) APIs (Fields, FieldsEnum, Terms, TermsEnum,
 9379     DocsEnum, DocsAndPositionsEnum). One big difference is that field
 9380     and terms are now enumerated separately: a TermsEnum provides a
 9381     BytesRef (wraps a byte[]) per term within a single field, not a
 9382     Term.  Another is that when asking for a Docs/AndPositionsEnum, you
 9383     now specify the skipDocs explicitly (typically this will be the
 9384     deleted docs, but in general you can provide any Bits).
 9385 
 9386   - The term vectors APIs (TermFreqVector, TermPositionVector, 
 9387     TermVectorMapper) have been removed in favor of the above
 9388     flexible indexing APIs, presenting a single-document inverted
 9389     index of the document from the term vectors.
 9390 
 9391   - MultiReader ctor now throws IOException
 9392 
 9393   - Directory.copy/Directory.copyTo now copies all files (not just
 9394     index files), since what is and isn't and index file is now
 9395     dependent on the codecs used.
 9396 
 9397   - UnicodeUtil now uses BytesRef for UTF-8 output, and some method
 9398     signatures have changed to CharSequence.  These are internal APIs
 9399     and subject to change suddenly.
 9400 
 9401   - Positional queries (PhraseQuery, *SpanQuery) will now throw an
 9402     exception if use them on a field that omits positions during
 9403     indexing (previously they silently returned no results).
 9404 
 9405   - FieldCache.{Byte,Short,Int,Long,Float,Double}Parser's API has
 9406     changed -- each parse method now takes a BytesRef instead of a
 9407     String.  If you have an existing Parser, a simple way to fix it is
 9408     invoke BytesRef.utf8ToString, and pass that String to your
 9409     existing parser.  This will work, but performance would be better
 9410     if you could fix your parser to instead operate directly on the
 9411     byte[] in the BytesRef.
 9412 
 9413   - The internal (experimental) API of NumericUtils changed completely
 9414     from String to BytesRef. Client code should never use this class,
 9415     so the change would normally not affect you. If you used some of
 9416     the methods to inspect terms or create TermQueries out of
 9417     prefix encoded terms, change to use BytesRef. Please note:
 9418     Do not use TermQueries to search for single numeric terms.
 9419     The recommended way is to create a corresponding NumericRangeQuery
 9420     with upper and lower bound equal and included. TermQueries do not
 9421     score correct, so the constant score mode of NRQ is the only
 9422     correct way to handle single value queries.
 9423   
 9424   - NumericTokenStream now works directly on byte[] terms. If you
 9425     plug a TokenFilter on top of this stream, you will likely get
 9426     an IllegalArgumentException, because the NTS does not support
 9427     TermAttribute/CharTermAttribute. If you want to further filter
 9428     or attach Payloads to NTS, use the new NumericTermAttribute.
 9429     
 9430   (Mike McCandless, Robert Muir, Uwe Schindler, Mark Miller, Michael Busch)
 9431 
 9432 * LUCENE-2858, LUCENE-3733: IndexReader was refactored into abstract
 9433   AtomicReader, CompositeReader, and DirectoryReader. To open Directory-
 9434   based indexes use DirectoryReader.open(), the corresponding method in
 9435   IndexReader is now deprecated for easier migration. Only DirectoryReader
 9436   supports commits, versions, and reopening with openIfChanged(). Terms,
 9437   postings, docvalues, and norms can from now on only be retrieved using
 9438   AtomicReader; DirectoryReader and MultiReader extend CompositeReader,
 9439   only offering stored fields and access to the sub-readers (which may be
 9440   composite or atomic). SlowCompositeReaderWrapper (LUCENE-2597) can be
 9441   used to emulate atomic readers on top of composites.
 9442   Please review MIGRATE.txt for information how to migrate old code.
 9443   (Uwe Schindler, Robert Muir, Mike McCandless)
 9444 
 9445 * LUCENE-2265: FuzzyQuery and WildcardQuery now operate on Unicode codepoints,
 9446   not unicode code units. For example, a Wildcard "?" represents any unicode
 9447   character. Furthermore, the rest of the automaton package and RegexpQuery use 
 9448   true Unicode codepoint representation.  (Robert Muir, Mike McCandless)
 9449 
 9450 * LUCENE-2380: The String-based FieldCache methods (getStrings,
 9451   getStringIndex) have been replaced with BytesRef-based equivalents
 9452   (getTerms, getTermsIndex).  Also, the sort values (returned in
 9453   FieldDoc.fields) when sorting by SortField.STRING or
 9454   SortField.STRING_VAL are now BytesRef instances.  See MIGRATE.txt
 9455   for more details. (yonik, Mike McCandless)
 9456  
 9457 * LUCENE-2480: Though not a change in backwards compatibility policy, pre-3.0 
 9458   indexes are no longer supported. You should upgrade to 3.x first, then run
 9459   optimize(), or reindex. (Shai Erera, Earwin Burrfoot)
 9460   
 9461 * LUCENE-2484: Removed deprecated TermAttribute. Use CharTermAttribute
 9462   and TermToBytesRefAttribute instead.  (Uwe Schindler)
 9463 
 9464 * LUCENE-2600: Remove IndexReader.isDeleted in favor of
 9465   AtomicReader.getDeletedDocs().  (Mike McCandless)
 9466 
 9467 * LUCENE-2667: FuzzyQuery's defaults have changed for more performant 
 9468   behavior: the minimum similarity is 2 edit distances from the word,
 9469   and the priority queue size is 50. To support this, FuzzyQuery now allows
 9470   specifying unscaled edit distances (foobar~2). If your application depends
 9471   upon the old defaults of 0.5 (scaled) minimum similarity and Integer.MAX_VALUE
 9472   priority queue size, you can use FuzzyQuery(Term, float, int, int) to specify 
 9473   those explicitly.
 9474   
 9475 * LUCENE-2674: MultiTermQuery.TermCollector.collect now accepts the
 9476   TermsEnum as well.  (Robert Muir, Mike McCandless)
 9477 
 9478 * LUCENE-588: WildcardQuery and QueryParser now allows escaping with 
 9479   the '\' character. Previously this was impossible (you could not escape */?,
 9480   for example).  If your code somehow depends on the old behavior, you will
 9481   need to change it (e.g. using "\\" to escape '\' itself).  
 9482   (Sunil Kamath, Terry Yang via Robert Muir)
 9483 
 9484 * LUCENE-2837: Collapsed Searcher, Searchable into IndexSearcher;
 9485   removed contrib/remote and MultiSearcher (Mike McCandless); absorbed
 9486   ParallelMultiSearcher into IndexSearcher as an optional
 9487   ExecutorServiced passed to its ctor.  (Mike McCandless)
 9488 
 9489 * LUCENE-2908, LUCENE-4037: Removed serialization code from lucene classes. 
 9490   It is recommended that you serialize user search needs at a higher level 
 9491   in your application.
 9492   (Robert Muir, Benson Margulies)
 9493 
 9494 * LUCENE-2831: Changed Weight#scorer, Weight#explain & Filter#getDocIdSet to
 9495   operate on a AtomicReaderContext instead of directly on IndexReader to enable
 9496   searches to be aware of IndexSearcher's context. (Simon Willnauer)
 9497   
 9498 * LUCENE-2839: Scorer#score(Collector,int,int) is now public because it is
 9499   called from other classes and part of public API. (Uwe Schindler)
 9500   
 9501 * LUCENE-2865: Weight#scorer(AtomicReaderContext, boolean, boolean) now accepts
 9502   a ScorerContext struct instead of booleans.(Simon Willnauer)
 9503   
 9504 * LUCENE-2882: Cut over SpanQuery#getSpans to AtomicReaderContext to enforce
 9505   per segment semantics on SpanQuery & Spans. (Simon Willnauer)
 9506 
 9507 * LUCENE-2236: Similarity can now be configured on a per-field basis. See the
 9508   migration notes in MIGRATE.txt for more details.  (Robert Muir, Doron Cohen)
 9509 
 9510 * LUCENE-2315: AttributeSource's methods for accessing attributes are now final,
 9511   else it's easy to corrupt the internal states.  (Uwe Schindler)
 9512 
 9513 * LUCENE-2814: The IndexWriter.flush method no longer takes "boolean
 9514   flushDocStores" argument, as we now always flush doc stores (index
 9515   files holding stored fields and term vectors) while flushing a
 9516   segment.  (Mike McCandless)
 9517 
 9518 * LUCENE-2548: Field names (eg in Term, FieldInfo) are no longer
 9519   interned.  (Mike McCandless)
 9520 
 9521 * LUCENE-2883: The contents of o.a.l.search.function has been consolidated into
 9522   the queries module and can be found at o.a.l.queries.function.  See 
 9523   MIGRATE.txt for more information (Chris Male)
 9524 
 9525 * LUCENE-2392, LUCENE-3299: Decoupled vector space scoring from 
 9526   Query/Weight/Scorer. If you extended Similarity directly before, you should 
 9527   extend TFIDFSimilarity instead.  Similarity is now a lower-level API to 
 9528   implement other scoring algorithms.  See MIGRATE.txt for more details.
 9529   (David Nemeskey, Simon Willnauer, Mike McCandless, Robert Muir)
 9530 
 9531 * LUCENE-3330: The expert visitor API in Scorer has been simplified and
 9532   extended to support arbitrary relationships. To navigate to a scorer's 
 9533   children, call Scorer.getChildren().  (Robert Muir)
 9534 
 9535 * LUCENE-2308: Field is now instantiated with an instance of IndexableFieldType,
 9536   of which there is a core implementation FieldType.  Most properties 
 9537   describing a Field have been moved to IndexableFieldType.  See MIGRATE.txt 
 9538   for more details.  (Nikola Tankovic, Mike McCandless, Chris Male)
 9539 
 9540 * LUCENE-3396: ReusableAnalyzerBase.TokenStreamComponents.reset(Reader) now 
 9541   returns void instead of boolean.  If a Component cannot be reset, it should 
 9542   throw an Exception.  (Chris Male)
 9543 
 9544 * LUCENE-3396: ReusableAnalyzerBase has been renamed to Analyzer.  All Analyzer
 9545   implementations must now use Analyzer.TokenStreamComponents, rather than
 9546   overriding .tokenStream() and .reusableTokenStream() (which are now final). 
 9547   (Chris Male)
 9548 
 9549 * LUCENE-3346: Analyzer.reusableTokenStream() has been renamed to tokenStream()
 9550   with the old tokenStream() method removed.  Consequently it is now mandatory
 9551   for all Analyzers to support reusability. (Chris Male)
 9552 
 9553 * LUCENE-3473: AtomicReader.getUniqueTermCount() no longer throws UOE when
 9554   it cannot be easily determined. Instead, it returns -1 to be consistent with
 9555   this behavior across other index statistics.
 9556   (Robert Muir)
 9557 
 9558 * LUCENE-1536: The abstract FilteredDocIdSet.match() method is no longer
 9559   allowed to throw IOException. This change was required to make it conform
 9560   to the Bits interface. This method should never do I/O for performance reasons.
 9561   (Mike McCandless, Uwe Schindler, Robert Muir, Chris Male, Yonik Seeley,
 9562   Jason Rutherglen, Paul Elschot)
 9563 
 9564 * LUCENE-3559: The methods "docFreq" and "maxDoc" on IndexSearcher were removed,
 9565   as these are no longer used by the scoring system. See MIGRATE.txt for more
 9566   details.  (Robert Muir)
 9567   
 9568 * LUCENE-3533: Removed SpanFilters, they created large lists of objects and 
 9569   did not scale. (Robert Muir)
 9570   
 9571 * LUCENE-3606: IndexReader and subclasses were made read-only. It is no longer
 9572   possible to delete or undelete documents using IndexReader; you have to use
 9573   IndexWriter now. As deleting by internal Lucene docID is no longer possible,
 9574   this requires adding a unique identifier field to your index. Deleting/
 9575   relying upon Lucene docIDs is not recommended anyway, because they can
 9576   change. Consequently commit() was removed and DirectoryReader.open(),
 9577   openIfChanged() no longer take readOnly booleans or IndexDeletionPolicy
 9578   instances. Furthermore, IndexReader.setNorm() was removed. If you need
 9579   customized norm values, the recommended way to do this is by modifying
 9580   Similarity to use an external byte[] or one of the new DocValues
 9581   fields (LUCENE-3108). Alternatively, to dynamically change norms (boost
 9582   *and* length norm) at query time, wrap your AtomicReader using
 9583   FilterAtomicReader, overriding FilterAtomicReader.norms(). To persist the
 9584   changes on disk, copy the FilteredIndexReader to a new index using
 9585   IndexWriter.addIndexes().  (Uwe Schindler, Robert Muir)
 9586 
 9587 * LUCENE-3640: Removed IndexSearcher.close(), because IndexSearcher no longer
 9588   takes a Directory and no longer "manages" IndexReaders, it is a no-op.
 9589   (Robert Muir)
 9590 
 9591 * LUCENE-3684: Add offsets into DocsAndPositionsEnum, and a few
 9592   FieldInfo.IndexOption: DOCS_AND_POSITIONS_AND_OFFSETS.  (Robert
 9593   Muir, Mike McCandless)
 9594 
 9595 * LUCENE-2858, LUCENE-3770: FilterIndexReader was renamed to 
 9596   FilterAtomicReader and now extends AtomicReader. If you want to filter 
 9597   composite readers like DirectoryReader or MultiReader, filter their 
 9598   atomic leaves and build a new CompositeReader (e.g. MultiReader) around 
 9599   them. (Uwe Schindler, Robert Muir) 
 9600 
 9601 * LUCENE-3736: ParallelReader was split into ParallelAtomicReader
 9602   and ParallelCompositeReader. Lucene 3.x's ParallelReader is now
 9603   ParallelAtomicReader; but the new composite variant has improved performance
 9604   as it works on the atomic subreaders. It requires that all parallel
 9605   composite readers have the same subreader structure. If you cannot provide this,
 9606   you can use SlowCompositeReaderWrapper to make all parallel readers atomic
 9607   and use ParallelAtomicReader.  (Uwe Schindler, Mike McCandless, Robert Muir)
 9608 
 9609 * LUCENE-2000: clone() now returns covariant types where possible. (ryan)
 9610 
 9611 * LUCENE-3970: Rename Fields.getUniqueFieldCount -> .size() and
 9612   Terms.getUniqueTermCount -> .size().  (Iulius Curt via Mike McCandless)
 9613 
 9614 * LUCENE-3514: IndexSearcher.setDefaultFieldSortScoring was removed
 9615   and replaced with per-search control via new expert search methods
 9616   that take two booleans indicating whether hit scores and max
 9617   score should be computed.  (Mike McCandless)
 9618 
 9619 * LUCENE-4055: You can't put foreign files into the index dir anymore.
 9620 
 9621 * LUCENE-3866: CompositeReader.getSequentialSubReaders() now returns
 9622   unmodifiable List<? extends IndexReader>. ReaderUtil.Gather was
 9623   removed, as IndexReaderContext.leaves() is now the preferred way
 9624   to access sub-readers.  (Uwe Schindler)
 9625 
 9626 * LUCENE-4155: oal.util.ReaderUtil, TwoPhaseCommit, TwoPhaseCommitTool
 9627   classes were moved to oal.index package. oal.util.CodecUtil class was moved
 9628   to oal.codecs package. oal.util.DummyConcurrentLock was removed
 9629   (no longer used in Lucene 4.0).  (Uwe Schindler)
 9630 
 9631 Changes in Runtime Behavior
 9632 
 9633 * LUCENE-2846: omitNorms now behaves like omitTermFrequencyAndPositions, if you
 9634   omitNorms(true) for field "a" for 1000 documents, but then add a document with
 9635   omitNorms(false) for field "a", all documents for field "a" will have no 
 9636   norms.  Previously, Lucene would fill the first 1000 documents with 
 9637   "fake norms" from Similarity.getDefault(). (Robert Muir, Mike McCandless)
 9638 
 9639 * LUCENE-2846: When some documents contain field "a", and others do not, the
 9640   documents that don't have the field get a norm byte value of 0. Previously, 
 9641   Lucene would populate "fake norms" with Similarity.getDefault() for these 
 9642   documents.  (Robert Muir, Mike McCandless)
 9643   
 9644 * LUCENE-2720: IndexWriter throws IndexFormatTooOldException on open, rather 
 9645   than later when e.g. a merge starts. 
 9646   (Shai Erera, Mike McCandless, Uwe Schindler)
 9647 
 9648 * LUCENE-2881: FieldInfos is now tracked per segment.  Before it was tracked
 9649   per IndexWriter session, which resulted in FieldInfos that had the FieldInfo
 9650   properties from all previous segments combined. Field numbers are now tracked
 9651   globally across IndexWriter sessions and persisted into a _X.fnx file on
 9652   successful commit. The corresponding file format changes are backwards-
 9653   compatible. (Michael Busch, Simon Willnauer)
 9654   
 9655 * LUCENE-2956, LUCENE-2573, LUCENE-2324, LUCENE-2555: Changes from 
 9656   DocumentsWriterPerThread:
 9657 
 9658   - IndexWriter now uses a DocumentsWriter per thread when indexing documents.
 9659     Each DocumentsWriterPerThread indexes documents in its own private segment,
 9660     and the in memory segments are no longer merged on flush.  Instead, each
 9661     segment is separately flushed to disk and subsequently merged with normal
 9662     segment merging.
 9663 
 9664   - DocumentsWriterPerThread (DWPT) is now flushed concurrently based on a
 9665     FlushPolicy.  When a DWPT is flushed, a fresh DWPT is swapped in so that
 9666     indexing may continue concurrently with flushing.  The selected
 9667     DWPT flushes all its RAM resident documents do disk.  Note: Segment flushes
 9668     don't flush all RAM resident documents but only the documents private to
 9669     the DWPT selected for flushing. 
 9670   
 9671   - Flushing is now controlled by FlushPolicy that is called for every add,
 9672     update or delete on IndexWriter. By default DWPTs are flushed either on
 9673     maxBufferedDocs per DWPT or the global active used memory. Once the active
 9674     memory exceeds ramBufferSizeMB only the largest DWPT is selected for
 9675     flushing and the memory used by this DWPT is subtracted from the active
 9676     memory and added to a flushing memory pool, which can lead to temporarily
 9677     higher memory usage due to ongoing indexing.
 9678     
 9679   - IndexWriter now can utilize ramBufferSize > 2048 MB. Each DWPT can address
 9680     up to 2048 MB memory such that the ramBufferSize is now bounded by the max
 9681     number of DWPT available in the used DocumentsWriterPerThreadPool.
 9682     IndexWriters net memory consumption can grow far beyond the 2048 MB limit if
 9683     the application can use all available DWPTs. To prevent a DWPT from
 9684     exhausting its address space IndexWriter will forcefully flush a DWPT if its
 9685     hard memory limit is exceeded. The RAMPerThreadHardLimitMB can be controlled
 9686     via IndexWriterConfig and defaults to 1945 MB. 
 9687     Since IndexWriter flushes DWPT concurrently not all memory is released
 9688     immediately. Applications should still use a ramBufferSize significantly
 9689     lower than the JVMs available heap memory since under high load multiple
 9690     flushing DWPT can consume substantial transient memory when IO performance
 9691     is slow relative to indexing rate.
 9692     
 9693   - IndexWriter#commit now doesn't block concurrent indexing while flushing all
 9694     'currently' RAM resident documents to disk. Yet, flushes that occur while a
 9695     a full flush is running are queued and will happen after all DWPT involved
 9696     in the full flush are done flushing. Applications using multiple threads
 9697     during indexing and trigger a full flush (eg call commit() or open a new
 9698     NRT reader) can use significantly more transient memory.
 9699     
 9700   - IndexWriter#addDocument and IndexWriter.updateDocument can block indexing
 9701     threads if the number of active + number of flushing DWPT exceed a
 9702     safety limit. By default this happens if 2 * max number available thread
 9703     states (DWPTPool) is exceeded. This safety limit prevents applications from
 9704     exhausting their available memory if flushing can't keep up with
 9705     concurrently indexing threads.  
 9706     
 9707   - IndexWriter only applies and flushes deletes if the maxBufferedDelTerms
 9708     limit is reached during indexing. No segment flushes will be triggered
 9709     due to this setting.
 9710     
 9711   - IndexWriter#flush(boolean, boolean) doesn't synchronized on IndexWriter
 9712     anymore. A dedicated flushLock has been introduced to prevent multiple full-
 9713     flushes happening concurrently. 
 9714     
 9715   - DocumentsWriter doesn't write shared doc stores anymore. 
 9716   
 9717   (Mike McCandless, Michael Busch, Simon Willnauer)
 9718 
 9719 * LUCENE-3309: Stored fields no longer record whether they were
 9720   tokenized or not.  In general you should not rely on stored fields
 9721   to record any "metadata" from indexing (tokenized, omitNorms,
 9722   IndexOptions, boost, etc.)  (Mike McCandless)
 9723 
 9724 * LUCENE-3309: Fast vector highlighter now inserts the
 9725   MultiValuedSeparator for NOT_ANALYZED fields (in addition to
 9726   ANALYZED fields).  To ensure your offsets are correct you should
 9727   provide an analyzer that returns 1 from the offsetGap method.
 9728   (Mike McCandless)
 9729 
 9730 * LUCENE-2621: Removed contrib/instantiated.  (Robert Muir)
 9731  
 9732 * LUCENE-1768: StandardQueryTreeBuilder no longer uses RangeQueryNodeBuilder
 9733   for RangeQueryNodes, since theses two classes were removed;
 9734   TermRangeQueryNodeProcessor now creates TermRangeQueryNode,
 9735   instead of RangeQueryNode; the same applies for numeric nodes;
 9736   (Vinicius Barros via Uwe Schindler)
 9737 
 9738 * LUCENE-3455: QueryParserBase.newFieldQuery() will throw a ParseException if
 9739   any of the calls to the Analyzer throw an IOException.  QueryParseBase.analyzeRangePart()
 9740   will throw a RuntimeException if an IOException is thrown by the Analyzer.
 9741 
 9742 * LUCENE-4127: IndexWriter will now throw IllegalArgumentException if
 9743   the first token of an indexed field has 0 positionIncrement
 9744   (previously it silently corrected it to 1, possibly masking bugs).
 9745   OffsetAttributeImpl will throw IllegalArgumentException if startOffset
 9746   is less than endOffset, or if offsets are negative.
 9747   (Robert Muir, Mike McCandless)
 9748 
 9749 API Changes
 9750 
 9751 * LUCENE-2302, LUCENE-1458, LUCENE-2111, LUCENE-2514: Terms are no longer
 9752   required to be character based. Lucene views a term as an arbitrary byte[]:
 9753   during analysis, character-based terms are converted to UTF8 byte[],
 9754   but analyzers are free to directly create terms as byte[]
 9755   (NumericField does this, for example).  The term data is buffered as
 9756   byte[] during indexing, written as byte[] into the terms dictionary,
 9757   and iterated as byte[] (wrapped in a BytesRef) by IndexReader for
 9758   searching.
 9759 
 9760 * LUCENE-1458, LUCENE-2111: AtomicReader now directly exposes its
 9761   deleted docs (getDeletedDocs), providing a new Bits interface to
 9762   directly query by doc ID.
 9763 
 9764 * LUCENE-2691: IndexWriter.getReader() has been made package local and is now
 9765   exposed via open and reopen methods on DirectoryReader.  The semantics of the
 9766   call is the same as it was prior to the API change.
 9767   (Grant Ingersoll, Mike McCandless)
 9768 
 9769 * LUCENE-2566: QueryParser: Unary operators +,-,! will not be treated as
 9770   operators if they are followed by whitespace. (yonik)
 9771   
 9772 * LUCENE-2831: Weight#scorer, Weight#explain, Filter#getDocIdSet,
 9773   Collector#setNextReader & FieldComparator#setNextReader now expect an
 9774   AtomicReaderContext instead of an IndexReader. (Simon Willnauer)
 9775 
 9776 * LUCENE-2892: Add QueryParser.newFieldQuery (called by getFieldQuery by 
 9777   default) which takes Analyzer as a parameter, for easier customization by 
 9778   subclasses.  (Robert Muir)
 9779 
 9780 * LUCENE-2953: In addition to changes in 3.x, PriorityQueue#initialize(int)
 9781   function was moved into the ctor. (Uwe Schindler, Yonik Seeley)
 9782 
 9783 * LUCENE-3219: SortField type properties have been moved to an enum
 9784   SortField.Type.  In be consistent, CachedArrayCreator.getSortTypeID() has
 9785   been changed CachedArrayCreator.getSortType(). (Chris Male)
 9786 
 9787 * LUCENE-3225: Add TermsEnum.seekExact for faster seeking when you
 9788   don't need the ceiling term; renamed existing seek methods to either
 9789   seekCeil or seekExact; changed seekExact(ord) to return no value.
 9790   Fixed MemoryCodec and SimpleTextCodec to optimize the seekExact
 9791   case, and fixed places in Lucene to use seekExact when possible.
 9792   (Mike McCandless)
 9793 
 9794 * LUCENE-1536: Filter.getDocIdSet() now takes an acceptDocs Bits interface (like
 9795   Scorer) limiting the documents that can appear in the returned DocIdSet.
 9796   Filters are now required to respect these acceptDocs, otherwise deleted documents
 9797   may get returned by searches. Most filters will pass these Bits down to DocsEnum,
 9798   but those, e.g. working on FieldCache, may need to use BitsFilteredDocIdSet.wrap()
 9799   to exclude them.
 9800   (Mike McCandless, Uwe Schindler, Robert Muir, Chris Male, Yonik Seeley,
 9801   Jason Rutherglen, Paul Elschot)
 9802 
 9803 * LUCENE-3722: Similarity methods and collection/term statistics now take
 9804   long instead of int (to enable distributed scoring of > 2B docs).
 9805   (Yonik Seeley, Andrzej Bialecki, Robert Muir)
 9806 
 9807 * LUCENE-3761: Generalize SearcherManager into an abstract ReferenceManager.
 9808   SearcherManager remains a concrete class, but due to the refactoring, the 
 9809   method maybeReopen has been deprecated in favor of maybeRefresh().
 9810   (Shai Erera, Mike McCandless, Simon Willnauer)
 9811 
 9812 * LUCENE-3859: AtomicReader.hasNorms(field) is deprecated, instead you
 9813   can inspect the FieldInfo yourself to see if norms are present, which
 9814   also allows you to get the type.  (Robert Muir)
 9815 
 9816 * LUCENE-2606: Changed RegexCapabilities interface to fix thread 
 9817   safety, serialization, and performance problems. If you have
 9818   written a custom RegexCapabilities it will need to be updated
 9819   to the new API.  (Robert Muir, Uwe Schindler)
 9820 
 9821 * LUCENE-2638 MakeHighFreqTerms.TermStats public to make it more useful
 9822   for API use. (Andrzej Bialecki)
 9823 
 9824 * LUCENE-2912: The field-specific hashmaps in SweetSpotSimilarity were removed.
 9825   Instead, use PerFieldSimilarityWrapper to return different SweetSpotSimilaritys
 9826   for different fields, this way all parameters (such as TF factors) can be 
 9827   customized on a per-field basis.  (Robert Muir)
 9828 
 9829 * LUCENE-3308: DuplicateFilter keepMode and processingMode have been converted to
 9830   enums DuplicateFilter.KeepMode and DuplicateFilter.ProcessingMode respectively.
 9831 
 9832 * LUCENE-3483: Move Function grouping collectors from Solr to grouping module.
 9833   (Martijn van Groningen)
 9834    
 9835 * LUCENE-3606: FieldNormModifier was deprecated, because IndexReader's
 9836   setNorm() was deprecated. Furthermore, this class is broken, as it does
 9837   not take position overlaps into account while recalculating norms.
 9838   (Uwe Schindler, Robert Muir)
 9839 
 9840 * LUCENE-3936: Renamed StringIndexDocValues to DocTermsIndexDocValues.
 9841   (Martijn van Groningen)
 9842 
 9843 * LUCENE-1768: Deprecated Parametric(Range)QueryNode, RangeQueryNode(Builder),
 9844   ParametricRangeQueryNodeProcessor were removed. (Vinicius Barros via Uwe Schindler)
 9845 
 9846 * LUCENE-3820: Deprecated constructors accepting pattern matching bounds. The input
 9847   is buffered and matched in one pass. (Dawid Weiss)
 9848 
 9849 * LUCENE-2413: Deprecated PatternAnalyzer in common/miscellaneous, in favor 
 9850   of the pattern package (CharFilter, Tokenizer, TokenFilter).  (Robert Muir)
 9851 
 9852 * LUCENE-2413: Removed the AnalyzerUtil in common/miscellaneous.  (Robert Muir)
 9853 
 9854 * LUCENE-1370: Added ShingleFilter option to output unigrams if no shingles
 9855   can be generated. (Chris Harris via Steven Rowe)
 9856    
 9857 * LUCENE-2514, LUCENE-2551: JDK and ICU CollationKeyAnalyzers were changed to
 9858   use pure byte keys when Version >= 4.0. This cuts sort key size approximately
 9859   in half. (Robert Muir)
 9860 
 9861 * LUCENE-3400: Removed DutchAnalyzer.setStemDictionary (Chris Male)
 9862 
 9863 * LUCENE-3431: Removed QueryAutoStopWordAnalyzer.addStopWords* deprecated methods
 9864   since they prevented reuse.  Stopwords are now generated at instantiation through
 9865   the Analyzer's constructors. (Chris Male)
 9866 
 9867 * LUCENE-3434: Removed ShingleAnalyzerWrapper.set* and PerFieldAnalyzerWrapper.addAnalyzer
 9868   since they prevent reuse.  Both Analyzers should be configured at instantiation.
 9869   (Chris Male)
 9870 
 9871 * LUCENE-3765: Stopset ctors that previously took Set<?> or Map<?,String> now take
 9872   CharArraySet and CharArrayMap respectively. Previously the behavior was confusing,
 9873   and sometimes different depending on the type of set, and ultimately a CharArraySet
 9874   or CharArrayMap was always used anyway.  (Robert Muir)
 9875 
 9876 * LUCENE-3830: Switched to NormalizeCharMap.Builder to create
 9877   immutable instances of NormalizeCharMap. (Dawid Weiss, Mike
 9878   McCandless)
 9879 
 9880 * LUCENE-4063: FrenchLightStemmer no longer deletes repeated digits.
 9881   (Tanguy Moal via Steve Rowe)
 9882 
 9883 * LUCENE-4122: Replace Payload with BytesRef. (Andrzej Bialecki)
 9884 
 9885 * LUCENE-4132: IndexWriter.getConfig() now returns a LiveIndexWriterConfig object 
 9886   which can be used to change the IndexWriter's live settings. IndexWriterConfig 
 9887   is used only for initializing the IndexWriter. (Shai Erera)
 9888 
 9889 * LUCENE-3866: IndexReaderContext.leaves() is now the preferred way to access
 9890   atomic sub-readers of any kind of IndexReader (for AtomicReaders it returns
 9891   itself as only leaf with docBase=0).  (Uwe Schindler)
 9892 
 9893 New features
 9894 
 9895 * LUCENE-2604: Added RegexpQuery support to QueryParser. Regular expressions
 9896   are directly supported by the standard queryparser via 
 9897      fieldName:/expression/ OR /expression against default field/
 9898   Users who wish to search for literal "/" characters are advised to 
 9899   backslash-escape or quote those characters as needed. 
 9900   (Simon Willnauer, Robert Muir)
 9901 
 9902 * LUCENE-1606, LUCENE-2089: Adds AutomatonQuery, a MultiTermQuery that 
 9903   matches terms against a finite-state machine. Implement WildcardQuery
 9904   and FuzzyQuery with finite-state methods. Adds RegexpQuery.
 9905   (Robert Muir, Mike McCandless, Uwe Schindler, Mark Miller)
 9906 
 9907 * LUCENE-3662: Add support for levenshtein distance with transpositions
 9908   to LevenshteinAutomata, FuzzyTermsEnum, and DirectSpellChecker.
 9909   (Jean-Philippe Barrette-LaPierre, Robert Muir)
 9910 
 9911 * LUCENE-2321: Cutover to a more RAM efficient packed-ints based
 9912   representation for the in-memory terms dict index.  (Mike
 9913   McCandless)
 9914 
 9915 * LUCENE-2126: Add new classes for data (de)serialization: DataInput
 9916   and DataOutput.  IndexInput and IndexOutput extend these new classes.
 9917   (Michael Busch)
 9918 
 9919 * LUCENE-1458, LUCENE-2111: With flexible indexing it is now possible
 9920   for an application to create its own postings codec, to alter how
 9921   fields, terms, docs and positions are encoded into the index.  The
 9922   standard codec is the default codec. IndexWriter accepts a Codec
 9923   class to obtain codecs for newly written segments.
 9924 
 9925 * LUCENE-1458, LUCENE-2111: Some experimental codecs have been added
 9926   for flexible indexing, including pulsing codec (inlines
 9927   low-frequency terms directly into the terms dict, avoiding seeking
 9928   for some queries), sep codec (stores docs, freqs, positions, skip
 9929   data and payloads in 5 separate files instead of the 2 used by
 9930   standard codec), and int block (really a "base" for using
 9931   block-based compressors like PForDelta for storing postings data).
 9932   
 9933 * LUCENE-1458, LUCENE-2111: The in-memory terms index used by standard
 9934   codec is more RAM efficient: terms data is stored as block byte
 9935   arrays and packed integers.  Net RAM reduction for indexes that have
 9936   many unique terms should be substantial, and initial open time for
 9937   IndexReaders should be faster.  These gains only apply for newly
 9938   written segments after upgrading.
 9939 
 9940 * LUCENE-1458, LUCENE-2111: Terms data are now buffered directly as
 9941   byte[] during indexing, which uses half the RAM for ascii terms (and
 9942   also numeric fields).  This can improve indexing throughput for
 9943   applications that have many unique terms, since it reduces how often
 9944   a new segment must be flushed given a fixed RAM buffer size.
 9945 
 9946 * LUCENE-2489: Added PerFieldCodecWrapper (in oal.index.codecs) which
 9947   lets you set the Codec per field (Mike McCandless)
 9948 
 9949 * LUCENE-2373: Extend Codec to use SegmentInfosWriter and
 9950   SegmentInfosReader to allow customization of SegmentInfos data.
 9951   (Andrzej Bialecki)
 9952 
 9953 * LUCENE-2504: FieldComparator.setNextReader now returns a
 9954   FieldComparator instance.  You can "return this", to just reuse the
 9955   same instance, or you can return a comparator optimized to the new
 9956   segment.  (yonik, Mike McCandless)
 9957   
 9958 * LUCENE-2648: PackedInts.Iterator now supports to advance by more than a
 9959   single ordinal. (Simon Willnauer) 
 9960   
 9961 * LUCENE-2649: Objects in the FieldCache can optionally store Bits
 9962   that mark which docs have real values in the native[] (ryan)
 9963 
 9964 * LUCENE-2664: Add SimpleText codec, which stores all terms/postings
 9965   data in a single text file for transparency (at the expense of poor
 9966   performance).  (Sahin Buyrukbilen via Mike McCandless)
 9967 
 9968 * LUCENE-2589: Add a VariableSizedIntIndexInput, which, when used w/
 9969   Sep*, makes it simple to take any variable sized int block coders
 9970   (like Simple9/16) and use them in a codec.  (Mike McCandless)
 9971 
 9972 * LUCENE-2597: Add oal.index.SlowCompositeReaderWrapper, to wrap a
 9973   composite reader (eg MultiReader or DirectoryReader), making it
 9974   pretend it's an atomic reader.  This is a convenience class (you can
 9975   use MultiFields static methods directly, instead) if you need to use
 9976   the flex APIs directly on a composite reader.  (Mike McCandless)
 9977 
 9978 * LUCENE-2690: MultiTermQuery boolean rewrites per segment.
 9979   (Uwe Schindler, Robert Muir, Mike McCandless, Simon Willnauer)
 9980 
 9981 * LUCENE-996: The QueryParser now accepts mixed inclusive and exclusive
 9982   bounds for range queries.  Example: "{3 TO 5]"
 9983   QueryParser subclasses that overrode getRangeQuery will need to be changed
 9984   to use the new getRangeQuery method.  (Andrew Schurman, Mark Miller, yonik)
 9985 
 9986 * LUCENE-2742: Add native per-field postings format support. Codec lets you now
 9987   register a postings format for each field and which is in turn recorded 
 9988   into the index. Postings formats are maintained on a per-segment basis and be
 9989   resolved without knowing the actual postings format used for writing the segment.
 9990   (Simon Willnauer)
 9991 
 9992 * LUCENE-2741: Add support for multiple codecs that use the same file
 9993   extensions within the same segment. Codecs now use their per-segment codec
 9994   ID in the file names. (Simon Willnauer)
 9995 
 9996 * LUCENE-2843: Added a new terms index impl,
 9997   VariableGapTermsIndexWriter/Reader, that accepts a pluggable
 9998   IndexTermSelector for picking which terms should be indexed in the
 9999   terms dict.  This impl stores the indexed terms in an FST, which is
10000   much more RAM efficient than FixedGapTermsIndex.  (Mike McCandless)
10001 
10002 * LUCENE-2862: Added TermsEnum.totalTermFreq() and
10003   Terms.getSumTotalTermFreq().  (Mike McCandless, Robert Muir)
10004 
10005 * LUCENE-3290: Added Terms.getSumDocFreq()  (Mike McCandless, Robert Muir)
10006 
10007 * LUCENE-3003: Added new expert class oal.index.DocTermsOrd,
10008   refactored from Solr's UnInvertedField, for accessing term ords for
10009   multi-valued fields, per document.  This is similar to FieldCache in
10010   that it inverts the index to compute the ords, but differs in that
10011   it's able to handle multi-valued fields and does not hold the term
10012   bytes in RAM. (Mike McCandless)
10013   
10014 * LUCENE-3108, LUCENE-2935, LUCENE-2168, LUCENE-1231: Changes from 
10015   DocValues (ColumnStrideFields):
10016   
10017   - IndexWriter now supports typesafe dense per-document values stored in
10018     a column like storage. DocValues are stored on a per-document
10019     basis where each documents field can hold exactly one value of a given
10020     type. DocValues are provided via Fieldable and can be used in
10021     conjunction with stored and indexed values.
10022      
10023   - DocValues provides an entirely RAM resident document id to value
10024     mapping per field as well as a DocIdSetIterator based disk-resident
10025     sequential access API relying on filesystem-caches.
10026     
10027   - Both APIs are exposed via IndexReader and the Codec / Flex API allowing
10028     expert users to integrate customized DocValues reader and writer
10029     implementations by extending existing Codecs.
10030     
10031   - DocValues provides implementations for primitive datatypes like int,
10032     long, float, double and arrays of byte. Byte based implementations further
10033     provide storage variants like straight or dereferenced stored bytes, fixed
10034     and variable length bytes as well as index time sorted based on 
10035     user-provided comparators.
10036   
10037   (Mike McCandless, Simon Willnauer)  
10038 
10039 * LUCENE-3209: Added MemoryCodec, which stores all terms & postings in
10040   RAM as an FST; this is good for primary-key fields if you frequently
10041   need to lookup by that field or perform deletions against it, for
10042   example in a near-real-time setting. (Mike McCandless)
10043 
10044 * SOLR-2533: Added support for rewriting Sort and SortFields using an
10045   IndexSearcher.  SortFields can have SortField.REWRITEABLE type which
10046   requires they are rewritten before they are used. (Chris Male)
10047 
10048 * LUCENE-3203: FSDirectory can now limit the max allowed write rate
10049   (MB/sec) of all running merges, to reduce impact ongoing merging has
10050   on searching, NRT reopen time, etc.  (Mike McCandless)
10051 
10052 * LUCENE-2793: Directory#createOutput & Directory#openInput now accept an
10053   IOContext instead of a buffer size to allow low level optimizations for 
10054   different usecases like merging, flushing and reading.
10055   (Simon Willnauer, Mike McCandless, Varun Thacker)
10056 
10057 * LUCENE-3354: FieldCache can cache DocTermOrds. (Martijn van Groningen)
10058 
10059 * LUCENE-3376: ReusableAnalyzerBase has been moved from modules/analysis/common
10060   into lucene/src/java/org/apache/lucene/analysis (Chris Male)
10061 
10062 * LUCENE-3423: add Terms.getDocCount(), which returns the number of documents
10063   that have at least one term for a field.  (Yonik Seeley, Robert Muir)
10064 
10065 * LUCENE-2959: Added a variety of different relevance ranking systems to Lucene.
10066 
10067   - Added Okapi BM25, Language Models, Divergence from Randomness, and 
10068     Information-Based Models. The models are pluggable, support all of lucene's 
10069     features (boosts, slops, explanations, etc) and queries (spans, etc).
10070 
10071   - All models default to the same index-time norm encoding as 
10072     DefaultSimilarity, so you can easily try these out/switch back and 
10073     forth/run experiments and comparisons without reindexing. Note: most of 
10074     the models do rely upon index statistics that are new in Lucene 4.0, so 
10075     for existing 3.x indexes it's a good idea to upgrade your index to the 
10076     new format with IndexUpgrader first.
10077 
10078   - Added a new subclass SimilarityBase which provides a simplified API 
10079     for plugging in new ranking algorithms without dealing with all of the
10080     nuances and implementation details of Lucene. 
10081 
10082   - For example, to use BM25 for all fields: 
10083      searcher.setSimilarity(new BM25Similarity());
10084 
10085     If you instead want to apply different similarities (e.g. ones with 
10086     different parameter values or different algorithms entirely) to different
10087     fields, implement PerFieldSimilarityWrapper with your per-field logic.
10088 
10089   (David Mark Nemeskey via Robert Muir)
10090 
10091 * LUCENE-3396: ReusableAnalyzerBase now provides a ReuseStrategy abstraction 
10092   which controls how TokenStreamComponents are reused per request.  Two 
10093   implementations are provided - GlobalReuseStrategy which implements the
10094   current behavior of sharing components between all fields, and 
10095   PerFieldReuseStrategy which shares per field.  (Chris Male)
10096   
10097 * LUCENE-2309: Added IndexableField.tokenStream(Analyzer) which is now 
10098   responsible for creating the TokenStreams for Fields when they are to 
10099   be indexed.  (Chris Male)
10100   
10101 * LUCENE-3433: Added random access for non RAM resident IndexDocValues. RAM
10102   resident and disk resident IndexDocValues are now exposed via the Source
10103   interface. ValuesEnum has been removed in favour of Source. (Simon Willnauer)
10104 
10105 * LUCENE-1536: Filters can now be applied down-low, if their DocIdSet implements
10106   a new bits() method, returning all documents in a random access way. If the
10107   DocIdSet is not too sparse, it will be passed as acceptDocs down to the Scorer
10108   as replacement for AtomicReader's live docs.
10109   In addition, FilteredQuery backs now IndexSearcher's filtering search methods.
10110   Using FilteredQuery you can chain Filters in a very performant way
10111   [new FilteredQuery(new FilteredQuery(query, filter1), filter2)], which was not
10112   possible with IndexSearcher's methods. FilteredQuery also allows to override
10113   the heuristics used to decide if filtering should be done random access or
10114   using a conjunction on DocIdSet's iterator().
10115   (Mike McCandless, Uwe Schindler, Robert Muir, Chris Male, Yonik Seeley,
10116   Jason Rutherglen, Paul Elschot)
10117 
10118 * LUCENE-3638: Added sugar methods to IndexReader and IndexSearcher to
10119   load only certain fields when loading a document.  (Peter Chang via
10120   Mike McCandless)
10121   
10122 * LUCENE-3628: Norms are represented as DocValues. AtomicReader exposes
10123   a #normValues(String) method to obtain norms per field. (Simon Willnauer) 
10124 
10125 * LUCENE-3687: Similarity#computeNorm(FieldInvertState, Norm) allows to compute
10126   norm values or arbitrary precision. Instead of returning a fixed single byte
10127   value, custom similarities can now set a integer, float or byte value to the
10128   given Norm object. (Simon Willnauer)  
10129 
10130 * LUCENE-2604, LUCENE-4103: Added RegexpQuery support to contrib/queryparser.
10131   (Simon Willnauer, Robert Muir, Daniel Truemper)
10132 
10133 * LUCENE-2373: Added a Codec implementation that works with append-only
10134   filesystems (such as e.g. Hadoop DFS). SegmentInfos writing/reading
10135   code is refactored to support append-only FS, and to allow for future
10136   customization of per-segment information. (Andrzej Bialecki)
10137 
10138 * LUCENE-2479: Added ability to provide a sort comparator for spelling suggestions along
10139   with two implementations.  The existing comparator (score, then frequency) is the default (Grant Ingersoll)
10140 
10141 * LUCENE-2608: Added the ability to specify the accuracy at method time in the SpellChecker.  The per class
10142   method is also still available.  (Grant Ingersoll)
10143     
10144 * LUCENE-2507: Added DirectSpellChecker, which retrieves correction candidates directly 
10145   from the term dictionary using levenshtein automata.  (Robert Muir)
10146 
10147 * LUCENE-3527: Add LuceneLevenshteinDistance, which computes string distance in a compatible
10148   way as DirectSpellChecker. This can be used to merge top-N results from more than one
10149   SpellChecker.  (James Dyer via Robert Muir)
10150 
10151 * LUCENE-3496: Support grouping by DocValues. (Martijn van Groningen)
10152 
10153 * LUCENE-2795: Generified DirectIOLinuxDirectory to work across any
10154   unix supporting the O_DIRECT flag when opening a file (tested on
10155   Linux and OS X but likely other Unixes will work), and improved it
10156   so it can be used for indexing and searching.  The directory uses
10157   direct IO when doing large merges to avoid  unnecessarily evicting
10158   cached IO pages due to large merges.  (Varun Thacker, Mike
10159   McCandless)
10160 
10161 * LUCENE-3827: DocsAndPositionsEnum from MemoryIndex implements
10162   start/endOffset, if offsets are indexed. (Alan Woodward via Mike
10163   McCandless)
10164 
10165 * LUCENE-3802, LUCENE-3856: Support for grouped faceting. (Martijn van Groningen)
10166 
10167 * LUCENE-3444: Added a second pass grouping collector that keeps track of distinct
10168   values for a specified field for the top N group. (Martijn van Groningen)
10169 
10170 * LUCENE-3778: Added a grouping utility class that makes it easier to use result
10171   grouping for pure Lucene apps. (Martijn van Groningen)
10172 
10173 * LUCENE-2341: A new analysis/ filter: Morfologik - a dictionary-driven lemmatizer 
10174   (accurate stemmer) for Polish (includes morphosyntactic annotations).
10175   (Michał Dybizbański, Dawid Weiss)
10176 
10177 * LUCENE-2413: Consolidated Lucene/Solr analysis components into analysis/common. 
10178   New features from Solr now available to Lucene users include:
10179    - o.a.l.analysis.commongrams: Constructs n-grams for frequently occurring terms
10180      and phrases. 
10181    - o.a.l.analysis.charfilter.HTMLStripCharFilter: CharFilter that strips HTML 
10182      constructs.
10183    - o.a.l.analysis.miscellaneous.WordDelimiterFilter: TokenFilter that splits words 
10184      into subwords and performs optional transformations on subword groups.
10185    - o.a.l.analysis.miscellaneous.RemoveDuplicatesTokenFilter: TokenFilter which 
10186      filters out Tokens at the same position and Term text as the previous token.
10187    - o.a.l.analysis.miscellaneous.TrimFilter: Trims leading and trailing whitespace 
10188      from Tokens in the stream.
10189    - o.a.l.analysis.miscellaneous.KeepWordFilter: A TokenFilter that only keeps tokens 
10190      with text contained in the required words (inverse of StopFilter).
10191    - o.a.l.analysis.miscellaneous.HyphenatedWordsFilter: A TokenFilter that puts 
10192      hyphenated words broken into two lines back together.
10193    - o.a.l.analysis.miscellaneous.CapitalizationFilter: A TokenFilter that applies
10194      capitalization rules to tokens.
10195    - o.a.l.analysis.pattern: Package for pattern-based analysis, containing a 
10196      CharFilter, Tokenizer, and TokenFilter for transforming text with regexes.
10197    - o.a.l.analysis.synonym.SynonymFilter: A synonym filter that supports multi-word
10198      synonyms.
10199    - o.a.l.analysis.phonetic: Package for phonetic search, containing various
10200      phonetic encoders such as Double Metaphone.
10201 
10202    Some existing analysis components changed packages:
10203     - o.a.l.analysis.KeywordAnalyzer -> o.a.l.analysis.core.KeywordAnalyzer
10204     - o.a.l.analysis.KeywordTokenizer -> o.a.l.analysis.core.KeywordTokenizer
10205     - o.a.l.analysis.LetterTokenizer -> o.a.l.analysis.core.LetterTokenizer
10206     - o.a.l.analysis.LowerCaseFilter -> o.a.l.analysis.core.LowerCaseFilter
10207     - o.a.l.analysis.LowerCaseTokenizer -> o.a.l.analysis.core.LowerCaseTokenizer
10208     - o.a.l.analysis.SimpleAnalyzer -> o.a.l.analysis.core.SimpleAnalyzer
10209     - o.a.l.analysis.StopAnalyzer -> o.a.l.analysis.core.StopAnalyzer
10210     - o.a.l.analysis.StopFilter -> o.a.l.analysis.core.StopFilter
10211     - o.a.l.analysis.WhitespaceAnalyzer -> o.a.l.analysis.core.WhitespaceAnalyzer
10212     - o.a.l.analysis.WhitespaceTokenizer -> o.a.l.analysis.core.WhitespaceTokenizer
10213     - o.a.l.analysis.PorterStemFilter -> o.a.l.analysis.en.PorterStemFilter
10214     - o.a.l.analysis.ASCIIFoldingFilter -> o.a.l.analysis.miscellaneous.ASCIIFoldingFilter
10215     - o.a.l.analysis.ISOLatin1AccentFilter -> o.a.l.analysis.miscellaneous.ISOLatin1AccentFilter
10216     - o.a.l.analysis.KeywordMarkerFilter -> o.a.l.analysis.miscellaneous.KeywordMarkerFilter
10217     - o.a.l.analysis.LengthFilter -> o.a.l.analysis.miscellaneous.LengthFilter
10218     - o.a.l.analysis.PerFieldAnalyzerWrapper -> o.a.l.analysis.miscellaneous.PerFieldAnalyzerWrapper
10219     - o.a.l.analysis.TeeSinkTokenFilter -> o.a.l.analysis.sinks.TeeSinkTokenFilter
10220     - o.a.l.analysis.CharFilter -> o.a.l.analysis.charfilter.CharFilter
10221     - o.a.l.analysis.BaseCharFilter -> o.a.l.analysis.charfilter.BaseCharFilter
10222     - o.a.l.analysis.MappingCharFilter -> o.a.l.analysis.charfilter.MappingCharFilter
10223     - o.a.l.analysis.NormalizeCharMap -> o.a.l.analysis.charfilter.NormalizeCharMap
10224     - o.a.l.analysis.CharArraySet -> o.a.l.analysis.util.CharArraySet
10225     - o.a.l.analysis.CharArrayMap -> o.a.l.analysis.util.CharArrayMap
10226     - o.a.l.analysis.ReusableAnalyzerBase -> o.a.l.analysis.util.ReusableAnalyzerBase
10227     - o.a.l.analysis.StopwordAnalyzerBase -> o.a.l.analysis.util.StopwordAnalyzerBase
10228     - o.a.l.analysis.WordListLoader -> o.a.l.analysis.util.WordListLoader
10229     - o.a.l.analysis.CharTokenizer -> o.a.l.analysis.util.CharTokenizer
10230     - o.a.l.util.CharacterUtils -> o.a.l.analysis.util.CharacterUtils
10231 
10232    All analyzers in contrib/analyzers and contrib/icu were moved to the
10233    analysis/ module.  The 'smartcn' and 'stempel' components now depend on 'common'.
10234    (Chris Male, Robert Muir)
10235 
10236 * LUCENE-4004: Add DisjunctionMaxQuery support to the xml query parser.
10237   (Benson Margulies via Robert Muir)
10238 
10239 * LUCENE-4025: Add maybeRefreshBlocking to ReferenceManager, to let a caller 
10240   block until the refresh logic has been executed. (Shai Erera, Mike McCandless)
10241   
10242 * LUCENE-4039: Add AddIndexesTask to benchmark, which uses IW.addIndexes. 
10243   (Shai Erera)
10244 
10245 * LUCENE-3514: Added IndexSearcher.searchAfter when Sort is used,
10246   returning results after a specified FieldDoc for deep
10247   paging.  (Mike McCandless)
10248 
10249 * LUCENE-4043: Added scoring support via score mode for query time joining.
10250   (Martijn van Groningen, Mike McCandless)
10251   
10252 * LUCENE-3523: Added oal.search.spell.WordBreakSpellChecker, which 
10253     generates suggestions by combining two or more terms and/or 
10254     breaking terms into multiple words.  See Javadocs for usage. (James Dyer)
10255 
10256 * LUCENE-4019: Added improved parsing of Hunspell Dictionaries so those
10257   rules missing the required number of parameters either ignored or 
10258   cause a ParseException (depending on whether strict parsing is enabled).
10259   (Luca Cavanna via Chris Male) 
10260 
10261 * LUCENE-3440: Add ordered fragments feature with IDF-weighted terms for FVH.
10262   (Sebastian Lutze via Koji Sekiguchi)
10263 
10264 * LUCENE-4082: Added explain to ToParentBlockJoinQuery.
10265   (Christoph Kaser, Martijn van Groningen)
10266 
10267 * LUCENE-4108: add replaceTaxonomy to DirectoryTaxonomyWriter, which replaces
10268   the taxonomy in place with the given one. (Shai Erera)
10269 
10270 * LUCENE-3030: new BlockTree terms dictionary (used by the default
10271   Lucene40 postings format) uses less RAM (for the terms index) and
10272   disk space (for all terms and metadata) and gives sizable
10273   performance gains for terms dictionary intensive operations like
10274   FuzzyQuery, direct spell checker and primary-key lookup (Mike
10275   McCandless).
10276 
10277 Optimizations
10278 
10279 * LUCENE-2588: Don't store unnecessary suffixes when writing the terms
10280   index, saving RAM in IndexReader; change default terms index
10281   interval from 128 to 32, because the terms index now requires much
10282   less RAM.  (Robert Muir, Mike McCandless)
10283 
10284 * LUCENE-2669: Optimize NumericRangeQuery.NumericRangeTermsEnum to
10285   not seek backwards when a sub-range has no terms. It now only seeks
10286   when the current term is less than the next sub-range's lower end.
10287   (Uwe Schindler, Mike McCandless)
10288   
10289 * LUCENE-2694: Optimize MultiTermQuery to be single pass for Term lookups.
10290   MultiTermQuery now stores TermState per leaf reader during rewrite to re-
10291   seek the term dictionary in TermQuery / TermWeight.
10292   (Simon Willnauer, Mike McCandless, Robert Muir)
10293 
10294 * LUCENE-3292: IndexWriter no longer shares the same SegmentReader
10295   instance for merging and NRT readers, which enables directory impls
10296   to separately tune IO flags for each.  (Varun Thacker, Simon
10297   Willnauer, Mike McCandless)
10298 
10299 * LUCENE-3328: BooleanQuery now uses a specialized ConjunctionScorer if all
10300   boolean clauses are required and instances of TermQuery. 
10301   (Simon Willnauer, Robert Muir)
10302   
10303 * LUCENE-3643: FilteredQuery and IndexSearcher.search(Query, Filter,...)
10304   now optimize the special case query instanceof MatchAllDocsQuery to
10305   execute as ConstantScoreQuery.  (Uwe Schindler)
10306 
10307 * LUCENE-3509: Added fasterButMoreRam option for docvalues. This option controls whether the space for packed ints
10308   should be rounded up for better performance. This option only applies for docvalues types bytes fixed sorted
10309   and bytes var sorted. (Simon Willnauer, Martijn van Groningen)
10310   
10311 * LUCENE-3795: Replace contrib/spatial with modules/spatial.  This includes 
10312   a basic spatial strategy interface.  (David Smiley, Chris Male, ryan)
10313     
10314 * LUCENE-3932: Lucene3x codec loads terms index faster, by
10315   pre-allocating the packed ints array based on the .tii file size
10316   (Sean Bridges via Mike McCandless)
10317 
10318 * LUCENE-3468: Replaced last() and remove() with pollLast() in
10319   FirstPassGroupingCollector (Martijn van Groningen)
10320 
10321 * LUCENE-3830: Changed MappingCharFilter/NormalizeCharMap to use an
10322   FST under the hood, which requires less RAM.  NormalizeCharMap no
10323   longer accepts empty string match (it did previously, but ignored
10324   it).  (Dawid Weiss, Mike McCandless)
10325 
10326 * LUCENE-4061: improve synchronization in DirectoryTaxonomyWriter.addCategory
10327   and few general improvements to DirectoryTaxonomyWriter.
10328   (Shai Erera, Gilad Barkai)
10329 
10330 * LUCENE-4062: Add new aligned packed bits impls for faster lookup
10331   performance; add float acceptableOverheadRatio to getWriter and
10332   getMutable API to give packed ints freedom to pick faster
10333   implementations (Adrien Grand via Mike McCandless)
10334 
10335 * LUCENE-2357: Reduce transient RAM usage when merging segments in
10336   IndexWriter. (Adrien Grand)
10337 
10338 * LUCENE-4098: Add bulk get/set methods to PackedInts (Adrien Grand
10339   via Mike McCandless)
10340   
10341 * LUCENE-4156: DirectoryTaxonomyWriter.getSize is no longer synchronized.
10342   (Shai Erera, Sivan Yogev)
10343 
10344 * LUCENE-4163: Improve concurrency of MMapIndexInput.clone() by using
10345   the new WeakIdentityMap on top of a ConcurrentHashMap to manage
10346   the cloned instances. WeakIdentityMap was extended to support
10347   iterating over its keys.  (Uwe Schindler)
10348 
10349 Bug fixes
10350 
10351 * LUCENE-2803: The FieldCache can miss values if an entry for a reader
10352   with more document deletions is requested before a reader with fewer
10353   deletions, provided they share some segments. (yonik)
10354 
10355 * LUCENE-2645: Fix false assertion error when same token was added one
10356   after another with 0 posIncr.  (David Smiley, Kurosaka Teruhiko via Mike
10357   McCandless)
10358 
10359 * LUCENE-3348: Fix thread safety hazards in IndexWriter that could
10360   rarely cause deletions to be incorrectly applied.  (Yonik Seeley,
10361   Simon Willnauer, Mike McCandless)
10362 
10363 * LUCENE-3515: Fix terrible merge performance versus 3.x, especially
10364   when the directory isn't MMapDirectory, due to failing to reuse
10365   DocsAndPositionsEnum while merging (Marc Sturlese, Erick Erickson,
10366   Robert Muir, Simon Willnauer, Mike McCandless)
10367 
10368 * LUCENE-3589: BytesRef copy(short) didn't set length.
10369   (Peter Chang via Robert Muir)
10370 
10371 * LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
10372   not lowercasing the key before checking for the tag (Adriano Crestani)
10373 
10374 * LUCENE-3890: Fixed NPE for grouped faceting on multi-valued fields.
10375   (Michael McCandless, Martijn van Groningen)
10376 
10377 * LUCENE-2945: Fix hashCode/equals for surround query parser generated queries.
10378   (Paul Elschot, Simon Rosenthal, gsingers via ehatcher)
10379 
10380 * LUCENE-3971: MappingCharFilter could return invalid final token position.
10381   (Dawid Weiss, Robert Muir)
10382 
10383 * LUCENE-3820: PatternReplaceCharFilter could return invalid token positions. 
10384   (Dawid Weiss)
10385 
10386 * LUCENE-3969: Throw IAE on bad arguments that could cause confusing errors in
10387   CompoundWordTokenFilterBase, PatternTokenizer, PositionFilter,
10388   SnowballFilter, PathHierarchyTokenizer, ReversePathHierarchyTokenizer, 
10389   WikipediaTokenizer, and KeywordTokenizer. ShingleFilter and 
10390   CommonGramsFilter now populate PositionLengthAttribute. Fixed
10391   PathHierarchyTokenizer to reset() all state. Protect against AIOOBE in
10392   ReversePathHierarchyTokenizer if skip is large. Fixed wrong final
10393   offset calculation in PathHierarchyTokenizer. 
10394   (Mike McCandless, Uwe Schindler, Robert Muir)
10395 
10396 * LUCENE-4060: Fix a synchronization bug in 
10397   DirectoryTaxonomyWriter.addTaxonomies(). Also, the method has been renamed to
10398   addTaxonomy and now takes only one Directory and one OrdinalMap.
10399   (Shai Erera, Gilad Barkai)
10400 
10401 * LUCENE-3590: Fix AIOOBE in BytesRef/CharsRef copyBytes/copyChars when 
10402   offset is nonzero, fix off-by-one in CharsRef.subSequence, and fix
10403   CharsRef's CharSequence methods to throw exceptions in boundary cases
10404   to properly meet the specification.  (Robert Muir)
10405 
10406 * LUCENE-4084: Attempting to reuse a single IndexWriterConfig instance
10407   across more than one IndexWriter resulted in a cryptic exception.
10408   This is now fixed, but requires that certain members of
10409   IndexWriterConfig (MergePolicy, FlushPolicy,
10410   DocumentsWriterThreadPool) implement clone.  (Robert Muir, Simon
10411   Willnauer, Mike McCandless)
10412 
10413 * LUCENE-4079: Fixed loading of Hunspell dictionaries that use aliasing (AF rules)
10414   (Ludovic Boutros via Chris Male)
10415 
10416 * LUCENE-4077: Expose the max score and per-group scores from
10417   ToParentBlockJoinCollector (Christoph Kaser, Mike McCandless)
10418 
10419 * LUCENE-4114: Fix int overflow bugs in BYTES_FIXED_STRAIGHT and
10420   BYTES_FIXED_DEREF doc values implementations (Walt Elder via Mike McCandless).
10421 
10422 * LUCENE-4147: Fixed thread safety issues when rollback() and commit()
10423   are called simultaneously.  (Simon Willnauer, Mike McCandless)
10424 
10425 * LUCENE-4165: Removed closing of the Reader used to read the affix file in
10426   HunspellDictionary.  Consumers are now responsible for closing all InputStreams
10427   once the Dictionary has been instantiated. (Torsten Krah, Uwe Schindler, Chris Male)
10428 
10429 Documentation
10430 
10431 * LUCENE-3958: Javadocs corrections for IndexWriter.
10432   (Iulius Curt via Robert Muir)
10433 
10434 Build
10435 
10436 * LUCENE-4047: Cleanup of LuceneTestCase: moved blocks of initialization/ cleanup
10437   code into JUnit instance and class rules. (Dawid Weiss)
10438 
10439 * LUCENE-4016: Require ANT 1.8.2+ for the build. 
10440 
10441 * LUCENE-3808: Refactoring of testing infrastructure to use randomizedtesting
10442   package: http://labs.carrotsearch.com/randomizedtesting.html (Dawid Weiss)
10443 
10444 * LUCENE-3964: Added target stage-maven-artifacts, which stages
10445   Maven release artifacts to a Maven staging repository in preparation
10446   for release.  (Steve Rowe)
10447 
10448 * LUCENE-2845: Moved contrib/benchmark to lucene/benchmark.
10449 
10450 * LUCENE-2995: Moved contrib/spellchecker into lucene/suggest.
10451 
10452 * LUCENE-3285: Moved contrib/queryparser into lucene/queryparser
10453 
10454 * LUCENE-3285: Moved contrib/xml-query-parser's demo into lucene/demo
10455 
10456 * LUCENE-3271: Moved contrib/queries BooleanFilter, BoostingQuery, 
10457   ChainedFilter, FilterClause and TermsFilter into lucene/queries
10458 
10459 * LUCENE-3381: Moved contrib/queries regex.*, DuplicateFilter,
10460   FuzzyLikeThisQuery and SlowCollated* into lucene/sandbox.
10461   Removed contrib/queries.
10462 
10463 * LUCENE-3286: Moved remainder of contrib/xml-query-parser to lucene/queryparser.
10464   Classes now found at org.apache.lucene.queryparser.xml.*
10465 
10466 * LUCENE-4059: Improve ANT task prepare-webpages (used by documentation
10467   tasks) to correctly encode build file names as URIs for later processing by
10468   XSL.  (Greg Bowyer, Uwe Schindler)
10469 
10470 
10471 ======================= Lucene 3.6.2 =======================
10472 
10473 Bug Fixes
10474 
10475 * LUCENE-4234: Exception when FacetsCollector is used with ScoreFacetRequest,
10476   and the number of matching documents is too large. (Gilad Barkai via Shai Erera)
10477 
10478 * LUCENE-2686, LUCENE-3505, LUCENE-4401: Fix BooleanQuery scorers to
10479   return correct freq().
10480   (Koji Sekiguchi, Mike McCandless, Liu Chao, Robert Muir)
10481 
10482 * LUCENE-2501: Fixed rare thread-safety issue that could cause
10483   ArrayIndexOutOfBoundsException inside ByteBlockPool (Robert Muir,
10484   Mike McCandless)
10485 
10486 * LUCENE-4297: BooleanScorer2 would multiply the coord() factor
10487   twice for conjunctions: for most users this is no problem, but
10488   if you had a customized Similarity that returned something other
10489   than 1 when overlap == maxOverlap (always the case for conjunctions),
10490   then the score would be incorrect.  (Pascal Chollet, Robert Muir)
10491 
10492 * LUCENE-4300: BooleanQuery's rewrite was not always safe: if you
10493   had a custom Similarity where coord(1,1) != 1F, then the rewritten
10494   query would be scored differently.  (Robert Muir)
10495 
10496 * LUCENE-4398: If you index many different field names in your
10497   documents then due to a bug in how it measures its RAM
10498   usage, IndexWriter would flush each segment too early eventually
10499   reaching the point where it flushes after every doc.  (Tim Smith via
10500   Mike McCandless)
10501 
10502 * LUCENE-4411: when sampling is enabled for a FacetRequest, its depth
10503   parameter is reset to the default (1), even if set otherwise.
10504   (Gilad Barkai via Shai Erera)
10505 
10506 * LUCENE-4635: Fixed ArrayIndexOutOfBoundsException when in-memory
10507   terms index requires more than 2.1 GB RAM (indices with billions of
10508   terms).  (Tom Burton-West via Mike McCandless)
10509 
10510 Documentation
10511 
10512 * LUCENE-4302: Fix facet userguide to have HTML loose doctype like
10513   all other javadocs.  (Karl Nicholas via Uwe Schindler)
10514 
10515 
10516 ======================= Lucene 3.6.1 =======================
10517 More information about this release, including any errata related to the 
10518 release notes, upgrade instructions, or other changes may be found online at:
10519    https://wiki.apache.org/lucene-java/Lucene3.6.1
10520 
10521 Bug Fixes
10522 
10523 * LUCENE-3969: Throw IAE on bad arguments that could cause confusing 
10524   errors in KeywordTokenizer. 
10525   (Uwe Schindler, Mike McCandless, Robert Muir)
10526 
10527 * LUCENE-3971: MappingCharFilter could return invalid final token position.
10528   (Dawid Weiss, Robert Muir)
10529 
10530 * LUCENE-4023: DisjunctionMaxScorer now implements visitSubScorers().
10531   (Uwe Schindler)
10532 
10533 * LUCENE-2566: + - operators allow any amount of whitespace (yonik, janhoy)
10534 
10535 * LUCENE-3590: Fix AIOOBE in BytesRef/CharsRef copyBytes/copyChars when 
10536   offset is nonzero, fix off-by-one in CharsRef.subSequence, and fix
10537   CharsRef's CharSequence methods to throw exceptions in boundary cases
10538   to properly meet the specification.  (Robert Muir)
10539 
10540 * LUCENE-4222: TieredMergePolicy.getFloorSegmentMB was returning the
10541   size in bytes not MB (Chris Fuller via Mike McCandless)
10542 
10543 API Changes
10544 
10545 * LUCENE-4023: Changed the visibility of Scorer#visitSubScorers() to
10546   public, otherwise it's impossible to implement Scorers outside
10547   the Lucene package.  (Uwe Schindler)
10548 
10549 Optimizations
10550 
10551 * LUCENE-4163: Improve concurrency of MMapIndexInput.clone() by using
10552   the new WeakIdentityMap on top of a ConcurrentHashMap to manage
10553   the cloned instances. WeakIdentityMap was extended to support
10554   iterating over its keys.  (Uwe Schindler)
10555 
10556 Tests
10557 
10558 * LUCENE-3873: add MockGraphTokenFilter, testing analyzers with
10559   random graph tokens.  (Mike McCandless)
10560 
10561 * LUCENE-3968: factor out LookaheadTokenFilter from 
10562   MockGraphTokenFilter (Mike McCandless)
10563 
10564 
10565 ======================= Lucene 3.6.0 =======================
10566 More information about this release, including any errata related to the 
10567 release notes, upgrade instructions, or other changes may be found online at:
10568    https://wiki.apache.org/lucene-java/Lucene3.6
10569 
10570 Changes in backwards compatibility policy
10571 
10572 * LUCENE-3594: The protected inner class (never intended to be visible)
10573   FieldCacheTermsFilter.FieldCacheTermsFilterDocIdSet was removed and
10574   replaced by another internal implementation.  (Uwe Schindler)
10575 
10576 * LUCENE-3620: FilterIndexReader now overrides all methods of IndexReader that 
10577   it should (note that some are still not overridden, as they should be 
10578   overridden by sub-classes only). In the process, some methods of IndexReader 
10579   were made final. This is not expected to affect many apps, since these methods
10580   already delegate to abstract methods, which you had to already override 
10581   anyway. (Shai Erera)
10582 
10583 * LUCENE-3636: Added SearcherFactory, used by SearcherManager and NRTManager
10584   to create new IndexSearchers. You can provide your own implementation to
10585   warm new searchers, set an ExecutorService, set a custom Similarity, or
10586   even return your own subclass of IndexSearcher. The SearcherWarmer and
10587   ExecutorService parameters on these classes were removed, as they are
10588   subsumed by SearcherFactory.  (Shai Erera, Mike McCandless, Robert Muir)
10589 
10590 * LUCENE-3644: The expert ReaderFinishedListener api suffered problems (propagated
10591   down to subreaders, but was not called on SegmentReaders, unless they were
10592   the owner of the reader core, and other ambiguities). The API is revised:
10593   You can set ReaderClosedListeners on any IndexReader, and onClose is called
10594   when that reader is closed.  SegmentReader has CoreClosedListeners that you
10595   can register to know when a shared reader core is closed.  
10596   (Uwe Schindler, Mike McCandless, Robert Muir)
10597 
10598 * LUCENE-3652: The package org.apache.lucene.messages was moved to
10599   contrib/queryparser. If you have used those classes in your code
10600   just add the lucene-queryparser.jar file to your classpath.
10601   (Uwe Schindler)
10602 
10603 * LUCENE-3681: FST now stores labels for BYTE2 input type as 2 bytes
10604   instead of vInt; this can make FSTs smaller and faster, but it is a
10605   break in the binary format so if you had built and saved any FSTs
10606   then you need to rebuild them. (Robert Muir, Mike McCandless)
10607 
10608 * LUCENE-3679: The expert IndexReader.getFieldNames(FieldOption) API
10609   has been removed and replaced with the experimental getFieldInfos
10610   API.  All IndexReader subclasses must implement getFieldInfos.
10611   (Mike McCandless)
10612 
10613 * LUCENE-3695: Move confusing add(X) methods out of FST.Builder into
10614   FST.Util.  (Robert Muir, Mike McCandless)
10615 
10616 * LUCENE-3701: Added an additional argument to the expert FST.Builder
10617   ctor to take FreezeTail, which you can use to (very-expertly) customize
10618   the FST construction process. Pass null if you want the default
10619   behavior.  Added seekExact() to FSTEnum, and added FST.save/read 
10620   from a File. (Mike McCandless, Dawid Weiss, Robert Muir)
10621   
10622 * LUCENE-3712: Removed unused and untested ReaderUtil#subReader methods.
10623   (Uwe Schindler)
10624   
10625 * LUCENE-3672: Deprecate Directory.fileModified,
10626   IndexCommit.getTimestamp and .getVersion and
10627   IndexReader.lastModified and getCurrentVersion (Andrzej Bialecki,
10628   Robert Muir, Mike McCandless)
10629 
10630 * LUCENE-3760: In IndexReader/DirectoryReader, deprecate static
10631   methods getCurrentVersion and getCommitUserData, and non-static
10632   method getCommitUserData (use getIndexCommit().getUserData()
10633   instead).  (Ryan McKinley, Robert Muir, Mike McCandless)
10634   
10635 * LUCENE-3867: Deprecate instance creation of RamUsageEstimator, instead
10636   the new static method sizeOf(Object) should be used. As the algorithm
10637   is now using Hotspot(TM) internals (reference size, header sizes,
10638   object alignment), the abstract o.a.l.util.MemoryModel class was
10639   completely removed (without replacement). The new static methods
10640   no longer support String intern-ness checking, interned strings
10641   now count to memory usage as any other Java object.
10642   (Dawid Weiss, Uwe Schindler, Shai Erera)
10643   
10644 * LUCENE-3738: All readXxx methods in BufferedIndexInput were made
10645   final. Subclasses should only override protected readInternal /
10646   seekInternal.  (Uwe Schindler)
10647 
10648 * LUCENE-2599: Deprecated the spatial contrib module, which was buggy and not
10649   well maintained.  Lucene 4 includes a new spatial module that replaces this.
10650   (David Smiley, Ryan McKinley, Chris Male)
10651   
10652 Changes in Runtime Behavior
10653 
10654 * LUCENE-3796, SOLR-3241: Throw an exception if you try to set an index-time
10655   boost on a field that omits norms. Because the index-time boost
10656   is multiplied into the norm, previously your boost would be 
10657   silently discarded.  (Tomás Fernández Löbbe, Hoss Man, Robert Muir)
10658 
10659 * LUCENE-3848: Fix tokenstreams to not produce a stream with an initial
10660   position increment of 0: which is out of bounds (overlapping with a
10661   non-existent previous term). Consumers such as IndexWriter and QueryParser
10662   still check for and silently correct this situation today, but at some point 
10663   in the future they may throw an exception.  (Mike McCandless, Robert Muir)
10664   
10665 * LUCENE-3738: DataInput/DataOutput no longer allow negative vLongs. Negative
10666   vInts are still supported (for index backwards compatibility), but
10667   should not be used in new code. The read method for negative vLongs
10668   was already broken since Lucene 3.1.
10669   (Uwe Schindler, Mike McCandless, Robert Muir)
10670   
10671 Security fixes
10672 
10673 * LUCENE-3588: Try harder to prevent SIGSEGV on cloned MMapIndexInputs:
10674   Previous versions of Lucene could SIGSEGV the JVM if you try to access
10675   the clone of an IndexInput retrieved from MMapDirectory. This security fix
10676   prevents this as best as it can by throwing AlreadyClosedException
10677   also on clones.  (Uwe Schindler, Robert Muir)
10678 
10679 API Changes
10680 
10681 * LUCENE-3606: IndexReader will be made read-only in Lucene 4.0, so all
10682   methods allowing to delete or undelete documents using IndexReader were
10683   deprecated; you should use IndexWriter now. Consequently
10684   IndexReader.commit() and all open(), openIfChanged(), clone() methods
10685   taking readOnly booleans (or IndexDeletionPolicy instances) were
10686   deprecated. IndexReader.setNorm() is superfluous and was deprecated.
10687   If you have to change per-document boost use CustomScoreQuery.
10688   If you want to dynamically change norms (boost *and* length norm) at
10689   query time, wrap your IndexReader using FilterIndexReader, overriding
10690   FilterIndexReader.norms(). To persist the changes on disk, copy the
10691   FilteredIndexReader to a new index using IndexWriter.addIndexes().
10692   In Lucene 4.0, SimilarityProvider will allow you to customize scoring
10693   using external norms, too.  (Uwe Schindler, Robert Muir)
10694 
10695 * LUCENE-3735: PayloadProcessorProvider was changed to return a
10696   ReaderPayloadProcessor instead of DirPayloadProcessor. The selection
10697   of the provider to return for the factory is now based on the IndexReader
10698   to be merged. To mimic the old behaviour, just use IndexReader.directory()
10699   for choosing the provider by Directory.  (Uwe Schindler)
10700 
10701 * LUCENE-3765: Deprecated StopFilter ctor that took ignoreCase, because
10702   in some cases (if the set is a CharArraySet), the argument is ignored.
10703   Deprecated StandardAnalyzer and ClassicAnalyzer ctors that take File,
10704   please use the Reader ctor instead.  (Robert Muir)
10705 
10706 * LUCENE-3766: Deprecate no-arg ctors of Tokenizer. Tokenizers are
10707   TokenStreams with Readers: tokenizers with null Readers will not be
10708   supported in Lucene 4.0, just use a TokenStream.  
10709   (Mike McCandless, Robert Muir)
10710 
10711 * LUCENE-3769: Simplified NRTManager by requiring applyDeletes to be
10712   passed to ctor only; if an app needs to mix and match it's free to
10713   create two NRTManagers (one always applying deletes and the other
10714   never applying deletes).  (MJB, Shai Erera, Mike McCandless)
10715 
10716 * LUCENE-3761: Generalize SearcherManager into an abstract ReferenceManager.
10717   SearcherManager remains a concrete class, but due to the refactoring, the 
10718   method maybeReopen has been deprecated in favor of maybeRefresh().
10719   (Shai Erera, Mike McCandless, Simon Willnauer)
10720 
10721 * LUCENE-3776: You now acquire/release the IndexSearcher directly from
10722   NRTManager.  (Mike McCandless)
10723   
10724 New Features
10725 
10726 * LUCENE-3593: Added a FieldValueFilter that accepts all documents that either
10727   have at least one or no value at all in a specific field. (Simon Willnauer,
10728   Uwe Schindler, Robert Muir)
10729 
10730 * LUCENE-3586: CheckIndex and IndexUpgrader allow you to specify the
10731   specific FSDirectory implementation to use (with the new -dir-impl
10732   command-line option).  (Luca Cavanna via Mike McCandless)
10733 
10734 * LUCENE-3634: IndexReader's static main method was moved to a new
10735   tool, CompoundFileExtractor, in contrib/misc.  (Robert Muir, Mike
10736   McCandless)
10737 
10738 * LUCENE-995: The QueryParser now interprets * as an open end for range
10739   queries.  Literal asterisks may be represented by quoting or escaping
10740   (i.e. \* or "*")  Custom QueryParser subclasses overriding getRangeQuery()
10741   will be passed null for any open endpoint. (Ingo Renner, Adriano
10742   Crestani, yonik, Mike McCandless 
10743 
10744 * LUCENE-3121: Add sugar reverse lookup (given an output, find the
10745   input mapping to it) for FSTs that have strictly monotonic long
10746   outputs (such as an ord).  (Mike McCandless)
10747   
10748 * LUCENE-3671: Add TypeTokenFilter that filters tokens based on
10749   their TypeAttribute.  (Tommaso Teofili via Uwe Schindler)
10750 
10751 * LUCENE-3690,LUCENE-3913: Added HTMLStripCharFilter, a CharFilter that strips
10752   HTML markup. (Steve Rowe)
10753 
10754 * LUCENE-3725: Added optional packing to FST building; this uses extra
10755   RAM during building but results in a smaller FST.  (Mike McCandless)
10756 
10757 * LUCENE-3714: Add top N shortest cost paths search for FST.
10758   (Robert Muir, Dawid Weiss, Mike McCandless)
10759 
10760 * LUCENE-3789: Expose MTQ TermsEnum via RewriteMethod for non package private
10761   access (Simon Willnauer)
10762   
10763 * LUCENE-3881: Added UAX29URLEmailAnalyzer: a standard analyzer that recognizes
10764   URLs and emails. (Steve Rowe)
10765 
10766 Bug fixes
10767 
10768 * LUCENE-3595: Fixed FieldCacheRangeFilter and FieldCacheTermsFilter
10769   to correctly respect deletions on reopened SegmentReaders. Factored out
10770   FieldCacheDocIdSet to be a top-level class.  (Uwe Schindler, Simon Willnauer)
10771 
10772 * LUCENE-3627: Don't let an errant 0-byte segments_N file corrupt the index.
10773   (Ken McCracken via Mike McCandless)
10774 
10775 * LUCENE-3630: The internal method MultiReader.doOpenIfChanged(boolean doClone)
10776   was overriding IndexReader.doOpenIfChanged(boolean readOnly), so changing the
10777   contract of the overridden method. This method was renamed and made private.
10778   In ParallelReader the bug was not existent, but the implementation method
10779   was also made private.  (Uwe Schindler)
10780 
10781 * LUCENE-3641: Fixed MultiReader to correctly propagate readerFinishedListeners
10782   to clones/reopened readers.  (Uwe Schindler)
10783 
10784 * LUCENE-3642, SOLR-2891, LUCENE-3717: Fixed bugs in CharTokenizer, n-gram tokenizers/filters, 
10785   compound token filters, thai word filter, icutokenizer, pattern analyzer, 
10786   wikipediatokenizer, and smart chinese where they would create invalid offsets in 
10787   some situations, leading to problems in highlighting.  
10788   (Max Beutel, Edwin Steiner via Robert Muir)
10789 
10790 * LUCENE-3639: TopDocs.merge was incorrectly setting TopDocs.maxScore to
10791   Float.MIN_VALUE when it should be Float.NaN, when there were 0
10792   hits.  Improved age calculation in SearcherLifetimeManager, to have
10793   double precision and to compute age to be how long ago the searcher
10794   was replaced with a new searcher (Mike McCandless)
10795 
10796 * LUCENE-3658: Corrected potential concurrency issues with
10797   NRTCachingDir, fixed createOutput to overwrite any previous file,
10798   and removed invalid asserts (Robert Muir, Mike McCandless)
10799 
10800 * LUCENE-3605: don't sleep in a retry loop when trying to locate the
10801   segments_N file (Robert Muir, Mike McCandless)
10802 
10803 * LUCENE-3711: SentinelIntSet with a small initial size can go into
10804   an infinite loop when expanded.  This can affect grouping using
10805   TermAllGroupsCollector or TermAllGroupHeadsCollector if instantiated with a
10806   non default small size. (Martijn van Groningen, yonik)
10807 
10808 * LUCENE-3727: When writing stored fields and term vectors, Lucene
10809   checks file sizes to detect a bug in some Sun JREs (LUCENE-1282),
10810   however, on some NFS filesystems File.length() could be stale,
10811   resulting in false errors like "fdx size mismatch while indexing".
10812   These checks now use getFilePointer instead to avoid this.
10813   (Jamir Shaikh, Mike McCandless, Robert Muir)
10814 
10815 * LUCENE-3816: Fixed problem in FilteredDocIdSet, if null was returned
10816   from the delegate DocIdSet.iterator(), which is allowed to return
10817   null by DocIdSet specification when no documents match.
10818   (Shay Banon via Uwe Schindler)
10819   
10820 * LUCENE-3821: SloppyPhraseScorer missed documents that ExactPhraseScorer finds
10821   When phrase query had repeating terms (e.g. "yes no yes")  
10822   sloppy query missed documents that exact query matched. 
10823   Fixed except when for repeating multiterms (e.g. "yes no yes|no").
10824   (Robert Muir, Doron Cohen)
10825 
10826 * LUCENE-3841: Fix CloseableThreadLocal to also purge stale entries on
10827   get(); this fixes certain cases where we were holding onto objects
10828   for dead threads for too long (Matthew Bellew, Mike McCandless)
10829 
10830 * LUCENE-3872: IndexWriter.close() now throws IllegalStateException if
10831   you call it after calling prepareCommit() without calling commit()
10832   first.  (Tim Bogaert via Mike McCandless)
10833 
10834 * LUCENE-3874: Throw IllegalArgumentException from IndexWriter (rather
10835   than producing a corrupt index), if a positionIncrement would cause
10836   integer overflow. This can happen, for example when using a buggy
10837   TokenStream that forgets to call clearAttributes() in combination
10838   with a StopFilter. (Robert Muir)
10839 
10840 * LUCENE-3876: Fix bug where positions for a document exceeding
10841   Integer.MAX_VALUE/2 would produce a corrupt index.  
10842   (Simon Willnauer, Mike McCandless, Robert Muir)
10843 
10844 * LUCENE-3880: UAX29URLEmailTokenizer now recognizes emails when the mailto:
10845   scheme is prepended. (Kai Gülzau, Steve Rowe)
10846     
10847 Optimizations
10848 
10849 * LUCENE-3653: Improve concurrency in VirtualMethod and AttributeSource by
10850   using a WeakIdentityMap based on a ConcurrentHashMap.  (Uwe Schindler,
10851   Gerrit Jansen van Vuuren)
10852 
10853 Documentation
10854 
10855 * LUCENE-3597: Fixed incorrect grouping documentation. (Martijn van Groningen,
10856   Robert Muir)
10857 
10858 * LUCENE-3926: Improve documentation of RAMDirectory, because this
10859   class is not intended to work with huge indexes. Everything beyond
10860   several hundred megabytes will waste resources (GC cycles), because
10861   it uses an internal buffer size of 1024 bytes, producing millions of 
10862   byte[1024] arrays. This class is optimized for small memory-resident
10863   indexes. It also has bad concurrency on multithreaded environments.
10864   It is recommended to materialize large indexes on disk and use
10865   MMapDirectory, which is a high-performance directory implementation
10866   working directly on the file system cache of the operating system,
10867   so copying data to Java heap space is not useful.  (Uwe Schindler,
10868   Mike McCandless, Robert Muir)
10869 
10870 Build
10871 
10872 * LUCENE-3857: exceptions from other threads in beforeclass/etc do not fail 
10873   the test (Dawid Weiss)
10874 
10875 * LUCENE-3847: LuceneTestCase will now check for modifications of System 
10876   properties before and after each test (and suite). If changes are detected,
10877   the test will fail. A rule can be used to reset system properties to
10878   before-scope state (and this has been used to make Solr tests pass).
10879   (Dawid Weiss, Uwe Schindler).
10880 
10881 * LUCENE-3228: Stop downloading external javadoc package-list files:
10882 
10883   - Added package-list files for Oracle Java javadocs and JUnit javadocs to
10884     Lucene/Solr subversion.
10885 
10886   - The Oracle Java javadocs package-list file is excluded from Lucene and
10887     Solr source release packages.
10888 
10889   - Regardless of network connectivity, javadocs built from a subversion
10890     checkout contain links to Oracle & JUnit javadocs.
10891 
10892   - Building javadocs from a source release package will download the Oracle
10893     Java package-list file if it isn't already present.
10894 
10895   - When the Oracle Java package-list file is not present and download fails,
10896     the javadocs targets will not fail the build, though an error will appear
10897     in the build log.  In this case, the built javadocs will not contain links
10898     to Oracle Java javadocs.
10899 
10900   - Links from Solr javadocs to Lucene's javadocs are enabled. When building
10901     a X.Y.Z-SNAPSHOT version, the links are to the most recently built nightly
10902     Jenkins javadocs. When building a release version, links are to the
10903     Lucene release javadocs for the same version.
10904 
10905   (Steve Rowe, hossman)
10906 
10907 * LUCENE-3753: Restructure the Lucene build system:
10908   - Created a new Lucene-internal module named "core" by moving the java/
10909     and test/ directories from lucene/src/ to lucene/core/src/.
10910   - Eliminated lucene/src/ by moving all its directories up one level.
10911   - Each internal module (core/, test-framework/, and tools/) now has its own
10912     build.xml, from which it is possible to run module-specific targets.
10913     lucene/build.xml delegates all build tasks (via
10914     <ant dir="internal-module-dir"> calls) to these modules' build.xml files.
10915   (Steve Rowe)
10916 
10917 * LUCENE-3774: Optimized and streamlined license and notice file validation
10918   by refactoring the build task into an ANT task and modifying build scripts
10919   to perform top-level checks. (Dawid Weiss, Steve Rowe, Robert Muir)
10920 
10921 * LUCENE-3762: Upgrade JUnit to 4.10, refactor state-machine of detecting 
10922   setUp/tearDown call chaining in LuceneTestCase. (Dawid Weiss, Robert Muir)
10923 
10924 * LUCENE-3944: Make the 'generate-maven-artifacts' target use filtered POMs
10925   placed under lucene/build/poms/, rather than in each module's base
10926   directory.  The 'clean' target now removes them.
10927   (Steve Rowe, Robert Muir)
10928 
10929 * LUCENE-3930: Changed build system to use Apache Ivy for retrival of 3rd
10930   party JAR files.  Please review BUILD.txt for instructions.
10931   (Robert Muir, Chris Male, Uwe Schindler, Steven Rowe, Hossman)
10932 
10933 
10934 ======================= Lucene 3.5.0 =======================
10935 
10936 Changes in backwards compatibility policy
10937 
10938 * LUCENE-3390: The first approach in Lucene 3.4.0 for missing values
10939   support for sorting had a design problem that made the missing value
10940   be populated directly into the FieldCache arrays during sorting,
10941   leading to concurrency issues. To fix this behaviour, the method
10942   signatures had to be changed:
10943   - FieldCache.getUnValuedDocs() was renamed to FieldCache.getDocsWithField()
10944     returning a Bits interface (backported from Lucene 4.0).
10945   - FieldComparator.setMissingValue() was removed and added to
10946     constructor
10947   As this is expert API, most code will not be affected.
10948   (Uwe Schindler, Doron Cohen, Mike McCandless)
10949 
10950 * LUCENE-3541: Remove IndexInput's protected copyBuf. If you want to
10951   keep a buffer in your IndexInput, do this yourself in your implementation, 
10952   and be sure to do the right thing on clone()!  (Robert Muir)
10953   
10954 * LUCENE-2822: TimeLimitingCollector now expects a counter clock instead of
10955   relying on a private daemon thread. The global time limiting clock thread
10956   has been exposed and is now lazily loaded and fully optional.
10957   TimeLimitingCollector now supports setting clock baseline manually to include
10958   prelude of a search. Previous versions set the baseline on construction time, 
10959   now baseline is set once the first IndexReader is passed to the collector
10960   unless set before. (Simon Willnauer)
10961 
10962 Changes in runtime behavior
10963 
10964 * LUCENE-3520: IndexReader.openIfChanged, when passed a near-real-time
10965   reader, will now return null if there are no changes.  The API has
10966   always reserved the right to do this; it's just that in the past for
10967   near-real-time readers it never did. (Mike McCandless)
10968 
10969 Bug fixes
10970 
10971 * LUCENE-3412: SloppyPhraseScorer was returning non-deterministic results 
10972   for queries with many repeats (Doron Cohen)
10973 
10974 * LUCENE-3421: PayloadTermQuery's explain was wrong when includeSpanScore=false.
10975   (Edward Drapkin via Robert Muir)
10976 
10977 * LUCENE-3432: IndexWriter.expungeDeletes with TieredMergePolicy
10978   should ignore the maxMergedSegmentMB setting (v.sevel via Mike
10979   McCandless)
10980   
10981 * LUCENE-3442: TermQuery.TermWeight.scorer() returns null for non-atomic
10982   IndexReaders (optimization bug, introcuced by LUCENE-2829), preventing
10983   QueryWrapperFilter and similar classes to get a top-level DocIdSet.
10984   (Dan C., Uwe Schindler)
10985 
10986 * LUCENE-3390: Corrected handling of missing values when two parallel searches
10987   using different missing values for sorting: the missing value was populated
10988   directly into the FieldCache arrays during sorting, leading to concurrency
10989   issues.  (Uwe Schindler, Doron Cohen, Mike McCandless)
10990 
10991 * LUCENE-3439: Closing an NRT reader after the writer was closed was
10992   incorrectly invoking the DeletionPolicy and (then possibly deleting
10993   files) on the closed IndexWriter (Robert Muir, Mike McCandless)
10994 
10995 * LUCENE-3215: SloppyPhraseScorer sometimes computed Infinite freq
10996   (Robert Muir, Doron Cohen)  
10997 
10998 * LUCENE-3503: DisjunctionSumScorer would give slightly different scores
10999   for a document depending if you used nextDoc() versus advance().
11000   (Mike McCandless, Robert Muir)
11001 
11002 * LUCENE-3529: Properly support indexing an empty field with empty term text.
11003   Previously, if you had assertions enabled you would receive an error during
11004   flush, if you didn't, you would get an invalid index.  
11005   (Mike McCandless, Robert Muir)
11006 
11007 * LUCENE-2633: PackedInts Packed32 and Packed64 did not support internal
11008   structures larger than 256MB (Toke Eskildsen via Mike McCandless)
11009 
11010 * LUCENE-3540: LUCENE-3255 dropped support for pre-1.9 indexes, but the
11011   error message in IndexFormatTooOldException was incorrect. (Uwe Schindler,
11012   Mike McCandless)
11013 
11014 * LUCENE-3541: IndexInput's default copyBytes() implementation was not safe 
11015   across multiple threads, because all clones shared the same buffer.
11016   (Robert Muir)
11017   
11018 * LUCENE-3548: Fix CharsRef#append to extend length of the existing char[]
11019   and preserve existing chars. (Simon Willnauer) 
11020 
11021 * LUCENE-3582: Normalize NaN values in NumericUtils.floatToSortableInt() /
11022   NumericUtils.doubleToSortableLong(), so this is consistent with stored
11023   fields. Also fix NumericRangeQuery to not falsely hit NaNs on half-open
11024   ranges (one bound is null). Because of normalization, NumericRangeQuery
11025   can now be used to hit NaN values by creating a query with
11026   upper == lower == NaN (inclusive).  (Dawid Weiss, Uwe Schindler)
11027 
11028 API Changes
11029 
11030 * LUCENE-3454: Rename IndexWriter.optimize to forceMerge to discourage
11031   use of this method since it is horribly costly and rarely justified
11032   anymore.  MergePolicy.findMergesForOptimize was renamed to
11033   findForcedMerges.  IndexReader.isOptimized was
11034   deprecated. IndexCommit.isOptimized was replaced with
11035   getSegmentCount. (Robert Muir, Mike McCandless)
11036 
11037 * LUCENE-3205: Deprecated MultiTermQuery.getTotalNumerOfTerms() [and
11038   related methods], as the numbers returned are not useful
11039   for multi-segment indexes. They were only needed for tests of
11040   NumericRangeQuery.  (Mike McCandless, Uwe Schindler)
11041 
11042 * LUCENE-3574: Deprecate outdated constants in org.apache.lucene.util.Constants
11043   and add new ones for Java 6 and Java 7.  (Uwe Schindler)
11044 
11045 * LUCENE-3571: Deprecate IndexSearcher(Directory). Use the constructors
11046   that take IndexReader instead.  (Robert Muir)
11047 
11048 * LUCENE-3577: Rename IndexWriter.expungeDeletes to forceMergeDeletes,
11049   and revamped the javadocs, to discourage
11050   use of this method since it is horribly costly and rarely
11051   justified.  MergePolicy.findMergesToExpungeDeletes was renamed to
11052   findForcedDeletesMerges. (Robert Muir, Mike McCandless)
11053 
11054 * LUCENE-3464: IndexReader.reopen has been renamed to
11055   IndexReader.openIfChanged (a static method), and now returns null
11056   (instead of the old reader) if there are no changes in the index, to
11057   prevent the common pitfall of accidentally closing the old reader.
11058 
11059 New Features
11060 
11061 * LUCENE-3448: Added FixedBitSet.and(other/DISI), andNot(other/DISI).
11062   (Uwe Schindler)
11063 
11064 * LUCENE-2215: Added IndexSearcher.searchAfter which returns results after a 
11065   specified ScoreDoc (e.g. last document on the previous page) to support deep
11066   paging use cases.  (Aaron McCurry, Grant Ingersoll, Robert Muir) 
11067 
11068 * LUCENE-1990: Adds internal packed ints implementation, to be used
11069   for more efficient storage of int arrays when the values are
11070   bounded, for example for storing the terms dict index (Toke
11071   Eskildsen via Mike McCandless)
11072   
11073 * LUCENE-3558: Moved SearcherManager, NRTManager & SearcherLifetimeManager into
11074   core. All classes are contained in o.a.l.search. (Simon Willnauer)
11075 
11076 Optimizations
11077 
11078 * LUCENE-3426: Add NGramPhraseQuery which extends PhraseQuery and tries to 
11079   reduce the number of terms of the query when rewrite(), in order to improve 
11080   performance.  (Robert Muir, Koji Sekiguchi)
11081   
11082 * LUCENE-3494: Optimize FilteredQuery to remove a multiply in score()
11083   (Uwe Schindler, Robert Muir)
11084 
11085 * LUCENE-3534: Remove filter logic from IndexSearcher and delegate to
11086   FilteredQuery's Scorer. This is a partial backport of a cleanup in
11087   FilteredQuery/IndexSearcher added by LUCENE-1536 to Lucene 4.0.
11088   (Uwe Schindler)
11089 
11090 * LUCENE-2205: Very substantial (3-5X) RAM reduction required to hold
11091   the terms index on opening an IndexReader (Aaron McCurry via Mike McCandless)
11092 
11093 * LUCENE-3443: FieldCache can now set docsWithField, and create an
11094   array, in a single pass.  This results in faster init time for apps
11095   that need both (such as sorting by a field with a missing value).
11096   (Mike McCandless)
11097 
11098 Test Cases
11099 
11100 * LUCENE-3420: Disable the finalness checks in TokenStream and Analyzer
11101   for implementing subclasses in different packages, where assertions are not
11102   enabled. (Uwe Schindler)
11103 
11104 * LUCENE-3506: tests relying on assertions being enabled were no-op because
11105   they ignored AssertionError. With this fix now entire test framework 
11106   (every test) fails if assertions are disabled, unless 
11107   -Dtests.asserts.gracious=true is specified. (Doron Cohen)
11108 
11109 Build
11110 
11111 * SOLR-2849: Fix dependencies in Maven POMs. (David Smiley via Steve Rowe)
11112 
11113 * LUCENE-3561: Fix maven xxx-src.jar files that were missing resources.
11114   (Uwe Schindler)
11115 
11116 ======================= Lucene 3.4.0 =======================
11117 
11118 Bug fixes
11119 
11120 * LUCENE-3251: Directory#copy failed to close target output if opening the
11121   source stream failed. (Simon Willnauer)
11122 
11123 * LUCENE-3255: If segments_N file is all zeros (due to file
11124   corruption), don't read that to mean the index is empty.  (Gregory
11125   Tarr, Mark Harwood, Simon Willnauer, Mike McCandless)
11126 
11127 * LUCENE-3254: Fixed minor bug in deletes were written to disk,
11128   causing the file to sometimes be larger than it needed to be.  (Mike
11129   McCandless)
11130 
11131 * LUCENE-3224: Fixed a big where CheckIndex would incorrectly report a
11132   corrupt index if a term with docfreq >= 16 was indexed more than once
11133   at the same position.  (Robert Muir)
11134 
11135 * LUCENE-3339: Fixed deadlock case when multiple threads use the new
11136   block-add (IndexWriter.add/updateDocuments) methods.  (Robert Muir,
11137   Mike McCandless)
11138 
11139 * LUCENE-3340: Fixed case where IndexWriter was not flushing at
11140   exactly maxBufferedDeleteTerms (Mike McCandless)
11141 
11142 * LUCENE-3358, LUCENE-3361: StandardTokenizer and UAX29URLEmailTokenizer 
11143   wrongly discarded combining marks attached to Han or Hiragana characters, 
11144   this is fixed if you supply Version >= 3.4 If you supply a previous 
11145   lucene version, you get the old buggy behavior for backwards compatibility.  
11146   (Trejkaz, Robert Muir)
11147 
11148 * LUCENE-3368: IndexWriter commits segments without applying their buffered
11149   deletes when flushing concurrently. (Simon Willnauer, Mike McCandless)
11150   
11151 * LUCENE-3365: Create or Append mode determined before obtaining write lock
11152   can cause IndexWriter overriding an existing index.
11153   (Geoff Cooney via Simon Willnauer)
11154 
11155 * LUCENE-3380: Fixed a bug where FileSwitchDirectory's listAll() would wrongly
11156   throw NoSuchDirectoryException when all files written so far have been
11157   written to one directory, but the other still has not yet been created on the
11158   filesystem.  (Robert Muir)
11159 
11160 * LUCENE-3409: IndexWriter.deleteAll was failing to close pooled NRT
11161   SegmentReaders, leading to unused files accumulating in the
11162   Directory.  (tal steier via Mike McCandless)
11163 
11164 * LUCENE-3418: Lucene was failing to fsync index files on commit,
11165   meaning an operating system or hardware crash, or power loss, could
11166   easily corrupt the index.  (Mark Miller, Robert Muir, Mike
11167   McCandless)
11168   
11169 New Features
11170 
11171 * LUCENE-3290: Added FieldInvertState.numUniqueTerms 
11172   (Mike McCandless, Robert Muir)
11173 
11174 * LUCENE-3280: Add FixedBitSet, like OpenBitSet but is not elastic
11175   (grow on demand if you set/get/clear too-large indices).  (Mike
11176   McCandless)
11177 
11178 * LUCENE-2048: Added the ability to omit positions but still index
11179   term frequencies, you can now control what is indexed into
11180   the postings via AbstractField.setIndexOptions:
11181    DOCS_ONLY: only documents are indexed: term frequencies and positions are omitted
11182    DOCS_AND_FREQS: only documents and term frequencies are indexed: positions are omitted
11183    DOCS_AND_FREQS_AND_POSITIONS: full postings: documents, frequencies, and positions
11184   AbstractField.setOmitTermFrequenciesAndPositions is deprecated,
11185   you should use DOCS_ONLY instead.  (Robert Muir)
11186 
11187 * LUCENE-3097: Added a new grouping collector that can be used to retrieve all most relevant
11188   documents per group. This can be useful in situations when one wants to compute grouping
11189   based facets / statistics on the complete query result. (Martijn van Groningen)
11190 
11191 * LUCENE-3334: If Java7 is detected, IOUtils.closeSafely() will log
11192   suppressed exceptions in the original exception, so stack trace
11193   will contain them.  (Uwe Schindler)
11194 
11195 Optimizations
11196 
11197 * LUCENE-3201, LUCENE-3218: CompoundFileSystem code has been consolidated 
11198   into a Directory implementation. Reading is optimized for MMapDirectory,
11199   NIOFSDirectory and SimpleFSDirectory to only map requested parts of the
11200   CFS into an IndexInput. Writing to a CFS now tries to append to the CF
11201   directly if possible and merges separately written files on the fly instead
11202   of during close. (Simon Willnauer, Robert Muir)
11203 
11204 * LUCENE-3289: When building an FST you can now tune how aggressively
11205   the FST should try to share common suffixes.  Typically you can
11206   greatly reduce RAM required during building, and CPU consumed, at
11207   the cost of a somewhat larger FST.  (Mike McCandless)
11208 
11209 Test Cases
11210 
11211 * LUCENE-3327: Fix AIOOBE when TestFSTs is run with -Dtests.verbose=true
11212  (James Dyer via Mike McCandless)
11213 
11214 Build
11215 
11216 * LUCENE-3406: Add ant target 'package-local-src-tgz' to Lucene and Solr
11217   to package sources from the local working copy.
11218   (Seung-Yeoul Yang via Steve Rowe)
11219 
11220 
11221 ======================= Lucene 3.3.0 =======================
11222 
11223 Changes in backwards compatibility policy
11224 
11225 * LUCENE-3140: IndexOutput.copyBytes now takes a DataInput (superclass
11226   of IndexInput) as its first argument.  (Robert Muir, Dawid Weiss,
11227   Mike McCandless)
11228 
11229 * LUCENE-3191: FieldComparator.value now returns an Object not
11230   Comparable; FieldDoc.fields also changed from Comparable[] to
11231   Object[] (Uwe Schindler, Mike McCandless)
11232 
11233 * LUCENE-3208: Made deprecated methods Query.weight(Searcher) and
11234   Searcher.createWeight() final to prevent override. If you have
11235   overridden one of these methods, cut over to the non-deprecated
11236   implementation. (Uwe Schindler, Robert Muir, Yonik Seeley)
11237 
11238 * LUCENE-3238: Made MultiTermQuery.rewrite() final, to prevent
11239   problems (such as not properly setting rewrite methods, or 
11240   not working correctly with things like SpanMultiTermQueryWrapper).
11241   To rewrite to a simpler form, instead return a simpler enum
11242   from getEnum(IndexReader). For example, to rewrite to a single term, 
11243   return a SingleTermEnum.  (ludovic Boutros, Uwe Schindler, Robert Muir)
11244 
11245 Changes in runtime behavior
11246 
11247 * LUCENE-2834: the hash used to compute the lock file name when the
11248   lock file is not stored in the index has changed.  This means you
11249   will see a different lucene-XXX-write.lock in your lock directory.
11250   (Robert Muir, Uwe Schindler, Mike McCandless)
11251   
11252 * LUCENE-3146: IndexReader.setNorm throws IllegalStateException if the field
11253   does not store norms. (Shai Erera, Mike McCandless)
11254 
11255 * LUCENE-3198: On Linux, if the JRE is 64 bit and supports unmapping,
11256   FSDirectory.open now defaults to MMapDirectory instead of
11257   NIOFSDirectory since MMapDirectory gives better performance.  (Mike
11258   McCandless)
11259   
11260 * LUCENE-3200: MMapDirectory now uses chunk sizes that are powers of 2.
11261   When setting the chunk size, it is rounded down to the next possible
11262   value. The new default value for 64 bit platforms is 2^30 (1 GiB),
11263   for 32 bit platforms it stays unchanged at 2^28 (256 MiB).
11264   Internally, MMapDirectory now only uses one dedicated final IndexInput
11265   implementation supporting multiple chunks, which makes Hotspot's life
11266   easier.  (Uwe Schindler, Robert Muir, Mike McCandless)
11267   
11268 Bug fixes
11269 
11270 * LUCENE-3147,LUCENE-3152: Fixed open file handles leaks in many places in the 
11271   code. Now MockDirectoryWrapper (in test-framework) tracks all open files, 
11272   including locks, and fails if the test fails to release all of them.
11273   (Mike McCandless, Robert Muir, Shai Erera, Simon Willnauer)
11274 
11275 * LUCENE-3102: CachingCollector.replay was failing to call setScorer
11276   per-segment (Martijn van Groningen via Mike McCandless)
11277 
11278 * LUCENE-3183: Fix rare corner case where seeking to empty term
11279   (field="", term="") with terms index interval 1 could hit
11280   ArrayIndexOutOfBoundsException (selckin, Robert Muir, Mike
11281   McCandless)
11282 
11283 * LUCENE-3208: IndexSearcher had its own private similarity field
11284   and corresponding get/setter overriding Searcher's implementation. If you
11285   setted a different Similarity instance on IndexSearcher, methods implemented
11286   in the superclass Searcher were not using it, leading to strange bugs.
11287   (Uwe Schindler, Robert Muir)
11288 
11289 * LUCENE-3197: Fix core merge policies to not over-merge during
11290   background optimize when documents are still being deleted
11291   concurrently with the optimize (Mike McCandless)
11292 
11293 * LUCENE-3222: The RAM accounting for buffered delete terms was
11294   failing to measure the space required to hold the term's field and
11295   text character data.  (Mike McCandless)
11296 
11297 * LUCENE-3238: Fixed bug where using WildcardQuery("prefix*") inside
11298   of a SpanMultiTermQueryWrapper rewrote incorrectly and returned
11299   an error instead.  (ludovic Boutros, Uwe Schindler, Robert Muir)
11300 
11301 API Changes
11302 
11303 * LUCENE-3208: Renamed protected IndexSearcher.createWeight() to expert
11304   public method IndexSearcher.createNormalizedWeight() as this better describes
11305   what this method does. The old method is still there for backwards
11306   compatibility. Query.weight() was deprecated and simply delegates to
11307   IndexSearcher. Both deprecated methods will be removed in Lucene 4.0.
11308   (Uwe Schindler, Robert Muir, Yonik Seeley)
11309 
11310 * LUCENE-3197: MergePolicy.findMergesForOptimize now takes
11311   Map<SegmentInfo,Boolean> instead of Set<SegmentInfo> as the second
11312   argument, so the merge policy knows which segments were originally
11313   present vs produced by an optimizing merge (Mike McCandless)
11314 
11315 Optimizations
11316 
11317 * LUCENE-1736: DateTools.java general improvements. 
11318   (David Smiley via Steve Rowe)
11319 
11320 New Features
11321 
11322 * LUCENE-3140: Added experimental FST implementation to Lucene.
11323   (Robert Muir, Dawid Weiss, Mike McCandless)
11324 
11325 * LUCENE-3193: A new TwoPhaseCommitTool allows running a 2-phase commit 
11326   algorithm over objects that implement the new TwoPhaseCommit interface (such
11327   as IndexWriter). (Shai Erera)
11328 
11329 * LUCENE-3191: Added TopDocs.merge, to facilitate merging results from
11330   different shards (Uwe Schindler, Mike McCandless)
11331 
11332 * LUCENE-3179: Added OpenBitSet.prevSetBit (Paul Elschot via Mike McCandless)
11333 
11334 * LUCENE-3210: Made TieredMergePolicy more aggressive in reclaiming
11335   segments with deletions; added new methods
11336   set/getReclaimDeletesWeight to control this.  (Mike McCandless)
11337 
11338 Build
11339 
11340 * LUCENE-1344: Create OSGi bundle using dev-tools/maven.
11341   (Nicolas Lalevée, Luca Stancapiano via ryan)
11342   
11343 * LUCENE-3204: The maven-ant-tasks jar is now included in the source tree;
11344   users of the generate-maven-artifacts target no longer have to manually
11345   place this jar in the Ant classpath.  NOTE: when Ant looks for the 
11346   maven-ant-tasks jar, it looks first in its pre-existing classpath, so
11347   any copies it finds will be used instead of the copy included in the 
11348   Lucene/Solr source tree.  For this reason, it is recommeded to remove
11349   any copies of the maven-ant-tasks jar in the Ant classpath, e.g. under
11350   ~/.ant/lib/ or under the Ant installation's lib/ directory. (Steve Rowe) 
11351   
11352 
11353 ======================= Lucene 3.2.0 =======================
11354 
11355 Changes in backwards compatibility policy
11356 
11357 * LUCENE-2953: PriorityQueue's internal heap was made private, as subclassing
11358   with generics can lead to ClassCastException. For advanced use (e.g. in Solr)
11359   a method getHeapArray() was added to retrieve the internal heap array as a
11360   non-generic Object[].  (Uwe Schindler, Yonik Seeley)
11361 
11362 * LUCENE-1076: IndexWriter.setInfoStream now throws IOException
11363   (Mike McCandless, Shai Erera)
11364 
11365 * LUCENE-3084: MergePolicy.OneMerge.segments was changed from
11366   SegmentInfos to a List<SegmentInfo>. SegmentInfos itself was changed
11367   to no longer extend Vector<SegmentInfo> (to update code that is using
11368   Vector-API, use the new asList() and asSet() methods returning unmodifiable
11369   collections; modifying SegmentInfos is now only possible through
11370   the explicitely declared methods). IndexWriter.segString() now takes
11371   Iterable<SegmentInfo> instead of List<SegmentInfo>. A simple recompile
11372   should fix this. MergePolicy and SegmentInfos are internal/experimental
11373   APIs not covered by the strict backwards compatibility policy.
11374   (Uwe Schindler, Mike McCandless)
11375 
11376 Changes in runtime behavior
11377 
11378 * LUCENE-3065: When a NumericField is retrieved from a Document loaded
11379   from IndexReader (or IndexSearcher), it will now come back as
11380   NumericField not as a Field with a string-ified version of the
11381   numeric value you had indexed.  Note that this only applies for
11382   newly-indexed Documents; older indices will still return Field
11383   with the string-ified numeric value. If you call Document.get(),
11384   the value comes still back as String, but Document.getFieldable()
11385   returns NumericField instances. (Uwe Schindler, Ryan McKinley,
11386   Mike McCandless)
11387 
11388 * LUCENE-1076: Changed the default merge policy from
11389   LogByteSizeMergePolicy to TieredMergePolicy, as of Version.LUCENE_32
11390   (passed to IndexWriterConfig), which is able to merge non-contiguous
11391   segments. This means docIDs no longer necessarily stay "in order"
11392   during indexing.  If this is a problem then you can use either of
11393   the LogMergePolicy impls.  (Mike McCandless)
11394   
11395 New features
11396 
11397 * LUCENE-3082: Added index upgrade tool oal.index.IndexUpgrader
11398   that allows to upgrade all segments to last recent supported index
11399   format without fully optimizing.  (Uwe Schindler, Mike McCandless)
11400 
11401 * LUCENE-1076: Added TieredMergePolicy which is able to merge non-contiguous 
11402   segments, which means docIDs no longer necessarily stay "in order".
11403   (Mike McCandless, Shai Erera)
11404 
11405 * LUCENE-3071: Adding ReversePathHierarchyTokenizer, added skip parameter to 
11406   PathHierarchyTokenizer (Olivier Favre via ryan)
11407 
11408 * LUCENE-1421, LUCENE-3102: added CachingCollector which allow you to cache 
11409   document IDs and scores encountered during the search, and "replay" them to 
11410   another Collector. (Mike McCandless, Shai Erera)
11411 
11412 * LUCENE-3112: Added experimental IndexWriter.add/updateDocuments,
11413   enabling a block of documents to be indexed, atomically, with
11414   guaranteed sequential docIDs.  (Mike McCandless)
11415 
11416 API Changes
11417 
11418 * LUCENE-3061: IndexWriter's getNextMerge() and merge(OneMerge) are now public
11419   (though @lucene.experimental), allowing for custom MergeScheduler 
11420   implementations. (Shai Erera)
11421 
11422 * LUCENE-3065: Document.getField() was deprecated, as it throws
11423   ClassCastException when loading lazy fields or NumericFields.
11424   (Uwe Schindler, Ryan McKinley, Mike McCandless)
11425 
11426 * LUCENE-2027: Directory.touchFile is deprecated and will be removed
11427   in 4.0.  (Mike McCandless)
11428 
11429 Optimizations
11430 
11431 * LUCENE-2990: ArrayUtil/CollectionUtil.*Sort() methods now exit early
11432   on empty or one-element lists/arrays.  (Uwe Schindler)
11433 
11434 * LUCENE-2897: Apply deleted terms while flushing a segment.  We still
11435   buffer deleted terms to later apply to past segments.  (Mike McCandless)
11436 
11437 * LUCENE-3126: IndexWriter.addIndexes copies incoming segments into CFS if they
11438   aren't already and MergePolicy allows that. (Shai Erera)
11439 
11440 Bug fixes
11441 
11442 * LUCENE-2996: addIndexes(IndexReader) did not flush before adding the new 
11443   indexes, causing existing deletions to be applied on the incoming indexes as 
11444   well. (Shai Erera, Mike McCandless)
11445 
11446 * LUCENE-3024: Index with more than 2.1B terms was hitting AIOOBE when
11447   seeking TermEnum (eg used by Solr's faceting) (Tom Burton-West, Mike
11448   McCandless)
11449 
11450 * LUCENE-3042: When a filter or consumer added Attributes to a TokenStream
11451   chain after it was already (partly) consumed [or clearAttributes(),
11452   captureState(), cloneAttributes(),... was called by the Tokenizer],
11453   the Tokenizer calling clearAttributes() or capturing state after addition 
11454   may not do this on the newly added Attribute. This bug affected only
11455   very special use cases of the TokenStream-API, most users would not
11456   have recognized it.  (Uwe Schindler, Robert Muir)
11457 
11458 * LUCENE-3054: PhraseQuery can in some cases stack overflow in
11459   SorterTemplate.quickSort(). This fix also adds an optimization to
11460   PhraseQuery as term with lower doc freq will also have less positions.
11461   (Uwe Schindler, Robert Muir, Otis Gospodnetic)
11462 
11463 * LUCENE-3068: sloppy phrase query failed to match valid documents when multiple 
11464   query terms had same position in the query. (Doron Cohen)
11465 
11466 * LUCENE-3012: Lucene writes the header now for separate norm files (*.sNNN)
11467   (Robert Muir)
11468 
11469 Build
11470 
11471 * LUCENE-3006: Building javadocs will fail on warnings by default. 
11472   Override with -Dfailonjavadocwarning=false (sarowe, gsingers)
11473 
11474 * LUCENE-3128: "ant eclipse" creates a .project file for easier Eclipse
11475   integration (unless one already exists). (Daniel Serodio via Shai Erera)
11476 
11477 Test Cases
11478 
11479 * LUCENE-3002: added 'tests.iter.min' to control 'tests.iter' by allowing to 
11480   stop iterating if at least 'tests.iter.min' ran and a failure occured. 
11481   (Shai Erera, Chris Hostetter)
11482 
11483 ======================= Lucene 3.1.0 =======================
11484 
11485 Changes in backwards compatibility policy
11486 
11487 * LUCENE-2719: Changed API of internal utility class
11488   org.apache.lucene.util.SorterTemplate to support faster quickSort using
11489   pivot values and also merge sort and insertion sort. If you have used
11490   this class, you have to implement two more methods for handling pivots.
11491   (Uwe Schindler, Robert Muir, Mike McCandless)
11492 
11493 * LUCENE-1923: Renamed SegmentInfo & SegmentInfos segString method to
11494   toString.  These are advanced APIs and subject to change suddenly.
11495   (Tim Smith via Mike McCandless)
11496 
11497 * LUCENE-2190: Removed deprecated customScore() and customExplain()
11498   methods from experimental CustomScoreQuery.  (Uwe Schindler)
11499 
11500 * LUCENE-2286: Enabled DefaultSimilarity.setDiscountOverlaps by default.
11501   This means that terms with a position increment gap of zero do not
11502   affect the norms calculation by default.  (Robert Muir)
11503 
11504 * LUCENE-2320: MergePolicy.writer is now of type SetOnce, which allows setting
11505   the IndexWriter for a MergePolicy exactly once. You can change references to
11506   'writer' from <code>writer.doXYZ()</code> to <code>writer.get().doXYZ()</code>
11507   (it is also advisable to add an <code>assert writer != null;</code> before you
11508   access the wrapped IndexWriter.)
11509 
11510   In addition, MergePolicy only exposes a default constructor, and the one that
11511   took IndexWriter as argument has been removed from all MergePolicy extensions.
11512   (Shai Erera via Mike McCandless)
11513 
11514 * LUCENE-2328: SimpleFSDirectory.SimpleFSIndexInput is moved to
11515   FSDirectory.FSIndexInput. Anyone extending this class will have to
11516   fix their code on upgrading. (Earwin Burrfoot via Mike McCandless)
11517 
11518 * LUCENE-2302: The new interface for term attributes, CharTermAttribute,
11519   now implements CharSequence. This requires the toString() methods of
11520   CharTermAttribute, deprecated TermAttribute, and Token to return only
11521   the term text and no other attribute contents. LUCENE-2374 implements
11522   an attribute reflection API to no longer rely on toString() for attribute
11523   inspection. (Uwe Schindler, Robert Muir)
11524 
11525 * LUCENE-2372, LUCENE-2389: StandardAnalyzer, KeywordAnalyzer,
11526   PerFieldAnalyzerWrapper, WhitespaceTokenizer are now final.  Also removed
11527   the now obsolete and deprecated Analyzer.setOverridesTokenStreamMethod().
11528   Analyzer and TokenStream base classes now have an assertion in their ctor,
11529   that check subclasses to be final or at least have final implementations
11530   of incrementToken(), tokenStream(), and reusableTokenStream().
11531   (Uwe Schindler, Robert Muir)
11532 
11533 * LUCENE-2316: Directory.fileLength contract was clarified - it returns the
11534   actual file's length if the file exists, and throws FileNotFoundException
11535   otherwise. Returning length=0 for a non-existent file is no longer allowed. If
11536   you relied on that, make sure to catch the exception. (Shai Erera)
11537 
11538 * LUCENE-2386: IndexWriter no longer performs an empty commit upon new index
11539   creation. Previously, if you passed an empty Directory and set OpenMode to
11540   CREATE*, IndexWriter would make a first empty commit. If you need that
11541   behavior you can call writer.