"Fossies" - the Fresh Open Source Software Archive

Member "solr-8.4.1/lucene/CHANGES.txt" (10 Jan 2020, 742863 Bytes) of package /linux/www/solr-8.4.1-src.tgz:

As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "CHANGES.txt" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 8.4.0_vs_8.4.1.

    1 Lucene Change Log
    3 For more information on past and future Lucene versions, please see:
    4 http://s.apache.org/luceneversions
    6 ======================= Lucene 8.4.1 =======================
    8 Bug Fixes
    9 ---------------------
   10 (No changes)
   12 ======================= Lucene 8.4.0 =======================
   14 API Changes
   16 * LUCENE-9029: Deprecate SloppyMath toRadians/toDegrees in favor of Java Math.
   17   (Jack Conradson via Adrien Grand)
   19 New Features
   21 * LUCENE-8620: Add CONTAINS support for LatLonShape and XYShape. (Ignacio Vera)
   23 Improvements
   25 * LUCENE-9002: Skip costly caching clause in LRUQueryCache if it makes the query
   26   many times slower. (Guoqiang Jiang)
   28 * LUCENE-9006: WordDelimiterGraphFilter's catenateAll token is now ordered before any token parts, like WDF did.
   29   (David Smiley)
   31 * LUCENE-9028: introducing Intervals.multiterm() (Mikhail Khludnev)
   33 * LUCENE-9018: ConcatenateGraphFilter now has a configurable separator. (Stanislav Mikulchik, David Smiley)
   35 * LUCENE-9036: ExitableDirectoryReader may interupt scaning over DocValues (Mikhail Khludnev)
   37 * LUCENE-9062: QueryVisitor now has a consumeTermsMatching() method, allowing queries
   38   that match a class of terms to pass a ByteRunAutomaton matching those that class
   39   back to the visitor. (Alan Woodward, David Smiley)
   41 * LUCENE-9073: IntervalQuery to respond field on toString() and explain() (Mikhail Khludnev)
   43 Optimizations
   45 * LUCENE-8928: When building a kd-tree for dimensions n > 2, compute exact bounds for an inner node every N splits
   46   to improve the quality of the tree. N is defined by SPLITS_BEFORE_EXACT_BOUNDS which is set to 4.
   47   (Ignacio Vera, Adrien Grand)
   49 * BaseDirectoryReader no longer sums up the `LeafReader#numDocs` of its leaves
   50   eagerly. This especially helps when creating views of readers that hide
   51   documents, since computing the number of live documents is an expensive
   52   operation. (Adrien Grand)
   54 * LUCENE-8992: TopFieldCollector and TopScoreDocCollector can now share minimum scores across leaves
   55   concurrently. (Adrien Grand, Atri Sharma, Jim Ferenczi)
   57 * LUCENE-8932: BKDReader's index is now stored off-heap when the IndexInput is
   58   an instance of ByteBufferIndexInput. (Jack Conradson via Adrien Grand)
   60 * LUCENE-9024: IntroSelector now falls back to the median of medians algorithm
   61   instead of sorting when the maximum recursion level is exceeded, providing
   62   better worst-case runtime. (Paul Sanwald via Adrien Grand)
   64 * LUCENE-8920: The denser arcs of FST now index labels with a bitset in order
   65   to provide near constant time access. (Bruno Roustant, Mike Sokolov via Adrien Grand)
   67 * LUCENE-9027: Use SIMD instructions to decode postings. (Adrien Grand)
   69 * LUCENE-9049: Remove FST cached root arcs now redundant with labels indexed by bitset.
   70   This frees some on-heap FST space. (Jack Conradson via Bruno Roustant)
   72 * LUCENE-9045: Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat. (Bruno Roustant)
   74 Bug Fixes
   76 * LUCENE-9001: Fix race condition in SetOnce. (Przemko Robakowski)
   78 * LUCENE-9030: Fix WordnetSynonymParser behaviour so it behaves similar to
   79   SolrSynonymParser. (Christoph Buescher via Alan Woodward)
   81 * LUCENE-9054: Fix reproduceJenkinsFailures.py to not overwrite junit XML files when retrying (hossman)
   83 * LUCENE-9031: UnsupportedOperationException on MatchesIterator.getQuery() (Alan Woodward, Mikhail Khludnev)
   85 * LUCENE-8996: maxScore was sometimes missing from distributed grouped responses.
   86   (Julien Massenet, Diego Ceccarelli, Munendra S N, Christine Poerschke)
   88 * LUCENE-9055: Fix the detection of lines crossing triangles through edge points.
   89   (Ignacio Vera)
   91 * LUCENE-9103: Disjunctions can miss some hits in some rare conditions. (Adrien Grand)
   93 Other
   95 * LUCENE-8979: Code Cleanup: Use entryset for map iteration wherever possible. - Part 2 (Koen De Groote)
   97 * LUCENE-8746: Refactor EdgeTree - Introduce a Component tree that represents the tree of components (e.g polygons).
   98   Edge tree is now just a tree of edges. (Ignacio Vera)
  100 * LUCENE-8994: Code Cleanup - Pass values to list constructor instead of empty constructor followed by addAll(). (Koen De Groote)
  102 * LUCENE-9046: Fix wrong example in Javadoc of TermInSetQuery (Namgyu Kim)
  104 * LUCENE-8983: Add sandbox PhraseWildcardQuery to control multi-terms expansions in a phrase. (Bruno Roustant)
  106 * LUCENE-9067: Polygon2D#contains() is now thread safe. (Ignacio Vera)
  108 Build
  110 * Upgrade forbiddenapis to version 2.7; upgrade Groovy to 2.4.17.  (Uwe Schindler)
  112 * LUCENE-9041: Upgrade ecj to 3.19.0 to fix sporadic precommit javadoc issues (Kevin Risden)
  114 ======================= Lucene 8.3.1 =======================
  116 Bug Fixes
  118 * LUCENE-9050: MultiTermIntervalsSource.visit() was not calling back to its
  119   visitor. (Alan Woodward)
  121 ======================= Lucene 8.3.0 =======================
  123 API Changes
  125 * LUCENE-8909: IndexWriter#getFieldNames() method is used to get fields present in index. After LUCENE-8316, this
  126   method is no longer required. Hence, deprecate IndexWriter#getFieldNames() method. (Adrien Grand, Munendra S N)
  128 * LUCENE-8755: SpatialPrefixTreeFactory now consumes the "version" parsed with Lucene's Version class.  The quad
  129   and packed quad prefix trees are sensitive to this.  It's recommended to pass the version like you
  130   should do likewise for analysis components for tokenized text, or else changes to the encoding in future versions
  131   may be incompatible with older indexes.  (Chongchen Chen, David Smiley)
  133 * LUCENE-8956: QueryRescorer now only sorts the first topN hits instead of all
  134   initial hits. (Paul Sanwald via Adrien Grand)
  136 * LUCENE-8921: IndexSearcher.termStatistics() no longer takes a TermStates; it takes the docFreq and totalTermFreq.
  137   And don't call if docFreq <= 0.  The previous implementation survives as deprecated and final.  It's removed in 9.0.
  138   (Bruno Roustant, David Smiley, Alan Woodward)
  140 * LUCENE-8990: PointValues#estimateDocCount(visitor) estimates the number of documents that would be matched by
  141   the given IntersectVisitor. THe method is used to compute the cost() of ScorerSuppliers instead of
  142   PointValues#estimatePointCount(visitor). (Ignacio Vera, Adrien Grand)
  144 New Features
  146 * LUCENE-8936: Add SpanishMinimalStemFilter (vinod kumar via Tomoko Uchida)
  148 * LUCENE-8764 LUCENE-8945: Add "export all terms and doc freqs" feature to Luke with delimiters. (Leonardo Menezes, Amish Shah via Tomoko Uchida)
  150 * LUCENE-8747: Composite Matches from multiple subqueries now allow access to
  151   their submatches, and a new NamedMatches API allows marking of subqueries
  152   and a simple way to find which subqueries have matched on a given document
  153   (Alan Woodward, Jim Ferenczi)
  155 * LUCENE-8769: Introduce Range Query For Multiple Connected Ranges (Atri Sharma)
  157 * LUCENE-8960: Introduce LatLonDocValuesPointInPolygonQuery for LatLonDocValuesField (Ignacio Vera)
  159 * LUCENE-8753: New UniformSplitPostingsFormat (name "UniformSplit") primarily benefiting in simplicity and
  160   extensibility.  New STUniformSplitPostingsFormat (name "SharedTermsUniformSplit") that shares a single internal
  161   term dictionary across fields.  (Bruno Roustant, Juan Rodriguez, David Smiley)
  163 Improvements
  165 * LUCENE-8874: Show SPI names instead of class names in Luke Analysis tab. (Tomoko Uchida)
  167 * LUCENE-8894: Add APIs to find SPI names for Tokenizer/CharFilter/TokenFilter factory classes. (Tomoko Uchida)
  169 * LUCENE-8914: move the logic for discarding inner modes in FloatPointNearestNeighbor to the IntersectVisitor
  170   so we take advantage of the change introduced in LUCENE-7862. (Ignacio Vera)
  172 * LUCENE-8955: move the logic for discarding inner modes in LatLonPoint NearestNeighbor to the IntersectVisitor
  173   so we take advantage of the change introduced in LUCENE-7862. (Ignacio Vera)
  175 * LUCENE-8918: PhraseQuery throws exceptions at construction time if it is passed
  176   null arguments. (Alan Woodward)
  178 * LUCENE-8916: GraphTokenStreamFiniteStrings preserves all Token attributes
  179   through its finite strings TokenStreams (Alan Woodward)
  181 * LUCENE-8933: Check kuromoji user dictionary beforehand to avoid unexpected runtime exceptions. (Tomoko Uchida
  183 * LUCENE-8906: Expose Lucene50PostingsFormat.IntBlockTermState as public so that other postings formats can re-use it.
  184   (Bruno Roustant)
  186 * LUCENE-8942: Remove redundant parameters and improve visibility strictness in
  187   LRUQueryCache (Atri Sharma)
  189 * SOLR-13663: Introduce <SpanPositionRange> into XML Query Parser (Alessandro Benedetti via Mikhail Khludnev)
  191 * LUCENE-8952: Use a sort key instead of true distance in NearestNeighbor (Julie Tibshirani).
  193 * LUCENE-8620: Tessellator labels the edges of the generated triangles whether they belong to
  194   the original polygon. This information is added to the triangle encoding. (Ignacio Vera)
  196 * LUCENE-8964: Fix geojson shape parsing on string arrays in properties
  197    (Alexander Reelsen)
  199 * LUCENE-8976: Use exact distance between point and bounding rectangle in FloatPointNearestNeighbor. (Ignacio Vera)
  201 * LUCENE-8966: The Korean analyzer now splits tokens on boundaries between digits and alphabetic characters. (Jim Ferenczi)
  203 * LUCENE-8984: MoreLikeThis MLT is biased for uncommon fields (Andy Hind via Anshum Gupta)
  205 Optimizations
  207 * LUCENE-8922: DisjunctionMaxQuery more efficiently leverages impacts to skip
  208   non-competitive hits. (Adrien Grand)
  210 * LUCENE-8935: BooleanQuery with no scoring clause can now early terminate the query when
  211   the total hits is not requested. (Jim Ferenczi)
  213 * LUCENE-8941: Matches on wildcard queries will defer building their full
  214   disjunction until a MatchesIterator is pulled (Alan Woodward)
  216 * LUCENE-8755: spatial-extras quad and packed quad prefix trees now index points faster.
  217   (Chongchen Chen, David Smiley)
  219 * LUCENE-8860: add additional leaf node level optimizations in LatLonShapeBoundingBoxQuery.
  220   (Igor Motov via Ignacio Vera)
  222 * LUCENE-8968: Improve performance of WITHIN and DISJOINT queries for Shape queries by
  223   doing just one pass whenever possible. (Ignacio Vera)
  225 * LUCENE-8939: Introduce shared count based early termination across multiple slices
  226   (Atri Sharma)
  228 * LUCENE-8980: Blocktree's seekExact now short-circuits false if the term isn't in the min-max range of the segment.
  229   Large perf gain for ID/time like data when populated sequentially.  (Guoqiang Jiang)
  231 Bug Fixes
  233 * LUCENE-8755: spatial-extras quad and packed quad prefix trees could throw a
  234   NullPointerException for certain cell edge coordinates (Chongchen Chen, David Smiley)
  236 * LUCENE-9005: BooleanQuery.visit() would pull subVisitors from its parent visitor, rather
  237   than from a visitor for its own specific query.  This could cause problems when BQ was
  238   nested under another BQ. Instead, we now pull a MUST subvisitor, pass it to any MUST
  239   subclauses, and then pull SHOULD, MUST_NOT and FILTER visitors from it rather than from
  240   the parent.  (Alan Woodward)
  242 Other
  244 * LUCENE-8778 LUCENE-8911 LUCENE-8957: Define analyzer SPI names as static final fields and document the names in Javadocs.
  245   (Tomoko Uchida, Uwe Schindler)
  247 * LUCENE-8758: QuadPrefixTree: removed levelS and levelN fields which weren't used. (Amish Shah)
  249 * LUCENE-8975: Code Cleanup: Use entryset for map iteration wherever possible. (Koen De Groote)
  251 * LUCENE-8993, LUCENE-8807: Changed all repository and download references in build files
  252   to HTTPS. (Uwe Schindler)
  254 * LUCENE-8998: Fix OverviewImplTest.testIsOptimized reproducible failure. (Tomoko Uchida)
  256 * LUCENE-8999: LuceneTestCase.expectThrows now propogates assert/assumption failures up to the test
  257   w/o wrapping in a new assertion failure unless the caller has explicitly expected them (hossman)
  259 * LUCENE-8062: GlobalOrdinalsWithScoreQuery is no longer eligible for query caching. (Jim Ferenczi)
  262 ======================= Lucene 8.2.0 =======================
  264 API Changes
  266 * LUCENE-8865: IndexSearcher now uses Executor instead of ExecutorSerivce.
  267   This change is fully backwards compatible since ExecutorService directly
  268   implements Executor. (Simon Willnauer)
  270 * LUCENE-8856: Intervals queries have moved from the sandbox to the queries
  271   module. (Alan Woodward)
  273 * LUCENE-8893: Intervals.wildcard() and Intervals.prefix() methods now take
  274   BytesRef rather than String. (Alan Woodward)
  276 New Features
  278 * LUCENE-8632: New XYShape Field and Queries for indexing and searching general cartesian
  279   geometries. (Nick Knize)
  281 * LUCENE-8891: Snowball stemmer/analyzer for the Estonian language.
  282  (Gert Morten Paimla via Tomoko Uchida)
  284 * LUCENE-8815: Provide a DoubleValues implementation for retrieving the value of features without
  285   requiring a separate numeric field. Note that as feature values are stored with only 8 bits of 
  286   mantissa the values returned may have a delta from the original values indexed.
  287   (Colin Goodheart-Smithe via Adrien Grand)
  289 * LUCENE-8803: Provide a FeatureSortfield to allow sorting search hits by descending value of a
  290   feature. This is exposed via the factory method FeatureField#newFeatureSort.
  291   (Colin Goodheart-Smithe via Adrien Grand)
  293 * LUCENE-8784: The KoreanTokenizer now preserves punctuations if discardPunctuation is set
  294   to false (defaults to true).
  295   (Namgyu Kim via Jim Ferenczi)
  297 * LUCENE-8812: Add new KoreanNumberFilter that can change Hangul character to number
  298   and process decimal point. It is similar to the JapaneseNumberFilter.
  299   (Namgyu Kim)
  301 * LUCENE-8362: Add doc-value support to range fields. (Atri Sharma via Adrien Grand)
  303 * LUCENE-8766: Add monitor subproject (previously Luwak monitoring library). This
  304   allows a stream of documents to be matched against a set of registered queries
  305   in an efficient manner, for use as a monitoring or classification tool.
  306   (Alan Woodward)
  308 * LUCENE-7714: Add a numeric range query in sandbox that takes advantage of index sorting.
  309   (Julie Tibshirani via Jim Ferenczi)
  311 * LUCENE-8859: The completion suggester's postings format now have an option to
  312   load its internal FST off-heap. (Jim Ferenczi)
  314 Bug Fixes
  316 * LUCENE-8831: Fixed LatLonShapeBoundingBoxQuery .hashCode methods. (Ignacio Vera)
  318 * LUCENE-8775: Improve tessellator to handle better cases where a hole share a vertex
  319   with the polygon. (Ignacio Vera)
  321 * LUCENE-8785: Ensure new threadstates are locked before retrieving the number of active threadstates.
  322   This causes assertion errors and potentially broken field attributes in the IndexWriter when
  323   IndexWriter#deleteAll is called while actively indexing. (Simon Willnauer)
  325 * LUCENE-8804: Forbid calls to putAttribute on frozen FieldType instances.
  326   (Vamshi Vijay Nakkirtha via Adrien Grand)
  328 * LUCENE-8828: Removes the buggy 'disallow overlaps' boolean from Intervals.unordered(),
  329   and replaces it with a new Intervals.unorderedNoOverlaps() method (Alan Woodward)
  331 * LUCENE-8843: Don't ignore exceptions that are thrown when trying to open a
  332   file in IOUtils#fsync. (Jason Tedor via Adrien Grand)
  334 * LUCENE-8835: FileSwitchDirectory now respects the file extension when listing directory
  335   contents to ensure we don't expose pending deletes if both directory point to the same
  336   underlying filesystem directory. (Simon Willnauer)
  338 * LUCENE-8853: FileSwitchDirectory now applies best effort to place tmp files in the same
  339   directory as the target files. (Simon Willnauer)
  341 * LUCENE-8892: Add missing closing parentheses in MultiBoolFunction's description() (Florian Diebold, Munendra S N)
  343 Improvements
  345 * LUCENE-7840: Non-scoring BooleanQuery now removes SHOULD clauses before building the scorer supplier
  346   as opposed to eliminating them during scoring construction. (Atri Sharma via Jim Ferenczi)
  348 * LUCENE-8770: BlockMaxConjunctionScorer now leverages two-phase iterators in order to avoid
  349   executing the second phase when scorers don't intersect. (Adrien Grand, Jim Ferenczi)
  351 * LUCENE-8781: FST lookup performance has been improved in many cases by
  352   encoding Arcs using full-sized arrays with gaps. The new encoding is
  353   enabled for postings in the default codec and for suggesters. (Mike Sokolov)
  355 * LUCENE-8818: Fix smokeTestRelease.py encoding bug (janhoy)
  357 * LUCENE-8845: Allow Intervals.prefix() and Intervals.wildcard() to specify
  358   their maximum allowed expansions (Alan Woodward)
  360 * LUCENE-8875: Introduce a Collector optimized for use cases when large
  361   number of hits are requested (Atri Sharma)
  363 * LUCENE-8848 LUCENE-7757 LUCENE-8492: The UnifiedHighlighter now detects that parts of the query are not understood by
  364   it, and thus it should not make optimizations that result in no highlights or slow highlighting.  This generally works
  365   best for WEIGHT_MATCHES mode.  Consequently queries produced by ComplexPhraseQueryParser and the surround QueryParser
  366   will now highlight correctly. (David Smiley)
  368 * LUCENE-8793: Luke enhanced UI for CustomAnalyzer: show detailed analysis steps. (Jun Ohtani via Tomoko Uchida)
  370 * LUCENE-8855: Add Accountable to some Query implementations (ab, Adrien Grand)
  372 Optimizations
  374 * LUCENE-8796: Use exponential search instead of binary search in
  375   IntArrayDocIdSet#advance method (Luca Cavanna via Adrien Grand)
  377 * LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor.
  378   Now caller threads execute at least one search on an index even if there is
  379   an executor provided to minimize thread context switching. (Simon Willnauer)
  381 * LUCENE-8868: New storing strategy for BKD tree leaves with low cardinality.
  382   It stores the distinct values once with the cardinality value reducing the
  383   storage cost. (Ignacio Vera)
  385 * LUCENE-8885: Optimise BKD reader by exploiting cardinality information stored
  386   on leaves. (Ignacio Vera)
  388 * LUCENE-8896: Override default implementation of IntersectVisitor#visit(DocIDSetBuilder, byte[])
  389   for several queries. (Ignacio Vera)
  391 * LUCENE-8901: Load frequencies lazily only when needed in BlockDocsEnum and
  392   BlockImpactsEverythingEnum (Mayya Sharipova).
  394 * LUCENE-8888: Optimize distribution of points with data dimensions in
  395   BKD tree leaves. (Ignacio Vera)
  397 * LUCENE-8311: Phrase queries now leverage impacts. (Adrien Grand)
  399 Test Framework
  401 * LUCENE-8825: CheckHits now display the shard index in case of mismatch
  402   between top hits. (Atri Sharma via Adrien Grand)
  404 Other
  406 * LUCENE-8847: Code Cleanup: Remove StringBuilder.append with concatenated
  407   strings. (Koen De Groote via Uwe Schindler)
  409 * LUCENE-8861: Script to find open Github PRs that needs attention (janhoy)
  411 * LUCENE-8852: ReleaseWizard tool for release managers (janhoy)
  413 * LUCENE-8838: Remove support for Steiner points on Tessellator. (Ignacio Vera)
  415 * LUCENE-8879: Improve BKDRadixSelector tests. (Ignacio Vera)
  417 * LUCENE-8886: Fix TestMutablePointsReaderUtils tests. (Ignacio Vera)
  419 ======================= Lucene 8.1.1 =======================
  420 (No Changes)
  422 ======================= Lucene 8.1.0 =======================
  424 API Changes
  426 * LUCENE-3041: A query introspection API has been added.  Queries should
  427   implement a visit() method, taking a QueryVisitor, and either pass the
  428   visitor down to any child queries, or call a visitX() or consumeX() method
  429   on it.  All locations in the code that called Weight.extractTerms() 
  430   have been changed to use this API, and the extractTerms() method has 
  431   been deprecated. (Alan Woodward, Simon Willnauer, David Smiley, Luca
  432   Cavanna)
  434 * LUCENE-8735: Directory.getPendingDeletions is now abstract to ensure
  435   subclasses override it. FilterDirectory now delegates the call, ensuring
  436   correct default behaviour for subclasses. (Henning Andersen)
  438 New Features
  440 * LUCENE-2562: The well-known graphical user interface for inspecting Lucene
  441   indexes "Luke" was added as a Lucene module. It can be started from the
  442   binary distribution by calling the shell scripts in the module folder
  443   or from the source checkout by using `ant -f lucene/luke/build.xml run`.
  444   Luke provides a Swing-based user interface and can be used to open
  445   Lucene or Solr (or Elasticsearch) indexes, inspect documents, check index
  446   commits and segments, or test (custom) analyzers. It also has maintenance
  447   functions to check index structures and force merge indexes for archival.
  448   Luke was originally developed by Andrzej Bialecki, later maintained by
  449   Dmitry Kan and finally rewritten by Tomoko Uchida to use the ASF licensing
  450   compatible Swing framework (as shipped with JDKs).
  451   (Tomoko Uchida, Uwe Schindler)
  453 Bug fixes
  455 * LUCENE-8736: LatLonShapePolygonQuery returns incorrect WITHIN results
  456   with shared boundaries. Point in Polygon now correctly includes boundary
  457   points. Box and Polygon relations with triangles have also been improved to
  458   correctly include boundary points. (Nick Knize)
  460 * LUCENE-8712: Polygon2D does not detect crossings through segment edges.
  461   (Ignacio Vera)
  463 * LUCENE-8720: NameIntCacheLRU (in the facets module) had an int
  464   overflow bug that disabled cleaning of the cache (Russell A Brown)
  466 * LUCENE-8726: ValueSource.asDoubleValuesSource() could leak a reference to
  467   IndexSearcher (Alan Woodward, Yury Pakhomov)
  469 * LUCENE-8719: FixedShingleFilter can miss shingles at the end of a token stream if
  470   there are multiple paths with different lengths. (Alan Woodward)
  472 * LUCENE-8688: TieredMergePolicy#findForcedMerges now tries to create the
  473   cheapest merges that allow the index to go down to `maxSegmentCount` segments
  474   or less. (Armin Braun via Adrien Grand)
  476 * LUCENE-8477: Interval disjunctions could miss valid hits if some of the
  477   clauses of the disjunction are minimized away.  We now rewrite intervals
  478   if a source contains a disjunction and the internal gaps matter for
  479   matching.  This behaviour can be disabled if users are more interested
  480   in speed rather than accuracy of matching. (Alan Woodward, Jim Ferenczi)
  482 * LUCENE-8741: ValueSource.fromDoubleValuesSource() was casting to
  483   Scorer instead of Scorable, leading to ClassCastExceptions (Markus Jelsma,
  484   Alan Woodward)
  486 * LUCENE-8754: Fix ConcurrentModificationException in SegmentInfo if
  487   attributes are accessed in MergePolicy while the merge is running (Simon Willnauer)
  489 * LUCENE-8765: Fixed validation of the number of added points in KD trees.
  490   (Zhao Yang via Adrien Grand)
  492 Improvements
  494 * LUCENE-8673: Use radix partitioning when merging dimensional points instead
  495   of sorting all dimensions before hand. (Ignacio Vera, Adrien Grand)
  497 * LUCENE-8687: Optimise radix partitioning for points on heap. (Ignacio Vera)
  499 * LUCENE-8699: Change HeapPointWriter to use a single byte array instead to a list
  500   of byte arrays. In addition a new interface PointValue is added to abstract out
  501   the different formats between offline and on-heap writers. (Ignacio Vera)
  503 * LUCENE-8703: Build point writers in the BKD tree only when they are needed.
  504   (Ignacio Vera)
  506 * LUCENE-8652: SynonymQuery can now deboost the document frequency of each term when
  507   blending the score of the synonym. (Jim Ferenczi)
  509 * LUCENE-8631: The Korean's user dictionary now picks the longest-matching word and discards
  510   the other matches. (Yeongsu Kim via Jim Ferenczi)
  512 * LUCENE-8732: ConstantScoreQuery can now early terminate the query if the minimum score is
  513   greater than the constant score and total hits are not requested. (Jim Ferenczi)
  515 * LUCENE-8750: Implements setMissingValue() on sort fields produced from 
  516   DoubleValuesSource and LongValuesSource (Mike Sokolov via Alan Woodward)
  518 * LUCENE-8701: ToParentBlockJoinQuery now creates a child scorer that disallows skipping over
  519   non-competitive documents if the score of a parent depends on the score of multiple
  520   children (avg, max, min). Additionally the score mode `none` that assigns a constant score to
  521   each parent can early terminate top scores's collection. (Jim Ferenczi)
  523 * LUCENE-8751: Weight#matches now use the ScorerSupplier to build scorers with a lead cost of 1
  524   (single document). (Jim Ferenczi)
  526 * LUCENE-8752: Japanese new era name '令和' (Reiwa) is added to the dictionary used in
  527   JapaneseTokenizer so that the analyzer handles the era name correctly.
  528   Reiwa is set to replace the Heisei Era on May 1, 2019. (Tomoko Uchida)
  530 * LUCENE-8671: Introduced reader attributes allows a per IndexReader configuration
  531   of codec internals. This enables a per reader configuration if FSTs are on- or off-heap on a
  532   per field basis (Simon Willnauer)
  534 * LUCENE-8787: spatial-extras DateRangePrefixTree used to only parse ISO-8601 timestamps with 0 or 3
  535   digits of milliseconds precision but now parses other lengths (although > 3 not used).
  536   (Thomas Lemmé via David Smiley)
  538 Changes in Runtime Behavior
  540 * LUCENE-8671: Load FST off-heap also for ID-like fields if reader is not opened
  541   from an IndexWriter. (Simon Willnauer)
  543 * LUCENE-8730: WordDelimiterGraphFilter always emits its original token first.  This
  544   brings its behaviour into line with the deprecated WordDelimiterFilter, so that
  545   the only difference in output between the two is in the position length
  546   attribute.  (Alan Woodward, Jim Ferenczi)
  548 * LUCENE-7386: Disjunctions nested in disjunctions are now flattened. This might
  549   trigger changes in the produced scores due to changes to the order in which
  550   scores of sub clauses are summed up. (Adrien Grand)
  552 * LUCENE-8756: MoreLikeThisQuery now respects custom term frequencies
  553   (TermFrequencyAttribute) at search time (Olli Kuonanoja)
  555 Other
  557 * LUCENE-8680: Refactor EdgeTree#relateTriangle method. (Ignacio Vera)
  559 * LUCENE-8685: Refactor LatLonShape tests. (Ignacio Vera)
  561 * LUCENE-8713: Add Line2D tests. (Ignacio Vera)
  563 * LUCENE-8729: Workaround: Disable accessibility doclints (Java 13+),
  564   so compilation with recent JDK succeeds.  (Uwe Schindler)
  566 * LUCENE-8725: Make TermsQuery.SeekingTermSetTermsEnum a top level class and public (noble)
  568 ======================= Lucene 8.0.0 =======================
  570 API Changes
  572 * LUCENE-8662: TermsEnum.seekExact(BytesRef) to abstract and delegate seekExact(BytesRef)
  573   in FilterLeafReader.FilterTermsEnum. (Jeffery Yuan via Tomás Fernández Löbbe, Simon Willnauer)
  575 * LUCENE-8469: Deprecated StringHelper.compare has been removed. (Dawid Weiss)
  577 * LUCENE-8039: Introduce a "delta distance" method set to GeoDistance.  This
  578   allows distance calculations, especially for paths, to take into account an
  579   "excursion" to include the specified point.
  581 * LUCENE-8007: Index statistics Terms.getSumDocFreq(), Terms.getDocCount() are
  582   now required to be stored by codecs. Additionally, TermsEnum.totalTermFreq()
  583   and Terms.getSumTotalTermFreq() are now required: if frequencies are not
  584   stored they are equal to TermsEnum.docFreq() and Terms.getSumDocFreq(),
  585   respectively, because all freq() values equal 1. (Adrien Grand, Robert Muir)
  587 * LUCENE-8038: Deprecated PayloadScoreQuery constructors have been removed (Alan
  588   Woodward)
  590 * LUCENE-8014: Similarity.computeSlopFactor() and
  591   Similarity.computePayloadFactor() have been removed (Alan Woodward)
  593 * LUCENE-7996: Queries are now required to produce positive scores.
  594   (Adrien Grand)
  596 * LUCENE-8099: CustomScoreQuery, BoostedQuery and BoostingQuery have been
  597   removed (Alan Woodward)
  599 * LUCENE-8012: Explanation now takes Number rather than float (Alan Woodward,
  600   Robert Muir)
  602 * LUCENE-8116: SimScorer now only takes a frequency and a norm as per-document
  603   scoring factors. (Adrien Grand)
  605 * LUCENE-8113: TermContext has been renamed to TermStates, and can now be
  606   constructed lazily if term statistics are not required (Alan Woodward)
  608 * LUCENE-8242: Deprecated method IndexSearcher#createNormalizedWeight() has
  609   been removed (Alan Woodward)
  611 * LUCENE-8267: Memory codecs removed from the codebase (MemoryPostings,
  612   MemoryDocValues). (Dawid Weiss)
  614 * LUCENE-8144: Moved QueryCachingPolicy.ALWAYS_CACHE to the test framework.
  615   (Nhat Nguyen via Adrien Grand)
  617 * LUCENE-8356: StandardFilter and StandardFilterFactory have been removed
  618   (Alan Woodward)
  620 * LUCENE-8373: StandardAnalyzer.ENGLISH_STOP_WORD_SET has been removed
  621   (Alan Woodward)
  623 * LUCENE-8388: Unused PostingsEnum#attributes() method has been removed 
  624   (Alan Woodward)
  626 * LUCENE-8405: TopDocs.maxScore is removed. IndexSearcher and TopFieldCollector
  627   no longer have an option to compute the maximum score when sorting by field.
  628   (Adrien Grand)
  630 * LUCENE-8411: TopFieldCollector no longer takes a fillFields option, it now
  631   always fills fields. (Adrien Grand)
  633 * LUCENE-8412: TopFieldCollector no longer takes a trackDocScores option. Scores
  634   need to be set on top hits via TopFieldCollector#populateScores instead.
  635   (Adrien Grand)
  637 * LUCENE-6228: A new Scorable abstract class has been added, containing only those
  638   methods from Scorer that should be called from Collectors.  LeafCollector.setScorer()
  639   now takes a Scorable rather than a Scorer. (Alan Woodward, Adrien Grand)
  641 * LUCENE-8475: Deprecated constants have been removed from RamUsageEstimator.
  642   (Dimitrios Athanasiou)
  644 * LUCENE-8483: Scorers may no longer take null as a Weight (Alan Woodward)
  646 * LUCENE-8352: TokenStreamComponents is now final, and can take a Consumer<Reader>
  647   in its constructor (Mark Harwood, Alan Woodward, Adrien Grand)
  649 * LUCENE-8498: LowerCaseTokenizer has been removed, and CharTokenizer no longer
  650   takes a normalizer function. (Alan Woodward)
  652 * LUCENE-7875: Moved MultiFields static methods out of the class.  getLiveDocs is now
  653   in MultiBits which is now public.  getMergedFieldInfos and getIndexedFields are now in
  654   FieldInfos.  getTerms is now in MultiTerms.  getTermPositionsEnum and getTermDocsEnum
  655   were collapsed and renamed to just getTermPostingsEnum and moved to MultiTerms.
  656   (David Smiley)
  658 * LUCENE-8513: MultiFields.getFields is now removed.  Please avoid this class,
  659   and Fields in general, when possible. (David Smiley)
  661 * LUCENE-8497: MultiTermAwareComponent has been removed, and in its place
  662   TokenFilterFactory and CharFilterFactory now expose type-safe normalize()
  663   methods.  This decouples normalization from tokenization entirely.
  664   (Mayya Sharipova, Alan Woodward)
  666 * LUCENE-8597: IntervalIterator now exposes a gaps() method that reports the
  667   number of gaps between its component sub-intervals.  This can be used in a 
  668   new filter available via Intervals.maxgaps().  (Alan Woodward)
  670 * LUCENE-8609: Remove IndexWriter#numDocs() and IndexWriter#maxDoc() in favor
  671   of IndexWriter#getDocStats(). (Simon Willnauer)
  673 * LUCENE-8292: Make TermsEnum fully abstract. (Simon Willnauer)
  675 Changes in Runtime Behavior
  677 * LUCENE-8333: Switch MoreLikeThis.setMaxDocFreqPct to use maxDoc instead of
  678   numDocs. (Robert Muir, Dawid Weiss).
  680 * LUCENE-7837: Indices that were created before the previous major version
  681   will now fail to open even if they have been merged with the previous major
  682   version. (Adrien Grand)
  684 * LUCENE-8020: Similarities are no longer passed terms that don't exist by
  685   queries such as SpanOrQuery, so scoring formulas no longer require
  686   divide-by-zero hacks.  IndexSearcher.termStatistics/collectionStatistics return null
  687   instead of returning bogus values for a non-existent term or field. (Robert Muir)
  689 * LUCENE-7996: FunctionQuery and FunctionScoreQuery now return a score of 0
  690   when the function produces a negative value. (Adrien Grand)
  692 * LUCENE-8116: Similarities now score fields that omit norms as if the norm was
  693   1. This might change score values on fields that omit norms. (Adrien Grand)
  695 * LUCENE-8134: Index options are no longer automatically downgraded.
  696   (Adrien Grand)
  698 * LUCENE-8031: Length normalization correctly reflects omission of term frequencies.
  699   (Robert Muir, Adrien Grand)
  701 * LUCENE-7444: StandardAnalyzer no longer defaults to removing English stopwords
  702   (Alan Woodward)
  704 * LUCENE-8060: IndexSearcher's search and searchAfter methods now only compute
  705   total hit counts accurately up to 1,000 in order to enable top-hits
  706   optimizations such as block-max WAND (LUCENE-8135). (Adrien Grand)
  708 * LUCENE-8505: IndexWriter#addIndices will now fail if the target index is sorted but
  709   the candidate is not. (Jim Ferenczi)
  711 * LUCENE-8535: Highlighter and FVH doesn't support ToParent and ToChildBlockJoinQuery out of the
  712   box anymore. In order to highlight on Block-Join Queries a custom WeightedSpanTermExtractor / FieldQuery
  713   should be used. (Simon Willnauer, Jim Ferenczi, Julie Tibshirani)
  715 * LUCENE-8563: BM25 scores don't include the (k1+1) factor in their numerator
  716   anymore. This doesn't affect ordering as this is a constant factor which is
  717   the same for every document. (Luca Cavanna via Adrien Grand)
  719 * LUCENE-8509: WordDelimiterGraphFilter will no longer set the offsets of internal
  720   tokens by default, preventing a number of bugs when the filter is chained with
  721   tokenfilters that change the length of their tokens (Alan Woodward)
  723 * LUCENE-8633: IntervalQuery scores do not use term weighting any more, the score
  724   is instead calculated as a function of the sloppy frequency of the matching
  725   intervals. (Alan Woodward, Jim Ferenczi)
  727 * LUCENE-8635: FSTs can now remain off-heap, accessed via
  728   IndexInput, and the default codec's term dictionary
  729   (BlockTreeTermsReader) will now leave the FST for the terms index
  730   off-heap for non-primary-key fields using MMapDirectory, reducing
  731   heap usage for such fields. (Ankit Jain)
  733 New Features
  735 * LUCENE-8340: LongPoint#newDistanceFeatureQuery may be used to boost scores based on
  736   how close a value of a long field is from an configurable origin. This is
  737   typically useful to boost by recency. (Adrien Grand)
  739 * LUCENE-8482: LatLonPoint#newDistanceFeatureQuery may be used to boost scores
  740   based on the haversine distance of a LatLonPoint field to a provided point. This is
  741   typically useful to boost by distance. (Ignacio Vera)
  743 * LUCENE-8216: Added a new BM25FQuery in sandbox to blend statistics across several fields
  744   using the BM25F formula. (Adrien Grand, Jim Ferenczi)
  746 * LUCENE-8564: GraphTokenFilter is an abstract class useful for token filters that need
  747   to read-ahead in the token stream and take into account graph structures.  This
  748   also changes FixedShingleFilter to extend GraphTokenFilter (Alan Woodward)
  750 * LUCENE-8612: Intervals.extend() treats an interval as if it covered a wider
  751   span than it actually does, allowing users to force minimum gaps between
  752   intervals in a phrase. (Alan Woodward)
  754 * LUCENE-8629: New interval functions: Intervals.before(), Intervals.after(),
  755   Intervals.within() and Intervals.overlapping(). (Alan Woodward)
  757 * LUCENE-8622: Adds a minimum-should-match interval function that produces intervals
  758   spanning a subset of a set of sources. (Alan Woodward)
  760 * LUCENE-8645: Intervals.fixField() allows you to report intervals from one field
  761   as if they came from another. (Alan Woodward)
  763 * LUCENE-8646: New interval functions: Intervals.prefix() and Intervals.wildcard()
  764   (Alan Woodward)
  766 * LUCENE-8655: Add a getter in FunctionScoreQuery class in order to access to the 
  767   underlying DoubleValuesSource. (Gérald Quaire via Alan Woodward)
  769 * LUCENE-8697: GraphTokenStreamFiniteStrings correctly handles side paths 
  770   containing gaps (Alan Woodward)
  772 * LUCENE-8702: Simplify intervals returned from vararg Intervals factory methods
  773   (Alan Woodward)
  775 Improvements
  777 * LUCENE-7997: Add BaseSimilarityTestCase to sanity check similarities.
  778   SimilarityBase switches to 64-bit doubles internally to help avoid common numeric issues.
  779   Add missing range checks for similarity parameters.
  780   Improve BM25 and ClassicSimilarity's explanations. (Robert Muir)
  782 * LUCENE-8011: Improved similarity explanations.
  783   (Mayya Sharipova via Adrien Grand)
  785 * LUCENE-4198: Codecs now have the ability to index score impacts.
  786   (Adrien Grand)
  788 * LUCENE-8135: Boolean queries now implement the block-max WAND algorithm in
  789   order to speed up selection of top scored documents. (Adrien Grand)
  791 * LUCENE-8279: CheckIndex now cross-checks terms with norms. (Adrien Grand)
  793 * LUCENE-8660: TopDocsCollectors now return an accurate count (instead of a lower bound)
  794   if the total hit count is equal to the provided threshold. (Adrien Grand, Jim Ferenczi)
  796 Optimizations
  798 * LUCENE-8040: Optimize IndexSearcher.collectionStatistics, avoiding MultiFields/MultiTerms
  799   (David Smiley, Robert Muir)
  801 * LUCENE-4100: Disjunctions now support faster collection of top hits when the
  802   total hit count is not required. (Stefan Pohl, Adrien Grand, Robert Muir)
  804 * LUCENE-7993: Phrase queries are now faster if total hit counts are not
  805   required. (Adrien Grand)
  807 * LUCENE-8109: Boolean queries propagate information about the minimum
  808   competitive score in order to make collection faster if there are disjunctions
  809   or phrase queries as sub queries, which know how to leverage this information
  810   to run faster. (Adrien Grand)
  812 * LUCENE-8439: Disjunction max queries can skip blocks to select the top documents
  813   if the total hit count is not required. (Jim Ferenczi, Adrien Grand)
  815 * LUCENE-8204: Boolean queries with a mix of required and optional clauses are
  816   now faster if the total hit count is not required. (Jim Ferenczi, Adrien Grand)
  818 * LUCENE-8448: Boolean queries now propagates the mininum score to their sub-scorers.
  819   (Jim Ferenczi, Adrien Grand)
  821 * LUCENE-8511: MultiFields.getIndexedFields is now optimized; does not call getMergedFieldInfos
  822   (David Smiley)
  824 * LUCENE-8507: TopFieldCollector can now update the minimum competitive score if the primary sort
  825   is by relevancy and the total hit count is not required. (Jim Ferenczi)
  827 * LUCENE-8464: ConstantScoreScorer now implements setMinCompetitveScore in order
  828   to early terminate the iterator if the minimum score is greater than the constant
  829   score. (Christophe Bismuth via Jim Ferenczi)
  831 * LUCENE-8607: MatchAllDocsQuery can shortcut when total hit count is not
  832   required (Alan Woodward, Adrien Grand)
  834 * LUCENE-8585: Index-time jump-tables for DocValues, for O(1) advance when retrieving doc values.
  835   (Toke Eskildsen, Adrien Grand)
  837 ======================= Lucene 7.7.2 =======================
  839 Bug fixes
  841 * LUCENE-8726: ValueSource.asDoubleValuesSource() could leak a reference to
  842   IndexSearcher (Alan Woodward, Yury Pakhomov)
  844 * LUCENE-8735: FilterDirectory.getPendingDeletions now forwards to the delegate
  845   even the method is not abstract in the super class. This prevents issues
  846   where our best effort in carrying on generations in the IndexWriter since pending
  847   deletions are swallowed by the FilterDirectory. (Henning Andersen, Simon Willnauer)
  849 * LUCENE-8688: TieredMergePolicy#findForcedMerges now tries to create the
  850   cheapest merges that allow the index to go down to `maxSegmentCount` segments
  851   or less. (Armin Braun via Adrien Grand)
  853 * LUCENE-8785: Ensure new threadstates are locked before retrieving the number of active threadstates.
  854   This causes assertion errors and potentially broken field attributes in the IndexWriter when
  855   IndexWriter#deleteAll is called while actively indexing. (Simon Willnauer)
  857 * LUCENE-8720: NameIntCacheLRU (in the facets module) had an int
  858   overflow bug that disabled cleaning of the cache (Russell A Brown)
  860 * LUCENE-8809: Refresh and rollback concurrently can leave segment states unclosed (Nhat Nguyen)
  862 ======================= Lucene 7.7.1 =======================
  863 (No Changes)
  865 ======================= Lucene 7.7.0 =======================
  867 Changes in Runtime Behavior
  869 * LUCENE-8527: StandardTokenizer and UAX29URLEmailTokenizer now support Unicode 9.0,
  870   and provide Unicode UTS#51 v11.0 Emoji tokenization with the "<EMOJI>" token type. 
  872 Build
  874 * LUCENE-8611: Update randomizedtesting to 2.7.2, JUnit to 4.12, add hamcrest-core 
  875   dependency. (Dawid Weiss)
  877 * LUCENE-8537: ant test command fails under lucene/tools (Peter Somogyi)
  879 Bug fixes:
  881 * LUCENE-8669: Fix LatLonShape WITHIN queries that fail with Multiple search Polygons
  882   that share the dateline. (Nick Knize)
  884 * LUCENE-8603: Fix the inversion of right ids for additional nouns in the Korean user dictionary.
  885   (Yoo Jeongin via Jim Ferenczi)
  887 * LUCENE-8624: int overflow in ByteBuffersDataOutput.size(). (Mulugeta Mammo, 
  888   Dawid Weiss)
  890 * LUCENE-8625: int overflow in ByteBuffersDataInput.sliceBufferList. (Mulugeta Mammo,
  891   Dawid Weiss)
  893 * LUCENE-8639: Newly created threadstates while flushing / refreshing can cause duplicated
  894   sequence IDs on IndexWriter. (Simon Willnauer)
  896 * LUCENE-8649: LatLonShape's within and disjoint queries can return false positives with
  897   indexed multi-shapes. (Ignacio Vera)
  899 * LUCENE-8654: Polygon2D#relateTriangle returns the wrong answer if polygon is inside
  900   the triangle. (Ignacio Vera)
  902 * LUCENE-8650: ConcatenatingTokenStream did not correctly clear its state in reset(), and
  903   was not propagating final position increments from its child streams correctly.
  904   (Dan Meehl, Alan Woodward)
  906 * LUCENE-8676: The Korean tokenizer does not update the last position if the backtrace is caused
  907   by a big buffer (1024 chars). (Jim Ferenczi)
  909 New Features
  911 * LUCENE-8026: ExitableDirectoryReader may now time out queries that run on
  912   points such as range queries or geo queries.
  913   (Christophe Bismuth via Adrien Grand)
  915 * LUCENE-8508: IndexWriter can now set the created version via
  916   IndexWriterConfig#setIndexCreatedVersionMajor. This is an expert feature.
  917   (Adrien Grand)
  919 * LUCENE-8601: Attributes set in the IndexableFieldType for each field during indexing will
  920   now be recorded into the corresponding FieldInfo's attributes, accessible at search
  921   time (Murali Krishna P)
  923 Improvements
  925 * LUCENE-8463: TopFieldCollector can now early-terminates queries when sorting by SortField.DOC.
  926   (Christophe Bismuth via Jim Ferenczi)
  928 * LUCENE-8562: Speed up merging segments of points with data dimensions by only sorting on the indexed
  929   dimensions. (Ignacio Vera)
  931 * LUCENE-8529: TopSuggestDocsCollector will now use the completion key to tiebreak completion
  932   suggestion with identical scores. (Jim Ferenczi)
  934 * LUCENE-8575: SegmentInfos#toString now includes attributes and diagnostics.
  935   (Namgyu Kim via Adrien Grand)
  937 * LUCENE-8548: The KoreanTokenizer no longer splits unknown words on combining diacritics and
  938   detects script boundaries more accurately with Character#UnicodeScript#of.
  939   (Christophe Bismuth, Jim Ferenczi)
  941 * LUCENE-8581: Change LatLonShape encoding to use 4 bytes Per Dimension.
  942   (Ignacio Vera, Nick Knize, Adrien Grand)
  944 * LUCENE-8527: Upgrade JFlex dependency to 1.7.0; in StandardTokenizer and UAX29URLEmailTokenizer,
  945   increase supported Unicode version from 6.3 to 9.0, and support Unicode UTS#51 v11.0 Emoji tokenization.
  947 * LUCENE-8640: Date Range format validation (Lucky Sharma, David Smiley via Mikhail Khludnev)
  949 Optimizations
  951 * LUCENE-8552: FieldInfos.getMergedFieldInfos no longer does any merging if there is <= 1 segment.
  952   (Christophe Bismuth via David Smiley)
  954 * LUCENE-8590: BufferedUpdates now uses an optimized storage for buffering docvalues updates that
  955   can safe up to 80% of the heap used compared to the previous implementation and uses non-object
  956   based datastructures. (Simon Willnauer, Mike McCandless, Shai Erera, Adrien Grand)
  958 * LUCENE-8598: Moved to the default accepted overhead ratio for packet ints in DocValuesFieldUpdats
  959   yields an up-to 4x performance improvement when applying doc values updates. (Simon Willnauer, Adrien Grand)
  961 * LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates. 
  962   (Simon Willnauer, Adrien Grand)
  964 * LUCENE-8600: Doc-value updates get applied faster by sorting with quicksort,
  965   rather than an in-place mergesort, which needs to perform fewer swaps.
  966   (Adrien Grand)
  968 * LUCENE-8623: Decrease I/O pressure when merging high dimensional points. (Ignacio Vera)
  970 Test Framework
  972 * LUCENE-8604: TestRuleLimitSysouts now has an optional "hard limit" of bytes that can be written
  973   to stderr and stdout (anything beyond the hard limit is ignored). The default hard limit is 2 GB of 
  974   logs per test class. (Dawid Weiss)
  976 Other
  978 * LUCENE-8573: BKDWriter now uses FutureArrays#mismatch to compute shared prefixes.
  979   (Christoph Büscher via Adrien Grand)
  981 * LUCENE-8605: Separate bounding box spatial logic from query logic on LatLonShapeBoundingBoxQuery.
  982   (Ignacio Vera)
  984 * LUCENE-8609: Deprecated IndexWriter#numDocs() and IndexWriter#maxDoc() in favor of IndexWriter#getDocStats()
  985   that allows to get consistent numDocs and maxDoc stats that are not subject to concurrent changes.
  986   (Simon Willnauer, Nhat Nguyen)
  988 ======================= Lucene 7.6.0 =======================
  990 Build
  992 * LUCENE-8504: Upgrade forbiddenapis to version 2.6.  (Uwe Schindler)
  994 * LUCENE-8493: Stop publishing insecure .sha1 files with releases (janhoy)
  996 Bug fixes
  998 * LUCENE-8479: QueryBuilder#analyzeGraphPhrase now throws TooManyClause exception
  999   if the number of expanded path reaches the BooleanQuery#maxClause limit. (Jim Ferenczi)
 1001 * LUCENE-8522: throw InvalidShapeException when constructing a polygon and
 1002   all points are coplanar. (Ignacio Vera)
 1004 * LUCENE-8531: QueryBuilder#analyzeGraphPhrase now creates one phrase query per finite strings
 1005   in the graph if the slop is greater than 0. Span queries cannot be used in this case because
 1006   they don't handle slop the same way than phrase queries. (Steve Rowe, Uwe Schindler, Jim Ferenczi)
 1008 * LUCENE-8524: Add the Hangul Letter Araea (interpunct) as a separator in Nori's tokenizer.
 1009   This change also removes empty terms and trim surface form in Nori's Korean dictionary. (Trey Jones, Jim Ferenczi)
 1011 * LUCENE-8550: Fix filtering of coplanar points when creating linked list on
 1012   polygon tesselator. (Ignacio Vera)
 1014 * LUCENE-8549: Polygon tessellator throws an error if some parts of the shape
 1015    could not be processed. (Ignacio Vera)
 1017 * LUCENE-8540: Better handling of min/max values for Geo3d encoding. (Ignacio Vera)
 1019 * LUCENE-8534: Fix incorrect computation for triangles intersecting polygon edges in
 1020   shape tessellation. (Ignacio Vera)
 1022 * LUCENE-8559: Fix bug where polygon edges were skipped when checking for intersections.
 1023   (Ignacio Vera)
 1025 * LUCENE-8556: Use latitude and longitude instead of encoding values to check if triangle is ear
 1026   when using morton optimisation. (Ignacio Vera)
 1028 * LUCENE-8586: Intervals.or() could get stuck in an infinite loop on certain indexes
 1029   (Alan Woodward)
 1031 * LUCENE-8595: Fix interleaved DV update and reset. Interleaved update and reset value
 1032   to the same doc in the same updates package looses an update if the reset comes before
 1033   the update as well as loosing the reset if the update comes frist. (Simon Willnauer, Adrien Grand)
 1035 * LUCENE-8592: Fix index sorting corruption due to numeric overflow. The merge of sorted segments
 1036   can produce an invalid sort if the sort field is an Integer/Long that uses reverse order and contains
 1037   values equal to Integer/Long#MIN_VALUE. These values are always sorted first during a merge
 1038   (instead of last because of the reverse order) due to this bug. Indices affected by the bug can be
 1039   detected by running the CheckIndex command on a distribution that contains the fix (7.6+).
 1040   (Jim Ferenczi, Adrien Grand, Mike McCandless, Simon Willnauer)
 1042 New Features
 1044 * LUCENE-8496: Selective indexing - modify BKDReader/BKDWriter to allow users
 1045   to select a fewer number of dimensions to be used for creating the index than
 1046   the total number of dimensions used for field encoding. i.e., dimensions 0 to N
 1047   may be used to determine how to split the inner nodes, and dimensions N+1 to D
 1048   are ignored and stored as data dimensions at the leaves. (Nick Knize)
 1050 * LUCENE-8538: Add a Simple WKT Shape Parser for creating Lucene Geometries (Polygon, Line,
 1051   Rectangle) from WKT format. (Nick Knize)
 1053 * LUCENE-8462: Adds an Arabic snowball stemmer based on
 1054   https://github.com/snowballstem/snowball/blob/master/algorithms/arabic.sbl 
 1055   (Ryadh Dahimene via Jim Ferenczi)
 1057 * LUCENE-8554: Add new LatLonShapeLineQuery that queries indexed LatLonShape fields
 1058   by arbitrary lines. (Nick Knize)
 1060 * LUCENE-8555: Add dateline crossing support to LatLonShapeBoundingBoxQuery. (Nick Knize)
 1062 Improvements
 1064 * LUCENE-8521: Change LatLonShape encoding to 7 dimensions instead of 6; where the
 1065   first 4 are index dimensions defining the bounding box of the Triangle and the
 1066   remaining 3 data dimensions define the vertices of the triangle. (Nick Knize)
 1068 * LUCENE-8557: LeafReader.getFieldInfos is now documented and tested that it ought to return
 1069   the same cached instance.  MemoryIndex's impl now pre-creates the FieldInfos instead of
 1070   re-calculating a new instance each time.  (Tim Underwood, David Smiley)
 1072 * LUCENE-8558: Replace O(N) lookup with O(1) lookup in PerFieldMergeState#FilterFieldInfos.
 1073   (Kranthi via Simon Willnauer)
 1075 Other
 1077 * LUCENE-8523: Correct typo in JapaneseNumberFilterFactory javadocs (Ankush Jhalani
 1078   via Alan Woodward)
 1080 * LUCENE-8533: Fix Javadocs of DataInput#readVInt(): Negative numbers are
 1081   supported, but should be avoided. (Vladimir Dolzhenko via Uwe Schindler)
 1083 ======================= Lucene 7.5.1 =======================
 1085 Bug Fixes
 1087 * LUCENE-8454: Fix incorrect vertex indexing and other computation errors in
 1088   shape tessellation that would sometimes cause an infinite loop. (Nick Knize)
 1090 ======================= Lucene 7.5.0 =======================
 1092 API Changes
 1094 * LUCENE-8467: RAMDirectory, RAMFile, RAMInputStream, RAMOutputStream are deprecated
 1095   (Dawid Weiss)
 1097 * LUCENE-8356: StandardFilter is deprecated (Alan Woodward)
 1099 * LUCENE-8373: ENGLISH_STOP_WORD_SET on StandardAnalyzer is deprecated.  Instead
 1100   use EnglishAnalyzer.ENGLISH_STOP_WORD_SET.  The default constructor for
 1101   StopAnalyzer is also deprecated, and a stop word set should be explicitly
 1102   passed to the constructor.  (Alan Woodward)
 1104 * LUCENE-8378: Add DocIdSetIterator.range static method to return an iterator
 1105   matching a range of docids (Mike McCandless)
 1107 * LUCENE-8379: Add experimental TermQuery.getTermStates method (Mike McCandless)
 1109 * LUCENE-8407: Add experimental SpanTermQuery.getTermStates method (David Smiley)
 1111 * LUCENE-8390: MatchesIteratorSupplier replaced by IOSupplier (Alan Woodward,
 1112   David Smiley)
 1114 * LUCENE-8397: Add DirectoryTaxonomyWriter.getCache (Mike McCandless)
 1116 * LUCENE-8387: Add experimental IndexSearcher.getSlices API to see which slices
 1117   IndexSearcher is searching concurrently when it's created with an ExecutorService
 1118   (Mike McCandless)
 1120 * LUCENE-8263: TieredMergePolicy's reclaimDeletesWeight has been replaced with a
 1121   new deletesPctAllowed setting to control how aggressively deletes should be
 1122   reclaimed. (Erick Erickson, Adrien Grand)
 1124 * LUCENE-7314: Graduate LatLonPoint and query classes to core (Nick Knize)
 1126 * LUCENE-8428: The way that oal.util.PriorityQueue creates sentinel objects has
 1127   been changed from a protected method to a java.util.function.Supplier as a
 1128   constructor argument. (Adrien Grand)
 1130 * LUCENE-8437: CheckIndex.Status.cantOpenSegments and missingSegmentVersion
 1131   have been removed as they were not computed correctly. (Adrien Grand)
 1133 * LUCENE-8286: The UnifiedHighlighter has a new HighlightFlag.WEIGHT_MATCHES flag that
 1134   will tell this highlighter to use the new MatchesIterator API as the underlying
 1135   approach to navigate matching hits for a query.  This mode will highlight more
 1136   accurately than any other highlighter, and can mark up phrases as one span instead of
 1137   word-by-word.  The UH's public internal APIs changed a bit in the process.
 1138   (David Smiley)
 1140 * LUCENE-8471: IndexWriter.getFlushingBytes() returns how many bytes are currently
 1141   being flushed to disk. (Alan Woodward)
 1143 * LUCENE-8422: Static helper functions for Matches and MatchesIterator implementations
 1144   have been moved from Matches to MatchesUtils (Alan Woodward)
 1146 * LUCENE-8343: Suggesters now require Long (versus long, previously) from weight() method
 1147   while indexing, and provide double (versus long, previously) scores at lookup time
 1148   (Alessandro Benedetti)
 1150 * LUCENE-8459: SearcherTaxonomyManager now has a constructor taking already opened
 1151   IndexReaders, allowing the caller to pass a FilterDirectoryReader, for example.
 1152   (Mike McCandless)
 1154 Bug Fixes
 1156 * LUCENE-8445: Tighten condition when two planes are identical to prevent constructing
 1157   bogus tiles when building GeoPolygons. (Ignacio Vera)
 1159 * LUCENE-8444: Prevent building functionally identical plane bounds when constructing
 1160   DualCrossingEdgeIterator . (Ignacio Vera)
 1162 * LUCENE-8380: UTF8TaxonomyWriterCache inconsistency. (Ruslan Torobaev, Dawid Weiss)
 1164 * LUCENE-8164: IndexWriter silently accepts broken payload. This has been fixed
 1165   via LUCENE-8165 since we are now checking for offset+length going out of bounds.
 1166   (Robert Muir, Nhat Nyugen, Simon Willnauer)
 1168 * LUCENE-8370: Reproducing 
 1169   TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()
 1170   failures (Erick Erickson)
 1172 * LUCENE-8376, LUCENE-8371: ConditionalTokenFilter.end() would not propagate correctly
 1173   if the last token in the stream was subsequently dropped; FixedShingleFilter did
 1174   not set position increment in end() (Alan Woodward)
 1176 * LUCENE-8395: WordDelimiterGraphFilter would incorrectly insert a hole into a
 1177   TokenStream if a token consisting entirely of delimiter characters was 
 1178   encountered, but preserve_original was set. (Alan Woodward)
 1180 * LUCENE-8398: TieredMergePolicy.getMaxMergedSegmentMB has rounding error (Erick Erickson)
 1182 * LUCENE-8429: DaciukMihovAutomatonBuilder is no longer prone to stack
 1183   overflows by enforcing a maximum term length. (Adrien Grand)
 1185 * LUCENE-8441: IndexWriter now checks doc value type for index sort fields
 1186   and fails the document if they are not compatible. (Jim Ferenczi, Mike McCandless)
 1188 * LUCENE-8458: Adjust initialization condition of PendingSoftDeletes and ensures
 1189   it is initialized before accepting deletes (Simon Willnauer, Nhat Nguyen)
 1191 * LUCENE-8466: IndexWriter.deleteDocs(Query... query) incorrectly applies deletes on flush
 1192   if the index is sorted. (Adrien Grand, Jim Ferenczi, Vish Ramachandran)
 1194 * LUCENE-8502: Allow access to delegate in FilterCodecReader. FilterCodecReader didn't 
 1195   allow access to it's delegate like other filter readers. This adds a new #getDelegate method
 1196   to access the wrapped reader. (Simon Willnauer)
 1198 Changes in Runtime Behavior
 1200 * LUCENE-7976: TieredMergePolicy now respects maxSegmentSizeMB by default when executing
 1201   findForcedMerges and findForcedDeletesMerges (Erick Erickson)
 1203 * LUCENE-8263: TieredMergePolicy now reclaims deleted documents more
 1204   aggressively by default ensuring that no more than ~1/3 of the index size is
 1205   used by deleted documents. (Adrien Grand)
 1207 * LUCENE-8503: Call #getDelegate instead of direct member access during unwrap. 
 1208   Filter*Reader instances access the member or the delegate directly instead of 
 1209   calling getDelegate(). In order to track access of the delegate these methods
 1210   should call #getDelegate() (Simon Willnauer)
 1212 Improvements
 1214 * LUCENE-8468: A ByteBuffer based Directory implementation. (Dawid Weiss)
 1216 * LUCENE-8447: Add DISJOINT and WITHIN support to LatLonShape queries. (Nick Knize)
 1218 * LUCENE-8440: Add support for indexing and searching Line and Point shapes using LatLonShape encoding (Nick Knize)
 1220 * LUCENE-8435: Add new LatLonShapePolygonQuery for querying indexed LatLonShape fields by arbitrary polygons (Nick Knize)
 1222 * LUCENE-8367: Make per-dimension drill down optional for each facet dimension (Mike McCandless)
 1224 * LUCENE-8396: Add Points Based Shape Indexing and Search that decomposes shapes
 1225   into a triangular mesh and indexes individual triangles as a 6 dimension point (Nick Knize)
 1227 * LUCENE-8345, GitHub PR #392: Remove instantiation of redundant wrapper classes for primitives;
 1228   add wrapper class constructors to forbiddenapis.  (Michael Braun via Uwe Schindler)
 1230 * LUCENE-8415: Clean up Directory contracts and JavaDoc comments. (Dawid Weiss)
 1232 * LUCENE-8414: Make segmentInfos private in IndexWriter (Simon Willnauer, Nhat Nguyen)
 1234 * LUCENE-8446: The UnifiedHighlighter's DefaultPassageFormatter now treats overlapping matches in
 1235   the passage as merged (as if one larger match).  (David Smiley)
 1237 * LUCENE-8460: Better argument validation in StoredField. (Namgyu Kim)
 1239 * LUCENE-8432: TopFieldComparator stops comparing documents if the index is
 1240   sorted, even if hits still need to be visited to compute the hit count.
 1241   (Nikolay Khitrin)
 1243 * LUCENE-8422: IntervalQuery now returns useful Matches (Alan Woodward)
 1245 * LUCENE-7862: Store the real bounds of the leaf cells in the BKD index when the
 1246   number of dimensions is bigger than 1. It improves performance when there is
 1247   correlation between the dimensions, for example ranges. (Ignacio Vera, Adrien Grand)
 1249 Build
 1251 * LUCENE-5143: Stop publishing KEYS file with each version, use topmost lucene/KEYS file only. 
 1252   The buildAndPushRelease.py script validates that RM's PGP key is in the KEYS file.
 1253   Remove unused 'copy-to-stage' and '-dist-keys' targets from ant build. (janhoy)
 1255 Other
 1257 * LUCENE-8485: Update randomizedtesting to version 2.6.4. (Dawid Weiss)
 1259 * LUCENE-8366: Upgrade to ICU 62.1. Emoji handling now uses Unicode 11's
 1260   Extended_Pictographic property. (Robert Muir)
 1262 * LUCENE-8408: original Highlighter:  Remove obsolete static AttributeFactory instance
 1263   in TokenStreamFromTermVector.  (Michael Braun, David Smiley)
 1265 * LUCENE-8420: Upgrade OpenNLP to 1.9.0 so OpenNLP tool can read the new model format which 1.8.x
 1266   cannot read. 1.9.0 can read the old format. (Koji Sekiguchi)
 1268 * LUCENE-8453: Add documentation to analysis factories of Korean (Nori) analyzer
 1269   module.  (Tomoko Uchida via Uwe Schindler)
 1271 * LUCENE-8455: Upgrade ECJ compiler to 4.6.1 in lucene/common-build.xml (Erick Erickson)
 1273 * LUCENE-8456: Upgrade Apache Commons Compress to v1.18 (Steve Rowe)
 1275 * LUCENE-765: Improved org.apache.lucene.index javadocs. (Mike Sokolov)
 1277 * LUCENE-8476: Remove redundant nullity check and switch to optimized List.sort in the
 1278   Korean's user dictionary. (Namgyu Kim)
 1280 ======================= Lucene 7.4.1 =======================
 1282 Bug Fixes
 1284  * LUCENE-8365: Fix ArrayIndexOutOfBoundsException in UnifiedHighlighter. This fixes 
 1285    a "off by one" error in the UnifiedHighlighter's code that is only triggered when
 1286    two nested SpanNearQueries contain the same term. (Marc-Andre Morissette via Simon Willnauer)
 1288  * LUCENE-8381: Fix IndexWriter incorrectly interprets hard-deletes as soft-deletes
 1289    while wrapping reader for merges. (Simon Willnauer, Nhat Nguyen)
 1291  * LUCENE-8384: Fix missing advance docValues generation while handling docValues
 1292    update in PendingSoftDeletes. (Simon Willnauer, Nhat Nguyen)
 1294  * LUCENE-8472: Always rewrite the soft-deletes merge retention query. (Adrien Grand, Nhat Nguyen)
 1296 ======================= Lucene 7.4.0 =======================
 1298 Upgrading
 1300 * LUCENE-8344: If you are using the AnalyzingSuggester or FuzzySuggester subclass, and if you
 1301   explicitly use the preservePositionIncrements=false setting (not the default), then you ought
 1302   to rebuild your suggester index. If you don't, queries or indexed data with trailing position
 1303   gaps (e.g. stop words) may not work correctly. (David Smiley, Jim Ferenczi)
 1305 API Changes
 1307 * LUCENE-8242: IndexSearcher.createNormalizedWeight() has been deprecated.
 1308   Instead use IndexSearcher.createWeight(), rewriting the query first.
 1309   (Alan Woodward)
 1311 * LUCENE-8248: MergePolicyWrapper is renamed to FilterMergePolicy and now
 1312   also overrides getMaxCFSSegmentSizeMB (Mike Sokolov via Mike McCandless)
 1314 * LUCENE-8303: LiveDocsFormat is now only responsible for (de)serialization of
 1315   live docs. (Adrien Grand)
 1317 Changes in Runtime Behavior
 1319 * LUCENE-8309: Live docs are no longer backed by a FixedBitSet. (Adrien Grand)
 1321 * LUCENE-8330: Detach IndexWriter from MergePolicy. MergePolicy now instead of 
 1322   requiring IndexWriter as a hard dependency expects a MergeContext which
 1323   IndexWriter implements. (Simon Willnauer, Robert Muir, Dawid Weiss, Mike McCandless)
 1325 New Features
 1327 * LUCENE-8200: Allow doc-values to be updated atomically together
 1328   with a document. Doc-Values updates now can be used as a soft-delete
 1329   mechanism to all keeping several version of a document or already 
 1330   deleted documents around for later reuse. See "IW.softUpdateDocument(...)"
 1331   for reference. (Simon Willnauer)
 1333 * LUCENE-8197: A new FeatureField makes it easy and efficient to integrate
 1334   static relevance signals into the final score. (Adrien Grand, Robert Muir)
 1336 * LUCENE-8202: Add a FixedShingleFilter (Alan Woodward, Adrien Grand, Jim
 1337   Ferenczi)
 1339 * LUCENE-8125: ICUTokenizer support for emoji/emoji sequence tokens. (Robert Muir)
 1341 * LUCENE-8196, LUCENE-8300: A new IntervalQuery in the sandbox allows efficient proximity
 1342   searches based on minimum-interval semantics. (Alan Woodward, Adrien Grand,
 1343   Jim Ferenczi, Simon Willnauer, Matt Weber)
 1345 * LUCENE-8233: Add support for soft deletes to IndexWriter delete accounting. 
 1346   Soft deletes are accounted for inside the index writer and therefor also
 1347   by merge policies. A SoftDeletesRetentionMergePolicy is added that allows
 1348   to selectively carry over soft_deleted document across merges for retention
 1349   policies (Simon Willnauer, Mike McCandless, Robert Muir)
 1351 * LUCENE-8237: Add a SoftDeletesDirectoryReaderWrapper that allows to respect
 1352   soft deletes if the reader is opened form a directory. (Simon Willnauer,
 1353   Mike McCandless, Uwe Schindler, Adrien Grand)
 1355 * LUCENE-8229, LUCENE-8270: Add a method Weight.matches(LeafReaderContext, doc) 
 1356   that returns an iterator over matching positions for a given query and document.
 1357   This allows exact hit extraction and will enable implementation of accurate 
 1358   highlighters. (Alan Woodward, Adrien Grand, David Smiley)
 1360 * LUCENE-8249: Implement Matches API for phrase queries (Alan Woodward, Adrien 
 1361   Grand)
 1363 * LUCENE-8246: Allow to customize the number of deletes a merge claims. This
 1364   helps merge policies in the soft-delete case to correctly implement retention
 1365   policies without triggering uncessary merges. (Simon Willnauer, Mike McCandless)
 1367 * LUCENE-8231: A new analysis module (nori) similar to Kuromoji
 1368   but to handle Korean using mecab-ko-dic and morphological analysis.
 1369   (Robert Muir, Jim Ferenczi)
 1371 * LUCENE-8265: WordDelimter/GraphFilter now have an option to skip tokens
 1372   marked with KeywordAttribute (Mike Sokolov via Mike McCandless)
 1374 * LUCENE-8297: Add IW#tryUpdateDocValues(Reader, int, Fields...) IndexWriter can
 1375   update doc values for a specific term but this might affect all documents 
 1376   containing the term. With tryUpdateDocValues users can update doc-values 
 1377   fields for individual documents. This allows for instance to soft-delete
 1378   individual documents. (Simon Willnauer)
 1380 * LUCENE-8298: Allow DocValues updates to reset a value. Passing a DV field with a null
 1381   value to IW#updateDocValues or IW#tryUpdateDocValues will now remove the value from the 
 1382   provided document. This allows to undelete a soft-deleted document unless it's been claimed
 1383   by a merge. (Simon Willnauer) 
 1385 * LUCENE-8273: ConditionalTokenFilter allows analysis chains to skip particular token
 1386   filters based on the attributes of the current token. This generalises the keyword
 1387   token logic currently used for stemmers and WDF.  It is integrated into 
 1388   CustomAnalyzer by using the `when` and `whenTerm` builder methods, and a new
 1389   ProtectedTermFilter is added as an example.  (Alan Woodward, Robert Muir,
 1390   David Smiley, Steve Rowe, Mike Sokolov)
 1392 * LUCENE-8310: Ensure IndexFileDeleter accounts for pending deletes. Today we fail
 1393   creating the IndexWriter when the directory has a pending delete. Yet, this
 1394   is mainly done to prevent writing still existing files more than once.
 1395   IndexFileDeleter already accounts for that for existing files which we can
 1396   now use to also take pending deletes into account which ensures that all file
 1397   generations per segment always go forward. (Simon Willnauer)
 1399 * LUCENE-7960: Add preserveOriginal option to the NGram and EdgeNGram filters.
 1400   (Ingomar Wesp, Shawn Heisey via Robert Muir)
 1402 * LUCENE-8335: Enforce soft-deletes field up-front. Soft deletes field must be marked
 1403   as such once it's introduced and can't be changed after the fact.
 1404   (Nhat Nguyen via Simon Willnauer)
 1406 * LUCENE-8332: New ConcatenateGraphFilter for concatenating all tokens into one (or more
 1407   in the event of a graph input).  This is useful for fast analyzed exact-match lookup,
 1408   suggesters, and as a component of a named entity recognition system.  This was excised
 1409   out of CompletionTokenStream in the NRT doc suggester.  (David Smiley, Jim Ferenczi)
 1411 Bug Fixes
 1413 * LUCENE-8221: MoreLikeThis.setMaxDocFreqPct can easily int-overflow on larger
 1414   indexes.
 1416 * LUCENE-8266: Detect bogus tiles when creating a standard polygon and
 1417   throw a TileException. (Ignacio Vera)
 1419 * LUCENE-8234: Fixed bug in how spatial relationship is computed for
 1420   GeoStandardCircle when it covers the whole world. (Ignacio Vera)
 1422 * LUCENE-8236: Filter duplicated points when creating GeoPath shapes to
 1423   avoid creation of bogus planes. (Ignacio Vera)
 1425 * LUCENE-8243: IndexWriter.addIndexes(Directory[]) did not properly preserve
 1426   index file names for updated doc values fields (Simon Willnauer,
 1427   Michael McCandless, Nhat Nguyen)
 1429 * LUCENE-8275: Push up #checkPendingDeletes to Directory to ensure IW fails if 
 1430   the directory has pending deletes files even if the directory is filtered or 
 1431   a FileSwitchDirectory (Simon Willnauer, Robert Muir)
 1433 * LUCENE-8244: Do not leak open file descriptors in SearcherTaxonomyManager's
 1434   refresh on exception (Mike McCandless)
 1436 * LUCENE-8305: ComplexPhraseQuery.rewrite now handles an embedded MultiTermQuery
 1437   that rewrites to a MatchNoDocsQuery instead of throwing an exception.
 1438   (Bjarke Mortensen, Andy Tran via David Smiley)
 1440 * LUCENE-8287: Ensure that empty regex completion queries always return no results.
 1441   (Julie Tibshirani via Jim Ferenczi)
 1443 * LUCENE-8317: Prevent concurrent deletes from being applied during full flush.
 1444   Future deletes could potentially be exposed to flushes/commits/refreshes if the 
 1445   amount of RAM used by deletes is greater than half of the IW RAM buffer. (Simon Willnauer)
 1447 * LUCENE-8320: Fix WindowsFS to correctly account for rename and hardlinks. 
 1448   (Simon Willnauer, Nhat Nguyen)
 1450 * LUCENE-8328: Ensure ReadersAndUpdates consistently executes under lock.
 1451   (Nhat Nguyen via Simon Willnauer)
 1453 * LUCENE-8325: Fixed the smartcn tokenizer to not split UTF-16 surrogate pairs.
 1454   (chengpohi via Jim Ferenczi)
 1456 * LUCENE-8186: LowerCaseTokenizerFactory now lowercases text in multi-term
 1457   queries. (Tim Allison via Adrien Grand)
 1459 * LUCENE-8278: Some end-of-input no-scheme domain-only URL tokens are typed as
 1460   <ALPHANUM> rather than <URL>.  (Junte Zhang, Steve Rowe)
 1462 * LUCENE-8355: Prevent IW from opening an already dropped segment while DV updates
 1463   are written. (Nhat Nguyen via Simon Willnauer)
 1465 * LUCENE-8344: TokenStreamToAutomaton (used by some suggesters) was not ignoring a trailing
 1466   position increment when the preservePositionIncrement setting is false.
 1467   (David Smiley, Jim Ferenczi)
 1469 * LUCENE-8357: FunctionScoreQuery.boostByQuery() and boostByValue() were
 1470   producing truncated Explanations (Markus Jelsma, Alan Woodward)
 1472 * LUCENE-8360: NGramTokenFilter and EdgeNGramTokenFilter did not correctly
 1473   set position increments in end() (Alan Woodward)
 1475 Other
 1477 * LUCENE-8301: Update randomizedtesting to 2.6.0. (Dawid Weiss)
 1479 * LUCENE-8299: Geo3D wrapper uses new polygon method factory that gives better
 1480   support for polygons with many points (>100). (Ignacio vera)
 1482 * LUCENE-8261: InterpolatedProperties.interpolate and recursive property
 1483   references. (Steve Rowe, Dawid Weiss)
 1485 * LUCENE-8228: removed obsolete IndexDeletionPolicy clone() requirements from
 1486   the javadoc. (Dawid Weiss)
 1488 * LUCENE-8219: Use a realistic estimate of the number of nodes and links in
 1489    LevensteinAutomaton.java, to save reallocation of arrays.
 1490    (Christian Ziech)
 1492 * LUCENE-8214: Improve selection of testPoint for GeoComplexPolygon.
 1493   (Ignacio Vera)
 1495 * SOLR-10912: Add automatic patch validation. (Mano Kovacs, Steve Rowe)
 1497 * LUCENE-8122, LUCENE-8175: Upgrade analysis/icu to ICU 61.1.
 1498   (Robert Muir, Adrien Grand, Uwe Schindler)
 1500 * LUCENE-8291: Remove QueryTemplateManager utility class from XML queryparser.
 1501   This class is just a general XML transforming tool (using property files and
 1502   XSLT) and has nothing to do with query parsing. It can easily be implemented
 1503   using more sophisticated libraries or using XSL transformers from the JDK.
 1504   This change also removes the Lucene demo webapp to prevent XSS issues in
 1505   untested/unmaintained code. (Uwe Schindler)
 1507 Build
 1509 * LUCENE-7935: Publish .sha512 hash files with the release artifacts and stop
 1510   publishing .md5 hashes since the algorithm is broken (janhoy)
 1512 * LUCENE-8230: Upgrade forbiddenapis to version 2.5.  (Uwe Schindler)
 1514 Documentation
 1516 * LUCENE-8238: Improve WordDelimiterFilter and WordDelimiterGraphFilter javadocs
 1517   (Mike Sokolov via Mike McCandless)
 1519 ======================= Lucene 7.3.1 =======================
 1521 Bug fixes
 1523 * LUCENE-8254: LRUQueryCache could cause IndexReader to hang on close, when
 1524   shared with another reader with no CacheHelper (Alan Woodward, Simon Willnauer,
 1525   Adrien Grand)
 1527 ======================= Lucene 7.3.0 =======================
 1529 API Changes
 1531 * LUCENE-8051: LevensteinDistance renamed to LevenshteinDistance.
 1532   (Pulak Ghosh via Adrien Grand)
 1534 * LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery and BoostingQuery.
 1535   Users should instead use FunctionScoreQuery, possibly combined with
 1536   a lucene expression (Alan Woodward)
 1538 * LUCENE-8104: Remove facets module compile-time dependency on queries
 1539   (Alan Woodward)
 1541 * LUCENE-8145: UnifiedHighlighter now uses a unitary OffsetsEnum rather
 1542   than a list of enums (Alan Woodward, David Smiley, Jim Ferenczi, Timothy
 1543   Rodriguez)
 1545 New Features
 1547 * LUCENE-2899: Add new module analysis/opennlp, with analysis components
 1548   to perform tokenization, part-of-speech tagging, lemmatization and phrase
 1549   chunking by invoking the corresponding OpenNLP tools. Named entity
 1550   recognition is also provided as a Solr update request processor.
 1551   (Lance Norskog, Grant Ingersoll, Joern Kottmann, Em, Kai Gülzau,
 1552   Rene Nederhand, Robert Muir, Steven Bower, Steve Rowe)
 1554 * LUCENE-8126: Add new spatial prefix tree (SPT) based on google S2 geometry.
 1555   It can only be used currently with Geo3D spatial context and it provides
 1556   improvements on indexing time for non-points shapes and on query performance.
 1557   (Ignacio Vera, David Smiley).
 1559 Improvements
 1561 * LUCENE-8081: Allow IndexWriter to opt out of flushing on indexing threads
 1562   Index/Update Threads try to help out flushing pending document buffers to
 1563   disk. This change adds an expert setting to opt ouf of this behavior unless
 1564   flusing is falling behind. (Simon Willnauer)
 1566 * LUCENE-8086: spatial-extras Geo3dFactory: Use GeoExactCircle with
 1567   configurable precision for non-spherical planet models.
 1568   (Ignacio Vera via David Smiley)
 1570 * LUCENE-8093: TrimFilterFactory implements MultiTermAwareComponent (Alan Woodward)
 1572 * LUCENE-8094: TermInSetQuery.toString now returns "field:(A B C)" (Mike McCandless)
 1574 * LUCENE-8121: UnifiedHighlighter passage relevancy is improved for terms that are
 1575   position sensitive (e.g. part of a phrase) by having an accurate freq.
 1576   (David Smiley)
 1578 * LUCENE-8129: A Unicode set filter can now be specified when using ICUFoldingFilter.
 1579   (Ere Maijala)
 1581 * LUCENE-7966: Build Multi-Release JARs to enable usage of optimized intrinsic methods
 1582   from Java 9 for index bounds checking and array comparison/mismatch. This change
 1583   introduces Java 8 replacements for those Java 9 methods and patches the compiled
 1584   classes to use the optimized variants through the MR-JAR mechanism.
 1585   (Uwe Schindler, Robert Muir, Adrien Grand, Mike McCandless)
 1587 * LUCENE-8127: Speed up rewriteNoScoring when there are no MUST clauses.
 1588   (Michael Braun via Adrien Grand)
 1590 * LUCENE-8152: Improve consumption of doc-value iterators. (Horatiu Lazu via
 1591   Adrien Grand)
 1593 * LUCENE-8033: FieldInfos now always use a dense encoding. (Mayya Sharipova
 1594   via Adrien Grand)
 1596 * LUCENE-8190: Specialized cell interface to allow any spatial prefix tree to
 1597   benefit from the setting setPruneLeafyBranches on RecursivePrefixTreeStrategy.
 1598   (Ignacio Vera)
 1600 Bug Fixes
 1602 * LUCENE-8077: Fixed bug in how CheckIndex verifies doc-value iterators.
 1603   (Xiaoshan Sun via Adrien Grand)
 1605 * SOLR-11758: Fixed FloatDocValues.boolVal to correctly return true for all values != 0.0F
 1606   (Munendra S N via hossman)
 1608 * LUCENE-8121: The UnifiedHighlighter would highlight some terms within some nested
 1609   SpanNearQueries at positions where it should not have.  It's fixed in the UH by
 1610   switching to the SpanCollector API.  The original Highlighter still has this
 1611   problem (LUCENE-2287, LUCENE-5455, LUCENE-6796).  Some public but internal parts of
 1612   the UH were refactored. (David Smiley, Steve Davids)
 1614 * LUCENE-8120: Fix LatLonBoundingBox's toString() method (Martijn van Groningen, Adrien Grand)
 1616 * LUCENE-8130: Fix NullPointerException from TermStates.toString() (Mike McCandless)
 1618 * LUCENE-8124: Fixed HyphenationCompoundWordTokenFilter to handle correctly
 1619   hyphenation patterns with indicator >= 7. (Holger Bruch via Adrien Grand)
 1621 * LUCENE-8163: BaseDirectoryTestCase could produce random filenames that fail
 1622   on Windows (Alan Woodward)
 1624 * LUCENE-8174: Fixed {Float,Double,Int,Long}Range.toString(). (Oliver Kaleske
 1625   via Adrien Grand)
 1627 * LUCENE-8182: Fixed BoostingQuery to apply the context boost instead of the parent query
 1628   boost (Jim Ferenczi)
 1630 * LUCENE-8188: Fixed bugs in OpenNLPOpsFactory that were causing InputStreams fetched from the
 1631   ResourceLoader to be leaked (hossman)
 1634 Other
 1636 * LUCENE-8111: IndexOrDocValuesQuery Javadoc references outdated method name.
 1637   (Kai Chan via Adrien Grand)
 1639 * LUCENE-8106: Add script (reproduceJenkinsFailures.py) to attempt to reproduce
 1640   failing tests from a Jenkins log. (Steve Rowe)
 1642 * LUCENE-8075: Removed unnecessary null check in IntersectTermsEnum.
 1643   (Pulak Ghosh via Adrien Grand)
 1645 * LUCENE-8156: Require users to not have ASM on the Ant classpath during build.
 1646   This is required by LUCENE-7966. (Adrien Grand, Uwe Schindler)
 1648 * LUCENE-8161: spatial-extras: the Spatial4j dependency has been updated from 0.6 to 0.7,
 1649   which is drop-in compatible (Lucene doesn't expressly use any of the few API differences).
 1650   Spatial4j 0.7 is compatible with JTS 1.15.0 and not any prior version.  JTS 1.15.0 is
 1651   dual-licensed to include BSD; prior versions were LGPL.  (David Smiley)
 1653 * LUCENE-8155: Add back support in smoke tester to run against later Java versions.
 1654   (Uwe Schindler)
 1656 * LUCENE-8169: Migrated build to use OpenClover 4.2.1 for checking code coverage.
 1657   (Uwe Schindler)
 1659 * LUCENE-8170: Improve OpenClover reports (separate test from production code);
 1660   enable coverage reports inside test-frameworks.  (Uwe Schindler)
 1662 Build
 1664 * LUCENE-8168: Moved Groovy scripts in build files to separate files.
 1665   Update Groovy to 2.4.13.  (Uwe Schindler)
 1667 * LUCENE-8176: HttpReplicatorTest awaits more than a minute for stopping Jetty threads
 1668   (Mikhail Khludnev)
 1670 ======================= Lucene 7.2.1 =======================
 1672 Bug Fixes
 1674 * LUCENE-8117: Fix advanceExact on SortedNumericDocValues produced by Lucene54DocValues. (Jim Ferenczi).
 1676 ======================= Lucene 7.2.0 =======================
 1678 API Changes
 1680 * LUCENE-8017, LUCENE-8042: Weight, DoubleValuesSource and related objects
 1681   now implement a SegmentCacheable interface, with a single method
 1682   isCacheable(LeafReaderContext) determining whether or not the object may
 1683   be cached against a LeafReader. (Alan Woodward, Robert Muir)
 1685 * LUCENE-8038: Payload factors for scoring in PayloadScoreQuery are now
 1686   calculated by a PayloadDecoder, instead of delegating to the Similarity.
 1687   (Alan Woodward)
 1689 * LUCENE-8014: Similarity.computeSlopFactor() and
 1690   Similarity.computePayloadFactor() have been deprecated. (Alan Woodward)
 1692 * LUCENE-6278: Scorer.freq() has been removed (Alan Woodward)
 1694 * LUCENE-7736: DoubleValuesSource and LongValuesSource now expose a
 1695   rewrite(IndexSearcher) function. (Alan Woodward)
 1697 * LUCENE-7998: DoubleValuesSource.fromQuery() allows you to use the scores
 1698   from a Query as a DoubleValuesSource. (Alan Woodward)
 1700 * LUCENE-8049: IndexWriter.getMergingSegments()'s return type was changed from
 1701   Collection to Set to more accurately reflect it's nature. (David Smiley)
 1703 * LUCENE-8059: TopFieldDocCollector can now early terminate collection when
 1704   the sort order is compatible with the index order. As a consequence,
 1705   EarlyTerminatingSortingCollector is now deprecated. (Adrien Grand)
 1707 New Features
 1709 * LUCENE-8061: Add convenience factory methods to create BBoxes and XYZSolids
 1710   directly from bounds objects.
 1712 * LUCENE-7736: IndexReaderFunctions expose various IndexReader statistics as
 1713   DoubleValuesSources. (Alan Woodward)
 1715 * LUCENE-8068: Allow IndexWriter to write a single DWPT to disk Adds a
 1716   flushNextBuffer method to IndexWriter that allows the caller to
 1717   synchronously move the next pending or the biggest non-pending index buffer to
 1718   disk. This enables flushing selected buffer to disk without highjacking an
 1719   indexing thread. This is for instance useful if more than one IW (shards) must
 1720   be maintained in a single JVM / system. (Simon Willnauer)
 1722 Bug Fixes
 1724 * LUCENE-8076: Normalize Vincenti distance calculation for planet models that aren't normalized.
 1725   (Ignacio Vera)
 1727 * LUCENE-8057: Exact circle bounds computation was incorrect.
 1728   (Ignacio Vera)
 1730 * LUCENE-8056: Exact circle segment bounding suffered from precision errors.
 1731   (Karl Wright)
 1733 * LUCENE-8054: Fix the exact circle case where relationships fail when the
 1734   planet model has c <= ab, because the planes are constructed incorrectly.
 1735   (Ignacio Vera)
 1737 * LUCENE-7991: KNearestNeighborDocumentClassifier.knnSearch no longer applies
 1738   a previous boosted field's factor to subsequent unboosted fields.
 1739   (Christine Poerschke)
 1741 * LUCENE-7999: Switch from int to long to track the name for the next
 1742   segment to write, so that very long lived indices with very frequent
 1743   refreshes or commits, and high indexing thread counts, do not
 1744   overflow an int (Mykhailo Demianenko via Mike McCandless)
 1746 * LUCENE-8025: Use sumTotalTermFreq=sumDocFreq when scoring DOCS_ONLY fields
 1747   that omit term frequency information, as it is equivalent in that case.
 1748   Previously bogus numbers were used, and many similarities would
 1749   completely degrade. (Robert Muir, Adrien Grand)
 1751 * LUCENE-8045: ParallelLeafReader did not correctly report FieldInfo.dvGen
 1752   (Alan Woodward)
 1754 * LUCENE-8034: Use subtraction instead of addition to sidestep int
 1755   overflow in SpanNotQuery.  (Hari Menon via Mike McCandless)
 1757 * LUCENE-8078: The query cache should not cache instances of
 1758   MatchNoDocsQuery. (Jon Harper via Adrien Grand)
 1760 * LUCENE-8048: Filesystems do not guarantee order of directories updates
 1761   (Nikolay Martynov, Simon Willnauer, Erick Erickson)
 1763 Optimizations
 1765 * LUCENE-8018: Smaller FieldInfos memory footprint by not retaining unnecessary
 1766   references to TreeMap entries. (Julian Vassev via Adrien Grand)
 1768 * LUCENE-7994: Use int/int scatter map to gather facet counts when the
 1769   number of hits is small relative to the number of unique facet labels
 1770   (Dawid Weiss, Robert Muir, Mike McCandless)
 1772 * LUCENE-8062: GlobalOrdinalsQuery is no longer eligible for caching. (Jim Ferenczi)
 1774 * LUCENE-8058: Large instances of TermInSetQuery are no longer eligible for
 1775   caching as they could break memory accounting of the query cache.
 1776   (Adrien Grand)
 1778 * LUCENE-8055: MemoryIndex.MemoryDocValuesIterator returns 2 documents
 1779   instead of 1. (Simon Willnauer)
 1781 * LUCENE-8043: Fix document accounting in IndexWriter to prevent writing too many
 1782   documents. Once this happens, Lucene refuses to open the index and throws a
 1783   CorruptIndexException. (Simon Willnauer, Yonik Seeley, Mike McCandless)
 1785 Tests
 1787 * LUCENE-8035: Run tests with JDK-specific options: --illegal-access=deny
 1788   on Java 9+.  (Uwe Schindler)
 1790 Build
 1792 * LUCENE-6144: Upgrade Ivy to 2.4.0; 'ant ivy-bootstrap' now removes old Ivy
 1793   jars in ~/.ant/lib/.  (Shawn Heisey, Steve Rowe)
 1796 ======================= Lucene 7.1.0 =======================
 1798 Changes in Runtime Behavior
 1800 * Resolving of external entities in queryparser/xml/CoreParser is disallowed
 1801   by default. See SOLR-11477 for details.
 1803 New Features
 1805 * LUCENE-7970: Add a shape to Geo3D that consists of multiple planes that
 1806   approximate a true circle, rather than an ellipse, for non-spherical planet models.
 1807   (Karl Wright, Ignacio Vera)
 1809 * LUCENE-7955: Add support for the concept of "nearest distance" to Geo3D's
 1810   GeoPath abstraction, which is the distance along the path to the point that is
 1811   closest to the provided point. (Karl Wright)
 1813 * LUCENE-7906: Add spatial relationships between all currently-defined Geo shapes.
 1814   (Ignacio Vera)
 1816 * LUCENE-7955: Add support for zero-width paths. (Karl Wright)
 1818 * LUCENE-7936: Add serialization and deserialization support to Geo3D. (Karl Wright,
 1819   Ignacio Vera)
 1821 * LUCENE-7942: Distance computations now have the ability to accurately aggregate
 1822   distances, rather than just doing sums. (Karl Wright)
 1824 * LUCENE-7934: Add a planet model interface. (Karl Wright)
 1826 * LUCENE-7918: Revamp the API for composites so that it's generic and can be used
 1827   for many kinds of shapes. (Ignacio Vera)
 1829 * LUCENE-7621: Add CoveringQuery, a query whose required number of matching
 1830   clauses can be defined per document. (Adrien Grand)
 1832 * LUCENE-7927: Add LongValueFacetCounts, to compute facet counts for individual
 1833   numeric values (Mike McCandless)
 1835 * LUCENE-7940: Add BengaliAnalyzer. (Md. Abdulla-Al-Sun via Robert Muir)
 1837 * LUCENE-7392: Add point based LatLonBoundingBox as new RangeField Type.
 1838   (Nick Knize)
 1840 * LUCENE-7951: Spatial-extras has much better Geo3d support by implementing Spatial4j
 1841   abstractions: SpatialContextFactory, ShapeFactory, BinaryCodec, DistanceCalculator.
 1842   (Ignacio Vera, David Smiley)
 1844 * LUCENE-7973: Update dictionary version for Ukrainian analyzer to 3.9.0 (Andriy
 1845   Rysin via Dawid Weiss)
 1847 * LUCENE-7974: Add FloatPointNearestNeighbor, an N-dimensional FloatPoint
 1848   K-nearest-neighbor search implementation.  (Steve Rowe)
 1850 * LUCENE-7975: Change the default taxonomy facets cache to a faster
 1851   byte[] (UTF-8) based cache.  (Mike McCandless)
 1853 * LUCENE-7972: DirectoryTaxonomyReader, in Lucene's facet module, now
 1854   implements Accountable, so you can more easily track how much heap
 1855   it's using.  (Mike McCandless)
 1857 * LUCENE-7982: A new NormsFieldExistsQuery matches documents that have
 1858   norms in a specified field (Colin Goodheart-Smithe via Mike McCandless)
 1860 Optimizations
 1862 * LUCENE-7905: Optimize how OrdinalMap (used by
 1863   SortedSetDocValuesFacetCounts and others) builds its map (Robert
 1864   Muir, Adrien Grand, Mike McCandless)
 1866 * LUCENE-7655: Speed up geo-distance queries in case of dense single-valued
 1867   fields when most documents match. (Maciej Zasada via Adrien Grand)
 1869 * LUCENE-7897: IndexOrDocValuesQuery now requires the range cost to be more
 1870   than 8x greater than the cost of the lead iterator in order to use doc values.
 1871   (Murali Krishna P via Adrien Grand)
 1873 * LUCENE-7925: Collapse duplicate SHOULD or MUST clauses by summing up their
 1874   boosts. (Adrien Grand)
 1876 * LUCENE-7939: MinShouldMatchSumScorer now leverages two-phase iteration in
 1877   order to be faster when used in conjunctions. (Adrien Grand)
 1879 * LUCENE-7827: AnalyzingInfixSuggester doesn't create "textgrams" 
 1880   when minPrefixChar=0 (Mikhail Khludnev)
 1882 Bug Fixes
 1884 * LUCENE-8066: It was still possible to construct a concave GeoExactCircle, so use
 1885    a sector approach to prevent that. (Ignacio Vera)
 1887 * LUCENE-7967: The GeoDegeneratePoint isWithin() method needed allowance for
 1888    numerical precision. (Karl Wright)
 1890 * LUCENE-7965: GeoBBoxFactory was constructing the wrong shape at the poles
 1891   if the longitude span was greater than 180 degrees. (Karl Wright)
 1893 * LUCENE-7916: Prevent ArrayIndexOutOfBoundsException if ICUTokenizer is used
 1894   with a different ICU JAR version than it is compiled against. Note, this is
 1895   not recommended, lucene-analyzers-icu contains binary data structures
 1896   specific to ICU/Unicode versions it is built against. (Chris Koenig, Robert Muir)
 1898 * LUCENE-7891: Lucene's taxonomy facets now uses a non-buggy LRU cache
 1899   by default.  (Jan-Willem van den Broek via Mike McCandless)
 1901 * LUCENE-7959: Improve NativeFSLockFactory's exception message if it cannot create
 1902   write.lock for an empty index due to bad permissions/read-only filesystem/etc.
 1903   (Erick Erickson, Shawn Heisey, Robert Muir)
 1905 * LUCENE-7968: AnalyzingSuggester would sometimes order suggestions incorrectly,
 1906   it did not properly break ties on the surface forms when both the weights and
 1907   the analyzed forms were equal. (Robert Muir)
 1909 * LUCENE-7957: ConjunctionScorer.getChildren was failing to return all
 1910   child scorers (Adrien Grand, Mike McCandless)
 1912 * SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser
 1913   by default. (Michael Stepankin, Olga Barinova, Uwe Schindler, Christine Poerschke)
 1915 Build
 1917 * SOLR-11181: Switch order of maven artifact publishing procedure: deploy first
 1918   instead of locally installing first, to workaround a double repository push of
 1919   *-sources.jar and *-javadoc.jar files.  (Lynn Monson via Steve Rowe)
 1921 * LUCENE-6673: Maven build fails for target javadoc:jar.  
 1922   (Ramkumar Aiyengar, Daniel Collins via Steve Rowe)
 1924 * LUCENE-7985: Upgrade forbiddenapis to 2.4.1.  (Uwe Schindler)
 1926 Other
 1928 * LUCENE-7948, LUCENE-7937: Upgrade randomizedtesting to 2.5.3 (minor fixes 
 1929   in test filtering for IDEs). (Mike Sokolov, Dawid Weiss)
 1931 * LUCENE-7933: LongBitSet now validates the numBits parameter (Won
 1932   Jonghoon, Mike McCandless)
 1934 * LUCENE-7978: Add some more documentation about setting up build
 1935   environment.  (Anton R. Yuste via Uwe Schindler)
 1937 * LUCENE-7983: IndexWriter.IndexReaderWarmer is now a functional interface
 1938   instead of an abstract class with a single method (Dawid Weiss)
 1940 * LUCENE-5753: Update TLDs recognized by UAX29URLEmailTokenizer. (Steve Rowe)
 1943 ======================= Lucene 7.0.1 =======================
 1945 Bug Fixes
 1947 * LUCENE-7957: ConjunctionScorer.getChildren was failing to return all
 1948   child scorers (Adrien Grand, Mike McCandless)
 1950 ======================= Lucene 7.0.0 =======================
 1952 New Features
 1954 * LUCENE-7703: SegmentInfos now record the major Lucene version at index
 1955   creation time. (Adrien Grand)
 1957 * LUCENE-7756: LeafReader.getMetaData now exposes the index created version as
 1958   well as the oldest Lucene version that contributed to the segment.
 1959   (Adrien Grand)
 1961 * LUCENE-7854: The new TermFrequencyAttribute used during analysis
 1962   with a custom token stream allows indexing custom term frequencies
 1963   (Mike McCandless)
 1965 * LUCENE-7866: Add a new DelimitedTermFrequencyTokenFilter that allows to
 1966   mark tokens with a custom term frequency (LUCENE-7854). It parses a numeric
 1967   value after a separator char ('|') at the end of each token and changes
 1968   the term frequency to this value.  (Uwe Schindler, Robert Muir, Mike
 1969   McCandless)
 1971 * LUCENE-7868: Multiple threads can now resolve deletes and doc values
 1972   updates concurrently, giving sizable speedups in update-heavy
 1973   indexing use cases (Simon Willnauer, Mike McCandless)
 1975 * LUCENE-7823: Pure query based naive bayes classifier using BM25 scores (Tommaso Teofili)
 1977 * LUCENE-7838: Knn classifier based on fuzzified term queries (Tommaso Teofili)
 1979 * LUCENE-7855: Added advanced options of the Wikipedia tokenizer to its factory.
 1980   (Juan Pedro via Adrien Grand)
 1982 API Changes
 1984 * LUCENE-2605: Classic QueryParser no longer splits on whitespace by default.
 1985   Use setSplitOnWhitespace(true) to get the old behavior.  (Steve Rowe)
 1987 * LUCENE-7369: Similarity.coord and BooleanQuery.disableCoord are removed.
 1988   (Adrien Grand)
 1990 * LUCENE-7368: Removed query normalization. (Adrien Grand)
 1992 * LUCENE-7355: AnalyzingQueryParser has been removed as its functionality has
 1993   been folded into the classic QueryParser. (Adrien Grand)
 1995 * LUCENE-7407: Doc values APIs have been switched from random access
 1996   to iterators, enabling future codec compression improvements. (Mike
 1997   McCandless)
 1999 * LUCENE-7475: Norms now support sparsity, allowing to pay for what is
 2000   actually used. (Adrien Grand)
 2002 * LUCENE-7494: Points now have a per-field API, like doc values. (Adrien Grand)
 2004 * LUCENE-7410: Cache keys and close listeners have been refactored in order
 2005   to be less trappy. See IndexReader.getReaderCacheHelper and
 2006   LeafReader.getCoreCacheHelper. (Adrien Grand)
 2008 * LUCENE-6819: Index-time boosts are not supported anymore. As a replacement,
 2009   index-time scoring factors should be indexed into a doc value field and
 2010   combined at query time using eg. FunctionScoreQuery. (Adrien Grand)
 2012 * LUCENE-7734: FieldType's copy constructor was widened to accept any IndexableFieldType.
 2013   (David Smiley)
 2015 * LUCENE-7701: Grouping collectors have been refactored, such that groups are
 2016   now defined by a GroupSelector implementation. (Alan Woodward)
 2018 * LUCENE-7741: DoubleValuesSource now has an explain() method (Alan Woodward,
 2019   Adrien Grand)
 2021 * LUCENE-7815: Removed the PostingsHighlighter; you should use the UnifiedHighlighter
 2022   instead, which derived from the UH.  WholeBreakIterator and
 2023   CustomSeparatorBreakIterator were moved to UH's package. (David Smiley)
 2025 * LUCENE-7850: Removed support for legacy numerics. (Adrien Grand)
 2027 * LUCENE-7500: Removed abstract LeafReader.fields(); instead terms(fieldName)
 2028   has been made abstract, fomerly was final.  Also, MultiFields.getTerms
 2029   was optimized to work directly instead of being implemented on getFields.
 2030   (David Smiley)
 2032 * LUCENE-7872: TopDocs.totalHits is now a long. (Adrien Grand, hossman)
 2034 * LUCENE-7868: IndexWriterConfig.setMaxBufferedDeleteTerms is
 2035   removed. (Simon Willnauer, Mike McCandless)
 2037 * LUCENE-7877: PrefixAwareTokenStream is replaced with ConcatenatingTokenStream
 2038   (Alan Woodward, Uwe Schindler, Adrien Grand)
 2040 * LUCENE-7867: The deprecated Token class is now only available in the test
 2041   framework (Alan Woodward, Adrien Grand)
 2043 * LUCENE-7723: DoubleValuesSource enforces implementation of equals() and
 2044   hashCode() (Alan Woodward)
 2046 * LUCENE-7737: The spatial-extras module no longer has a dependency on the
 2047   queries module.  All uses of ValueSource are either replaced with core
 2048   DoubleValuesSource extensions, or with the new ShapeValuesSource and
 2049   ShapeValuesPredicate classes (Alan Woodward, David Smiley)
 2051 * LUCENE-7892: Doc-values query factory methods have been renamed so that their
 2052   name contains "slow" in order to cleary indicate that they would usually be a
 2053   bad choice. (Adrien Grand)
 2055 * LUCENE-7899: FieldValueQuery is renamed to DocValuesFieldExistsQuery
 2056   (Adrien Grand, Mike McCandless)
 2058 Bug Fixes
 2060 * LUCENE-7626: IndexWriter will no longer accept broken token offsets
 2061   (Mike McCandless)
 2063 * LUCENE-7859: Spatial-extras PackedQuadPrefixTree bug that only revealed itself
 2064   with the new pointsOnly optimizations in LUCENE-7845. (David Smiley)
 2066 * LUCENE-7871: fix false positive match in BlockJoinSelector when children have no value, introducing 
 2067   wrap methods accepting children as DISI. Extracting ToParentDocValues (Mikhail Khludnev)
 2069 * LUCENE-7914: Add a maximum recursion level in automaton recursive
 2070   functions (Operations.isFinite and Operations.topsortState) to prevent
 2071   large automaton to overflow the stack (Robert Muir, Adrien Grand, Jim Ferenczi)
 2073 * LUCENE-7864: IndexMergeTool is not using intermediate hard links (even
 2074   if possible). (Dawid Weiss)
 2076 * LUCENE-7956: Fixed potential stack overflow error in ICUNormalizer2CharFilter.
 2077   (Adrien Grand)
 2079 * LUCENE-7963: Remove useless getAttribute() in DefaultIndexingChain that
 2080   causes performance drop, introduced by LUCENE-7626.  (Daniel Mitterdorfer
 2081   via Uwe Schindler)
 2083 Improvements
 2085 * LUCENE-7489: Better storage of sparse doc-values fields with the default
 2086   codec. (Adrien Grand)
 2088 * LUCENE-7730: More accurate encoding of the length normalization factor
 2089   thanks to the removal of index-time boosts. (Adrien Grand)
 2091 * LUCENE-7901: Original Highlighter now eagerly throws an exception if you
 2092   provide components that are null. (Jason Gerlowski, David Smiley)
 2094 * LUCENE-7841: Normalize ґ to г in Ukrainian analyzer. (Andriy Rysin via Dawid Weiss)
 2096 Optimizations
 2098 * LUCENE-7416: BooleanQuery optimizes queries that have queries that occur both
 2099   in the sets of SHOULD and FILTER clauses, or both in MUST/FILTER and MUST_NOT
 2100   clauses. (Spyros Kapnissis via Adrien Grand, Uwe Schindler)
 2102 * LUCENE-7506: FastTaxonomyFacetCounts should use CPU in proportion to
 2103   the size of the intersected set of hits from the query and documents
 2104   that have a facet value, so sparse faceting works as expected
 2105   (Adrien Grand via Mike McCandless)
 2107 * LUCENE-7519: Add optimized APIs to compute browse-only top level
 2108   facets (Mike McCandless)
 2110 * LUCENE-7589: Numeric doc values now have the ability to encode blocks of
 2111   values using different numbers of bits per value if this proves to save
 2112   storage. (Adrien Grand)
 2114 * LUCENE-7845: Enhance spatial-extras RecursivePrefixTreeStrategy queries when the
 2115   query is a point (for 2D) or a is a simple date interval (e.g. 1 month).  When
 2116   the strategy is marked as pointsOnly, the results is a TermQuery. (David Smiley)
 2118 * LUCENE-7874: DisjunctionMaxQuery rewrites to a BooleanQuery when tiebreaker is set to 1. (Jim Ferenczi)
 2120 * LUCENE-7828: Speed up range queries on range fields by improving how we
 2121   compute the relation between the query and inner nodes of the BKD tree.
 2122   (Adrien Grand)
 2124 Other
 2126 * LUCENE-7923: Removed FST.Arc.node field (unused). (Dawid Weiss)
 2128 * LUCENE-7328: Remove LegacyNumericEncoding from GeoPointField. (Nick Knize)
 2130 * LUCENE-7360: Remove Explanation.toHtml() (Alan Woodward)
 2132 * LUCENE-7681: MemoryIndex uses new DocValues API (Alan Woodward)
 2134 * LUCENE-7753: Make fields static when possible.
 2135   (Daniel Jelinski via Adrien Grand)
 2137 * LUCENE-7540: Upgrade ICU to 59.1 (Mike McCandless, Jim Ferenczi)
 2139 * LUCENE-7852: Correct copyright year(s) in lucene/LICENSE.txt file.
 2140   (Christine Poerschke, Steve Rowe)
 2142 * LUCENE-7719: Generalized the UnifiedHighlighter's support for AutomatonQuery
 2143   for character & binary automata. Added AutomatonQuery.isBinary. (David Smiley)
 2145 * LUCENE-7873: Due to serious problems with context class loaders in several
 2146   frameworks (OSGI, Java 9 Jigsaw), the lookup of Codecs, PostingsFormats,
 2147   DocValuesFormats and all analysis factories was changed to only inspect the
 2148   current classloader that defined the interface class (lucene-core.jar).
 2149   See MIGRATE.txt for more information!  (Uwe Schindler, Dawid Weiss)
 2151 * LUCENE-7883: Lucene no longer uses the context class loader when resolving
 2152   resources in CustomAnalyzer or ClassPathResourceLoader. Resources are only
 2153   resolved against Lucene's class loader by default. Please use another builder
 2154   method to change to a custom classloader.  (Uwe Schindler)
 2156 * LUCENE-5822: Convert README to Markdown (Jason Gerlowski via Mike Drob)
 2158 * LUCENE-7773: Remove unused/deprecated token types from StandardTokenizer.
 2159   (Ahmet Arslan via Steve Rowe)
 2161 * LUCENE-7800: Remove code that potentially rethrows checked exceptions
 2162   from methods that don't declare them ("sneaky throw" hack). (Robert Muir,
 2163   Uwe Schindler, Dawid Weiss)
 2165 * LUCENE-7876: Avoid calls to LeafReader.fields() and MultiFields.getFields()
 2166   that are trivially replaced by LeafReader.terms() and MultiFields.getTerms()
 2167   (David Smiley)
 2169 ======================= Lucene 6.6.5 =======================
 2170 (No Changes)
 2172 ======================= Lucene 6.6.4 =======================
 2173 (No Changes)
 2175 ======================= Lucene 6.6.3 =======================
 2177 Build
 2179 * LUCENE-6144: Upgrade Ivy to 2.4.0; 'ant ivy-bootstrap' now removes old Ivy
 2180   jars in ~/.ant/lib/.  (Shawn Heisey, Steve Rowe)
 2182 ======================= Lucene 6.6.2 =======================
 2184 Changes in Runtime Behavior
 2186 * Resolving of external entities in queryparser/xml/CoreParser is disallowed
 2187   by default. See SOLR-11477 for details.
 2189 Bug Fixes
 2191 * SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser
 2192   by default. (Michael Stepankin, Olga Barinova, Uwe Schindler, Christine Poerschke)
 2194 ======================= Lucene 6.6.1 =======================
 2196 Bug Fixes
 2198 * LUCENE-7869: Changed MemoryIndex to sort 1d points. In case of 1d points, the PointInSetQuery.MergePointVisitor expects
 2199   that these points are visited in ascending order. The memory index doesn't do this and this can result in document
 2200   with multiple points that should match to not match. (Martijn van Groningen)
 2202 * LUCENE-7878: Fix query builder to keep the SHOULD clause that wraps multi-word synonyms. (Jim Ferenczi)
 2204 ======================= Lucene 6.6.0 =======================
 2206 New Features
 2208 * LUCENE-7811: Add a concurrent SortedSet facets implementation.
 2209   (Mike McCandless)
 2211 Bug Fixes
 2213 * LUCENE-7777: ByteBlockPool.readBytes sometimes throws
 2214   ArrayIndexOutOfBoundsException when byte blocks larger than 32 KB
 2215   were added (Mike McCandless)
 2217 * LUCENE-7797: The static FSDirectory.listAll(Path) method was always
 2218   returning an empty array.  (Atkins Chang via Mike McCandless)
 2220 * LUCENE-7481: Fixed missing rewrite methods for SpanPayloadCheckQuery
 2221   and PayloadScoreQuery. (Erik Hatcher)
 2223 * LUCENE-7808: Fixed PayloadScoreQuery and SpanPayloadCheckQuery
 2224   .equals and .hashCode methods.  (Erik Hatcher)
 2226 * LUCENE-7798: Add .equals and .hashCode to ToParentBlockJoinSortField 
 2227   (Mikhail Khludnev)
 2229 * LUCENE-7814: DateRangePrefixTree (in spatial-extras) had edge-case bugs for
 2230   years >= 292,000,000. (David Smiley)
 2232 * LUCENE-5365, LUCENE-7818: Fix incorrect condition in queryparser's
 2233   QueryNodeOperation#logicalAnd().  (Olivier Binda, Amrit Sarkar,
 2234   AppChecker via Uwe Schindler)
 2236 * LUCENE-7821: The classic and flexible query parsers, as well as Solr's
 2237  "lucene"/standard query parser, should require " TO " in range queries,
 2238   and accept "TO" as endpoints in range queries. (hossman, Steve Rowe)
 2240 * LUCENE-7824: Fix graph query analysis for multi-word synonym rules with common terms (eg. new york, new york city).
 2241   (Jim Ferenczi)
 2243 * LUCENE-7817: Pass cached query to onQueryCache instead of null.
 2244   (Christoph Kaser via Adrien Grand)
 2246 * LUCENE-7831: CodecUtil should not seek to negative offsets. (Adrien Grand)
 2248 * LUCENE-7833: ToParentBlockJoinQuery computed the min score instead of the max
 2249   score with ScoreMode.MAX. (Adrien Grand)
 2251 * LUCENE-7847: Fixed all-docs-match optimization of range queries on range
 2252   fields. (Adrien Grand)
 2254 * LUCENE-7810: Fix equals() and hashCode() methods of several join queries.
 2255   (Hossman, Adrien Grand, Martijn van Groningen)
 2257 Improvements
 2259 * LUCENE-7782: OfflineSorter now passes the total number of items it
 2260   will write to getWriter (Mike McCandless)
 2262 * LUCENE-7785: Move dictionary for Ukrainian analyzer to external dependency.
 2263   (Andriy Rysin via Steve Rowe, Dawid Weiss)
 2265 * LUCENE-7801: SortedSetDocValuesReaderState now implements
 2266   Accountable so you can see how much RAM it's using (Robert Muir,
 2267   Mike McCandless)
 2269 * LUCENE-7792: OfflineSorter can now run concurrently if you pass it
 2270   an optional ExecutorService (Dawid Weiss, Mike McCandless)
 2272 * LUCENE-7811: Sorted set facets now use sparse storage when
 2273   collecting hits, when appropriate.  (Mike McCandless)
 2275 Optimizations
 2277 * LUCENE-7787: spatial-extras HeatmapFacetCounter will now short-circuit it's
 2278   work when Bits.MatchNoBits is passed. (David Smiley)
 2280 Other
 2282 * LUCENE-7796: Make IOUtils.reThrow idiom declare Error return type so 
 2283   callers may use it in a way that compiler knows subsequent code is 
 2284   unreachable. reThrow is now deprecated in favor of IOUtils.rethrowAlways
 2285   with a slightly different semantics (see javadoc). (Hossman, Robert Muir, 
 2286   Dawid Weiss)
 2288 * LUCENE-7754: Inner classes should be static whenever possible.
 2289   (Daniel Jelinski via Adrien Grand)
 2291 * LUCENE-7751: Avoid boxing primitives only to call compareTo.
 2292   (Daniel Jelinski via Adrien Grand)
 2294 * LUCENE-7743: Never call new String(String).
 2295   (Daniel Jelinski via Adrien Grand)
 2297 * LUCENE-7761: Fixed comment in ReqExclScorer.
 2298   (Pablo Pita Leira via Adrien Grand)
 2300 ======================= Lucene 6.5.1 =======================
 2302 Bug Fixes
 2304 * LUCENE-7755: Fixed join queries to not reference IndexReaders, as it could
 2305   cause leaks if they are cached. (Adrien Grand)
 2307 * LUCENE-7749: Made LRUQueryCache delegate the scoreSupplier method.
 2308   (Martin Amirault via Adrien Grand)
 2310 * LUCENE-7769: The UnifiedHighligter wasn't highlighting portions of the query
 2311   wrapped in BoostQuery or SpanBoostQuery. (David Smiley, Dmitry Malinin)
 2313 Other
 2315 * LUCENE-7763: Remove outdated comment in IndexWriterConfig.setIndexSort javadocs.
 2316   (马可阳 via Christine Poerschke)
 2318 ======================= Lucene 6.5.0 =======================
 2320 API Changes
 2322 * LUCENE-7740: Refactor Range Fields to remove Field suffix (e.g., DoubleRange),
 2323   move InetAddressRange and InetAddressPoint from sandbox to misc module, and
 2324   refactor all other range fields from sandbox to core. (Nick Knize)
 2326 * LUCENE-7624: TermsQuery has been renamed as TermInSetQuery and moved to core.
 2327   (Alan Woodward)
 2329 * LUCENE-7637: TermInSetQuery requires that all terms come from the same field.
 2330   (Adrien Grand)
 2332 * LUCENE-7644: FieldComparatorSource.newComparator() and
 2333   SortField.getComparator() no longer throw IOException (Alan Woodward)
 2335 * LUCENE-7643: Replaced doc-values queries in lucene/sandbox with factory
 2336   methods on the *DocValuesField classes. (Adrien Grand)
 2338 * LUCENE-7659: Added a IndexWriter#getFieldNames() method (experimental) to return
 2339   all field names as visible from the IndexWriter. This would be useful for
 2340   IndexWriter#updateDocValues() calls, to prevent calling with non-existent
 2341   docValues fields (Ishan Chattopadhyaya, Adrien Grand, Mike McCandless)
 2343 * LUCENE-6959: Removed ToParentBlockJoinCollector in favour of
 2344   ParentChildrenBlockJoinQuery, that can return the matching children documents per
 2345   parent document. This query should be executed for each matching parent document
 2346   after the main query has been executed. (Adrien Grand, Martijn van Groningen,
 2347   Mike McCandless)
 2349 * LUCENE-7628: Scorer.getChildren() now only returns Scorers that are
 2350   positioned on the current document, and can throw an IOException.
 2351   AssertingScorer checks that getChildren() is not called on an unpositioned
 2352   Scorer.  (Alan Woodward, Adrien Grand)
 2354 * LUCENE-7702: Removed GraphQuery in favour of simple boolean query. (Matt Webber via Jim Ferenczi)
 2356 * LUCENE-7707: TopDocs.merge now takes a boolean option telling it
 2357   when to use the incoming shard index versus when to assign the shard
 2358   index itself, allowing users to merge shard responses incrementally
 2359   instead of once all shard responses are present. (Simon Willnauer,
 2360   Mike McCandless)
 2362 * LUCENE-7700: A cleanup of merge throughput control logic. Refactored all the
 2363   code previously scattered throughout the IndexWriter and 
 2364   ConcurrentMergeScheduler into a more accessible set of public methods (see 
 2365   MergePolicy.OneMergeProgress, MergeScheduler.wrapForMerge and 
 2366   OneMerge.mergeInit). (Dawid Weiss, Mike McCandless).
 2368 * LUCENE-7734: FieldType's copy constructor was widened to accept any IndexableFieldType.
 2369   (David Smiley)
 2371 New Features
 2373 * LUCENE-7738: Add new InetAddressRange for indexing and querying InetAddress
 2374   ranges. (Nick Knize)
 2376 * LUCENE-7449: Add CROSSES relation support to RangeFieldQuery. (Nick Knize)
 2378 * LUCENE-7623: Add FunctionScoreQuery and FunctionMatchQuery (Alan Woodward,
 2379   Adrien Grand, David Smiley)
 2381 * LUCENE-7619: Add WordDelimiterGraphFilter, just like
 2382   WordDelimiterFilter except it produces correct token graphs so that
 2383   proximity queries at search time will produce correct results (Mike
 2384   McCandless)
 2386 * LUCENE-7656: Added the LatLonDocValuesField.new(Box/Distance)Query() factory
 2387   methods that are the equivalent of factory methods on LatLonPoint but operate
 2388   on doc values. These new methods should be wrapped in an IndexOrDocValuesQuery
 2389   for best performance. (Adrien Grand)
 2391 * LUCENE-7673: Added MultiValued[Int/Long/Float/Double]FieldSource that given a
 2392   SortedNumericSelector.Type can give a ValueSource view of a 
 2393   SortedNumericDocValues field. (Tomás Fernández Löbbe)
 2395 * LUCENE-7465: Add SimplePatternTokenizer and
 2396   SimplePatternSplitTokenizer, using Lucene's regexp/automaton
 2397   implementation for analysis/tokenization (Clinton Gormley, Mike
 2398   McCandless)
 2400 * LUCENE-7688: Add OneMergeWrappingMergePolicy class.
 2401   (Keith Laban, Christine Poerschke)
 2403 * LUCENE-7686: The near-real-time document suggester can now
 2404   efficiently filter out duplicate suggestions (Uwe Schindler, Mike
 2405   McCandless)
 2407 * LUCENE-7712: SimpleQueryParser now supports default fuzziness
 2408   syntax, mapping foo~ to a FuzzyQuery with edit distance 2.  (Lee
 2409   Hinman, David Pilato via Mike McCandless)
 2411 Bug Fixes
 2413 * LUCENE-7630: Fix (Edge)NGramTokenFilter to no longer drop payloads
 2414   and preserve all attributes. (Nathan Gass via Uwe Schindler)
 2416 * LUCENE-7679: MemoryIndex was ignoring omitNorms settings on passed-in
 2417   IndexableFields. (Alan Woodward)
 2419 * LUCENE-7692: PatternReplaceCharFilterFactory now implements MultiTermAware.
 2420   (Adrien Grand)
 2422 * LUCENE-7685: ToParentBlockJoinQuery and ToChildBlockJoinQuery now use the
 2423   rewritten child query in their equals and hashCode implementations.
 2424   (Adrien Grand)
 2426 * LUCENE-7698: CommonGramsQueryFilter was producing a disconnected
 2427   token graph, messing up phrase queries when it was used during query
 2428   parsing (Ere Maijala via Mike McCandless)
 2430 * LUCENE-7708: ShingleFilter without unigram was producing a disconnected
 2431   token graph, messing up queries when it was used during query
 2432   parsing (Jim Ferenczi)
 2434 Improvements
 2436 * LUCENE-7055: Added Weight#scorerSupplier, which allows to estimate the cost
 2437   of a Scorer before actually building it, in order to optimize how the query
 2438   should be run, eg. using points or doc values depending on costs of other
 2439   parts of the query. (Adrien Grand)
 2441 * LUCENE-7643: IndexOrDocValuesQuery allows to execute range queries using
 2442   either points or doc values depending on which one is more efficient.
 2443   (Adrien Grand)
 2445 * LUCENE-7662: If index files are missing, throw CorruptIndexException instead
 2446   of the less descriptive FileNotFound or NoSuchFileException (Mike Drob via 
 2447   Mike McCandless, Erick Erickson)
 2449 * LUCENE-7680: UsageTrackingQueryCachingPolicy never caches term filters anymore
 2450   since they are plenty fast. This also has the side-effect of leaving more
 2451   space in the history for costly filters. (Adrien Grand)
 2453 * LUCENE-7677: UsageTrackingQueryCachingPolicy now caches compound queries a bit
 2454   earlier than regular queries in order to improve cache efficiency.
 2455   (Adrien Grand)
 2457 * LUCENE-7710: BlockPackedReader throws CorruptIndexException and includes
 2458   IndexInput description instead of plain IOException (Mike Drob via
 2459   Mike McCandless)
 2461 * LUCENE-7695: ComplexPhraseQueryParser to support query time synonyms (Markus Jelsma
 2462   via Mikhail Khludnev) 
 2464 * LUCENE-7747: QueryBuilder now iterates lazily over the possible paths when building a graph query
 2465   (Jim Ferenczi)
 2467 Optimizations
 2469 * LUCENE-7641: Optimized point range queries to compute documents that do not
 2470   match the range on single-valued fields when more than half the documents in
 2471   the index would match. (Adrien Grand)
 2473 * LUCENE-7656: Speed up for LatLonPointDistanceQuery by computing distances even
 2474   less often. (Adrien Grand)
 2476 * LUCENE-7661: Speed up for LatLonPointInPolygonQuery by pre-computing the
 2477   relation of the polygon with a grid. (Adrien Grand)
 2479 * LUCENE-7660: Speed up LatLonPointDistanceQuery by improving the detection of
 2480   whether BKD cells are entirely within the distance close to the dateline.
 2481   (Adrien Grand)
 2483 * LUCENE-7654: ToParentBlockJoinQuery now implements two-phase iteration and
 2484   computes scores lazily in order to be faster when used in conjunctions.
 2485   (Adrien Grand)
 2487 * LUCENE-7667: BKDReader now calls `IntersectVisitor.grow()` on larger
 2488   increments. (Adrien Grand)
 2490 * LUCENE-7638: Query parsers now analyze the token graph for articulation
 2491   points (or cut vertices) in order to create more efficient queries for
 2492   multi-token synonyms. (Jim Ferenczi)
 2494 * LUCENE-7699: Query parsers now use span queries to produce more efficient
 2495   phrase queries for multi-token synonyms. (Matt Webber via Jim Ferenczi)
 2497 * LUCENE-7742: Fix places where we were unboxing and then re-boxing
 2498   according to FindBugs (Daniel Jelinski via Mike McCandless)
 2500 * LUCENE-7739: Fix places where we unnecessarily boxed while parsing
 2501   a numeric value according to FindBugs (Daniel Jelinski via Mike
 2502   McCandless)
 2504 Build
 2506 * LUCENE-7653: Update randomizedtesting to version 2.5.0. (Dawid Weiss)
 2508 * LUCENE-7665: Remove grouping dependency from the join module.
 2509   (Martijn van Groningen)
 2511 * SOLR-10023: Add non-recursive 'test-nocompile' target: Only runs unit tests.
 2512   Jars are not downloaded; compilation is not updated; and Clover is not enabled.
 2513   (Steve Rowe)
 2515 * LUCENE-7694: Update forbiddenapis to version 2.3. (Uwe Schindler)
 2517 * LUCENE-7693: Replace "org.apache." logic in GetMavenDependenciesTask.
 2518   (Daniel Collins, Christine Poerschke)
 2520 * LUCENE-7726: Fix HTML entity bugs in Javadocs to be able to build with
 2521   Java 9. (Uwe Schindler, Hossman)
 2523 * LUCENE-7727: Replace end-of-life Markdown parser "Pegdown" by "Flexmark"
 2524   for compatibility with Java 9. (Uwe Schindler)
 2526 Other
 2528 * LUCENE-7666: Fix typos in lucene-join package info javadoc.
 2529   (Tom Saleeba via Christine Poerschke)
 2531 * LUCENE-7658: queryparser/xml CoreParser now implements SpanQueryBuilder interface.
 2532   (Daniel Collins, Christine Poerschke)
 2534 * LUCENE-7715: NearSpansUnordered simplifications.
 2535   (Paul Elschot via Adrien Grand)
 2537 ======================= Lucene 6.4.2 =======================
 2539 Bug Fixes
 2541 * LUCENE-7676: Fixed FilterCodecReader to override more super-class methods.
 2542   Also added TestFilterCodecReader class. (Christine Poerschke)
 2544 * LUCENE-7717: The UnifiedHighlighter and PostingsHighlighter were not highlighting
 2545   prefix queries with multi-byte characters. TermRangeQuery is affected too.
 2546   (Dmitry Malinin, David Smiley)
 2548 ======================= Lucene 6.4.1 =======================
 2550 Build
 2552 * LUCENE-7651: Fix Javadocs build for Java 8u121 by injecting "Google Code
 2553   Prettify" without adding Javascript to Javadocs's -bottom parameter.
 2554   Also update Prettify to latest version to fix Google Chrome issue.
 2555   (Uwe Schindler)
 2557 Bug Fixes
 2559 * LUCENE-7657: Fixed potential memory leak in the case that a (Span)TermQuery
 2560   with a TermContext is cached. (Adrien Grand)
 2562 * LUCENE-7647: Made stored fields reclaim native memory more aggressively when
 2563   configured with BEST_COMPRESSION. This could otherwise result in out-of-memory
 2564   issues. (Adrien Grand)
 2566 * LUCENE-7670: AnalyzingInfixSuggester should not immediately open an
 2567   IndexWriter over an already-built index. (Steve Rowe)
 2569 ======================= Lucene 6.4.0 =======================
 2571 API Changes
 2573 * LUCENE-7533: Classic query parser no longer allows autoGeneratePhraseQueries
 2574   to be set to true when splitOnWhitespace is false (and vice-versa).
 2576 * LUCENE-7607: LeafFieldComparator.setScorer and SimpleFieldComparator.setScorer
 2577   are declared as throwing IOException (Alan Woodward)
 2579 * LUCENE-7617: Collector construction for two-pass grouping queries is
 2580   abstracted into a new Grouper class, which can be passed as a constructor
 2581   parameter to GroupingSearch.  The abstract base classes for the different
 2582   grouping Collectors are renamed to remove the Abstract* prefix.
 2583   (Alan Woodward, Martijn van Groningen)
 2585 * LUCENE-7609: The expressions module now uses the DoubleValuesSource API, and 
 2586   no longer depends on the queries module.  Expression#getValueSource() is
 2587   replaced with Expression#getDoubleValuesSource(). (Alan Woodward, Adrien
 2588   Grand)
 2590 * LUCENE-7610: The facets module now uses the DoubleValuesSource API, and
 2591   methods that take ValueSource parameters are deprecated (Alan Woodward)
 2593 * LUCENE-7611: DocumentValueSourceDictionary now takes a LongValuesSource
 2594   as a parameter, and the ValueSource equivalent is deprecated (Alan Woodward)
 2596 New features
 2598 * LUCENE-5867: Added BooleanSimilarity. (Robert Muir, Adrien Grand)
 2600 * LUCENE-7466: Added AxiomaticSimilarity. (Peilin Yang via Tommaso Teofili)
 2602 * LUCENE-7590: Added DocValuesStatsCollector to compute statistics on DocValues
 2603   fields. (Shai Erera)
 2605 * LUCENE-7587: The new FacetQuery and MultiFacetQuery helper classes
 2606   make it simpler to execute drill down when drill sideways counts are
 2607   not needed (Emmanuel Keller via Mike McCandless)
 2609 * LUCENE-6664: A new SynonymGraphFilter outputs a correct graph
 2610   structure for multi-token synonyms, separating out a
 2611   FlattenGraphFilter that is hardwired into the current
 2612   SynonymFilter.  This finally makes it possible to implement
 2613   correct multi-token synonyms at search time.  See
 2614   http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html
 2615   for details. (Mike McCandless)
 2617 * LUCENE-5325: Added LongValuesSource and DoubleValuesSource, intended as
 2618   type-safe replacements for ValueSource in the queries module.  These
 2619   expose per-segment LongValues or DoubleValues iterators. (Alan Woodward, Adrien Grand)
 2621 * LUCENE-7603: Graph token streams are now handled accurately by query
 2622   parsers, by enumerating all paths and creating the corresponding
 2623   query/ies as sub-clauses (Matt Weber via Mike McCandless)
 2625 * LUCENE-7588: DrillSideways can now run queries concurrently, and
 2626   supports an IndexSearcher using an executor service to run each query
 2627   concurrently across all segments in the index (Emmanuel Keller via
 2628   Mike McCandless)
 2630 * LUCENE-7627: Added .intersect methods to SortedDocValues and 
 2631   SortedSetDocValues to allow filtering their TermsEnums with a
 2632   CompiledAutomaton (Alan Woodward, Mike McCandless)
 2634 Bug Fixes
 2636 * LUCENE-7547: JapaneseTokenizerFactory was failing to close the
 2637   dictionary file it opened (Markus via Mike McCandless)
 2639 * LUCENE-7562: CompletionFieldsConsumer sometimes throws
 2640   NullPointerException on ghost fields (Oliver Eilhard via Mike McCandless)
 2642 * LUCENE-7533: Classic query parser: disallow autoGeneratePhraseQueries=true
 2643   when splitOnWhitespace=false (and vice-versa). (Steve Rowe)
 2645 * LUCENE-7536: ASCIIFoldingFilterFactory used to return an illegal multi-term
 2646   component when preserveOriginal was set to true. (Adrien Grand)
 2648 * LUCENE-7576: Fix Terms.intersect in the default codec to detect when
 2649   the incoming automaton is a special case and throw a clearer
 2650   exception than NullPointerException (Tom Mortimer via Mike McCandless)
 2652 * LUCENE-6989: Fix Exception handling in MMapDirectory's unmap hack
 2653   support code to work with Java 9's new InaccessibleObjectException
 2654   that does not extend ReflectiveAccessException in Java 9.
 2655   (Uwe Schindler)
 2657 * LUCENE-7581: Lucene now prevents updating a doc values field that is used
 2658   in the index sort, since this would lead to corruption.  (Jim
 2659   Ferenczi via Mike McCandless)
 2661 * LUCENE-7570: IndexWriter may deadlock if a commit is running while
 2662   there are too many merges running and one of the merges hits a
 2663   tragic exception (Joey Echeverria via Mike McCandless)
 2665 * LUCENE-7594: Fixed point range queries on floating-point types to recommend
 2666   using helpers for exclusive bounds that are consistent with Double.compare.
 2667   (Adrien Grand, Dawid Weiss)
 2669 * LUCENE-7606: Normalization with CustomAnalyzer would only apply the last
 2670   token filter. (Adrien Grand)
 2672 * LUCENE-7612: Removed an unused dependency from the suggester to the misc
 2673   module. (Alan Woodward)
 2675 Improvements
 2677 * LUCENE-7532: Add back lost codec file format documentation
 2678   (Shinichiro Abe via Mike McCandless)
 2680 * LUCENE-6824: TermAutomatonQuery now rewrites to TermQuery,
 2681   PhraseQuery or MultiPhraseQuery when the word automaton is simple
 2682   (Mike McCandless)
 2684 * LUCENE-7431: Allow a certain amount of overlap to be specified between the include
 2685   and exclude arguments of SpanNotQuery via negative pre and/or post arguments.
 2686   (Marc Morissette via David Smiley)
 2688 * LUCENE-7544: UnifiedHighlighter: add extension points for handling custom queries.
 2689   (Michael Braun, David Smiley)
 2691 * LUCENE-7538: Asking IndexWriter to store a too-massive text field
 2692   now throws IllegalArgumentException instead of a cryptic exception
 2693   that closes your IndexWriter (Steve Chen via Mike McCandless)
 2695 * LUCENE-7524: Added more detailed explanation of how IDF is computed in
 2696   ClassicSimilarity and BM25Similarity. (Adrien Grand)
 2698 * LUCENE-7564: AnalyzingInfixSuggester should close its IndexWriter by default
 2699   at the end of build(). (Steve Rowe)
 2701 * LUCENE-7526: Enhanced UnifiedHighlighter's passage relevancy for queries with
 2702   wildcards and sometimes just terms. Added shouldPreferPassageRelevancyOverSpeed()
 2703   which can be overridden to return false to eek out more speed in some cases.
 2704   (Timothy M. Rodriguez, David Smiley)
 2706 * LUCENE-7560: QueryBuilder.createFieldQuery is no longer final,
 2707   giving custom query parsers subclassing QueryBuilder more freedom to
 2708   control how text is analyzed and converted into a query (Matt Weber
 2709   via Mike McCandless)
 2711 * LUCENE-7537: Index time sorting now supports multi-valued sorts
 2712   using selectors (MIN, MAX, etc.) (Jim Ferenczi via Mike McCandless)
 2714 * LUCENE-7575: UnifiedHighlighter can now highlight fields with queries that don't
 2715   necessarily refer to that field (AKA requireFieldMatch==false). Disabled by default.
 2716   See UH get/setFieldMatcher. (Jim Ferenczi via David Smiley)
 2718 * LUCENE-7592: If the segments file is truncated, we now throw
 2719   CorruptIndexException instead of the more confusing EOFException
 2720   (Mike Drob via Mike McCandless)
 2722 * LUCENE-6989: Make MMapDirectory's unmap hack work with Java 9 EA (b150+):
 2723   Unmapping uses new sun.misc.Unsafe#invokeCleaner(ByteBuffer).
 2724   Java 9 now needs same permissions like Java 8;
 2725   RuntimePermission("accessClassInPackage.jdk.internal.ref")
 2726   is no longer needed. Support for older Java 9 builds was removed.
 2727   (Uwe Schindler)
 2729 * LUCENE-7401: Changed the way BKD trees pick the split dimension in order to
 2730   ensure all dimensions are indexed. (Adrien Grand)
 2732 * LUCENE-7614: Complex Phrase Query parser ignores double quotes around single token 
 2733   prefix, wildcard, range queries (Mikhail Khludnev)
 2735 * LUCENE-7620: Added LengthGoalBreakIterator, a wrapper around another B.I. to skip breaks
 2736   that would create Passages that are too short.  Only for use with the UnifiedHighlighter
 2737   (and probably PostingsHighlighter).  (David Smiley)
 2739 Optimizations
 2741 * LUCENE-7568: Optimize merging when index sorting is used but the
 2742   index is already sorted (Jim Ferenczi via Mike McCandless)
 2744 * LUCENE-7563: The BKD in-memory index for dimensional points now uses
 2745   a compressed format, using substantially less RAM in some cases
 2746   (Adrien Grand, Mike McCandless)
 2748 * LUCENE-7583: BKD writing now buffers each leaf block in heap before
 2749   writing to disk, giving a small speedup in points-heavy use cases.
 2750   (Mike McCandless)
 2752 * LUCENE-7572: Doc values queries now cache their hash code. (Adrien Grand)
 2754 Other
 2756 * LUCENE-7546: Fixed references to benchmark wikipedia data and the Jenkins line-docs file
 2757   (David Smiley)
 2759 * LUCENE-7534: fix smokeTestRelease.py to run on Cygwin (Mikhail Khludnev)
 2761 * LUCENE-7559: UnifiedHighlighter: Make Passage and OffsetsEnum more exposed to allow
 2762   passage creation to be customized. (David Smiley)
 2764 * LUCENE-7599: Simplify TestRandomChains using Java's built-in Predicate and
 2765   Function interfaces. (Ahmet Arslan via Adrien Grand)
 2767 * LUCENE-7595: Improve RAMUsageTester in test-framework to estimate memory usage of
 2768   runtime classes and work with Java 9 EA (b148+). Disable static field heap usage
 2769   checker in LuceneTestCase.  (Uwe Schindler, Dawid Weiss)
 2771 Build
 2773 * LUCENE-7387: fix defaultCodec in build.xml to account for the line ending (hossman)
 2775 * LUCENE-7543: Make changes-to-html target an offline operation, by moving the
 2776   Lucene and Solr DOAP RDF files into the Git source repository under
 2777   dev-tools/doap/ and then pulling release dates from those files, rather than
 2778   from JIRA. (Mano Kovacs, hossman, Steve Rowe)
 2780 * LUCENE-7596: Update Groovy to version 2.4.8 to allow building with Java 9
 2781   build 148+. Also update JGit version for working-copy checks. (Uwe Schindler)
 2783 ======================= Lucene 6.3.0 =======================
 2785 API Changes
 2787 New Features
 2789 * LUCENE-7438: New "UnifiedHighlighter" derivative of the PostingsHighlighter that
 2790   can consume offsets from postings, term vectors, or analysis.  It can highlight phrases
 2791   as accurately as the standard Highlighter. Light term vectors can be used with offsets
 2792   in postings for fast wildcard (MultiTermQuery) highlighting.
 2793   (David Smiley, Timothy Rodriguez)
 2795 * LUCENE-7490: SimpleQueryParser now parses '*' to MatchAllDocsQuery
 2796   (Lee Hinman via Mike McCandless)
 2798 Bug Fixes
 2800 * LUCENE-7507: Upgrade morfologik-stemming to version 2.1.1 (fixes security
 2801   manager issue with Polish dictionary lookup). (Dawid Weiss)
 2803 * LUCENE-7472: MultiFieldQueryParser.getFieldQuery() drops queries that are
 2804   neither BooleanQuery nor TermQuery.  (Steve Rowe)
 2806 * LUCENE-7456: PerFieldPostings/DocValues was failing to delegate the
 2807   merge method (Julien MASSENET via Mike McCandless)
 2809 * LUCENE-7468: ASCIIFoldingFilter should not emit duplicated tokens when
 2810   preserve original is on. (David Causse via Adrien Grand)
 2812 * LUCENE-7484: FastVectorHighlighter failed to highlight SynonymQuery
 2813   (Jim Ferenczi via Mike McCandless)
 2815 * LUCENE-7476: JapaneseNumberFilter should not invoke incrementToken
 2816   on its input after it's exhausted (Andy Hind via Mike McCandless)
 2818 * LUCENE-7486: DisjunctionMaxQuery does not work correctly with queries that
 2819   return negative scores.  (Ivan Provalov, Uwe Schindler, Adrien Grand)
 2821 * LUCENE-7491: Suddenly turning on dimensional points for some fields
 2822   that already exist in an index but didn't previously index
 2823   dimensional points could cause unexpected merge exceptions (Hans
 2824   Lund, Mike McCandless)
 2826 * LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points.
 2827   (Hossman)
 2829 * LUCENE-7493: FacetCollector.search threw an unexpected exception if
 2830   you asked for zero hits but wanted facets (Mahesh via Mike McCandless)
 2832 * LUCENE-7505: AnalyzingInfixSuggester returned invalid results when
 2833   allTermsRequired is false and context filters are specified (Mike
 2834   McCandless)
 2836 * LUCENE-7429: AnalyzerWrapper can now modify the normalization chain too and
 2837   DelegatingAnalyzerWrapper does the right thing automatically. (Adrien Grand)
 2839 * LUCENE-7135: Lucene's check for 32 or 64 bit JVM now works around security
 2840   manager blocking access to some properties (Aaron Madlon-Kay via
 2841   Mike McCandless)
 2843 Improvements
 2845 * LUCENE-7439: FuzzyQuery now matches all terms within the specified
 2846   edit distance, even if they are short terms (Mike McCandless)
 2848 * LUCENE-7496: Better toString for SweetSpotSimilarity (janhoy)
 2850 * LUCENE-7520: Highlighter's WeightedSpanTermExtractor shouldn't attempt to expand a MultiTermQuery
 2851   when its field doesn't match the field the extraction is scoped to.
 2852   (Cao Manh Dat via David Smiley)
 2854 Optimizations
 2856 * LUCENE-7501: BKDReader should not store the split dimension explicitly in the
 2857   1D case. (Adrien Grand)
 2859 Other
 2861 * LUCENE-7513: Upgrade randomizedtesting to 2.4.0. (Dawid Weiss)
 2863 * LUCENE-7452: Block join query exception suggests how to find a doc, which 
 2864  violates orthogonality requirement. (Mikhail Khludnev)
 2866 * LUCENE-7438: Renovate the Benchmark module's support for benchmarking highlighting. All
 2867   highlighters are supported via SearchTravRetHighlight. (David Smiley)
 2869 Build
 2871 * LUCENE-7292: Fix build to use "--release 8" instead of "-release 8" on
 2872   Java 9 (this changed with recent EA build b135).  (Uwe Schindler)
 2874 ======================= Lucene 6.2.1 =======================
 2876 API Changes
 2878 * LUCENE-7436: MinHashFilter's constructor, and some of its default
 2879   settings, should be public.  (Doug Turnbull via Mike McCandless)
 2881 Bug Fixes
 2883 * LUCENE-7417: The standard Highlighter could throw an IllegalArgumentException when
 2884   trying to highlight a query containing a degenerate case of a MultiPhraseQuery with one
 2885   term.  (Thomas Kappler via David Smiley)
 2887 * LUCENE-7440: Document id skipping (PostingsEnum.advance) could throw an
 2888   ArrayIndexOutOfBoundsException exception on large index segments (>1.8B docs)
 2889   with large skips. (yonik)
 2891 * LUCENE-7442: MinHashFilter's ctor should validate its args.
 2892   (Cao Manh Dat via Steve Rowe)
 2894 * LUCENE-7318: Fix backwards compatibility issues around StandardAnalyzer
 2895   and its components, introduced with Lucene 6.2.0. The moved classes
 2896   were restored in their original packages: LowercaseFilter and StopFilter,
 2897   as well as several utility classes.  (Uwe Schindler, Mike McCandless)
 2899 ======================= Lucene 6.2.0 =======================
 2901 API Changes
 2903 * ScoringWrapperSpans was removed since it had no purpose or effect as of Lucene 5.5.
 2905 New Features
 2907 * LUCENE-7388: Add point based IntRangeField, FloatRangeField, LongRangeField along with
 2908   supporting queries and tests (Nick Knize)
 2910 * LUCENE-7381: Add point based DoubleRangeField and RangeFieldQuery for
 2911   indexing and querying on Ranges up to 4 dimensions (Nick Knize)
 2913 * LUCENE-6968: LSH Filter (Tommaso Teofili, Andy Hind, Cao Manh Dat)
 2915 * LUCENE-7302: IndexWriter methods that change the index now return a
 2916   long "sequence number" indicating the effective equivalent
 2917   single-threaded execution order (Mike McCandless)
 2919 * LUCENE-7335: IndexWriter's commit data is now late binding,
 2920   recording key/values from a provided iterable based on when the
 2921   commit actually takes place (Mike McCandless)
 2923 * LUCENE-7287: UkrainianMorfologikAnalyzer is a new dictionary-based
 2924   analyzer for the Ukrainian language (Andriy Rysin via Mike
 2925   McCandless)
 2927 * LUCENE-7373: Directory.renameFile, which did both renaming and fsync
 2928   of the directory metadata, has been deprecated; use the new separate
 2929   methods Directory.rename and Directory.syncMetaData instead (Robert Muir,
 2930   Uwe Schindler, Mike McCandless)
 2932 * LUCENE-7355: Added Analyzer#normalize(), which only applies normalization to
 2933   an input string. (Adrien Grand)
 2935 * LUCENE-7380: Add Polygon.fromGeoJSON for more easily creating
 2936   Polygon instances from a standard GeoJSON string (Robert Muir, Mike
 2937   McCandless)
 2939 * LUCENE-7395: PerFieldSimilarityWrapper requires a default similarity
 2940   for calculating query norm and coordination factor in Lucene 6.x.
 2941   Lucene 7 will no longer have those factors.  (Uwe Schindler, Sascha Markus)
 2943 * SOLR-9279: Queries module: new ComparisonBoolFunction base class
 2944   (Doug Turnbull via David Smiley)
 2946 Bug Fixes
 2948 * LUCENE-6662: Fixed potential resource leaks. (Rishabh Patel via Adrien Grand)
 2950 * LUCENE-7340: MemoryIndex.toString() could throw NPE; fixed. Renamed to toStringDebug().
 2951   (Daniel Collins, David Smiley)
 2953 * LUCENE-7382: Fix bug introduced by LUCENE-7355 that used the
 2954   wrong default AttributeFactory for new Tokenizers.
 2955   (Terry Smith, Uwe Schindler)
 2957 * LUCENE-7389: Fix FieldType.setDimensions(...) validation for the dimensionNumBytes
 2958   parameter. (Martijn van Groningen)
 2960 * LUCENE-7391: Fix performance regression in MemoryIndex's fields() introduced
 2961   in Lucene 6. (Steve Mason via David Smiley)
 2963 * LUCENE-7395, SOLR-9315: Fix PerFieldSimilarityWrapper to also delegate query
 2964   norm and coordination factor using a default similarity added as ctor param.
 2965   (Uwe Schindler, Sascha Markus)
 2967 * SOLR-9413: Fix analysis/kuromoji's CSVUtil.quoteEscape logic, add TestCSVUtil test.
 2968   (AppChecker, Christine Poerschke)
 2970 * LUCENE-7419: Fix performance bug with TokenStream.end(), where it would lookup
 2971   PositionIncrementAttribute every time. (Mike McCandless, Robert Muir)
 2973 Improvements
 2975 * LUCENE-7323: Compound file writing now verifies the incoming
 2976   sub-files' checkums and segment IDs, to catch hardware issues or
 2977   filesytem bugs earlier (Robert Muir, Mike McCandless)
 2979 * LUCENE-6766: Index time sorting has graduated from the misc module
 2980   to core, is much simpler to use, via
 2981   IndexWriter.setIndexSort, and now works with dimensional points.
 2982   (Adrien Grand, Mike McCandless)
 2984 * LUCENE-5931: Detect when an application tries to reopen an
 2985   IndexReader after (illegally) removing the old index and
 2986   reindexing (Vitaly Funstein, Robert Muir, Mike McCandless)
 2988 * LUCENE-6171: Lucene now passes the StandardOpenOption.CREATE_NEW
 2989   option when writing new files so the filesystem enforces our
 2990   write-once architecture, possibly catching externally caused
 2991   issues sooner (Robert Muir, Mike McCandless)
 2993 * LUCENE-7318: StandardAnalyzer has been moved from the analysis
 2994   module into core and is now the default analyzer in
 2995   IndexWriterConfig (Robert Muir, Mike McCandless)
 2997 * LUCENE-7345: RAMDirectory now enforces write-once files as well
 2998   (Robert Muir, Mike McCandless)
 3000 * LUCENE-7337: MatchNoDocsQuery now scores with 0 normalization factor
 3001   and empty boolean queries now rewrite to MatchNoDocsQuery instead of
 3002   vice/versa (Jim Ferenczi via Mike McCandless)
 3004 * LUCENE-7359: Add equals() and hashCode() to Explanation (Alan Woodward)
 3006 * LUCENE-7353: ScandinavianFoldingFilterFactory and
 3007   ScandinavianNormalizationFilterFactory now implement MultiTermAwareComponent.
 3008   (Adrien Grand)
 3010 * LUCENE-2605: Add classic QueryParser option setSplitOnWhitespace() to
 3011   control whether to split on whitespace prior to text analysis.  Default
 3012   behavior remains unchanged: split-on-whitespace=true. (Steve Rowe)
 3014 * LUCENE-7276: MatchNoDocsQuery now includes an optional reason for
 3015   why it was used (Jim Ferenczi via Mike McCandless)
 3017 * LUCENE-7355: AnalyzingQueryParser now only applies the subset of the analysis
 3018   chain that is about normalization for range/fuzzy/wildcard queries.
 3019   (Adrien Grand)
 3021 * LUCENE-7376: Add support for ToParentBlockJoinQuery to fast vector highlighter's
 3022   FieldQuery. (Martijn van Groningen)
 3024 * LUCENE-7385: Improve/fix assert messages in SpanScorer. (David Smiley)
 3026 * LUCENE-7393: Add ICUTokenizer option to parse Myanmar text as syllables instead of words,
 3027   because the ICU word-breaking algorithm has some issues. This allows for the previous 
 3028   tokenization used before Lucene 5. (AM, Robert Muir)
 3030 * LUCENE-7409: Changed MMapDirectory's unmapping to work safer, but still with
 3031   no guarantees. This uses a store-store barrier and yields the current thread
 3032   before unmapping to allow in-flight requests to finish. The new code no longer
 3033   uses WeakIdentityMap as it delegates all ByteBuffer reads throgh a new
 3034   ByteBufferGuard wrapper that is shared between all ByteBufferIndexInput clones.
 3035   (Robert Muir, Uwe Schindler)
 3037 Optimizations
 3039 * LUCENE-7330, LUCENE-7339: Speed up conjunction queries. (Adrien Grand)
 3041 * LUCENE-7356: SearchGroup tweaks. (Christine Poerschke)
 3043 * LUCENE-7351: Doc id compression for points. (Adrien Grand)
 3045 * LUCENE-7371: Point values are now better compressed using run-length
 3046   encoding. (Adrien Grand)
 3048 * LUCENE-7311: Cached term queries do not seek the terms dictionary anymore.
 3049   (Adrien Grand)
 3051 * LUCENE-7396, LUCENE-7399: Faster flush of points.
 3052   (Adrien Grand, Mike McCandless)
 3054 * LUCENE-7406: Automaton and PrefixQuery tweaks (fewer object (re)allocations).
 3055   (Christine Poerschke)
 3057 Other
 3059 * LUCENE-4787: Fixed some highlighting javadocs. (Michael Dodsworth via Adrien
 3060   Grand)
 3062 * LUCENE-7334: Update ASM dependency to 5.1.  (Uwe Schindler)
 3064 * LUCENE-7346: Update forbiddenapis to version 2.2.
 3065   (Uwe Schindler)
 3067 * LUCENE-7360: Explanation.toHtml() is deprecated. (Alan Woodward)
 3069 * LUCENE-7372: Factor out an org.apache.lucene.search.FilterWeight class.
 3070   (Christine Poerschke, Adrien Grand, David Smiley)
 3072 * LUCENE-7384: Removed ScoringWrapperSpans. And tweaked SpanWeight.buildSimWeight() to
 3073   reuse the existing Similarity instead of creating a new one. (David Smiley)
 3075 ======================= Lucene 6.1.0 =======================
 3077 New Features
 3079 * LUCENE-7099: Add LatLonDocValuesField.newDistanceSort to the sandbox.
 3080   (Robert Muir)
 3082 * LUCENE-7140: Add PlanetModel.bisection to spatial3d (Karl Wright via
 3083   Mike McCandless)
 3085 * LUCENE-7069: Add LatLonPoint.nearest, to find nearest N points to a
 3086   provided query point (Mike McCandless)
 3088 * LUCENE-7234: Added InetAddressPoint.nextDown/nextUp to easily generate range
 3089   queries with excluded bounds. (Adrien Grand)
 3091 * LUCENE-7300: The misc module now has a directory wrapper that uses hard-links if
 3092   applicable and supported when copying files from another FSDirectory in 
 3093   Directory#copyFrom. (Simon Willnauer)
 3095 API Changes
 3097 * LUCENE-7184: Refactor LatLonPoint encoding methods to new GeoEncodingUtils
 3098   helper class in core geo package. Also refactors LatLonPointTests to
 3099   TestGeoEncodingUtils (Nick Knize)
 3101 * LUCENE-7163: refactor GeoRect, Polygon, and GeoUtils tests to geo
 3102   package in core (Nick Knize)
 3104 * LUCENE-7152: Refactor GeoUtils from lucene-spatial package to
 3105   core (Nick Knize)
 3107 * LUCENE-7141: Switch OfflineSorter's ByteSequencesReader to
 3108   BytesRefIterator (Mike McCandless)
 3110 * LUCENE-7150: Spatial3d gets useful APIs to create common shape
 3111   queries, matching LatLonPoint.  (Karl Wright via Mike McCandless)
 3113 * LUCENE-7243: Removed the LeafReaderContext parameter from
 3114   QueryCachingPolicy#shouldCache. (Adrien Grand)
 3116 Optimizations
 3118 * LUCENE-7071: Reduce bytes copying in OfflineSorter, giving ~10%
 3119   speedup on merging 2D LatLonPoint values (Mike McCandless)
 3121 * LUCENE-7105, LUCENE-7215: Optimize LatLonPoint's newDistanceQuery. 
 3122   (Robert Muir)
 3124 * LUCENE-7097: IntroSorter now recurses to 2 * log_2(count) quicksort
 3125   stack depth before switching to heapsort (Adrien Grand, Mike McCandless)
 3127 * LUCENE-7115: Speed up FieldCache.CacheEntry toString by setting initial
 3128   StringBuilder capacity (Gregory Chanan)
 3130 * LUCENE-7147: Improve disjoint check for geo distance query traversal
 3131   (Ryan Ernst, Robert Muir, Mike McCandless)
 3133 * LUCENE-7153: GeoPointField and LatLonPoint polygon queries now support
 3134   multiple polygons and holes, with memory usage independent of
 3135   polygon complexity. (Karl Wright, Mike McCandless, Robert Muir)
 3137 * LUCENE-7159: Speed up LatLonPoint polygon performance. (Robert Muir, Ryan Ernst)
 3139 * LUCENE-7211: Reduce memory & GC for spatial RPT Intersects when the number of
 3140   matching docs is small. (Jeff Wartes, David Smiley)
 3142 * LUCENE-7235: LRUQueryCache should not take a lock for segments that it will
 3143   not cache on anyway. (Adrien Grand)
 3145 * LUCENE-7238: Explicitly disable the query cache in MemoryIndex#createSearcher.
 3146   (Adrien Grand)
 3148 * LUCENE-7237: LRUQueryCache now prefers returning an uncached Scorer than
 3149   waiting on a lock. (Adrien Grand)
 3151 * LUCENE-7261, LUCENE-7262, LUCENE-7264, LUCENE-7258: Speed up DocIdSetBuilder
 3152   (which is used by TermsQuery, multi-term queries and several point queries).
 3153   (Adrien Grand, Jeff Wartes, David Smiley)
 3155 * LUCENE-7299: Speed up BytesRefHash.sort() using radix sort. (Adrien Grand)
 3157 * LUCENE-7306: Speed up points indexing and merging using radix sort.
 3158   (Adrien Grand)
 3160 Bug Fixes
 3162 * LUCENE-7127: Fix corner case bugs in GeoPointDistanceQuery. (Robert Muir)
 3164 * LUCENE-7166: Fix corner case bugs in LatLonPoint/GeoPointField bounding box
 3165   queries. (Robert Muir)
 3167 * LUCENE-7168: Switch to stable encode for geo3d, remove quantization
 3168   test leniency, remove dead code (Mike McCandless)
 3170 * LUCENE-7301: Multiple doc values updates to the same document within
 3171   one update batch could be applied in the wrong order resulting in
 3172   the wrong updated value (Ishan Chattopadhyaya, hossman, Mike McCandless)
 3174 * LUCENE-7312: Fix geo3d's x/y/z double to int encoding to ensure it always
 3175   rounds down (Karl Wright, Mike McCandless)
 3177 * LUCENE-7132: BooleanQuery sometimes assigned too-low scores in cases
 3178   where ranges of documents had only a single clause matching while
 3179   other ranges had more than one clause matching (Ahmet Arslan,
 3180   hossman, Mike McCandless)
 3182 * LUCENE-7286: Added support for highlighting SynonymQuery. (Adrien Grand)
 3184 * LUCENE-7291: Spatial heatmap faceting could mis-count when the heatmap crosses the
 3185   dateline and indexed non-point shapes are much bigger than the heatmap region.
 3186   (David Smiley)
 3188 * LUCENE-7333: Fix test bug where randomSimpleString() generated a filename
 3189   that is a reserved device name on Windows.  (Uwe Schindler, Mike McCandless)
 3191 Other
 3193 * LUCENE-7295: TermAutomatonQuery.hashCode calculates Automaton.toDot().hash,
 3194   equivalence relationship replaced with object identity. (Dawid Weiss)
 3196 * LUCENE-7277: Make Query.hashCode and Query.equals abstract. (Paul Elschot, 
 3197   Dawid Weiss)
 3199 * LUCENE-7174: Upgrade randomizedtesting to 2.3.4. (Uwe Schindler, Dawid Weiss)
 3201 * LUCENE-7205: Remove repeated nl.getLength() calls in
 3202   (Boolean|DisjunctionMax|FuzzyLikeThis)QueryBuilder. (Christine Poerschke)
 3204 * LUCENE-7210: Make TestCore*Parser's analyzer choice override-able
 3205   (Christine Poerschke, Daniel Collins)
 3207 * LUCENE-7263: Make queryparser/xml/CoreParser's SpanQueryBuilderFactory
 3208   accessible to deriving classes. (Daniel Collins via Christine Poerschke)
 3210 * SOLR-9109/SOLR-9121: Allow specification of a custom Ivy settings file via system
 3211   property "ivysettings.xml". (Misha Dmitriev, Christine Poerschke, Uwe Schindler, Steve Rowe)
 3213 * LUCENE-7206: Improve the ToParentBlockJoinQuery's explain by including the explain
 3214   of the best matching child doc. (Ilya Kasnacheev, Jeff Evans via Martijn van Groningen)
 3216 * LUCENE-7307: Add getters to the PointInSetQuery and PointRangeQuery queries.
 3217   (Martijn van Groningen, Adrien Grand)
 3219 Build
 3221 * LUCENE-7292: Use '-release' instead of '-source/-target' during
 3222   compilation on Java 9+ to ensure real cross-compilation.
 3223   (Uwe Schindler)
 3225 * LUCENE-7296: Update forbiddenapis to version 2.1.
 3226   (Uwe Schindler)
 3228 ======================= Lucene 6.0.1 =======================
 3230 New Features
 3232 * LUCENE-7278: Spatial-extras DateRangePrefixTree's Calendar is now configurable, to
 3233   e.g. clear the Gregorian Change Date.  Also, toString(cal) is now identical to
 3234   DateTimeFormatter.ISO_INSTANT. (David Smiley)
 3236 Bug Fixes
 3238 * LUCENE-7187: Block join queries' Weight#extractTerms(...) implementations
 3239   should delegate to the wrapped weight. (Martijn van Groningen)
 3241 * LUCENE-7209: Fixed explanations of FunctionScoreQuery. (Adrien Grand)
 3243 * LUCENE-7232: Fixed InetAddressPoint.newPrefixQuery, which was generating an
 3244   incorrect query when the prefix length was not a multiple of 8. (Adrien Grand)
 3246 * LUCENE-7279: JapaneseTokenizer throws ArrayIndexOutOfBoundsException
 3247   on some valid inputs (Mike McCandless)
 3249 * LUCENE-7188: remove incorrect sanity check in NRTCachingDirectory.listAll()
 3250   that led to IllegalStateException being thrown when nothing was wrong.
 3251   (David Smiley, yonik)
 3253 * LUCENE-7219: Make queryparser/xml (Point|LegacyNumeric)RangeQuery builders
 3254   match the underlying queries' (lower|upper)Term optionality logic.
 3255   (Kaneshanathan Srivisagan, Christine Poerschke)
 3257 * LUCENE-7257: Fixed PointValues#size(IndexReader, String), docCount,
 3258   minPackedValue and maxPackedValue to skip leaves that do not have points
 3259   rather than raising an IllegalStateException. (Adrien Grand)
 3261 * LUCENE-7284: GapSpans needs to implement positionsCost(). (Daniel Bigham, Alan
 3262   Woodward)
 3264 * LUCENE-7231: WeightedSpanTermExtractor didn't deal correctly with single-term
 3265   phrase queries. (Eva Popenda, Alan Woodward)
 3267 * LUCENE-7293: Don't try to highlight GeoPoint queries (Britta Weber,
 3268   Nick Knize, Mike McCandless, Uwe Schindler)
 3270 Documentation
 3272 * LUCENE-7223: Improve XXXPoint javadocs to make it clear that you
 3273   should separately add StoredField if you want to retrieve these
 3274   field values at search time (Greg Huber, Robert Muir, Mike McCandless)
 3276 ======================= Lucene 6.0.0 =======================
 3278 System Requirements
 3280 * LUCENE-5950: Move to Java 8 as minimum Java version.
 3281   (Ryan Ernst, Uwe Schindler)
 3283 * LUCENE-6069: Lucene Core now gets compiled with Java 8 "compact1" profile,
 3284   all other modules with "compact2".  (Robert Muir, Uwe Schindler)
 3286 New Features
 3288 * LUCENE-6631: Lucene Document classification (Tommaso Teofili, Alessandro Benedetti)
 3290 * LUCENE-6747: FingerprintFilter is a TokenFilter that outputs a single
 3291   token which is a concatenation of the sorted and de-duplicated set of 
 3292   input tokens. Useful for normalizing short text in clustering/linking 
 3293   tasks. (Mark Harwood, Adrien Grand)
 3295 * LUCENE-5735: NumberRangePrefixTreeStrategy now includes interval/range faceting
 3296   for counting ranges that align with the underlying terms as defined by the
 3297   NumberRangePrefixTree (e.g. familiar date units like days).  (David Smiley)
 3299 * LUCENE-6711: Use CollectionStatistics.docCount() for IDF and average field
 3300   length computations, to avoid skew from documents that don't have the field.
 3301   (Ahmet Arslan via Robert Muir)
 3303 * LUCENE-6758: Use docCount+1 for DefaultSimilarity's IDF, so that queries 
 3304   containing nonexistent fields won't screw up querynorm. (Terry Smith, Robert Muir)
 3306 * SOLR-7876: The QueryTimeout interface now has a isTimeoutEnabled method
 3307   that can return false to exit from ExitableDirectoryReader wrapping at
 3308   the point fields() is called. (yonik)
 3310 * LUCENE-6825: Add low-level support for block-KD trees (Mike McCandless)
 3312 * LUCENE-6852, LUCENE-6975: Add support for points (dimensionally
 3313   indexed values) to index, document and codec APIs, including a
 3314   simple text implementation.  (Mike McCandless)
 3316 * LUCENE-6861: Create Lucene60Codec, supporting points.
 3317   (Mike McCandless)
 3319 * LUCENE-6879: Allow to define custom CharTokenizer instances without
 3320   subclassing using Java 8 lambdas or method references. (Uwe Schindler)
 3322 * LUCENE-6881: Cutover all BKD implementations to points
 3323   (Mike McCandless)
 3325 * LUCENE-6837: Add N-best output support to JapaneseTokenizer.
 3326   (Hiroharu Konno via Christian Moen)
 3328 * LUCENE-6962: Add per-dimension min/max to points
 3329   (Mike McCandless)
 3331 * LUCENE-6975: Add ExactPointQuery, to match a single N-dimensional
 3332   point (Robert Muir, Mike McCandless)
 3334 * LUCENE-6989: Add preliminary support for MMapDirectory unmapping in Java 9.
 3335   (Uwe Schindler, Chris Hegarty, Peter Levart)
 3337 * LUCENE-7040: Upgrade morfologik-stemming to version 2.1.0.
 3338   (Dawid Weiss)
 3340 * LUCENE-7048: Add XXXPoint.newSetQuery, to create a query that
 3341   efficiently matches all documents containing any of the specified
 3342   point values.  This is the analog of TermsQuery, but for points
 3343   instead.   (Adrien Grand, Robert Muir, Mike McCandless)
 3345 API Changes
 3347 * LUCENE-7094: BBoxStrategy and PointVectorStrategy now support
 3348   PointValues (in addition to legacy numeric trie).  Their APIs
 3349   were changed a little and also made more consistent.  PointValues/Trie
 3350   is optional, DocValues is optional, stored value is optional.
 3351   (Nick Knize, David Smiley)
 3353 * LUCENE-6067: Accountable.getChildResources has a default
 3354   implementation returning the empty list.  (Robert Muir)
 3356 * LUCENE-6583: FilteredQuery has been removed. Instead, you can construct a
 3357   BooleanQuery with one MUST clause for the query, and one FILTER clause for
 3358   the filter. (Adrien Grand)
 3360 * LUCENE-6651: AttributeImpl#reflectWith(AttributeReflector) was made
 3361   abstract and has no reflection-based default implementation anymore.
 3362   (Uwe Schindler)
 3364 * LUCENE-6706: PayloadTermQuery and PayloadNearQuery have been removed.
 3365   Instead, use PayloadScoreQuery to wrap any SpanQuery. (Alan Woodward)
 3367 * LUCENE-6829: OfflineSorter, and the classes that use it (suggesters,
 3368   hunspell) now do all temporary file IO via Directory instead of
 3369   directly through java's temp dir.  Directory.createTempOutput
 3370   creates a uniquely named IndexOutput, and the new
 3371   IndexOutput.getName returns its name (Dawid Weiss, Robert Muir, Mike
 3372   McCandless)
 3374 * LUCENE-6917: Deprecate and rename NumericXXX classes to
 3375   LegacyNumericXXX in favor of points (Mike McCandless)
 3377 * LUCENE-6947: SortField.missingValue is now protected. You can read its
 3378   value using the new SortField.getMissingValue getter. (Adrien Grand)
 3380 * LUCENE-7028: Remove duplicate method in LegacyNumericUtils.
 3381   (Uwe Schindler)
 3383 * LUCENE-7052, LUCENE-7053: Remove custom comparators from BytesRef
 3384   class and solely use natural byte[] comparator throughout codebase.
 3385   This also simplifies API of BytesRefHash. It also replaces the natural
 3386   comparator in ArrayUtil by Java 8's Comparator#naturalOrder().
 3387   (Mike McCandless, Uwe Schindler, Robert Muir)
 3389 * LUCENE-7060: Update Spatial4j to 0.6.  The package com.spatial4j.core
 3390   is now org.locationtech.spatial4j. (David Smiley)
 3392 * LUCENE-7058: Add getters to various Query implementations (Guillaume Smet via
 3393   Alan Woodward)
 3395 * LUCENE-7064: MultiPhraseQuery is now immutable and should be constructed
 3396   with MultiPhraseQuery.Builder. (Luc Vanlerberghe via Adrien Grand)
 3398 * LUCENE-7072: Geo3DPoint always uses WGS84 planet model.
 3399   (Robert Muir, Mike McCandless)
 3401 * LUCENE-7056: Geo3D classes are in different packages now. (David Smiley)
 3403 * LUCENE-6952: These classes are now abstract: FilterCodecReader, FilterLeafReader,
 3404   FilterCollector, FilterDirectory.  And some Filter* classes in
 3405   lucene-test-framework too. (David Smiley)
 3407 * SOLR-8867: FunctionValues.getRangeScorer now takes a LeafReaderContext instead
 3408   of an IndexReader, and avoids matching documents without a value in the field
 3409   for numeric fields. (yonik)
 3411 Optimizations
 3413 * LUCENE-6891: Use prefix coding when writing points in 
 3414   each leaf block in the default codec, to reduce the index
 3415   size (Mike McCandless)
 3417 * LUCENE-6901: Optimize points indexing: use faster
 3418   IntroSorter instead of InPlaceMergeSorter, and specialize 1D
 3419   merging to merge sort the already sorted segments instead of
 3420   re-indexing (Mike McCandless)
 3422 * LUCENE-6793: LegacyNumericRangeQuery.hashCode() is now less subject to hash
 3423   collisions. (J.B. Langston via Adrien Grand)
 3425 * LUCENE-7050: TermsQuery is now cached more aggressively by the default
 3426   query caching policy. (Adrien Grand)
 3428 * LUCENE-7066: PointRangeQuery got optimized for the case that all documents
 3429   have a value and all points from the segment match. (Adrien Grand)
 3431 Changes in Runtime Behavior
 3433 * LUCENE-6789: IndexSearcher's default Similarity is changed to BM25Similarity.
 3434   Use ClassicSimilarity to get the old vector space DefaultSimilarity. (Robert Muir)
 3436 * LUCENE-6886: Reserve the .tmp file name extension for temp files,
 3437   and codec components are no longer allowed to use this extension
 3438   (Robert Muir, Mike McCandless)
 3440 * LUCENE-6835: Directory.listAll now returns entries in sorted order,
 3441   to not leak platform-specific behavior, and "retrying file deletion"
 3442   is now the responsibility of Directory.deleteFile, not the caller.
 3443   (Robert Muir, Mike McCandless)
 3445 Tests
 3447 * LUCENE-7009: Add expectThrows utility to LuceneTestCase. This uses a lambda
 3448   expression to encapsulate a statement that is expected to throw an exception.
 3449   (Ryan Ernst)
 3451 Bug Fixes
 3453 * LUCENE-7065: Fix the explain for the global ordinals join query. Before the
 3454   explain would also indicate that non matching documents would match.
 3455   On top of that with score mode average, the explain would fail with a NPE.
 3456   (Martijn van Groningen)
 3458 * LUCENE-7101: OfflineSorter had O(N^2) merge cost, and used too many
 3459   temporary file descriptors, for large sorts (Mike McCandless)
 3461 * LUCENE-7111: DocValuesRangeQuery.newLongRange behaves incorrectly for
 3462   Long.MAX_VALUE and Long.MIN_VALUE (Ishan Chattopadhyaya via Steve Rowe)
 3464 * LUCENE-7139: Fix bugs in geo3d's Vincenty surface distance
 3465   implementation (Karl Wright via Mike McCandless)
 3467 * LUCENE-7112: WeightedSpanTermExtractor.extractUnknownQuery is only called
 3468   on queries that could not be extracted. (Adrien Grand)
 3470 * LUCENE-7126: Remove GeoPointDistanceRangeQuery. This query was implemented
 3471   with boolean NOT, and incorrect for multi-valued documents. (Robert Muir)
 3473 * LUCENE-7158: Consistently use earth's WGS84 mean radius wherever our
 3474   geo search implementations approximate the earth as a sphere (Karl
 3475   Wright via Mike McCandless)
 3477 Other
 3479 * LUCENE-7035: Upgrade icu4j to 56.1/unicode 8. (Robert Muir)
 3481 * LUCENE-7087: Let MemoryIndex#fromDocument(...) accept 'Iterable<? extends IndexableField>'
 3482   as document instead of 'Document'. (Martijn van Groningen)
 3484 * LUCENE-7091: Add doc values support to MemoryIndex
 3485   (Martijn van Groningen, David Smiley)
 3487 * LUCENE-7093: Add point values support to MemoryIndex
 3488   (Martijn van Groningen, Mike McCandless)
 3490 * LUCENE-7095: Add point values support to the numeric field query time join.
 3491   (Martijn van Groningen, Mike McCandless)
 3493 ======================= Lucene 5.5.5 =======================
 3495 Changes in Runtime Behavior
 3497 * Resolving of external entities in queryparser/xml/CoreParser is disallowed
 3498   by default. See SOLR-11477 for details.
 3500 Bug Fixes
 3502 * LUCENE-7419: Fix performance bug with TokenStream.end(), where it would lookup
 3503   PositionIncrementAttribute every time. (Mike McCandless, Robert Muir)
 3505 * SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser
 3506   by default. (Michael Stepankin, Olga Barinova, Uwe Schindler, Christine Poerschke)
 3508 ======================= Lucene 5.5.4 =======================
 3510 Bug Fixes
 3512 * LUCENE-7417: The standard Highlighter could throw an IllegalArgumentException when
 3513   trying to highlight a query containing a degenerate case of a MultiPhraseQuery with one
 3514   term.  (Thomas Kappler via David Smiley)
 3516 * LUCENE-7657: Fixed potential memory leak in the case that a (Span)TermQuery
 3517   with a TermContext is cached. (Adrien Grand)
 3519 * LUCENE-7647: Made stored fields reclaim native memory more aggressively when
 3520   configured with BEST_COMPRESSION. This could otherwise result in out-of-memory
 3521   issues. (Adrien Grand)
 3523 * LUCENE-7562: CompletionFieldsConsumer sometimes throws
 3524   NullPointerException on ghost fields (Oliver Eilhard via Mike McCandless)
 3526 * LUCENE-7547: JapaneseTokenizerFactory was failing to close the
 3527   dictionary file it opened (Markus via Mike McCandless)
 3529 * LUCENE-6914: Fixed DecimalDigitFilter in case of supplementary code points.
 3530   (Hossman)
 3532 * LUCENE-7440: Document id skipping (PostingsEnum.advance) could throw an
 3533   ArrayIndexOutOfBoundsException exception on large index segments (>1.8B docs)
 3534   with large skips. (yonik)
 3536 * LUCENE-7570: IndexWriter may deadlock if a commit is running while
 3537   there are too many merges running and one of the merges hits a
 3538   tragic exception (Joey Echeverria via Mike McCandless)
 3540 Other
 3542 * LUCENE-6989: Backport MMapDirectory's unmapping code from Lucene 6.4 to use
 3543   MethodHandles. This allows it to work with Java 9 (EA build 150 and later).
 3544   (Uwe Schindler)
 3546 Build
 3548 * LUCENE-7543: Make changes-to-html target an offline operation, by moving the
 3549   Lucene and Solr DOAP RDF files into the Git source repository under
 3550   dev-tools/doap/ and then pulling release dates from those files, rather than
 3551   from JIRA. (Mano Kovacs, hossman, Steve Rowe)
 3553 * LUCENE-7596: Update Groovy to version 2.4.8 to allow building with Java 9
 3554   build 148+. Also update JGit version for working-copy checks. This does not
 3555   fix all issues with Java 9, but allows to build the distribution.
 3556   (Uwe Schindler)
 3558 * LUCENE-7651: Backport (Lucene 6.4.1) fix for Java 8u121 to allow documentation
 3559   build to inject "Google Code Prettify" without adding Javascript to Javadocs's
 3560   -bottom parameter. Unfortunately, this fix disables Prettify if Javadocs are
 3561   built with Java 7, as there is no generic way in Java 7 to inject Javascript
 3562   without breaking Java 8 (and possible paid Java 7 security updates). This
 3563   fix also updates Prettify to latest version to work around a Google Chrome
 3564   issue. (Uwe Schindler)
 3566 ======================= Lucene 5.5.3 =======================
 3567 (No Changes)
 3569 ======================= Lucene 5.5.2 =======================
 3571 Bug Fixes
 3573 * LUCENE-7065: Fix the explain for the global ordinals join query. Before the
 3574   explain would also indicate that non matching documents would match.
 3575   On top of that with score mode average, the explain would fail with a NPE.
 3576   (Martijn van Groningen)
 3578 * LUCENE-7111: DocValuesRangeQuery.newLongRange behaves incorrectly for
 3579   Long.MAX_VALUE and Long.MIN_VALUE (Ishan Chattopadhyaya via Steve Rowe)
 3581 * LUCENE-7139: Fix bugs in geo3d's Vincenty surface distance
 3582   implementation (Karl Wright via Mike McCandless)
 3584 * LUCENE-7187: Block join queries' Weight#extractTerms(...) implementations
 3585   should delegate to the wrapped weight. (Martijn van Groningen)
 3587 * LUCENE-7279: JapaneseTokenizer throws ArrayIndexOutOfBoundsException
 3588   on some valid inputs (Mike McCandless)
 3590 * LUCENE-7219: Make queryparser/xml (Point|LegacyNumeric)RangeQuery builders
 3591   match the underlying queries' (lower|upper)Term optionality logic.
 3592   (Kaneshanathan Srivisagan, Christine Poerschke)
 3594 * LUCENE-7284: GapSpans needs to implement positionsCost(). (Daniel Bigham, Alan
 3595   Woodward)
 3597 * LUCENE-7231: WeightedSpanTermExtractor didn't deal correctly with single-term
 3598   phrase queries. (Eva Popenda, Alan Woodward)
 3600 * LUCENE-7301: Multiple doc values updates to the same document within
 3601   one update batch could be applied in the wrong order resulting in
 3602   the wrong updated value (Ishan Chattopadhyaya, hossman, Mike McCandless)
 3604 * LUCENE-7132: BooleanQuery sometimes assigned too-low scores in cases
 3605   where ranges of documents had only a single clause matching while
 3606   other ranges had more than one clause matching (Ahmet Arslan,
 3607   hossman, Mike McCandless)
 3609 * LUCENE-7291: Spatial heatmap faceting could mis-count when the heatmap crosses the
 3610   dateline and indexed non-point shapes are much bigger than the heatmap region.
 3611   (David Smiley)
 3613 ======================= Lucene 5.5.1 =======================
 3615 Bug fixes
 3617 * LUCENE-7112: WeightedSpanTermExtractor.extractUnknownQuery is only called
 3618   on queries that could not be extracted. (Adrien Grand)
 3620 * LUCENE-7188: remove incorrect sanity check in NRTCachingDirectory.listAll()
 3621   that led to IllegalStateException being thrown when nothing was wrong.
 3622   (David Smiley, yonik)
 3624 * LUCENE-7209: Fixed explanations of FunctionScoreQuery. (Adrien Grand)
 3626 ======================= Lucene 5.5.0 =======================
 3628 New Features
 3630 * LUCENE-5868: JoinUtil.createJoinQuery(..,NumericType,..) query-time join 
 3631   for LONG and INT fields with NUMERIC and SORTED_NUMERIC doc values.
 3632   (Alexey Zelin via Mikhail Khludnev) 
 3634 * LUCENE-6939: Add exponential reciprocal scoring to
 3635   BlendedInfixSuggester, to even more strongly favor suggestions that
 3636   match closer to the beginning (Arcadius Ahouansou via Mike McCandless)
 3638 * LUCENE-6958: Improved CustomAnalyzer to take class references to factories
 3639   as alternative to their SPI name. This enables compile-time safety when
 3640   defining analyzer's components.  (Uwe Schindler, Shai Erera)
 3642 * LUCENE-6818, LUCENE-6986: Add DFISimilarity implementing the divergence 
 3643   from independence model. (Ahmet Arslan via Robert Muir)
 3645 * SOLR-4619: Added removeAllAttributes() to AttributeSource, which removes
 3646   all previously added attributes.
 3648 * LUCENE-7010: Added MergePolicyWrapper to allow easy wrapping of other policies.
 3649   (Shai Erera)
 3651 API Changes
 3653 * LUCENE-6997: refactor sandboxed GeoPointField and query classes to lucene-spatial 
 3654   module under new lucene.spatial.geopoint package (Nick Knize)
 3656 * LUCENE-6908: GeoUtils static relational methods have been refactored to new 
 3657   GeoRelationUtils and now correctly handle large irregular rectangles, and
 3658   pole crossing distance queries. (Nick Knize)
 3660 * LUCENE-6900: Grouping sortWithinGroup variables used to allow null to mean
 3661   Sort.RELEVANCE.  Null is no longer permitted.  (David Smiley)
 3663 * LUCENE-6919: The Scorer class has been refactored to expose an iterator
 3664   instead of extending DocIdSetIterator. asTwoPhaseIterator() has been renamed
 3665   to twoPhaseIterator() for consistency. (Adrien Grand)
 3667 * LUCENE-6973: TeeSinkTokenFilter no longer accepts a SinkFilter (the latter
 3668   has been removed). If you wish to filter the sinks, you can wrap them with
 3669   any other TokenFilter (e.g. a FilteringTokenFilter). Also, you can no longer
 3670   add a SinkTokenStream to an existing TeeSinkTokenFilter. If you need to
 3671   share multiple streams with a single sink, chain them with multiple
 3672   TeeSinkTokenFilters.
 3673   DateRecognizerSinkFilter was renamed to DateRecognizerFilter and moved under
 3674   analysis/common. TokenTypeSinkFilter was removed (use TypeTokenFilter instead).
 3675   TokenRangeSinkFilter was removed. (Shai Erera, Uwe Schindler)
 3677 * LUCENE-6980: Default applyAllDeletes to true when opening
 3678   near-real-time readers (Mike McCandless)
 3680 * LUCENE-6981: SpanQuery.getTermContexts() helper methods are now public, and
 3681   SpanScorer has a public getSpans() method. (Alan Woodward)
 3683 * LUCENE-6932: IndexInput.seek implementations now throw EOFException
 3684   if you seek beyond the end of the file (Adrien Grand, Mike McCandless)
 3686 * LUCENE-6988: IndexableField.tokenStream() no longer throws IOException
 3687   (Alan Woodward)
 3689 * LUCENE-7028: Deprecate a duplicate method in NumericUtils.
 3690   (Uwe Schindler)
 3692 Optimizations
 3694 * LUCENE-6930: Decouple GeoPointField from NumericType by using a custom
 3695   and efficient GeoPointTokenStream and TermEnum designed for GeoPoint prefix
 3696   terms. (Nick Knize)
 3698 * LUCENE-6951: Improve GeoPointInPolygonQuery using point orientation based
 3699   line crossing algorithm, and adding result for multi-value docs when least
 3700   1 point satisfies polygon criteria. (Nick Knize)
 3702 * LUCENE-6889: BooleanQuery.rewrite now performs some query optimization, in
 3703   particular to rewrite queries that look like: "+*:* #filter" to a
 3704   "ConstantScore(filter)". (Adrien Grand)
 3706 * LUCENE-6912: Grouping's Collectors now calculate a response to needsScores()
 3707   instead of always 'true'. (David Smiley)
 3709 * LUCENE-6815: DisjunctionScorer now advances two-phased iterators lazily,
 3710   stopping to evaluate them as soon as a single one matches. The other iterators
 3711   will be confirmed lazily when computing score() or freq(). (Adrien Grand)
 3713 * LUCENE-6926: MUST_NOT clauses now use the match cost API to run the slow bits
 3714   last whenever possible. (Adrien Grand)
 3716 * LUCENE-6944: BooleanWeight no longer creates sub-scorers if BS1 is not
 3717   applicable. (Adrien Grand)
 3719 * LUCENE-6940: MUST_NOT clauses execute faster, especially when they are sparse.
 3720   (Adrien Grand)
 3722 * LUCENE-6470: Improve efficiency of TermsQuery constructors. (Robert Muir)
 3724 Bug Fixes
 3726 * LUCENE-6976: BytesRefTermAttributeImpl.copyTo NPE'ed if BytesRef was null.
 3727   Added equals & hashCode, and a new test for these things. (David Smiley)
 3729 * LUCENE-6932: RAMDirectory's IndexInput was failing to throw
 3730   EOFException in some cases (Stéphane Campinas, Adrien Grand via Mike
 3731   McCandless)
 3733 * LUCENE-6896: Don't treat the smallest possible norm value as an infinitely
 3734   long document in SimilarityBase or BM25Similarity. Add more warnings to sims
 3735   that will not work well with extreme tf values. (Ahmet Arslan, Robert Muir)
 3737 * LUCENE-6984: SpanMultiTermQueryWrapper no longer modifies its wrapped query.
 3738   (Alan Woodward, Adrien Grand)
 3740 * LUCENE-6998: Fix a couple places to better detect truncated index files
 3741   as corruption.  (Robert Muir, Mike McCandless)
 3743 * LUCENE-7002: Fixed MultiCollector to not throw a NPE if setScorer is called
 3744   after one of the sub collectors is done collecting. (John Wang, Adrien Grand)
 3746 * LUCENE-7027: Fixed NumericTermAttribute to not throw IllegalArgumentException
 3747   after NumericTokenStream was exhausted.  (Uwe Schindler, Lee Hinman,
 3748   Mike McCandless)
 3750 * LUCENE-7018: Fix GeoPointTermQueryConstantScoreWrapper to add document on
 3751   first GeoPointField match. (Nick Knize)
 3753 * LUCENE-7019: Add two-phase iteration to GeoPointTermQueryConstantScoreWrapper.
 3754   (Robert Muir via Nick Knize)
 3756 * LUCENE-6989: Improve MMapDirectory's unmapping checks to catch more non-working
 3757   cases. The unmap-hack does not yet work with recent Java 9. Official support
 3758   will come with Lucene 6.  (Uwe Schindler)
 3760 Other
 3762 * LUCENE-6924: Upgrade randomizedtesting to 2.3.2. (Dawid Weiss)
 3764 * LUCENE-6920: Improve custom function checks in expressions module
 3765   to use MethodHandles and work without extra security privileges.
 3766   (Uwe Schindler, Robert Muir)
 3768 * LUCENE-6921: Fix SPIClassIterator#isParentClassLoader to don't
 3769   require extra permissions.  (Uwe Schindler)
 3771 * LUCENE-6923: Fix RamUsageEstimator to access private fields inside
 3772   AccessController block for computing size. (Robert Muir)
 3774 * LUCENE-6907: make TestParser extendable, rename test/.../xml/
 3775   NumericRangeQueryQuery.xml to NumericRangeQuery.xml
 3776   (Christine Poerschke)
 3778 * LUCENE-6925: add ForceMergePolicy class in test-framework
 3779   (Christine Poerschke)
 3781 * LUCENE-6945: factor out TestCorePlus(Queries|Extensions)Parser from
 3782   TestParser, rename TestParser to TestCoreParser (Christine Poerschke)
 3784 * LUCENE-6949: fix (potential) resource leak in SynonymFilterFactory
 3785   (https://scan.coverity.com/projects/5620 CID 120656)
 3786   (Christine Poerschke, Coverity Scan (via Rishabh Patel))
 3788 * LUCENE-6961: Improve Exception handling in AnalysisFactories /
 3789   AnalysisSPILoader: Don't wrap exceptions occuring in factory's
 3790   ctor inside InvocationTargetException.  (Uwe Schindler)
 3792 * LUCENE-6965: Expression's JavascriptCompiler now throw ParseException 
 3793   with bad function names or bad arity instead of IllegalArgumentException.
 3794   (Tomás Fernández Löbbe, Uwe Schindler, Ryan Ernst)
 3796 * LUCENE-6964: String-based signatures in JavascriptCompiler replaced
 3797   with better compile-time-checked MethodType; generated class files
 3798   are no longer marked as synthetic.  (Uwe Schindler)
 3800 * LUCENE-6978: Refactor several code places that lookup locales
 3801   by string name to use BCP47 locale tag instead. LuceneTestCase
 3802   now also prints locales on failing tests this way.
 3803   Locale#forLanguageTag() and Locale#toString() were placed on list
 3804   of forbidden signatures.  (Uwe Schindler, Robert Muir)
 3806 * LUCENE-6988: You can now add IndexableFields directly to a MemoryIndex,
 3807   and create a MemoryIndex from a lucene Document.  (Alan Woodward)
 3809 * LUCENE-7005: TieredMergePolicy tweaks (>= vs. >, @see get vs. set)
 3810   (Christine Poerschke)
 3812 * LUCENE-7006: increase BaseMergePolicyTestCase use (TestNoMergePolicy and
 3813   TestSortingMergePolicy now extend it, TestUpgradeIndexMergePolicy added)
 3814   (Christine Poerschke)
 3816 ======================= Lucene 5.4.1 =======================
 3818 Bug Fixes
 3820 * LUCENE-6910: fix 'if ... > Integer.MAX_VALUE' check in
 3821   (Binary|Numeric)DocValuesFieldUpdates.merge
 3822   (https://scan.coverity.com/projects/5620 CID 119973 and CID 120081)
 3823   (Christine Poerschke, Coverity Scan (via Rishabh Patel))
 3825 * LUCENE-6946: SortField.equals now takes the missingValue parameter into
 3826   account. (Adrien Grand)
 3828 * LUCENE-6918: LRUQueryCache.onDocIdSetEviction is only called when at least
 3829   one DocIdSet is being evicted. (Adrien Grand)
 3831 * LUCENE-6929: Fix SpanNotQuery rewriting to not drop the pre/post parameters.
 3832   (Tim Allison via Adrien Grand)
 3834 * LUCENE-6950: Fix FieldInfos handling of UninvertingReader, e.g. do not
 3835   hide the true docvalues update generation or other properties.
 3836   (Ishan Chattopadhyaya via Robert Muir)
 3838 * LUCENE-6948: Fix ArrayIndexOutOfBoundsException in PagedBytes$Reader.fill
 3839   by removing an unnecessary long-to-int cast.
 3840   (Michael Lawley via Christine Poerschke)
 3842 * SOLR-7865: BlendedInfixSuggester was returning too many results
 3843   (Arcadius Ahouansou via Mike McCandless)
 3845 * LUCENE-6970: Fixed off-by-one error in Lucene54DocValuesProducer that could
 3846   potentially corrupt doc values. (Adrien Grand)
 3848 * LUCENE-2229: Fix Highlighter's SimpleSpanFragmenter when multiple adjacent
 3849   stop words following a span can unduly make the fragment way too long.
 3850   (Elmer Garduno, Lukhnos Liu via David Smiley)
 3852 ======================= Lucene 5.4.0 =======================
 3854 New Features
 3856 * LUCENE-6875: New Serbian Filter. (Nikola Smolenski via Robert Muir, 
 3857   Dawid Weiss)
 3859 * LUCENE-6720: New FunctionRangeQuery wrapper around ValueSourceScorer
 3860   (returned from ValueSource/FunctionValues.getRangeScorer()). (David Smiley)
 3862 * LUCENE-6724: Add utility APIs to GeoHashUtils to compute neighbor
 3863   geohash cells (Nick Knize via Mike McCandless).
 3865 * LUCENE-6737: Add DecimalDigitFilter which folds unicode digits to basic latin.
 3866   (Robert Muir)
 3868 * LUCENE-6699: Add integration of BKD tree and geo3d APIs to give
 3869   fast, very accurate query to find all indexed points within an
 3870   earth-surface shape (Karl Wright, Mike McCandless)
 3872 * LUCENE-6838: Added IndexSearcher#getQueryCache and #getQueryCachingPolicy.
 3873   (Adrien Grand)
 3875 * LUCENE-6844: PayloadScoreQuery can include or exclude underlying span scores
 3876   from its score calculations (Bill Bell, Alan Woodward)
 3878 * LUCENE-6778: Add GeoPointDistanceRangeQuery, to search for points
 3879   within a "ring" (beyond a minimum distance and below a maximum
 3880   distance) (Nick Knize via Mike McCandless)
 3882 * LUCENE-6874: Add a new UnicodeWhitespaceTokenizer to analysis/common
 3883   that uses Unicode character properties extracted from ICU4J to tokenize
 3884   text on whitespace. This tokenizer will split on non-breaking
 3885   space (NBSP), too.  (David Smiley, Uwe Schindler, Steve Rowe)
 3887 API Changes
 3889 * LUCENE-6590: Query.setBoost(), Query.getBoost() and Query.clone() are gone.
 3890   In order to apply boosts, you now need to wrap queries in a BoostQuery.
 3891   (Adrien Grand)
 3893 * LUCENE-6716: SpanPayloadCheckQuery now takes a List<BytesRef> rather than
 3894   a Collection<byte[]>. (Alan Woodward)
 3896 * LUCENE-6489: The various span payload queries have been moved to the queries
 3897   submodule, and PayloadSpanUtil is now in sandbox. (Alan Woodward)
 3899 * LUCENE-6650: The spatial module no longer uses Filter in any way.  All
 3900   spatial Filters are now subclass Query.  The spatial heatmap/facet API
 3901   now accepts a Bits parameter to filter counts. (David Smiley, Adrien Grand)
 3903 * LUCENE-6803: Deprecate sandbox Regexp Query. (Uwe Schindler)
 3905 * LUCENE-6301: org.apache.lucene.search.Filter is now deprecated. You should use
 3906   Query objects instead of Filters, and the BooleanClause.Occur.FILTER clause in
 3907   order to let Lucene know that a Query should be used for filtering but not
 3908   scoring.
 3910 * LUCENE-6939: SpanOrQuery.addClause is now deprecated, clauses should all be
 3911   provided at construction time. (Paul Elschot via Adrien Grand)
 3913 * LUCENE-6855: CachingWrapperQuery is deprecated and will be removed in 6.0.
 3914   (Adrien Grand)
 3916 * LUCENE-6870: DisjunctionMaxQuery#add is now deprecated, clauses should all be
 3917   provided at construction time. (Adrien Grand) 
 3919 * LUCENE-6884: Analyzer.tokenStream() and Tokenizer.setReader() are no longer
 3920   declared as throwing IOException. (Alan Woodward)
 3922 * LUCENE-6849: Expose IndexWriter.flush() method, to move all
 3923   in-memory segments to disk without opening a near-real-time reader
 3924   nor calling fsync (Robert Muir, Simon Willnauer, Mike McCandless)
 3926 * LUCENE-6911: Add correct StandardQueryParser.getMultiFields() method,
 3927   deprecate no-op StandardQueryParser.getMultiFields(CharSequence[]) method.
 3928   (Christine Poerschke, Mikhail Khludnev, Coverity Scan (via Rishabh Patel))
 3930 Optimizations
 3932 * LUCENE-6708: TopFieldCollector does not compute the score several times on the
 3933   same document anymore. (Adrien Grand)
 3935 * LUCENE-6720: ValueSourceScorer, returned from
 3936   FunctionValues.getRangeScorer(), now uses TwoPhaseIterator. (David Smiley)
 3938 * LUCENE-6756: MatchAllDocsQuery now has a dedicated BulkScorer for better
 3939   performance when used as a top-level query. (Adrien Grand)
 3941 * LUCENE-6746: DisjunctionMaxQuery, BoostingQuery and BoostedQuery now create
 3942   sub weights through IndexSearcher so that they can be cached. (Adrien Grand)
 3944 * LUCENE-6754: Optimized IndexSearcher.count for the cases when it can use
 3945   index statistics instead of collecting all matches. (Adrien Grand)
 3947 * LUCENE-6773: Nested conjunctions now iterate over documents as if clauses
 3948   were all at the same level. (Adrien Grand)
 3950 * LUCENE-6777: Reuse BytesRef when visiting term ranges in
 3951   GeoPointTermsEnum to reduce GC pressure (Nick Knize via Mike
 3952   McCandless)
 3954 * LUCENE-6779: Reduce memory allocated by CompressingStoredFieldsWriter to write
 3955   strings larger than 64kb by an amount equal to string's utf8 size.
 3956   (Dawid Weiss, Robert Muir, shalin)
 3958 * LUCENE-6850: Optimize BooleanScorer for sparse clauses. (Adrien Grand)
 3960 * LUCENE-6840: Ordinal indexes for SORTED_SET/SORTED_NUMERIC fields and
 3961   addresses for BINARY fields are now stored on disk instead of in memory.
 3962   (Adrien Grand)
 3964 * LUCENE-6878: Speed up TopDocs.merge. (Daniel Jelinski via Adrien Grand)
 3966 * LUCENE-6885: StandardDirectoryReader (initialCapacity) tweaks
 3967   (Christine Poerschke)
 3969 * LUCENE-6863: Optimized storage requirements of doc values fields when less
 3970   than 1% of documents have a value. (Adrien Grand)
 3972 * LUCENE-6892: various lucene.index initialCapacity tweaks
 3973   (Christine Poerschke)
 3975 * LUCENE-6276: Added TwoPhaseIterator.matchCost() which allows to confirm the
 3976   least costly TwoPhaseIterators first. (Paul Elschot via Adrien Grand)
 3978 * LUCENE-6898: In the default codec, the last stored field value will not
 3979   be fully read from disk if the supplied StoredFieldVisitor doesn't want it.
 3980   So put your largest text field value last to benefit. (David Smiley)
 3982 * LUCENE-6909: Remove unnecessary synchronized from
 3983   FacetsConfig.getDimConfig for better concurrency (Sanne Grinovero
 3984   via Mike McCandless)
 3986 * SOLR-7730: Speed up SlowCompositeReaderWrapper.getSortedSetDocValues() by
 3987   avoiding merging FieldInfos just to check doc value type.
 3988   (Paul Vasilyev, Yuriy Pakhomov, Mikhail Khludnev, yonik)
 3990 Bug Fixes
 3992 * LUCENE-6905: Unwrap center longitude for dateline crossing 
 3993   GeoPointDistanceQuery. (Nick Knize)
 3995 * LUCENE-6817: ComplexPhraseQueryParser.ComplexPhraseQuery does not display 
 3996   slop in toString(). (Ahmet Arslan via Dawid Weiss)
 3998 * LUCENE-6730: Hyper-parameter c is ignored in term frequency NormalizationH1.
 3999   (Ahmet Arslan via Robert Muir)
 4001 * LUCENE-6742: Lovins & Finnish implementation of SnowballFilter was
 4002   fixed to behave exactly as specified. A bug in the snowball compiler
 4003   caused differences in output of the filter in comparison to the original
 4004   test data.  In addition, the performance of those filters was improved
 4005   significantly.  (Uwe Schindler, Robert Muir)
 4007 * LUCENE-6783: Removed side effects from FuzzyLikeThisQuery.rewrite.
 4008   (Adrien Grand)
 4010 * LUCENE-6776: Fix geo3d math to handle randomly squashed planet
 4011   models (Karl Wright via Mike McCandless) 
 4013 * LUCENE-6792: Fix TermsQuery.toString() to work with binary terms.
 4014   (Ruslan Muzhikov, Robert Muir)
 4016 * LUCENE-5503: When Highlighter's WeightedSpanTermExtractor converts a
 4017   PhraseQuery to an equivalent SpanQuery, it would sometimes use a slop that is
 4018   too low (no highlight) or determine inOrder wrong.
 4019   (Tim Allison via David Smiley)
 4021 * LUCENE-6790: Fix IndexWriter thread safety when one thread is
 4022   handling a tragic exception but another is still committing (Mike
 4023   McCandless)
 4025 * LUCENE-6810: Upgrade to Spatial4j 0.5 -- fixes some edge-case bugs in the
 4026   spatial module. See https://github.com/locationtech/spatial4j/blob/master/CHANGES.md
 4027   (David Smiley)
 4029 * LUCENE-6813: OfflineSorter no longer removes its output Path up
 4030   front, and instead opens it for write with the
 4031   StandardCopyOption.REPLACE_EXISTING to overwrite any prior file, so
 4032   that callers can safely use Files.createTempFile for the output.
 4033   This change also fixes OfflineSorter's default temp directory when
 4034   running tests to use mock filesystems so e.g. we detect file handle
 4035   leaks (Dawid Weiss, Robert Muir, Mike McCandless)
 4037 * LUCENE-6813: RangeTreeWriter was failing to close all file handles
 4038   it opened, leading to intermittent failures on Windows (Dawid Weiss,
 4039   Robert Muir, Mike McCandless)
 4041 * LUCENE-6826: Fix ClassCastException when merging a field that has no
 4042   terms because they were filtered out by e.g. a FilterCodecReader
 4043   (Trejkaz via Mike McCandless)
 4045 * LUCENE-6823: LocalReplicator should use System.nanoTime as its clock
 4046   source for checking for expiration (Ishan Chattopadhyaya via Mike
 4047   McCandless)
 4049 * LUCENE-6856: The Weight wrapper used by LRUQueryCache now delegates to the
 4050   original Weight's BulkScorer when applicable. (Adrien Grand)
 4052 * LUCENE-6858: Fix ContextSuggestField to correctly wrap token stream
 4053   when using CompletionAnalyzer. (Areek Zillur)
 4055 * LUCENE-6872: IndexWriter handles any VirtualMachineError, not just OOM,
 4056   as tragic. (Robert Muir)
 4058 * LUCENE-6814: PatternTokenizer no longer hangs onto heap sized to the
 4059   maximum input string it's ever seen, which can be a large memory
 4060   "leak" if you tokenize large strings with many threads across many
 4061   indices (Alex Chow via Mike McCandless)
 4063 * LUCENE-6888: Explain output of map() function now also prints default value (janhoy)
 4065 Other
 4067 * LUCENE-6899: Upgrade randomizedtesting to 2.3.1. (Dawid Weiss)
 4069 * LUCENE-6478: Test execution can hang with java.security.debug. (Dawid Weiss)
 4071 * LUCENE-6862: Upgrade of RandomizedRunner to version 2.2.0. (Dawid Weiss)
 4073 * LUCENE-6857: Validate StandardQueryParser with NOT operator 
 4074   with-in parantheses. (Jigar Shah via Dawid Weiss)
 4076 * LUCENE-6827: Use explicit capacity ArrayList instead of a LinkedList 
 4077   in MultiFieldQueryNodeProcessor. (Dawid Weiss).
 4079 * LUCENE-6812: Upgrade RandomizedTesting to 2.1.17. (Dawid Weiss)
 4081 * LUCENE-6174: Improve "ant eclipse" to select right JRE for building.
 4082   (Uwe Schindler, Dawid Weiss)
 4084 * LUCENE-6417, LUCENE-6830: Upgrade ANTLR used in expressions module
 4085   to version 4.5.1-1.  (Jack Conradson, Uwe Schindler)
 4087 * LUCENE-6729: Upgrade ASM used in expressions module to version 5.0.4.
 4088   (Uwe Schindler)
 4090 * LUCENE-6738: remove IndexWriterConfig.[gs]etIndexingChain
 4091   (Christine Poerschke)
 4093 * LUCENE-6755: more tests of ToChildBlockJoinScorer.advance (hossman)
 4095 * LUCENE-6571: fix some private access level javadoc errors and warnings
 4096   (Cao Manh Dat, Christine Poerschke)
 4098 * LUCENE-6768: AbstractFirstPassGroupingCollector.groupSort private member
 4099   is not needed. (Christine Poerschke)
 4101 * LUCENE-6761: MatchAllDocsQuery's Scorers do not expose approximations
 4102   anymore. (Adrien Grand)
 4104 * LUCENE-6775, LUCENE-6833: Improved MorfologikFilterFactory to allow 
 4105   loading of custom dictionaries from ResourceLoader. Upgraded 
 4106   Morfologik to version 2.0.1. The 'dictionary' attribute has been
 4107   reverted back and now points at the dictionary resource to be 
 4108   loaded instead of the default Polish dictionary.
 4109   (Uwe Schindler, Dawid Weiss)
 4111 * LUCENE-6797: Make GeoCircle an interface and use a factory to create
 4112   it, to eventually handle degenerate cases (Karl Wright via Mike
 4113   McCandless)
 4115 * LUCENE-6800: Use XYZSolidFactory to create XYZSolids (Karl Wright
 4116   via Mike McCandless)
 4118 * LUCENE-6798: Geo3d now models degenerate (too tiny) circles as a
 4119   single point (Karl Wright via Mike McCandless)
 4121 * LUCENE-6770: Add javadocs that FSDirectory canonicalizes the path.
 4122   (Uwe Schindler, Vladimir Kuzmin)
 4124 * LUCENE-6795: Fix various places where code used
 4125   AccessibleObject#setAccessible() without a privileged block. Code
 4126   without a hard requirement to do reflection were rewritten. This
 4127   makes Lucene and Solr ready for Java 9 Jigsaw's module system, where
 4128   reflection on Java's runtime classes is very restricted.
 4129   (Robert Muir, Uwe Schindler)
 4131 * LUCENE-6467: Simplify Query.equals. (Paul Elschot via Adrien Grand)
 4133 * LUCENE-6845: SpanScorer is now merged into Spans (Alan Woodward, David Smiley)
 4135 * LUCENE-6887: DefaultSimilarity is deprecated, use ClassicSimilarity for equivalent behavior,
 4136   or consider switching to BM25Similarity which will become the new default in Lucene 6.0 (hossman)
 4138 * LUCENE-6893: factor out CorePlusQueriesParser from CorePlusExtensionsParser
 4139   (Christine Poerschke)
 4141 * LUCENE-6902: Don't retry to fsync files / directories; fail
 4142   immediately. (Daniel Mitterdorfer, Uwe Schindler)
 4144 * LUCENE-6801: Clarify JavaDocs of PhraseQuery that it in fact supports terms
 4145   at the same position (as does MultiPhraseQuery), treated like a conjunction.
 4146   Added test. (David Smiley, Adrien Grand)
 4148 Build
 4150 * LUCENE-6732: Improve checker for invalid source patterns to also
 4151   detect javadoc-style license headers. Use Groovy to implement the
 4152   checks instead of plain Ant.  (Uwe Schindler)
 4154 * LUCENE-6594: Update forbiddenapis to 2.0.  (Uwe Schindler)
 4156 Tests
 4158 * LUCENE-6752: Add Math#random() to forbiddenapis.  (Uwe Schindler,
 4159   Mikhail Khludnev, Andrei Beliakov)
 4161 Changes in Backwards Compatibility Policy
 4163 * LUCENE-6742: The Lovins & Finnish implementation of SnowballFilter
 4164   were fixed to now behave exactly like the original Snowball stemmer.
 4165   If you have indexed text using those stemmers you may need to reindex.
 4166   (Uwe Schindler, Robert Muir)
 4168 Changes in Runtime Behavior
 4170 * LUCENE-6772: MultiCollector now catches CollectionTerminatedException and
 4171   removes the collector that threw this exception from the list of sub
 4172   collectors to collect. (Adrien Grand)
 4174 * LUCENE-6784: IndexSearcher's query caching is enabled by default. Run
 4175   indexSearcher.setQueryCache(null) to disable. (Adrien Grand)
 4177 * LUCENE-6305: BooleanQuery.equals and hashcode do not depend on the order of
 4178   clauses anymore. (Adrien Grand)
 4180 ======================= Lucene 5.3.2 =======================
 4182 Bug Fixes
 4184 * SOLR-7865: BlendedInfixSuggester was returning too many results
 4185   (Arcadius Ahouansou via Mike McCandless)
 4187 ======================= Lucene 5.3.1 =======================
 4189 Bug Fixes
 4191 * LUCENE-6774: Remove classloader hack in MorfologikFilter. (Robert Muir,
 4192   Uwe Schindler)
 4194 * LUCENE-6748: UsageTrackingQueryCachingPolicy no longer caches trivial queries
 4195   like MatchAllDocsQuery. (Adrien Grand)
 4197 * LUCENE-6781: Fixed BoostingQuery to rewrite wrapped queries. (Adrien Grand)
 4199 Tests
 4201 * LUCENE-6760, SOLR-7958: Move TestUtil#randomWhitespace to the only
 4202   Solr test that is using it. The method is not useful for Lucene tests
 4203   (and easily breaks, e.g., in Java 9 caused by Unicode version updates).
 4204   (Uwe Schindler)
 4207 ======================= Lucene 5.3.0 =======================
 4209 New Features
 4211 * LUCENE-6485: Add CustomSeparatorBreakIterator to postings
 4212   highlighter which splits on any character. For example, it 
 4213   can be used with getMultiValueSeparator render whole field
 4214   values.  (Luca Cavanna via Robert Muir)
 4216 * LUCENE-6459: Add common suggest API that mirrors Lucene's
 4217   Query/IndexSearcher APIs for Document based suggester.
 4218   Adds PrefixCompletionQuery, RegexCompletionQuery,
 4219   FuzzyCompletionQuery and ContextQuery.
 4220   (Areek Zillur via Mike McCandless)
 4222 * LUCENE-6487: Spatial Geo3D API now has a WGS84 ellipsoid world model option.
 4223   (Karl Wright via David Smiley)
 4225 * LUCENE-6477: Add experimental BKD geospatial tree doc values format
 4226   and queries, for fast "bbox/polygon contains lat/lon points" (Mike
 4227   McCandless)
 4229 * LUCENE-6526: Asserting(Query|Weight|Scorer) now ensure scores are not computed
 4230   if they are not needed. (Adrien Grand)
 4232 * LUCENE-6481: Add GeoPointField, GeoPointInBBoxQuery,
 4233   GeoPointInPolygonQuery for simple "indexed lat/lon point in
 4234   bbox/shape" searching.  (Nick Knize via Mike McCandless)
 4236 * LUCENE-5954: The segments_N commit point now stores the Lucene
 4237   version that wrote the commit as well as the lucene version that
 4238   wrote the oldest segment in the index, for faster checking of "too
 4239   old" indices (Ryan Ernst, Robert Muir, Mike McCandless)
 4241 * LUCENE-6519: BKDPointInPolygonQuery is much faster by avoiding
 4242   the per-hit polygon check when a leaf cell is fully contained by the
 4243   polygon.  (Nick Knize, Mike McCandless)
 4245 * LUCENE-6549: Add preload option to MMapDirectory. (Robert Muir)
 4247 * LUCENE-6504: Add Lucene53Codec, with norms implemented directly
 4248   via the Directory's RandomAccessInput api. (Robert Muir)
 4250 * LUCENE-6539: Add new DocValuesNumbersQuery, to match any document
 4251   containing one of the specified long values.  This change also
 4252   moves the existing DocValuesTermsQuery and DocValuesRangeQuery
 4253   to Lucene's sandbox module, since in general these queries are
 4254   quite slow and are only fast in specific cases.  (Adrien Grand,
 4255   Robert Muir, Mike McCandless)
 4257 * LUCENE-6577: Give earlier and better error message for invalid CRC.
 4258   (Robert Muir)
 4260 * LUCENE-6544: Geo3D: (1) Regularize path & polygon construction, (2) add
 4261   PlanetModel.surfaceDistance() (ellipsoidal calculation), (3) cache lat & lon
 4262   in GeoPoint, (4) add thread-safety where missing -- Geo3dShape. (Karl Wright,
 4263   David Smiley)
 4265 * LUCENE-6606: SegmentInfo.toString now confesses how the documents
 4266   were sorted, when SortingMergePolicy was used (Christine Poerschke
 4267   via Mike McCandless)
 4269 * LUCENE-6524: IndexWriter can now be initialized from an already open
 4270   near-real-time or non-NRT reader.  (Boaz Leskes, Robert Muir, Mike
 4271   McCandless)
 4273 * LUCENE-6578: Geo3D can now compute the distance from a point to a shape, both
 4274   inner distance and to an outside edge. Multiple distance algorithms are
 4275   available.  (Karl Wright, David Smiley)
 4277 * LUCENE-6632: Geo3D: Compute circle planes more accurately.
 4278   (Karl Wright via David Smiley)
 4280 * LUCENE-6653: Added general purpose BytesTermAttribute to basic token
 4281   attributes package that can be used for TokenStreams that solely produce
 4282   binary terms.  (Uwe Schindler)
 4284 * LUCENE-6365: Add Operations.topoSort, to run topological sort of the
 4285   states in an Automaton (Markus Heiden via Mike McCandless)
 4287 * LUCENE-6365: Replace Operations.getFiniteStrings with a
 4288   more scalable iterator API (FiniteStringsIterator) (Markus Heiden
 4289   via Mike McCandless)
 4291 * LUCENE-6589: Add a new org.apache.lucene.search.join.CheckJoinIndex class
 4292   that can be used to validate that an index has an appropriate structure to
 4293   run join queries. (Adrien Grand)
 4295 * LUCENE-6659: Remove IndexWriter's unnecessary hard limit on max concurrency
 4296   (Robert Muir, Mike McCandless)
 4298 * LUCENE-6547: Add GeoPointDistanceQuery, matching all points within
 4299   the specified distance from the center point.  Fix
 4300   GeoPointInBBoxQuery to handle dateline crossing.
 4302 * LUCENE-6694: Add LithuanianAnalyzer and LithuanianStemmer.
 4303   (Dainius Jocas via Robert Muir)
 4305 * LUCENE-6695: Added a new BlendedTermQuery to blend statistics across several
 4306   terms. (Simon Willnauer, Adrien Grand)
 4308 * LUCENE-6706: Added a new PayloadScoreQuery that generalises the behaviour of
 4309   PayloadTermQuery and PayloadNearQuery to all Span queries. (Alan Woodward)
 4311 * LUCENE-6697: Add experimental range tree doc values format and
 4312   queries, based on a 1D version of the spatial BKD tree, for a faster
 4313   and smaller alternative to postings-based numeric and binary term
 4314   filtering.  Range trees can also handle values larger than 64 bits.
 4315   (Adrien Grand, Mike McCandless)
 4317 * LUCENE-6647: Add GeoHash string utility APIs (Nick Knize via Mike
 4318   McCandless).
 4320 * LUCENE-6710: GeoPointField now uses full 64 bits (up from 62) to encode
 4321   lat/lon (Nick Knize via Mike McCandless).
 4323 * LUCENE-6580: SpanNearQuery now allows defined-width gaps in its subqueries
 4324   (Alan Woodward, Adrien Grand).
 4326 * LUCENE-6712: Use doc values to post-filter GeoPointField hits that
 4327   fall in boundary cells, resulting in smaller index, faster searches
 4328   and less heap used for each query (Nick Knize via Mike McCandless).
 4330 API Changes
 4332 * LUCENE-6508: Simplify Lock api, there is now just 
 4333   Directory.obtainLock() which returns a Lock that can be 
 4334   released (or fails with exception). Add lock verification 
 4335   to IndexWriter. Improve exception messages when locking fails.
 4336   (Uwe Schindler, Mike McCandless, Robert Muir)
 4338 * LUCENE-6371, LUCENE-6490: Payload collection from Spans is moved to a more generic
 4339   SpanCollector framework.  Spans no longer implements .hasPayload() and
 4340   .getPayload() methods, and instead exposes a collect() method that allows
 4341   the collection of arbitrary postings information. SpanPayloadCheckQuery and
 4342   SpanPayloadNearCheckQuery have moved from the .spans package to the .payloads
 4343   package. (Alan Woodward, David Smiley, Paul Elschot, Robert Muir)
 4345 * LUCENE-6529: Removed an optimization in UninvertingReader that was causing
 4346   incorrect results for Numeric fields using precisionStep
 4347   (hossman, Robert Muir)
 4349 * LUCENE-6551: Add missing ConcurrentMergeScheduler.getAutoIOThrottle
 4350   getter (Simon Willnauer, Mike McCandless)
 4352 * LUCENE-6552: Add MergePolicy.OneMerge.getMergeInfo and rename
 4353   setInfo to setMergeInfo (Simon Willnauer, Mike McCandless)
 4355 * LUCENE-6525: Deprecate IndexWriterConfig's writeLockTimeout.
 4356   (Robert Muir)
 4358 * LUCENE-6583: FilteredQuery is deprecated and will be removed in 6.0. It should
 4359   be replaced with a BooleanQuery which handle the query as a MUST clause and
 4360   the filter as a FILTER clause. (Adrien Grand)
 4362 * LUCENE-6553: The postings, spans and scorer APIs no longer take an acceptDocs
 4363   parameter. Live docs are now always checked on top of these APIs.
 4364   (Adrien Grand)
 4366 * LUCENE-6634: PKIndexSplitter now takes a Query instead of a Filter to decide
 4367   how to split an index. (Adrien Grand)
 4369 * LUCENE-6643: GroupingSearch from lucene/grouping was changed to take a Query
 4370   object to define groups instead of a Filter. (Adrien Grand)
 4372 * LUCENE-6554: ToParentBlockJoinFieldComparator was removed because of a bug
 4373   with missing values that could not be fixed. ToParentBlockJoinSortField now
 4374   works with string or numeric doc values selectors. Sorting on anything else
 4375   than a string or numeric field would require to implement a custom selector.
 4376   (Adrien Grand)
 4378 * LUCENE-6648: All lucene/facet APIs now take Query objects where they used to
 4379   take Filter objects. (Adrien Grand)
 4381 * LUCENE-6640: Suggesters now take a BitsProducer object instead of a Filter
 4382   object to reduce the scope of doc IDs that may be returned, emphasizing the
 4383   fact that these objects need to support random-access. (Adrien Grand)
 4385 * LUCENE-6646: Make EarlyTerminatingCollector take a Sort object directly
 4386   instead of a SortingMergePolicy. (Christine Poerschke via Adrien Grand)
 4388 * LUCENE-6649: BitDocIdSetFilter and BitDocIdSetCachingWrapperFilter are now
 4389   deprecated in favour of BitSetProducer and QueryBitSetProducer, which do not
 4390   extend oal.search.Filter. (Adrien Grand)
 4392 * LUCENE-6607: Factor out geo3d into its own spatial3d module.  (Karl
 4393   Wright, Nick Knize, David Smiley, Mike McCandless)
 4395 * LUCENE-6531: PhraseQuery is now immutable and can be built using the
 4396   PhraseQuery.Builder class. (Adrien Grand)
 4398 * LUCENE-6570: BooleanQuery is now immutable and can be built using the
 4399   BooleanQuery.Builder class. (Adrien Grand)
 4401 * LUCENE-6702: NRTSuggester: Add a method to inject context values at index time
 4402   in ContextSuggestField. Simplify ContextQuery logic for extracting contexts and
 4403   add dedicated method to consider all context values at query time.
 4404   (Areek Zillur, Mike McCandless)
 4406 * LUCENE-6719: NumericUtils getMinInt, getMaxInt, getMinLong, getMaxLong now
 4407   return null if there are no terms for the specified field, previously these
 4408   methods returned primitive values and raised an undocumented NullPointerException
 4409   if there were no terms for the field. (hossman, Timothy Potter)
 4411 Bug fixes
 4413 * LUCENE-6500: ParallelCompositeReader did not always call
 4414   closed listeners. This was fixed by LUCENE-6501.
 4415   (Adrien Grand, Uwe Schindler)
 4417 * LUCENE-6520: Geo3D GeoPath.done() would throw an NPE if adjacent path
 4418   segments were co-linear. (Karl Wright via David Smiley)
 4420 * LUCENE-5805: QueryNodeImpl.removeFromParent was doing nothing in a
 4421   costly manner (Christoph Kaser, Cao Manh Dat via Mike McCAndless)
 4423 * LUCENE-6533: SlowCompositeReaderWrapper no longer caches its live docs
 4424   instance since this can prevent future improvements like a
 4425   disk-backed live docs (Adrien Grand, Mike McCandless)
 4427 * LUCENE-6558: Highlighters now work with CustomScoreQuery (Cao Manh
 4428   Dat via Mike McCandless)
 4430 * LUCENE-6560: BKDPointInBBoxQuery now handles "dateline crossing"
 4431   correctly (Nick Knize, Mike McCandless)
 4433 * LUCENE-6564: Change PrintStreamInfoStream to use thread safe Java 8
 4434   ISO-8601 date formatting (in Lucene 5.x use Java 7 FileTime#toString
 4435   as workaround); fix output of tests to use same format.  (Uwe Schindler,
 4436   Ramkumar Aiyengar)
 4438 * LUCENE-6593: Fixed ToChildBlockJoinQuery's scorer to not refuse to advance
 4439   to a document that belongs to the parent space. (Adrien Grand)
 4441 * LUCENE-6591: Never write a negative vLong (Robert Muir, Ryan Ernst,
 4442   Adrien Grand, Mike McCandless)
 4444 * LUCENE-6588: Fix how ToChildBlockJoinQuery deals with acceptDocs.
 4445   (Christoph Kaser via Adrien Grand)
 4447 * LUCENE-6597: Geo3D's GeoCircle now supports a world-globe diameter.
 4448   (Karl Wright via David Smiley)
 4450 * LUCENE-6608: Fix potential resource leak in BigramDictionary.
 4451   (Rishabh Patel via Uwe Schindler)
 4453 * LUCENE-6614: Improve partition detection in IOUtils#spins() so it
 4454   works with NVMe drives.  (Uwe Schindler, Mike McCandless)
 4456 * LUCENE-6586: Fix typo in GermanStemmer, causing possible wrong value
 4457   for substCount.  (Christoph Kaser via Mike McCandless)
 4459 * LUCENE-6658: Fix IndexUpgrader to also upgrade indexes without any
 4460   segments.  (Trejkaz, Uwe Schindler)
 4462 * LUCENE-6677: QueryParserBase fails to enforce maxDeterminizedStates when
 4463   creating a WildcardQuery (David Causse via Mike McCandless)
 4465 * LUCENE-6680: Preserve two suggestions that have same key and weight but
 4466   different payloads (Arcadius Ahouansou via Mike McCandless)
 4468 * LUCENE-6681: SortingMergePolicy must override MergePolicy.size(...).
 4469   (Christine Poerschke via Adrien Grand)
 4471 * LUCENE-6682: StandardTokenizer performance bug: scanner buffer is
 4472   unnecessarily copied when maxTokenLength doesn't change.  Also stop silently
 4473   maxing out buffer size (and effectively also max token length) at 1M chars,
 4474   but instead throw an exception from setMaxTokenLength() when the given
 4475   length is greater than 1M chars.  (Piotr Idzikowski, Steve Rowe)
 4477 * LUCENE-6696: Fix FilterDirectoryReader.close() to never close the
 4478   underlying reader several times. (Adrien Grand)
 4480 * LUCENE-6334: FastVectorHighlighter failed to highlight phrases across
 4481   more than one value in a multi-valued field. (Chris Earle, Nik Everett
 4482   via Mike McCandless)
 4484 * LUCENE-6704: GeoPointDistanceQuery was visiting too many term ranges,
 4485   consuming too much heap for a large radius (Nick Knize via Mike McCandless)
 4487 * SOLR-5882: fix ScoreMode.Min at ToParentBlockJoinQuery (Mikhail Khludnev)
 4489 * LUCENE-6718: JoinUtil.createJoinQuery failed to rewrite queries before
 4490   creating a Weight. (Adrien Grand)
 4492 * LUCENE-6713: TooComplexToDeterminizeException claims to be serializable
 4493   but wasn't (Simon Willnauer, Mike McCandless)
 4495 * LUCENE-6723: Fix date parsing problems in Java 9 with date formats using
 4496   English weekday/month names.  (Uwe Schindler)
 4498 * LUCENE-6618: Properly set MMapDirectory.UNMAP_SUPPORTED when it is now allowed
 4499   by security policy. (Robert Muir)
 4501 Changes in Runtime Behavior
 4503 * LUCENE-6501: The subreader structure in ParallelCompositeReader
 4504   was flattened, because the current implementation had too many
 4505   hidden bugs regarding refounting and close listeners.
 4506   If you create a new ParallelCompositeReader, it will just take
 4507   all leaves of the passed readers and form a flat structure of
 4508   ParallelLeafReaders instead of trying to assemble the original
 4509   structure of composite and leaf readers.  (Adrien Grand,
 4510   Uwe Schindler)
 4512 * LUCENE-6537: NearSpansOrdered no longer tries to minimize its
 4513   Span matches.  This means that the matching algorithm is entirely
 4514   lazy.  All spans returned by the previous implementation are still 
 4515   reported, but matching documents may now also return additional 
 4516   spans that were previously discarded in preference to shorter 
 4517   overlapping ones. (Alan Woodward, Adrien Grand, Paul Elschot)
 4519 * LUCENE-6538: Also include java.vm.version and java.runtime.version
 4520   in per-segment diagnostics (Robert Muir, Mike McCandless)
 4522 * LUCENE-6569: Optimize MultiFunction.anyExists and allExists to eliminate
 4523   excessive array creation in common 2 argument usage (Jacob Graves, hossman)
 4525 * LUCENE-2880: Span queries now score more consistently with regular queries.
 4526   (Robert Muir, Adrien Grand)
 4528 * LUCENE-6601: FilteredQuery now always rewrites to a BooleanQuery which handles
 4529   the query as a MUST clause and the filter as a FILTER clause.
 4531   guarantee anymore which iterator will be advanced first, it will depend on the
 4532   respective costs of the iterators. QUERY_FIRST_FILTER_STRATEGY and
 4533   RANDOM_ACCESS_FILTER_STRATEGY still consume the filter using its random-access
 4534   API, however the returned bits may be called on different documents compared
 4535   to before. (Adrien Grand)
 4537 * LUCENE-6542: FSDirectory's ctor now works with security policies or file systems
 4538   that restrict write access.  (Trejkaz, hossman, Uwe Schindler)
 4540 * LUCENE-6651: The default implementation of AttributeImpl#reflectWith(AttributeReflector)
 4541   now uses AccessControler#doPrivileged() to do the reflection. Please consider
 4542   implementing this method in all your custom attributes, because the method will be
 4543   made abstract in Lucene 6.  (Uwe Schindler)
 4545 * LUCENE-6639: LRUQueryCache and CachingWrapperQuery now consider a query as
 4546   "used" when the first Scorer is pulled instead of when a Scorer is pulled on
 4547   the first segment on an index. (Terry Smith, Adrien Grand)
 4549 * LUCENE-6579: IndexWriter now sacrifices (closes) itself to protect the index
 4550   when an unexpected, tragic exception strikes while merging. (Robert
 4551   Muir, Mike McCandless) 
 4553 * LUCENE-6691: SortingMergePolicy.isSorted now considers FilterLeafReader instances.
 4554   EarlyTerminatingSortingCollector.terminatedEarly accessor added.
 4555   TestEarlyTerminatingSortingCollector.testTerminatedEarly test added.
 4556   (Christine Poerschke)
 4558 * LUCENE-6609: Add getSortField impls to many subclasses of FieldCacheSource which return
 4559   the most direct SortField implementation.  In many trivial sort by ValueSource usages, this
 4560   will result in less RAM, and more precise sorting of extreme values due to no longer
 4561   converting to double. (hossman)
 4563 Optimizations
 4565 * LUCENE-6548: Some optimizations for BlockTree's intersect with very
 4566   finite automata (Mike McCandless)
 4568 * LUCENE-6585: Flatten conjunctions and conjunction approximations into
 4569   parent conjunctions. For example a sloppy phrase query of "foo bar"~5
 4570   with a filter of "baz" will internally leapfrog foo,bar,baz as one
 4571   conjunction. (Ryan Ernst, Robert Muir, Adrien Grand)
 4573 * LUCENE-6325: Reduce RAM usage of FieldInfos, and speed up lookup by
 4574   number, by using an array instead of TreeMap except in very sparse
 4575   cases (Robert Muir, Mike McCandless)
 4577 * LUCENE-6617: Reduce heap usage for small FSTs (Mike McCandless)
 4579 * LUCENE-6616: IndexWriter now lists the files in the index directory
 4580   only once on init, and IndexFileDeleter no longer suppresses
 4581   FileNotFoundException and NoSuchFileException.  This also improves
 4582   IndexFileDeleter to delete segments_N files last, so that in the
 4583   presence of a virus checker, the index is never left in a state
 4584   where an expired segments_N references non-existing files (Robert
 4585   Muir, Mike McCandless)
 4587 * LUCENE-6645: Optimized the way we merge postings lists in multi-term queries
 4588   and TermsQuery. This should especially help when there are lots of small
 4589   postings lists. (Adrien Grand, Mike McCandless)
 4591 * LUCENE-6668: Optimized storage for sorted set and sorted numeric doc values
 4592   in the case that there are few unique sets of values.
 4593   (Adrien Grand, Robert Muir)
 4595 * LUCENE-6690: Sped up MultiTermsEnum.next() on high-cardinality fields.
 4596   (Adrien Grand)
 4598 * LUCENE-6621: Removed two unused variables in analysis/stempel/src/java/org/
 4599   egothor/stemmer/Compile.java
 4600   (Rishabh Patel via Christine Poerschke)
 4602 Build
 4604 * LUCENE-6518: Don't report false thread leaks from IBM J9
 4605   ClassCache Reaper in test framework. (Dawid Weiss)
 4607 * LUCENE-6567: Simplify payload checking in SpanPayloadCheckQuery (Alan
 4608   Woodward)
 4610 * LUCENE-6568: Make rat invocation depend on ivy configuration being set up
 4611   (Ramkumar Aiyengar)
 4613 * LUCENE-6683: ivy-fail goal directs people to non-existent page
 4614   (Mike Drob via Steve Rowe)
 4616 * LUCENE-6693: Updated Groovy to 2.4.4, Pegdown to 1.5, Svnkit to 1.8.10.
 4617   Also fixed some PermGen errors while running full build caused by
 4618   these updates: Tasks are now installed from root's build.xml.
 4619   (Uwe Schindler)
 4621 * LUCENE-6741: Fix jflex files to regenerate the java files correctly.
 4622   (Uwe Schindler)
 4624 Test Framework
 4626 * LUCENE-6637: Fix FSTTester to not violate file permissions
 4627   on -Dtests.verbose=true.  (Mesbah M. Alam, Uwe Schindler)
 4629 * LUCENE-6542: LuceneTestCase now has runWithRestrictedPermissions() to run
 4630   an action with reduced permissions. This can be used to simulate special
 4631   environments (e.g., read-only dirs). If tests are running without a security
 4632   manager, an assume cancels test execution automatically.  (Uwe Schindler)
 4634 * LUCENE-6652: Removed lots of useless Byte(s)TermAttributes all over test
 4635   infrastructure.  (Uwe Schindler)
 4637 * LUCENE-6563: Improve MockFileSystemTestCase.testURI to check if a path
 4638   can be encoded according to local filesystem requirements. Otherwise
 4639   stop test execution.  (Christine Poerschke via Uwe Schindler)
 4641 Changes in Backwards Compatibility Policy
 4643 * LUCENE-6553: The iterator returned by the LeafReader.postings method now
 4644   always includes deleted docs, so you have to check for deleted documents on
 4645   top of the iterator. (Adrien Grand)
 4647 * LUCENE-6633: DuplicateFilter has been deprecated and will be removed in 6.0.
 4648   DiversifiedTopDocsCollector can be used instead with a maximum number of hits
 4649   per key equal to 1. (Adrien Grand)
 4651 * LUCENE-6653: The workflow for consuming the TermToBytesRefAttribute was changed:
 4652   getBytesRef() now does all work and is called on each token, fillBytesRef()
 4653   was removed. The implementation is free to reuse the internal BytesRef
 4654   or return a new one on each call.  (Uwe Schindler)
 4656 * LUCENE-6682: StandardTokenizer.setMaxTokenLength() now throws an exception if
 4657   a length greater than 1M chars is given.  Previously the effective max token
 4658   length (the scanner's buffer) was capped at 1M chars, but getMaxTokenLength()
 4659   incorrectly returned the previously requested length, even when it exceeded 1M.
 4660   (Piotr Idzikowski, Steve Rowe)
 4663 ======================= Lucene 5.2.1 =======================
 4665 Bug Fixes
 4667 * LUCENE-6482: Fix class loading deadlock relating to Codec initialization,
 4668   default codec and SPI discovery.  (Shikhar Bhushan, Uwe Schindler)
 4670 * LUCENE-6523: NRT readers now reflect a new commit even if there is
 4671   no change to the commit user data (Mike McCandless)
 4673 * LUCENE-6527: Queries now get a dummy Similarity when scores are not needed
 4674   in order to not load unnecessary information like norms. (Adrien Grand)
 4676 * LUCENE-6559: TimeLimitingCollector now also checks for timeout when a new
 4677   leaf reader is pulled ie. if we move from one segment to another even without
 4678   collecting a hit. (Simon Willnauer)
 4680 ======================= Lucene 5.2.0 =======================
 4682 New Features
 4684 * LUCENE-6308, LUCENE-6385, LUCENE-6391: Span queries now share 
 4685   document conjunction/intersection
 4686   code with boolean queries, and use two-phased iterators for
 4687   faster intersection by avoiding loading positions in certain cases.
 4688   (Paul Elschot, Terry Smith, Robert Muir via Mike McCandless)
 4690 * LUCENE-6393: Add two-phase support to SpanPositionCheckQuery
 4691   and its subclasses: SpanPositionRangeQuery, SpanPayloadCheckQuery, 
 4692   SpanNearPayloadCheckQuery, SpanFirstQuery. (Paul Elschot, Robert Muir)
 4694 * LUCENE-6394: Add two-phase support to SpanNotQuery and refactor
 4695   FilterSpans to just have an accept(Spans candidate) method for
 4696   subclasses. (Robert Muir)
 4698 * LUCENE-6373: SpanOrQuery shares disjunction logic with boolean
 4699   queries, and supports two-phased iterators to avoid loading
 4700   positions when possible. (Paul Elschot via Robert Muir)
 4702 * LUCENE-6352, LUCENE-6472: Added a new query time join to the join module
 4703   that uses global ordinals, which is faster for subsequent joins between
 4704   reopens. (Martijn van Groningen, Adrien Grand)
 4706 * LUCENE-5879: Added experimental auto-prefix terms to BlockTree terms
 4707   dictionary, exposed as AutoPrefixPostingsFormat (Adrien Grand,
 4708   Uwe Schindler, Robert Muir, Mike McCandless)
 4710 * LUCENE-5579: New CompositeSpatialStrategy combines speed of RPT with
 4711   accuracy of SDV. Includes optimized Intersect predicate to avoid many
 4712   geometry checks. Uses TwoPhaseIterator. (David Smiley)
 4714 * LUCENE-5989: Allow passing BytesRef to StringField to make it easier
 4715   to index arbitrary binary tokens, and change the experimental
 4716   StoredFieldVisitor.stringField API to take UTF-8 byte[] instead of
 4717   String (Mike McCandless)
 4719 * LUCENE-6389: Added ScoreMode.Min that aggregates the lowest child score
 4720   to the parent hit. (Martijn van Groningen, Adrien Grand)
 4722 * LUCENE-6423: New LimitTokenOffsetFilter that limits tokens to those before
 4723   a configured maximum start offset. (David Smiley)
 4725 * LUCENE-6422: New spatial PackedQuadPrefixTree, a generally more efficient
 4726   choice than QuadPrefixTree, especially for high precision shapes.
 4727   When used, you should typically disable RPT's pruneLeafyBranches option.
 4728   (Nick Knize, David Smiley)
 4730 * LUCENE-6451: Expressions now support bindings keys that look like
 4731   zero arg functions (Jack Conradson via Ryan Ernst)
 4733 * LUCENE-6083: Add SpanWithinQuery and SpanContainingQuery that return
 4734   spans inside of / containing another spans. (Paul Elschot via Robert Muir)
 4736 * LUCENE-6454: Added distinction between member variable and method in
 4737   expression helper VariableContext
 4738   (Jack Conradson via Ryan Ernst)
 4740 * LUCENE-6196: New Spatial "Geo3d" API with partial Spatial4j integration.
 4741   It is a set of shapes implemented using 3D planar geometry for calculating
 4742   spatial relations on the surface of a sphere. Shapes include Point, BBox,
 4743   Circle, Path (buffered line string), and Polygon.
 4744   (Karl Wright via David Smiley)
 4746 * LUCENE-6464: Add a new expert lookup method to
 4747   AnalyzingInfixSuggester to accept an arbitrary BooleanQuery to
 4748   express how contexts should be filtered. (Arcadius Ahouansou via
 4749   Mike McCandless)
 4751 Optimizations
 4753 * LUCENE-6379: IndexWriter.deleteDocuments(Query...) now detects if
 4754   one of the queries is MatchAllDocsQuery and just invokes the much
 4755   faster IndexWriter.deleteAll in that case (Robert Muir, Adrien
 4756   Grand, Mike McCandless)
 4758 * LUCENE-6388: Optimize SpanNearQuery when payloads are not present.
 4759   (Robert Muir)
 4761 * LUCENE-6421: Defer reading of positions in MultiPhraseQuery until
 4762   they are needed. (Robert Muir)
 4764 * LUCENE-6392: Highligher- reduce memory of tokens in
 4765   TokenStreamFromTermVector, and add maxStartOffset limit. (David Smiley)
 4767 * LUCENE-6456: Queries that generate doc id sets that are too large for the
 4768   query cache are not cached instead of evicting everything. (Adrien Grand)
 4770 * LUCENE-6455: Require a minimum index size to enable query caching in order
 4771   not to cache eg. on MemoryIndex. (Adrien Grand)
 4773 * LUCENE-6330: BooleanScorer (used for top-level disjunctions) does not decode
 4774   norms when not necessary anymore. (Adrien Grand)
 4776 * LUCENE-6350: TermsQuery is now compressed with PrefixCodedTerms.
 4777   (Robert Muir, Mike McCandless, Adrien Grand)
 4779 * LUCENE-6458: Multi-term queries matching few terms per segment now execute
 4780   like a disjunction. (Adrien Grand)
 4782 * LUCENE-6360: TermsQuery rewrites to a disjunction when there are 16 matching
 4783   terms or less. (Adrien Grand)
 4785 Bug Fixes
 4787 * LUCENE-329: Fix FuzzyQuery defaults to rank exact matches highest.
 4788   (Mark Harwood, Adrien Grand)
 4790 * LUCENE-6378: Fix all RuntimeExceptions to throw the underlying root cause.
 4791   (Varun Thacker, Adrien Grand, Mike McCandless)
 4793 * LUCENE-6415: TermsQuery.extractTerms is a no-op (used to throw an
 4794   UnsupportedOperationException). (Adrien Grand)
 4796 * LUCENE-6416: BooleanQuery.extractTerms now only extracts terms from scoring
 4797   clauses. (Adrien Grand)
 4799 * LUCENE-6409: Fixed integer overflow in LongBitSet.ensureCapacity.
 4800   (Luc Vanlerberghe via Adrien Grand)
 4802 * LUCENE-6424, LUCENE-6430: Fix many bugs with mockfs filesystems in the 
 4803   test-framework: always consistently wrap Path, fix buggy behavior for 
 4804   globs, implement equals/hashcode for filtered Paths, etc.  
 4805   (Ryan Ernst, Simon Willnauer, Robert Muir)
 4807 * LUCENE-6426: Fix FieldType's copy constructor to also copy over the numeric
 4808   precision step. (Adrien Grand)
 4810 * LUCENE-6345: Null check terms/fields in Lucene queries (Lee
 4811   Hinman via Mike McCandless)
 4813 * LUCENE-6400: SolrSynonymParser should preserve original token instead
 4814   of replacing it with a synonym, when expand=true and there is no
 4815   explicit mapping (Ian Ribas, Robert Muir, Mike McCandless)
 4817 * LUCENE-6449: Don't throw NullPointerException if some segments are
 4818   missing the field being highlighted, in PostingsHighlighter (Roman
 4819   Khmelichek via Mike McCandless)
 4821 * LUCENE-6427: Added assertion about the presence of ghost bits in
 4822   (Fixed|Long)BitSet. (Luc Vanlerberghe via Adrien Grand)
 4824 * LUCENE-6468: Fixed NPE with empty Kuromoji user dictionary.
 4825   (Jun Ohtani via Christian Moen)
 4827 * LUCENE-6483: Ensure core closed listeners are called on the same cache key as
 4828   the reader which has been used to register the listener. (Adrien Grand)
 4830 * LUCENE-6486 DocumentDictionary iterator no longer skips
 4831   documents with no payloads and now returns an empty BytesRef instead
 4832   (Marius Grama via Michael McCandless)
 4834 * LUCENE-6505: NRT readers now reflect segments_N filename and commit
 4835   user data from previous commits (Mike McCandless)
 4837 * LUCENE-6507: Don't let NativeFSLock.close() release other locks
 4838   (Simon Willnauer, Robert Muir, Uwe Schindler, Mike McCandless)
 4840 API Changes
 4842 * LUCENE-6377: SearcherFactory#newSearcher now accepts the previous reader
 4843   to simplify warming logic during opening new searchers. (Simon Willnauer)
 4845 * LUCENE-6410: Removed unused "reuse" parameter to
 4846   Terms.iterator. (Robert Muir, Mike McCandless)
 4848 * LUCENE-6425: Replaced Query.extractTerms with Weight.extractTerms.
 4849   (Adrien Grand)
 4851 * LUCENE-6446: Simplified Explanation API. (Adrien Grand)
 4853 * LUCENE-6445: Two new methods in Highlighter's TokenSources; the existing
 4854   methods are now marked deprecated. (David Smiley)
 4856 * LUCENE-6484: Removed EliasFanoDocIdSet, which was unused.
 4857   (Paul Elschot via Adrien Grand)
 4859 * LUCENE-6466: Moved SpanQuery.getSpans() and .extractTerms() to SpanWeight
 4860   (Alan Woodward, Robert Muir)
 4862 * LUCENE-6497: Allow subclasses of FieldType to check frozen state
 4863   (Ryan Ernst)
 4865 Other
 4867 * LUCENE-6413: Test runner should report the number of suites completed/ 
 4868   remaining. (Dawid Weiss)
 4870 * LUCENE-5439: Add 'ant jacoco' build target. (Robert Muir)
 4872 * LUCENE-6315: Simplify the private iterator Lucene uses internally
 4873   when resolving deleted terms to matched docids. (Robert Muir, Adrien
 4874   Grand, Mike McCandless)
 4876 * LUCENE-6399: Benchmark module's QueryMaker.resetInputs should call setConfig
 4877   so queries can react to property changes in new rounds. (David Smiley)
 4879 * LUCENE-6382: Lucene now enforces that positions never exceed the
 4880   maximum value IndexWriter.MAX_POSITION.  (Robert Muir, Mike McCandless)
 4882 * LUCENE-6372: Simplified and improved equals/hashcode of span queries.
 4883   (Paul Elschot via Adrien Grand)
 4885 Build
 4887 * LUCENE-6420: Update forbiddenapis to v1.8  (Uwe Schindler)
 4889 Test Framework
 4891 * LUCENE-6419: Added two-phase iteration assertions to AssertingQuery.
 4892   (Adrien Grand)
 4894 * LUCENE-6437: Randomly set CPU core count and spins, derived from
 4895   test's master seed, used by ConcurrentMergeScheduler to set dynamic
 4896   defaults, for better test randomization and to help tests reproduce
 4897   (Robert Muir, Mike McCandless)
 4899 ======================= Lucene 5.1.0 =======================
 4901 New Features
 4903 * LUCENE-6066: Added DiversifiedTopDocsCollector to misc for collecting no more 
 4904   than a given number of results under a choice of key. Introduces new remove 
 4905   method to core's PriorityQueue. (Mark Harwood)
 4907 * LUCENE-6191: New spatial 2D heatmap faceting for PrefixTreeStrategy. (David Smiley)
 4909 * LUCENE-6227: Added BooleanClause.Occur.FILTER to filter documents without
 4910   participating in scoring (on the contrary to MUST). (Adrien Grand)
 4912 * LUCENE-6294: Added oal.search.CollectorManager to allow for parallelization
 4913   of the document collection process on IndexSearcher. (Adrien Grand)
 4915 * LUCENE-6303: Added filter caching baked into IndexSearcher, disabled by
 4916   default. (Adrien Grand)
 4918 * LUCENE-6304: Added a new MatchNoDocsQuery that matches no documents.
 4919   (Lee Hinman via Adrien Grand)
 4921 * LUCENE-6341: Add a -fast option to CheckIndex. (Robert Muir)
 4923 * LUCENE-6355: IndexWriter's infoStream now also logs time to write FieldInfos
 4924   during merge (Lee Hinman via Mike McCandless)
 4926 * LUCENE-6339: Added Near-real time Document Suggester via custom postings format
 4927   (Areek Zillur, Mike McCandless, Simon Willnauer)
 4929 Bug Fixes
 4931 * LUCENE-6368: FST.save can truncate output (BufferedOutputStream may be closed 
 4932   after the underlying stream). (Ippei Matsushima via Dawid Weiss)
 4934 * LUCENE-6249: StandardQueryParser doesn't support pure negative clauses. 
 4935   (Dawid Weiss)
 4937 * LUCENE-6190: Spatial pointsOnly flag on PrefixTreeStrategy shouldn't switch all predicates to
 4938   Intersects. (David Smiley)
 4940 * LUCENE-6242: Ram usage estimation was incorrect for SparseFixedBitSet when
 4941   object alignment was different from 8. (Uwe Schindler, Adrien Grand)
 4943 * LUCENE-6293: Fixed TimSorter bug. (Adrien Grand)
 4945 * LUCENE-6001: DrillSideways hits NullPointerException for certain
 4946   BooleanQuery searches.  (Dragan Jotannovic, jane chang via Mike
 4947   McCandless)
 4949 * LUCENE-6311: Fix NIOFSDirectory and SimpleFSDirectory so that the
 4950   toString method of IndexInputs confess when they are from a compound
 4951   file. (Robert Muir, Mike McCandless)
 4953 * LUCENE-6381: Add defensive wait time limit in
 4954   DocumentsWriterStallControl to prevent hangs during indexing if we
 4955   miss a .notify/All somewhere (Mike McCandless)
 4957 * LUCENE-6386: Correct IndexWriter.forceMerge documentation to state
 4958   that up to 3X (X = current index size) spare disk space may be needed
 4959   to complete forceMerge(1).  (Robert Muir, Shai Erera, Mike McCandless)
 4961 * LUCENE-6395: Seeking by term ordinal was failing to set the term's
 4962   bytes in MemoryIndex (Mike McCandless)
 4964 * LUCENE-6429: Removed the TermQuery(Term,int) constructor which could lead to
 4965   inconsistent term statistics. (Adrien Grand, Robert Muir)
 4967 Optimizations
 4969 * LUCENE-6183, LUCENE-5647: Avoid recompressing stored fields
 4970   and term vectors when merging segments without deletions. 
 4971   Lucene50Codec's BEST_COMPRESSION mode uses a higher deflate 
 4972   level for more compact storage.  (Robert Muir)
 4974 * LUCENE-6184: Make BooleanScorer only score windows that contain
 4975   matches. (Adrien Grand)
 4977 * LUCENE-6161: Speed up resolving of deleted terms to docIDs by doing
 4978   a combined merge sort between deleted terms and segment terms
 4979   instead of a separate merge sort for each segment.  In delete-heavy
 4980   use cases this can be a sizable speedup. (Mike McCandless)
 4982 * LUCENE-6201: BooleanScorer can now deal with values of minShouldMatch that
 4983   are greater than one and is used when queries produce dense result sets.
 4984   (Adrien Grand)
 4986 * LUCENE-6218: Don't decode frequencies or match all positions when scoring
 4987   is not needed. (Robert Muir)
 4989 * LUCENE-6233 Speed up CheckIndex when the index has term vectors
 4990   (Robert Muir, Mike McCandless)
 4992 * LUCENE-6198: Added the TwoPhaseIterator API, exposed on scorers which
 4993   is for now only used on phrase queries and conjunctions in order to check
 4994   positions lazily if the phrase query is in a conjunction with other queries.
 4995   (Robert Muir, Adrien Grand, David Smiley)
 4997 * LUCENE-6244, LUCENE-6251: All boolean queries but those that have a
 4998   minShouldMatch > 1 now either propagate or take advantage of the two-phase
 4999   iteration capabilities added in LUCENE-6198. (Adrien Grand, Robert Muir)
 5001 * LUCENE-6241: FSDirectory.listAll() doesnt filter out subdirectories anymore,
 5002   for faster performance. Subdirectories don't matter to Lucene. If you need to
 5003   filter out non-index files with some custom usage, you may want to look at 
 5004   the IndexFileNames class. (Robert Muir)
 5006 * LUCENE-6262: ConstantScoreQuery does not wrap the inner weight anymore when
 5007   scores are not required. (Adrien Grand)
 5009 * LUCENE-6263: MultiCollector automatically caches scores when several
 5010   collectors need them. (Adrien Grand)
 5012 * LUCENE-6275: SloppyPhraseScorer now uses the same logic as ConjunctionScorer
 5013   in order to advance doc IDs, which takes advantage of the cost() API.
 5014   (Adrien Grand)
 5016 * LUCENE-6290: QueryWrapperFilter propagates approximations and FilteredQuery
 5017   rewrites to a BooleanQuery when the filter is a QueryWrapperFilter in order
 5018   to leverage approximations. (Adrien Grand)
 5020 * LUCENE-6318: Reduce RAM usage of FieldInfos when there are many fields.
 5021   (Mike McCandless, Robert Muir)
 5023 * LUCENE-6320: Speed up CheckIndex. (Robert Muir)
 5025 * LUCENE-4942: Optimized the encoding of PrefixTreeStrategy indexes for
 5026   non-point data: 33% smaller index, 68% faster indexing, and 44% faster
 5027   searching. YMMV (David Smiley)
 5029 API Changes
 5031 * LUCENE-6204, LUCENE-6208: Simplify CompoundFormat: remove files()
 5032   and remove files parameter to write(). (Robert Muir)
 5034 * LUCENE-6217: Add IndexWriter.isOpen and getTragicException.  (Simon
 5035   Willnauer, Mike McCandless)
 5037 * LUCENE-6218, LUCENE-6220: Add Collector.needsScores() and needsScores
 5038   parameter to Query.createWeight(). (Robert Muir, Adrien Grand)
 5040 * LUCENE-4524, LUCENE-6246, LUCENE-6256, LUCENE-6271: Merge DocsEnum and DocsAndPositionsEnum
 5041   into a single PostingsEnum iterator.  TermsEnum.docs() and TermsEnum.docsAndPositions()
 5042   are replaced by TermsEnum.postings(). 
 5043   (Alan Woodward, Simon Willnauer, Robert Muir, Ryan Ernst)
 5045 * LUCENE-6222: Removed TermFilter, use a QueryWrapperFilter(TermQuery)
 5046   instead. This will be as efficient now that queries can opt out from
 5047   scoring. (Adrien Grand)
 5049 * LUCENE-6269: Removed BooleanFilter, use a QueryWrapperFilter(BooleanQuery)
 5050   instead. (Adrien Grand)
 5052 * LUCENE-6270: Replaced TermsFilter with TermsQuery, use a
 5053   QueryWrapperFilter(TermsQuery) instead. (Adrien Grand)
 5055 * LUCENE-6223: Move BooleanQuery.BooleanWeight to BooleanWeight.
 5056   (Robert Muir)
 5058 * LUCENE-1518: Make Filter extend Query and return 0 as score.
 5059   (Uwe Schindler, Adrien Grand)
 5061 * LUCENE-6245: Force Filter subclasses to implement toString API from Query.
 5062   (Ryan Ernst)
 5064 * LUCENE-6268: Replace FieldValueFilter and DocValuesRangeFilter with equivalent
 5065   queries that support approximations. (Adrien Grand)
 5067 * LUCENE-6289: Replace DocValuesRangeFilter with DocValuesRangeQuery which
 5068   supports approximations. (Adrien Grand)
 5070 * LUCENE-6266: Remove unnecessary Directory params from SegmentInfo.toString, 
 5071   SegmentInfos.files/toString, and SegmentCommitInfo.toString. (Robert Muir)
 5073 * LUCENE-6272: Scorer extends DocSetIdIterator rather than DocsEnum (Alan
 5074   Woodward)
 5076 * LUCENE-6281: Removed support for slow collations from lucene/sandbox. Better
 5077   performance would be achieved through CollationKeyAnalyzer or
 5078   ICUCollationKeyAnalyzer. (Adrien Grand)
 5080 * LUCENE-6286: Removed IndexSearcher methods that take a Filter object.
 5081   A BooleanQuery with a filter clause must be used instead. (Adrien Grand)
 5083 * LUCENE-6300: PrefixFilter, TermRangeFilter and NumericRangeFilter have been
 5084   removed. Use PrefixQuery, TermRangeQuery and NumericRangeQuery instead.
 5085   (Adrien Grand)
 5087 * LUCENE-6303: Replaced FilterCache with QueryCache and CachingWrapperFilter
 5088   with CachingWrapperQuery. (Adrien Grand)
 5090 * LUCENE-6317: Deprecate DataOutput.writeStringSet and writeStringStringMap.
 5091   Use writeSetOfStrings/Maps instead. (Mike McCandless, Robert Muir)
 5093 * LUCENE-6307: Rename SegmentInfo.getDocCount -> .maxDoc,
 5094   SegmentInfos.totalDocCount -> .totalMaxDoc, MergeInfo.totalDocCount
 5095   -> .totalMaxDoc and MergePolicy.OneMerge.totalDocCount ->
 5096   .totalMaxDoc (Adrien Grand, Robert Muir, Mike McCandless)
 5098 * LUCENE-6367: PrefixQuery now subclasses AutomatonQuery, removing the
 5099   specialized PrefixTermsEnum.  (Robert Muir, Mike McCandless)
 5101 Other
 5103 * LUCENE-6248: Remove unused odd constants from StandardSyntaxParser.jj
 5104   (Dawid Weiss)
 5106 * LUCENE-6193: Collapse identical catch branches in try-catch statements.
 5107   (shalin)
 5109 * LUCENE-6239: Removed RAMUsageEstimator's sun.misc.Unsafe calls.
 5110   (Robert Muir, Dawid Weiss, Uwe Schindler)
 5112 * LUCENE-6292: Seed StringHelper better. (Robert Muir)
 5114 * LUCENE-6333: Refactored queries to delegate their equals and hashcode
 5115   impls to the super class. (Lee Hinman via Adrien Grand)
 5117 * LUCENE-6343: DefaultSimilarity javadocs had the wrong float value to
 5118   demonstrate precision of encoded norms (András Péteri via Mike McCandless)
 5120 Changes in Runtime Behavior
 5122 * LUCENE-6255: PhraseQuery now ignores leading holes and requires that
 5123   positions are positive and added in order. (Adrien Grand)
 5125 * LUCENE-6298: SimpleQueryParser returns an empty query rather than
 5126   null, if e.g. the terms were all stopwords. (Lee Hinman via Robert Muir)
 5128 ======================= Lucene 5.0.0 =======================
 5130 New Features
 5132 * LUCENE-5945: All file handling converted to NIO.2 apis. (Robert Muir)
 5134 * LUCENE-5946: SimpleFSDirectory now uses Files.newByteChannel, for 
 5135   portability with custom FileSystemProviders. If you want the old
 5136   non-interruptible behavior of RandomAccessFile, use RAFDirectory
 5137   in the misc/ module. (Uwe Schindler, Robert Muir)
 5139 * SOLR-3359: Added analyzer attribute/property to SynonymFilterFactory.
 5140   (Ryo Onodera via Koji Sekiguchi)
 5142 * LUCENE-5648: Index and search date ranges, particularly multi-valued ones. It's
 5143   implemented in the spatial module as DateRangePrefixTree used with
 5144   NumberRangePrefixTreeStrategy. (David Smiley)
 5146 * LUCENE-5895: Lucene now stores a unique id per-segment and per-commit to aid
 5147   in accurate replication of index files (Robert Muir, Mike McCandless)
 5149 * LUCENE-5889: Add commit method to AnalyzingInfixSuggester, and allow just using .add
 5150   to build up the suggester.  (Varun Thacker via Mike McCandless)
 5152 * LUCENE-5123: Add a "pull" option to the postings writing API, so
 5153   that a PostingsFormat now receives a Fields instance and it is
 5154   responsible for iterating through all fields, terms, documents and
 5155   positions.  (Robert Muir, Mike McCandless)
 5157 * LUCENE-5268: Full cutover of all postings formats to the "pull"
 5158   FieldsConsumer API, removing PushFieldsConsumer.  Added new
 5159   PushPostingsWriterBase for single-pass push of docs/positions to the
 5160   postings format.  (Mike McCandless)
 5162 * LUCENE-5906: Use Files.delete everywhere instead of File.delete, so that
 5163   when things go wrong, you get a real exception message why.
 5164   (Uwe Schindler, Robert Muir)
 5166 * LUCENE-5933: Added FilterSpans for easier wrapping of Spans instance. (Shai Erera)
 5168 * LUCENE-5925: Remove fallback logic from opening commits, instead use
 5169   Directory.renameFile so that in-progress commits are never visible.
 5170   (Robert Muir)
 5172 * LUCENE-5820: SuggestStopFilter should have a factory.
 5173   (Varun Thacker via Steve Rowe)
 5175 * LUCENE-5949: Add Accountable.getChildResources(). (Robert Muir)
 5177 * SOLR-5986: Added ExitableDirectoryReader that extends FilterDirectoryReader and enables
 5178   exiting requests that take too long to enumerate over terms. (Anshum Gupta, Steve Rowe,
 5179   Robert Muir)
 5181 * LUCENE-5911: Add MemoryIndex.freeze() to allow thread-safe searching over a 
 5182   MemoryIndex. (Alan Woodward, David Smiley, Robert Muir)
 5184 * LUCENE-5969: Lucene 5.0 has a new index format with mismatched file detection,
 5185   improved exception handling, and indirect norms encoding for sparse fields.
 5186   (Mike McCandless, Ryan Ernst, Robert Muir)
 5188 * LUCENE-6053: Add Serbian analyzer.  (Nikola Smolenski via Robert Muir, Mike McCandless)
 5190 * LUCENE-4400: Add support for new NYSIIS Apache commons phonetic
 5191   codec (Thomas Neidhart via Mike McCandless)
 5193 * LUCENE-6059: Add Daitch-Mokotoff Soundex phonetic Apache commons
 5194   phonetic codec, and upgrade to Apache commons codec 1.10. (Thomas
 5195   Neidhart via Mike McCandless)
 5197 * LUCENE-6058: With the upgrade to Apache commons codec 1.10, the
 5198   experimental BeiderMorseFilter has changed its behavior, so any
 5199   index using it will need to be rebuilt.  (Thomas
 5200   Neidhart via Mike McCandless)
 5202 * LUCENE-6050: Accept MUST and MUST_NOT (in addition to SHOULD) for
 5203   each context passed to Analyzing/BlendedInfixSuggester (Arcadius
 5204   Ahouansou, jane chang via Mike McCandless)
 5206 * LUCENE-5929: Also extract terms to highlight from block join
 5207   queries. (Julie Tibshirani via Mike McCandless)
 5209 * LUCENE-6063: Allow overriding whether/how ConcurrentMergeScheduler
 5210   stalls incoming threads when merges are falling behind (Mike
 5211   McCandless)
 5213 * LUCENE-5833: DocumentDictionary now enumerates each value separately
 5214   in a multi-valued field (not just the first value), so you can build
 5215   suggesters from multi-valued fields.  (Varun Thacker via Mike
 5216   McCandless)
 5218 * LUCENE-6077: Added a filter cache. (Adrien Grand, Robert Muir)
 5220 * LUCENE-6088: TermsFilter implements Accountable. (Adrien Grand)
 5222 * LUCENE-6034: The default highlighter when used with QueryScorer will highlight payload-sensitive
 5223   queries provided that term vectors with positions, offsets, and payloads are present. This is the
 5224   only highlighter that can highlight such queries accurately. (David Smiley)
 5226 * LUCENE-5914: Add an option to Lucene50Codec to support either BEST_SPEED
 5227   or BEST_COMPRESSION for stored fields. (Adrien Grand, Robert Muir)
 5229 * LUCENE-6119: Add auto-IO-throttling to ConcurrentMergeScheduler, to
 5230   rate limit IO writes for each merge depending on incoming merge
 5231   rate.  (Mike McCandless)
 5233 * LUCENE-6155: Add payload support to MemoryIndex. The default highlighter's
 5234   QueryScorer and WeighedSpanTermExtractor now have setUsePayloads(bool).
 5235   (David Smiley)
 5237 * LUCENE-6166: Deletions (alone) can now trigger new merges.  (Mike McCandless)
 5239 * LUCENE-6177: Add CustomAnalyzer that allows to configure analyzers
 5240   like you do in Solr's index schema. This class has a builder API to configure
 5241   Tokenizers, TokenFilters, and CharFilters based on their SPI names
 5242   and parameters as documented by the corresponding factories.
 5243   (Uwe Schindler)
 5245 Optimizations
 5247 * LUCENE-5960: Use a more efficient bitset, not a Set<Integer>, to
 5248   track visited states.  (Markus Heiden via Mike McCandless)
 5250 * LUCENE-5959: Don't allocate excess memory when building automaton in
 5251   finish. (Markus Heiden via Mike McCandless)
 5253 * LUCENE-5963: Reduce memory allocations in
 5254   AnalyzingSuggester. (Markus Heiden via Mike McCandless)
 5256 * LUCENE-5938: MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE is now faster on
 5257   queries that match few documents by using a sparse bit set implementation.
 5258   (Adrien Grand)
 5260 * LUCENE-5969: Refactor merging to be more efficient, checksum calculation is
 5261   per-segment/per-producer, and norms and doc values merging no longer cause 
 5262   RAM spikes for latent fields. (Mike McCandless, Robert Muir)
 5264 * LUCENE-5983: CachingWrapperFilter now uses a new DocIdSet implementation
 5265   called RoaringDocIdSet instead of WAH8DocIdSet. (Adrien Grand)
 5267 * LUCENE-6022: DocValuesDocIdSet checks live docs before doc values.
 5268   (Adrien Grand)
 5270 * LUCENE-6030: Add norms patched compression for a small number of common values
 5271   (Ryan Ernst)
 5273 * LUCENE-6040: Speed up EliasFanoDocIdSet through broadword bit selection.
 5274   (Paul Elschot)
 5276 * LUCENE-6033: CachingTokenFilter now uses ArrayList not LinkedList, and has new
 5277   isCached() method. (David Smiley)
 5279 * LUCENE-6031: TokenSources (in the default highlighter) converts term vectors into a
 5280   TokenStream much faster in linear time (not N*log(N) using less memory, and with reset()
 5281   implemented.  Only one of offsets or positions are required of the term vector.
 5282   (David Smiley)
 5284 * LUCENE-6089, LUCENE-6090: Tune CompressionMode.HIGH_COMPRESSION for 
 5285   better compression and less cpu usage. (Adrien Grand, Robert Muir)
 5287 * LUCENE-6034: QueryScorer, used by the default highlighter, needn't re-index the provided
 5288   TokenStream with MemoryIndex when it comes from TokenSources (term vectors) with offsets and
 5289   positions. (David Smiley)
 5291 * LUCENE-5951: ConcurrentMergeScheduler detects whether the index is on SSD or not
 5292   and does a better job defaulting its settings.  This only works on Linux for now;
 5293   other OS's will continue to use the previous defaults (tuned for spinning disks).
 5294   (Robert Muir, Uwe Schindler, hossman, Mike McCandless)
 5296 * LUCENE-6131: Optimize SortingMergePolicy. (Robert Muir)
 5298 * LUCENE-6133: Improve default StoredFieldsWriter.merge() to be more efficient. 
 5299   (Robert Muir)
 5301 * LUCENE-6145: Make EarlyTerminatingSortingCollector able to early-terminate
 5302   when the sort order is a prefix of the index-time order. (Adrien Grand)
 5304 * LUCENE-6178: Score boolean queries containing MUST_NOT clauses with BooleanScorer2,
 5305   to use skip list data and avoid unnecessary scoring. (Adrien Grand, Robert Muir)
 5307 API Changes
 5309 * LUCENE-5900: Deprecated more constructors taking Version in *InfixSuggester and
 5310   ICUCollationKeyAnalyzer, and removed TEST_VERSION_CURRENT from the test framework.
 5311   (Ryan Ernst)
 5313 * LUCENE-4535: oal.util.FilterIterator is now an internal API.
 5314   (Adrien Grand)
 5316 * LUCENE-4924: DocIdSetIterator.docID() must now return -1 when the iterator is
 5317   not positioned. This change affects all classes that inherit from
 5318   DocIdSetIterator, including DocsEnum and DocsAndPositionsEnum. (Adrien Grand)
 5320 * LUCENE-5127: Reduce RAM usage of FixedGapTermsIndex. Remove 
 5321   IndexWriterConfig.setTermIndexInterval, IndexWriterConfig.setReaderTermsIndexDivisor,
 5322   and termsIndexDivisor from StandardDirectoryReader. These options have been no-ops
 5323   with the default codec since Lucene 4.0. If you want to configure the interval for
 5324   this term index, pass it directly in your codec, where it can also be configured
 5325   per-field. (Robert Muir)
 5327 * LUCENE-5388: Remove Reader from Tokenizer's constructor and from
 5328   Analyzer's createComponents. TokenStreams now always get their input
 5329   via setReader.  
 5330   (Benson Margulies via Robert Muir - pull request #16)
 5332 * LUCENE-5527: The Collector API has been refactored to use a dedicated Collector
 5333   per leaf. (Shikhar Bhushan, Adrien Grand)
 5335 * LUCENE-5702: The FieldComparator API has been refactor to a per-leaf API, just
 5336   like Collectors. (Adrien Grand)
 5338 * LUCENE-4246: IndexWriter.close now always closes, even if it throws
 5339   an exception.  The new IndexWriterConfig.setCommitOnClose (default
 5340   true) determines whether close() should commit before closing.
 5342 * LUCENE-5608, LUCENE-5565: Refactor SpatialPrefixTree/Cell API. Doesn't use Strings
 5343   as tokens anymore, and now iterates cells on-demand during indexing instead of
 5344   building a collection.  RPT now has more setters. (David Smiley)
 5346 * LUCENE-5666: Change uninverted access (sorting, faceting, grouping, etc)
 5347   to use the DocValues API instead of FieldCache. For FieldCache functionality,
 5348   use UninvertingReader in lucene/misc (or implement your own FilterReader).
 5349   UninvertingReader is more efficient: supports multi-valued numeric fields,
 5350   detects when a multi-valued field is single-valued, reuses caches
 5351   of compatible types (e.g. SORTED also supports BINARY and SORTED_SET access
 5352   without insanity).  "Insanity" is no longer possible unless you explicitly want it. 
 5353   Rename FieldCache* and DocTermOrds* classes in the search package to DocValues*. 
 5354   Move SortedSetSortField to core and add SortedSetFieldSource to queries/, which
 5355   takes the same selectors. Add helper methods to DocValues.java that are better 
 5356   suited for search code (never return null, etc).  (Mike McCandless, Robert Muir)
 5358 * LUCENE-5871: Remove Version from IndexWriterConfig. Use
 5359   IndexWriterConfig.setCommitOnClose to change the behavior of IndexWriter.close().
 5360   The default has been changed to match that of 4.x.
 5361   (Ryan Ernst, Mike McCandless)
 5363 * LUCENE-5965: CorruptIndexException requires a String or DataInput resource.
 5364   (Robert Muir)
 5366 * LUCENE-5972: IndexFormatTooOldException and IndexFormatTooNewException now
 5367                extend from IOException.
 5368   (Ryan Ernst, Robert Muir)
 5370 * LUCENE-5569: *AtomicReader/AtomicReaderContext have been renamed to *LeafReader/LeafReaderContext.
 5371   (Ryan Ernst)
 5373 * LUCENE-5938: Removed MultiTermQuery.ConstantScoreAutoRewrite as
 5374   MultiTermQuery.CONSTANT_SCORE_FILTER_REWRITE is usually better. (Adrien Grand)
 5376 * LUCENE-5924: Rename CheckIndex -fix option to -exorcise. This option does not
 5377   actually fix the index, it just drops data.  (Robert Muir)
 5379 * LUCENE-5969: Add Codec.compoundFormat, which handles the encoding of compound 
 5380   files. Add getMergeInstance() to codec producer APIs, which can be overridden
 5381   to return an instance optimized for merging instead of searching. Add 
 5382   Terms.getStats() which can return additional codec-specific statistics about a field.
 5383   Change instance method SegmentInfos.read() to two static methods: SegmentInfos.readCommit()
 5384   and SegmentInfos.readLatestCommit().
 5385   (Mike McCandless, Robert Muir)
 5387 * LUCENE-5992: Remove FieldInfos from SegmentInfosWriter.write API. (Robert Muir, Mike McCandless)
 5389 * LUCENE-5998: Simplify Field/SegmentInfoFormat to read+write methods.
 5390   (Robert Muir)
 5392 * LUCENE-6000: Removed StandardTokenizerInterface.  Tokenizers now use
 5393   their jflex impl directly.
 5394   (Ryan Ernst)
 5396 * LUCENE-6006: Removed FieldInfo.normType since it's redundant: it
 5397   will be DocValuesType.NUMERIC if the field indexed and does not omit
 5398   norms, else null.  (Robert Muir, Mike McCandless)
 5400 * LUCENE-6013: Removed indexed boolean from IndexableFieldType and
 5401   FieldInfo, since it's redundant with IndexOptions != null. (Robert
 5402   Muir, Mike McCandless)
 5404 * LUCENE-6021: FixedBitSet.nextSetBit now returns DocIdSetIterator.NO_MORE_DOCS
 5405   instead of -1 when there are no more bits which are set. (Adrien Grand)
 5407 * LUCENE-5953: Directory and LockFactory APIs were restructured: Locking is
 5408   now under the responsibility of the Directory implementation. LockFactory is
 5409   only used by subclasses of BaseDirectory to delegate locking to an impl
 5410   class. LockFactories are now singletons and are responsible to create a Lock
 5411   instance based on a Directory implementation passed to the factory method.
 5412   See MIGRATE.txt for more details.  (Uwe Schindler, Robert Muir)
 5414 * LUCENE-6062: Throw exception instead of silently doing nothing if you try to 
 5415   sort/group/etc on a misconfigured field (e.g. no docvalues, no UninvertingReader, etc).  
 5416   (Robert Muir)
 5418 * LUCENE-6068: LeafReader.fields() never returns null. (Robert Muir)
 5420 * LUCENE-6082: Remove abort() from codec apis. (Robert Muir)
 5422 * LUCENE-6084: IndexOutput's constructor now requires a String
 5423   resourceDescription so its toString is sane (Robert Muir, Mike
 5424   McCandless)
 5426 * LUCENE-6087: Allow passing custom DirectoryReader to SearcherManager
 5427   (Mike McCandless)
 5429 * LUCENE-6085: Undeprecate SegmentInfo attributes, but add safety so they
 5430   won't be trappy if codec tries to use them during docvalues updates.
 5431   (Robert Muir)
 5433 * LUCENE-6097: Remove dangerous / overly expert
 5434   IndexWriter.abortMerges and waitForMerges methods.  (Robert Muir,
 5435   Mike McCandless)
 5437 * LUCENE-6099: Add FilterDirectory.unwrap and
 5438   FilterDirectoryReader.unwrap (Simon Willnauer, Mike McCandless)
 5440 * LUCENE-6121: CachingTokenFilter.reset() now propagates to its input if called before
 5441   incrementToken().  You must call reset() now on this filter instead of doing it a-priori on the
 5442   input(), which previously didn't work.  (David Smiley, Robert Muir)
 5444 * LUCENE-6147: Make the core Accountables.namedAccountable function public
 5445   (Ryan Ernst)
 5447 * LUCENE-6150: Remove staleFiles set and onIndexOutputClosed() from FSDirectory.
 5448   (Uwe Schindler, Robert Muir, Mike McCandless)
 5450 * LUCENE-6146: Replaced Directory.copy() with Directory.copyFrom().
 5451   (Robert Muir)
 5453 * LUCENE-6149: Infix suggesters' highlighting and allTermsRequired can
 5454   be set at the constructor for non-contextual lookup.
 5455   (Boon Low, Tomás Fernández Löbbe)
 5457 * LUCENE-6158, LUCENE-6165: IndexWriter.addIndexes(IndexReader...) changed to
 5458   addIndexes(CodecReader...) (Robert Muir)
 5460 * LUCENE-6179: Out-of-order scoring is not allowed anymore, so
 5461   Weight.scoresDocsOutOfOrder and LeafCollector.acceptsDocsOutOfOrder have been
 5462   removed and boolean queries now always score in order.
 5464 * LUCENE-6212: IndexWriter no longer accepts per-document Analyzer to
 5465   add/updateDocument.  These methods were trappy as they made it
 5466   easy to accidentally index tokens that were not easily
 5467   searchable. (Mike McCandless)
 5469 Bug Fixes
 5471 * LUCENE-5650: Enforce read-only access to any path outside the temporary
 5472   folder via security manager, and make test temp dirs absolute.
 5473   (Ryan Ernst, Dawid Weiss)
 5475 * LUCENE-5948: RateLimiter now fully inits itself on init.  (Varun
 5476   Thacker via Mike McCandless)
 5478 * LUCENE-5981: CheckIndex obtains write.lock, since with some parameters it
 5479   may modify the index, and to prevent false corruption reports, as it does
 5480   not have the regular "spinlock" of DirectoryReader.open. It now implements
 5481   Closeable and you must close it to release the lock.  (Mike McCandless, Robert Muir)
 5483 * LUCENE-6004: Don't highlight the LookupResult.key returned from
 5484   AnalyzingInfixSuggester (Christian Reuschling, jane chang via Mike McCandless)
 5486 * LUCENE-5980: Don't let document length overflow. (Robert Muir)
 5488 * LUCENE-5961: Fix the exists() method for FunctionValues returned by many ValueSources to
 5489   behave properly when wrapping other ValueSources which do not exist for the specified document
 5490   (hossman)
 5492 * LUCENE-6039: Add IndexOptions.NONE and DocValuesType.NONE instead of
 5493   using null to mean not index and no doc values, renamed
 5494   IndexOptions.DOCS_ONLY to DOCS, and pulled IndexOptions and
 5495   DocValues out of FieldInfo into their own classes in
 5496   org.apache.lucene.index (Simon Willnauer, Robert Muir, Mike
 5497   McCandless)
 5499 * LUCENE-6041: Remove sugar methods FieldInfo.isIndexed and
 5500   FieldInfo.hasDocValues.  (Robert Muir, Mike McCandless)
 5502 * LUCENE-6044: Fix backcompat support for token filters with enablePositionIncrements=false.
 5503   Also fixed backcompat for TrimFilter with updateOffsets=true.  These options
 5504   are supported with a match version before 4.4, and no longer valid at all with 5.0.
 5505   (Ryan Ernst) 
 5507 * LUCENE-6042: CustomScoreQuery explain was incorrect in some cases,
 5508   such as when nested inside a boolean query. (Denis Lantsman via Robert Muir)
 5510 * LUCENE-6046: Add maxDeterminizedStates safety to determinize (which has
 5511   an exponential worst case) so that if it would create too many states, it
 5512   now throws an exception instead of exhausting CPU/RAM.  (Nik
 5513   Everett via Mike McCandless)
 5515 * LUCENE-6054: Allow repeating the empty automaton (Nik Everett via
 5516   Mike McCandless)
 5518 * LUCENE-6049: Don't throw cryptic exception writing a segment when
 5519   the only docs in it had fields that hit non-aborting exceptions
 5520   during indexing but also had doc values.  (Mike McCandless)
 5522 * LUCENE-6055: PayloadAttribute.clone() now does a deep clone of the underlying
 5523   bytes. (Shai Erera)
 5525 * LUCENE-6060: Remove dangerous IndexWriter.unlock method (Simon
 5526   Willnauer, Mike McCandless)
 5528 * LUCENE-6062: Pass correct fieldinfos to docvalues producer when the
 5529   segment has updates. (Mike McCandless, Shai Erera, Robert Muir)
 5531 * LUCENE-6075: Don't overflow int in SimpleRateLimiter (Boaz Leskes
 5532   via Mike McCandless)
 5534 * LUCENE-5987: IndexWriter will now forcefully close itself on
 5535   aborting exception (an exception that would otherwise cause silent
 5536   data loss).  (Robert Muir, Mike McCandless)
 5538 * LUCENE-6094: Allow IW.rollback to stop ConcurrentMergeScheduler even
 5539   when it's stalling because there are too many merges. (Mike McCandless)
 5541 * LUCENE-6105: Don't cache FST root arcs if the number of root arcs is
 5542   small, or if the cache would be > 20% of the size of the FST.
 5543   (Robert Muir, Mike McCandless)
 5545 * LUCENE-6124: Fix double-close() problems in codec and store APIs.
 5546   (Robert Muir)
 5548 * LUCENE-6152: Fix double close problems in OutputStreamIndexOutput.
 5549   (Uwe Schindler)
 5551 * LUCENE-6139: Highlighter: TokenGroup start & end offset getters should have
 5552   been returning the offsets of just the matching tokens in the group when
 5553   there's a distinction. (David Smiley)
 5555 * LUCENE-6173: NumericTermAttribute and spatial/CellTokenStream do not clone
 5556   their BytesRef(Builder)s. Also equals/hashCode was missing.  (Uwe Schindler)
 5558 * LUCENE-6205: Fixed intermittent concurrency issue that could cause
 5559   FileNotFoundException when writing doc values updates at the same
 5560   time that a merge kicks off.  (Mike McCandless)
 5562 * LUCENE-6192: Fix int overflow corruption case in skip data for
 5563   high frequency terms in extremely large indices (Robert Muir, Mike
 5564   McCandless)
 5566 * LUCENE-6093: Don't throw NullPointerException from
 5567   BlendedInfixSuggester for lookups that do not end in a prefix
 5568   token.  (jane chang via Mike McCandless)
 5570 * LUCENE-6214: Fixed IndexWriter deadlock when one thread is
 5571   committing while another opens a near-real-time reader and an
 5572   unrecoverable (tragic) exception is hit.  (Simon Willnauer, Mike
 5573   McCandless)
 5575 Documentation
 5577 * LUCENE-5392: Add/improve analysis package documentation to reflect
 5578   analysis API changes.  (Benson Margulies via Robert Muir - pull request #17)
 5580 * LUCENE-6057: Improve Sort(SortField) docs (Martin Braun via Mike McCandless)
 5582 * LUCENE-6112: Fix compile error in FST package example code
 5583   (Tomoko Uchida via Koji Sekiguchi)
 5585 Tests
 5587 * LUCENE-5957: Add option for tests to not randomize codec
 5588   (Ryan Ernst)
 5590 * LUCENE-5974: Add check that backcompat indexes use default codecs
 5591   (Ryan Ernst)
 5593 * LUCENE-5971: Create addBackcompatIndexes.py script to build and add 
 5594   backcompat test indexes for a given lucene version. Also renamed backcompat
 5595   index files to use Version.toString() in filename.
 5596   (Ryan Ernst)
 5598 * LUCENE-6002: Monster tests no longer fail.  Most of them now have an 80 hour
 5599   timeout, effectively removing the timeout.  The tests that operate near the 2
 5600   billion limit now use IndexWriter.MAX_DOCS instead of Integer.MAX_VALUE.
 5601   Some of the slow Monster tests now explicitly choose the default codec.
 5602   (Mike McCandless, Shawn Heisey)
 5604 * LUCENE-5968: Improve error message when 'ant beast' is run on top-level
 5605   modules.  (Ramkumar Aiyengar, Uwe Schindler)
 5607 * LUCENE-6120: Fix MockDirectoryWrapper's close() handling.
 5608   (Mike McCandless, Robert Muir)
 5610 Build
 5612 * LUCENE-5909: Smoke tester now has better command line parsing and
 5613   optionally also runs on Java 8.  (Ryan Ernst, Uwe Schindler)
 5615 * LUCENE-5902: Add bumpVersion.py script to manage version increase after release branch is cut.
 5617 * LUCENE-5962: Rename diffSources.py to createPatch.py and make it work with all text file types.
 5618   (Ryan Ernst)
 5620 * LUCENE-5995: Upgrade ICU to 54.1 (Robert Muir)
 5622 * LUCENE-6070: Upgrade forbidden-apis to 1.7 (Uwe Schindler)
 5624 Other
 5626 * LUCENE-5563: Removed sep layout: which has fallen behind on features and doesn't
 5627   perform as well as other options.  (Robert Muir)
 5629 * LUCENE-4086: Removed support for Lucene 3.x indexes. See migration guide for
 5630   more information.  (Robert Muir)
 5632 * LUCENE-5858: Moved Lucene 4 compatibility codecs to 'lucene-backward-codecs.jar'. 
 5633   (Adrien Grand, Robert Muir)
 5635 * LUCENE-5915: Remove Pulsing postings format. (Robert Muir)
 5637 * LUCENE-6213: Add useful exception message when commit contains segments from legacy codecs.
 5638   (Ryan Ernst)
 5640 ======================= Lucene 4.10.4 ======================
 5642 Bug fixes
 5644 * LUCENE-6019, LUCENE-6117: Remove -Dtests.assert to make IndexWriter
 5645   infoStream sane.  (Robert Muir, Mike McCandless)
 5647 * LUCENE-6161: Resolving deletes was failing to reuse DocsEnum likely
 5648   causing substantial performance cost for use cases that frequently
 5649   delete old documents (Mike McCandless)
 5651 * LUCENE-6192: Fix int overflow corruption case in skip data for
 5652   high frequency terms in extremely large indices (Robert Muir, Mike
 5653   McCandless)
 5655 * LUCENE-6207: Fixed consumption of several terms enums on the same
 5656   sorted (set) doc values instance at the same time.
 5657   (Tom Shally, Robert Muir, Adrien Grand)
 5659 * LUCENE-6093: Don't throw NullPointerException from
 5660   BlendedInfixSuggester for lookups that do not end in a prefix
 5661   token.  (jane chang via Mike McCandless)
 5663 * LUCENE-6279: Don't let an abusive leftover _N_upgraded.si in the
 5664   index directory cause index corruption on upgrade (Robert Muir, Mike
 5665   McCandless)
 5667 * LUCENE-6287: Fix concurrency bug in IndexWriter that could cause
 5668   index corruption (missing _N.si files) the first time 4.x kisses a
 5669   3.x index if merges are also running.  (Simon Willnauer, Mike
 5670   McCandless)
 5672 * LUCENE-6205: Fixed intermittent concurrency issue that could cause
 5673   FileNotFoundException when writing doc values updates at the same
 5674   time that a merge kicks off.  (Mike McCandless)
 5676 * LUCENE-6214: Fixed IndexWriter deadlock when one thread is
 5677   committing while another opens a near-real-time reader and an
 5678   unrecoverable (tragic) exception is hit.  (Simon Willnauer, Mike
 5679   McCandless)
 5681 * LUCENE-6105: Don't cache FST root arcs if the number of root arcs is
 5682   small, or if the cache would be > 20% of the size of the FST.
 5683   (Robert Muir, Mike McCandless)
 5685 * LUCENE-6001: DrillSideways hits NullPointerException for certain
 5686   BooleanQuery searches.  (Dragan Jotannovic, jane chang via Mike
 5687   McCandless)
 5689 * LUCENE-6306: Merging of doc values and norms now checks whether the
 5690   merge was aborted so IndexWriter.rollback can more promptly abort a
 5691   running merge. (Robert Muir, Mike McCandless)
 5693 API Changes
 5695 * LUCENE-6212: Deprecate IndexWriter APIs that accept per-document Analyzer.
 5696   These methods were trappy as they made it easy to accidentally index
 5697   tokens that were not easily searchable and will be removed in 5.0.0.
 5698   (Mike McCandless)
 5700 ======================= Lucene 4.10.3 ======================
 5702 Bug fixes
 5704 * LUCENE-6046: Add maxDeterminizedStates safety to determinize (which has
 5705   an exponential worst case) so that if it would create too many states, it
 5706   now throws an exception instead of exhausting CPU/RAM.  (Nik
 5707   Everett via Mike McCandless)
 5709 * LUCENE-6054: Allow repeating the empty automaton (Nik Everett via
 5710   Mike McCandless)
 5712 * LUCENE-6049: Don't throw cryptic exception writing a segment when
 5713   the only docs in it had fields that hit non-aborting exceptions
 5714   during indexing but also had doc values.  (Mike McCandless)
 5716 * LUCENE-6060: Deprecate IndexWriter.unlock (Simon Willnauer, Mike
 5717   McCandless)
 5719 * LUCENE-3229: Overlapping ordered SpanNearQuery spans should not match.
 5720   (Ludovic Boutros, Paul Elschot, Greg Dearing, ehatcher)
 5722 * LUCENE-6004: Don't highlight the LookupResult.key returned from
 5723   AnalyzingInfixSuggester (Christian Reuschling, jane chang via Mike McCandless)
 5725 * LUCENE-6075: Don't overflow int in SimpleRateLimiter (Boaz Leskes
 5726   via Mike McCandless)
 5728 * LUCENE-5980: Don't let document length overflow. (Robert Muir)
 5730 * LUCENE-6042: CustomScoreQuery explain was incorrect in some cases,
 5731   such as when nested inside a boolean query. (Denis Lantsman via Robert Muir)
 5733 * LUCENE-5948: RateLimiter now fully inits itself on init.  (Varun
 5734   Thacker via Mike McCandless)
 5736 * LUCENE-6055: PayloadAttribute.clone() now does a deep clone of the underlying
 5737   bytes. (Shai Erera)
 5739 * LUCENE-6094: Allow IW.rollback to stop ConcurrentMergeScheduler even
 5740   when it's stalling because there are too many merges. (Mike McCandless)
 5742 Documentation
 5744 * LUCENE-6057: Improve Sort(SortField) docs (Martin Braun via Mike McCandless)
 5746 ======================= Lucene 4.10.2 ======================
 5748 Bug fixes
 5750 * LUCENE-5977: Fix tokenstream safety checks in IndexWriter to properly
 5751   work across multi-valued fields. Previously some cases across multi-valued
 5752   fields would happily create a corrupt index. (Dawid Weiss, Robert Muir)
 5754 * LUCENE-6019: Detect when DocValuesType illegally changes for the
 5755   same field name.  Also added -Dtests.asserts=true|false so we can
 5756   run tests with and without assertions. (Simon Willnauer, Robert
 5757   Muir, Mike McCandless).
 5759 ======================= Lucene 4.10.1 ======================
 5761 Bug fixes
 5763 * LUCENE-5934: Fix backwards compatibility for 4.0 indexes.
 5764   (Ian Lea, Uwe Schindler, Robert Muir, Ryan Ernst)
 5766 * LUCENE-5939: Regenerate old backcompat indexes to ensure they were built with
 5767   the exact release
 5768   (Ryan Ernst, Uwe Schindler)
 5770 * LUCENE-5952: Improve error messages when version cannot be parsed;
 5771   don't check for too old or too new major version (it's too low level
 5772   to enforce here); use simple string tokenizer.  (Ryan Ernst, Uwe Schindler,
 5773   Robert Muir, Mike McCandless)
 5775 * LUCENE-5958: Don't let exceptions during checkpoint corrupt the index. 
 5776   Refactor existing OOM handling too, so you don't need to handle OOM special
 5777   for every IndexWriter method: instead such disasters will cause IW to close itself
 5778   defensively. (Robert Muir, Mike McCandless)
 5780 * LUCENE-5904: Fixed a corruption case that can happen when 1)
 5781   IndexWriter is uncleanly shut-down (OS crash, power loss, etc.), 2)
 5782   on startup, when a new IndexWriter is created, a virus checker is
 5783   holding some of the previously written but unused files open and
 5784   preventing deletion, 3) IndexWriter writes these files again during
 5785   the course of indexing, then the files can later be deleted, causing
 5786   corruption.  This case was detected by adding evilness to
 5787   MockDirectoryWrapper to have it simulate a virus checker holding a
 5788   file open and preventing deletion (Robert Muir, Mike McCandless)
 5790 * LUCENE-5916: Static scope test components should be consistent between
 5791   tests (and test iterations). Fix for FaultyIndexInput in particular.
 5792   (Dawid Weiss)
 5794 * LUCENE-5975: Fix reading of 3.0-3.3 indexes, where bugs in these old
 5795   index formats would result in CorruptIndexException "did not read all
 5796   bytes from file" when reading the deleted docs file. (Patrick Mi, Robert MUir)
 5798 Tests
 5800 * LUCENE-5936: Add backcompat checks to verify what is tested matches known versions
 5801   (Ryan Ernst)
 5803 ======================= Lucene 4.10.0 ======================
 5805 New Features
 5807 * LUCENE-5778: Support hunspell morphological description fields/aliases.
 5808   (Robert Muir)
 5810 * LUCENE-5801: Added (back) OrdinalMappingAtomicReader for merging search
 5811   indexes that contain category ordinals from separate taxonomy indexes.
 5812   (Nicola Buso via Shai Erera)
 5814 * LUCENE-4175, LUCENE-5714, LUCENE-5779: Index and search rectangles with spatial
 5815   BBoxSpatialStrategy using most predicates.  Sort documents by relative overlap
 5816   of query areas or just by indexed shape area. (Ryan McKinley, David Smiley)
 5818 * LUCENE-5806: Extend expressions grammar to support array access in variables.
 5819   Added helper class VariableContext to parse complex variable into pieces.
 5820   (Ryan Ernst)
 5822 * LUCENE-5826: Support proper hunspell case handling, LANG, KEEPCASE, NEEDAFFIX,
 5823   and ONLYINCOMPOUND flags.  (Robert Muir)
 5825 * LUCENE-5815: Add TermAutomatonQuery, a proximity query allowing you
 5826   to create an arbitrary automaton, using terms on the transitions,
 5827   expressing which sequence of sequential terms (including a special
 5828   "any" term) are allowed.  This is a generalization of
 5829   MultiPhraseQuery and span queries, and enables "correct" (including
 5830   position) length search-time graph synonyms.  (Mike McCandless)
 5832 * LUCENE-5819: Add OrdsLucene41 block tree terms dict and postings
 5833   format, to include term ordinals in the index so the optional
 5834   TermsEnum.ord() and TermsEnum.seekExact(long ord) APIs work.  (Mike
 5835   McCandless)
 5837 * LUCENE-5835: TermValComparator can sort missing values last. (Adrien Grand)
 5839 * LUCENE-5825: Benchmark module can use custom postings format, e.g.:
 5840  codec.postingsFormat=Memory (Varun Shenoy, David Smiley)
 5842 * LUCENE-5842: When opening large files (where it's too expensive to compare
 5843   checksum against all the bytes), retrieve checksum to validate structure
 5844   of footer, this can detect some forms of corruption such as truncation.
 5845   (Robert Muir)
 5847 * LUCENE-5739: Added DataInput.readZ(Int|Long) and DataOutput.writeZ(Int|Long)
 5848   to read and write small signed integers. (Adrien Grand)
 5850 API Changes
 5852 * LUCENE-5752: Simplified Automaton API to be immutable. (Mike McCandless)
 5854 * LUCENE-5793: Add equals/hashCode to FieldType. (Shay Banon, Robert Muir)
 5856 * LUCENE-5692: DisjointSpatialFilter is deprecated (used by RecursivePrefixTreeStrategy)
 5857   (David Smiley)
 5859 * LUCENE-5771: SpatialOperation's predicate names are now aliased to OGC standard names.
 5860   Thus you can use: Disjoint, Equals, Intersects, Overlaps, Within, Contains, Covers,
 5861   CoveredBy. The area requirement on the predicates was removed, and Overlaps' definition
 5862   was fixed. (David Smiley)
 5864 * LUCENE-5850: Made Version handling more robust and extensible. Deprecated
 5865   Constants.LUCENE_MAIN_VERSION, Constants.LUCENE_VERSION and current Version
 5866   constants of the form LUCENE_X_Y. Added version constants that include bugfix
 5867   number of form LUCENE_X_Y_Z.  Changed Version.LUCENE_CURRENT to Version.LATEST.
 5868   CheckIndex now prints the Lucene version used to write each segment.
 5869   (Ryan Ernst, Uwe Schindler, Robert Muir, Mike McCandless)
 5871 * LUCENE-5836: BytesRef has been splitted into BytesRef, whose intended usage is
 5872   to be just a reference to a section of a larger byte[] and BytesRefBuilder
 5873   which is a StringBuilder-like class for BytesRef instances. (Adrien Grand)
 5875 * LUCENE-5883: You can now change the MergePolicy instance on a live IndexWriter,
 5876   without first closing and reopening the writer. This allows to e.g. run a special
 5877   merge with UpgradeIndexMergePolicy without reopening the writer. Also, MergePolicy
 5878   no longer implements Closeable; if you need to release your custom MergePolicy's
 5879   resources, you need to implement close() and call it explicitly. (Shai Erera)
 5881 * LUCENE-5859: Deprecate Analyzer constructors taking Version.  Use Analyzer.setVersion()
 5882   to set the version an analyzer to replicate behavior from a specific release.
 5883   (Ryan Ernst, Robert Muir)
 5886 Optimizations
 5888 * LUCENE-5780: Make OrdinalMap more memory-efficient, especially in case the
 5889   first segment has all values. (Adrien Grand, Robert Muir)
 5891 * LUCENE-5782: OrdinalMap now sorts enums before being built in order to
 5892   improve compression. (Adrien Grand)
 5894 * LUCENE-5798: Optimize MultiDocsEnum reuse. (Robert Muir)
 5896 * LUCENE-5799: Optimize numeric docvalues merging. (Robert Muir)
 5898 * LUCENE-5797: Optimize norms merging (Adrien Grand, Robert Muir)
 5900 * LUCENE-5803: Add DelegatingAnalyzerWrapper, an optimized variant
 5901   of AnalyzerWrapper that doesn't allow to wrap components or readers.
 5902   This wrapper class is the base class of all analyzers that just delegate
 5903   to another analyzer, e.g. per field name: PerFieldAnalyzerWrapper and
 5904   Solr's schema support.  (Shay Banon, Uwe Schindler, Robert Muir)
 5906 * LUCENE-5795: MoreLikeThisQuery now only collects the top N terms instead
 5907   of collecting all terms from the like text when building the query. 
 5908   (Alex Ksikes, Simon Willnauer)
 5910 * LUCENE-5681: Fix RAMDirectory's IndexInput to not do double buffering
 5911   on slices (causes useless data copying, especially on random access slices).
 5912   This also improves slices of NRTCachingDirectory, because the cache
 5913   is based on RAMDirectory. BufferedIndexInput.wrap() was marked with a
 5914   warning in javadocs. It is almost always a better idea to implement
 5915   slicing on your own!  (Uwe Schindler, Robert Muir)
 5917 * LUCENE-5834: Empty sorted set and numeric doc values are now singletons.
 5918   (Adrien Grand)
 5920 * LUCENE-5841: Improve performance of block tree terms dictionary when
 5921   assigning terms to blocks.  (Mike McCandless)
 5923 * LUCENE-5856: Optimize Fixed/Open/LongBitSet to remove unnecessary AND. 
 5924   (Robert Muir)
 5926 * LUCENE-5884: Optimize FST.ramBytesUsed.  (Adrien Grand, Robert Muir,
 5927   Mike McCandless)
 5929 * LUCENE-5882: Add Lucene410DocValuesFormat, with faster term lookups
 5930   for SORTED/SORTED_SET fields.  (Robert Muir)
 5932 * LUCENE-5887: Remove WeakIdentityMap caching in AttributeFactory,
 5933   AttributeSource, and VirtualMethod in favour of Java 7's ClassValue.
 5934   Always use MethodHandles to create AttributeImpl classes.
 5935   (Uwe Schindler)
 5937 Bug Fixes
 5939 * LUCENE-5796: Fixes the Scorer.getChildren() method for two combinations 
 5940   of BooleanQuery. (Terry Smith via Robert Muir)
 5942 * LUCENE-5790: Fix compareTo in MutableValueDouble and MutableValueBool, this caused 
 5943   incorrect results when grouping on fields with missing values. 
 5944   (海老澤 志信, hossman)
 5946 * LUCENE-5817: Fix hunspell zero-affix handling: previously only zero-strips worked
 5947   correctly.  (Robert Muir)
 5949 * LUCENE-5818, LUCENE-5823: Fix hunspell overgeneration for short strings that also 
 5950   match affixes, words are only stripped to a zero-length string if FULLSTRIP option
 5951   is specified in the dictionary.  (Robert Muir)
 5953 * LUCENE-5824: Fix hunspell 'long' flag handling. (Robert Muir)
 5955 * LUCENE-5838: Fix hunspell when the .aff file has over 64k affixes. (Robert Muir)
 5957 * LUCENE-5869: Added restriction to positive values for maxExpansions in
 5958   FuzzyQuery.  (Ryan Ernst)
 5960 * LUCENE-5672: IndexWriter.addIndexes() calls maybeMerge(), to ensure the index stays
 5961   healthy. If you don't want merging use NoMergePolicy instead. (Robert Muir)
 5963 * LUCENE-5908: Fix Lucene43NGramTokenizer to be final
 5965 Test Framework
 5967 * LUCENE-5786: Unflushed/ truncated events file (hung testing subprocess).
 5968   (Dawid Weiss)
 5970 * LUCENE-5881: Add "beasting" of tests: repeats the whole "test" Ant target
 5971   N times with "ant beast -Dbeast.iters=N".  (Uwe Schindler, Robert Muir,
 5972   Ryan Ernst, Dawid Weiss)
 5974 Build
 5976 * LUCENE-5770: Upgrade to JFlex 1.6, which has direct support for
 5977   supplementary code points - as a result, ICU4J is no longer used
 5978   to generate surrogate pairs to augment JFlex scanner specifications.
 5979   (Steve Rowe)
 5981 * SOLR-6358: Remove VcsDirectoryMappings from idea configuration 
 5982   vcs.xml (Ramkumar Aiyengar via Steve Rowe)
 5984 ======================= Lucene 4.9.1 ======================
 5986 Bug fixes
 5988 * LUCENE-5907: Fix corruption case when opening a pre-4.x index with
 5989   IndexWriter, then opening an NRT reader from that writer, then
 5990   calling commit from the writer, then closing the NRT reader.  This
 5991   case would remove the wrong files from the index leading to a
 5992   corrupt index.  (Mike McCandless)
 5994 * LUCENE-5919: Fix exception handling inside IndexWriter when
 5995   deleteFile throws an exception, to not over-decRef index files,
 5996   possibly deleting a file that's still in use in the index, leading
 5997   to corruption.  (Mike McCandless)
 5999 * LUCENE-5922: DocValuesDocIdSet on 5.x and FieldCacheDocIdSet on 4.x
 6000   are not cacheable. (Adrien Grand)
 6002 * LUCENE-5843: Added IndexWriter.MAX_DOCS which is the maximum number
 6003   of documents allowed in a single index, and any operations that add
 6004   documents will now throw IllegalStateException if the max count
 6005   would be exceeded, instead of silently creating an unusable
 6006   index.  (Mike McCandless)
 6008 * LUCENE-5844: ArrayUtil.grow/oversize now returns a maximum of
 6009   Integer.MAX_VALUE - 8 for the maximum array size.  (Robert Muir,
 6010   Mike McCandless)
 6012 * LUCENE-5827: Make all Directory implementations correctly fail with
 6013   IllegalArgumentException if slices are out of bounds.  (Uwe Schindler)
 6015 * LUCENE-5897, LUCENE-5400: JFlex-based tokenizers StandardTokenizer and
 6016   UAX29URLEmailTokenizer tokenize extremely slowly over long sequences of
 6017   text partially matching certain grammar rules.  The scanner default
 6018   buffer size was reduced, and scanner buffer growth was disabled, resulting
 6019   in much, much faster tokenization for these text sequences.  
 6020   (Chris Geeringh, Robert Muir, Steve Rowe)
 6022 ======================= Lucene 4.9.0 =======================
 6024 Changes in Runtime Behavior
 6026 * LUCENE-5611: Changing the term vector options for multiple field
 6027   instances by the same name in one document is not longer accepted;
 6028   IndexWriter will now throw IllegalArgumentException.  (Robert Muir,
 6029   Mike McCandless)
 6031 * LUCENE-5646: Remove rare/undertested bulk merge algorithm in 
 6032   CompressingStoredFieldsWriter. (Robert Muir, Adrien Grand)
 6034 New Features
 6036 * LUCENE-5610: Add Terms.getMin and Terms.getMax to get the lowest and
 6037   highest terms, and NumericUtils.get{Min/Max}{Int/Long} to get the
 6038   minimum numeric values from the provided Terms.  (Robert Muir, Mike
 6039   McCandless)
 6041 * LUCENE-5675: Add IDVersionPostingsFormat, a postings format
 6042   optimized for primary-key (ID) fields that also record a version
 6043   (long) for each ID.  (Robert Muir, Mike McCandless)
 6045 * LUCENE-5680: Add ability to atomically update a set of DocValues
 6046   fields. (Shai Erera)
 6048 * LUCENE-5717: Add support for multiterm queries nested inside
 6049   filtered and constant-score queries to postings highlighter.
 6050   (Luca Cavanna via Robert Muir)
 6052 * LUCENE-5731, LUCENE-5760: Add RandomAccessInput, a random access API for directory.
 6053   Add DirectReader/Writer, optimized for reading packed integers directly 
 6054   from Directory. Add Lucene49Codec and Lucene49DocValuesFormat that make
 6055   use of these.  (Robert Muir)
 6057 * LUCENE-5743: Add Lucene49NormsFormat, which can compress in some cases
 6058   such as very short fields.  (Ryan Ernst, Adrien Grand, Robert Muir)
 6060 * LUCENE-5748: Add SORTED_NUMERIC docvalues type, which is efficient
 6061   for processing numeric fields with multiple values.  (Robert Muir)
 6063 * LUCENE-5754: Allow "$" as part of variable and function names in
 6064   expressions module.  (Uwe Schindler)
 6066 Changes in Backwards Compatibility Policy
 6068 * LUCENE-5634: Add reuse argument to IndexableField.tokenStream. This
 6069   can be used by custom fieldtypes, which don't use the Analyzer, but
 6070   implement their own TokenStream.  (Uwe Schindler, Robert Muir)
 6072 * LUCENE-5640: AttributeSource.AttributeFactory was moved to a
 6073   top-level class: org.apache.lucene.util.AttributeFactory
 6074   (Uwe Schindler, Robert Muir)
 6076 * LUCENE-4371: Removed IndexInputSlicer and Directory.createSlicer() and replaced
 6077   with IndexInput.slice(). (Robert Muir)
 6079 * LUCENE-5727, LUCENE-5678: Remove IndexOutput.seek, IndexOutput.setLength().
 6080   (Robert Muir, Uwe Schindler)
 6082 API Changes
 6084 * LUCENE-5756: IndexWriter now implements Accountable and IW#ramSizeInBytes()
 6085   has been deprecated in favor of IW#ramBytesUsed() (Simon Willnauer)
 6087 * LUCENE-5725: MoreLikeThis#like now accepts multiple values per field.
 6088   The pre-existing method has been deprecated in favor of a variable arguments
 6089   for the like text. (Alex Ksikes via Simon Willnauer)
 6091 * LUCENE-5711: MergePolicy accepts an IndexWriter instance
 6092   on each method rather than holding state against a single
 6093   IndexWriter instance. (Simon Willnauer)
 6095 * LUCENE-5582: Deprecate IndexOutput.length (just use
 6096   IndexOutput.getFilePointer instead) and IndexOutput.setLength.
 6097   (Mike McCandless)
 6099 * LUCENE-5621: Deprecate IndexOutput.flush: this is not used by Lucene.
 6100   (Robert Muir)
 6102 * LUCENE-5611: Simplified Lucene's default indexing chain / APIs.
 6103   AttributeSource/TokenStream.getAttribute now returns null if the
 6104   attribute is not present (previously it threw
 6105   IllegalArgumentException).  StoredFieldsWriter.startDocument no
 6106   longer receives the number of fields that will be added (Robert
 6107   Muir, Mike McCandless)
 6109 * LUCENE-5632: In preparation for coming Lucene versions, the Version
 6110   enum constants were renamed to make them better readable. The constant
 6111   for Lucene 4.9 is now "LUCENE_4_9". Version.parseLeniently() is still
 6112   able to parse the old strings ("LUCENE_49"). The old identifiers got
 6113   deprecated and will be removed in Lucene 5.0.  (Uwe Schindler,
 6114   Robert Muir)
 6116 * LUCENE-5633: Change NoMergePolicy to a singleton with no distinction between
 6117   compound and non-compound types. (Shai Erera)
 6119 * LUCENE-5640: The Token class was deprecated. Since Lucene 2.9, TokenStreams
 6120   are using Attributes, Token is no longer used.  (Uwe Schindler, Robert Muir)
 6122 * LUCENE-5679: Consolidated IndexWriter.deleteDocuments(Term) and 
 6123   IndexWriter.deleteDocuments(Query) with their varargs counterparts.
 6124   (Shai Erera)
 6126 * LUCENE-5701: Core closed listeners are now available in the AtomicReader API,
 6127   they used to sit only in SegmentReader. (Adrien Grand, Robert Muir)
 6129 * LUCENE-5706: Removed the option to unset a DocValues field through DocValues
 6130   updates. (Shai Erera)
 6132 * LUCENE-5700: Added oal.util.Accountable that is now implemented by all
 6133   classes whose memory usage can be estimated. (Robert Muir, Adrien Grand)
 6135 * LUCENE-5708: Remove IndexWriterConfig.clone, so now IndexWriter
 6136   simply uses the IndexWriterConfig you pass it, and you must create a
 6137   new IndexWriterConfig for each IndexWriter.  (Mike McCandless)
 6139 * LUCENE-5678: IndexOutput no longer allows seeking, so it is no longer required
 6140   to use RandomAccessFile to write Indexes. Lucene now uses standard FileOutputStream
 6141   wrapped with OutputStreamIndexOutput to write index data. BufferedIndexOutput was
 6142   removed, because buffering and checksumming is provided by FilterOutputStreams,
 6143   provided by the JDK.  (Uwe Schindler, Mike McCandless)
 6145 * LUCENE-5703: BinaryDocValues API changed to work like TermsEnum and not allocate/
 6146   copy bytes on each access, you are responsible for cloning if you want to keep
 6147   data around. (Adrien Grand)
 6149 * LUCENE-5695: DocIdSet implements Accountable. (Adrien Grand)
 6151 * LUCENE-5757: Moved RamUsageEstimator's reflection-based processing to RamUsageTester
 6152   in the test-framework module. (Robert Muir)
 6154 * LUCENE-5761: Removed DiskDocValuesFormat, it was very inefficient and saved very little
 6155   RAM over the default codec. (Robert Muir)
 6157 * LUCENE-5775: Deprecate JaspellLookup. (Mike McCandless)
 6159 Optimizations
 6161 * LUCENE-5603: hunspell stemmer more efficiently strips prefixes
 6162   and suffixes.  (Robert Muir)
 6164 * LUCENE-5599: HttpReplicator did not properly delegate bulk read() to wrapped
 6165   InputStream. (Christoph Kaser via Shai Erera)
 6167 * LUCENE-5591: pass an IOContext with estimated flush size when applying DV
 6168   updates. (Shai Erera)
 6170 * LUCENE-5634: IndexWriter reuses TokenStream instances for String and Numeric
 6171   fields by default. (Uwe Schindler, Shay Banon, Mike McCandless, Robert Muir)
 6173 * LUCENE-5638, LUCENE-5640: TokenStream uses a more performant AttributeFactory
 6174   by default, that packs the core attributes into one implementation
 6175   (PackedTokenAttributeImpl), for faster clearAttributes(), saveState(), and
 6176   restoreState(). In addition, AttributeFactory uses Java 7 MethodHandles for
 6177   instantiating Attribute implementations.  (Uwe Schindler, Robert Muir)
 6179 * LUCENE-5609: Changed the default NumericField precisionStep from 4
 6180   to 8 (for int/float) and 16 (for long/double), for faster indexing
 6181   time and smaller indices. (Robert Muir, Uwe Schindler, Mike McCandless)
 6183 * LUCENE-5670: Add skip/FinalOutput to FST Outputs.  (Christian
 6184   Ziech via Mike McCandless).
 6186 * LUCENE-4236: Optimize BooleanQuery's in-order scoring. This speeds up
 6187   some types of boolean queries.  (Robert Muir)
 6189 * LUCENE-5694: Don't score() subscorers in DisjunctionSumScorer or 
 6190   DisjunctionMaxScorer unless score() is called.  (Robert Muir)
 6192 * LUCENE-5720: Optimize DirectPackedReader's decompression. (Robert Muir)
 6194 * LUCENE-5722: Optimize ByteBufferIndexInput#seek() by specializing
 6195   implementations. This improves random access as used by docvalues codecs
 6196   if used with MMapDirectory.  (Robert Muir, Uwe Schindler)
 6198 * LUCENE-5730: FSDirectory.open returns MMapDirectory for 64-bit operating
 6199   systems, not just Linux and Windows. (Robert Muir)
 6201 * LUCENE-5703: BinaryDocValues producers don't allocate or copy bytes on 
 6202   each access anymore.  (Adrien Grand)
 6204 * LUCENE-5721: Monotonic compression doesn't use zig-zag encoding anymore.
 6205   (Robert Muir, Adrien Grand)
 6207 * LUCENE-5750: Speed up monotonic addressing for BINARY and SORTED_SET 
 6208   docvalues. (Robert Muir)
 6210 * LUCENE-5751: Speed up MemoryDocValues. (Adrien Grand, Robert Muir)
 6212 * LUCENE-5767: OrdinalMap optimizations, that mostly help on low cardinalities.
 6213   (Martijn van Groningen, Adrien Grand) 
 6215 * LUCENE-5769: SingletonSortedSetDocValues now supports random access ordinals.
 6216   (Robert Muir)
 6218 Bug fixes
 6220 * LUCENE-5738: Ensure NativeFSLock prevents opening the file channel for the
 6221   lock if the lock is already obtained by the JVM. Trying to obtain an already
 6222   obtained lock in the same JVM can unlock the file might allow other processes
 6223   to lock the file even without explicitly unlocking the FileLock. This behavior
 6224   is operating system dependent. (Simon Willnauer) 
 6226 * LUCENE-5673: MMapDirectory: Work around a "bug" in the JDK that throws
 6227   a confusing OutOfMemoryError wrapped inside IOException if the FileChannel
 6228   mapping failed because of lack of virtual address space. The IOException is
 6229   rethrown with more useful information about the problem, omitting the
 6230   incorrect OutOfMemoryError.  (Robert Muir, Uwe Schindler)
 6232 * LUCENE-5682: NPE in QueryRescorer when Scorer is null
 6233   (Joel Bernstein, Mike McCandless)
 6235 * LUCENE-5691: DocTermOrds lookupTerm(BytesRef) would return incorrect results
 6236   if the underlying TermsEnum supports ord() and the insertion point would
 6237   be at the end. (Robert Muir)
 6239 * LUCENE-5618, LUCENE-5636: SegmentReader referenced unneeded files following 
 6240   doc-values updates. Now doc-values field updates are written in separate file
 6241   per field. (Shai Erera, Robert Muir)
 6243 * LUCENE-5684: Make best effort to detect invalid usage of Lucene,
 6244   when IndexReader is reopened after all files in its index were
 6245   removed and recreated by the application (the proper way to do
 6246   this is IndexWriter.deleteAll, or opening an IndexWriter with
 6247   OpenMode.CREATE)  (Mike McCandless)
 6249 * LUCENE-5704: Fix compilation error with Java 8u20.  (Uwe Schindler)
 6251 * LUCENE-5710: Include the inner exception as the cause and in the
 6252   exception message when an immense term is hit during indexing (Lee
 6253   Hinman via Mike McCandless)
 6255 * LUCENE-5724: CompoundFileWriter was failing to pass through the
 6256   IOContext in some cases, causing NRTCachingDirectory to cache
 6257   compound files when it shouldn't, then causing OOMEs.  (Mike
 6258   McCandless)
 6260 * LUCENE-5747: Project-specific settings for the eclipse development
 6261   environment will prevent automatic code reformatting. (Shawn Heisey)
 6263 * LUCENE-5768, LUCENE-5777: Hunspell condition checks containing character classes
 6264   were buggy. (Clinton Gormley, Robert Muir)
 6266 Test Framework
 6268 * LUCENE-5622: Fail tests if they print over the given limit of bytes to 
 6269   System.out or System.err. (Robert Muir, Dawid Weiss)
 6271 * LUCENE-5619: Added backwards compatibility tests to ensure we can update existing
 6272   indexes with doc-values updates. (Shai Erera, Robert Muir)
 6274 Build
 6276 * LUCENE-5442: The Ant check-lib-versions target now runs Ivy resolution
 6277   transitively, then fails the build when it finds a version conflict: when a
 6278   transitive dependency's version is more recent than the direct dependency's
 6279   version specified in lucene/ivy-versions.properties.  Exceptions are
 6280   specifiable in lucene/ivy-ignore-conflicts.properties.
 6281   (Steve Rowe)
 6283 * LUCENE-5715: Upgrade direct dependencies known to be older than transitive
 6284   dependencies: com.sun.jersey.version:1.8->1.9; com.sun.xml.bind:jaxb-impl:2.2.2->2.2.3-1;
 6285   commons-beanutils:commons-beanutils:1.7.0->1.8.3; commons-digester:commons-digester:2.0->2.1;
 6286   commons-io:commons-io:2.1->2.3; commons-logging:commons-logging:1.1.1->1.1.3;
 6287   io.netty:netty:3.6.2.Final->3.7.0.Final; javax.activation:activation:1.1->1.1.1;
 6288   javax.mail:mail:1.4.1->1.4.3; log4j:log4j:1.2.16->1.2.17; org.apache.avro:avro:1.7.4->1.7.5;
 6289   org.tukaani:xz:1.2->1.4; org.xerial.snappy:snappy-java:>1.0.5 (Steve Rowe)
 6291 ======================= Lucene 4.8.1 =======================
 6293 Bug fixes
 6295 * LUCENE-5639: Fix PositionLengthAttribute implementation in Token class.
 6296   (Uwe Schindler, Robert Muir)
 6298 * LUCENE-5635: IndexWriter didn't properly handle IOException on TokenStream.reset(),
 6299   which could leave the analyzer in an inconsistent state.  (Robert Muir)
 6301 * LUCENE-5599: HttpReplicator did not properly delegate bulk read() to wrapped
 6302   InputStream. (Christoph Kaser via Shai Erera)
 6304 * LUCENE-5600: HttpClientBase did not properly consume a connection if a server
 6305   error occurred. (Christoph Kaser via Shai Erera)
 6307 * LUCENE-5628: Change getFiniteStrings to iterative not recursive
 6308   implementation, so that building suggesters on a long suggestion
 6309   doesn't risk overflowing the stack; previously it consumed one Java
 6310   stack frame per character in the expanded suggestion.  If you are building
 6311   a suggester this is a nasty trap. (Robert Muir, Simon Willnauer,
 6312   Mike McCandless).
 6314 * LUCENE-5559: Add additional argument validation for CapitalizationFilter
 6315   and CodepointCountFilter. (Ahmet Arslan via Robert Muir)
 6317 * LUCENE-5641: SimpleRateLimiter would silently rate limit at 8 MB/sec
 6318   even if you asked for higher rates.  (Mike McCandless)
 6320 * LUCENE-5644: IndexWriter clears which threads use which internal
 6321   thread states on flush, so that if an application reduces how many
 6322   threads it uses for indexing, that results in a reduction of how
 6323   many segments are flushed on a full-flush (e.g. to obtain a
 6324   near-real-time reader).  (Simon Willnauer, Mike McCandless)
 6326 * LUCENE-5653: JoinUtil with ScoreMode.Avg on a multi-valued field
 6327   with more than 256 values would throw exception.
 6328   (Mikhail Khludnev via Robert Muir)
 6330 * LUCENE-5654: Fix various close() methods that could suppress 
 6331   throwables such as OutOfMemoryError, instead returning scary messages
 6332   that look like index corruption.  (Mike McCandless, Robert Muir)
 6334 * LUCENE-5656: Fix rare fd leak in SegmentReader when multiple docvalues
 6335   fields have been updated with IndexWriter.updateXXXDocValue and one
 6336   hits exception. (Shai Erera, Robert Muir)
 6338 * LUCENE-5660: AnalyzingSuggester.build will now throw IllegalArgumentException if
 6339   you give it a longer suggestion than it can handle (Robert Muir, Mike McCandless)
 6341 * LUCENE-5662: Add missing checks to Field to prevent IndexWriter.abort
 6342   if a stored value is null. (Robert Muir)
 6344 * LUCENE-5668: Fix off-by-one in TieredMergePolicy (Mike McCandless)
 6346 * LUCENE-5671: Upgrade ICU version to fix an ICU concurrency problem that
 6347   could cause exceptions when indexing. (feedly team, Robert Muir)
 6349 ======================= Lucene 4.8.0 =======================
 6351 System Requirements
 6353 * LUCENE-4747, LUCENE-5514: Move to Java 7 as minimum Java version.
 6354   (Robert Muir, Uwe Schindler)
 6356 Changes in Runtime Behavior
 6358 * LUCENE-5472: IndexWriter.addDocument will now throw an IllegalArgumentException
 6359   if a Term to be indexed exceeds IndexWriter.MAX_TERM_LENGTH.  To recreate previous
 6360   behavior of silently ignoring these terms, use LengthFilter in your Analyzer.
 6361   (hossman, Mike McCandless, Varun Thacker)
 6363 New Features
 6365 * LUCENE-5356: Morfologik filter can accept custom dictionary resources.
 6366   (Michal Hlavac, Dawid Weiss)
 6368 * LUCENE-5454: Add SortedSetSortField to lucene/sandbox, to allow sorting
 6369   on multi-valued field. (Robert Muir)
 6371 * LUCENE-5478: CommonTermsQuery now allows to create custom term queries
 6372   similar to the query parser by overriding a newTermQuery method.
 6373   (Simon Willnauer)
 6375 * LUCENE-5477: AnalyzingInfixSuggester now supports near-real-time
 6376   additions and updates (to change weight or payload of an existing
 6377   suggestion).  (Mike McCandless)
 6379 * LUCENE-5482: Improve default TurkishAnalyzer by adding apostrophe
 6380   handling suitable for Turkish.  (Ahmet Arslan via Robert Muir)
 6382 * LUCENE-5479: FacetsConfig subclass can now customize the default
 6383   per-dim facets configuration.  (Rob Audenaerde via Mike McCandless)
 6385 * LUCENE-5485: Add circumfix support to HunspellStemFilter. (Robert Muir)
 6387 * LUCENE-5224: Add iconv, oconv, and ignore support to HunspellStemFilter.
 6388   (Robert Muir)
 6390 * LUCENE-5493: SortingMergePolicy, and EarlyTerminatingSortingCollector
 6391   support arbitrary Sort specifications.  
 6392   (Robert Muir, Mike McCandless, Adrien Grand)
 6394 * LUCENE-3758: Allow the ComplexPhraseQueryParser to search order or 
 6395   un-order proximity queries. (Ahmet Arslan via Erick Erickson)
 6397 * LUCENE-5530: ComplexPhraseQueryParser throws ParseException for fielded queries.
 6398   (Erick Erickson via Tomas Fernandez Lobbe and Ahmet Arslan)
 6400 * LUCENE-5513: Add IndexWriter.updateBinaryDocValue which lets
 6401   you update the value of a BinaryDocValuesField without reindexing the
 6402   document(s). (Shai Erera)
 6404 * LUCENE-4072: Add ICUNormalizer2CharFilter, which lets you do unicode normalization
 6405   with offset correction before the tokenizer. (David Goldfarb, Ippei UKAI via Robert Muir)
 6407 * LUCENE-5476: Add RandomSamplingFacetsCollector for computing facets on a sampled
 6408   set of matching hits, in cases where there are millions of hits.
 6409   (Rob Audenaerde, Gilad Barkai, Shai Erera)
 6411 * LUCENE-4984: Add SegmentingTokenizerBase, abstract class for tokenizers
 6412   that want to do two-pass tokenization such as by sentence and then by word.
 6413   (Robert Muir)
 6415 * LUCENE-5489: Add Rescorer/QueryRescorer, to resort the hits from a
 6416   first pass search using scores from a more costly second pass
 6417   search. (Simon Willnauer, Robert Muir, Mike McCandless)
 6419 * LUCENE-5528: Add context to suggesters (InputIterator and Lookup
 6420   classes), and fix AnalyzingInfixSuggester to handle contexts.
 6421   Suggester contexts allow you to filter suggestions.  (Areek Zillur,
 6422   Mike McCandless)
 6424 * LUCENE-5545: Add SortRescorer and Expression.getRescorer, to
 6425   resort the hits from a first pass search using a Sort or an
 6426   Expression. (Simon Willnauer, Robert Muir, Mike McCandless)
 6428 * LUCENE-5558: Add TruncateTokenFilter which truncates terms to
 6429   the specified length.  (Ahmet Arslan via Robert Muir)
 6431 * LUCENE-2446: Added checksums to lucene index files. As of 4.8, the last 8 
 6432   bytes of each file contain a zlib-crc32 checksum. Small metadata files are
 6433   verified on load. Larger files can be checked on demand via 
 6434   AtomicReader.checkIntegrity. You can configure this to happen automatically
 6435   before merges by enabling IndexWriterConfig.setCheckIntegrityAtMerge.
 6436   (Robert Muir)
 6438 * LUCENE-5580: Checksums are automatically verified on the default stored
 6439   fields format when performing a bulk merge. (Adrien Grand)
 6441 * LUCENE-5602: Checksums are automatically verified on the default term
 6442   vectors format when performing a bulk merge. (Adrien Grand, Robert Muir)
 6444 * LUCENE-5583: Added DataInput.skipBytes. ChecksumIndexInput can now seek, but
 6445   only forward. (Adrien Grand, Mike McCandless, Simon Willnauer, Uwe Schindler)
 6447 * LUCENE-5588: Lucene now calls fsync() on the index directory, ensuring
 6448   that all file metadata is persisted on disk in case of power failure.
 6449   This does not work on all file systems and operating systems, but Linux
 6450   and MacOSX are known to work. On Windows, fsyncing a directory is not
 6451   possible with Java APIs.  (Mike McCandless, Uwe Schindler)
 6453 API Changes
 6455 * LUCENE-5454: Add RandomAccessOrds, an optional extension of SortedSetDocValues
 6456   that supports random access to the ordinals in a document. (Robert Muir)
 6458 * LUCENE-5468: Move offline Sort (from suggest module) to OfflineSort. (Robert Muir)
 6460 * LUCENE-5493: SortingMergePolicy and EarlyTerminatingSortingCollector take
 6461   Sort instead of Sorter. BlockJoinSorter is removed, replaced with 
 6462   BlockJoinComparatorSource, which can take a Sort for ordering of parents
 6463   and a separate Sort for ordering of children within a block. 
 6464   (Robert Muir, Mike McCandless, Adrien Grand)
 6466 * LUCENE-5516: MergeScheduler#merge() now accepts a MergeTrigger as well as
 6467   a boolean that indicates if a new merge was found in the caller thread before
 6468   the scheduler was called. (Simon Willnauer)
 6470 * LUCENE-5487: Separated bulk scorer (new Weight.bulkScorer method) from
 6471   normal scoring (Weight.scorer) for those queries that can do bulk
 6472   scoring more efficiently, e.g. BooleanQuery in some cases.  This
 6473   also simplified the Weight.scorer API by removing the two confusing
 6474   booleans.  (Robert Muir, Uwe Schindler, Mike McCandless)
 6476 * LUCENE-5519: TopNSearcher now allows to retrieve incomplete results if the max
 6477   size of the candidate queue is unknown. The queue can still be bound in order
 6478   to apply pruning while retrieving the top N but will not throw an exception if
 6479   too many results are rejected to guarantee an absolutely correct top N result.
 6480   The TopNSearcher now returns a struct like class that indicates if the result
 6481   is complete in the sense of the top N or not. Consumers of this API should assert
 6482   on the completeness if the bounded queue size is know ahead of time. (Simon Willnauer)
 6484 * LUCENE-4984: Deprecate ThaiWordFilter and smartcn SentenceTokenizer and WordTokenFilter.
 6485   These filters would not work correctly with CharFilters and could not be safely placed
 6486   at an arbitrary position in the analysis chain. Use ThaiTokenizer and HMMChineseTokenizer
 6487   instead. (Robert Muir)
 6489 * LUCENE-5543: Remove/deprecate Directory.fileExists (Mike McCandless)
 6491 * LUCENE-5573: Move docvalues constants and helper methods to o.a.l.index.DocValues.
 6492   (Dawid Weiss, Robert Muir)
 6494 * LUCENE-5604: Switched BytesRef.hashCode to MurmurHash3 (32 bit).
 6495   TermToBytesRefAttribute.fillBytesRef no longer returns the hash
 6496   code.  BytesRefHash now uses MurmurHash3 for its hashing.  (Robert
 6497   Muir, Mike McCandless)
 6499 Optimizations
 6501 * LUCENE-5468: HunspellStemFilter uses 10 to 100x less RAM. It also loads
 6502   all known openoffice dictionaries without error, and supports an additional 
 6503   longestOnly option for a less aggressive approach.  (Robert Muir)
 6505 * LUCENE-4848: Use Java 7 NIO2-FileChannel instead of RandomAccessFile
 6506   for NIOFSDirectory and MMapDirectory. This allows to delete open files
 6507   on Windows if NIOFSDirectory is used, mmapped files are still locked.
 6508   (Michael Poindexter, Robert Muir, Uwe Schindler)
 6510 * LUCENE-5515: Improved TopDocs#merge to create a merged ScoreDoc
 6511   array with length of at most equal to the specified size instead of length
 6512   equal to at most from + size as was before. (Martijn van Groningen)
 6514 * LUCENE-5529: Spatial search of non-point indexed shapes should be a little
 6515   faster due to skipping intersection tests on redundant cells. (David Smiley)
 6517 Bug fixes
 6519 * LUCENE-5483: Fix inaccuracies in HunspellStemFilter. Multi-stage affix-stripping,
 6520   prefix-suffix dependencies, and COMPLEXPREFIXES now work correctly according
 6521   to the hunspell algorithm. Removed recursionCap parameter, as it's no longer needed, rules for
 6522   recursive affix application are driven correctly by continuation classes in the affix file.
 6523   (Robert Muir)
 6525 * LUCENE-5497: HunspellStemFilter properly handles escaped terms and affixes without conditions.
 6526   (Robert Muir)
 6528 * LUCENE-5505: HunspellStemFilter ignores BOM markers in dictionaries and handles varying
 6529   types of whitespace in SET/FLAG commands. (Robert Muir)
 6531 * LUCENE-5507: Fix HunspellStemFilter loading of dictionaries with large amounts of aliases
 6532   etc before the encoding declaration.  (Robert Muir)
 6534 * LUCENE-5111: Fix WordDelimiterFilter to return offsets in correct order.  (Robert Muir)
 6536 * LUCENE-5555: Fix SortedInputIterator to correctly encode/decode contexts in presence of payload (Areek Zillur)
 6538 * LUCENE-5559: Add missing argument checks to tokenfilters taking
 6539   numeric arguments.  (Ahmet Arslan via Robert Muir)
 6541 * LUCENE-5568: Benchmark module's "default.codec" option didn't work. (David Smiley)
 6543 * SOLR-5983: HTMLStripCharFilter is treating CDATA sections incorrectly.
 6544   (Dan Funk, Steve Rowe)
 6546 * LUCENE-5615: Validate per-segment delete counts at write time, to
 6547   help catch bugs that might otherwise cause corruption (Mike McCandless)
 6549 * LUCENE-5612: NativeFSLockFactory no longer deletes its lock file. This cannot be done
 6550   safely without the risk of deleting someone else's lock file. If you use NativeFSLockFactory,
 6551   you may see write.lock hanging around from time to time: it's harmless.  
 6552   (Uwe Schindler, Mike McCandless, Robert Muir)
 6554 * LUCENE-5624: Ensure NativeFSLockFactory does not leak file handles if it is unable
 6555   to obtain the lock. (Uwe Schindler, Robert Muir)
 6557 * LUCENE-5626: Fix bug in SimpleFSLockFactory's obtain() that sometimes throwed
 6558   IOException (ERROR_ACCESS_DENIED) on Windows if the lock file was created
 6559   concurrently. This error is now handled the same way like in NativeFSLockFactory
 6560   by returning false.  (Uwe Schindler, Robert Muir, Dawid Weiss)
 6562 * LUCENE-5630: Add missing META-INF entry for UpperCaseFilterFactory.
 6563   (Robert Muir)
 6565 Tests
 6567 * LUCENE-5630: Fix TestAllAnalyzersHaveFactories to correctly check for existence
 6568   of class and corresponding Map<String,String> ctor.  (Uwe Schindler, Robert Muir)
 6570 Test Framework
 6572 * LUCENE-5592: Incorrectly reported uncloseable files. (Dawid Weiss)
 6574 * LUCENE-5577: Temporary folder and file management (and cleanup facilities)
 6575   (Mark Miller, Uwe Schindler, Dawid Weiss)
 6577 * LUCENE-5567: When a suite fails with zombie threads failure marker and count 
 6578   is not propagated properly. (Dawid Weiss)
 6580 * LUCENE-5449: Rename _TestUtil and _TestHelper to remove the leading _.
 6582 * LUCENE-5501: Added random out-of-order collection testing (when the collector
 6583   supports it) to AssertingIndexSearcher. (Adrien Grand)
 6585 Build
 6587 * LUCENE-5463: RamUsageEstimator.(human)sizeOf(Object) is now a forbidden API.
 6588   (Adrien Grand, Robert Muir)
 6590 * LUCENE-5512: Remove redundant typing (use diamond operator) throughout
 6591   the codebase.  (Furkan KAMACI via Robert Muir)
 6593 * LUCENE-5614: Enable building on Java 8 using Apache Ant 1.8.3 or 1.8.4
 6594   by adding a workaround for the Ant bug.  (Uwe Schindler)
 6596 * LUCENE-5612: Add a new Ant target in lucene/core to test LockFactory
 6597   implementations: "ant test-lock-factory".  (Uwe Schindler, Mike McCandless,
 6598   Robert Muir)
 6600 Documentation
 6602 * LUCENE-5534: Add javadocs to GreekStemmer methods. 
 6603   (Stamatis Pitsios via Robert Muir)
 6605 ======================= Lucene 4.7.2 =======================
 6607 Bug Fixes
 6609 * LUCENE-5574: Closing a near-real-time reader no longer attempts to
 6610   delete unreferenced files if the original writer has been closed;
 6611   this could cause index corruption in certain cases where index files
 6612   were directly changed (deleted, overwritten, etc.) in the index
 6613   directory outside of Lucene.  (Simon Willnauer, Shai Erera, Robert
 6614   Muir, Mike McCandless)
 6616 * LUCENE-5570: Don't let FSDirectory.sync() create new zero-byte files, instead throw
 6617   exception if a file is missing.  (Uwe Schindler, Mike McCandless, Robert Muir)
 6619 ======================= Lucene 4.7.1 =======================
 6621 Changes in Runtime Behavior
 6623 * LUCENE-5532: AutomatonQuery.equals is no longer implemented as "accepts same language".
 6624   This was inconsistent with hashCode, and unnecessary for any subclasses in Lucene.
 6625   If you desire this in a custom subclass, minimize the automaton.  (Robert Muir)
 6627 Bug Fixes
 6629 * LUCENE-5450: Fix getField() NPE issues with SpanOr/SpanNear when they have an 
 6630   empty list of clauses. This can happen for example,  when a wildcard matches 
 6631   no terms.  (Tim Allison via Robert Muir)
 6633 * LUCENE-5473: Throw IllegalArgumentException, not
 6634   NullPointerException, if the synonym map is empty when creating
 6635   SynonymFilter (帅广应 via Mike McCandless)
 6637 * LUCENE-5432: EliasFanoDocIdSet: Fix number of index entry bits when the maximum
 6638   entry is a power of 2. (Paul Elschot via Adrien Grand)
 6640 * LUCENE-5466: query is always null in countDocsWithClass() of SimpleNaiveBayesClassifier.
 6641   (Koji Sekiguchi)
 6643 * LUCENE-5502: Fixed TermsFilter.equals that could return true for different
 6644   filters. (Igor Motov via Adrien Grand)
 6646 * LUCENE-5522: FacetsConfig didn't add drill-down terms for association facet 
 6647   fields labels. (Shai Erera)
 6649 * LUCENE-5520: ToChildBlockJoinQuery would hit
 6650   ArrayIndexOutOfBoundsException if a parent document had no children
 6651   (Sally Ang via Mike McCandless)
 6653 * LUCENE-5532: AutomatonQuery.hashCode was not thread-safe. (Robert Muir)
 6655 * LUCENE-5525: Implement MultiFacets.getAllDims, so you can do sparse
 6656   facets through DrillSideways, for example.  (Jose Peleteiro, Mike
 6657   McCandless)
 6659 * LUCENE-5481: IndexWriter.forceMerge used to run a merge even if there was a
 6660   single segment in the index. (Adrien Grand, Mike McCandless)
 6662 * LUCENE-5538: Fix FastVectorHighlighter bug with index-time synonyms when the
 6663   query is more complex than a single phrase.  (Robert Muir)
 6665 * LUCENE-5544: Exceptions during IndexWriter.rollback could leak file handles
 6666   and the write lock. (Robert Muir)
 6668 * LUCENE-4978: Spatial RecursivePrefixTree queries could result in false-negatives for
 6669   indexed shapes within 1/2 maxDistErr from the edge of the query shape.  This meant
 6670   searching for a point by the same point as a query rarely worked.  (David Smiley)
 6672 * LUCENE-5553: IndexReader#ReaderClosedListener is not always invoked when 
 6673   IndexReader#close() is called or if refCount is 0. If an exception is 
 6674   thrown during internal close or on any of the close listeners some or all
 6675   listeners might be missed. This can cause memory leaks if the core listeners
 6676   are used to clear caches. (Simon Willnauer)
 6678 Build
 6680 * LUCENE-5511: "ant precommit" / "ant check-svn-working-copy" now work again
 6681   with any working copy format (thanks to svnkit 1.8.4).  (Uwe Schindler)
 6683 ======================= Lucene 4.7.0 =======================
 6685 New Features
 6687 * LUCENE-5336: Add SimpleQueryParser: parser for human-entered queries.
 6688   (Jack Conradson via Robert Muir)
 6690 * LUCENE-5337: Add Payload support to FileDictionary (Suggest) and make it more
 6691   configurable (Areek Zillur via Erick Erickson)
 6693 * LUCENE-5329: suggest: DocumentDictionary and
 6694   DocumentExpressionDictionary are now lenient for dirty documents
 6695   (missing the term, weight or payload).  (Areek Zillur via
 6696   Mike McCandless)
 6698 * LUCENE-5404: Add .getCount method to all suggesters (Lookup); persist count 
 6699   metadata on .store(); Dictionary returns InputIterator; Dictionary.getWordIterator
 6700   renamed to .getEntryIterator. (Areek Zillur)
 6702 * SOLR-1871: The RangeMapFloatFunction accepts an arbitrary ValueSource
 6703   as target and default values. (Chris Harris, shalin)
 6705 * LUCENE-5371: Speed up Lucene range faceting from O(N) per hit to
 6706   O(log(N)) per hit using segment trees; this only really starts to
 6707   matter in practice if the number of ranges is over 10 or so.  (Mike
 6708   McCandless)
 6710 * LUCENE-5379: Add Analyzer for Kurdish.  (Robert Muir)
 6712 * LUCENE-5369: Added an UpperCaseFilter to make UPPERCASE tokens. (ryan)
 6714 * LUCENE-5345: Add a new BlendedInfixSuggester, which is like
 6715   AnalyzingInfixSuggester but boosts suggestions that matched tokens
 6716   with lower positions.  (Remi Melisson via Mike McCandless)
 6718 * LUCENE-5399: When sorting by String (SortField.STRING), you can now
 6719   specify whether missing values should be sorted first (the default),
 6720   using SortField.setMissingValue(SortField.STRING_FIRST), or last,
 6721   using SortField.setMissingValue(SortField.STRING_LAST). (Rob Muir,
 6722   Mike McCandless)
 6724 * LUCENE-5099: QueryNode should have the ability to detach from its node
 6725   parent. Added QueryNode.removeFromParent() that allows nodes to be 
 6726   detached from its parent node. (Adriano Crestani)
 6728 * LUCENE-5395 LUCENE-5451: Upgrade to Spatial4j 0.4.1: Parses WKT (including
 6729   ENVELOPE) with extension "BUFFER"; buffering a point results in a Circle.
 6730   JTS isn't needed for WKT any more but remains required for Polygons. New
 6731   Shapes: ShapeCollection and BufferedLineString. Various other improvements and
 6732   bug fixes too. More info:
 6733   https://github.com/spatial4j/spatial4j/blob/master/CHANGES.md  (David Smiley)
 6735 * LUCENE-5415: Add multitermquery (wildcards,prefix,etc) to PostingsHighlighter.
 6736   (Mike McCandless, Robert Muir)
 6738 * LUCENE-3069: Add two memory resident dictionaries (FST terms dictionary and 
 6739   FSTOrd terms dictionary) to improve primary key lookups. The PostingsBaseFormat 
 6740   API is also changed so that term dictionaries get the ability to block 
 6741   encode term metadata, and all dictionary implementations can now plug in any 
 6742   PostingsBaseFormat. (Han Jiang, Mike McCandless)
 6744 * LUCENE-5353: ShingleFilter's filler token should be configurable.
 6745   (Ahmet Arslan, Simon Willnauer, Steve Rowe)
 6747 * LUCENE-5320: Add SearcherTaxonomyManager over search and taxonomy index
 6748   directories (i.e. not only NRT). (Shai Erera)
 6750 * LUCENE-5410: Add fuzzy and near support via '~' operator to SimpleQueryParser.
 6751   (Lee Hinman via Robert Muir)
 6753 * LUCENE-5426: Make SortedSetDocValuesReaderState abstract to allow
 6754   custom implementations for Lucene doc values faceting (John Wang via
 6755   Mike McCandless)
 6757 * LUCENE-5434: NRT support for file systems that do no have delete on last
 6758   close or cannot delete while referenced semantics. 
 6759   (Mark Miller, Mike McCandless)
 6761 * LUCENE-5418: Drilling down or sideways on a Lucene facet range
 6762   (using Range.getFilter()) is now faster for costly filters (uses
 6763   random access, not iteration); range facet counts now accept a
 6764   fast-match filter to avoid computing the value for documents that
 6765   are out of bounds, e.g. using a bounding box filter with distance
 6766   range faceting.  (Mike McCandless)
 6768 * LUCENE-5440: Add LongBitSet for managing more than 2.1B bits (otherwise use
 6769   FixedBitSet). (Shai Erera)
 6771 * LUCENE-5437: ASCIIFoldingFilter now has an option to preserve the original token
 6772   and emit it on the same position as the folded token only if the actual token was
 6773   folded. (Simon Willnauer, Nik Everett)
 6775 * LUCENE-5408: Add spatial SerializedDVStrategy that serializes a binary
 6776   representations of a shape into BinaryDocValues. It supports exact geometry
 6777   relationship calculations. (David Smiley)
 6779 * LUCENE-5457: Add SloppyMath.earthDiameter(double latitude) that returns an
 6780   approximate value of the diameter of the earth at the given latitude.
 6781   (Adrien Grand)
 6783 * LUCENE-5979: FilteredQuery uses the cost API to decide on whether to use
 6784   random-access or leap-frog to intersect the filter with the query.
 6785   (Adrien Grand)
 6787 Build
 6789 * LUCENE-5217,LUCENE-5420: Maven config: get dependencies from Ant+Ivy config;
 6790   disable transitive dependency resolution for all depended-on artifacts by
 6791   putting an exclusion for each transitive dependency in the
 6792   <dependencyManagement> section of the grandparent POM. (Steve Rowe)
 6794 * LUCENE-5322: Clean up / simplify Maven-related Ant targets.
 6795   (Steve Rowe)
 6797 * LUCENE-5347: Upgrade forbidden-apis checker to version 1.4.
 6798   (Uwe Schindler)
 6800 * LUCENE-4381: Upgrade analysis/icu to 52.1. (Robert Muir)
 6802 * LUCENE-5357: Upgrade StandardTokenizer and UAX29URLEmailTokenizer to
 6803   Unicode 6.3; update UAX29URLEmailTokenizer's recognized top level
 6804   domains in URLs and Emails from the IANA Root Zone Database.
 6805   (Steve Rowe)
 6807 * LUCENE-5360: Add support for developing in Netbeans IDE.
 6808   (Michal Hlavac, Uwe Schindler, Steve Rowe)
 6810 * SOLR-5590: Upgrade HttpClient/HttpComponents to 4.3.x.
 6811   (Karl Wright via Shawn Heisey)
 6813 * LUCENE-5385: "ant precommit" / "ant check-svn-working-copy" now work
 6814   for SVN 1.8 or GIT checkouts. The ANT target prints a warning instead
 6815   of failing. It also instructs the user, how to run on SVN 1.8 working
 6816   copies.  (Robert Muir, Uwe Schindler)
 6818 * LUCENE-5383: fix changes2html to link pull requests (Steve Rowe)
 6820 * LUCENE-5411: Upgrade to released JFlex 1.5.0; stop requiring
 6821   a locally built JFlex snapshot jar. (Steve Rowe)
 6823 * LUCENE-5465: Solr Contrib "map-reduce" breaks Manifest of all other
 6824   JAR files by adding a broken Main-Class attribute.
 6825   (Uwe Schindler, Steve Rowe)
 6827 Bug fixes
 6829 * LUCENE-5285: Improved highlighting of multi-valued fields with
 6830   FastVectorHighlighter. (Nik Everett via Adrien Grand)
 6832 * LUCENE-5391: UAX29URLEmailTokenizer should not tokenize no-scheme
 6833   domain-only URLs that are followed by an alphanumeric character.
 6834   (Chris Geeringh, Steve Rowe)
 6836 * LUCENE-5405: If an analysis component throws an exception, Lucene
 6837   logs the field name to the info stream to assist in
 6838   diagnosis. (Benson Margulies)
 6840 * SOLR-5661: PriorityQueue now refuses to allocate itself if the
 6841   incoming maxSize is too large (Raintung Li via Mike McCandless)
 6843 * LUCENE-5228: IndexWriter.addIndexes(Directory[]) now acquires a
 6844   write lock in each Directory, to ensure that no open IndexWriter is
 6845   changing the incoming indices.  This also means that you cannot pass
 6846   the same Directory to multiple concurrent addIndexes calls (which is
 6847   anyways unusual).  (Robert Muir, Mike McCandless)
 6849 * LUCENE-5415: SpanMultiTermQueryWrapper didn't handle its boost in
 6850   hashcode/equals/tostring/rewrite.  (Robert Muir)
 6852 * LUCENE-5409: ToParentBlockJoinCollector.getTopGroups would fail to
 6853   return any groups when the joined query required more than one
 6854   rewrite step (Peng Cheng via Mike McCandless)
 6856 * LUCENE-5398: NormValueSource was incorrectly casting the long value
 6857   to byte, before calling Similarity.decodeNormValue.  (Peng Cheng via
 6858   Mike McCandless)
 6860 * LUCENE-5436: ReferenceManager#accquire can result in infinite loop if
 6861   managed resource is abused outside of the ReferenceManager. Decrementing
 6862   the reference without a corresponding incRef() call can cause an infinite
 6863   loop. ReferenceManager now throws IllegalStateException if currently managed
 6864   resources ref count is 0. (Simon Willnauer)
 6866 * LUCENE-5443: Lucene45DocValuesProducer.ramBytesUsed() may throw
 6867   ConcurrentModificationException. (Shai Erera, Simon Willnauer)
 6869 * LUCENE-5444: MemoryIndex didn't respect the analyzers offset gap and
 6870   offsets were corrupted if multiple fields with the same name were
 6871   added to the memory index. (Britta Weber, Simon Willnauer)
 6873 * LUCENE-5447: StandardTokenizer should break at consecutive chars matching
 6874   Word_Break = MidLetter, MidNum and/or MidNumLet (Steve Rowe)
 6876 * LUCENE-5462: RamUsageEstimator.sizeOf(Object) is not used anymore to
 6877   estimate memory usage of segments. This used to make
 6878   SegmentReader.ramBytesUsed very CPU-intensive. (Adrien Grand)
 6880 * LUCENE-5461: ControlledRealTimeReopenThread would sometimes wait too
 6881   long (up to targetMaxStaleSec) when a searcher is waiting for a
 6882   specific generation, when it should have waited for at most
 6883   targetMinStaleSec. (Hans Lund via Mike McCandless)
 6885 API Changes
 6887 * LUCENE-5339: The facet module was simplified/reworked to make the
 6888   APIs more approachable to new users. Note: when migrating to the new 
 6889   API, you must pass the Document that is returned from FacetConfig.build() 
 6890   to IndexWriter.addDocument(). (Shai Erera, Gilad Barkai, Rob
 6891   Muir, Mike McCandless)
 6893 * LUCENE-5405: Make ShingleAnalyzerWrapper.getWrappedAnalyzer() public final (gsingers)
 6895 * LUCENE-5395: The SpatialArgsParser now only reads WKT, no more "lat, lon"
 6896   etc. but it's easy to override the parseShape method if you wish. (David
 6897   Smiley)
 6899 * LUCENE-5414: DocumentExpressionDictionary was renamed to 
 6900   DocumentValueSourceDictionary and all dependencies to the lucene-expression
 6901   module were removed from lucene-suggest. DocumentValueSourceDictionary now
 6902   only accepts a ValueSource instead of a convenience ctor for an expression
 6903   string. (Simon Willnauer)
 6905 * LUCENE-3069: PostingsWriterBase and PostingsReaderBase are no longer
 6906   responsible for encoding/decoding a block of terms.  Instead, they
 6907   should encode/decode each term to/from a long[] and byte[].  (Han
 6908   Jiang, Mike McCandless)
 6910 * LUCENE-5425: FacetsCollector and MatchingDocs use a general DocIdSet, 
 6911   allowing for custom implementations to be used when faceting.
 6912   (John Wang, Lei Wang, Shai Erera) 
 6914 Optimizations
 6916 * LUCENE-5372: Replace StringBuffer by StringBuilder, where possible.
 6917   (Joshua Hartman via Uwe Schindler, Dawid Weiss, Mike McCandless)
 6919 * LUCENE-5271: A slightly more accurate SloppyMath distance.
 6920   (Gilad Barkai via Ryan Ernst)
 6922 * LUCENE-5399: Deep paging using IndexSearcher.searchAfter when
 6923   sorting by fields is faster (Rob Muir, Mike McCandless)
 6925 Changes in Runtime Behavior
 6927 * LUCENE-5362: IndexReader and SegmentCoreReaders now throw 
 6928   AlreadyClosedException if the refCount in incremented but
 6929   is less that 1. (Simon Willnauer)
 6931 Documentation
 6933 * LUCENE-5384: Add some tips for making tokenfilters and tokenizers 
 6934   to the analysis package overview.  
 6935   (Benson Margulies via Robert Muir - pull request #12)
 6937 * LUCENE-5389: Add more guidance in the analysis documentation 
 6938   package overview.
 6939   (Benson Margulies via Robert Muir - pull request #14)
 6941 ======================= Lucene 4.6.1 =======================
 6943 Bug fixes
 6945 * LUCENE-5373: Memory usage of
 6946   [Lucene40/Lucene42/Memory/Direct]DocValuesFormat was over-estimated.
 6947   (Shay Banon, Adrien Grand, Robert Muir)
 6949 * LUCENE-5361: Fixed handling of query boosts in FastVectorHighlighter.
 6950   (Nik Everett via Adrien Grand)
 6952 * LUCENE-5374: IndexWriter processes internal events after the it
 6953   closed itself internally. This rare condition can happen if an
 6954   IndexWriter has internal changes that were not fully applied yet
 6955   like when index / flush requests happen concurrently to the close or
 6956   rollback call. (Simon Willnauer)
 6958 * LUCENE-5394: Fix TokenSources.getTokenStream to return payloads if
 6959   they were indexed with the term vectors. (Mike McCandless)
 6961 * LUCENE-5344: Flexible StandardQueryParser behaves differently than
 6962   ClassicQueryParser. (Adriano Crestani)
 6964 * LUCENE-5375: ToChildBlockJoinQuery works harder to detect mis-use,
 6965   when the parent query incorrectly returns child documents, and throw
 6966   a clear exception saying so. (Dr. Oleg Savrasov via Mike McCandless)
 6968 * LUCENE-5401: Field.StringTokenStream#end() calls super.end() now,
 6969   preventing wrong term positions for fields that use
 6970   StringTokenStream. (Michael Busch)
 6972 * LUCENE-5377: IndexWriter.addIndexes(Directory[]) would cause corruption
 6973   on Lucene 4.6 if any index segments were Lucene 4.0-4.5.
 6974   (Littlestar, Mike McCandless, Shai Erera, Robert Muir)
 6976 ======================= Lucene 4.6.0 =======================
 6978 New Features
 6980 * LUCENE-4906: PostingsHighlighter can now render to custom Object,
 6981   for advanced use cases where String is too restrictive (Luca
 6982   Cavanna, Robert Muir, Mike McCandless)
 6984 * LUCENE-5133: Changed AnalyzingInfixSuggester.highlight to return
 6985   Object instead of String, to allow for advanced use cases where
 6986   String is too restrictive (Robert Muir, Shai Erera, Mike
 6987   McCandless)
 6989 * LUCENE-5207, LUCENE-5334: Added expressions module for customizing ranking
 6990   with script-like syntax. 
 6991   (Jack Conradson, Ryan Ernst, Uwe Schindler via Robert Muir)
 6993 * LUCENE-5180: ShingleFilter now creates shingles with trailing holes,
 6994   for example if a StopFilter had removed the last token.  (Mike
 6995   McCandless)
 6997 * LUCENE-5219: Add support to SynonymFilterFactory for custom
 6998   parsers.  (Ryan Ernst via Robert Muir)
 7000 * LUCENE-5235: Tokenizers now throw an IllegalStateException if the
 7001   consumer does not call reset() before consuming the stream. Previous
 7002   versions throwed NullPointerException or ArrayIndexOutOfBoundsException
 7003   on best effort which was not user-friendly.
 7004   (Uwe Schindler, Robert Muir)
 7006 * LUCENE-5240: Tokenizers now throw an IllegalStateException if the
 7007   consumer neglects to call close() on the previous stream before consuming
 7008   the next one. (Uwe Schindler, Robert Muir)
 7010 * LUCENE-5214: Add new FreeTextSuggester, to predict the next word
 7011   using a simple ngram language model.  This is useful for the "long
 7012   tail" suggestions, when a primary suggester fails to find a
 7013   suggestion.  (Mike McCandless)
 7015 * LUCENE-5251: New DocumentDictionary allows building suggesters via
 7016   contents of existing field, weight and optionally payload stored
 7017   fields in an index (Areek Zillur via Mike McCandless)
 7019 * LUCENE-5261: Add QueryBuilder, a simple API to build queries from
 7020   the analysis chain directly, or to make it easier to implement
 7021   query parsers.  (Robert Muir, Uwe Schindler)
 7023 * LUCENE-5270: Add Terms.hasFreqs, to determine whether a given field
 7024   indexed per-doc term frequencies.  (Mike McCandless)
 7026 * LUCENE-5269: Add CodepointCountFilter. (Robert Muir)
 7028 * LUCENE-5294: Suggest module: add DocumentExpressionDictionary to
 7029   compute each suggestion's weight using a javascript expression.
 7030   (Areek Zillur via Mike McCandless)
 7032 * LUCENE-5274: FastVectorHighlighter now supports highlighting against several
 7033   indexed fields. (Nik Everett via Adrien Grand)
 7035 * LUCENE-5304: SingletonSortedSetDocValues can now return the wrapped
 7036   SortedDocValues (Robert Muir, Adrien Grand)
 7038 * LUCENE-2844: The benchmark module can now test the spatial module. See
 7039   spatial.alg  (David Smiley, Liviy Ambrose)
 7041 * LUCENE-5302: Make StemmerOverrideMap's methods public (Alan Woodward)
 7043 * LUCENE-5296: Add DirectDocValuesFormat, which holds all doc values
 7044   in heap as uncompressed java native arrays.  (Mike McCandless)
 7046 * LUCENE-5189: Add IndexWriter.updateNumericDocValues, to update
 7047   numeric DocValues fields of documents, without re-indexing them.
 7048   (Shai Erera, Mike McCandless, Robert Muir)
 7050 * LUCENE-5298: Add SumValueSourceFacetRequest for aggregating facets by
 7051   a ValueSource, such as a NumericDocValuesField or an expression.
 7052   (Shai Erera)
 7054 * LUCENE-5323: Add .sizeInBytes method to all suggesters (Lookup).
 7055   (Areek Zillur via Mike McCandless)
 7057 * LUCENE-5312: Add BlockJoinSorter, a new Sorter implementation that makes sure
 7058   to never split up blocks of documents indexed with IndexWriter.addDocuments.
 7059   (Adrien Grand)
 7061 * LUCENE-5297: Allow to range-facet on any ValueSource, not just
 7062   NumericDocValues fields. (Shai Erera)
 7064 Bug Fixes
 7066 * LUCENE-5272: OpenBitSet.ensureCapacity did not modify numBits, causing 
 7067   false assertion errors in fastSet. (Shai Erera)
 7069 * LUCENE-5303: OrdinalsCache did not use coreCacheKey, resulting in
 7070   over caching across multiple threads. (Mike McCandless, Shai Erera)
 7072 * LUCENE-5307: Fix topScorer inconsistency in handling QueryWrapperFilter
 7073   inside ConstantScoreQuery, which now rewrites to a query removing the
 7074   obsolete QueryWrapperFilter.  (Adrien Grand, Uwe Schindler)
 7076 * LUCENE-5330: IndexWriter didn't process all internal events on 
 7077   #getReader(), #close() and #rollback() which causes files to be
 7078   deleted at a later point in time. This could cause short-term disk
 7079   pollution or OOM if in-memory directories are used. (Simon Willnauer)
 7081 * LUCENE-5342: Fixed bulk-merge issue in CompressingStoredFieldsFormat which
 7082   created corrupted segments when mixing chunk sizes.
 7083   Lucene41StoredFieldsFormat is not impacted. (Adrien Grand, Robert Muir)
 7085 API Changes
 7087 * LUCENE-5222: Add SortField.needsScores(). Previously it was not possible
 7088   for a custom Sort that makes use of the relevance score to work correctly
 7089   with IndexSearcher when an ExecutorService is specified.
 7090   (Ryan Ernst, Mike McCandless, Robert Muir)
 7092 * LUCENE-5275: Change AttributeSource.toString() to display the current
 7093   state of attributes. (Robert Muir)
 7095 * LUCENE-5277: Modify FixedBitSet copy constructor to take an additional
 7096   numBits parameter to allow growing/shrinking the copied bitset. You can
 7097   use FixedBitSet.clone() if you only need to clone the bitset. (Shai Erera)
 7099 * LUCENE-5260: Use TermFreqPayloadIterator for all suggesters; those
 7100   suggesters that can't support payloads will throw an exception if
 7101   hasPayloads() is true.  (Areek Zillur via Mike McCandless)
 7103 * LUCENE-5280: Rename TermFreqPayloadIterator -> InputIterator, along
 7104   with associated suggest/spell classes.  (Areek Zillur via Mike
 7105   McCandless)
 7107 * LUCENE-5157: Rename OrdinalMap methods to clarify API and internal structure.
 7108   (Boaz Leskes via Adrien Grand)
 7110 * LUCENE-5313: Move preservePositionIncrements from setter to ctor in
 7111   Analyzing/FuzzySuggester.  (Areek Zillur via Mike McCandless)
 7113 * LUCENE-5321: Remove Facet42DocValuesFormat. Use DirectDocValuesFormat if you
 7114   want to load the category list into memory. (Shai Erera, Mike McCandless)
 7116 * LUCENE-5324: AnalyzerWrapper.getPositionIncrementGap and getOffsetGap can now
 7117   be overridden. (Adrien Grand)
 7119 Optimizations
 7121 * LUCENE-5225: The ToParentBlockJoinQuery only keeps tracks of the the child
 7122   doc ids and child scores if the ToParentBlockJoinCollector is used.
 7123   (Martijn van Groningen)
 7125 * LUCENE-5236: EliasFanoDocIdSet now has an index and uses broadword bit
 7126   selection to speed-up advance(). (Paul Elschot via Adrien Grand)
 7128 * LUCENE-5266: Improved number of read calls and branches in DirectPackedReader. (Ryan Ernst)
 7130 * LUCENE-5300: Optimized SORTED_SET storage for fields which are single-valued.
 7131   (Adrien Grand)
 7133 Documentation
 7135 * LUCENE-5211: Better javadocs and error checking of 'format' option in 
 7136   StopFilterFactory, as well as comments in all snowball formatted files
 7137   about specifying format option.  (hossman)
 7139 Changes in backwards compatibility policy
 7141 * LUCENE-5235: Sub classes of Tokenizer have to call super.reset()
 7142   when implementing reset(). Otherwise the consumer will get an
 7143   IllegalStateException because the Reader is not correctly assigned.
 7144   It is important to never change the "input" field on Tokenizer
 7145   without using setReader(). The "input" field must not be used
 7146   outside reset(), incrementToken(), or end() - especially not in
 7147   the constructor.  (Uwe Schindler, Robert Muir)
 7149 * LUCENE-5204: Directory doesn't have default implementations for
 7150   LockFactory-related methods, which have been moved to BaseDirectory. If you
 7151   had a custom Directory implementation that extended Directory, you need to
 7152   extend BaseDirectory instead. (Adrien Grand)
 7154 Build
 7156 * LUCENE-5283: Fail the build if ant test didn't execute any tests 
 7157   (everything filtered out). (Dawid Weiss, Uwe Schindler)
 7159 * LUCENE-5249, LUCENE-5257: All Lucene/Solr modules should use the same
 7160   dependency versions. (Steve Rowe)
 7162 * LUCENE-5273: Binary artifacts in Lucene and Solr convenience binary
 7163   distributions accompanying a release, including on Maven Central,
 7164   should be identical across all distributions. (Steve Rowe, Uwe Schindler,
 7165   Shalin Shekhar Mangar)
 7167 * LUCENE-4753: Run forbidden-apis Ant task per module. This allows more
 7168   improvements and prevents OOMs after the number of class files
 7169   raised recently.  (Uwe Schindler)
 7171 Tests
 7173 * LUCENE-5278: Fix MockTokenizer to work better with more regular expression
 7174   patterns. Previously it could only behave like CharTokenizer (where a character
 7175   is either a "word" character or not), but now it gives a general longest-match
 7176   behavior.  (Nik Everett via Robert Muir)
 7178 ======================= Lucene 4.5.1 =======================
 7180 Bug Fixes
 7182 * LUCENE-4998: Fixed a few places to pass IOContext.READONCE instead
 7183   of IOContext.READ (Shikhar Bhushan via Mike McCandless)
 7185 * LUCENE-5242: DirectoryTaxonomyWriter.replaceTaxonomy did not fully reset
 7186   its state, which could result in exceptions being thrown, as well as
 7187   incorrect ordinals returned from getParent. (Shai Erera)
 7189 * LUCENE-5254: Fixed bounded memory leak, where objects like live
 7190   docs bitset were not freed from an starting reader after reopening
 7191   to a new reader and closing the original one.  (Shai Erera, Mike
 7192   McCandless)
 7194 * LUCENE-5262: Fixed file handle leaks when multiple attempts to open an
 7195   NRT reader hit exceptions. (Shai Erera)
 7197 * LUCENE-5263: Transient IOExceptions, e.g. due to disk full or file
 7198   descriptor exhaustion, hit at unlucky times inside IndexWriter could
 7199   lead to silently losing deletions. (Shai Erera, Mike McCandless)
 7201 * LUCENE-5264: CommonTermsQuery ignored minMustMatch if only high-frequent
 7202   terms were present in the query and the high-frequent operator was set
 7203   to SHOULD. (Simon Willnauer)
 7205 * LUCENE-5269: Fix bug in NGramTokenFilter where it would sometimes count
 7206   unicode characters incorrectly. (Mike McCandless, Robert Muir)
 7208 * LUCENE-5289: IndexWriter.hasUncommittedChanges was returning false
 7209   when there were buffered delete-by-Term.  (Shalin Shekhar Mangar,
 7210   Mike McCandless)
 7212 ======================= Lucene 4.5.0 =======================
 7214 New features
 7216 * LUCENE-5084: Added new Elias-Fano encoder, decoder and DocIdSet
 7217   implementations. (Paul Elschot via Adrien Grand)
 7219 * LUCENE-5081: Added WAH8DocIdSet, an in-memory doc id set implementation based
 7220   on word-aligned hybrid encoding. (Adrien Grand)
 7222 * LUCENE-5098: New broadword utility methods in oal.util.BroadWord.
 7223   (Paul Elschot via Adrien Grand, Dawid Weiss)
 7225 * LUCENE-5030: FuzzySuggester now supports optional unicodeAware
 7226   (default is false).  If true then edits are measured in Unicode code
 7227   points instead of UTF8 bytes.  (Artem Lukanin via Mike McCandless)
 7229 * LUCENE-5118: SpatialStrategy.makeDistanceValueSource() now has an optional
 7230   multiplier for scaling degrees to another unit. (David Smiley)
 7232 * LUCENE-5091: SpanNotQuery can now be configured with pre and post slop to act
 7233   as a hypothetical SpanNotNearQuery. (Tim Allison via David Smiley)
 7235 * LUCENE-4985: FacetsAccumulator.create() is now able to create a 
 7236   MultiFacetsAccumulator over a mixed set of facet requests. MultiFacetsAccumulator
 7237   allows wrapping multiple FacetsAccumulators, allowing to easily mix
 7238   existing and custom ones. TaxonomyFacetsAccumulator supports any
 7239   FacetRequest which implements createFacetsAggregator and was indexed
 7240   using the taxonomy index. (Shai Erera)
 7242 * LUCENE-5153: AnalyzerWrapper.wrapReader allows wrapping the Reader given to 
 7243   inputReader. (Shai Erera)
 7245 * LUCENE-5155: FacetRequest.getValueOf and .getFacetArraysSource replaced by
 7246   FacetsAggregator.createOrdinalValueResolver. This gives better options for
 7247   resolving an ordinal's value by FacetAggregators. (Shai Erera)
 7249 * LUCENE-5165: Add SuggestStopFilter, to be used with analyzing
 7250   suggesters, so that a stop word at the very end of the lookup query,
 7251   and without any trailing token characters, will be preserved.  This
 7252   enables query "a" to suggest apple; see 
 7253   http://blog.mikemccandless.com/2013/08/suggeststopfilter-carefully-removes.html
 7254   for details.
 7256 * LUCENE-5178: Added support for missing values to DocValues fields.
 7257   AtomicReader.getDocsWithField returns a Bits of documents with a value,
 7258   and FieldCache.getDocsWithField forwards to that for DocValues fields. Things like 
 7259   SortField.setMissingValue, FunctionValues.exists, and FieldValueFilter now
 7260   work with DocValues fields.  (Robert Muir)
 7262 * LUCENE-5124: Lucene 4.5 has a new Lucene45Codec with Lucene45DocValues,
 7263   supporting missing values and with most datastructures residing off-heap.
 7264   Added "Memory" docvalues format that works entirely in heap, and "Disk"
 7265   loads no datastructures into RAM. Both of these also support missing values.
 7266   Added DiskNormsFormat (in case you want norms entirely on disk).  (Robert Muir)
 7268 * LUCENE-2750: Added PForDeltaDocIdSet, an in-memory doc id set implementation
 7269   based on the PFOR encoding. (Adrien Grand)
 7271 * LUCENE-5186: Added CachingWrapperFilter.getFilter in order to be able to get
 7272   the wrapped filter. (Trejkaz via Adrien Grand)
 7274 * LUCENE-5197: Added SegmentReader.ramBytesUsed to return approximate heap RAM
 7275   used by index datastructures. (Areek Zillur via Robert Muir)
 7277 Bug Fixes
 7279 * LUCENE-5116: IndexWriter.addIndexes(IndexReader...) should drop empty (or all
 7280   deleted) segments. (Robert Muir, Shai Erera)
 7282 * LUCENE-5132: Spatial RecursivePrefixTree Contains predicate will throw an NPE
 7283   when there's no indexed data and maybe in other circumstances too. (David Smiley)
 7285 * LUCENE-5146: AnalyzingSuggester sort comparator read part of the input key as the
 7286   weight that caused the sorter to never sort by weight first since the weight is only
 7287   considered if the input is equal causing the malformed weight to be identical as well.
 7288   (Simon Willnauer)
 7290 * LUCENE-5151: Associations FacetsAggregators could enter an infinite loop when
 7291   some result documents were missing category associations. (Shai Erera)
 7293 * LUCENE-5152: Fix MemoryPostingsFormat to not modify borrowed BytesRef from FSTEnum
 7294   seek/lookup which can cause side effects if done on a cached FST root arc.
 7295   (Simon Willnauer)
 7297 * LUCENE-5160: Handle the case where reading from a file or FileChannel returns -1,
 7298   which could happen in rare cases where something happens to the file between the
 7299   time we start the read loop (where we check the length) and when we actually do
 7300   the read. (gsingers, yonik, Robert Muir, Uwe Schindler)
 7302 * LUCENE-5166: PostingsHighlighter would throw IOOBE if a term spanned the maxLength
 7303   boundary, made it into the top-N and went to the formatter.
 7304   (Manuel Amoabeng, Michael McCandless, Robert Muir)
 7306 * LUCENE-4583: Indexing core no longer enforces a limit on maximum
 7307   length binary doc values fields, but individual codecs (including
 7308   the default one) have their own limits (David Smiley, Robert Muir,
 7309   Mike McCandless)
 7311 * LUCENE-3849: TokenStreams now set the position increment in end(),
 7312   so we can handle trailing holes.  If you have a custom TokenStream
 7313   implementing end() then be sure it calls super.end().  (Robert Muir,
 7314   Mike McCandless)
 7316 * LUCENE-5192: IndexWriter could allow adding same field name with different
 7317   DocValueTypes under some circumstances. (Shai Erera)
 7319 * LUCENE-5191: SimpleHTMLEncoder in Highlighter module broke Unicode
 7320   outside BMP because it encoded UTF-16 chars instead of codepoints.
 7321   The escaping of codepoints > 127 was removed (not needed for valid HTML)
 7322   and missing escaping for ' and / was added.  (Uwe Schindler)
 7324 * LUCENE-5201: Fixed compression bug in LZ4.compressHC when the input is highly
 7325   compressible and the start offset of the array to compress is > 0.
 7326   (Adrien Grand)
 7328 * LUCENE-5221: SimilarityBase did not write norms the same way as DefaultSimilarity
 7329   if discountOverlaps == false and index-time boosts are present for the field.
 7330   (Yubin Kim via Robert Muir)
 7332 * LUCENE-5223: Fixed IndexUpgrader command line parsing: -verbose is not required
 7333   and -dir-impl option now works correctly.  (hossman)
 7335 * LUCENE-5245: Fix MultiTermQuery's constant score rewrites to always
 7336   return a ConstantScoreQuery to make scoring consistent. Previously it
 7337   returned an empty unwrapped BooleanQuery, if no terms were available,
 7338   which has a different query norm.  (Nik Everett, Uwe Schindler)
 7340 * LUCENE-5218: In some cases, trying to retrieve or merge a 0-length
 7341   binary doc value would hit an ArrayIndexOutOfBoundsException.
 7342   (Littlestar via Mike McCandless)
 7344 API Changes
 7346 * LUCENE-5094: Add ramBytesUsed() to MultiDocValues.OrdinalMap.
 7347   (Robert Muir)
 7349 * LUCENE-5114: Remove unused boolean useCache parameter from
 7350   TermsEnum.seekCeil and .seekExact (Mike McCandless)
 7352 * LUCENE-5128: IndexSearcher.searchAfter throws IllegalArgumentException if 
 7353   searchAfter exceeds the number of documents in the reader. 
 7354   (Crocket via Shai Erera)
 7356 * LUCENE-5129: CategoryAssociationsContainer no longer supports null
 7357   association values for categories. If you want to index categories without
 7358   associations, you should add them using FacetFields. (Shai Erera)
 7360 * LUCENE-4876: IndexWriter no longer clones the given IndexWriterConfig. If you
 7361   need to use the same config more than once, e.g. when sharing between multiple 
 7362   writers, make sure to clone it before passing to each writer.
 7363   (Shai Erera, Mike McCandless)
 7365 * LUCENE-5144: StandardFacetsAccumulator renamed to OldFacetsAccumulator, and all
 7366   associated classes were moved under o.a.l.facet.old. The intention to remove it
 7367   one day, when the features it covers (complements, partitions, sampling) will be
 7368   migrated to the new FacetsAggregator and FacetsAccumulator API. Also,
 7369   FacetRequest.createAggregator was replaced by OldFacetsAccumulator.createAggregator.
 7370   (Shai Erera)
 7372 * LUCENE-5149: CommonTermsQuery now allows to set the minimum number of terms that 
 7373   should match for its high and low frequent sub-queries. Previously this was only
 7374   supported on the low frequent terms query. (Simon Willnauer)  
 7376 * LUCENE-5156: CompressingTermVectors TermsEnum no longer supports ord().
 7377   (Robert Muir)
 7379 * LUCENE-5161, LUCENE-5164: Fix default chunk sizes in FSDirectory to not be
 7380   unnecessarily large (now 8192 bytes); also use chunking when writing to index
 7381   files. FSDirectory#setReadChunkSize() is now deprecated and will be removed
 7382   in Lucene 5.0.  (Uwe Schindler, Robert Muir, gsingers)
 7384 * LUCENE-5170: Analyzer.ReuseStrategy instances are now stateless and can
 7385   be reused in other Analyzer instances, which was not possible before.
 7386   Lucene ships now with stateless singletons for per field and global reuse.
 7387   Legacy code can still instantiate the deprecated implementation classes,
 7388   but new code should use the constants. Implementors of custom strategies
 7389   have to take care of new method signatures. AnalyzerWrapper can now be
 7390   configured to use a custom strategy, too, ideally the one from the wrapped
 7391   Analyzer. Analyzer adds a getter to retrieve the strategy for this use-case.
 7392   (Uwe Schindler, Robert Muir, Shay Banon)
 7394 * LUCENE-5173: Lucene never writes segments with 0 documents anymore.
 7395   (Shai Erera, Uwe Schindler, Robert Muir)
 7397 * LUCENE-5178: SortedDocValues always returns -1 ord when a document is missing
 7398   a value for the field. Previously it only did this if the SortedDocValues
 7399   was produced by uninversion on the FieldCache.  (Robert Muir)
 7401 * LUCENE-5183: remove BinaryDocValues.MISSING. In order to determine a document
 7402   is missing a field, use getDocsWithField instead.  (Robert Muir)
 7404 Changes in Runtime Behavior
 7406 * LUCENE-5178: DocValues codec consumer APIs (iterables) return null values 
 7407   when the document has no value for the field. (Robert Muir)
 7409 * LUCENE-5200: The HighFreqTerms command-line tool returns the true top-N
 7410   by totalTermFreq when using the -t option, it uses the term statistics (faster)
 7411   and now always shows totalTermFreq in the output.  (Robert Muir)
 7413 Optimizations
 7415 * LUCENE-5088: Added TermFilter to filter docs by a specific term.
 7416   (Martijn van Groningen)
 7418 * LUCENE-5119: DiskDV keeps the document-to-ordinal mapping on disk for 
 7419   SortedDocValues.  (Robert Muir)
 7421 * LUCENE-5145: New AppendingPackedLongBuffer, a new variant of the former
 7422   AppendingLongBuffer which assumes values are 0-based.
 7423   (Boaz Leskes via Adrien Grand)
 7425 * LUCENE-5145: All Appending*Buffer now support bulk get.
 7426   (Boaz Leskes via Adrien Grand)
 7428 * LUCENE-5140: Fixed a performance regression of span queries caused by
 7429   LUCENE-4946. (Alan Woodward, Adrien Grand)
 7431 * LUCENE-5150: Make WAH8DocIdSet able to inverse its encoding in order to
 7432   compress dense sets efficiently as well. (Adrien Grand)
 7434 * LUCENE-5159: Prefix-code the sorted/sortedset value dictionaries in DiskDV.
 7435   (Robert Muir)
 7437 * LUCENE-5170: Fixed several wrapper analyzers to inherit the reuse strategy
 7438   of the wrapped Analyzer.  (Uwe Schindler, Robert Muir, Shay Banon)
 7440 * LUCENE-5006: Simplified DocumentsWriter and DocumentsWriterPerThread
 7441   synchronization and concurrent interaction with IndexWriter. DWPT is now
 7442   only setup once and has no reset logic. All segment publishing and state
 7443   transition from DWPT into IndexWriter is now done via an Event-Queue
 7444   processed from within the IndexWriter in order to prevent situations
 7445   where DWPT or DW calling int IW causing deadlocks. (Simon Willnauer)
 7447 * LUCENE-5182: Terminate phrase searches early if max phrase window is 
 7448   exceeded in FastVectorHighlighter to prevent very long running phrase
 7449   extraction if phrase terms are high frequent. (Simon Willnauer)
 7451 * LUCENE-5188: CompressingStoredFieldsFormat now slices chunks containing big
 7452   documents into fixed-size blocks so that requesting a single field does not
 7453   necessarily force to decompress the whole chunk. (Adrien Grand)
 7455 * LUCENE-5101: CachingWrapper makes it easier to plug-in a custom cacheable
 7456   DocIdSet implementation and uses WAH8DocIdSet by default, which should be
 7457   more memory efficient than FixedBitSet on average as well as faster on small
 7458   sets. (Robert Muir)
 7460 Documentation
 7462 * LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
 7463   package's documentation and classes javadocs. (Shai Erera)
 7465 * LUCENE-5206: Clarify FuzzyQuery's unexpected behavior on short
 7466   terms. (Tim Allison via Mike McCandless)
 7468 Changes in backwards compatibility policy
 7470 * LUCENE-5141: CheckIndex.fixIndex(Status,Codec) is now
 7471   CheckIndex.fixIndex(Status). If you used to pass a codec to this method, just
 7472   remove it from the arguments. (Adrien Grand)
 7474 * LUCENE-5089, SOLR-5126: Update to Morfologik 1.7.1. MorfologikAnalyzer and MorfologikFilter 
 7475   no longer support multiple "dictionaries" as there is only one dictionary available.
 7476   (Dawid Weiss)
 7478 * LUCENE-5170: Changed method signatures of Analyzer.ReuseStrategy to take
 7479   Analyzer. Closeable interface was removed because the class was changed to
 7480   be stateless.  (Uwe Schindler, Robert Muir, Shay Banon)
 7482 * LUCENE-5187: SlowCompositeReaderWrapper constructor is now private,
 7483   SlowCompositeReaderWrapper.wrap should be used instead. (Adrien Grand)
 7485 * LUCENE-5101: CachingWrapperFilter doesn't always return FixedBitSet instances
 7486   anymore. Users of the join module can use
 7487   oal.search.join.FixedBitSetCachingWrapperFilter instead. (Adrien Grand)
 7489 Build
 7491 * SOLR-5159: Manifest includes non-parsed maven variables. 
 7492   (Artem Karpenko via Steve Rowe)
 7494 * LUCENE-5193: Add jar-src as top-level target to generate all Lucene and Solr
 7495   *-src.jar. (Steve Rowe, Shai Erera)
 7497 ======================= Lucene 4.4.0 =======================
 7499 Changes in backwards compatibility policy
 7501 * LUCENE-5085: MorfologikFilter will no longer stem words marked as keywords
 7502   (Dawid Weiss, Grzegorz Sobczyk)
 7504 * LUCENE-4955: NGramTokenFilter now emits all n-grams for the same token at the
 7505   same position and preserves the position length and the offsets of the
 7506   original token. (Simon Willnauer, Adrien Grand)
 7508 * LUCENE-4955: NGramTokenizer now emits n-grams in a different order
 7509   (a, ab, b, bc, c) instead of (a, b, c, ab, bc) and doesn't trim trailing
 7510   whitespaces. (Adrien Grand)
 7512 * LUCENE-5042: The n-gram and edge n-gram tokenizers and filters now correctly
 7513   handle supplementary characters, and the tokenizers have the ability to
 7514   pre-tokenize the input stream similarly to CharTokenizer. (Adrien Grand)
 7516 * LUCENE-4967: NRTManager is replaced by
 7517   ControlledRealTimeReopenThread, for controlling which requests must
 7518   see which indexing changes, so that it can work with any
 7519   ReferenceManager (Mike McCandless)
 7521 * LUCENE-4973: SnapshotDeletionPolicy no longer requires a unique
 7522   String id (Mike McCandless, Shai Erera)
 7524 * LUCENE-4946: The internal sorting API (SorterTemplate, now Sorter) has been
 7525   completely refactored to allow for a better implementation of TimSort.
 7526   (Adrien Grand, Uwe Schindler, Dawid Weiss)
 7528 * LUCENE-4963: Some TokenFilter options that generate broken TokenStreams have
 7529   been deprecated: updateOffsets=true on TrimFilter and
 7530   enablePositionIncrements=false on all classes that inherit from
 7531   FilteringTokenFilter: JapanesePartOfSpeechStopFilter, KeepWordFilter,
 7532   LengthFilter, StopFilter and TypeTokenFilter. (Adrien Grand)
 7534 * LUCENE-4963: In order not to take position increments into account in
 7535   suggesters, you now need to call setPreservePositionIncrements(false) instead
 7536   of configuring the token filters to not increment positions. (Adrien Grand)
 7538 * LUCENE-3907: EdgeNGramTokenizer now supports maxGramSize > 1024, doesn't trim
 7539   the input, sets position increment = 1 for all tokens and doesn't support
 7540   backward grams anymore. (Adrien Grand)
 7542 * LUCENE-3907: EdgeNGramTokenFilter does not support backward grams and does
 7543   not update offsets anymore. (Adrien Grand)
 7545 * LUCENE-4981: PositionFilter is now deprecated as it can corrupt token stream
 7546   graphs. Since it main use-case was to make query parsers generate boolean
 7547   queries instead of phrase queries, it is now advised to use
 7548   QueryParser.setAutoGeneratePhraseQueries(false) (for simple cases) or to
 7549   override QueryParser.newFieldQuery. (Adrien Grand, Steve Rowe)
 7551 * LUCENE-5018: CompoundWordTokenFilterBase and its children
 7552   DictionaryCompoundWordTokenFilter and HyphenationCompoundWordTokenFilter don't
 7553   update offsets anymore. (Adrien Grand)
 7555 * LUCENE-5015: SamplingAccumulator no longer corrects the counts of the sampled 
 7556   categories. You should set TakmiSampleFixer on SamplingParams if required (but 
 7557   notice that this means slower search). (Rob Audenaerde, Gilad Barkai, Shai Erera)
 7559 * LUCENE-4933: Replace ExactSimScorer/SloppySimScorer with just SimScorer. Previously
 7560   there were 2 implementations as a performance hack to support tableization of
 7561   sqrt(), but this caching is removed, as sqrt is implemented in hardware with modern 
 7562   jvms and it's faster not to cache.  (Robert Muir)
 7564 * LUCENE-5038: MergePolicy now has a default implementation for useCompoundFile based
 7565   on segment size and noCFSRatio. The default implementation was pulled up from
 7566   TieredMergePolicy. (Simon Willnauer)
 7568 * LUCENE-5063: FieldCache.get(Bytes|Shorts), SortField.Type.(BYTE|SHORT) and
 7570   deprecated. These methods/types assume that data is stored as strings although
 7571   Lucene has much better support for numeric data through (Int|Long)Field,
 7572   NumericRangeQuery and FieldCache.get(Int|Long)s. (Adrien Grand)
 7574 * LUCENE-5078: TfIDFSimilarity lets you encode the norm value as any arbitrary long.
 7575   As a result, encode/decodeNormValue were made abstract with their signatures changed.
 7576   The default implementation was moved to DefaultSimilarity, which encodes the norm as
 7577   a single-byte value. (Shai Erera)
 7579 Bug Fixes
 7581 * LUCENE-4890: QueryTreeBuilder.getBuilder() only finds interfaces on the 
 7582   most derived class. (Adriano Crestani)
 7584 * LUCENE-4997: Internal test framework's tests are sensitive to previous 
 7585   test failures and tests.failfast. (Dawid Weiss, Shai Erera)
 7587 * LUCENE-4955: NGramTokenizer now supports inputs larger than 1024 chars.
 7588   (Adrien Grand)
 7590 * LUCENE-4959: Fix incorrect return value in
 7591   SimpleNaiveBayesClassifier.assignClass. (Alexey Kutin via Adrien Grand)
 7593 * LUCENE-4972: DirectoryTaxonomyWriter created empty commits even if no changes
 7594   were made. (Shai Erera, Michael McCandless)
 7596 * LUCENE-949: AnalyzingQueryParser can't work with leading wildcards.
 7597   (Tim Allison, Robert Muir, Steve Rowe)
 7599 * LUCENE-4980: Fix issues preventing mixing of RangeFacetRequest and
 7600   non-RangeFacetRequest when using DrillSideways.  (Mike McCandless,
 7601   Shai Erera)
 7603 * LUCENE-4996: Ensure DocInverterPerField always includes field name
 7604   in exception messages.  (Markus Jelsma via Robert Muir)
 7606 * LUCENE-4992: Fix constructor of CustomScoreQuery to take FunctionQuery
 7607   for scoringQueries. Instead use QueryValueSource to safely wrap arbitrary 
 7608   queries and use them with CustomScoreQuery.  (John Wang, Robert Muir)
 7610 * LUCENE-5016: SamplingAccumulator returned inconsistent label if asked to
 7611   aggregate a non-existing category. Also fixed a bug in RangeAccumulator if
 7612   some readers did not have the requested numeric DV field.
 7613   (Rob Audenaerde, Shai Erera)
 7615 * LUCENE-5028: Remove pointless and confusing doShare option in FST's
 7616   PositiveIntOutputs (Han Jiang via Mike McCandless)
 7618 * LUCENE-5032: Fix IndexOutOfBoundsExc in PostingsHighlighter when
 7619   multi-valued fields exceed maxLength (Tomás Fernández Löbbe
 7620   via Mike McCandless)
 7622 * LUCENE-4933: SweetSpotSimilarity didn't apply its tf function to some
 7623   queries (SloppyPhraseQuery, SpanQueries).  (Robert Muir)
 7625 * LUCENE-5033: SlowFuzzyQuery was accepting too many terms (documents) when
 7626   provided minSimilarity is an int > 1 (Tim Allison via Mike McCandless)
 7628 * LUCENE-5045: DrillSideways.search did not work on an empty index. (Shai Erera)
 7630 * LUCENE-4995: CompressingStoredFieldsReader now only reuses an internal buffer
 7631   when there is no more than 32kb to decompress. This prevents from running
 7632   into out-of-memory errors when working with large stored fields.
 7633   (Adrien Grand)
 7635 * LUCENE-5062: If the spatial data for a document was comprised of multiple
 7636   overlapping or adjacent parts then a CONTAINS predicate query might not match
 7637   when the sum of those shapes contain the query shape but none do individually.
 7638   A flag was added to use the original faster algorithm. (David Smiley)
 7640 * LUCENE-4971: Fixed NPE in AnalyzingSuggester when there are too many
 7641   graph expansions.  (Alexey Kudinov via Mike McCandless)
 7643 * LUCENE-5080: Combined setMaxMergeCount and setMaxThreadCount into one
 7644   setter in ConcurrentMergePolicy: setMaxMergesAndThreads.  Previously these
 7645   setters would not work unless you invoked them very carefully.
 7646   (Robert Muir, Shai Erera)
 7648 * LUCENE-5068: QueryParserUtil.escape() does not escape forward slash.
 7649   (Matias Holte via Steve Rowe)
 7651 * LUCENE-5103: A join on A single-valued field with deleted docs scored too few
 7652   docs. (David Smiley)
 7654 * LUCENE-5090: Detect mismatched readers passed to
 7655   SortedSetDocValuesReaderState and SortedSetDocValuesAccumulator.
 7656   (Robert Muir, Mike McCandless)
 7658 * LUCENE-5120: AnalyzingSuggester modified its FST's cached root arc if payloads
 7659   are used and the entire output resided on the root arc on the first access. This
 7660   caused subsequent suggest calls to fail. (Simon Willnauer)
 7662 Optimizations
 7664 * LUCENE-4936: Improve numeric doc values compression in case all values share
 7665   a common divisor. In particular, this improves the compression ratio of dates
 7666   without time when they are encoded as milliseconds since Epoch. Also support
 7667   TABLE compressed numerics in the Disk codec.  (Robert Muir, Adrien Grand)
 7669 * LUCENE-4951: DrillSideways uses the new Scorer.cost() method to make
 7670   better decisions about which scorer to use internally.  (Mike McCandless)
 7672 * LUCENE-4976: PersistentSnapshotDeletionPolicy writes its state to a
 7673   single snapshots_N file, and no longer requires closing (Mike
 7674   McCandless, Shai Erera)
 7676 * LUCENE-5035: Compress addresses in FieldCacheImpl.SortedDocValuesImpl more
 7677   efficiently. (Adrien Grand, Robert Muir)
 7679 * LUCENE-4941: Sort "from" terms only once when using JoinUtil.
 7680   (Martijn van Groningen)
 7682 * LUCENE-5050: Close the stored fields and term vectors index files as soon as
 7683   the index has been loaded into memory to save file descriptors. (Adrien Grand)
 7685 * LUCENE-5086: RamUsageEstimator now uses official Java 7 API or a proprietary
 7686   Oracle Java 6 API to get Hotspot MX bean, preventing AWT classes to be
 7687   loaded on MacOSX.  (Shay Banon, Dawid Weiss, Uwe Schindler)
 7689 New Features
 7691 * LUCENE-5085: MorfologikFilter will no longer stem words marked as keywords
 7692   (Dawid Weiss, Grzegorz Sobczyk)
 7694 * LUCENE-5064: Added PagedMutable (internal), a paged extension of
 7695   PackedInts.Mutable which allows for storing more than 2B values. (Adrien Grand)
 7697 * LUCENE-4766: Added a PatternCaptureGroupTokenFilter that uses Java regexes to 
 7698   emit multiple tokens one for each capture group in one or more patterns.
 7699   (Simon Willnauer, Clinton Gormley)
 7701 * LUCENE-4952: Expose control (protected method) in DrillSideways to
 7702   force all sub-scorers to be on the same document being collected.
 7703   This is necessary when using collectors like
 7704   ToParentBlockJoinCollector with DrillSideways.  (Mike McCandless)
 7706 * SOLR-4761: Add SimpleMergedSegmentWarmer, which just initializes terms,
 7707   norms, docvalues, and so on. (Mark Miller, Mike McCandless, Robert Muir)
 7709 * LUCENE-4964: Allow arbitrary Query for per-dimension drill-down to
 7710   DrillDownQuery and DrillSideways, to support future dynamic faceting
 7711   methods (Mike McCandless)
 7713 * LUCENE-4966: Add CachingWrapperFilter.sizeInBytes() (Mike McCandless)
 7715 * LUCENE-4965: Add dynamic (no taxonomy index used) numeric range
 7716   faceting to Lucene's facet module (Mike McCandless, Shai Erera)
 7718 * LUCENE-4979: LiveFieldFields can work with any ReferenceManager, not
 7719   just ReferenceManager<IndexSearcher> (Mike McCandless).
 7721 * LUCENE-4975: Added a new Replicator module which can replicate index 
 7722   revisions between server and client. (Shai Erera, Mike McCandless)
 7724 * LUCENE-5022: Added FacetResult.mergeHierarchies to merge multiple
 7725   FacetResult of the same dimension into a single one with the reconstructed
 7726   hierarchy. (Shai Erera)
 7728 * LUCENE-5026: Added PagedGrowableWriter, a new internal packed-ints structure
 7729   that grows the number of bits per value on demand, can store more than 2B
 7730   values and supports random write and read access. (Adrien Grand)
 7732 * LUCENE-5025: FST's Builder can now handle more than 2.1 billion
 7733   "tail nodes" while building a minimal FST.  (Aaron Binns, Adrien
 7734   Grand, Mike McCandless)
 7736 * LUCENE-5063: FieldCache.DEFAULT.get(Ints|Longs) now uses bit-packing to save
 7737   memory. (Adrien Grand)
 7739 * LUCENE-5079: IndexWriter.hasUncommittedChanges() returns true if there are
 7740   changes that have not been committed. (yonik, Mike McCandless, Uwe Schindler)
 7742 * SOLR-4565: Extend NorwegianLightStemFilter and NorwegianMinimalStemFilter 
 7743   to handle "nynorsk" (Erlend Garåsen, janhoy via Robert Muir)
 7745 * LUCENE-5087: Add getMultiValuedSeparator to PostingsHighlighter, for cases
 7746   where you want a different logical separator between field values. This can
 7747   be set to e.g. U+2029 PARAGRAPH SEPARATOR if you never want passes to span
 7748   values. (Mike McCandless, Robert Muir)
 7750 * LUCENE-5013: Added ScandinavianFoldingFilterFactory and
 7751   ScandinavianNormalizationFilterFactory (Karl Wettin via janhoy)
 7753 * LUCENE-4845: AnalyzingInfixSuggester finds suggestions based on
 7754   matches to any tokens in the suggestion, not just based on pure
 7755   prefix matching.  (Mike McCandless, Robert Muir)
 7757 API Changes
 7759 * LUCENE-5077: Make it easier to use compressed norms. Lucene42NormsFormat takes
 7760   an overhead parameter, so you can easily pass a different value other than
 7761   PackedInts.FASTEST from your own codec.  (Robert Muir)
 7763 * LUCENE-5097: Analyzer now has an additional tokenStream(String fieldName,
 7764   String text) method, so wrapping by StringReader for common use is no
 7765   longer needed. This method uses an internal reusable reader, which was
 7766   previously only used by the Field class.  (Uwe Schindler, Robert Muir)
 7768 * LUCENE-4542: HunspellStemFilter's maximum recursion level is now configurable.
 7769   (Piotr, Rafał Kuć via Adrien Grand)
 7771 Build
 7773 * LUCENE-4987: Upgrade randomized testing to version 2.0.10: 
 7774   Test framework may fail internally due to overly aggressive J9 optimizations. 
 7775   (Dawid Weiss, Shai Erera)
 7777 * LUCENE-5043: The eclipse target now uses the containing directory for the
 7778   project name.  This also enforces UTF-8 encoding when files are copied with
 7779   filtering.
 7781 * LUCENE-5055: "rat-sources" target now checks also build.xml, ivy.xml,
 7782   forbidden-api signatures, and parts of resources folders.  (Ryan Ernst,
 7783   Uwe Schindler)
 7785 * LUCENE-5072: Automatically patch javadocs generated by JDK versions
 7786   before 7u25 to work around the frame injection vulnerability (CVE-2013-1571,
 7787   VU#225657).  (Uwe Schindler)
 7789 Tests
 7791 * LUCENE-4901: TestIndexWriterOnJRECrash should work on any 
 7792   JRE vendor via Runtime.halt().
 7793   (Mike McCandless, Robert Muir, Uwe Schindler, Rodrigo Trujillo, Dawid Weiss)
 7795 Changes in runtime behavior
 7797 * LUCENE-5038: New segments written by IndexWriter are now wrapped into CFS
 7798   by default. DocumentsWriterPerThread doesn't consult MergePolicy anymore 
 7799   to decide if a CFS must be written, instead IndexWriterConfig now has a
 7800   property to enable / disable CFS for newly created segments. (Simon Willnauer)
 7802 * LUCENE-5107: Properties files by Lucene are now written in UTF-8 encoding,
 7803   Unicode is no longer escaped. Reading of legacy properties files with
 7804   \u escapes is still possible.  (Uwe Schindler, Robert Muir)
 7806 ======================= Lucene 4.3.1 =======================
 7808 Bug Fixes
 7810 * SOLR-4813: Fix SynonymFilterFactory to allow init parameters for
 7811   tokenizer factory used when parsing synonyms file.  (Shingo Sasaki, hossman)
 7813 * LUCENE-4935: CustomScoreQuery wrongly applied its query boost twice
 7814   (boost^2).  (Robert Muir)
 7816 * LUCENE-4948: Fixed ArrayIndexOutOfBoundsException in PostingsHighlighter
 7817   if you had a 64-bit JVM without compressed OOPS: IBM J9, or Oracle with
 7818   large heap/explicitly disabled.  (Mike McCandless, Uwe Schindler, Robert Muir)
 7820 * LUCENE-4953: Fixed ParallelCompositeReader to inform ReaderClosedListeners of
 7821   its synthetic subreaders. FieldCaches keyed on the atomic children will be purged
 7822   earlier and FC insanity prevented.  In addition, ParallelCompositeReader's
 7823   toString() was changed to better reflect the reader structure.
 7824   (Mike McCandless, Uwe Schindler)
 7826 * LUCENE-4968: Fixed ToParentBlockJoinQuery/Collector: correctly handle parent
 7827   hits that had no child matches, don't throw IllegalArgumentEx when
 7828   the child query has no hits, more aggressively catch cases where childQuery
 7829   incorrectly matches parent documents (Mike McCandless)
 7831 * LUCENE-4970: Fix boost value of rewritten NGramPhraseQuery.
 7832   (Shingo Sasaki via Adrien Grand)
 7834 * LUCENE-4974: CommitIndexTask was broken if no params were set. (Shai Erera)
 7836 * LUCENE-4986: Fixed case where a newly opened near-real-time reader
 7837   fails to reflect a delete from IndexWriter.tryDeleteDocument (Reg,
 7838   Mike McCandless)
 7840 * LUCENE-4994: Fix PatternKeywordMarkerFilter to have public constructor.
 7841   (Uwe Schindler)
 7843 * LUCENE-4993: Fix BeiderMorseFilter to preserve custom attributes when
 7844   inserting tokens with position increment 0.  (Uwe Schindler)
 7846 * LUCENE-4991: Fix handling of synonyms in classic QueryParser.getFieldQuery for
 7847   terms not separated by whitespace. PositionIncrementAttribute was ignored, so with
 7848   default AND synonyms wrongly became mandatory clauses, and with OR, the
 7849   coordination factor was wrong.  (李威, Robert Muir)
 7851 * LUCENE-5002: IndexWriter#deleteAll() caused a deadlock in DWPT / DWSC if a
 7852   DwPT was flushing concurrently while deleteAll() aborted all DWPT. The IW
 7853   should never wait on DWPT via the flush control while holding on to the IW
 7854   Lock. (Simon Willnauer)
 7856 Optimizations
 7858 * LUCENE-4938: Don't use an unnecessarily large priority queue in IndexSearcher
 7859   methods that take top-N.  (Uwe Schindler, Mike McCandless, Robert Muir)
 7862 ======================= Lucene 4.3.0 =======================
 7864 Changes in backwards compatibility policy
 7866 * LUCENE-4810: EdgeNGramTokenFilter no longer increments position for
 7867   multiple ngrams derived from the same input token. (Walter Underwood
 7868   via Mike McCandless)
 7870 * LUCENE-4822: KeywordTokenFilter is now an abstract class. Subclasses
 7871   need to implement #isKeyword() in order to mark terms as keywords. 
 7872   The existing functionality has been factored out into a new 
 7873   SetKeywordTokenFilter class. (Simon Willnauer, Uwe Schindler)
 7875 * LUCENE-4642: Remove Tokenizer's and subclasses' ctors taking
 7876   AttributeSource. (Renaud Delbru, Uwe Schindler, Steve Rowe)
 7878 * LUCENE-4833: IndexWriterConfig used to use LogByteSizeMergePolicy when
 7879   calling setMergePolicy(null) although the default merge policy is
 7880   TieredMergePolicy. IndexWriterConfig setters now throw an exception when
 7881   passed null if null is not a valid value. (Adrien Grand)
 7883 * LUCENE-4849: Made ParallelTaxonomyArrays abstract with a concrete
 7884   implementation for DirectoryTaxonomyWriter/Reader. Also moved it under
 7885   o.a.l.facet.taxonomy. (Shai Erera)
 7887 * LUCENE-4876: IndexDeletionPolicy is now an abstract class instead of an
 7888   interface. IndexDeletionPolicy, MergeScheduler and InfoStream now implement
 7889   Cloneable. (Adrien Grand)
 7891 * LUCENE-4874: FilterAtomicReader and related classes (FilterTerms,
 7892   FilterDocsEnum, ...) don't forward anymore to the filtered instance when the
 7893   method has a default implementation through other abstract methods.
 7894   (Adrien Grand, Robert Muir)
 7896 * LUCENE-4642, LUCENE-4877: Implementors of TokenizerFactory, TokenFilterFactory,
 7897   and CharFilterFactory now need to provide at least one constructor taking
 7898   Map<String,String> to be able to be loaded by the SPI framework (e.g., from Solr).
 7899   In addition, TokenizerFactory needs to implement the abstract
 7900   create(AttributeFactory,Reader) method.  (Renaud Delbru, Uwe Schindler,
 7901   Steve Rowe, Robert Muir)
 7903 API Changes
 7905 * LUCENE-4896: Made PassageFormatter abstract in PostingsHighlighter, made
 7906   members of DefaultPassageFormatter protected.  (Luca Cavanna via Robert Muir)
 7908 * LUCENE-4844: removed TaxonomyReader.getParent(), you should use
 7909   TaxonomyReader.getParallelArrays().parents() instead. (Shai Erera)
 7911 * LUCENE-4742: Renamed spatial 'Node' to 'Cell', along with any method names
 7912   and variables using this terminology. (David Smiley)
 7914 New Features
 7916 * LUCENE-4815: DrillSideways now allows more than one FacetRequest per
 7917   dimension (Mike McCandless)
 7919 * LUCENE-3918: IndexSorter has been ported to 4.3 API and now supports
 7920   sorting documents by a numeric DocValues field, or reverse the order of
 7921   the documents in the index. Additionally, apps can implement their own
 7922   sort criteria. (Anat Hashavit, Shai Erera)
 7924 * LUCENE-4817: Added KeywordRepeatFilter that allows to emit a token twice
 7925   once as a keyword and once as an ordinary token allow stemmers to emit
 7926   a stemmed version along with the un-stemmed version. (Simon Willnauer)
 7928 * LUCENE-4822: PatternKeywordTokenFilter can mark tokens as keywords based
 7929   on regular expressions. (Simon Willnauer, Uwe Schindler)
 7931 * LUCENE-4821: AnalyzingSuggester now uses the ending offset to
 7932   determine whether the last token was finished or not, so that a
 7933   query "i " will no longer suggest "Isla de Muerta" for example.
 7934   (Mike McCandless)
 7936 * LUCENE-4642: Add create(AttributeFactory) to TokenizerFactory and
 7937   subclasses with ctors taking AttributeFactory. 
 7938   (Renaud Delbru, Uwe Schindler, Steve Rowe)
 7940 * LUCENE-4820: Add payloads to Analyzing/FuzzySuggester, to record an
 7941   arbitrary byte[] per suggestion (Mike McCandless)
 7943 * LUCENE-4816: Add WholeBreakIterator to PostingsHighlighter
 7944   for treating the entire content as a single Passage.  (Robert
 7945   Muir, Mike McCandless)
 7947 * LUCENE-4827: Add additional ctor to PostingsHighlighter PassageScorer
 7948   to provide bm25 k1,b,avgdl parameters. (Robert Muir)
 7950 * LUCENE-4607: Add DocIDSetIterator.cost() and Spans.cost() for optimizing
 7951   scoring.  (Simon Willnauer, Robert Muir)
 7953 * LUCENE-4795: Add SortedSetDocValuesFacetFields and
 7954   SortedSetDocValuesAccumulator, to compute topK facet counts from a
 7955   field's SortedSetDocValues.  This method only supports flat
 7956   (dim/label) facets, is a bit (~25%) slower, has added cost
 7957   per-IndexReader-open to compute its ordinal map, but it requires no
 7958   taxonomy index and it tie-breaks facet labels in an understandable
 7959   (by Unicode sort order) way.  (Robert Muir, Mike McCandless)
 7961 * LUCENE-4843: Add LimitTokenPositionFilter: don't emit tokens with 
 7962   positions that exceed the configured limit.  (Steve Rowe)
 7964 * LUCENE-4832: Add ToParentBlockJoinCollector.getTopGroupsWithAllChildDocs, to retrieve
 7965   all children in each group.  (Aleksey Aleev via Mike McCandless)
 7967 * LUCENE-4846: PostingsHighlighter subclasses can override where the
 7968   String values come from (it still defaults to pulling from stored
 7969   fields).  (Robert Muir, Mike McCandless)
 7971 * LUCENE-4853: Add PostingsHighlighter.highlightFields method that
 7972   takes int[] docIDs instead of TopDocs.  (Robert Muir, Mike
 7973   McCandless)
 7975 * LUCENE-4856: If there are no matches for a given field, return the
 7976   first maxPassages sentences (Robert Muir, Mike McCandless)
 7978 * LUCENE-4859: IndexReader now exposes Terms statistics: getDocCount,
 7979   getSumDocFreq, getSumTotalTermFreq. (Shai Erera)
 7981 * LUCENE-4862: It is now possible to terminate collection of a single
 7982   IndexReader leaf by throwing a CollectionTerminatedException in
 7983   Collector.collect. (Adrien Grand, Shai Erera)
 7985 * LUCENE-4752: New SortingMergePolicy (in lucene/misc) that sorts documents
 7986   before merging segments. (Adrien Grand, Shai Erera, David Smiley)
 7988 * LUCENE-4860: Customize scoring and formatting per-field in
 7989   PostingsHighlighter by subclassing and overriding the getFormatter
 7990   and/or getScorer methods.  This also changes Passage.getMatchTerms()
 7991   to return BytesRef[] instead of Term[].  (Robert Muir, Mike
 7992   McCandless)
 7994 * LUCENE-4839: Added SorterTemplate.timSort, a O(n log n) stable sort algorithm
 7995   that performs well on partially sorted data. (Adrien Grand)
 7997 * LUCENE-4644: Added support for the "IsWithin" spatial predicate for
 7998   RecursivePrefixTreeStrategy. It's for matching non-point indexed shapes; if
 7999   you only have points (1/doc) then "Intersects" is equivalent and faster.
 8000   See the javadocs.  (David Smiley)
 8002 * LUCENE-4861: Make BreakIterator per-field in PostingsHighlighter. This means
 8003   you can override getBreakIterator(String field) to use different mechanisms
 8004   for e.g. title vs. body fields.  (Mike McCandless, Robert Muir)
 8006 * LUCENE-4645: Added support for the "Contains" spatial predicate for
 8007   RecursivePrefixTreeStrategy.  (David Smiley)
 8009 * LUCENE-4898: DirectoryReader.openIfChanged now allows opening a reader
 8010   on an IndexCommit starting from a near-real-time reader (previously
 8011   this would throw IllegalArgumentException).  (Mike McCandless)
 8013 * LUCENE-4905: Made the maxPassages parameter per-field in PostingsHighlighter.
 8014   (Robert Muir)
 8016 * LUCENE-4897: Added TaxonomyReader.getChildren for traversing a category's 
 8017   children. (Shai Erera)
 8019 * LUCENE-4902: Added FilterDirectoryReader to allow easy filtering of a
 8020   DirectoryReader's subreaders. (Alan Woodward, Adrien Grand, Uwe Schindler)
 8022 * LUCENE-4858: Added EarlyTerminatingSortingCollector to be used in conjunction
 8023   with SortingMergePolicy, which allows to early terminate queries on sorted
 8024   indexes, when the sort order matches the index order. (Adrien Grand, Shai Erera)
 8026 * LUCENE-4904: Added descending sort order to NumericDocValuesSorter. (Shai Erera)
 8028 * LUCENE-3786: Added SearcherTaxonomyManager, to manage access to both
 8029   IndexSearcher and DirectoryTaxonomyReader for near-real-time
 8030   faceting.  (Shai Erera, Mike McCandless)
 8032 * LUCENE-4915: DrillSideways now allows drilling down on fields that
 8033   are not faceted. (Mike McCandless)
 8035 * LUCENE-4895: Added support for the "IsDisjointTo" spatial predicate for
 8036   RecursivePrefixTreeStrategy.  (David Smiley)
 8038 * LUCENE-4774: Added FieldComparator that allows sorting parent documents based on
 8039   fields on the child / nested document level. (Martijn van Groningen)
 8041 Optimizations
 8043 * LUCENE-4839: SorterTemplate.merge can now be overridden in order to replace
 8044   the default implementation which merges in-place by a faster implementation
 8045   that could require fewer swaps at the expense of some extra memory.
 8046   ArrayUtil and CollectionUtil override it so that their mergeSort and timSort
 8047   methods are faster but only require up to 1% of extra memory. (Adrien Grand)
 8049 * LUCENE-4571: Speed up BooleanQuerys with minNrShouldMatch to use
 8050   skipping.  (Stefan Pohl via Robert Muir)
 8052 * LUCENE-4863: StemmerOverrideFilter now uses a FST to represent its overrides
 8053   in memory. (Simon Willnauer)
 8055 * LUCENE-4889: UnicodeUtil.codePointCount implementation replaced with a
 8056   non-array-lookup version. (Dawid Weiss)
 8058 * LUCENE-4923: Speed up BooleanQuerys processing of in-order disjunctions.
 8059   (Robert Muir)
 8061 * LUCENE-4926: Speed up DisjunctionMatchQuery.  (Robert Muir)
 8063 * LUCENE-4930: Reduce contention in older/buggy JVMs when using
 8064   AttributeSource#addAttribute() because java.lang.ref.ReferenceQueue#poll()
 8065   is implemented using synchronization.  (Christian Ziech, Karl Wright,
 8066   Uwe Schindler)
 8068 Bug Fixes
 8070 * LUCENE-4868: SumScoreFacetsAggregator used an incorrect index into
 8071   the scores array. (Shai Erera)
 8073 * LUCENE-4882: FacetsAccumulator did not allow to count ROOT category (i.e.
 8074   count dimensions). (Shai Erera)
 8076 * LUCENE-4876: IndexWriterConfig.clone() now clones its MergeScheduler,
 8077   IndexDeletionPolicy and InfoStream in order to make an IndexWriterConfig and
 8078   its clone fully independent. (Adrien Grand)
 8080 * LUCENE-4893: Facet counts were multiplied as many times as 
 8081   FacetsCollector.getFacetResults() is called. (Shai Erera)
 8083 * LUCENE-4888: Fixed SloppyPhraseScorer, MultiDocs(AndPositions)Enum and
 8084   MultiSpansWrapper which happened to sometimes call DocIdSetIterator.advance
 8085   with target<=current (in this case the behavior of advance is undefined).
 8086   (Adrien Grand)
 8088 * LUCENE-4899: FastVectorHighlighter failed with StringIndexOutOfBoundsException
 8089   if a single highlight phrase or term was greater than the fragCharSize producing
 8090   negative string offsets. (Simon Willnauer)
 8092 * LUCENE-4877: Throw exception for invalid arguments in analysis factories.
 8093   (Steve Rowe, Uwe Schindler, Robert Muir)
 8095 * LUCENE-4914: SpatialPrefixTree's Node/Cell.reset() forgot to reset the 'leaf'
 8096   flag.  It affects SpatialRecursivePrefixTreeStrategy on non-point indexed
 8097   shapes, as of Lucene 4.2. (David Smiley)
 8099 * LUCENE-4913: FacetResultNode.ordinal was always 0 when all children
 8100   are returned. (Mike McCandless)
 8102 * LUCENE-4918: Highlighter closes the given IndexReader if QueryScorer
 8103   is used with an external IndexReader. (Simon Willnauer, Sirvan Yahyaei)
 8105 * LUCENE-4880: Fix MemoryIndex to consume empty terms from the tokenstream consistent
 8106   with IndexWriter. Previously it discarded them.  (Timothy Allison via Robert Muir)
 8108 * LUCENE-4885: FacetsAccumulator did not set the correct value for
 8109   FacetResult.numValidDescendants. (Mike McCandless, Shai Erera)
 8111 * LUCENE-4925: Fixed IndexSearcher.search when the argument list contains a Sort
 8112   and one of the sort fields is the relevance score. Only IndexSearchers created
 8113   with an ExecutorService are concerned. (Adrien Grand)
 8115 * LUCENE-4738, LUCENE-2727, LUCENE-2812: Simplified
 8116   DirectoryReader.indexExists so that it's more robust to transient
 8117   IOExceptions (e.g. due to issues like file descriptor exhaustion),
 8118   but this will also cause it to err towards returning true for
 8119   example if the directory contains a corrupted index or an incomplete
 8120   initial commit.  In addition, IndexWriter with OpenMode.CREATE will
 8121   now succeed even if the directory contains a corrupted index (Billow
 8122   Gao, Robert Muir, Mike McCandless)
 8124 * LUCENE-4928: Stored fields and term vectors could become super slow in case
 8125   of tiny documents (a few bytes). This is especially problematic when switching
 8126   codecs since bulk-merge strategies can't be applied and the same chunk of
 8127   documents can end up being decompressed thousands of times. A hard limit on
 8128   the number of documents per chunk has been added to fix this issue.
 8129   (Robert Muir, Adrien Grand)
 8131 * LUCENE-4934: Fix minor equals/hashcode problems in facet/DrillDownQuery, 
 8132   BoostingQuery, MoreLikeThisQuery, FuzzyLikeThisQuery, and block join queries.
 8133   (Robert Muir, Uwe Schindler)
 8135 * LUCENE-4504: Fix broken sort comparator in ValueSource.getSortField,
 8136   used when sorting by a function query.  (Tom Shally via Robert Muir)
 8138 * LUCENE-4937: Fix incorrect sorting of float/double values (+/-0, NaN).
 8139   (Robert Muir, Uwe Schindler)
 8141 Documentation
 8143 * LUCENE-4841: Added example SimpleSortedSetFacetsExample to show how
 8144   to use the new SortedSetDocValues backed facet implementation.
 8145   (Shai Erera, Mike McCandless)
 8147 Build
 8149 * LUCENE-4879: Upgrade randomized testing to version 2.0.9: 
 8150   Filter stack traces on console output. (Dawid Weiss, Robert Muir)
 8153 ======================= Lucene 4.2.1 =======================
 8155 Bug Fixes
 8157 * LUCENE-4713: The SPI components used to load custom codecs or analysis
 8158   components were fixed to also scan the Lucene ClassLoader in addition
 8159   to the context ClassLoader, so Lucene is always able to find its own
 8160   codecs. The special case of a null context ClassLoader is now also
 8161   supported.  (Christian Kohlschütter, Uwe Schindler)
 8163 * LUCENE-4819: seekExact(BytesRef, boolean) did not work correctly with
 8164   Sorted[Set]DocValuesTermsEnum.  (Robert Muir)
 8166 * LUCENE-4826: PostingsHighlighter was not returning the top N best
 8167   scoring passages. (Robert Muir, Mike McCandless)
 8169 * LUCENE-4854: Fix DocTermOrds.getOrdTermsEnum() to not return negative
 8170   ord on initial next().  (Robert Muir)
 8172 * LUCENE-4836: Fix SimpleRateLimiter#pause to return the actual time spent
 8173   sleeping instead of the wakeup timestamp in nano seconds. (Simon Willnauer)
 8175 * LUCENE-4828: BooleanQuery no longer extracts terms from its MUST_NOT
 8176   clauses.  (Mike McCandless)
 8178 * SOLR-4589: Fixed CPU spikes and poor performance in lazy field loading 
 8179   of multivalued fields. (hossman)
 8181 * LUCENE-4870: Fix bug where an entire index might be deleted by the IndexWriter
 8182   due to false detection if an index exists in the directory when
 8183   OpenMode.CREATE_OR_APPEND is used. This might also affect application that set
 8184   the open mode manually using DirectoryReader#indexExists. (Simon Willnauer)
 8186 * LUCENE-4878: Override getRegexpQuery in MultiFieldQueryParser to prevent
 8187   NullPointerException when regular expression syntax is used with
 8188   MultiFieldQueryParser. (Simon Willnauer, Adam Rauch)
 8190 Optimizations
 8192 * LUCENE-4819: Added Sorted[Set]DocValues.termsEnum(), and optimized the
 8193   default codec for improved enumeration performance.  (Robert Muir)
 8195 * LUCENE-4854: Speed up TermsEnum of FieldCache.getDocTermOrds.
 8196   (Robert Muir)
 8198 * LUCENE-4857: Don't unnecessarily copy stem override map in
 8199   StemmerOverrideFilter. (Simon Willnauer)
 8201 ======================= Lucene 4.2.0 =======================
 8203 Changes in backwards compatibility policy
 8205 * LUCENE-4602: FacetFields now stores facet ordinals in a DocValues field, 
 8206   rather than a payload. This forces rebuilding existing indexes, or do a
 8207   one time migration using FacetsPayloadMigratingReader. Since DocValues 
 8208   support in-memory caching, CategoryListCache was removed too.
 8209   (Shai Erera, Michael McCandless)
 8211 * LUCENE-4697: FacetResultNode is now a concrete class with public members
 8212   (instead of getter methods). (Shai Erera)
 8214 * LUCENE-4600: FacetsCollector is now an abstract class with two
 8215   implementations: StandardFacetsCollector (the old version of 
 8216   FacetsCollector) and CountingFacetsCollector. FacetsCollector.create()
 8217   returns the most optimized collector for the given parameters.
 8218   (Shai Erera, Michael McCandless)
 8220 * LUCENE-4700: OrdinalPolicy is now per CategoryListParams, and is no longer
 8221   an interface, but rather an enum with values NO_PARENTS and ALL_PARENTS.
 8222   PathPolicy was removed, you should extend FacetFields and DrillDownStream
 8223   to control which categories are added as drill-down terms. (Shai Erera)
 8225 * LUCENE-4547: DocValues improvements:
 8226   - Simplified codec API: codecs are now only responsible for encoding and
 8227     decoding docvalues, they do not need to do buffering or RAM accounting.
 8228   - Per-Field support: added PerFieldDocValuesFormat, which allows you to
 8229     use a different DocValuesFormat per field (like postings).
 8230   - Unified with FieldCache api: DocValues can be accessed via FieldCache API,
 8231     so it works automatically with grouping/join/sort/function queries, etc.
 8232   - Simplified types: There are only 3 types (NUMERIC, BINARY, SORTED), so it's
 8233     not necessary to specify for example that all of your binary values have
 8234     the same length. Instead it's easy for the Codec API to optimize encoding
 8235     based on any properties of the content.
 8236   (Simon Willnauer, Adrien Grand, Mike McCandless, Robert Muir)
 8238 * LUCENE-4757: Cleanup and refactoring of FacetsAccumulator, FacetRequest,
 8239   FacetsAggregator and FacetResultsHandler API. If your application did 
 8240   FacetsCollector.create(), you should not be affected, but if you wrote
 8241   an Aggregator, then you should migrate it to the per-segment
 8242   FacetsAggregator. You can still use StandardFacetsAccumulator, which works
 8243   with the old API (for now). (Shai Erera) 
 8245 * LUCENE-4761: Facet packages reorganized. Should be easy to fix your import
 8246   statements, if you use an IDE such as Eclipse. (Shai Erera)
 8248 * LUCENE-4750: Convert DrillDown to DrillDownQuery, so you can initialize it
 8249   and add drill-down categories to it. (Michael McCandless, Shai Erera)
 8251 * LUCENE-4759: remove FacetRequest.SortBy; result categories are always 
 8252   sorted by value, while ties are broken by category ordinal. (Shai Erera)
 8254 * LUCENE-4772: Facet associations moved to new FacetsAggregator API. You
 8255   should override FacetsAccumulator and return the relevant aggregator,
 8256   for aggregating the association values. (Shai Erera)
 8258 * LUCENE-4748: A FacetRequest on a non-existent field now returns an
 8259   empty FacetResult instead of skipping it.  (Shai Erera, Mike McCandless)
 8261 * LUCENE-4806: The default category delimiter character was changed
 8262   from U+F749 to U+001F, since the latter uses 1 byte vs 3 bytes for
 8263   the former.  Existing facet indices must be reindexed.  (Robert
 8264   Muir, Shai Erera, Mike McCandless)
 8266 Optimizations
 8268 * LUCENE-4687: BloomFilterPostingsFormat now lazily initializes delegate
 8269   TermsEnum only if needed to do a seek or get a DocsEnum. (Simon Willnauer) 
 8271 * LUCENE-4677, LUCENE-4682: unpacked FSTs now use vInt to encode the node target,
 8272   to reduce their size (Mike McCandless)
 8274 * LUCENE-4678: FST now uses a paged byte[] structure instead of a
 8275   single byte[] internally, to avoid large memory spikes during
 8276   building (James Dyer, Mike McCandless)
 8278 * LUCENE-3298: FST can now be larger than 2.1 GB / 2.1 B nodes.
 8279   (James Dyer, Mike McCandless)
 8281 * LUCENE-4690: Performance improvements and non-hashing versions
 8282   of NumericUtils.*ToPrefixCoded() (yonik)
 8284 * LUCENE-4715: CategoryListParams.getOrdinalPolicy now allows to return a
 8285   different OrdinalPolicy per dimension, to better tune how you index
 8286   facets. Also added OrdinalPolicy.ALL_BUT_DIMENSION. 
 8287   (Shai Erera, Michael McCandless)
 8289 * LUCENE-4740: Don't track clones of MMapIndexInput if unmapping
 8290   is disabled. This reduces GC overhead. (Kristofer Karlsson, Uwe Schindler)
 8292 * LUCENE-4733: The default Lucene 4.2 codec now uses a more compact
 8293   TermVectorsFormat (Lucene42TermVectorsFormat) based on
 8294   CompressingTermVectorsFormat. (Adrien Grand)
 8296 * LUCENE-3729: The default Lucene 4.2 codec now uses a more compact
 8297   DocValuesFormat (Lucene42DocValuesFormat). Sorted values are stored in an
 8298   FST, Numerics and Ordinals use a number of strategies (delta-compression, 
 8299   table-compression, etc), and memory addresses use MonotonicBlockPackedWriter.
 8300   (Simon Willnauer, Adrien Grand, Mike McCandless, Robert Muir)
 8302 * LUCENE-4792: Reduction of the memory required to build the doc ID maps used
 8303   when merging segments. (Adrien Grand)
 8305 * LUCENE-4794: Spatial RecursivePrefixTreeStrategy's search filter: Skip calls
 8306   to termsEnum.seek() when the next term is known to follow the current cell.
 8307   (David Smiley)
 8309 New Features
 8311 * LUCENE-4686: New specialized DGapVInt8IntEncoder for facets (now the 
 8312   default). (Shai Erera)
 8314 * LUCENE-4703: Add simple PrintTaxonomyStats tool to see summary
 8315   information about the facets taxonomy index.  (Mike McCandless)
 8317 * LUCENE-4599: New oal.codecs.compressing.CompressingTermVectorsFormat which
 8318   compresses term vectors into chunks of documents similarly to
 8319   CompressingStoredFieldsFormat. (Adrien Grand)
 8321 * LUCENE-4695: Added LiveFieldValues utility class, for getting the
 8322   current (live, real-time) value for any indexed doc/field.  The
 8323   class buffers recently indexed doc/field values until a new
 8324   near-real-time reader is opened that contains those changes.
 8325   (Robert Muir, Mike McCandless)
 8327 * LUCENE-4723: Add AnalyzerFactoryTask to benchmark, and enable analyzer
 8328   creation via the resulting factories using NewAnalyzerTask.  (Steve Rowe)
 8330 * LUCENE-4728: Unknown and not explicitly mapped queries are now rewritten
 8331   against the highlighting IndexReader to obtain primitive queries before 
 8332   discarding the query entirely. WeightedSpanTermExtractor now builds a
 8333   MemoryIndex only once even if multiple fields are highlighted.
 8334   (Simon Willnauer)
 8336 * LUCENE-4035: Added ICUCollationDocValuesField, more efficient
 8337   support for Locale-sensitive sort and range queries for
 8338   single-valued fields.  (Robert Muir)
 8340 * LUCENE-4547: Added MonotonicBlockPacked(Reader/Writer), which provide 
 8341   efficient random access to large amounts of monotonically increasing 
 8342   positive values (e.g. file offsets). Each block stores the minimum value
 8343   and the average gap, and values are encoded as signed deviations from
 8344   the expected value.  (Adrien Grand)
 8346 * LUCENE-4547: Added AppendingLongBuffer, an append-only buffer that packs
 8347   signed long values in memory and provides an efficient iterator API.
 8348   (Adrien Grand) 
 8350 * LUCENE-4540: It is now possible for a codec to represent norms with
 8351   less than 8 bits per value. For performance reasons this is not done
 8352   by default, but you can customize your codec (e.g. pass PackedInts.DEFAULT
 8353   to Lucene42DocValuesConsumer) if you want to make this tradeoff.
 8354   (Adrien Grand, Robert Muir)
 8356 * LUCENE-4764: A new Facet42Codec and Facet42DocValuesFormat provide
 8357   faster but more RAM-consuming facet performance.  (Shai Erera, Mike
 8358   McCandless) 
 8360 * LUCENE-4769: Added OrdinalsCache and CachedOrdsCountingFacetsAggregator
 8361   which uses the cache to obtain a document's ordinals. This aggregator
 8362   is faster than others, however consumes much more RAM.
 8363   (Michael McCandless, Shai Erera)
 8365 * LUCENE-4778: Add a getter for the delegate in RateLimitedDirectoryWrapper.
 8366   (Mark Miller)
 8368 * LUCENE-4765: Add a multi-valued docvalues type (SORTED_SET). This is equivalent
 8369   to building a FieldCache.getDocTermOrds at index-time.  (Robert Muir)
 8371 * LUCENE-4780: Add MonotonicAppendingLongBuffer: an append-only buffer for
 8372   monotonically increasing values.  (Adrien Grand)
 8374 * LUCENE-4748: Added DrillSideways utility class for computing both
 8375   drill-down and drill-sideways counts for a DrillDownQuery.  (Mike
 8376   McCandless)
 8378 API Changes
 8380 * LUCENE-4709: FacetResultNode no longer has a residue field. (Shai Erera)
 8382 * LUCENE-4716: DrillDown.query now takes Occur, allowing to specify if
 8383   categories should be OR'ed or AND'ed. (Shai Erera)
 8385 * LUCENE-4695: ReferenceManager.RefreshListener.afterRefresh now takes
 8386   a boolean indicating whether a new reference was in fact opened, and
 8387   a new beforeRefresh method notifies you when a refresh attempt is
 8388   starting.  (Robert Muir, Mike McCandless)
 8390 * LUCENE-4794: Spatial RecursivePrefixTreeFilter replaced by
 8391   IntersectsPrefixTreeFilter and some extensible base classes. (David Smiley)
 8393 Bug Fixes
 8395 * LUCENE-4705: Pass on FilterStrategy in FilteredQuery if the filtered query is 
 8396   rewritten. (Simon Willnauer)
 8398 * LUCENE-4712: MemoryIndex#normValues() throws NPE if field doesn't exist. 
 8399   (Simon Willnauer, Ricky Pritchett)
 8401 * LUCENE-4550: Shapes wider than 180 degrees would use too much accuracy for the
 8402   PrefixTree based SpatialStrategy. For a pathological case of nearly 360
 8403   degrees and barely any height, it would generate so many indexed terms
 8404   (> 500k) that it could even cause an OutOfMemoryError. Fixed. (David Smiley)
 8406 * LUCENE-4704: Make join queries override hashcode and equals methods.
 8407   (Martijn van Groningen)
 8409 * LUCENE-4724: Fix bug in CategoryPath which allowed passing null or empty
 8410   string components. This is forbidden now (throws an exception). Note that if
 8411   you have a taxonomy index created with such strings, you should rebuild it.
 8412   (Michael McCandless, Shai Erera)
 8414 * LUCENE-4732: Fixed TermsEnum.seekCeil/seekExact on term vectors.
 8415   (Adrien Grand, Robert Muir)
 8417 * LUCENE-4739: Fixed bugs that prevented FSTs more than ~1.1GB from
 8418   being saved and loaded (Adrien Grand, Mike McCandless)
 8420 * LUCENE-4717: Fixed bug where Lucene40DocValuesFormat would sometimes write 
 8421   an extra unused ordinal for sorted types. The bug is detected and corrected
 8422   on-the-fly for old indexes.  (Robert Muir)
 8424 * LUCENE-4547: Fixed bug where Lucene40DocValuesFormat was unable to encode
 8425   segments that would exceed 2GB total data. This could happen in some surprising
 8426   cases, for example if you had an index with more than 260M documents and a 
 8427   VAR_INT field.  (Simon Willnauer, Adrien Grand, Mike McCandless, Robert Muir)
 8429 * LUCENE-4775: Remove SegmentInfo.sizeInBytes() and make
 8430   MergePolicy.OneMerge.totalBytesSize thread safe (Josh Bronson via
 8431   Robert Muir, Mike McCandless)
 8433 * LUCENE-4770: If spatial's TermQueryPrefixTreeStrategy was used to search
 8434   indexed non-point shapes, then there was an edge case where a query should
 8435   find a shape but it didn't. The fix is the removal of an optimization that
 8436   simplifies some leaf cells into a parent. The index data for such a field is
 8437   now ~20% larger. This optimization is still done for the query shape, and for
 8438   indexed data for RecursivePrefixTreeStrategy. Furthermore, this optimization
 8439   is enhanced to roll up beyond the bottom cell level. (David Smiley,
 8440   Florian Schilling)
 8442 * LUCENE-4790: Fix FieldCacheImpl.getDocTermOrds to not bake deletes into the
 8443   cached datastructure. Otherwise this can cause inconsistencies with readers
 8444   at different points in time.  (Robert Muir)
 8446 * LUCENE-4791: A conjunction of terms (ConjunctionTermScorer) scanned on
 8447   the lowest frequency term instead of skipping, leading to potentially
 8448   large performance impacts for many non-random or non-uniform
 8449   term distributions.  (John Wang, yonik)
 8451 * LUCENE-4798: PostingsHighlighter's formatter sometimes didn't highlight 
 8452   matched terms.  (Robert Muir)
 8454 * LUCENE-4796, SOLR-4373: Fix concurrency issue in NamedSPILoader and
 8455   AnalysisSPILoader when doing reload (e.g. from Solr).
 8456   (Uwe Schindler, Hossman)
 8458 * LUCENE-4802: Don't compute norms for drill-down facet fields. (Mike McCandless)
 8460 * LUCENE-4804: PostingsHighlighter sometimes applied terms to the wrong passage,
 8461   if they started exactly on a passage boundary.  (Robert Muir)
 8463 Documentation
 8465 * LUCENE-4718: Fixed documentation of oal.queryparser.classic.
 8466   (Hayden Muhl via Adrien Grand)
 8468 * LUCENE-4784, LUCENE-4785, LUCENE-4786: Fixed references to deprecated classes
 8469   SinkTokenizer, ValueSourceQuery and RangeQuery. (Hao Zhong via Adrien Grand) 
 8471 Build
 8473 * LUCENE-4654: Test duration statistics from multiple test runs should be 
 8474   reused. (Dawid Weiss)
 8476 * LUCENE-4636: Upgrade ivy to 2.3.0 (Shawn Heisey via Robert Muir)
 8478 * LUCENE-4570: Use the Policeman Forbidden API checker, released separately
 8479   from Lucene and downloaded via Ivy.  (Uwe Schindler, Robert Muir)
 8481 * LUCENE-4758: 'ant jar', 'ant compile', and 'ant compile-test' should
 8482   recurse.  (Steve Rowe)
 8484 ======================= Lucene 4.1.0 =======================
 8486 Changes in backwards compatibility policy
 8488 * LUCENE-4514: Scorer's freq() method returns an integer value indicating
 8489   the number of times the scorer matches the current document. Previously
 8490   this was only sometimes the case, in some cases it returned a (meaningless)
 8491   floating point value.  Scorer now extends DocsEnum so it has attributes().
 8492   (Robert Muir)
 8494 * LUCENE-4543: TFIDFSimilarity's index-time computeNorm is now final to
 8495   match the fact that its query-time norm usage requires a FIXED_8 encoding. 
 8496   Override lengthNorm and/or encode/decodeNormValue to change the specifics,
 8497   like Lucene 3.x. (Robert Muir)
 8499 * LUCENE-3441: The facet module now supports NRT. As a result, the following
 8500   changes were made:
 8501   - DirectoryTaxonomyReader has a new constructor which takes a 
 8502     DirectoryTaxonomyWriter. You should use that constructor in order to get
 8503     the NRT support (or the old one for non-NRT).
 8504   - TaxonomyReader.refresh() removed in exchange for TaxonomyReader.openIfChanged
 8505     static method. Similar to DirectoryReader, the method either returns null
 8506     if no changes were made to the taxonomy, or a new TR instance otherwise.
 8507     Instead of calling refresh(), you should write similar code to how you reopen
 8508     a regular DirectoryReader.
 8509   - TaxonomyReader.openIfChanged (previously refresh()) no longer throws
 8510     InconsistentTaxonomyException, and supports recreate. InconsistentTaxoEx
 8511     was removed.
 8512   - ChildrenArrays was pulled out of TaxonomyReader into a top-level class.
 8513   - TaxonomyReader was made an abstract class (instead of an interface), with
 8514     methods such as close() and reference counting management pulled from
 8515     DirectoryTaxonomyReader, and made final. The rest of the methods, remained
 8516     abstract.
 8517   (Shai Erera, Gilad Barkai)
 8519 * LUCENE-4576: Remove CachingWrapperFilter(Filter, boolean). This recacheDeletes
 8520   option gave less than 1% speedup at the expense of cache churn (filters were
 8521   invalidated on reopen if even a single delete was posted against the segment).
 8522   (Robert Muir)
 8524 * LUCENE-4575: Replace IndexWriter's commit/prepareCommit versions that take
 8525   commitData with setCommitData(). That allows committing changes to IndexWriter
 8526   even if the commitData is the only thing that changes.
 8527   (Shai Erera, Michael McCandless)
 8529 * LUCENE-4565: TaxonomyReader.getParentArray and .getChildrenArrays consolidated
 8530   into one getParallelTaxonomyArrays(). You can obtain the 3 arrays that the
 8531   previous two methods returned by calling parents(), children() or siblings()
 8532   on the returned ParallelTaxonomyArrays. (Shai Erera)
 8534 * LUCENE-4585: Spatial PrefixTree based Strategies (either TermQuery or
 8535   RecursivePrefix based) MAY want to re-index if used for point data. If a
 8536   re-index is not done, then an indexed point is ~1/2 the smallest grid cell
 8537   larger and as such is slightly more likely to match a query shape.
 8538   (David Smiley)
 8540 * LUCENE-4604: DefaultOrdinalPolicy removed in favor of OrdinalPolicy.ALL_PARENTS.
 8541   Same for DefaultPathPolicy (now PathPolicy.ALL_CATEGORIES). In addition, you
 8542   can use OrdinalPolicy.NO_PARENTS to never write any parent category ordinal
 8543   to the fulltree posting payload (but note that you need a special 
 8544   FacetsAccumulator - see javadocs). (Shai Erera)
 8546 * LUCENE-4594: Spatial PrefixTreeStrategy no longer indexes center points of
 8547   non-point shapes.  If you want to call makeDistanceValueSource() based on
 8548   shape centers, you need to do this yourself in another spatial field.
 8549   (David Smiley)
 8551 * LUCENE-4615: Replace IntArrayAllocator and FloatArrayAllocator by ArraysPool.
 8552   FacetArrays no longer takes those allocators; if you need to reuse the arrays,
 8553   you should use ReusingFacetArrays. (Shai Erera, Gilad Barkai)
 8555 * LUCENE-4621: FacetIndexingParams is now a concrete class (instead of DefaultFIP).
 8556   Also, the entire IndexingParams chain is now immutable. If you need to override
 8557   a setting, you should extend the relevant class.
 8558   Additionally, FacetSearchParams is now immutable, and requires all FacetRequests
 8559   to specified at initialization time. (Shai Erera)
 8561 * LUCENE-4647: CategoryDocumentBuilder and EnhancementsDocumentBuilder are replaced
 8562   by FacetFields and AssociationsFacetFields respectively. CategoryEnhancement and
 8563   AssociationEnhancement were removed in favor of a simplified CategoryAssociation
 8564   interface, with CategoryIntAssociation and CategoryFloatAssociation 
 8565   implementations.
 8566   NOTE: indexes that contain category enhancements/associations are not supported
 8567   by the new code and should be recreated. (Shai Erera)
 8569 * LUCENE-4659: Massive cleanup to CategoryPath API. Additionally, CategoryPath is 
 8570   now immutable, so you don't need to clone() it. (Shai Erera)
 8572 * LUCENE-4670: StoredFieldsWriter and TermVectorsWriter have new finish* callbacks
 8573   which are called after a doc/field/term has been completely added.
 8574   (Adrien Grand, Robert Muir)
 8576 * LUCENE-4620: IntEncoder/Decoder were changed to do bulk encoding/decoding. As a
 8577   result, few other classes such as Aggregator and CategoryListIterator were
 8578   changed to handle bulk category ordinals. (Shai Erera)
 8580 * LUCENE-4683: CategoryListIterator and Aggregator are now per-segment. As such
 8581   their implementations no longer take a top-level IndexReader in the constructor
 8582   but rather implement a setNextReader. (Shai Erera)
 8584 New Features
 8586 * LUCENE-4226: New experimental StoredFieldsFormat that compresses chunks of
 8587   documents together in order to improve the compression ratio. (Adrien Grand)
 8589 * LUCENE-4426: New ValueSource implementations (in lucene/queries) for
 8590   DocValues fields. (Adrien Grand)
 8592 * LUCENE-4410: FilteredQuery now exposes a FilterStrategy that exposes
 8593   how filters are applied during query execution. (Simon Willnauer) 
 8595 * LUCENE-4404: New ListOfOutputs (in lucene/misc) for FSTs wraps
 8596   another Outputs implementation, allowing you to store more than one
 8597   output for a single input.  UpToTwoPositiveIntsOutputs was moved
 8598   from lucene/core to lucene/misc.  (Mike McCandless)
 8600 * LUCENE-3842: New AnalyzingSuggester, for doing auto-suggest
 8601   using an analyzer.  This can create powerful suggesters: if the analyzer
 8602   remove stop words then "ghost chr..." could suggest "The Ghost of
 8603   Christmas Past"; if SynonymFilter is used to map wifi and wireless
 8604   network to hotspot, then "wirele..." could suggest "wifi router";
 8605   token normalization likes stemmers, accent removal, etc. would allow
 8606   the suggester to ignore such variations. (Robert Muir, Sudarshan
 8607   Gaikaiwari, Mike McCandless)
 8609 * LUCENE-4446: Lucene 4.1 has a new default index format (Lucene41Codec)
 8610   that incorporates the previously experimental "Block" postings format
 8611   for better search performance. 
 8612   (Han Jiang, Adrien Grand, Robert Muir, Mike McCandless)
 8614 * LUCENE-3846: New FuzzySuggester, like AnalyzingSuggester except it
 8615   also finds completions allowing for fuzzy edits in the input string.
 8616   (Robert Muir, Simon Willnauer, Mike McCandless)
 8618 * LUCENE-4515: MemoryIndex now supports adding the same field multiple
 8619   times. (Simon Willnauer)
 8621 * LUCENE-4489: Added consumeAllTokens option to LimitTokenCountFilter
 8622   (hossman, Robert Muir)
 8624 * LUCENE-4566: Add NRT/SearcherManager.RefreshListener/addListener to
 8625   be notified whenever a new searcher was opened. (selckin via Shai
 8626   Erera, Mike McCandless)
 8628 * SOLR-4123: Add per-script customizability to ICUTokenizerFactory via
 8629   rule files in the ICU RuleBasedBreakIterator format.
 8630   (Shawn Heisey, Robert Muir, Steve Rowe)
 8632 * LUCENE-4590: Added WriteEnwikiLineDocTask - a benchmark task for writing
 8633   Wikipedia category pages and non-category pages into separate line files.
 8634   extractWikipedia.alg was changed to use this task, so now it creates two
 8635   files. (Doron Cohen)
 8637 * LUCENE-4290: Added PostingsHighlighter to the highlighter module. It uses
 8638   offsets from the postings lists to highlight documents. (Robert Muir)
 8640 * LUCENE-4628: Added CommonTermsQuery that executes high-frequency terms
 8641   in a optional sub-query to prevent slow queries due to "common" terms 
 8642   like stopwords. (Simon Willnauer)
 8644 API Changes
 8646 * LUCENE-4399: Deprecated AppendingCodec. Lucene's term dictionaries
 8647   no longer seek when writing.  (Adrien Grand, Robert Muir)
 8649 * LUCENE-4479: Rename TokenStream.getTokenStream(IndexReader, int, String)
 8650   to TokenStream.getTokenStreamWithOffsets, and return null on failure
 8651   rather than throwing IllegalArgumentException.  (Alan Woodward)
 8653 * LUCENE-4472: MergePolicy now accepts a MergeTrigger that provides 
 8654   information about the trigger of the merge ie. merge triggered due
 8655   to a segment merge or a full flush etc. (Simon Willnauer)
 8657 * LUCENE-4415: TermsFilter is now immutable. All terms need to be provided
 8658   as constructor argument. (Simon Willnauer)
 8660 * LUCENE-4520: ValueSource.getSortField no longer throws IOExceptions
 8661   (Alan Woodward)
 8663 * LUCENE-4537: RateLimiter is now separated from FSDirectory and exposed via
 8664   RateLimitingDirectoryWrapper. Any Directory can now be rate-limited.
 8665   (Simon Willnauer)  
 8667 * LUCENE-4591: CompressingStoredFields{Writer,Reader} now accept a segment
 8668   suffix as a constructor parameter. (Renaud Delbru via Adrien Grand)
 8670 * LUCENE-4605: Added DocsEnum.FLAG_NONE which can be passed instead of 0 as
 8671   the flag to .docs() and .docsAndPositions(). (Shai Erera)
 8673 * LUCENE-4617: Remove FST.pack() method. Previously to make a packed FST,
 8674   you had to make a Builder with willPackFST=true (telling it you will later pack it),
 8675   create your fst with finish(), and then call pack() to get another FST.
 8676   Instead just pass true for doPackFST to Builder and finish() returns a packed FST.
 8677   (Robert Muir)
 8679 * LUCENE-4663: Deprecate IndexSearcher.document(int, Set). This was not intended
 8680   to be final, nor named document(). Use IndexSearcher.doc(int, Set) instead.
 8681   (Robert Muir)
 8683 * LUCENE-4684: Made DirectSpellChecker extendable.
 8684   (Martijn van Groningen)
 8686 Bug Fixes
 8688 * LUCENE-1822: BaseFragListBuilder hard-coded 6 char margin is too naive.
 8689   (Alex Vigdor, Arcadius Ahouansou, Koji Sekiguchi)
 8691 * LUCENE-4468: Fix rareish integer overflows in Lucene41 postings 
 8692   format. (Robert Muir)
 8694 * LUCENE-4486: Add support for ConstantScoreQuery in Highlighter.
 8695  (Simon Willnauer)
 8697 * LUCENE-4485: When CheckIndex terms, terms/docs pairs and tokens,
 8698   these counts now all exclude deleted documents.  (Mike McCandless)
 8700 * LUCENE-4479: Highlighter works correctly for fields with term vector
 8701   positions, but no offsets.  (Alan Woodward)
 8703 * SOLR-3906: JapaneseReadingFormFilter in romaji mode will return
 8704   romaji even for out-of-vocabulary kana cases (e.g. half-width forms).
 8705   (Robert Muir)
 8707 * LUCENE-4511: TermsFilter might return wrong results if a field is not 
 8708   indexed or doesn't exist in the index. (Simon Willnauer)
 8710 * LUCENE-4521: IndexWriter.tryDeleteDocument could return true
 8711   (successfully deleting the document) but then on IndexWriter
 8712   close/commit fail to write the new deletions, if no other changes
 8713   happened in the IndexWriter instance.  (Ivan Vasilev via Mike
 8714   McCandless)
 8716 * LUCENE-4513: Fixed that deleted nested docs are scored into the
 8717   parent doc when using ToParentBlockJoinQuery. (Martijn van Groningen)
 8719 * LUCENE-4534: Fixed WFSTCompletionLookup and Analyzing/FuzzySuggester
 8720   to allow 0 byte values in the lookup keys.  (Mike McCandless)
 8722 * LUCENE-4532: DirectoryTaxonomyWriter use a timestamp to denote taxonomy
 8723   index re-creation, which could cause a bug in case machine clocks were
 8724   not synced. Instead, it now tracks an 'epoch' version, which is incremented
 8725   whenever the taxonomy is re-created, or replaced. (Shai Erera)
 8727 * LUCENE-4544: Fixed off-by-1 in ConcurrentMergeScheduler that would
 8728   allow 1+maxMergeCount merges threads to be created, instead of just
 8729   maxMergeCount (Radim Kolar, Mike McCandless)
 8731 * LUCENE-4567: Fixed NullPointerException in analyzing, fuzzy, and
 8732   WFST suggesters when no suggestions were added (selckin via Mike
 8733   McCandless)
 8735 * LUCENE-4568: Fixed integer overflow in
 8736   PagedBytes.PagedBytesData{In,Out}put.getPosition. (Adrien Grand)
 8738 * LUCENE-4581: GroupingSearch.setAllGroups(true) was failing to
 8739   actually compute allMatchingGroups (dizh@neusoft.com via Mike
 8740   McCandless)
 8742 * LUCENE-4009: Improve TermsFilter.toString (Tim Costermans via Chris
 8743   Male, Mike McCandless)
 8745 * LUCENE-4588: Benchmark's EnwikiContentSource was discarding last wiki 
 8746   document and had leaking threads in 'forever' mode. (Doron Cohen)   
 8748 * LUCENE-4585: Spatial RecursivePrefixTreeFilter had some bugs that only
 8749   occurred when shapes were indexed.  In what appears to be rare circumstances,
 8750   documents with shapes near a query shape were erroneously considered a match.
 8751   In addition, it wasn't possible to index a shape representing the entire
 8752   globe.
 8754 * LUCENE-4595: EnwikiContentSource had a thread safety problem (NPE) in 
 8755   'forever' mode (Doron Cohen)
 8757 * LUCENE-4587: fix WordBreakSpellChecker to not throw AIOOBE when presented
 8758   with 2-char codepoints, and to correctly break/combine terms containing
 8759   non-latin characters. (James Dyer, Andreas Hubold)
 8761 * LUCENE-4596: fix a concurrency bug in DirectoryTaxonomyWriter.
 8762   (Shai Erera)
 8764 * LUCENE-4594: Spatial PrefixTreeStrategy would index center-points in addition
 8765   to the shape to index if it was non-point, in the same field.  But sometimes
 8766   the center-point isn't actually in the shape (consider a LineString), and for
 8767   highly precise shapes it could cause makeDistanceValueSource's cache to load
 8768   parts of the shape's boundary erroneously too.  So center points aren't
 8769   indexed any more; you should use another spatial field. (David Smiley)
 8771 * LUCENE-4629: IndexWriter misses to delete documents if a document block is
 8772   indexed and the Iterator throws an exception. Documents were only rolled back
 8773   if the actual indexing process failed. (Simon Willnauer)
 8775 * LUCENE-4608: Handle large number of requested fragments better.
 8776   (Martijn van Groningen)
 8778 * LUCENE-4633: DirectoryTaxonomyWriter.replaceTaxonomy did not refresh its
 8779   internal reader, which could cause an existing category to be added twice.
 8780   (Shai Erera)
 8782 * LUCENE-4461: If you added the same FacetRequest more than once, you would get
 8783   inconsistent results. (Gilad Barkai via Shai Erera)
 8785 * LUCENE-4656: Fix regression in IndexWriter to work with empty TokenStreams
 8786   that have no TermToBytesRefAttribute (commonly provided by CharTermAttribute),
 8787   e.g., oal.analysis.miscellaneous.EmptyTokenStream.
 8788   (Uwe Schindler, Adrien Grand, Robert Muir)
 8790 * LUCENE-4660: ConcurrentMergeScheduler was taking too long to
 8791   un-pause incoming threads it had paused when too many merges were
 8792   queued up. (Mike McCandless)
 8794 * LUCENE-4662: Add missing elided articles and prepositions to FrenchAnalyzer's
 8795   DEFAULT_ARTICLES list passed to ElisionFilter.  (David Leunen via Steve Rowe)
 8797 * LUCENE-4671: Fix CharsRef.subSequence method.  (Tim Smith via Robert Muir)
 8799 * LUCENE-4465: Let ConstantScoreQuery's Scorer return its child scorer.
 8800   (selckin via Uwe Schindler)
 8802 Changes in Runtime Behavior
 8804 * LUCENE-4586: Change default ResultMode of FacetRequest to PER_NODE_IN_TREE.
 8805   This only affects requests with depth>1. If you execute such requests and
 8806   rely on the facet results being returned flat (i.e. no hierarchy), you should
 8807   set the ResultMode to GLOBAL_FLAT. (Shai Erera, Gilad Barkai) 
 8809 * LUCENE-1822: Improves the text window selection by recalculating the starting margin
 8810   once all phrases in the fragment have been identified in FastVectorHighlighter. This
 8811   way if a single word is matched in a fragment, it will appear in the middle of the highlight,
 8812   instead of 6 characters from the beginning. This way one can also guarantee that
 8813   the entirety of short texts are represented in a fragment by specifying a large
 8814   enough fragCharSize.
 8816 Optimizations
 8818 * LUCENE-2221: oal.util.BitUtil was modified to use Long.bitCount and
 8819   Long.numberOfTrailingZeros (which are intrinsics since Java 6u18) instead of
 8820   pure java bit twiddling routines in order to improve performance on modern
 8821   JVMs/hardware. (Dawid Weiss, Adrien Grand)
 8823 * LUCENE-4509: Enable stored fields compression by default in the Lucene 4.1
 8824   default codec. (Adrien Grand)
 8826 * LUCENE-4536: PackedInts on-disk format is now byte-aligned (it used to be
 8827   long-aligned), saving up to 7 bytes per array of values.
 8828   (Adrien Grand, Mike McCandless)
 8830 * LUCENE-4512: Additional memory savings for CompressingStoredFieldsFormat.
 8831   (Adrien Grand, Robert Muir)
 8833 * LUCENE-4443: Lucene41PostingsFormat no longer writes unnecessary offsets 
 8834   into the skipdata. (Robert Muir)
 8836 * LUCENE-4459: Improve WeakIdentityMap.keyIterator() to remove GCed keys
 8837   from backing map early instead of waiting for reap(). This makes test
 8838   failures in TestWeakIdentityMap disappear, too.
 8839   (Uwe Schindler, Mike McCandless, Robert Muir)
 8841 * LUCENE-4473: Lucene41PostingsFormat encodes offsets more efficiently
 8842   for low frequency terms (< 128 occurrences).  (Robert Muir)
 8844 * LUCENE-4462: DocumentsWriter now flushes deletes, segment infos and builds
 8845   CFS files if necessary during segment flush and not during publishing. The latter
 8846   was a single threaded process while now all IO and CPU heavy computation is done
 8847   concurrently in DocumentsWriterPerThread. (Simon Willnauer)
 8849 * LUCENE-4496: Optimize Lucene41PostingsFormat when requesting a subset of
 8850   the postings data (via flags to TermsEnum.docs/docsAndPositions) to use
 8851   ForUtil.skipBlock.  (Robert Muir)
 8853 * LUCENE-4497: Don't write PosVIntCount to the positions file in 
 8854   Lucene41PostingsFormat, as it's always totalTermFreq % BLOCK_SIZE. (Robert Muir)
 8856 * LUCENE-4498: In Lucene41PostingsFormat, when a term appears in only one document, 
 8857   Instead of writing a file pointer to a VIntBlock containing the doc id, just 
 8858   write the doc id.  (Mike McCandless, Robert Muir)
 8860 * LUCENE-4515: MemoryIndex now uses Byte/IntBlockPool internally to hold terms and 
 8861   posting lists. All index data is represented as consecutive byte/int arrays to 
 8862   reduce GC cost and memory overhead. (Simon Willnauer) 
 8864 * LUCENE-4538: DocValues now caches direct sources in a ThreadLocal exposed via SourceCache. 
 8865   Users of this API can now simply obtain an instance via DocValues#getDirectSource per thread.
 8866   (Simon Willnauer)
 8868 * LUCENE-4580: DrillDown.query variants return a ConstantScoreQuery with boost set to 0.0f
 8869   so that documents scores are not affected by running a drill-down query. (Shai Erera)  
 8871 * LUCENE-4598: PayloadIterator no longer uses top-level IndexReader to iterate on the
 8872   posting's payload. (Shai Erera, Michael McCandless)
 8874 * LUCENE-4661: Drop default maxThreadCount to 1 and maxMergeCount to 2
 8875   in ConcurrentMergeScheduler, for faster merge performance on
 8876   spinning-magnet drives (Mike McCandless)
 8878 Documentation
 8880 * LUCENE-4483: Refer to BytesRef.deepCopyOf in Term's constructor that takes BytesRef.
 8881   (Paul Elschot via Robert Muir)
 8883 Build
 8885 * LUCENE-4650: Upgrade randomized testing to version 2.0.8: make the
 8886   test framework more robust under low memory conditions. (Dawid Weiss)
 8888 * LUCENE-4603: Upgrade randomized testing to version 2.0.5: print forked
 8889   JVM PIDs on heartbeat from hung tests (Dawid Weiss)
 8891 * Upgrade randomized testing to version 2.0.4: avoid hangs on shutdown
 8892   hooks hanging forever by calling Runtime.halt() in addition to 
 8893   Runtime.exit() after a short delay to allow graceful shutdown (Dawid Weiss)
 8895 * LUCENE-4451: Memory leak per unique thread caused by 
 8896   RandomizedContext.contexts static map. Upgrade randomized testing
 8897   to version 2.0.2 (Mike McCandless, Dawid Weiss)
 8899 * LUCENE-4589: Upgraded benchmark module's Nekohtml dependency to version
 8900   1.9.17, removing the workaround in Lucene's HTML parser for the
 8901   Turkish locale.  (Uwe Schindler)
 8903 * LUCENE-4601: Fix ivy availability check to use typefound, so it works
 8904   if called from another build file.  (Ryan Ernst via Robert Muir)
 8907 ======================= Lucene 4.0.0 =======================
 8909 Changes in backwards compatibility policy
 8911 * LUCENE-4392: Class org.apache.lucene.util.SortedVIntList has been removed.
 8912   (Adrien Grand)
 8914 * LUCENE-4393: RollingCharBuffer has been moved to the o.a.l.analysis.util
 8915   package of lucene-analysis-common. (Adrien Grand)
 8917 New Features
 8919 * LUCENE-1888: Added the option to store payloads in the term
 8920   vectors (IndexableFieldType.storeTermVectorPayloads()). Note 
 8921   that you must store term vector positions to store payloads.
 8922   (Robert Muir)
 8924 * LUCENE-3892: Add a new BlockPostingsFormat that bulk-encodes docs,
 8925   freqs and positions in large (size 128) packed-int blocks for faster
 8926   search performance.  This was from Han Jiang's 2012 Google Summer of
 8927   Code project (Han Jiang, Adrien Grand, Robert Muir, Mike McCandless)
 8929 * LUCENE-4323: Added support for an absolute maximum CFS segment size
 8930   (in MiB) to LogMergePolicy and TieredMergePolicy.
 8931   (Alexey Lef via Uwe Schindler)
 8933 * LUCENE-4339: Allow deletes against 3.x segments for easier upgrading.
 8934   Lucene3x Codec is still otherwise read-only, you should not set it
 8935   as the default Codec on IndexWriter, because it cannot write new segments.
 8936   (Mike McCandless, Robert Muir)
 8938 * SOLR-3441: ElisionFilterFactory is now MultiTermAware
 8939   (Jack Krupansky via hossman)
 8941 API Changes
 8943 * LUCENE-4391, LUCENE-4440: All methods of Lucene40Codec but
 8944   getPostingsFormatForField are now final. To reuse functionality 
 8945   of Lucene40, you should extend FilterCodec and delegate to Lucene40
 8946   instead of extending Lucene40Codec.  (Adrien Grand, Shai Erera,
 8947   Robert Muir, Uwe Schindler)
 8949 * LUCENE-4299: Added Terms.hasPositions() and Terms.hasOffsets().
 8950   Previously you had no real way to know that a term vector field
 8951   had positions or offsets, since this can be configured on a 
 8952   per-field-per-document basis. (Robert Muir)
 8954 * Removed DocsAndPositionsEnum.hasPayload() and simplified the
 8955   contract of getPayload(). It returns null if there is no payload,
 8956   otherwise returns the current payload. You can now call it multiple
 8957   times per position if you want. (Robert Muir)
 8959 * Removed FieldsEnum. Fields API instead implements Iterable<String>
 8960   and exposes Iterator, so you can iterate over field names with
 8961   for (String field : fields) instead.  (Robert Muir)
 8963 * LUCENE-4152: added IndexReader.leaves(), which lets you enumerate
 8964   the leaf atomic reader contexts for all readers in the tree.
 8965   (Uwe Schindler, Robert Muir)
 8967 * LUCENE-4304: removed PayloadProcessorProvider. If you want to change
 8968   payloads (or other things) when merging indexes, it's recommended
 8969   to just use a FilterAtomicReader + IndexWriter.addIndexes. See the
 8970   OrdinalMappingAtomicReader and TaxonomyMergeUtils in the facets
 8971   module if you want an example of this.
 8972   (Mike McCandless, Uwe Schindler, Shai Erera, Robert Muir)
 8974 * LUCENE-4304: Make CompositeReader.getSequentialSubReaders()
 8975   protected. To get atomic leaves of any IndexReader use the new method
 8976   leaves() (LUCENE-4152), which lists AtomicReaderContexts including
 8977   the doc base of each leaf.  (Uwe Schindler, Robert Muir)
 8979 * LUCENE-4307: Renamed IndexReader.getTopReaderContext to 
 8980   IndexReader.getContext.  (Robert Muir)
 8982 * LUCENE-4316: Deprecate Fields.getUniqueTermCount and remove it from
 8983   AtomicReader. If you really want the unique term count across all
 8984   fields, just sum up Terms.size() across those fields. This method
 8985   only exists so that this statistic can be accessed for Lucene 3.x 
 8986   segments, which don't support Terms.size().  (Uwe Schindler, Robert Muir)
 8988 * LUCENE-4321: Change CharFilter to extend Reader directly, as FilterReader
 8989   overdelegates (read(), read(char[], int, int), skip, etc). This made it
 8990   hard to implement CharFilters that were correct. Instead only close() is
 8991   delegated by default: read(char[], int, int) and correct(int) are abstract
 8992   so that it's obvious which methods you should implement.  The protected 
 8993   inner Reader is 'input' like CharFilter in the 3.x series, instead of 'in'.  
 8994   (Dawid Weiss, Uwe Schindler, Robert Muir)
 8996 * LUCENE-3309: The expert FieldSelector API, used to load only certain
 8997   fields in a stored document, has been replaced with the simpler
 8998   StoredFieldVisitor API.  (Mike McCandless)
 9000 * LUCENE-4343: Made Tokenizer.setReader final. This is a setter that should
 9001   not be overridden by subclasses: per-stream initialization should happen
 9002   in reset().  (Robert Muir)
 9004 * LUCENE-4377: Remove IndexInput.copyBytes(IndexOutput, long). 
 9005   Use DataOutput.copyBytes(DataInput, long) instead.
 9006   (Mike McCandless, Robert Muir)
 9008 * LUCENE-4355: Simplify AtomicReader's sugar methods such as termDocsEnum,
 9009   termPositionsEnum, docFreq, and totalTermFreq to only take Term as a
 9010   parameter. If you want to do expert things such as pass a different
 9011   Bits as liveDocs, then use the flex apis (fields(), terms(), etc) directly.
 9012   (Mike McCandless, Robert Muir)
 9014 * LUCENE-4425: clarify documentation of StoredFieldVisitor.binaryValue
 9015   and simplify the api to binaryField(FieldInfo, byte[]).
 9016   (Adrien Grand, Robert Muir)
 9018 Bug Fixes
 9020 * LUCENE-4423: DocumentStoredFieldVisitor.binaryField ignored offset and
 9021   length. (Adrien Grand)
 9023 * LUCENE-4297: BooleanScorer2 would multiply the coord() factor
 9024   twice for conjunctions: for most users this is no problem, but
 9025   if you had a customized Similarity that returned something other
 9026   than 1 when overlap == maxOverlap (always the case for conjunctions),
 9027   then the score would be incorrect.  (Pascal Chollet, Robert Muir)
 9029 * LUCENE-4298: MultiFields.getTermDocsEnum(IndexReader, Bits, String, BytesRef)
 9030   did not work at all, it would infinitely recurse.
 9031   (Alberto Paro via Robert Muir)
 9033 * LUCENE-4300: BooleanQuery's rewrite was not always safe: if you
 9034   had a custom Similarity where coord(1,1) != 1F, then the rewritten
 9035   query would be scored differently.  (Robert Muir)
 9037 * Don't allow negatives in the positions file. If you have an index
 9038   from 2.4.0 or earlier with such negative positions, and you already 
 9039   upgraded to 3.x, then to Lucene 4.0-ALPHA or -BETA, you should run 
 9040   CheckIndex. If it fails, then you need to upgrade again to 4.0  (Robert Muir)
 9042 * LUCENE-4303: PhoneticFilterFactory and SnowballPorterFilterFactory load their
 9043   encoders / stemmers via the ResourceLoader now instead of Class.forName().
 9044   Solr users should now no longer have to embed these in its war. (David Smiley)
 9046 * SOLR-3737: StempelPolishStemFilterFactory loaded its stemmer table incorrectly.
 9047   Also, ensure immutability and use only one instance of this table in RAM (lazy
 9048   loaded) since it's quite large. (sausarkar, Steven Rowe, Robert Muir)
 9050 * LUCENE-4310: MappingCharFilter was failing to match input strings
 9051   containing non-BMP Unicode characters.  (Dawid Weiss, Robert Muir,
 9052   Mike McCandless)
 9054 * LUCENE-4224: Add in-order scorer to query time joining and the
 9055   out-of-order scorer throws an UOE. (Martijn van Groningen, Robert Muir)
 9057 * LUCENE-4333: Fixed NPE in TermGroupFacetCollector when faceting on mv fields.
 9058   (Jesse MacVicar, Martijn van Groningen)
 9060 * LUCENE-4218: Document.get(String) and Field.stringValue() again return
 9061   values for numeric fields, like Lucene 3.x and consistent with the documentation.
 9062   (Jamie, Uwe Schindler, Robert Muir)
 9064 * NRTCachingDirectory was always caching a newly flushed segment in
 9065   RAM, instead of checking the estimated size of the segment
 9066   to decide whether to cache it. (Mike McCandless)
 9068 * LUCENE-3720: fix memory-consumption issues with BeiderMorseFilter.
 9069   (Thomas Neidhart via Robert Muir)
 9071 * LUCENE-4401: Fix bug where DisjunctionSumScorer would sometimes call score()
 9072   on a subscorer that had already returned NO_MORE_DOCS.  (Liu Chao, Robert Muir)
 9074 * LUCENE-4411: when sampling is enabled for a FacetRequest, its depth
 9075   parameter is reset to the default (1), even if set otherwise.
 9076   (Gilad Barkai via Shai Erera)
 9078 * LUCENE-4455: Fix bug in SegmentInfoPerCommit.sizeInBytes() that was
 9079   returning 2X the true size, inefficiently.  Also fixed bug in
 9080   CheckIndex that would report no deletions when a segment has
 9081   deletions, and vice/versa.  (Uwe Schindler, Robert Muir, Mike McCandless)
 9083 * LUCENE-4456: Fixed double-counting sizeInBytes for a segment
 9084   (affects how merge policies pick merges); fixed CheckIndex's
 9085   incorrect reporting of whether a segment has deletions; fixed case
 9086   where on abort Lucene could remove files it didn't create; fixed
 9087   many cases where IndexWriter could leave leftover files (on
 9088   exception in various places, on reuse of a segment name after crash
 9089   and recovery.  (Uwe Schindler, Robert Muir, Mike McCandless)
 9091 Optimizations
 9093 * LUCENE-4322: Decrease lucene-core JAR size. The core JAR size had increased a
 9094   lot because of generated code introduced in LUCENE-4161 and LUCENE-3892.
 9095   (Adrien Grand)
 9097 * LUCENE-4317: Improve reuse of internal TokenStreams and StringReader
 9098   in oal.document.Field.  (Uwe Schindler, Chris Male, Robert Muir)
 9100 * LUCENE-4327: Support out-of-order scoring in FilteredQuery for higher
 9101   performance.  (Mike McCandless, Robert Muir)
 9103 * LUCENE-4364: Optimize MMapDirectory to not make a mapping per-cfs-slice,
 9104   instead one map per .cfs file. This reduces the total number of maps.
 9105   Additionally factor out a (package-private) generic 
 9106   ByteBufferIndexInput from MMapDirectory.  (Uwe Schindler, Robert Muir)
 9108 Build
 9110 * LUCENE-4406, LUCENE-4407: Upgrade to randomizedtesting 2.0.1.
 9111   Workaround for broken test output XMLs due to non-XML text unicode
 9112   chars in strings. Added printing of failed tests at the end of a
 9113   test run (Dawid Weiss)
 9115 * LUCENE-4252: Detect/Fail tests when they leak RAM in static fields
 9116   (Robert Muir, Dawid Weiss)
 9118 * LUCENE-4360: Support running the same test suite multiple times in 
 9119   parallel (Dawid Weiss)
 9121 * LUCENE-3985: Upgrade to randomizedtesting 2.0.0. Added support for 
 9122   thread leak detection. Added support for suite timeouts. (Dawid Weiss)
 9124 * LUCENE-4354: Corrected maven dependencies to be consistent with
 9125   the licenses/ folder and the binary release. Some had different
 9126   versions or additional unnecessary dependencies. (selckin via Robert Muir)
 9128 * LUCENE-4340: Move all non-default codec, postings format and terms
 9129   dictionary implementations to lucene/codecs. (Adrien Grand)
 9131 Documentation
 9133 * LUCENE-4302: Fix facet userguide to have HTML loose doctype like
 9134   all other javadocs.  (Karl Nicholas via Uwe Schindler)
 9136 ======================= Lucene 4.0.0-BETA =======================
 9138 New features
 9140 * LUCENE-4249: Changed the explanation of the PayloadTermWeight to use the
 9141   underlying PayloadFunction's explanation as the explanation
 9142   for the payload score. (Scott Smerchek via Robert Muir)
 9144 * LUCENE-4069: Added BloomFilteringPostingsFormat for use with low-frequency terms
 9145   such as primary keys (Mark Harwood, Mike McCandless) 
 9147 * LUCENE-4201: Added JapaneseIterationMarkCharFilter to normalize Japanese
 9148   iteration marks. (Robert Muir, Christian Moen)
 9150 * LUCENE-3832: Added BasicAutomata.makeStringUnion method to efficiently
 9151   create automata from a fixed collection of UTF-8 encoded BytesRef
 9152   (Dawid Weiss, Robert Muir)
 9154 * LUCENE-4153: Added option to fast vector highlighting via BaseFragmentsBuilder to
 9155   respect field boundaries in the case of highlighting for multivalued fields.
 9156   (Martijn van Groningen)
 9158 * LUCENE-4227: Added DirectPostingsFormat, to hold all postings in
 9159   memory as uncompressed simple arrays.  This uses a tremendous amount
 9160   of RAM but gives good search performance gains.  (Mike McCandless)
 9162 * LUCENE-2510, LUCENE-4044: Migrated Solr's Tokenizer-, TokenFilter-, and
 9163   CharFilterFactories to the lucene-analysis module. The API is still
 9164   experimental.  (Chris Male, Robert Muir, Uwe Schindler)
 9166 * LUCENE-4230: When pulling a DocsAndPositionsEnum you can now
 9167   specify whether or not you require payloads (in addition to
 9168   offsets); turning one or both off may allow some codec
 9169   implementations to optimize the enum implementation.  (Robert Muir,
 9170   Mike McCandless)
 9172 * LUCENE-4203: Add IndexWriter.tryDeleteDocument(AtomicReader reader,
 9173   int docID), to attempt deletion by docID as long as the provided
 9174   reader is an NRT reader, and the segment has not yet been merged
 9175   away (Mike McCandless).
 9177 * LUCENE-4286: Added option to CJKBigramFilter to always also output
 9178   unigrams. This can be used for a unigram+bigram approach, or at 
 9179   index-time only for better support of short queries.
 9180   (Tom Burton-West, Robert Muir)
 9182 API Changes
 9184 * LUCENE-4138: update of morfologik (Polish morphological analyzer) to 1.5.3.
 9185   The tag attribute class has been renamed to MorphosyntacticTagsAttribute and
 9186   has a different API (carries a list of tags instead of a compound tag). Upgrade
 9187   of embedded morfologik dictionaries to version 1.9. (Dawid Weiss)
 9189 * LUCENE-4178: set 'tokenized' to true on FieldType by default, so that if you
 9190   make a custom FieldType and set indexed = true, it's analyzed by the analyzer.
 9191   (Robert Muir)
 9193 * LUCENE-4220: Removed the buggy JavaCC-based HTML parser in the benchmark
 9194   module and replaced by NekoHTML. HTMLParser interface was cleaned up while
 9195   changing method signatures.  (Uwe Schindler, Robert Muir)
 9197 * LUCENE-2191: Rename Tokenizer.reset(Reader) to Tokenizer.setReader(Reader).
 9198   The purpose of this method was always to set a new Reader on the Tokenizer,
 9199   reusing the object. But the name was often confused with TokenStream.reset().
 9200   (Robert Muir)
 9202 * LUCENE-4228: Refactored CharFilter to extend java.io.FilterReader. CharFilters
 9203   filter another reader and you override correct() for offset correction.
 9204   (Robert Muir)
 9206 * LUCENE-4240: Analyzer api now just takes fieldName for getOffsetGap. If the
 9207   field is not analyzed (e.g. StringField), then the analyzer is not invoked
 9208   at all. If you want to tweak things like positionIncrementGap and offsetGap,
 9209   analyze the field with KeywordTokenizer instead.  (Grant Ingersoll, Robert Muir)
 9211 * LUCENE-4250: Pass fieldName to the PayloadFunction explain method, so it
 9212   parallels with docScore and the default implementation is correct.
 9213   (Robert Muir)
 9215 * LUCENE-3747: Support Unicode 6.1.0. (Steve Rowe)
 9217 * LUCENE-3884: Moved ElisionFilter out of org.apache.lucene.analysis.fr
 9218   package into org.apache.lucene.analysis.util.  (Robert Muir)
 9220 * LUCENE-4230: When pulling a DocsAndPositionsEnum you now pass an int
 9221   flags instead of the previous boolean needOffsets.  Currently
 9222   recognized flags are DocsAndPositionsEnum.FLAG_PAYLOADS and
 9223   DocsAndPositionsEnum.FLAG_OFFSETS (Robert Muir, Mike McCandless)
 9225 * LUCENE-4273: When pulling a DocsEnum, you can pass an int flags
 9226   instead of the previous boolean needsFlags; consistent with the changes
 9227   for DocsAndPositionsEnum in LUCENE-4230. Currently the only flag
 9228   is DocsEnum.FLAG_FREQS. (Robert Muir, Mike McCandless)
 9230 * LUCENE-3616: TextField(String, Reader, Store) was reduced to TextField(String, Reader),
 9231   as the Store parameter didn't make sense: if you supplied Store.YES, you would only 
 9232   receive an exception anyway. (Robert Muir)
 9234 Optimizations
 9236 * LUCENE-4171: Performance improvements to Packed64.
 9237   (Toke Eskildsen via Adrien Grand)
 9239 * LUCENE-4184: Performance improvements to the aligned packed bits impl.
 9240   (Toke Eskildsen, Adrien Grand)
 9242 * LUCENE-4235: Remove enforcing of Filter rewrite for NRQ queries.
 9243   (Uwe Schindler)
 9245 * LUCENE-4279: Regenerated snowball Stemmers from snowball r554,
 9246   making them substantially more lightweight. Behavior is unchanged. 
 9247   (Robert Muir)
 9249 * LUCENE-4291: Reduced internal buffer size for Jflex-based tokenizers 
 9250   such as StandardTokenizer from 32kb to 8kb.  
 9251   (Raintung Li, Steven Rowe, Robert Muir)
 9253 Bug Fixes
 9255 * LUCENE-4109: BooleanQueries are not parsed correctly with the 
 9256   flexible query parser. (Karsten Rauch via Robert Muir)
 9258 * LUCENE-4176: Fix AnalyzingQueryParser to analyze range endpoints as bytes,
 9259   so that it works correctly with Analyzers that produce binary non-UTF-8 terms
 9260   such as CollationAnalyzer. (Nattapong Sirilappanich via Robert Muir) 
 9262 * LUCENE-4209: Fix FSTCompletionLookup to close its sorter, so that it won't
 9263   leave temp files behind in /tmp. Fix SortedTermFreqIteratorWrapper to not
 9264   leave temp files behind in /tmp on Windows. Fix Sort to not leave
 9265   temp files behind when /tmp is a separate volume. (Uwe Schindler, Robert Muir)
 9267 * LUCENE-4221: Fix overeager CheckIndex validation for term vector offsets.
 9268   (Robert Muir)
 9270 * LUCENE-4222: TieredMergePolicy.getFloorSegmentMB was returning the
 9271   size in bytes not MB (Chris Fuller via Mike McCandless)
 9273 * LUCENE-3505: Fix bug (Lucene 4.0alpha only) where boolean conjunctions
 9274   were sometimes scored incorrectly. Conjunctions of only termqueries where
 9275   at least one term omitted term frequencies (IndexOptions.DOCS_ONLY) would 
 9276   be scored as if all terms omitted term frequencies.  (Robert Muir)
 9278 * LUCENE-2686, LUCENE-3505: Fixed BooleanQuery scorers to return correct 
 9279   freq().  Added support for scorer navigation API (Scorer.getChildren) to 
 9280   all queries.  Made Scorer.freq() abstract. 
 9281   (Koji Sekiguchi, Mike McCandless, Robert Muir)
 9283 * LUCENE-4234: Exception when FacetsCollector is used with ScoreFacetRequest, 
 9284   and the number of matching documents is too large. (Gilad Barkai via Shai Erera)
 9286 * LUCENE-4245: Make IndexWriter#close() and MergeScheduler#close()
 9287   non-interruptible.  (Mark Miller, Uwe Schindler)
 9289 * LUCENE-4190: restrict allowed filenames that a codec may create to
 9290   the patterns recognized by IndexFileNames.  This also fixes
 9291   IndexWriter to only delete files matching this pattern from an index
 9292   directory, to reduce risk when the wrong index path is accidentally
 9293   passed to IndexWriter (Robert Muir, Mike McCandless)
 9295 * LUCENE-4277: Fix IndexWriter deadlock during rollback if flushable DWPT
 9296   instance are already checked out and queued up but not yet flushed. 
 9297   (Simon Willnauer)
 9299 * LUCENE-4282: Automaton FuzzyQuery didn't always deliver all results.
 9300   (Johannes Christen, Uwe Schindler, Robert Muir)
 9302 * LUCENE-4289: Fix minor idf inconsistencies/inefficiencies in highlighter.
 9303   (Robert Muir)
 9305 Changes in Runtime Behavior
 9307 * LUCENE-4109: Enable position increments in the flexible queryparser by default.
 9308   (Karsten Rauch via Robert Muir)
 9310 * LUCENE-3616: Field throws exception if you try to set a boost on an 
 9311   unindexed field or one that omits norms. (Robert Muir)
 9313 Build
 9315 * LUCENE-4094: Support overriding file.encoding on forked test JVMs
 9316   (force via -Drandomized.file.encoding=XXX). (Dawid Weiss)
 9318 * LUCENE-4189: Test output should include timestamps (start/end for each 
 9319   test/ suite). Added -Dtests.timestamps=[off by default]. (Dawid Weiss)
 9321 * LUCENE-4110: Report long periods of forked jvm inactivity (hung tests/ suites).
 9322   Added -Dtests.heartbeat=[seconds] with the default of 60 seconds.
 9323   (Dawid Weiss)
 9325 * LUCENE-4160: Added a property to quit the tests after a given
 9326   number of failures has occurred. This is useful in combination
 9327   with -Dtests.iters=N (you can start N iterations and wait for M
 9328   failures, in particular M = 1). -Dtests.maxfailures=M. Alternatively,
 9329   specify -Dtests.failfast=true to skip all tests after the first failure.
 9330   (Dawid Weiss)
 9332 * LUCENE-4115: JAR resolution/ cleanup should be done automatically for ant 
 9333   clean/ eclipse/ resolve (Dawid Weiss)
 9335 * LUCENE-4199, LUCENE-4202, LUCENE-4206: Add a new target "check-forbidden-apis"
 9336   that parses all generated .class files for use of APIs that use default
 9337   charset, default locale, or default timezone and fail build if violations
 9338   found. This ensures, that Lucene / Solr is independent on local configuration
 9339   options.  (Uwe Schindler, Robert Muir, Dawid Weiss)
 9341 * LUCENE-4217: Add the possibility to run tests with Atlassian Clover
 9342   loaded from IVY. A development License solely for Apache code was added in
 9343   the tools/ folder, but is not included in releases.  (Uwe Schindler)
 9345 Documentation
 9347 * LUCENE-4195: Added package documentation and examples for 
 9348   org.apache.lucene.codecs (Alan Woodward via Robert Muir)
 9350 ======================= Lucene 4.0.0-ALPHA =======================
 9352 More information about this release, including any errata related to the 
 9353 release notes, upgrade instructions, or other changes may be found online at:
 9354    https://wiki.apache.org/lucene-java/Lucene4.0
 9356 For "contrib" changes prior to 4.0, please see:
 9357 http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_3_6_0/lucene/contrib/CHANGES.txt
 9359 Changes in backwards compatibility policy
 9361 * LUCENE-1458, LUCENE-2111, LUCENE-2354: Changes from flexible indexing:
 9363   - On upgrading to 4.0, if you do not fully reindex your documents,
 9364     Lucene will emulate the new flex API on top of the old index,
 9365     incurring some performance cost (up to ~10% slowdown, typically).
 9366     To prevent this slowdown, use oal.index.IndexUpgrader
 9367     to upgrade your indexes to latest file format (LUCENE-3082).
 9369     Mixed flex/pre-flex indexes are perfectly fine -- the two
 9370     emulation layers (flex API on pre-flex index, and pre-flex API on
 9371     flex index) will remap the access as required.  So on upgrading to
 9372     4.0 you can start indexing new documents into an existing index.
 9373     To get optimal performance, use oal.index.IndexUpgrader
 9374     to upgrade your indexes to latest file format (LUCENE-3082).
 9376   - The postings APIs (TermEnum, TermDocsEnum, TermPositionsEnum)
 9377     have been removed in favor of the new flexible
 9378     indexing (flex) APIs (Fields, FieldsEnum, Terms, TermsEnum,
 9379     DocsEnum, DocsAndPositionsEnum). One big difference is that field
 9380     and terms are now enumerated separately: a TermsEnum provides a
 9381     BytesRef (wraps a byte[]) per term within a single field, not a
 9382     Term.  Another is that when asking for a Docs/AndPositionsEnum, you
 9383     now specify the skipDocs explicitly (typically this will be the
 9384     deleted docs, but in general you can provide any Bits).
 9386   - The term vectors APIs (TermFreqVector, TermPositionVector, 
 9387     TermVectorMapper) have been removed in favor of the above
 9388     flexible indexing APIs, presenting a single-document inverted
 9389     index of the document from the term vectors.
 9391   - MultiReader ctor now throws IOException
 9393   - Directory.copy/Directory.copyTo now copies all files (not just
 9394     index files), since what is and isn't and index file is now
 9395     dependent on the codecs used.
 9397   - UnicodeUtil now uses BytesRef for UTF-8 output, and some method
 9398     signatures have changed to CharSequence.  These are internal APIs
 9399     and subject to change suddenly.
 9401   - Positional queries (PhraseQuery, *SpanQuery) will now throw an
 9402     exception if use them on a field that omits positions during
 9403     indexing (previously they silently returned no results).
 9405   - FieldCache.{Byte,Short,Int,Long,Float,Double}Parser's API has
 9406     changed -- each parse method now takes a BytesRef instead of a
 9407     String.  If you have an existing Parser, a simple way to fix it is
 9408     invoke BytesRef.utf8ToString, and pass that String to your
 9409     existing parser.  This will work, but performance would be better
 9410     if you could fix your parser to instead operate directly on the
 9411     byte[] in the BytesRef.
 9413   - The internal (experimental) API of NumericUtils changed completely
 9414     from String to BytesRef. Client code should never use this class,
 9415     so the change would normally not affect you. If you used some of
 9416     the methods to inspect terms or create TermQueries out of
 9417     prefix encoded terms, change to use BytesRef. Please note:
 9418     Do not use TermQueries to search for single numeric terms.
 9419     The recommended way is to create a corresponding NumericRangeQuery
 9420     with upper and lower bound equal and included. TermQueries do not
 9421     score correct, so the constant score mode of NRQ is the only
 9422     correct way to handle single value queries.
 9424   - NumericTokenStream now works directly on byte[] terms. If you
 9425     plug a TokenFilter on top of this stream, you will likely get
 9426     an IllegalArgumentException, because the NTS does not support
 9427     TermAttribute/CharTermAttribute. If you want to further filter
 9428     or attach Payloads to NTS, use the new NumericTermAttribute.
 9430   (Mike McCandless, Robert Muir, Uwe Schindler, Mark Miller, Michael Busch)
 9432 * LUCENE-2858, LUCENE-3733: IndexReader was refactored into abstract
 9433   AtomicReader, CompositeReader, and DirectoryReader. To open Directory-
 9434   based indexes use DirectoryReader.open(), the corresponding method in
 9435   IndexReader is now deprecated for easier migration. Only DirectoryReader
 9436   supports commits, versions, and reopening with openIfChanged(). Terms,
 9437   postings, docvalues, and norms can from now on only be retrieved using
 9438   AtomicReader; DirectoryReader and MultiReader extend CompositeReader,
 9439   only offering stored fields and access to the sub-readers (which may be
 9440   composite or atomic). SlowCompositeReaderWrapper (LUCENE-2597) can be
 9441   used to emulate atomic readers on top of composites.
 9442   Please review MIGRATE.txt for information how to migrate old code.
 9443   (Uwe Schindler, Robert Muir, Mike McCandless)
 9445 * LUCENE-2265: FuzzyQuery and WildcardQuery now operate on Unicode codepoints,
 9446   not unicode code units. For example, a Wildcard "?" represents any unicode
 9447   character. Furthermore, the rest of the automaton package and RegexpQuery use 
 9448   true Unicode codepoint representation.  (Robert Muir, Mike McCandless)
 9450 * LUCENE-2380: The String-based FieldCache methods (getStrings,
 9451   getStringIndex) have been replaced with BytesRef-based equivalents
 9452   (getTerms, getTermsIndex).  Also, the sort values (returned in
 9453   FieldDoc.fields) when sorting by SortField.STRING or
 9454   SortField.STRING_VAL are now BytesRef instances.  See MIGRATE.txt
 9455   for more details. (yonik, Mike McCandless)
 9457 * LUCENE-2480: Though not a change in backwards compatibility policy, pre-3.0 
 9458   indexes are no longer supported. You should upgrade to 3.x first, then run
 9459   optimize(), or reindex. (Shai Erera, Earwin Burrfoot)
 9461 * LUCENE-2484: Removed deprecated TermAttribute. Use CharTermAttribute
 9462   and TermToBytesRefAttribute instead.  (Uwe Schindler)
 9464 * LUCENE-2600: Remove IndexReader.isDeleted in favor of
 9465   AtomicReader.getDeletedDocs().  (Mike McCandless)
 9467 * LUCENE-2667: FuzzyQuery's defaults have changed for more performant 
 9468   behavior: the minimum similarity is 2 edit distances from the word,
 9469   and the priority queue size is 50. To support this, FuzzyQuery now allows
 9470   specifying unscaled edit distances (foobar~2). If your application depends
 9471   upon the old defaults of 0.5 (scaled) minimum similarity and Integer.MAX_VALUE
 9472   priority queue size, you can use FuzzyQuery(Term, float, int, int) to specify 
 9473   those explicitly.
 9475 * LUCENE-2674: MultiTermQuery.TermCollector.collect now accepts the
 9476   TermsEnum as well.  (Robert Muir, Mike McCandless)
 9478 * LUCENE-588: WildcardQuery and QueryParser now allows escaping with 
 9479   the '\' character. Previously this was impossible (you could not escape */?,
 9480   for example).  If your code somehow depends on the old behavior, you will
 9481   need to change it (e.g. using "\\" to escape '\' itself).  
 9482   (Sunil Kamath, Terry Yang via Robert Muir)
 9484 * LUCENE-2837: Collapsed Searcher, Searchable into IndexSearcher;
 9485   removed contrib/remote and MultiSearcher (Mike McCandless); absorbed
 9486   ParallelMultiSearcher into IndexSearcher as an optional
 9487   ExecutorServiced passed to its ctor.  (Mike McCandless)
 9489 * LUCENE-2908, LUCENE-4037: Removed serialization code from lucene classes. 
 9490   It is recommended that you serialize user search needs at a higher level 
 9491   in your application.
 9492   (Robert Muir, Benson Margulies)
 9494 * LUCENE-2831: Changed Weight#scorer, Weight#explain & Filter#getDocIdSet to
 9495   operate on a AtomicReaderContext instead of directly on IndexReader to enable
 9496   searches to be aware of IndexSearcher's context. (Simon Willnauer)
 9498 * LUCENE-2839: Scorer#score(Collector,int,int) is now public because it is
 9499   called from other classes and part of public API. (Uwe Schindler)
 9501 * LUCENE-2865: Weight#scorer(AtomicReaderContext, boolean, boolean) now accepts
 9502   a ScorerContext struct instead of booleans.(Simon Willnauer)
 9504 * LUCENE-2882: Cut over SpanQuery#getSpans to AtomicReaderContext to enforce
 9505   per segment semantics on SpanQuery & Spans. (Simon Willnauer)
 9507 * LUCENE-2236: Similarity can now be configured on a per-field basis. See the
 9508   migration notes in MIGRATE.txt for more details.  (Robert Muir, Doron Cohen)
 9510 * LUCENE-2315: AttributeSource's methods for accessing attributes are now final,
 9511   else it's easy to corrupt the internal states.  (Uwe Schindler)
 9513 * LUCENE-2814: The IndexWriter.flush method no longer takes "boolean
 9514   flushDocStores" argument, as we now always flush doc stores (index
 9515   files holding stored fields and term vectors) while flushing a
 9516   segment.  (Mike McCandless)
 9518 * LUCENE-2548: Field names (eg in Term, FieldInfo) are no longer
 9519   interned.  (Mike McCandless)
 9521 * LUCENE-2883: The contents of o.a.l.search.function has been consolidated into
 9522   the queries module and can be found at o.a.l.queries.function.  See 
 9523   MIGRATE.txt for more information (Chris Male)
 9525 * LUCENE-2392, LUCENE-3299: Decoupled vector space scoring from 
 9526   Query/Weight/Scorer. If you extended Similarity directly before, you should 
 9527   extend TFIDFSimilarity instead.  Similarity is now a lower-level API to 
 9528   implement other scoring algorithms.  See MIGRATE.txt for more details.
 9529   (David Nemeskey, Simon Willnauer, Mike McCandless, Robert Muir)
 9531 * LUCENE-3330: The expert visitor API in Scorer has been simplified and
 9532   extended to support arbitrary relationships. To navigate to a scorer's 
 9533   children, call Scorer.getChildren().  (Robert Muir)
 9535 * LUCENE-2308: Field is now instantiated with an instance of IndexableFieldType,
 9536   of which there is a core implementation FieldType.  Most properties 
 9537   describing a Field have been moved to IndexableFieldType.  See MIGRATE.txt 
 9538   for more details.  (Nikola Tankovic, Mike McCandless, Chris Male)
 9540 * LUCENE-3396: ReusableAnalyzerBase.TokenStreamComponents.reset(Reader) now 
 9541   returns void instead of boolean.  If a Component cannot be reset, it should 
 9542   throw an Exception.  (Chris Male)
 9544 * LUCENE-3396: ReusableAnalyzerBase has been renamed to Analyzer.  All Analyzer
 9545   implementations must now use Analyzer.TokenStreamComponents, rather than
 9546   overriding .tokenStream() and .reusableTokenStream() (which are now final). 
 9547   (Chris Male)
 9549 * LUCENE-3346: Analyzer.reusableTokenStream() has been renamed to tokenStream()
 9550   with the old tokenStream() method removed.  Consequently it is now mandatory
 9551   for all Analyzers to support reusability. (Chris Male)
 9553 * LUCENE-3473: AtomicReader.getUniqueTermCount() no longer throws UOE when
 9554   it cannot be easily determined. Instead, it returns -1 to be consistent with
 9555   this behavior across other index statistics.
 9556   (Robert Muir)
 9558 * LUCENE-1536: The abstract FilteredDocIdSet.match() method is no longer
 9559   allowed to throw IOException. This change was required to make it conform
 9560   to the Bits interface. This method should never do I/O for performance reasons.
 9561   (Mike McCandless, Uwe Schindler, Robert Muir, Chris Male, Yonik Seeley,
 9562   Jason Rutherglen, Paul Elschot)
 9564 * LUCENE-3559: The methods "docFreq" and "maxDoc" on IndexSearcher were removed,
 9565   as these are no longer used by the scoring system. See MIGRATE.txt for more
 9566   details.  (Robert Muir)
 9568 * LUCENE-3533: Removed SpanFilters, they created large lists of objects and 
 9569   did not scale. (Robert Muir)
 9571 * LUCENE-3606: IndexReader and subclasses were made read-only. It is no longer
 9572   possible to delete or undelete documents using IndexReader; you have to use
 9573   IndexWriter now. As deleting by internal Lucene docID is no longer possible,
 9574   this requires adding a unique identifier field to your index. Deleting/
 9575   relying upon Lucene docIDs is not recommended anyway, because they can
 9576   change. Consequently commit() was removed and DirectoryReader.open(),
 9577   openIfChanged() no longer take readOnly booleans or IndexDeletionPolicy
 9578   instances. Furthermore, IndexReader.setNorm() was removed. If you need
 9579   customized norm values, the recommended way to do this is by modifying
 9580   Similarity to use an external byte[] or one of the new DocValues
 9581   fields (LUCENE-3108). Alternatively, to dynamically change norms (boost
 9582   *and* length norm) at query time, wrap your AtomicReader using
 9583   FilterAtomicReader, overriding FilterAtomicReader.norms(). To persist the
 9584   changes on disk, copy the FilteredIndexReader to a new index using
 9585   IndexWriter.addIndexes().  (Uwe Schindler, Robert Muir)
 9587 * LUCENE-3640: Removed IndexSearcher.close(), because IndexSearcher no longer
 9588   takes a Directory and no longer "manages" IndexReaders, it is a no-op.
 9589   (Robert Muir)
 9591 * LUCENE-3684: Add offsets into DocsAndPositionsEnum, and a few
 9592   FieldInfo.IndexOption: DOCS_AND_POSITIONS_AND_OFFSETS.  (Robert
 9593   Muir, Mike McCandless)
 9595 * LUCENE-2858, LUCENE-3770: FilterIndexReader was renamed to 
 9596   FilterAtomicReader and now extends AtomicReader. If you want to filter 
 9597   composite readers like DirectoryReader or MultiReader, filter their 
 9598   atomic leaves and build a new CompositeReader (e.g. MultiReader) around 
 9599   them. (Uwe Schindler, Robert Muir) 
 9601 * LUCENE-3736: ParallelReader was split into ParallelAtomicReader
 9602   and ParallelCompositeReader. Lucene 3.x's ParallelReader is now
 9603   ParallelAtomicReader; but the new composite variant has improved performance
 9604   as it works on the atomic subreaders. It requires that all parallel
 9605   composite readers have the same subreader structure. If you cannot provide this,
 9606   you can use SlowCompositeReaderWrapper to make all parallel readers atomic
 9607   and use ParallelAtomicReader.  (Uwe Schindler, Mike McCandless, Robert Muir)
 9609 * LUCENE-2000: clone() now returns covariant types where possible. (ryan)
 9611 * LUCENE-3970: Rename Fields.getUniqueFieldCount -> .size() and
 9612   Terms.getUniqueTermCount -> .size().  (Iulius Curt via Mike McCandless)
 9614 * LUCENE-3514: IndexSearcher.setDefaultFieldSortScoring was removed
 9615   and replaced with per-search control via new expert search methods
 9616   that take two booleans indicating whether hit scores and max
 9617   score should be computed.  (Mike McCandless)
 9619 * LUCENE-4055: You can't put foreign files into the index dir anymore.
 9621 * LUCENE-3866: CompositeReader.getSequentialSubReaders() now returns
 9622   unmodifiable List<? extends IndexReader>. ReaderUtil.Gather was
 9623   removed, as IndexReaderContext.leaves() is now the preferred way
 9624   to access sub-readers.  (Uwe Schindler)
 9626 * LUCENE-4155: oal.util.ReaderUtil, TwoPhaseCommit, TwoPhaseCommitTool
 9627   classes were moved to oal.index package. oal.util.CodecUtil class was moved
 9628   to oal.codecs package. oal.util.DummyConcurrentLock was removed
 9629   (no longer used in Lucene 4.0).  (Uwe Schindler)
 9631 Changes in Runtime Behavior
 9633 * LUCENE-2846: omitNorms now behaves like omitTermFrequencyAndPositions, if you
 9634   omitNorms(true) for field "a" for 1000 documents, but then add a document with
 9635   omitNorms(false) for field "a", all documents for field "a" will have no 
 9636   norms.  Previously, Lucene would fill the first 1000 documents with 
 9637   "fake norms" from Similarity.getDefault(). (Robert Muir, Mike McCandless)
 9639 * LUCENE-2846: When some documents contain field "a", and others do not, the
 9640   documents that don't have the field get a norm byte value of 0. Previously, 
 9641   Lucene would populate "fake norms" with Similarity.getDefault() for these 
 9642   documents.  (Robert Muir, Mike McCandless)
 9644 * LUCENE-2720: IndexWriter throws IndexFormatTooOldException on open, rather 
 9645   than later when e.g. a merge starts. 
 9646   (Shai Erera, Mike McCandless, Uwe Schindler)
 9648 * LUCENE-2881: FieldInfos is now tracked per segment.  Before it was tracked
 9649   per IndexWriter session, which resulted in FieldInfos that had the FieldInfo
 9650   properties from all previous segments combined. Field numbers are now tracked
 9651   globally across IndexWriter sessions and persisted into a _X.fnx file on
 9652   successful commit. The corresponding file format changes are backwards-
 9653   compatible. (Michael Busch, Simon Willnauer)
 9655 * LUCENE-2956, LUCENE-2573, LUCENE-2324, LUCENE-2555: Changes from 
 9656   DocumentsWriterPerThread:
 9658   - IndexWriter now uses a DocumentsWriter per thread when indexing documents.
 9659     Each DocumentsWriterPerThread indexes documents in its own private segment,
 9660     and the in memory segments are no longer merged on flush.  Instead, each
 9661     segment is separately flushed to disk and subsequently merged with normal
 9662     segment merging.
 9664   - DocumentsWriterPerThread (DWPT) is now flushed concurrently based on a
 9665     FlushPolicy.  When a DWPT is flushed, a fresh DWPT is swapped in so that
 9666     indexing may continue concurrently with flushing.  The selected
 9667     DWPT flushes all its RAM resident documents do disk.  Note: Segment flushes
 9668     don't flush all RAM resident documents but only the documents private to
 9669     the DWPT selected for flushing. 
 9671   - Flushing is now controlled by FlushPolicy that is called for every add,
 9672     update or delete on IndexWriter. By default DWPTs are flushed either on
 9673     maxBufferedDocs per DWPT or the global active used memory. Once the active
 9674     memory exceeds ramBufferSizeMB only the largest DWPT is selected for
 9675     flushing and the memory used by this DWPT is subtracted from the active
 9676     memory and added to a flushing memory pool, which can lead to temporarily
 9677     higher memory usage due to ongoing indexing.
 9679   - IndexWriter now can utilize ramBufferSize > 2048 MB. Each DWPT can address
 9680     up to 2048 MB memory such that the ramBufferSize is now bounded by the max
 9681     number of DWPT available in the used DocumentsWriterPerThreadPool.
 9682     IndexWriters net memory consumption can grow far beyond the 2048 MB limit if
 9683     the application can use all available DWPTs. To prevent a DWPT from
 9684     exhausting its address space IndexWriter will forcefully flush a DWPT if its
 9685     hard memory limit is exceeded. The RAMPerThreadHardLimitMB can be controlled
 9686     via IndexWriterConfig and defaults to 1945 MB. 
 9687     Since IndexWriter flushes DWPT concurrently not all memory is released
 9688     immediately. Applications should still use a ramBufferSize significantly
 9689     lower than the JVMs available heap memory since under high load multiple
 9690     flushing DWPT can consume substantial transient memory when IO performance
 9691     is slow relative to indexing rate.
 9693   - IndexWriter#commit now doesn't block concurrent indexing while flushing all
 9694     'currently' RAM resident documents to disk. Yet, flushes that occur while a
 9695     a full flush is running are queued and will happen after all DWPT involved
 9696     in the full flush are done flushing. Applications using multiple threads
 9697     during indexing and trigger a full flush (eg call commit() or open a new
 9698     NRT reader) can use significantly more transient memory.
 9700   - IndexWriter#addDocument and IndexWriter.updateDocument can block indexing
 9701     threads if the number of active + number of flushing DWPT exceed a
 9702     safety limit. By default this happens if 2 * max number available thread
 9703     states (DWPTPool) is exceeded. This safety limit prevents applications from
 9704     exhausting their available memory if flushing can't keep up with
 9705     concurrently indexing threads.  
 9707   - IndexWriter only applies and flushes deletes if the maxBufferedDelTerms
 9708     limit is reached during indexing. No segment flushes will be triggered
 9709     due to this setting.
 9711   - IndexWriter#flush(boolean, boolean) doesn't synchronized on IndexWriter
 9712     anymore. A dedicated flushLock has been introduced to prevent multiple full-
 9713     flushes happening concurrently. 
 9715   - DocumentsWriter doesn't write shared doc stores anymore. 
 9717   (Mike McCandless, Michael Busch, Simon Willnauer)
 9719 * LUCENE-3309: Stored fields no longer record whether they were
 9720   tokenized or not.  In general you should not rely on stored fields
 9721   to record any "metadata" from indexing (tokenized, omitNorms,
 9722   IndexOptions, boost, etc.)  (Mike McCandless)
 9724 * LUCENE-3309: Fast vector highlighter now inserts the
 9725   MultiValuedSeparator for NOT_ANALYZED fields (in addition to
 9726   ANALYZED fields).  To ensure your offsets are correct you should
 9727   provide an analyzer that returns 1 from the offsetGap method.
 9728   (Mike McCandless)
 9730 * LUCENE-2621: Removed contrib/instantiated.  (Robert Muir)
 9732 * LUCENE-1768: StandardQueryTreeBuilder no longer uses RangeQueryNodeBuilder
 9733   for RangeQueryNodes, since theses two classes were removed;
 9734   TermRangeQueryNodeProcessor now creates TermRangeQueryNode,
 9735   instead of RangeQueryNode; the same applies for numeric nodes;
 9736   (Vinicius Barros via Uwe Schindler)
 9738 * LUCENE-3455: QueryParserBase.newFieldQuery() will throw a ParseException if
 9739   any of the calls to the Analyzer throw an IOException.  QueryParseBase.analyzeRangePart()
 9740   will throw a RuntimeException if an IOException is thrown by the Analyzer.
 9742 * LUCENE-4127: IndexWriter will now throw IllegalArgumentException if
 9743   the first token of an indexed field has 0 positionIncrement
 9744   (previously it silently corrected it to 1, possibly masking bugs).
 9745   OffsetAttributeImpl will throw IllegalArgumentException if startOffset
 9746   is less than endOffset, or if offsets are negative.
 9747   (Robert Muir, Mike McCandless)
 9749 API Changes
 9751 * LUCENE-2302, LUCENE-1458, LUCENE-2111, LUCENE-2514: Terms are no longer
 9752   required to be character based. Lucene views a term as an arbitrary byte[]:
 9753   during analysis, character-based terms are converted to UTF8 byte[],
 9754   but analyzers are free to directly create terms as byte[]
 9755   (NumericField does this, for example).  The term data is buffered as
 9756   byte[] during indexing, written as byte[] into the terms dictionary,
 9757   and iterated as byte[] (wrapped in a BytesRef) by IndexReader for
 9758   searching.
 9760 * LUCENE-1458, LUCENE-2111: AtomicReader now directly exposes its
 9761   deleted docs (getDeletedDocs), providing a new Bits interface to
 9762   directly query by doc ID.
 9764 * LUCENE-2691: IndexWriter.getReader() has been made package local and is now
 9765   exposed via open and reopen methods on DirectoryReader.  The semantics of the
 9766   call is the same as it was prior to the API change.
 9767   (Grant Ingersoll, Mike McCandless)
 9769 * LUCENE-2566: QueryParser: Unary operators +,-,! will not be treated as
 9770   operators if they are followed by whitespace. (yonik)
 9772 * LUCENE-2831: Weight#scorer, Weight#explain, Filter#getDocIdSet,
 9773   Collector#setNextReader & FieldComparator#setNextReader now expect an
 9774   AtomicReaderContext instead of an IndexReader. (Simon Willnauer)
 9776 * LUCENE-2892: Add QueryParser.newFieldQuery (called by getFieldQuery by 
 9777   default) which takes Analyzer as a parameter, for easier customization by 
 9778   subclasses.  (Robert Muir)
 9780 * LUCENE-2953: In addition to changes in 3.x, PriorityQueue#initialize(int)
 9781   function was moved into the ctor. (Uwe Schindler, Yonik Seeley)
 9783 * LUCENE-3219: SortField type properties have been moved to an enum
 9784   SortField.Type.  In be consistent, CachedArrayCreator.getSortTypeID() has
 9785   been changed CachedArrayCreator.getSortType(). (Chris Male)
 9787 * LUCENE-3225: Add TermsEnum.seekExact for faster seeking when you
 9788   don't need the ceiling term; renamed existing seek methods to either
 9789   seekCeil or seekExact; changed seekExact(ord) to return no value.
 9790   Fixed MemoryCodec and SimpleTextCodec to optimize the seekExact
 9791   case, and fixed places in Lucene to use seekExact when possible.
 9792   (Mike McCandless)
 9794 * LUCENE-1536: Filter.getDocIdSet() now takes an acceptDocs Bits interface (like
 9795   Scorer) limiting the documents that can appear in the returned DocIdSet.
 9796   Filters are now required to respect these acceptDocs, otherwise deleted documents
 9797   may get returned by searches. Most filters will pass these Bits down to DocsEnum,
 9798   but those, e.g. working on FieldCache, may need to use BitsFilteredDocIdSet.wrap()
 9799   to exclude them.
 9800   (Mike McCandless, Uwe Schindler, Robert Muir, Chris Male, Yonik Seeley,
 9801   Jason Rutherglen, Paul Elschot)
 9803 * LUCENE-3722: Similarity methods and collection/term statistics now take
 9804   long instead of int (to enable distributed scoring of > 2B docs).
 9805   (Yonik Seeley, Andrzej Bialecki, Robert Muir)
 9807 * LUCENE-3761: Generalize SearcherManager into an abstract ReferenceManager.
 9808   SearcherManager remains a concrete class, but due to the refactoring, the 
 9809   method maybeReopen has been deprecated in favor of maybeRefresh().
 9810   (Shai Erera, Mike McCandless, Simon Willnauer)
 9812 * LUCENE-3859: AtomicReader.hasNorms(field) is deprecated, instead you
 9813   can inspect the FieldInfo yourself to see if norms are present, which
 9814   also allows you to get the type.  (Robert Muir)
 9816 * LUCENE-2606: Changed RegexCapabilities interface to fix thread 
 9817   safety, serialization, and performance problems. If you have
 9818   written a custom RegexCapabilities it will need to be updated
 9819   to the new API.  (Robert Muir, Uwe Schindler)
 9821 * LUCENE-2638 MakeHighFreqTerms.TermStats public to make it more useful
 9822   for API use. (Andrzej Bialecki)
 9824 * LUCENE-2912: The field-specific hashmaps in SweetSpotSimilarity were removed.
 9825   Instead, use PerFieldSimilarityWrapper to return different SweetSpotSimilaritys
 9826   for different fields, this way all parameters (such as TF factors) can be 
 9827   customized on a per-field basis.  (Robert Muir)
 9829 * LUCENE-3308: DuplicateFilter keepMode and processingMode have been converted to
 9830   enums DuplicateFilter.KeepMode and DuplicateFilter.ProcessingMode respectively.
 9832 * LUCENE-3483: Move Function grouping collectors from Solr to grouping module.
 9833   (Martijn van Groningen)
 9835 * LUCENE-3606: FieldNormModifier was deprecated, because IndexReader's
 9836   setNorm() was deprecated. Furthermore, this class is broken, as it does
 9837   not take position overlaps into account while recalculating norms.
 9838   (Uwe Schindler, Robert Muir)
 9840 * LUCENE-3936: Renamed StringIndexDocValues to DocTermsIndexDocValues.
 9841   (Martijn van Groningen)
 9843 * LUCENE-1768: Deprecated Parametric(Range)QueryNode, RangeQueryNode(Builder),
 9844   ParametricRangeQueryNodeProcessor were removed. (Vinicius Barros via Uwe Schindler)
 9846 * LUCENE-3820: Deprecated constructors accepting pattern matching bounds. The input
 9847   is buffered and matched in one pass. (Dawid Weiss)
 9849 * LUCENE-2413: Deprecated PatternAnalyzer in common/miscellaneous, in favor 
 9850   of the pattern package (CharFilter, Tokenizer, TokenFilter).  (Robert Muir)
 9852 * LUCENE-2413: Removed the AnalyzerUtil in common/miscellaneous.  (Robert Muir)
 9854 * LUCENE-1370: Added ShingleFilter option to output unigrams if no shingles
 9855   can be generated. (Chris Harris via Steven Rowe)
 9857 * LUCENE-2514, LUCENE-2551: JDK and ICU CollationKeyAnalyzers were changed to
 9858   use pure byte keys when Version >= 4.0. This cuts sort key size approximately
 9859   in half. (Robert Muir)
 9861 * LUCENE-3400: Removed DutchAnalyzer.setStemDictionary (Chris Male)
 9863 * LUCENE-3431: Removed QueryAutoStopWordAnalyzer.addStopWords* deprecated methods
 9864   since they prevented reuse.  Stopwords are now generated at instantiation through
 9865   the Analyzer's constructors. (Chris Male)
 9867 * LUCENE-3434: Removed ShingleAnalyzerWrapper.set* and PerFieldAnalyzerWrapper.addAnalyzer
 9868   since they prevent reuse.  Both Analyzers should be configured at instantiation.
 9869   (Chris Male)
 9871 * LUCENE-3765: Stopset ctors that previously took Set<?> or Map<?,String> now take
 9872   CharArraySet and CharArrayMap respectively. Previously the behavior was confusing,
 9873   and sometimes different depending on the type of set, and ultimately a CharArraySet
 9874   or CharArrayMap was always used anyway.  (Robert Muir)
 9876 * LUCENE-3830: Switched to NormalizeCharMap.Builder to create
 9877   immutable instances of NormalizeCharMap. (Dawid Weiss, Mike
 9878   McCandless)
 9880 * LUCENE-4063: FrenchLightStemmer no longer deletes repeated digits.
 9881   (Tanguy Moal via Steve Rowe)
 9883 * LUCENE-4122: Replace Payload with BytesRef. (Andrzej Bialecki)
 9885 * LUCENE-4132: IndexWriter.getConfig() now returns a LiveIndexWriterConfig object 
 9886   which can be used to change the IndexWriter's live settings. IndexWriterConfig 
 9887   is used only for initializing the IndexWriter. (Shai Erera)
 9889 * LUCENE-3866: IndexReaderContext.leaves() is now the preferred way to access
 9890   atomic sub-readers of any kind of IndexReader (for AtomicReaders it returns
 9891   itself as only leaf with docBase=0).  (Uwe Schindler)
 9893 New features
 9895 * LUCENE-2604: Added RegexpQuery support to QueryParser. Regular expressions
 9896   are directly supported by the standard queryparser via 
 9897      fieldName:/expression/ OR /expression against default field/
 9898   Users who wish to search for literal "/" characters are advised to 
 9899   backslash-escape or quote those characters as needed. 
 9900   (Simon Willnauer, Robert Muir)
 9902 * LUCENE-1606, LUCENE-2089: Adds AutomatonQuery, a MultiTermQuery that 
 9903   matches terms against a finite-state machine. Implement WildcardQuery
 9904   and FuzzyQuery with finite-state methods. Adds RegexpQuery.
 9905   (Robert Muir, Mike McCandless, Uwe Schindler, Mark Miller)
 9907 * LUCENE-3662: Add support for levenshtein distance with transpositions
 9908   to LevenshteinAutomata, FuzzyTermsEnum, and DirectSpellChecker.
 9909   (Jean-Philippe Barrette-LaPierre, Robert Muir)
 9911 * LUCENE-2321: Cutover to a more RAM efficient packed-ints based
 9912   representation for the in-memory terms dict index.  (Mike
 9913   McCandless)
 9915 * LUCENE-2126: Add new classes for data (de)serialization: DataInput
 9916   and DataOutput.  IndexInput and IndexOutput extend these new classes.
 9917   (Michael Busch)
 9919 * LUCENE-1458, LUCENE-2111: With flexible indexing it is now possible
 9920   for an application to create its own postings codec, to alter how
 9921   fields, terms, docs and positions are encoded into the index.  The
 9922   standard codec is the default codec. IndexWriter accepts a Codec
 9923   class to obtain codecs for newly written segments.
 9925 * LUCENE-1458, LUCENE-2111: Some experimental codecs have been added
 9926   for flexible indexing, including pulsing codec (inlines
 9927   low-frequency terms directly into the terms dict, avoiding seeking
 9928   for some queries), sep codec (stores docs, freqs, positions, skip
 9929   data and payloads in 5 separate files instead of the 2 used by
 9930   standard codec), and int block (really a "base" for using
 9931   block-based compressors like PForDelta for storing postings data).
 9933 * LUCENE-1458, LUCENE-2111: The in-memory terms index used by standard
 9934   codec is more RAM efficient: terms data is stored as block byte
 9935   arrays and packed integers.  Net RAM reduction for indexes that have
 9936   many unique terms should be substantial, and initial open time for
 9937   IndexReaders should be faster.  These gains only apply for newly
 9938   written segments after upgrading.
 9940 * LUCENE-1458, LUCENE-2111: Terms data are now buffered directly as
 9941   byte[] during indexing, which uses half the RAM for ascii terms (and
 9942   also numeric fields).  This can improve indexing throughput for
 9943   applications that have many unique terms, since it reduces how often
 9944   a new segment must be flushed given a fixed RAM buffer size.
 9946 * LUCENE-2489: Added PerFieldCodecWrapper (in oal.index.codecs) which
 9947   lets you set the Codec per field (Mike McCandless)
 9949 * LUCENE-2373: Extend Codec to use SegmentInfosWriter and
 9950   SegmentInfosReader to allow customization of SegmentInfos data.
 9951   (Andrzej Bialecki)
 9953 * LUCENE-2504: FieldComparator.setNextReader now returns a
 9954   FieldComparator instance.  You can "return this", to just reuse the
 9955   same instance, or you can return a comparator optimized to the new
 9956   segment.  (yonik, Mike McCandless)
 9958 * LUCENE-2648: PackedInts.Iterator now supports to advance by more than a
 9959   single ordinal. (Simon Willnauer) 
 9961 * LUCENE-2649: Objects in the FieldCache can optionally store Bits
 9962   that mark which docs have real values in the native[] (ryan)
 9964 * LUCENE-2664: Add SimpleText codec, which stores all terms/postings
 9965   data in a single text file for transparency (at the expense of poor
 9966   performance).  (Sahin Buyrukbilen via Mike McCandless)
 9968 * LUCENE-2589: Add a VariableSizedIntIndexInput, which, when used w/
 9969   Sep*, makes it simple to take any variable sized int block coders
 9970   (like Simple9/16) and use them in a codec.  (Mike McCandless)
 9972 * LUCENE-2597: Add oal.index.SlowCompositeReaderWrapper, to wrap a
 9973   composite reader (eg MultiReader or DirectoryReader), making it
 9974   pretend it's an atomic reader.  This is a convenience class (you can
 9975   use MultiFields static methods directly, instead) if you need to use
 9976   the flex APIs directly on a composite reader.  (Mike McCandless)
 9978 * LUCENE-2690: MultiTermQuery boolean rewrites per segment.
 9979   (Uwe Schindler, Robert Muir, Mike McCandless, Simon Willnauer)
 9981 * LUCENE-996: The QueryParser now accepts mixed inclusive and exclusive
 9982   bounds for range queries.  Example: "{3 TO 5]"
 9983   QueryParser subclasses that overrode getRangeQuery will need to be changed
 9984   to use the new getRangeQuery method.  (Andrew Schurman, Mark Miller, yonik)
 9986 * LUCENE-2742: Add native per-field postings format support. Codec lets you now
 9987   register a postings format for each field and which is in turn recorded 
 9988   into the index. Postings formats are maintained on a per-segment basis and be
 9989   resolved without knowing the actual postings format used for writing the segment.
 9990   (Simon Willnauer)
 9992 * LUCENE-2741: Add support for multiple codecs that use the same file
 9993   extensions within the same segment. Codecs now use their per-segment codec
 9994   ID in the file names. (Simon Willnauer)
 9996 * LUCENE-2843: Added a new terms index impl,
 9997   VariableGapTermsIndexWriter/Reader, that accepts a pluggable
 9998   IndexTermSelector for picking which terms should be indexed in the
 9999   terms dict.  This impl stores the indexed terms in an FST, which is
10000   much more RAM efficient than FixedGapTermsIndex.  (Mike McCandless)
10002 * LUCENE-2862: Added TermsEnum.totalTermFreq() and
10003   Terms.getSumTotalTermFreq().  (Mike McCandless, Robert Muir)
10005 * LUCENE-3290: Added Terms.getSumDocFreq()  (Mike McCandless, Robert Muir)
10007 * LUCENE-3003: Added new expert class oal.index.DocTermsOrd,
10008   refactored from Solr's UnInvertedField, for accessing term ords for
10009   multi-valued fields, per document.  This is similar to FieldCache in
10010   that it inverts the index to compute the ords, but differs in that
10011   it's able to handle multi-valued fields and does not hold the term
10012   bytes in RAM. (Mike McCandless)
10014 * LUCENE-3108, LUCENE-2935, LUCENE-2168, LUCENE-1231: Changes from 
10015   DocValues (ColumnStrideFields):
10017   - IndexWriter now supports typesafe dense per-document values stored in
10018     a column like storage. DocValues are stored on a per-document
10019     basis where each documents field can hold exactly one value of a given
10020     type. DocValues are provided via Fieldable and can be used in
10021     conjunction with stored and indexed values.
10023   - DocValues provides an entirely RAM resident document id to value
10024     mapping per field as well as a DocIdSetIterator based disk-resident
10025     sequential access API relying on filesystem-caches.
10027   - Both APIs are exposed via IndexReader and the Codec / Flex API allowing
10028     expert users to integrate customized DocValues reader and writer
10029     implementations by extending existing Codecs.
10031   - DocValues provides implementations for primitive datatypes like int,
10032     long, float, double and arrays of byte. Byte based implementations further
10033     provide storage variants like straight or dereferenced stored bytes, fixed
10034     and variable length bytes as well as index time sorted based on 
10035     user-provided comparators.
10037   (Mike McCandless, Simon Willnauer)  
10039 * LUCENE-3209: Added MemoryCodec, which stores all terms & postings in
10040   RAM as an FST; this is good for primary-key fields if you frequently
10041   need to lookup by that field or perform deletions against it, for
10042   example in a near-real-time setting. (Mike McCandless)
10044 * SOLR-2533: Added support for rewriting Sort and SortFields using an
10045   IndexSearcher.  SortFields can have SortField.REWRITEABLE type which
10046   requires they are rewritten before they are used. (Chris Male)
10048 * LUCENE-3203: FSDirectory can now limit the max allowed write rate
10049   (MB/sec) of all running merges, to reduce impact ongoing merging has
10050   on searching, NRT reopen time, etc.  (Mike McCandless)
10052 * LUCENE-2793: Directory#createOutput & Directory#openInput now accept an
10053   IOContext instead of a buffer size to allow low level optimizations for 
10054   different usecases like merging, flushing and reading.
10055   (Simon Willnauer, Mike McCandless, Varun Thacker)
10057 * LUCENE-3354: FieldCache can cache DocTermOrds. (Martijn van Groningen)
10059 * LUCENE-3376: ReusableAnalyzerBase has been moved from modules/analysis/common
10060   into lucene/src/java/org/apache/lucene/analysis (Chris Male)
10062 * LUCENE-3423: add Terms.getDocCount(), which returns the number of documents
10063   that have at least one term for a field.  (Yonik Seeley, Robert Muir)
10065 * LUCENE-2959: Added a variety of different relevance ranking systems to Lucene.
10067   - Added Okapi BM25, Language Models, Divergence from Randomness, and 
10068     Information-Based Models. The models are pluggable, support all of lucene's 
10069     features (boosts, slops, explanations, etc) and queries (spans, etc).
10071   - All models default to the same index-time norm encoding as 
10072     DefaultSimilarity, so you can easily try these out/switch back and 
10073     forth/run experiments and comparisons without reindexing. Note: most of 
10074     the models do rely upon index statistics that are new in Lucene 4.0, so 
10075     for existing 3.x indexes it's a good idea to upgrade your index to the 
10076     new format with IndexUpgrader first.
10078   - Added a new subclass SimilarityBase which provides a simplified API 
10079     for plugging in new ranking algorithms without dealing with all of the
10080     nuances and implementation details of Lucene. 
10082   - For example, to use BM25 for all fields: 
10083      searcher.setSimilarity(new BM25Similarity());
10085     If you instead want to apply different similarities (e.g. ones with 
10086     different parameter values or different algorithms entirely) to different
10087     fields, implement PerFieldSimilarityWrapper with your per-field logic.
10089   (David Mark Nemeskey via Robert Muir)
10091 * LUCENE-3396: ReusableAnalyzerBase now provides a ReuseStrategy abstraction 
10092   which controls how TokenStreamComponents are reused per request.  Two 
10093   implementations are provided - GlobalReuseStrategy which implements the
10094   current behavior of sharing components between all fields, and 
10095   PerFieldReuseStrategy which shares per field.  (Chris Male)
10097 * LUCENE-2309: Added IndexableField.tokenStream(Analyzer) which is now 
10098   responsible for creating the TokenStreams for Fields when they are to 
10099   be indexed.  (Chris Male)
10101 * LUCENE-3433: Added random access for non RAM resident IndexDocValues. RAM
10102   resident and disk resident IndexDocValues are now exposed via the Source
10103   interface. ValuesEnum has been removed in favour of Source. (Simon Willnauer)
10105 * LUCENE-1536: Filters can now be applied down-low, if their DocIdSet implements
10106   a new bits() method, returning all documents in a random access way. If the
10107   DocIdSet is not too sparse, it will be passed as acceptDocs down to the Scorer
10108   as replacement for AtomicReader's live docs.
10109   In addition, FilteredQuery backs now IndexSearcher's filtering search methods.
10110   Using FilteredQuery you can chain Filters in a very performant way
10111   [new FilteredQuery(new FilteredQuery(query, filter1), filter2)], which was not
10112   possible with IndexSearcher's methods. FilteredQuery also allows to override
10113   the heuristics used to decide if filtering should be done random access or
10114   using a conjunction on DocIdSet's iterator().
10115   (Mike McCandless, Uwe Schindler, Robert Muir, Chris Male, Yonik Seeley,
10116   Jason Rutherglen, Paul Elschot)
10118 * LUCENE-3638: Added sugar methods to IndexReader and IndexSearcher to
10119   load only certain fields when loading a document.  (Peter Chang via
10120   Mike McCandless)
10122 * LUCENE-3628: Norms are represented as DocValues. AtomicReader exposes
10123   a #normValues(String) method to obtain norms per field. (Simon Willnauer) 
10125 * LUCENE-3687: Similarity#computeNorm(FieldInvertState, Norm) allows to compute
10126   norm values or arbitrary precision. Instead of returning a fixed single byte
10127   value, custom similarities can now set a integer, float or byte value to the
10128   given Norm object. (Simon Willnauer)  
10130 * LUCENE-2604, LUCENE-4103: Added RegexpQuery support to contrib/queryparser.
10131   (Simon Willnauer, Robert Muir, Daniel Truemper)
10133 * LUCENE-2373: Added a Codec implementation that works with append-only
10134   filesystems (such as e.g. Hadoop DFS). SegmentInfos writing/reading
10135   code is refactored to support append-only FS, and to allow for future
10136   customization of per-segment information. (Andrzej Bialecki)
10138 * LUCENE-2479: Added ability to provide a sort comparator for spelling suggestions along
10139   with two implementations.  The existing comparator (score, then frequency) is the default (Grant Ingersoll)
10141 * LUCENE-2608: Added the ability to specify the accuracy at method time in the SpellChecker.  The per class
10142   method is also still available.  (Grant Ingersoll)
10144 * LUCENE-2507: Added DirectSpellChecker, which retrieves correction candidates directly 
10145   from the term dictionary using levenshtein automata.  (Robert Muir)
10147 * LUCENE-3527: Add LuceneLevenshteinDistance, which computes string distance in a compatible
10148   way as DirectSpellChecker. This can be used to merge top-N results from more than one
10149   SpellChecker.  (James Dyer via Robert Muir)
10151 * LUCENE-3496: Support grouping by DocValues. (Martijn van Groningen)
10153 * LUCENE-2795: Generified DirectIOLinuxDirectory to work across any
10154   unix supporting the O_DIRECT flag when opening a file (tested on
10155   Linux and OS X but likely other Unixes will work), and improved it
10156   so it can be used for indexing and searching.  The directory uses
10157   direct IO when doing large merges to avoid  unnecessarily evicting
10158   cached IO pages due to large merges.  (Varun Thacker, Mike
10159   McCandless)
10161 * LUCENE-3827: DocsAndPositionsEnum from MemoryIndex implements
10162   start/endOffset, if offsets are indexed. (Alan Woodward via Mike
10163   McCandless)
10165 * LUCENE-3802, LUCENE-3856: Support for grouped faceting. (Martijn van Groningen)
10167 * LUCENE-3444: Added a second pass grouping collector that keeps track of distinct
10168   values for a specified field for the top N group. (Martijn van Groningen)
10170 * LUCENE-3778: Added a grouping utility class that makes it easier to use result
10171   grouping for pure Lucene apps. (Martijn van Groningen)
10173 * LUCENE-2341: A new analysis/ filter: Morfologik - a dictionary-driven lemmatizer 
10174   (accurate stemmer) for Polish (includes morphosyntactic annotations).
10175   (Michał Dybizbański, Dawid Weiss)
10177 * LUCENE-2413: Consolidated Lucene/Solr analysis components into analysis/common. 
10178   New features from Solr now available to Lucene users include:
10179    - o.a.l.analysis.commongrams: Constructs n-grams for frequently occurring terms
10180      and phrases. 
10181    - o.a.l.analysis.charfilter.HTMLStripCharFilter: CharFilter that strips HTML 
10182      constructs.
10183    - o.a.l.analysis.miscellaneous.WordDelimiterFilter: TokenFilter that splits words 
10184      into subwords and performs optional transformations on subword groups.
10185    - o.a.l.analysis.miscellaneous.RemoveDuplicatesTokenFilter: TokenFilter which 
10186      filters out Tokens at the same position and Term text as the previous token.
10187    - o.a.l.analysis.miscellaneous.TrimFilter: Trims leading and trailing whitespace 
10188      from Tokens in the stream.
10189    - o.a.l.analysis.miscellaneous.KeepWordFilter: A TokenFilter that only keeps tokens 
10190      with text contained in the required words (inverse of StopFilter).
10191    - o.a.l.analysis.miscellaneous.HyphenatedWordsFilter: A TokenFilter that puts 
10192      hyphenated words broken into two lines back together.
10193    - o.a.l.analysis.miscellaneous.CapitalizationFilter: A TokenFilter that applies
10194      capitalization rules to tokens.
10195    - o.a.l.analysis.pattern: Package for pattern-based analysis, containing a 
10196      CharFilter, Tokenizer, and TokenFilter for transforming text with regexes.
10197    - o.a.l.analysis.synonym.SynonymFilter: A synonym filter that supports multi-word
10198      synonyms.
10199    - o.a.l.analysis.phonetic: Package for phonetic search, containing various
10200      phonetic encoders such as Double Metaphone.
10202    Some existing analysis components changed packages:
10203     - o.a.l.analysis.KeywordAnalyzer -> o.a.l.analysis.core.KeywordAnalyzer
10204     - o.a.l.analysis.KeywordTokenizer -> o.a.l.analysis.core.KeywordTokenizer
10205     - o.a.l.analysis.LetterTokenizer -> o.a.l.analysis.core.LetterTokenizer
10206     - o.a.l.analysis.LowerCaseFilter -> o.a.l.analysis.core.LowerCaseFilter
10207     - o.a.l.analysis.LowerCaseTokenizer -> o.a.l.analysis.core.LowerCaseTokenizer
10208     - o.a.l.analysis.SimpleAnalyzer -> o.a.l.analysis.core.SimpleAnalyzer
10209     - o.a.l.analysis.StopAnalyzer -> o.a.l.analysis.core.StopAnalyzer
10210     - o.a.l.analysis.StopFilter -> o.a.l.analysis.core.StopFilter
10211     - o.a.l.analysis.WhitespaceAnalyzer -> o.a.l.analysis.core.WhitespaceAnalyzer
10212     - o.a.l.analysis.WhitespaceTokenizer -> o.a.l.analysis.core.WhitespaceTokenizer
10213     - o.a.l.analysis.PorterStemFilter -> o.a.l.analysis.en.PorterStemFilter
10214     - o.a.l.analysis.ASCIIFoldingFilter -> o.a.l.analysis.miscellaneous.ASCIIFoldingFilter
10215     - o.a.l.analysis.ISOLatin1AccentFilter -> o.a.l.analysis.miscellaneous.ISOLatin1AccentFilter
10216     - o.a.l.analysis.KeywordMarkerFilter -> o.a.l.analysis.miscellaneous.KeywordMarkerFilter
10217     - o.a.l.analysis.LengthFilter -> o.a.l.analysis.miscellaneous.LengthFilter
10218     - o.a.l.analysis.PerFieldAnalyzerWrapper -> o.a.l.analysis.miscellaneous.PerFieldAnalyzerWrapper
10219     - o.a.l.analysis.TeeSinkTokenFilter -> o.a.l.analysis.sinks.TeeSinkTokenFilter
10220     - o.a.l.analysis.CharFilter -> o.a.l.analysis.charfilter.CharFilter
10221     - o.a.l.analysis.BaseCharFilter -> o.a.l.analysis.charfilter.BaseCharFilter
10222     - o.a.l.analysis.MappingCharFilter -> o.a.l.analysis.charfilter.MappingCharFilter
10223     - o.a.l.analysis.NormalizeCharMap -> o.a.l.analysis.charfilter.NormalizeCharMap
10224     - o.a.l.analysis.CharArraySet -> o.a.l.analysis.util.CharArraySet
10225     - o.a.l.analysis.CharArrayMap -> o.a.l.analysis.util.CharArrayMap
10226     - o.a.l.analysis.ReusableAnalyzerBase -> o.a.l.analysis.util.ReusableAnalyzerBase
10227     - o.a.l.analysis.StopwordAnalyzerBase -> o.a.l.analysis.util.StopwordAnalyzerBase
10228     - o.a.l.analysis.WordListLoader -> o.a.l.analysis.util.WordListLoader
10229     - o.a.l.analysis.CharTokenizer -> o.a.l.analysis.util.CharTokenizer
10230     - o.a.l.util.CharacterUtils -> o.a.l.analysis.util.CharacterUtils
10232    All analyzers in contrib/analyzers and contrib/icu were moved to the
10233    analysis/ module.  The 'smartcn' and 'stempel' components now depend on 'common'.
10234    (Chris Male, Robert Muir)
10236 * LUCENE-4004: Add DisjunctionMaxQuery support to the xml query parser.
10237   (Benson Margulies via Robert Muir)
10239 * LUCENE-4025: Add maybeRefreshBlocking to ReferenceManager, to let a caller 
10240   block until the refresh logic has been executed. (Shai Erera, Mike McCandless)
10242 * LUCENE-4039: Add AddIndexesTask to benchmark, which uses IW.addIndexes. 
10243   (Shai Erera)
10245 * LUCENE-3514: Added IndexSearcher.searchAfter when Sort is used,
10246   returning results after a specified FieldDoc for deep
10247   paging.  (Mike McCandless)
10249 * LUCENE-4043: Added scoring support via score mode for query time joining.
10250   (Martijn van Groningen, Mike McCandless)
10252 * LUCENE-3523: Added oal.search.spell.WordBreakSpellChecker, which 
10253     generates suggestions by combining two or more terms and/or 
10254     breaking terms into multiple words.  See Javadocs for usage. (James Dyer)
10256 * LUCENE-4019: Added improved parsing of Hunspell Dictionaries so those
10257   rules missing the required number of parameters either ignored or 
10258   cause a ParseException (depending on whether strict parsing is enabled).
10259   (Luca Cavanna via Chris Male) 
10261 * LUCENE-3440: Add ordered fragments feature with IDF-weighted terms for FVH.
10262   (Sebastian Lutze via Koji Sekiguchi)
10264 * LUCENE-4082: Added explain to ToParentBlockJoinQuery.
10265   (Christoph Kaser, Martijn van Groningen)
10267 * LUCENE-4108: add replaceTaxonomy to DirectoryTaxonomyWriter, which replaces
10268   the taxonomy in place with the given one. (Shai Erera)
10270 * LUCENE-3030: new BlockTree terms dictionary (used by the default
10271   Lucene40 postings format) uses less RAM (for the terms index) and
10272   disk space (for all terms and metadata) and gives sizable
10273   performance gains for terms dictionary intensive operations like
10274   FuzzyQuery, direct spell checker and primary-key lookup (Mike
10275   McCandless).
10277 Optimizations
10279 * LUCENE-2588: Don't store unnecessary suffixes when writing the terms
10280   index, saving RAM in IndexReader; change default terms index
10281   interval from 128 to 32, because the terms index now requires much
10282   less RAM.  (Robert Muir, Mike McCandless)
10284 * LUCENE-2669: Optimize NumericRangeQuery.NumericRangeTermsEnum to
10285   not seek backwards when a sub-range has no terms. It now only seeks
10286   when the current term is less than the next sub-range's lower end.
10287   (Uwe Schindler, Mike McCandless)
10289 * LUCENE-2694: Optimize MultiTermQuery to be single pass for Term lookups.
10290   MultiTermQuery now stores TermState per leaf reader during rewrite to re-
10291   seek the term dictionary in TermQuery / TermWeight.
10292   (Simon Willnauer, Mike McCandless, Robert Muir)
10294 * LUCENE-3292: IndexWriter no longer shares the same SegmentReader
10295   instance for merging and NRT readers, which enables directory impls
10296   to separately tune IO flags for each.  (Varun Thacker, Simon
10297   Willnauer, Mike McCandless)
10299 * LUCENE-3328: BooleanQuery now uses a specialized ConjunctionScorer if all
10300   boolean clauses are required and instances of TermQuery. 
10301   (Simon Willnauer, Robert Muir)
10303 * LUCENE-3643: FilteredQuery and IndexSearcher.search(Query, Filter,...)
10304   now optimize the special case query instanceof MatchAllDocsQuery to
10305   execute as ConstantScoreQuery.  (Uwe Schindler)
10307 * LUCENE-3509: Added fasterButMoreRam option for docvalues. This option controls whether the space for packed ints
10308   should be rounded up for better performance. This option only applies for docvalues types bytes fixed sorted
10309   and bytes var sorted. (Simon Willnauer, Martijn van Groningen)
10311 * LUCENE-3795: Replace contrib/spatial with modules/spatial.  This includes 
10312   a basic spatial strategy interface.  (David Smiley, Chris Male, ryan)
10314 * LUCENE-3932: Lucene3x codec loads terms index faster, by
10315   pre-allocating the packed ints array based on the .tii file size
10316   (Sean Bridges via Mike McCandless)
10318 * LUCENE-3468: Replaced last() and remove() with pollLast() in
10319   FirstPassGroupingCollector (Martijn van Groningen)
10321 * LUCENE-3830: Changed MappingCharFilter/NormalizeCharMap to use an
10322   FST under the hood, which requires less RAM.  NormalizeCharMap no
10323   longer accepts empty string match (it did previously, but ignored
10324   it).  (Dawid Weiss, Mike McCandless)
10326 * LUCENE-4061: improve synchronization in DirectoryTaxonomyWriter.addCategory
10327   and few general improvements to DirectoryTaxonomyWriter.
10328   (Shai Erera, Gilad Barkai)
10330 * LUCENE-4062: Add new aligned packed bits impls for faster lookup
10331   performance; add float acceptableOverheadRatio to getWriter and
10332   getMutable API to give packed ints freedom to pick faster
10333   implementations (Adrien Grand via Mike McCandless)
10335 * LUCENE-2357: Reduce transient RAM usage when merging segments in
10336   IndexWriter. (Adrien Grand)
10338 * LUCENE-4098: Add bulk get/set methods to PackedInts (Adrien Grand
10339   via Mike McCandless)
10341 * LUCENE-4156: DirectoryTaxonomyWriter.getSize is no longer synchronized.
10342   (Shai Erera, Sivan Yogev)
10344 * LUCENE-4163: Improve concurrency of MMapIndexInput.clone() by using
10345   the new WeakIdentityMap on top of a ConcurrentHashMap to manage
10346   the cloned instances. WeakIdentityMap was extended to support
10347   iterating over its keys.  (Uwe Schindler)
10349 Bug fixes
10351 * LUCENE-2803: The FieldCache can miss values if an entry for a reader
10352   with more document deletions is requested before a reader with fewer
10353   deletions, provided they share some segments. (yonik)
10355 * LUCENE-2645: Fix false assertion error when same token was added one
10356   after another with 0 posIncr.  (David Smiley, Kurosaka Teruhiko via Mike
10357   McCandless)
10359 * LUCENE-3348: Fix thread safety hazards in IndexWriter that could
10360   rarely cause deletions to be incorrectly applied.  (Yonik Seeley,
10361   Simon Willnauer, Mike McCandless)
10363 * LUCENE-3515: Fix terrible merge performance versus 3.x, especially
10364   when the directory isn't MMapDirectory, due to failing to reuse
10365   DocsAndPositionsEnum while merging (Marc Sturlese, Erick Erickson,
10366   Robert Muir, Simon Willnauer, Mike McCandless)
10368 * LUCENE-3589: BytesRef copy(short) didn't set length.
10369   (Peter Chang via Robert Muir)
10371 * LUCENE-3045: fixed QueryNodeImpl.containsTag(String key) that was
10372   not lowercasing the key before checking for the tag (Adriano Crestani)
10374 * LUCENE-3890: Fixed NPE for grouped faceting on multi-valued fields.
10375   (Michael McCandless, Martijn van Groningen)
10377 * LUCENE-2945: Fix hashCode/equals for surround query parser generated queries.
10378   (Paul Elschot, Simon Rosenthal, gsingers via ehatcher)
10380 * LUCENE-3971: MappingCharFilter could return invalid final token position.
10381   (Dawid Weiss, Robert Muir)
10383 * LUCENE-3820: PatternReplaceCharFilter could return invalid token positions. 
10384   (Dawid Weiss)
10386 * LUCENE-3969: Throw IAE on bad arguments that could cause confusing errors in
10387   CompoundWordTokenFilterBase, PatternTokenizer, PositionFilter,
10388   SnowballFilter, PathHierarchyTokenizer, ReversePathHierarchyTokenizer, 
10389   WikipediaTokenizer, and KeywordTokenizer. ShingleFilter and 
10390   CommonGramsFilter now populate PositionLengthAttribute. Fixed
10391   PathHierarchyTokenizer to reset() all state. Protect against AIOOBE in
10392   ReversePathHierarchyTokenizer if skip is large. Fixed wrong final
10393   offset calculation in PathHierarchyTokenizer. 
10394   (Mike McCandless, Uwe Schindler, Robert Muir)
10396 * LUCENE-4060: Fix a synchronization bug in 
10397   DirectoryTaxonomyWriter.addTaxonomies(). Also, the method has been renamed to
10398   addTaxonomy and now takes only one Directory and one OrdinalMap.
10399   (Shai Erera, Gilad Barkai)
10401 * LUCENE-3590: Fix AIOOBE in BytesRef/CharsRef copyBytes/copyChars when 
10402   offset is nonzero, fix off-by-one in CharsRef.subSequence, and fix
10403   CharsRef's CharSequence methods to throw exceptions in boundary cases
10404   to properly meet the specification.  (Robert Muir)
10406 * LUCENE-4084: Attempting to reuse a single IndexWriterConfig instance
10407   across more than one IndexWriter resulted in a cryptic exception.
10408   This is now fixed, but requires that certain members of
10409   IndexWriterConfig (MergePolicy, FlushPolicy,
10410   DocumentsWriterThreadPool) implement clone.  (Robert Muir, Simon
10411   Willnauer, Mike McCandless)
10413 * LUCENE-4079: Fixed loading of Hunspell dictionaries that use aliasing (AF rules)
10414   (Ludovic Boutros via Chris Male)
10416 * LUCENE-4077: Expose the max score and per-group scores from
10417   ToParentBlockJoinCollector (Christoph Kaser, Mike McCandless)
10419 * LUCENE-4114: Fix int overflow bugs in BYTES_FIXED_STRAIGHT and
10420   BYTES_FIXED_DEREF doc values implementations (Walt Elder via Mike McCandless).
10422 * LUCENE-4147: Fixed thread safety issues when rollback() and commit()
10423   are called simultaneously.  (Simon Willnauer, Mike McCandless)
10425 * LUCENE-4165: Removed closing of the Reader used to read the affix file in
10426   HunspellDictionary.  Consumers are now responsible for closing all InputStreams
10427   once the Dictionary has been instantiated. (Torsten Krah, Uwe Schindler, Chris Male)
10429 Documentation
10431 * LUCENE-3958: Javadocs corrections for IndexWriter.
10432   (Iulius Curt via Robert Muir)
10434 Build
10436 * LUCENE-4047: Cleanup of LuceneTestCase: moved blocks of initialization/ cleanup
10437   code into JUnit instance and class rules. (Dawid Weiss)
10439 * LUCENE-4016: Require ANT 1.8.2+ for the build. 
10441 * LUCENE-3808: Refactoring of testing infrastructure to use randomizedtesting
10442   package: http://labs.carrotsearch.com/randomizedtesting.html (Dawid Weiss)
10444 * LUCENE-3964: Added target stage-maven-artifacts, which stages
10445   Maven release artifacts to a Maven staging repository in preparation
10446   for release.  (Steve Rowe)
10448 * LUCENE-2845: Moved contrib/benchmark to lucene/benchmark.
10450 * LUCENE-2995: Moved contrib/spellchecker into lucene/suggest.
10452 * LUCENE-3285: Moved contrib/queryparser into lucene/queryparser
10454 * LUCENE-3285: Moved contrib/xml-query-parser's demo into lucene/demo
10456 * LUCENE-3271: Moved contrib/queries BooleanFilter, BoostingQuery, 
10457   ChainedFilter, FilterClause and TermsFilter into lucene/queries
10459 * LUCENE-3381: Moved contrib/queries regex.*, DuplicateFilter,
10460   FuzzyLikeThisQuery and SlowCollated* into lucene/sandbox.
10461   Removed contrib/queries.
10463 * LUCENE-3286: Moved remainder of contrib/xml-query-parser to lucene/queryparser.
10464   Classes now found at org.apache.lucene.queryparser.xml.*
10466 * LUCENE-4059: Improve ANT task prepare-webpages (used by documentation
10467   tasks) to correctly encode build file names as URIs for later processing by
10468   XSL.  (Greg Bowyer, Uwe Schindler)
10471 ======================= Lucene 3.6.2 =======================
10473 Bug Fixes
10475 * LUCENE-4234: Exception when FacetsCollector is used with ScoreFacetRequest,
10476   and the number of matching documents is too large. (Gilad Barkai via Shai Erera)
10478 * LUCENE-2686, LUCENE-3505, LUCENE-4401: Fix BooleanQuery scorers to
10479   return correct freq().
10480   (Koji Sekiguchi, Mike McCandless, Liu Chao, Robert Muir)
10482 * LUCENE-2501: Fixed rare thread-safety issue that could cause
10483   ArrayIndexOutOfBoundsException inside ByteBlockPool (Robert Muir,
10484   Mike McCandless)
10486 * LUCENE-4297: BooleanScorer2 would multiply the coord() factor
10487   twice for conjunctions: for most users this is no problem, but
10488   if you had a customized Similarity that returned something other
10489   than 1 when overlap == maxOverlap (always the case for conjunctions),
10490   then the score would be incorrect.  (Pascal Chollet, Robert Muir)
10492 * LUCENE-4300: BooleanQuery's rewrite was not always safe: if you
10493   had a custom Similarity where coord(1,1) != 1F, then the rewritten
10494   query would be scored differently.  (Robert Muir)
10496 * LUCENE-4398: If you index many different field names in your
10497   documents then due to a bug in how it measures its RAM
10498   usage, IndexWriter would flush each segment too early eventually
10499   reaching the point where it flushes after every doc.  (Tim Smith via
10500   Mike McCandless)
10502 * LUCENE-4411: when sampling is enabled for a FacetRequest, its depth
10503   parameter is reset to the default (1), even if set otherwise.
10504   (Gilad Barkai via Shai Erera)
10506 * LUCENE-4635: Fixed ArrayIndexOutOfBoundsException when in-memory
10507   terms index requires more than 2.1 GB RAM (indices with billions of
10508   terms).  (Tom Burton-West via Mike McCandless)
10510 Documentation
10512 * LUCENE-4302: Fix facet userguide to have HTML loose doctype like
10513   all other javadocs.  (Karl Nicholas via Uwe Schindler)
10516 ======================= Lucene 3.6.1 =======================
10517 More information about this release, including any errata related to the 
10518 release notes, upgrade instructions, or other changes may be found online at:
10519    https://wiki.apache.org/lucene-java/Lucene3.6.1
10521 Bug Fixes
10523 * LUCENE-3969: Throw IAE on bad arguments that could cause confusing 
10524   errors in KeywordTokenizer. 
10525   (Uwe Schindler, Mike McCandless, Robert Muir)
10527 * LUCENE-3971: MappingCharFilter could return invalid final token position.
10528   (Dawid Weiss, Robert Muir)
10530 * LUCENE-4023: DisjunctionMaxScorer now implements visitSubScorers().
10531   (Uwe Schindler)
10533 * LUCENE-2566: + - operators allow any amount of whitespace (yonik, janhoy)
10535 * LUCENE-3590: Fix AIOOBE in BytesRef/CharsRef copyBytes/copyChars when 
10536   offset is nonzero, fix off-by-one in CharsRef.subSequence, and fix
10537   CharsRef's CharSequence methods to throw exceptions in boundary cases
10538   to properly meet the specification.  (Robert Muir)
10540 * LUCENE-4222: TieredMergePolicy.getFloorSegmentMB was returning the
10541   size in bytes not MB (Chris Fuller via Mike McCandless)
10543 API Changes
10545 * LUCENE-4023: Changed the visibility of Scorer#visitSubScorers() to
10546   public, otherwise it's impossible to implement Scorers outside
10547   the Lucene package.  (Uwe Schindler)
10549 Optimizations
10551 * LUCENE-4163: Improve concurrency of MMapIndexInput.clone() by using
10552   the new WeakIdentityMap on top of a ConcurrentHashMap to manage
10553   the cloned instances. WeakIdentityMap was extended to support
10554   iterating over its keys.  (Uwe Schindler)
10556 Tests
10558 * LUCENE-3873: add MockGraphTokenFilter, testing analyzers with
10559   random graph tokens.  (Mike McCandless)
10561 * LUCENE-3968: factor out LookaheadTokenFilter from 
10562   MockGraphTokenFilter (Mike McCandless)
10565 ======================= Lucene 3.6.0 =======================
10566 More information about this release, including any errata related to the 
10567 release notes, upgrade instructions, or other changes may be found online at:
10568    https://wiki.apache.org/lucene-java/Lucene3.6
10570 Changes in backwards compatibility policy
10572 * LUCENE-3594: The protected inner class (never intended to be visible)
10573   FieldCacheTermsFilter.FieldCacheTermsFilterDocIdSet was removed and
10574   replaced by another internal implementation.  (Uwe Schindler)
10576 * LUCENE-3620: FilterIndexReader now overrides all methods of IndexReader that 
10577   it should (note that some are still not overridden, as they should be 
10578   overridden by sub-classes only). In the process, some methods of IndexReader 
10579   were made final. This is not expected to affect many apps, since these methods
10580   already delegate to abstract methods, which you had to already override 
10581   anyway. (Shai Erera)
10583 * LUCENE-3636: Added SearcherFactory, used by SearcherManager and NRTManager
10584   to create new IndexSearchers. You can provide your own implementation to
10585   warm new searchers, set an ExecutorService, set a custom Similarity, or
10586   even return your own subclass of IndexSearcher. The SearcherWarmer and
10587   ExecutorService parameters on these classes were removed, as they are
10588   subsumed by SearcherFactory.  (Shai Erera, Mike McCandless, Robert Muir)
10590 * LUCENE-3644: The expert ReaderFinishedListener api suffered problems (propagated
10591   down to subreaders, but was not called on SegmentReaders, unless they were
10592   the owner of the reader core, and other ambiguities). The API is revised:
10593   You can set ReaderClosedListeners on any IndexReader, and onClose is called
10594   when that reader is closed.  SegmentReader has CoreClosedListeners that you
10595   can register to know when a shared reader core is closed.  
10596   (Uwe Schindler, Mike McCandless, Robert Muir)
10598 * LUCENE-3652: The package org.apache.lucene.messages was moved to
10599   contrib/queryparser. If you have used those classes in your code
10600   just add the lucene-queryparser.jar file to your classpath.
10601   (Uwe Schindler)
10603 * LUCENE-3681: FST now stores labels for BYTE2 input type as 2 bytes
10604   instead of vInt; this can make FSTs smaller and faster, but it is a
10605   break in the binary format so if you had built and saved any FSTs
10606   then you need to rebuild them. (Robert Muir, Mike McCandless)
10608 * LUCENE-3679: The expert IndexReader.getFieldNames(FieldOption) API
10609   has been removed and replaced with the experimental getFieldInfos
10610   API.  All IndexReader subclasses must implement getFieldInfos.
10611   (Mike McCandless)
10613 * LUCENE-3695: Move confusing add(X) methods out of FST.Builder into
10614   FST.Util.  (Robert Muir, Mike McCandless)
10616 * LUCENE-3701: Added an additional argument to the expert FST.Builder
10617   ctor to take FreezeTail, which you can use to (very-expertly) customize
10618   the FST construction process. Pass null if you want the default
10619   behavior.  Added seekExact() to FSTEnum, and added FST.save/read 
10620   from a File. (Mike McCandless, Dawid Weiss, Robert Muir)
10622 * LUCENE-3712: Removed unused and untested ReaderUtil#subReader methods.
10623   (Uwe Schindler)
10625 * LUCENE-3672: Deprecate Directory.fileModified,
10626   IndexCommit.getTimestamp and .getVersion and
10627   IndexReader.lastModified and getCurrentVersion (Andrzej Bialecki,
10628   Robert Muir, Mike McCandless)
10630 * LUCENE-3760: In IndexReader/DirectoryReader, deprecate static
10631   methods getCurrentVersion and getCommitUserData, and non-static
10632   method getCommitUserData (use getIndexCommit().getUserData()
10633   instead).  (Ryan McKinley, Robert Muir, Mike McCandless)
10635 * LUCENE-3867: Deprecate instance creation of RamUsageEstimator, instead
10636   the new static method sizeOf(Object) should be used. As the algorithm
10637   is now using Hotspot(TM) internals (reference size, header sizes,
10638   object alignment), the abstract o.a.l.util.MemoryModel class was
10639   completely removed (without replacement). The new static methods
10640   no longer support String intern-ness checking, interned strings
10641   now count to memory usage as any other Java object.
10642   (Dawid Weiss, Uwe Schindler, Shai Erera)
10644 * LUCENE-3738: All readXxx methods in BufferedIndexInput were made
10645   final. Subclasses should only override protected readInternal /
10646   seekInternal.  (Uwe Schindler)
10648 * LUCENE-2599: Deprecated the spatial contrib module, which was buggy and not
10649   well maintained.  Lucene 4 includes a new spatial module that replaces this.
10650   (David Smiley, Ryan McKinley, Chris Male)
10652 Changes in Runtime Behavior
10654 * LUCENE-3796, SOLR-3241: Throw an exception if you try to set an index-time
10655   boost on a field that omits norms. Because the index-time boost
10656   is multiplied into the norm, previously your boost would be 
10657   silently discarded.  (Tomás Fernández Löbbe, Hoss Man, Robert Muir)
10659 * LUCENE-3848: Fix tokenstreams to not produce a stream with an initial
10660   position increment of 0: which is out of bounds (overlapping with a
10661   non-existent previous term). Consumers such as IndexWriter and QueryParser
10662   still check for and silently correct this situation today, but at some point 
10663   in the future they may throw an exception.  (Mike McCandless, Robert Muir)
10665 * LUCENE-3738: DataInput/DataOutput no longer allow negative vLongs. Negative
10666   vInts are still supported (for index backwards compatibility), but
10667   should not be used in new code. The read method for negative vLongs
10668   was already broken since Lucene 3.1.
10669   (Uwe Schindler, Mike McCandless, Robert Muir)
10671 Security fixes
10673 * LUCENE-3588: Try harder to prevent SIGSEGV on cloned MMapIndexInputs:
10674   Previous versions of Lucene could SIGSEGV the JVM if you try to access
10675   the clone of an IndexInput retrieved from MMapDirectory. This security fix
10676   prevents this as best as it can by throwing AlreadyClosedException
10677   also on clones.  (Uwe Schindler, Robert Muir)
10679 API Changes
10681 * LUCENE-3606: IndexReader will be made read-only in Lucene 4.0, so all
10682   methods allowing to delete or undelete documents using IndexReader were
10683   deprecated; you should use IndexWriter now. Consequently
10684   IndexReader.commit() and all open(), openIfChanged(), clone() methods
10685   taking readOnly booleans (or IndexDeletionPolicy instances) were
10686   deprecated. IndexReader.setNorm() is superfluous and was deprecated.
10687   If you have to change per-document boost use CustomScoreQuery.
10688   If you want to dynamically change norms (boost *and* length norm) at
10689   query time, wrap your IndexReader using FilterIndexReader, overriding
10690   FilterIndexReader.norms(). To persist the changes on disk, copy the
10691   FilteredIndexReader to a new index using IndexWriter.addIndexes().
10692   In Lucene 4.0, SimilarityProvider will allow you to customize scoring
10693   using external norms, too.  (Uwe Schindler, Robert Muir)
10695 * LUCENE-3735: PayloadProcessorProvider was changed to return a
10696   ReaderPayloadProcessor instead of DirPayloadProcessor. The selection
10697   of the provider to return for the factory is now based on the IndexReader
10698   to be merged. To mimic the old behaviour, just use IndexReader.directory()
10699   for choosing the provider by Directory.  (Uwe Schindler)
10701 * LUCENE-3765: Deprecated StopFilter ctor that took ignoreCase, because
10702   in some cases (if the set is a CharArraySet), the argument is ignored.
10703   Deprecated StandardAnalyzer and ClassicAnalyzer ctors that take File,
10704   please use the Reader ctor instead.  (Robert Muir)
10706 * LUCENE-3766: Deprecate no-arg ctors of Tokenizer. Tokenizers are
10707   TokenStreams with Readers: tokenizers with null Readers will not be
10708   supported in Lucene 4.0, just use a TokenStream.  
10709   (Mike McCandless, Robert Muir)
10711 * LUCENE-3769: Simplified NRTManager by requiring applyDeletes to be
10712   passed to ctor only; if an app needs to mix and match it's free to
10713   create two NRTManagers (one always applying deletes and the other
10714   never applying deletes).  (MJB, Shai Erera, Mike McCandless)
10716 * LUCENE-3761: Generalize SearcherManager into an abstract ReferenceManager.
10717   SearcherManager remains a concrete class, but due to the refactoring, the 
10718   method maybeReopen has been deprecated in favor of maybeRefresh().
10719   (Shai Erera, Mike McCandless, Simon Willnauer)
10721 * LUCENE-3776: You now acquire/release the IndexSearcher directly from
10722   NRTManager.  (Mike McCandless)
10724 New Features
10726 * LUCENE-3593: Added a FieldValueFilter that accepts all documents that either
10727   have at least one or no value at all in a specific field. (Simon Willnauer,
10728   Uwe Schindler, Robert Muir)
10730 * LUCENE-3586: CheckIndex and IndexUpgrader allow you to specify the
10731   specific FSDirectory implementation to use (with the new -dir-impl
10732   command-line option).  (Luca Cavanna via Mike McCandless)
10734 * LUCENE-3634: IndexReader's static main method was moved to a new
10735   tool, CompoundFileExtractor, in contrib/misc.  (Robert Muir, Mike
10736   McCandless)
10738 * LUCENE-995: The QueryParser now interprets * as an open end for range
10739   queries.  Literal asterisks may be represented by quoting or escaping
10740   (i.e. \* or "*")  Custom QueryParser subclasses overriding getRangeQuery()
10741   will be passed null for any open endpoint. (Ingo Renner, Adriano
10742   Crestani, yonik, Mike McCandless 
10744 * LUCENE-3121: Add sugar reverse lookup (given an output, find the
10745   input mapping to it) for FSTs that have strictly monotonic long
10746   outputs (such as an ord).  (Mike McCandless)
10748 * LUCENE-3671: Add TypeTokenFilter that filters tokens based on
10749   their TypeAttribute.  (Tommaso Teofili via Uwe Schindler)
10751 * LUCENE-3690,LUCENE-3913: Added HTMLStripCharFilter, a CharFilter that strips
10752   HTML markup. (Steve Rowe)
10754 * LUCENE-3725: Added optional packing to FST building; this uses extra
10755   RAM during building but results in a smaller FST.  (Mike McCandless)
10757 * LUCENE-3714: Add top N shortest cost paths search for FST.
10758   (Robert Muir, Dawid Weiss, Mike McCandless)
10760 * LUCENE-3789: Expose MTQ TermsEnum via RewriteMethod for non package private
10761   access (Simon Willnauer)
10763 * LUCENE-3881: Added UAX29URLEmailAnalyzer: a standard analyzer that recognizes
10764   URLs and emails. (Steve Rowe)
10766 Bug fixes
10768 * LUCENE-3595: Fixed FieldCacheRangeFilter and FieldCacheTermsFilter
10769   to correctly respect deletions on reopened SegmentReaders. Factored out
10770   FieldCacheDocIdSet to be a top-level class.  (Uwe Schindler, Simon Willnauer)
10772 * LUCENE-3627: Don't let an errant 0-byte segments_N file corrupt the index.
10773   (Ken McCracken via Mike McCandless)
10775 * LUCENE-3630: The internal method MultiReader.doOpenIfChanged(boolean doClone)
10776   was overriding IndexReader.doOpenIfChanged(boolean readOnly), so changing the
10777   contract of the overridden method. This method was renamed and made private.
10778   In ParallelReader the bug was not existent, but the implementation method
10779   was also made private.  (Uwe Schindler)
10781 * LUCENE-3641: Fixed MultiReader to correctly propagate readerFinishedListeners
10782   to clones/reopened readers.  (Uwe Schindler)
10784 * LUCENE-3642, SOLR-2891, LUCENE-3717: Fixed bugs in CharTokenizer, n-gram tokenizers/filters, 
10785   compound token filters, thai word filter, icutokenizer, pattern analyzer, 
10786   wikipediatokenizer, and smart chinese where they would create invalid offsets in 
10787   some situations, leading to problems in highlighting.  
10788   (Max Beutel, Edwin Steiner via Robert Muir)
10790 * LUCENE-3639: TopDocs.merge was incorrectly setting TopDocs.maxScore to
10791   Float.MIN_VALUE when it should be Float.NaN, when there were 0
10792   hits.  Improved age calculation in SearcherLifetimeManager, to have
10793   double precision and to compute age to be how long ago the searcher
10794   was replaced with a new searcher (Mike McCandless)
10796 * LUCENE-3658: Corrected potential concurrency issues with
10797   NRTCachingDir, fixed createOutput to overwrite any previous file,
10798   and removed invalid asserts (Robert Muir, Mike McCandless)
10800 * LUCENE-3605: don't sleep in a retry loop when trying to locate the
10801   segments_N file (Robert Muir, Mike McCandless)
10803 * LUCENE-3711: SentinelIntSet with a small initial size can go into
10804   an infinite loop when expanded.  This can affect grouping using
10805   TermAllGroupsCollector or TermAllGroupHeadsCollector if instantiated with a
10806   non default small size. (Martijn van Groningen, yonik)
10808 * LUCENE-3727: When writing stored fields and term vectors, Lucene
10809   checks file sizes to detect a bug in some Sun JREs (LUCENE-1282),
10810   however, on some NFS filesystems File.length() could be stale,
10811   resulting in false errors like "fdx size mismatch while indexing".
10812   These checks now use getFilePointer instead to avoid this.
10813   (Jamir Shaikh, Mike McCandless, Robert Muir)
10815 * LUCENE-3816: Fixed problem in FilteredDocIdSet, if null was returned
10816   from the delegate DocIdSet.iterator(), which is allowed to return
10817   null by DocIdSet specification when no documents match.
10818   (Shay Banon via Uwe Schindler)
10820 * LUCENE-3821: SloppyPhraseScorer missed documents that ExactPhraseScorer finds
10821   When phrase query had repeating terms (e.g. "yes no yes")  
10822   sloppy query missed documents that exact query matched. 
10823   Fixed except when for repeating multiterms (e.g. "yes no yes|no").
10824   (Robert Muir, Doron Cohen)
10826 * LUCENE-3841: Fix CloseableThreadLocal to also purge stale entries on
10827   get(); this fixes certain cases where we were holding onto objects
10828   for dead threads for too long (Matthew Bellew, Mike McCandless)
10830 * LUCENE-3872: IndexWriter.close() now throws IllegalStateException if
10831   you call it after calling prepareCommit() without calling commit()
10832   first.  (Tim Bogaert via Mike McCandless)
10834 * LUCENE-3874: Throw IllegalArgumentException from IndexWriter (rather
10835   than producing a corrupt index), if a positionIncrement would cause
10836   integer overflow. This can happen, for example when using a buggy
10837   TokenStream that forgets to call clearAttributes() in combination
10838   with a StopFilter. (Robert Muir)
10840 * LUCENE-3876: Fix bug where positions for a document exceeding
10841   Integer.MAX_VALUE/2 would produce a corrupt index.  
10842   (Simon Willnauer, Mike McCandless, Robert Muir)
10844 * LUCENE-3880: UAX29URLEmailTokenizer now recognizes emails when the mailto:
10845   scheme is prepended. (Kai Gülzau, Steve Rowe)
10847 Optimizations
10849 * LUCENE-3653: Improve concurrency in VirtualMethod and AttributeSource by
10850   using a WeakIdentityMap based on a ConcurrentHashMap.  (Uwe Schindler,
10851   Gerrit Jansen van Vuuren)
10853 Documentation
10855 * LUCENE-3597: Fixed incorrect grouping documentation. (Martijn van Groningen,
10856   Robert Muir)
10858 * LUCENE-3926: Improve documentation of RAMDirectory, because this
10859   class is not intended to work with huge indexes. Everything beyond
10860   several hundred megabytes will waste resources (GC cycles), because
10861   it uses an internal buffer size of 1024 bytes, producing millions of 
10862   byte[1024] arrays. This class is optimized for small memory-resident
10863   indexes. It also has bad concurrency on multithreaded environments.
10864   It is recommended to materialize large indexes on disk and use
10865   MMapDirectory, which is a high-performance directory implementation
10866   working directly on the file system cache of the operating system,
10867   so copying data to Java heap space is not useful.  (Uwe Schindler,
10868   Mike McCandless, Robert Muir)
10870 Build
10872 * LUCENE-3857: exceptions from other threads in beforeclass/etc do not fail 
10873   the test (Dawid Weiss)
10875 * LUCENE-3847: LuceneTestCase will now check for modifications of System 
10876   properties before and after each test (and suite). If changes are detected,
10877   the test will fail. A rule can be used to reset system properties to
10878   before-scope state (and this has been used to make Solr tests pass).
10879   (Dawid Weiss, Uwe Schindler).
10881 * LUCENE-3228: Stop downloading external javadoc package-list files:
10883   - Added package-list files for Oracle Java javadocs and JUnit javadocs to
10884     Lucene/Solr subversion.
10886   - The Oracle Java javadocs package-list file is excluded from Lucene and
10887     Solr source release packages.
10889   - Regardless of network connectivity, javadocs built from a subversion
10890     checkout contain links to Oracle & JUnit javadocs.
10892   - Building javadocs from a source release package will download the Oracle
10893     Java package-list file if it isn't already present.
10895   - When the Oracle Java package-list file is not present and download fails,
10896     the javadocs targets will not fail the build, though an error will appear
10897     in the build log.  In this case, the built javadocs will not contain links
10898     to Oracle Java javadocs.
10900   - Links from Solr javadocs to Lucene's javadocs are enabled. When building
10901     a X.Y.Z-SNAPSHOT version, the links are to the most recently built nightly
10902     Jenkins javadocs. When building a release version, links are to the
10903     Lucene release javadocs for the same version.
10905   (Steve Rowe, hossman)
10907 * LUCENE-3753: Restructure the Lucene build system:
10908   - Created a new Lucene-internal module named "core" by moving the java/
10909     and test/ directories from lucene/src/ to lucene/core/src/.
10910   - Eliminated lucene/src/ by moving all its directories up one level.
10911   - Each internal module (core/, test-framework/, and tools/) now has its own
10912     build.xml, from which it is possible to run module-specific targets.
10913     lucene/build.xml delegates all build tasks (via
10914     <ant dir="internal-module-dir"> calls) to these modules' build.xml files.
10915   (Steve Rowe)
10917 * LUCENE-3774: Optimized and streamlined license and notice file validation
10918   by refactoring the build task into an ANT task and modifying build scripts
10919   to perform top-level checks. (Dawid Weiss, Steve Rowe, Robert Muir)
10921 * LUCENE-3762: Upgrade JUnit to 4.10, refactor state-machine of detecting 
10922   setUp/tearDown call chaining in LuceneTestCase. (Dawid Weiss, Robert Muir)
10924 * LUCENE-3944: Make the 'generate-maven-artifacts' target use filtered POMs
10925   placed under lucene/build/poms/, rather than in each module's base
10926   directory.  The 'clean' target now removes them.
10927   (Steve Rowe, Robert Muir)
10929 * LUCENE-3930: Changed build system to use Apache Ivy for retrival of 3rd
10930   party JAR files.  Please review BUILD.txt for instructions.
10931   (Robert Muir, Chris Male, Uwe Schindler, Steven Rowe, Hossman)
10934 ======================= Lucene 3.5.0 =======================
10936 Changes in backwards compatibility policy
10938 * LUCENE-3390: The first approach in Lucene 3.4.0 for missing values
10939   support for sorting had a design problem that made the missing value
10940   be populated directly into the FieldCache arrays during sorting,
10941   leading to concurrency issues. To fix this behaviour, the method
10942   signatures had to be changed:
10943   - FieldCache.getUnValuedDocs() was renamed to FieldCache.getDocsWithField()
10944     returning a Bits interface (backported from Lucene 4.0).
10945   - FieldComparator.setMissingValue() was removed and added to
10946     constructor
10947   As this is expert API, most code will not be affected.
10948   (Uwe Schindler, Doron Cohen, Mike McCandless)
10950 * LUCENE-3541: Remove IndexInput's protected copyBuf. If you want to
10951   keep a buffer in your IndexInput, do this yourself in your implementation, 
10952   and be sure to do the right thing on clone()!  (Robert Muir)
10954 * LUCENE-2822: TimeLimitingCollector now expects a counter clock instead of
10955   relying on a private daemon thread. The global time limiting clock thread
10956   has been exposed and is now lazily loaded and fully optional.
10957   TimeLimitingCollector now supports setting clock baseline manually to include
10958   prelude of a search. Previous versions set the baseline on construction time, 
10959   now baseline is set once the first IndexReader is passed to the collector
10960   unless set before. (Simon Willnauer)
10962 Changes in runtime behavior
10964 * LUCENE-3520: IndexReader.openIfChanged, when passed a near-real-time
10965   reader, will now return null if there are no changes.  The API has
10966   always reserved the right to do this; it's just that in the past for
10967   near-real-time readers it never did. (Mike McCandless)
10969 Bug fixes
10971 * LUCENE-3412: SloppyPhraseScorer was returning non-deterministic results 
10972   for queries with many repeats (Doron Cohen)
10974 * LUCENE-3421: PayloadTermQuery's explain was wrong when includeSpanScore=false.
10975   (Edward Drapkin via Robert Muir)
10977 * LUCENE-3432: IndexWriter.expungeDeletes with TieredMergePolicy
10978   should ignore the maxMergedSegmentMB setting (v.sevel via Mike
10979   McCandless)
10981 * LUCENE-3442: TermQuery.TermWeight.scorer() returns null for non-atomic
10982   IndexReaders (optimization bug, introcuced by LUCENE-2829), preventing
10983   QueryWrapperFilter and similar classes to get a top-level DocIdSet.
10984   (Dan C., Uwe Schindler)
10986 * LUCENE-3390: Corrected handling of missing values when two parallel searches
10987   using different missing values for sorting: the missing value was populated
10988   directly into the FieldCache arrays during sorting, leading to concurrency
10989   issues.  (Uwe Schindler, Doron Cohen, Mike McCandless)
10991 * LUCENE-3439: Closing an NRT reader after the writer was closed was
10992   incorrectly invoking the DeletionPolicy and (then possibly deleting
10993   files) on the closed IndexWriter (Robert Muir, Mike McCandless)
10995 * LUCENE-3215: SloppyPhraseScorer sometimes computed Infinite freq
10996   (Robert Muir, Doron Cohen)  
10998 * LUCENE-3503: DisjunctionSumScorer would give slightly different scores
10999   for a document depending if you used nextDoc() versus advance().
11000   (Mike McCandless, Robert Muir)
11002 * LUCENE-3529: Properly support indexing an empty field with empty term text.
11003   Previously, if you had assertions enabled you would receive an error during
11004   flush, if you didn't, you would get an invalid index.  
11005   (Mike McCandless, Robert Muir)
11007 * LUCENE-2633: PackedInts Packed32 and Packed64 did not support internal
11008   structures larger than 256MB (Toke Eskildsen via Mike McCandless)
11010 * LUCENE-3540: LUCENE-3255 dropped support for pre-1.9 indexes, but the
11011   error message in IndexFormatTooOldException was incorrect. (Uwe Schindler,
11012   Mike McCandless)
11014 * LUCENE-3541: IndexInput's default copyBytes() implementation was not safe 
11015   across multiple threads, because all clones shared the same buffer.
11016   (Robert Muir)
11018 * LUCENE-3548: Fix CharsRef#append to extend length of the existing char[]
11019   and preserve existing chars. (Simon Willnauer) 
11021 * LUCENE-3582: Normalize NaN values in NumericUtils.floatToSortableInt() /
11022   NumericUtils.doubleToSortableLong(), so this is consistent with stored
11023   fields. Also fix NumericRangeQuery to not falsely hit NaNs on half-open
11024   ranges (one bound is null). Because of normalization, NumericRangeQuery
11025   can now be used to hit NaN values by creating a query with
11026   upper == lower == NaN (inclusive).  (Dawid Weiss, Uwe Schindler)
11028 API Changes
11030 * LUCENE-3454: Rename IndexWriter.optimize to forceMerge to discourage
11031   use of this method since it is horribly costly and rarely justified
11032   anymore.  MergePolicy.findMergesForOptimize was renamed to
11033   findForcedMerges.  IndexReader.isOptimized was
11034   deprecated. IndexCommit.isOptimized was replaced with
11035   getSegmentCount. (Robert Muir, Mike McCandless)
11037 * LUCENE-3205: Deprecated MultiTermQuery.getTotalNumerOfTerms() [and
11038   related methods], as the numbers returned are not useful
11039   for multi-segment indexes. They were only needed for tests of
11040   NumericRangeQuery.  (Mike McCandless, Uwe Schindler)
11042 * LUCENE-3574: Deprecate outdated constants in org.apache.lucene.util.Constants
11043   and add new ones for Java 6 and Java 7.  (Uwe Schindler)
11045 * LUCENE-3571: Deprecate IndexSearcher(Directory). Use the constructors
11046   that take IndexReader instead.  (Robert Muir)
11048 * LUCENE-3577: Rename IndexWriter.expungeDeletes to forceMergeDeletes,
11049   and revamped the javadocs, to discourage
11050   use of this method since it is horribly costly and rarely
11051   justified.  MergePolicy.findMergesToExpungeDeletes was renamed to
11052   findForcedDeletesMerges. (Robert Muir, Mike McCandless)
11054 * LUCENE-3464: IndexReader.reopen has been renamed to
11055   IndexReader.openIfChanged (a static method), and now returns null
11056   (instead of the old reader) if there are no changes in the index, to
11057   prevent the common pitfall of accidentally closing the old reader.
11059 New Features
11061 * LUCENE-3448: Added FixedBitSet.and(other/DISI), andNot(other/DISI).
11062   (Uwe Schindler)
11064 * LUCENE-2215: Added IndexSearcher.searchAfter which returns results after a 
11065   specified ScoreDoc (e.g. last document on the previous page) to support deep
11066   paging use cases.  (Aaron McCurry, Grant Ingersoll, Robert Muir) 
11068 * LUCENE-1990: Adds internal packed ints implementation, to be used
11069   for more efficient storage of int arrays when the values are
11070   bounded, for example for storing the terms dict index (Toke
11071   Eskildsen via Mike McCandless)
11073 * LUCENE-3558: Moved SearcherManager, NRTManager & SearcherLifetimeManager into
11074   core. All classes are contained in o.a.l.search. (Simon Willnauer)
11076 Optimizations
11078 * LUCENE-3426: Add NGramPhraseQuery which extends PhraseQuery and tries to 
11079   reduce the number of terms of the query when rewrite(), in order to improve 
11080   performance.  (Robert Muir, Koji Sekiguchi)
11082 * LUCENE-3494: Optimize FilteredQuery to remove a multiply in score()
11083   (Uwe Schindler, Robert Muir)
11085 * LUCENE-3534: Remove filter logic from IndexSearcher and delegate to
11086   FilteredQuery's Scorer. This is a partial backport of a cleanup in
11087   FilteredQuery/IndexSearcher added by LUCENE-1536 to Lucene 4.0.
11088   (Uwe Schindler)
11090 * LUCENE-2205: Very substantial (3-5X) RAM reduction required to hold
11091   the terms index on opening an IndexReader (Aaron McCurry via Mike McCandless)
11093 * LUCENE-3443: FieldCache can now set docsWithField, and create an
11094   array, in a single pass.  This results in faster init time for apps
11095   that need both (such as sorting by a field with a missing value).
11096   (Mike McCandless)
11098 Test Cases
11100 * LUCENE-3420: Disable the finalness checks in TokenStream and Analyzer
11101   for implementing subclasses in different packages, where assertions are not
11102   enabled. (Uwe Schindler)
11104 * LUCENE-3506: tests relying on assertions being enabled were no-op because
11105   they ignored AssertionError. With this fix now entire test framework 
11106   (every test) fails if assertions are disabled, unless 
11107   -Dtests.asserts.gracious=true is specified. (Doron Cohen)
11109 Build
11111 * SOLR-2849: Fix dependencies in Maven POMs. (David Smiley via Steve Rowe)
11113 * LUCENE-3561: Fix maven xxx-src.jar files that were missing resources.
11114   (Uwe Schindler)
11116 ======================= Lucene 3.4.0 =======================
11118 Bug fixes
11120 * LUCENE-3251: Directory#copy failed to close target output if opening the
11121   source stream failed. (Simon Willnauer)
11123 * LUCENE-3255: If segments_N file is all zeros (due to file
11124   corruption), don't read that to mean the index is empty.  (Gregory
11125   Tarr, Mark Harwood, Simon Willnauer, Mike McCandless)
11127 * LUCENE-3254: Fixed minor bug in deletes were written to disk,
11128   causing the file to sometimes be larger than it needed to be.  (Mike
11129   McCandless)
11131 * LUCENE-3224: Fixed a big where CheckIndex would incorrectly report a
11132   corrupt index if a term with docfreq >= 16 was indexed more than once
11133   at the same position.  (Robert Muir)
11135 * LUCENE-3339: Fixed deadlock case when multiple threads use the new
11136   block-add (IndexWriter.add/updateDocuments) methods.  (Robert Muir,
11137   Mike McCandless)
11139 * LUCENE-3340: Fixed case where IndexWriter was not flushing at
11140   exactly maxBufferedDeleteTerms (Mike McCandless)
11142 * LUCENE-3358, LUCENE-3361: StandardTokenizer and UAX29URLEmailTokenizer 
11143   wrongly discarded combining marks attached to Han or Hiragana characters, 
11144   this is fixed if you supply Version >= 3.4 If you supply a previous 
11145   lucene version, you get the old buggy behavior for backwards compatibility.  
11146   (Trejkaz, Robert Muir)
11148 * LUCENE-3368: IndexWriter commits segments without applying their buffered
11149   deletes when flushing concurrently. (Simon Willnauer, Mike McCandless)
11151 * LUCENE-3365: Create or Append mode determined before obtaining write lock
11152   can cause IndexWriter overriding an existing index.
11153   (Geoff Cooney via Simon Willnauer)
11155 * LUCENE-3380: Fixed a bug where FileSwitchDirectory's listAll() would wrongly
11156   throw NoSuchDirectoryException when all files written so far have been
11157   written to one directory, but the other still has not yet been created on the
11158   filesystem.  (Robert Muir)
11160 * LUCENE-3409: IndexWriter.deleteAll was failing to close pooled NRT
11161   SegmentReaders, leading to unused files accumulating in the
11162   Directory.  (tal steier via Mike McCandless)
11164 * LUCENE-3418: Lucene was failing to fsync index files on commit,
11165   meaning an operating system or hardware crash, or power loss, could
11166   easily corrupt the index.  (Mark Miller, Robert Muir, Mike
11167   McCandless)
11169 New Features
11171 * LUCENE-3290: Added FieldInvertState.numUniqueTerms 
11172   (Mike McCandless, Robert Muir)
11174 * LUCENE-3280: Add FixedBitSet, like OpenBitSet but is not elastic
11175   (grow on demand if you set/get/clear too-large indices).  (Mike
11176   McCandless)
11178 * LUCENE-2048: Added the ability to omit positions but still index
11179   term frequencies, you can now control what is indexed into
11180   the postings via AbstractField.setIndexOptions:
11181    DOCS_ONLY: only documents are indexed: term frequencies and positions are omitted
11182    DOCS_AND_FREQS: only documents and term frequencies are indexed: positions are omitted
11183    DOCS_AND_FREQS_AND_POSITIONS: full postings: documents, frequencies, and positions
11184   AbstractField.setOmitTermFrequenciesAndPositions is deprecated,
11185   you should use DOCS_ONLY instead.  (Robert Muir)
11187 * LUCENE-3097: Added a new grouping collector that can be used to retrieve all most relevant
11188   documents per group. This can be useful in situations when one wants to compute grouping
11189   based facets / statistics on the complete query result. (Martijn van Groningen)
11191 * LUCENE-3334: If Java7 is detected, IOUtils.closeSafely() will log
11192   suppressed exceptions in the original exception, so stack trace
11193   will contain them.  (Uwe Schindler)
11195 Optimizations
11197 * LUCENE-3201, LUCENE-3218: CompoundFileSystem code has been consolidated 
11198   into a Directory implementation. Reading is optimized for MMapDirectory,
11199   NIOFSDirectory and SimpleFSDirectory to only map requested parts of the
11200   CFS into an IndexInput. Writing to a CFS now tries to append to the CF
11201   directly if possible and merges separately written files on the fly instead
11202   of during close. (Simon Willnauer, Robert Muir)
11204 * LUCENE-3289: When building an FST you can now tune how aggressively
11205   the FST should try to share common suffixes.  Typically you can
11206   greatly reduce RAM required during building, and CPU consumed, at
11207   the cost of a somewhat larger FST.  (Mike McCandless)
11209 Test Cases
11211 * LUCENE-3327: Fix AIOOBE when TestFSTs is run with -Dtests.verbose=true
11212  (James Dyer via Mike McCandless)
11214 Build
11216 * LUCENE-3406: Add ant target 'package-local-src-tgz' to Lucene and Solr
11217   to package sources from the local working copy.
11218   (Seung-Yeoul Yang via Steve Rowe)
11221 ======================= Lucene 3.3.0 =======================
11223 Changes in backwards compatibility policy
11225 * LUCENE-3140: IndexOutput.copyBytes now takes a DataInput (superclass
11226   of IndexInput) as its first argument.  (Robert Muir, Dawid Weiss,
11227   Mike McCandless)
11229 * LUCENE-3191: FieldComparator.value now returns an Object not
11230   Comparable; FieldDoc.fields also changed from Comparable[] to
11231   Object[] (Uwe Schindler, Mike McCandless)
11233 * LUCENE-3208: Made deprecated methods Query.weight(Searcher) and
11234   Searcher.createWeight() final to prevent override. If you have
11235   overridden one of these methods, cut over to the non-deprecated
11236   implementation. (Uwe Schindler, Robert Muir, Yonik Seeley)
11238 * LUCENE-3238: Made MultiTermQuery.rewrite() final, to prevent
11239   problems (such as not properly setting rewrite methods, or 
11240   not working correctly with things like SpanMultiTermQueryWrapper).
11241   To rewrite to a simpler form, instead return a simpler enum
11242   from getEnum(IndexReader). For example, to rewrite to a single term, 
11243   return a SingleTermEnum.  (ludovic Boutros, Uwe Schindler, Robert Muir)
11245 Changes in runtime behavior
11247 * LUCENE-2834: the hash used to compute the lock file name when the
11248   lock file is not stored in the index has changed.  This means you
11249   will see a different lucene-XXX-write.lock in your lock directory.
11250   (Robert Muir, Uwe Schindler, Mike McCandless)
11252 * LUCENE-3146: IndexReader.setNorm throws IllegalStateException if the field
11253   does not store norms. (Shai Erera, Mike McCandless)
11255 * LUCENE-3198: On Linux, if the JRE is 64 bit and supports unmapping,
11256   FSDirectory.open now defaults to MMapDirectory instead of
11257   NIOFSDirectory since MMapDirectory gives better performance.  (Mike
11258   McCandless)
11260 * LUCENE-3200: MMapDirectory now uses chunk sizes that are powers of 2.
11261   When setting the chunk size, it is rounded down to the next possible
11262   value. The new default value for 64 bit platforms is 2^30 (1 GiB),
11263   for 32 bit platforms it stays unchanged at 2^28 (256 MiB).
11264   Internally, MMapDirectory now only uses one dedicated final IndexInput
11265   implementation supporting multiple chunks, which makes Hotspot's life
11266   easier.  (Uwe Schindler, Robert Muir, Mike McCandless)
11268 Bug fixes
11270 * LUCENE-3147,LUCENE-3152: Fixed open file handles leaks in many places in the 
11271   code. Now MockDirectoryWrapper (in test-framework) tracks all open files, 
11272   including locks, and fails if the test fails to release all of them.
11273   (Mike McCandless, Robert Muir, Shai Erera, Simon Willnauer)
11275 * LUCENE-3102: CachingCollector.replay was failing to call setScorer
11276   per-segment (Martijn van Groningen via Mike McCandless)
11278 * LUCENE-3183: Fix rare corner case where seeking to empty term
11279   (field="", term="") with terms index interval 1 could hit
11280   ArrayIndexOutOfBoundsException (selckin, Robert Muir, Mike
11281   McCandless)
11283 * LUCENE-3208: IndexSearcher had its own private similarity field
11284   and corresponding get/setter overriding Searcher's implementation. If you
11285   setted a different Similarity instance on IndexSearcher, methods implemented
11286   in the superclass Searcher were not using it, leading to strange bugs.
11287   (Uwe Schindler, Robert Muir)
11289 * LUCENE-3197: Fix core merge policies to not over-merge during
11290   background optimize when documents are still being deleted
11291   concurrently with the optimize (Mike McCandless)
11293 * LUCENE-3222: The RAM accounting for buffered delete terms was
11294   failing to measure the space required to hold the term's field and
11295   text character data.  (Mike McCandless)
11297 * LUCENE-3238: Fixed bug where using WildcardQuery("prefix*") inside
11298   of a SpanMultiTermQueryWrapper rewrote incorrectly and returned
11299   an error instead.  (ludovic Boutros, Uwe Schindler, Robert Muir)
11301 API Changes
11303 * LUCENE-3208: Renamed protected IndexSearcher.createWeight() to expert
11304   public method IndexSearcher.createNormalizedWeight() as this better describes
11305   what this method does. The old method is still there for backwards
11306   compatibility. Query.weight() was deprecated and simply delegates to
11307   IndexSearcher. Both deprecated methods will be removed in Lucene 4.0.
11308   (Uwe Schindler, Robert Muir, Yonik Seeley)
11310 * LUCENE-3197: MergePolicy.findMergesForOptimize now takes
11311   Map<SegmentInfo,Boolean> instead of Set<SegmentInfo> as the second
11312   argument, so the merge policy knows which segments were originally
11313   present vs produced by an optimizing merge (Mike McCandless)
11315 Optimizations
11317 * LUCENE-1736: DateTools.java general improvements. 
11318   (David Smiley via Steve Rowe)
11320 New Features
11322 * LUCENE-3140: Added experimental FST implementation to Lucene.
11323   (Robert Muir, Dawid Weiss, Mike McCandless)
11325 * LUCENE-3193: A new TwoPhaseCommitTool allows running a 2-phase commit 
11326   algorithm over objects that implement the new TwoPhaseCommit interface (such
11327   as IndexWriter). (Shai Erera)
11329 * LUCENE-3191: Added TopDocs.merge, to facilitate merging results from
11330   different shards (Uwe Schindler, Mike McCandless)
11332 * LUCENE-3179: Added OpenBitSet.prevSetBit (Paul Elschot via Mike McCandless)
11334 * LUCENE-3210: Made TieredMergePolicy more aggressive in reclaiming
11335   segments with deletions; added new methods
11336   set/getReclaimDeletesWeight to control this.  (Mike McCandless)
11338 Build
11340 * LUCENE-1344: Create OSGi bundle using dev-tools/maven.
11341   (Nicolas Lalevée, Luca Stancapiano via ryan)
11343 * LUCENE-3204: The maven-ant-tasks jar is now included in the source tree;
11344   users of the generate-maven-artifacts target no longer have to manually
11345   place this jar in the Ant classpath.  NOTE: when Ant looks for the 
11346   maven-ant-tasks jar, it looks first in its pre-existing classpath, so
11347   any copies it finds will be used instead of the copy included in the 
11348   Lucene/Solr source tree.  For this reason, it is recommeded to remove
11349   any copies of the maven-ant-tasks jar in the Ant classpath, e.g. under
11350   ~/.ant/lib/ or under the Ant installation's lib/ directory. (Steve Rowe) 
11353 ======================= Lucene 3.2.0 =======================
11355 Changes in backwards compatibility policy
11357 * LUCENE-2953: PriorityQueue's internal heap was made private, as subclassing
11358   with generics can lead to ClassCastException. For advanced use (e.g. in Solr)
11359   a method getHeapArray() was added to retrieve the internal heap array as a
11360   non-generic Object[].  (Uwe Schindler, Yonik Seeley)
11362 * LUCENE-1076: IndexWriter.setInfoStream now throws IOException
11363   (Mike McCandless, Shai Erera)
11365 * LUCENE-3084: MergePolicy.OneMerge.segments was changed from
11366   SegmentInfos to a List<SegmentInfo>. SegmentInfos itself was changed
11367   to no longer extend Vector<SegmentInfo> (to update code that is using
11368   Vector-API, use the new asList() and asSet() methods returning unmodifiable
11369   collections; modifying SegmentInfos is now only possible through
11370   the explicitely declared methods). IndexWriter.segString() now takes
11371   Iterable<SegmentInfo> instead of List<SegmentInfo>. A simple recompile
11372   should fix this. MergePolicy and SegmentInfos are internal/experimental
11373   APIs not covered by the strict backwards compatibility policy.
11374   (Uwe Schindler, Mike McCandless)
11376 Changes in runtime behavior
11378 * LUCENE-3065: When a NumericField is retrieved from a Document loaded
11379   from IndexReader (or IndexSearcher), it will now come back as
11380   NumericField not as a Field with a string-ified version of the
11381   numeric value you had indexed.  Note that this only applies for
11382   newly-indexed Documents; older indices will still return Field
11383   with the string-ified numeric value. If you call Document.get(),
11384   the value comes still back as String, but Document.getFieldable()
11385   returns NumericField instances. (Uwe Schindler, Ryan McKinley,
11386   Mike McCandless)
11388 * LUCENE-1076: Changed the default merge policy from
11389   LogByteSizeMergePolicy to TieredMergePolicy, as of Version.LUCENE_32
11390   (passed to IndexWriterConfig), which is able to merge non-contiguous
11391   segments. This means docIDs no longer necessarily stay "in order"
11392   during indexing.  If this is a problem then you can use either of
11393   the LogMergePolicy impls.  (Mike McCandless)
11395 New features
11397 * LUCENE-3082: Added index upgrade tool oal.index.IndexUpgrader
11398   that allows to upgrade all segments to last recent supported index
11399   format without fully optimizing.  (Uwe Schindler, Mike McCandless)
11401 * LUCENE-1076: Added TieredMergePolicy which is able to merge non-contiguous 
11402   segments, which means docIDs no longer necessarily stay "in order".
11403   (Mike McCandless, Shai Erera)
11405 * LUCENE-3071: Adding ReversePathHierarchyTokenizer, added skip parameter to 
11406   PathHierarchyTokenizer (Olivier Favre via ryan)
11408 * LUCENE-1421, LUCENE-3102: added CachingCollector which allow you to cache 
11409   document IDs and scores encountered during the search, and "replay" them to 
11410   another Collector. (Mike McCandless, Shai Erera)
11412 * LUCENE-3112: Added experimental IndexWriter.add/updateDocuments,
11413   enabling a block of documents to be indexed, atomically, with
11414   guaranteed sequential docIDs.  (Mike McCandless)
11416 API Changes
11418 * LUCENE-3061: IndexWriter's getNextMerge() and merge(OneMerge) are now public
11419   (though @lucene.experimental), allowing for custom MergeScheduler 
11420   implementations. (Shai Erera)
11422 * LUCENE-3065: Document.getField() was deprecated, as it throws
11423   ClassCastException when loading lazy fields or NumericFields.
11424   (Uwe Schindler, Ryan McKinley, Mike McCandless)
11426 * LUCENE-2027: Directory.touchFile is deprecated and will be removed
11427   in 4.0.  (Mike McCandless)
11429 Optimizations
11431 * LUCENE-2990: ArrayUtil/CollectionUtil.*Sort() methods now exit early
11432   on empty or one-element lists/arrays.  (Uwe Schindler)
11434 * LUCENE-2897: Apply deleted terms while flushing a segment.  We still
11435   buffer deleted terms to later apply to past segments.  (Mike McCandless)
11437 * LUCENE-3126: IndexWriter.addIndexes copies incoming segments into CFS if they
11438   aren't already and MergePolicy allows that. (Shai Erera)
11440 Bug fixes
11442 * LUCENE-2996: addIndexes(IndexReader) did not flush before adding the new 
11443   indexes, causing existing deletions to be applied on the incoming indexes as 
11444   well. (Shai Erera, Mike McCandless)
11446 * LUCENE-3024: Index with more than 2.1B terms was hitting AIOOBE when
11447   seeking TermEnum (eg used by Solr's faceting) (Tom Burton-West, Mike
11448   McCandless)
11450 * LUCENE-3042: When a filter or consumer added Attributes to a TokenStream
11451   chain after it was already (partly) consumed [or clearAttributes(),
11452   captureState(), cloneAttributes(),... was called by the Tokenizer],
11453   the Tokenizer calling clearAttributes() or capturing state after addition 
11454   may not do this on the newly added Attribute. This bug affected only
11455   very special use cases of the TokenStream-API, most users would not
11456   have recognized it.  (Uwe Schindler, Robert Muir)
11458 * LUCENE-3054: PhraseQuery can in some cases stack overflow in
11459   SorterTemplate.quickSort(). This fix also adds an optimization to
11460   PhraseQuery as term with lower doc freq will also have less positions.
11461   (Uwe Schindler, Robert Muir, Otis Gospodnetic)
11463 * LUCENE-3068: sloppy phrase query failed to match valid documents when multiple 
11464   query terms had same position in the query. (Doron Cohen)
11466 * LUCENE-3012: Lucene writes the header now for separate norm files (*.sNNN)
11467   (Robert Muir)
11469 Build
11471 * LUCENE-3006: Building javadocs will fail on warnings by default. 
11472   Override with -Dfailonjavadocwarning=false (sarowe, gsingers)
11474 * LUCENE-3128: "ant eclipse" creates a .project file for easier Eclipse
11475   integration (unless one already exists). (Daniel Serodio via Shai Erera)
11477 Test Cases
11479 * LUCENE-3002: added 'tests.iter.min' to control 'tests.iter' by allowing to 
11480   stop iterating if at least 'tests.iter.min' ran and a failure occured. 
11481   (Shai Erera, Chris Hostetter)
11483 ======================= Lucene 3.1.0 =======================
11485 Changes in backwards compatibility policy
11487 * LUCENE-2719: Changed API of internal utility class
11488   org.apache.lucene.util.SorterTemplate to support faster quickSort using
11489   pivot values and also merge sort and insertion sort. If you have used
11490   this class, you have to implement two more methods for handling pivots.
11491   (Uwe Schindler, Robert Muir, Mike McCandless)
11493 * LUCENE-1923: Renamed SegmentInfo & SegmentInfos segString method to
11494   toString.  These are advanced APIs and subject to change suddenly.
11495   (Tim Smith via Mike McCandless)
11497 * LUCENE-2190: Removed deprecated customScore() and customExplain()
11498   methods from experimental CustomScoreQuery.  (Uwe Schindler)
11500 * LUCENE-2286: Enabled DefaultSimilarity.setDiscountOverlaps by default.
11501   This means that terms with a position increment gap of zero do not
11502   affect the norms calculation by default.  (Robert Muir)
11504 * LUCENE-2320: MergePolicy.writer is now of type SetOnce, which allows setting
11505   the IndexWriter for a MergePolicy exactly once. You can change references to
11506   'writer' from <code>writer.doXYZ()</code> to <code>writer.get().doXYZ()</code>
11507   (it is also advisable to add an <code>assert writer != null;</code> before you
11508   access the wrapped IndexWriter.)
11510   In addition, MergePolicy only exposes a default constructor, and the one that
11511   took IndexWriter as argument has been removed from all MergePolicy extensions.
11512   (Shai Erera via Mike McCandless)
11514 * LUCENE-2328: SimpleFSDirectory.SimpleFSIndexInput is moved to
11515   FSDirectory.FSIndexInput. Anyone extending this class will have to
11516   fix their code on upgrading. (Earwin Burrfoot via Mike McCandless)
11518 * LUCENE-2302: The new interface for term attributes, CharTermAttribute,
11519   now implements CharSequence. This requires the toString() methods of
11520   CharTermAttribute, deprecated TermAttribute, and Token to return only
11521   the term text and no other attribute contents. LUCENE-2374 implements
11522   an attribute reflection API to no longer rely on toString() for attribute
11523   inspection. (Uwe Schindler, Robert Muir)
11525 * LUCENE-2372, LUCENE-2389: StandardAnalyzer, KeywordAnalyzer,
11526   PerFieldAnalyzerWrapper, WhitespaceTokenizer are now final.  Also removed
11527   the now obsolete and deprecated Analyzer.setOverridesTokenStreamMethod().
11528   Analyzer and TokenStream base classes now have an assertion in their ctor,
11529   that check subclasses to be final or at least have final implementations
11530   of incrementToken(), tokenStream(), and reusableTokenStream().
11531   (Uwe Schindler, Robert Muir)
11533 * LUCENE-2316: Directory.fileLength contract was clarified - it returns the
11534   actual file's length if the file exists, and throws FileNotFoundException
11535   otherwise. Returning length=0 for a non-existent file is no longer allowed. If
11536   you relied on that, make sure to catch the exception. (Shai Erera)
11538 * LUCENE-2386: IndexWriter no longer performs an empty commit upon new index
11539   creation. Previously, if you passed an empty Directory and set OpenMode to
11540   CREATE*, IndexWriter would make a first empty commit. If you need that
11541   behavior you can call writer.