"Fossies" - the Fresh Open Source Software Archive

Member "chandler-1.0.3/external/PyLucene/PyLucene-2.3.1-3-418/lucene-java-2.3.1/contrib/highlighter/src/java/org/apache/lucene/search/highlight/package.html" (2 Jan 2008, 3864 Bytes) of package /windows/misc/Chandler_src_1.0.3.tar.gz:

Caution: In this restricted "Fossies" environment the current HTML page may not be correctly presentated and may have some non-functional links. Alternatively you can here view or download the uninterpreted raw source code. A member file download can also be achieved by clicking within a package contents listing on the according byte size field.

The highlight package contains classes to provide "keyword in context" features typically used to highlight search terms in the text of results pages. The Highlighter class is the central component and can be used to extract the most interesting sections of a piece of text and highlight them, with the help of Fragmenter, FragmentScorer, Formatter classes.

Example Usage

  //... Above, create documents with two fields, one with term vectors (tv) and one without (notv)
  IndexSearcher searcher = new IndexSearcher(directory);
  QueryParser parser = new QueryParser("notv", analyzer);
  Query query = parser.parse("million");
  //query = query.rewrite(reader); //required to expand search terms
  Hits hits = searcher.search(query);

  SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter();
  Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query));
  for (int i = 0; i < 10; i++) {
    String text = hits.doc(i).get("notv");
    TokenStream tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), hits.id(i), "notv", analyzer);
    TextFragment[] frag = highlighter.getBestTextFragments(tokenStream, text, false, 10);//highlighter.getBestFragments(tokenStream, text, 3, "...");
    for (int j = 0; j < frag.length; j++) {
      if ((frag[j] != null) && (frag[j].getScore() > 0)) {
    //Term vector
    text = hits.doc(i).get("tv");
    tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), hits.id(i), "tv", analyzer);
    frag = highlighter.getBestTextFragments(tokenStream, text, false, 10);
    for (int j = 0; j < frag.length; j++) {
      if ((frag[j] != null) && (frag[j].getScore() > 0)) {

New features 06/02/2005

This release adds options for encoding (thanks to Nicko Cadell). An "Encoder" implementation such as the new SimpleHTMLEncoder class can be passed to the highlighter to encode all those non-xhtml standard characters such as & into legal values. This simple class may not suffice for some languages - Commons Lang has an implementation that could be used: escapeHtml(String) in http://svn.apache.org/viewcvs.cgi/jakarta/commons/proper/lang/trunk/src/java/org/apache/commons/lang/StringEscapeUtils.java?rev=137958&view=markup

New features 22/12/2004

This release adds some new capabilities:
  1. Faster highlighting using Term vector support
  2. New formatting options to use color intensity to show informational value
  3. Options for better summarization by using term IDF scores to influence fragment selection

The highlighter takes a TokenStream as input. Until now these streams have typically been produced using an Analyzer but the new class TokenSources provides helper methods for obtaining TokenStreams from the new TermVector position support (see latest CVS version).

The new class GradientFormatter can use a scale of colors to highlight terms according to their score. A subtle use of color can help emphasise the reasons for matching (useful when doing "MoreLikeThis" queries and you want to see what the basis of the similarities are).

The QueryScorer class has a new constructor which can use an IndexReader to derive the IDF (inverse document frequency) for each term in order to influcence the score. This is useful for helping to extracting the most significant sections of a document and in supplying scores used by the new GradientFormatter to color significant words more strongly. The QueryScorer.getMaxWeight method is useful when passed to the GradientFormatter constructor to define the top score which is associated with the top color.