<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.16 2008/07/22 22:05:24 tgl Exp $ -->

<chapter id="GIN">
<title>GIN Indexes</title>

   <indexterm>
    <primary>index</primary>
    <secondary>GIN</secondary>
   </indexterm>

<sect1 id="gin-intro">
 <title>Introduction</title>

 <para>
   <acronym>GIN</acronym> stands for Generalized Inverted Index.  It is
   an index structure storing a set of (key, posting list) pairs, where
   a <quote>posting list</> is a set of rows in which the key occurs. Each
   indexed value can contain many keys, so the same row ID can appear in
   multiple posting lists.
 </para>

 <para>
   It is generalized in the sense that a <acronym>GIN</acronym> index
   does not need to be aware of the operation that it accelerates.
   Instead, it uses custom strategies defined for particular data types.
 </para>

 <para>
  One advantage of <acronym>GIN</acronym> is that it allows the development
  of custom data types with the appropriate access methods, by
  an expert in the domain of the data type, rather than a database expert.
  This is much the same advantage as using <acronym>GiST</acronym>.
 </para>

 <para>
  The <acronym>GIN</acronym>
  implementation in <productname>PostgreSQL</productname> is primarily
  maintained by Teodor Sigaev and Oleg Bartunov. There is more
  information about <acronym>GIN</acronym> on their
  <ulink url="http://www.sai.msu.su/~megera/wiki/Gin">website</ulink>.
 </para>
</sect1>

<sect1 id="gin-extensibility">
 <title>Extensibility</title>

 <para>
   The <acronym>GIN</acronym> interface has a high level of abstraction,
   requiring the access method implementer only to implement the semantics of
   the data type being accessed.  The <acronym>GIN</acronym> layer itself
   takes care of concurrency, logging and searching the tree structure.
 </para>

 <para>
   All it takes to get a <acronym>GIN</acronym> access method working is to
   implement four (or five) user-defined methods, which define the behavior of
   keys in the tree and the relationships between keys, indexed values,
   and indexable queries. In short, <acronym>GIN</acronym> combines
   extensibility with generality, code reuse, and a clean interface.
 </para>

 <para>
   The four methods that an operator class for
   <acronym>GIN</acronym> must provide are:
 </para>

 <variablelist>
    <varlistentry>
     <term>int compare(Datum a, Datum b)</term>
     <listitem>
      <para>
       Compares keys (not indexed values!) and returns an integer less than
       zero, zero, or greater than zero, indicating whether the first key is
       less than, equal to, or greater than the second.
      </para>
     </listitem>
    </varlistentry>

    <varlistentry>
     <term>Datum *extractValue(Datum inputValue, int32 *nkeys)</term>
     <listitem>
      <para>
       Returns an array of keys given a value to be indexed.  The
       number of returned keys must be stored into <literal>*nkeys</>.
      </para>
     </listitem>
    </varlistentry>

    <varlistentry>
     <term>Datum *extractQuery(Datum query, int32 *nkeys,
        StrategyNumber n, bool **pmatch)</term>
     <listitem>
      <para>
       Returns an array of keys given a value to be queried; that is,
       <literal>query</> is the value on the right-hand side of an
       indexable operator whose left-hand side is the indexed column.
       <literal>n</> is the strategy number of the operator within the
       operator class (see <xref linkend="xindex-strategies">).
       Often, <function>extractQuery</> will need
       to consult <literal>n</> to determine the data type of
       <literal>query</> and the key values that need to be extracted.
       The number of returned keys must be stored into <literal>*nkeys</>.
       If the query contains no keys then <function>extractQuery</> 
       should store 0 or -1 into <literal>*nkeys</>, depending on the
       semantics of the operator.  0 means that every
       value matches the <literal>query</> and a sequential scan should be 
       produced.  -1 means nothing can match the <literal>query</>. 
       <literal>pmatch</> is an output argument for use when partial match
       is supported.  To use it, <function>extractQuery</> must allocate
       an array of <literal>*nkeys</> booleans and store its address at
       <literal>*pmatch</>.  Each element of the array should be set to TRUE
       if the corresponding key requires partial match, FALSE if not.
       If <literal>*pmatch</> is set to NULL then GIN assumes partial match
       is not required.  The variable is initialized to NULL before call,
       so this argument can simply be ignored by operator classes that do
       not support partial match.
      </para>

     </listitem>
    </varlistentry>

    <varlistentry>
     <term>bool consistent(bool check[], StrategyNumber n, Datum query, bool *recheck)</term>
     <listitem>
      <para>
       Returns TRUE if the indexed value satisfies the query operator with
       strategy number <literal>n</> (or might satisfy, if the recheck
       indication is returned).  The <literal>check</> array has
       the same length as the number of keys previously returned by
       <function>extractQuery</> for this query.  Each element of the
       <literal>check</> array is TRUE if the indexed value contains the
       corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
       <function>extractQuery</> result array is present in the indexed value.
       The original <literal>query</> datum (not the extracted key array!) is
       passed in case the <function>consistent</> method needs to consult it.
       On success, <literal>*recheck</> should be set to TRUE if the heap
       tuple needs to be rechecked against the query operator, or FALSE if
       the index test is exact.
      </para>
     </listitem>
    </varlistentry>

  </variablelist>

 <para>
  Optionally, an operator class for
  <acronym>GIN</acronym> can supply a fifth method:
 </para>

  <variablelist>

    <varlistentry>
     <term>int comparePartial(Datum partial_key, Datum key, StrategyNumber n)</term>
     <listitem>
      <para>
       Compare a partial-match query to an index key.  Returns an integer
       whose sign indicates the result: less than zero means the index key
       does not match the query, but the index scan should continue; zero
       means that the index key does match the query; greater than zero
       indicates that the index scan should stop because no more matches
       are possible.  The strategy number <literal>n</> of the operator
       that generated the partial match query is provided, in case its
       semantics are needed to determine when to end the scan.
      </para>
     </listitem>
    </varlistentry>

  </variablelist>

 <para>
  To support <quote>partial match</> queries, an operator class must
  provide the <function>comparePartial</> method, and its
  <function>extractQuery</> method must set the <literal>pmatch</>
  parameter when a partial-match query is encountered.  See
  <xref linkend="gin-partial-match"> for details.
 </para>

</sect1>

<sect1 id="gin-implementation">
 <title>Implementation</title>

 <para>
  Internally, a <acronym>GIN</acronym> index contains a B-tree index
  constructed over keys, where each key is an element of the indexed value
  (a member of an array, for example) and where each tuple in a leaf page is
  either a pointer to a B-tree over heap pointers (PT, posting tree), or a
  list of heap pointers (PL, posting list) if the list is small enough.
 </para>

 <sect2 id="gin-partial-match">
  <title>Partial match algorithm</title>
  
  <para>
   GIN can support <quote>partial match</> queries, in which the query
   does not determine an exact match for one or more keys, but the possible
   matches fall within a reasonably narrow range of key values (within the
   key sorting order determined by the <function>compare</> support method).
   The <function>extractQuery</> method, instead of returning a key value
   to be matched exactly, returns a key value that is the lower bound of
   the range to be searched, and sets the <literal>pmatch</> flag true.
   The key range is then searched using the <function>comparePartial</>
   method.  <function>comparePartial</> must return zero for an actual
   match, less than zero for a non-match that is still within the range
   to be searched, or greater than zero if the index key is past the range
   that could match.
  </para>

  <para>
   During a partial-match scan, all <literal>itemPointer</>s for matching keys
   are OR'ed into a <literal>TIDBitmap</>.
   The scan fails if the <literal>TIDBitmap</> becomes lossy.
   In this case an error message will be reported with advice
   to increase <literal>work_mem</>.
  </para>
 </sect2>

</sect1>

<sect1 id="gin-tips">
<title>GIN tips and tricks</title>

 <variablelist>
  <varlistentry>
   <term>Create vs insert</term>
   <listitem>
    <para>
     In most cases, insertion into a <acronym>GIN</acronym> index is slow
     due to the likelihood of many keys being inserted for each value.
     So, for bulk insertions into a table it is advisable to drop the GIN
     index and recreate it after finishing bulk insertion.
    </para>
   </listitem>
  </varlistentry>

  <varlistentry>
   <term><xref linkend="guc-maintenance-work-mem"></term>
   <listitem>
    <para>
     Build time for a <acronym>GIN</acronym> index is very sensitive to
     the <varname>maintenance_work_mem</> setting; it doesn't pay to
     skimp on work memory during index creation.
    </para>
   </listitem>
  </varlistentry>

  <varlistentry>
   <term><xref linkend="guc-gin-fuzzy-search-limit"></term>
   <listitem>
    <para>
     The primary goal of developing <acronym>GIN</acronym> indexes was
     to create support for highly scalable, full-text search in
     <productname>PostgreSQL</productname>, and there are often situations when
     a full-text search returns a very large set of results.  Moreover, this
     often happens when the query contains very frequent words, so that the
     large result set is not even useful.  Since reading many
     tuples from the disk and sorting them could take a lot of time, this is
     unacceptable for production.  (Note that the index search itself is very
     fast.)
    </para>
    <para>
     To facilitate controlled execution of such queries
     <acronym>GIN</acronym> has a configurable soft upper limit on the
     number of rows returned, the
     <varname>gin_fuzzy_search_limit</varname> configuration parameter.
     It is set to 0 (meaning no limit) by default.
     If a non-zero limit is set, then the returned set is a subset of
     the whole result set, chosen at random.
    </para>
    <para>
     <quote>Soft</quote> means that the actual number of returned results
     could differ slightly from the specified limit, depending on the query
     and the quality of the system's random number generator.
    </para>
   </listitem>
  </varlistentry>
 </variablelist>

</sect1>

<sect1 id="gin-limit">
 <title>Limitations</title>

 <para>
  <acronym>GIN</acronym> doesn't support full index scans: because there are
  often many keys per value, each heap pointer would be returned many times,
  and there is no easy way to prevent this.
 </para>

 <para>
  When <function>extractQuery</function> returns zero keys,
  <acronym>GIN</acronym> will emit an error.  Depending on the operator,
  a void query might match all, some, or none of the indexed values (for
  example, every array contains the empty array, but does not overlap the
  empty array), and <acronym>GIN</acronym> cannot determine the correct
  answer, nor produce a full-index-scan result if it could determine that
  that was correct.
 </para>

 <para>
  It is not an error for <function>extractValue</> to return zero keys,
  but in this case the indexed value will be unrepresented in the index.
  This is another reason why full index scan is not useful &mdash; it would
  miss such rows.
 </para>

 <para>
  It is possible for an operator class to circumvent the restriction against
  full index scan.  To do that, <function>extractValue</> must return at least
  one (possibly dummy) key for every indexed value, and
  <function>extractQuery</function> must convert an unrestricted search into
  a partial-match query that will scan the whole index.  This is inefficient
  but might be necessary to avoid corner-case failures with operators such
  as LIKE.  Note however that failure could still occur if the intermediate
  <literal>TIDBitmap</> becomes lossy.
 </para>
</sect1>

<sect1 id="gin-examples">
 <title>Examples</title>

 <para>
  The <productname>PostgreSQL</productname> source distribution includes
  <acronym>GIN</acronym> operator classes for <type>tsvector</> and
  for one-dimensional arrays of all internal types.  Prefix searching in
  <type>tsvector</> is implemented using the <acronym>GIN</> partial match
  feature.
  The following <filename>contrib</> modules also contain
  <acronym>GIN</acronym> operator classes:
 </para>

 <variablelist>
  <varlistentry>
   <term>hstore</term>
   <listitem>
    <para>Module for storing (key, value) pairs</para>
   </listitem>
  </varlistentry>

  <varlistentry>
   <term>intarray</term>
   <listitem>
    <para>Enhanced support for int4[]</para>
   </listitem>
  </varlistentry>

  <varlistentry>
   <term>pg_trgm</term>
   <listitem>
    <para>Text similarity using trigram matching</para>
   </listitem>
  </varlistentry>
 </variablelist>
</sect1>

</chapter>
