gin.sgml 9.26 KB
Newer Older
1
<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.12 2007/11/13 23:36:26 tgl Exp $ -->
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

<chapter id="GIN">
<title>GIN Indexes</title>

   <indexterm>
    <primary>index</primary>
    <secondary>GIN</secondary>
   </indexterm>

<sect1 id="gin-intro">
 <title>Introduction</title>

 <para>
   <acronym>GIN</acronym> stands for Generalized Inverted Index.  It is
   an index structure storing a set of (key, posting list) pairs, where
17
   a <quote>posting list</> is a set of rows in which the key occurs. Each
18
   indexed value can contain many keys, so the same row ID can appear in
19
   multiple posting lists.
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
 </para>

 <para>
   It is generalized in the sense that a <acronym>GIN</acronym> index
   does not need to be aware of the operation that it accelerates.
   Instead, it uses custom strategies defined for particular data types.
 </para>

 <para>
  One advantage of <acronym>GIN</acronym> is that it allows the development
  of custom data types with the appropriate access methods, by
  an expert in the domain of the data type, rather than a database expert.
  This is much the same advantage as using <acronym>GiST</acronym>.
 </para>

35 36 37 38 39 40 41
 <para>
  The <acronym>GIN</acronym>
  implementation in <productname>PostgreSQL</productname> is primarily
  maintained by Teodor Sigaev and Oleg Bartunov. There is more
  information about <acronym>GIN</acronym> on their
  <ulink url="http://www.sai.msu.su/~megera/oddmuse/index.cgi/Gin">website</ulink>.
 </para>
42 43 44 45 46 47 48
</sect1>

<sect1 id="gin-extensibility">
 <title>Extensibility</title>

 <para>
   The <acronym>GIN</acronym> interface has a high level of abstraction,
49
   requiring the access method implementer only to implement the semantics of
50 51 52 53 54 55 56
   the data type being accessed.  The <acronym>GIN</acronym> layer itself
   takes care of concurrency, logging and searching the tree structure.
 </para>

 <para>
   All it takes to get a <acronym>GIN</acronym> access method working
   is to implement four user-defined methods, which define the behavior of
57 58 59
   keys in the tree and the relationships between keys, indexed values,
   and indexable queries. In short, <acronym>GIN</acronym> combines
   extensibility with generality, code reuse, and a clean interface.
60 61 62
 </para>

 <para>
63 64
   The four methods that an index operator class for
   <acronym>GIN</acronym> must provide are:
65 66 67 68
 </para>

 <variablelist>
    <varlistentry>
69
     <term>int compare(Datum a, Datum b)</term>
70 71
     <listitem>
      <para>
72 73 74
       Compares keys (not indexed values!) and returns an integer less than
       zero, zero, or greater than zero, indicating whether the first key is
       less than, equal to, or greater than the second.
75 76 77 78 79
      </para>
     </listitem>
    </varlistentry>

    <varlistentry>
80
     <term>Datum* extractValue(Datum inputValue, int32 *nkeys)</term>
81 82
     <listitem>
      <para>
83 84
       Returns an array of keys given a value to be indexed.  The
       number of returned keys must be stored into <literal>*nkeys</>.
85 86 87 88 89
      </para>
     </listitem>
    </varlistentry>

    <varlistentry>
90
     <term>Datum* extractQuery(Datum query, int32 *nkeys,
91
        StrategyNumber n)</term>
92 93
     <listitem>
      <para>
94 95 96 97 98 99 100 101 102
       Returns an array of keys given a value to be queried; that is,
       <literal>query</> is the value on the right-hand side of an
       indexable operator whose left-hand side is the indexed column.
       <literal>n</> is the strategy number of the operator within the
       operator class (see <xref linkend="xindex-strategies">).
       Often, <function>extractQuery</> will need
       to consult <literal>n</> to determine the data type of
       <literal>query</> and the key values that need to be extracted.
       The number of returned keys must be stored into <literal>*nkeys</>.
103 104 105 106 107 108
       If number of keys is equal to zero then <function>extractQuery</> 
       should store 0 or -1 into <literal>*nkeys</>. 0 means that any 
       row matches the <literal>query</> and sequence scan should be 
       produced. -1 means nothing can satisfy <literal>query</>. 
       Choice of value should be based on semantics meaning of operation with 
       given strategy number.
109 110 111 112 113
      </para>
     </listitem>
    </varlistentry>

    <varlistentry>
114
     <term>bool consistent(bool check[], StrategyNumber n, Datum query)</term>
115 116
     <listitem>
      <para>
117
       Returns TRUE if the indexed value satisfies the query operator with
118
       strategy number <literal>n</> (or would satisfy, if the operator is
119 120 121 122 123 124 125 126
       marked RECHECK in the operator class).  The <literal>check</> array has
       the same length as the number of keys previously returned by
       <function>extractQuery</> for this query.  Each element of the
       <literal>check</> array is TRUE if the indexed value contains the
       corresponding query key, ie, if (check[i] == TRUE) the i-th key of the
       <function>extractQuery</> result array is present in the indexed value.
       The original <literal>query</> datum (not the extracted key array!) is
       passed in case the <function>consistent</> method needs to consult it.
127 128 129 130 131 132 133 134
      </para>
     </listitem>
    </varlistentry>

  </variablelist>

</sect1>

135 136 137 138 139 140 141 142 143 144 145 146 147
<sect1 id="gin-implementation">
 <title>Implementation</title>

 <para>
  Internally, a <acronym>GIN</acronym> index contains a B-tree index
  constructed over keys, where each key is an element of the indexed value
  (a member of an array, for example) and where each tuple in a leaf page is
  either a pointer to a B-tree over heap pointers (PT, posting tree), or a
  list of heap pointers (PL, posting list) if the list is small enough.
 </para>

</sect1>

148
<sect1 id="gin-tips">
149
<title>GIN tips and tricks</title>
150 151 152 153 154

 <variablelist>
  <varlistentry>
   <term>Create vs insert</term>
   <listitem>
155 156 157 158 159 160
    <para>
     In most cases, insertion into a <acronym>GIN</acronym> index is slow
     due to the likelihood of many keys being inserted for each value.
     So, for bulk insertions into a table it is advisable to drop the GIN
     index and recreate it after finishing bulk insertion.
    </para>
161 162 163 164
   </listitem>
  </varlistentry>

  <varlistentry>
165
   <term><xref linkend="guc-gin-fuzzy-search-limit"></term>
166
   <listitem>
167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190
    <para>
     The primary goal of developing <acronym>GIN</acronym> indexes was
     to create support for highly scalable, full-text search in
     <productname>PostgreSQL</productname>, and there are often situations when
     a full-text search returns a very large set of results.  Moreover, this
     often happens when the query contains very frequent words, so that the
     large result set is not even useful.  Since reading many
     tuples from the disk and sorting them could take a lot of time, this is
     unacceptable for production.  (Note that the index search itself is very
     fast.)
    </para>
    <para>
     To facilitate controlled execution of such queries
     <acronym>GIN</acronym> has a configurable soft upper limit on the size
     of the returned set, the
     <varname>gin_fuzzy_search_limit</varname> configuration parameter.
     It is set to 0 (meaning no limit) by default.
     If a non-zero limit is set, then the returned set is a subset of
     the whole result set, chosen at random.
    </para>
    <para>
     <quote>Soft</quote> means that the actual number of returned results
     could differ slightly from the specified limit, depending on the query
     and the quality of the system's random number generator.
191 192 193
    </para>
   </listitem>
  </varlistentry>
Teodor Sigaev's avatar
Teodor Sigaev committed
194
 </variablelist>
195 196 197 198 199 200 201

</sect1>

<sect1 id="gin-limit">
 <title>Limitations</title>

 <para>
202 203 204
  <acronym>GIN</acronym> doesn't support full index scans: because there are
  often many keys per value, each heap pointer would be returned many times,
  and there is no easy way to prevent this.
205 206 207
 </para>

 <para>
208
  When <function>extractQuery</function> returns zero keys,
209 210 211
  <acronym>GIN</acronym> will emit an error.  Depending on the operator,
  a void query might match all, some, or none of the indexed values (for
  example, every array contains the empty array, but does not overlap the
212
  empty array), and <acronym>GIN</acronym> cannot determine the correct
213 214
  answer, nor produce a full-index-scan result if it could determine that
  that was correct.
215 216 217
 </para>

 <para>
218 219 220 221 222 223 224
  It is not an error for <function>extractValue</> to return zero keys,
  but in this case the indexed value will be unrepresented in the index.
  This is another reason why full index scan is not useful &mdash; it would
  miss such rows.
 </para>

 <para>
225
  <acronym>GIN</acronym> searches keys only by equality matching.  This might
226 227 228
  be improved in future.
 </para>
</sect1>
Tom Lane's avatar
Tom Lane committed
229

230 231 232 233 234
<sect1 id="gin-examples">
 <title>Examples</title>

 <para>
  The <productname>PostgreSQL</productname> source distribution includes
235 236
  <acronym>GIN</acronym> operator classes for <type>tsvector</> and
  for one-dimensional arrays of all internal types.  The following
237
  <filename>contrib</> modules also contain <acronym>GIN</acronym>
238
  operator classes:
239
 </para>
240

241
 <variablelist>
242 243 244 245 246 247 248
  <varlistentry>
   <term>hstore</term>
   <listitem>
    <para>Module for storing (key, value) pairs</para>
   </listitem>
  </varlistentry>

249 250 251 252 253 254 255 256
  <varlistentry>
   <term>intarray</term>
   <listitem>
    <para>Enhanced support for int4[]</para>
   </listitem>
  </varlistentry>

  <varlistentry>
257
   <term>pg_trgm</term>
258
   <listitem>
259
    <para>Text similarity using trigram matching</para>
260 261
   </listitem>
  </varlistentry>
Tom Lane's avatar
Tom Lane committed
262 263
 </variablelist>
</sect1>
264 265

</chapter>