Skip to content

P
Postgres FD Implementation
Commit 6efae5bf authored by Tom Lane's avatar Tom Lane
Browse files
Options

Another round of editorialization on the text search documentation. 

Notably, standardize on using "token" for the strings output by a parser,
while "lexeme" is reserved for the normalized strings produced by a
dictionary.
parent cb0d539d
Show whitespace changes
Inline Side-by-side
Showing with 361 additions and 242 deletions
doc/src/sgml/ref/alter_tsdictionary.sgml
View file @ 6efae5bf
<!-- <!--
$PostgreSQL: pgsql/doc/src/sgml/ref/alter_tsdictionary.sgml,v 1.2 2007/08/22 01:39:44 tgl Exp $ $PostgreSQL: pgsql/doc/src/sgml/ref/alter_tsdictionary.sgml,v 1.3 2007/10/17 01:01:28 tgl Exp $
PostgreSQL documentation PostgreSQL documentation
--> -->
...@@ -122,6 +122,21 @@ ALTER TEXT SEARCH DICTIONARY my_dict ( StopWords = newrussian ); ...@@ -122,6 +122,21 @@ ALTER TEXT SEARCH DICTIONARY my_dict ( StopWords = newrussian );
<programlisting> <programlisting>
ALTER TEXT SEARCH DICTIONARY my_dict ( language = dutch, StopWords ); ALTER TEXT SEARCH DICTIONARY my_dict ( language = dutch, StopWords );
</programlisting> </programlisting>
<para>
The following example command <quote>updates</> the dictionary's
definition without actually changing anything.
<programlisting>
ALTER TEXT SEARCH DICTIONARY my_dict ( dummy );
</programlisting>
(The reason this works is that the option removal code doesn't complain
if there is no such option.) This trick is useful when changing
configuration files for the dictionary: the <command>ALTER</> will
force existing database sessions to re-read the configuration files,
which otherwise they would never do if they had read them earlier.
</para>
</refsect1> </refsect1>
<refsect1> <refsect1>
......
doc/src/sgml/textsearch.sgml
View file @ 6efae5bf
<!-- $PostgreSQL: pgsql/doc/src/sgml/textsearch.sgml,v 1.19 2007/10/15 21:39:57 tgl Exp $ --> <!-- $PostgreSQL: pgsql/doc/src/sgml/textsearch.sgml,v 1.20 2007/10/17 01:01:27 tgl Exp $ -->
<chapter id="textsearch"> <chapter id="textsearch">
<title id="textsearch-title">Full Text Search</title> <title id="textsearch-title">Full Text Search</title>
...@@ -75,11 +75,11 @@ ...@@ -75,11 +75,11 @@
<itemizedlist mark="none"> <itemizedlist mark="none">
<listitem> <listitem>
<para> <para>
<emphasis>Parsing documents into <firstterm>lexemes</></emphasis>. It is <emphasis>Parsing documents into <firstterm>tokens</></emphasis>. It is
useful to identify various classes of lexemes, e.g. numbers, words, useful to identify various classes of tokens, e.g. numbers, words,
complex words, email addresses, so that they can be processed complex words, email addresses, so that they can be processed
differently. In principle lexeme classes depend on the specific differently. In principle token classes depend on the specific
application but for an ordinary search it is useful to have a predefined application, but for most purposes it is adequate to use a predefined
set of classes. set of classes.
<productname>PostgreSQL</productname> uses a <firstterm>parser</> to <productname>PostgreSQL</productname> uses a <firstterm>parser</> to
perform this step. A standard parser is provided, and custom parsers perform this step. A standard parser is provided, and custom parsers
...@@ -89,11 +89,18 @@ ...@@ -89,11 +89,18 @@
<listitem> <listitem>
<para> <para>
<emphasis>Converting lexemes into <firstterm>normalized <emphasis>Converting tokens into <firstterm>lexemes</></emphasis>.
form</></emphasis>. This allows searches to find variant forms of the A lexeme is a string, just like a token, but it has been
<firstterm>normalized</> so that different forms of the same word
are made alike. For example, normalization almost always includes
folding upper-case letters to lower-case, and often involves removal
of suffixes (such as <literal>s</> or <literal>es</> in English).
This allows searches to find variant forms of the
same word, without tediously entering all the possible variants. same word, without tediously entering all the possible variants.
Also, this step typically eliminates <firstterm>stop words</>, which Also, this step typically eliminates <firstterm>stop words</>, which
are words that are so common that they are useless for searching. are words that are so common that they are useless for searching.
(In short, then, tokens are raw fragments of the document text, while
lexemes are words that are believed useful for indexing and searching.)
<productname>PostgreSQL</productname> uses <firstterm>dictionaries</> to <productname>PostgreSQL</productname> uses <firstterm>dictionaries</> to
perform this step. Various standard dictionaries are provided, and perform this step. Various standard dictionaries are provided, and
custom ones can be created for specific needs. custom ones can be created for specific needs.
...@@ -105,17 +112,17 @@ ...@@ -105,17 +112,17 @@
<emphasis>Storing preprocessed documents optimized for <emphasis>Storing preprocessed documents optimized for
searching</emphasis>. For example, each document can be represented searching</emphasis>. For example, each document can be represented
as a sorted array of normalized lexemes. Along with the lexemes it is as a sorted array of normalized lexemes. Along with the lexemes it is
desirable to store positional information to use for <firstterm>proximity often desirable to store positional information to use for
ranking</firstterm>, so that a document that contains a more <firstterm>proximity ranking</firstterm>, so that a document that
<quote>dense</> region of query words is contains a more <quote>dense</> region of query words is
assigned a higher rank than one with scattered query words. assigned a higher rank than one with scattered query words.
</para> </para>
</listitem> </listitem>
</itemizedlist> </itemizedlist>
<para> <para>
Dictionaries allow fine-grained control over how lexemes are normalized. Dictionaries allow fine-grained control over how tokens are normalized.
With dictionaries you can: With appropriate dictionaries, you can:
</para> </para>
<itemizedlist spacing="compact" mark="bullet"> <itemizedlist spacing="compact" mark="bullet">
...@@ -155,9 +162,11 @@ ...@@ -155,9 +162,11 @@
<para> <para>
A data type <type>tsvector</type> is provided for storing preprocessed A data type <type>tsvector</type> is provided for storing preprocessed
documents, along with a type <type>tsquery</type> for representing processed documents, along with a type <type>tsquery</type> for representing processed
queries (<xref linkend="datatype-textsearch">). Also, a full text search queries (<xref linkend="datatype-textsearch">). There are many
operator <literal>@@</literal> is defined for these data types (<xref functions and operators available for these data types
linkend="textsearch-searches">). Full text searches can be accelerated (<xref linkend="functions-textsearch">), the most important of which is
the match operator <literal>@@</literal>, which we introduce in
<xref linkend="textsearch-searches">. Full text searches can be accelerated
using indexes (<xref linkend="textsearch-indexes">). using indexes (<xref linkend="textsearch-indexes">).
</para> </para>
...@@ -221,45 +230,78 @@ WHERE mid = did AND mid = 12; ...@@ -221,45 +230,78 @@ WHERE mid = did AND mid = 12;
<para> <para>
Full text searching in <productname>PostgreSQL</productname> is based on Full text searching in <productname>PostgreSQL</productname> is based on
the operator <literal>@@</literal>, which tests whether a <type>tsvector</type> the match operator <literal>@@</literal>, which returns
(document) matches a <type>tsquery</type> (query). Also, this operator <literal>true</literal> if a <type>tsvector</type>
supports <type>text</type> input, allowing explicit conversion of a text (document) matches a <type>tsquery</type> (query).
string to <type>tsvector</type> to be skipped. The variants available It doesn't matter which data type is written first:
are:
<programlisting> <programlisting>
tsvector @@ tsquery SELECT 'a fat cat sat on a mat and ate a fat rat'::tsvector @@ 'cat &amp; rat'::tsquery;
tsquery @@ tsvector ?column?
text @@ tsquery ----------
text @@ text t
SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector;
?column?
----------
f
</programlisting> </programlisting>
</para> </para>
<para> <para>
The match operator <literal>@@</literal> returns <literal>true</literal> if As the above example suggests, a <type>tsquery</type> is not just raw
the <type>tsvector</type> matches the <type>tsquery</type>. It doesn't text, any more than a <type>tsvector</type> is. A <type>tsquery</type>
matter which data type is written first: contains search terms, which must be already-normalized lexemes, and may
contain AND, OR, and NOT operators.
(For details see <xref linkend="datatype-textsearch">.) There are
functions <function>to_tsquery</> and <function>plainto_tsquery</>
that are helpful in converting user-written text into a proper
<type>tsquery</type>, for example by normalizing words appearing in
the text. Similarly, <function>to_tsvector</> is used to parse and
normalize a document string. So in practice a text search match would
look more like this:
<programlisting> <programlisting>
SELECT 'cat &amp; rat'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector; SELECT to_tsvector('fat cats ate fat rats') @@ to_tsquery('fat &amp; rat');
?column? ?column?
---------- ----------
t t
</programlisting>
SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector; Observe that this match would not succeed if written as
<programlisting>
SELECT 'fat cats ate fat rats'::tsvector @@ to_tsquery('fat &amp; rat');
?column? ?column?
---------- ----------
f f
</programlisting> </programlisting>
since here no normalization of the word <literal>rats</> will occur:
the elements of a <type>tsvector</> are lexemes, which are assumed
already normalized.
</para> </para>
<para> <para>
The <literal>@@</literal> operator also
supports <type>text</type> input, allowing explicit conversion of a text
string to <type>tsvector</type> or <type>tsquery</> to be skipped
in simple cases. The variants available are:
<programlisting>
tsvector @@ tsquery
tsquery @@ tsvector
text @@ tsquery
text @@ text
</programlisting>
</para>
<para>
The first two of these we saw already.
The form <type>text</type> <literal>@@</literal> <type>tsquery</type> The form <type>text</type> <literal>@@</literal> <type>tsquery</type>
is equivalent to <literal>to_tsvector(x) @@ y</literal>. is equivalent to <literal>to_tsvector(x) @@ y</literal>.
The form <type>text</type> <literal>@@</literal> <type>text</type> The form <type>text</type> <literal>@@</literal> <type>text</type>
is equivalent to <literal>to_tsvector(x) @@ plainto_tsquery(y)</literal>. is equivalent to <literal>to_tsvector(x) @@ plainto_tsquery(y)</literal>.
<xref linkend="functions-textsearch"> contains a complete list of full text
search functions and operators.
</para> </para>
</sect2> </sect2>
...@@ -305,14 +347,14 @@ SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t ...@@ -305,14 +347,14 @@ SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t
<itemizedlist spacing="compact" mark="bullet"> <itemizedlist spacing="compact" mark="bullet">
<listitem> <listitem>
<para> <para>
<firstterm>Text search parsers</> break documents into lexemes <firstterm>Text search parsers</> break documents into tokens
and classify each lexeme (for example, as words or numbers). and classify each token (for example, as words or numbers).
</para> </para>
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
<firstterm>Text search dictionaries</> convert lexemes to normalized <firstterm>Text search dictionaries</> convert tokens to normalized
form and reject stop words. form and reject stop words.
</para> </para>
</listitem> </listitem>
...@@ -328,7 +370,7 @@ SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t ...@@ -328,7 +370,7 @@ SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t
<listitem> <listitem>
<para> <para>
<firstterm>Text search configurations</> specify a parser and a set <firstterm>Text search configurations</> specify a parser and a set
of dictionaries to use to normalize the lexemes produced by the parser. of dictionaries to use to normalize the tokens produced by the parser.
</para> </para>
</listitem> </listitem>
</itemizedlist> </itemizedlist>
...@@ -401,6 +443,13 @@ ORDER BY dlm DESC LIMIT 10; ...@@ -401,6 +443,13 @@ ORDER BY dlm DESC LIMIT 10;
in one of the two fields. in one of the two fields.
</para> </para>
<para>
Although these queries will work without an index, most applications
will find this approach too slow, except perhaps for occasional ad-hoc
queries. Practical use of text searching usually requires creating
an index.
</para>
</sect2> </sect2>
<sect2 id="textsearch-tables-index"> <sect2 id="textsearch-tables-index">
...@@ -408,7 +457,7 @@ ORDER BY dlm DESC LIMIT 10; ...@@ -408,7 +457,7 @@ ORDER BY dlm DESC LIMIT 10;
<para> <para>
We can create a <acronym>GIN</acronym> index (<xref We can create a <acronym>GIN</acronym> index (<xref
linkend="textsearch-indexes">) to speed up the search: linkend="textsearch-indexes">) to speed up text searches:
<programlisting> <programlisting>
CREATE INDEX pgweb_idx ON pgweb USING gin(to_tsvector('english', body)); CREATE INDEX pgweb_idx ON pgweb USING gin(to_tsvector('english', body));
...@@ -562,21 +611,26 @@ SELECT to_tsvector('english', 'a fat cat sat on a mat - it ate a fat rats'); ...@@ -562,21 +611,26 @@ SELECT to_tsvector('english', 'a fat cat sat on a mat - it ate a fat rats');
<para> <para>
The <function>to_tsvector</function> function internally calls a parser The <function>to_tsvector</function> function internally calls a parser
which breaks the document (<literal>a fat cat sat on a mat - it ate a which breaks the <quote>document</> text into tokens and assigns a type to
fat rats</literal>) into words and corresponding types. The default parser each token. The default parser recognizes 23 token types.
recognizes 23 types. Each word, depending on its type, passes through a For each token, a list of
group of dictionaries (<xref linkend="textsearch-dictionaries">). At the dictionaries (<xref linkend="textsearch-dictionaries">) is consulted,
end of this step we obtain <emphasis>lexemes</emphasis>. For example, where the list can vary depending on the token type. The first dictionary
that <firstterm>recognizes</> the token emits one or more normalized
<firstterm>lexemes</firstterm> to represent the token. For example,
<literal>rats</literal> became <literal>rat</literal> because one of the <literal>rats</literal> became <literal>rat</literal> because one of the
dictionaries recognized that the word <literal>rats</literal> is a plural dictionaries recognized that the word <literal>rats</literal> is a plural
form of <literal>rat</literal>. Some words are treated as "stop words" form of <literal>rat</literal>. Some words are recognized as <quote>stop
(<xref linkend="textsearch-stopwords">) and ignored since they occur too words</> (<xref linkend="textsearch-stopwords">), which causes them to
frequently and have little informational value. In our example these are be ignored since they occur too frequently to be useful in searching.
In our example these are
<literal>a</literal>, <literal>on</literal>, and <literal>it</literal>. <literal>a</literal>, <literal>on</literal>, and <literal>it</literal>.
The punctuation sign <literal>-</literal> was also ignored because its If no dictionary in the list recognizes the token then it is also ignored.
type (<literal>Space symbols</literal>) is not indexed. The choice of In this example that happened to the punctuation sign <literal>-</literal>
parser, dictionaries and what types of lexemes to index is determined by because there are in fact no dictionaries assigned for its token type
the selected text search configuration (<xref (<literal>Space symbols</literal>), meaning space tokens will never be
indexed. The choices of parser, dictionaries and which types of tokens to
index are determined by the selected text search configuration (<xref
linkend="textsearch-tables-configuration">). It is possible to have linkend="textsearch-tables-configuration">). It is possible to have
many different configurations in the same database, and predefined many different configurations in the same database, and predefined
configurations are available for various languages. In our example configurations are available for various languages. In our example
...@@ -619,13 +673,18 @@ SELECT * FROM ts_debug('english','a fat cat sat on a mat - it ate a fat rats'); ...@@ -619,13 +673,18 @@ SELECT * FROM ts_debug('english','a fat cat sat on a mat - it ate a fat rats');
lword | Latin word | rats | {english} | english: {rat} lword | Latin word | rats | {english} | english: {rat}
(24 rows) (24 rows)
</programlisting> </programlisting>
A more extensive example of <function>ts_debug</function> output
appears in <xref linkend="textsearch-debugging">.
</para> </para>
<para> <para>
The function <function>setweight()</function> is used to label the entries The function <function>setweight()</function> can be used to label the
of a <type>tsvector</type> with a given <firstterm>weight</>. The typical entries of a <type>tsvector</type> with a given <firstterm>weight</>,
usage of this is to mark entries coming from where a weight is one of the letters <literal>A</>, <literal>B</>,
different parts of a document, perhaps by importance. Later, this can be <literal>C</>, or <literal>D</>.
This is typically used to mark entries coming from
different parts of a document. Later, this information can be
used for ranking of search results in addition to positional information used for ranking of search results in addition to positional information
(distance between query terms). If no ranking is required, positional (distance between query terms). If no ranking is required, positional
information can be removed from <type>tsvector</type> using the information can be removed from <type>tsvector</type> using the
...@@ -649,12 +708,14 @@ UPDATE tt SET ti = ...@@ -649,12 +708,14 @@ UPDATE tt SET ti =
Here we have used <function>setweight()</function> to label the source Here we have used <function>setweight()</function> to label the source
of each lexeme in the finished <type>tsvector</type>, and then merged of each lexeme in the finished <type>tsvector</type>, and then merged
the labeled <type>tsvector</type> values using the concatenation the labeled <type>tsvector</type> values using the <type>tsvector</>
operator <literal>||</>. concatenation operator <literal>||</>.
</para> </para>
<para> <para>
The following functions allow manual parsing control: The following functions allow manual parsing control. They would
not normally be used during actual text searches, but they are very
useful for debugging purposes:
<variablelist> <variablelist>
...@@ -674,8 +735,8 @@ UPDATE tt SET ti = ...@@ -674,8 +735,8 @@ UPDATE tt SET ti =
<para> <para>
Parses the given <replaceable>document</replaceable> and returns a Parses the given <replaceable>document</replaceable> and returns a
series of records, one for each token produced by parsing. Each record series of records, one for each token produced by parsing. Each record
includes a <varname>tokid</varname> giving its type and a includes a <varname>tokid</varname> showing the assigned token type
<varname>token</varname> which gives its content: and a <varname>token</varname> which is the text of the token.
<programlisting> <programlisting>
SELECT * FROM ts_parse('default','123 - a number'); SELECT * FROM ts_parse('default','123 - a number');
...@@ -705,12 +766,13 @@ SELECT * FROM ts_parse('default','123 - a number'); ...@@ -705,12 +766,13 @@ SELECT * FROM ts_parse('default','123 - a number');
<listitem> <listitem>
<para> <para>
Returns a table which describes each kind of token the Returns a table which describes each type of token the
<replaceable>parser</replaceable> can recognize. For each token <replaceable>parser</replaceable> can recognize. For each token
type the table gives the <varname>tokid</varname> which the type the table gives the integer <varname>tokid</varname> that the
<replaceable>parser</replaceable> uses to label a <replaceable>parser</replaceable> uses to label a
token of that type, the <varname>alias</varname> which token of that type, the <varname>alias</varname> that
names the token type, and a short <varname>description</varname>: names the token type in configuration commands,
and a short <varname>description</varname>:
<programlisting> <programlisting>
SELECT * FROM ts_token_type('default'); SELECT * FROM ts_token_type('default');
...@@ -755,8 +817,8 @@ SELECT * FROM ts_token_type('default'); ...@@ -755,8 +817,8 @@ SELECT * FROM ts_token_type('default');
<para> <para>
Ranking attempts to measure how relevant documents are to a particular Ranking attempts to measure how relevant documents are to a particular
query by inspecting the number of times each search word appears in the query, typically by checking the number of times each search term appears
document, and whether different search terms occur near each other. in the document and whether the search terms occur near each other.
<productname>PostgreSQL</productname> provides two predefined ranking <productname>PostgreSQL</productname> provides two predefined ranking
functions, which take into account lexical, functions, which take into account lexical,
proximity, and structural information. However, the concept of proximity, and structural information. However, the concept of
...@@ -792,7 +854,8 @@ SELECT * FROM ts_token_type('default'); ...@@ -792,7 +854,8 @@ SELECT * FROM ts_token_type('default');
<listitem> <listitem>
<para> <para>
This ranking function offers the ability to weigh word instances more The optional <replaceable class="PARAMETER">weights</replaceable>
argument offers the ability to weigh word instances more or less
heavily depending on how you have classified them. The weights specify heavily depending on how you have classified them. The weights specify
how heavily to weigh each category of word: how heavily to weigh each category of word:
...@@ -1088,7 +1151,8 @@ ORDER BY rank DESC LIMIT 10) AS foo; ...@@ -1088,7 +1151,8 @@ ORDER BY rank DESC LIMIT 10) AS foo;
<para> <para>
Dictionaries are used to eliminate words that should not be considered in a Dictionaries are used to eliminate words that should not be considered in a
search (<firstterm>stop words</>), and to <firstterm>normalize</> words so search (<firstterm>stop words</>), and to <firstterm>normalize</> words so
that different derived forms of the same word will match. Aside from that different derived forms of the same word will match. A successfully
normalized word is called a <firstterm>lexeme</>. Aside from
improving search quality, normalization and removal of stop words reduce the improving search quality, normalization and removal of stop words reduce the
size of the <type>tsvector</type> representation of a document, thereby size of the <type>tsvector</type> representation of a document, thereby
improving performance. Normalization does not always have linguistic meaning improving performance. Normalization does not always have linguistic meaning
...@@ -1108,7 +1172,8 @@ ORDER BY rank DESC LIMIT 10) AS foo; ...@@ -1108,7 +1172,8 @@ ORDER BY rank DESC LIMIT 10) AS foo;
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
Identical <acronym>URL</acronym> locations are identified and canonicalized: <acronym>URL</acronym> locations can be canonicalized to make
equivalent URLs match:
<itemizedlist spacing="compact" mark="bullet"> <itemizedlist spacing="compact" mark="bullet">
<listitem> <listitem>
...@@ -1131,14 +1196,15 @@ ORDER BY rank DESC LIMIT 10) AS foo; ...@@ -1131,14 +1196,15 @@ ORDER BY rank DESC LIMIT 10) AS foo;
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
Color names are substituted by their hexadecimal values, e.g., Color names can be replaced by their hexadecimal values, e.g.,
<literal>red, green, blue, magenta -> FF0000, 00FF00, 0000FF, FF00FF</literal> <literal>red, green, blue, magenta -> FF0000, 00FF00, 0000FF, FF00FF</literal>
</para> </para>
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
Remove some numeric fractional digits to reduce the range of possible If indexing numbers, we can
numbers, so <emphasis>3.14</emphasis>159265359, remove some fractional digits to reduce the range of possible
numbers, so for example <emphasis>3.14</emphasis>159265359,
<emphasis>3.14</emphasis>15926, <emphasis>3.14</emphasis> will be the same <emphasis>3.14</emphasis>15926, <emphasis>3.14</emphasis> will be the same
after normalization if only two digits are kept after the decimal point. after normalization if only two digits are kept after the decimal point.
</para> </para>
...@@ -1148,22 +1214,23 @@ ORDER BY rank DESC LIMIT 10) AS foo; ...@@ -1148,22 +1214,23 @@ ORDER BY rank DESC LIMIT 10) AS foo;
</para> </para>
<para> <para>
A dictionary is a program that accepts lexemes as A dictionary is a program that accepts a token as
input and returns: input and returns:
<itemizedlist spacing="compact" mark="bullet"> <itemizedlist spacing="compact" mark="bullet">
<listitem> <listitem>
<para> <para>
an array of lexemes if the input lexeme is known to the dictionary an array of lexemes if the input token is known to the dictionary
(notice that one token can produce more than one lexeme)
</para> </para>
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
an empty array if the dictionary knows the lexeme, but it is a stop word an empty array if the dictionary knows the token, but it is a stop word
</para> </para>
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
<literal>NULL</literal> if the dictionary does not recognize the input lexeme <literal>NULL</literal> if the dictionary does not recognize the input token
</para> </para>
</listitem> </listitem>
</itemizedlist> </itemizedlist>
...@@ -1180,14 +1247,14 @@ ORDER BY rank DESC LIMIT 10) AS foo; ...@@ -1180,14 +1247,14 @@ ORDER BY rank DESC LIMIT 10) AS foo;
<para> <para>
A text search configuration binds a parser together with a set of A text search configuration binds a parser together with a set of
dictionaries to process the parser's output lexemes. For each token dictionaries to process the parser's output tokens. For each token
type that the parser can return, a separate stack of dictionaries is type that the parser can return, a separate list of dictionaries is
specified by the configuration. When a lexeme of that type is found specified by the configuration. When a token of that type is found
by the parser, each dictionary in the stack is consulted in turn, by the parser, each dictionary in the list is consulted in turn,
until some dictionary recognizes it as a known word. If it is identified until some dictionary recognizes it as a known word. If it is identified
as a stop word, or if no dictionary recognizes the lexeme, it will be as a stop word, or if no dictionary recognizes the token, it will be
discarded and not indexed or searched for. discarded and not indexed or searched for.
The general rule for configuring a stack of dictionaries The general rule for configuring a list of dictionaries
is to place first the most narrow, most specific dictionary, then the more is to place first the most narrow, most specific dictionary, then the more
general dictionaries, finishing with a very general dictionary, like general dictionaries, finishing with a very general dictionary, like
a <application>Snowball</> stemmer or <literal>simple</>, which a <application>Snowball</> stemmer or <literal>simple</>, which
...@@ -1246,11 +1313,22 @@ SELECT ts_rank_cd ('{1,1,1,1}', to_tsvector('english','list stop words'), to_tsq ...@@ -1246,11 +1313,22 @@ SELECT ts_rank_cd ('{1,1,1,1}', to_tsvector('english','list stop words'), to_tsq
behavior is an attempt to decrease noise. behavior is an attempt to decrease noise.
</para> </para>
</sect2>
<sect2 id="textsearch-simple-dictionary">
<title>Simple Dictionary</title>
<para>
The <literal>simple</> dictionary template operates by converting the
input token to lower case and checking it against a file of stop words.
If it is found in the file then <literal>NULL</> is returned, causing
the token to be discarded. If not, the lower-cased form of the word
is returned as the normalized lexeme.
</para>
<para> <para>
Here is an example of a dictionary that returns the input word as lowercase Here is an example of a dictionary definition using the <literal>simple</>
or <literal>NULL</literal> if it is a stop word; it also specifies the name template:
of a file of stop words. It uses the <literal>simple</> dictionary as
a template:
<programlisting> <programlisting>
CREATE TEXT SEARCH DICTIONARY public.simple_dict ( CREATE TEXT SEARCH DICTIONARY public.simple_dict (
...@@ -1259,6 +1337,20 @@ CREATE TEXT SEARCH DICTIONARY public.simple_dict ( ...@@ -1259,6 +1337,20 @@ CREATE TEXT SEARCH DICTIONARY public.simple_dict (
); );
</programlisting> </programlisting>
Here, <literal>english</literal> is the base name of a file of stop words.
The file's full name will be
<filename>$SHAREDIR/tsearch_data/english.stop</>,
where <literal>$SHAREDIR</> means the
<productname>PostgreSQL</productname> installation's shared-data directory,
often <filename>/usr/local/share/postgresql</> (use <command>pg_config
--sharedir</> to determine it if you're not sure).
The file format is simply a list
of words, one per line. Blank lines and trailing spaces are ignored,
and upper case is folded to lower case, but no other processing is done
on the file contents.
</para>
<para>
Now we can test our dictionary: Now we can test our dictionary:
<programlisting> <programlisting>
...@@ -1276,10 +1368,21 @@ SELECT ts_lexize('public.simple_dict','The'); ...@@ -1276,10 +1368,21 @@ SELECT ts_lexize('public.simple_dict','The');
<caution> <caution>
<para> <para>
Most types of dictionaries rely on configuration files, such as files of stop Most types of dictionaries rely on configuration files, such as files of
words. These files <emphasis>must</> be stored in UTF-8 encoding. They will stop words. These files <emphasis>must</> be stored in UTF-8 encoding.
be translated to the actual database encoding, if that is different, when they They will be translated to the actual database encoding, if that is
are read into the server. different, when they are read into the server.
</para>
</caution>
<caution>
<para>
Normally, a database session will read a dictionary configuration file
only once, when it is first used within the session. If you modify a
configuration file and want to force existing sessions to pick up the
new contents, issue an <command>ALTER TEXT SEARCH DICTIONARY</> command
on the dictionary. This can be a <quote>dummy</> update that doesn't
actually change any parameter values.
</para> </para>
</caution> </caution>
...@@ -1291,7 +1394,7 @@ SELECT ts_lexize('public.simple_dict','The'); ...@@ -1291,7 +1394,7 @@ SELECT ts_lexize('public.simple_dict','The');
<para> <para>
This dictionary template is used to create dictionaries that replace a This dictionary template is used to create dictionaries that replace a
word with a synonym. Phrases are not supported (use the thesaurus word with a synonym. Phrases are not supported (use the thesaurus
dictionary (<xref linkend="textsearch-thesaurus">) for that). A synonym template (<xref linkend="textsearch-thesaurus">) for that). A synonym
dictionary can be used to overcome linguistic problems, for example, to dictionary can be used to overcome linguistic problems, for example, to
prevent an English stemmer dictionary from reducing the word 'Paris' to prevent an English stemmer dictionary from reducing the word 'Paris' to
'pari'. It is enough to have a <literal>Paris paris</literal> line in the 'pari'. It is enough to have a <literal>Paris paris</literal> line in the
...@@ -1304,8 +1407,10 @@ SELECT * FROM ts_debug('english','Paris'); ...@@ -1304,8 +1407,10 @@ SELECT * FROM ts_debug('english','Paris');
lword | Latin word | Paris | {english_stem} | english_stem: {pari} lword | Latin word | Paris | {english_stem} | english_stem: {pari}
(1 row) (1 row)
CREATE TEXT SEARCH DICTIONARY synonym CREATE TEXT SEARCH DICTIONARY synonym (
(TEMPLATE = synonym, SYNONYMS = my_synonyms); TEMPLATE = synonym,
SYNONYMS = my_synonyms
);
ALTER TEXT SEARCH CONFIGURATION english ALTER TEXT SEARCH CONFIGURATION english
ALTER MAPPING FOR lword WITH synonym, english_stem; ALTER MAPPING FOR lword WITH synonym, english_stem;
...@@ -1318,6 +1423,20 @@ SELECT * FROM ts_debug('english','Paris'); ...@@ -1318,6 +1423,20 @@ SELECT * FROM ts_debug('english','Paris');
</programlisting> </programlisting>
</para> </para>
<para>
The only parameter required by the <literal>synonym</> template is
<literal>SYNONYMS</>, which is the base name of its configuration file
&mdash; <literal>my_synonyms</> in the above example.
The file's full name will be
<filename>$SHAREDIR/tsearch_data/my_synonyms.syn</>
(where <literal>$SHAREDIR</> means the
<productname>PostgreSQL</> installation's shared-data directory).
The file format is just one line
per word to be substituted, with the word followed by its synonym,
separated by white space. Blank lines and trailing spaces are ignored,
and upper case is folded to lower case.
</para>
</sect2> </sect2>
<sect2 id="textsearch-thesaurus"> <sect2 id="textsearch-thesaurus">
...@@ -1333,11 +1452,10 @@ SELECT * FROM ts_debug('english','Paris'); ...@@ -1333,11 +1452,10 @@ SELECT * FROM ts_debug('english','Paris');
<para> <para>
Basically a thesaurus dictionary replaces all non-preferred terms by one Basically a thesaurus dictionary replaces all non-preferred terms by one
preferred term and, optionally, preserves them for indexing. Thesauruses preferred term and, optionally, preserves the original terms for indexing
are used during indexing so any change in the thesaurus <emphasis>requires</emphasis> as well. <productname>PostgreSQL</>'s current implementation of the
reindexing. The current implementation of the thesaurus thesaurus dictionary is an extension of the synonym dictionary with added
dictionary is an extension of the synonym dictionary with added <firstterm>phrase</firstterm> support. A thesaurus dictionary requires
<emphasis>phrase</emphasis> support. A thesaurus dictionary requires
a configuration file of the following format: a configuration file of the following format:
<programlisting> <programlisting>
...@@ -1352,7 +1470,7 @@ more sample word(s) : more indexed word(s) ...@@ -1352,7 +1470,7 @@ more sample word(s) : more indexed word(s)
</para> </para>
<para> <para>
A thesaurus dictionary uses a <emphasis>subdictionary</emphasis> (which A thesaurus dictionary uses a <firstterm>subdictionary</firstterm> (which
is defined in the dictionary's configuration) to normalize the input text is defined in the dictionary's configuration) to normalize the input text
before checking for phrase matches. It is only possible to select one before checking for phrase matches. It is only possible to select one
subdictionary. An error is reported if the subdictionary fails to subdictionary. An error is reported if the subdictionary fails to
...@@ -1367,8 +1485,8 @@ more sample word(s) : more indexed word(s) ...@@ -1367,8 +1485,8 @@ more sample word(s) : more indexed word(s)
</para> </para>
<para> <para>
Stop words recognized by the subdictionary are replaced by a 'stop word Stop words recognized by the subdictionary are replaced by a <quote>stop word
placeholder' to record their position. To break possible ties the thesaurus placeholder</quote> to record their position. To break possible ties the thesaurus
uses the last definition. To illustrate this, consider a thesaurus (with uses the last definition. To illustrate this, consider a thesaurus (with
a <parameter>simple</parameter> subdictionary) with pattern a <parameter>simple</parameter> subdictionary) with pattern
<replaceable>swsw</>, where <replaceable>s</> designates any stop word and <replaceable>swsw</>, where <replaceable>s</> designates any stop word and
...@@ -1391,19 +1509,26 @@ the one a two : swsw2 ...@@ -1391,19 +1509,26 @@ the one a two : swsw2
uses these assignments to check if it should handle the next word or stop uses these assignments to check if it should handle the next word or stop
accumulation. The thesaurus dictionary must be configured accumulation. The thesaurus dictionary must be configured
carefully. For example, if the thesaurus dictionary is assigned to handle carefully. For example, if the thesaurus dictionary is assigned to handle
only the <token>lword</token> lexeme, then a thesaurus dictionary only the <literal>lword</literal> token, then a thesaurus dictionary
definition like ' one 7' will not work since lexeme type definition like ' one 7' will not work since token type
<token>uint</token> is not assigned to the thesaurus dictionary. <literal>uint</literal> is not assigned to the thesaurus dictionary.
</para> </para>
</sect2> <caution>
<para>
Thesauruses are used during indexing so any change in the thesaurus
dictionary's parameters <emphasis>requires</emphasis> reindexing.
For most other dictionary types, small changes such as adding or
removing stopwords does not force reindexing.
</para>
</caution>
<sect2 id="textsearch-thesaurus-config"> <sect3 id="textsearch-thesaurus-config">
<title>Thesaurus Configuration</title> <title>Thesaurus Configuration</title>
<para> <para>
To define a new thesaurus dictionary one can use the thesaurus template. To define a new thesaurus dictionary, use the <literal>thesaurus</>
For example: template. For example:
<programlisting> <programlisting>
CREATE TEXT SEARCH DICTIONARY thesaurus_simple ( CREATE TEXT SEARCH DICTIONARY thesaurus_simple (
...@@ -1417,29 +1542,30 @@ CREATE TEXT SEARCH DICTIONARY thesaurus_simple ( ...@@ -1417,29 +1542,30 @@ CREATE TEXT SEARCH DICTIONARY thesaurus_simple (
<itemizedlist spacing="compact" mark="bullet"> <itemizedlist spacing="compact" mark="bullet">
<listitem> <listitem>
<para> <para>
<literal>thesaurus_simple</literal> is the thesaurus dictionary name <literal>thesaurus_simple</literal> is the new dictionary's name
</para> </para>
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
<literal>mythesaurus</literal> is the base name of the thesaurus file <literal>mythesaurus</literal> is the base name of the thesaurus
(its full name will be <filename>$SHAREDIR/tsearch_data/mythesaurus.ths</>, configuration file.
where <literal>$SHAREDIR</> means the installation shared-data directory, (Its full name will be <filename>$SHAREDIR/tsearch_data/mythesaurus.ths</>,
often <filename>/usr/local/share</>). where <literal>$SHAREDIR</> means the installation shared-data
directory.)
</para> </para>
</listitem> </listitem>
<listitem> <listitem>
<para> <para>
<literal>pg_catalog.english_stem</literal> is the dictionary (Snowball <literal>pg_catalog.english_stem</literal> is the subdictionary (here,
English stemmer) to use for thesaurus normalization. Notice that the a Snowball English stemmer) to use for thesaurus normalization.
<literal>english_stem</> dictionary has its own configuration (for example, Notice that the subdictionary will have its own
stop words), which is not shown here. configuration (for example, stop words), which is not shown here.
</para> </para>
</listitem> </listitem>
</itemizedlist> </itemizedlist>
Now it is possible to bind the thesaurus dictionary <literal>thesaurus_simple</literal> Now it is possible to bind the thesaurus dictionary <literal>thesaurus_simple</literal>
and selected <literal>tokens</literal>, for example: to the desired token types, for example:
<programlisting> <programlisting>
ALTER TEXT SEARCH CONFIGURATION russian ALTER TEXT SEARCH CONFIGURATION russian
...@@ -1447,9 +1573,9 @@ ALTER TEXT SEARCH CONFIGURATION russian ...@@ -1447,9 +1573,9 @@ ALTER TEXT SEARCH CONFIGURATION russian
</programlisting> </programlisting>
</para> </para>
</sect2> </sect3>
<sect2 id="textsearch-thesaurus-examples"> <sect3 id="textsearch-thesaurus-examples">
<title>Thesaurus Example</title> <title>Thesaurus Example</title>
<para> <para>
...@@ -1475,11 +1601,11 @@ ALTER TEXT SEARCH CONFIGURATION russian ...@@ -1475,11 +1601,11 @@ ALTER TEXT SEARCH CONFIGURATION russian
ADD MAPPING FOR lword, lhword, lpart_hword WITH thesaurus_astro, english_stem; ADD MAPPING FOR lword, lhword, lpart_hword WITH thesaurus_astro, english_stem;
</programlisting> </programlisting>
Now we can see how it works. Note that <function>ts_lexize</function> cannot Now we can see how it works.
be used for testing the thesaurus (see description of <function>ts_lexize</function> is not very useful for testing a thesaurus,
<function>ts_lexize</function>), but we can use because it treats its input as a single token. Instead we can use
<function>plainto_tsquery</function> and <function>to_tsvector</function> <function>plainto_tsquery</function> and <function>to_tsvector</function>
which accept <literal>text</literal> arguments, not lexemes: which will break their input strings into multiple tokens:
<programlisting> <programlisting>
SELECT plainto_tsquery('supernova star'); SELECT plainto_tsquery('supernova star');
...@@ -1504,13 +1630,14 @@ SELECT to_tsquery('''supernova star'''); ...@@ -1504,13 +1630,14 @@ SELECT to_tsquery('''supernova star''');
</programlisting> </programlisting>
Notice that <literal>supernova star</literal> matches <literal>supernovae Notice that <literal>supernova star</literal> matches <literal>supernovae
stars</literal> in <literal>thesaurus_astro</literal> because we specified the stars</literal> in <literal>thesaurus_astro</literal> because we specified
<literal>english_stem</literal> stemmer in the thesaurus definition. the <literal>english_stem</literal> stemmer in the thesaurus definition.
The stemmer removed the <literal>e</>.
</para> </para>
<para> <para>
To keep an original phrase in full text indexing just add it to the right part To index the original phrase as well as the substitute, just include it
of the definition: in the right-hand part of the definition:
<programlisting> <programlisting>
supernovae stars : sn supernovae stars supernovae stars : sn supernovae stars
...@@ -1522,18 +1649,29 @@ SELECT plainto_tsquery('supernova star'); ...@@ -1522,18 +1649,29 @@ SELECT plainto_tsquery('supernova star');
</programlisting> </programlisting>
</para> </para>
</sect3>
</sect2> </sect2>
<sect2 id="textsearch-ispell-dictionary"> <sect2 id="textsearch-ispell-dictionary">
<title>Ispell Dictionary</title> <title><application>Ispell</> Dictionary</title>
<para>
The <application>Ispell</> dictionary template supports
<firstterm>morphological dictionaries</>, which can normalize many
different linguistic forms of a word into the same lexeme. For example,
an English <application>Ispell</> dictionary can match all declensions and
conjugations of the search term <literal>bank</literal>, e.g.
<literal>banking</>, <literal>banked</>, <literal>banks</>,
<literal>banks'</>, and <literal>bank's</>.
</para>
<para> <para>
The <application>Ispell</> template dictionary for full text allows the The standard <productname>PostgreSQL</productname> distribution does
creation of morphological dictionaries based on <ulink not include any <application>Ispell</> configuration files.
url="http://ficus-www.cs.ucla.edu/geoff/ispell.html">Ispell</ulink>, which Dictionaries for a large number of languages are available from <ulink
supports a large number of languages. This dictionary tries to change an url="http://ficus-www.cs.ucla.edu/geoff/ispell.html">Ispell</ulink>.
input word to its normalized form. Also, more modern spelling dictionaries Also, some more modern dictionary file formats are supported &mdash; <ulink
are supported - <ulink
url="http://en.wikipedia.org/wiki/MySpell">MySpell</ulink> (OO &lt; 2.0.1) url="http://en.wikipedia.org/wiki/MySpell">MySpell</ulink> (OO &lt; 2.0.1)
and <ulink url="http://sourceforge.net/projects/hunspell">Hunspell</ulink> and <ulink url="http://sourceforge.net/projects/hunspell">Hunspell</ulink>
(OO &gt;= 2.0.2). A large list of dictionaries is available on the <ulink (OO &gt;= 2.0.2). A large list of dictionaries is available on the <ulink
...@@ -1542,36 +1680,8 @@ SELECT plainto_tsquery('supernova star'); ...@@ -1542,36 +1680,8 @@ SELECT plainto_tsquery('supernova star');
</para> </para>
<para> <para>
The <application>Ispell</> dictionary allows searches without bothering To create an <application>Ispell</> dictionary, use the built-in
about different linguistic forms of a word. For example, a search on <literal>ispell</literal> template and specify several parameters:
<literal>bank</literal> would return hits of all declensions and
conjugations of the search term <literal>bank</literal>, e.g.
<literal>banking</>, <literal>banked</>, <literal>banks</>,
<literal>banks'</>, and <literal>bank's</>.
<programlisting>
SELECT ts_lexize('english_ispell','banking');
ts_lexize
-----------
{bank}
SELECT ts_lexize('english_ispell','bank''s');
ts_lexize
-----------
{bank}
SELECT ts_lexize('english_ispell','banked');
ts_lexize
-----------
{bank}
</programlisting>
</para>
<para>
To create an ispell dictionary one should use the built-in
<literal>ispell</literal> template and specify several
parameters:
</para> </para>
<programlisting> <programlisting>
...@@ -1585,19 +1695,22 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( ...@@ -1585,19 +1695,22 @@ CREATE TEXT SEARCH DICTIONARY english_ispell (
<para> <para>
Here, <literal>DictFile</>, <literal>AffFile</>, and <literal>StopWords</> Here, <literal>DictFile</>, <literal>AffFile</>, and <literal>StopWords</>
specify the names of the dictionary, affixes, and stop-words files. specify the base names of the dictionary, affixes, and stop-words files.
The stop-words file has the same format explained above for the
<literal>simple</> dictionary type. The format of the other files is
not specified here but is available from the above-mentioned web sites.
</para> </para>
<para> <para>
Ispell dictionaries usually recognize a restricted set of words so they Ispell dictionaries usually recognize a limited set of words, so they
should be used in conjunction with another broader dictionary; for should be followed by another broader dictionary; for
example, a stemming dictionary, which recognizes everything. example, a Snowball dictionary, which recognizes everything.
</para> </para>
<para> <para>
Ispell dictionaries support splitting compound words based on an Ispell dictionaries support splitting compound words.
ispell dictionary. This is a nice feature and full text searching This is a nice feature and
in <productname>PostgreSQL</productname> supports it. <productname>PostgreSQL</productname> supports it.
Notice that the affix file should specify a special flag using the Notice that the affix file should specify a special flag using the
<literal>compoundwords controlled</literal> statement that marks dictionary <literal>compoundwords controlled</literal> statement that marks dictionary
words that can participate in compound formation: words that can participate in compound formation:
...@@ -1606,7 +1719,7 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( ...@@ -1606,7 +1719,7 @@ CREATE TEXT SEARCH DICTIONARY english_ispell (
compoundwords controlled z compoundwords controlled z
</programlisting> </programlisting>
Several examples for the Norwegian language: Here are some examples for the Norwegian language:
<programlisting> <programlisting>
SELECT ts_lexize('norwegian_ispell','overbuljongterningpakkmesterassistent'); SELECT ts_lexize('norwegian_ispell','overbuljongterningpakkmesterassistent');
...@@ -1620,15 +1733,15 @@ SELECT ts_lexize('norwegian_ispell','sjokoladefabrikk'); ...@@ -1620,15 +1733,15 @@ SELECT ts_lexize('norwegian_ispell','sjokoladefabrikk');
<para> <para>
<application>MySpell</> does not support compound words. <application>MySpell</> does not support compound words.
<application>Hunspell</> has sophisticated support for compound words. At <application>Hunspell</> has sophisticated support for compound words. At
present, full text searching implements only the basic compound word present, <productname>PostgreSQL</productname> implements only the basic
operations of Hunspell. compound word operations of Hunspell.
</para> </para>
</note> </note>
</sect2> </sect2>
<sect2 id="textsearch-stemming-dictionary"> <sect2 id="textsearch-snowball-dictionary">
<title><application>Snowball</> Stemming Dictionary</title> <title><application>Snowball</> Dictionary</title>
<para> <para>
The <application>Snowball</> dictionary template is based on the project The <application>Snowball</> dictionary template is based on the project
...@@ -1637,24 +1750,29 @@ SELECT ts_lexize('norwegian_ispell','sjokoladefabrikk'); ...@@ -1637,24 +1750,29 @@ SELECT ts_lexize('norwegian_ispell','sjokoladefabrikk');
many languages (see the <ulink url="http://snowball.tartarus.org">Snowball many languages (see the <ulink url="http://snowball.tartarus.org">Snowball
site</ulink> for more information). Each algorithm understands how to site</ulink> for more information). Each algorithm understands how to
reduce common variant forms of words to a base, or stem, spelling within reduce common variant forms of words to a base, or stem, spelling within
its language. A Snowball dictionary requires a language parameter to its language. A Snowball dictionary requires a <literal>language</>
identify which stemmer to use, and optionally can specify a stopword file parameter to identify which stemmer to use, and optionally can specify a
name that gives a list of words to eliminate. <literal>stopword</> file name that gives a list of words to eliminate.
(<productname>PostgreSQL</productname>'s standard stopword lists are also (<productname>PostgreSQL</productname>'s standard stopword lists are also
provided by the Snowball project.) provided by the Snowball project.)
For example, there is a built-in definition equivalent to For example, there is a built-in definition equivalent to
<programlisting> <programlisting>
CREATE TEXT SEARCH DICTIONARY english_stem ( CREATE TEXT SEARCH DICTIONARY english_stem (
TEMPLATE = snowball, Language = english, StopWords = english TEMPLATE = snowball,
Language = english,
StopWords = english
); );
</programlisting> </programlisting>
The stopword file format is the same as already explained.
</para> </para>
<para> <para>
The <application>Snowball</> dictionary recognizes everything, so it is best A <application>Snowball</> dictionary recognizes everything, whether
to place it at the end of the dictionary stack. It it useless to have it or not it is able to simplify the word, so it should be placed
before any other dictionary because a lexeme will never pass through it to at the end of the dictionary list. It it useless to have it
before any other dictionary because a token will never pass through it to
the next dictionary. the next dictionary.
</para> </para>
...@@ -1676,15 +1794,15 @@ CREATE TEXT SEARCH DICTIONARY english_stem ( ...@@ -1676,15 +1794,15 @@ CREATE TEXT SEARCH DICTIONARY english_stem (
<term> <term>
<synopsis> <synopsis>
ts_lexize(<replaceable class="PARAMETER">dict_name</replaceable> text, <replaceable class="PARAMETER">lexeme</replaceable> text) returns text[] ts_lexize(<replaceable class="PARAMETER">dict_name</replaceable> text, <replaceable class="PARAMETER">token</replaceable> text) returns text[]
</synopsis> </synopsis>
</term> </term>
<listitem> <listitem>
<para> <para>
Returns an array of lexemes if the input Returns an array of lexemes if the input
<replaceable>lexeme</replaceable> is known to the dictionary <replaceable>token</replaceable> is known to the dictionary
<replaceable>dict_name</replaceable>, or an empty array if the lexeme <replaceable>dict_name</replaceable>, or an empty array if the token
is known to the dictionary but it is a stop word, or is known to the dictionary but it is a stop word, or
<literal>NULL</literal> if it is an unknown word. <literal>NULL</literal> if it is an unknown word.
</para> </para>
...@@ -1709,7 +1827,7 @@ SELECT ts_lexize('english_stem', 'a'); ...@@ -1709,7 +1827,7 @@ SELECT ts_lexize('english_stem', 'a');
<note> <note>
<para> <para>
The <function>ts_lexize</function> function expects a The <function>ts_lexize</function> function expects a
<replaceable>lexeme</replaceable>, not text. Below is an example: <replaceable>token</replaceable>, not text. Below is an example:
<programlisting> <programlisting>
SELECT ts_lexize('thesaurus_astro','supernovae stars') is null; SELECT ts_lexize('thesaurus_astro','supernovae stars') is null;
...@@ -1720,7 +1838,7 @@ SELECT ts_lexize('thesaurus_astro','supernovae stars') is null; ...@@ -1720,7 +1838,7 @@ SELECT ts_lexize('thesaurus_astro','supernovae stars') is null;
The thesaurus dictionary <literal>thesaurus_astro</literal> does know The thesaurus dictionary <literal>thesaurus_astro</literal> does know
<literal>supernovae stars</literal>, but <function>ts_lexize</> fails since it <literal>supernovae stars</literal>, but <function>ts_lexize</> fails since it
does not parse the input text and considers it as a single lexeme. Use does not parse the input text and considers it as a single token. Use
<function>plainto_tsquery</> and <function>to_tsvector</> to test thesaurus <function>plainto_tsquery</> and <function>to_tsvector</> to test thesaurus
dictionaries: dictionaries:
...@@ -1745,36 +1863,26 @@ SELECT plainto_tsquery('supernovae stars'); ...@@ -1745,36 +1863,26 @@ SELECT plainto_tsquery('supernovae stars');
<para> <para>
A text search configuration specifies all options necessary to transform a A text search configuration specifies all options necessary to transform a
document into a <type>tsvector</type>: the parser breaks text into tokens, document into a <type>tsvector</type>: the parser to use to break text
and the dictionaries transform each token into a lexeme. Every call to into tokens, and the dictionaries to use to transform each token into a
<function>to_tsvector()</function> and <function>to_tsquery()</function> lexeme. Every call of
needs a configuration to perform its processing. To facilitate management <function>to_tsvector()</function> or <function>to_tsquery()</function>
of text search objects, a set of <acronym>SQL</acronym> commands needs a text search configuration to perform its processing.
is available, and there are several psql commands that display information
about text search objects (<xref linkend="textsearch-psql">).
</para>
<para>
The configuration parameter The configuration parameter
<xref linkend="guc-default-text-search-config"> <xref linkend="guc-default-text-search-config">
specifies the name of the current default configuration, which is the specifies the name of the current default configuration, which is the
one used by text search functions when an explicit configuration one used by text search functions if an explicit configuration
parameter is omitted. parameter is omitted.
It can be set in <filename>postgresql.conf</filename>, or set for an It can be set in <filename>postgresql.conf</filename>, or set for an
individual session using the <command>SET</> command. individual session using the <command>SET</> command.
</para> </para>
<para> <para>
Several predefined text search configurations are available in the Several predefined text search configurations are available, and
<literal>pg_catalog</literal> schema. If you need a custom configuration you can create custom configurations easily. To facilitate management
you can create a new text search configuration and modify it using SQL of text search objects, a set of <acronym>SQL</acronym> commands
commands. is available, and there are several psql commands that display information
</para> about text search objects (<xref linkend="textsearch-psql">).
<para>
New text search objects are created in the current schema by default
(usually the <literal>public</literal> schema), but a schema-qualified
name can be used to create objects in the specified schema.
</para> </para>
<para> <para>
...@@ -1791,7 +1899,7 @@ CREATE TEXT SEARCH CONFIGURATION public.pg ( COPY = english ); ...@@ -1791,7 +1899,7 @@ CREATE TEXT SEARCH CONFIGURATION public.pg ( COPY = english );
<para> <para>
We will use a PostgreSQL-specific synonym list We will use a PostgreSQL-specific synonym list
and store it in <filename>share/tsearch_data/pg_dict.syn</filename>. and store it in <filename>$SHAREDIR/tsearch_data/pg_dict.syn</filename>.
The file contents look like: The file contents look like:
<programlisting> <programlisting>
...@@ -1809,11 +1917,8 @@ CREATE TEXT SEARCH DICTIONARY pg_dict ( ...@@ -1809,11 +1917,8 @@ CREATE TEXT SEARCH DICTIONARY pg_dict (
); );
</programlisting> </programlisting>
</para> Next we register the <productname>ispell</> dictionary
<literal>english_ispell</literal>:
<para>
Then register the <productname>ispell</> dictionary
<literal>english_ispell</literal> using the <literal>ispell</literal> template:
<programlisting> <programlisting>
CREATE TEXT SEARCH DICTIONARY english_ispell ( CREATE TEXT SEARCH DICTIONARY english_ispell (
...@@ -1823,20 +1928,16 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( ...@@ -1823,20 +1928,16 @@ CREATE TEXT SEARCH DICTIONARY english_ispell (
StopWords = english StopWords = english
); );
</programlisting> </programlisting>
</para>
<para> Now modify the mappings for Latin words for configuration <literal>pg</>:
Now modify mappings for Latin words for configuration <literal>pg</>:
<programlisting> <programlisting>
ALTER TEXT SEARCH CONFIGURATION pg ALTER TEXT SEARCH CONFIGURATION pg
ALTER MAPPING FOR lword, lhword, lpart_hword ALTER MAPPING FOR lword, lhword, lpart_hword
WITH pg_dict, english_ispell, english_stem; WITH pg_dict, english_ispell, english_stem;
</programlisting> </programlisting>
</para>
<para> We do not index or search some token types:
We do not index or search some tokens:
<programlisting> <programlisting>
ALTER TEXT SEARCH CONFIGURATION pg ALTER TEXT SEARCH CONFIGURATION pg
...@@ -1894,8 +1995,10 @@ SHOW default_text_search_config; ...@@ -1894,8 +1995,10 @@ SHOW default_text_search_config;
<para> <para>
There are two kinds of indexes that can be used to speed up full text There are two kinds of indexes that can be used to speed up full text
operators (<xref linkend="textsearch-searches">). searches.
Note that indexes are not mandatory for full text searching. Note that indexes are not mandatory for full text searching, but in
cases where a column is searched on a regular basis, an index will
usually be desirable.
<variablelist> <variablelist>
...@@ -2072,7 +2175,7 @@ EXPLAIN SELECT * FROM apod WHERE textsearch @@@ to_tsquery('supernovae:a'); ...@@ -2072,7 +2175,7 @@ EXPLAIN SELECT * FROM apod WHERE textsearch @@@ to_tsquery('supernovae:a');
the indexes are faster for lookups. For dynamic data, GiST indexes are the indexes are faster for lookups. For dynamic data, GiST indexes are
faster to update. Specifically, <acronym>GiST</acronym> indexes are very faster to update. Specifically, <acronym>GiST</acronym> indexes are very
good for dynamic data and fast if the number of unique words (lexemes) is good for dynamic data and fast if the number of unique words (lexemes) is
under 100,000, while <acronym>GIN</acronym> handles +100,000 lexemes better under 100,000, while <acronym>GIN</acronym> handles 100,000+ lexemes better
but is slower to update. but is slower to update.
</para> </para>
...@@ -2228,12 +2331,12 @@ Parser: "pg_catalog.default" ...@@ -2228,12 +2331,12 @@ Parser: "pg_catalog.default"
=&gt; \dFp+ =&gt; \dFp+
Text search parser "pg_catalog.default" Text search parser "pg_catalog.default"
Method | Function | Description Method | Function | Description
------------------+----------------+------------- -----------------+----------------+-------------
Start parse | prsd_start | Start parse | prsd_start |
Get next token | prsd_nexttoken | Get next token | prsd_nexttoken |
End parse | prsd_end | End parse | prsd_end |
Get headline | prsd_headline | Get headline | prsd_headline |
Get lexeme types | prsd_lextype | Get token types | prsd_lextype |
Token types for parser "pg_catalog.default" Token types for parser "pg_catalog.default"
Token name | Description Token name | Description
...@@ -2303,22 +2406,22 @@ Parser: "pg_catalog.default" ...@@ -2303,22 +2406,22 @@ Parser: "pg_catalog.default"
text search features are: text search features are:
<itemizedlist spacing="compact" mark="bullet"> <itemizedlist spacing="compact" mark="bullet">
<listitem> <listitem>
<para>The length of each lexeme must be less than 2K bytes </para> <para>The length of each lexeme must be less than 2K bytes</para>
</listitem> </listitem>
<listitem> <listitem>
<para>The length of a <type>tsvector</type> (lexemes + positions) must be less than 1 megabyte </para> <para>The length of a <type>tsvector</type> (lexemes + positions) must be less than 1 megabyte</para>
</listitem> </listitem>
<listitem> <listitem>
<para>The number of lexemes must be less than 2<superscript>64</superscript> </para> <para>The number of lexemes must be less than 2<superscript>64</superscript></para>
</listitem> </listitem>
<listitem> <listitem>
<para>Positional information must be greater than 0 and less than 16,383 </para> <para>Positional information must be greater than 0 and less than 16,383</para>
</listitem> </listitem>
<listitem> <listitem>
<para>No more than 256 positions per lexeme </para> <para>No more than 256 positions per lexeme</para>
</listitem> </listitem>
<listitem> <listitem>
<para>The number of nodes (lexemes + operations) in tsquery must be less than 32,768 </para> <para>The number of nodes (lexemes + operations) in a <type>tsquery</type> must be less than 32,768</para>
</listitem> </listitem>
</itemizedlist> </itemizedlist>
</para> </para>
...@@ -2406,15 +2509,16 @@ SELECT * FROM ts_debug('public.english','The Brightest supernovaes'); ...@@ -2406,15 +2509,16 @@ SELECT * FROM ts_debug('public.english','The Brightest supernovaes');
<para> <para>
In this example, the word <literal>Brightest</> was recognized by the In this example, the word <literal>Brightest</> was recognized by the
parser as a <literal>Latin word</literal> (alias <literal>lword</literal>). parser as a <literal>Latin word</literal> (alias <literal>lword</literal>).
For this token type the dictionary stack is For this token type the dictionary list is
<literal>public.english_ispell</> and <literal>public.english_ispell</> and
<literal>pg_catalog.english_stem</literal>. The word was recognized by <literal>pg_catalog.english_stem</literal>. The word was recognized by
<literal>public.english_ispell</literal>, which reduced it to the noun <literal>public.english_ispell</literal>, which reduced it to the noun
<literal>bright</literal>. The word <literal>supernovaes</literal> is unknown <literal>bright</literal>. The word <literal>supernovaes</literal> is unknown
to the <literal>public.english_ispell</literal> dictionary so it was passed to to the <literal>public.english_ispell</literal> dictionary so it was passed to
the next dictionary, and, fortunately, was recognized (in fact, the next dictionary, and, fortunately, was recognized (in fact,
<literal>public.english_stem</literal> is a stemming dictionary and recognizes <literal>public.english_stem</literal> is a Snowball dictionary which
everything; that is why it was placed at the end of the dictionary stack). recognizes everything; that is why it was placed at the end of the
dictionary list).
</para> </para>
<para> <para>
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment