Fix strange behavior (and possible crashes) in full text phrase search.

In an attempt to simplify the tsquery matching engine, the original
phrase search patch invented rewrite rules that would rearrange a
tsquery so that no AND/OR/NOT operator appeared below a PHRASE operator.
But this approach had numerous problems.  The rearrangement step was
missed by ts_rewrite (and perhaps other places), allowing tsqueries
to be created that would cause Assert failures or perhaps crashes at
execution, as reported by Andreas Seltenreich.  The rewrite rules
effectively defined semantics for operators underneath PHRASE that were
buggy, or at least unintuitive.  And because rewriting was done in
tsqueryin() rather than at execution, the rearrangement was user-visible,
which is not very desirable --- for example, it might cause unexpected
matches or failures to match in ts_rewrite.

As a somewhat independent problem, the behavior of nested PHRASE operators
was only sane for left-deep trees; queries like "x <-> (y <-> z)" did not
behave intuitively at all.

To fix, get rid of the rewrite logic altogether, and instead teach the
tsquery execution engine to manage AND/OR/NOT below a PHRASE operator
by explicitly computing the match location(s) and match widths for these
operators.

This requires introducing some additional fields into the publicly visible
ExecPhraseData struct; but since there's no way for third-party code to
pass such a struct to TS_phrase_execute, it shouldn't create an ABI problem
as long as we don't move the offsets of the existing fields.

Another related problem was that index searches supposed that "!x <-> y"
could be lossily approximated as "!x & y", which isn't correct because
the latter will reject, say, "x q y" which the query itself accepts.
This required some tweaking in TS_execute_ternary along with the main
tsquery engine.

Back-patch to 9.6 where phrase operators were introduced.  While this
could be argued to change behavior more than we'd like in a stable branch,
we have to do something about the crash hazards and index-vs-seqscan
inconsistency, and it doesn't seem desirable to let the unintuitive
behaviors induced by the rewriting implementation stand as precedent.

Discussion: https://postgr.es/m/28215.1481999808@sss.pgh.pa.us
Discussion: https://postgr.es/m/26706.1482087250@sss.pgh.pa.us

doc/src/sgml/datatype.sgml

@@ -3959,15 +3959,7 @@ SELECT 'fat & rat & ! cat'::tsquery;
         tsquery         
 ------------------------
  'fat' & 'rat' & !'cat'
-
-SELECT '(fat | rat) <-> cat'::tsquery;
-              tsquery
------------------------------------
- 'fat' <-> 'cat' | 'rat' <-> 'cat'
 </programlisting>
-
-     The last example demonstrates that <type>tsquery</type> sometimes
-     rearranges nested operators into a logically equivalent formulation.
     </para>
 
     <para>

doc/src/sgml/textsearch.sgml

@@ -264,7 +264,7 @@ SELECT 'fat & cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::t
     text, any more than a <type>tsvector</type> is.  A <type>tsquery</type>
     contains search terms, which must be already-normalized lexemes, and
     may combine multiple terms using AND, OR, NOT, and FOLLOWED BY operators.
-    (For details see <xref linkend="datatype-tsquery">.)  There are
+    (For syntax details see <xref linkend="datatype-tsquery">.)  There are
     functions <function>to_tsquery</>, <function>plainto_tsquery</>,
     and <function>phraseto_tsquery</>
     that are helpful in converting user-written text into a proper
@@ -323,6 +323,8 @@ text @@ text
     at least one of its arguments must appear, while the <literal>!</> (NOT)
     operator specifies that its argument must <emphasis>not</> appear in
     order to have a match.
+    For example, the query <literal>fat &amp; ! rat</> matches documents that
+    contain <literal>fat</> but not <literal>rat</>.
    </para>
 
    <para>
@@ -377,6 +379,28 @@ SELECT phraseto_tsquery('the cats ate the rats');
     then <literal>&amp;</literal>, then <literal>&lt;-&gt;</literal>,
     and <literal>!</literal> most tightly.
    </para>
+
+   <para>
+    It's worth noticing that the AND/OR/NOT operators mean something subtly
+    different when they are within the arguments of a FOLLOWED BY operator
+    than when they are not, because within FOLLOWED BY the exact position of
+    the match is significant.  For example, normally <literal>!x</> matches
+    only documents that do not contain <literal>x</> anywhere.
+    But <literal>!x &lt;-&gt; y</> matches <literal>y</> if it is not
+    immediately after an <literal>x</>; an occurrence of <literal>x</>
+    elsewhere in the document does not prevent a match.  Another example is
+    that <literal>x &amp; y</> normally only requires that <literal>x</>
+    and <literal>y</> both appear somewhere in the document, but
+    <literal>(x &amp; y) &lt;-&gt; z</> requires <literal>x</>
+    and <literal>y</> to match at the same place, immediately before
+    a <literal>z</>.  Thus this query behaves differently from
+    <literal>x &lt;-&gt; z &amp; y &lt;-&gt; z</>, which will match a
+    document containing two separate sequences <literal>x z</> and
+    <literal>y z</>.  (This specific query is useless as written,
+    since <literal>x</> and <literal>y</> could not match at the same place;
+    but with more complex situations such as prefix-match patterns, a query
+    of this form could be useful.)
+   </para>
   </sect2>
 
   <sect2 id="textsearch-intro-configurations">

src/backend/utils/adt/tsginidx.c

@@ -212,7 +212,7 @@ checkcondition_gin(void *checkval, QueryOperand *val, ExecPhraseData *data)
  * Evaluate tsquery boolean expression using ternary logic.
  */
 static GinTernaryValue
-TS_execute_ternary(GinChkVal *gcv, QueryItem *curitem)
+TS_execute_ternary(GinChkVal *gcv, QueryItem *curitem, bool in_phrase)
 {
 	GinTernaryValue val1,
 				val2,
@@ -230,7 +230,10 @@ TS_execute_ternary(GinChkVal *gcv, QueryItem *curitem)
 	switch (curitem->qoperator.oper)
 	{
 		case OP_NOT:
-			result = TS_execute_ternary(gcv, curitem + 1);
+			/* In phrase search, always return MAYBE since we lack positions */
+			if (in_phrase)
+				return GIN_MAYBE;
+			result = TS_execute_ternary(gcv, curitem + 1, in_phrase);
 			if (result == GIN_MAYBE)
 				return result;
 			return !result;
@@ -238,17 +241,21 @@ TS_execute_ternary(GinChkVal *gcv, QueryItem *curitem)
 		case OP_PHRASE:
 
 			/*
-			 * GIN doesn't contain any information about positions, treat
+			 * GIN doesn't contain any information about positions, so treat
 			 * OP_PHRASE as OP_AND with recheck requirement
 			 */
-			*gcv->need_recheck = true;
+			*(gcv->need_recheck) = true;
+			/* Pass down in_phrase == true in case there's a NOT below */
+			in_phrase = true;
+
 			/* FALL THRU */
 
 		case OP_AND:
-			val1 = TS_execute_ternary(gcv, curitem + curitem->qoperator.left);
+			val1 = TS_execute_ternary(gcv, curitem + curitem->qoperator.left,
+									  in_phrase);
 			if (val1 == GIN_FALSE)
 				return GIN_FALSE;
-			val2 = TS_execute_ternary(gcv, curitem + 1);
+			val2 = TS_execute_ternary(gcv, curitem + 1, in_phrase);
 			if (val2 == GIN_FALSE)
 				return GIN_FALSE;
 			if (val1 == GIN_TRUE && val2 == GIN_TRUE)
@@ -257,10 +264,11 @@ TS_execute_ternary(GinChkVal *gcv, QueryItem *curitem)
 				return GIN_MAYBE;
 
 		case OP_OR:
-			val1 = TS_execute_ternary(gcv, curitem + curitem->qoperator.left);
+			val1 = TS_execute_ternary(gcv, curitem + curitem->qoperator.left,
+									  in_phrase);
 			if (val1 == GIN_TRUE)
 				return GIN_TRUE;
-			val2 = TS_execute_ternary(gcv, curitem + 1);
+			val2 = TS_execute_ternary(gcv, curitem + 1, in_phrase);
 			if (val2 == GIN_TRUE)
 				return GIN_TRUE;
 			if (val1 == GIN_FALSE && val2 == GIN_FALSE)
@@ -307,7 +315,7 @@ gin_tsquery_consistent(PG_FUNCTION_ARGS)
 
 		res = TS_execute(GETQUERY(query),
 						 &gcv,
-						 TS_EXEC_CALC_NOT | TS_EXEC_PHRASE_AS_AND,
+						 TS_EXEC_CALC_NOT | TS_EXEC_PHRASE_NO_POS,
 						 checkcondition_gin);
 	}
 
@@ -343,7 +351,7 @@ gin_tsquery_triconsistent(PG_FUNCTION_ARGS)
 		gcv.map_item_operand = (int *) (extra_data[0]);
 		gcv.need_recheck = &recheck;
 
-		res = TS_execute_ternary(&gcv, GETQUERY(query));
+		res = TS_execute_ternary(&gcv, GETQUERY(query), false);
 
 		if (res == GIN_TRUE && recheck)
 			res = GIN_MAYBE;

src/backend/utils/adt/tsgistidx.c

@@ -359,12 +359,11 @@ gtsvector_consistent(PG_FUNCTION_ARGS)
 		if (ISALLTRUE(key))
 			PG_RETURN_BOOL(true);
 
-		PG_RETURN_BOOL(TS_execute(
-								  GETQUERY(query),
+		/* since signature is lossy, cannot specify CALC_NOT here */
+		PG_RETURN_BOOL(TS_execute(GETQUERY(query),
 								  (void *) GETSIGN(key),
-								  TS_EXEC_PHRASE_AS_AND,
-								  checkcondition_bit
-								  ));
+								  TS_EXEC_PHRASE_NO_POS,
+								  checkcondition_bit));
 	}
 	else
 	{							/* only leaf pages */
@@ -372,12 +371,10 @@ gtsvector_consistent(PG_FUNCTION_ARGS)
 
 		chkval.arrb = GETARR(key);
 		chkval.arre = chkval.arrb + ARRNELEM(key);
-		PG_RETURN_BOOL(TS_execute(
-								  GETQUERY(query),
+		PG_RETURN_BOOL(TS_execute(GETQUERY(query),
 								  (void *) &chkval,
-								  TS_EXEC_PHRASE_AS_AND | TS_EXEC_CALC_NOT,
-								  checkcondition_arr
-								  ));
+								  TS_EXEC_PHRASE_NO_POS | TS_EXEC_CALC_NOT,
+								  checkcondition_arr));
 	}
 }
 

src/backend/utils/adt/tsquery.c

@@ -557,13 +557,11 @@ findoprnd_recurse(QueryItem *ptr, uint32 *pos, int nnodes, bool *needcleanup)
 				   curitem->oper == OP_OR ||
 				   curitem->oper == OP_PHRASE);
 
-			if (curitem->oper == OP_PHRASE)
-				*needcleanup = true;	/* push OP_PHRASE down later */
-
 			(*pos)++;
 
 			/* process RIGHT argument */
 			findoprnd_recurse(ptr, pos, nnodes, needcleanup);
+
 			curitem->left = *pos - tmp; /* set LEFT arg's offset */
 
 			/* process LEFT argument */
@@ -574,8 +572,9 @@ findoprnd_recurse(QueryItem *ptr, uint32 *pos, int nnodes, bool *needcleanup)
 
 
 /*
- * Fills in the left-fields previously left unfilled. The input
- * QueryItems must be in polish (prefix) notation.
+ * Fill in the left-fields previously left unfilled.
+ * The input QueryItems must be in polish (prefix) notation.
+ * Also, set *needcleanup to true if there are any QI_VALSTOP nodes.
  */
 static void
 findoprnd(QueryItem *ptr, int size, bool *needcleanup)
@@ -687,15 +686,17 @@ parse_tsquery(char *buf,
 	memcpy((void *) GETOPERAND(query), (void *) state.op, state.sumlen);
 	pfree(state.op);
 
-	/* Set left operand pointers for every operator. */
+	/*
+	 * Set left operand pointers for every operator.  While we're at it,
+	 * detect whether there are any QI_VALSTOP nodes.
+	 */
 	findoprnd(ptr, query->size, &needcleanup);
 
 	/*
-	 * QI_VALSTOP nodes should be cleaned and OP_PHRASE should be pushed
-	 * down
+	 * If there are QI_VALSTOP nodes, delete them and simplify the tree.
 	 */
 	if (needcleanup)
-		return cleanup_fakeval_and_phrase(query);
+		query = cleanup_tsquery_stopwords(query);
 
 	return query;
 }
@@ -1088,6 +1089,9 @@ tsqueryrecv(PG_FUNCTION_ARGS)
 	 */
 	findoprnd(item, size, &needcleanup);
 
+	/* Can't have found any QI_VALSTOP nodes */
+	Assert(!needcleanup);
+
 	/* Copy operands to output struct */
 	for (i = 0; i < size; i++)
 	{
@@ -1105,9 +1109,6 @@ tsqueryrecv(PG_FUNCTION_ARGS)
 
 	SET_VARSIZE(query, len + datalen);
 
-	if (needcleanup)
-		PG_RETURN_TSQUERY(cleanup_fakeval_and_phrase(query));
-
 	PG_RETURN_TSQUERY(query);
 }
 

src/backend/utils/adt/tsquery_cleanup.c

@@ -25,19 +25,6 @@ typedef struct NODE
 	QueryItem  *valnode;
 } NODE;
 
-/*
- * To simplify walking on query tree and pushing down of phrase operator
- * we define some fake priority here: phrase operator has highest priority
- * of any other operators (and we believe here that OP_PHRASE is a highest
- * code of operations) and value node has ever highest priority.
- * Priority values of other operations don't matter until they are less than
- * phrase operator and value node.
- */
-#define VALUE_PRIORITY			(OP_COUNT + 1)
-#define NODE_PRIORITY(x) \
-	( ((x)->valnode->qoperator.type == QI_OPR) ? \
-		(x)->valnode->qoperator.oper : VALUE_PRIORITY )
-
 /*
  * make query tree from plain view of query
  */
@@ -368,227 +355,6 @@ clean_stopword_intree(NODE *node, int *ladd, int *radd)
 	return node;
 }
 
-static NODE *
-copyNODE(NODE *node)
-{
-	NODE	   *cnode = palloc(sizeof(NODE));
-
-	/* since this function recurses, it could be driven to stack overflow. */
-	check_stack_depth();
-
-	cnode->valnode = palloc(sizeof(QueryItem));
-	*(cnode->valnode) = *(node->valnode);
-
-	if (node->valnode->type == QI_OPR)
-	{
-		cnode->right = copyNODE(node->right);
-		if (node->valnode->qoperator.oper != OP_NOT)
-			cnode->left = copyNODE(node->left);
-	}
-
-	return cnode;
-}
-
-static NODE *
-makeNODE(int8 op, NODE *left, NODE *right)
-{
-	NODE	   *node = palloc(sizeof(NODE));
-
-	/* zeroing allocation to prevent difference in unused bytes */
-	node->valnode = palloc0(sizeof(QueryItem));
-
-	node->valnode->qoperator.type = QI_OPR;
-	node->valnode->qoperator.oper = op;
-
-	node->left = left;
-	node->right = right;
-
-	return node;
-}
-
-/*
- * Move operation with high priority to the leaves. This guarantees
- * that the phrase operator will be near the bottom of the tree.
- * An idea behind is do not store position of lexemes during execution
- * of ordinary operations (AND, OR, NOT) because it could be expensive.
- * Actual transformation will be performed only on subtrees under the
- * <-> (<n>) operation since it's needed solely for the phrase operator.
- *
- * Rules:
- *	  a  <->  (b | c)	=>	(a <-> b)  |   (a <-> c)
- *	 (a | b)  <->	 c	   =>	(a <-> c)  |   (b <-> c)
- *	  a  <->	!b	   =>		a	  &  !(a <-> b)
- *	 !a  <->	 b	   =>		b	  &  !(a <-> b)
- *
- * Warnings for readers:
- *		  a <-> b	   !=	   b <-> a
- *
- *	  a <n> (b <n> c)	!=	 (a <n> b) <n> c since the phrase lengths are:
- *			 n					2n-1
- */
-static NODE *
-normalize_phrase_tree(NODE *node)
-{
-	/* there should be no stop words at this point */
-	Assert(node->valnode->type != QI_VALSTOP);
-
-	if (node->valnode->type == QI_VAL)
-		return node;
-
-	/* since this function recurses, it could be driven to stack overflow. */
-	check_stack_depth();
-
-	Assert(node->valnode->type == QI_OPR);
-
-	if (node->valnode->qoperator.oper == OP_NOT)
-	{
-		NODE	   *orignode = node;
-
-		/* eliminate NOT sequence */
-		while (node->valnode->type == QI_OPR &&
-		node->valnode->qoperator.oper == node->right->valnode->qoperator.oper)
-		{
-			node = node->right->right;
-		}
-
-		if (orignode != node)
-			/* current node isn't checked yet */
-			node = normalize_phrase_tree(node);
-		else
-			node->right = normalize_phrase_tree(node->right);
-	}
-	else if (node->valnode->qoperator.oper == OP_PHRASE)
-	{
-		int16		distance;
-		NODE	   *X;
-
-		node->left = normalize_phrase_tree(node->left);
-		node->right = normalize_phrase_tree(node->right);
-
-		/*
-		 * if subtree contains only nodes with higher "priority" then we are
-		 * done. See comment near NODE_PRIORITY()
-		 */
-		if (NODE_PRIORITY(node) <= NODE_PRIORITY(node->right) &&
-			NODE_PRIORITY(node) <= NODE_PRIORITY(node->left))
-			return node;
-
-		/*
-		 * We can't swap left-right and works only with left child because of
-		 * a <-> b	!=	b <-> a
-		 */
-
-		distance = node->valnode->qoperator.distance;
-
-		if (node->right->valnode->type == QI_OPR)
-		{
-			switch (node->right->valnode->qoperator.oper)
-			{
-				case OP_AND:
-					/* a <-> (b & c)  =>  (a <-> b) & (a <-> c) */
-					node = makeNODE(OP_AND,
-									makeNODE(OP_PHRASE,
-											 node->left,
-											 node->right->left),
-									makeNODE(OP_PHRASE,
-											 copyNODE(node->left),
-											 node->right->right));
-					node->left->valnode->qoperator.distance =
-						node->right->valnode->qoperator.distance = distance;
-					break;
-				case OP_OR:
-					/* a <-> (b | c)  =>  (a <-> b) | (a <-> c) */
-					node = makeNODE(OP_OR,
-									makeNODE(OP_PHRASE,
-											 node->left,
-											 node->right->left),
-									makeNODE(OP_PHRASE,
-											 copyNODE(node->left),
-											 node->right->right));
-					node->left->valnode->qoperator.distance =
-						node->right->valnode->qoperator.distance = distance;
-					break;
-				case OP_NOT:
-					/* a <-> !b  =>  a & !(a <-> b) */
-					X = node->right;
-					node->right = node->right->right;
-					X->right = node;
-					node = makeNODE(OP_AND,
-									copyNODE(node->left),
-									X);
-					break;
-				case OP_PHRASE:
-					/* no-op */
-					break;
-				default:
-					elog(ERROR, "Wrong type of tsquery node: %d",
-						 node->right->valnode->qoperator.oper);
-			}
-		}
-
-		if (node->left->valnode->type == QI_OPR &&
-			node->valnode->qoperator.oper == OP_PHRASE)
-		{
-			/*
-			 * if the node is still OP_PHRASE, check the left subtree,
-			 * otherwise the whole node will be transformed later.
-			 */
-			switch (node->left->valnode->qoperator.oper)
-			{
-				case OP_AND:
-					/* (a & b) <-> c  =>  (a <-> c) & (b <-> c) */
-					node = makeNODE(OP_AND,
-									makeNODE(OP_PHRASE,
-											 node->left->left,
-											 node->right),
-									makeNODE(OP_PHRASE,
-											 node->left->right,
-											 copyNODE(node->right)));
-					node->left->valnode->qoperator.distance =
-						node->right->valnode->qoperator.distance = distance;
-					break;
-				case OP_OR:
-					/* (a | b) <-> c  =>  (a <-> c) | (b <-> c) */
-					node = makeNODE(OP_OR,
-									makeNODE(OP_PHRASE,
-											 node->left->left,
-											 node->right),
-									makeNODE(OP_PHRASE,
-											 node->left->right,
-											 copyNODE(node->right)));
-					node->left->valnode->qoperator.distance =
-						node->right->valnode->qoperator.distance = distance;
-					break;
-				case OP_NOT:
-					/* !a <-> b  =>  b & !(a <-> b) */
-					X = node->left;
-					node->left = node->left->right;
-					X->right = node;
-					node = makeNODE(OP_AND,
-									X,
-									copyNODE(node->right));
-					break;
-				case OP_PHRASE:
-					/* no-op */
-					break;
-				default:
-					elog(ERROR, "Wrong type of tsquery node: %d",
-						 node->left->valnode->qoperator.oper);
-			}
-		}
-
-		/* continue transformation */
-		node = normalize_phrase_tree(node);
-	}
-	else	/* AND or OR */
-	{
-		node->left = normalize_phrase_tree(node->left);
-		node->right = normalize_phrase_tree(node->right);
-	}
-
-	return node;
-}
-
 /*
  * Number of elements in query tree
  */
@@ -613,8 +379,11 @@ calcstrlen(NODE *node)
 	return size;
 }
 
+/*
+ * Remove QI_VALSTOP (stopword) nodes from TSQuery.
+ */
 TSQuery
-cleanup_fakeval_and_phrase(TSQuery in)
+cleanup_tsquery_stopwords(TSQuery in)
 {
 	int32		len,
 				lenstr,
@@ -642,9 +411,6 @@ cleanup_fakeval_and_phrase(TSQuery in)
 		return out;
 	}
 
-	/* push OP_PHRASE nodes down */
-	root = normalize_phrase_tree(root);
-
 	/*
 	 * Build TSQuery from plain view
 	 */

src/backend/utils/adt/tsquery_op.c

@@ -104,7 +104,7 @@ tsquery_or(PG_FUNCTION_ARGS)
 	PG_FREE_IF_COPY(a, 0);
 	PG_FREE_IF_COPY(b, 1);
 
-	PG_RETURN_POINTER(query);
+	PG_RETURN_TSQUERY(query);
 }
 
 Datum
@@ -140,7 +140,7 @@ tsquery_phrase_distance(PG_FUNCTION_ARGS)
 	PG_FREE_IF_COPY(a, 0);
 	PG_FREE_IF_COPY(b, 1);
 
-	PG_RETURN_POINTER(cleanup_fakeval_and_phrase(query));
+	PG_RETURN_TSQUERY(query);
 }
 
 Datum

src/include/tsearch/ts_utils.h

@@ -113,8 +113,8 @@ extern text *generateHeadline(HeadlineParsedText *prs);
  * struct ExecPhraseData is passed to a TSExecuteCallback function if we need
  * lexeme position data (because of a phrase-match operator in the tsquery).
  * The callback should fill in position data when it returns true (success).
- * If it cannot return position data, it may ignore its "data" argument, but
- * then the caller of TS_execute() must pass the TS_EXEC_PHRASE_AS_AND flag
+ * If it cannot return position data, it may leave "data" unchanged, but
+ * then the caller of TS_execute() must pass the TS_EXEC_PHRASE_NO_POS flag
  * and must arrange for a later recheck with position data available.
  *
  * The reported lexeme positions must be sorted and unique.  Callers must only
@@ -123,13 +123,21 @@ extern text *generateHeadline(HeadlineParsedText *prs);
  * portion of a tsvector value.  If "allocated" is true then the pos array
  * is palloc'd workspace and caller may free it when done.
  *
+ * "negate" means that the pos array contains positions where the query does
+ * not match, rather than positions where it does.  "width" is positive when
+ * the match is wider than one lexeme.  Neither of these fields normally need
+ * to be touched by TSExecuteCallback functions; they are used for
+ * phrase-search processing within TS_execute.
+ *
  * All fields of the ExecPhraseData struct are initially zeroed by caller.
  */
 typedef struct ExecPhraseData
 {
 	int			npos;			/* number of positions reported */
 	bool		allocated;		/* pos points to palloc'd data? */
+	bool		negate;			/* positions are where query is NOT matched */
 	WordEntryPos *pos;			/* ordered, non-duplicate lexeme positions */
+	int			width;			/* width of match in lexemes, less 1 */
 } ExecPhraseData;
 
 /*
@@ -139,7 +147,9 @@ typedef struct ExecPhraseData
  * val: lexeme to test for presence of
  * data: to be filled with lexeme positions; NULL if position data not needed
  *
- * Return TRUE if lexeme is present in data, else FALSE
+ * Return TRUE if lexeme is present in data, else FALSE.  If data is not
+ * NULL, it should be filled with lexeme positions, but function can leave
+ * it as zeroes if position data is not available.
  */
 typedef bool (*TSExecuteCallback) (void *arg, QueryOperand *val,
 											   ExecPhraseData *data);
@@ -151,15 +161,18 @@ typedef bool (*TSExecuteCallback) (void *arg, QueryOperand *val,
 /*
  * If TS_EXEC_CALC_NOT is not set, then NOT expressions are automatically
  * evaluated to be true.  Useful in cases where NOT cannot be accurately
- * computed (GiST) or it isn't important (ranking).
+ * computed (GiST) or it isn't important (ranking).  From TS_execute's
+ * perspective, !CALC_NOT means that the TSExecuteCallback function might
+ * return false-positive indications of a lexeme's presence.
  */
 #define TS_EXEC_CALC_NOT		(0x01)
 /*
- * Treat OP_PHRASE as OP_AND.  Used when positional information is not
- * accessible, like in consistent methods of GIN/GiST indexes; rechecking
- * must occur later.
+ * If TS_EXEC_PHRASE_NO_POS is set, allow OP_PHRASE to be executed lossily
+ * in the absence of position information: a TRUE result indicates that the
+ * phrase might be present.  Without this flag, OP_PHRASE always returns
+ * false if lexeme position information is not available.
  */
-#define TS_EXEC_PHRASE_AS_AND	(0x02)
+#define TS_EXEC_PHRASE_NO_POS	(0x02)
 
 extern bool TS_execute(QueryItem *curitem, void *arg, uint32 flags,
 		   TSExecuteCallback chkcond);
@@ -228,7 +241,7 @@ extern Datum gin_tsquery_consistent_oldsig(PG_FUNCTION_ARGS);
  * TSQuery Utilities
  */
 extern QueryItem *clean_NOT(QueryItem *ptr, int32 *len);
-extern TSQuery cleanup_fakeval_and_phrase(TSQuery in);
+extern TSQuery cleanup_tsquery_stopwords(TSQuery in);
 
 typedef struct QTNode
 {

src/test/regress/expected/tsdicts.out

@@ -470,15 +470,15 @@ SELECT to_tsquery('hunspell_tst', 'footballyklubber:b & rebookings:A & sky');
 (1 row)
 
 SELECT to_tsquery('hunspell_tst', 'footballyklubber:b <-> sky');
-                           to_tsquery                            
------------------------------------------------------------------
- 'foot':B <-> 'sky' & 'ball':B <-> 'sky' & 'klubber':B <-> 'sky'
+                   to_tsquery                    
+-------------------------------------------------
+ ( 'foot':B & 'ball':B & 'klubber':B ) <-> 'sky'
 (1 row)
 
 SELECT phraseto_tsquery('hunspell_tst', 'footballyklubber sky');
-                     phraseto_tsquery                      
------------------------------------------------------------
- 'foot' <-> 'sky' & 'ball' <-> 'sky' & 'klubber' <-> 'sky'
+             phraseto_tsquery              
+-------------------------------------------
+ ( 'foot' & 'ball' & 'klubber' ) <-> 'sky'
 (1 row)
 
 -- Test ispell dictionary with hunspell affix with FLAG long in configuration

src/test/regress/expected/tsearch.out

@@ -556,15 +556,15 @@ SELECT plainto_tsquery('english', 'foo bar') && 'asd | fg';
 
 -- Check stop word deletion, a and s are stop-words
 SELECT to_tsquery('english', '!(a & !b) & c');
- to_tsquery 
-------------
- 'b' & 'c'
+ to_tsquery  
+-------------
+ !!'b' & 'c'
 (1 row)
 
 SELECT to_tsquery('english', '!(a & !b)');
  to_tsquery 
 ------------
- 'b'
+ !!'b'
 (1 row)
 
 SELECT to_tsquery('english', '(1 <-> 2) <-> a');
@@ -1240,15 +1240,15 @@ SELECT ts_rewrite('1 & (2 <2> 3)', 'SELECT keyword, sample FROM test_tsquery'::t
 (1 row)
 
 SELECT ts_rewrite('5 <-> (1 & (2 <-> 3))', 'SELECT keyword, sample FROM test_tsquery'::text );
-              ts_rewrite               
----------------------------------------
- '5' <-> '1' & '5' <-> ( '2' <-> '3' )
+       ts_rewrite        
+-------------------------
+ '5' <-> ( '2' <-> '4' )
 (1 row)
 
 SELECT ts_rewrite('5 <-> (6 | 8)', 'SELECT keyword, sample FROM test_tsquery'::text );
-        ts_rewrite         
----------------------------
- '5' <-> '7' | '5' <-> '8'
+      ts_rewrite       
+-----------------------
+ '5' <-> ( '6' | '8' )
 (1 row)
 
 -- Check empty substitution
@@ -1386,6 +1386,26 @@ SELECT ts_rewrite( query, 'SELECT keyword, sample FROM test_tsquery' ) FROM to_t
  'citi' & 'foo' & ( 'bar' | 'qq' ) & ( 'nyc' | 'big' & 'appl' | 'new' & 'york' )
 (1 row)
 
+SELECT ts_rewrite(tsquery_phrase('foo', 'foo'), 'foo', 'bar | baz');
+               ts_rewrite                
+-----------------------------------------
+ ( 'bar' | 'baz' ) <-> ( 'bar' | 'baz' )
+(1 row)
+
+SELECT to_tsvector('foo bar') @@
+  ts_rewrite(tsquery_phrase('foo', 'foo'), 'foo', 'bar | baz');
+ ?column? 
+----------
+ f
+(1 row)
+
+SELECT to_tsvector('bar baz') @@
+  ts_rewrite(tsquery_phrase('foo', 'foo'), 'foo', 'bar | baz');
+ ?column? 
+----------
+ t
+(1 row)
+
 RESET enable_seqscan;
 --test GUC
 SET default_text_search_config=simple;

src/test/regress/expected/tstypes.out

@@ -366,133 +366,6 @@ SELECT '!!a & !!b'::tsquery;
  !!'a' & !!'b'
 (1 row)
 
--- phrase transformation
-SELECT 'a <-> (b|c)'::tsquery;
-          tsquery          
----------------------------
- 'a' <-> 'b' | 'a' <-> 'c'
-(1 row)
-
-SELECT '(a|b) <-> c'::tsquery;
-          tsquery          
----------------------------
- 'a' <-> 'c' | 'b' <-> 'c'
-(1 row)
-
-SELECT '(a|b) <-> (d|c)'::tsquery;
-                        tsquery                        
--------------------------------------------------------
- 'a' <-> 'd' | 'b' <-> 'd' | 'a' <-> 'c' | 'b' <-> 'c'
-(1 row)
-
-SELECT 'a <-> (b&c)'::tsquery;
-          tsquery          
----------------------------
- 'a' <-> 'b' & 'a' <-> 'c'
-(1 row)
-
-SELECT '(a&b) <-> c'::tsquery;
-          tsquery          
----------------------------
- 'a' <-> 'c' & 'b' <-> 'c'
-(1 row)
-
-SELECT '(a&b) <-> (d&c)'::tsquery;
-                        tsquery                        
--------------------------------------------------------
- 'a' <-> 'd' & 'b' <-> 'd' & 'a' <-> 'c' & 'b' <-> 'c'
-(1 row)
-
-SELECT 'a <-> !b'::tsquery;
-        tsquery         
-------------------------
- 'a' & !( 'a' <-> 'b' )
-(1 row)
-
-SELECT '!a <-> b'::tsquery;
-        tsquery         
-------------------------
- !( 'a' <-> 'b' ) & 'b'
-(1 row)
-
-SELECT '!a <-> !b'::tsquery;
-              tsquery               
-------------------------------------
- !'a' & !( !( 'a' <-> 'b' ) & 'b' )
-(1 row)
-
-SELECT 'a <-> !(b&c)'::tsquery;
-               tsquery                
---------------------------------------
- 'a' & !( 'a' <-> 'b' & 'a' <-> 'c' )
-(1 row)
-
-SELECT 'a <-> !(b|c)'::tsquery;
-               tsquery                
---------------------------------------
- 'a' & !( 'a' <-> 'b' | 'a' <-> 'c' )
-(1 row)
-
-SELECT  '!(a&b) <-> c'::tsquery;
-               tsquery                
---------------------------------------
- !( 'a' <-> 'c' & 'b' <-> 'c' ) & 'c'
-(1 row)
-
-SELECT  '!(a|b) <-> c'::tsquery;
-               tsquery                
---------------------------------------
- !( 'a' <-> 'c' | 'b' <-> 'c' ) & 'c'
-(1 row)
-
-SELECT  '(!a|b) <-> c'::tsquery;
-               tsquery                
---------------------------------------
- !( 'a' <-> 'c' ) & 'c' | 'b' <-> 'c'
-(1 row)
-
-SELECT  '(!a&b) <-> c'::tsquery;
-               tsquery                
---------------------------------------
- !( 'a' <-> 'c' ) & 'c' & 'b' <-> 'c'
-(1 row)
-
-SELECT  'c <-> (!a|b)'::tsquery;
-               tsquery                
---------------------------------------
- 'c' & !( 'c' <-> 'a' ) | 'c' <-> 'b'
-(1 row)
-
-SELECT  'c <-> (!a&b)'::tsquery;
-               tsquery                
---------------------------------------
- 'c' & !( 'c' <-> 'a' ) & 'c' <-> 'b'
-(1 row)
-
-SELECT  '(a|b) <-> !c'::tsquery;
-                    tsquery                     
-------------------------------------------------
- ( 'a' | 'b' ) & !( 'a' <-> 'c' | 'b' <-> 'c' )
-(1 row)
-
-SELECT  '(a&b) <-> !c'::tsquery;
-                  tsquery                   
---------------------------------------------
- 'a' & 'b' & !( 'a' <-> 'c' & 'b' <-> 'c' )
-(1 row)
-
-SELECT  '!c <-> (a|b)'::tsquery;
-                     tsquery                     
--------------------------------------------------
- !( 'c' <-> 'a' ) & 'a' | !( 'c' <-> 'b' ) & 'b'
-(1 row)
-
-SELECT  '!c <-> (a&b)'::tsquery;
-                     tsquery                     
--------------------------------------------------
- !( 'c' <-> 'a' ) & 'a' & !( 'c' <-> 'b' ) & 'b'
-(1 row)
-
 --comparisons
 SELECT 'a' < 'b & c'::tsquery as "true";
  true 
@@ -568,33 +441,33 @@ SELECT 'foo & bar'::tsquery && 'asd | fg';
 (1 row)
 
 SELECT 'a' <-> 'b & d'::tsquery;
-         ?column?          
----------------------------
- 'a' <-> 'b' & 'a' <-> 'd'
+       ?column?        
+-----------------------
+ 'a' <-> ( 'b' & 'd' )
 (1 row)
 
 SELECT 'a & g' <-> 'b & d'::tsquery;
-                       ?column?                        
--------------------------------------------------------
- 'a' <-> 'b' & 'g' <-> 'b' & 'a' <-> 'd' & 'g' <-> 'd'
+            ?column?             
+---------------------------------
+ ( 'a' & 'g' ) <-> ( 'b' & 'd' )
 (1 row)
 
 SELECT 'a & g' <-> 'b | d'::tsquery;
-                       ?column?                        
--------------------------------------------------------
- 'a' <-> 'b' & 'g' <-> 'b' | 'a' <-> 'd' & 'g' <-> 'd'
+            ?column?             
+---------------------------------
+ ( 'a' & 'g' ) <-> ( 'b' | 'd' )
 (1 row)
 
 SELECT 'a & g' <-> 'b <-> d'::tsquery;
-                     ?column?                      
----------------------------------------------------
- 'a' <-> ( 'b' <-> 'd' ) & 'g' <-> ( 'b' <-> 'd' )
+             ?column?              
+-----------------------------------
+ ( 'a' & 'g' ) <-> ( 'b' <-> 'd' )
 (1 row)
 
 SELECT tsquery_phrase('a <3> g', 'b & d', 10);
-               tsquery_phrase                
----------------------------------------------
- 'a' <3> 'g' <10> 'b' & 'a' <3> 'g' <10> 'd'
+         tsquery_phrase         
+--------------------------------
+ 'a' <3> 'g' <10> ( 'b' & 'd' )
 (1 row)
 
 -- tsvector-tsquery operations
@@ -749,25 +622,152 @@ SELECT to_tsvector('simple', '1 2 3 4') @@ '(1 <-> 2) <-> 3' AS "true";
  t
 (1 row)
 
-SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <-> (2 <-> 3)' AS "false";
+SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <-> (2 <-> 3)' AS "true";
+ true 
+------
+ t
+(1 row)
+
+SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <2> (2 <-> 3)' AS "false";
  false 
 -------
  f
 (1 row)
 
-SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <2> (2 <-> 3)' AS "true";
+SELECT to_tsvector('simple', '1 2 1 2 3 4') @@ '(1 <-> 2) <-> 3' AS "true";
  true 
 ------
  t
 (1 row)
 
-SELECT to_tsvector('simple', '1 2 1 2 3 4') @@ '(1 <-> 2) <-> 3' AS "true";
+SELECT to_tsvector('simple', '1 2 1 2 3 4') @@ '1 <-> 2 <-> 3' AS "true";
  true 
 ------
  t
 (1 row)
 
-SELECT to_tsvector('simple', '1 2 1 2 3 4') @@ '1 <-> 2 <-> 3' AS "true";
+-- without position data, phrase search does not match
+SELECT strip(to_tsvector('simple', '1 2 3 4')) @@ '1 <-> 2 <-> 3' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'q x q y') @@ 'q <-> (x & y)' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'q x') @@ 'q <-> (x | y <-> z)' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'q y') @@ 'q <-> (x | y <-> z)' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'q y z') @@ 'q <-> (x | y <-> z)' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'q y x') @@ 'q <-> (x | y <-> z)' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'q x y') @@ 'q <-> (x | y <-> z)' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'q x') @@ '(x | y <-> z) <-> q' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'x q') @@ '(x | y <-> z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'x y q') @@ '(x | y <-> z) <-> q' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'x y z') @@ '(x | y <-> z) <-> q' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'x y z q') @@ '(x | y <-> z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'y z q') @@ '(x | y <-> z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'y y q') @@ '(x | y <-> z) <-> q' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'y y q') @@ '(!x | y <-> z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'x y q') @@ '(!x | y <-> z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'y y q') @@ '(x | y <-> !z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'x q') @@ '(x | y <-> !z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'x q') @@ '(!x | y <-> z) <-> q' AS "false";
+ false 
+-------
+ f
+(1 row)
+
+select to_tsvector('simple', 'z q') @@ '(!x | y <-> z) <-> q' AS "true";
+ true 
+------
+ t
+(1 row)
+
+select to_tsvector('simple', 'x y q y') @@ '!x <-> y' AS "true";
  true 
 ------
  t
@@ -1002,6 +1002,12 @@ SELECT 'a:1 b:3'::tsvector @@ 'a <3> b'::tsquery AS "false";
  f
 (1 row)
 
+SELECT 'a:1 b:3'::tsvector @@ 'a <0> a:*'::tsquery AS "true";
+ true 
+------
+ t
+(1 row)
+
 -- tsvector editing operations
 SELECT strip('w:12B w:13* w:12,5,6 a:1,3* a:3 w asd:1dc asd'::tsvector);
      strip     

src/test/regress/sql/tsearch.sql

@@ -447,6 +447,12 @@ SELECT ts_rewrite( query, 'SELECT keyword, sample FROM test_tsquery' ) FROM to_t
 SELECT ts_rewrite( query, 'SELECT keyword, sample FROM test_tsquery' ) FROM to_tsquery('english', 'moscow & hotel') AS query;
 SELECT ts_rewrite( query, 'SELECT keyword, sample FROM test_tsquery' ) FROM to_tsquery('english', 'bar &  new & qq & foo & york') AS query;
 
+SELECT ts_rewrite(tsquery_phrase('foo', 'foo'), 'foo', 'bar | baz');
+SELECT to_tsvector('foo bar') @@
+  ts_rewrite(tsquery_phrase('foo', 'foo'), 'foo', 'bar | baz');
+SELECT to_tsvector('bar baz') @@
+  ts_rewrite(tsquery_phrase('foo', 'foo'), 'foo', 'bar | baz');
+
 RESET enable_seqscan;
 
 --test GUC

src/test/regress/sql/tstypes.sql

@@ -64,34 +64,6 @@ SELECT 'a & !!b'::tsquery;
 SELECT '!!a & b'::tsquery;
 SELECT '!!a & !!b'::tsquery;
 
--- phrase transformation
-SELECT 'a <-> (b|c)'::tsquery;
-SELECT '(a|b) <-> c'::tsquery;
-SELECT '(a|b) <-> (d|c)'::tsquery;
-
-SELECT 'a <-> (b&c)'::tsquery;
-SELECT '(a&b) <-> c'::tsquery;
-SELECT '(a&b) <-> (d&c)'::tsquery;
-
-SELECT 'a <-> !b'::tsquery;
-SELECT '!a <-> b'::tsquery;
-SELECT '!a <-> !b'::tsquery;
-
-SELECT 'a <-> !(b&c)'::tsquery;
-SELECT 'a <-> !(b|c)'::tsquery;
-SELECT  '!(a&b) <-> c'::tsquery;
-SELECT  '!(a|b) <-> c'::tsquery;
-
-SELECT  '(!a|b) <-> c'::tsquery;
-SELECT  '(!a&b) <-> c'::tsquery;
-SELECT  'c <-> (!a|b)'::tsquery;
-SELECT  'c <-> (!a&b)'::tsquery;
-
-SELECT  '(a|b) <-> !c'::tsquery;
-SELECT  '(a&b) <-> !c'::tsquery;
-SELECT  '!c <-> (a|b)'::tsquery;
-SELECT  '!c <-> (a&b)'::tsquery;
-
 --comparisons
 SELECT 'a' < 'b & c'::tsquery as "true";
 SELECT 'a' > 'b & c'::tsquery as "false";
@@ -146,10 +118,33 @@ SELECT to_tsvector('simple', '1 2 11 3') @@ '1:* <-> 3' AS "true";
 
 SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <-> 2 <-> 3' AS "true";
 SELECT to_tsvector('simple', '1 2 3 4') @@ '(1 <-> 2) <-> 3' AS "true";
-SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <-> (2 <-> 3)' AS "false";
-SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <2> (2 <-> 3)' AS "true";
+SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <-> (2 <-> 3)' AS "true";
+SELECT to_tsvector('simple', '1 2 3 4') @@ '1 <2> (2 <-> 3)' AS "false";
 SELECT to_tsvector('simple', '1 2 1 2 3 4') @@ '(1 <-> 2) <-> 3' AS "true";
 SELECT to_tsvector('simple', '1 2 1 2 3 4') @@ '1 <-> 2 <-> 3' AS "true";
+-- without position data, phrase search does not match
+SELECT strip(to_tsvector('simple', '1 2 3 4')) @@ '1 <-> 2 <-> 3' AS "false";
+
+select to_tsvector('simple', 'q x q y') @@ 'q <-> (x & y)' AS "false";
+select to_tsvector('simple', 'q x') @@ 'q <-> (x | y <-> z)' AS "true";
+select to_tsvector('simple', 'q y') @@ 'q <-> (x | y <-> z)' AS "false";
+select to_tsvector('simple', 'q y z') @@ 'q <-> (x | y <-> z)' AS "true";
+select to_tsvector('simple', 'q y x') @@ 'q <-> (x | y <-> z)' AS "false";
+select to_tsvector('simple', 'q x y') @@ 'q <-> (x | y <-> z)' AS "true";
+select to_tsvector('simple', 'q x') @@ '(x | y <-> z) <-> q' AS "false";
+select to_tsvector('simple', 'x q') @@ '(x | y <-> z) <-> q' AS "true";
+select to_tsvector('simple', 'x y q') @@ '(x | y <-> z) <-> q' AS "false";
+select to_tsvector('simple', 'x y z') @@ '(x | y <-> z) <-> q' AS "false";
+select to_tsvector('simple', 'x y z q') @@ '(x | y <-> z) <-> q' AS "true";
+select to_tsvector('simple', 'y z q') @@ '(x | y <-> z) <-> q' AS "true";
+select to_tsvector('simple', 'y y q') @@ '(x | y <-> z) <-> q' AS "false";
+select to_tsvector('simple', 'y y q') @@ '(!x | y <-> z) <-> q' AS "true";
+select to_tsvector('simple', 'x y q') @@ '(!x | y <-> z) <-> q' AS "true";
+select to_tsvector('simple', 'y y q') @@ '(x | y <-> !z) <-> q' AS "true";
+select to_tsvector('simple', 'x q') @@ '(x | y <-> !z) <-> q' AS "true";
+select to_tsvector('simple', 'x q') @@ '(!x | y <-> z) <-> q' AS "false";
+select to_tsvector('simple', 'z q') @@ '(!x | y <-> z) <-> q' AS "true";
+select to_tsvector('simple', 'x y q y') @@ '!x <-> y' AS "true";
 
 --ranking
 SELECT ts_rank(' a:1 s:2C d g'::tsvector, 'a | s');
@@ -193,6 +188,7 @@ SELECT 'a:1 b:3'::tsvector @@ 'a <0> b'::tsquery AS "false";
 SELECT 'a:1 b:3'::tsvector @@ 'a <1> b'::tsquery AS "false";
 SELECT 'a:1 b:3'::tsvector @@ 'a <2> b'::tsquery AS "true";
 SELECT 'a:1 b:3'::tsvector @@ 'a <3> b'::tsquery AS "false";
+SELECT 'a:1 b:3'::tsvector @@ 'a <0> a:*'::tsquery AS "true";
 
 -- tsvector editing operations