Hi!

INTERSECT and EXCEPT is available for postgresql-v6.4!

The patch against v6.4 is included at the end of the current text
(in uuencoded form!)

I also included the text of my Master's Thesis. (a postscript
version). I hope that you find something of it useful and would be
happy if parts of it find their way into the PostgreSQL documentation
project (If so, tell me, then I send the sources of the document!)

The contents of the document are:
  -) The first chapter might be of less interest as it gives only an
     overview on SQL.

  -) The second chapter gives a description on much of PostgreSQL's
     features (like user defined types etc. and how to use these features)

  -) The third chapter starts with an overview of PostgreSQL's internal
     structure with focus on the stages a query has to pass (i.e. parser,
     planner/optimizer, executor). Then a detailed description of the
     implementation of the Having clause and the Intersect/Except logic is
     given.

Originally I worked on v6.3.2 but never found time enough to prepare
and post a patch. Now I applied the changes to v6.4 to get Intersect
and Except working with the new version. Chapter 3 of my documentation
deals with the changes against v6.3.2, so keep that in mind when
comparing the parts of the code printed there with the patched sources
of v6.4.

Here are some remarks on the patch. There are some things that have
still to be done but at the moment I don't have time to do them
myself. (I'm doing my military service at the moment) Sorry for that
:-(

-) I used a rewrite technique for the implementation of the Except/Intersect
   logic which rewrites the query to a semantically equivalent query before
   it is handed to the rewrite system (for views, rules etc.), planner,
   executor etc.

-) In v6.3.2 the types of the attributes of two select statements
   connected by the UNION keyword had to match 100%. In v6.4 the types
   only need to be familiar (i.e. int and float can be mixed). Since this
   feature did not exist when I worked on Intersect/Except it
   does not work correctly for Except/Intersect queries WHEN USED IN
   COMBINATION WITH UNIONS! (i.e. sometimes the wrong type is used for the
   resulting table. This is because until now the types of the attributes of
   the first select statement have been used for the resulting table.
   When Intersects and/or Excepts are used in combination with Unions it
   might happen, that the first select statement of the original query
   appears at another position in the query which will be executed. The reason
   for this is the technique used for the implementation of
   Except/Intersect which does a query rewrite!)
   NOTE: It is NOT broken for pure UNION queries and pure INTERSECT/EXCEPT
         queries!!!

-) I had to add the field intersect_clause to some data structures
   but did not find time to implement printfuncs for the new field.
   This does NOT break the debug modes but when an Except/Intersect
   is used the query debug output will be the already rewritten query.

-) Massive changes to the grammar rules for SELECT and INSERT statements
   have been necessary (see comments in gram.y and documentation for
   deatails) in order to be able to use mixed queries like
   (SELECT ... UNION (SELECT ... EXCEPT SELECT)) INTERSECT SELECT...;

-) When using UNION/EXCEPT/INTERSECT you will get:
   NOTICE: equal: "Don't know if nodes of type xxx are equal".
   I did not have  time to add comparsion support for all the needed nodes,
   but the default behaviour of the function equal met my requirements.
   I did not dare to supress this message!

   That's the reason why the regression test for union will fail: These
   messages are also included in the union.out file!

-) Somebody of you changed the union_planner() function for v6.4
   (I copied the targetlist to new_tlist and that was removed and
   replaced by a cleanup of the original targetlist). These chnages
   violated some having queries executed against views so I changed
   it back again. I did not have time to examine the differences between the
   two versions but now it works :-)
   If you want to find out, try the file queries/view_having.sql on
   both versions and compare the results . Two queries won't produce a
   correct result with your version.

regards

    Stefan

src/backend/executor/execAmi.c

@@ -5,7 +5,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- *  $Id: execAmi.c,v 1.28 1998/12/14 08:11:02 scrappy Exp $
+ *  $Id: execAmi.c,v 1.29 1999/01/18 00:09:45 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -39,6 +39,9 @@
 #include "executor/nodeNestloop.h"
 #include "executor/nodeHashjoin.h"
 #include "executor/nodeHash.h"
+/***S*I***/
+#include "executor/nodeGroup.h"
+
 #include "executor/nodeAgg.h"
 #include "executor/nodeGroup.h"
 #include "executor/nodeResult.h"

src/backend/executor/nodeAgg.c

@@ -491,7 +491,10 @@ ExecAgg(Agg *node)
 		 * As long as the retrieved group does not match the
 		 * qualifications it is ignored and the next group is fetched
 		 */
+		if(node->plan.qual != NULL){
 		  qual_result = ExecQual(fix_opids(node->plan.qual), econtext);
+		}
+		
 		if (oneTuple)
 			pfree(oneTuple);
 	}

src/backend/nodes/outfuncs.c

@@ -5,7 +5,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- *  $Id: outfuncs.c,v 1.58 1998/12/20 07:13:36 scrappy Exp $
+ *  $Id: outfuncs.c,v 1.59 1999/01/18 00:09:45 momjian Exp $
  *
  * NOTES
  *	  Every (plan) node in POSTGRES has an associated "out" routine which
@@ -227,6 +227,9 @@ _outQuery(StringInfo str, Query *node)
 			node->hasSubLinks ? "true" : "false");
 	_outNode(str, node->unionClause);
 
+	appendStringInfo(str, " :intersectClause ");
+	_outNode(str, node->intersectClause);
+
 	appendStringInfo(str, " :limitOffset ");
 	_outNode(str, node->limitOffset);
 

src/backend/nodes/readfuncs.c

@@ -7,7 +7,7 @@
  *
  *
  * IDENTIFICATION
- *	  $Header: /cvsroot/pgsql/src/backend/nodes/readfuncs.c,v 1.40 1998/12/14 00:01:47 thomas Exp $
+ *	  $Header: /cvsroot/pgsql/src/backend/nodes/readfuncs.c,v 1.41 1999/01/18 00:09:46 momjian Exp $
  *
  * NOTES
  *	  Most of the read functions for plan nodes are tested. (In fact, they
@@ -163,6 +163,11 @@ _readQuery()
 	token = lsptok(NULL, &length);		/* skip :unionClause */
 	local_node->unionClause = nodeRead(true);
 
+	/***S*I***/
+ 	token = lsptok(NULL, &length);		/* skip :intersectClause */
+ 	local_node->intersectClause = nodeRead(true);
+ 
+
 	token = lsptok(NULL, &length);		/* skip :limitOffset */
 	local_node->limitOffset = nodeRead(true);
 

src/backend/optimizer/plan/planner.c

@@ -7,7 +7,7 @@
  *
  *
  * IDENTIFICATION
- *	  $Header: /cvsroot/pgsql/src/backend/optimizer/plan/planner.c,v 1.35 1998/09/09 03:48:01 vadim Exp $
+ *	  $Header: /cvsroot/pgsql/src/backend/optimizer/plan/planner.c,v 1.36 1999/01/18 00:09:47 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -95,11 +95,17 @@ Plan *
 union_planner(Query *parse)
 {
 	List	   *tlist = parse->targetList;
-	int			tlist_len = length(tlist);
+
+	/***S*H***/
+	/* copy the original tlist, we will need the original one 
+	 * for the AGG node later on */
+	List    *new_tlist = new_unsorted_tlist(tlist);
+		
 	List	   *rangetable = parse->rtable;
+
 	Plan	   *result_plan = (Plan *) NULL;
-	Index		rt_index;
 
+	Index		rt_index;
 
 	if (parse->unionClause)
 	{
@@ -124,19 +130,37 @@ union_planner(Query *parse)
 	{
 	  List  **vpm = NULL;
 	  
-		/*
-		 * check_having_qual_for_vars takes the havingQual and the tlist
-		 * as arguments and recursively scans the havingQual for VAR nodes 
-		 * that are not contained in tlist yet. If so, it creates a new entry 
-		 * and attaches it to the tlist. Latter, we use tlist_len to 
-		 * truncate tlist - ie restore actual tlist...
-		 */
+	  /***S*H***/
+	  /* This is only necessary if aggregates are in use in queries like:
+	   * SELECT sid 
+	   * FROM part
+	   * GROUP BY sid
+	   * HAVING MIN(pid) > 1;  (pid is used but never selected for!!!)
+	   * because the function 'query_planner' creates the plan for the lefttree
+	   * of the 'GROUP' node and returns only those attributes contained in 'tlist'.
+	   * The original 'tlist' contains only 'sid' here and that's why we have to
+	   * to extend it to attributes which are not selected but are used in the 
+	   * havingQual. */
+	  	  
+	  /* 'check_having_qual_for_vars' takes the havingQual and the actual 'tlist'
+	   * as arguments and recursively scans the havingQual for attributes 
+	   * (VAR nodes) that are not contained in 'tlist' yet. If so, it creates
+	   * a new entry and attaches it to the list 'new_tlist' (consisting of the 
+	   * VAR node and the RESDOM node as usual with tlists :-)  ) */
 	  if (parse->hasAggs)
 	    {
 	      if (parse->havingQual != NULL)
-				tlist = check_having_qual_for_vars(parse->havingQual, tlist);
+		{
+		  new_tlist = check_having_qual_for_vars(parse->havingQual,new_tlist);
 		}
+	    }
+	  
+	  new_tlist = preprocess_targetlist(new_tlist,
+					    parse->commandType,
+					    parse->resultRelation,
+					    parse->rtable);
 	  
+	  /* Here starts the original (pre having) code */
 	  tlist = preprocess_targetlist(tlist,
 					parse->commandType,
 					parse->resultRelation,
@@ -150,7 +174,7 @@ union_planner(Query *parse)
 	  PlannerVarParam = lcons(vpm, PlannerVarParam);
 	  result_plan = query_planner(parse,
 				      parse->commandType,
-									tlist,
+				      new_tlist,
 				      (List *) parse->qual);
 	  PlannerVarParam = lnext(PlannerVarParam);
 	  if (vpm != NULL)
@@ -173,8 +197,10 @@ union_planner(Query *parse)
 		 */
 		tuplePerGroup = parse->hasAggs;
 
+		/***S*H***/
+		/* Use 'new_tlist' instead of 'tlist' */
 		result_plan =
-			make_groupPlan(&tlist,
+			make_groupPlan(&new_tlist,
 						   tuplePerGroup,
 						   parse->groupClause,
 						   result_plan);
@@ -185,6 +211,11 @@ union_planner(Query *parse)
 	 */
 	if (parse->hasAggs)
 	{
+	        int old_length=0, new_length=0;
+		
+		/* Create the AGG node but use 'tlist' not 'new_tlist' as target list because we
+		 * don't want the additional attributes (only used for the havingQual, see above)
+		 * to show up in the result */
 		result_plan = (Plan *) make_agg(tlist, result_plan);
 
 		/*
@@ -195,31 +226,15 @@ union_planner(Query *parse)
 		  set_agg_tlist_references((Agg *) result_plan); 
 
 
-		if (parse->havingQual != NULL)
+		/***S*H***/
+		if(parse->havingQual!=NULL) 
 		  {
 		    List	   *clause;
 		    List	  **vpm = NULL;
 		    
-			/* 
-			 * Restore target list: get rid of Vars added for havingQual.
-			 * Assumption: tlist_len > 0...
-			 */
-			{
-				List   *l;
-				int		tlen = 0;
-			
-				foreach (l, ((Agg *) result_plan)->plan.targetlist)
-				{
-					if (++tlen == tlist_len)
-						break;
-				}
-				lnext(l) = NIL;
-			}
 		    
-			/*
-			 * stuff copied from above to handle the use of attributes
-			 * from outside in subselects
-			 */
+		    /* stuff copied from above to handle the use of attributes from outside
+		     * in subselects */
 
 		    if (parse->rtable != NULL)
 		      {
@@ -228,35 +243,47 @@ union_planner(Query *parse)
 		      }
 		    PlannerVarParam = lcons(vpm, PlannerVarParam);
 		    
-			/*
-			 * There is a subselect in the havingQual, so we have to
-			 * process it using the same function as for a subselect in
-			 * 'where'
-			 */
-			if (parse->hasSubLinks)
-				parse->havingQual = SS_process_sublinks((Node *) parse->havingQual);
 
 		    /* convert the havingQual to conjunctive normal form (cnf) */
-			parse->havingQual = (Node *) cnfify((Expr *) (Node *) parse->havingQual, true);
+		    (List *) parse->havingQual=cnfify((Expr *)(Node *) parse->havingQual,true);
 
-			/*
-			 * Calculate the opfids from the opnos (=select the correct
-			 * functions for the used VAR datatypes)
-			 */
-			parse->havingQual = (Node *) fix_opids((List *) parse->havingQual);
+		    /* There is a subselect in the havingQual, so we have to process it
+                     * using the same function as for a subselect in 'where' */
+		    if (parse->hasSubLinks)
+		      {
+			(List *) parse->havingQual = 
+			  (List *) SS_process_sublinks((Node *) parse->havingQual);
+		      }
 		    		    
-			((Agg *) result_plan)->plan.qual = (List *) parse->havingQual;
 		    
-			/*
-			 * Check every clause of the havingQual for aggregates used
-			 * and append them to result_plan->aggs
-			 */
+		    /* Calculate the opfids from the opnos (=select the correct functions for
+		     * the used VAR datatypes) */
+		    (List *) parse->havingQual=fix_opids((List *) parse->havingQual);
+		    
+		    ((Agg *) result_plan)->plan.qual=(List *) parse->havingQual;
+
+		    /* Check every clause of the havingQual for aggregates used and append
+		     * them to result_plan->aggs */
 		    foreach(clause, ((Agg *) result_plan)->plan.qual)
 		      {
+			/* Make sure there are aggregates in the havingQual 
+			 * if so, the list must be longer after check_having_qual_for_aggs */
+			old_length=length(((Agg *) result_plan)->aggs);			
+			
 			((Agg *) result_plan)->aggs = nconc(((Agg *) result_plan)->aggs,
 			    check_having_qual_for_aggs((Node *) lfirst(clause),
 				       ((Agg *) result_plan)->plan.lefttree->targetlist,
 				       ((List *) parse->groupClause)));
+
+			/* Have a look at the length of the returned list. If there is no
+			 * difference, no aggregates have been found and that means, that
+			 * the Qual belongs to the where clause */
+			if (((new_length=length(((Agg *) result_plan)->aggs)) == old_length) ||
+			    (new_length == 0))
+			  {
+			    elog(ERROR,"This could have been done in a where clause!!");
+			    return (Plan *)NIL;
+			  }
 		      }
 		    PlannerVarParam = lnext(PlannerVarParam);
 		    if (vpm != NULL)
@@ -277,14 +304,14 @@ union_planner(Query *parse)
 	{
 		Plan	   *sortplan = make_sortplan(tlist, parse->sortClause, result_plan);
 
-		return (Plan *) make_unique(tlist, sortplan, parse->uniqueFlag);
+		return ((Plan *) make_unique(tlist, sortplan, parse->uniqueFlag));
 	}
 	else
 	{
 		if (parse->sortClause)
-			return make_sortplan(tlist, parse->sortClause, result_plan);
+			return (make_sortplan(tlist, parse->sortClause, result_plan));
 		else
-			return (Plan *) result_plan;
+			return ((Plan *) result_plan);
 	}
 
 }

src/backend/optimizer/plan/setrefs.c

@@ -7,7 +7,7 @@
  *
  *
  * IDENTIFICATION
- *	  $Header: /cvsroot/pgsql/src/backend/optimizer/plan/setrefs.c,v 1.29 1998/12/14 00:02:10 thomas Exp $
+ *	  $Header: /cvsroot/pgsql/src/backend/optimizer/plan/setrefs.c,v 1.30 1999/01/18 00:09:48 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -923,7 +923,7 @@ del_agg_clause(Node *clause)
 	return NULL;
 }
 
-
+/***S*H***/ 
 /* check_having_qual_for_vars takes the the havingQual and the actual targetlist as arguments
  * and recursively scans the havingQual for attributes that are not included in the targetlist
  * yet. Attributes contained in the havingQual but not in the targetlist show up with queries
@@ -952,8 +952,10 @@ check_having_qual_for_vars(Node *clause, List *targetlist_so_far)
        */
 
       /* Check if the VAR is already contained in the targetlist */
-		if (tlist_member((Var *) clause, (List *) targetlist_so_far) == NULL)
-			add_tl_element(&tmp_rel, (Var *) clause);
+      if (tlist_member((Var *)clause, (List *)targetlist_so_far) == NULL)
+	{
+	  add_tl_element(&tmp_rel, (Var *)clause); 
+	} 
 	    
       return tmp_rel.targetlist;
     }
@@ -967,7 +969,9 @@ check_having_qual_for_vars(Node *clause, List *targetlist_so_far)
        * arguments...
        */
       foreach(t, ((Expr *) clause)->args)
+	{
 	  targetlist_so_far = check_having_qual_for_vars(lfirst(t), targetlist_so_far);
+	}
       return targetlist_so_far;
     }
   else if (IsA(clause, Aggreg))
@@ -985,9 +989,13 @@ check_having_qual_for_vars(Node *clause, List *targetlist_so_far)
        * expression and its index expression...
        */
       foreach(t, aref->refupperindexpr)
+	{
 	  targetlist_so_far = check_having_qual_for_vars(lfirst(t), targetlist_so_far);
+	}
       foreach(t, aref->reflowerindexpr)
+	{
 	  targetlist_so_far = check_having_qual_for_vars(lfirst(t), targetlist_so_far);
+	}
       targetlist_so_far = check_having_qual_for_vars(aref->refexpr, targetlist_so_far);
       targetlist_so_far = check_having_qual_for_vars(aref->refassgnexpr, targetlist_so_far);
       
@@ -1010,25 +1018,23 @@ check_having_qual_for_vars(Node *clause, List *targetlist_so_far)
       
       return targetlist_so_far;
     }
-	else if (IsA(clause, Param) ||IsA(clause, Const))
+  else if (IsA(clause, Param) || IsA(clause, Const))
     {
       /* do nothing! */
       return targetlist_so_far;
     }
-
-	/*
-	 * If we get to a sublink, then we only have to check the lefthand
-	 * side of the expression to see if there are any additional VARs
-	 */
+  /* If we get to a sublink, then we only have to check the lefthand side of the expression
+   * to see if there are any additional VARs */
   else if (IsA(clause, SubLink))
     {
-		foreach(t, ((List *) ((SubLink *) clause)->lefthand))
+      foreach(t,((List *)((SubLink *)clause)->lefthand))
+	{
 	  targetlist_so_far = check_having_qual_for_vars(lfirst(t), targetlist_so_far);
+	}
       return targetlist_so_far;
     }
   else
     {
-
       /*
        * Ooops! we can not handle that!
        */
@@ -1045,8 +1051,7 @@ check_having_qual_for_vars(Node *clause, List *targetlist_so_far)
 List *
 check_having_qual_for_aggs(Node *clause, List *subplanTargetList, List *groupClause)
 {
-	List	   *t,
-			   *l1;
+	List	   *t, *l1;
 	List	   *agg_list = NIL;
 
 	int contained_in_group_clause = 0;
@@ -1062,26 +1067,43 @@ check_having_qual_for_aggs(Node *clause, List *subplanTargetList, List *groupCla
 	  subplanVar = match_varid((Var *) clause, subplanTargetList);
 	  
 	  /*
-		 * Change the varno & varattno fields of the var node to point to
-		 * the resdom->resno fields of the subplan (lefttree)
+	   * Change the varno & varattno fields of the var node to point to the resdom->resno
+	   * fields of the subplan (lefttree) 
 	   */	  
 	  ((Var *) clause)->varattno = subplanVar->resdom->resno;
 
 	  return NIL;
 
 	}
+        /***S*H***/
 	else if (is_funcclause(clause) || not_clause(clause) || 
 		 or_clause(clause) || and_clause(clause))
 	{
+	  int new_length=0, old_length=0;
+	  
 		/*
 		 * This is a function. Recursively call this routine for its
 		 * arguments... (i.e. for AND, OR, ... clauses!)
 		 */
 		foreach(t, ((Expr *) clause)->args)
 		{
+		  old_length=length((List *)agg_list);
+
 		  agg_list = nconc(agg_list,
 				   check_having_qual_for_aggs(lfirst(t), subplanTargetList,
 							      groupClause));
+
+		  /* The arguments of OR or AND clauses are comparisons or relations 
+		   * and because we are in the havingQual there must be at least one operand
+		   * using an aggregate function. If so, we will find it and the length of the
+		   * agg_list will be increased after the above call to 
+		   * check_having_qual_for_aggs. If there are no aggregates used, the query
+                   * could have been formulated using the 'where' clause */
+		  if(((new_length=length((List *)agg_list)) == old_length) || (new_length == 0))
+		    {
+		      elog(ERROR,"This could have been done in a where clause!!");
+		      return NIL;
+		    } 
 		}
 		return agg_list;
 	}
@@ -1141,26 +1163,20 @@ check_having_qual_for_aggs(Node *clause, List *subplanTargetList, List *groupCla
 
 		return agg_list;
 	}
-	else if (IsA(clause, Param) ||IsA(clause, Const))
+	else if (IsA(clause, Param) || IsA(clause, Const))
 	{
 		/* do nothing! */
 		return NIL;
 	}
-
-	/*
-	 * This is for Sublinks which show up as EXPR nodes. All the other
-	 * EXPR nodes (funcclauses, and_clauses, or_clauses) were caught above
-	 */
+	/* This is for Sublinks which show up as EXPR nodes. All the other EXPR nodes
+         * (funcclauses, and_clauses, or_clauses) were caught above */
 	else if (IsA(clause, Expr))
 	  {
-
-		/*
-		 * Only the lefthand side of the sublink has to be checked for
-		 * aggregates to be attached to result_plan->aggs (see
-		 * planner.c:union_planner() )
+	    /* Only the lefthand side of the sublink has to be checked for aggregates
+             * to be attached to result_plan->aggs (see planner.c:union_planner() )
 	     */	    
-		foreach(t, ((List *) ((SubLink *) ((SubPlan *)
-						   ((Expr *) clause)->oper)->sublink)->lefthand))
+	    foreach(t,((List *)((SubLink *)((SubPlan *)
+		       ((Expr *)clause)->oper)->sublink)->lefthand)) 
 	      {
 		agg_list = 
 		  nconc(agg_list,
@@ -1168,27 +1184,39 @@ check_having_qual_for_aggs(Node *clause, List *subplanTargetList, List *groupCla
 						   subplanTargetList, groupClause));
 	      }
 
+            /* The first argument of ...->oper has also to be checked */
+	    {	      
+	      List *tmp_ptr;
 	      
-		/*
-		 * All arguments to the Sublink node are attributes from outside
-		 * used within the sublink. Here we have to check that only
-		 * attributes that is grouped for are used!
-		 */
-		foreach(t, ((Expr *) clause)->args)
+	      foreach(tmp_ptr, ((SubLink *)((SubPlan *)
+					    ((Expr *)clause)->oper)->sublink)->oper)
+		{		
+  		  agg_list = 
+		    nconc(agg_list,
+			  check_having_qual_for_aggs((Node *)lfirst(((Expr *)
+								     lfirst(tmp_ptr))->args), 
+						     subplanTargetList, groupClause));
+		}
+	    }
+	      		
+	    /* All arguments to the Sublink node are attributes from outside used within
+	     * the sublink. Here we have to check that only attributes that is grouped for
+	     * are used! */
+	    foreach(t,((Expr *)clause)->args) 
 	      {	
 		contained_in_group_clause = 0;
 
-			foreach(l1, groupClause)
+		foreach(l1,groupClause)
 		  {
-				if (tlist_member(lfirst(t), lcons(((GroupClause *) lfirst(l1))->entry, NIL)) !=
+		    if (tlist_member(lfirst(t),lcons(((GroupClause *)lfirst(l1))->entry,NIL)) != 
 			NULL)
-					contained_in_group_clause = 1;
+		      {
+			contained_in_group_clause=1;
+		      }
 		  }
 		
-			/*
-			 * If the use of the attribute is allowed (i.e. it is in the
-			 * groupClause) we have to adjust the varnos and varattnos
-			 */
+		/* If the use of the attribute is allowed (i.e. it is in the groupClause)
+		 * we have to adjust the varnos and varattnos */
 		if (contained_in_group_clause)
 		  {
 		    agg_list = 
@@ -1198,7 +1226,7 @@ check_having_qual_for_aggs(Node *clause, List *subplanTargetList, List *groupCla
 		  }
 		else
 		  {
-				elog(ERROR, "You must group by the attribute used from outside!");
+		    elog(ERROR,"You must group by the attribute used from outside!");
 		    return NIL;
 		  }		
 	      }
@@ -1214,3 +1242,4 @@ check_having_qual_for_aggs(Node *clause, List *subplanTargetList, List *groupCla
 	    return NIL;
 	  }
 }
+/***S*H***/ /* End */

src/backend/optimizer/plan/subselect.c

@@ -405,20 +405,6 @@ SS_process_sublinks(Node *expr)
 			SS_process_sublinks((Node *) ((Expr *) expr)->args);
 	else if (IsA(expr, SubLink))/* got it! */
 	{
-
-		/*
-		 * Hack to make sure expr->oper->args points to the same VAR node
-		 * as expr->lefthand does. Needed for subselects in the havingQual
-		 * when used on views. Otherwise aggregate functions will fail
-		 * later on (at execution time!) Reason: The rewite System makes
-		 * several copies of the VAR nodes and in this case it should not
-		 * do so :-(
-		 */
-		if (((SubLink *) expr)->lefthand != NULL)
-		{
-			lfirst(((Expr *) lfirst(((SubLink *) expr)->oper))->args) =
-				lfirst(((SubLink *) expr)->lefthand);
-		}
 		expr = _make_subplan((SubLink *) expr);
 	}
 

src/backend/parser/analyze.c

@@ -5,7 +5,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- *  $Id: analyze.c,v 1.91 1998/12/14 06:50:32 scrappy Exp $
+ *  $Id: analyze.c,v 1.92 1999/01/18 00:09:49 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -26,6 +26,11 @@
 #include "parser/parse_node.h"
 #include "parser/parse_relation.h"
 #include "parser/parse_target.h"
+/***S*I***/
+#include "parser/parse_expr.h"
+#include "catalog/pg_type.h"
+#include "parse.h"
+
 #include "utils/builtins.h"
 #include "utils/mcxt.h"
 
@@ -384,7 +389,12 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 	 * unionClause may be false
 	 */
   	qry->unionall = stmt->unionall;	
-	qry->unionClause = transformUnionClause(stmt->unionClause, qry->targetList);
+
+ 	/***S*I***/
+ 	/* Just hand through the unionClause and intersectClause. 
+ 	 * We will handle it in the function Except_Intersect_Rewrite() */
+ 	qry->unionClause = stmt->unionClause;
+ 	qry->intersectClause = stmt->intersectClause;	
 
 	/*
 	 * If there is a havingQual but there are no aggregates, then there is
@@ -942,7 +952,12 @@ transformSelectStmt(ParseState *pstate, SelectStmt *stmt)
 	 * unionClause may be false
 	 */
 	qry->unionall = stmt->unionall;
-	qry->unionClause = transformUnionClause(stmt->unionClause, qry->targetList);
+
+ 	/***S*I***/
+ 	/* Just hand through the unionClause and intersectClause. 
+ 	 * We will handle it in the function Except_Intersect_Rewrite() */
+ 	qry->unionClause = stmt->unionClause;
+ 	qry->intersectClause = stmt->intersectClause;
 
 	/*
 	 * If there is a havingQual but there are no aggregates, then there is
@@ -1012,3 +1027,97 @@ transformCursorStmt(ParseState *pstate, SelectStmt *stmt)
 
 	return qry;
 }
+
+/***S*I***/
+/* This function steps through the tree
+ * built up by the select_w_o_sort rule
+ * and builds a list of all SelectStmt Nodes found
+ * The built up list is handed back in **select_list.
+ * If one of the SelectStmt Nodes has the 'unionall' flag
+ * set to true *unionall_present hands back 'true' */
+void 
+create_select_list(Node *ptr, List **select_list, bool *unionall_present)
+{
+  if(IsA(ptr, SelectStmt)) {
+    *select_list = lappend(*select_list, ptr);    
+    if(((SelectStmt *)ptr)->unionall == TRUE) *unionall_present = TRUE;    
+    return;    
+  }
+  
+  /* Recursively call for all arguments. A NOT expr has no lexpr! */
+  if (((A_Expr *)ptr)->lexpr != NULL) 
+     create_select_list(((A_Expr *)ptr)->lexpr, select_list, unionall_present);
+  create_select_list(((A_Expr *)ptr)->rexpr, select_list, unionall_present);
+}
+
+/* Changes the A_Expr Nodes to Expr Nodes and exchanges ANDs and ORs.
+ * The reason for the exchange is easy: We implement INTERSECTs and EXCEPTs 
+ * by rewriting these queries to semantically equivalent queries that use
+ * IN and NOT IN subselects. To be able to use all three operations 
+ * (UNIONs INTERSECTs and EXCEPTs) in one complex query we have to 
+ * translate the queries into Disjunctive Normal Form (DNF). Unfortunately
+ * there is no function 'dnfify' but there is a function 'cnfify'
+ * which produces DNF when we exchange ANDs and ORs before calling
+ * 'cnfify' and exchange them back in the result.
+ *
+ * If an EXCEPT or INTERSECT is present *intersect_present
+ * hands back 'true' */ 
+Node *A_Expr_to_Expr(Node *ptr, bool *intersect_present)
+{
+  Node *result;
+  
+  switch(nodeTag(ptr))
+    {
+    case T_A_Expr:
+      {
+	A_Expr *a = (A_Expr *)ptr;
+	
+	switch (a->oper)
+	  {
+	  case AND:
+	    {
+	      Expr *expr = makeNode(Expr);
+	      Node	   *lexpr = A_Expr_to_Expr(((A_Expr *)ptr)->lexpr, intersect_present);
+	      Node	   *rexpr = A_Expr_to_Expr(((A_Expr *)ptr)->rexpr, intersect_present);
+
+	      *intersect_present = TRUE;
+	      
+	      expr->typeOid = BOOLOID;
+	      expr->opType = OR_EXPR;
+	      expr->args = makeList(lexpr, rexpr, -1);
+	      result = (Node *) expr;
+	      break;	      
+	    }	  	  
+	  case OR:
+	    {
+	      Expr *expr = makeNode(Expr);
+	      Node	   *lexpr = A_Expr_to_Expr(((A_Expr *)ptr)->lexpr, intersect_present);
+	      Node	   *rexpr = A_Expr_to_Expr(((A_Expr *)ptr)->rexpr, intersect_present);
+
+	      expr->typeOid = BOOLOID;
+	      expr->opType = AND_EXPR;
+	      expr->args = makeList(lexpr, rexpr, -1);
+	      result = (Node *) expr;
+	      break;	      
+	    }
+	  case NOT:
+	    {
+	      Expr *expr = makeNode(Expr);
+	      Node	   *rexpr = A_Expr_to_Expr(((A_Expr *)ptr)->rexpr, intersect_present);
+
+	      expr->typeOid = BOOLOID;
+	      expr->opType = NOT_EXPR;
+	      expr->args = makeList(rexpr, -1);
+	      result = (Node *) expr;
+	      break;	      
+	    }
+	  }	
+	break;	
+      }
+    default:
+      {
+	result = ptr;
+      }      
+    }
+  return result;  
+}

src/backend/parser/keywords.c

@@ -7,7 +7,7 @@
  *
  *
  * IDENTIFICATION
- *	  $Header: /cvsroot/pgsql/src/backend/parser/keywords.c,v 1.50 1998/12/18 09:10:34 vadim Exp $
+ *	  $Header: /cvsroot/pgsql/src/backend/parser/keywords.c,v 1.51 1999/01/18 00:09:53 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -93,6 +93,9 @@ static ScanKeyword ScanKeywords[] = {
 	{"else", ELSE},
 	{"encoding", ENCODING},
 	{"end", END_TRANS},
+	/***S*I***/
+	{"except", EXCEPT},
+
 	{"execute", EXECUTE},
 	{"exists", EXISTS},
 	{"explain", EXPLAIN},
@@ -120,6 +123,9 @@ static ScanKeyword ScanKeywords[] = {
 	{"insensitive", INSENSITIVE},
 	{"insert", INSERT},
 	{"instead", INSTEAD},
+	/***S*I***/
+	{"intersect", INTERSECT},
+
 	{"interval", INTERVAL},
 	{"into", INTO},
 	{"is", IS},

src/backend/rewrite/rewriteManip.c

@@ -6,7 +6,7 @@
  *
  *
  * IDENTIFICATION
- *	  $Header: /cvsroot/pgsql/src/backend/rewrite/rewriteManip.c,v 1.23 1998/12/14 00:02:17 thomas Exp $
+ *	  $Header: /cvsroot/pgsql/src/backend/rewrite/rewriteManip.c,v 1.24 1999/01/18 00:09:56 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -152,7 +152,10 @@ OffsetVarNodes(Node *node, int offset, int sublevels_up)
 		case T_SubLink:
 			{
 				SubLink	*sub = (SubLink *)node;
+				List *tmp_oper, *tmp_lefthand;
 
+				/* We also have to adapt the variables used in sub->lefthand
+				 * and sub->oper */
 				OffsetVarNodes(
 						(Node *)(sub->lefthand),
 						offset,
@@ -162,6 +165,19 @@ OffsetVarNodes(Node *node, int offset, int sublevels_up)
 						(Node *)(sub->subselect),
 						offset,
 						sublevels_up + 1);
+
+				/***S*I***/
+				/* Make sure the first argument of sub->oper points to the
+				 * same var as sub->lefthand does otherwise we will
+				 * run into troubles using aggregates (aggno will not be
+				 * set correctly) */
+				tmp_lefthand = sub->lefthand;				
+				foreach(tmp_oper, sub->oper)
+				  {				    
+				    lfirst(((Expr *) lfirst(tmp_oper))->args) = 
+				      lfirst(tmp_lefthand);
+				    tmp_lefthand = lnext(tmp_lefthand);
+				  }	
 			}
 			break;
 
@@ -364,6 +380,7 @@ ChangeVarNodes(Node *node, int rt_index, int new_index, int sublevels_up)
 		case T_SubLink:
 			{
 				SubLink	*sub = (SubLink *)node;
+				List *tmp_oper, *tmp_lefthand;
 				
 				ChangeVarNodes(
 						(Node *)(sub->lefthand),
@@ -376,6 +393,19 @@ ChangeVarNodes(Node *node, int rt_index, int new_index, int sublevels_up)
 						rt_index,
 						new_index,
 						sublevels_up + 1);
+				
+				/***S*I***/
+				/* Make sure the first argument of sub->oper points to the
+				 * same var as sub->lefthand does otherwise we will
+				 * run into troubles using aggregates (aggno will not be
+				 * set correctly) */
+				tmp_lefthand = sub->lefthand;
+				foreach(tmp_oper, sub->oper)
+				  {				    
+				    lfirst(((Expr *) lfirst(tmp_oper))->args) = 
+				      lfirst(tmp_lefthand);
+				    tmp_lefthand = lnext(tmp_lefthand);
+				  }	
 			}
 			break;
 
@@ -465,7 +495,10 @@ AddQual(Query *parsetree, Node *qual)
 	if (qual == NULL)
 		return;
 
-	copy = copyObject(qual);
+	/***S*I***/
+	/* copy = copyObject(qual); */
+	copy = qual;
+
 	old = parsetree->qual;
 	if (old == NULL)
 		parsetree->qual = copy;
@@ -485,7 +518,10 @@ AddHavingQual(Query *parsetree, Node *havingQual)
 	if (havingQual == NULL)
 		return;
 
-	copy = copyObject(havingQual);
+	/***S*I***/
+	copy = havingQual;
+	/* copy = copyObject(havingQual); */
+
 	old = parsetree->havingQual;
 	if (old == NULL)
 		parsetree->havingQual = copy;
@@ -494,6 +530,20 @@ AddHavingQual(Query *parsetree, Node *havingQual)
 			(Node *) make_andclause(makeList(parsetree->havingQual, copy, -1));
 }
 
+void
+AddNotHavingQual(Query *parsetree, Node *havingQual)
+{
+	Node	   *copy;
+
+	if (havingQual == NULL)
+		return;
+
+	/***S*I***/
+	/* copy = (Node *)make_notclause( (Expr *)copyObject(havingQual)); */
+	copy = (Node *) make_notclause((Expr *)havingQual);
+
+	AddHavingQual(parsetree, copy);
+}
 
 void
 AddNotQual(Query *parsetree, Node *qual)
@@ -503,7 +553,9 @@ AddNotQual(Query *parsetree, Node *qual)
 	if (qual == NULL)
 		return;
 
-	copy = (Node *) make_notclause(copyObject(qual));
+	/***S*I***/
+	/* copy = (Node *) make_notclause((Expr *)copyObject(qual)); */
+	copy = (Node *) make_notclause((Expr *)qual);
 
 	AddQual(parsetree, copy);
 }
@@ -835,7 +887,7 @@ HandleRIRAttributeRule(Query *parsetree,
 							   rt_index, attr_num, modified, badsql, 0);
 }
 
-
+#ifdef NOT_USED
 static void
 nodeHandleViewRule(Node **nodePtr,
 				   List *rtable,
@@ -976,10 +1028,19 @@ nodeHandleViewRule(Node **nodePtr,
 			{
 				SubLink    *sublink = (SubLink *) node;
 				Query	   *query = (Query *) sublink->subselect;
+				List *tmp_lefthand, *tmp_oper;
+				
 
 				nodeHandleViewRule((Node **) &(query->qual), rtable, targetlist,
 								   rt_index, modified, sublevels_up + 1);
 
+				/***S*H*D***/
+				nodeHandleViewRule((Node **) &(query->havingQual), rtable, targetlist,
+						   rt_index, modified, sublevels_up + 1);
+				nodeHandleViewRule((Node **) &(query->targetList), rtable, targetlist,
+						   rt_index, modified, sublevels_up + 1);
+
+
 				/*
 				 * We also have to adapt the variables used in
 				 * sublink->lefthand and sublink->oper
@@ -993,9 +1054,16 @@ nodeHandleViewRule(Node **nodePtr,
 				 * will run into troubles using aggregates (aggno will not
 				 * be set correctly
 				 */
-				pfree(lfirst(((Expr *) lfirst(sublink->oper))->args));
-				lfirst(((Expr *) lfirst(sublink->oper))->args) =
-					lfirst(sublink->lefthand);
+				/* pfree(lfirst(((Expr *) lfirst(sublink->oper))->args)); */
+
+				/***S*I***/
+ 				tmp_lefthand = sublink->lefthand;				
+ 				foreach(tmp_oper, sublink->oper)
+ 				  {				    
+ 				    lfirst(((Expr *) lfirst(tmp_oper))->args) = 
+ 				      lfirst(tmp_lefthand);
+ 				    tmp_lefthand = lnext(tmp_lefthand);
+ 				  }								
   			}
 			break;
 		default:
@@ -1004,7 +1072,6 @@ nodeHandleViewRule(Node **nodePtr,
 	}
 }
 
-#ifdef NOT_USED
 void
 HandleViewRule(Query *parsetree,
 			   List *rtable,

src/backend/tcop/postgres.c

@@ -7,7 +7,7 @@
  *
  *
  * IDENTIFICATION
- *	  $Header: /cvsroot/pgsql/src/backend/tcop/postgres.c,v 1.96 1999/01/17 06:18:42 momjian Exp $
+ *	  $Header: /cvsroot/pgsql/src/backend/tcop/postgres.c,v 1.97 1999/01/18 00:09:56 momjian Exp $
  *
  * NOTES
  *	  this is the "main" module of the postgres backend and
@@ -448,6 +448,14 @@ pg_parse_and_plan(char *query_string,	/* string to execute */
 
 		querytree = querytree_list->qtrees[i];
 
+ 		/***S*I***/
+ 		/* Rewrite Union, Intersect and Except Queries
+ 		 * to normal Union Queries using IN and NOT IN subselects */
+ 		if(querytree->intersectClause != NIL) 
+ 		  {	  
+ 		    querytree = Except_Intersect_Rewrite(querytree);
+ 		  }
+
 		if (DebugPrintQuery)
 		{
 			if (DebugPrintQuery > 3)
@@ -1527,7 +1535,7 @@ PostgresMain(int argc, char *argv[], int real_argc, char *real_argv[])
 	if (!IsUnderPostmaster)
 	{
 		puts("\nPOSTGRES backend interactive interface ");
-		puts("$Revision: 1.96 $ $Date: 1999/01/17 06:18:42 $\n");
+		puts("$Revision: 1.97 $ $Date: 1999/01/18 00:09:56 $\n");
 	}
 
 	/* ----------------

src/include/executor/nodeGroup.h

@@ -6,7 +6,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- * $Id: nodeGroup.h,v 1.7 1998/09/01 04:35:56 momjian Exp $
+ * $Id: nodeGroup.h,v 1.8 1999/01/18 00:10:02 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -22,5 +22,8 @@ extern bool ExecInitGroup(Group *node, EState *estate, Plan *parent);
 extern int	ExecCountSlotsGroup(Group *node);
 extern void ExecEndGroup(Group *node);
 extern void ExecReScanGroup(Group *node, ExprContext *exprCtxt, Plan *parent);
+/***S*I***/
+extern void ExecReScanGroup(Group *node, ExprContext *exprCtxt, Plan *parent);
+
 
 #endif	 /* NODEGROUP_H */

src/include/nodes/parsenodes.h

@@ -6,7 +6,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- * $Id: parsenodes.h,v 1.65 1999/01/05 15:45:49 vadim Exp $
+ * $Id: parsenodes.h,v 1.66 1999/01/18 00:10:06 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -58,6 +58,9 @@ typedef struct Query
 								 * BY */
 	Node	   *havingQual;		/* qualification of each group */
 
+        /***S*I***/
+	List	   *intersectClause;	
+
 	List	   *unionClause;	/* unions are linked under the previous
 								 * query */
 	Node	   *limitOffset;	/* # of result tuples to skip */
@@ -606,6 +609,8 @@ typedef struct InsertStmt
 	Node	   *havingClause;	/* having conditional-expression */
 	List	   *unionClause;	/* union subselect parameters */
 	bool	   unionall;		/* union without unique sort */
+        /***S*I***/
+        List       *intersectClause;  
 } InsertStmt;
 
 /* ----------------------
@@ -646,6 +651,10 @@ typedef struct SelectStmt
 	Node	   *whereClause;	/* qualifications */
 	List	   *groupClause;	/* group by clause */
 	Node	   *havingClause;	/* having conditional-expression */
+        /***S*I***/
+        List       *intersectClause;
+        List       *exceptClause;
+  
 	List	   *unionClause;	/* union subselect parameters */
 	List	   *sortClause;		/* sort clause (a list of SortGroupBy's) */
 	char	   *portalname;		/* the portal (cursor) to create */

src/include/parser/analyze.h

@@ -5,7 +5,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- * $Id: analyze.h,v 1.4 1998/09/01 04:37:25 momjian Exp $
+ * $Id: analyze.h,v 1.5 1999/01/18 00:10:11 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -15,5 +15,8 @@
 #include <parser/parse_node.h>
 
 extern QueryTreeList *parse_analyze(List *pl, ParseState *parentParseState);
+/***S*I***/
+extern void create_select_list(Node *ptr, List **select_list, bool *unionall_present);
+extern Node *A_Expr_to_Expr(Node *ptr, bool *intersect_present);
 
 #endif	 /* ANALYZE_H */

src/include/rewrite/rewriteHandler.h

@@ -6,7 +6,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- * $Id: rewriteHandler.h,v 1.6 1998/09/01 04:38:01 momjian Exp $
+ * $Id: rewriteHandler.h,v 1.7 1999/01/18 00:10:12 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -34,5 +34,9 @@ typedef struct _rewrite_meta_knowledge RewriteInfo;
 
 
 extern List *QueryRewrite(Query *parsetree);
-
+/***S*I***/
+extern Query *Except_Intersect_Rewrite(Query *parsetree);
+extern void create_list(Node *ptr, List **intersect_list);
+extern Node *intersect_tree_analyze(Node *tree, Node *first_select, Node *parsetree);
+extern void check_targetlists_are_compatible(List *prev_target, List *current_target);
 #endif	 /* REWRITEHANDLER_H */

src/include/rewrite/rewriteManip.h

@@ -6,7 +6,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- * $Id: rewriteManip.h,v 1.11 1998/10/21 16:21:29 momjian Exp $
+ * $Id: rewriteManip.h,v 1.12 1999/01/18 00:10:16 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -25,6 +25,8 @@ void		AddQual(Query *parsetree, Node *qual);
 void		AddHavingQual(Query *parsetree, Node *havingQual);
 
 void		AddNotQual(Query *parsetree, Node *qual);
+void            AddNotHavingQual(Query *parsetree, Node *havingQual);
+
 void		FixNew(RewriteInfo *info, Query *parsetree);
 
 void HandleRIRAttributeRule(Query *parsetree, List *rtable, List *targetlist,

src/include/utils/elog.h

@@ -6,7 +6,7 @@
  *
  * Copyright (c) 1994, Regents of the University of California
  *
- * $Id: elog.h,v 1.8 1998/09/01 04:39:03 momjian Exp $
+ * $Id: elog.h,v 1.9 1999/01/18 00:10:17 momjian Exp $
  *
  *-------------------------------------------------------------------------
  */
@@ -29,7 +29,12 @@
 #define ABORTX	0x4000			/* abort process after logging */
 #endif
 
-#define ELOG_MAXLEN 4096
+/***S*I***/
+/* Increase this to be able to use postmaster -d 3 with complex
+ * view definitions (which are transformed to very, very large INSERT statements
+ * and if -d 3 is used the query string of these statements is printed using
+ * vsprintf which expects enough memory reserved! */
+#define ELOG_MAXLEN 12288
 
 
 /* uncomment the following if you want your elog's to be timestamped */