Implement jsonpath .datetime() method

This commit implements jsonpath .datetime() method as it's specified in
SQL/JSON standard.  There are no-argument and single-argument versions of
this method.  No-argument version selects first of ISO datetime formats
matching input string.  Single-argument version accepts template string as
its argument.

Additionally to .datetime() method itself this commit also implements
comparison ability of resulting date and time values.  There is some difficulty
because exising jsonb_path_*() functions are immutable, while comparison of
timezoned and non-timezoned types involves current timezone.  At first, current
timezone could be changes in session.  Moreover, timezones themselves are not
immutable and could be updated.  This is why we let existing immutable functions
throw errors on such non-immutable comparison.  In the same time this commit
provides jsonb_path_*_tz() functions which are stable and support operations
involving timezones.  As new functions are added to the system catalog,
catversion is bumped.

Support of .datetime() method was the only blocker prevents T832 from being
marked as supported.  sql_features.txt is updated correspondingly.

Extracted from original patch by Nikita Glukhov, Teodor Sigaev, Oleg Bartunov.
Heavily revised by me.  Comments were adjusted by Liudmila Mantrova.

Discussion: https://postgr.es/m/fcc6fc6a-b497-f39a-923d-aa34d0c588e8%402ndQuadrant.com
Discussion: https://postgr.es/m/CAPpHfdsZgYEra_PeCLGNoXOWYx6iU-S3wF8aX0ObQUcZU%2B4XTw%40mail.gmail.com
Author: Alexander Korotkov, Nikita Glukhov, Teodor Sigaev, Oleg Bartunov, Liudmila Mantrova
Reviewed-by: Anastasia Lubennikova, Peter Eisentraut

doc/src/sgml/func.sgml

@@ -11910,16 +11910,6 @@ table2-mapping
   </para>
 
   <itemizedlist>
-   <listitem>
-    <para>
-     <literal>.datetime()</literal> item method is not implemented yet
-     mainly because immutable <type>jsonpath</type> functions and operators
-     cannot reference session timezone, which is used in some datetime
-     operations.  Datetime support will be added to <type>jsonpath</type>
-     in future versions of <productname>PostgreSQL</productname>.
-    </para>
-   </listitem>
-
    <listitem>
     <para>
      A path expression can be a Boolean predicate, although the SQL/JSON
@@ -12190,6 +12180,20 @@ table2-mapping
         <entry><literal>$.z.abs()</literal></entry>
         <entry><literal>0.3</literal></entry>
        </row>
+       <row>
+        <entry><literal>datetime()</literal></entry>
+        <entry>Date/time value converted from a string</entry>
+        <entry><literal>["2015-8-1", "2015-08-12"]</literal></entry>
+        <entry><literal>$[*] ? (@.datetime() &lt; "2015-08-2". datetime())</literal></entry>
+        <entry><literal>2015-8-1</literal></entry>
+       </row>
+       <row>
+        <entry><literal>datetime(<replaceable>template</replaceable>)</literal></entry>
+        <entry>Date/time value converted from a string using the specified template</entry>
+        <entry><literal>["12:30", "18:40"]</literal></entry>
+        <entry><literal>$[*].datetime("HH24:MI")</literal></entry>
+        <entry><literal>"12:30:00", "18:40:00"</literal></entry>
+       </row>
        <row>
         <entry><literal>keyvalue()</literal></entry>
         <entry>
@@ -12207,6 +12211,37 @@ table2-mapping
      </tgroup>
     </table>
 
+    <note>
+     <para>
+      The result type of <literal>datetime()</literal> and
+      <literal>datetime(<replaceable>template</replaceable>)</literal>
+      methods can be <type>date</type>, <type>timetz</type>, <type>time</type>,
+      <type>timestamptz</type>, or <type>timestamp</type>.
+      Both methods determine the result type dynamically.
+     </para>
+     <para>
+      The <literal>datetime()</literal> method sequentially tries ISO formats
+      for <type>date</type>, <type>timetz</type>, <type>time</type>,
+      <type>timestamptz</type>, and <type>timestamp</type>. It stops on
+      the first matching format and the corresponding data type.
+     </para>
+     <para>
+      The <literal>datetime(<replaceable>template</replaceable>)</literal>
+      method determines the result type by the provided template string.
+     </para>
+     <para>
+      The <literal>datetime()</literal> and
+      <literal>datetime(<replaceable>template</replaceable>)</literal> methods
+      use the same parsing rules as <literal>to_timestamp</literal> SQL
+      function does (see <xref linkend="functions-formatting"/>) with three
+      exceptions.  At first, these methods doesn't allow unmatched template
+      patterns.  At second, only following separators are allowed in the
+      template string: minus sign, period, solidus, comma, apostrophe,
+      semicolon, colon and space.  At third, separators in the template string
+      must exactly match the input string.
+     </para>
+    </note>
+
     <table id="functions-sqljson-filter-ex-table">
      <title><type>jsonpath</type> Filter Expression Elements</title>
      <tgroup cols="5">
@@ -12350,6 +12385,15 @@ table2-mapping
       </tbody>
      </tgroup>
     </table>
+
+    <note>
+     <para>
+      When different date/time values are compared, an implicit cast is
+      applied. A <type>date</type> value can be cast to <type>timestamp</type>
+      or <type>timestamptz</type>, <type>timestamp</type> can be cast to
+      <type>timestamptz</type>, and <type>time</type> &mdash; to <type>timetz</type>.
+     </para>
+    </note>
    </sect3>
 
   </sect2>
@@ -12582,7 +12626,7 @@ table2-mapping
    <para>
     The <literal>@?</literal> and <literal>@@</literal> operators suppress
     the following errors: lacking object field or array element, unexpected
-    JSON item type, and numeric errors.
+    JSON item type, datetime and numeric errors.
     This behavior might be helpful while searching over JSON document
     collections of varying structure.
    </para>
@@ -12851,18 +12895,33 @@ table2-mapping
   <indexterm>
    <primary>jsonb_path_exists</primary>
   </indexterm>
+  <indexterm>
+   <primary>jsonb_path_exists_tz</primary>
+  </indexterm>
   <indexterm>
    <primary>jsonb_path_match</primary>
   </indexterm>
+  <indexterm>
+   <primary>jsonb_path_match_tz</primary>
+  </indexterm>
   <indexterm>
    <primary>jsonb_path_query</primary>
   </indexterm>
+  <indexterm>
+   <primary>jsonb_path_query_tz</primary>
+  </indexterm>
   <indexterm>
    <primary>jsonb_path_query_array</primary>
   </indexterm>
+  <indexterm>
+   <primary>jsonb_path_query_array_tz</primary>
+  </indexterm>
   <indexterm>
    <primary>jsonb_path_query_first</primary>
   </indexterm>
+  <indexterm>
+   <primary>jsonb_path_query_first_tz</primary>
+  </indexterm>
 
   <table id="functions-json-processing-table">
     <title>JSON Processing Functions</title>
@@ -13202,6 +13261,9 @@ table2-mapping
          <para><literal>
            jsonb_path_exists(target jsonb, path jsonpath [, vars jsonb [, silent bool]])
          </literal></para>
+         <para><literal>
+           jsonb_path_exists_tz(target jsonb, path jsonpath [, vars jsonb, silent bool])
+         </literal></para>
         </entry>
         <entry><type>boolean</type></entry>
         <entry>
@@ -13222,6 +13284,9 @@ table2-mapping
          <para><literal>
            jsonb_path_match(target jsonb, path jsonpath [, vars jsonb, silent bool])
          </literal></para>
+         <para><literal>
+           jsonb_path_match_tz(target jsonb, path jsonpath [, vars jsonb, silent bool])
+         </literal></para>
         </entry>
         <entry><type>boolean</type></entry>
         <entry>
@@ -13243,6 +13308,9 @@ table2-mapping
          <para><literal>
            jsonb_path_query(target jsonb, path jsonpath [, vars jsonb, silent bool])
          </literal></para>
+         <para><literal>
+           jsonb_path_query_tz(target jsonb, path jsonpath [, vars jsonb, silent bool])
+         </literal></para>
         </entry>
         <entry><type>setof jsonb</type></entry>
         <entry>
@@ -13271,6 +13339,9 @@ table2-mapping
          <para><literal>
            jsonb_path_query_array(target jsonb, path jsonpath [, vars jsonb, silent bool])
          </literal></para>
+         <para><literal>
+           jsonb_path_query_array_tz(target jsonb, path jsonpath [, vars jsonb, silent bool])
+         </literal></para>
         </entry>
         <entry><type>jsonb</type></entry>
         <entry>
@@ -13291,6 +13362,9 @@ table2-mapping
          <para><literal>
            jsonb_path_query_first(target jsonb, path jsonpath [, vars jsonb, silent bool])
          </literal></para>
+         <para><literal>
+           jsonb_path_query_first_tz(target jsonb, path jsonpath [, vars jsonb, silent bool])
+         </literal></para>
         </entry>
         <entry><type>jsonb</type></entry>
         <entry>
@@ -13433,11 +13507,8 @@ table2-mapping
 
   <note>
    <para>
-    The <literal>jsonb_path_exists</literal>, <literal>jsonb_path_match</literal>,
-    <literal>jsonb_path_query</literal>, <literal>jsonb_path_query_array</literal>, and
-    <literal>jsonb_path_query_first</literal>
-    functions have optional <literal>vars</literal> and <literal>silent</literal>
-    arguments.
+    The <literal>jsonb_path_*</literal> functions have optional
+    <literal>vars</literal> and <literal>silent</literal> arguments.
    </para>
    <para>
     If the <parameter>vars</parameter> argument is specified, it provides an
@@ -13451,6 +13522,20 @@ table2-mapping
    </para>
   </note>
 
+  <note>
+   <para>
+    Some of the <literal>jsonb_path_*</literal> functions have the
+    <literal>_tz</literal> suffix. These functions have been implemented to
+    support comparison of date/time values that involves implicit
+    timezone-aware casts. Since operations with time zones are not immutable,
+    these functions are qualified as stable. Their counterparts without the
+    suffix do not support such casts, so they are immutable and can be used for
+    such use-cases as expression indexes
+    (see <xref linkend="indexes-expressional"/>). There is no difference
+    between these functions for other <type>jsonpath</type> operations.
+   </para>
+  </note>
+
   <para>
     See also <xref linkend="functions-aggregate"/> for the aggregate
     function <function>json_agg</function> which aggregates record

src/backend/catalog/sql_features.txt

@@ -544,7 +544,7 @@ T828	JSON_QUERY			NO
 T829	JSON_QUERY: array wrapper options			NO	
 T830	Enforcing unique keys in SQL/JSON constructor functions			NO	
 T831	SQL/JSON path language: strict mode			YES	
-T832	SQL/JSON path language: item method			NO	datetime() not yet implemented
+T832	SQL/JSON path language: item method			YES	
 T833	SQL/JSON path language: multiple subscripts			YES	
 T834	SQL/JSON path language: wildcard member accessor			YES	
 T835	SQL/JSON path language: filter expressions			YES	

src/backend/catalog/system_views.sql

@@ -1287,6 +1287,46 @@ LANGUAGE INTERNAL
 STRICT IMMUTABLE PARALLEL SAFE
 AS 'jsonb_path_query_first';
 
+CREATE OR REPLACE FUNCTION
+  jsonb_path_exists_tz(target jsonb, path jsonpath, vars jsonb DEFAULT '{}',
+                    silent boolean DEFAULT false)
+RETURNS boolean
+LANGUAGE INTERNAL
+STRICT STABLE PARALLEL SAFE
+AS 'jsonb_path_exists_tz';
+
+CREATE OR REPLACE FUNCTION
+  jsonb_path_match_tz(target jsonb, path jsonpath, vars jsonb DEFAULT '{}',
+                   silent boolean DEFAULT false)
+RETURNS boolean
+LANGUAGE INTERNAL
+STRICT STABLE PARALLEL SAFE
+AS 'jsonb_path_match_tz';
+
+CREATE OR REPLACE FUNCTION
+  jsonb_path_query_tz(target jsonb, path jsonpath, vars jsonb DEFAULT '{}',
+                   silent boolean DEFAULT false)
+RETURNS SETOF jsonb
+LANGUAGE INTERNAL
+STRICT STABLE PARALLEL SAFE
+AS 'jsonb_path_query_tz';
+
+CREATE OR REPLACE FUNCTION
+  jsonb_path_query_array_tz(target jsonb, path jsonpath, vars jsonb DEFAULT '{}',
+                         silent boolean DEFAULT false)
+RETURNS jsonb
+LANGUAGE INTERNAL
+STRICT STABLE PARALLEL SAFE
+AS 'jsonb_path_query_array_tz';
+
+CREATE OR REPLACE FUNCTION
+  jsonb_path_query_first_tz(target jsonb, path jsonpath, vars jsonb DEFAULT '{}',
+                         silent boolean DEFAULT false)
+RETURNS jsonb
+LANGUAGE INTERNAL
+STRICT STABLE PARALLEL SAFE
+AS 'jsonb_path_query_first_tz';
+
 --
 -- The default permissions for functions mean that anyone can execute them.
 -- A number of functions shouldn't be executable by just anyone, but rather

src/backend/utils/adt/jsonpath.c

@@ -337,12 +337,14 @@ flattenJsonPathParseItem(StringInfo buf, JsonPathParseItem *item,
 		case jpiPlus:
 		case jpiMinus:
 		case jpiExists:
+		case jpiDatetime:
 			{
 				int32		arg = reserveSpaceForItemPointer(buf);
 
-				chld = flattenJsonPathParseItem(buf, item->value.arg,
-												nestingLevel + argNestingLevel,
-												insideArraySubscript);
+				chld = !item->value.arg ? pos :
+					flattenJsonPathParseItem(buf, item->value.arg,
+											 nestingLevel + argNestingLevel,
+											 insideArraySubscript);
 				*(int32 *) (buf->data + arg) = chld - pos;
 			}
 			break;
@@ -692,6 +694,15 @@ printJsonPathItem(StringInfo buf, JsonPathItem *v, bool inKey,
 		case jpiDouble:
 			appendBinaryStringInfo(buf, ".double()", 9);
 			break;
+		case jpiDatetime:
+			appendBinaryStringInfo(buf, ".datetime(", 10);
+			if (v->content.arg)
+			{
+				jspGetArg(v, &elem);
+				printJsonPathItem(buf, &elem, false, false);
+			}
+			appendStringInfoChar(buf, ')');
+			break;
 		case jpiKeyValue:
 			appendBinaryStringInfo(buf, ".keyvalue()", 11);
 			break;
@@ -754,6 +765,8 @@ jspOperationName(JsonPathItemType type)
 			return "floor";
 		case jpiCeiling:
 			return "ceiling";
+		case jpiDatetime:
+			return "datetime";
 		default:
 			elog(ERROR, "unrecognized jsonpath item type: %d", type);
 			return NULL;
@@ -889,6 +902,7 @@ jspInitByBuffer(JsonPathItem *v, char *base, int32 pos)
 		case jpiPlus:
 		case jpiMinus:
 		case jpiFilter:
+		case jpiDatetime:
 			read_int32(v->content.arg, base, pos);
 			break;
 		case jpiIndexArray:
@@ -913,7 +927,8 @@ jspGetArg(JsonPathItem *v, JsonPathItem *a)
 		   v->type == jpiIsUnknown ||
 		   v->type == jpiExists ||
 		   v->type == jpiPlus ||
-		   v->type == jpiMinus);
+		   v->type == jpiMinus ||
+		   v->type == jpiDatetime);
 
 	jspInitByBuffer(a, v->base, v->content.arg);
 }
@@ -961,6 +976,7 @@ jspGetNext(JsonPathItem *v, JsonPathItem *a)
 			   v->type == jpiFloor ||
 			   v->type == jpiCeiling ||
 			   v->type == jpiDouble ||
+			   v->type == jpiDatetime ||
 			   v->type == jpiKeyValue ||
 			   v->type == jpiStartsWith);
 

src/backend/utils/adt/jsonpath_gram.y

@@ -94,12 +94,14 @@ static JsonPathParseItem *makeItemLikeRegex(JsonPathParseItem *expr,
 %token	<str>		LESS_P LESSEQUAL_P EQUAL_P NOTEQUAL_P GREATEREQUAL_P GREATER_P
 %token	<str>		ANY_P STRICT_P LAX_P LAST_P STARTS_P WITH_P LIKE_REGEX_P FLAG_P
 %token	<str>		ABS_P SIZE_P TYPE_P FLOOR_P DOUBLE_P CEILING_P KEYVALUE_P
+%token	<str>		DATETIME_P
 
 %type	<result>	result
 
 %type	<value>		scalar_value path_primary expr array_accessor
 					any_path accessor_op key predicate delimited_predicate
 					index_elem starts_with_initial expr_or_predicate
+					datetime_template opt_datetime_template
 
 %type	<elems>		accessor_expr
 
@@ -247,9 +249,20 @@ accessor_op:
 	| array_accessor				{ $$ = $1; }
 	| '.' any_path					{ $$ = $2; }
 	| '.' method '(' ')'			{ $$ = makeItemType($2); }
+	| '.' DATETIME_P '(' opt_datetime_template ')'
+									{ $$ = makeItemUnary(jpiDatetime, $4); }
 	| '?' '(' predicate ')'			{ $$ = makeItemUnary(jpiFilter, $3); }
 	;
 
+datetime_template:
+	STRING_P						{ $$ = makeItemString(&$1); }
+	;
+
+opt_datetime_template:
+	datetime_template				{ $$ = $1; }
+	| /* EMPTY */					{ $$ = NULL; }
+	;
+
 key:
 	key_name						{ $$ = makeItemKey(&$1); }
 	;
@@ -272,6 +285,7 @@ key_name:
 	| FLOOR_P
 	| DOUBLE_P
 	| CEILING_P
+	| DATETIME_P
 	| KEYVALUE_P
 	| LAST_P
 	| STARTS_P

src/backend/utils/adt/jsonpath_scan.l

@@ -323,6 +323,7 @@ static const JsonPathKeyword keywords[] = {
 	{ 6, false,	STRICT_P,	"strict"},
 	{ 7, false,	CEILING_P,	"ceiling"},
 	{ 7, false,	UNKNOWN_P,	"unknown"},
+	{ 8, false,	DATETIME_P,	"datetime"},
 	{ 8, false,	KEYVALUE_P,	"keyvalue"},
 	{ 10,false, LIKE_REGEX_P, "like_regex"},
 };

src/backend/utils/errcodes.txt

@@ -207,6 +207,7 @@ Section: Class 22 - Data Exception
 2200S    E    ERRCODE_INVALID_XML_COMMENT                                    invalid_xml_comment
 2200T    E    ERRCODE_INVALID_XML_PROCESSING_INSTRUCTION                     invalid_xml_processing_instruction
 22030    E    ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE                        duplicate_json_object_key_value
+22031    E    ERRCODE_INVALID_ARGUMENT_FOR_JSON_DATETIME_FUNCTION            invalid_argument_for_json_datetime_function
 22032    E    ERRCODE_INVALID_JSON_TEXT                                      invalid_json_text
 22033    E    ERRCODE_INVALID_SQL_JSON_SUBSCRIPT                             invalid_sql_json_subscript
 22034    E    ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM                            more_than_one_sql_json_item

src/include/catalog/catversion.h

@@ -53,6 +53,6 @@
  */
 
 /*							yyyymmddN */
-#define CATALOG_VERSION_NO	201909214
+#define CATALOG_VERSION_NO	201909251
 
 #endif

src/include/catalog/pg_proc.dat

@@ -9332,6 +9332,28 @@
   proname => 'jsonb_path_match', prorettype => 'bool',
   proargtypes => 'jsonb jsonpath jsonb bool', prosrc => 'jsonb_path_match' },
 
+{ oid => '1177', descr => 'jsonpath exists test with timezone',
+  proname => 'jsonb_path_exists_tz', provolatile => 's',
+  prorettype => 'bool', proargtypes => 'jsonb jsonpath jsonb bool',
+  prosrc => 'jsonb_path_exists_tz' },
+{ oid => '1179', descr => 'jsonpath query with timezone',
+  proname => 'jsonb_path_query_tz', provolatile => 's',
+  prorows => '1000', proretset => 't',
+  prorettype => 'jsonb', proargtypes => 'jsonb jsonpath jsonb bool',
+  prosrc => 'jsonb_path_query_tz' },
+{ oid => '1180', descr => 'jsonpath query wrapped into array with timezone',
+  proname => 'jsonb_path_query_array_tz', provolatile => 's',
+  prorettype => 'jsonb', proargtypes => 'jsonb jsonpath jsonb bool',
+  prosrc => 'jsonb_path_query_array_tz' },
+{ oid => '2023', descr => 'jsonpath query first item with timezone',
+  proname => 'jsonb_path_query_first_tz', provolatile => 's',
+  prorettype => 'jsonb', proargtypes => 'jsonb jsonpath jsonb bool',
+  prosrc => 'jsonb_path_query_first_tz' },
+{ oid => '2030', descr => 'jsonpath match with timezone',
+  proname => 'jsonb_path_match_tz', provolatile => 's',
+  prorettype => 'bool', proargtypes => 'jsonb jsonpath jsonb bool',
+  prosrc => 'jsonb_path_match_tz' },
+
 { oid => '4010', descr => 'implementation of @? operator',
   proname => 'jsonb_path_exists_opr', prorettype => 'bool',
   proargtypes => 'jsonb jsonpath', prosrc => 'jsonb_path_exists_opr' },

src/include/utils/jsonpath.h

@@ -79,6 +79,7 @@ typedef enum JsonPathItemType
 	jpiFloor,					/* .floor() item method */
 	jpiCeiling,					/* .ceiling() item method */
 	jpiDouble,					/* .double() item method */
+	jpiDatetime,				/* .datetime() item method */
 	jpiKeyValue,				/* .keyvalue() item method */
 	jpiSubscript,				/* array subscript: 'expr' or 'expr TO expr' */
 	jpiLast,					/* LAST array subscript */

src/test/regress/expected/jsonpath.out

@@ -395,6 +395,18 @@ select '$.keyvalue().key'::jsonpath;
  $.keyvalue()."key"
 (1 row)
 
+select '$.datetime()'::jsonpath;
+   jsonpath   
+--------------
+ $.datetime()
+(1 row)
+
+select '$.datetime("datetime template")'::jsonpath;
+            jsonpath             
+---------------------------------
+ $.datetime("datetime template")
+(1 row)
+
 select '$ ? (@ starts with "abc")'::jsonpath;
         jsonpath         
 -------------------------

src/test/regress/sql/jsonpath.sql

@@ -71,6 +71,8 @@ select '"aaa".type()'::jsonpath;
 select 'true.type()'::jsonpath;
 select '$.double().floor().ceiling().abs()'::jsonpath;
 select '$.keyvalue().key'::jsonpath;
+select '$.datetime()'::jsonpath;
+select '$.datetime("datetime template")'::jsonpath;
 
 select '$ ? (@ starts with "abc")'::jsonpath;
 select '$ ? (@ starts with $var)'::jsonpath;