Further improvement of make_greater_string.

Make sure that it considers all the possibilities that the old code did,
instead of trying only one possibility per character position.  To keep the
runtime in bounds, instead tweak the character incrementers to not try
every possible multibyte character code.  Remove unnecessary logic to
restore the old character value on failure.  Additional comment and
formatting cleanup.

src/backend/utils/adt/selfuncs.c

@@ -5701,13 +5701,23 @@ byte_increment(unsigned char *ptr, int len)
  * and "9" is seen as largest by the collation, and append that to the given
  * prefix before trying to find a string that compares as larger.
  *
- * If we max out the righthand byte, truncate off the last character
- * and start incrementing the next.  For example, if "z" were the last
- * character in the sort order, then we could produce "foo" as a
- * string greater than "fonz".
+ * To search for a greater string, we repeatedly "increment" the rightmost
+ * character, using an encoding-specific character incrementer function.
+ * When it's no longer possible to increment the last character, we truncate
+ * off that character and start incrementing the next-to-rightmost.
+ * For example, if "z" were the last character in the sort order, then we
+ * could produce "foo" as a string greater than "fonz".
  *
  * This could be rather slow in the worst case, but in most cases we
  * won't have to try more than one or two strings before succeeding.
+ *
+ * Note that it's important for the character incrementer not to be too anal
+ * about producing every possible character code, since in some cases the only
+ * way to get a larger string is to increment a previous character position.
+ * So we don't want to spend too much time trying every possible character
+ * code at the last position.  A good rule of thumb is to be sure that we
+ * don't try more than 256*K values for a K-byte character (and definitely
+ * not 256^K, which is what an exhaustive search would approach).
  */
 Const *
 make_greater_string(const Const *str_const, FmgrInfo *ltproc, Oid collation)
@@ -5779,17 +5789,19 @@ make_greater_string(const Const *str_const, FmgrInfo *ltproc, Oid collation)
 		}
 	}
 
+	/* Select appropriate character-incrementer function */
 	if (datatype == BYTEAOID)
-		charinc = &byte_increment;
+		charinc = byte_increment;
 	else
 		charinc = pg_database_encoding_character_incrementer();
 
+	/* And search ... */
 	while (len > 0)
 	{
-		int		charlen;
+		int			charlen;
 		unsigned char *lastchar;
-		Const	   *workstr_const;
 
+		/* Identify the last character --- for bytea, just the last byte */
 		if (datatype == BYTEAOID)
 			charlen = 1;
 		else
@@ -5799,9 +5811,15 @@ make_greater_string(const Const *str_const, FmgrInfo *ltproc, Oid collation)
 		/*
 		 * Try to generate a larger string by incrementing the last character
 		 * (for BYTEA, we treat each byte as a character).
+		 *
+		 * Note: the incrementer function is expected to return true if it's
+		 * generated a valid-per-the-encoding new character, otherwise false.
+		 * The contents of the character on false return are unspecified.
 		 */
-		if (charinc(lastchar, charlen))
+		while (charinc(lastchar, charlen))
 		{
+			Const	   *workstr_const;
+
 			if (datatype == BYTEAOID)
 				workstr_const = string_to_bytea_const(workstr, len);
 			else
@@ -5825,7 +5843,8 @@ make_greater_string(const Const *str_const, FmgrInfo *ltproc, Oid collation)
 		}
 
 		/*
-		 * Truncate off the last character or byte.
+		 * No luck here, so truncate off the last character and try to
+		 * increment the next one.
 		 */
 		len -= charlen;
 		workstr[len] = '\0';