Noticed via -fsanitize=undefined.

Backpatch to all branches, for the same reasons as 46ab07ffda9.

Discussion: https://postgr.es/m/20220323173537.ll7klrglnp4gn2um@alap3.anarazel.de
Backpatch: 10-