This reverts commit d1c0b613.  The patch has some downsides that require
more attention, as discussed with Noah Misch.

Backpatch-through: 9.5