If either bound is infinite, then we shouldn't even try to perform a
comparison of the values themselves.  Rearrange the logic so that
we don't.

Per buildfarm member skink and Tom Lane.