Adjust costsize calculations to avoid introducing unnecessary roundoff

error.  This seems to explain the differing choice of plan that's been
causing geometry regress test to fail for the last few days.