Fix broken mmap failure-detection code, and improve error message.

Per an observation by Thom Brown that my previous commit made an
overly large shmem allocation crash the server, on Linux.