Remove some redundant tests and improve comments in next_token().

Cosmetic, but it might make this a bit less confusing to the next reader.