Repair crash with unsortable data in grouping sets.

Previously the code would generate incorrect results, assertion
failures, or crashes if given unsortable (but hashable) columns in
grouping sets.  Handle by throwing an error instead.

Report and patch by Pavan Deolasee (though I changed the error
wording slightly); regression test by me.

(This affects 9.5 only since the planner was refactored in 9.6.)