Rework join-removal logic as per recent discussion. In particular this

fixes things so that it works for cases where nested removals are possible.
The overhead of the optimization should be significantly less, as well.