Add tests (and associated fixes) to restore coverage of rtree.c.

FossilOrigin-Name: b06f4695bdab244d9c764c082cd434a764dc5c29