Remove explicit uses of Buffer

Unfortunately doesn't fix the circular dependencies warning because
we still indirectly depend on Buffer.

Address #32