Use std::unique_ptr for the Annotation data members wherever appropriate [Rebased]

This is a rebased version of !20 (closed) . For some reason I couldn't update the original merge request.

Please give this some testing. The rebasing was quite a bit of manual work, not all of it obvious. Regressions may have slipped in.