Make it be a vector of unique pointers.

That way we don't have to worry to delete the pointers on error cases,
they are deleted automatically.

This actually fixes some leaks because in some cases we were deleting
them on the error case but in some others we were not