Discovered on accident when I forgot to link a sink after vtenc and the error message was a strange looking assertion error (because of this line).
Mostly a cosmetic change to display an error that somewhat points towards the cause. gst_vtenc_negotiate_specific_format_details was already returning whether it succeeded, so I just added a check for that.