vtenc: Handle missing sink scenario more gracefully

Discovered on accident when I forgot to link a sink after vtenc and the error message was a strange looking assertion error (because of this line).

Mostly a cosmetic change to display an error that somewhat points towards the cause. gst_vtenc_negotiate_specific_format_details was already returning whether it succeeded, so I just added a check for that.