nvenc: Do not access to broken encode session

If an encode session failed in initializing, the encode session would be broken and the next nvenc API will cause crash.

Fixes: #1179 (closed)