vabaseenc: Move out encoder validation from assertion.
Assertion can be disabled at compilation time. Still it's important to validate it the encoder object was opened by the subclass. This patch removes the assertion and returns if the encoder is open.