av1enc: Use correct enum type with libaom >= 3

This fixes, among other things, a compiler warning with clang.

Also add static assertions that our own enum values match with the ones from libaom.