enum_param should return -ENOENT when the param is not known.
When negotiating buffers handle -ENOENT. This means the port does
not know about the property and we should assume anything is fine
so just use the filter from the first port.