We take a ref on the pad's buffer at the beginning so we need to unref when we are done in all cases.