cleanup: Use allow/deny lists instead of color based ones

There was nothing racist on those names here (what? Do human races even exist?!), but let's avoid any confusion.