Move the port property logic from the adapter to the port itself.

The port was already doing some of the same work as a fallback but can
just as well do everything. This also makes things more unified when
there is no adapter used.