splitmuxsrc: don't consider unlinked pads when deactivating part

If splitmuxsrc exposes multiple pads, but only one is linked, part pads will never see an EOS event. This shouldn't prevent the part from being eventually deactivated.