WIP: Don't crash out when the reference/test directory is missing

Here's a workaround to prevent crashes when we pass non-existing path to os.scan(). The first patch attempts to resolve this is a better (IMHO) way, yet it falls short.

See the follow-up commit message with details.

Keeping this as WIP, since it's not perfect, yet useful as-is. To any reviewers, you're welcome to take ownership and finish this.

Thanks