Minio can get stuck in a exception loop
This manifested itself in jobs hitting the "tarball bug", and the minio logs were full of:
an 30 15:55:55 mupuf-gateway sh[12037]: created by github.com/minio/minio/cmd.(*transitionState).updateWorkers
Jan 30 15:55:55 mupuf-gateway sh[12037]: /home/mupuf/src/minio/cmd/bucket-lifecycle.go:334 +0x29
Jan 30 15:55:55 mupuf-gateway sh[12037]: goroutine 456 [select]:
Jan 30 15:55:55 mupuf-gateway sh[12037]: runtime.gopark(0xc00393c930?, 0x3?, 0x0?, 0x0?, 0xc00393c91a?)
Jan 30 15:55:55 mupuf-gateway sh[12037]: /usr/lib/go/src/runtime/proc.go:381 +0xd6 fp=0xc00393c500 sp=0xc00393c4e0 pc=0x43dd56
Jan 30 15:55:55 mupuf-gateway sh[12037]: runtime.selectgo(0xc00393c930, 0xc00393c914, 0x0?, 0x0, 0x0?, 0x1)
Jan 30 15:55:55 mupuf-gateway sh[12037]: /usr/lib/go/src/runtime/select.go:327 +0x7be fp=0xc00393c640 sp=0xc00393c500 pc=0x44e29e
Jan 30 15:55:55 mupuf-gateway sh[12037]: github.com/minio/minio/cmd.(*transitionState).worker(0xc0002a8fc0, {0x4fb00f8, 0xc0038f8000})
Jan 30 15:55:55 mupuf-gateway sh[12037]: /home/mupuf/src/minio/cmd/bucket-lifecycle.go:272 +0x125 fp=0xc00393cfb8 sp=0xc00393c640 pc=0x21092a5
Jan 30 15:55:55 mupuf-gateway sh[12037]: github.com/minio/minio/cmd.(*transitionState).updateWorkers.func2()
Jan 30 15:55:55 mupuf-gateway sh[12037]: /home/mupuf/src/minio/cmd/bucket-lifecycle.go:334 +0x2e fp=0xc00393cfe0 sp=0xc00393cfb8 pc=0x2109ece
Jan 30 15:55:55 mupuf-gateway sh[12037]: runtime.goexit()
Jan 30 15:55:55 mupuf-gateway sh[12037]: /usr/lib/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00393cfe8 sp=0xc00393cfe0 pc=0x4724c1
I wonder if we should have some sort of heartbeat self-tests that would restart the service when they fail.