When our proxy is removed or when we destroy it, remove our listeners
and go to the unconnected state.

Destroy the proxy in all cases.