In my experience it only happens when someone disconnects from the lobby, i.e. not by quitting.
It also happened when someone quit via Logout when the game was about to start.
I think a lobby that experienced at least 1 player disconnecting from it is doomed to failload.