Fix a long standing bug where device lists would remain cached when remote users left and rejoined the last room shared with the local homeserver.