I leave my adapters on my lenses. I just bought three new lenses; I fear I'll have to get three more adapters.
Leaving them on the lenses is simply so much faster than fiddling first with the bayonetting (?) the adapter on and then still having to screw on the LTM lens. The whole point of the M-bayonet is that it's so much simpler for switching lenses.