I had a problem just like yours. The goggles in the Elmarit are probably misaligned. The very definitive test would be to run some film in the camera using that lens and see if the focusing quirk has any effect on the photos themselves (open the lens wide and examine the negatives).
I had three Leicas when I wound up with my Elmarit. In two of them, no problem. With my M6TTL... there was NO perfect adjustment. I ran a roll of cheap supermarket film and all the photos came out perfectly focused. However, I sent the lens to Don Goldberg in Wisconsin, and he told me that the goggles were misaligned. Of course, he also fixed them.
What version is your lens? Is it the one that takes series VII filters or does it take 55mm filters? Either way, it may be the lens.