I’ve owned both—I left GXR for an X-E1, later an X-Pro. I doubt you’ll see any difference in IQ, and the larger GXR modules, like the 33/2.5, are perfectly matched to the sensor. Also, Ricoh built its M module with experience building its own ltm/m lenses. The M sensor is in there. That’s not going to be the case with the old X-Trans sensor.
*If* you do not have the Ricoh EVF (you did not mention VF in the OP), and you can’t find one in good shape, then the Fuji has an advantage with its OVF/EVF. I assumed you do have the Ricoh EVF and use it, in which case the Fuji advantages are *its* XF lenses, and the analogue aperture/speed controls.
If you have the money to play with, enjoy the experiment. I would not expect a big change (and in my case, after a few years I left Fuji (which had displaced GXR) for FF Sonys with Kolari sensors, then Leica digital). If you want a big change from APSC with your M lenses, try an A7 with a Kolari sensor, or use an M camera. M8s seem to be enjoying a sentimental renaissance with RFF users who once found fault with them, and they’re about as affordable as M2/3/4/5s.