I use both cameras.
The XA is the one I most often use.
If wearing a jacket, both cameras can be paced in the pocket. If in just shirt sleeves, the XA is the way to go.
The XA is much faster to set-up. A quick slide of the clamshell, focus, and make the photograph. It's much more quiet as well.
I also think the Zuiko is the better lens.