When on my bike, I let carts go by all the time. I ride a hybrid specifically so I can safely exit the pavement onto the grass anytime I need to and ride on the grass for a short distance while the cart passes.
My main goal is of course self preservation
, but a close second is courtesy to the cart. It causes me no heartburn whatsoever...and almost no delay.