And there may be no place to conveniently put their hat; not to mention the likelihood that if they take it off they will forget and leave it behind.
The only time I ever wore a hat at the table, especially when eating out, was in Texas where I was wearing a cowboy hat. I went into a restaurant where there were about 40 people, mostly men, and all of the men were wearing cowboy hats at the table. There was no place to hang hats in the room.
I wore my hat throughout the meal. When in Texas, do as the Texans do.