Is it rude to ask for gas money when someone requests to ride along to an out of town event?

I don't think so. But I get a lot of flak for bringing this up. We often drive two hours out of town (plus garage parking) to recreational events that happen to be a mutual destination for other family members and/or friends. Naturally, the group tends to coalesce into a carpool arrangement at the request of said people, since our car is large enough to accommodate a few extra bodies. They never offer to contribute gas money, and after asking them to chip in, I usually get something to the extent of "well, you'd be going there anyways". It drives me crazy, as this happens often and I get painted as the bad guy for even bringing it up. Gas is expensive! It's like the cost of gas is some intangible throwaway expense that is rude to even bring up. I am not wealthy, and I think it's more than fair for people to contribute their fair share.