No you're not. The son of a wealthy local businessman decided to get married in the Napa Valley. Plane fare, (expensive) hotel rates, tux rent, etc was too much for my co-worker's son, who was asked to be a groomsman. I would've begged off, too. Sometimes people just assume way too much.
I agree. If you are having a destination wedding or somewhere that is far from either family/friends you should a) be prepared that not everyone is going to want to travel that much and b) know your potential wedding party well enough that you shouldn't have to put anybody in the awkward position of having to back out due to cost.
