I don't know which is a worse gig, basketball ref or baseball/softball umpire. Both are close to the fan base and they are asked to make a number judgement calls.
In youth and high school sports I don't think there is a worse environment than "travel" sports tournaments. Parents have spent hundreds if not thousands of dollars to be there and they are insufferable.
Sadly a close second can be beer league rec program adult softball and basketball games. You always get the "my high school coach screwed me, I could have been great," person combined with the occassional NCAA Division 3 hero.