Almost no one works on Thanksgiving, Easter, or Christmas unless it's medical related, in law enforcement, or at a convenience store.
For the holidays that a lot of people do work on I would say Memorial Day is the worst to work on. Most people have to work the following Tuesday so they are doing stuff earlier in the day. If you are working, once you are off work most people are wrapping up their day.
I don't mind working 4th of July at all. I personally have little interest in parades or carnivals so as long as I'm off in time to grill in the evening, have some beers, and watch the fireworks I am happy.
My family and friends have never done anything on Labor Day so I forget that it's a holiday most of the time.