Lived with my parents, who we get along with great, for 8 weeks while our house finished up. We had a 3 and 2 year old.
The worst part by far is just having alone time with just the four of us. I remember a 3 day span where my parents left town, we forgot how good it felt to be by ourselves.
If I had to do it over, I probably would have chosen an apartment or rented a home. You will get super antsy to get out of their home and an into your new home. Remember, to give your contractors the time to finish it up. We had a few projects finished up after we moved it--it was a lot more cleaning.