Yeah, the monthly "cost to own" is what matters, not the rate or principal on their own. The two obviously have an inverse relationship, so higher rates are going to push prices down.
So it might not net out too differently for buyers, but certainly less of a sellers' market.
This is why I've always been skeptical of anybody proposing to improve "affordability" by making something easier to finance (e.g., student loans). The seller isn't stupid. If they know you've got more ability to finance behind you, then they know they can charge more upfront to reap that windfall.
But hey, sometimes that indirect transfer of income and wealth (either to homeowners or to colleges and universities themselves) is much the point, a feature rather than a bug of the system.