Yes, exactly. This is why it is a pain in the butt.
Using the phone can be a pain in the butt. It does work really well with Google Home though, I usually just tell it what channel to turn to and rarely pull up anything on the phone. Even when the receiver and TV are off the chromecast can turn them on/off via hdmi and the Google Home voice commands.