I think your are right. My 2nd guess is the Toledo game.
That is a good guess, too, but without the collapse against Kansas State later that same season, then we probably bring Rhoads back for another year.
Rhoads might have kept holding on; Campbell might have been gone by then, even that offseason (to say Missouri) or certainly within 2-3 seasons of that.
I do not think the road game against Toledo was the one that "courted" Campbell. I think that was the game in Ames the previous season, with so much lore around it about how much Ames and JTS impressed Campbell with their potential, but ISU won that game.
Other ideas that I had...
-- the OT loss against Nebraska... chance to go 4/4 for bowl games for Rhoads, send Nebraska out of the conference on a 0/2 streak against Iowa State, maybe "turns the corner" as a program... but part of me also thinks the Rhoads era would have still peaked and declined after that no matter what, so maybe it would not ultimately matter...
-- losing to KU in 2005 and missing a chance at the Big 12 North title and championship game... maybe the loss that "sent McCarney down the road to ruin," but even if we make that game, Texas smokes us on its way to a showdown with U$C at the Rose Bowl, and the problems with the program under Mac would have continued...
-- bowl losses to Rutgers/Tulsa... again, chance for Rhoads to "turn the corner," but I am not sure the outcome of those two games alone really matters that much long-term
It has to be something bedrock about the direction of the program, which likely means us coming to acquire Matt Campbell and his staff, so I like yours/mine.
Ohio State losing the 1978 Gator Bowl and Woody Hayes being fired for sucker-punching a Clemson player on the sideline also came to mind, but that was not technically an Iowa State loss. But one can imagine Hayes continuing on with the Buckeyes without that game, Bruce sticking around Iowa State long-term, and perhaps filling the same niche that Nebraska eventually would as the dominant team from the northern reaches of the Big Eight competing with Oklahoma for the title of best team in the conference (or even more than that).