The "program" wasn't on the trajectory you claim because the coach responsible for said trajectory was looking for a way OUT as soon as he was hired.
Even if Fred stays, there is nothing that guarantees the program continues to ascend. Indeed, cracks in the foundation were showing in 2015, remembering...
-- they still could not play defense, which meant if they were not shooting well on a particular night, they were capable of losing to absolutely anybody
-- inexplicable losses to trash teams like South Carolina, Texas Tech, and (sorry that I have to say this) UAB that limited the ceiling for the team in the rankings
-- blowing winning the Big 12 regular season title when all they had to do was close out mediocre Baylor and Kansas State teams to accomplish it
-- various disciplinary problems... BDJ as a whole, the "frosted tips," it came off as a team that was arrogant and cocksure and not really locked in as needed
-- the transfer market was already becoming tighter, so not sure Fred could have kept mining that as efficiently as he had in previous spring recruiting cycles
Fred took the program from a dumpster fire to respectable and competitive, but going from 15-20th nationally to beating out the blue bloods routinely on the court and with recruiting (such as, say, Virginia has, to give an example) is a whole new thing. The level of competition at that ratified of air for college basketball is murderous. I kind of wonder if Fred was finding his peak in 2014 and 2015 given his strengths and weaknesses.
That being said, that was still a pretty fun era. I would be happy to return to that. But the people acting like Fred would have just kept winning 2-3 more games every year and advancing one more round in the NCAA tournament in a linear fashion like clockwork need to have their head examined. That is not how basketball works.