I think when it comes to whether a program fades into mediocrity or not depends on the type of coach that is hired to replace the departing legendary coach. Case in point, both Kentucky and Indiana basketball. Kentucky's fade to irrelevancy began when Rick Pitino decided to take the Boston Celtic's job (even though Tubby Smith did win a championship a year later, with mostly Rick Pitino players), and definitely became irrelevant upon the firing of Smith - Gillespie has done nothing of note there yet. As far as Indiana, they for the most part have been irrelevant since Bobby Knight was fired.
To counter the argument, take North Carolina and Kansas, another two legendary programs. North Carolina hires Williams away from Kansas and goes on to win a championship the very next year and has been a top-10 team ever since. No fading into obscurity there. Kansas replaces Williams with Self, and while there's an initial struggle of trying to get out of the first round of the NCAA Tournament, Self brings the first championship to KU in 20+ years. That and he's taken a team that everyone assumed was "middle-of-the-pack" in a weak Big XII conference and has them in first place.
If Duke takes the route of North Carolina and does their homework and hires a proven, established, successful head coach to replace Coach K when he decides to leave, Duke basketball will be just fine in the future. If it decides to do either what Kentucky or Indiana did, there could be problems for them in the future.