They are basing it sole on the number of fans these schools have. It's not about the casuals really. They know Bama, Ohio State, Texas, Georgia, etc. etc. have massive fan bases and every single one of them will be watching their team play. It's a safe scenario for the networks. Maybe they lose the casuals who would want to see some different teams, but between just those massive fanbases and the college football junkies they will get their number.
Massive fanbases are chock full of casual/bandwagoners, so if a big brand is eliminated and not in there, the networks know that's a ton of people not tuning in because their team isn't in and they don't really care about college football.