The problem with those 'census' numbers is they dont show the degree of fandom. You call up any random group of people, and you'll probably hear 'yeah, im a fan of *local team here*'. What they dont show is how passionate that fandom is. For instance, many may answer they're a fan of Illinois, but be much more interested in the Bears. Or they may be a fan but by no means fan enough to be willing to drop the kind of cash needed for a bowl trip (or on season tickets). On the other hand, you may have a smaller number that identify themselves as fans of another school, but they may be on the whole more passionate fans of that school willing to put up the money for tickets and bowl trips.
Those are good points, but what TV networks care about are how many fans will watch the game. This still doesn't answer that, but there will be many more that watch than those that go to the games.