There's your problem. The fact of the matter is, you can pull the "actually it was a foul", but within each game what really is a foul is set during the course of a game. By rule we could call about 80 fouls a game and everyone would be dragging their walkons in to end the games. But that'd be a horrible entertainment product.
When a standard has been set for what's called during the game, there are effectively a new set of rules on the floor that varies from the strict definition in the rulebook. To make a call like that when the de facto rules have been different the first 39 minutes, is a bad call.