The refs have called it right. Just because there's contact doesn't mean it's a foul. If the defender is there and jumps straight up and stays in that plane, it isn't a foul.
You forgot the Niang corollary to the "straight up in the plane" no-call. If it's Niang, that's a foul.