There's a lot more to it. Every player that goes in for a hard tackle is aware of their tackling technique and what the consequences are if they arrive late, from the wrong angle, or with studs up. All challenges are not equal.
Regarding the first part of your post, good refs will establish how much contact they allow and stay mostly consistent with it. The issue becomes when contact is allowed once and then never again.
In general, the same contact is not treated the same if it's not affecting the flow of the game. That's why you'll see a lot of tugging on jerseys get ignored or a defender shielding a ball over the endline that gets laid out with no whistle.