This is just not true my friend. They very much do care - depending on where that scent is encountered. If they encounter it on a hiking trail, a place where they have repeatedly encountered it, they will be more tolerant. But if Mr big is coming back to his bedding in the AM, does his customary down wind J-hook to scent check the bedding area before he proceeds in, and he detects human odor, I can tell you - from experience - he very much cares indeed. Additionally, if Mr. Big is coming back to his bedding, approaches from the downwind side and comes across a scent trail from where you had hunted there a day before, he very much cares. In each of those circumstances, I can guarantee with nearly 100% certainty that the bedding area that had been getting used, will go desolate.
If they pick up human scent where they've come to expect it, they'll still hang around. But as soon as they start to detect human scent where they don't, like in the areas they bed and stage, they will vacate the area. They may return if it's a source of food. But only under cover of darkness. Eventually, they will come back, if the human scent goes away. It may take a month if you only go in there once. But if you hammer the spot, it could take years! I've personally seen this in the woods enough times to know it as fact.
Here is food for thought. Consider that a deer's nose is 4 dimensional! That's right - chew on that for a second. Let me unpack what I mean now. Say I said meet me at my favorite bar. A pretty simple statement. But consider what is required information to make that happen. You need to know where the bar is (that would be your x, y, and z coordinates) but you also need a 4th dimension..... TIME.
A deer's nose gives them all that information. Just from catching a whiff from downwind they can determine the location of the scent's source, the direction of travel, and what time the source was there.
So if you hunt a feeding area, like an oak flat for example, the deer will know you were there. But they'll also know you were there only during the day. So the deer may still use the area at night. They've patterened you! I have seen this first hand as well - confirmed via years and years of trail cam photos.
These were things I learned through the hunting school of hard knocks, and from a few very knowledgable and helpful people right here on this very forum, and a couple others I frequent.