No, they can lie to you. Don't ask questions. Just say "I don't consent to a search and I don't consent for you to enter my home/business. I won't answer questions without my attorney. Leave me your business card and i will have my attorney contact you." They will respond with a different question or statement. Don't answer their question, just repeat the above.Question to piggy back off this one: If you just flat out asked any federal agent "Do I have to let you in", are they obligated to answer truthfully?
But it isn't going to happen this way. Audits don't start with an auditor showing up at your door. Audits start with a letter.