So, I get the concern, but you have to learn to trust your gun and training. When I first got my LTC, I carried unloaded. Nothing in it. It took about 3 days for me to start carrying a loaded mag, and after a week of the same cocked indicator not moving, I trusted the gun to not fire uncommanded.
You want to get a CC pistol out and into the fight as quickly as possible. That means training a good draw, trigger finger control, and how to naturally have your firearm in the needed orientation to instantly have proper sight alignment.
If a manual safety means feeling comfortable enough to carry with a round chambered, do it. With training, there is no difference in speed of draw to flick that lever to fire.
I personally want it as simple as humanly possible, because when it's time to go, its time to GO.