I'm 55, have some of the same issues. I bought a SuperRedhawk 7 1/2" 44mag (actually, Lady Radtekk bought it for me as a wedding gift) a few years back. First time I shot it with the factory grips I almost cried. The trigger guard slammed back into the front of my middle finger like a steel hammer. Put Hogue hardwood grooved grips on it, lowers the fingers down below the trigger guard, it was like a different gun. I shoot it with 44spcl, 44mag, and 340gr Buffalo Bore for fun. The BB is pricey, so I only shoot a couple of rounds to remain familiar with it, but the grips made a HUGE difference. Yes, the'll change the bore axis and increase muzzle flip a little, but the gun won't hurt to shoot.
To be fair, I have BIG hands, recoil itself doesn't bother me. I shoot a 12ga one-handed with a pistol grip using slugs, will shoot 357 mag all day, and once went to an Appleseeds event using a Mosin. The Mosin was steaming hot in July so I put it away after lunch and focused on making sure my 12 y/o was having a good time with her 22. Point is, spend some money on the grips, it'll be worth EVERY penny.
As to group size, I'm willing to be there's another factor. Somebody mentioned grip strength and using finger bands. I'd suggest getting a couple of 1-2 pound weights (a 1 liter water bottle would work) and using that to strengthen your arms by repeatedly raising it to a shooting position and holding for a 10 count. I notice when there's a long time between range trips that my gun tends to get heavy FAST, even the polymer xD I use for EDC. That SRH feels like it weighs a TON after a dozen rounds or so.