Let's start with the basics, how to read a box of shotshells:
Gauge: gauge is theoretically determined by taking a one-pound piece of lead and dividing it into fractional lead spheres. 12-gauge is 1/12 of a pound of lead, which comes out to 1 1⁄8oz of lead. 16-gauge is 1/16 of a pound of lead, or 1oz. And so on and so forth. Because we're talking fractions of a pound, the smaller the number, the bigger the payload being shot, and thus, more recoil.
Shell length: shotgun shells come in different lengths depending on the purpose of the shell. Lighter-recoiling loads in 12-gauge are typically 2 ¾ inches long when fired. Then there's 3" and 3 1/2" shells. Longer shells typically contain more powder and lead than shorter shells, but don't assume this, and make sure to fully understand what you're buying by looking at all the variables.
Dram equivalent: this is the theoretic amount of black powder in a shell. This can at times be used as a poor man's recoil calculator, but isn't very accurate a measurement of recoil. The lower the dram equivalent, generally, the less recoil the shells create.
Shot weight: again, as mentioned before, gauge is a fraction of a pound of lead. As mentioned above, in theory, a 12-gauge shell should contain 1 1⁄8oz of lead. Thus, if you want less recoil, you can get shells with less shot in them. For example, 1oz 12-gauge is very common when someone wants lighter loads. However, don't go too light, because this can lead to inconsistent shot patterns. Remember, shot is a column of pellets being launched at a target. Less shot means less hits. But more shot means more recoil.
Shot size: see the diagram above. The higher the number from 0 (and any iteration of zero, such as 0000, 000, 00, or 0) to 9-12, the smaller the individual pellets are. Generally, bigger pellets mean more recoil. However, this is a huge generalization. Are you going to notice much recoil difference between 1 1⁄8oz of 7.5 versus 1 1⁄8oz of 8? Probably not, especially as a new shooter.
Velocity: different shells come with different velocities. More velocity means the shot needs to be propelled faster, which generally means more recoil.
Choke: choke is a different discussion than shotgun shells, but what choke you're using should sync with whatever projectiles you're shooting. If you're shooting a shotgun slug, for instance, its not recommended to use a choke tighter than say improved cylinder. Choke is something you want to keep in the back of your mind.
With all that said, how do you determine which shells to buy?
(1) What are you shooting at?
(2) Does where you're shooting have any particular rules regarding shells?
(3) Do you need nontoxic shot?
(4) How much are you going to be shooting?
(5) How much are you willing to spend per round?
Generally, most clay pigeon shooting (sporting clays, skeet, trap, 3-gun) is going to involve lighter loads of birdshot from sizes 7.5 to 9. Size 8 shot is typically associated with trap and size 9 is typically associated with skeet. Almost every gun club I've been to has had rules saying which shot sizes you can use for different gun games. Usually between 7.5 and 9.
Hunting is a lot more complicated. There's everything from slugs to tiny bismuth shot for small birds.
Home defense is typically either 00 buckshot or slugs, though some people get creative with things like turkey loads or heavier bird shot. But the baseline is 00 buckshot.
