Two suggestions.
a) Try turning the whole assembly upside down and tap with a light rubber or plastic mallet. If the key moves "up" (down, with inverted assembly), insert a drift pin. A full-sized pin my not fit (key didn't move all the way), so start with a smaller one and work the key "up." End up with a full-sized drift pin and leave it in place. Now, start the actual pin (from the correct side), using it to force the drift pin out.
b) Consider a small dab of grease to hold the key more or less where it will show at least some daylight through the pin hole. Then follow above suggetions.