Methinks you are making my main point for me. Silver goes up and down like any other speculative commodity. And like you (I assume), I prefer to buy low and sell high vs. buying high and selling low. I was willing to buy at $20.00. Now at around $25.00? No thanks. I can wait for it to come back down again... and it will. But beyond that, the premium over spot on current American Eagles has gone bonkers and I have no interest in paying such crazy premiums to hold silver in that form. I can wait. Nothing in my world demands or even suggests that I need to buy American Eagles at current prices.
Like you, some of the silver coins I've bought in the past are worth more than I paid for them and others not so much. Take out the cost of procuring and the cost of selling and I'm probably not much better than even. There have always been better investments... and worse.
But that's not why I hold physical silver. It is not to make money. It's not to pay this month's bills. It's a "what happens when the government Ponzi scheme collapses" kind of thing. To quote Google: "Silver's real value is as a hedge and safe haven asset and the primary reason to own silver is not about wealth accumulation but about wealth protection and hedging risk in an uncertain world."
I agree with that.