They start earlier because they have much more aggressive loss.
It's not guaranteed, but I suspect that people starting later might be more prone to sides for two reasons:
1) Your hormones are already on the decline due to age versus someone who starts in their early 20s while still at their hormonal peak.
2) If you're one of the people who notice hair loss after the age of 30, I'd suspect you're not very high in androgens to begin with.
Not saying you will get sides, it's a hunch mostly from my own experience. I didn't notice hair loss until the age of 29, and I immediately got bad sides from finasteride.
If you have no bald relatives, you want to rule out other possibilities, including thyroid.