I take something called medroxyprogesterone acetate or provera at 10 mg that lowers testosterone but I felt absolutely no negative sides while I had serious and persistent sides on spironolactone. It's not clear exactly what the purposes of AA's are compared with just taking more E2. MPA works differently than the major three AA's including their analogues among the 'mides. Instead of blocking male receptors, MPA effectuates its effects more so by greatly lowering testosterone. MPA is valuable because it can be taken orally whereas progesterone tends not to make it though the liver, hence the Dr. Powers' crowd using progesterone soft gels rectally which makes a horrible mess.
E2-only, except for feminization which is necessary at least for the scalp to experience hair regrowth appears to have no negative side effects except for the one's that MtF's want. With me, after going off spironolactone, I saw an immediate return to form in terms of strength as spironolactone greatly weakened me, much more so than estrogen. It made me sweat, causes fatigue for many and many claim dulled thought processes on spironolactone. Some folks seem to tolerate it but I sure didn't. I blame it for a horrible shed to baldness whereas estrogen alone might have improved hair without a shed.
E2-only is a reaction to the fact that AA's are unneeded for most now that liver-safe forms of estrogen are available. The best rule of thumb for avoiding breast growth is to not mimic cis-female adolescent amounts of estrogen, starting at ~12 years. Some people seem to feminize quickly in 18 months or less while others can have a second puberty of up to ~8 years, comparable to 12 to 20 of puberty for cis-females. Some of the successful have used more than one type of estrogen or mixed ethiny estradiol with pure E2 or mixed conjugated equine estrogens, or CEE's.
Based upon my experiences with spironolactone, yes, AA's are more likely to lead to penile dysfunction by taking away all sexual desire. After a short period of time if someone hasn't been promoting or experiencing daily erections, the penis can become erectile dysfunctional and size and girth can be lost in the flaccid state. It all appears reversible for those not following the orchiectomy path so no penile size is permanently lost. Viable sperm can be regained at some point helped by things like clomid. There are also Serms for the folks who want a more complicated path to avoid breast growth.