ACTUALLY, if you want to get medical, estrogen has a far larger protective role in females than testosterone does in men. Estrogen is billed as being one of the primary reasons why women have lower incidences of early-onset cardiovascular disease and why they are less likely to get a multitude of diseases, whichi makes their life span on average several years longer than men's. So to go back to what you said, I sincerely doubt the direct decrease of T is making the effect... instead, when T is converted to estrogen at androgen receptors, I'd bet my huge med school debt that it's the subsequent decrease in estrogen that is negative with respect to health.
AND like another poster mentioned, Testosterone is transiently INCREASED on finasteride (propecia), although a few weeks later the body should adjust and the T levels should return to normal values. T AND DHT ARE DIFFERENT!