Both. Too much estrogen will give you reduced libido and ED - or in worst case scenario lead to loss of sex drive and make you impotent, and so will deficiency.
Use large amounts of zinc picolinate if you want to adjust the androgen/estrogen ratio. I guarantee you that you'll get reduced libido and ED, if you start messing around with prescription AI's - without having a severe problem with estrogen.