I know how you feel man. I was on finasteride for 2 years, trying to convince my self there were no side effects.
During this time I have had a steady girlfriend, so my loss in sex drive I attributed it to being with the same person for a long period of time. I stopped finasteride about 2-3 month ago and let me tell you everything is way more better.
Now I am not saying this is for everyone, but my effects were some what hidden, I mean I could get an erection when I needed to, but I often just didn't feel like having sex. In fact, I would use the "I have a headache".
Anyways I just thought I would let you know. WHen I first started I had similar experiences, I would actually find it difficult to get an erection. People told me to stay on it, and the side effects would disapear. It no longer became difficult, but my Labido was gone.