Anxiety and depression is not a documented side effect of Finasteride use, but it is an extremely common side effect of hair loss! Have you considered that it is just coincidence that your anxiety begun when you were on Finasteride?
I have been through the worst of anxiety; it most commonly starts at around the age of 20 (which is when it started for me). I had crippling anxiety for about 5 years, and have only recently gotten over it. None of this was related to hair loss; it was caused by certain events that happened in my life that I couldn't let go, and that caused constant worry and rumination. It was also exacerbated by drugs and alcohol (BAD coping strategy!), but that's another story.
I would consider myself a non-medical expert on anxiety and depression, it's causes, symptoms and treatments, since I spent 5 years of my life studying everything I could so that I could understand what was wrong with me.
But anyways, I do beleive some people with a very unfortunate genetic makeup may have very negative reactions to Finasteride, I don't think it's worth worrying about. The fact that this drug has been out for 10+ years, millions of people have taken it, and I have yet to see one medically documented case that conclusively states Finasteride can cause these problems. Hell, look up "severe side effects" for any medication, and you will find pages and pages of people claiming terrible life changing side effects. Not to say that these people arent experiencing these symptoms, but whatever drug you are taking while experiencing them is usually the most convenient place to aim blame.
Again, i'm NOT saying that I know everything; I think that it is very possible for Finasteride to cause permanent sexual side effects. However, I would put it in the realm of 1 out of 10,000 people at most.