It’s completely understandable to stop due to libido concerns. But you sound surprised that you lost a ton of ground. Of course you’re going to start losing what you gained through finasteride if you stopped taking finasteride. Now that you’re back on why would you want to take a break again? It’s illogical. Unless of course you have other concerns that are far more crucial than hair such as libido. Your libido is now through the roof likely due to the spike in testosterone levels caused by finasteride. Or perhaps your libido is just regaining normality from staying off finasteride after a few weeks, and you getting back on finasteride is a coincidence as far as timing goes. Stay on this time if you value your hair. However, if in the upcoming weeks and months you notice your libido taking a hit again then you know for sure it’s from finasteride.
The alternative route is to take tadalafil 2.5mg daily to counteract any potential libido loss while simultaneously taking finasteride. If you do indeed suffered loss of libido from finasteride the tadalafil will allow you a chance to see if you can ride it out, many times the side effect goes away on it’s own.
Side note: You could have at least replaced finasteride with something else to minimize your loss, such as ketokonazole 2% ed or eod.