Women - even mid-20's girls - definitely notice muscle/build. I'm 6'1" and naturally broad, and my entire life people always asked me if I was a bouncer or a police officer. Since I've been working out the past 2.5 years, and added some more size (nothing crazy I don't think, but I'm very well-proportioned with high-set, broad shoulders and a prominent chest), many, many girls have commented on my build. They'll either say "wow, do you work out?" with a flirty voice, or just come up and feel my chest sometimes.
One hot early 20's girl comes up to me at this rooftop bar last year as I was just standing there and feels my shoulders and then my chest and goes "wow, are those real?" She was well-endowed herself, so I said "yeah, all natural - how about yours?"
It does help that I'm attractive guy facially, of course, but even though girls always were attracted to me, they more frequently make comments about my physique now. The point is that girls do notice. Once you get past a certain point, however, I'm sure they're turned off by it (as someone said above, they'd prefer the Hemsworth physique over that musclebound guy in the pool video above, as would I), but 95+% of people don't have the genetics or inclination to get to that point anyway, so don't let that stop you from working out. The amount of comments/respect I get from other men (even men my size or larger than me) is pretty absurd, too, especially since I don't think I'm that big. But my proportions are good, which is what I think people notice.
Working out and seeing physical improvements in yourself helps you psychologically/chemically in a whole host of other ways, and improves your confidence overall. I wish I started when I was 20 years old instead of when I was 36 lol.