Yeah you have got it right. You have to be very careful with oral minoxidil. At high doses (I've seen people say they take up to 50+ mg/day) my bigger concern would be sudden death, organ failure, brain damage, etc.
Consider though that a small amount of topical minoxidil will also systematic. So even with topical application, you'll be having minor systematic effects. So if you have a serious cardiovascular condition I'd stay away from even the topical stuff.
Personally, I'm taking a very small dose of oral minoxidil (just over 1mg/day). Long term, my goal is to get off of it once it's helped with regrowth, partially because of any potential negative cardiovascular effects.