It never dies. But it can miniturize a bit. Many bald guys thin on the sides too, just not nearly as bad as the top.
If your grafts disappear, that means your entire head would be bald, including the sides. You'd be like NW10 or something. But no one goes that bald. Most likely your grafts will still be there.