My wizard self-buffs to 517 HP and still wears heavy fort. Hit points are irrelevant. Heavy fort is an essential gear slot for every class. You can't just say "oh, since I don't have aggro and I'm not in melee range, I don't need HP or fort." Because when Murphy's Law happens to your party, things like that will determine whether you become a soul stone like everyone else, or whether you're able to save the quest.
Reminds me of a cleric in an Epic VoN6 who didn't have fort (or Quicken)...you know how useful he was to the party?