I suppose part of the loss/gain analysis depends on what you're taking with those four other levels. Since you mentioned unarmed damage, I guess you're planning on going something like 12 monk/8 fighter or 12 fighter/8 monk.
Ignoring for now what your other levels would be in this multiclass character, logic simply dictates that Fighter 12 > Fighter 8. With Fighter 12, you unlock Kensai II which is a very nice enhancement. You'd get:
- Power Surge (+8 STR for 60 seconds)
- Kensai <insert weapon here> Mastery II (+1 to hit, damage and +2 to confirm crits and critical damage, according to the compendium)
Other nice enhancements you would receive include:
- Fighter Haste Boost IV (at lvl 10)
- Fighter Extra Action Boost (at lvl 11)
- Fighter Strength III (at lvl 10)
- Fighter Toughness IV (at lvl 10)
You also get two more class feats, one of which must be Greater Weapon Specialization (+2 damage).
So it looks like going Fighter 12 gives you +5 to hit (+4 Power Surge, +1 Mastery II enhancement) and +8 to damage (+4 Power Surge, +2 Mastery II, +2 Greater Weapon Specialization). Also, if Fighter Strength III puts you at another even number, those numbers go up by +1. You also can't overlook the other enhancements, particularly Haste Boost IV and Extra Action Boost, both of which are very nice to have.
I may have missed something. Regardless, I hope this helps.