Your loot is your loot.

People have the right to decide to group with you or not based on these types of circumstances.

Both of you are correct in your stances.

The person who had the fit should have silently dealt with it instead, and then not grouped with you afterwards.