Just join and don't heal anyone. When someone dies and starts yelling at you for not healing, tell them "Sorry, I was busy making room in my inventory for your stone. Just because I rolled a cleric doesn't mean I can heal, best to send a tell before a random invite next time.
"
If they all die, pick up their stones and proceed to solo the dungeon, looting every chest along the way and skipping bunches of mobs.
When they ask for a rezz, tell them "Sorry, I wasn't planning on grouping, so didn't bother memorizing a spell slot for a rezz."
This could be classified as greifing, and probably should be avoided, but it would be fun to do at least once.
Usually I just accept and continue doing what I was doing before the invite...I may even turn off party chat while I'm at it.