That's what you can expect from blind invites.
I've long ago stopped accepting blind invites on my cleric, just because you don't know
where it will take you.
And i would strongly recommend you to do the same.
I do enjoy pugging too, but not that kind of pug you described.
If you get into such a pug, speak up. You are the cleric, you are expected to heal people during fights.
Not that it's your job or your final duty, but people expect it from the party cleric and
you're good at it, so you can aswell use some of your SP to prevent them from dying.
But if they don't wait for you (i know how slow clerics can be) they can't expect you to do anything.
Tell them exactly that: "Wait for me or you might die horribly, as i couldn't even heal you if i wanted to, clerics are much slower than monk/ranger/barb splashes!"
If they do so, fine, seems they can at least read/listen.
If not, leave them, you won't have any fun running with them and all they'd do would
be complaining.
Clerics/FvS are welcome in almost every PUG, so you won't have trouble finding a decent one.