The vampire is just using a typical dominate spell, and Mindflayers use psionic dominate. Both use these dominations to make the player stand still, rather than attack their teammates. This is evidenced by the fact that the vampire usually accompanies a dominate attempt with the statement "You, stand there and make no actions". So, theoretically the vampire / mind flayer could make the player attack his comraders like any other charm spell, they just don't.