Have you tried simply removing the invisible wall?
My solution was to simply lob fireballs at him. Like so many other locations in this game they ignore the barrier. If you don't have the spell and can't UMD the scrolls the hireling Azala Fireshaper can do it.