I think that making it go away after a few seconds is a good idea. I'm trying out a Bardbarian on Risia and I used Rage a few times in Misery's Peak. I noticed a significant decrease in brightness after raging and it made it a bit easier to to get turned around or miss a ledge. It's especially bad in that quest where it's a bit difficult to see which steps are just steps and which are a shortcut to the bottom.
I also had trouble telling colors through the red. That could be really bad in some quests and make some puzzles impossible to do.
Now Barbarian rage isn't such a problem. At low levels it doesn't last too long and you can always dismiss it. The real problem would be with the spell. There's no way to cancel that.