I feel like concentration should apply when you are trying to use a lever or open a door and some archer is pinging you from across the room. If you can cast a spell while something is throwing rocks at your head why can't you open a door or press a button? Or change clothes... or recall for that matter.
I know the first instinct is to yell that it would unbalance things, but that is actually not true. Its not a feat, its a skill. So those skill points have to come from somewhere.