I don't have a huge problem with the shield bonus, but I think it should not be all or nothing. Let those not using a shield and weapon get a 50% of the bonus.
The cooldown should go though. The stalwart is weakening his defenses to swap to a scroll but he should not have to wait a full minute after switching back to his sword and shield to regain defensive benefits.