You're correct. We HAVE come to the core of the issue.
You feel entitled to it (the bonus) even though you didn't do the work (switching the item out).
I'm sorry to inform you of this, but you are NOT entitled to a bonus unless you are willing to do the work to get the bonus. Someone else stated it very clearly. If you want the bonus from smashing crates and other breakables, you have to smash crates and other breakables. You're not entitled to it. If you want to make veteran characters, you have to earn enough favor to build veteran characters (or buy it). You're not entitled to it. If you want that nice Greensteel sword you have to grind it out in the Vale. You are not entitled to it.
Similarly, if you want an extra 5% XP you must be willing to switch to the item before the end of the quest. You are not entitled to it.