Oh I can totally see the perspective of that, no question, if you wanted to main tree Sacred Defender, that is the way to go. And it is sound logic in that scenario.
I am not going further into Sacred Defender than 13 AP. I need all of the extra AP that I can get in Drow and Monk Trees, and Vistani if I swap to Daggers. For Daggers + Sacred Fist that is minimum 24/23 AP in those trees, locking out full investment in Sacred Defender. If you go mostly Sacred Fist and Drow trees and pickup nice stuff from Monk, same deal, you are not going to have the AP for getting to 31 AP in Defender anyway.
It is a totally different discussion about how it may be possible to achieve higher defenses with 13 AP in Defender, and Embracing the Zen of the Monk lifestyle. Versus 31 AP in defender with just 1 Monk level, and ignoring the Zen entirely, but perhaps going into Unyielding Sentinal for even more Lay on hands and recharging there. There are some arguments to be made either way.
I am embracing the Zen and with my setup, and I totally do not need the lay on hands outside of emergency situations.