Disclosure: Not a huge fan of the new system. Think there are simpler ways to handle it. Absolutely don't want to lose the d20 if possible. I further hate the diminishing returns, and if we were going to be d100, I'd want it to be linear.
That said ....
You're not on the d20 anymore. You're faking it and shoe-horning it, but the system isn't d20. These issues are entirely around the abstraction layer your design goals have created. As my grandmother would have said (god bless her) "!@#@ or get off the pot".
By not being on the d20 but by faking it, when you apply each bit of modifier you're creating that abstracted value case. Things that modify before and after now have non-equal value. One part of the system is d100, the other is faked to be d20. A modifier on the d20 of 5% is different than one on the d100; definitely so if it adds to some reduced or diminished return.
You guys need to commit ... d20, d100 or hybrid. Recognize with each there are going to be people who don't like it. Recognize with each there's balance.
This hybrid thing where you're trying to straddle a fence or be half in/out of the pool means you have to balance to things on top of the cranky players who don't like change.
Commit ... and do it now before you have to rework too many things when **** breaks later. Turn AC and to-hits into d100s if you need to (with corresponding increases in feats, equipment, etc.). You may even want to consider fractional BAB, etc.
This balance mess is self-inflicted because you're trying to show something different on the presentation layer and rationalize that to the players versus what the system is really doing.