Not true.
Most of the so called 'decline' came number wise from several missteps. None of the more than dozen guild members from my guild moved on because they thought things were too casual. The challenge thing is purely driven by elite players usually completionists that have all the gear and goodies and want something to challenge their incremental power. It's not driven by your average player who don't fly through or play EE. Simply not true.
I bet that raid bypasses have had more impact on the day to day player population and the lack of a upper cap and goal than anything. When you got everything and you can blow through content 20 runs in 2-3 days for the 20th there's really no longterm stability. Most of the base are now running content through guild and a small friend list which hurts average gamers play. If anything the xp group event is a resounding success since it expands upper end players to invite anyone instead of their regular small clique.
Feel free to ignore that dynamic and think that if you just put the thumb screws on and drag everyone through glass there's going to be a magic influx of players. To me the answer is really simple. Reintroduce a long term goal for everyone and things like xp grouping events to have the entire base from top to bottom enjoy gaming. The challenge piece you're talking about only effects a small vocal clique. Most others simply see very little point running on empty LFMs with little group dynamic and having already accomplished most things they set out to do.
Personally I look forward to E shroud as a longterm goal and I hope that Turbine nukes the raid bypasses for all new raids for at least 6 months. If they don't E shroud will be another fail event where everyone bombs through it in a month or 2 and then it's completely dead.