If you start out at level 6, you'll only miss your bravery bonus on the first quest (and you'll still get full XP with no penalties) and you shouldn't have any problem getting through the entire chain on Elite. The last two quests really aren't at all bad once you know how to run through the traps in the final chapter. A full party that has been through the chain at least once or twice before shouldn't have a serious problem on Elite.
...and if you do have a problem getting through the quest on Elite, you can certainly go through on Normal or Hard. There's no law that says everyone has to do every quest on Elite.