So, pretty much, turning off the EXP share in X/Y doesn't actually pump up the difficulty.
All it does is make you lose more battles.

When is there going to be a legitimate difficulty setting in a Pokemon game?