Unlocking challenge mode post E4 actually makes for tougher fights. If you have CM on the whole way, you're getting experience points from the higher levels and fuller parties. When you get it post game, you don't have that advantage. The post-game level bump + challenge mode level bump made for a really fun experience. I am sure they didn't intend it, but it worked out alright for me
