>>12881491G/S/C did this better, I think. There, your rival's a delusional cocky bastard, but after you beat him enough times, he finally realizes he can't beat you as he is and allows you to pass. He resolves to better himself and finally beat you some day.
R/S/E took it a step too far, but it was a fairly logical direction to take it. Your rival's pretty much destined to lose to you from start to finish, so a newbie trainer like Wally makes it comparatively understandable. His goal is to beat you. YOU'RE the invincible rival that Gary was SUPPOSED to be, and you're the single ultimate obstacle in his path. You're everything Gary wasn't.
Since then, Gen IV and V basically refined this method. Realistically, a rival will never be a very large obstacle to you, the player, because the rival exists to lose against you. Gary was a failure. He was supposed to be some amazing goal you were supposed to surpass, except you end up surpassing him before you even leave the starting line and he never catches up. Sure, he's always physically ahead of you, but all that really means is that he's a slightly faster walker than you, that's all.
