>>184387975-77 gr .223/5.56 ammo out of your AR should do the trick against deer. Pic related was killed by 75gr TAP .223 ammo and left a fuck huge wound cavity. Modern .223/5.56 is incredibly more powerful than the shit that was around decades ago.
If you want to go bigger to be on the safer side, buy an upper receiver in 6.5 Grendel or 6.8 SPC II. Pretty close in performance to .308 yet doesn’t require an entire new rifle.