>>3479991I took some high-speed video and I think I found the issue. When the shutter opens the whole lens assembly is twitching since its connection to the body must not be quite tight. Since it only moves when the shutter opens it would explain why it got worse at faster shutter speeds.
Oddly, this doesn't happen if I use the self-timer.