Most of what you hear about white men fetishizing Asian women is just Asian women projecting their own fetish for white men onto them.
Dating statistics show the vast majority of white men are with white women and have no interest in Asian women; in contrast Asian women disproportionately prefer white men to the exclusion of all other races.

Among Asian women born in the U.S., there are more of them cohabitating with white men than with men of their own race. Many Asian women have a "white only" dating policy that is eerily reminiscent of the dating habits they accuse white men with yellow fever to have.

By all reasonable standards "yellow fever" is barely a fringe phenomenon whereas nobody adresses the elephant in the room, namely "white fever".