There were a bunch of factors.
-Hasbro saw "popular Japanese monster franchise" and assumed it'd be the second coming of the Pokemon fad just on those merits, leading them to vastly overproduce the first wave of merchandise. This fucked over distribution for later waves and resulted in the early stuff getting clearanced (making stores even more hesitant to carry the line).
-Nintendo of America shat the bed with the first game's marketing, mostly focusing on the one fart joke yokai rather than everything else the game has to offer. This wasn't helped by Smash autists screeching every time Nintendo tried to promote the game because they wanted to know the DLC ballot results, or Pokemon fans immediately perceiving any monster-collecting game other than their own as a direct threat.
-For the second game, they copied the Japanese release's three-version approach over two years rather than consolidating them all into one title. They learned from this mistake for YW3, but the 3DS is dead now so it's too little too late.
-The show only aired on Disney XD, but didn't even have the benefit of the mouse house's marketing juggernaut to go on.
-The toyline chose to focus almost entirely on the cute/silly side of the franchise rather than trying to highlight some of the more "cool" characters. We got about 500 different nyans and komas but not a single Kyubi or Venoct. Abandoning the transforming toy idea and the sofubis in favor of making a single figure line with a wider pool of characters would've helped the toyline catch on with more kids.
