Look at how poorly the "Wii U" sold after Nintendo abandoned their core audience to chase the Wii fad.
A big problem with the Wii U was that it was not targeting the Wii crowd enough. A lot of what sold the original Wii was the bundled Wii Sports, something groups of people could play together at gatherings, while showing off what the unique controllers could do, allowing the console to become popular through word of mouth. The Wii U didn't have that. There was no new Wii Sports title even available for purchase on the platform until they started trickling out Wii Sports Club minigames, sold individually through their online marketplace, a full year after the console's launch. The basic version of the console, which was priced $50 higher than what the Wii launched for, didn't include any bundled game whatsoever. For yet another $50, the premium bundle added some more internal storage and a copy of Nintendo Land, but that mini-game collection didn't really have the appeal for "non-gamers" that Wii Sports had.
And on the hardware side of things, the Wii U didn't even include a Wii-style motion controller in the box, instead expecting people to use existing MotionPlus controllers from the Wii for any games that required them. Its main gimmick was a large touchscreen, but it came at a time when tablets were already doing that better in many ways, and at lower prices. It wasn't a standalone portable device either, with the touchscreen controller needing to be within range of the main console, and often games required an external screen to function.
The marketing had major issues too. With a name like Wii U, it wasn't clear to consumers whether it was a new console, or just some kind of expensive touchscreen add-on for the aging Wii, leaving many unsure of what it was intended to do that their existing Wii and tablet/smartphone didn't already do. Announcing that they were discontinuing many of the Wii's online services right after the Wii U launched didn't instill much faith in the longevity of the new device's services either.
The Wii U had a lot going wrong with it from many perspectives even before it launched. It might not have really been the gaming device long-time Nintendo fans were hoping for, but it wasn't really targeted properly at the wider Wii audience either.
As for exclusively targeting the "core" Nintendo audience, Nintendo tried that with the Gamecube, and that console also sold poorly. Nintendo ended up having to officially drop its price to $99 (half its launch price) not even two years after the console came out. Actually, up until the Wii, Nintendo's console sales had been decreasing every generation since the NES. The NES sold around 62 million units worldwide, the SNES around 49 million, the N64 around 33 million, and the GameCube under 22 million. It's clear that Nintendo's console sales were on a steep downward descent, and something had to change to keep them viable in the console space. Then, the Wii came along, and managed to sell over 100 million units, making it Nintendo's best-selling console hardware ever. The Wii U obviously didn't sell well at around 14 million units, but looking at Nintendo's console sales trends prior to the Wii, those numbers were right in line with what might have been expected of a GameCube successor.
With the Switch, it's a bit less clear, since that device has effectively taken on the role of both their console and portable devices, and their portable devices have always tended to sell well. Combining the two platforms does seem like a good way to allow them to concentrate their resources in one place though.