I wish we could get a dev from back then to comment on discussions about the N64 hardware at the time.
What happened to the N64 isn't really a mystery though.
Nintendo bet on cartridges and Silicon Graphics. CD's were slow so their assumption that their games needed faster loading times wasn't really crazy. Plus SGI was the market leader at the time so it made sense to work with them.
I know people like to dunk at Nintendo to go with CD's but if you look at what happened around the time they made that call with the SEGA CD and the likes, their position makes sense.
They strayed too far away from Gunpei's philosophy though, possibly because of the VB disaster and that meant that the Ultra 64 chasing the tech crown was only top of the line as the downgraded N64 until PC's overtook it about a year after it launched.
What mostly held it back was their weird ass ideas about Dream Teams and providing select developers with all the support of which only Rare turned out to be successful.
Sony made all the right decisions, the downsides of the PlayStation hardware were not severe enough to sink it and the advantages of the N64 were cancelled out with far better third party support.
Nintendo also underestimated how fast their high-end machine would age. In fact it aged so fast that they considered to launch a new 3DO/CagEnt MX console by Christmas 1999.