Dunno what the big deal is; I've got two kids, so we're ahead of the average.
Things are definitely difficult though. Another bit which I don't see addressed is that work hours here are egregiously long, and exhausting. That's everywhere. My brother-in-law was working as a school manager; he was putting in 7-day-weeks, 10-11 hours a day for five years or more. Now he's a politician, and can be called on at any hour by his constituency.
In many Japanese companies, women are hired as "Office Ladies" -- basically secretaries, they make tea and do admin stuff, and many of them eventually meet their husband at the company. It's generally expected that they'll quit working and become a housewife after they become pregnant.
At my last company, there weren't many married guys. It was a game company, so there weren't even many women working in the office. There was no-one at the office, and no time outside the office to meet a partner.
It was depressing.
If anyone's wondering why the population is nosediving, I'd look at foolish, needless work-related exhaustion as a cause.