This has been done in administrations past.
Actually, it's been fairly common since the creation of the National Security Council and the effect that has had on the Secretary of State. We basically had a figurehead Secretary of State from 1961-1973. Rusk was basically ignored by the Kennedy's who chose to do foreign policy themselves and through a trio of undersecretaries (JFK actually wanted to eliminate the State Department) and McGeorge Bundy, LBJ initially started to bring him in and then came to the same conclusion that he wanted to run foreign policy himself and then Rogers was shoved aside as Kissinger played to Nixon's personality better and eventually Nixon just replaced Rogers officially.
National Security Advisor doesn't have to be approved by the Senate unless you appoint a General and he wants to stay in the military. Brzezinski, Anthony Lake, and both the Rice's also were arguably closer to the President and had more influence on foreign policy than the sitting Secretary of State. (Cyus Vance, Warren Christopher, Colin Powell and John Kerry)
Kerry was arguably unique in that he found a way to work himself out of this situation and play a key role late in the administration. The last two administrations also had Vice Presidents who played fairly prominent roles on foreign policy decision making in Cheney and Biden. Reagan also leaned on H.W. Bush more as his administration went on.
Speaking of some of those names, Cheney, Rumsfeld, Ford and H.W. Bush tried to collectively box out Kissinger in a similar way because Ford was too afraid to fire him.
H. W. Bush's failed appointment of John Tower to Secretary of Defense has been considered somewhat similar, as Bush, Baker, and Scowcroft were going to run policy anyway and it was assumed that Colin Powell was going to be the effective voice at the Pentagon. But when Tower was canned by the Senate and Bush turned to Cheney, Cheney was a hardliner about civilian control so started a decades long feud with Powell instead by telling him to be quiet unless asked.
In theory, I'm not sure what prevents a President from not appointing anyone to a Cabinet post or other Senate confirmed position if they don't mind the Acting person. In Bill Clinton's second term he had an acting Secretary of Veteran Affairs for like half of the term. Elaine Duke was acting Secretary of Homeland Security for like six months last year. Obama had Rand Beers as acting Secretary for about three months in 2013.
