Tokayev immediately proposed changing the name of the Central Asian nation's capital from Astana to Nursultan, or "Sultan of Light" in Kazakh, and parliament approved the change within hours.
The senate also appointed Nazarbayev's eldest daughter Dariga Nazarbayeva as speaker, setting her up as a potential contender to succeed her father.
Tokayev, 65, will serve out the rest of Nazarbayev's mandate until elections due in April next year, though the former president retains significant powers in the country he ruled for nearly three decades.
Tokayev told lawmakers that Nazarbayev had "shown wisdom" by deciding to step down, a rare move in ex-Soviet Central Asia where other leaders have stayed in power until death.
"Yesterday the world witnessed a historic event," Tokayev said, hailing Nazarbayev as a visionary reformer.
"The results of an independent Kazakhstan are there for all to see," Tokayev said.
Nazarbayev changed the capital from Kazakhstan's largest city Almaty to Astana in 1997, transforming it from a minor provincial town into a futuristic city of skyscrapers rising from the steppes.
Its name meant "capital" in Kazakh and there had long been speculation of a renaming after the leader who shaped it.
The city is central to government propaganda highlighting the achievements of Nazarbayev's reign and his journey to build it was recently the subject of a state-funded film, "Leader's Path: Astana".
Nazarbayev, 78, ruled Kazakhstan since before it gained independence with the 1991 collapse of the Soviet Union.
He steered the country through a major transformation, developing huge energy reserves and boosting its international influence, but was accused of cracking down on dissent and tolerating little opposition.
Nazarbayev will continue to enjoy significant powers thanks to his constitutional status as "Leader of the Nation", life-time position as chief of the security council and head of the ruling Nur Otan party.
Tokayev appeared to be in pole position to take over in the long term until senators voted shortly after his swearing-in to name Nazarbayeva, 55, as their new chief.
She is the most politically prominent of Nazarbayev's three children and has long been mooted as a potential successor.
Kazakhstan's deputy prime minister from 2015 to 2016, Nazarbayeva has significant influence over the media.
Analysts said it was too early to declare a clear frontrunner to become the next elected president, with the recently named prime minister, 53-year-old Askar Mamin, another possible contender.
Seat warmer or contender?
Tokayev, the interim president, has a strong diplomatic record dating back to the Soviet period and has twice been foreign minister.
This should go some way to reassuring Kazakhstan's major partners including China, the European Union, Russia and the United States that the move will not threaten key relationships.
But analysts are in two minds over whether he has the attributes to take the job beyond the term that Nazarbayev would have served.
As Tokayev alternated between Russian and Kazakh in his speech in the parliament on Wednesday, he appeared notably more comfortable in Russian.
Independent political analyst Dosym Satpayev said that language skills are closely scrutinised by Kazakh speakers tired of the dominance of Russian.
"In terms of Kazakh language, we cannot say that Tokayev is as comfortable as the first president. This is an important issue in the long term as over 60 percent of the population is Kazakh and this demographic is expanding," Satpayev said.
But Satpayev also said that Tokayev's diplomatic skills had helped him forge a careful career path through an elite prone to clannishness and regionalism.
"He is capable of negotiating between different groups and taking different interests into account. He lacks popular recognition to some extent but he is a heavyweight in the bureaucracy."
The new leader will need to tackle growing discontent over falling living standards after Kazakhstan's economy was hit by the 2014 drop in oil prices and sanctions against Russia, a key trading partner.
Kassym-Jomart Tokayev took office in a pomp-filled ceremony less than 24 hours after Nursultan Nazarbayev, the only leader an independent Kazakhstan had ever known, suddenly announced he was stepping down.