The Texas native, 64, was cleared for full Senate approval in a 56-43 vote.
The vote came after Senate Republicans changed the rules in order to approve Mr Trump's nominees for health and treasury, despite a Democratic boycott.
Mr Tillerson, who has never held political office, faced intense scrutiny over his ties to Russia.
As Mr Tillerson was sworn by the vice-president in the Oval Office, the president told him: "Your whole life has prepared you for this moment."
Mr Trump - who had just returned from a Delaware air base where he met the returning remains of a Navy SEAL killed in a weekend raid in Yemen, the first known US combat casualty since he took office - was sombre and subdued in tone.
America's new top diplomat thanked him and said: "As I serve this president, I will always represent the interests of the American people at all times."
The former oil chief forged multibillion-dollar deals with Russia's state oil company, Rosneft, and was awarded the Order of Friendship by the Kremlin in 2013.
In his Senate confirmation hearing, he admitted that the West had reason to be alarmed by Russian aggression, but he refused under questioning to label Mr Putin a war criminal.
As secretary of state, Mr Tillerson will handle relations with countries including China and Russia and negotiate matters such as climate change and human rights, on behalf of President Trump.The Senate voted largely along party lines, despite speculation that some Republicans would not approve his nomination.
Though some critics suggested Mr Tillerson would not be able to give up his corporate interests, supporters contended his background as a global dealmaker would bring fresh perspective to the role.
Mr Tillerson's confirmation was more contentious than those of his predecessors.
The Senate confirmed President Barack Obama's selections John Kerry 94-3 and Hillary Clinton 94-2.
President George W Bush's secretary of state, Condoleezza Rice, was confirmed 85-13 while Colin Powell was approved by a voice vote.
One of Mr Tillerson's first challenges may be allaying fears at the State Department over President Trump's order to ban travel or immigration from seven mostly Muslim nations.
Nearly 1,000 US diplomats and Foreign Service officers signed a memo of dissent on Tuesday regarding the travel ban.
Mr Tillerson will be tasked with restoring relations with those affected countries as well as outraged European leaders.
British Prime Minister Theresa May, fresh off a US diplomatic visit, opposed the ban on Monday.
Also infuriated is Mexican President Enrique Pena Nieto, who cancelled his planned Washington trip over Mr Trump's insistence that the neighbouring country would pay for a wall along the US border.
"He will start at a disadvantage and will have to play catch-up to build trust with his foreign counterparts and with State Department employees," said John Bellinger, a State Department legal adviser under Republican President George W Bush.
Also on Wednesday, Republicans on a US Senate committee pushed through the approval of Mr Trump's Treasury nominee Steven Mnuchin and Health and Human Services pick, Tom Price.
After Democrats boycotted committee votes on both nominees, Republicans changed the rules to approve the candidates without their votes.
Alabama Senator Jeff Sessions was also confirmed as Mr Trump's attorney general, despite heated exchanges between Senate committee members.
But Senate Democrats managed to stall a confirmation for Mr Trump's pick to lead the Environmental Protection Agency by refusing to show up.
The Texas native, 64, was cleared for full Senate approval in a 56-43 vote.
The vote came after Senate Republicans changed the rules in order to approve Mr Trump's nominees for health and treasury, despite a Democratic boycott.
Mr Tillerson, who has never held political office, faced intense scrutiny over his ties to Russia.
As Mr Tillerson was sworn by the vice-president in the Oval Office, the president told him: "Your whole life has prepared you for this moment."
Mr Trump - who had just returned from a Delaware air base where he met the returning remains of a Navy SEAL killed in a weekend raid in Yemen, the first known US combat casualty since he took office - was sombre and subdued in tone.
America's new top diplomat thanked him and said: "As I serve this president, I will always represent the interests of the American people at all times."
The former oil chief forged multibillion-dollar deals with Russia's state oil company, Rosneft, and was awarded the Order of Friendship by the Kremlin in 2013.
In his Senate confirmation hearing, he admitted that the West had reason to be alarmed by Russian aggression, but he refused under questioning to label Mr Putin a war criminal.
As secretary of state, Mr Tillerson will handle relations with countries including China and Russia and negotiate matters such as climate change and human rights, on behalf of President Trump.
