Some say a second-class mind makes for a first-class leader, others that madness is an essential feature of the role. From Trump and Obama, to Blair and Boris Johnson, which personalities are born to rule?

There is a story that often gets told about modern presidents and prime ministers, and sometimes gets told by them as well. The politician spends half a lifetime working tirelessly towards the top job, with the goal of making a real difference once he or she gets there. They issue their instructions. Dutiful officials nod along encouragingly. But nothing really changes. Once the door to the Oval Office or No 10 closes behind them, and they settle their feet under the desk, the new president or prime minister finds out that it’s just another room and just another desk. It feels as if true power is still somewhere out of reach.

In politics you should never assume that there is a pot of gold at the end of the rainbow. It’s better to know how little is waiting for you, like a weird inversion of the parable of The Wizard of Oz. In place of the Yellow Brick Road is the greasy pole, which has to be ascended to reach the Emerald City. Yet the successful climber finds that his or her fate is not to encounter a shrunken wizard at the end of it. Instead it is to become that person: the impostor behind the curtain.

How do politicians react when they discover themselves in that position? Some, like George W Bush, never quite acknowledge it. Others, like Tony Blair, decide to do something about it. Blair concluded that he had to build the machinery that would enable his administration to deliver on its ambitions. He called this instrument “the delivery unit”. It was designed to make sure that the levers in Downing Street were connected to the rest of government. Yet even after 10 years in power, Blair was frustrated with how little he had managed to achieve. One reason he was reluctant to leave office at the end was a nagging feeling that he was only just beginning to get the hang of it.

Real power still felt out of reach, somewhere over the horizon. Since quitting frontline politics, Blair has made himself into a salesman for the idea of “Deliverology”, which promises to help politicians around the world with the problem of actually getting things done. It’s a pretty threadbare prospectus. Perma-tanned and increasingly wizened, Blair cuts a tawdry figure these days. Here is another version of the morality tale: the wizard has spent so long behind the curtain that he doesn’t realise how diminished he appears when he steps out in front.

Despite her reputation for knowing her own mind, Thatcher was surprisingly fragile in her confidence and scatty in her convictions

There are other ways of responding to the deficit of power at the summit of politics. The truly paranoid politician believes that the reason the levers are not working is that someone has cut the strings. Faced with the frustrations of office, it is always tempting to imagine that there is a conspiracy at work to prevent meaningful change. Blame it on the “deep state” – or, in the politer British version, on “Sir Humphrey”. Probably no one is immune from this suspicion, especially in the long reaches of the night. Most politicians have had moments when they believed that dark forces were at work to prevent them getting their way. But only one has turned this belief into his governing philosophy. Donald Trump’s response to any setback is to claim that he is the victim of a deliberate attempt to subvert his authority. He cannot accept that there are inherent limits to the power of his office. So any manifestations of those limits become further evidence of the conspiracy against him. These are the dangers of electing a narcissist to an office that is not as powerful as it seems.

That said, it is not true that none of the levers work. Some do, all too well. Another temptation is to keep pulling until you find one that produces a direct response. Inevitably, for both presidents and prime ministers, this tends to be the lever that links to the armed forces. Again, Blair is emblematic here, but he is far from unusual. Chastened by his inability to get traction with his domestic agenda, Blair latched gratefully on to the opportunity presented by 9/11 to turn himself into a player on the international stage. It wasn’t just that he wanted to put the world to rights according to his own lights. It was also that he was able to do so, more easily than he could put his own government right, because the instrument to hand was military force. The same was true of Bush. But it has equally been true of politicians as otherwise different from each other as Margaret Thatcher and Bill Clinton, Barack Obama and Donald Trump. The power of the office they hold is least constrained when it comes to taking action abroad. That is why presidents and PMs, who almost never get elected on a foreign policy platform, often find that foreign policy is what ends up defining their tenure at the top. The international arena is where they can make the biggest difference, for better or for worse.

The trauma of 9/11 – and what followed – reveals another fundamental truth. The power of presidents and prime ministers is hugely dependent on the accidents of history. For Margaret Thatcher, the Falklands warcompletely altered what it was possible for her to achieve in office. Without Argentina’s invasion of April 1982, over which she had no control, her premiership would have been very different. For Gordon Brown, the financial crisis of 2007–8 reconfigured what he could accomplish, as it did for Obama’s presidency. Such unforeseen events do not suddenly endow presidents and prime ministers with superpowers: the frustrations of trying to get things done remain just as intense (as both Brown and Obama discovered). But they do provide an opportunity to break out of a rut.

It helps to arrive in office knowing who you are. Obama, always poised and self-contained, knew his own worth full well. Yet he accomplished less as a result, because he never really pushed the limits of what he could achieve. Thatcher, despite her reputation for knowing her own mind, was surprisingly fragile in her confidence and scatty in her convictions. Yet she understood, with an almost uncanny instinct, what someone with her personality could achieve as prime minister, especially when the occasion arose. Blair was far more self-confident and sure of what he believed than Thatcher was. But it did him little good. He persisted in trying to make the office of PM suit his convictions, rather than trying to adapt those convictions to the limits of what the office made possible. Bill Clinton was perhaps the most intelligent man ever to occupy the Oval Office. His mind was voracious in its appetite for new information and fresh insights. It was too much. He couldn’t contain the intelligence he had within the space of the role he occupied. It spilled out, and too often it went nowhere.

It is sometimes said that it doesn’t do for a president or prime minister to be too smart: a second-class mind is more likely to make for a first-class leader. Like many such generalisations about politics, there’s some truth to it, but also plenty that it misses. There are lots of different ways for a politician to be intelligent, and there are many things that politicians can know which are unknown to anyone on the outside. What matters is whether they know what it is that they know. And if they do, whether they understand what that knowledge is good for.

This brings us to Trump, who in many ways exemplifies the idea that the personality of the politician reveals the character of the office he occupies. His persona is not going to change. He is disturbingly consistent in what he has always been: showman, chauvinist, charlatan. What he is doing is testing how far a man like that is able to push the boundaries of what a president can be. He has been more successful than many people believed was possible. His willingness to say anything – and possibly to believe anything – in order to get his way turns out to be a surprisingly effective means of maximising his authority. Given that a majority of Americans revile him, he has done quite a lot with the limited power he has. Perhaps he too bucks the fable of The Wizard of Oz. He simply refuses to acknowledge the existence of the curtain. He wants people to see!

Donald Trump in Cincinnati, Ohio. Photograph: Andrew Spear/Getty Images

What makes Trump so unnerving, however, is his seeming lack of any self-knowledge. He is not really probing for the limits of what the presidency allows, because that would require him to accept that there are limits. He does what he does regardless. Trump is both more and less than a president should be. More, because he is behaving as though his power were truly as he believes it to be. Less, because he is also behaving as though the presidency were just another job (businessman, reality TV host). Much of the time he does not seem to appreciate where he is. Why would someone whose personality is so fixed be so unpredictable in office? Because that personality makes him incapable of seeing the presidency as its previous occupants have seen it, as an office that comes with certain expectations of how to behave. Trump is, in institutional terms, unhinged.

Does that mean he is mentally unfit to be president? During his time in the White House Trump has been dogged by repeated rumours about his intellectual incapacity – he has been variously described as a “fucking moron”, “like an 11-year-old child”, a “dope”, an “idiot” and “dumb as shit”. And that’s just how his former aides and colleagues speak about him. Plenty of psychiatrists have pushed back hard against the so-called “Goldwater rule”, which prevents them from diagnosing the psychological failings of public figures at second hand. For many, this injunction bars them from simply stating the obvious: Trump is out of his mind.

While in the White House Trump has been dogged by repeated rumours about his intell­ectual incapacity

But we should be wary of assuming that this is enough to place Trump beyond the pale of conventional political leadership. Blair was widely thought to have lost his mental bearings during and after the Iraq war. Brown was pursued by stories of titanic rages and prolonged depressions. I have spoken to people who believe that the true story of Theresa May’s premiership has been suppressed, that as a type-1 diabetic she faces cognitive handicaps, including an inability either to process new information or to change her mind. These kinds of accusations come with the territory: presidents and prime ministers are often thought to be psychologically undone by office. It is one of the ways we express our discomfort with anyone aspiring to that kind of power.

Max Weber, writing 100 years ago, made the case that the risk of madness is not simply an accidental byproduct of high office. It is an essential part of it: a feature, not a bug. Presidents and prime ministers have to deal with the mental strain of bearing enormous political responsibility without necessarily having the personal authority to match. The leaders of modern states hold the lives of millions in their hands, and yet they often can’t even get the people in the next room to do what they want. It might make anyone a little crazy. Leadership is a constant tug of war between the rules of political accountability and the law of unintended consequences. That has not changed in the time since Weber wrote. One reason it is so hard to be a president or a prime minister is that the voters hold the wizard responsible for what happens. Even though the wizard is just the impostor behind the curtain.

By far the sanest president or prime minister of recent times was Obama. He went out of his way to maintain an even keel, even in the face of the most outrageous provocation. He made sure that he stayed connected with his family and that he got enough downtime. Is it possible to be too sane for the presidency? Certainly there were moments when Obama’s insistence on keeping his cool looked like a missed opportunity. Sometimes one longed to see him let rip. But that was not his style. Nor was it what got him elected.

Obama and Clinton pictured in 2008. Photograph: Michal Czerwonka/EPA

This is the other deep tension that resides in the character of anyone who pursues the highest office. The personality traits that can win you the crucial election may not be the ones that suit the role to which you have been elected. Campaigning, as the slogan goes, is not governing. Obama the candidate was known to his inner circle as “‘No drama’ Obama”. During his long, bruising, underdog campaign in 2008 to wrest the Democratic nomination from Hillary Clinton, and then for the duration of the shorter but equally high-wire act needed to win a general election that was taking place as the world economy was having a heart attack, his level temperament was a golden asset. Refusing to get ruffled got him over the line. But when he became president he needed other skills too. His preference for cool analysis over impulsive decision-making and his insistence that he would not be baited by his baying opponents – as his wife, Michelle, memorably said in 2016, “when they go low, we go high” – were still assets. But they were not enough.

Personal development is very difficult over the course of a political career, especially when being who you are has made you what you have become. Why would political leaders whose approach to politics won them office abandon that approach once they get there? Trump stormed to the Republican nomination and ultimately the presidency in 2016 with a scorched earth campaign that recognised no limits and took no prisoners. He defeated his rivals by mocking them, belittling them and, when the opportunity arose, lying about them. That is how he has carried on conducting himself as president, to continuing howls of outrage from his opponents. But who is to say he is wrong? If high office doesn’t change who politicians really are, we shouldn’t expect them to change how it makes them behave.

Some politicians reach the top without having to win an election to get there. When May faced a general election in 2017, by choice rather than by necessity, the skill set that had got her to Downing Street let her down. The tenacious, colourless, undeviating politician who tiptoed over the corpses to inherit the crown in the aftermath of the Brexit vote morphed on the campaign trail into the deathless Maybot: cold, mechanical and seemingly without a personality of her own. On the stump, Jeremy Corbyn, a politician for whom campaigning has been the lifeblood of his entire career, ran rings round her.

Qualities that we once appreciated can become loathed: Blair’s sincerity came to seem like sanctimony, while Obama’s coolness turned into aloofness

Change is hard for politicians. But what makes political leadership so precarious is that it is relatively easy for voters. If we get tired of our politicians, we can get rid of them. The qualities that we once appreciated in a leader can turn surprisingly quickly into what we loathe about them. Blair’s sincerity came to seem like sanctimony. Obama’s coolness turned into aloofness. May’s steadfastness made her come across like a robot. Sooner or later Corbyn’s simplicity will come to appear like idiocy. Another truism of democratic life is that all political careers end in failure. Perhaps it would be easier to say that no political personality is built to last at the very top. The demands of the highest office are never set.

Yet if there is one quality that is indispensable for anyone at or near the summit of political life, it is stamina. This does not have to be physical stamina, though that helps. Corbyn has a reputation for being a lazy politician. He rarely strays outside his comfort zone. But there has been nothing lazy about his political career. As a backbench MP he stuck it out for decades, even when he had very little to show for it. Corbyn hung in there, where others might have given up. As a result, he was still around when his chance came.

The British and US political systems are different, so the kinds of chances they throw up for resilient politicians will be different too. The presidency is open to genuine outsiders in a way that the prime ministership is not. No one could reach No 10, as Trump reached the White House, without ever having stood for election before in any capacity. A presidential campaign, running over two or more years, with its relentless requirements for fundraising and attention-seeking, makes distinctive demands. It suits personalities as different as Obama’s and Trump’s – the one unflappable, the other unembarrassable. It would not suit May. British politics is more self-contained, its campaigns shorter and the role of its parliament in selecting political leaders much more pronounced. Corbyn, in turning Labour into a mass movement that downplays the role of its elected MPs, is as close to an outsider as British politics has ever had near its summit. Yet he has had no career outside parliament.

Trump’s stamina remains an underrated weapon in his political arsenal. He didn’t just outfight his opponents in 2016, he outlasted them, withstanding setbacks that would have felled a less resilient candidate. His appetite for gruelling speaking engagements has not diminished since he won the presidency. In fact, he seems happiest when he is hammering away at the podium. He has chewed up and spat out a remarkable number of staffers in his relatively short political career. He seems to measure his political success in large part by the number of bodies he can pile up at his door, regardless of their prior allegiances. By that measure, he may be the most successful president in modern American history.

If Boris Johnson wants you to do something, he’ll grab you by the arm. He's a charmer, a wheedler, a flatterer, a bully

So where does Boris Johnson fit into this story of high ambition and mismatched expectations? Johnson can have few illusions about the limits of the power of the office he now holds, given that his predecessor’s tenure was a case study in exposing them. May arrived in Downing Street promising both to deliver Brexit and to remedy some of the social injustices that stoked the divisions which lay behind it. She failed to do either. Instead, she spent three years searching in vain for the tools that would get others to comply with her wishes. She couldn’t find them, no matter how doggedly she looked. Calling a general election to supply the missing leverage over her colleagues in parliament only made the situation worse. Yet here is Johnson, making exactly the same promises and threatening exactly the same remedy: do as I say or the voters will tell you! It’s as though the last three years never happened. What does he know that she didn’t? What does he have that she lacked?

It can’t be true belief, which Brexiters sometimes say was the crucial ingredient missing from May’s prime ministerial portfolio. Yes, she campaigned for remain whereas Johnson led for leave. But he is also the man who wrote two columns, one for each side of the question, each serviceable depending on which way he decided to jump. He is not a conviction politician. He is a jobbing journalist, for whom convictions are simply the touchpaper to ignite this week’s copy.

Nor can it be greater political experience or political skills. What Johnson does have is a certain bearish presence: he enjoys the rough and tumble of politics and has no problem with physical intimacy. If he wants you to do something, he’ll grab you by the arm or pull you by the ear. He is a charmer, a wheedler, a flatterer and an occasional bully. Johnson appears to see these qualities as a substitute for hard work and a grasp of detail but he is not going to get his (or any other) version of Brexit through parliament by grappling his colleagues over the line.

Perhaps the new prime minister thinks that he has finally assembled in Downing Street what Blair sought but never quite achieved: a delivery unit that can actually deliver. It’s a one-man unit named Dominic Cummings, the man who delivered victory for Vote Leave in the referendum. However, winning the referendum is very different from delivering Brexit itself. Cummings may yet find a way to engineer another victory at the ballot box. But the bureaucratic grind of forging a lasting settlement looks beyond him. Johnson’s delivery unit is in the business of getting him through the next few months. What happens after that is anyone’s guess.

Johnson is behaving as though Brexit were just another foreign policy challenge, giving him the same licence his predecessors have sometimes enjoyed when taking on foreigners. This is international politics, so he doesn’t see why he has to stand on ceremony. He can throw his weight around, unencumbered by democratic niceties. The May government, according to this version of events, was far too worried about creating the right impression. But Brexit is not just another foreign policy challenge. It is a legal, economic and diplomatic nightmare.

No amount of sub-Churchillian rhetoric can change the fact that a successful exit strategy is almost entirely dependent on the goodwill of others, ranging from Emmanuel Macron and Angela Merkel to the Irish government and the European Commission, not to mention the foreign currency and bond markets. Treating it like a war doesn’t make it one. It just makes it more likely that the limits of the power of the prime minister will once again be exposed.

In the end, it is hard to avoid the conclusion that the only game-changing asset Johnson has that his predecessor may have lacked is a willingness to embrace his inner Trump. What Johnson brings to the table is an appetite for creative destruction: not testing the limits of the office he holds but hoping to ignore them altogether. He is relying on energy, animal spirits and a disregard not just for the conventional wisdom but for the rules of the game to carry him through. If he succeeds, it will be a clear indication that the rules are indeed changing. The story of modern politics – of presidents and prime ministers frustrated by and chafing against the limits of their power – will have a new chapter.

In many ways, Johnson is not Trump. He is more of a hybrid politician, whose character contains bits of the past that cling to him like debris from another age. He tries to ape Thatcherite resolve, Blairish charisma, Clintonian chutzpah about his personal indiscretions. But being Thatcher or Blair or Clinton won’t get Johnson out of the Brexit mess that he helped to create and which he has now inherited. If he finds a way through it will be because his obvious weaknesses and his disregard for the truth turned out not to matter. Like Trump, he will have pulled back the curtain and got away with it.

Can this approach succeed? I do not know what comes next, any more than Johnson does. But if the result of the turmoil of the past few years and of the months and years to come is that we end up with less faith in the power of presidents and prime ministers to make all the difference, that will be no bad thing. Maybe what comes after the myth of the strong leader is the idea of leaderless democracy. There are far worse ideas.

When Thatcher, the closest Britain has had to a strong leader in modern times, died in 2013, many of her diehard critics celebrated on Twitter by announcing: “Ding, Dong! The Witch is dead!” But it’s not the return of the Wicked Witch we should fear. It’s the revenge of the frustrated Wizard.

THE GUARDIAN