What lessons from history can Keir Starmer draw on when choosing his chief of staff?

As Tony Blair entered Downing Street for the first time as prime minister in May 1997, a few steps ahead of him walked a new, powerful political constitutional creature: his No 10 chief of staff.

With the precedent established, every prime minister since has followed the Blair innovation in one form or another. Now Keir Starmer is searching for a candidate that could fulfil that role, leading preparations for power and then easing the transition to government if Labour wins.

Since Jonathan Powell, the chief of staff is the most important piece in the advisory jigsaw that potential prime ministers need to assemble. For Blair, the aim was a powerful group of political advisers able to provide the high-quality central direction and leadership from within Downing Street that was believed to be absent from John Major’s government. In addition, they were to address the perception that the government machine had absorbed the values and instincts of the Conservative government that had been in power for 18 years.

Of these advisers, Powell would never compete with Alastair Campbell as the best known of Blair’s inner circle. Indeed, Powell rarely gave interviews, and curious journalists initially even found it difficult to find photographs of him. Neither would he compete with Anji Hunter as the voice of Blair’s middle England political project. Nor could he hope to rival Sally Morgan’s grasp of the machinations of Labour Party politics.

Yet Powell brought something else: a knowledge and experience of government while Blair was in opposition, and a capacity to engage with the machinery of government and get things done for the prime minister throughout his period in office. This contribution meant he stayed the distance, outlasting all of Blair’s other senior advisers and having a central role in the Blair administration. He was security cleared, attended cabinet, and answered to the PM alone.

Of course, there had been powerful prime ministerial advisers in the past who broke with precedent as the powers of the PM have evolved. For example, the political closeness of Marcia Williams to Harold Wilson and her position in Downing Street was a source of friction with the civil service and other advisers. Lord Rothschild (the first head of the Central Policy Review Staff) under Edward Heath brought a creative mindset to long-term policy problems that Heath sought to solve. Bernard Donoughue, plucked from the London School of Economics, set up the Number 10 Policy Unit under Wilson and served Callaghan. The businessman David Wolfson had also served as an unpaid adviser with the chief of staff title under Margaret Thatcher, but with limited scope and powers.

Powell was different in the importance of the role he would fulfil for Blair and the combination of administrative and political support he provided. By background he was a career diplomat, recruited to Blair’s office in 1995 after initially meeting him in Washington in 1993.

Working at the UK embassy, Powell had spotted Bill Clinton’s potential as a presidential candidate during the Democrat primaries, and developed a strong relationship with the campaign team. The New Democrat model became of interest to New Labour, and Powell’s expertise was seen as an importance resource. Previously he had diplomatic experience of complex negotiations in Hong Kong and Germany.

Administration and politics were in his blood. His eldest brother, Charles Powell (pronounced Pole), was Thatcher’s civil service private secretary on foreign affairs, and a close, long-term confidant to her. Powell has acknowledged that as a civil servant he became too identified with Thatcher, and so left soon after she departed.

Another brother, Chris Powell, advised Labour leaders on advertising strategy and as a volunteer set up Neil Kinnock’s Shadow Communications Agency with Philip Gould, Peter Mandelson and Deborah Mattinson in 1986.

Once in place, from 1995 Powell oversaw Blair’s preparations for office. He led discussions with the civil service on the initial priorities of a New Labour government and facilitated the training of ministers.

While others were engaged with how to win the election, Powell’s immediate priority was to think about how to govern. An early flavour of what to expect came from his remarks to senior civil servants that the Downing Street centre of government was to be strengthened. A change from “feudal barons” to something more “Napoleonic”.

In fact, the role that Powell was to play once in government remained ill-defined at the start. Sir Robin Butler, the cabinet secretary, was concerned that Blair wanted Powell to fill the key civil service role of principal private secretary (PPS) to the prime minister. When this idea was scotched in favour of the chief of staff title, with Powell emphatically a political appointment as a special adviser (spad) rather than a civil servant, Butler continued to worry that his role potentially absorbed or duplicated aspects of the work of the PPS as well as his own as cabinet secretary.

Could this unsettle the “golden triangle” of the prime minister’s PPS, the cabinet secretary and the Queen’s PPS? Was the new and more powerful breed of Labour advisers a challenge to the permanent civil service and to their political neutrality?

In the end, a good Whitehall compromise was fashioned that gave Blair the more powerful Downing Street he wanted, while still protecting constitutional probity. Part of that arrangement was giving Powell and Campbell the power via an Order in Council to direct civil servants. It was the first time that political appointees could officially exercise this power, but not the first time they had done so.

Controversial at the time, it was not a power Campbell or Powell had sought and it was subsequently rescinded. The numerous chiefs of staff since have never complained that its absence has limited their ability to get what the PM wants done.

How the chief of staff role was integrated with the traditional prime ministerial support of Downing Street and the Cabinet Office continued to be a source of friction with subsequent cabinet secretaries, however; particularly with Sir Richard Wilson from 1998 to 2002. Numerous different structures were put in place or considered for Downing Street as Blair sought to create an effective central operation, combining political and policy advice and a centralised drive to implement reforms.

Powell’s role was not always well understood by the PM’s political inner circle either, particularly during the early phase of Blair’s premiership. Although accounts differ, criticisms were made of Powell’s political judgement and difficulties in managing friction with the cabinet secretary.

Powell feared on several occasions that he lacked Blair’s full support. The need to straddle the political and administrative aspects of a government can easily generate discontent on either or both sides. Those equally skilled at both are few and far between.

It was on Northern Ireland that the role of the chief of staff first came into its own. As negotiations started to open up, Powell emerged as the chief negotiator trusted by both the Unionist and the Republican sides, and played a central role in the team that delivered the Good Friday Agreement.

To make progress on decommissioning, he secretly travelled to Northern Ireland with no security protection at the request of Martin McGuinness for a one-to-one meeting. Given that the IRA had shot and injured his father and put his brother on a death list it was a feat of personal courage to do so.

He remained constantly involved throughout the process of putting in place the power-sharing arrangement, exploring back channels and being urged by Blair to find solutions to seemingly intractable problems. It is an important illustration that the chief of staff role must be able to act for the PM in addressing their priorities.

Yet the role also requires an important strategic capacity, contributing to the longer-term direction of the government. Here under Blair, the chief of staff’s importance would grow through Blair’s second term.

More dominant advisers had left Downing Street and greater focus was placed on policy development – particularly public service reform – and foreign affairs rather than communications. That was a strength for Powell with his policy and diplomatic background.

As the geography of power shifted within Downing Street, so too did Powell’s role in shaping the machinery of government by which so much either works or doesn’t. As principal private secretary to the prime minister between 1997 and 1999, John Holmes was clear about how the dynamic worked in Downing Street.

Holmes told Jon Davis and John Rentoul, the authors of Heroes or Villains? The Blair Government Reconsidered: “Jonathan was the boss, there was no question of that.”

Jeremy Heywood, who succeeded Holmes as PPS in 1999, became a highly influential and powerful part of Blair’s inner circle. It was Heywood who quickly took over from Powell in becoming the broker of government business with Ed Balls, as they managed the rock around which the Blair government had to be negotiated – the relationship with Gordon Brown known as “the TB/GBs”. Heywood, however, still reported to Powell as chief of staff.

Heywood’s successor as PPS in 2003, Ivan Rogers, did not have such an all-encompassing role and so Powell filled the gaps in capacity that emerged. As such, adapting the machinery of government ahead of a third term became a key focus for Powell.

This was a repeat of the situation ahead of the 2001 general election, when Blair exclaimed to Heywood and Powell: “I’m going to lose Alastair [Campbell] so now more than ever, I need Whitehall to deliver.”

In the weeks before the 2001 election, a new Prime Minister’s Delivery Unit was designed, based on a proposal that Michael Barber, an adviser in the education department, had drafted with Powell and Heywood.

The Delivery Unit was set up shortly after the election, with Powell and Heywood acting as midwives. Its accomplishments under Barber’s leadership were such that Blair remarked upon leaving office that not only was it a success but that it was “quite revolutionary, more so than we realised at the time”.

Though history does not repeat itself, there is value in learning about how to govern from the decade-long Blair premiership. Not least as competence is the name of the game once again. There is a big difference between saying and doing, and any credible opposition will feel that shift immediately on entering office. As the victory became clear on election night in 1997, Blair reflected: “Now we would enter a new and foreign land. I was afraid because I felt instinctively that its obstacles were of an altogether different order of complexity and difficulty.”

The political strategy of opposition – even of an election-winning machine – does not survive first contact with government. It must adapt. By 1998, Blair laughed at the modesty of New Labour’s first set of five pledges: “The challenge wasn’t meeting them. The challenge was: so what?”

Though, as chief of staff, Powell became a powerful operator of the machinery of government, there was discord. Powell has acknowledged: “I remember Tony complaining to me after … six months, that the government machinery was like a Rolls-Royce parked outside Downing Street which he wasn’t allowed to drive because he was still driving through our guerrilla operation.”

Prime ministers and special advisers must find the balance between activating the machinery of government and restraining the need to blame those who operate it. After complaining of “scars on his back” in 1999, Blair recognised by 2002 that his career in part depended on the civil service.

As Ed Balls has since reflected, a change of mindset and intellectual direction may be required after long periods of one party being in power; but that is the role of effective advisers – to mould the machine to a new political reality.

For any special adviser at the very centre of government, there are traits that will make them more successful. A savvy chief of staff entering Downing Street may want to learn from the extraordinary alliance-building skills of Anji Hunter; the instinctual understanding of how party and policy must be managed like Sally Morgan; the ability to be frank with the prime minister as Liz Lloyd often exhibited; the iron fist in the softest of velvet gloves of Michael Barber; the relentless focus and clarity of Alastair Campbell, and the hard reality check of focus groups diligently offered by Philip Gould.

As with all of the above, those advisers closest to a possible future prime minister Keir Starmer will have to be sure that they act on the basis of knowing the mind of their principal. If they say, “Keir wants” and it in fact is not what the PM wants, they will be found out quickly.

The chief of staff above all (bar the cabinet secretary of the day) will have to make endless decisions that sift the urgent from the important. Downing Street can easily become focused on the former to the PM’s detriment. With this in mind, senior advisers will have to remember the need for them to be responsive to prime ministerial instinct. If a PM has decisively changed direction, there will need to be swift adaptation.

It is the PM’s prerogative to focus not only on short-term wins but also on what comes next; constant renewal is the lifeblood of a successful premiership. To do this, a PM’s chief of staff and closest advisers will have to manage the resilience of their leader, and where possible find what Walter Bagehot called “mind in reserve” to think widely.

As the premiership rolls on, especially if successive terms are won, the skills which a PM needs to govern will change. A wise adviser will either change where possible, and where not, know when it is time to move on – and perhaps continue to provide advice from the outside.

Norman Tebbit, former cabinet minister in Margaret Thatcher’s government, lamented: “The windows in this building are very big when the prime minister first comes in, and every year they get smaller and smaller and smaller, and after 10 years it is very difficult for a prime minister to see the world outside as opposed to the world of No 10.”

As the windows of Downing Street shrink, and a bunker mentality looms, it is the function of the most senior advisers to be a lifeline. To keep the prime minister connected to the changing world outside, and thus extend their time on the political stage, as well as to offer careful guidance about how best to manage their exit.

History tells us that the clarity and coherence of the team around the prime minister casts the die of success much of the time. But in truth, the team can only be as successful as their prime minister allows them to be.

An effective prime minister benefits from retaining a lightness of character so that those closest of advisers keep telling the truth – a great strength of Tony Blair’s. They must also possess the self-awareness to recognise their flaws and capacities so that gaps can be addressed; another of Blair’s most powerful traits. An enduring lesson of the first term for Blair was his self-confessed hoarding of political capital. It may be the most valued resource in Whitehall, but political capital must be invested if the compound interest is to be gained.

And so when Keir Starmer is deciding who will be his chief of staff, he should remember that when in 1995, Blair chose Jonathan Powell, little did he know that Powell would be the last man standing with him in 2007. The leader of His Majesty’s opposition may be choosing the person who will be at his side for the duration of not just opposition but perhaps, if fortune favours Labour, an entire premiership.

Patrick Law is a PhD candidate at King’s College London, looking at the role of the Downing Street chief of staff and Jonathan Powell

Dr Michelle Clement is a lecturer and researcher in reform and delivery in government at The Strand Group, King’s College London