The Richmond Park By-Election in Perspective

Introduction

In the course of writing various analytical pieces on Lib Dem electoral performance in recent years, I’ve become rather used to ending up as a voice of doom and gloom. “You’ve rather cornered the market in depressing Liberal Democrats”, observed one colleague. This is slightly different. It’s actually a welcome change of pace to be the bearer of good news. For the Liberal Democrats, the Richmond Park by-election was an unambiguously good result. In an age where expectations are constantly being managed up or down, this pamphlet seeks to put the by-election result in perspective.

Much of this was written in the immediate aftermath of the Richmond Park by-election of December 2016, but held over for release until after the Stoke-upon-Trent Central and Copeland by-elections of February 2017, for revision and reconsideration in light of subsequent developments. In the event, very little needed tweaking, though as I add the finishing touches to this with the next by-election having just been triggered in Manchester Gorton, I am more convinced than ever of the points made about the predominance of by-elections in Labour-held seats

Richmond Park constituency in context

By any standards, the Lib Dem performance in the Richmond by-election was an impressive achievement. A 21.7% swing overturned a majority of 23,015 votes, coming on the back of a 19.3% swing in the previous Witney by-election, and followed by a favourable 4.0% swing in the rather less promising territory of Sleaford and North Hykeham. The aim here is to put the Richmond result into some longer-term perspective. There is a long history of viewing by-elections as a test of the government’s fortunes, going back to the nineteenth century — in the days before opinion polls, they were the primary means of tracking a government’s fortunes.^[1] Liberal Democrats and their predecessor parties have long made a specialty of pulling off by-election coups, and views differ widely on the long-term benefit of this. As Table 1 shows, the swing in Richmond Park was very much in the mid-range of Liberal by-election gains; not quite competing with the stellar swings of Bermondsey and Christchurch, at 44.2% and 35.4% respectively, but certainly more impressive than the more modest swings of 8.6% and 7.3% in the by-elections of Romsey and Roxburgh, Selkirk & Peebles, where the party was already in a strong place to begin with.

It is worth taking a moment to look at the Richmond Park seat itself. Over a third of the present constituency is drawn from Kingston Borough rather than Richmond Borough, and successive Richmond constituencies have been the subject of boundary changes over the decades. A Richmond constituency has existed since 1918, when it was carved out of the larger Kingston seat, and saw boundary changes in 1950, 1983 (when it was renamed Richmond and Barnes), 1997 (when it was renamed Richmond Park), and 2010 (when it adopted its current boundaries). 

Table 1 — Post-war Liberal/SDP/Lib Dem by-election gains, ranked by swing

Not included in the table are Liberal/SDP/Lib Dem by-election holds, namely Montgomery in 1962, Truro in 1987, Winchester in 1997, Cheadle in 2005, and Eastleigh in 2013. Nor does the table contain by-election losses, namely Carmarthen in 1957, and Mitcham and Morden in 1982.

For a quarter of a century before the Lib Dems finally took the seat, the Richmond constituency was one of the great “What might have beens” of Liberal electoral politics. Besides having a noticeable concentration of liberal-minded voters, the seat benefitted from a number of well-known Liberals who all lived in the constituency, not least Jo Grimond. A string of high-profile candidates — John Baker, Peter Sheldon-Williams, Stanley Rundle, Alan Watson and Jenny Tonge — successively contested the seat.^[2] Through the 1950s and 1960s, although the party remained in third place, its share of the vote was noticeably above-average for the seats it contested, holding its deposit from 1959 onwards.

Figure 1 — Votes polled in Richmond (Surrey), 1945-1979, Richmond and Barnes, 1983-92, and Richmond Park, 1997-present

Source: The Times Guide to the House of Commons, 1945, 1950, 1951, 1955, 1959 , 1964, 1966, 1970, February 1974, October 1974, 1979, 1983, 1987, 1992, 1997, 2001, 2005, 2010, 2015, 19 vols. (London: The Times, 1945-2015). Note that no adjustment has been made for notional results around changed boundaries — these are the original election results.

The February 1974 general election had a transformative effect on the party’s fortunes in Richmond, with Stanley Rundle moving into second place, and slashing the Conservative majority to under 4,000. (Rundle had previously been the candidate in neighbouring Kingston; and while a May 1972 by-election in Kingston yielded a modest 11.3%, the February 1974 election proved to be transformative in Kingston as well, with the Liberals more than doubling their vote and moving into second place for the first time since 1923.) Thereafter, Richmond was keenly fought by the Liberals. In the 1979 general election, David Steel speculated on election night that Richmond might be a gain for the party, although it ended up remaining Conservative by just over 2,500 votes.^[3] In 1983, the Liberals again believed that they would gain Richmond, only to see the seat stay Conservative by 74 votes. It was in the Conservative rout of 1997 that Richmond Park finally fell to the Lib Dems. As with the 2016 by-election, it was actually the third of the constituency lying within the London Borough of Kingston which provided the Lib Dems’ strongest electoral base (and their majority), and it is perhaps no coincidence that the election on which the Richmond seat finally fell to the Liberals for the first time was also the first one in which a large Kingston element was added to the electorate.

Throughout the 1970s and 1980s, local indicators were strong as well — in 1973, Richmond elected Rundle as one of the first two Liberals to the GLC; and in 1981, Adrian Slade was returned by Richmond as the only Liberal on the GLC. In the aftermath of the December 1972 Sutton and Cheam by-election, Liberals in Kingston and Richmond were visited by Trevor Jones, who had served as campaign manager in the Sutton by-election, and offered a masterclass on how to win council seats, with these techniques being put to good use; by May 1974, two-thirds of Liberal council seats in London were either in Richmond, Kingston or Sutton. Liberal support on the local council increased so that by 1982, Richmond Council was tied 26-all between the Liberals and Conservatives, and in a by-election early in 1986, the Liberals commenced a 16-year stint of running Richmond Council. By the late 1990s, it reportedly had the largest Lib Dem local party membership in the UK. As such, it is an area that has long been favourable to Liberals in good years — but which has often provided bitter electoral disappointment as well.^[4]

The 2016 Richmond Park result — and the Witney by-election the month before — demonstrated a palpable break from the party’s showing across the 2010-5 parliament, a losing streak which continued into the 4 by-elections held in the first year of the 2015 parliament. Across the 2010-5 parliament, the party averaged 8.4% of the vote, and lost an average of 11.7% of votes; however calamitous this seemed at the time, the party’s by-election performance in the last parliament was actually a slightly better performance than the party’s eventual vote share 2015 general election, on 7.9%. The party lost its deposit in 10 of the 19 by-elections contested in 2010-5 — a harbinger of what happened in 340 of 631 contested seats at the 2015 election. The 2010-5 parliament was (with the exception of the six-month parliament of March-September 1974, which contained only one by-election) the first parliament since 1951-5 to not have any Liberal by-election gain. And this trend seemed to continue early into the present parliament, with the first four 2015-6 by-elections indicating support having “bottomed out”, the party averaging 3.9% support, and averaging a swing to the party of just 0.05% from its 2015 election result. The five post-referendum by-elections of Witney, Richmond Park, Sleaford & North Hikeham, Copeland and Stoke-on-Trent Central signal, then, a marked change in the UK political landscape.^[5]

This has been reflected in council by-elections since the last round of local elections in May 2016, which have seen the Lib Dems gain a net 30 council seats (33 gains, 3 losses), while the Conservatives have lost a net 23 seats, Labour has lost a net 11 seats, Plaid Cymru are up 3 seats, the Greens up 1 seat, Independents up 6 seats, the SNP are down a net 2 seats, and UKIP down a net 4 seats.^[6] The average swing to Lib Dems in these council by-elections has been 8%, and this pattern is reflected in seats across the UK, spanning a wide range of demographics, geographies, and levels of turnout. This is not something easy to dismiss. Therefore, much as political pundits overestimated the level of Lib Dem support in the last parliament, there is good reason to believe that national pundits are currently underestimating that level of support — as with the last parliament, the national poll ratings are not correlating with the party’s electoral performance on the ground. In the case of the current polls, there is reason to suspect a sampling error. Present polling consistently shows a level of “false recall” over respondents claiming to have voted Remain in the EU referendum; respondents may be lying, they may be disproportionately represented among those interviewed, or they may just have “false recall” over how they voted. Whatever the reason, pollsters tend to weigh this demographic down. Since this is also the demographic disproportionately likely to vote Lib Dem, the party’s polling rating is also weighed down.

Who triggers by-elections, and how often?

As Table 1 makes clear, most Liberal and Lib Dem by-election gains have been from the Conservatives — 22 in total, compared to just 8 from Labour. And the first three post-Brexit by-elections contested by the party were all in Conservative-held seats.^[7] The Copeland and Stoke-on-Trent Central by-elections stood as the Liberal Democrats’ first post-referendum electoral test in Labour-held seats, and each showed a promising swing to the Liberal Democrats – in the case of Stoke, the largest increase in votes to any party was to the Liberal Democrats.

Labour seats have often proved a much harder nut to crack for Liberals. While Liberals have been making by-election gains from Labour since 1912,^[8] the first post-war Liberal gain from Labour in a by-election, in Birmingham Ladywood in 1969, was won on a 37.1% turnout in the constituency containing the smallest electorate in the UK at the time (just 25,294 voters). The next gain, Rochdale in 1972, was fought in exceptional circumstances, with the Liberals already in a strong second place, and with a popular local Liberal candidate with deep roots in the local Labour Party, after 18 years as a Labour activist. The next gain, at Liverpool Edge Hill in 1979, was another exceptional by-election, the last in a Parliament with a general election imminently expected. Finally, Bermondsey in 1983 proved to be the last ever Liberal gain from Labour prior to the merger, and although it led to a revival of Alliance polling ratings in the run-up to the 1983 general election, it was closely followed by the bitter disappointment of the Darlington by-election, where an expected SDP gain from Labour turned into a disappointing third place.

Despite the SDP’s roots in Labour, it was noticeable that three out of four of the SDP’s by-election gains were from the Conservatives, as at Crosby (1981), Glasgow Hillhead (1982) and Portsmouth South (1984). Indeed, this reflected a marked cultural and philosophical difference from the Liberal Party’s approach to such by-elections; since at least the 1970s, Liberals had been attacking Labour-held seats in by-elections from the left, calling for policies like profit-sharing, and workers’ co-operatives. (The same applies to the Liberal general election gain of Leeds West in 1983 — the first Liberal general election gain from Labour since 1929.) By contrast, in keeping with their roots among the parliamentary party on the Labour right, SDP by-election campaigns tended to attack Labour from the right or the centre; a tactic which met with very limited success in Labour strongholds. Only in Greenwich in 1987 (where the pre-existing Labour share of the vote, at 38.2%, was not that high to begin with) did the SDP finally succeed in gaining its only Labour seat in a by-election — and this proved to be the last third-party by-election gain from Labour in 16 years. And while the 1990s saw the Liberal Democrats notch up an impressive tally of by-election gains from the government, consolidating their reputation as the masters of by-election victories in the most improbable circumstances (they actually made more by-election gains in the 1992-7 Parliament than any other party, even though New Labour was at its height in the opinion polls), it was noticeable that none of these wins were from Labour — a pattern that continued throughout the 1997-2001 Parliament. (Against this, it should be recognised that the 1990s by-elections in Eastleigh and Littleborough & Saddleworth both represented at least as much of a fight against Labour as well as the Conservatives; in each case, a resurgent Labour Party pushed the incumbent Conservatives into third place, and the Lib Dem challenge was as much to withstand a Labour surge as to displace the incumbent Conservatives.)

It was only the aftermath of the Iraq War which saw a noticeable change to this pattern, with a run of three gains in Brent, Leicester and Dunfermline in 2004-6, which again bolstered the party’s credentials in Labour-facing seats; an area in which it had been vulnerable since the 1920s.

As such, the lesson to take home is that it would be really quite exceptional for the Lib Dems to make further by-election gains from Labour (which is a different matter to gaining Tory-held seats with Labour tactical votes). Even though the Labour Party is currently suffering an implosion of electoral support, and is registering early-1980s levels of support in national opinion polls, losing city voters to Lib Dems and town voters to UKIP, it should be remembered that even in its grimmest days of the early 1980s, Labour only ever lost two by-elections to the Liberals or SDP. Other by-elections where the Alliance parties made inroads — like Warrington (1981), Chesterfield (1984) and Knowsley North (1986) were still Labour holds.

Yet it is worth not giving up on Labour-facing seats altogether, for two good reasons. Firstly, there is obviously an historic opportunity in many Labour-held seats, with the party’s difficulties under Jeremy Corbyn. The Labour loss of Copeland signaled the first by-election loss of an opposition seat to the government since 1982 (with the Mitcham and Morden by-election having been held in exceptional circumstances during the Falklands War), and in this case, in a seat held by Labour at every election since 1935.The Stoke-on-Trent result was scarcely more encouraging for Labour, with a further contraction of its vote share to just 37% in a one-time ‘heartland’ seat. Labour may hold many seats, but its rump share of the vote within such seats is continuing to diminish.

And secondly, by-elections are historically more likely to come up in Labour-held seats. In fact, Labour MPs have triggered more by-elections than any other party’s MPs in all of the last seven Parliaments, including in the 2001-2005 Parliament, where all six by-elections triggered were in Labour-held seats. There have been numerous historical reasons for this, including Labour MPs representing (and often coming from) areas with poorer areas marked by shorter life expectancy, making deaths in office more likely; and Labour MPs traditionally having less financial independence than Conservative MPs, increasing the probability of a resignation mid-Parliament to pursue a new career. Whatever the reasons, of the 13 Parliaments since 1966, Labour have triggered more by-elections than any other party in 12 of them.^[9]

Figure 2 — Percentage of by-elections in each Parliament triggered by Labour, the Conservatives, or other parties, 1955-present

Source: F.W.S. Craig, Chronology of British By-Elections, 1832–1987 (Chichester: Parliamentary Research Services, 1987); Sarah Priddy, UK Parliamentary By-Elections Since 1945: House of Commons Library Research Paper No. 02383, 9 December 2016 (London: House of Commons, 2016).

This has gone hand in hand with by-elections generally becoming rarer. There are many reasons for this, most of them guided by political parties’ dread of triggering the uncertainty of a by-election: over the past few decades, parties have increasingly selected ever-younger candidates who are less likely to die in the middle of a parliament;^[10] Prime Ministers are increasingly reluctant to make Lords appointments from among serving MPs;^[11] and whips put MPs under considerable pressure to not quit their seats mid-parliament. The result of this is that, while the frequency of by-elections is once again on the rise from the low point of 2001-5, the overall trend has still been very much downwards over the last 60 years. Although all parties try to anticipate by-elections, keeping an eye out for dissatisfied MPs likely to quit politics, MPs at high risk of death, legal proceedings likely to unseat incumbent MPs, and other ‘high risk’ scenarios, the triggering of many by-elections still comes as a surprise.

Figure 3 — Average number of by-elections per month in each parliament, 1955-present

Source: F.W.S. Craig, Chronology of British By-Elections, 1832–1987 (Chichester: Parliamentary Research Services, 1987); Sarah Priddy, UK Parliamentary By-Elections Since 1945: House of Commons Library Research Paper No. 02383, 9 December 2016 (London: House of Commons, 2016).

This all means that the opportunities to secure by-election upsets are broadly becoming rarer, as fewer by-elections are held, and those which are triggered are more likely to be in Labour-facing seats, where Liberals have traditionally had a more uphill struggle.

Near misses and wins each parliament

It is not enough to look at by-election wins. It is also instructive to look at by-election “near misses”. The Liberal Democrats were able to mount a credible challenge in Richmond Park, in part due to the momentum from the massive swing secured in Witney barely over a month earlier, and this is part of a wider pattern of by-elections coming on the back of “near misses”. Often, such wins on the back of massive swings only seem credible in the aftermath of securing large swings elsewhere.

In looking at “near misses”, I’m going to define them in entirely subjective terms, as by-elections where there was either (a) a narrowly close result, sometimes surprisingly so, or (b) there was reason to believe, rightly or wrongly, that the Liberals/SDP/Liberal Democrats might conceivably take the seat, or (c) there was a conspicuously large swing to the party. Due to the subjective nature of such a list, no doubt it is highly debatable, and alternative lists of “near misses” can be compiled.^[12] Nonetheless, it is apparent that the broad thrust of rising support for a third party is as measurable by such “near misses” as by the total number of outright wins.

Table 2 — Liberal/SDP/Lib Dem by-election gains, holds and “neart misses”, 1955-present

Source: F.W.S. Craig, Chronology of British By-Elections, 1832–1987 (Chichester: Parliamentary Research Services, 1987); Sarah Priddy, UK Parliamentary By-Elections Since 1945: House of Commons Library Research Paper No. 02383, 9 December 2016 (London: House of Commons, 2016).  

Table 2 therefore sets out how even in parliaments where Liberals won an exceptionally large number of by-elections (and benefitted from the accompanying boost to their credibility), they were seldom competitive in more than a fraction of by-elections. The lesson that can be taken home is that it is not necessary to seriously contest all by-elections to be taken seriously. The Liberal Democrats were remembered for their four impressive by-election victories going into the 1997 general election, which granted them a degree of electoral credibility — but they had not been seriously competitive in 13 out of 17 mainland by-elections in the preceding parliament. Only in the Parliament of 2001-5, with its record low of just 6 by-elections across four years, were Lib Dems competitive in more than half. Each win was a substantial electoral achievement, taken seriously by pundits. In themselves, “good second places” went almost as unnoticed as trailing in third. Yet cumulatively, a run of good second places was noticed.

But if there is a further lesson from Table 2, it is this: by-election gains on huge swings rarely come out of the blue. With a few exceptions like Brent East in 2003 (which again, came after the longest recorded break without a single by-election, some 17 months), by-election gains usually happen on the back of one or more “narrow misses” which help build up momentum and plausibility. As such, the recent by-election result in Witney, with its 19.3% swing, was at least as important as the 21.7% swing in Richmond Park, in showing that the party was once again gaining support, and in providing enough of a morale boost to encourage activists to invest serious time into the subsequent Richmond Park by-election. There is therefore a strong incentive for Liberal Democrats to seek strong by-election showings in at least a selected number of seats, however remote the odds of victory may be.  

The need to innovate

Election success depends on innovation — a truth reflected in the eager way the last 30 years have seen British political campaigners descend upon U.S. Presidential and mid-term elections, in part to appropriate the latest electoral techniques. Most of the campaign techniques the Liberal Democrats are best known for were developed at a particularly fertile period in the evolution of political thought and action, between the 1960s and 1980s, as various strands of “community politics” evolved.^[13]

Yet since at least the early 2000s, the party’s tactics have grown stale. Part of this was simply that the other parties caught up. Long having been mystified by the Lib Dems’ curious by-election appeal, the other parties started to amass a formidable cache of Lib Dem campaigning literature, shamelessly copying the techniques contained therein. So exhaustive was the Conservative Party’s trawl of Lib Dem campaigning manuals, there was reputedly a time when CCHQ held more copies of the Association of Liberal Democrat Councillors’ manuals than the Lib Dems’ own party HQ did. I can well remember turning up to an election count a decade ago, where the Conservatives won a narrow victory over the Lib Dems, and triumphantly brandished an ALDC campaigns manual at the defeated candidate, with the words, “I don’t understand a word of this stuff, but it bloody well works!” Moreover, the other parties have been able to build upon such techniques, augmenting them with far more sophisticated voter segmentation and data operations, and in many cases, an avalanche of money. And it is no coincidence that this period of campaigning “fossilisation” saw the party lose ground — it must be remembered that even before the electoral massacres of the coalition years, the preceding half-decade saw the party make losses at local, European and Scottish elections.

It is therefore timely ask to what community politics was about in the first place. Bernard Greaves and Gordon Lishman’s seminal pamphlet on the subject starts with a list of what community politics is not. It may come as some news to today’s “community politicians” in all parties:

Community Politics is not a technique for the winning of Local government elections. Community Politics is not a technique. It is an ideology, a system of ideas for social transformation. For those ideas to become a reality there is a need for a strategy of political action. For that strategy to be successful it needs to develop effective techniques of political campaigning. Those techniques are a means to an end. If they become an end in themselves, the ideas they were designed to promote will have been lost.^[14]

They went on to write that it “is not local. It is universal…[It] is not government. It is about people…[It] is not about elections.”^[15] If a Lib Dem revival is to be at all sustainable, then up-to-date campaigning techniques need to be paired with a strong sense of why liberals believe in community, as well as an open-mindedness to how today’s communities are not necessarily recognisable as the same communities of old — online communities, long-distance communities, cultures and subcultures all recognisably display the characteristics of a physical community in a neighbourhood, and as well as a potentially rich electoral harvest, these fundamentally liberal communities are currently under-represented in parliament. Or to frame this in reverse, post-Brexit: is it not the case that the Referendum has made pro-Leave politicians first asked themselves what voters most closely aligned with their values, and then asked in which communities those voters could be found? Can Liberal Democrats not do this as well with the Remain vote, on a national as well as local scale? Of course, the biggest reason for remaining stuck in existing electoral practices is the First Past the Post electoral system, under which elections need to be won. It rewards parties with geographical concentrations of voters, rather than with voters equally spread across the map (something learned the hard way by the SDP-Liberal Alliance, when it polled over a quarter of the vote in 1983, but this simply amounted to coming second in hundreds of seats). But with the new data-driven techniques of voter identification, the time may well be ripe for Liberal Democrats to look to practicing “community politics” on a national scale, seeking out communities of liberal-minded voters.

It would be strategically imprudent for the party to publicly spell out what it did differently this time around in Richmond Park. Nonetheless, it is important to recognise that the party’s campaigning in Witney and Richmond Park in particular did differ from some of its recent campaigning activity, and if it is to be successful, it needs to continue to innovate. For decades, Liberals and Liberal Democrats were the masters of by-elections when they were building on established techniques, but constantly refining them. Many of the local parties that have enjoyed some of the biggest electoral swings are those that have not been afraid to experiment. 

Picking and choosing battles

Table 3 — Where victorious Liberal/SDP/Lib Dem by-election candidates had previously been placed, 1958-present

If the Liberal Democrats were campaigning on the base of their pre-2010 levels of election support, then all of the above would be a viable strategy, and all the old campaign clichés would come flooding back: “breaking the mould”, “one more heave”, etc. Unfortunately, this is not the base line that Liberal Democrats are starting from. As has been noted elsewhere, in 2010, the Liberal Democrats were very much a national party, coming first or second in 299 seats — therefore nearly half the country was ‘winnable seat’ territory.^[16] By contrast, the comparable figure in 2015 was 71. (Figures 4 & 5 overleaf) The party will have relatively few seats where it can plausibly say it is the main challenger. Liberal Democrats would do well to etch these maps onto their collective memory. If the party is serious about punching above its weight — as now seems feasible after Richmond — then it is all the more important to target its efforts accordingly. And as we have seen, it is not necessary for the party to win many by-elections in a Parliament to be taken seriously, nor even to do well in by-elections across the board, but to pick and choose a selected number of seats it can win in.

            Of course, it could be argued that many by-elections have been won from third place and a derisory share of the vote to begin with — Brent East comes to mind, where the party started from a base of third place and just 10.6% support. But this argument should not be taken too far. Firstly, it is worth looking at the spread of previous positions for Liberal/SDP/Lib Dem candidates who went on to win a by-election:

*Includes Crosby and Glasgow Hillhead, where the SDP had not stood before, but the Liberals had previously come third; and Torrington and the Isle of Ely, where the Liberals had not stood at the previous election.

Figure 4 — Lib Dem first and second places, 2010 general election

Figure 5 — Lib Dem first and second

 
           
 places, 2015 general election

As can be seen in Table 3, aside from the 5 candidates who were defending held seats, and disregarding the 4 rather unusual cases of standing where the party had not stood before, such by-election gains have been evenly spread between second-placed candidates and third-placed candidates. No Liberal, SDP or Lib Dem by-election candidate has ever been victorious when previously placed fourth or worse. This is of considerable concern to Liberal Democrats, because in addition to the 71 candidates who came first or second in 2015, only another 36 came third. The remaining 524 constituencies had candidates starting from a base of fourth place or worse. Given such historic performances, it would be stretching credulity to expect many fourth- or fifth-placed candidates to win, or to devise a strategy revolving around their doing so.

Secondly, it is also worth remembering that a party’s previous election performance is less important than whether they can mount a convincing campaign in which they cast themselves as the principal challenger. There are numerous reasons why a third-placed challenger might still be able to plausibly argue they are best-placed to win — results from other recent elections, past election history, the incapacity of one or more of the challenging parties. But even assuming a “narrative of victory” is possible in more seats, the Liberal Democrats could plausibly construct a narrative of victory in up to 107 seats — but that is still only one-sixth of all seats. Indeed, the case of Witney rather underlines this point. A “narrative of victory” was forged in what was one of the 225 seats where Liberal Democrats had previously (in 2005 and 2010) come second, and a phenomenal swing was achieved — but starting from so far behind, the party was still only able to obtain a good “near miss”. There is much to be said for such near misses, for all the reasons set out above, and it was critical in building a sense of momentum prior to Richmond Park. But “near misses” should not in themselves be the objective of wider by-election strategy.

These are, of course, short-term points, dealing with the next election only. In the longer term, the party would no doubt be looking to develop the number of second places in ‘development’ seats; and bolstered by big post-referendum shifts in opinion, it would undoubtedly do considerably better than in it did even a year ago. In January 2016, the party was averaging 6% in polls and just 5% of pro-European voters. Today, it is averaging 11% of the electorate, and 19% of pro-European voters.[17] These are encouraging signs, but the challenge will be in making these healthier approval ratings among the wider population correlate with winnable constituencies.

And there is a direct applicability to Richmond: while the party’s performance was impressive, the Liberal Democrats were lucky that a by-election came up in such a winnable seat in the first place. Of the 9 other by-elections triggered in this parliament, 7 have been in seats where the Lib Dems previously came fourth, and 2 in fifth-placed seats.  When an all-too-rare (in this parliament) second place came up, it was not only a golden opportunity, but also one that the party could ill afford to lose. 

Did the Green Party’s decision not to stand in Richmond, and its endorsement of the Lib Dems, make a difference on the final result? It is hard to know. At the Witney by-election six weeks earlier, Liz Leffman’s insurgent campaign apparently squeezed the Green vote, halving it from 2,970 to 1,363. This was still not enough to account for the shortfall of 5,702 votes from the Conservative vote. In Richmond Park, the 3,548 Green votes from 2015 may have been enough to account for Sarah Olney’s majority. But against that, we should consider UKIP’s decision not to stand, and their suggestion that their 2,464 voters from 2015 should back Zac Goldsmith. It is likely that the two actions broadly cancelled one another out, though with a slightly larger impact for the Green decision.

It is also worth acknowledging a wider significance to the Green Party’s endorsement, and for that matter, to that of the Women’s Equality Party. In themselves, each endorsement only shifted a relatively small number of votes from one column to the other. But each endorsement arguably had an incalculably greater effect, in signalling to anti-conservative voters that there was a coalescing of support around one viable anti-Conservative challenger. It also encouraged the existing liberal-inclined vote in such a constituency — part of which had stayed at home in 2015 — to hold their heads up high and start supporting the party again. In other words, the Liberal Democrats started to look like winners again.

Labour tactical voting undoubtedly played a major part, too, and when canvassing in the constituency, the Labour vote became noticeably ‘softer’ as polling day grew nearer. Although Labour’s candidate Christian Wolmar put in a bullish performance, doggedly attacking the Lib Dems, it was not a theme Richmond’s voters warmed to, and nearly four-fifths of Labour’s vote deserted it. Labour memorably ended up polling fewer votes than it has members in the constituency.^[18]

Much has been written of forging a ‘Progressive Alliance’, with research on this commissioned by the Joseph Rowntree Reform Trust and conducted by former Lib Dem MP Andrew George;^[19] and Labour's Lisa Nandy, the Greens’s Caroline Lucas, and the Lib Dems’s Chris Bowers recently edited The Alternative, setting out the case for how and why such an alliance could work.^[20] Some, such as the Telegraph’s Asa Bennett, have gone so far as to identify “10 seats Remainers may hope to take from Tory Brexiteers”.^[21] This is not the place to fully analyse such a plan, but it is worth noting the considerable obstacles to such an arrangement, which seriously undermined the SDP-Liberal Alliance through rancorous seat negotiations throughout much of 1981-3.^[22] Similarly, it is worth noting that parties only tend to show an interest in such alliances when they do not think they can win in their own right — once a recovery sets in, interest tends to wane.

But that is not to say such difficulties are insurmountable. Indeed, it should be noted that a series of ad hoc arrangements have sprung up. In Richmond Park, the Green and Women’s Equality Party decision to not stand and to instead endorse the Lib Dems was very much an ad hoc arrangement. Similarly, a more recent Lib Dem pact with the Greens in Broxtowe, ahead of the forthcoming 2017 County Council elections, is another standalone pact.^[23] These are by no means unprecedented. Liberal Democrats have previously stood down candidates in selected seats, as with Martin Bell’s Independent candidature in 1997, and Dr Richard Taylor’s Independent candidature in 2001 and 2005; and there was a measure of informal co-operation with Labour in 1997, in agreeing to turn most of the two parties’ national resources onto Conservative-facing seats, minimising the number of well-resourced Lib-Lab marginal battles.^[24] And, of course, there is a much longer Liberal tradition of such pacts. When the party was at its lowest ebb in the 1950s, returning just 6 MPs in general elections, 2 of its 6 MPs (Arthur Holt in Bolton West and Donald West in Huddersfield West) were returned in local electoral pacts. Jo Grimond’s entire ‘realignment of the left’ strategy was based upon wooing over and making pacts with Labour moderates; and after the strategy failed to materialise, Grimond would occasionally favour ad hoc pacts, as in the March 1970 South Ayrshire by-election, when he advocated an electoral pact with the SNP, proposing that the Liberals should support the SNP candidate. In 1973, the Liberals opted to endorse Dick Taverne as an independent Social Democrat in his by-election. And most notably, the entire Liberal-SDP Alliance was structured around a nationwide electoral pact between two parties.

If there are to be pacts, it is most likely that they will be on the sort of bottom-up, localised level recently seen in Broxtowe and Richmond Park. Nonetheless, the very nature of by-elections mean that they are in many ways far more amenable to being testbeds for such pacts than general elections are.

Conclusion

By any definition, the 2016 Richmond Park by-election was a remarkable result, with a remarkable swing. But the Liberal Democrats were also quite “lucky” in having had such a relatively winnable seat come up for a by-election in the first place. There is no shame in that — by their very nature, by-elections are unusual, and the patchwork of different by-elections held across a parliament seldom forms a ‘typical’ cross-section of seats.

Little has been said here of the Richmond Park by-election’s strategy, themes, campaigning — in many ways, that is not the point of this work, since there is no shortage of existing analysis of that elsewhere. The aim has been more to ask “Where next?”, and to try to set some context as to the scale and scope of the result, compared to other Liberal by-election gains.

For those with longer-term electoral memories, the by-election result seems to be sign of a return to “business as usual” for the Liberal Democrats. After a grueling 2010-5 parliament, the party is once again able to pull off by-election gains, on a swing that is around average for such election gains, yet high by by-election standards.

The prospects for further gains against Labour have seldom been high. Only in the latter half of the Blair government did a third party start to make serious inroads in Labour seats in by-elections. The vast majority of Liberal by-election gains have been from the Conservatives, and Liberals have traditionally done well against incumbent Conservative governments in by-elections, often on the back of Labour votes. Yet paradoxically, by-elections are far more likely to be triggered in Labour-held constituencies. That, combined with fewer by-elections being triggered than 30 or 40 years ago, means that the scope for many more by-election gains in a parliament is limited. But this need not be a cause for excessive concern. Even when the Liberals have secured their best runs of by-election victories, these have seldom amounted to more than four or five by-elections in a parliament — and often just one or two. The number and proportion of by-election victories is less important than the credibility of having secured such victories in the first place; they have an immediacy with real places and real people which opinion polls do not. Sometimes, as with Liverpool Edge Hill providing a welcome Liberal boost going into the 1979 general election, one single by-election gain is quite enough. And cumulatively, improving the vote share and coming a good second in some seats helps add credibility to a third party — but only for pundits and analysts. Swing voters sometimes cite a party “Looking like winners” as a reason for switching their support to them. Swing voters seldom cite “Who came second in a by-election” as a critical reason for switching.

As such, Liberal Democrats should be realistic in aiming to make more by-election gains this parliament — but not to imagine that a streak of four or five such victories is likely, given the diminishing number of by-elections triggered. One more, maybe two, would be quite enough to show that the party is winning elections again.

Moreover, given the party’s reduced starting position in much of the UK, the opportunities will be fewer for such wins, and so serious thought should be given to ad hoc arrangements where the party might back another, better-placed candidate to win — and to strengthen their hand in occasionally expecting others to stand down in their favour.

Sarah Olney’s victory certainly owed much to Labour and Green tactical votes; and if other Lib Dem candidates are placed to benefit from such tactical votes, it stands to reason that other progressive candidates are better-placed to win with Lib Dem tactical votes. What is suggested here is nothing as broad as a fully fleshed-out, top-down “Progressive Alliance” across the board for contesting all elections, but simply that Liberal Democrats keep their options open in considering standing down in occasional by-elections, if the local party feels able to support another, better-placed candidate. What matters to the national party’s profile is being seen to win, being seen to not be humiliated, and not being dismissed as “spoilers”. Picking and choosing where Lib Dems stand in by-elections helps ensure all three of these objectives. It also acts to “test-bed” the idea of cross-party co-operation, while in no way committing the party to standing down in seats at the next general election. Given the Liberal Democrats’ current starting point, it is an approach through which the party has nothing to lose, and a great deal to gain. 

About the author

Dr Seth Thévoz (b. 1984) is a political, cultural and social historian at Nuffield College, Oxford, where he is research assistant to Michael Crick on the official biography of the psephologist David Butler, the “Sultan of Swing”. He holds degrees from the Universities of Cambridge, London and Warwick, and has worked in politics for over a decade. He has held various positions within the Liberal Democrats, at both a local and national level, since first joining the party in 1997.