The cost and availability of housing has become a potent political issue in the UK, where younger people especially are being priced out of the market as their parents and grandparents benefit from decades of above-inflation rises in home values.

The ruling Conservatives, traditionally the party of home ownership, now find themselves shunned by millennial voters frustrated by spiralling housing costs. So far, the response has been to provide subsidy to renters and buyers, and to exhort the construction industry to build more. Philip Hammond, the chancellor, is under pressure to announce more money for housing in Wednesday’s budget.

Yet building more homes is unlikely to meaningfully reduce prices, especially in the short term. One reason is that, whatever the other dysfunctions of the housing market there is — nationally at least — no shortage of homes.

Those on the left attribute this to the sharp fall in the construction of social housing since the 1980s. Those on the right blame the planning system for throwing sand in the wheels of the free market. Both broadly agree that the solution is to build many more homes. In 2015, ministers set a target of 1m new dwellings by 2020. On Sunday, Mr Hammond pledged to build 300,000 homes a year, though he did not specify a timeframe.

That number echoes the promise made by Harold Macmillan, housing minister in the 1951-55 Conservative government, whose achievements are hailed by those in the ruling party calling for more ambition on housing. The 1951 administration came to power on a promise to “give housing a priority second only to national defence” and presided over a construction boom in an era when austerity included food rationing.

Many prognostications about the housing market cite government data on new builds. But what matters more is the net change in overall supply, which includes property converted to residential from other uses, and subtracts the homes demolished.

The gap between these two was substantial in the Macmillan era and in the 1960s, when housebuilding was at its peak but thousands of condemned homes were also being flattened in slum clearances. It is still fairly wide today, but in the opposite direction. In the year to March 31 2017, 183,570 homes were built in England, the most since 2008. But conversions and changes of use took the total net new supply to 217,350.

All that would be academic if the supply were consistently behind demand, which at first glance does appear to be the case. In recent years, net new supply has been below the 210,000 dwellings that the government estimates will be needed each year from 2014 to 2039 in England. But long-range forecasts on household formation require big assumptions about longevity, fertility, household size and migration, and are subject to large margins of error. In 2008, the government estimated that 280,000 homes would be needed in the UK each year until 2016. In 2012, after the global financial crisis, that had dropped to 231,000 a year.

Not only are the figures a moving target — the statisticians now have to factor in the impact of Brexit on migration, for instance — but they are also some way off the reality. The 2012 forecasts predicted 27.7m households by 2016. Figures from the Office for National Statistics say there are now 27.2m households — around 500,000 fewer than predicted.

As for dwellings, there were 28m in 2014, the last year for which UK-wide figures are available. Even allowing for second homes — around 200,000 in England, down from over 300,000 a decade ago — and those that are temporarily empty, there are clearly more dwellings than there are households. That may be because there is a surplus of housing in some areas, such as old industrial towns. But Ian Mulheirn, director of consulting at Oxford Economics, says that even London and the south-east added more dwellings than households from 2001 to 2015.

One possible explanation for this contradiction is that high prices have suppressed household growth. Households cannot form because 30-year-olds are still living in their parents’ spare bedrooms. The data do not really bear out the anecdotal evidence of the “boomerang generation”. Around a quarter of people aged 20-34 still live in the parental home. In 1996, when house prices were much lower relative to earnings, the proportion was still a fifth.

But many more young people are now renting. A study by the Resolution Foundation in September found that two in five millennials (those born between 1981 and 2000) were living in private rented accommodation at the age of 30. For baby boomers (born 1946 to 1965), the equivalent figure was one in 10. Rented households are still households, but it may be that the individuals who rent together may prefer to have bought individually. The Resolution Foundation also found that, on average, renters were living in less space further from their place of work than would have been the case in the past.

Julie Rugg, a research fellow at the University of York’s Centre for Housing Policy, also cautions that national data on rents conceal wide regional variations. “There are parts of the country with housing oversupply, and where private rents may sit somewhere below social housing rents,” she says. Nevertheless, aggregate rents have risen far more slowly than aggregate prices — suggesting that high prices are about something more than just the supply and demand of homes.

For Toby Lloyd, director of housing policy at Shelter, the housing charity, that something is actually two things, “land and money”. Not just the cheap credit that followed the global financial crisis and pushed up real estate prices in cities around the world, but a whole series of policy interventions over several decades.

In Macmillan’s era, only building societies could extend mortgages, and only then on conservative terms. Liberalisation of credit and tax cuts led to the “Barber boom” of the early 1970s, where wages and prices (including house prices) rose sharply. More relaxing of credit regulation at the start of the 1980s drove an even larger expansion in lending. Mortgage securitisation facilitated further growth, as did the Basel II reforms cutting the risk weights applied to real estate. This made mortgage lending less capital-intensive for banks.

The fuelling of the housing boom

1. 1963 Tax on imputed rents scrapped
2. 1969 Mortgage interest relief at source (Miras) introduced
3. September 1971 Competition, Credit & Control Act (allows banks to borrow from wholesale market)
4. October 1980 Housing Act gives council tenants right to buy their homes
5. 1988 Assured shorthold tenancies introduced, giving landlords more rights
6. 1996 First buy-to-let mortgages sold
7. 2000 Miras scrapped
8. 2007 Collapse of Northern Rock and onset of financial crisis

A similar thing happened in other European countries, notably Spain and Ireland. “After about 2003 [the boom] became more about rates and money,” says Kieran McQuinn, a research professor at the Economic and Social Research Institute in Dublin. “Lots of overseas banks set up shop in Ireland. More of the buying became about investment and retirement. Even business loans were increasingly tied to property.” In the peak year for home construction, 2006, this country of just 4m people built over 90,000 homes — yet prices still rose 11 per cent that year.

The expansion of credit was only one part of what Neal Hudson, a property market analyst, calls the “financialisation of housing”. In 1963, “schedule A” personal income tax, an annual levy on the imputed rent of an owned home, was abolished. Housing profits were exempted from the new capital gains tax. In 1969 came “Miras”, a tax break on mortgage interest that endured until 2000.

Council house sales in the 1980s gave rise to a private rental market, whose development was accelerated by the reform of assured shorthold tenancies and the advent of buy-to-let mortgages in the 1990s. These came with their own advantages: landlords only paid the interest, not principal, and until this year interest costs could be offset against profit in full. More recently came the Help to Buy equity loan scheme, and increased allowances for property within inheritance tax.

Such favourable treatment of property in the legal and tax systems, and the ready availability of cheap credit and government support, did more than just nurture a long house price boom. They created behavioural effects that no econometric house price model could capture.

“People like houses as an investment because they are tangible,” says Greg Davies, a behavioural economist. “They feel they understand them far more than funds or shares or bonds. People assume that property is safe because of its familiarity.”

They may also feel that policymakers are standing behind the market. The government spends £24bn a year on housing benefit, and recently earmarked another £10bn to the Help to Buy scheme. Andrew Lilico, an economist who subscribes to the idea that there is no housing crisis, points to a key Bank of England meeting in August 2005 when, according to the minutes, “the ongoing adjustment in the housing market” was one factor that resulted in a narrow vote to cut rates to 4.5 per cent. “We took a decision to protect those who had paid too much for houses at the expense of those who wanted to buy,” says Mr Lilico.

From the mid-19th century until the start of the 1970s, the price of land moved more or less in line with the prices of homes, according to Paul Cheshire, professor of economic geography at the London School of Economics. After that, land prices tended to act as a leveraged play on house prices, with considerable volatility.

This was partly down to credit conditions. But it was also due to changes in the planning regime, especially the 1961 Land Compensation Act. This forced local authorities to pay a price for land that reflects its likely future use, rather than its current value, and gradually curtailed the involvement of local authorities in constructing new housing.

The steep rises in the value of land as it moves through the planning process are a major driver of profits for landowners and housebuilders, who have the legal expertise and financial resources to endure the often lengthy process. But the fact that land with permission to build is so costly, up to half the final selling price in some areas, is a major reason why British houses are small, poor-quality and expensive.

The existence of “green belts” around many conurbations, designed to limit further development, makes it hard for councils to approve new housing in response to local conditions. In the five years to 2017, more homes were built in Barnsley (3,480) a former mining town in South Yorkshire, than were built in booming Cambridge (2,490).

Few would dispute that building more homes is a good idea, even if aggregate data suggest there is no acute shortage. “All of the other problems become easier to solve if there are more houses to start with,” says Mr Lloyd. But few expect that building more homes to buy will reduce prices in anything but the long term — meaning over five years.

The problem for Mr Hammond is that other options are either ineffectual or politically difficult. No politician is going to turn off the credit tap. Appropriating unearned property profits from older homeowners — who in aggregate are under-occupying housing — results in howls of outrage from voters. Reforming the planning system to permanently lower the cost of development land risks incurring the wrath of voters in the leafy, prosperous counties that surround London.

Interventions such as taxing foreigners who buy UK property play well politically but have little impact. The same pattern is evident elsewhere; overseas investors buying property in Hong Kong, Singapore, Vancouver and some Australian cities pay additional stamp duty of up to 15 per cent, but such eye-watering levies have done little to stem the appetite for real estate in those cities.

Perhaps the most promising way to reduce housing costs — if not house prices — is to increase the supply of homes for rent. “The nub of the whole debate is that the market in housing assets is different from the market in housing services,” says Mr Mulheirn. “House prices do not set rents. Landlords charge what the tenant market will bear.” Yet even here, the current administration lacks the pragmatism of past ones.

Harold Macmillan encouraged private developers, but also thought nothing of mobilising the resources of the state to ensure that his target of 300,000 homes a year was met. In some of the most open markets in the world, such as Hong Kong and Singapore, the state owns most of the land and up to half the population is housed in government-owned apartments.

Some have lobbied for more state intervention. Last week Sajid Javid, the UK secretary of state for communities and local government, said he would alter the rules so that housing associations are treated as private companies and can borrow more freely. But this is a timid reform compared with the one that many academics, think tanks and even former ministers have advocated: legislating to change planning rules so that the cost of land comes down. Mr Hammond appears unmoved. His comments so far suggest the Budget will offer more of the same: relying on benefits, mortgage subsidies, tweaks to planning rules and increased private-sector construction to improve the affordability of houses to buy. Like previous attempts, it looks doomed to fail.