We’re squeezed in, hundreds of sweaty bodies pressed against one another in the aptly named Greenhouse tent. It’s 30 December 2014, and two days of tropical downpours have been followed by a full day’s searing heat. We’re at Woodford Folk Festival, north-west of Brisbane, waiting to hear a speaker. It’s either deliciously tropical or downright disgusting in here, depending on your frame of mind.

In Queensland, Woodford is an institution, attended by 130,000 over its six days. It’s part holiday, part music festival, part political movement – either out of touch with the real world or a better version of it. The festival has run for nearly 30 years since Bill Hauritz founded it in nearby Maleny with the help of his local ALP branch, some unions, the music crowd and his friends.

I’m here on the festival’s fourth day with a crowd of old Labor Party friends – a one-time caucus colleague from my time in the state governments of Peter Beattie and Anna Bligh, a former party president, some party officials and hangers-on. They’ve all come to Woodford since the beginning, as volunteers, spectators, speakers, musicians.

Right now, we’re here to listen to Ian Lowe, a well-known environmentalist and scientist who was until recently the president of the Australian Conservation Foundation. We know Ian; we’ve followed his work for years. His topic is ‘How To Solve a Problem Like Denial’. Having recently seen Julia Gillard – our woman – torn down for taxing carbon, we’re keen to hear what Ian will say.

Lowe is greeted warmly when he walks out. This is his part of the world and his audience. From the opening gambit Lowe is strident. For years he put the case to governments for a more sustainable approach, but now he’s burning the whole place down. He condemns the Bligh and Abbott governments – Queensland’s most progressive and Australia’s most right-wing – as if they are one and the same.

One hundred per cent of state and federal politicians today, he proclaims, think people are consumers and not citizens, that the world’s resources are inexhaustible, that for man to have dominion over nature is a God-given right. It’s nonsense, but the crowd lap it up. There’s a mixture of laughter and applause.

Lowe moves on to the pointy end, the solutions that he sees. While he’s an environmental campaigner, not a political Green, Lowe’s words have a familiar ring: Australia needs a small population, a quick transition to renewable energy, and an abandonment of politics as we know it in favour of community problem-solving. While he touches only briefly on economic affairs, there is the hint of an idea that is starting to gain legs – it’s the economy itself that’s the problem; it’s a consumption machine. We need to rein in capitalism in favour of something warm-feeling but ill-defined. The answer is not greener growth but no growth, coupled with a new era of sharing the fruits of the land. It’s a mantra that’s familiar if you remember the old left.

When Lowe finishes, the crowd applaud passionately and a good proportion rise to their feet. Both his analysis of the problem and his solutions might be simplistic but he’s telling these people what they want to hear.

Later, sitting on a hay bale in one of Woodford’s makeshift bars, I ask the Labor crowd what they make of it all. How is it, do they think, that our party can be so easily derided, dismissed in the very place and by the very people we once thought our own? Their lack of an answer takes me by surprise. They don’t agree with Lowe’s answers – some of these old lefties have tried them before – but there’s a resignation here, a sense of being caught in the middle, and an acceptance that among the bright, creative types who care about the future, modern Labor is losing the battle of ideas.

When Bill Shorten finished 2014 ahead in the polls, he seemed genuinely surprised. He had set out as a new federal Opposition leader not to attack but to get his own house in order – uniting a bruised and divided Labor Party and beginning to establish a relationship with the electorate, one he hoped would be founded on trust. That Tony Abbott’s Coalition government was imploding under the weight of its own mean-spirited incompetence less than two years into its first term was a surprise to many, not least the ALP.

Shorten responded in the only way he could, belatedly painting himself into the frame. He hastily labelled 2014 Labor’s “year of resistance”. For this year he promised a “year of ideas”.

In early 2015 it is becoming increasingly clear that Australian politics is undergoing a seismic shift. In the past six months, two first-term conservative governments have been swept away – in Queensland, spectacularly – and the federal government, elected not 18 months ago, has entirely failed to win the public’s trust.

The question for oppositions is no longer whether you can win an election – it’s likely that Shorten can. The question now is, if you do win, can you survive the experience and actually govern?

A comprehensive plan for government is nowhere to be seen, on either side of parliament. While politics is often spoken about in terms of values, listening and communication style, the business of government largely remains a hard-edged economic affair. Governments influence the economic environment (albeit less than they used to) with policy decisions on trade and foreign investment, productivity and tax policy, and they apply their agenda around the Expenditure Review Committee table.

For all the talk of leadership style in the Abbott government – of knighthoods, message control and Peta Credlin – its failures are economic at heart. Tony Abbott went to the election with an implausible plan: he would balance the budget with a formula that involved adding the massive costs of the Direct Action climate-change policy and paid parental leave while ruling out both new taxes and new savings in health, education, the ABC and SBS. The idea that he could do it was always wishful thinking. A less generous soul might call it a lie.

As the Abbott government spun out of control in February, Tom Bentley, a policy analyst and the former deputy chief of staff to Julia Gillard, seized on this point. Writing for the Guardian, he condemned the government as innately conflicted on a range of key issues, suggesting it was little wonder they governed from slogans with contradictions like these:

“austerity budgeting and increased defence spending; greater surveillance of personal data and extreme freedom of speech; trade liberalisation and agriculture-led regional development; privileging of financial services and diversifying the economy; supporting families and deregulating the labour market; starting an infrastructure boom and reducing public debt; stimulating economic growth and reducing the deficit.”

While Bentley’s attack was directed at the Coalition, Labor’s policy platform is no less conflicted.

The roots of many of Labor’s modern economic contradictions date back to 1996 when, shocked at its defeat and the desertion of part of its traditional base to a new cohort of “Howard’s battlers”, the party bought the idea of “reform fatigue”. It believed the electorate had had enough and that it would be better now to tone it down.

While Paul Keating’s agenda of floating the dollar, tariff reduction (a path, it’s often forgotten, Gough Whitlam started Labor on), and support for innovation, competition policy and privatisation may have been difficult for Labor’s old protectionist Left, it created the modern Australia. It was a reform path Labor largely left behind.

In the years since the era of Keating and Bob Hawke, Labor has never resolved the internal debate between the party’s Left, with its preference for state ownership, more social spending and higher taxes, and its Right, long the bastion of economic reform. While avoiding that debate might have been good for party unity, the result has been a party with economic populism, not principle, at its very heart.

Throughout John Howard’s prime ministership, on issues such as the GST, the privatisation of Telstra and industrial relations, Labor simply chose the path of opposition.

The underlying uncertainty of the party’s economic principles bedevilled the Rudd–Gillard years. In 2007, Kevin Rudd went to an election with twin, though contradictory, promises: that he was a “fiscal conservative” (enough to win the Australian’s endorsement), and that he wanted to “live in a country that makes things” – a nod to 1950s industry policy that reportedly brought tears of joy to Kim Carr’s eyes. The global financial crisis shifted this to a Keynesian philosophy with a big push for government to spend its way out of the bust. Then, following the crisis, the internal contradictions inevitably played out, as the government introduced “great big new taxes” on mining and carbon without net revenue gain, embarked on a nationalised broadband network, introduced the Gonski education reforms and the National Disability Insurance Scheme, subsidised a car industry that ultimately shut down anyway, and repeatedly delayed the promise of a return to surplus as revenue forecasts were written down again and again. Its reforms, in other words, were a significant departure from the legacy bequeathed by Keating.

As he formulates his policy program, Shorten will embrace a competitive market, just as the two successful Opposition leaders, Whitlam and Bill Hayden, did before him. He will likely accept the Keating legacy as a starting point, resisting the constant calls for a backslide into rent-seeking and protection, but once he has established that position, what else can we expect?

The Keating legacy created two constraints on economic policy today. The first, as both Laura Tingle and Mark Latham have written about extensively, is that the liberalising reforms – floating the dollar, corporatising utilities and moving monetary policy to the independent Reserve Bank – have left government’s hands tied, compared to preceding eras.

The creation of competitive markets for the provision of everything from utilities to petrol meant giving up much of the government’s power. It’s no doubt tempting for oppositions to connect with voters by promising to do something about “the cost of living”, but as Rudd’s Grocery Watch debacle demonstrated, it more often serves to highlight the government’s impotence.

The second constraint, as Keating’s economic adviser John Edwards points out, is that the bulk of the big structural economic reforms have already been done. You can’t float the dollar twice, as the saying goes. The next generation of reform lies, he argues, not in changing structures but in the capacity of the Australian people and in our engagement with Asia.

In Beyond the Boom, an essay about Australia’s future economic direction, Edwards writes:

“It seems to me the arguments over big changes to industrial relations and big changes to the tax mix are now conducted at the expense of a more relevant conversation about Australia’s future. Australia’s continuing prosperity will be far more influenced by its success or failure in engaging the rapidly expanding Asian economic community, which is closely related to its capacity to keep improving its human capital, than it will by any conceivable change to the tax mix or industrial relations.”

The Edwards view is that the great opportunity for Australia is the rise of the Asian market – not so much for Australian resources but for our services. He argues that the downturn in the mining boom need not greatly affect the Australian economy if the nation has the capacity to become a service centre for the region in the Asian century.

That view is a long way from the mainstream of Australian economic debate – a long way from the Labor nostalgists who preach a return to Keating-style economic rationalism, and a million miles from the Coalition agenda of budget cuts, GST increases and the spectre of labour-market deregulation through the current Productivity Commission inquiry.

Having seen the Abbott government fail spectacularly in its pursuit of spending cuts, Labor has likely already accepted the argument put by John Edwards, Ken Henry and a growing body of economists, that the structural deficit finds its main causes not in overblown spending but in falling revenue originating from the tax cuts of the late Howard years, and the collapse of commodity prices as the mining boom falls away.

The broader Edwards thesis is that additional economic activity in housing construction and infrastructure, a lower Australian dollar stimulating exports, and monetary policy settings that encourage expansion and growth will combine to ease Australia’s economic problems as it makes the transition from mining to a more service-oriented economy.

The problem with Edwards’ view is clear: evidence suggests that, even if his thesis eventually turns out to be the true, the short-term forecast is for collapsing revenue and bigger budget blowouts. While Australians were comfortable enough to accept a budget flung into deficit by stimulus spending at the height of the GFC, a seemingly entrenched structural deficit looks not Keynesian but simply lazy.

On the spending side, there are no low-hanging fruit. The public has reacted savagely to the Coalition’s unannounced cuts, and Labor could expect a similar response if it embarked on a similar strategy.

Shorten and his shadow treasurer, Chris Bowen, do have a range of revenue options. Every one of them is hard. Privatisation has become politically difficult; cracking down on multinationals that shift profits offshore is an obvious target but easier said than done; and winding back superannuation tax concessions, cutting diesel fuel rebates and raising taxes all create obvious losers who will fight ferociously.

Any changes to negative gearing would affect the property market and create a world of political pain. A stricter assets test on aged pensions would pit an army of near-retirees against Labor. Changes to business taxes might move the focus away from voting households, but they too would create political problems for a party seeking actively to build its pro-business credentials.

Even such a cursory glance highlights the challenge Labor faces around the shadow cabinet table. A neat line of argument does indeed link these ideas – that in a fair, modern Australia, multinational corporations and high-income earners need to pay their share – but every measure of reform will be resisted with a palpable force.

And the fights would be on several fronts. There’s the public and the industry lobby groups, and then there’s the Labor Party itself.

It is ironic perhaps that, due to Kevin Rudd’s reform of the rules regarding leadership changes (one requires the leader be elected by both caucus and branch member votes, another requires a 75% caucus vote to remove a leader), Shorten has both a broader power base and greater room to move in a policy sense and is less beholden to the caucus and factions than any leader who has come before.

Nevertheless, there are risks from within, especially on proposals for liberal microeconomic or productivity reforms. Both Anna Bligh and the former NSW premier Bob Carr, for instance, found themselves facing a massive backlash from the left unions in the broader community and on the conference floor when they publicly committed to privatisation as the centrepiece of their economic strategy.

The chance of a wild internecine fight is not so great at a national level; even so, policy contests would allow Shorten to stamp his authority on the party, leaving no one in any doubt as to where he stands. It’s often said in politics that it’s only through your fights that the people know who you really are.

The Labor Party has a folklore of core shared beliefs. One of the most central is that, as a party of progress, Labor is inherently different from the Tories sitting opposite. Labor tells itself that, while the conservatives can win as a small target, urging people to vote for them because they’re not Labor (as Malcolm Fraser, John Howard and Tony Abbott have all done in living memory), people vote for Labor out of hope for a better future, not simply out of fear. To win, Labor must develop an agenda, sell it to the people, and convince them that the party is united and competent enough to risk making a change.

The adage has been borne out in much of Labor’s history. Whitlam won on his famous “Program”, Hawke on the back of Bill Hayden’s diligent, economically focused Opposition leadership. Rudd won in a blaze of promises: not content to change the country, it seemed he was attempting to change the world. For the true believers the platform matters; unlike the conservatives, Labor can’t be a small target and still win.

In 2015, the adage may no longer be true.

Voters are more impatient than ever. The faith that has been lost cannot be rebuilt, regardless of who is in power. Gillard infamously promised “no carbon tax under a government I lead” and bore the consequences in perpetuity; the Abbott government hasn’t recovered from its 2014 budget disaster. The Queensland election in January again proved that voters will punish a government deemed untruthful, regardless of the consequences. It’s about dishing out the medicine, not waiting for the other side to come good. In these circumstances, Labor may get over the line as a small target, but can it govern from there?

“If you vote Labor at the next election, it’ll be about science, education and health, jobs, small business, the regions and infrastructure. We’re working out policies …”

In a routine interview on Radio National in mid-February, the Opposition leader was still giving little away. While he continued to resist all calls to spell out a concrete program, Shorten, echoing the Edwards view, said that Labor saw education and health not as social justice measures or even a social safety net but as core economic policy. He spoke about infrastructure and a focus on small business before claiming that productivity would lie at the heart of a Labor agenda. He said that Labor measured productivity gains not in terms of driving down wages but in terms of keeping women and older workers engaged and improving training.

Shorten’s dilemma, which is also Labor’s, is ostensibly this: he can present a detailed policy agenda and risk some of his political capital for the chance to govern successfully, if he makes it that far, or he can play the standard hand for an Opposition leader, releasing few policies, aiming to be merely less unpopular than his opponent, understanding that he may have no mandate if he wins.

But Shorten can only be truly credible when he lays out a policy program that explains how he plans to go about what is now, at the bare minimum, $40 billion in budget repair. Polling will provide little guidance. When questioned in detail, Australians consistently say the same thing: they want smaller government and better services. The only conclusion to be drawn is that a leader must choose a program and be willing to stand up and spell it out, over and again.

Shorten faces a nation whose faith in its major parties and their leaders is diminishing, and who may not be open to hearing bad news. He might be surprised, though: it’s been a long time since the public heard a political leader mounting a difficult case for new revenue measures, speaking frankly about the limits of government power, or explaining that you can’t cut taxes and improve services. People might just like it.

If Labor is to win the battle of ideas, first it needs to present some. Labor is a centrist party committed to creating prosperity through the growth of a market economy and to redistributing the fruits of that prosperity to social services and, increasingly, to protect the environment. The Greens’ focus is more pointedly on redistribution (and the environment). The Liberals seem ideologically content only when making punitive cuts. Shorten’s job is to make these distinctions clear.

I first met Bill Shorten not long after the 2007 federal election at a ministerial council meeting on disability services held in the inappropriately plush surrounds of Melbourne’s Langham hotel.

I remember our first exchange, for I was a tad put out. Bill approached and asked awkwardly if I too was “new”. In my third term as an MP by then, I was a bit fed up with constantly being asked if I’d been elected five minutes before. My pique was not just personal, though.

Disability was a tough, thankless portfolio. The room was occupied by earnest triers: the junior state Labor ministers (they were indeed all Labor at the time), public servants and welfare groups who’d been shouldering the burden of disability services for years, seemingly alone. Everyone knew the shocking state of disability support. We were the people who dealt with the families and responded to the crises, the people who’d given up on expecting change. There’d been funding increases over the years but never the great big shake. Disability reform was thought to be politically risky and unaffordable. Radical reform was beyond the realm of most people’s hopes.

It was from this perspective that Shorten was being judged. When Shorten was appointed parliamentary secretary for disabilities and children’s services, an offsider to the minister for community services, Jenny Macklin, it was widely seen as a comedown for him. Shorten had been elected as the member for Maribyrnong in the Ruddslide of 2007. Already bearing a public profile from his work with the Beaconsfield miners and his national leadership of the Australian Workers Union, he was a feted character – a Labor man with society connections and an MBA. People spoke of him as a future leader before he even joined the parliament. He was the inheritor, it was said, of Bob Hawke’s hugely popular place. The expectation among disability types was that this star in our midst wouldn’t be with us for long. People were interested to meet him but there was little hope of change.

Contrary to all those expectations, Shorten did indeed engage. He worked hard, he listened, and he brought about a revolutionary change.

When he began to speak on that day in Melbourne, he did so with a compassion and understanding that was surprising for one so new. He described Australian disability services as being in a shocking state and announced an inquiry to try to sort it out.

In the years that followed, he pushed that inquiry through, and championed the Productivity Commission’s findings for a National Disability Insurance Scheme. He engaged the families and disability groups who needed the support and later, as assistant treasurer, built the economic case and found the money to make the scheme work. It won bipartisan support. By the time it was finalised, Bill Shorten was no longer in the disabilities role and he allowed his new prime minister, Julia Gillard, to take the win.

When no one was expecting it, he quietly stepped up. He had the diligence, the intellect and the courage to see a thankless task to the end. He changed the lives of thousands and had the humility and the loyalty to stand in the background when it was done.

These are not qualities widely expected of political types.

Shorten is not the next Bob Hawke. He may have been in the public eye for a decade now, but he remains an enigmatic figure. In press conferences he adopts an almost pleading earnestness, and his megaphone addresses to bigger crowds have attracted ridicule. In private, though, Shorten is well liked and respected – it’s telling that in the leadership contest of 2013 he gained nearly two thirds of his caucus colleagues’ votes.

The challenges he faces are enormous: a distrustful and inward-looking electorate, a structurally unbalanced budget, a party that for the first time in 100 years is being squeezed from both left and right, and a people looking to their leadership for something different, something more.

Labor’s love lost: Which hill is Labor’s light on again?