We all thought it would be so much better. Having dispatched John Howard, it seemed we were on the cusp of something fine. When I think back to those long, long months of campaigning in 2007 – when Labor took the fight right into Howard’s own backyard of Bennelong – a lot of us dreamed a different dream.

We all knew, as we walked the streets and doorknocked house to house, that Australians were far more complex and interesting and subtle than the odious divide on which Howard had traded for years – the beloved ‘battlers’ versus the despised ‘elites’. We would build the future, go “for the big ones” as Paul Keating did, and redefine the ‘light on the hill’ for a new generation of activists. We would banish the rancour and belligerence.

Four years later, it all sounds pretty hokey. For those of us naive enough to still think of politics as a cause and not a career, it’s hard even to summon up the memory. The bitter reality is that the Howard years are still with us. And then some.

Tony Abbott has morphed into one of the great political ferals while the new Labor dreaming came up with the Malaysia Solution. Julia Gillard has issued a call for the party to recruit 8000 new members. They would join, because …?

Troy Bramston’s Looking for the Light on the Hill: Modern Labor’s Challenges (Scribe, 288pp; $32.95) charts the deep disappointment of supporters as they survey the wreckage that has resulted in a dismal primary level of support that, for months now, has been consistently below 30%.

Bramston argues for boldness, reinvention. He correctly identifies policy timidity and the laziness of Labor’s lost decade in the Opposition years from 1996 to 2007 as a core weakness for Labor in government. The party, he argues, is facing “an existential crisis” with no clarity around core beliefs or identity, and is increasingly crippled by ossified structures.

Bramston’s book is a comprehensive effort: strong on Labor’s history, reverential towards the modern greats – Whitlam, Hayden, Hawke and Keating – and, in the end, more idealistic than fatalistic.

With a background as a party activist and policy adviser (he wrote speeches for Kevin Rudd when he was leader of the Opposition), Bramston has researched widely and taken on board a range of views.

Brian Howe, a minister in both the Hawke and Keating cabinets, says the party needs to recover a “sense that it is primarily a movement, not an institution”. Then there’s John Faulkner, the senior NSW senator who has battled the party’s Neanderthals for years, who says: “I think we’ll survive but survival ain’t enough. I want us to be more than just vaguely viable.” You can hear the gloom in Faulkner’s voice.

For his part, Bramston asks a lot of the right questions. Among them: Why maintain a commitment to the socialist objective? For Immigration Minister Chris Bowen, the answer is clear: “Socialism is a failed idea.” Bowen talks instead about the need to rebrand around what he calls ‘social liberalism’.

But the bitter truth is that the only successful rebranding has been to Labor’s detriment. Thanks to Abbott’s cut-through clarity, Australians now doubt the basic credentials of the Gillard government, seeing it as beholden to Bob Brown and his political agenda.

Having signed an agreement with the Greens (unnecessary and a “huge mistake”, according to Bramston) the prime minister can’t seem to make up her mind about whether the Greens are partners or opponents. Again, something that needs to be thought through before that 8000-membership drive gets underway. People might want to know before they hand over the joining fee.

Coming from New South Wales, a state that has made an artform of decapitating leaders, Bramston doesn’t always follow the logic of his own material and is coy about a rudimentary question. Who owns and wields power in the Labor Party? Who makes the decisions? As Senator Faulkner would say, it ain’t the branch members.

The machine bosses and union leaders, who operate a seigneurial system of preferment and patronage, exist in a world of their own making. It is feudal in construction and delivers exactly what is intended – a quiescent caucus. Bramston seems to understand this. He tells his readers, “the perception, indeed the reality, is that a bunch of powerbrokers can sit around a table in a Chinese restaurant and decide who will enter parliament and who will not.”

Of course, the powerbrokers think this is just fine and dandy. They boast about it. If WikiLeaks is right, Kathy Jackson, the Health Services Union’s national secretary, told US embassy officials that unionists, by dominating pre-selections, are far more influential than parliamentarians. A case of ‘we decide who sits where and the manner of their voting’. It follows that it takes no time at all to knock off a parliamentary leader when the owners ring around the owned. Just ask Morris Iemma. Nathan Rees. Kevin Rudd.

Bramston ignores the destructive handiwork of the powerbrokers. A bit of journalistic nous would have helped him a lot. There aren’t enough ‘yes, but …’ queries.

For a book that is so NSW-centric, the voice that is missing is that of Morris Iemma. The deposed former premier will probably chew his arm off reading onetime NSW and national party secretary Karl Bitar chiding Labor leaders and ministers for being insufficiently courageous in addressing reformist policy!

Iemma was shopped out of office when the Sussex Street bosses went weak at the knees over his attempt to privatise the states’ electricity assets. The favoured assassination weapon is the ‘party research’ that, not surprisingly, always seems to support an argument to ditch the leader.

Which brings us to Rudd, and the events of June 2010. With no source or supporting documentation, Bramston writes: “In the weeks after the May budget the private polling revealed that the government’s vote was in freefall and that around 30 seats were deemed lost or at-risk.” Really? Who sighted the polling? The raw data?

The facts from the published Newspoll of the period from April to June last year tell a very different story. Labor’s lowest polling was recorded over the period from 30 April to 2 May with Newspoll showing the Coalition ahead on a two-party preferred vote of 51/49. There is no gainsaying that this was an ugly time with the Rudd government fighting on multiple fronts and seemingly pleasing no one. For all this, it’s worth paying close attention to the three sets of Newspoll figures in the lead-up to Rudd’s removal on 24 June. They record a consistent rise in Labor’s vote.

Two days before the owners of the Labor Party struck again, this time to remove a first-term prime minister who had defeated the four-term-winning John Howard, the published Newspoll showed a two-party preferred lead for Labor of 52/48. As the then Parliamentary Secretary Bob McMullan told colleagues that week, it was a winning position. In fact, it was consistent with the two-party preferred vote that had secured victory for Labor in November 2007.

Bramston ignores these figures, and instead sticks to the increasingly thin narrative that “caucus members were simply united in their dislike of Rudd”. I can’t recall the simplicity. Or the unity.

John Faulkner, someone Bramston clearly respects, is on the record as saying “the misuse and manipulation of party research to influence internal party affairs or parliamentary party ballots is just plain unconscionable”.

Without a bit more honesty about this, the prospects for any kind of breakthrough reform at Labor’s National Conference next month are not good. 

Bramston’s closing chapters offer a more ambitious reform blueprint: a UK-style direct election of the parliamentary leader by all party members, as well as adopting a proposal that goes back to Gough Whitlam – the direct election of local branch members to the party conference. As for trade union representation at the conference, he proposes reducing this from the present 50% to 30%.

These issues aren’t even up for debate. In December Labor’s powerbrokers will talk the talk on reform but they will convince no one. The continuing denial of the devastating consequences of their cavalier removal of Rudd taints everything.

Australian voters know this, as do the Labor faithful. The latter, as Troy Bramston writes, “still want to change the world”. Before that can happen, Labor itself must change.

Troy Bramston’s 'Looking for the Light on the Hill' New Labor Dreaming