This moment is a turning point in the climate change debate. Not because the report released Monday addresses every concern raised by critics of the Intergovernmental Panel on Climate Change (IPCC), but because it knocks the IPCC off its pedestal.

Those who challenge the IPCC’s authority are often ignored. Numerous science academies have blessed its efforts, so who are we to question? This week those academies began to act like grownups in relation to this wayward child. The report, authored by a committee assembled by the InterAcademy Council (a collection of science bodies from around the world), blows smoking holes through just about everything the IPCC’s chairman, Rajendra Pachauri, has been telling us. [113-page report PDF]

(Feb. 2, 2011 note: A later version of the report, dated October 2010 and consisting of 123 pages, was subsequently uploaded to the identical web address.)

TRANSPARENCY
He boasts that his organization carries out its work with “complete transparency.” But this report says transparency is in short supply. Some stages of the IPCC process, it finds, “are poorly understood, even to many scientists and government representatives who participate in the process.”

The report says the IPCC has never established any formal criteria for selecting its most senior personnel, its lead authors, or other key participants. Nor are there any guidelines about what scientific and technical information the IPCC should consider when it carries out its literature review. How these decisions have been made for the past two decades is, therefore, anyone’s guess – a situation rather opposite to complete transparency.

The report says a preliminary outline is drawn up by a select group of individuals at the beginning of the IPCC process, but how this happens – and who participates – is a mystery to those who aren’t invited. Nor does anything in the following sentence provide comfort to IPCC partisans:

The absence of a transparent author selection process or well-defined criteria for author selection can raise questions of bias and undermine the confidence of scientists and others in the credibility of the assessment…

PEER-REVIEWED LITERATURE
In February 2008 Pachauri declared to a committee of the North Carolina legislature (as he has in many other contexts before and since), that:

…we carry out an assessment of climate change based on peer-reviewed literature, so everything that we look at and take into account in our assessments has to carry [the] credibility of peer-reviewed publications, we don’t settle for anything less than that. [bold added]

But the InterAcademy report matter-of-factly tells the world that an analysis of the IPCC’s third assessment report found only 84% of the source material cited by Working Group 1 was peer-reviewed, only 59% cited by Working Group 2 was, and only 36% cited by Working Group 3 met this standard. (An analysis of the IPCC’s fourth assessment report references, organized by yours truly, produced similar results.)

Procedures regarding the use of non-peer-reviewed literature are in place, but the report says “it is clear that these procedures are not always followed.” The rules say non-peer-reviewed sources are supposed to be identified as such when listed among the IPCC’s references. Yet the InterAcademy report says it “found few instances of information flagged” in this manner. As in almost none. According to my colleague, Hilary Ostrov, only 6 of the 5,587 non-peer-reviewed references in the 2007 IPCC report were properly identified.

In a nutshell, the IPCC doesn’t follow it’s own procedures. Or, in the more diplomatic phrasing of the report: “stronger mechanisms for enforcing [these procedures] are needed.”

ROBUST PROCESSES
Pachauri also told the North Carolina lawmakers that the IPCC’s “writing and review process is very robust, very vigorous.” Yet the InterAcademy report confirms that, no matter how loudly the IPCC’s expert reviewers and each chapter’s review editors might protest, the lead authors “have the final say on the content of their chapter.” In other words, the IPCC’s vaunted review process amounts to window-dressing.

In June 2007, Pachauri pompously told an interviewer:

I can’t think of a better process. There is not a parallel on this planet, in any field of endeavour as you have in the case of the IPCC.

Yet the InterAcademy report finds “significant shortcomings in each major step of [the] IPCC’s assessment process.” For example, it notes there’s “near universal” agreement that the IPCC needs “to strengthen the authority of the Review Editors.” But doing so, it admits, still “would not make the review process truly independent.” This is because the IPCC’s review editors are selected by other IPCC personnel:

To be independent, the selection of Review Editors would have to be made by an individual or group that is not engaged in writing the report, and Review Editors would report directly to that individual or group.

Another key recommendation is that, from now on, IPCC lead authors should “document that they have considered the full range of thoughtful views, even if these views do not appear” in the final IPCC reports. The InterAcademy committee observes that the IPCC’s embarrassing Himalayan glacier error could have been avoided had it merely listened to its own expert reviewers. The mistake was noticed, but the IPCC “did not change the text.”

In that instance alone, the IPCC system failed in three ways. First, the IPCC authors chose to rely on an unsubstantiated claim in a non-peer-reviewed document. Then these authors failed to take seriously the feedback from the IPCC’s expert reviewers – who pointed out that peer-reviewed material contained more cautious and equivocal conclusions. Finally, the review editors for that chapter failed to ensure that the expert feedback was properly addressed.

One of the more damning sections of the InterAcademy report says the IPCC claimed to have “high confidence in some statements for which there is little evidence.” “Many of the 71 conclusions” in one section of the IPCC report, it says, “are imprecise statements” that contain too little detail to be meaningful. Thus:

The Working Group II Summary for Policy Makers in the Fourth Assessment Report contains many vague statements of “high confidence” that are not supported sufficiently in the literature, not put into perspective, or are difficult to refute. [bold added]

This situation occurred, says the InterAcademy committee, partly because IPCC authors failed to follow existing IPCC guidelines.

Another area of concern relates to the fact that, despite the highly contested nature of the climate debate, and that billions in expenditures around the world are profoundly influenced by the IPCC’s findings, this organization has no conflict-of-interest policy.

In a February 4, 2010 interview with The Economist, Pachauri was asked: “Isn’t it rather remarkable that you should have this organisation that does not have any procedure for dealing with conflict of interest”? His response to this line of questioning took the form of declarations such as:

I think if the governments who govern the IPCC determine that there should be something of this nature I’m sure that will be put in place.

…Why would I raise something, unless there is a reason for me to raise it?

…So I’ve never felt the need for it. If somebody else feels the need for it go ahead. My behaviour is above reproach…

The InterAcademy report says the IPCC should “adopt a rigorous conflict of interest policy that applies to all individuals directly involved in the preparation of IPCC reports.” The committee didn’t investigate complaints it received regarding the fact that individuals who write IPCC assessments routinely pass judgment on their own work (as well as that of their critics). But it says the IPCC should “pay special attention to issues of independence and bias to maintain the integrity of, and public confidence in, its results” [italics added].

SUBTLE, BUT DEADLY
In many ways this is a subtle report. It doesn’t say outright that Pachauri should resign. Rather, it says no chairman should serve more than a single term  – and that a “12-year appointment (two terms) is too long for a field as dynamic and contested as climate change.” Pachauri is well into his second term and, in the event that he is not replaced, will be what the report describes as “the leader and the face” of the IPCC until 2014.

The InterAcademy committee admits it had only four months to carry out its investigation – and that some areas of concern deserve further scrutiny. It says that “significant improvements” of the IPCC process are “necessary.” Even with these improvements, it acknowledges that the IPCC’s review process will still not be “truly independent.” Moreover, it recommends the adoption of new policies while observing that the already existing IPCC policies aren’t being followed.

Even though at least one media account has characterized this as a “damning report,” the InterAcademy committee nevertheless declares that “the IPCC assessment process has been successful overall and has served society well.”

Perhaps that was the politic thing for them to say. But we the public must make up our own minds on that question. And what is beyond dispute is that the IPCC’s high priest, Rajendra Pachauri, has spent years jetting around the world making declarations that this report firmly contradicts.

Let us not forget that we have been advised – by no less an authority than Pachauri himself – that the reason we should believe the IPCC’s global warming prognostications is because the IPCC’s process is so rigorous, so airtight, so unparalleled on the planet. As he argued in 2007:

So you can’t think of a more transparent process, you can’t think of a better set of qualified people than what we have in the IPCC. I would only put that forward as valid reasons to accept the science and the scientific assessments that are carried out. [bold added]

Three years ago, Pachauri said the nature of the IPCC process was a valid reason to “accept the science.” Now that we know this process is deeply flawed, it follows that this is a valid reason not to accept the science.