by Judith Curry, President (co-owner) of Climate Forecast Applications Network (CFAN) and previously Professor and Chair of the School of Earth and Atmospheric Sciences at the Georgia Institute of Technology.
JC's column puts those who like to accuse others of being "climate deniers" in perspective. Perhaps they are the climate deniers.
Here are some excerpts.
--------------------------------------------------------
I’ve just completed rereading Vol I of the NCA4. There is so much here of concern that it is difficult to know where to start.
-----
To illustrate the overconfidence problem with the NCA4 Report, consider the following Key Conclusion from Chapter 1 Our Globally Changing Climate:
“Longer-term climate records over past centuries and millennia indicate that average temperatures in recent decades over much of the world have been much higher, and have risen faster during this time period, than at any time in the past 1,700 years or more, the time period for which the global distribution of surface temperatures can be reconstructed. (High confidence)”
This statement really struck me, since it is at odds with the conclusion from the IPCC AR5 WG1 Chapter 5 on paleoclimate:
“For average annual NH temperatures, the period 1983–2012 was very likely the warmest 30-year period of the last 800 years (high confidence) and likely the warmest 30-year period of the last 1400 years (medium confidence).
While my knowledge of paleoclimate is relatively limited, I don’t find the AR5 conclusion to be unreasonable, but it seems rather overconfident with the conclusion regarding the last 1400 years. The NCA4 conclusion, which is stronger than the AR5 conclusion and with greater confidence, made me wonder whether there was some new research that I was unaware of, and whether the authors included young scientists with a new perspective.
Fortunately, the NCA includes a section at the end of each Chapter that provides a traceability analysis for each of the key conclusions:
“Traceable Accounts for each Key Finding: 1) document the process and rationale the authors used in reaching the conclusions in their Key Finding, 2) provide additional information to readers about the quality of the information used, 3) allow traceability to resources and data, and 4) describe the level of likelihood and confidence in the Key Finding. Thus, the Traceable Accounts represent a synthesis of the chapter author team’s judgment of the validity of findings, as determined through evaluation of evidence and agreement in the scientific literature.”
Here is text from the traceability account for the paleoclimate conclusion:
“Description of evidence base. The Key Finding and supporting text summarizes extensive evidence documented in the climate science literature and are similar to statements made in previous national (NCA3) and international assessments. There are many recent studies of the paleoclimate leading to this conclusion including those cited in the report (e.g., Mann et al. 2008; PAGES 2k Consortium 2013).”
“Major uncertainties: Despite the extensive increase in knowledge in the last few decades, there are still many uncertainties in understanding the hemispheric and global changes in climate over Earth’s history, including that of the last few millennia. Additional research efforts in this direction can help reduce those uncertainties.”
“Assessment of confidence based on evidence and agreement, including short description of nature of evidence and level of agreement : There is high confidence for current temperatures to be higher than they have been in at least 1,700 years and perhaps much longer.
I read all this with acute cognitive dissonance. Apart from Steve McIntyre’s takedown of Mann et al. 2008 and PAGES 2K Consortium (for the latest, see PAGES2K: North American Tree Ring Proxies), how can you ‘square’ high confidence with “there are still many uncertainties in understanding the hemispheric and global changes in climate over Earth’s history, including that of the last few millennia”?
Further, Chapter 5 of the AR5 includes 1+ pages on uncertainties in temperature reconstructions for the past 200o years (section 5.3.5.2), a few choice quotes:
“Reconstructing NH, SH or global-mean temperature variations over the last 2000 years remains a challenge due to limitations of spatial sampling, uncertainties in individual proxy records and challenges associated with the statistical methods used to calibrate and integrate multi-proxy information”
“A key finding is that the methods used for many published reconstructions can underestimate the amplitude of the low-frequency variability”
“data are still sparse in the tropics, SH and over the oceans”
“Limitations in proxy data and reconstruction methods suggest that published uncertainties will underestimate the full range of uncertainties of large-scale temperature reconstructions.”
Heck, does all this even justify the AR5’s ‘medium’ confidence level?
I checked the relevant references in the NCA4 Chapter 1; only two (Mann et al., 2008; PAGES 2013), both of which were referenced by the AR5. The one figure from this section was from — you guessed it — Mann et al. (2008).
I next wondered: exactly who were the paleoclimate experts that came up with this stuff? Here is the author list for Chapter 1:
Wuebbles, D.J., D.R. Easterling, K. Hayhoe, T. Knutson, R.E. Kopp, J.P. Kossin, K.E. Kunkel, A.N. LeGrande, C. Mears, W.V. Sweet, P.C. Taylor, R.S. Vose, and M.F. Wehner
I am fairly familiar with half of these scientists (a few of them I have a great deal of respect for), somewhat familiar with another 25%, and unfamiliar with the rest. I looked these up to see which of them were the paleoclimate experts. There are only two authors (Kopp and LeGrande) that appear to have any expertise in paleoclimate, albeit on topics that don’t directly relate to the Key Finding. This is in contrast to an entire chapter in the IPCC AR5 being devoted to paleoclimate, with substantial expertise among the authors.
A pretty big lapse, not having an expert on your author team related to one of 6 key findings. This isn’t to say that a non-expert can’t do a good job of assessing this topic with a sufficient level of effort. However the level of effort here didn’t seem to extend to reading the IPCC AR5 Chapter 5, particularly section 5.3.5.2.
Why wasn’t this caught by the reviewers? The NCA4 advertises an extensive in house and external review process, including the National Academies.
I took some heat for my Report On Sea Level Rise and Climate Change, since it had only a single author and wasn’t peer reviewed. Well, the NCA provides a good example of how multiple authors and peer review is no panacea for providing a useful assessment report.
And finally, does this issue related to whether current temperatures were warmer than the medieval warm period really matter? Well yes, it is very important in context of detection and attribution arguments (which will be the subject of forthcoming posts).
This is but one example of overconfidence in the NCA4.
-----
Nearly everyone is overconfident. See these previous articles:- The lure of incredible certitude
- We are all confident idiots.
- It pays to be overconfident, even if you have no idea what you are doing
I don’t regard myself as overconfident with regards to climate science; in fact some have accused me of being underconfident. My experience in owning a company that makes weather and climate predictions (whose skill is regularly evaluated) has been extremely humbling in this regard. Further, I study and read the literature from philosophy of science, risk management, social psychology and law regarding uncertainty, evidence, judgement, confidence, argumentation.
The most disturbing point here is that overconfidence seems to ‘pay’ in terms of influence of an individual in political debates about science. There doesn’t seem to be much downside for the individuals/groups to eventually being proven wrong. So scientific overconfidence seems to be a victimless crime, with the only ‘victim’ being science itself and then the public who has to live with inappropriate decisions based on this overconfident information
So what are the implications of all this for understanding overconfidence in the IPCC and particularly the NCA? Cognitive biases in the context of an institutionalized consensus building process have arguably resulted in the consensus becoming increasingly confirmed in a self-reinforcing way, with ever growing confidence. The ‘merchants of doubt’ meme has motivated activist scientists (as well as the institutions that support and assess climate science) to downplay uncertainty and overhype confidence in the interests of motivating action on mitigation.
There are numerous strategies that have been studied and employed to help avoid overconfidence in scientific judgments. However, the IPCC and particularly the NCA introduces systemic bias through the assessment process, including consensus seeking.
As a community, we need to do better — a LOT better. The IPCC actually reflects on these issues in terms of carefully considering uncertainty guidance and selection of a relatively diverse group of authors, although the core problems still remain. The NCA appears not to reflect on any of this, resulting in a document with poorly justified and overconfident conclusions.
Climate change is a very serious issue — depending on your perspective, there will be much future loss and damage from either climate change itself or from the policies designed to prevent climate change. Not only do we need to think harder and more carefully about this, but we need to think better, with better ways justifying our arguments and assessing uncertainty, confidence and ignorance.
Sub-personal biases are unavoidable, although as scientists we should work hard to be aware and try to overcome these biases. Multiple scientists with different perspectives can be a big help, but it doesn’t help if you assign a group of ‘pals’ to do the assessment. The issue of systemic bias introduced by institutional constraints and guidelines is of greatest concern.
The task of synthesis and assessment is an important one, and it requires some different skills than a researcher pursuing a narrow research problem. First and foremost, the assessors need to do their homework and read tons of papers, consider multiple perspectives, understand sources of and reasons for disagreement, play ‘devils advocate’, and ask ‘how could we be wrong?’
Instead, what we see in at least some of the sections of the NCA4 is bootstrapping on previous assessments and then inflating the confidence without justification.
No comments:
Post a Comment