Brian Christian Consulting

“Brian, we’re being asked to predict the weather in 2007.”

That statement was made to me in my office shortly after I went to work for the then-TCEQ Chairman as his senior advisor.  The speaker coordinated the various moving parts of the State Implementation Plan for complying with the National Ambient Air Quality Standards—the scientific work, the rule writing, and the occasional black magic needed to make the numbers foot.  He was specifically referring to the modeling necessary to demonstrate Houston-Galveston-Brazoria’s compliance with the one-hour ozone standard by November 2007.  He made that comment in… November 2001.

This memory was conjured recently while I read the news.  During the past few spectacularly bad months, a lot of people have been talking about, and making trillion-dollar decisions based on, projections out into the future.  One outcome of our present situation is that a harsh light has been thrown on modeling and its use in public policy development and government response.

This has me thinking about modeling—the good, the bad, and the ugly.

No doubt—models have their place.  They can inform decisions.  They can help describe what might happen for any-given scenario.  They are often required for regulatory decisions, such as the SIP-planning process.  And, they can be continually refined and improved with better data and assumptions.

I fear, however, that their results are inadequately described by most policymakers, and badly reported on by the media.  This can, at best, lead to a poor understanding by the public of what any given model is, and is not, telling us.  At worst, and depending on what is being modeled, people can be made unduly frightened, with dire consequences to their health and our society.

So, what to do?  I think there are at least three things one should consider when they see the word “modeling” in a study (governmental or NGO), report, or news story.  I would further suggest that these considerations would help modeling entities, government officials, and reporters frame how they talk about modeling results.  Obviously, my wheelhouse is environmental policymaking, not epidemiology or other science.  I do think, however, that the broad concepts below can apply when thinking about the application of any modeling to a problem.  We’ll call them, “The Three Laws for How to Think About Modeling.”

First Law:  All models are wrong; some models are useful.

I do not take credit for this pithy statement.  Several super-smart people at TCEQ have been telling that to anyone who would listen for years.  What they are saying is this–it is a simple fact that no model will describe a guaranteed outcome.  Some models might get close, but others will wildly miss the mark.  Sometimes an invalid model will even accidentally yield the correct answer.  It is crucially important to start from this simplest of concepts.

Second Law:  Models are only as good as the data and assumptions that go into them.

This one is easy.  Trash in, trash out, right?  And, conversely, better data and assumptions grounded in the real-world, better results.  Though an easy point, this often seems completely overlooked when modeling results are being described to the public.  This is partly due to an understandable need to distill complex information into a user-friendly form.  Sometimes it may be for more insidious reasons, such as the desire to push a narrative.  Whatever the case, to anyone I would suggest asking the following questions when you are told a model’s conclusions:

• What were the underlying assumptions that went into the model?
• How good was the data?
• How does this model compare with others?
• What are the model’s limitations because of the assumptions and the data?
• How will the model be continually improved as data gets better?
• Can the model be validated with another dataset?

An aside about journalists.  If you have read any of my other pieces, you know I hold the concept of good journalism in high regard, and journalists to high standards of fairness and accuracy.  When it comes to talking about models, the news media should show greater restraint and provide a more balanced narrative on what any given model may be telling us.  A reporter could start by asking the same questions I have posed above.  Headline writers should also dial-back the screaming headlines (they usually summarize the worst-case result of a model).  Whatever the political slant of a news outlet is (there is, after all no objective journalism), it owes it to its consumers to get it right.  I know asking the media for restraint is a fool’s errand, but I gotta be me.

Third Law:  A model should never be the sole basis of any policy decision.

If only it were so simple, but, no.  The real world is too complex, random, and variable to make important decisions based on only one input.  I recognize, too, that some decisions must be made quickly, but even so, other factors must always be considered.  These include socio-economic impacts, the law, cost, human behaviors, politics, unintended consequences, and relative risks of one course of action over others.  Once upon a time in Texas, we lowered speed limits and limited the hours commercial landscaping could be performed because a model showed there would be an air quality benefit.  How did that work out?

I think these basic laws can be applied by any political leader, government official, reporter, or reader when presented with modeling results.  Indeed, I think they represent a healthy skepticism that is, appropriately, in the spirit of scientific thought and rigor.  Asking these questions can help put, and keep, modeling in its proper place in decision-making.

A final thought that ties to my point above about human behavior.  As I write this, my wife (who is a healthcare professional) observed that humans seem to cope better with the randomness of the world than with a feeling of powerlessness.  I think her profound observation gets to what I attempted to say above, namely that important decisions cannot be based on any single input (like a model).  Indeed, perhaps the paramount consideration for any decision maker is how their decision will affect people and how those people view of the world.  Afterall,  the world is random, complex, confounding, and rarely hews to the designs and plans of governments and the scientists advising them.