What is the difference between models and tools? EDR Group has been working in this field for 20 years now, and yet we still see ample confusion on this topic. Here’s a try at answering the question:
- A TOOL helps people carry out a function, so a “decision support tool” can be any method or process that helps people assess alternatives. In the case of transportation investments, this includes benefit-cost calculators (such as Cal-B/C spreadsheet or MBCA from TREDIS), and case studies search and reporting systems (such as EconWorks case studies).
- A MODEL is a special type of tool – a construct that captures relationships between stimuli variables and behavior responses variables, enabling policy analysts to predict the expected effects of proposed projects, policies or other actions. (This can include the behavior of atoms in the atmosphere, tires on icy roads, traffic on a congested highway, or businesses facing a change in transportation costs.) These “if-then” relationships underlie travel demand and air quality models, as well as economic impact models (including TREDIS and REMI).
Models also need a factual basis of observed data. In the case of economic impact models, we need retrospective information on past impacts of actual projects (ex post analysis) to calibrate the response factors enabling predicted future impacts of proposed new projects (ex ante analysis). We’ve been doing both. Staff of EDR Group led the development of the first online national database of transportation project impact case studies (TPICS, now EconWorks) and also pioneered applications of the REMI and later TREDIS predictive economic models. And you can see from the documentation of these predictive models that their response factors draw directly on published statistical research studies that relate transportation price/cost and access changes to productivity, investment and income outcomes.
But using past statistical research is not enough. We also need to validate that economic impact models actually work, and that takes time, since large scale projects can take 10 or 20 years to take place, and we need at least 5-10 years afterwards to observe actual results. At this point, what we do know for sure is that transportation projects do have economic impacts of a magnitude predicted by economic models. We can draw that conclusion by looking at case studies in the EconWorks database -- such as Wisconsin Highway 29, the Boston Central Artery and NY Southern Tier Freeway – and comparing their actual economic impact outcomes to economic impact modeling done before those projects were started. (I plan to assemble more on this matter in a later white paper, so these facts can be more easily found by interested readers.) But being in the right magnitude is not the same as “nailing” a prediction of future changes in the economy.
So, can impact models accurately predict the future? The answer is probably not, for the basic reason that our developing cities and our evolving economy are open systems, and many factors can affect them over time. So there can be wide variation in predicted vs actual outcomes of projects, as there will always be intervening factors that could not have been anticipated at the beginning. But that’s OK – as long as the economic models enable us to consider complex factors and system interactions to make more informed decisions about investment priorities and plans for new projects. And as long as we accomplish that, allowing us to make the best investment and planning decisions possible at a given point in time, then the models will have been useful. A parallel point can be drawn from a discussion of travel model accuracy that was sponsored by FHWA’s Travel Model Improvement Program (TMIP).
Now where do non-model tools fit in? While economic models cannot capture all external factors affecting observed outcomes, we can extract further insights about outside factors by using case study information. In particular, the EconWorks case study search tool provides a treasure trove of insight for planners that no model alone can provide. The tool allows us to see the range of actual observed impacts among roughly similar projects and to read about how some projects had their economic impacts enhanced by positive local factors, while other projects had their economic impacts reduced by negative local factors. As the case studies identify unique, qualitative factors affecting observed outcomes in each case, they provide a context for public discussion regarding how to mitigate potential problems and achieve better results by addressing local land use, business tax/regulation and economic development factors. This aspect makes the case study database particularly useful for early stage public debate, when the very concept of a project is first considered, and before there is enough detail about a proposed investment to support quantitative modeling. To support such uses, the EconWorks toolkit also includes an “Assess My Project” tool that allows interested parties to specify a given type of planned project, and see the range of economic impacts that would be likely to be associated with them based on the available case study experience. This use is discussed in greater detail in the EconWorks Handbook for Practitioners (see pp. 3-4). https://planningtools.transportation.org/files/117.pdf
Yet care must be taken to avoid confusion between the simple presentation of case study results (as in EconWorks) and the sophisticated analytics involved in economic impact models. While there are warnings about these differences in the documentation cited above, some people do not read it. So to simplify the point, consider the graphic to the left. As shown, the EconWorks “Assess My Project” tool is a database summarization process that allows analysts to input information about a type of project, its size and setting, and then see the range of expected outcomes based on case study experience. Yet it totally lacks any consideration of existing transportation systems, their conditions and levels of use, as well as existing economic activities. In other words, it cannot distinguish between (1) a bridge that would serve high traffic volumes, expand worker access and reduce freight costs, and (2) a bridge that would be seldom used by anyone. Quite simply, it is a database lookup tool that can be just right for early stage planning uses as described above, but was never intended to replicate or replace the transportation and economic behavioral relationships embodied in an economic impact model (as shown on the right side of the graphic).
So, for example, the case study database tool can be just right for very early stage planning, where we want to consider bridges in general and do not yet have information on traffic volumes and travel times. But once we get to more detailed consideration of project options and alternatives, then the more specific capabilities of predictive economic impact models become absolutely critical. As professionals, we should work to improve both ex post case study information and ex ante predictive models since the two are complementary to each other and to planning processes.