Economic impact studies are an increasingly common feature of government, regulatory body, business and tribunal decision-making. Governments and regulatory bodies use impact studies to understand the potential effects of their decisions, to add an economic context, and businesses use these studies to gain support for major projects they are undertaking or to advocate for (or against) government policy changes. Tribunals may also use them to take into account the broader economic impact of their decision on a disrupted matter.

Given the increasing prevalence of these types of studies, it’s worth asking: do they accomplish their purpose and can they be relied on as the basis for major decisions? To date, the authors have encountered many economic studies for which the answer to these questions is decidedly ‘no’.  Too many economic studies emphasize the application of off-the-shelf models without providing decision-makers with sufficiently broad economic insight, and so are not suited as a basis for regulatory, policy or board decisions.

When procuring or reviewing economic impact studies, counsel should be aware of the common pitfalls. To help cut through economic jargon and evaluate the methodologies of economic impact studies, here are some essential questions to ask your trusted advisor:

  1. Does your analysis distinguish between the footprint of economic activities and the “creation” of economic impacts?

We often see economic impact studies that centre around the mechanical application of economic models (for instance, Statistics Canada’s Input Output Model) without consideration of their output. Specifically, these models provide estimates of the total economic footprint of an activity, in terms of the labour it employs and the suppliers of the industries it affects. Further analysis is required to determine whether, as is often claimed,  the jobs and output generated from these activities are additive rather than a redirection from other areas of the economy.

Take the simple example of the opening of a big box retailer in a small town. Employment estimates from basic models of the retailer’s impact are often misinterpreted as net new jobs, when in fact some employees are likely to have come from impacted retailers in the area and have simply changed employers. An understanding of the business environment and work force dynamics is a necessity for these types of analysis.

  1. Does your analysis take into account the unique features of the labour market you are analyzing?

Labour markets vary significantly by region, industry and job type, and while  simple economic multiplier-based analyses do not take into account this uniqueness, economists should.

For example, will slack in a local labour market for employees with a certain skill-set mean that the impact of a potential project will be amplified, in terms of avoiding the social and public costs of employment or underemployment? Conversely, could potential job losses from a decision be mitigated based on the demand for the skills of impacted employees? In a recent study in relation to a government initiative, we found that short-term job losses in connection with a major policy change could result in a positive gain for the economy by freeing up in-demand skilled workers  for other sectors, provided relocation or retraining assistance could be provided in some cases. A mechanical analysis could have reached the opposite conclusion without actionable insight for policymakers

  1. Does your analysis take into account the likely response of market participants?

Business decisions and government policy formulation do not take place in a vacuum, and comprehensive economic impact analysis should take into account how competitors, suppliers, employees and governments will respond to the impacts at issue. These responses can fully offset, partially mitigate or significantly exaggerate any impacts in the medium- and long-term, and therefore should be considered in any economic impact analysis on which decisions will be based. Focusing only on short-term impacts can lead to incorrect conclusions and misguided decisions.

In a recent case before a regulatory board, we reviewed an economic impact analysis that failed to consider these dynamics while claiming that a new regulation would benefit consumers. In fact, long-term analysis suggested that the viability of businesses would be threatened and consumers would eventually pay higher prices for reduced variety. A simplistic analysis could have led the board to make a decision based on incorrect economic findings, to the detriment of consumers and the economy as a whole.

  1. Does your analysis consider the likely impact on competition, innovation, the environment and consumers?

A narrow analysis that ignores these factors would fail to provide decision- makers with a holistic view of the impact of their decisions. In a recent case, the Competition Tribunal considered and adopted our analysis of the broader impacts of banning certain practices that the Competition Board deemed to be anti-competitive. On the basis of our analysis the Tribunal decided that the costs to the broader economy are higher than the benefits that would accrue to certain consumers as a result of banning these practices. Analysis that ignores these elements is of limited use for policymakers and tribunals.

We recommend that counsel consult experienced advisors to avoid potentially exposing themselves and their clients to vulnerabilities that may arise during questioning related to submissions to regulatory bodies and tribunals. Similarly, counsel should ensure that studies presented by opposing parties take into account at least these important points to help avoid decisions based on flawed analysis.