WHO’S ASSESSING THE ASSESSMENTS?

When the weather forecaster predicts the weather the prediction is soon tested against reality. The people who practise that complex science — or is it art? — soon find out how well it is doing. Ever wonder if the same is true for the complex business of environmental impact assessments?

Despite the inevitable errors, most of us would agree that the predictive ability of the meteorologists has improved. The many factors which interact to shape the atmospheric environment are captured in complex modelling systems. Comparing the actual result with the predicted one allows meteorologists to improve their models and so their predictions. Does the same apply to the complex ecological, economic and social factors which make up an environmental assessment? Given the cost and effort they involve, how sure are we that the predictions are accurate? And how are they improved? What is the feedback loop, the iterative process, for evaluating environmental assessments? How do policy makers, project engineers and planners improve the quality of their assessment work, and how do they acquire long-term data needed to test their initial hypotheses?

Joanna Treweek is a research scientist with the Institute of Terrestrial Ecology at Monks Wood in Cambridgeshire in the U.K. In an academic paper published in the journal Impact Assessment in 1995 she characterizes impact assessments as “essentially an exercise in prediction, but predictive ability is generally recognized as weak, largely due to the complexity of ecosystems and the shortage of long-term datasets.”

Treweek also says, “As a general rule, ecological assessment procedures end with the submission of the impact statement, and remarkably few studies have been carried out to test the accuracy of impact predictions to monitor the longer-term consequences of development actions.” It is important to note that her remark applies to ecological assessments in the U.K. Would it also be true in Canada? Is it true of the many other aspects of impact assessment? The answer appears to be a resounding maybe.

In Ontario the stated purpose of the Environmental Assessment Act is “the betterment of the people … by providing for the protection, conservation and wise management … of the environment.” “Environment” is defined very broadly and includes “social, economic and cultural conditions” as well as the built environment. Impact assessments study the various aspects of a project. They allow the proponent to make changes which limit the project’s undesirable impacts. The impact assessments are, however, hypotheses and need to be tested. Wise management depends on cumulative knowledge developed over time by the iterative process of testing the assessment hypothesis against actual follow-up data. If this does not happen, how wise is the management?

Ann Joyner lectures in environmental impact assessment and bioregional planning in York University’s Faculty of Environmental Studies and is a partner (environmental planning) at Dillon Consulting. Joyner does not know of any follow-up studies which looked at earlier environmental impact assessments to see how accurate the predictions were. A key reason for this lack, she believes, is budgetary. Most projects do not provide consulting companies with the funds for follow-up studies. “We use formulas, for example, in the case of waste sites to predict how far from residential areas they should be sited for noise, dust, traffic etc. but seldom go back afterwards to find out whether the residents were affected in the ways predicted.”

Joyner’s opinion is echoed by others. Paul McDonald is president of the Ontario Association for Impact Assessment (OAIA). This organization arranges seminars and other opportunities for its members to share their knowledge in a professional forum. While McDonald believes that environmental assessments have a high degree of reliability, he also agrees that the field has failed to integrate knowledge from past into present environmental assessments. “We tend to (conduct) study after study, over and over. There is no mechanism for taking what has been learned and plugging it back in. The process is disjointed, project to project.” His colleague in the organization, Betty Hansen of Ecosystems International, agrees that we need a more integrated approach.

Follow-up studies involving the entire environmental impact assessment of a project may be rare, but, according to McDonald, there are instances where monitoring is adequate. This is particularly true where monitoring is required as a condition of a compliance permit or a regulation. The use of the monitoring data has led, for example, to improvements in the development of water treatment projects.

It is interesting that the monitoring data needed for improving the environmental assessment process is produced when there are regulatory requirements for it. The OAIA wants governments, federal or provincial, to take more responsibility for follow-up in the interests of the public good. The organization wants the same process that is already used in the case of social programs — careful monitoring to make sure that health, education, etc. are producing the intended benefits — used for the environment. This is an unlikely prospect at the moment, at least in Ontario where the government has cut back environmental programs. The chair of the federal Commons Environment Committee recently voted against the newly passed Environmental Protection Act on the grounds that the Act did not protect the environment. This does not bode well for stronger regulation.

In the absence of monitoring data, what other opportunities are there for improving environmental assessments? Two come to mind: the public consultation process and the complaints process. Both of these are only as effective as the people involved. During the environmental assessment a well designed consultation process can produce valuable changes. Complaints, on the other hand, may take the form of opposition, leading in extreme cases to outright cancellation of a project. Or the concerns expressed by affected parties may result in adjustments to the proposed activity.

After a project is completed, complaints also form a de facto kind of monitoring. Probably the least welcome and effective way of monitoring a project, complaints nevertheless provide some follow-up data about an environmental impact assessment, incomplete and unsystematic as that data may be.

While the human parties affected have some ability to protect the social, cultural and economic aspects of their environment, other elements affected may not be so lucky. The lack of systematic monitoring by government or other interested parties can have grave consequences for the physical environment which the Environmental Assessment Act is supposed to protect. Plants and animals don’t have much say unless people decide to complain on their behalf. This hasn’t happened often enough judging by the opinions expressed about ecological impact assessments.

The work by Joanne Treweek quoted earlier reaches some grim conclusions about ecological assessments. The lack of monitoring, according to Treweek, “Has greatly hampered the development of predictive methods and effectively renders ecological impact assessment useless as a mechanism for environmental regulation.”

As if this stunning statement were not enough, Treweek quotes R.C. Buckley who published the results of another study in 1991. Buckley reviewed 181 Australian environmental impact statements and found that predictions were less than 50 per cent accurate on average, and occasionally more than two orders of magnitude out. In other words, one might say, the results were worse than they would have been by basing decisions on the flip of a coin. While these studies deal with the situation abroad, there is no reason to believe things are any better in Canada.

The accuracy of the weather forecast is quickly verified, but it takes time for the negative impacts of engineering projects to become clear. We now, however, have plenty of evidence to ponder. Hydro-electric dams have
silted up producing less electricity than planned. Communities and cultures have been disturbed never to recover. Natural environments have been effectively destroyed. Some of these works were undertaken before the environmental assessment process existed. Indeed the process came into being because of these consequences. For other locations, environmental assessments exist but without a follow-up process to determine their effectiveness, their use is limited. Without the feedback loop which improves the accuracy of the predictions made by environmental assessments there is little chance the profession can improve its collective knowledge.

The environmental assessment process is no doubt a complex one, but so is the weather. With improved computer models and exciting technical aids like geographical information systems, it is possible to improve the state of our environmental assessment knowledge if the regulatory environment provides the long-term data needed for the task. Wise management will only be possible when the mechanism exists to generate the required wisdom.CCE

Rosalind Cairncross, P.Eng. is a contributing editor of Canadian Consulting Engineer.

By Rosalind Cairncross, P.Eng.

Comments on the above article or any other matter are welcome. Please write to the Editor, Canadian Consulting Engineer, 1450 Don Mills Road, Toronto, Ont., M3B 2X7.

Or e-mail

bledger@corporate.southam.ca