The methods employed to evaluate recycling policy initiatives vary greatly, and differ depending on the questions being asked, the scale of the evaluation and the resources available to conduct an evaluation (Conley et al, 2003). While experimental methods and multivariate correlation analysis have historically been used to establish cause and effect relationships between initiative characteristics and outcomes, there is an increasing emphasis being placed on qualitative models of evaluation (Patton, 1986, Leach 2000).
The following section describes several of the predominant models used to evaluate recycling policy initiatives.
Multivariate Correlation Analysis
Multivariate correlation analysis (within the context of recycling policy) aims to establish causal relationships between the outcomes of a particular initiative with individual project characteristics. This is largely a quantitative exercise, using statistical techniques such as regression and log linear analysis to calculate the strength of the relationship between the dependent (i.e diversion/recycling rates) and independent (i.e. promotion and education rates, curbside collection etc) variables. While this technique remains extremely popular in disciplines such as economics and ecology, its applicability as a standalone measure to issues related to recycling is debated.
One of the primary challenges of correlation analysis is a paucity of reliable data. Such methods require sufficiently large sample sizes to draw statistically meaningful conclusions, and often have difficulty accommodating the complex and dynamic nature of recycling initiatives (Sidique et al, 2010). When multivariate approaches are employed, it is usually based on structured surveys and/or province/state wide data on household recycling activity. Several researchers (see Chen, Rossi, 1987) have been critical of this approach, as there is a propensity to lose sight of “contextual factors and circumstances” when analyzing empirical data (Chen et al, 1987). Conversely, one could contend that quantifiable measures of a project’s success/failure provide objective and easily communicable results. While it is important to recognize the shortcomings of a multi-variate approach, we must be cautious of dismissing it all together. Such techniques have an extremely long history in issues related to resource management, and as such, must remain in our “tool box” of evaluative strategies.
Participatory Approaches to Recycling Policy
Participatory evaluative models of recycling initiatives directly engage recycling stakeholders, soliciting input as to the perceived successes, failures and experiences of a given project. Typically, respondents are asked to participate in surveys or interviews to assess a project’s outcomes, the factors that led to those outcomes, and the appropriateness of the processes used (Lee, 2011). Participatory models may also be used to glean information about stakeholder attitudes, opinions and relationships. Mendoza and Prabhu (2002) have noted that the strength of participatory evaluative models can be attributed to:
- Participatory models are useful in capturing behavioural patterns and change among stakeholders
- Participatory models are effective at capturing people’s perceptions, particularly those that are difficult to quantify
- Participatory models are generally more accommodating and less intimidating to stakeholders
While participatory modelling is decidedly qualitative in its approach, survey responses can be used to help inform quantitative methods such as multivariate correlation analysis (described above).
As discussed by House (1999), the subjective nature of participant perceptions and values may subvert the credibility of a participatory approach. Though the approach is often lauded for capturing the full range of stakeholder experiences, it is seen as a less appropriate mechanism for measuring tangible outcomes (Mendoza et al, 2002). Furthermore, participatory models of evaluation are often resource and time intensive. Depending on the scope of a recycling initiative, it may be difficult to gather responses from a meaningful sample of participants (Conley et al, 2003). With that being said, participatory evaluative models are gaining traction as a preferred approach in assessing the efficacy of recycling initiatives, as they provide greater insights into the opinions and perspectives of recycling stakeholders.
Measuring Tangible Outcomes
Outcome evaluation is often predicated on comparing observed outcomes with desired objectives. As noted by Conley and Moote (2003), outcome evaluation can be applied when outcomes of a given initiative are readily quantified, and where there is sufficient baseline information to allow reliable comparisons over time and between cases. Within the context of recycling initiatives, some quantifiable metrics include:
- Municipal diversion levels
- Recycling program costs
- Access to recycling services
- Household recycling participation rates
Assuming that sufficient baseline data has been collected, two relative system states can be compared (pre and post recycling initiative) to evaluate the efficacy of a given initiative. Outcome evaluations are often seen as more objective than participatory evaluative models, as it is generally not prone to issues of stakeholder bias, values and perceptions. However, critics of outcome based evaluations often question the “black box” nature of the approach (Patton, 1986). Unlike multivariate correlation analysis, outcome-based approaches do not explore the relationship between project outcomes and characteristics (Patton, 1986). As such, evaluators are unable to determine which variable (project characteristic) leads to a given outcome. Relationships under an outcome based approach are inferred, perhaps even erroneously. Furthermore, outcome based approaches give little insight into perceptual factors, like mutual learning among stakeholders, perceived fairness of the process, or outcome and conflict abatement (Conley et al, 2003) . Despite these criticisms, outcome based evaluation remains a popular evaluative approach.
Reconciling two or more evaluative models may be seen as a potential strategy for overcoming some of the methodological shortcomings described above. The intersection of empirical and ethnographic approaches (such as multi-variate and participatory modeling), captures both the nuances and complexity of stakeholder experiences with the empirical rigor of conventional multi-variate analysis (Mendoza et al, 2002). Hybrid approaches attempt to overcome the methodological pitfalls attributable to any one approach. Multivariate correlation analysis is employed to check for the relationships among policy characteristics, while oral interviews establish a contextual narrative among affected stakeholders. Furthermore, incongruences between survey responses and stated experiences (via interviews) can be readily identified and examined further.
While the benefits of a hybrid approach are readily apparent, there remain practical impediments to applying such a model to all recycling policy initiatives. The foremost of these challenges is the cost of undertaking this approach. A combination of both participatory and multivariate models are both resource and time intensive, often requiring a longitudinal approach that may not be feasible for the purposes of informing decision making (Innes, 1999). As such, one must carefully consider the intended purpose and timescale of the evaluation before employing a hybrid evaluative model. I was able to overcome the resource/data gathering challenges traditionally associated with the hybrid model by partnering with Waste Diversion Ontario, who graciously provided me with access to data from the municipal data call. This not only greatly reduced the time and resource burden of acquiring data, but provided me with access to Canada’s largest and oldest database related to municipal waste management.