Introduction
Empirical research has shown that measures of educational attainment alone may not be sufficient to capture the extent to which human capital triggers economic growth and impacts individual labor market outcomes. Research shows that concrete measures of academic achievement and cognitive skills, along with educational attainment, are strongly correlated with labor market outcomes, such as earnings and unemployment (Currie and Thomas, 2001, Hanushek and Woessmann, 2008, Murnane et al., 2000, Murnane et al., 1995, Neal and Johnson, 1996).1 A number of studies have documented the specific importance of mathematical abilities in adult socioeconomic success (e.g. Ingram and Neumann, 2006, Murnane et al., 1995, Rose and Betts, 2004). Evidence suggests that skills in mathematics have long lasting effects.2
Developing these skills should be of great policy interest, and so should understanding which policy may help (or hinder) the development of these skills. A large body of research in the literature on human capital has investigated the impact of different inputs in the educational production function on achievement. Teacher quality has been shown to be of great importance in predicting the success of students.3 Other types of resources (e.g. per student expenditure, school facilities, class size) have generally been shown to be poor predictors of student performance.4 While the impact of school resources has been widely studied, few studies in economics have addressed the importance of what is being taught and how it is being taught.5
In this paper, we estimate the impact of Québec's (the second most populated province in Canada) ambitious and universal school reform implemented in the early 2000's on children's mathematical ability throughout primary and secondary school. At the time of the reform, the performance of students in the province of Québec was comparable to that of students from the top performing countries in international assessments. Nonetheless, the educational system in Québec was still subject to severe criticism at home due to its alarmingly large high school dropout rate, especially among male students.6 To ensure the success of all students, the province decided to implement an ambitious reform introducing a new program in each and every school across the province which drastically changed the way teaching was delivered to all children in primary and secondary schools. The Québec education program (MELS, 2001, MELS, 2003, MELS, 2007) relied on a socio-constructivist teaching approach focused on problem-based and self-directed learning. This approach mainly moved teaching away from the traditional/academic approaches of memorization, repetitions and activity books, to a much more comprehensive approach focused on learning in a contextual setting in which children are expected to find answers for themselves.
Evidence from Bulle (2011) suggests that most OECD countries are moving away (or have long moved away) from the traditional (more academic) teaching approach. More specifically, the teaching approach promoted by the Québec reform is comparable to the reform-oriented teaching approach in the United States. As of 2006, this approach was widely spread across the United States (although more traditional approaches remained dominant) and it was supported by leading organizations such as the National Council of Teachers of Mathematics, the National Research Council, and the American Association for the Advancement of Science. Yet few studies in economics have addressed the impact of various teaching approaches, let alone the approach promoted by the Québec reform.
The Québec reform/experiment provides important advantages for the purpose of evaluation. First, Québec's Department of Education implemented the reform and all schools (public and private) were compelled to apply the new education program. Combined with a rich data set, this allows us to estimate the impact of a universal reform, on students of different abilities. Second, because teaching in the Rest of Canada (RofC) continued to be delivered in the same way throughout the period, it is possible to estimate the effect of the reform using students from the other provinces as controls. Third, the reform combined with our data set allows us to observe students treated for up to nine years (as opposed to three and less in the studies cited below). This longer observation period also allows us to estimate (to some extent) the long term effects of the reform.7 Fourth, although the treatment period is long, because the reform was implemented in steps, all of our estimates are derived from a comparison of students in the same grade close by in time, which reduces the possible bias induced by other reforms taking place at the same time.
We use the Canadian National Longitudinal Survey of Children and Youth (NLSCY) for the analysis, which provides students’ test scores in mathematics. We estimate the effect of the reform on a standardized measure of mathematical ability using two econometric methods. First, we apply the ‘Difference-In-Differences’ (DID) framework, a method largely used for evaluating the effects of policy changes (Angrist and Krueger, 1999, Blundell and Costa Dias, 2009). Second, the estimations are conducted with the ‘Changes-in-Changes’ (CIC) non parametric estimator, developed by Athey and Imbens (2006), which generalizes the DID model. Using the CIC framework, we assess the impact of the reform at different points on the skills distribution, from the least performing students to high achievers.
Studying this reform contributes to the literature by further identifying the determinants of mathematical abilities, and more specifically by identifying the causal impact of an increasingly popular teaching approach. It is the first paper to exploit a universal school reform of this magnitude to identify the causal effect of the teaching approach on the development of the mathematical skills of students. Our results suggest that the reform had negative effects on the development of students’ mathematical abilities as measured by the CAT/2 test and that the effects were larger the longer the exposure to the reform and hence to the teaching method.
The outline of the paper is as follows. Section 2 highlights the distinctive features of the education system in Québec compared to the other Canadian provinces, and describes the school reform implemented as of 2000. International research on comparable teaching approaches are also discussed. Section 3 exposes the econometric methodology used to identify the causal effect (treatment on the treated effect) of the school reform on achievement in math. Section 4 describes the data set used and presents descriptive statistics of key variables. The estimated effects of the reform are presented in Section 5. In Section 6, we provide further evidence on the impact of the reform using international assessment data and behavioral outcomes available in our main data set. The last section offers concluding remarks.