Economics of Education Review

Elsevier

Published by

Elsevier

Highlights

  • •
    We evaluate the effects of teaching method reform on math scores in grades 2–10.
  • •
    Evaluation based on a natural experiment implemented in the early 2000's in Canada.
  • •
    Impact estimated for the distribution of scores and mean scores.
  • •
    Strong negative effects estimated on all points of the distribution and means.
  • •
    More negative effect obtained in the later grades of high school.

Abstract

We investigate the impact of an ambitious provincial school reform in Canada on students’ mathematical achievements. It is the first paper to exploit a universal school reform of this magnitude to identify the causal effect of a widely supported teaching approach on students’ math scores. Our data set allows us to differentiate impacts according to the number of years of treatment and the timing of treatment. Using the changes-in-changes model, we find that the reform had negative effects on students’ scores at all points on the skills distribution and that the effects were larger the longer the exposure to the reform.

JEL classification

A21
;
I22

Keywords

Natural experiment
;
Teaching reform
;
Math scores
;
Distributional effect

Introduction

Empirical research has shown that measures of educational attainment alone may not be sufficient to capture the extent to which human capital triggers economic growth and impacts individual labor market outcomes. Research shows that concrete measures of academic achievement and cognitive skills, along with educational attainment, are strongly correlated with labor market outcomes, such as earnings and unemployment (Currie and Thomas, 2001, Hanushek and Woessmann, 2008, Murnane et al., 2000, Murnane et al., 1995, Neal and Johnson, 1996).1 A number of studies have documented the specific importance of mathematical abilities in adult socioeconomic success (e.g. Ingram and Neumann, 2006, Murnane et al., 1995, Rose and Betts, 2004). Evidence suggests that skills in mathematics have long lasting effects.2
Developing these skills should be of great policy interest, and so should understanding which policy may help (or hinder) the development of these skills. A large body of research in the literature on human capital has investigated the impact of different inputs in the educational production function on achievement. Teacher quality has been shown to be of great importance in predicting the success of students.3 Other types of resources (e.g. per student expenditure, school facilities, class size) have generally been shown to be poor predictors of student performance.4 While the impact of school resources has been widely studied, few studies in economics have addressed the importance of what is being taught and how it is being taught.5
In this paper, we estimate the impact of Québec's (the second most populated province in Canada) ambitious and universal school reform implemented in the early 2000's on children's mathematical ability throughout primary and secondary school. At the time of the reform, the performance of students in the province of Québec was comparable to that of students from the top performing countries in international assessments. Nonetheless, the educational system in Québec was still subject to severe criticism at home due to its alarmingly large high school dropout rate, especially among male students.6 To ensure the success of all students, the province decided to implement an ambitious reform introducing a new program in each and every school across the province which drastically changed the way teaching was delivered to all children in primary and secondary schools. The Québec education program (MELS, 2001, MELS, 2003, MELS, 2007) relied on a socio-constructivist teaching approach focused on problem-based and self-directed learning. This approach mainly moved teaching away from the traditional/academic approaches of memorization, repetitions and activity books, to a much more comprehensive approach focused on learning in a contextual setting in which children are expected to find answers for themselves.
Evidence from Bulle (2011) suggests that most OECD countries are moving away (or have long moved away) from the traditional (more academic) teaching approach. More specifically, the teaching approach promoted by the Québec reform is comparable to the reform-oriented teaching approach in the United States. As of 2006, this approach was widely spread across the United States (although more traditional approaches remained dominant) and it was supported by leading organizations such as the National Council of Teachers of Mathematics, the National Research Council, and the American Association for the Advancement of Science. Yet few studies in economics have addressed the impact of various teaching approaches, let alone the approach promoted by the Québec reform.
The Québec reform/experiment provides important advantages for the purpose of evaluation. First, Québec's Department of Education implemented the reform and all schools (public and private) were compelled to apply the new education program. Combined with a rich data set, this allows us to estimate the impact of a universal reform, on students of different abilities. Second, because teaching in the Rest of Canada (RofC) continued to be delivered in the same way throughout the period, it is possible to estimate the effect of the reform using students from the other provinces as controls. Third, the reform combined with our data set allows us to observe students treated for up to nine years (as opposed to three and less in the studies cited below). This longer observation period also allows us to estimate (to some extent) the long term effects of the reform.7 Fourth, although the treatment period is long, because the reform was implemented in steps, all of our estimates are derived from a comparison of students in the same grade close by in time, which reduces the possible bias induced by other reforms taking place at the same time.
We use the Canadian National Longitudinal Survey of Children and Youth (NLSCY) for the analysis, which provides students’ test scores in mathematics. We estimate the effect of the reform on a standardized measure of mathematical ability using two econometric methods. First, we apply the ‘Difference-In-Differences’ (DID) framework, a method largely used for evaluating the effects of policy changes (Angrist and Krueger, 1999, Blundell and Costa Dias, 2009). Second, the estimations are conducted with the ‘Changes-in-Changes’ (CIC) non parametric estimator, developed by Athey and Imbens (2006), which generalizes the DID model. Using the CIC framework, we assess the impact of the reform at different points on the skills distribution, from the least performing students to high achievers.
Studying this reform contributes to the literature by further identifying the determinants of mathematical abilities, and more specifically by identifying the causal impact of an increasingly popular teaching approach. It is the first paper to exploit a universal school reform of this magnitude to identify the causal effect of the teaching approach on the development of the mathematical skills of students. Our results suggest that the reform had negative effects on the development of students’ mathematical abilities as measured by the CAT/2 test and that the effects were larger the longer the exposure to the reform and hence to the teaching method.
The outline of the paper is as follows. Section 2 highlights the distinctive features of the education system in Québec compared to the other Canadian provinces, and describes the school reform implemented as of 2000. International research on comparable teaching approaches are also discussed. Section 3 exposes the econometric methodology used to identify the causal effect (treatment on the treated effect) of the school reform on achievement in math. Section 4 describes the data set used and presents descriptive statistics of key variables. The estimated effects of the reform are presented in Section 5. In Section 6, we provide further evidence on the impact of the reform using international assessment data and behavioral outcomes available in our main data set. The last section offers concluding remarks.

Organizational access

Get full-text access by signing in with your organisationAccess through your organization

Other access options

Purchase PDF

Section snippets

Québec's school system and teaching reform

In Canada, education is regulated and administered at the provincial level. The overall structure of the education system is comparable across all ten provinces, except for Québec where it is slightly different. In all of Canada, children start school in kindergarten at age 5, and then move on to primary school, where they complete six years of education (grades 1–6); children then pursue their education in high school. In Québec, high school consists of five years of education, grades 7–11,

Empirical strategy

In economics, difference-in-differences (DID) methods have often been used to estimate the effects of policy reforms. Angrist and Krueger (1999) and Blundell and Costa Dias (2009) describe applications and give an overview of the methodology. This approach can be used in settings where some individuals of a population are subject to a policy reform (or a treatment) while others are not, and comparable groups of individuals are observed prior to the policy intervention. The standard DID has

Data set

The data set used for our empirical analysis is Statistics Canada's National Longitudinal Survey of Children and Youth (NLSCY),21 a long-term biennial survey designed to provide information about the development and well-being of Canadian children and youth. The survey covers a comprehensive range of topics including childcare, schooling, physical development, cognitive skills and behavior of the child as

Effects of the reform on math scores

Table 5 presents the empirical results using DID and MDID. Estimated impacts using DID and MDID with three matching techniques, suggest that the reform had significant negative effects on mathematical abilities. Given the complex sampling design of the NLSCY, all estimations are performed using the sampling weights provided by Statistics Canada. To account for the clustering and stratification of the NLSCY, the standard errors are estimated using the 1000 bootstrap weights provided by

Cognitive measures

The NLSCY is not the only source of information providing evidence on the reform. International assessments in which Canada participated can also be used for the analysis. Only a few of the ten provinces participated in the TIMSS.41

Conclusion

We estimated the impact of the Québec school reform, based on a complete revision of teaching methods, on grade 2–10 students using math scores provided by the NLSCY. To our knowledge, no econometric evidence-based evaluation of the reform has been conducted to date. We find strong evidence of negative effects of the reform on the development of students’ mathematical abilities. More specifically, using the changes-in-changes estimator, we show that the impact of the reform increases with

Acknowledgements

We gratefully acknowledge financial support from the Fonds québécois de la recherche sur la société et la culture (grant 2011-SE-137016). We thank the Québec Inter-University Centre for Social Statistics (QICSS) for great support throughout this project. The analysis is based on Statistics Canada's National Longitudinal Survey of Children and Youth (NLSCY) restricted-access Micro Data Files available at the QICSS. All computations on these micro-data were prepared by the authors who assume

References (54)

View full text