Calculating the Productivity and Efficiency of an Educational Product: Transposition of Mario Godard’s Method

The Office of the educational material’s approval is responsible for approving educational material. The approval process does not include use of the material by students. The productivity of educational products is unknown. The goal of this article is to find a way to calculate the productivity and efficiency of an educational product, to improve it in order to provide a truly effective tool. To do this, we were inspired by the book written by Professor Mario Godard (2010). We will propose ways to measure the productivity and efficiency of an educational product. Finally, we will discuss the difficulties caused by this calculation. RÉSUMÉ. Le Bureau d’approbation du matériel didactique a pour tâche d’approuver le matériel didactique. Cette évaluation ne comprend pas une utilisation par les élèves. La productivité et l’efficience du processus de conception d’un produit pédagogique et l’impact de l’utilisation de ce produit sur la productivité du processus d’utilisation sont inconnus. Ce texte a pour objectif de trouver une façon de calculer la productivité et l’efficience du processus de conception d’un produit et l’impact de l’utilisation d’un produit pédagogique sur la productivité du processus de formation. Pour ce faire, nous nous sommes inspirés du livre du professeur Mario Godard (2010). Indexing terms/


INTRODUCTION
Since 1980, the role of the Office of the educational material's approval is to evaluate and approve educational material. A judgment is made based on several criteria including educational aspects. So, the approach of the proposed material must be compatible with the learning by skills recommend by the Programme de formation de l'école québécoise. The prescribed contents of the Program must be in it, the evaluation activities must contribute to the development of skills, the material must provide cultural references, the didactic contents must be accurate and the material must facilitate teaching. These educational aspects are analyzed by educational advisers and teachers. In Quebec, educational reforms occur approximately every 20 years. We can thus consider an educational product to have a maximum expected lifetime of about two decadesor a generation. In summary, it takes three months for a group of specialists to evaluate material that may be in our classrooms for 20 years.
During these three months of evaluation, the teaching material is never tested by a student, its main user. At no time is it used in a classroom, its main user environment. By analogy, in the medical field, a medicine that has never been tested on an animal or a human and whose side effects and efficiency we ignore, is never marketed. Educational products are usually not tested and that is the reality. In education, the economics of education, created in the 60s, focused on the evaluation of the efforts granted regarding results (Michaud 1981). More specifically, they were interested in the profitability of education and employability of students (Cousineau 1984;Johnes and Johnes 2004;Lemelin 1988;Paul 2007;Vaillancourt and Henriques 1986). It was performed to show the capacity of students to find employment job according to their educational level, the level of health of students according to their education and their social participation according to their education. The economics of education analyze the education product. According to Legendre (1995), the education product is a "set of resources in the cycle of education, which is located at the end of the education process of which it results (free translation)." The productivity of the pedagogical process that uses educational products in schools is ignored. In theory, their use is productive, but is this really the case?
The purpose of this article is to find a way to calculate the efficiency and/or productivity change of the design process of an adapted textbook and of the pedagogical process using this textbook to educate pupils facing certain difficulties. In order to do this, we were inspired by the book written by professor Mario Godard (2010). The present publication addresses the specific case of the use of an adapted textbook, but the approach can be used with any educational product or program. The adapted textbook is a simplified version of the original Signet (ERPI, 3rd grade) textbook. It is identical to the original textbook (same cover, subjects, illustrations, number of pages and answers to the first question of every page), but the contents are simplified to make it easier for pupils who do not know how to read in 3rd grade. First, we revise the productivity change and efficiency measurement systems as proposed by Godard (2010). Then, we apply these

Productivity and efficiency measurement
Productivity and efficiency must be measured to establish the performance of different processes. We usually try to identify the factors explaining the difference in overall cost for those processes. If we know the real cost associated to certain processes for a period and have the same type of real cost for the same processes for another period, a reference period, we want to identify the part of the cost variance that is caused by productivity change. If we do not have the cost of a reference period but have the planned cost for the same period, we want to identify the part of the cost variance that is caused by efficiency.
Literature proposes many measurement systems. If we want to be able to accurately measure productivity change and efficiency and also want to be able to isolate the cost variance caused by productivity change or efficiency, the only systems that can give this information,are those proposed by Godard (2010). These systems are summarized briefly in the next sections. The information presented here is useful to understand how we will transfer Godard's method (2010) to the measurement of educational processes.

Productivity measurement
According to Godard (2010), productivity is "the relation between the quantities of goods produced and all the factors of production (total productivity) used or one of these factors (partial productivity)." This calculation is useful, for example, to know the quantity of goods produced per hour of work; however, this relation is useless if we want to know whether or not the productivity is good or bad. That is why the partial pure productivity index takes into account partial productivity during a reference period, named o throughout this text, and partial productivity during the real period, named n in this text (see Where: IOPij: partial pure productivity index of input i to make output j between period n and period o qijo: real quantity of input i used for the manufacturing of output j during period o qijn: real quantity of input i used for the manufacturing of output j during period n Qjo: real quantity of output j made during period o Qjn: real quantity of output j made during period n To understand how to calculate a partial pure productivity index, the example of a toy factory making cars and tractors is used (Table 1). Factory employees are paid $10/hour, during period o. Given that the partial pure productivity index of input 1 to equal output 1 (IOP11) is 1, the productivity remained the same for periods o and n (see Equation 1.2). If we do the same calculation with input 1 and output 2, we can find the productivity change of input 1 to produce output 2 (see Equation 1.3).   [1.4] This change is useful in the numerator to obtain the quantity of input during period n that would be necessary to produce the quantity of output of period n, with the productivity of period o. Let us transfer this equation to our example (Equation 1.5). For period o, input 1 is 4 hours and output 2 is 1 tractor, while during period n, the factory produces 2 tractors in 2 hours of work. The numerator of Equation 1.5 allows us to know, with the productivity of period o, how much time would be needed to produce 2 tractors (the quantity of output during period n). With the productivity of period o, 8 hours would be required to produce 2 tractors. The partial pure productivity index of Equation 1.5 is identical to the one found in 1.3. With this change, we have a quantity of input in the numerator and denominator. We can therefore multiply those quantities by the unit cost of the input considered. The index value is not influenced. We can use the unit cost of period o or period n. The interpretation will vary depending on the unit cost used. The notation must take this into consideration. If we use the unit cost of period o, we will obtain Equation 1.6. The second exponent in this equation, added to the IOP, means that the productivity index is calculated with the quantity of output j during period n and the unit cost of input i during period o. The first exponent of the notation refers to the quantity of output that is used to calculate the index.
In the previous example, employees receive an average pay of $10/hour during period o. Thus, if we apply Equation 1.6, we will find that it costs $80 to produce 2 tractors during period o and $20 during period n. Subtracting the numerator and denominator of 1.6 is useful to find the cost variance caused by the productivity change of input i to produce output j, calculated with the output of period n and the unit cost of period o. If we go back to our example, the cost variance caused by the productivity change of input 1, to produce output 2, quantity of tractors, is $60. It is a $60 gain during period n compared to reference o, but calculated with the unit cost of period o. If we use the unit cost of period n in equations similar to Equation 1.6 and 1.7, we have type nn results, because it is calculated with the quantity and unit cost of period n. These types of results are the most interesting for CEOs. No and nn indices will yield the same result, when we consider one input with one output. This is not the case for the cost variance. For our example, if we suppose that the unit cost for input 1 is $15 for period n, the cost variance caused by the productivity change will be $90 instead of $60.
To calculate the total productivity when we have many inputs or outputs, we calculate a new numerator by adding all the numerators of the partial pure productivity indices involved and calculate a new denominator by adding all the denominators of the same partial pure productivity indices. We can also calculate type oo by using the quantity of output j during period o and the unit cost of input i during period o. Oo indices may be considered the more pure indices because the quantities of outputs and the unit cost are the values of the reference period. For our example, we can calculate type oo cost variance from the result of Equation 1.9 by subtracting the numerator from the denominator. Given that the quantity of output o is less than the quantity of output n, the cost variance is smaller. It is $30 instead of the $60 obtained for the no cost variance.  With all the nn indices and cost variances, what makes Godard's approach interesting is that we can accurately isolate the exact amount caused by the productivity change and unit cost change and also the part due only to the change in output quantities, from the total cost variance between two periods. With all the types (oo, no, nn), we can isolate other important factors explaining the total cost performance. When we only have one output and one input, these different measures are less useful, but as soon as we have many outputs and many inputs, we can have a more complete analysis of the causes of the productivity change.
Before we start transposing these equations into an educational product, it is important first to study the constraints imposed by the calculation of productivity. We must be able to measure the quantity of input for each output to use this type of measure. Outputs, like inputs, have to be quantitative measures. The nature of output j during period o and period n must be relatively similar. For example, if a factory makes cars during period o and tractors during period n, we cannot calculate the productivity evolution. On the other hand, the result helps to observe if the manufacturing of tractors requires more hours or less than the manufacturing of cars. In the third section of this text, these equations are transposed into the productivity evaluation of a pedagogical process.

Efficiency measurement
When there is no previous production of a similar output, we cannot take a reference period; however we can calculate the efficiency if we have planned or budgeted quantities (Godard 2010). Efficiency is "comparing the real quantities of inputs used to produce outputs with the quantities of inputs that would have been necessary to take (planned or budgeted) according to a standard or some scale, during the same period (p. 109)." The budgeted period is named, b, throughout this article.
The development of the efficiency measurement system is exactly the same as the productivity change measurement system. As for the productivity index, the efficiency can be calculated with the quantity of output j during period n and with the unit cost of period b (type nb) or the quantity of output j during period b and with the unit cost of period b (type bb) and finally with the quantity of output j during period n and with the unit cost of period n (type nn). Here we only provide the nb equation. With all nn indices and cost variances, we can isolate the exact amount caused by efficiency and unit cost change and also the part caused only by the change in output quantities, from the total cost variance between the real costs and planned costs.
In the next section, Formula 1.10 is adapted to the case of the design process.

Efficiency of the design process of an adapted textbook
First, we present a short resume of the design process of an adapted textbook. Then, we explain how to measure the efficiency of that design process. We choose to develop the efficiency measurement because data for a real reference period is unavailable.

Brief presentation of the design process of an adapted textbook
The design process we used is mainly based on research conducted by a research group from a Montreal university, the DEFI Accessibility Group. The latter specializes in the study of children with intellectual disabilities. Given that these disabilities mainly affect reading skills and that these skills are essential to learn other subjects, the group decided to design an adapted textbook to help these children. For the purpose of this article, we did not include all of the group's research in the design process. We limited the present design phase to the year when the researchers focused more extensively on the theoretical framework of the adapted textbook. The human resources involved in this phase include one main researcher from the research group, one student-researcher, one computer graphics specialist (layout of the adapted textbooks) and a team of three research assistants. We identify these resources as the design team.
As for any design, the team first had to define the problem and identify user needs. To analyze the needs of potential users, the questionnaire and interview proposed by Mayer and Ouellet (1991) were used. The questionnaire was built from the potential needs of the users highlighted by the literature review. The second method, the needs analysis, consisted in meeting key advisers in the field. The semi-structured interview technique was used to know the parents' needs and their child's (Mayer and Ouellet 1991).
Once these two needs analyses were conducted (questionnaire and interview), the needs were transposed into functions. A function is a "characteristic role of a product, service, process or system toward needs and satisfactions it will get to its user. All needs can be transformed into functions" (Petitdemange 1985). In education, a function is a "characteristic role of an educational product toward the needs of the Subject users and Agents of a specific educational situation" (Legendre 2005; Rocque and al. 1998). For example, the literature review and needs analyses showed that 3rd graders who were two years behind in reading, need to progress urgently in this field because the Programme de formation de l'école québécoise prescribes that it is important to develop reading skills to be able to learn other subjects (Ministère de l'Éducation, du Loisir et du Sport 2001).
This work led to the functional specifications, which include the selection of functions, their ranking by importance and their characterization (Petitdemange 1985; Robichaud 2010; Rocque and al. 1998). The functional specifications play two roles: they are prescriptive for the design team, which has to find solutions to fill the functions, and they also are normative because they are a reference to estimate the prototype (Rocque and al. 1998).
The design phase of the adapted textbooks lasted over a year. The design team participated in numerous meetings to discuss the frame and functions of the textbooks. The constraint function, which stipulated that the adapted textbook had to encourage an inclusive approach for the pupil's education, raised several problems because the possible adaptations were always obstacles to an inclusive approach. These problems led to an improvement of the rules for text simplification proposed in Simple et Accessible (Duquette, Rocque, Langevin, Beaulieu & Chalghoumi, to be published) with new simplification rules specific to educational texts. Once the simplification rules were written, a team of assistants was trained by the research group to simplify the original textbook Signet (ERPI, 3rd grade). A list of strategies targeting educational agents using adapted textbooks was created. These strategies target the similarities between the original textbook and the adapted textbook, inciting educational agents to encourage pupils using the adapted textbooks to participate on these specific points. A computer graphics specialist did the layout of the adapted textbooks, and Les Éditions du renouveau pédagogique Inc. printed the prototype. Fig.1 Adapted textbook design process Figure 1 shows the design process of an adapted textbook. The inputs considered are the research assistants, studentresearcher, computer graphics specialist, main researcher and a certain number of prototypes. The output is of course the adapted textbook. If a prototype is essential to the design and a test is prescribed by designers to improve the prototype, it must be considered in the design process.

Efficiency measurement of the design process
Measurement of the productivity change requires a period of reference and similar j outputs during period o and period n. Given that we did not find other adapted textbooks, comparing the quantities of inputs used, between a real reference period and a real comparison period, is impossible for the design process. So we cannot develop a productivity index for the design process. Instead, we calculate a measure of efficiency, which is detailed in this part of the text. By definition, the efficiency is an index comparing the real productivity ratio of a period to the budgeted or planned productivity ratio from the same period. To be able to calculate the efficiency of a process, we must have planned or budgeted quantities.

DESIGN PROCESS
Adapted textbook Inputs for the calculation of the efficiency of the design process of the adapted textbooks are: the research assistants, student-researcher, computer graphics specialist, main researcher and a certain number of prototypes. Equation 1.11 gives the efficiency for all inputs considered to produce a proper adapted textbook. The letter t used in the name of this productivity index indicates that it includes all inputs.  .11 is presented in its most general form. For this type of design process, Qsn and Qsb will be egal to 1.0. The cost variance caused by efficiency can be calculated by subtracting the denominator from the numerator of the previous equation. As for the productivity index, the efficiency should not be the only information considered. For an educational product, or any other situation, the researcher has to complete his analysis with additional analyses. In this precise case, it would be interesting to compare the difference in the costs of the pedagogical process with or without an adapted textbook with the quality of the students' education, a type of cost-profit analysis. The quality of the students' education will certainly impact the cost of their future education.

Inputs Outputs
In the next section of this article, we focus on the pedagogical process using an adapted textbook. Isolating this process allows us to consider pupils with the desired characteristics (reading age, motivation and inclusive situation) as outputs. The productivity of the educational process with an adapted textbook can be analyzed according to the quantity of resources required in comparison with the pupil's success.

Brief presentation of the pedagogical process using an adapted textbook
The pedagogical process consists in educating a group of pupils in a classroom that includes children with some intellectual disabilities and "normal" children. For this paper, we focus on children with disabilities. The purpose is to educate them, at a predetermined level, by giving them more support with an adapted textbook and teacher. Fig. 2 shows the pedagogical process. The inputs used are: the teacher, resource teacher, children with disabilities and adapted textbooks. The outputs are the pupils educated with the desired characteristics. In the next part of this text, we will focus on the calculation of the productivity of this process.

Productivity measurement of a pedagogical process using an adapted textbook
This section proposes an adaptation of Godard's method to the process presented previously. For this process, we can capitalize on the fact that at this point, pupils are integrated into regular classes and work with material adapted by their teacher. Period o, the reference period, can be the period before the adapted textbook is used; the period when the pupil uses homemade adapted material. Period n, said to be the real perios, is when pupils use the adapted textbook. During the results analysis, we can observe the evolution of productivity because even if the homemade adapted material (adapted by the teacher) is very different from adapted textbooks, the outputs of the pedagogical process using an adapted textbook or the homemade adapted material are the same. In both cases, we want to educate a certain number of pupils with some learning difficulties. They are the process outputs. At the end of the pedagogical process, these pupils must have certain characteristics at a given level (motivation of the pupil, success in learning how to read and inclusive approach in school). These output characteristics stem from functions of the functional specifications of the pedagogical process, using an adapted textbook. A tool, in our case an adapted textbook, can help to learn more easily and with fewer resources. Here we develop the question of the impact of using an adapted textbook on the productivity of the pedagogical process. The inputs of the pedagogical process for the productivity evaluation are: the teacher and resource teacher; children with some disabilities and the adapted textbooks.
As seen in Part 1.1, the output must be expressed in the quantitative data for us to find a productivity index. A qualitative part is required to improve and qualify the productivity indices and cost variance caused by the productivity change. For example, the relation between school and family is treated in a more qualitative way. To specify if the outputs used to calculate the productivity index of the pedagogical process are good outputs, we use the functional specifications as quality characteristics.  .13 gives the partial productivity index for the adapted textbook. For this measurement, we are faced with a special case. Given that the same textbook is not used for the two comparison periods, the unit cost is difficult to define. Do we use the unit cost of the homemade textbook or the adapted textbook? For the partial productivity index, the result is not influenced by the unit cost. So we can use the unit cost of the homemade textbook for period o and the unit cost of the adapted textbook for period n. With this approach, we can evaluate the cost variance caused by the change in unit cost as part of a more global cost analysis. We can measure the productivity index for the input time of the teacher [1.12] and the productivity index for the input adapted textbook [1.13]. By subtracting the denominator of the numerator in Equations 1.12 and 1.13, we can calculate the cost variance caused by the productivity change given that the average hourly wage of a teacher and the unit cost of an adapted textbook are known.
If the value of one hour of work of a teacher is known, the monetary value of pupils in the beginning of the pedagogical process has to be determined. We can suppose that at the beginning of the pedagogical process, the unit cost of a pupil is the unit cost of all his previous education. With this hypothesis, we can calculate the partial productivity index for the input pupils with Equation 1.14 and can also find the cost variance caused by the productivity change by subtracting the denominator from the numerator of that equation. In sum, in the case of adapted textbooks, there are three type no productivity indices: with the input time of the teacher, with the input adapted textbook and with the input number of pupils. These productivity indices take into account inputs and outputs for the stage named pedagogical process, in the life cycle of the product.
If we use unit cost n, we can calculate all type nn indices and cost variances. With this information, it is possible to isolate the exact amount caused by the productivity change and unit cost change and also the part caused by the change in output quantities from the total cost variance between the real cost of the pedagogical process using an adapted textbook and the real cost of the pedagogical process using a homemade textbook.
The analysis of this example shows that in education, analyzing productivity does not have to be limited. It seems beneficial that a qualitative part comes to qualify the indices and improve them.

Difficulties in obtaining data
When we initially tried to measure the productivity change of a pedagogical process using an adapted textbook, we had some problems obtaining our data. The teacher time for the educational inclusion, during period o, is difficult to find. Indeed, we did not find data on this subject in literature. We thus want to make a collection of large-scale data to quantify this time. We would like a representative sample of Quebec's teachers. It would thus be necessary to have different teachers with varied experiences, age, socioeconomic circles and clienteles as part of our sample. To be representative, our sample would need to include 1,000 Quebec teachers. It is very difficult to have a large sample for education research. Finally, we know that a comparison (in time o and in time n) will be made between homemade adapted material and adapted textbooks stemming from the research. Given that both types of products are relatively different, the results should be nuanced. Yet, in the future, we can measure the results of schools using the adapted textbook and not using it during the same academic year. For future designs, we will be able to calculate its productivity evolution using data gathered during the first design.

Conclusion
In conclusion, this text showed that it is possible to calculate the productivity and efficiency of an educational product. We also explained that, like for any type of organization, use of those measures must be accompanied with another analysis in order to qualify the results. Calculations of productivity and efficiency are tools that can quantify the impacts of an educational product. To be effective, they must be implemented on a continuous basis, as in organizations concerned by their continuous improvement. This text is a first try of transposition into education, which must be improved. The approach proposed must be experimented to find the gaps, to improve it.