Indicators of pedagogical quality for the design of a Massive Open Online Course for teacher training

Massive Open Online Courses (MOOCs) have generated high expectations and revolutionized some educational practices by providing open educational resources for reference, usage and adaptation; therefore, their pedagogical quality is often questioned. The objective of this study is to identify indicators related to pedagogical, functional, technological and time factors in order to assess the quality of the MOOC entitled “Liderazgo en gestión educativa estratégica a través del uso de la tecnología” (Leadership in strategic educational management through the use of technology), offered as a teacher training program through Coursera to 10.161 participants. Via the Delphi method, a group of 55 experts agreed that time is a key factor to be considered in the design of learning activities. It was concluded that without measuring results, the success of a MOOC could not be evaluated; thus, institutions and consortia must establish evaluation indicators to focus their efforts on the enhancement of pedagogical quality. By providing relevant information, the learning potential of educational resources based on connectivism principles can be evaluated, and so can the quality of MOOCs. The goal is to contribute to a vision of a future in which everyone has access to a world-class education. Los cursos en línea, masivos y abiertos (MOOC) han generado importantes expectativas y han revolucionado algunas prácticas educativas, al ofrecer recursos educativos abiertos para su consulta, uso y adaptación; sin embargo, con frecuencia se cuestiona su calidad pedagógica. El objetivo de este estudio es identificar indicadores relacionados con factores pedagógicos, funcionales, tecnológicos y de tiempo, para evaluar la calidad del MOOC Liderazgo en gestión educativa estratégica a través del uso de la tecnología, ofrecido como recurso de actualización docente en Coursera a 10.161 participantes. Mediante el método Delphi, un grupo de 55 expertos acordó que el tiempo es un factor clave a considerarse en el diseño de las actividades de aprendizaje. Se concluye que sin medición de los resultados no se puede valorar el éxito de un MOOC, por ello instituciones y consorcios deben establecer indicadores de evaluación para enfocar sus esfuerzos para la mejora de su calidad pedagógica. Si se proporciona información relevante se podrá evaluar el potential de aprendizaje que poseen los recursos educativos basados en principios conectivistas y reconocer la calidad pedagógica de los MOOC, con el objetivo de coadyuvar a la visión de un futuro en el que todos tengan acceso a una educación de clase mundial.


Introduction
Currently, Massive Open Online Courses, or MOOCs, have generated high expectations and revolutionized pedagogical practices by providing open educational resources for reference, usage and adaptation (UNESCO, 2012). c-MOOCs, which gave rise to this phenomenon, adopted a pedagogical strategy with an epistemological basis grounded in connectivism; they also promoted educational change, not only through technological advances, but also through the theoretical developments that emerged from this field of study (Zapata-Ros, 2013). They provided access opportunities to the general public and were facilitated by renowned professionals who took a vital role.
Thus, c-MOOCs made knowledge from a wide range of prestigious universities available to the whole world at a speed, scope, scale and price that no traditional course would be in a position to offer (Bell, 2012). Just a decade ago, this type of training would have been impossible without the current advances of technology (Friedman, 2013;Skiba, 2013). Siemens (2004) posits that connectivism is an educational theory characterized by considering learning as an extension of a personal network through which participants learn, share knowledge and comprehend. He claimed that the most important contribution of MOOCs resided in their potential to change the relationship between students and teachers, academia and the community at large, by offering a broad and diverse virtual space, a meeting place for the exchange of ideas. He emphasized that anyone enrolling on a MOOC would probably find learning in its most exposed form, through platforms that not only invited participants to see and hear, but also to participate and collaborate.
Despite its rapid integration into the educational offering, the Institute for Prospective Technological Studies claims that the criteria for evaluating the educational quality of a MOOC lack transparency and warns that, for the moment, the implemented educational models are evidently not sustainable (Aceto, Borotis, Devine, & Fischer, 2014).
For Sangrà and Wheeler (2013), the massive aspect of the courses, which is promoted as something positive, has never been a feature of successful training. They believe that, in MOOCs, informal learning has found a perfect ally in ICT in general, and in online learning in particular. This emphasizes the need to investigate MOOCs in depth in order to establish whether they represent real opportunities for learning in informal settings, or if they are simply attempts to formalize the informal.
This article presents the results of a study aimed at identifying indicators for the educational quality of a MOOC, which will greatly improve the design of the course entitled "Liderazgo en gestión educativa estratégica a través del uso de la tecnología" (Leadership in strategic educational management through the use of technology), offered as a teacher training update strategy by a higher education institution. In the analysis, the assessment of the indicators by a group of 55 experts is presented, and the profile and expectations of 10,161 participants in the first edition of the course are described.

Background
Although the history of MOOCs may seem short in absolute terms, Boven (2013) locates its origins in open and distance education. He notes that many emerging movements have adopted the principles advocated by educational reformers, such as those proposed by the "Education for All" ideal. This perhaps explains why MOOCs  RUSC Vol. 12 No. 1 | Universitat Oberta de Catalunya and University of New England | Barcelona, January 2015 CC L. Y. Alemán de la Garza, T. Sancho-Vinuesa and M. G. Gómez Zermeño | CC by FUOC, 2015 | Indicators of pedagogical quality for the design… have so often been described as "recurrent resources in the discourse of educational openness" (McAuley, Stewart, Siemens, & Cormier, 2010, p. 46).
As its name suggests, a MOOC is an online course accessible to virtually anyone who wishes to participate with unlimited attendance (EDUCAUSE, 2011). For Tschofen and Mackness (2012), MOOCs are online courses that appeal to a wide variety of participants around the world; they are massive because literally thousands of people can participate in them, and they are open because participants can openly share resources, ideas and experiences without any requirements. They constitute a collective creation of knowledge, resulting in a composition greater than the sum of its parts. In many ways, they are a microcosm of a nation (Liyanagunawardena, Adams, & Williams, 2013).
While reviewing the educational research undertaken, ever-increasing interest in the study of MOOCs in recent years was noted. This contemporary interest is manifested in the evolution of their conceptual references and the definition of their main features. According to Rodriguez (2012), the term 'MOOC' was coined by Dave Cormier when the number of students on the course entitled "Connectivism and Connective Knowledge (CCK08)" offered by George Siemens and Stephen Downes reached a total of 2,300. After analyzing the results of their experience, they could see MOOCs as a new modality of online education and stated that its implementation required conceptual changes in the processes of teaching and learning, from the perspective of teachers and students alike.

Theoretical perspectives on learning
In pedagogy, learning theories are the basis of educational processes, and they often refer to principles of behaviorism, cognitivism and constructivism to create instructional environments. Adding technology to existing learning theories raises many questions and prompts theorists to review them continuously, in order to adapt them as learning conditions change. These three theories were developed at a time when learning had not been impacted by the use of technology in education. At some point, the underlying conditions have been so significantly altered that additional modifications are no longer feasible. An interconnected world allows us to explore how information is acquired; connectivist environments have facilitated the creation of networks to share resources, ideas and experiences, and they must be evaluated. Faced with this new reality, a completely different approach is necessary (Siemens, 2004).
The SCOPEO (2013) report confirms that the "first generation" of MOOCs was based on connectivism, a pedagogical theory that posits that personal knowledge is created from a network that provides its members with information. In turn, these members provide feedback with additional information generated within the same network. The process ends when this information, which may come from different nodes, transforms and alters the knowledge base, and generates new learning for individuals. For Chiecher and Donolo (2013), MOOCs have broken curricular rigidity, questioning the ability of traditional teaching to meet training needs in changing conditions.
Training offerings in which teaching differs from the what is commonly referred to as "traditional classroom education" are being overwhelmingly adopted. This has given rise to critical reflection on the perception of learners -and is something that demands a paradigm shift.
Addressing the future of MOOCs, some experts predict that they will mutate into different forms of learning with a greater emphasis on participants' support and a decrease on their number to allow adequate attention. Experts emphasize the need to implement evaluation systems to establish pedagogical quality criteria that go beyond limited assessments, based on the "reputation" of the educational institution offering them (Menéndez, 2013

Indicators of the educational quality of a MOOC
For the educational community, it is a reality that the use of technology in teaching practices has led to the emergence of modern social structures and organizational forms, in which the traditional space and time referents are no longer valid (Garrido, 2003). Given the lack of space and the growing demand for admission to educational institutions, online education is renowned for offering opportunities to expand the educational range and coverage.
For Moore (2013), distance education is a psychological construct that depends on macro factors such as dialogue, structure and autonomy.
In relation to their pedagogical design and to the learning expectations of students, MOOCs imply a change of instructional schemes. At first, their structure was thought of in a minimalist way to allow participants to design their own learning, but research findings have since revealed that, in order to improve their pedagogical quality, multiple factors related to how, where and when participants learn must be considered (The New Media Consortium & Universitat Oberta de Catalunya, 2012).
Although MOOCs imply the design of new schemes that recognize studies outside the classroom, the lack of standards for evaluating their pedagogic quality is often questioned (Bernal, Molina, & Perez, 2013). For Gómez-Zermeño and Alemán (2012), the integration of technology into educational processes requires the establishment of both theoretical foundations and evaluation mechanisms to identify their numerous sources of influence. They point out that the design of technology-based strategies that seek to strengthen access to quality education poses significant challenges to educational researchers yet offers attractive advantages that could encourage their adoption.
In the MOOC Quality Project by the European Foundation for Quality eLearning, it became apparent that the For Barbera, Gros, and Kirschner (2012), time is a critical factor that has also been used as a quality measure, since it is related to the amount and the sequence in which people learn through the accumulation of experiences. In collaborative learning environments, the implementation of strategies that promote participants' self-regulation is recommended (Franco-Casamitjana, Barbera, & Romero, 2013).

Method
The objectives of this research can be addressed via different methodologies. Given the object of study, a quasiexperimental design was adopted. Cross (2013)  research into an educational innovation may have a quasi-experimental design that does not require evaluation of a random sample. Moreover, the results are measurable with one post-test group (Shadish, Cook, & Campbell, 2002).
Regarding quality indicators to strengthen the design of a course of this nature, a set of indicators was selected from studies by Arias (2007), Barbera et al. (2012), Cabero and Romero (2007) (2011), and Roig et al. (2013). A group of experts reviewed and validated these indicators using the Delphi method. This method involves selecting a group of experts, who are asked their opinion on issues relating to the future, implementing consecutive anonymous rounds to ensure the autonomy of participants. The predictive power of this method is based on the systematic use of intuitive judgment by all experts (Astigarraga, 2003).
Once the set of selected indicators had been identified, a questionnaire entitled "MOOC-I-Quality Indicators" was designed, consisting of closed questions that assess 50 indicators on a 4-point Likert scale. For the analysis, the indicators were classified into 15 subcategories related to Pedagogical, Functional, Technological and Time factors (Table 1). This instrument was applied to a group of 55 experts involved in the design, development and delivery of MOOCs, as well as in distance education courses and open learning resources offered by the institution. This group was formed by 14 lecturers responsible for design and content generation, and a total of 41 tutors, instructional designers, graphic designers, programmers and audiovisual producers.
Based on studies by Breslow, Pritchard, DeBoer, Stump, Ho, and Seaton (2013), the "MOOC-I-Participants' Information" questionnaire, made up of 30 questions gathering demographic, employment, educational update, expectation and opinion data from those on the course, was designed. Taking into consideration the assessment made by the experts for indicators of pedagogical quality, the MOOC entitled "Liderazgo en gestión educativa estratégica a través del uso de la tecnología" (Leadership in strategic educational management through the use of technology) was designed and offered as a teacher training update strategy; this questionnaire was administered to 10,161 participants on the first edition of the MOOC.

Results Analysis
Aligned with the objectives of this research, the analysis of two different issues is presented. First, the assessment of quality indicators for the design of MOOCs and, second, after the MOOC in question had been designed and implemented, the students' profiles and expectations.

Indicators for assessing the pedagogical quality of a MOOC
The results of the Delphi method show that while experts rated the indicators related to the Time factor higher, they also recognized that the quality of a MOOC must take the other factors into account (Figure 1). It is worth noting that the indicator for the time to Take exams was rated the highest. Studies by Barbera et al. (2012) corroborate that time remains a crucial strategic issue in online teaching, requiring explicit attention from teachers and designers, since it affects students' learning. For Franco-Casamitjana et al. (2013), time management skills determine self-regulation of students and members of a group alike (Guitert, 2011). In distance education, autonomy refers to the extent to which students decide "what to learn, how to learn, and how much they learn" (Moore 2013, p. 68). When analyzing the results by subcategory (Table 1)  In the experts' opinion, MOOCs enable the comprehensive development of competitive citizens who are able to position themselves as agents of change. Experts recognize that to meet the growing demand for education, the adoption of innovative models that develop commitment and linkage between participants is required (Brown, 2013). Such models should also emphasize the need to implement an evaluation system in order to improve the educational quality of resources like MOOCs.

Participants' profiles and expectations
Based on the quality indicators assessed by experts, the MOOC entitled "Liderazgo en gestión educativa estratégica a través del uso de la tecnología" (Leadership in strategic educational management through the use of technology) was designed and implemented using the Coursera'® platform. For three weeks, the call for mass participation in the course was disseminated via institutional electronic media and social networks. A total of 10,161 participants enrolled on the MOOC in question. The participants were from 79 countries located in all continents: The Americas 90.0%, Europe 9.0%, Asia 0.6%, Oceania 0.1% and Africa 0.1%. According to Coursera, 85% of the study's population came from emerging economies. One of the main features of a MOOC is its openness to enrolment; so all the participants who confirmed their interest in voluntarily registering for free by answering the questionnaire "MOOC-I-Participants' Information" formed the population of this study. Table 2 describes their main features. On completion, the MOOC achieved a terminal efficiency rate exceeding 22% (see MOOC 12 in Figure 2), which is considered "atypical", as well as a higher commitment rate from participants, which reached 52.15%. It is noted that the average terminal efficiency rate of all MOOCs this institution has offered is 4%, which coincides with the percentage reported in studies by the University of Pennsylvania Graduate School of Education (Penn GSE, 2013).

Conclusions
In this paper, quality indicators endorsing the design of a MOOC on Leadership in strategic educational management have been identified, as have the profiles of participants in its first edition and the rate of terminal efficiency.
The main conclusion from this research is that the success of a MOOC cannot be evaluated without measuring results. Thus, institutions and consortia should establish indicators to focus efforts on improving their pedagogical quality. Design changes and development cannot be planned either, as these actions require the implementation of appropriate mechanisms to measure the participants' performance, which should be integrated into the standards of their educational model. When applying quality criteria, it should be considered that the indicators cannot operate in isolation, so it is necessary to define them in order to build a complete picture of the educational system that will receive feedback.
According to 55 experts, time is a key factor that impacts the pedagogical quality of a MOOC; however, they emphasize that the results are a logical consequence of the interrelationship between the Pedagogical, Functional, Technological and Time factors. When designing a MOOC, the time it will take participants to review the content, videos, resources, exercises and tests, and to take an active part in activities and collaborative learning discussion boards should be considered (Gros, Barbera, & Kirschner, 2010). The designed system involves promoting selfregulation skills, and must include the elements and approaches that support the theory of connectivism in order to be able to offer open educational resources that provide a real strategic opportunity to improve the quality of education (UNESCO, 2012).
As the use of technology in education advances, the understanding of the elements and principles of connectivism will be challenged, and educational research will provide evidence on the network capacities and possibilities intertwined with the various learning styles of individuals. By providing pertinent information, the learning potential of MOOCs to improve teacher training may be evaluated. Only then can the pedagogical quality