1. Introduction
Currently, Massive Open Online Courses, or MOOCs, have generated high expectations and revolutionized pedagogical practices by providing open educational resources for reference, usage and adaptation (UNESCO, 2012). c-MOOCs, which gave rise to this phenomenon, adopted a pedagogical strategy with an epistemological basis grounded in connectivism; they also promoted educational change, not only through technological advances, but also through the theoretical developments that emerged from this field of study (Zapata-Ros, 2013). They provided access opportunities to the general public and were facilitated by renowned professionals who took a vital role. Thus, c-MOOCs made knowledge from a wide range of prestigious universities available to the whole world at a speed, scope, scale and price that no traditional course would be in a position to offer (Bell, 2012). Just a decade ago, this type of training would have been impossible without the current advances of technology (Friedman, 2013; Skiba, 2013).
Siemens (2004) posits that connectivism is an educational theory characterized by considering learning as an extension of a personal network through which participants learn, share knowledge and comprehend. He claimed that the most important contribution of MOOCs resided in their potential to change the relationship between students and teachers, academia and the community at large, by offering a broad and diverse virtual space, a meeting place for the exchange of ideas. He emphasized that anyone enrolling on a MOOC would probably find learning in its most exposed form, through platforms that not only invited participants to see and hear, but also to participate and collaborate.
Despite its rapid integration into the educational offering, the Institute for Prospective Technological Studies claims that the criteria for evaluating the educational quality of a MOOC lack transparency and warns that, for the moment, the implemented educational models are evidently not sustainable (Aceto, Borotis, Devine, & Fischer, 2014).
For Sangrà and Wheeler (2013), the massive aspect of the courses, which is promoted as something positive, has never been a feature of successful training. They believe that, in MOOCs, informal learning has found a perfect ally in ICT in general, and in online learning in particular. This emphasizes the need to investigate MOOCs in depth in order to establish whether they represent real opportunities for learning in informal settings, or if they are simply attempts to formalize the informal.
This article presents the results of a study aimed at identifying indicators for the educational quality of a MOOC, which will greatly improve the design of the course entitled “Liderazgo en gestión educativa estratégica a través del uso de la tecnología” (Leadership in strategic educational management through the use of technology), offered as a teacher training update strategy by a higher education institution. In the analysis, the assessment of the indicators by a group of 55 experts is presented, and the profile and expectations of 10,161 participants in the first edition of the course are described.
1.1. Background
Although the history of MOOCs may seem short in absolute terms, Boven (2013) locates its origins in open and distance education. He notes that many emerging movements have adopted the principles advocated by educational reformers, such as those proposed by the “Education for All” ideal. This perhaps explains why MOOCs have so often been described as “recurrent resources in the discourse of educational openness” (McAuley, Stewart, Siemens, & Cormier, 2010, p. 46).
As its name suggests, a MOOC is an online course accessible to virtually anyone who wishes to participate with unlimited attendance (EDUCAUSE, 2011). For Tschofen and Mackness (2012), MOOCs are online courses that appeal to a wide variety of participants around the world; they are massive because literally thousands of people can participate in them, and they are open because participants can openly share resources, ideas and experiences without any requirements. They constitute a collective creation of knowledge, resulting in a composition greater than the sum of its parts. In many ways, they are a microcosm of a nation (Liyanagunawardena, Adams, & Williams, 2013).
While reviewing the educational research undertaken, ever-increasing interest in the study of MOOCs in recent years was noted. This contemporary interest is manifested in the evolution of their conceptual references and the definition of their main features. According to Rodriguez (2012), the term ‘MOOC’ was coined by Dave Cormier when the number of students on the course entitled “Connectivism and Connective Knowledge (CCK08)” offered by George Siemens and Stephen Downes reached a total of 2,300. After analyzing the results of their experience, they could see MOOCs as a new modality of online education and stated that its implementation required conceptual changes in the processes of teaching and learning, from the perspective of teachers and students alike.
1.2. Theoretical perspectives on learning
In pedagogy, learning theories are the basis of educational processes, and they often refer to principles of behaviorism, cognitivism and constructivism to create instructional environments. Adding technology to existing learning theories raises many questions and prompts theorists to review them continuously, in order to adapt them as learning conditions change. These three theories were developed at a time when learning had not been impacted by the use of technology in education. At some point, the underlying conditions have been so significantly altered that additional modifications are no longer feasible. An interconnected world allows us to explore how information is acquired; connectivist environments have facilitated the creation of networks to share resources, ideas and experiences, and they must be evaluated. Faced with this new reality, a completely different approach is necessary (Siemens, 2004).
The SCOPEO (2013) report confirms that the “first generation” of MOOCs was based on connectivism, a pedagogical theory that posits that personal knowledge is created from a network that provides its members with information. In turn, these members provide feedback with additional information generated within the same network. The process ends when this information, which may come from different nodes, transforms and alters the knowledge base, and generates new learning for individuals. For Chiecher and Donolo (2013), MOOCs have broken curricular rigidity, questioning the ability of traditional teaching to meet training needs in changing conditions. Training offerings in which teaching differs from the what is commonly referred to as “traditional classroom education” are being overwhelmingly adopted. This has given rise to critical reflection on the perception of learners – and is something that demands a paradigm shift.
Addressing the future of MOOCs, some experts predict that they will mutate into different forms of learning with a greater emphasis on participants’ support and a decrease on their number to allow adequate attention. Experts emphasize the need to implement evaluation systems to establish pedagogical quality criteria that go beyond limited assessments, based on the “reputation” of the educational institution offering them (Menéndez, 2013).
1.3. Indicators of the educational quality of a MOOC
For the educational community, it is a reality that the use of technology in teaching practices has led to the emergence of modern social structures and organizational forms, in which the traditional space and time referents are no longer valid (Garrido, 2003). Given the lack of space and the growing demand for admission to educational institutions, online education is renowned for offering opportunities to expand the educational range and coverage. For Moore (2013), distance education is a psychological construct that depends on macro factors such as dialogue, structure and autonomy.
In relation to their pedagogical design and to the learning expectations of students, MOOCs imply a change of instructional schemes. At first, their structure was thought of in a minimalist way to allow participants to design their own learning, but research findings have since revealed that, in order to improve their pedagogical quality, multiple factors related to how, where and when participants learn must be considered (The New Media Consortium & Universitat Oberta de Catalunya, 2012).
Although MOOCs imply the design of new schemes that recognize studies outside the classroom, the lack of standards for evaluating their pedagogic quality is often questioned (Bernal, Molina, & Perez, 2013). For Gómez-Zermeño and Alemán (2012), the integration of technology into educational processes requires the establishment of both theoretical foundations and evaluation mechanisms to identify their numerous sources of influence. They point out that the design of technology-based strategies that seek to strengthen access to quality education poses significant challenges to educational researchers yet offers attractive advantages that could encourage their adoption.
In the MOOC Quality Project by the European Foundation for Quality eLearning, it became apparent that the evaluation of MOOCs was more complicated than in other online offerings due to the lack of an agreed definition on what they are, to the involvement of perceived external factors and to the lack of consensus on their purpose (EFQUEL, 2013). Bernal et al. (2013) recommend that MOOCs should apply the same quality standards used in formal open and distance courses. The fact that they are massive, open and online calls for greater rigor in their quality in order to overcome dissimilar profiles and a lack of monitoring of results and objective attainment.
In assessing the educational quality of distance education and open learning resources, Arias (2007), Cabero and Romero (2007), Gómez-Zermeño (2012), Gómez-Zermeño, Rodríguez, and Márquez (2013), Domingo and Marquès (2011), and Roig et al. (2013) used indicators related to pedagogical, functional and technological factors. For Barbera, Gros, and Kirschner (2012), time is a critical factor that has also been used as a quality measure, since it is related to the amount and the sequence in which people learn through the accumulation of experiences. In collaborative learning environments, the implementation of strategies that promote participants’ self-regulation is recommended (Franco-Casamitjana, Barbera, & Romero, 2013).
2. Method
The objectives of this research can be addressed via different methodologies. Given the object of study, a quasi-experimental design was adopted. Cross (2013) claims that educational research into MOOCs generates large methodological and interpretive challenges, as it poses new dynamics in the teaching-learning process. Strengthening its design involves considering the relationship between research and educational innovation. For Schmelkes (2001), research into an educational innovation may have a quasi-experimental design that does not require evaluation of a random sample. Moreover, the results are measurable with one post-test group (Shadish, Cook, & Campbell, 2002).
Regarding quality indicators to strengthen the design of a course of this nature, a set of indicators was selected from studies by Arias (2007), Barbera et al. (2012), Cabero and Romero (2007), Franco-Casamitjana et al. (2013), Gómez-Zermeño (2012), Gómez-Zermeño et al. (2013), Domingo and Marquès (2011), and Roig et al. (2013). A group of experts reviewed and validated these indicators using the Delphi method. This method involves selecting a group of experts, who are asked their opinion on issues relating to the future, implementing consecutive anonymous rounds to ensure the autonomy of participants. The predictive power of this method is based on the systematic use of intuitive judgment by all experts (Astigarraga, 2003).
Once the set of selected indicators had been identified, a questionnaire entitled “MOOC-I-Quality Indicators” was designed, consisting of closed questions that assess 50 indicators on a 4-point Likert scale. For the analysis, the indicators were classified into 15 subcategories related to Pedagogical, Functional, Technological and Time factors (Table 1). This instrument was applied to a group of 55 experts involved in the design, development and delivery of MOOCs, as well as in distance education courses and open learning resources offered by the institution. This group was formed by 14 lecturers responsible for design and content generation, and a total of 41 tutors, instructional designers, graphic designers, programmers and audiovisual producers.
Based on studies by Breslow, Pritchard, DeBoer, Stump, Ho, and Seaton (2013), the “MOOC-I-Participants’ Information” questionnaire, made up of 30 questions gathering demographic, employment, educational update, expectation and opinion data from those on the course, was designed. Taking into consideration the assessment made by the experts for indicators of pedagogical quality, the MOOC entitled “Liderazgo en gestión educativa estratégica a través del uso de la tecnología” (Leadership in strategic educational management through the use of technology) was designed and offered as a teacher training update strategy; this questionnaire was administered to 10,161 participants on the first edition of the MOOC.
3. Results Analysis
Aligned with the objectives of this research, the analysis of two different issues is presented. First, the assessment of quality indicators for the design of MOOCs and, second, after the MOOC in question had been designed and implemented, the students’ profiles and expectations.
3.1. Indicators for assessing the pedagogical quality of a MOOC
The results of the Delphi method show that while experts rated the indicators related to the Time factor higher, they also recognized that the quality of a MOOC must take the other factors into account (Figure 1). It is worth noting that the indicator for the time to Take exams was rated the highest. Studies by Barbera et al. (2012) corroborate that time remains a crucial strategic issue in online teaching, requiring explicit attention from teachers and designers, since it affects students’ learning. For Franco-Casamitjana et al. (2013), time management skills determine self-regulation of students and members of a group alike (Guitert, 2011). In distance education, autonomy refers to the extent to which students decide “what to learn, how to learn, and how much they learn” (Moore 2013, p. 68).
When analyzing the results by subcategory (Table 1), the experts considered the pedagogical quality of the content of a MOOC to be a relevant factor. They also recommended a review of the pedagogical approach, the provision of tutorials and the specification of the evaluation process of educational activities. For Moore (1983), the content or topic of study determines the dialogue between teachers and learners; it also constitutes one of the main characteristics of open educational resources. Thus, excellent quality should be ensured (UNESCO, 2012).
Category |
Subcategory |
Result |
---|---|---|
Pedagogical |
Contents Pedagogical approach Tutorial and evaluation Adequacy and adaptation to users Motivational capacity Resources |
3.60 3.47 3.44 3.29 3.27 3.15 |
Functional |
Ease of use Autonomy and user control Functionality of the documentation |
3.72 3.44 3.03 |
Technological |
Interaction and dialogues Navigation Visual environment Design and technology Versatility |
3.61 3.52 3.52 3.44 2.97 |
Time |
Take exams Perform activities Perform exercises Study the topics Calendar / Schedule Participate in discussion boards |
3.64 3.62 3.60 3.60 3.53 3.22 |
Regarding the subcategories of functional factors, experts believe that Ease of use is an important factor, followed by Autonomy and user control, and Functionality of the documentation. Under the subcategory Ease of use, the indicator “Exercise instructions are clear and easy to understand” was positively valued, while “The activities suggest the use of additional documents (blogs, wikis, etc.)” was considered less important. Among the Technological factors, the subcategories Interaction and dialogues, Navigation and Visual environment were rated highly, while Versatility was not considered relevant. The experts mentioned, in relation to the indicators of Interaction and dialogues, that communication is one of the most important elements, emphasizing the importance of making means of communication available to foster exchanges among and between participants (discussion boards, news, etc.) and tutors.
In the experts’ opinion, MOOCs enable the comprehensive development of competitive citizens who are able to position themselves as agents of change. Experts recognize that to meet the growing demand for education, the adoption of innovative models that develop commitment and linkage between participants is required (Brown, 2013). Such models should also emphasize the need to implement an evaluation system in order to improve the educational quality of resources like MOOCs.
3.2. Participants’ profiles and expectations
Based on the quality indicators assessed by experts, the MOOC entitled “Liderazgo en gestión educativa estratégica a través del uso de la tecnología” (Leadership in strategic educational management through the use of technology) was designed and implemented using the Coursera’® platform. For three weeks, the call for mass participation in the course was disseminated via institutional electronic media and social networks. A total of 10,161 participants enrolled on the MOOC in question. The participants were from 79 countries located in all continents: The Americas 90.0%, Europe 9.0%, Asia 0.6%, Oceania 0.1% and Africa 0.1%. According to Coursera, 85% of the study’s population came from emerging economies. One of the main features of a MOOC is its openness to enrolment; so all the participants who confirmed their interest in voluntarily registering for free by answering the questionnaire “MOOC-I- Participants’ Information” formed the population of this study. Table 2 describes their main features.
Data |
Characteristics |
Responses |
---|---|---|
SOCIODEMOGRAPHIC |
Country of residence |
Mexico 57%, Colombia 7%, Peru 6%, Argentina 3%, Chile 3%, Ecuador 2%, United States 2 %, Venezuela 2%, Dominican Republic 2%, Guatemala 1%, Brazil 1%, Honduras 0.7 %, El Salvador 0.7%, Costa Rica 0.6%, Bolivia 0.5%, Uruguay 0.5%, Puerto Rico 0.3%, Nicaragua 0.3%, Paraguay 0.3%, Panama 0.2%, Canada 0.2%, and with 0.1% Trinidad and Tobago, Haiti, French Guyana |
Spain 7%, United Kingdom 0.3%, Italy 0.2%, Russian Federation 0.2%, Portugal 0.2%, Germany 0.2%, France 0.2%, and with 0.1% Greece, Poland, Switzerland, Belgium, Ireland, Denmark, Ukraine, Turkey, Norway, The Netherlands, Czech Republic, Hungary, Austria, Sweden, Serbia, Romania, Moldavia, Malta, Macedonia, Lithuania, Latvia, Cyprus, Croatia, Aruba, Andorra |
||
China 0.3%, and with 0.1% Hong Kong, Korea, India, Vietnam, Taiwan, Australia, Philippines, Thailand, Singapore, New Zealand, New Caledonia, Mauritius, Malaysia, Japan, Morocco, South Africa, Angola, Algeria, Pakistan, Kazakhstan, Israel, Iran, Islamic Republic |
||
Gender |
Female 59.4% and Male 40.6% |
|
Age |
Average of 37 years 9 months old, highest frequency 34 years old, range from 14 to 76; 75.0% is 45 or younger, 25% is older than 45. |
|
OCCUPATIONAL |
Occupation |
Teacher 60.9%, Principal 11.0%, Pedagogical advisor10.9%, Higher 2.5%, Inspector 0.4%, Student 14.2% |
Educational level |
Preschool 8.5%, Elementary School 17.6%, Secondary School 18.7%, High School 21.4%, Superior 33.8% |
|
Geographical zone |
Urban 76.0%, Rural 11.3% and Urban/Rural 12.7% |
|
Type of school |
Public 56.2%, Private 32.8% and Public/private 11.0% |
|
School equipment |
Media room 20.1%, Internet in media room 18.8%, Classroom 10.4%, Internet in classroom 12.7%, Principal’s office 17.5%, Internet in Principal’s office 17.8% and none 2.7% |
|
TEACHER TRAINING UPDATE |
Educational level |
Undergraduate 52%, Graduate 37%, High School 7%, PhD 4% |
Years of Service |
5 years or less 28.6%, 6 to 10 years 20.6%, 11 to 15 years 14.6%, 16 to 20 years 11.7%, 21 to 25 years 7.4%, 26 to 30 years 5.9%, more than 31 years 3.5%, No years of service 7.7% |
|
Frequency of teacher training update courses |
1 to 2 courses 53.7%, 3 to 4 courses 22.0%, 5 to 6 courses 4.5%, more than 7 courses 2.0%, does not participate in teacher training update courses 17.7% |
|
Types of teacher training update courses |
National Catalogue of Continuing Education 11.3%, Centre for Training and Teacher training Update 9.7%, National Program of Teaching Career 7.4%, Requested courses 21.6%, courses from public institutions 16.5% and courses from private institutions 18.7% and other courses 14.8% |
|
Modality |
Classroom 35.8%, Online 19.4%, Blended 40.7% and other 4.1% |
|
Use of ICT level |
None 0.6%, Basic 20.5%, Intermediate 38.5%, Advanced 32.7%, Expert 7.7% |
|
Development of ICT level |
None 3.0%, Basic 30.2%, Intermediate 39.3%, Advanced 22.7%, Expert 4.9% |
|
EXPECTATIVES |
Reason for participating |
Take a MOOC 15.4%, Model of Strategic Educational Management 31.1%, Technological tools 25.4%, Técnologico de Monterrey course 17.7%, Learn what a MOOC is 10.1% and Other 0.3% |
Main expectation |
Teacher professional development 46.5%, Points for Teaching Career 4.8%, Know a MOOC 14.7%, Evidence of MOOC participation 13.6%, Evidence of Tecnológico de Monterrey 18.2%, Economic stimulus 2.3% |
|
Workspace |
School 26.9%, Home 67.7% and Cybercafé 5.4% |
|
Hours participation |
No specific hour 27.1%, 8:00am-10:59am 9.1%, 11:00am-02:59pm 7.3%, 03:00pm-05:59pm 8.3%, 06:00pm-08:59pm 22.0%, 09:00pm-12:00am 26.2% |
|
Intention to complete the course |
I intend to finish the course 96.03%, I just want to know the agenda 2.01%, I just want to participate in some activities 1.52% and I do not intend to finish the course 0.45% |
In relation to their expectations, 43.3% of the participants believed that through MOOCs they could acquire enriched learning and 23.6% mentioned that they would experience higher learning proficiency compared to a classroom course, while 29.8% thought that they would be getting the same learning experience. It is relevant that 96% of the participants expressed commitment to the successful completion of the course, and 68.99% reaffirmed their assurance to strengthen their professional development through participation in other MOOCs offered as part of a teacher training update strategy.
On completion, the MOOC achieved a terminal efficiency rate exceeding 22% (see MOOC 12 in Figure 2), which is considered “atypical”, as well as a higher commitment rate from participants, which reached 52.15%. It is noted that the average terminal efficiency rate of all MOOCs this institution has offered is 4%, which coincides with the percentage reported in studies by the University of Pennsylvania Graduate School of Education (Penn GSE, 2013).
4. Conclusions
In this paper, quality indicators endorsing the design of a MOOC on Leadership in strategic educational management have been identified, as have the profiles of participants in its first edition and the rate of terminal efficiency.
The main conclusion from this research is that the success of a MOOC cannot be evaluated without measuring results. Thus, institutions and consortia should establish indicators to focus efforts on improving their pedagogical quality. Design changes and development cannot be planned either, as these actions require the implementation of appropriate mechanisms to measure the participants’ performance, which should be integrated into the standards of their educational model. When applying quality criteria, it should be considered that the indicators cannot operate in isolation, so it is necessary to define them in order to build a complete picture of the educational system that will receive feedback.
According to 55 experts, time is a key factor that impacts the pedagogical quality of a MOOC; however, they emphasize that the results are a logical consequence of the interrelationship between the Pedagogical, Functional, Technological and Time factors. When designing a MOOC, the time it will take participants to review the content, videos, resources, exercises and tests, and to take an active part in activities and collaborative learning discussion boards should be considered (Gros, Barbera, & Kirschner, 2010). The designed system involves promoting self-regulation skills, and must include the elements and approaches that support the theory of connectivism in order to be able to offer open educational resources that provide a real strategic opportunity to improve the quality of education (UNESCO, 2012).
As the use of technology in education advances, the understanding of the elements and principles of connectivism will be challenged, and educational research will provide evidence on the network capacities and possibilities intertwined with the various learning styles of individuals. By providing pertinent information, the learning potential of MOOCs to improve teacher training may be evaluated. Only then can the pedagogical quality of MOOCs offered by educational institutions be recognized and contribute to the vision of a future in which everyone has access to a world-class education.
4.1. Future research
There are significant areas of opportunity related to the evaluation of MOOCs, as standards regarding their pedagogical quality have yet to be agreed upon. Little is known about their uses and scope, or about the ways in which they are integrated into the educational models of different institutions and consortia. Separate lines of research are therefore proposed to address this issue. On the one hand, it is crucial to carry on studying in detail the results of each of the indicators that make up the different subcategories in order to understand the interplay of all factors and their level of impact on pedagogical quality. On the other hand, the definition of success of a similar course should go further than the rate of terminal efficiency and delve deeper into the analysis of traffic patterns to understand the interests and behavior of its participants.