2. Assessment of the quality of virtual higher education

The quality of virtual education is a complex matter, especially, given its multifactorial nature. Some authors link its quality to learning processes, products and services based on the use of information and communication technologies (ICTs) (Marúm-Espinosa, 2011). Others (Ardila-Rodríguez, 2011) refer to the quality of virtual education as user satisfaction, system excellence, positive academic results and good social impact. According to Silvio (2006), such quality should be understood as the union between academic excellence, equity and social relevance. In his opinion, bringing quality to virtual education means studying it from its singular and specific aspects, taking into consideration all variables, criteria and specific indicators arising from the characteristics of virtual education. According to that author, the quality of virtual education cannot be compared or reduced to the criteria applied to traditional education, as it requires specific assessment models and methodologies adapted to the different contexts of virtual education.

Rubio (2003) highlights two types of models that can be used to assess the quality of virtual education:

1. Partial focus based mainly on one of the following:
   • Assessment of the educational activity
   • Assessment of the educational material
   • Assessment of the technological platforms
   • Financial assessment
2. Global focus. It is the assessment based on a series of elements included in an e-learning solution in order to set out the criteria to be used to manage and assess its quality. Two trends can be distinguished within this focus:
   1. Assessment systems focused on Total Quality Management (TQM) models and/or standards
   2. Systems based on benchmarking

As can be observed, a university can assess the quality of its virtual education using different methods in order to reach the highest quality and the best results possible based on its competitiveness’ conditions. One of the methods is benchmarking, the purpose of which is to discover the very best practices.

3. The proposed methodology for performing international benchmarking of the quality of virtual higher education

Nowadays, there are several methodologies describing how benchmarking should be carried out. However, most of them have been designed for application to the industrial sector and, thus, it is difficult to apply them in order to assess the quality of virtual education, as they do not take into consideration its specific context. For this reason, the author proposes her own methodology to carry out such benchmarking, setting out all the necessary phases and subphases (see Figure 1).

According to Figure 1, the benchmarking process is divided into four phases and thirteen subphases, as described below.

Phase I. Planning of the benchmarking project

Subphase 1: Definition of the purposes of the benchmarking

The start point of each benchmarking process is to define its purposes and main tasks to be fulfilled.

Subphase 2: Definition of the quality variables and indicators

In this subphase, all the variables and indicators to be used to measure quality will be defined.

Subphase 3: Selection of the data collection methods and sources

In this subphase, the type of data necessary to carry out the research will be defined. Given the relevance of data collection, various data sources will be defined.

Subphase 4: Design and verification of the tools to be used

The last step of phase I consists in designing data collection tools to be used during the benchmarking process.

Phase II. Search for the best practices (identification of the partner university)

Subphase 5: Development of the criteria to be applied to select the partner university

The first subphase of phase II, consists in developing the criteria to be applied to select the universities to be considered for the final selection of the partner university (the university with the best practices).

Subphase 6: Identification of potential partner universities

This subphase requires exhaustive research using secondary data in order to identify the best university according to certain variables. Based on the results from that research, the universities with the best possible practices will be identified in order to select potential partner universities.

Subphase 7: Selection of the partner university

In this subphase, the university that best fulfills all the criteria set out in subphase 5 will be identified.

Subphase 8: Contacting the selected partner university

At this point, the first formal contact with the selected partner university will be made. For this purpose, it is necessary to contact the persons in charge of the that university, informing them of the purposes of the research, as well as of the reasons why the selected university is considered one of the best of its kind, asking them whether they are interested in participating in the research.

Phase III. Data collection and analysis

Subphase 9: Analysis of the university that requires improvements

Based on the indicators drawn up in subphase 3, all the internal data necessary to assess the performance of the university will be collected, using the collection sources and tools created in that subphase.

Subphase 10: Analysis of the partner university

Based on the indicators drawn up in subphase 3, all the internal data necessary to assess the performance of the partner university will be collected, using the collection sources and tools created in that subphase.

Subphase 11: Comparison of the data obtained and identification of the gaps

In this subphase, the results from the application of the criteria to the analyzed university will be compared to those obtained from their application to the partner university. The differences between the results will be defined in order to identify the gaps between the two universities.

Phase IV. Action

Subphase 12: Drafting of an action plan for quality improvement

In this phase, an improvement plan containing specific actions to be taken will be designed in order for the analyzed university to achieve the very best practices.

Subphase 13: Implementation of the action plan

This subphase consists in implementing the previously drawn up plan, and in following up and monitoring its implementation.

4. Empirical application of the proposed methodology. Research results

In order to verify the efficiency of the proposed methodology, the author applied it from April to August 2013 to the Polish Virtual University (PUW).1 The benchmarking was performed according to the methodology described in Figure 1.

Phase I. Planning of the benchmarking project

Subphase 1: Defining of the purposes of the benchmarking

The main purpose of the benchmarking was to assess the quality of the education offered by PUW in order to compare it to a leading university and define the gaps between them so as to identify the steps that should be taken in order to improve the quality of the education offered by PUW.

Subphase 2: Definition of the quality variables and indicators

Based on the analysis described in Table 1, it can be concluded that the most common benchmarking variables applicable to virtual education can be divided into five main groups:

• Strategic plan
• Institutional context
• Educational agents
• Teaching-learning processes
• Virtual platforms

The aforementioned variables were applied to this research. Table 2 below defines each of the aforementioned variables, as well as the criteria and indicators to be used for each of them.

Subphase 3: Selection of the data collection methods and sources

The author applied documentary analysis methods and social research techniques in order to collect all the necessary data. For this purpose, semi structured face-to-face interviews and non-participant observation of the virtual classroom were carried out. Both primary and secondary sources were used for documentary analysis. Regarding primary sources, the author used, among others, the documentation provided by the researched university for the variables described above, that is, strategic plans, curricula, course documents, student statistics, teaching guides and materials. Regarding secondary sources, the author used, among others, information materials provided by the Education Ministries of Poland and Spain, reports drafted by the Polish E-learning Association, the Polish Accrediting Commission, the Spanish National Agency for Quality Assessment and Accreditation (ANECA) and the Catalan University Quality Assurance Agency (AQU).

A total of eight semi-structured interviews were carried out in order to understand, as much as possible, the issues related to the scope of this research. The interviews were carried out with directors of virtual programs (2), human resources managers (2), managers of virtual classrooms (2) and teachers-tutors (2).

The non-participative observation of virtual classrooms allowed the quality of the virtual environment to be assessed. Furthermore, the author applied her knowledge of virtual education acquired through years of professional experience as a teacher-tutor at PUW and through her studies at various virtual universities, among others, the Open University of Catalonia (UOC), the National Distance Education University (UNED) of Spain and the Virtual Educa Teacher Training Institute of Argentina.

Source: Drawn up by the author based on Duart and Martínez (2001), Espinosa and González (2006), García Aretio (1998), Giorgetti et al. (2013), HEQC (1997), Ornellas and Muñoz (2012), Sánchez (2009) and Sarramona (2001)

Subphase 4: Design and verification of the tools to be used

In order to collect all the necessary data to carry out the benchmarking, the author drew up five assessment protocols to be used for the comparative assessment.

These were:

• A comparative assessment protocol of the strategic plans
• A comparative assessment protocol of the institutional context
• A comparative assessment protocol of the teaching-learning process
• A comparative assessment protocol of the educational agents
• A comparative assessment protocol of the virtual platform

The results from the application of the aforementioned protocols have been arranged according to the Likert scale, ranging from 1 (non-fulfillment of the indicator) to 4 (high fulfillment of the indicator). Furthermore, comments were made regarding each of the applied indicators (qualitative assessment), as the author considered that subjective values should be included in the assessment. For this reason, the objective data obtained from the application of the protocols was completed with the author’s comments on the results obtained (see Table 3). The drafted protocols were assessed by four experts in virtual higher education.

Phase II. Search for the best practices (identification of the partner university)

The author chose a Spanish university as partner university, given that Spanish universities offer highly developed virtual education, as shown by numerous international rankings, such as the Online MBA Listing 2013 by the Financial Times (Financial Times, 2013) or Ranking Mundial de Universidades en la Web (Consejo Superior de Investigaciones Científicas, 2013).

A description of the selection process for the partner university is given below:

– During subphase 5, the author created six selection criteria: online mode, access to ICTs, course variety, other services offered, quality and prestige.

– During subphase 6, a total of 41 Spanish universities offering virtual education were analyzed. As a result, four preselected universities were thoroughly assessed, applying the above-mentioned criteria.

– During subphase 7, UOC was selected as the partner university, as it satisfactorily fulfilled all the applied criteria.

– Subphase 8 consisted of sending a letter to the partner university, requesting its authorization to carry out the benchmarking and explaining the functioning of the entire process.

Phase III. Data collection and analysis

Subphases 9 and 10: Internal analysis of PUW and UOC

The purpose of these subphases is to collect data about the partner university and the quality of its virtual education. For this reason, prior to collecting that data, the quality of the virtual education offered by PUW was assessed in order to increase the possibility of detecting all areas to be improved. The assessment was carried out using comparative assessment protocols (see Table 3 for an example of those protocols) and the data collection sources described in subphase 3.

Subphase 11: Comparison of the data obtained and identification of the gaps

Once the quality of the virtual education offered by PUW had been assessed, its situation as compared to the partner university was defined in order to identify possible gaps (see Table 4) between the two universities in five main areas, as well as to determine their causes.

A brief summary of those gaps is given below.

The first variable used by the author was the strategic plan of each university. One of the biggest weaknesses of PUW in this area is that its vision and the values that it should abide by for its future development have not been developed. Furthermore, the strategic mission and objectives of PUW are significantly less well defined than those of UOC.

Another relevant variable that creates a positive and a negative gap in comparison to the partner university is the institutional context. The positive gap is that the teaching load of PUW (student/teacher ratio) is lower than that of the partner university. The negative gap is that PUW lacks not only a virtual library but also strategies for the availability of technological services and human resources management.

Regarding educational agents, there is a clear negative gap between the two universities, as PUW does not provide its students with any additional services, that is, it does not assist them with their access to the job market and it does not encourage them to participate in educational events or scientific research/programs. What is more, not even PUW’s teachers participate in such events. Moreover, PUW has not defined any teacher/tutor selection procedure.

Regarding the teaching-learning process variable, there is a very significant negative gap between the two universities. In this case, PUW has not set out any procedures for the updating or drafting of studies programs. Furthermore, it lacks educational material, learning strategies and technologies, and the activities it offers are not varied. Finally, it should be noted that the qualification criteria and assessment systems applied by PUW teachers are not detailed enough.

Explicaciones: PUW ♦ ♦; UOC • •

* The average score is the sum of scores obtained for each indicator divided by the total number of indicators making up each criterion

Regarding the last variable, that is, the virtual platform, the negative gap of PUW in comparison to the partner university is due to the fact that the virtual platform used by PUW offers very few tools for team projects and it does not offer any tool for students to create their own personal learning environment (PLE). Moreover, the platform offered by PUW has not had accessibility standards applied to it, so it does not offer students with special needs the possibility of studying.

Phase IV. Action

Subphase 12: Drafting of an action plan for quality improvement

In order to remove or reduce the gaps identified in phase III, an improvement action plan was designed, consisting of specific actions to be taken so that PUW could not only remove the said gaps, but also achieve the very best practices.

Table 5 shows an example of the plan drafted for PUW.

Subphase 13: Implementation of the action plan

In this subphase, the drawn-up action plan had to be implemented and monitored. This subphase is extremely important, as any failure could lead to the strategic objectives not being reached and, thus, the quality of the virtual education offered by PUW not being improved. In order to avoid that risk, a set of the most relevant indicators was drawn up in order to ensure that each step of this subphase would be carried out.