ARTICLES
PSYCHOLOGY AND EDUCATION

 

Course reaction scale in e-learning: adaptation and factor structure

 

Escala de reacción al curso en enseñanza virtual: adaptación y estructura factorial

 

 

Lara Barros Martins^I; Thaís Zerbini^II; Francisco José Medina^III

^ISouthern Collegel (IMED), Passo Fundo, RS, Brazil
^IIUniversity of São Paulo, Ribeirão Preto, SP, Brazil
^IIIUniversidad de Sevilla, Seville, Spain

Mailling address

 

 

---

ABSTRACT

This paper aims to adapt and analyze the factor structure and psychometric properties of the Reaction to Instructional Proceedings Scale in Distance Education (ERPI-EAD) and its relation with the effectiveness of training at work. In total, 3,600 employees from a Brazilian public bank participated in this study by answering the ERPI-EAD after taking part in an online course. We measured training transfer with a self-evaluation scale. Internal consistency, confirmatory factor analysis, and multiple regressions were conducted. A one-factor structure and a high level of fit for the model were found. The participants' satisfaction with the course was related to the effectiveness of training. We suggest using the ERPI-EAD as a reliable and valid measure that predicts the effectiveness of training, can contribute to collect relevant information to improve instructional aspects and ensure better performance results in the workplace.

Keywords: distance education; evaluation; satisfaction; scale; effectiveness of training.

---

RESUMEN

El objetivo de este estudio fue adaptar y examinar la estructura factorial y las propiedades psicométricas de la Escala de Reacción al Curso en Enseñanza Virtual (ERC-EV) y su relación con la eficacia de la formación en el trabajo. Un total de 3.600 empleados de un Banco estatal brasileño participó en el estudio respondiendo a la ERC-EV después de haber participado en una teleformación. Se midió la transferencia de la formación con una escala de autoevaluación. Análisis de consistencia interna, factoriales confirmatorios y regresiones múltiples fueron realizados. Una estructura de un único factor y un alto nivel de ajuste para el modelo fueron encontrados. La satisfacción de los participantes con el curso estuvo relacionada con la eficacia de la formación. Se sugiere el uso de la ERC-EV como una medida fiable y válida, que predice la eficacia de la formación, puede ayudar a recoger información relevante para la mejora de los aspectos de la instrucción y asegurar mejores resultados de rendimiento en el trabajo.

Palabras-clave: educación a distancia; evaluación; satisfacción; escala; eficacia de la formación.

---

 

 

Introduction

Reactions can be defined as trainees' opinions on several aspects of training and their satisfaction with it. Participants judge the quality of the instructional plan or its programming (goals, appropriateness, and sequence of content, duration, etc.), the quality of its execution, and the support offered (e.g., teaching facilities and materials). They are also indicators of the instruction delivery quality, reflecting participants' general satisfaction with the course (affective results), as well as contents relevance and usefulness- to what extent the graduates feel that the training was helpful to them (Alvarez, Salas, & Garofano, 2004; Klein, Noe, & Wang, 2006).

Reaction measures are actively present in studies on effectiveness evaluation of Training, Development, and Education (TD&E) programs. Current research concerns mainly focus on how to use reactions to improve training design and delivery. This measure is not used as the sole criterion of training effectiveness but associated with other variables that may lead to a better instructional planning, according to data on the evaluation of effects produced (Aguinis & Kraiger, 2009).

This information's measuring and collection aim at improving instruction procedures to facilitate the achievement of learning and performance results an organization seeks, as the adoption of strategies and teaching methods, combined with the choice of the instruction media, are appointed as relevant factors for the impact of training in work (Burke & Hutchins, 2007)- the leading indicator of the effectiveness of training actions at the individual level, which refers to the long-term effects of training on the graduate's behavior, regarding general performance at work (Abbad, Pilati, Borges-Andrade, & Sallorenzo, 2012; Grossman & Salas, 2011; Pilati & Abbad, 2005).

Instructional procedures are made up of different teaching strategies and principles, such as tools, methods, and the context itself, which are combined and integrated to create a training delivery approach, as the way it was planned, organized, and structured matters to be successful (Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012). Thus, it is necessary to choose appropriate instructional procedures according to previous results that showed links with the transfer process, i.e., the effective application to the work environment of the new skills learned (Burke & Hutchins, 2007).

It is necessary to measure the reactions to educational actions online and to improve the use of this information (Long, DuBois, & Faley, 2008), as e-learning contexts have particularities (e.g., technology-supported instruction procedures, online tutoring, virtual learning environment) that can influence its effectiveness. Overall, research appoints that the participants react favorably (Lim, Morris, & Kupritz, 2007; Vaughan and MacVicar, 2004), even demonstrating a preference for the distance method in comparison with other modes of instruction, such as traditional classroom education (DeRouin, Fritzsche, & Salas, 2005).

Satisfaction can lead to higher levels of commitment, learning, and success in e-learning (Sahin & Shelley, 2008), being linked to the skills transfer process and interfering with training results (Carvalho & Abbad, 2006; Zerbini & Abbad, 2010). Thus, knowing the factors that can influence the participants' (dis)satisfaction permits optimizing the impact of e-learning programs, providing support for effective interventions (Martins & Zerbini, 2016) and increasing learning retention (Chyung & Vachon, 2013).

There was a proposal of specific national reaction measures for courses of this nature to accompany new trends in the supply of e-learning programs. These measures aim to verify participants' satisfaction with the quality of course procedures (objectives, strategies and teaching aids, contents, learning assessments) and also with virtual interaction tools (links, forums, chats, etc.).

Zerbini and Abbad (2005) proposed the first version of the reaction to instructional proceedings measure, and it consisted of three factors: Traditional Procedures (9 items, α = 0.91), Web Resources (7 items, α = 0.89), and Activities and Exercises (7 items, α = 0.85). Borges-Ferreira (2005) revalidated the same measure- a one-factor structure (12 items, α = 0.89)- and later Zerbini and Abbad (2009) did the same- Traditional Procedures (12 items, α = 0.91) and Web Resources (3 items, α = 0.76). Subsequently, Martins and Zerbini (2015) proposed an adapted version with 17 items (α = 0.96) and a one-factor structure. The adaptations were necessary to match the measures to the participants' different samples and the application contexts (distance or hybrids), such as technical and vocational courses, open and free within Brazil, or undergraduate courses.

Although the measures mentioned presented good to excellent psychometric properties, in some cases, the large number of items in single-factor solutions and the high-reliability indices could suggest the repetition of content or overlapping of aspects evaluated by the items. In fact, there was the presence of multicollinearity, for example, in the instrument by Martins & Zerbini's (2015)- four pairs of highly correlated items.

Therefore, in this study, the aim was to adapt and analyze the factor structure and the psychometric characteristics of the Reaction to Instructional Proceedings Scale in Distance Education (ERPI-EAD) and its relation with the effectiveness of training at work. The confirmation of the factor structure of the ERPI-EAD is justified as several Brazilian surveys, including in different e-learning contexts (Borges-Ferreira & Abbad, 2009; Carvalho & Abbad, 2006; Martins & Zerbini, 2015; Zerbini & Abbad 2009), used the measure, which maintained the good psychometric qualities and the internal consistency indices considered excellent, but all had an exploratory design and, therefore, did not execute the statistical validation of the instrument by means of confirmatory factor analysis, maintaining this step as a research agenda. Then, this study intends to present validity evidence of the ERPI-EAD, permitting the investigation of e-learning course participants' satisfaction in different contexts.

 

Method

Participants

Participants were employees of a Brazilian state-owned bank, who had conducted an online self-instructional training at work about the theme of promoting operational efficiency in work activities ("Operational Efficiency") during the workday. The answers obtained on the demographic and functional characteristics of 1,639 respondents indicate that the majority is male (56.8%), between 46 and 55 years of age (26.1%), works in the Business and Management Support Unit (37.8%), has 1 to 3 years of experience on the job (20.3%) and holds an undergraduate degree (33.6%).

Instruments

The ERPI-EAD contains 9 items, associated with a quality scale ranging from 1 ("Poor") to 5 ("Excellent"), which assess the participants' satisfaction with the quality of the course procedures. It was adapted from the version proposed by Martins and Zerbini (2015): a one-factor empirical structure with 17 items, α = 0.96 and factor loadings between 0.60 and 0.86. Due to the presence of multicollinearity (i.e., pairs of highly correlated items) in the original structure, a more significant adaptation of the scale was needed, with the elimination of redundant items, to generate a more economical instrument. The ERPI-EAD was submitted to validation by experts (n = 2), specialists in the area, who sought to analyze its content when reviewing the items, verifying the theoretical adequacy of the instrument, the measuring scale and the instructions for completion.

To measure training transfer, we used an adapted version of the "Impact Scale in Workplace Training" (Pilati & Abbad, 2005), with 7 items (α = 0.89 and factor loadings: 0.62 to 0.86) and alternative answers scored from 1 ("I do not agree") to 5 ("I totally agree"). Training transfer refers to the indirect influence of the training on the overall performance, attitudes and motivation of the graduates (examples of items: "I carry out my work more quickly," "Improved the quality of my work," "Increased my motivation for work").

Data collection and analysis procedures

The data collection was carried out entirely at a distance, through the availability of the virtual questionnaire on the page of the Corporate University of the Bank to a potential sample of 3,600 employees who had participated in the course mentioned.

To perform the analysis, we used the SPSS (Statistical Package for the Social Science) and the SPSS 23.0 AMOS. Before the actual analysis, the 2,201 (61.1% rate of return) questionnaire responses, obtained in the self-assessment of the conducted training, underwent statistical treatment to verify the existence of absent data, from uni and multivariate outliers, and we respected the assumption of normality (asymmetry and kurtosis between- 2.0 and 2.0).

We submitted the 1,915 valid cases for the ERPI-EAD to confirmatory factor analysis (CFA, Maximum Likelihood Method) and internal consistency (Cronbach's alpha: α). To judge the goodness of fit of the model, the incremental adjustment measures (GFI, CFI, and TLI) should be higher than 0.90 (ideally > 0.95) and residual values (RMSR and RMSEA) should be inferior to 0.08 (ideally < 0.05). We used the BIC parsimony index to compare adjustments between models (Byrne, 2010). To analyze the predictive validity of the scale, we used multiple regression analysis, with the single ERPI-EAD factor as a predictor of the effectiveness of training at work (training transfer).

Regarding the ethical considerations involved, the study received approval from the Research Ethics Committee of the Faculty of Philosophy, Sciences, and Letters of Ribeirão Preto at the University of São Paulo (CAAE 46449615.0.0000.5407).

 

Results

Adaptation of ERPI-EAD

We analyzed the instrument by Martins and Zerbini (2015), considering its semantic aspects (content of the items and their factor distribution) and psychometric properties (factor loadings, multicollinearity, Cronbach's alpha if the item were excluded, etc.) for redundant items to be rewritten or deleted. This edition phase resulted in the exclusion of 8 items, focusing on the specific application context and sample of this study, that is, workers participating in distance training, but the instrument can also be applied to students in academic settings.

Theoretical Analysis

The suggestions from the expert validation consisted of modifying the scale, as it had no neutral point, equidistant from the extremes both qualitatively and quantitatively, and three of the five anchor points had a positive connotation. Therefore, for the points to be equidistant from each other and to respect an order, they were modified by including the labels at all points, but with a better distribution of the qualifiers: 1 "Poor," 2 "Bad," 3 "Regular," 4 "Good," and 5 "Excellent".

Descriptive Analyses

Descriptive statistics for ERPI-EAD items are displayed in Table 1.

In general, participants judged that the quality of the instruction procedures used throughout the course is good, as the means vary between 3.99 (SD = 0.73) and 4.30 (SD = 0.63) on a five-point scale, being satisfied with the procedures adopted. As the coefficients of the standard deviations were low, we can consider that the respondents agreed on the evaluated aspects, highlighting the language used in the course material, the presentation sequence of the content, and the correspondence between the proposed content and the course objectives.

Confirmatory factor analysis and internal consistency

The analysis of normality indicates that all items are within the established reference limit, indicating normal distribution- asymmetry between- 0.86 and- 0.48 and kurtosis between 1.06 and 1.94 -, except for item 6 with a coefficient slightly above the limit (2.03).

The initial hypothetical model (9-item one-factor empirical structure and α = 0.92) for the ERPI-EAD presents good fit indicators, except for CMIN/DF and RMSEA. By introducing correlations for the two pairs of errors (e1-e2 and e6-e7) appointed with higher values in the residue analysis (respecified model 1), we obtained increased values of the indicators and a fair RMSEA estimate. Items 1 ("Correspondence between the proposed content and the course objectives") and 2 ("Correspondence between course content and personal objectives"), and 6 ("Proposed activities/tasks at the end of each unit") and 7 ("Guidance for the solution of errors in activities/tasks"), present content similarity in their writing, which may have led the participants to understand the two pairs of items as the same measure, contributing to finding the covariances between them, although they are low: r = 0.30 (e1-e2) and r = 0.19 (e6-e7). When testing a new model (respecified model 2), only with the correlation between the pair of items 1 and 2, the values of the indices slightly increased and the value of BIC decreased (ΔBIC = 146.07), indicating that this is more adjusted than the initial hypothetical model (see Table 2 and Figure 1), therefore being more economical.

 

 

The ERPI-EAD has an excellent Cronbach's Alpha (α = 0.92), totaling nine items (factor loadings between 0.65 and 0.81) that measure the reactions to instructional proceedings of online courses.

Influence of ERPI-EAD on Effectiveness of Training at Work

In relation the self-evaluated training, the course reactions were correlated with the training transfer (r = 0.52) and contributed significantly to its explanation (R² = 0.27; β = 0.52 p < 0.01), confirming the positive influence of the reactions to the occurrence of the training result phenomenon on the graduates' performance.

 

Discussion

The single-factor solution found is theoretically meaningful, and the results confirm the validity and reliability evidence of the measure, which serves as a useful assessment tool to measure the satisfaction of participants in online educational events regarding various aspects of the course. As there is empirical evidence that proves the explanatory power of the reactions and their contribution for the occurrence of training transfer at work and the learning process, the instrument can diagnose the quality and effectiveness of distance training in corporate contexts, as well as in educational institutions offering e-learning courses.

The adaptation of the instrument managed to eliminate a large number of items (8) which, supported by the presence of multicollinearity- the existence of highly correlated pairs of items -, indicated the repetition of content or overlapping of aspects evaluated by the original instrument (Martins & Zerbini, 2015). The previous studies using the measure chose not to exclude items because they were the first applications and tests of the instruments in specific e-learning contexts (Borges-Ferreira & Abbad, 2009; Zerbini & Abbad, 2009). The literature shows, however, that instruments to measure reactions usually consists of a very small number of items in sin-gle-factor solutions (e.g., Klein et al., 2006), indicating that the exclusion of items and a more significant adaptation of the scale was necessary to generate a more economical, quick, and easily applicable scale. We reached this objective and produced a new version that focused on the multicollinearity between the items of the measure, which was extinguished while maintaining the excellent psychometric qualities of the scale.

Despite the high level of adjustment found for respecified model 2, the factorial structure considered more adjusted, it includes the correlation between the errors of items 1 ("Correspondence between the proposed content and the objectives of the course") and 2 ("Correspondence between course content and personal goals"). It is observed that, in order to respond to the items of the ERPI-EAD, the respondent's reference concerns aspects related to the course and its instructional procedures, except for item 2, the only item that refers to the participant's personal goals. This may indicate that the respondent does not distinguish between items 1 and 2, perhaps due to the similar content in their formulation, or because the respondent is led to answer the items in relation to the course and only in this case (item 2) about the agreement between the course and personal goals. The choice not to remove item 2 from the structure, however, is justified because of its theoretical meaning.

The construction and use of specific measures for e-learning are essential, considering the great advance in the supply of internet-mediated courses at present. The application of the ERPI-EAD, in different contexts and samples, can contribute to the collection of relevant information to improve these educational actions with regard to the existing instructional aspects (see Salas et al., 2012), as satisfaction with these aspects positively affects the results and academic success in online learning environments, producing good effects on the graduates' behaviors.

These effects of the reactions to the successful transfer of the new skills learned in training for work were supported in this study, as the general performance of the graduates who reported satisfaction with the formal procedures of the course under evaluation was modified (training transfer). These results confirm findings that have already been compared in previous investigations (e.g., Giangreco, Carugati, Sebastiano, & Della Bella, 2010) and indicate the pertinence of continuing to measure post-training satisfaction levels to produce positive consequences on performance in the work of participants in instructional events. How satisfied the graduates were with the training matters and generates applied implications: training will be transferred to the work environment to the extent that these conditions have been met.

The instrument used to measure the impact of training at work is a self-reported measure. Therefore, the results indicate that, from the perspective of the graduates, their performance at work improved after having participated in the training. The self-assessment permits the measuring the graduates' perception of the effects of the training on their behavior, in terms of gains in the general performance at work.

It is worth mentioning that, to minimize the self-report bias, we also evaluated the training (N = 2,411; return rate of 66.9%), when the graduates' hierarchical superiors evaluated the impact of training on the work of their subordinates. Self- and hetero-evaluation allowed us to analyze the relationships between the average global levels of training transfer reported by the workers (training graduates) and their supervisors. The favorable descriptive results showed a high convergence between the two evaluations (X = 4.31 and SD = 0.70) and managers (X = 4.07 and SD = 0.83), showing that they strongly agreed on the contributions of the training, regarding the degree of manifestation of the skills at work.

One should continue evaluating the affective reactions (liked the training or not) and the usefulness of the training at the end of the program, being essential variables to explain the occurrence of the phenomenon of the training result in the graduates' performance. In this sense, future research should continue to certify the predictive capacity of the ERPI-EAD, both concerning learning outcomes and the impact of training in work.

 

References

Abbad, G., Pilati, R., Borges-Andrade, J. E., & Sallorenzo, L. H. (2012). Impacto do treinamento no trabalho - medida em amplitude. In G. Abbad, L. Mourão, P. P. M. Meneses, T. Zerbini, J. E. Borges-Andrade, & R. Vilas-Boas (Orgs.), Medidas de Avaliação em Treinamento, Desenvolvimento e Educação (pp. 145-162). Porto Alegre: Artmed.         [ Links ]

Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60(1),451-474. doi: 10.1146/annurev.psych.60.110707.163505        [ Links ]

Alvarez, K., Salas, E., & Garofano, C. M. (2004). An integrated model of training evaluation and effectiveness. Human Resource Development Review, 3(4),385-416. doi:10.1177/1534484304270820        [ Links ]

Borges-Ferreira, M. F. (2005). Avaliação de reações e aprendizagem em disciplinas de curso técnico profissionalizante oferecidas a distância. (Dissertação de Mestrado não publicada). Instituto de Psicologia, Universidade de Brasília, Brasília, DF.         [ Links ]

Borges-Ferreira, M. F., & Abbad, G. (2009). Avaliação de aprendizagem em disciplinas de curso técnico a distância. Estudos de Psicologia (Natal), 14(2),141-149.         [ Links ]

Burke, L. A., & Hutchins, H. M. (2007). Training transfer: An integrative literature review. Human Resource Development Review, 6(3),263-297. doi:10.1177/15344 84307303035        [ Links ]

Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming. New York/London: Routledge Taylor & Francis Group.         [ Links ]

Carvalho, R. S., & Abbad, G. (2006). Avaliação de treinamento a distância: Reação, suporte à transferência e impactos no trabalho. Revista de Administração Contemporânea, 10(1),95-116.         [ Links ]

Chyung, S. Y., & Vachon, M. (2013). An investigation of the profiles of satisfying and dissatisfying factors in e-learning. Performance Improvement Quarterly, 26(2),117-140. doi:10.1002/piq.21147        [ Links ]

DeRouin, R. E., Fritzsche, B. A., & Salas, E. (2005). E-learning in organizations. Journal of Management, 31(6),920-940. doi:10.1177/0149206305279815        [ Links ]

Giangreco, A., Carugati, A., Sebastiano, A., & Della Bella, D. (2010). Trainees' reactions to training: Shaping groups and courses for happier trainees. The International Journal of Human Resource Management, 21(13),2468-2487. doi:10.1080/09585192.2010.516598        [ Links ]

Grossman, R., & Salas, E. (2011). The transfer of training: What really matters. International Journal of Training and Development, 15(2),103-120. doi:10.1111/j.1468-2419.2011.00373.x        [ Links ]

Klein, H. J., Noe, R. A., & Wang, C. (2006). Motivation to learn and course outcomes: The impact of delivery mode, learning goal orientation, and perceived barriers and enablers. Personnel Psychology, 59(3),665-702. doi:10.1111/j.1744-6570.2006.00050.x        [ Links ]

Lim, D. H., Morris, M. L., & Kupritz, V. W. (2007). Online vs. blended learning: Differences in instructional outcomes and learner satisfaction. Journal of Asynchronous Learning Networks, 11(2),27-42.         [ Links ]

Long, L. K., DuBois, C. Z., & Faley, R. H. (2008). Online training: The value of capturing trainee reactions. Journal of Workplace Learning, 20(1),21-37. doi:10.1108/13665620810843629        [ Links ]

Martins, L. B., & Zerbini, T. (2014). Educação a distância em instituições de ensino superior: Uma revisão das pesquisas. Revista Psicologia: Organizações e Trabalho, 14(3),271-282.         [ Links ]

Martins, L. B., & Zerbini, T (2015). Evidências de validade de instrumentos de reações no ensino superior a distância. Estudos e Pesquisas em Psicologia (UERJ), 15(1),116-134.         [ Links ]

Martins, L. B., & Zerbini, T. (2016). Fatores influentes no desempenho acadêmico de universitários em ações educacionais a distância. Estudos de Psicologia (Natal), 21(3),317-327.         [ Links ]

Pilati, R., & Abbad, G. S. (2005). Análise fatorial confirmatória da escala de impacto no treinamento no trabalho. Psicologia: Teoria e Pesquisa, 21(1),43-51. doi:10.1590/S0102-37722005000100007        [ Links ]

Sahin, I., & Shelley, M. (2008). Considering students' perceptions: The distance education student satisfaction model. Educational Technology & Society, 11(3),216-223.         [ Links ]

Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2)74-101. doi:10.1177/1529100612436661        [ Links ]

Vaughan, K., & MacVicar, A. (2004). Employees' pre-implementation attitudes and perceptions to e-learning: A banking case study analysis. Journal of European Industrial Training, 28(5),400-413.         [ Links ]

Zerbini, T., & Abbad, G. (2005). Impacto de treinamento no trabalho via internet. Revista de Administração de Empresas- RAE- eletrônica, 4(2),1-21. doi:10.1590/S1676-56482005000200001        [ Links ]

Zerbini, T., & Abbad, G. (2009). Reação aos procedimentos instrucionais de um curso via internet: Validação de uma escala. Estudos de Psicologia (Campinas), 26(3),363-371.         [ Links ]

Zerbini, T., & Abbad, G. (2010). Qualificação profissional a distância: Avaliação da transferência de treinamento. Paideia (Ribeirão Preto), 20(47),313-323.         [ Links ]

 

 

Endereço para correspondência:
Lara Barros Martins
R. Senador Pinheiro, 304, Vila Rodrigues
Passo Fundo, RS. CEP: 99070-220
E-mail: lara_bmartins@hotmail.com

Submission: 7.10.2017
Acceptance: 11.12.2017