ARTIGOS

Measuring socioemotional skills of children and adolescents: development and validation of battery

Bruno Figueiredo Damásio^I; Grupo Semente Educação^II

^IDepartamento de Psicometria, Universidade Federal do Rio de Janeiro, Rio de Janeiro, RJ, Brazil
^IIPrograma Semente de Desenvolvimento de Habilidades Socioemocionais, São Paulo, SP, Brazil

Mailing address

ABSTRACT

Social emotional skills (SES) is comprehended as a multidimensional construct, composed of emotional, cognitive and behavioral variables that positively affect healthy development throughout the life span. Recent studies have shown that SES can be developed and learned. In this sense, there is currently a broad agreement that the educational system should not solely focus on cognitive development but also on the development of children and adolescents' social and emotional competencies. This technical note briefly describes the development process and the psychometric properties of six scales aiming to measure SES. The scales were designed to measure the Programa Semente - a newly developed Brazilian social emotional learning (SEL) program. The instruments measure the constructs proposed by Collaborative for Academic, Social, and Emotional Learning (CASEL): self-knowledge, self-control, perseverance, empathy, responsible decision-making and prossocial behaviors. All the instruments presented evidence of validity based on the internal structure (factorial structure, reliability, configural, metric and scalar invariance) as well as based on external measures (convergent and/or concurrent validity). The final battery is composed of 45 items. Together, the constructs form a second-order factor, called SES, which, also, presented adequate goodness-of-fit indexes.

Keywords: Social emotional skills, measurement, scale, CASEL, Programa Semente.

Socioemotional skills (SES) have been conceived as a multidimensional construct that consists of emotional variables (e.g., self-awareness and self-control), cognitive variables (e.g., empathy) and behavioral variables (e.g., perseverance, responsible decision making and prosocial behavior) that foster healthy development throughout the lifecycle (Weissberg, Durlak, Domitrovich, & Gullotta, 2015). SES differ from Social Skills (SS) in the sense that they are more wide-ranging, encompassing not only factors characteristic of social interactions, but also individual, cognitive, emotional and behavioral factors. Studies have demonstrated that individuals who have higher SES levels display self-attitudes that are more positive, including greater self-esteem, self-efficacy, greater perseverance as to objectives, better interpersonal relationships, greater commitment, better scholastic performance, etc. In the long term, higher SES levels are associated with a higher educational level (i.e. undergraduate and graduate degrees), greater career achievement, domestic and employment relationships that are more constructive, better mental health indicators, less psychopathology and reduced behavioral difficulties (Domitrovich, Durlak, Staley, & Weissberg, 2017; Hawkins, Kosterman, Catalano, Hill, & Abbott, 2008; Weissberg et al., 2015).

There is presently a broad range of intervention programs that, based on scientific evidence, strive to teach and develop SES. In 2011, Durlak, Weissberg, Dymnicki, Taylor and Schellinger (2011) performed a meta-analysis, seeking to comprehend the impact of interventions on the development of SES. Encompassing 213 intervention programs and 270,034 children, the analysis revealed that SES development leads to better psychological and social performance, as well as influencing scholastic performance. Overall, children who were involved in some form of SES-development intervention program exhibited scholastic performance that was 11% higher than that of children and adolescents who had never experienced such interventions.

The scientific evidence on the subject demonstrates that SES can be developed and learned (Weissberg et al., 2015). Along these lines, public policy researchers, educators and specialists now widely agree that the educational system should concentrate not only on cognitive development but also on the development of youngsters' social and emotional skills, preparing students for life (Durlak et al., 2011).

This global movement has contributed to the establishment of a series of intervention programs aimed at developing SES. Nowadays in Brazil, SES development initiatives are still scarce. One such initiative is the Programa Semente, which was developed by the Semente Educação group. The proposed intervention is based on the guidelines of the Collaborative for Academic, Social, and Emotional Learning (CASEL), a non-profit organization made up of a pioneering team of researchers who, for over twenty years, have dedicated themselves to evaluating SES impacts throughout the life cycle and have contributed to developing and disseminating SES that have proved to be effective. As set forth by the organization, intervention programs should focus on developing the following key areas:

1. Self-Awareness – the ability to recognize your emotions, life goals, and personal values, as well as your potentials and limitations;

2. Self-Control – the ability to control your emotions and behavior, including impulse control, delayed gratification, perseverance in pursuing goals, etc.;

3. Empathy – the ability to put yourself in other people's shoes, being aware of their feelings and emotions;

4. Responsible Decision Making – the ability to make constructive decisions in relation to your personal behavior and social relations, taking into consideration ethical, safe standards of conduct, avoiding risk behavior and continuously evaluating the consequences of your own attitudes;

5. Prosocial Behavior – the ability to develop and maintain healthy, positive social relationships, which entails communicating clearly and effectively, listening actively, cooperating, seeking to resolve conflicts constructively, etc. (Weissberg et al., 2015).

The Programa Semente encompasses four modules aimed at developing children and adolescents' SES, focusing on self-awareness, self-control, perseverance, empathy, responsible decision making, and prosocial behavior, as proposed by CASEL (2015). Seeking to evaluate the Programa Semente's impact on SES development, quasi-experimental studies are being conducted to appraise the proposal's effectiveness, featuring the delineation of pre- and post-tests for experimental and control groups (Robson & McCartan, 2016). In order to assess the intervention, we developed quantitative questionnaires that rate all of the five aforementioned areas. The scales were devised based principally on the Programa Semente's goals. Each instrument's items were designed with the aim of analyzing the program's theoretical framework, endeavoring to produce scales that would be entirely in line with the proposed intervention. Furthermore, an integrative review of the literature on the subject contributed to assessing the existing instruments of each of the constructs that came closest to the intervention. In all, we designed six specific questionnaires for the Programa Semente: the Self-Awareness Scale, Self-Control Scale, Perseverance Scale, Empathy Scale, Responsible Decision-Making Scale and Pro-social Behavior Scale. The questionnaires are answered according to a five-point Likert scale ranging between 0 (Completely unlike me) and 4 (Completely like me).

Scales were validated in accordance with the guidelines of the American Educational Research Association, the American Psychological Association and the National Council on Measurement in Education (AERA, APA, & NCME, 2014). We assessed each instrument's validity evidence based on its internal structure (factor structure and reliability) and on external measurements (convergent and concurrent validity; AERA et al., 2014).

At the outset, we conducted exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) in order to ascertain the scales' factor structure. The number of retained factors was determined via Parallel Analysis, with random permutation of the observed data (Timmerman & Lorenzo-Seva, 2011), and via the Hull Method (Lorenzo-Seva, Timmerman, & Kiers, 2011). EFAs were conducted via Minimum Rank Factor Analysis (ten Berge & Kiers, 1991), with oblique Promin rotation, performed on a polychoric correlation matrix, bearing in mind the data's ordinal nature. Confirmatory factor analyses (CFAs) were conducted using MPlus software, via the Weighted Least Squares Mean- and Variance-adjusted (WLSMV) extraction method, which is appropriate for ordinal data (Muthén & Muthén, 2012). The fit indices employed to assess the models, based on the CFA, were the Root Mean Square Error of Approximation (RMSEA), the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI).

The scales' reliability was estimated via composite reliability, because it takes the magnitude of the items' factor loadings into account (Raykov, 1997). In order to refine the scales, in the EFAs, we excluded items with a factor loading less than 0.40, items that exhibited cross-loading and/or items that did not load on their theoretically predicted factor structure. With respect to the confirmatory factor analysis, we removed items that displayed significant worsening in terms of model fit, which was evidenced by the modification indices (Brown, 2006).

We also performed multigroup confirmatory factor analysis (MGCFA), aiming at discovering whether the instruments' final versions were equivalent for boys and girls, as well as for sixth to ninth-graders'students. We tested three invariance models (configural, metric and scalar) for each instrument (Sass, 2011). In order to assess the instrument›s invariance, we employed the Comparative Fit Index (ΔCFI) and RMSEA (ΔRMSEA), adopting the cutoff points recommended by the literature (ΔCFI < 0.01 and ΔRMSEA < 0.005; Chen, 2007; Cheung & Rensvold, 2002).

The scale validation study enjoyed the participation of a total of 1,295 sixth to ninth-grade students (647 boys and 648 girls) from nine middle schools in the state of São Paulo. Their ages varied between 10 and 16 years (M = 11.80, SD = 1.19). A total of 445 of the students in the sample were in the 6th grade; 347, 7th grade; 272, 8th grade; and 231, 9th grade.

All of the instruments we developed exhibited a satisfactory factor structure. A total of 45 items made up the Programa Semente's SES Battery. Tables 1 and 2 display each finalversion scale's principal psychometric indicators. As one can observe, all of the scales exhibit favorable indicators, evidencing the proposed models' suitability. All of the models' fit indices were higher than the cutoff points that one normally comes across in the literature: RMSEA values were under 0.08, with an upper limit less than 0.10, and the CFI and TLI values were above 0.90 (Brown, 2006).

We performed multigroup confirmatory factor analyses (MGCFAs) in order to assess measurement invariance between the sexes (girls and boys) and between different school groups (i.e. between the group of sixth and seventh graders and the group of eighth and ninth graders). Tests for differences in the CFI and RMSEA (Chen, 2007; Cheung & Rensvold, 2002) revealed that all of the instruments possessed configural, metric and scalar invariance, evidencing the fact that there were no response biases between the sexes or between students of different grades. The evidence thus shows that the Programa Semente's SES battery can be applied to such diverse groups, enabling one to make a satisfactory comparison (Sass, 2011).

In addition to the validity evidence based on the internal structure, we assessed the correlations the scales exhibited with external scales. Three international scales were adapted to the Brazilian context, serving as measurements of convergent/concurrent validity for several of the scales we developed. The three adapted scales were the following: the Private Subscale of the Revised Self-Consciousness Scales for Children (Takishima-Lacasa, Higa-McMillan, Ebesutani, Smith, & Chorpita, 2014), which rates the self-awareness levels of the subjects; the Grit Scale for children (Duckworth & Quinn, 2009), which rates their perseverance in relation to pursuing goals; and the Basic Empathy Scale (BES; Jolliffe & Farrington, 2006), which gauges emotional and cognitive empathy levels.

The Programa Semente's Self-Awareness Scale exhibited a strong correlation with the Private Subscale of the Revised Self-Consciousness Scales for Children (r = 0.745, p < .001); the Programa Semente's Perseverance Scale exhibited a strong correlation with the Grit Scale (r = 0.830, p < .001); and the Empathy Scale of the Programa Semente exhibited significant correlations with the Basic Empathy Scale: The Programa Semente Empathy Scale's (Social Empathy factor) exhibited a correlation of r = 0.55 (p < .001) with the Cognitive Empathy factor of the Basic Empathy Scale and a correlation of r = 0.275 (p < .001) with the Emotional Empathy factor of the Basic Empathy Scale. In turn, the Programa Semente Empathy Scale's (Cognitive/Emotional factor) exhibited a correlation of r = 0.697 (p < .001) with the Basic Empathy Scale's Emotional Empathy factor and a correlation of r = 0.754 (p < .001) with the Basic Empathy Scale's Cognitive Empathy factor. With respect to the other scales (Responsible Decision Making and Prosocial Behavior), no external measurements were validated. Nonetheless, upon correlating these scales with the Programa Semente Battery's other measurements, we found moderate to high correlations (Table 3), as was theoretically expected (Weissberg et al., 2015).

In light of the fact that each scale, by itself, exhibited satisfactory psychometric indicators, we aimed at examining the extent to which all of the instruments together could form a single score, known as Socioemotional Skills (SES). Accordingly, we conducted a second-order confirmatory factor analysis in which the latent variables - self-awareness, self-control, perseverance, empathy (social and cognitive/emotional), responsible decision making and prosocial behavior - were grouped into an all-inclusive factor, labeled "SES." This model's results exhibited considerably favorable fit indices, indicating that all of the scales can actually be considered part of the Socioemotional Skills construct, as CASEL (2015) suggested. Figure 1 presents the final structure of the Programa Semente's Socioemotional Skills Assessment Battery. The model's fit indices are satisfactory: CFI = 0.931; TLI = 0.927; RMSEA (90% CI) = 0.040 (0.038 0.042). As one can observe, the Programa Semente Socioemotional Skills Assessment Battery boasts favorable fit indices, validating its appropriateness for the Brazilian context.

Conclusions

Socioemotional skills (SES) has been pointed out as predictive of various positive outcomes across the course of a lifetime. The development of measurement tools that satisfactorily assess this construct is extremely important because it fosters scientific progress in the field. The Programa Semente Social-Emotional Skills Assessment Battery displays satisfactory psychometric properties, substantiating the appropriateness of its use. The development of this battery will enable one to evaluate SES levels in children and adolescents, as well as making assessment of the Programa Semente's impact possible. It is worth stressing that, up to the present time in Brazil, there has been a total lack of psychometrics tools that combine all SES content as recommended by CASEL. The Programa Semente Social-Emotional Skills Assessment Battery thus contributes to overcoming such a lack by making it possible to assess several constructs via a comparatively brief instrument (45 items). The battery can thus be of great value to various interested parties, including researchers, educators and other professionals that deal with children and teenagers' SES.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Psychological Association.         [ Links ]

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press.         [ Links ]

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3),464-504. doi: https://doi.org/10.1080/10705510701301834        [ Links ]

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2),233-255. doi: http://dx.doi.org/10.1207/S15328007SEM0902_5        [ Links ]

Collaborative for Academic, Social, and Emotional Learning. (2015). Effective Social and Emotional Learning Programs. Preschool and Elementary School Edition. Chicago, IL: CASEL Guide.         [ Links ]

Duckworth, A. L, & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (GRIT-S). Journal of Personality Assessment, 91(2),166-174. doi: https://dx.doi.org/10.1080/00223890802634290        [ Links ]

Domitrovich, C. E., Durlak, J. A., Staley, K. C., & Weissberg, R. P. (2017). Social-emotional competence: An essential factor for promoting positive adjustment and reducing risk in school children. Child Development, 88(2),408-416. doi: https://doi.org/10.1111/cdev.12739        [ Links ]

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students' social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1),405-432. doi: https://doi.org/10.1111/j.1467-8624.2010.01564.x        [ Links ]

Hawkins, J. D., Kosterman, R., Catalano, R. F., Hill, K. G., & Abbott, R. D. (2008). Effects of Social Development Intervention in Childhood 15 Years Later. Archives of Pediatrics & Adolescent Medicine, 162(12),11-33. doi: https://doi.org/10.1001/archpedi.162.12.1133        [ Links ]

Jolliffe, D., & Farrington, D. P. (2006). Development and validation of the Basic Empathy Scale. Journal of Adolescence, 29(4),589-611. doi: https://doi.org/10.1016/j.adolescence.2005.08.010        [ Links ]

Lorenzo-Seva, U., Timmerman, M. E., & Kiers, H. A. (2011). The hull method for selecting the number of common factors. Multivariate Behavioral Research, 46(2),340-364. doi: http://dx.doi.org/10.1080/00273171.2011.564527        [ Links ]

Muthén L. K., & Muthén, B. O. (2012). Mplus: Statistical analysis with latent variables. User's guide. Los Angeles, CA: Authors.         [ Links ]

Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21(2),173-184. doi: https://doi.org/10.1177/01466216970212006        [ Links ]

Robson, C., & McCartan, K. (2016). Real world research: A resource for users of social research methods in applied settings (4^rd ed.). Chichester, UK: John Wily & Sons.         [ Links ]

Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29(4),347-363. doi: https://dx.doi. org/10.1177/0734282911406661        [ Links ]

Takishima-Lacasa, J. Y., Higa-McMillan, C. K., Ebesutani, C., Smith, R. L., & Chorpita, B. F. (2014). Self-consciousness and social anxiety in youth: The Revised Self-Consciousness Scales for Children. Psychological Assessment, 26(4),1292-1306. doi: https://dx.doi.org/10.1037/a0037386        [ Links ]

Ten Berge, J. M. F., & Kiers, H. A. L. (1991). A numerical approach to the exact and the approximate minimum rank of a covariance matrix. Psychometrika, 56(2),309-315. doi: https://dx.doi. org/10.1007/BF02294464        [ Links ]

Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2),209-220. doi: https://dx.doi.org/10.1037/a0023353        [ Links ]

Weissberg, R. P., Durlak, J. A., Domitrovich, C. E., & Gullotta, T. P. (2015). Social and emotional learning: Past, present, and future. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 3-19). New York: Guilford.         [ Links ]