SciELO - Scientific Electronic Library Online

 
vol.24 número1Ensino de leitura e escrita para alunos com deficiência intelectual baseado no paradigma de equivalência de estímulosIntervenções com grupos de mães e filhos expostos à VPI: uma revisão sistemática de literatura índice de autoresíndice de assuntospesquisa de artigos
Home Pagelista alfabética de periódicos  

Serviços Personalizados

Journal

artigo

Indicadores

Compartilhar


Psicologia: teoria e prática

versão impressa ISSN 1516-3687

Psicol. teor. prat. vol.24 no.1 São Paulo jan./abr. 2022

https://doi.org/10.5935/1980-6906/ePTPCP14125.en 

CLINICAL PSYCHOLOGY

 

Portuguese adolescents' perception of the Covid-19 pandemic: gender differences and relation with psychopathological symptoms

 

Perceção dos adolescentes portugueses da pandemia Covid-19: diferenças de género e relação com sintomas psicopatológicos

 

La percepción de los adolescentes portugueses sobre la pandemia de Covid-19: diferencias de género y relaciones con los síntomas psicopatológicos

 

 

Diogo Rafael V. CarreirasI; Paula Cristina de O. de C. FreitasI; Marina Isabel V. A. da CunhaI,II

IFaculty of Psychology and Educational Sciences, Center for Research in Neuropsychology and Cognitive and Behavioral Intervention (CINEICC), University of Coimbra
IIDepartament of Psychology, Miguel Torga Institute of Higher Education

Correspondence

 

 


ABSTRACT

The Covid-19 pandemic is having a great impact on people's mental health all over the world. Adolescents have been facing several potential stressors. The aim of this study was to explore how Portuguese adolescents were perceiving the Covid-19 outbreak and the relationship between the perceived impact of the pandemic and mental health. Participants were 130 adolescents (97 girls) who completed online questionnaires about the Covid-19 pandemic and the DASS-21. Around 73% of the adolescents agreed that the Covid-19 pandemic affected their lives and 45% felt emotionally affected. Feeling uncomfortable around family during quarantine was correlated with depression (r = .39), anxiety (r = .44), and stress symptoms (r = .37), and feeling capable to deal with the pandemic was negatively correlated with psychopathological symptoms. Girls seemed to be more emotionally affected. Results revealed the negative effect of the Covid-19 pandemic on Portuguese adolescents' mental health.

Keywords: Covid-19 pandemic, adolescents, anxiety, depression, stress


RESUMO

A pandemia de Covid-19 está a ter um grande impacto na saúde mental das pessoas em todo o mundo. Os adolescentes têm enfrentado diversos potenciais stressores. O objectivo deste estudo foi explorar como os adolescentes portugueses percecionaram o surto do Covid-19 e a relação entre o impacto percebido da pandemia e a saúde mental. Os participantes foram 130 adolescentes (97 do sexo feminino) que responderam a questionários online sobre a pandemia de Covid-19 e DASS-21. Cerca de 73% dos adolescentes concordaram que a pandemia afectou as suas vidas e 45% sentiu-se emocionalmente afectado. Sentir-se desconfortável perto da família durante a quarentena mostrou-se correlacionado com sintomas de depressão (r = .39), ansiedade (r = .44) e stress (r = .37) e sentir-se capaz de lidar com a pandemia esteve negativamente correlacionado com sintomas psicopatológicos. Adolescentes do sexo feminino pareceram ficar mais emocionalmente afectadas. Esses resultados apontam para o efeito negativo da pandemia de Covid-19 na saúde mental dos adolescentes portugueses.

Palavras-chave: pandemia Covid-19, adolescentes, ansiedade, depressão, stress


RESUMEN

La pandemia de Covid-19 está teniendo un gran impacto en la salud mental de las personas. Los adolescentes se han enfrentado a varios posibles estresores. El objetivo de este estudio fue explorar cómo los adolescentes portugueses percibieron el brote de Covid-19 y la relación entre el impacto percibido de la pandemia y la salud mental. Los participantes fueron 130 adolescentes (97 mujeres) que respondieron cuestionarios online sobre la pandemia Covid-19 y DASS-21. 73% de los adolescentes estuvo de acuerdo en que la pandemia de Covid-19 afectó sus vidas y el 45% se sintió emocionalmente afectado. Sentirse incómodo con la familia durante la cuarentena se correlacionó con síntomas de depresión (r = .39), ansiedad (r = .44) y estrés (r = .37), y sentirse capaz de lidiar con la pandemia se correlacionó negativamente con los síntomas psicopatológicos. Las adolescentes parecían estar más emocionalmente afectadas. Estos resultados apuntan al efecto negativo de la pandemia Covid-19 en la salud mental de los adolescentes portugueses.

Palabras clave: pandemia Covid-19, adolescentes, ansiedad, depresión, estrés


 

 

The Covid-19 pandemic is one of the major health crises that the world has faced in the last centuries. Contact restriction measures, isolation, and economic shutdown are important changes that have the potential to affect people's mental health (Fegert et al., 2020). Outbreaks of infectious diseases are related with psychopathological symptoms and psychological distress (Bao et al., 2020; Rajkumar, 2020). In fact, some studies have already showed the psychological effects that the Covid-19 pandemic have generated in people with different ages from the general population (Wang, Pan et al., 2020; Zandifar & Badrfam, 2020). Several stressors due to the pandemic may be responsible for these effects, such as fear related to being ill or dying, fear of losing family members or friends, diminished social contact, loss of employment or homelessness (Kavoor, 2020). Anxiety symptoms seem to be the most common ones, followed by depression symptoms and stress (Wang, Pan et al., 2020). Indeed, people with greater vulnerability to mental health issues would be at a higher risk to develop a psychopathology during the Covid-19 pandemic, for example depression or anxiety disorders, causing a burden in mental health services (Kavoor, 2020).

Based on our literature review and considering the recent outbreak of the Covid-19 pandemic, there is a lack of national and international studies with adolescent populations. Nevertheless, several changes might have an impact on adolescents' mental health. The closure of schools represented an interruption of important routines for children and adolescents, and social distancing measures may result in isolation from friends and significant teachers, which means being confined to their homes (Wang, Zhang et al., 2020). Being everyday with all family members for weeks in a period of uncertainty might generate conflicts and, in abusive familiar environments, violence may increase (Lee, 2020). Home-schooling and possible postponement of exams may represent additional stress for adolescents. Leisure time activities have been limited, and adolescents were restricted from their social groups, sports clubs, and other activities. Also, a drastic decrease of physical activity was reported by adolescents, which negatively impacts physical and mental health (Xiang & Zhang, 2020). Additionally, the economic situation indirectly affects children and adolescents by rising levels of unemployment and putting great pressure on the household (Fegert et al., 2020).

Liang et al. (2020) shed light on the negative impact that the Covid-19 pandemic had on adolescents' and young adults' mental health. The results revealed that 40% of participants, two weeks after the outbreak in China, tended to develop psychological problems. Chen et al. (2020) reported that girls were at an increased risk of depression and anxiety during the pandemic and that older adolescents presented higher depression symptoms than the younger ones. Anxiety symptoms, however, did not present an association with age. A comprehensive and non-systematic review in databases was conducted by Miranda et al. (2020) to appraise the pandemic's global effect on children's and adolescents' mental health. The authors concluded that, although the number and designs of recent studies are not very robust to draw consistent conclusions, children and adolescents are a vulnerable population, and important measures might be taken to protect them from developing mental health issues given the pandemic context.

Accordingly, the aim of this study was to understand how Portuguese adolescents perceived the impact of the Covid-19 pandemic one month after the outbreak in Portugal and to explore the relationship of this perception with mental health variables. Researchers hypothesized that the current pandemic is having a significant negative effect on adolescents' emotional state.

 

Method

Procedures

Participants were part of an ongoing data collection for a longitudinal study about the evolution of borderline features in adolescents from the general population (project reference: SFRH/BD/129985/2017), sponsored by the Portuguese Foundation for Science and Technology (FCT). Accordingly, parents and adolescents had already given their informed consent to the participant in the study. Inclusion criteria were: having between 13 and 18 years old and being of Portuguese nationality. Data were collected in April 2020, a month after the outbreak in Portugal, in schools from the center and north regions of Portugal, which had already agreed to collaborate in the research. Adolescents completed the longitudinal study questionnaires, and six questions were formulated to explore the perceived impact of the Covid-19 pandemic in their lives. Students were contacted by the class director via e-mail to complete the online self-report questionnaires (LimeSurvey online source, available at https://www.limesurvey.org), since they were confined at home. Around 300 adolescents were contacted, but only approximately half of them completed the questionnaires. A large rate of non-responses were expected, considering the lower response rates of online surveys in comparison to paper-based ones (Nulty, 2008).

All procedures take into account the ethical standards of the Ministry of Education and the National Commission for Data Protection of Portugal (number: 6713/ 2018), the Ethics and Deontology Commission of the Faculty of Psychology and Educational Sciences of University of Coimbra and the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Measures

The sociodemographic questionnaire included questions about gender, age, years of education and perceived socioeconomic status (1 = "Very low"; 5 = "Very high").

A questionnaire to assess the Covid-19 pandemic impact was devised by the study authors. Six items were formulated to be rated on a 5-point Likert scale (1 = "Strongly disagree"; 5 = "Strongly agree"). Items are "The Covid-19 pandemic has affected my life", "I have been feeling limited to do my usual activities due to the Covid-19 pandemic", "The Covid-19 pandemic has affected me emotionally (angry, scared, disturbed, depressed)", "I have been feeling uncomfortable to spend so much time with my family during quarantine", "I have been feeling isolated from my friends due to the Covid-19 pandemic", and "I have managed to deal with the Covid-19 pandemic challenges".

The Depression Anxiety Stress Scale (DASS-21; Lovibond & Lovibond, 1995; Portuguese version by Pais-Ribeiro et al., 2004) is a self-report questionnaire with 21 items to assess depression, anxiety, and stress symptoms. Items are rated on a 4-point Likert scale (0 = "Did not apply to me at all"; 3 = "Applied to me very much, or most of the time") and higher scores mean higher negative affect. The original version showed good internal consistency (α = .91 for Depression, α = .84 for Anxiety, and α = .90 for Stress). The Portuguese version also showed good internal consistency (α = .85 for Depression, α = .74 for Anxiety, and α = .81 for Stress). In this study, the Cronbach's alpha was .92 for Depression, .87 for Anxiety, and .90 for Stress.

Data Analyses

Data were analyzed using IBM SPSS Statistics version 23. Normality assumption was tested though Kolmogorov-Smirnov test and skewness (sk) and kurtosis (ku) values (normality assumption assumed with Sk < 3 and Ku < 8) (Kline, 2011). Outliers were explored with the boxplot diagram.

Descriptive statistics were conducted, and group differences were tested through student's t-tests for independent samples. Effect sizes were interpreted according to Cohen (1988), considering d values between .20 and .49 small, between .50 and .79 medium, and above .80 large. Pearson correlation coefficients were used to explore the relationship between variables. Correlation coefficients between .10 and .39 were considered weak; between .40 and .69 considered moderate; and above .70 considered strong (Dancey & Reidy, 2017).

Considering our sample of 130 cases (97 females and 33 males), the G*Power software (version 3.1; alpha of .05) estimated a power of .94 for correlations and a power of .69 for student's t-tests (independent samples). Statistical significance was considered for p values under .05.

 

Results

Participants

Participants were 130 adolescents from the general population, of which 97 (74.60%) were girls and 33 (25.40%) were boys. The mean age of participants was 16.40 years old (SD = 0.95; range = 15-18) and they completed an average of 10.98 years in school (SD = 0.72). The majority of participants (82%) reported a medium socioeconomic status, while 12% reported a high, 1% a very high and 5% a low status. Non-significant gender differences were found for age (t(128) = 1.10, p = .273), years of schooling (t(128) = 0.98, p = .327), and socioeconomic status (t(128) = 0.44, p = .663).

Preliminary Analyses

Kolmogorov-Smirnoff normality test was significant, but no severe violations of normality were found considering skewness and kurtosis values (Sk < 3 and Ku < 8-10) (Kline, 2011). Outliers were kept in order to maintain the natural variance and representation in the population. According to the preliminary results, parametric tests were conducted.

Perceived Impact of the Covid-19 Pandemic

In Table 1 are presented means, standard deviations, and frequencies for all the six items about the impact of the Covid-19 pandemic. Around 73% of adolescents agreed or strongly agreed that the Covid-19 pandemic affected their lives and around 45% agreed or strongly agreed that the Covid-19 pandemic affected them emotionally. In addition, 60% of adolescents considered that they were capable to manage the challenges raised due to the pandemic.

 

 

Demographic variables and the Covid-19 Pandemic Perceived Impact

Despite the disparity between boys and girls in the current sample, gender differences were explored regarding the perception of the impact of the new coronavirus pandemic. A student's t-test (t(128) = 3.82, p < .001) showed that girls presented a higher score (M = 3.2; SD = 1.3) in item 3 ("The Covid-19 pandemic have affected me emotionally") compared to boys (M = 2.2; SD = 1.5), with a medium effect size (d = 0.75). Non-significant differences were found between gender groups in the remaining items. Additionally, non-significant correlations were found between the six Covid-19-related items and age, years of education, and socioeconomic status.

Gender Differences in Depression, Anxiety, and Stress Symptoms During the Pandemic

Means and standard deviations for the total sample and gender groups are presented in Table 2. Results showed that girls exhibited higher levels of anxiety in comparison to boys (t(128) = 1.95, p = .05), with a small effect size (d = 0.42). A similar result was obtain for stress, with girls presenting higher scores than boys (t(128) = 2.78, p = .01), with a medium effect size (d = 0.55).

 

 

Relationship Between the Covid-19 Pandemic Perceived Impact and Negative Emotional States

Pearson correlations were conducted in order to explore the association between the perceived impact of Covid-19 and depression, anxiety, and stress symptoms (Table 3). Feeling emotionally affected was moderately correlated with stress (r = .45, p < .001) and depression (r = .43, p < .001) and weakly correlated with anxiety (r = .39, p < .001). Feeling uncomfortable to spend time with family during quarantine was moderately correlated with anxiety (r = .44, p < .001) and weakly correlated with depression (r = .39, p < .001) and stress (r = .37, p < .001). Isolation from friends was weakly correlated with depression (r = .22, p < .05). Feeling able to deal with the pandemic challenges was weakly and negatively correlated with depression (r = -.29, p < .001), anxiety (r = -.32, p < .001), and stress (r = -.30, p < .001).

 

 

Discussion

Considering the worldwide Covid-19 pandemic, the aim of the current study was to understand how Portuguese adolescents perceived the impact of the pandemic and to explore the association with mental health variables, a month after the outbreak. Previous studies have reported the negative impact of the Covid-19 pandemic, showing an association with depression, anxiety, and stress symptoms (Wang, Zhang et al., 2020; Xiang & Zhang, 2020); however there is a lack of studies with adolescent samples on this topic.

In the current study, the Covid-19 pandemic perceived impact appeared to have no relationship with age, which means that experienced difficulties about the pandemic and being able to manage them are not related to being younger or older. However, girls reported feeling more emotionally affected by the pandemic context than boys. Moreover, results showed that female adolescents presented higher anxiety and stress levels during the pandemic. Previous literature has discussed that female adolescents are more vulnerable to develop psychological distress, anxiety, and depression not only in general (Hayward & Sanborn, 2002; Van Droogenbroeck et al., 2018), but also in the pandemic context (Chen et al., 2020; Miranda et al., 2020). Moreover, no relation was found between the perceived impact of the Covid-19 pandemic and years of education and socioeconomic status.

Around 73% of adolescents agreed that their lives were affected, and around 43% reported being emotionally affected, including feeling angry, scared, disturbed or depressed, since the outbreak in Portugal. Relational, cognitive, scholar, and economic challenges related to the Covid-19 pandemic are associated with negative emotional affect, which is a defensive response to deal with this context. Fegert et al. (2020) affirmed that changes related to pandemic occurrences have the potential to psychologically affect people, and it was expected that the pandemic context would have a negative impact on adolescents' mental health. In general, these results align with previous studies about the negative impact of the Covid-19 pandemic (Dubey et al., 2020; Wang, Zhang et al., 2020; Xiang & Zhang, 2020).

Results showed that depression, anxiety, and stress symptoms were positively associated with feeling emotionally affected by the pandemic and with feeling uncomfortable for spending time with family during the quarantine. This last matter is concerning and might be related to more frequent familiar conflicts in a period of uncertainty and additional stressors (unemployment, economic shutdown, diminished privacy, emotional vulnerability), with violence potentially increasing in abusive environments (Lee, 2020). Depression symptoms were also weakly associated with being isolated from friends, highlighting the importance of social interactions with same-age peers (Rohrbeck & Gray, 2014). The weak association might be related to using social media to contact friends by messages and/or videocalls, probably decreasing the feeling of isolation. Moreover, our sample might overly include adolescents with more well-heeled backgrounds, with access to several resources that might protect them from negative symptoms. We must also consider that adolescents who already presented psychopathological symptoms before the pandemic might be more prone to experience negative affect, although we cannot directly answer this question considering the cross-sectional data.

On the other side, around 60% of adolescents reported feeling able to deal with the current situation and, in our view, this might be an indicative of positive future outcomes. However, mental health-related policies are encouraged to prevent the development of some clinical psychological symptoms that might persist. Adolescents with vulnerability to developing mental health problems are important risk populations to be watched over. Also, girls seemed to be at a greater risk of feeling emotionally disturbed during this period and parents, teachers and doctors are encouraged to be alert and refer them to clinical assessment and treatment when necessary.

The current study was one of the first Portuguese studies shedding light on the emotional and psychological impact of the Covid-19 pandemic on adolescents. Nevertheless, it presents some limitations. Participants were a convenience sample, collected online (youth without internet access at home are not represented in this study) and with an uneven proportion of boys and girls, which might not reflect a representative sample of Portuguese adolescents. Moreover, the cross-sectional data do not allow firmly inferring causality between variables, for example we cannot affirm whether psychopathological symptoms were caused by the pandemic or triggered by previous anxiety or stress symptoms. Although we know that a large number of adolescents reported feeling able to deal with the pandemic context, it would be important to further explore which coping strategies were employed and the potential effect of risk factors on mental health, such as violence exposure or adverse familial environment. Additionally, future studies could also investigate the effect of Covid-19 health literacy on mental health and emotional states, as well as the impact of the subsequently de-confinement measures. Overall, we must be prudent regarding our conclusions, and longitudinal and prospective studies are crucial to understand the evolution of emotional states during the Covid-19 pandemic in representative adolescent samples.

 

References

Bao, Y., Sun, Y., Meng, S., Shi, J., & Lu, L. (2020). 2019-nCoV epidemic: Address mental health care to empower society. The Lancet, 395,37-38. https://doi.org/10.1016/S0140-6736(20)30309-3        [ Links ]

Chen, F., Dan, Z., Liu, J., Yi, G., Guan, Z., & Lou, D. (2020). Depression and anxiety among adolescents during COVID-19: A cross-sectional study. Brain, Behavior, and Immunity, 88,36-38. https://doi.org/10.1016/j.bbi.2020.05.061        [ Links ]

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates Publishers.         [ Links ]

Dancey, C., & Reidy, J. (2017). Statistics without maths for psychology (7th ed.). Pearson Education.         [ Links ]

Dubey, S., Biswas, P., Ghosh, R., Chatterjee, S., Dubey, M. J., Chatterjee, S., Lahiri, D., & Lavie, C. J. (2020). Psychosocial impact of COVID-19. Diabetes and Metabolic Syndrome: Clinical Research and Reviews, 14(5),779-788. https://doi.org/10.1016/j.dsx.2020.05.035        [ Links ]

Fegert, J. M., Vitiello, B., Plener, P. L., & Clemens, V. (2020). Challenges and burden of the Coronavirus 2019 (COVID-19) pandemic for child and adolescent mental health: A narrative review to highlight clinical and research needs in the acute phase and the long return to normality. Child and Adolescent Psychiatry and Mental Health, 14(1),1-11. https://doi.org/10.1186/s13034-020-00329-3        [ Links ]

Hayward, C., & Sanborn, K. (2002). Puberty and the emergence of gender differences in psychopathology. Journal of Adolescent Health, 30(4),49-58. https://doi.org/10.1016/S1054-139X(02)00336-1        [ Links ]

Kavoor, A. R. (2020). COVID-19 in People with Mental Illness: Challenges and Vulnerabilities. Asian Journal of Psychiatry, 51, 102051. https://doi.org/10.1016/j.ajp.2020.102051        [ Links ]

Kline, R. (2011). Principles and practice of structural equation modelling (3rd ed.). The Guilford Press.         [ Links ]

Lee, J. (2020). Mental health effects of school closures during COVID-19. The Lancet Child and Adolescent Health, 4, 421. https://doi.org/10.1016/S2352-4642(20)30109-7        [ Links ]

Liang, L., Ren, H., Cao, R., Hu, Y., Qin, Z., Li, C., & Mei, S. (2020). The Effect of COVID-19 on Youth Mental Health. Psychiatric Quarterly, 91,841-852. https://doi.org/10.1007/s11126-020-09744-3        [ Links ]

Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33(3),335-343. https://doi.org/https://doi.org/10.1016/0005-7967(94)00075-U        [ Links ]

Miranda, D. M., Athanasio, B. da S., Oliveira, A. C. S., & Simoes-e-Silva, A. C. (2020). How is COVID-19 pandemic impacting mental health of children and adolescents? International Journal of Disaster Risk Reduction, 51, 101845. https://doi.org/10.1016/j.ijdrr.2020.101845        [ Links ]

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment and Evaluation in Higher Education, 33(3),301-314. https://doi.org/10.1080/02602930701293231        [ Links ]

Pais-Ribeiro, J. L., Honrado, A., & Leal, I. (2004). Contribuição para o estudo da adaptação portuguesa das Escalas de Ansiedade, Depressão e Stress (EADS) de 21 itens de Lovibond e Lovibond. Psicologia, Saúde & Doenças, 5(2),229-239. https://www.redalyc.org/articulo.oa?id=36250207        [ Links ]

Rajkumar, R. P. (2020). COVID-19 and mental health: A review of the existing literature. Asian Journal of Psychiatry, 52, 102066. https://doi.org/10.1016/j.ajp.2020.102066        [ Links ]

Rohrbeck, C. A., & Gray, L. S. (2014). Peer relationships: promoting positive peer relationships during childhood. In T. P. Gullotta & M. Bloom (Eds.), Encyclopedia of Primary Prevention and Health Promotion (pp. 828-836). Springer. https://doi.org/10.1007/978-1-4614-5999-6_137        [ Links ]

Van Droogenbroeck, F., Spruyt, B., & Keppens, G. (2018). Gender differences in mental health problems among adolescents and the role of social support: Results from the Belgian health interview surveys 2008 and 2013. BMC Psychiatry, 18(6),1-9. https://doi.org/10.1186/s12888-018-1591-4        [ Links ]

Wang, C., Pan, R., Wan, X., Tan, Y., Xu, L., Ho, C. S., & Ho, R. C. (2020). Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. International Journal of Environmental Research and Public Health, 17(5),1729. https://doi.org/10.3390/ijerph17051729        [ Links ]

Wang, G., Zhang, Y., Zhao, J., Zhang, J., & Jiang, F. (2020). Mitigate the effects of home confinement on children during the COVID-19 outbreak. The Lancet, 395(10228),945-947. https://doi.org/10.1016/S0140-6736(20)30547-X        [ Links ]

Xiang, M., & Zhang, Z. (2020). Impact of COVID-19 pandemic on children and adolescents' lifestyle behavior larger than expected. Progress in Cardiovascular Diseases, 63(4),531-532. https://doi.org/https://doi.org/10.1016/j.pcad.2020.04.013        [ Links ]

Zandifar, A., & Badrfam, R. (2020). Iranian mental health during the COVID-19 epidemic. Asian Journal of Psychiatry, 51, 101990. https://doi.org/10.1016/j.ajp.2020.101990        [ Links ]

 

 

Correspondence:
Diogo Rafael Veiga Carreiras
Center for Research in Neuropsychology and Cognitive and Behavioral Intervention (CINEICC), University of Coimbra
Rua do Colégio Novo, s/n 3000-115
Coimbra, Portugal
E-mail: diogocarreiras1@gmail.com

Received: August 26th, 2020
Accepted: May 14th, 2021

Creative Commons License Todo o conteúdo deste periódico, exceto onde está identificado, está licenciado sob uma Licença Creative Commons