THEMATIC SECTION - MOTHERHOOD, CHILDHOOD DEVELOPMENTAL CHANGES, AND SUPPORT NETWORKS

 

Children and adolescents’ psychotherapy: Evidences of validity of the Brazilian Portuguese Youth Outcome Questionnaire 2.01

 

Psicoterapia com crianças e adolescentes: Evidências de validade do Y-OQ 2.01 em português brasileiro

 

Psicoterapia con niños y adolescentes: Evidencias de validez para el Y-OQ 2.01 en portugués brasileño

 

 

Bruna Holst^I; Carolina Saraiva de Macedo Lisboa^II; Wagner de Lara Machado^III

^IPsicóloga e Psicoterapeuta. Mestre e Doutora em Psicologia Clínica pela Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS). Docente e supervisora do Curso de Formação em Psicoterapia Psicanalítica do ESIPP (Estudos Integrados de Psicoterapia Psicanalítica), Porto Alegre, RS, Brasil. bruholst@gmail.com
^IIPsicóloga e Psicoterapeuta. Mestre e Doutora em Psicologia pela Universidade Federal do Rio Grande do Sul (UFRGS). Professora da Graduação e da Pós-Graduação em Psicologia da Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS) e Coordenadora do Grupo de Pesquisa Relações Interpessoais e Violência (RIVI/PUCRS), Porto Alegre, RS, Brasil. lisboacaro@gmail.com
^IIIPsicólogo. Mestre e Doutor em Psicologia pela Universidade Federal do Rio Grande do Sul (UFRGS). Professor da Graduação e da Pós-Graduação em Psicologia da Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS) e Coordenador do Grupo de Pesquisa Avaliação em Bem-estar e Saúde Mental (ABES/PUCRS), Porto Alegre, RS, Brasil. wagner.machado@pucrs.br

 

 

---

ABSTRACT

Early and effective interventions in the area of mental health of children and adolescents are of unquestionable importance. Thus, there is a great need to develop reliable tools to monitor the success or failure of these interventions. The goal of this study is to investigate the psychometric properties of the Brazilian Portuguese Youth Outcome Questionnaire 2.01, an instrument that seeks to evaluate the process and outcome of psychotherapy with children and adolescents from the perspective of the parents or legal guardians. After a cross-cultural adaptation, confirmatory and exploratory factor analysis and a network analysis were performed. The sample consisted of 341 parents or legal guardians of children (172) and adolescents (163) from 1 to 19 years old; 51% of the youths were male. It was observed that confirmatory factorial analysis based on the original model did not present good fit indexes, but the exploratory factorial analysis performed yielded a two-factor model which showed good fit indexes. Evaluating the expected influence among the nodes that composed the network analysis, items were identified that seem to activate symptom networks. The conclusion was that there is evidence of validity for the Brazilian Portuguese Youth Outcome Questionnaire 2.01, which can be used by Brazilian psychotherapists to plan treatments and monitor psychotherapeutic results.

Keywords: validation studies; psychometrics; child; adolescent; efficacy-effectiveness of interventions.

---

RESUMO

É inquestionável a importância de intervenções precoces e efetivas na área da saúde mental com crianças e adolescentes, o que torna fundamental o desenvolvimento de ferramentas confiáveis para monitorar o êxito ou fracasso destas intervenções. O objetivo deste estudo é investigar as propriedades psicométricas da versão em português brasileiro do Youth Outcome Questionnaire 2.01, instrumento para avaliação de processo e resultado de psicoterapia com crianças e adolescentes a partir da perspectiva dos pais ou cuidadores. Após uma adaptação transcultural, realizaram-se análises fatoriais confirmatória e exploratória e também uma análise de rede. A amostra foi composta por 341 pais ou responsáveis legais de crianças (172) e adolescentes (163) de 1 a 19 anos de idade; 51% dos jovens eram do sexo masculino. Foi observado que a análise fatorial confirmatória com base no modelo original não apresentou bons índices de ajuste, mas a análise fatorial exploratória apontou um modelo de dois fatores com bons índices de ajuste. Avaliando a Expected Influence entre os nós que compuseram a análise de rede, identificaram-se itens que parecem ativar redes de sintomas. Concluiu-se que existem evidências de validade para o Y-OQ 2.01 em português brasileiro, que poderá ser usado por psicoterapeutas brasileiros para planejar tratamentos e monitorar resultados psicoterapêuticos.

Palavras-chave: estudos de validação; psicometria; criança; adolescente; eficácia-efetividade de intervenções.

---

RESUMEN

Es incuestionable la importancia de intervenciones tempranas y efectivas en la salud mental de niños y adolescentes, lo que hace fundamental el desarrollo de herramientas confiables para monitorear el éxito o fracaso de intervenciones. Lo objetivo de este estudio es investigar las propiedades psicométricas del Youth Outcome Questionnaire 2.01 en portugués brasileño, instrumento para la evaluación del proceso y los resultados de la psicoterapia con niños y adolescentes desde la perspectiva de los padres o cuidadores. Después de una adaptación transcultural, se realizaron análisis factoriales confirmatorias y exploratorias, y también un análisis de red. La muestra fue compuesta por 341 padres o cuidadores de niños (172) y adolescentes (163) de 1 a 19 años de edad; 51% de los jóvenes eran chicos. Resultó que el análisis factorial confirmatoria basado en el modelo original no presentó buenos índices de ajuste, pero el análisis factorial exploratoria apuntó un modelo de dos factores con buenos índices de ajuste. Evaluando la Expected Influence entre los nodos que compusieron el análisis de red, se identificaron ítems que parecen activar redes de síntomas. La conclusión es que existen evidencias de validez del Y-OQ 2.01 en portugués brasileño, que puede ser utilizado por psicoterapeutas para planificar tratamientos y evaluar los resultados psicoterapéuticos.

Palabras clave: estudios de validación; psicometría; niño; adolescente; eficacia-efectividad de intervenciones.

---

 

 

Introduction

Human, financial and scientific resources in the area of children and adolescents' mental health are scarce in emerging countries, where there is a gap between needs and provision of mental treatments (World Health Organization and Calouste Gulbenkian Foundation, 2014). Worldwide efforts have been made among academics and researchers to identify opportunities for improvements in youth mental health care, particularly in underdeveloped countries (Belfer, 2008; Klasen & Crombag, 2013). Fostering youth mental health is necessary to reduce the occurrence of early mental disorders, which have the potential to reduce educational and professional performance in adult life, aggravate legal problems, induce isolation or suicide, and increase physical frailty and other health problems (Costello & Maughan, 2015). Therefore, early and effective psychological interventions are extremely relevant for general human development.

Thus, the evaluation and measurement of these interventions are critical needs for researchers, psychologists, psychiatrists and the administrators of health systems. Outcome measures are valuable tools to monitor patient progress, which is essential in determining whether the intervention is working for that particular patient at that particular phase of the treatment (Burlingame et al., 2004). In addition, outcome measures are an important tool of evidence-based psychotherapy and are essential so that treatments can be adjusted according to the patient's response (American Psychological Association, 2006). In a study with 539 therapists of children and adolescents, regarding outcome assessment, 23% reported using standardized outcome measures with their adolescent patients (Bickman et al., 2000). The five main types of information that clinicians considered relevant to verify or identify the progress of psychotherapy and that emerged from these evaluations are the possibilities to investigate any history of abuse, evaluate past and present stressors, understand the patient's family functioning, observe the relationship quality between patients and their parents and evaluate therapeutic alliance strength (Bickman et al., 2000). However, the American Psychological Association (2008) reports a lack of reliable or valid measures to determine the efficacy and effectiveness of commonly applied therapeutic practices and that strengthening the evidence-based psychotherapy for children and adolescents requires the identification, measurement, and monitoring of psychotherapeutic processes and outcomes.

In Brazil, a systematic review conducted by Holst et al. (2018) sought to investigate research designs and instruments used in Brazilian studies describing psychotherapeutic interventions with children and adolescents. Five databases were surveyed, with no date limitation. The results showed that only 28 studies used some sort of instrument or assessment to measure results, which suggests the lack of research into mental health in this population. The most cited instrument to assess psychotherapy outcome was the Child Behavior Checklist (CBCL). Holst et al. (2018) also reported a lack of instrument descriptions by Brazilian studies in this field, which is essential for the evaluation of research methodological quality.

This result corroborates the study by Guimarães and Yoshida (2014) in Brazil, which sought to understand which factors are considered markers of progress by children's psychotherapists of different theoretical approaches. They pointed to the predominance of subjective evaluations of the process, mainly through the perception of the therapist, and the observation of children's play as the most common monitoring method, regardless of the psychotherapeutic approach. Although important, the therapist's subjective evaluation is not the only method to monitor psychotherapy and can be more easily biased, as it relies only on the therapist's perception.

Youth Outcome Questionnaire 2.01

Although the importance of psychotherapy for children and adolescents is evident, the process by which therapeutic change happens with this age group is still poorly studied (Kazdin, 2000). The study by McClendon et al. (2011) sought to examine the sensitivity to capture change process by comparing three commonly used instruments to assess child and adolescent treatment: the CBCL/6-18, the Behavior Assessment System for Children-2 (BASC-2) and the Youth Outcome Questionnaire 2.01 (Y-OQ 2.01). The results showed that, regarding measures that assess parents' perceptions, the Y-OQ 2.01 was the most sensitive instrument concerning symptom reduction, and the BASC-2 and CBCL were not significantly different from each other. This result can be traced back to the development methods of these scales: while the Y-OQ 2.01 was devised by clinicians, researchers, and health service administrators to assess psychotherapy change on a weekly basis, the CBCL and the BASC-2 were initially created to categorize and identify psychosocial problems and were then adapted as outcome measures, applied monthly. In addition, the Y-OQ 2.01 offers five variations of response, which allow greater sensitivity than the CBCL, with three variations on a Likert scale, and BASC-2, with four variations on a Likert scale (McClendon et al., 2011).

The original version of the Youth Outcome Questionnaire 2.01 provided adequate reliability indexes evidenced by Cronbach's alphas; the internal consistency analysis of the instrument in a nonclinical (N=1091) and clinical (N=2732) sample obtained a total score of 0.97 among the samples. Test-retest reliability investigation was also performed two and four weeks after a first application; interval scores produced a reliability coefficient of 0.83, and all correlations between the subscales on test-retest were significant (p<0.01), ranging from 0.56 to 0.82. Criterion validity of the Y-OQ 2.01 was also investigated in two studies comparing the total score and each Y-OQ 2.01 subscale with scores from the Child Behavior Checklist (Achenbach & Rescorla, 2001) and the Connor's Parent Rating Scale (Connors, 1990), two measures for behavioral problems of children and adolescents. The results indicate a strong relationship between the Y-OQ 2.01 and these measures. The construct validity of the Y-OQ 2.01 was verified in studies with nonclinical and clinical samples (outpatient and inpatient); the instrument was successful in discriminating symptom levels among different populations (Burlingame et al., 2005).

The Y-OQ 2.01 evaluates the behavior and psychological suffering of youth in a total score, composed of six subscales: Intrapersonal Distress, Somatic Distress, Interpersonal Relations, Social Problems, Behavioral Dysfunction, and Critical Items. The Intrapersonal Distress subscale evaluates the emotional distress of children and adolescents through manifestations of anxiety, depression, fear, hopelessness or self-harm. The Somatic Distress subscale measures somatic symptoms such as headaches and joint pain, dizziness, nausea and muscle weakness. The items of the Interpersonal Relations subscale evaluate the relationship between the child/adolescent and parents, other adults and peers, patterns of interaction with friends, and levels of cooperation or aggressiveness. The Social Problems subscale evaluates behavioral problems, such as aggressive or delinquent behavior. Although aggressiveness is present in the Interpersonal Relations subscale, the Social Problems items assess more serious aspects of aggression, involving the breaking of social norms, such as school absenteeism, promiscuous sexual behavior, running away from home and substance abuse. The Behavioral Dysfunction subscale evaluates the ability of the child/adolescent to organize and complete tasks, focus, and assess moments of inattention, hyperactivity and impulsivity. Finally, the Critical Items subscale evaluates problems commonly seen in inpatients, such as paranoia, hallucination, delusions, suicide and eating disorders (Burlingame et al., 2005).

To qualify psychological interventions, the objective of this study is to adapt and investigate evidence of the validity and psychometric properties of the Youth Outcome Questionnaire 2.01 in Brazilian Portuguese. This instrument seeks to evaluate the process and result of psychotherapy with children and adolescents from the perspective of parents and legal guardians. This instrument was chosen due to the evidence of its reliability and validity in its original English version - both in its full form, answered by parents or legal guardians, which is the focus of this study (Burlingame et al., 2004) - as well as for its short version (Dunn et al., 2005) and patient's self-report (Ridge et al., 2009).

 

Method

Sample

A total of 341 parents or legal guardians of children (N=172) and adolescents (N=163) from 1 to 19 years old (M=9.44; SD=4.20) were surveyed. We considered adolescence as starting at ten years old, following the World Health Organization (1995). Of the 341 participants, 92.5% reported being the main caregiver of the youth. In addition to mothers (82.4%) and fathers (15.5%), seven legal guardians also completed the questionnaire (2.1%). Although the questionnaire does not reveal the kinship of guardians and minors, 5 legal guardians presented themselves as the main caregiver. The sample size consisted of 5.3 participants per item of the Y-OQ 2.01 and was based on parameters that considered larger samples as more likely to produce accurate solutions for factorial analyses and a minimum of 5 subjects per item to be considered adequate (Costello & Osborne, 2005; Pasquali, 2010). The sample presented an adequate distribution of gender, age group and between youth with and without a history of psychotherapy. The social and demographic information of the children/adolescents and their caregivers are described in Table 1, and the data in parentheses (n) refers to the number of participants who answered each item of the questionnaire.

 

 

Measures

Two questionnaires were administered to participants. The first was a socio-demographic questionnaire developed for this study. The second was the Y-OQ 2.01, a 64-item questionnaire answered on a Likert scale of 1 to 5 by parents or legal guardians of children or adolescents throughout the evolution of psychotherapeutic treatment. There is also a self-report version of the instrument, the Y-OQ Self-Report (Ridge et al., 2009). OQ Measures granted a license and authorization for the adaptation and investigation of the psychometric properties of the Y-OQ 2.01 in Brazilian Portuguese.

Procedures

Two local psychotherapy schools participated in this study and agreed to the terms of the research. Participants completed the questionnaires at the time of the search for care for their children, before the first interview with the therapist. Participants were also recruited through the snowball nonprobabilistic sampling methods (Breakwell et al., 2010) and by completing online questionnaires published in social media. Individually collected data took approximately 25 minutes to be completed. In accordance with the Guidelines and Regulatory Norms involving research with human beings described in Resolution nº 510 of the Brazilian Health Council (CNS, 2016) and Resolution nº 016/2000 of the Brazilian Council of Psychology (CFP, 2000), all participants agreed to participate in the study and signed the Free and Informed Consent Form. In accordance with both Resolutions, the anonymity and lack of any kind of harm to the participants were guaranteed. In addition, the study was approved by the University Ethics Committee (Protocol nº 2.383.625 / Ethical Appreciation Certificate - CAAE nº 76229317400005336).

Translation, transcultural adaptation and content validity evidence

The procedure adopted for the Y-OQ 2.01 cross-cultural translation and adaptation followed the standardized process recommended by OQ Measures and described by Wild et al. (2005). The first step consisted of two independent translations into Brazilian Portuguese from the original instrument, performed by Portuguese native speakers fluent in English, with previous experience in translating instruments and working in the clinical psychology area. A synthesis of the two translations was carried out by a third independent translator and was sent to a committee of four experts in clinical psychology and psychotherapy to investigate evidence of content validity. Afterwards, a final semantic adjustment was made based on the suggestions of the expert committee. Then, backtranslation from Portuguese to English was completed and sent to the original authors for approval. Finally, operational equivalence was made for the Brazilian context, adjusting the questionnaire's format, the manner of giving instructions, and its correct administration based on the original instrument.

We sought to evaluate evidence of the content validity of the Y-OQ 2.01, such as clarity, relevance and representativeness of the instrument's items (Cassepp-Borges et al., 2010). For this purpose, we used the coefficient of content validity (CVC), proposed by Hernández-Nieto (2002) and cited by Cassepp-Borges et al. (2010), which assesses the strengths and weaknesses of instruments based on the opinion of at least three and at most five specialists. A committee of four experts - masters in psychology and specialists in the relevant clinical area - received the Portuguese version of the Y-OQ 2.01, along with a Likert-type questionnaire ranging from 1 (very unsatisfactory) to 5 (very satisfactory), with three categories for evaluation of the items: language clarity, practicality and theoretical relevance.

The CVC is calculated based on the average scores of each item attributed by the specialists, which is then divided by the maximum value that an item can reach. The measurement error (Pe) is then calculated to discount possible biases, which will subsequently be subtracted from the CVC value. Finally, the total CVC score of the scale uses the average CVCs of all questionnaire items, subtracted from the average errors of all items. The items that reach a CVCc>0.8 are considered adequate. The total instrument scored 0.97. For each category assessed, Y-OQ 2.01 scored 0.97 in language clarity, 0.99 in practical relevance and 0.98 in theoretical relevance. Only one of the 64 items scored below 0.8 in language clarity and was therefore rephrased (item 17, revised from Seems tense, frequently easily frightened to Seems tense or easily frightened frequently, after recommendations by the experts).

Investigation of psychometric properties

Several techniques were used to evaluate the psychometric properties of the Y-OQ 2.01. Using the SPSS statistical program, version 19.0 (IBM Corp., 2010), the frequencies, means, standard deviations and percentiles of the social and demographic data were calculated. Correlations between the Y-OQ 2.01 six subscales were also examined. Additionally, Y-OQ 2.01 construct validity evidence was investigated, defined as characteristics of an instrument with respect to the measurement of a property or quality that has not yet been operationally defined (Pasquali, 2010). For this purpose, a confirmatory factorial analysis (CFA) was performed based on its original factorial structure, since previous studies suggest a strong single factor (total scale score) underlying the six factors that constitute the instrument's subscales (Burlingame et al., 2005). Subsequently, an exploratory factorial analysis (EFA) with oblique rotation was performed (Costello & Osborne, 2005). To investigate the number of factors to be retained, the parallel analysis technique was used (Horn, 1965; Timmerman & Lorenzo-Seva, 2011). The Kaiser-Meyer-Olkin (KMO=0.9) and Bartlett sphericity tests (χ² [2016] =52083.63; p<0.001) assessed the adequacy of the sample for these analyses. As a criterion of good EFA indexes, factor loadings greater than 0.3 were considered in one or both factors (Costello & Osborne, 2005). Finally, CFA and EFA adjustment indexes were compared, and for the proposed model, Cronbach's alphas were calculated in order to verify the Y-OQ 2.01 internal consistency.

Afterwards, a network analysis applied to mental illness (Borsboom & Cramer, 2013; Schmittmann et al., 2011) was performed to investigate the pattern of relationship between the Y-OQ 2.01 items. The network is built in two stages. First, partial correlations between the network nodes (items) are estimated, controlling the effect of other variables or sets of variables. Subsequently, those conditionate relations are graphically represented, allowing the observation of the relationship pattern between the network's elements and the nodes. These nodes represent the symptoms described in the instrument's items, and the lines represent the relationships between the nodes. The premise of this analysis is that symptoms are not passive indicators of a disorder but active psychological variables capable of mutually influencing and contributing to the general network's activation (Schmittmann et al., 2011). From the network analysis, we focused on evaluating the expected influence (EI) metric (Robinaugh et al., 2016), proposed by the same authors. Instead of quantifying the influence of the node within the network (centrality measure), the EI measure is used to investigate the nature of this influence and identify its role in the activation, deactivation or persistence of the network. This analysis investigates whether lines connecting nodes are positive or negative, which provides information about the effect that line deactivations can have on the network. In this study, each node represents an item of the Y-OQ 2.01. Thus, it is observed how each symptom associates with the others in terms of both first and second degree. This analysis was performed with the qgraph package in R (Epskamp et al., 2012).

 

Results

First, a confirmatory factorial analysis (CFA) was performed based on the Y-OQ 2.01 original structure; it consisted of a single factor (total scale score) underlying six factors - which correspond to the six subscales (Burlingame et al., 2005). We used the robust method to correct the chi-square values (χ²) due to ordinal measurement level and nonnormality of the data distribution, thus presenting the value of Satorra-Bentler χ² (s-bχ²; Satorra & Bentler, 2001). In addition to the s-bχ² test and the respective degrees of freedom (df), the following CFA adjustment indexes were considered: comparative fit index (CFI), Tucker-Lewis index (TLI), standardized root mean residual (SRMR) and the root mean square error of approximation (RMSEA) and its respective confidence interval. For these indexes, the following adaptation parameters were adopted (Hu & Bentler, 1999; Schreiber et al., 2006): s-bχ²/df less than 5 (preferably less than 3), SRMR less than 0.08, RMSEA less than 0.06 (with an upper confidence interval not greater than 1.00), and CFI and TLI values greater than 0.90 - preferably greater than 0.95.

The CFA adjustment indexes were insufficient to validate the original Y-OQ 2.01 model (s-bχ²/df=1.79, CFI=0.86, TLI=0.86, SRMR=0.109, RMSEA=0.055 [0.052-0.058]), contrary to the hypothesis that this factorial structure would work for the scale's Portuguese version. In addition, considering the original model, correlations were calculated between the six subscales based on participants who completed all Y-OQ 2.01 items (N=264). Pearson and Spearman correlations were calculated for this sample, but since the results and conclusions of the significance tests were similar, it was decided to present the Spearman correlations. As shown in Table 2, strong correlations (p<0.001) were found between the Interpersonal Relations subscale and the Intrapersonal Distress and Behavioral Dysfunction subscales. Moderate correlations predominated among the subscales (all p<0.001), with the exception of a weak correlation between Somatic Distress and Social Problems.

An exploratory factorial analysis (EFA), using polychoric correlations and the minimum rank estimation method with oblique (direct oblimin) rotation, was later performed to investigate the elements of the factor load matrix in a free setting. The parallel analysis suggests that only two factors exhibited Eigenvalues greater than the simulated (Monte Carlo) and sample permutated values. The exploratory model had higher fit indexes than the confirmatory model and was structured with two major factors (TLI=0.95, RMSEA=0.08). Therefore, this model was used for the internal consistency and network analyses. The EFA result, which pointed to a two-factor model, agrees with other studies that investigated interrelationships between psychiatric disorders according to diagnostic manuals such as the DSM and that found a structure of two latent factors interpreted as internalizing and externalizing symptoms (Kotov et al., 2011; Krueger et al., 1998). Individuals in the first group tend to express pain internally, leading to depression and anxiety, while individuals in the second group tend to express their suffering to the outside world in an opposition manner, developing disorders such as substance abuse, conduct disorders, and/or antisocial behaviors. Factor loadings of the EFA are presented in Table 3, along with the items of the instrument, the subscales with items presented in the original six-factor structure and in the proposed two-factor structure, with the respective Cronbach's alphas and Guttman's lambda-6 coefficient.

The results of the Y-OQ 2.01 network analysis generated two graphical representations (Figure 1 and Figure 2) based on the two-factor structure that emerged in the EFA. These graphs show the expected influence (EI) metric analysis, which aims to identify the force of the nodes' influence in the networks of internalizing and externalizing symptoms. Positive EI scores (0 to 2) represent parallel activations between the nodes and the network - the activation of the node induces the activation of the network. Negative EI scores (-2 to 0) indicate opposite activations between the nodes and the network - the activation of the node leads to a deactivation of the network (Robinaugh et al., 2016).

 

 

 

 

Discussion

The results suggested that the Youth Outcome Questionnaire 2.01 demonstrates evidence of validity for its Brazilian Portuguese version. The CVC analysis verified the instrument's content validity. The methods used (CFA, EFA and network analysis) to evaluate the psychometric properties of the Y-OQ 2.01 indicated that, for the Brazilian context, a two-factor structure has a greater capacity to explain the latent variable of the instrument. The reliability of this structure was evaluated by two different methods (Cronbach's alpha and Guttmann's reliability coefficient), and the results were satisfactory. The model presented is based on the historical premise of psychoanalyst Karen Horney (1964), who proposed a differentiation between patients based on different psychological solutions used to resolve conflicts - both conflicts between individuals and the external world and conflicts between antagonistic internal attitudes. The classification of two different positions to deal with suffering is also present in traditional research on youth psychopathology (Achenbach & Edelbrock, 1978) and, as previously mentioned, is supported by studies on psychiatric disorders according to diagnostic manuals that also found a two-factor structure, which suggests a broader classification of internalizing and externalizing disorders (Kotov et al., 2011; Krueger et al., 1998).

The latent structure identified in this study also emerged in previous findings. In a longitudinal cohort study with a New Zealand epidemiological sample that evaluated ten common mental disorders through the Diagnostic Interview Schedule (DIS), Version III-R (Robins et al., 1989), using DSM-III-R classification, the results pointed to a model of two latent factors (internalization and externalization) underlying these ten disorders. This result highlights the diagnostic manuals' low sensitivity to capture comorbidities and adds to the discussion about the lack of specificities regarding predictors of psychopathology, proposing a broader understanding of these disorders based on different basic orientations towards the world (Krueger et al., 1998). Another study (Cosgrove et al., 2011) also found a latent two-factor model (internalization and externalization) to explain the interrelationships among six psychiatric disorders in adolescents - 1162 pairs of twins and 426 siblings. Genetic analyses of multivariate behavior were carried out, and it was concluded that the concomitant occurrence of internalizing and externalizing symptoms results from genetic and environmental influences.

Although this categorization is very recurrent in the literature (Connell & Goodman, 2002), it is known that comorbidities between internalizing and externalizing disorders in childhood exist and challenge the evaluation model underlying the diagnostic manuals (Krueger et al., 2001). There are many reasons for the presence of comorbidities, such as shared diagnoses, evaluation biases, shared causal factor or genesis; depression may, in some cases, inhibit concern for the consequences of actions, increasing the risk for antisocial behavior, and antisocial behavior can lead to a recurrent state of anxiety (Lilienfeld, 2003). Thus, the overlap of some items of the Y-OQ 2.01 within the two dimensions is supported by the complexity that characterizes human behavior in general.

With regard to the expected influence metric derived from the network analysis, Figure 1 shows the pattern of influence of the internalizing nodes in the network. At the positive pole of the graph, which is composed of nodes that induce the activation of the network, the one with the greatest positive influence is item number 41 (Has suicidal thoughts or says it would be better off to be dead). Suicidal ideation is a diagnostic criterion for depression in the DSM-V (American Psychiatric Association, 2013) and has a high prevalence among adolescents, even among the nonclinical population (Moreira & Bastos, 2015). Other nodes that positively influence the activation of the internalizing network describe symptoms of self-harm (21 - Hurts himself/herself intentionally), somatic (35 - Complains of stomach ache or feeling sick more than other children/adolescents of the same age) and social rejection (63 - Feels that he/she has no friends or that no one likes him/her). Although these items belong to three different subscales in the original Y-OQ 2.01 six-factor structure, they are all characteristic symptoms of internalizing conditions such as depression, anxiety, social isolation, and somatic or physical problems (Merrell, 2008).

Figure 1 also shows that in the negative pole of the graph, composed of nodes that induce the deactivation of the network and decrease the probability of internalizing manifestations, there are symptoms most related to the externalizing spectrum - but whose factor loadings were higher for Factor 1 in the EFA - such as substance use (22 - Uses alcohol or other drugs) and agitation and hyperactivity (14 - Is restless, agitated or hyperactive). The EI analysis seems to support that internalizing and externalizing manifestations, even when present within the same context - which is not uncommon (Krueger et al., 2001), remain two distinct psychological solutions to suffering that tend towards opposite directions. While hyperactivity and aggressive/antisocial behaviors are symptoms directed to the outside world and therefore easier to identify, internalized problems have an excessively controlled and secretive nature, which makes diagnosis difficult and increases the possibility of suffering negligence and noncomprehension (Merrell, 2008). The results of the EI metric demonstrate that for a young man prone to depression and anxiety, addressing such suffering to the outside world through externalization may mitigate his suicidal potential. These data are reinforced by the EFA result (Table 3), in which item 14 (Is restless, agitated or hyperactive) had significant factorial loadings in both factors, but it is negative for Factor 1 - internalizing (-0.333), which can be interpreted again as a protective aspect of the item for the internalizing dynamic.

In evaluating the EI metrics of Factor 2 - externalizing (Figure 2), we observe that delinquent and antisocial behaviors (31 - Violates rules, laws and expectations on purpose; 51 - Gets angry to the point of being a danger to others; 14 - Is restless, agitated or hyperactive) represent the nodes of greatest activation of this network - which is characterized by little control or self-regulation of emotional behaviors and expressions (Merrell, 2008). Deactivation of this path, which culminates in behavior pathologies, seems to be related to a more regressive behavior and disconnection to the external environment (20 - Sees, hears and believes in things that are not real; 18 - Urinates or defecates in his/her pants) and the presence of guilt and responsibility in relation to one's own behavior (47 - Feels guilt properly when he/she does something wrong; 45 - Deals appropriately with boredom and frustration). It is observed that the disconnection from the external world, along with the presence of guilt and frustration tolerance, has the potential to deactivate the externalizing network. The relationship between the absence of guilt and impulsive/antisocial behavior is already widely described in the DSM-V (American Psychiatric Association, 2013).

The observed patterns of influence help clinicians understand how the symptomatic pattern manifests and which direction the course of the disorder may take, and based on the symptoms, they help clinicians to estimate the activation potential of the network. The network analysis also allows the identification of symptom directions. For example, in Figure 1, which presents the set of internalizing symptoms, the most influential item is linked to suicide (item 41), which, once activated, is more likely to trigger other internalizing symptoms. The lowest activation item is related to alcohol and substance use (item 22). Therefore, regarding internalized suffering, symptoms either evolve to self-harm or suicide-related symptoms or progress to substance abuse disorders, which will not necessarily trigger symptoms of classical depression such as suicide attempts. The same happens with the externalizing network. The item that involves the largest activation of the network is item 31 (Violates rules, laws and expectations on purpose). When enabled, other symptoms linked to the externalizing network are more likely to appear. However, when item 20 (Sees, hears and believes in things that are not real) is present, the network is heading towards a deactivation - and disorders that include delusions and hallucinations are likely to develop.

Although understanding the Y-OQ 2.01 in two factors is supported by studies that found two latent variables to explain mental disorders (Cosgrove et al., 2011; Kotov et al., 2011; Krueger et al., 1998), the six factors that comprise the original structure of the Y-OQ 2.01 reveal a richness of detail to outline the behaviors that should be considered by the clinician. Considering possible overlaps in both models, it is observed that the six factors of the original structure are organized in a coherent way between the two dimensions proposed by the EFA. For example, the subscale of Interpersonal Stress, which evaluates anxiety and depression (typically internalizing), was divided into 14 items in the internalizing dimension and only four in the externalizing dimension, and these four items (15, 33, 49, and 64) are clear behavior manifestations of anxiety and sadness. The subscale of Somatic Distress, typically an internalizing dimension (Dhossche et al., 2002; Merrell, 2008), was decomposed into seven items in the internalizing dimension and only one in the externalizing dimension, item 18 (Urinates or defecates in his/her pants), understood as an exteriorization of suffering. The items that belong to the other four subscales of the instrument's original structure also seem to have a logical division between the two proposed dimensions (as observed in Table 3).

The relevance of the original model's structure can also be observed through the correlation pattern presented in Table 2. Strong correlations were found between Interpersonal Relations (variable containing both externalizing and internalizing elements) and two subscales: Intrapersonal Distress (typically internalizing) and Behavioral Dysfunction (externalizing). In addition, moderate correlations predominated among the subscales (all p<0.001), except for the weak correlation between Somatic Distress (typically internalizing) and Social Problems (externalizing). It is important to mention that these six dimensions were developed based on many methods of information gathering, such as meta-analytic analyses, focus groups with patients, parents, psychologists and psychiatrists, hospital registries, among others (Burlingame et al., 2005). Thus, the interpretation of the Y-OQ 2.01 based on six factors maintains a clinical relevance that must be considered in future studies.

 

Conclusion

This study demonstrated evidence of validity for the Brazilian Portuguese Y-OQ 2.01. Therapists seeking to evaluate psychotherapy outcomes and processes with children and adolescents may benefit from the use of this tool, especially given that this population has a lower capacity for self-observation of behavior and depends on external observers to determine psychological triggers, motivators, extent of mental distress, and improvements. This instrument may be even more relevant for preverbal children, whose monitoring of treatment is impossible without parental input. In addition, the possibility of obtaining the perception of other people from the child's environment about the treatment's response makes it possible to detect maladjustments and complements the evaluation of the patient with multiple perspectives of their evolution. Specifically, for adolescents who are particularly vulnerable to disorders originating from untreated problems in childhood (Klasen & Crombag, 2013), assessing their parents' perception of the treatment strengthens the support network.

Limitations of this study are based on the use of a nonprobabilistic and nonrepresentative sample of the Brazilian population. In addition, subsequent studies containing larger and probabilistic samples, including samples from other Brazilian regions, should be conducted. The hypothesis that the factorial structure of the Brazilian Portuguese Y-OQ 2.01 would be similar to its original version, consisting of six factors, was not confirmed, and a two-factor solution was proposed. It is expected that other studies continue to use this instrument in Brazil and that the structure for this sample will continue to be tested. The strengths of this study are the adequate sample distribution regarding gender and age group, since there are significant theoretical differences regarding internalizing and externalizing manifestations between boys and girls and children and adolescents (Rescorla et al., 2007). Considering the limited number of Brazilian studies examining psychotherapy with children and adolescents that use instruments to evaluate results and the scarce use of quantitative assessments to measure psychotherapy outcome (Holst et al., 2018), this study intends to provide a tool that helps the scientific community to demonstrate results of psychotherapeutic interventions of any theoretical base and foster evidence-based psychotherapy.

 

References

Achenbach, T. M.; Edelbrock, C. S. (1978). The classification of child psychopathology: A review and analysis of empirical efforts. Psychological Bulletin, 85(6), 1275-1301. https://doi.org/10.1037/0033-2909.85.6.1275        [ Links ]

Achenbach, T. M.; Rescorla, L. A. (2001). Manual for the ASEBA school-age forms and profiles. Burlington, VT: University of Vermont Research Center for Children, Youth, & Families.         [ Links ]

American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: American Psychiatric Association.         [ Links ]

American Psychological Association (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285.         [ Links ]

American Psychological Association (2008). Disseminating evidence-based practice for children and adolescents: A systems approach to enhancing care. Washington, DC: American Psychological Association.         [ Links ]

Belfer, M. L. (2008). Child and adolescent mental disorders: The magnitude of the problem across the globe. The Journal of Child Psychology and Psychiatry, 49, 226-236. https://doi.org/10.1111/j.1469-7610.2007.01855.x        [ Links ]

Bickman, L.; Lambert, E. W.; Andrade, A. R.; Penaloza, R. V. (2000). The Fort Bragg continuum of care for children and adolescents: Mental health outcomes over 5 years. Journal of Consulting and Clinical Psychology, 68, 710-716. https://doi.org/10.1037/0022-006X.68.4.710        [ Links ]

Borsboom, D.; Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. The Annual Review of Clinical Psychology, 9, 91-121. https://doi.org/10.1146/annurev-clinpsy-050212-185608        [ Links ]

Breakwell, G.; Hammond, S.; Fife-Schaw, C.; Smith, J. A. (2010). Método de pesquisa em psicologia (3ª ed.). Porto Alegre: Artmed.         [ Links ]

Burlingame, G. M.; Cox, J.; Wells, G.; Latkowski, M.; Justice, D.; Carter, C.; Lambert, M. (2005). The administration and scoring manual of the Youth Outcome Questionnaire. Salt Lake City, UT: OQ Measures.         [ Links ]

Burlingame, G. M.; Wells, M. G.; Lambert, M. J.; Cox, J. C. (2004). Youth Outcome Questionnaire (Y-OQ). In: M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcomes assessment: Instruments for children and adolescents, p. 235-273. Mahwah, NJ: Lawrence Erlbaum Associates.         [ Links ]

Cassepp-Borges, V.; Balbinotti, M. A. A.; Teodoro, M. L. M. (2010). Tradução e validação de conteúdo: Uma proposta para a adaptação de instrumentos. In: L. Pasquali e colaboradores (Orgs.), Instrumentação psicológica: Fundamentos e práticas, p. 506-520. Porto Alegre: Artmed.         [ Links ]

CFP - Conselho Federal de Psicologia (2000). Resolução nº 16, de 20/12/2000. Dispõe sobre a realização de pesquisa em Psicologia com seres humanos. Brasília: CFP. https://atosoficiais.com.br/cfp/resolucao-de-fiscalizacao-e-orientacao-n-16-2000-dispoe-sobre-a-realizacao-de-pesquisa-em-psicologia-com-seres-humanos        [ Links ]

CNS - Conselho Nacional de Saúde, Ministério da Saúde (2016). Resolução nº 510, de 07/04/2016. Normas aplicáveis a pesquisas em Ciências Humanas e Sociais. Brasília: Diário Oficial da União. https://www.in.gov.br/materia/-/asset_publisher/Kujrw0TZC2Mb/content/id/22917581        [ Links ]

Connell, A. M.; Goodman, S. H. (2002). The association between psychopathology in fathers versus mothers and children's internalizing and externalizing behavior problems: A meta-analysis. Psychological Bulletin, 128(5), 746-773. https://doi.org/10.1037/0033-2909.128.5.746        [ Links ]

Connors, C. K. (1990). Connor's rating scales manual. North Towanda, NY: Multi-Health Systems.         [ Links ]

Cosgrove, V. E.; Rhee, S. H.; Gelhorn, H. L.; Boeldt, D.; Corley, R. C.; Ehringer, M. A. (2011). Structure and etiology of co-occurring internalizing and externalizing disorders in adolescents. Journal of Abnormal Child Psychology, 39(1), 109-123. https://doi.org/10.1007/s10802-010-9444-8        [ Links ]

Costello, A. B.; Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, & Evaluation, 10, 1-9.         [ Links ]

Costello, E. J.; Maughan, B. (2015). Annual Research Review: Optimal outcomes of child and adolescent mental illness. The Journal of Child Psychology and Psychiatry, 56(3), 324-341. https://doi.org/10.1111/jcpp.12371        [ Links ]

Dhossche, D.; van der Steen, F.; Ferdinand, R. (2002). Somatoform disorders in children and adolescents: A comparison with other internalizing disorders. Annals of clinical psychiatry, 14(1), 23-31. https://pubmed.ncbi.nlm.nih.gov/12046637/        [ Links ]

Dunn, T.; Burlingame, G.; Walbridge, M.; Smith, J.; Crum, M. (2005). Outcome assessment for children and adolescents: Psychometric validation of the Youth Outcome Questionnaire 30.1. Clinical Psychology and Psychotherapy, 12(5), 388-401. https://doi.org/10.1002/cpp.461        [ Links ]

Epskamp, S.; Cramer, A. O. J.; Waldorp, L. J.; Schmittmann, V. D.; Borsboom, D. (2012). Qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software, 48, 1-18. https://doi.org/10.18637/jss.v048.i04        [ Links ]

Guimarães, L. P. M.; Yoshida, E. M. P. (2014). Criteria of progress in child psychotherapies according to psychotherapists. Paidéia, 24(57), 95-104. https://doi.org/10.1590/1982-43272457201412        [ Links ]

Hernández-Nieto, R. A. (2002). Contribuciones al análisis estadístico. Mérida, Venezuela: Universidad de Los Andes / IESINFO.         [ Links ]

Holst, B.; Lisboa, C. S. M.; Braga, A. Z.; Strey, A. M.; Souza, D. B. (2018). Systematic review on the use and description of measures to evaluate psychotherapeutic interventions with children and adolescents in Brazil. Trends in Psychiatry and Psychotherapy, 40(4), 342-351. https://doi.org/10.1590/2237-6089-2017-0067        [ Links ]

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrica, 30, 179-185.         [ Links ]

Horney, K. (1964). Nossos conflitos interiores. Rio de Janeiro: Civilização Brasileira.         [ Links ]

Hu, L.; Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118        [ Links ]

IBM Corp. (2010). IBM SPSS Statistics for Windows, Version 19.0. Armonk, NY: IBM Corp.         [ Links ]

Kazdin, A. E. (2000). Understanding change: From description to explanation in child and adolescent psychotherapy research. Journal of School Psychology, 38(4), 337-348.         [ Links ]

Klasen, H.; Crombag, A. C. (2013). What works where? A systematic review of child and adolescent mental health interventions for low and middle income countries. Social Psychiatry and Psychiatric Epidemiology, 48(4), 595-611. https://doi.org/10.1007/s00127-012-0566-x        [ Links ]

Kotov, R.; Chang, S. W.; Fochtmann, L. J.; Mojtabai, R.; Carlson, G. A.; Sedler, M. J.; Bromet, E. J. (2011). Schizophrenia in the internalizing-externalizing framework: A third dimension? Schizophrenia Bulletin, 37, 1168-1178. https://doi.org/10.1093/schbul/sbq024        [ Links ]

Krueger, R. F.; Caspi, A.; Moffitt, T. E.; Silva, P. A. (1998). The structure and stability of common mental disorders (DSM-III-R): A longitudinal-epidemiological study. Journal of Abnormal Psychology, 107(2), 216-227. https://doi.org/10.1037/0021-843X.107.2.216        [ Links ]

Krueger, R. F.; McGue, M.; Iacono, W. G. (2001). The higher-order structure of common DSM mental disorders: Internalization, externalization, and their connections to personality. Personality & Individual Differences, 30(7), 1245-1259. https://doi.org/10.1016/S0191-8869(00)00106-9        [ Links ]

Lilienfeld, S. O. (2003). Comorbidity between and within childhood externalizing and internalizing disorders: Reflections and directions. Journal of Abnormal Child Psychology, 31(3), 285-291. https://doi.org/10.1023/A:1023229529866        [ Links ]

McClendon, D. T.; Warren, J. S.; Green, K. M.; Burlingame, G. M.; Eggett, D. L.; McClendon, R. J. (2011). Sensitivity to change of youth treatment outcome measures: A comparison of the CBCL, BASC-2, and Y-OQ. Journal of Clinical Psychology, 67(1), 111-125. https://doi.org/10.1002/jclp.20746        [ Links ]

Merrell, K. W. (2008). The Guilford practical intervention in the schools series. Helping students overcome depression and anxiety: A practical guide (2nd ed.). New York: Guilford.         [ Links ]

Moreira, L. C. O.; Bastos, P. R. H. O. (2015). Prevalência e fatores associados à ideação suicida na adolescência: Revisão de literatura. Psicologia Escolar e Educacional, 19(3), 445-453. https://doi.org/10.1590/2175-3539/2015/0193857        [ Links ]

Pasquali, L. (2010). Testes referentes a construto: Teoria e modelo de construção. In: L. Pasquali e colaboradores (Orgs.), Instrumentação psicológica: Fundamentos e práticas, p. 165-198. Porto Alegre: Artmed.         [ Links ]

Rescorla, L.; Achenbach, T. M.; Ivanova, M. Y.; Dumenci, L.; Almqvist, F.; Bilenberg, N.; … Verhulst, F. (2007). Behavioral and emotional problems reported by parents of children ages 6 to 16 in 31 societies. Journal of Emotional and Behavioral Disorders, 15, 130-142. https://doi.org/10.1177/10634266070150030101        [ Links ]

Ridge, N. W.; Warren, J. S.; Burlingame, G. M.; Wells, M. G.; Tumblin, K. M. (2009). Reliability and validity of the Youth Outcome Questionnaire Self-Report. Journal of Clinical Psychology, 65, 1115-1126. https://doi.org/10.1002/jclp.20620        [ Links ]

Robinaugh, D. J.; Millner, A. J.; McNally, R. J. (2016). Identifying highly influential nodes in the complicated grief network. Journal of Abnormal Psychology, 125(6), 747-757. https://doi.org/10.1037/abn0000181        [ Links ]

Robins, L. N.; Helzer, J. E.; Cottier, L.; Goldring, E. (1989). Diagnostic Interview Schedule, Version III-R (unpublished manuscript). Washington University, St. Louis, MO.         [ Links ]

Satorra, A.; Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507-514. https://doi.org/10.1007/BF02296192        [ Links ]

Schmittmann, V. D.; Cramer, A. O. J.; Waldorp, L. J.; Epskamp, S.; Kievit, R. A.; Borsboom, D. (2011). Deconstructing the construct: A network perspective on psychological phenomena. New Ideas in Psychology, 31, 43-53. https://doi.org/10.1016/j.newideapsych.2011.02.007        [ Links ]

Schreiber, J. B.; Stage, F. K.; King, J.; Nora, A.; Barlow, E. A. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. Journal of Educational Research, 99(6), 323-337. https://doi.org/10.3200/JOER.99.6.323-338        [ Links ]

Timmerman, M. E.; Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209-220. https://doi.org/10.1037/a0023353        [ Links ]

WHO - World Health Organization (1995). Physical status: Use and interpretation of anthropometry. Geneva: World Health Organization.         [ Links ]

WHO - World Health Organization & Calouste Gulbenkian Foundation (2014). Social determinants of mental health. Geneva: World Health Organization.         [ Links ]

Wild, D.; Grove, A.; Martin, M.; Eremenco, S.; McElroy, S.; Verjee-Lorenz, A.; Erikson, P. (2005). Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for translation and cultural adaptation. Value in Health, 8(2), 94-104. https://doi.org/10.1111/j.1524-4733.2005.04054.x        [ Links ]

 

 

Recebido em 10 de março de 2019
Aceito para publicação em 13 de abril de 2021

 

 

 Este estudo foi financiado pela CAPES/PROSUP - Coordenação de Aperfeiçoamento de Pessoal de Ensino Superior / Apoio à Pós-Graduação de Instituições de Ensino Particular. O autor Wagner de Lara Machado é bolsista de produtividade 2 do CNPq - Conselho Nacional de Desenvolvimento Científico e Tecnológico.
 Este artigo é parte da tese de doutorado da primeira autora, intitulada Evidências de validade da escala Youth Outcome Questionnaire 2.01, versão pais e responsáveis legais, para o português brasileiro, apresentada no Programa de Pós-Graduação em Psicologia da Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS).