Participants

For the user’s participants, the inclusion criteria were aging 18 years or over; being in treatment for at least 3 months in one of the participating services, without restriction of diagnosis. Exclusion criteria were: having disagreed in participating in the research; and having severe cognitive impairment that prevented them from participating and answering the questions. Users were invited at the services, or during the therapeutic groups and others indicated by the staff. For family members, the inclusion criteria were: To be family members of users who were attending community mental health services for a period longer than three months, over the age of 18 years, able to communicate in Brazilian Portuguese, and without cognitive impairments.

Instruments

The sociodemographic questionnaire. A questionnaire was used to assess the characteristics of the research participants. The variables included in the questionnaire were: age, sex, race, education and income, diagnosis, length of treatment, service attended.

Recovery Self-Assessment (RSA-R) (^{O’Connell et al., 2005}, ²⁰⁰⁷). We used the revised Recovery Self-Assessment (RSA-R) Person in Recovery and Family members versions. The RSA-R assesses recovery-oriented practices on community mental health services. It consists of 32 items with five response options to be chosen: 1 (Strongly Disagree) to 5 (Strongly Agree) and two additional alternatives D/K (do not know) and N/A (not applicable), which should be coded as missing, constituting a 5-point Likert scale. In the RSA-R family members version, the instrument has 8 additional items that are deemed appendixes.

The items were categorized and grouped into 6 domains as follows: 1 (Life Goals); 2 (Involvement); 3 (Diversity of Treatment Options; 4 (Choice); 5 (Individually-tailored Services); 6 (Inviting Domain) (^{O’Connell et al., 2005}, ²⁰⁰⁷).

The instrument was cross-culturally adapted following the methodology indicated in the literature to achieve equivalence between source and target languages (^{Beaton et al., 2007}; ^{Wild et al., 2005}). Items were linguistically translated and cross-culturally adapted to maintain the validity of the content (^{Beaton et al., 2007}; ^{Guillemin, 1995}).

Procedures

Interviews were conducted by students of the Multiprofessional Residency in Mental Health at the University of Campinas (UNICAMP) and postgraduate students of the Public Health and Mental Health Interfaces Research Group at UNICAMP. Interviewers were preselected and qualified to apply the instrument by the research team. Training occurred in two meetings, 2h/each, in which we thoroughly discussed the instrument, management during invitations, application, and rules of retest. We also have a closed virtual group, in which everyone could share their doubts in real time, being solved by the researchers.

Ethical Considerations

This research is part of a larger project titled: Recovery: Instrumentos para sua aferição na realidade brasileira [Recovery: Instruments for measuring it within the Brazilian reality], approved by the Research Ethics Committee (CEP) with opinion no. 60826616.6.0000.5404, in March 2017, with an Informed Consent Form. Permission was obtained from the head of the fifteen community mental health centers included in the research. All participants signed an informed consent to participate in the study.

Data analysis

The database for inserting responses of the sociodemographic questionnaire, test, and retest was typed using the form built in Google forms, to aid in the control of errors, generating a standard Excel® file, which was subsequently transferred and analyzed. We used the SAS System for Windows (Statistical Analysis System) version 9.4. SAS® Institute Inc, Cary, NC, USA (^{The SAS System, 2013}).

The sociodemographic questionnaire was analyzed using descriptive statistics; frequencies and percentages of categorical variables, and the measures of central tendency and dispersion for numerical variables were calculated.

Reliability verifies homogeneity, redundancy, or heterogeneity of an instrument, by the ability to reproduce results, even under different conditions such as in the use of different items for a similar group of individuals (internal consistency - calculated by the α-Cr) over time (test-retest calculated by the ICC) (^{Brown & Prescott, 2006}). For RSA-R Person in Recovery and Family member versions, the reliability was assessed considering the six domains of the original US scale (^{O’Connell et al., 2005}, ²⁰⁰⁷). Internal consistency was calculated using the Cronbach’s alpha coefficient (α-Cr), and temporal stability was estimated using the Intraclass Correlation Coefficient (ICC). A value of ≥0.70 was considered adequate for both measures (^{Prinsen et al., 2016}). The level of significance adopted for the study was 5%.

Confirmatory factor analysis was used to verify the structure of the instrument RSA-R person in recovery, using the Polychoric correlation matrix (^{Brown & Prescott, 2006}; ^{Damásio, 2012}). Due to the high percentage of items without information (N/A and D/K), it was not possible to carry out the analysis of the validity of the factor structure of the RSA-R Family/Brazil. For adjustment quality indexes we used Goodness of Fit Index (GFI), Adjusted GFI (AGFI), Bentler Comparative Fit Index, Bentler-Bonett (NFI), and RMSEA Estimate (^{Brown & Prescott, 2006}; ^{Hair et al., 2005}). The level of significance adopted for the statistical tests was 5%, i.e., p-value <0.05.

It is recommended to verify validity of subjective constructs using exploratory factor analysis EFA). EFA provides the researcher with the amount of factors needed to represent the data, i.e., it is a tool for exploring the dimensionality of a set of items (^{Mokkink, Terwee, Patrick, et al., 2010}; ^{Polit, 2015}). In EFA, variables produce loads for all factors, contributing to the analysis of items and their loads concerning the construct (^{Damásio, 2012}). EFA is usually conducted when the researcher wants to confirm or refute the factorial structure of a given instrument (^{Brown & Prescott, 2006}).

The EFA in this study was evaluated by the method of principal components and Varimax orthogonal rotation. The following criteria were considered: 1. Kaiser - Meyer - Olkin measure of sample adequacy (KMO) >0.60 and Bartlett's sphericity test with p<0.05; 2. eigenvalue >1 in line with the Scree Plot plot and the explained percentage variance; 3. loading variables >0.40; 4. substantive meaning of item in each factor (^{Pituch, 2016}). The level of significance adopted for the statistical tests was 5%.

Sociodemographic profile

Table 1 shows the descriptive data of the samples of users and family members. 207 service users participated, mostly of CAPS and CAPS AD (72.4%); 6 years of treatment or over (54.6%); men (56.2%); The mean age was 47.56 year (±13.8) ranging from 20 to 101 years; white color/ethnicity (42.5%); single (79.7%); receiving some kind of benefit (62.8%) with family income between 2 and 3 minimum wage (40.6) and diagnosis of schizophrenia (29.0%) e mood disorders (30.0%).

Concerning the participating family members, the most were women (82.0%) with a mean age was 54.94 years (± 13.1) ranging from 21 to 84 years, married (54.9%), white (53.9%), with family income between less than 1 minimum wage and 2 minimum wages (64.9%). Regarding the degree of kinship, fathers/mothers were identified as the main caregivers of patients (53.4%). Of the total interviews, 123 (59.7%) were conducted at CAPS, 49 (23.8%) at CAPS AD, and 33 (16.5%) at other services.

Internal consistency

In the RSA-R Person in Recovery version were found three satisfactory domains, α-Cr>0.7 (life goal, involvement and Individually-tailored services), two acceptable domains (Diversity of treatment options and Choices), For domain 6, we could not calculate the alpha value by the number of items and, according to the literature, at least three to four items are accepted per domain (^{Hair et al., 2005}; ^{Polit & Beck, 2006}). Overall, the instrument accounts for α-Cr=0.9338 (Table 2).

Concerning the RSA-R family members Concerning the RSA-R family members version, the Cronbach’s alpha coefficients ranged from 0.68 (individually-tailored services) to 0.89 (life goals) (Table 2). According to the data, we verified adequate value for the global scale, with a Cronbach’s alpha of 0.94. The evaluation of internal consistency of each domain of the scale in case any item was removed, indicated that, for the majority, the Cronbach’s alpha would decrease, suggesting the contribution of the item to the internal consistency of the domain. Only the removal of item 29 would increase the Cronbach’s alpha in its corresponding domain. Nevertheless, this increase is not significant, considering that the domains are already reflected in an adequate reliability score.

Intraclass Correlation Coefficient (ICC)

In this research, analyses of stability in test/retest were verified by ICC. We used the interval between 10 and 20 days in the applications, being reapplied to 63 users and 70 family members.

In the temporal stability analysis data, the RSA-R Person in recovery version indicates that there is an acceptable correlation between the scores of the two applications of the instrument, both for the global scale (ICC=0.68) and for domains one, two, four; for domains three and five, we found moderate stability (table 3). Results concerning the ICC per item, ranged from -0.06 (item 11) to 0.78 (item 5). Seven items showed acceptable stability (items 1, 3, 16, 17, 18, 27 and 30). The remaining items did not meet the literature criteria (Table 3).

Concerning the RSA-R family members version, the ICC indicates that there is an adequate correlation between the scores of the two applications of the instrument, both for the global scale (ICC=0.83) and for domains one, four and five; for domains two and three, we found acceptable stability (Table 3). Results concerning the ICC per item, ranged from 0.13 (item 23) to 0.81 (item 2). Eight items showed good stability (items 2, 4, 5, 10, 14, 15, 20, and 40). The remaining items did not meet the literature criteria (^{Prinsen et al., 2016}) (Table 3).

Confirmatory Factor Analysis

In the CFA none of the 5 adequacy indices were satisfactory, the instrument cannot be considered validated by these psychometric parameters in this sample.

It is also noteworthy the missing data that we had in this confirmatory factor analysis, considering that of the 207 interviews, only 71 were eligible. This occurred because we had many “don’t know” (D/K) and “not applicable” (N/A) answers, which are deemed missing data, i.e., data unconsidered for this analysis (Table 4).

Considering these unsatisfactory results with CFA, we conducted an exploratory factor analysis (EFA) to understand how the items behaved in the domains and their factor loads. We firstly performed the Kaiser’s Measure of Sampling Adequacy (MSA) and the measurement of each sentence with all items to understand the behavior of each question. The overall MSA was=0.80733, and we opted to exclude the questions with MSA <70% (^{Brown & Prescott, 2006}). In this stage, we excluded items 4, 6, 14, and 29.

Then, we repeated the process and MSA accounted for = 0.085859 (n=105), indicating that the sample had good consistency to be used in the factor analysis. By the criterion of selection of factors with an eigenvalue higher than 1.7 domains were obtained, which explain 3.76% of the data variability. According to the scree plot, we opted to maintain the extraction of 3 domains, which explain 47.25% of the total variability (^{Brown & Prescott, 2006}). We describe in the table 5 the behavior by the factor loads:

Items 5, 15, and 21a had values below 0.4 and were excluded from this new model because they did not fit the distribution of the domains (^{Brown & Prescott, 2006}). On the other hand, Items 8, 13, 18, 26 and 32 had values between .4 and .5 (0.5< x >0.4) and, according to the indication of the studied literatures, they may be excluded depending on the researchers’ evaluations (^{Brown & Prescott, 2006}; ^{Damásio, 2012}). We also noticed throughout the collection and analyses that items 27 and 28, considering the Brazilian culture, resembled and appeared with repeated items, being the 28th easier to understand.

Thus, in total, the model would have 20 items (sentences), requiring a new field to test the format based on a confirmatory factor analysis (^{Brown & Prescott, 2006}; ^{Hair et al., 2005}). In Table 06 we demonstrate the new model, after conducting exploratory factor analyses:

Reliability of RSA-R Person in Recovery and Family Members versions

According to our results, the a priori six-domains structure of RSA-R Person in Recovery and Family/ Brazil is a reliable instrument to measure recovery-oriented practices of community mental health services from the perspective of family members. In other words, our results were consistently reproducible with few variations due to time and/or intercorrelations between items of the measure (^{Souza et al., 2017}).

For the RSA-R family version, the internal consistency values of three of the six domains of the construct were adequate for the test. Domains 4 (choices) and 5 (individually-tailored services) achieved acceptably consistency. Low scores of internal consistency were also reported for domain 5 in the study Sweden for the validation of the RSA-R Person in Recovery versión (^{Rosenberg et al., 2015}).

An explanation for the fragility of these domains may be related to the profile of services in the country, centered on the disease, drug treatment, and on the Brazilian authoritarian culture. Individuals continue to be perceived by the symptoms and not by their subjectivity. This suggests that users must undergo prolonged treatments with little decision-making power (^{Emerich et al., 2014}; ^{Silva et al., 2019}). Furthermore, the weakness of the Brazilian mental health system in terms of users’ participation and autonomy levels. According to studies, there is still a strong hierarchical relationship between the professionals and users and their family members. Sometimes, services can assume a place of control, and this may discourage the protagonism of both the user and the family member regarding the treatment (^{Emerich et al., 2014}; ^{Silva et al., 2019}).

According to the results of stability per domain, there were adequate levels of intraclass correlation coefficient for most domains. Nevertheless, domains 2 and 3 accounted for a moderate stability. These domains refer to the implementation of strategies for greater involvement of users in the services. In addition, they also refer to whether the services provide diversified treatment options with community integration.

In Brazilian community health services, the individually-tailored treatment centered on the user is a prerequisite for the attention and care of the users and their family, aiming at achieving greater involvement on their part and the diversification of their support networks. However, it often works as an unfulfilled premise, which causes the treatment to be reduced to the prescription of activities informed to the users, generally prioritizing the pharmacological treatment and consultations with the psychiatrist (^{Emerich et al., 2014}; ^{Silva et al., 2019}).

For the RSA-R person version, as there were 3 domains with satisfactory indices in the internal consistency values, this collaborated to follow with the test and retest steps, which were lower than the standard and led us to the exploratory analysis. Considering the phenomenological nature of the instrument and cultural singularities, it was not surprising that there was a need to extend statistical analysis.

Validity of RSA-R Person in Recovery

Confirmatory factor analysis demonstrated that the original, a priori six-factor structure of RSA-R Person in recovery could not be replicated in our sample. Thus, using EFA, we observed that such indicated a structure with fewer items and domains for the Brazilian version of the instrument. In the studied literature, the interpretation of these coefficients may vary and depends on the studied context and the objects (^{Cohen, 1988}). We are studying an instrument with subjective characteristics, on people who have moderate to severe mental disorders, with unfavorable social conditions, which demanded from the research team many adaptations in the instrument and data collection.

Different fields in the research can affect reliability, such as variability and number of people interviewed, methods used in the collection, the analyzed period, and statistical choices among others. Our sample was relatively homogeneous between men and women, with low income and mean education level. Highlights for some bias are people’s long time of treatment and the use of certain medications, which may imply certain chronicity, and different interviewers used in the fields.

Regarding validation (EFA), the sample decreased from 207 to 105 valid interviews, since we observed a large number of missing answers. Two thirds of the interviewees answered some items as “don’t know” (D/K) or “not applicable” (N/A). In our assessment, the “N/A” option may make sense for an instrument that intends to evaluate different services and programs. On the other hand, the “D/K” opens the possibility of people not positioning themselves based on their evaluation/ experience, for different reasons, damaging the performance of the instrument as a whole and the evaluation itself of the services.

With EFA, we perceive by numerical results that the exclusion of some items would not affect the overall set of the instrument, because these could be measurable by other items that remained, i.e., in the Brazilian reality some contents of the sentences were redundant and impaired the evaluative performance of the instrument.

On the other hand, some excluded items are directly linked to cultural differences between the country of origin of the instrument (USA) and the Brazilian culture. In Brazil, very patriarchal, tutelary, racist, and unequal relations are prevalent in multiple senses (social, economic, political, educational etc.), demonstrating a low degree of autonomy, citizenship, and empowerment, core elements for recovery. Examples of this fact were confirmed in the items excluded in the EFA (MSA and Varimax), which demonstrate the unequal power relations between users and services.

Therefore, for the RSA-R users’ version (Pt-Br) we would have 20 items grouped into three domains. Domain one, with six items, was designated as welcoming, as it debates the first contacts of services and people. It is understood that the professional attitude aimed at recovery is essential in resuming the hope of people in suffering. The second domain, with eight items, was designated as social involvement, as it is a necessary condition to keep connected the social struggles for rights. Domain three, with six items, was designated as choices, a dimension that is also fundamental in the recovery process, in which the centrality is in the voice of people and the preservation of their autonomy.

^{Konkolÿ Thege et al. (2017)} corroborates the need for revisions to make RSA-R a more solid instrument regarding its psychometric attributes, thus evaluating practices aimed at recovering mental health services in different countries (^{Konkolÿ Thege et al., 2017}). Results of the EFA resembled the RSA - brief version (RSA-B), with 12 items, which still lacks further studies for the model (^{Barbic et al., 2015}).

Study strengths and limitations

The RSA-R Person in Recovery version is widely used in international studies to assess recovery orientation in mental health services, highlighting the relevance of this construct in evaluating such services (^{Larivière et al., 2020}; ^{Wong et al., 2020}; ^{Sánchez-Guarnido et al., 2024}). This underscores the importance of validating and applying this scale in the Brazilian context.

In addition to the original study, this is the first study that focuses on assessing the psychometric properties of the revised version of the RSA Family. Only two studies evaluated the revised version, but focusing on the questionnaire for Person in Recovery and Provider (Sweden and Canada) (^{Konkolÿ Thege et al., 2017}; ^{Rosenberg et al., 2015}). Furthermore, only in Canada and China a cross-cultural validation was carried out in the interest group of family members; however, the authors adapted the original scale of 36 items (^{Bola et al., 2016}; ^{Kidd et al., 2010}). Therefore, this would be a pioneer study in providing evidence on the reliability of the RSA-R Family instrument in a different context from the USA.

Moreover, we found that only two studies performed the test-retest reliability analysis in addition to ours. One was conducted in Sweden, and the other was conducted in China, where the authors validated the RSA-R Person in Recovery (^{Konkolÿ Thege et al., 2017}; ^{Wong et al., 2020} ordem alfabética). This methodology is recommended in the literature for the cross-cultural validation of scales because it allows the assessment of the invariability of responses (^{Souza et al., 2017}). In our study, this technique allowed the identification of items that had lower scores between applications, indicating the possible limitations of adaptation of some items to the Brazilian context.

A limitation of the study was the high percentage of items without information (N/A and D/K), which made it impossible to carry out the analysis of the validity of factor structure of the RSA-R Family/Brazil. Some researchers (^{Barbic et al., 2015}; ^{Konkolÿ Thege et al., 2017}) had already identified the complexity of the Likert scale for measuring the agreement of the instrument, encouraging researchers to test the scale with fewer response categories. Considering this, we suggest future research on the scale in Brazil, in order to test the instrument without considering the categories of D/K and N/A and, thus, improve the mensuration potential of the scale.

Recent international studies on the RSA have primarily focused on the statistical analysis of reliability and validity tests, with little attention given to content aspects and the structure of the response options (^{Tan & Fernandez, 2018}). In preliminary cross-cultural adaptation studies conducted in Brazil, the structure of the response options already posed a challenge in pilot studies with users and family members (^{Erazo-Chavez, LaRotta et al., 2021}; ^{Ricci et al., 2020}).

^{Erazo-Chavez, Ricci et al. (2021)} identified several possible explanations for this issue, such as the scale requiring a certain level of education from participants (^{Erazo Chavez, Ricci et al., 2021}). In this regard, the socioeconomic differences between the countries that validated and continue to use the scale-such as the United States, Canada, Australia, China, and Germany, which generally have higher educational standards and better access to education compared to users and family members in Brazil-may result in greater difficulties in completing the scale. Furthermore, recovery-oriented practices are strongly encouraged in mental health policies in most of these countries. In Brazil, while mental health policy has made significant progress in expanding the network of community mental health services with a humanized approach, recovery has not yet been a central focus in policy (^{Erazo Chavez; Ricci et al., 2021}; ^{Onocko Campos et al., 2017}; ^{Ricci et al., 2020}).

The second limitation we found was the difficulty in detailing the answers and distinguishing them in the two end points of the scale (Strongly disagree X disagree; Strongly agree X agree). We perceived doubts between these gradations during the interviews, demanding interviewers to explain this variation. This difficulty with the 5-point scale for measuring the subjective phenomenon was also reported in another study using RSA (^{Barbic et al., 2015}).

One of the limitations reported in the interviewers’ field journals was the concern or fear of people when making a criticism or negative assessments of the services and staff when responding the questions. Several times we were asked whether they would lose the benefit, or whether they would suffer some kind of retaliation, or harm the workers. A similar situation occurred in Canada with the validation and assessments of services using RSA (^{Kidd et al., 2010}).

The third limitation may be the size of the sample, 207 participants to 33 items is the minimum that the literature requires for analyses to be done. We are dealing with a subjective construct, which in different cultures can generate different interpretations, in addition to the people’s difficulty in understanding the content of the instrument, since most of them have been treated for years and are diagnosed with severe disorders, which may cause temporary or permanent cognitive difficulties.

Future studies should focus on addressing these limitations by considering several approaches. First, reducing the complexity of the Likert scale, as suggested by previous research (^{Barbic et al., 2015}; ^{Konkolÿ Thege et al., 2017}), may improve the reliability of responses. Testing the RSA-R Family/Brazil with fewer response categories and excluding "D/K" (Don’t Know) and "N/A" (Not Applicable) options could help enhance the instrument’s ability to capture meaningful data. Additionally, further research should explore alternative ways to clearly differentiate between response options, particularly at the scale's endpoints, to reduce confusion. Researchers could also experiment with different formats or clearer descriptions to assist participants in understanding these variations, especially when subjective phenomena are being measured.

Finally, future studies should aim to increase sample sizes to strengthen statistical analyses, while also considering the specific cognitive challenges faced by participants with severe mental disorders. Ethical considerations regarding potential fear of retaliation when answering should also be addressed by emphasizing confidentiality and ensuring participants feel safe in expressing critical feedback.