Inglés (pdf)
Articulo en XML
Referencias del artículo
Traducción automática
Permalink
Authors summary
Why was this study done?
This pioneering research in Brazil presents a novelty in the methodological design of educational technologies to promote and protect the health of populations facing a more significant risk of death during pandemics. The research topic is very current and will raise international discussion among professionals and researchers investigating health education and risk communication related to public health emergencies.
What did the researchers do and find?
Our methodological study investigated the validity and reliability of an electronic booklet on COVID-19 for adults with cancer. Validation analyses by experts revealed that the content, appearance, language, motivation, and cultural aspects of the booklet developed by the ONCOV-19 study group are suitable for application in individual and collective health interventions. For the thematic analysis of the dissertation proposals, the profile of the evaluators contributed significantly to maximizing the quality of the educational technology. The Metalinguistic Thematic Analysis for Educational Technologies (MeTA-Edu) is a new analytical approach to speed up the validation process during crises.
What do these findings mean?
Considering the relevance of experts’ input for improving the booklet’s suitability, our findings indicate that qualitative methodologies are essential for validating educational technologies in health and can optimize the process, especially for risk communication. Thus, our study suggests a specific professional profile for the expert panel and a simplified analytical approach Metalinguistic Thematic Analysis for Educational Technologies (MeTA-Edu) to optimize the validation of health education technologies.
Highlights
Cancer patients are at increased risk of severe complications from COVID-19. They have been directly affected by heightened awareness of their fragility, disruptions to the healthcare system, and social isolation during the pandemic.
The massive flow of COVID-19 information and misinformation mitigates appropriate behaviors to cope with risk proactively, increasing the need for validated educational technologies during public health emergencies.
Expert validation of educational technologies with Content Validity Index, Suitability Assessment of Materials, and proposals analysis is an efficient methodology during a health system crisis.
The methodology of expert proposal analysis has been variably reported in the scientific literature.
The Metalinguistic Thematic Analysis for Educational Technologies (MeTA-Edu) is a new analytical approach to optimize the validation of health education technologies.
INTRODUCTION
With approximately 19 million new cases annually, more than 50 million people live with cancer worldwide1 . However, developing more accurate diagnostic methods and treatments increased survival in this population, and communicable diseases have become the leading cause of death among adults with cancer2 .
Among other risk factors, myelosuppressive therapies, impaired cellular immunity associated with the tumor, and cachexia contribute to infection susceptibility of cancer patients3 , including Coronavirus Disease 2019 (COVID-19). One of the earliest studies showing the effect of COVID-19 on cancer patients was conducted in China and established a worse prognosis for such patients, reporting a severe events frequency of 39%, while patients without cancer had only 8%4 . Later, a meta-analysis concluded that the mortality rate in cancer carriers with COVID-19 reached 7.6%, substantially higher than in those without comorbidities (1.4%)5 .
In Brazil, the excess mortality rate in cancer carriers drastically increased (82.1%) between March and December 2020 compared to 2019, although there was a reduction in observed deaths having cancer as the underlying cause (-9.6%)6 . In addition, the Brazilian Unified Health System (SUS) underwent significant mitigation of the provision of cancer screening, diagnosis, treatment, and hospitalization services between December 2019 and February 20227 .
Besides their clinical vulnerability, these patients are directly affected by heightened awareness of their fragility, disruptions to the healthcare system, and social isolation during health emergencies of international concern8 . With the increase in telehealth and lay medical websites providing remote healthcare as precautious measures, more than 90% of cancer patients access information online to make health decisions9 . However, more than 3,000 studies refute the unsuitability of health education materials10 .
In this scenario, the massive flow of COVID-19 information, misinformation, and disinformation mitigates appropriate behaviors to cope with risk proactively11 . Due to the positive effects of knowledge on patients and the public’s quality of life12 , validation of educational technologies (ETs) is essential to establishing their adequacy and reliability during public health emergencies15 . Hence, this study assessed the validity and reliability of an ET on COVID-19 prevention to promote and protect the health of adults with cancer.
METHODS
Location and Study Period
Data sampling and collection were conducted in a virtual environment from December 2021 to January 2022.
Study Population and Eligibility Criteria
The sampling was performed by convenience and snowball techniques type and comprised 22 experts16 , 17 . The sample size was calculated from the formula: n= Zα2 * P (1-P)/d2, where: Zα2 = 1,96; P = 0,85; d= 0,15. The sample size of this study reflects two central statistical criteria: the minimum proportion of 85% agreement concerning the pertinence of each component evaluated and the difference of 15% in agreement, including an interval of 70% to 100% of the same18 .
For convenience sampling, an initial search for Ph.D. holders in fields of interest, Health Sciences, Educational Technologies, and Cancerology, was conducted in the national curriculum vitae database, verified by the Department of Federal Police. The inclusion criteria consisted of being a Brazilian resident, having a valid e-mail address, a doctorate in one of the fields of interest, and adequacy in at least two of the selection criteria: (a) skills acquired by professional experience, (b) specialized knowledge that makes the professional an authority in the subject, (c) particular skill in one of the specific categories of the study, (d) high classification assigned by an authority, and (e) approval in a specific test to identify experts19 .
A new methodology was developed to analyze expert proposals based on the language type and function, the metalinguistic thematic analysis for ETs (MeTA-Edu). The creation of MeTA-Edu has foundations on existing methodological guidelines20 , 21 .
Subsequently, the Electronic Booklet Analysis Protocol included the content validation instruments for measuring the Content Validity Index (CVI) and Suitability Assessment of Materials (SAM).
Data Collection
Data were collected virtually through the Questionnaire for the Characterization of Experts and the Protocol for Analysis of the ET. The Expert’s Characterization Questionnaire included questions related to demographic variables, professional experience (education, work, teaching experience), and Jasper’s selection criteria.
Subsequently, the Digital Booklet Analysis Protocol included the content validation instruments for measuring the Content Validity Index (CVI) and Suitability Assessment of Materials (SAM). The CVI evaluates Language Clarity (LC), Theoretical Relevance (TR), and Practical Pertinence (PP), while SAM, content, literacy demand, interface (graphics, layout, and typography), learning motivation, and cultural suitability (CS).
A new methodology was developed for the analysis of dissertation proposals based on the pedagogical language, the metalinguistic thematic analysis for ETs (MeTA-Edu). The creation of MeTA-Edu has foundations on existing methodological guidelines20 , 21 .
Data Analysis
The CVI was used for the segmented analysis of the ET. The CVI measures the agreement between experts on a particular aspect of the instrument at the item (I-CVI) and scale (S-CVI) levels 22. I-CVI was calculated through the sum of ratings “4” or “5” on the Likert scale divided by the total number of ratings assigned16 .
The S-CVI consisted of the sum of all I-CVI calculated separately divided by the number of items in the instrument. Items with an I-CVI ≥ 0·78 and S-CVI ≥ 0·80 were validated. The intraclass correlation coefficient (ICC) was measured at a 95% confidence interval (CI 95%)23 .
S-CVI and SAM score calculations were performed for a comprehensive assessment. SAM constituted the percentage corresponding to each aspect assessed, considering “Superior” if the score was between 70% and 100%24 .
To analyze expert suggestions, proposals were discriminated, categorized using MeTA-Edu, and accepted or rejected, with justification
Figure 1 outlines the MeTA-Edu in two primary independent categorical levels based on the type and language function. The language type dimensions were visual, written, verbal, or mixed, while the language function dimensions were Informative (comprising LC and TR) and Motivational (comprising PP and CS).
Ethical and Legal Aspects
Participants were informed about the research objectives and requested informed consent according to Circular Letter 1/2021 issued by the National Research Ethics Commission on Guidelines for Procedures in Research with Any Stage in a virtual environment25 . The Research Ethics Committee approved the study under protocol number CAE: 54635916.7.0000.5054, according to the Brazilian National Health Council Resolution 466/201226 .
RESULTS
One hundred twenty-three experts were invited via e-mail, of which 36 agreed to participate in the research. After curricular verification via CL, 30 candidates met the study’s inclusion criteria and were ranked according to Jasper’s criteria. Thus, the 22 first classified were selected to compose the evaluation panel.
The demographic and academic profile of the 22 experts selected for the ET validation committee is shown in Table 1 .
Table 1 : Characterization of the expert panel.
| Variables | n | % |
|---|---|---|
| Sex | ||
| Female | 12 | 54.5 |
| Male | 10 | 45.5 |
| Region | ||
| Amazon | 2 | 9.1 |
| South Central | 15 | 68.2 |
| Northeast | 5 | 22.7 |
| Age | ||
| Mean | 48.32 | |
| Standard deviation | 8.85 | |
| Minimum | 33 | |
| Maximum | 68 | |
| Training areas | ||
| Nursing | 6 | 27.3 |
| Physical Therapy | 5 | 22.7 |
| Medicine | 5 | 22.7 |
| Nutrition | 2 | 9.1 |
| Other | 4 | 18.2 |
| Academic Degree | ||
| Doctorate | 22 | 100.0 |
| Postdoctoral | 2 | 9.1 |
| Areas of Expertise | ||
| Health Education | 19 | 86.4 |
| Cancer Care | 7 | 31.8 |
| Educational Technologies | 5 | 22.7 |
| Teaching experience time (in years) | ||
| Mean | 17.82 | |
| Standard deviation | 8.17 | |
| Minimum | 2 | |
| Maximum | 34 |
Table 2 shows the profile of the selected experts according to classification analysis using Jasper’s criteria.
Table 2 : Characterization of the expert panel according to Jasper's criteria.
| Selection Criteria for Experts | No | Yes | ||
|---|---|---|---|---|
| n | % | n | % | |
| 1. Skills or knowledge acquired by experience | - | 0.0 | 22 | 100.0 |
| More than ten years of professional teaching experience in one of the fields*. | 4 | 18.2 | 18 | 81.8 |
| Experience in carrying out individual and collective health education activities. | - | 0.0 | 22 | 100.0 |
| 2. Skills or specialized knowledge that grants professional authority in the fields of interest* | - | 0.0 | 22 | 100.0 |
| Participation in round tables or as a guest speaker at a national or international scientific event in one of the fields*. | - | 0.0 | 22 | 100.0 |
| Academic supervision of undergraduate studies with a theme related to one of the fields*. | - | 0.0 | 22 | 100.0 |
| Academic supervision of stricto sensu graduate studies (master or doctorate) with a theme related to one of the fields*. | 8 | 36.4 | 14 | 63.6 |
| Doctor's degree, with the thesis in a theme related to the fields*. | 13 | 59.1 | 9 | 40.9 |
| 3. Special skills in a specific category of study | - | 0.0 | 22 | 100.0 |
| Experience in developing scientific research in one of the fields*. | - | 0.0 | 22 | 100.0 |
| Authorship in a scientific article with themes related to one of the fields* in an indexed journal. | - | 0.0 | 22 | 100.0 |
| Participation in academic evaluation panels of undergraduate work with a theme related to one of the fields*. | - | 0.0 | 22 | 100.0 |
| Participation in academic evaluation panels of stricto sensu (master or doctorate) work with a theme related to one of the fields*. | 2 | 9.1 | 20 | 90.9 |
| 4. High rating assigned by an authority | 11 | 50.0 | 11 | 50.0 |
| Recipient of tribute, honorable mention, or recognition as an authority in the fields from a recognized scientific institution. | 14 | 63.6 | 8 | 36.4 |
| Recipient of an award in a national or international scientific event of which content refers to the fields of interest*. | 15 | 68.2 | 7 | 31.8 |
* Research fields of interest: Health Sciences; Cancerology; Educational Technologies.
Content Validation
The CVI presented values above 0.900 for LC (0.950), PP (0.953), TR (0.955), and the global CVI (0.953), demonstrating excellence in terms of the validity of the ET content (Suppl. 1). The high I-CVI, SCI/AVE, and S-CVI scores for LC and PP for 30 items and TR for 24 showed that most experts rated the items as “much” and “very much” recommended. In addition, the ICC was 0.871 for LC, 0.908 for PP, and 0.863 for TR. In addition, an overall analysis of 0.958 evidenced high agreement among the experts.
Suitability Validation
As for the convenience of the content, literacy demand, interface, learning motivation, and CS measured by the SAM instrument, it was observed that 90.6% of the answers were classified as “Superior,” evidencing that, in this regard, the booklet has high standards of adequacy ( Table 3 ).
Table 3 : Global analysis of educational technology through the suitability assessment of materials (SAM).
| SAM Criteria | Not Suitable (0) | Adequate (1) | Superior (2) | |||
|---|---|---|---|---|---|---|
| n | % | n | % | n | % | |
| The objective is evident, facilitating the prompt understanding of the material. | - | 0.0 | - | 0.0 | 22 | 100.0 |
| The content addresses information related to healthy behaviors that favor health-promoting actions. | 1 | 4.5 | - | 0.0 | 21 | 95.5 |
| The material proposal is aligned with the objectives so that the viewer can reasonably understand it in the time allowed. | - | 0.0 | 3 | 13.6 | 19 | 86.4 |
| The reading level is suitable for the target audience. | - | 0.0 | 3 | 13.6 | 19 | 86.4 |
| The conversation style makes it easy to understand the text. | - | 0.0 | 1 | 4.5 | 21 | 95.5 |
| Vocabulary uses common words. | - | 0.0 | 3 | 13.6 | 19 | 86.4 |
| The cover attracts attention and depicts the purpose of the material. | 1 | 4.5 | 3 | 13.6 | 18 | 81.8 |
| Illustrations present fundamental visual messages so that the reader understands the main ideas alone, without distractions. | - | 0.0 | 3 | 13.6 | 19 | 86.4 |
| Text or figures interact with the reader, leading them to solve problems, make choices, or demonstrate skills. | - | 0.0 | 3 | 13.6 | 19 | 86.4 |
| The desired behavior patterns are modeled or well demonstrated. | - | 0.0 | 1 | 4.5 | 21 | 95.5 |
| There is the motivation for developing healthy attitudes. People are motivated to learn by believing that tasks and behaviors are feasible. | 1 | 4.5 | 1 | 4.5 | 20 | 90.9 |
| The material is culturally suited to the target audience's logic, language, and experience. | - | 0.0 | 1 | 4.5 | 21 | 95.5 |
| It presents culturally appropriate images and examples. | - | 0.0 | 2 | 9.1 | 20 | 90.9 |
| Total/Mean | 3 | 1.0 | 24 | 8.4 | 259 | 90.6 |
* Research fields of interest: Health Sciences; Cancerology; Educational Technologies.
Metalinguistic Thematic Analysis (MeTA)
During the text analysis, 94 expert propositions were discriminated from the 96 recorded comments for the 28 evaluated items (14 pre-textual, 23 textual, and one post-textual). The MeTA-Edu categorical approach revealed that most of the suggestions were visual language type (47.9%), corresponding to appearance, and the informative language function, corresponding to the knowledge transfer stage of the learning process (68.1%), especially LC ( Table 4 ).
Table 4 : Validation Model for Metalinguistic Thematic Analysis of Educational Technologies (MeTA-Edu).
| Language Function | Visual | Writing | Mixed | TOTAL | ||||
|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | n | % | |
| Informative or Referential | 27 | 28.7 | 20 | 21.3 | 17 | 18.1 | 64 | 68.1 |
| Language Clarity | 26 | 27.7 | 16 | 17.0 | 8 | 8.5 | 50 | 53. 2 |
| Theoretical Relevance | 1 | 1.0 | 4 | 4.3 | 9 | 9.6 | 14 | 14. 9 |
| Motivational or Appealing | 18 | 19.2 | 5 | 5.3 | 7 | 7.4 | 30 | 31.9 |
| Practical Pertinence | 15 | 16.0 | 4 | 4.3 | 7 | 7.4 | 26 | 27.7 |
| Cultural Suitability | 3 | 3.2 | 1 | 1.0 | - | 0.0 | 4 | 4.2 |
| TOTAL | 45 | 47.9 | 25 | 26.6 | 24 | 25.5 | 94 | 100.0 |
Table 5 demonstrates the level of agreement between experts and independent researchers by language type and function. Furthermore, the frequency of propositions by category compared to the distribution by textual section of educational technology.
Table 5 : Evaluation of the propositions of experts by language type and function.
| Decision | TYPE | FUNCTION | Total | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Visual | Writing | Mixed | Informative | Motivational | ||||||||
| n | % | n | % | n | % | n | % | n | % | n | % | |
| Accepted | 41 | 43.6 | 21 | 22.3 | 16 | 17.0 | 55 | 58.5 | 23 | 24.5 | 78 | 83.0 |
| Rejected | 4 | 4.3 | 4 | 4.3 | 8 | 8.5 | 9 | 9.6 | 7 | 7.4 | 16 | 17.0 |
| TOTAL | 45 | 47.9 | 25 | 26.6 | 24 | 25.5 | 64 | 68.1 | 30 | 31.9 | 94 | 100.0 |
Lastly, Figure 2 illustrates the booklet cover for the original language, Portuguese, and translated versions in English, Spanish, and Simplified Chinese.
Figure 2 : Electronic booklet covers in (A) Portuguese, (B) English, (C) Spanish, and (D) Simplified Chinese.
The illustrations were sketched and finished using digital painting in the Adobe Photoshop program and diagramming, in the Adobe InDesign program , by a professional with experience in diagramming and graphic design of teaching materials.
The final version of the booklet was entitled “Uncomplicating COVID-19 for people with cancer” ( Figure 2 ) and consisted of 31 pages in half A5 page format (14.8 wide and 21.0 high), configured in landscape orientation (Suppl. 2).
The content was organized into six chapters that answered the following questions: “I. What do I need to know about COVID-19?”; “II. What are the COVID-19 risks for cancer carriers?”; “III. How to distinguish between COVID-19 and cancer symptoms?”; “IV. What should I do if I suspect I have COVID-19?”; “V. How do vaccines work?”; “VI. How can I stay healthy during a pandemic?”. The questions were elaborated based on an online survey inquiring about the primary health information needs and related seeking behavior of cancer patients treated at Brazilian oncology services during the COVID-19 pandemic with patients and healthcare professionals (Suppl. 3).
DISCUSSION
The expert committee confirmed the validity of the electronic booklet “Uncomplicating COVID-19 for people with cancer” regarding content, literacy demand, interface, learning motivation, and CS with a high degree of agreement. The results suggest the excellent suitability of the ET for application in health promotion interventions for adults with cancer during public health emergencies.
On the other hand, most informational materials about COVID-19 for cancer patients available on the internet are inadequate. Recently, an analysis of information on COVID-19 and cancer for cancer patients revealed that only 37 of the 398 accessible online sources in English listed by Google, Yippy, and Dogpile addressed COVID-19 in the context of a cancer diagnosis or treatment. None of these sites had a recommended reading level for public health communication (< 6.0 on the Flesch-Kincaid scale), and only 24% cited references27 .
In this context, the contribution of the Pan American Health Organization (PAHO) stands out, which published a digital information sheet on COVID-19 aimed at people with cancer in English during the first months of the pandemic28 . However, despite the document going through an internal validation system to be accepted in their Institutional Repository for Information Sharing (IRIS), the methodological description is not accessible within the publication.
Quantitative validation
The content assessment regarding LC, PP, and TR presented excellent scores per item and scale (universal S-CVI = 0.952). Likewise, Bana et al. (2020) developed and validated 16 information leaflets as part of an educational intervention on symptom self-management for cancer patients29 . The assessment panel, involving 48 patients and health professionals, evaluated the general I-CVI at 0.95 and 0.9 for the French and German versions, respectively30 .
The evaluation results by experts revealed a high degree of adequacy for the content, literacy demand, and graphics, as well as for learning motivation and cultural adequacy. Overall, the calculated SAM (90.6%) was excellent (>90%)24 . The TE achieved a complete evaluation regarding the objectives and a minimum evaluation regarding the cover theme and critical literacy.
Previous validation studies on ETs for cancer patients mostly showed unsatisfactory results, especially regarding literacy demand 31 . For example, a SAM analysis of printed and digital teaching materials on cancer reveals that most do not have minimum criteria for reading level and cultural adequacy, considering adequate only two of the seven materials evaluated25 .
Qualitative validation
In this study, the analysis of expert proposals assumed a central role in ET validation. Despite being approved by the quantitative methods, the accepted recommendations significantly expanded the scientific-pedagogical quality of the booklet, mainly in terms of interface, literacy demand, and motivation.
Interface Suitability: Simplifying Drawings
Studies show that a sequence of simple, realistic images with a brief explanation in a large sans-serif font (> 14) is best understood by people with low literacy levels who may not see any connection between the sequential images32 . Therefore, the design has been simplified and generally reorganized to make it easier for the lay public to understand without being distracted by irrelevant details (Suppl. 4, Page 7: E6 and E8).
Written Suitability: Clear and Objective Language
Plain language composed of short sentences, sometimes followed by brief definitions, should preferably be used by adults with low and medium literacy. Developing written digital content in plain language means the public will understand the information on their first reading33 .
Recent studies on available information for cancer patient education indicate that the reading level does not correspond to the required instruction level of the target audience, highlighting the importance of using concrete words, definitions, examples, and clear instructions for preventive measures34 . Consequently, the research team carefully selected the recommended websites according to content reliability, excluding those reported inappropriately in the literature (Suppl. 4, Page 28: E6 and E19).
Informative Suitability: Omission of Information
Informed health decision-making demonstrates several benefits, such as increased knowledge, reduced anxiety, and better therapeutic prognosis35 . Thus, relevant content was restructured, using language as a facilitating resource for adults’ participatory educational approach, but not omitted (Suppl. 4, Page 17: E6 and E8).
It was intended to escape from a traditional model of education in which there is no stimulus for knowledge creation13 . In this model, the educator is considered a superior being who teaches when considered ignorant, disabling him.
On the contrary, we sought to recognize the reader as a subject capable of developing a thoughtful criticism of their reality, enabling them to make decisions regarding individual and community health. Thus, based on Freire’s principles of popular health education, this study adopted health literacy as a transformative tool for promoting lasting healthy behavior and attitudes36 .
Motivational Suitability: Self-Management and Self-Efficacy
MeTA-Edu stood out for the significant number of expert suggestions for the motivational dimension (Suppl. 4; Page 6: E22; Page 13: E7). Thus, creating strategic educational content for social engagement is essential for an effective lifestyle change for the target audience. Engagement with a digital health education intervention is a precondition for its effectiveness/in achieving the intended outcomes37 .
Limitations
Due to the restrictions imposed by the pandemic, data collection took place virtually. Despite having enabled the simultaneous collection of data and the reduction of observer bias, this possibly interfered with the recruitment of candidates with low digital literacy and audiovisual deficiencies, the interpretation of the questions, the motivation for answering essay questions, and the interpretation of qualitative data.
Thus, we sought to minimize these effects by elaborating an invitation message explaining the objectives and importance of the research to the target audience. As questions and answers channel and accessibility tools, a telephone number, electronic address contacts, and automatic narration and audio-response recording options were provided, respectively.
Strengths
This pioneering research presents a novelty in the methodological design of ETs to promote and protect the health of populations facing a more significant risk of death during pandemics. The research topic is very current and will raise international discussion among professionals and researchers investigating health education and risk communication related to public health emergencies.
Considering the relevance of experts’ input for improving the booklet’s suitability, our findings indicate that qualitative methodologies are essential for validating ETs in health and can optimize the process, especially for risk communication. Thus, our study suggests a specific professional profile for the expert panel and a simplified analytical approach (MeTA-Edu) to maximize the efficiency (quality and speed) of ET validation for response and preparedness to health emergencies of international concern.
CONCLUSION
The expert evaluation confirmed the validity and reliability of the electronic booklet “Uncomplicating COVID-19 for people with cancer” concerning the content, literacy demand, interface, motivation, and CS for application in preventative interventions for adults with cancer. In addition, the expert committee profile with extensive teaching experience and specialization in education seems to have contributed to the relevance of dissertation proposals.
Thus, a systematic approach of dissertate analysis for ETs, MeTA-Edu, was developed and successfully applied to optimize the current study. More validation studies of ETs in communication crises are needed to refute these findings. The implications of health interventions with the booklet “Uncomplicating COVID-19 for people with cancer” should not only benefit cancer carriers who face more significant life-threatening risks but also support new methods of validating ETs.
