ORIGINAL RESEARCH

Evaluation of intervention programs with adolescents: limits, progress and prospects

Fabio Scorsolini-Comin^I; Alana Batistuta Manzi-Oliveira^II; Karin Aparecida Casarini^III; Roberta Cury Jacquemin^IV; Manoel Antônio dos Santos^V

^IProfessor do Departamento de Psicologia do Desenvolvimento, da Educação e do Trabalho da Universidade Federaldo Triângulo Mineiro. E-mail: scorsolini_usp@yahoo.com.br
^IIPsicóloga e mestranda em Psicologia pela Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto da Universidadede São Paulo. E-mail: alana.manzi@gmail.com
^IIIProfessora do Departamento de Psicologia da Saúde e Processos Básicos da Universidade Federal do TriânguloMineiro. E-mail: kacasarini@yahoo.com.br
^IVPsicóloga e mestranda em Psicologia da Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto da Universidadede São Paulo. E-mail: betacury@hotmail.com
^VProfessor Associado do Programa de Pós-graduação em Psicologia da Faculdade de Filosofia, Ciências e Letras deRibeirão Preto da Universidade de São Paulo. E-mail: masantos@ffclrp.usp.br

Correspondência para

ABSTRACT

The aim of this study is to describe the state of the art of the evaluation of intervention programs with adolescents. Through a literature review, six articles published between 2008 and 2010, were retrieved. Of these, four were empirical, one theoretical and one a literature review, with the majority of the publications by North American authors. The studies address instruments and methods used to evaluate programs and seek a connection between research and practice. According to the methodological approach, they were predominantly qualitative research. The studies indicate an interest linked to the development and refinement of interventions with adolescents and the sophistication of a critical discussion of the description studies of efficacy indicators. Among the contributions, there is the consideration of the context for evaluation research and its potential relationship with the improvement of the intervention programs. The results can be seen as guidelines for the professionals of the field, promoting the construction of practices and knowledge regarding the role of the evaluator.

Key words: evaluation; programs; adolescents; literature review.

INTRODUCTION

Adolescence is a developmental stage commonly characterized by extremes and excesses, but also by the construction of solid foundations for the adult personality, in a period marked by crisis and physical, psychological and social transformations. It is in the transition of this evolutionary stage that the individual will acquire maturational conditions to develop their social role, internalizing values, attitudes, beliefs, principles and habits that will be organized and assumed by the adolescent, serving as a foundation for the consolidation of their psychosocial development process^1-3.

Adolescence has attracted growing interest from the contemporary media, such as, social networks, virtual communities and social networking websites and is being increasingly contemplated in public policies of health and education. Adolescence can be considered a period in which some individuals become vulnerable to risk factors and psychosocial stressors present in their life context because this period includes physical, psychological and social changes. There is a greater proximity regarding the peer group, who increase their field of influence over the adolescent, and the experience of new situations without being prepared to deal with these situations in a constructive way⁴. In adolescence it is also common to engage in risky behavior and violation of the law, related to the developmental tasks of seeking increased autonomy and self-regulation⁵. Therefore, there is space for the intervention of education and health professionals, among whom the psychologist is highlighted as an important agent of mediation in this context of human development.

Based on this concept, several programs, aimed at promoting the health, welfare and education of adolescents, have been proposed as strategies in Brazil and also in other countries^6-8. These programs are aimed at not only meeting the needs of a significant portion of the population, but also allowing the assumption of adaptive and healthy practices in individuals who are in the transition to the adult phase, especially those subjected to risk situations and ill-treatment. The proposition of such programs brings with it the need to evaluate them in order to verify the achievement of their aims and also to provide a comprehensive view of their results at different stages of their implementation, in order to improve and optimize the resources applied. The sustainability of the intervention programs should be considered in operational terms and also in relation to the practices developed, employed and negotiated for, by and between the participants and the professionals involved^9-10.

In one of the most promising approaches, the evaluation of intervention programs with children and adolescents is addressed from the perspective that comprehends the evaluation process not only as a dimension of judgment, but also as a measure of the practices and constitutive dimensions of the socio-educative or psycho-educative actions⁶. From this perspective, the evaluation is not something that happens after the event, but is closely linked to any intervention process in all its phases. It should therefore be used in the diagnostic stage, in the planning, in the intervention itself and, subsequently, in a longitudinal follow-up.

Intervention programs with adolescents must show clear attempts to improve the skills, abilities and attitudes of these actors, their relationship with other young people and their contribution to the community to which they belong, acting as protective and supportive factors for healthy development¹¹. To detect the scope of these programs in an adequate manner, it is necessary to implement a contextualized and delineated evaluation process, in a clear and precise way, which problematizes the reality analyzed. Given these considerations, it could be asked: how has the evaluation of intervention programs with adolescents been taking place, according to the important scientific literature? Therefore, the aim of this study is to describe the state of the art of the evaluation of intervention programs with adolescents.

METHODS

This is an integrative literature review, which seeks to summarize research findings and to draw overall conclusions from a body of literature on a particular topic in order to contribute to discussions regarding research methods and results, as well as to provide reflections on the implementation of future investigations¹². Following specific procedures¹³, such as the establishment of criteria for inclusion and exclusion of the studies and the critical analysis of the results, it is observed that, although the methods for conducting integrative reviews vary, there are standards to be followed.

It is therefore necessary to follow standards of rigor and clarity in the review, so the reader can identify the main characteristics of the studies reviewed. In the performance of this review, the steps followed were: (a) selection of the thematic question; (b) choice of the database/journal; (c) establishment of the criteria for selection/exclusion of the sample; (d) analysis and interpretation of the results; (e) presentation of the review.

Thus, the following national and Latin American coverage indexing databases were searched: PePSIC, SciELO and LILACS. The organizations responsible for these databases are, respectively, the Brazilian Association of Scientific Editors in Psychology (ABECiPsi), the Latin American and Caribbean Center on Health Sciences Information (BIREME) and the São Paulo Research Foundation (FAPESP).

The consultation of such databases was not found to be appropriate for the retrieval of a significant proportion of the international studies, since, in a previous search, only a few articles, predominantly of theoretical work, were found, many of which were review studies as opposed to empirical research. As the aim of this study was to discuss not only recent studies but also those with the greatest impact in the area of evaluation, it was chosen to conduct a search using an international database.

Due to the volume of international publications found, it was decided to select an international journal prominent in the area of evaluation, as a strategy to gain access to recent publications related to the theme investigated. The journal selected was the American Journal of Evaluation - AJE (<http://aje.sagepub.com>). This journal explores the complex challenges of the area of evaluation from research reports derived from specific studies on the methods, theory and practice of evaluation.

The field of evaluation is diverse, with different intellectual traditions that are reflected in the multiplicity of methods, research strategies, approaches to the practice and application domains. Considering this, the journal selected publishes studies originating from different theoretical, multidisciplinary perspectives, relevant to education, public administration, behavioral sciences, services, health research, sociology, economics and criminology. Some of the critical themes covered in recent issues include: approaches to collaborative evaluation, comprehensive evaluation of community initiatives, development of evaluation capabilities, ethics evaluation, health and education evaluation, implementation of the evaluation process, international development, validation and reliability of measurement, methods of narrative evaluation, and statistical methods, among other themes of interest.

The AJE is published by the American Evaluation Association, which is an international professional organization of evaluators devoted to the application and exploration of the evaluation of programs, personnel, technology and other forms of measurement. The area of evaluation includes the strong and weak points of programs, personnel policies, products and organizations, aiming to improve their efficacy through improved practices and methods of evaluation, of the promotion of evaluation as a profession, and of the support and contribution of evaluation for the generation of theory and knowledge about effective human action.

The AJE contains special sections, such as: (A) ethical challenges, with the aim of stimulating discussions by professionals from various fields concerning how evaluators might respond to certain ethical dilemmas that are common in the practice of evaluation; (b) behind the scenes interviews with the evaluators whose work illustrates the application of different models, theories and principles; (c) debates on intriguing issues at the forefront of evaluation; (d) essays on the philosophical, ethical and practical dilemmas of evaluation as a profession; (e) reports on new or improved evaluation methods, techniques, tools, products and/or services; (f) practical case studies on the teaching of evaluation and training in diverse environments, including traditional academic, corporate and government environments, as well as nonprofit institutions and community contexts.

As inclusion and exclusion criteria, in order to restrict the search to only publications submitted to a rigorous methodological evaluation, only indexed articles were selected. This criterion was already covered in the choice of the database, since the AJE is an indexed international journal with a selective editorial policy. Studies that addressed issues distant from the theme and unrelated to interventions with adolescents were excluded. Regarding the language of publication, there was no restriction, although this journal allows retrieval of predominantly English language articles.

The search comprised the period from January 2008 to January 2010. This period was chosen in order to provide only recent publications in the area. Abstracts consistent with the adopted criteria were selected and from this preliminary search the complete studies were retrieved, as described below.

The search was conducted in May 2010 from the official webpage of the AJE. The electronic searches were conducted with the use of the descriptors "evaluation", "program", "adolescents" and the Boolean operator "and". In the search using the first two terms, 113 records were recovered. By adding the third term (adolescents), 10 records were recovered. This strategy was preferred, taking into consideration that it enables the identification of references that present the three descriptors.

The databases were configured to locate the references that presented the descriptors within the keywords and/or abstract. This procedure was adopted to allow more accurate result to be obtained than would be achieved without specifying the search fields¹⁴.

The option for the use of these descriptors was due to the fact that the three selected are recognized by national and international databases, as well as the database of the journal consulted, furthermore they are being used on an ongoing basis in the specialist scientific literature.

Data analysis

First, the abstracts of the 10 articles, located from the descriptors used, were read and analyzed in terms of their adherence to the theme investigated. Using the inclusion/exclusion criteria as a guide, three of these articles were excluded due to their distance from the theme: one article was dedicated to cultural competence evaluation in the context of an HIV/AIDS intervention program, one study was on ethical dilemmas in the evaluation of indigenous populations and, finally, one study covered the construction of a database regarding people affected by trauma. None of these articles fitted into the theme of the evaluation of intervention programs with adolescents.

Therefore, six articles were selected for further reading and full analysis. These publications were retrieved, analyzed in depth and categorized based on the following axes of analysis: (a) characterization of the studies; (b) aims; (c) methods employed; (d) theoretical frameworks used; (e) main results; (f) limits, trends and progress. Subsequently, the articles were discussed within each axis, seeking the points of intersection, the disagreements, and especially the contributions of each to the production of important knowledge in the area.

RESULTS

Table 1 summarizes the articles selected in terms of their country of origin, date of publication and identification. As can be seen, of the six articles retrieved, four are empirical, one theoretical and one a literature review. The majority of the publications are of North American origin, being four from the United States and one from Canada. The other publication is from Israel. As regards the year, two studies were published in 2008 and four in 2009.

DISCUSSION

As can be seen in Table 2, the first article¹⁵ recovered showed that, after disclosure of the list of programs approved by the government, specifying the disapproval of the D.A.R.E. program, six districts no longer maintained them as drug prevention programs used in the schools This number increased after two years, rising to eight, and a significant reduction was presented in six other districts. The reasons highlighted for the abandonment of the programs by the agents interviewed were the governmental demands for the use of approved programs in order to obtain funding, highlighting the criteria used by them in the judgment of the evaluation studies and in the decision to adopt certain intervention programs as privileged strategies. A more detailed examination of the evaluation studies of the programs revealed that they contained many methodological biases, compromising the validity of the results generated.

These biases included: (a) the inclusion of studies conducted by the authors of the programs as references for the their efficiency; (b) limited evidence of the positive results, without the discussion of shortcomings or failures; (c) lack of evaluation mechanisms regarding the adherence to the proposed implementation of the program (correspondence between the planning and the actions actually performed); ( d) inadequate comparisons between groups for validation of the results; (e) absence of long-term follow-up studies, among others. The article also points out that the criteria for inclusion of a program on the approved list were weak, since they did not consider the negative results and they were not applied in a consistent and homogeneous manner to the different programs. This situation may have contributed to the low confidence of the respondents in the evaluation studies¹⁵.

Concerning the second article¹⁶, which sought to verify the psychometric properties (reliability) of an intervention evaluation protocol (GAS), the data indicated high concordance between student respondents of a Social Work course in relation to their judgments regarding the case described (evaluation by judges method). This study indicates that the subjective clinical impressions of the 43 students in the use of the GAS obtained statistically significant scores, suggesting the reliability of the instrument¹⁶.

In the third article¹⁷, the authors sought, by means of a literature review, to examine fundamental aspects of the methodological warrants of the evaluation theories and their practices. The authors found that the majority of the studies developed in the area (38) were centered on the characteristics of the evaluations. These studies also sought characteristics of the evaluators or of the evaluation contexts, as well as aspects related to the use of the evaluation results or, in some cases, the relationship of the results to the decision-making process of the program. Twelve of these studies examined other issues of a more general character. Finally, five studies examined the level or the extent of use of the evaluation results. It should be noted that the majority of the studies related to the area of education.

The results of the fourth study¹⁸ retrieved indicated that the nutritionists were concerned with three primary aims during the performance of the workshop on nutrition: (a) involvement of the students regarding their future health; (b) optimization of the functioning of the workshop on nutrition; and (c) creation of conditions to ensure the long-term stability of the intervention in the school, which could be evidence from an adequate evaluation.

The fifth article¹⁹ showed that, among the factors that differentiate the programs that maintained their sustainability from those that failed are: multiple finance funds, programs linked to NGOs, managers who invested in the support of the organizations, community mobilization to raise funds, programs that were seen as priorities within the organizations with efforts by these organizations to ensure the sustainability of the programs, articulation with other programs within the organization, investment in marketing targeted toward funding agencies and networks of relationships with potential donors and financiers, and close relationship with the local and extended community. The study highlights the fundamental role of the human factor in the organizations, especially regarding the managers, who, in the case of programs that maintained their sustainability, presented initiative capacity, flexibility and dedication to the pursuit of partnerships¹⁹.

Finally, the sixth article²⁰ described and explained models for integration between theory and practice, from application examples in the context of intervention programs. According to the authors, the evaluation and the planning of programs gave life to the efforts employed to carry out such integration.

Methodologies of expression of theories, of identification of needs and of dissemination of results, as well as the construction of scientific systems and Evidence-Based Practice (EBP), allow the identification of places where the evaluation efforts and research literature meet, illustrated by the authors through the metaphor of the "golden spike". This point synthesizes the connection between research and practice, the orientation of the network of pathways and the greatest potential of the evaluation. From this perspective, the evaluation is conceived as part of a dynamic process, which must occur throughout the whole period of the program.

Among the limitations, progress and future perspectives highlighted in the articles, it was noted that the use of case studies related to specific intervention programs could prevent the generalization of the findings to other realities and contexts of diverse application¹⁵. However, the study highlights the progress presented, in the sense that it affirms the importance of the context for the evaluation studies and its potential contribution for the improvement of the programs, considering the local characteristics of the population and of the resources invested, and the political and financial aspects involved, among other dimensions Other progress that may be mentioned refers to the critical discussion of the results of government funding policies, and of research designs and their impacts on the behavior of the professionals. Finally, it is evidenced how the harnessing of financing to the scientific approval of a program, detached from the social reality in which we live, does not promote greater fulfillment of the needs of the professionals responsible for implementing the program, nor of those of its users.

In another publication¹⁶, the authors consider that the scores obtained in the reliability study of the instrument may be higher than those that exist in practice for two reasons: (a) the judges were trained uniformly within a short period of time before the study; and (b) the evaluations were performed in relation to a pre-organized case, which did not allow access to the interpretations of the students. Although not prominent trends, the article puts forward the fact that the data can contribute to the use of an evaluation instrument composed of constructed criteria based on a collaborative relationship between professionals and clients. Another contribution of this study is to highlight the importance of training of the evaluators in the use of the instrument, which is crucial so that the evaluation can reflect a given reality investigated by the scientific research.

In the third article retrieved from databases¹⁷, the authors speculated that the methodological foundations used for the conclusions regarding the use of evaluation (and its instruments) are not as strong as is desirable. According to the authors, the existing studies cannot provide definitive evidence concerning the effects of the use, although they provide suggestions. Without further evidence of the psychometric quality of the data collection instruments, the validity of the results is still unknown. In relation to progress achieved, the evaluation professionals, seeking practical advice regarding the use of research results, can see the findings of the studies on the use of evaluation as evidence. In addition, the results of the studies on the use of the evaluation show that many variables can be useful for the evaluators, in terms of promoting the use of these techniques.

The fourth article¹⁸ presents as a limitation the fact that the authors provide a theoretical framework that helps to refocus and promote reflection on what a program is and what it becomes - that is, how it will develop and transform - during its execution. However, the authors recommend the realization of further studies to systematically define what happens in practice. The progress is related to the fact that the nutritionists have strategically translated components of the program as a means of negotiation with the participants and interested parties, which comes close to the work of the third article discussed here. The results of this study support the theoretical proposition that the execution of the program is a process of expansion of a socio-technical network.

The limitations raised by the fifth article¹⁹ highlight the need for more studies specifically focused on the sustainability of programs, working toward the elaboration of a theory of sustainability. The second phase of the study, not presented in this publication, will be based on quantitative data, contributing to a more complete and improved practice in relation to the evaluation of programs. The progress mentioned considers the sustainability as part of the evaluation of the programs, a recent trend clearly evolving in terms of scientific research. In addition, the study presents a review of the literature in the area, combining the results encountered with the studies already produced. It also systematizes into categories of analysis the factors associated with sustainability, which until then were grouped in the literature, without, however, dismissing their complexity and their dynamic character.

The sixth article reviewed²⁰ provides a comprehensive systematic model for the evaluation of programs. Regarding the trends explained, the article highlights that the task for the proponents who elaborate the program is to guide the range of the successive evaluations towards the points of the "golden spike", which may be related to predetermined research pathways. For the researchers, the challenge is to develop lines of research with higher levels of coverage, and ones that are able to unite medium-term results with greater and more adaptive effects for the target population. The challenge with regard to the evaluation is precisely to provide an environment and support systems to determine whether such connections are reasonable.

The appraisal of scientific production related to the evaluation of intervention programs directed towards adolescents, presented in the AJE over past two years, indicates an interest related to the development and improvement of interventions together with this population and a sophistication of the critical discussion of the studies that provide a description of efficacy indicators.

As already evidenced, adolescence constitutes a period of greater psychological vulnerability, which sheds light on the importance of developing preventive actions directed towards specific needs⁴, while the actions of health promotion seek a more general scope. The actions of prevention, when implemented in the form of intervention programs, should rely on evaluation mechanisms that can affirm their adequacy given the characteristics of the users.

In this context, it was observed that the articles identified in this review are predominantly directed towards the analysis of factors that can influence the accuracy of the findings of the intervention program evaluations. These factors include broader cultural and social aspects, which can interfere in the way the intervention programs reach different sections of the population, political and economic aspects that may favor the adoption of certain recommendations of the programs in a way absolutely unconnected to the reality experienced by the users (intervention for the sake of intervention), and methodological characteristics of the studies of evaluation, related to the form and to the criteria used for considering the efficiency or adequacy of the results.

The articles reviewed are directed toward the search for the refinement of evaluation strategies, affirming the need for rigorous and expanded scientific analysis, establishing relationships between diverse aspects of the social context, of the reality of the users, and of the academic world. They also stress the trend of the evaluations of intervention programs being constructed from a interlocution between users, professional implementers and researchers, in order to favor the approach of a knowledge of the real conditions of implementation of the intervention programs and needs of the users. This aspect confirms the indications derived from other studies^6,9-11 on the realization of evaluations, which include the consideration of immediate results and those of the medium and long term, providing information regarding the maintenance and evolution of interventions, to match planned objectives/achieved objectives/needs of the users, as well as regarding the adjustment of the social and educational measures and actions.

Therefore, attempts to construct longitudinal evaluation instruments are especially important, in order to encourage the expression of the perspective of the users, including the establishment of goals, forms of measurement/identification of the results and negotiation of the practices adopted. Furthermore, it also seems to be important to consider the territorial, social and phenomenal specificities involved in a given intervention program, as well as the challenge of finding ways to provide generalizability for the results encountered.

Some of the articles promote an analysis of how the evaluation process of the intervention programs can be used as a directing instrument for the results and for the policies related to the actions contemplated by the programs, serving the different interests. In this sense, studies on sustainability¹⁹ can contribute to confer greater impartiality to this process.

According to the literature review, one of the main points that should be highlighted concerns the role of the evaluator. Their importance should be emphasized, not only as diagnosticians of a reality and planners of an intervention, but as professionals capable of establishing an open dialogue between theoretical and practical knowledge. From the studies retrieved in the corpus of this study, this ability is seen as a necessity and also a trend in the overview of the evolution of the area. Thus, the evaluation would fulfill its role in the comprehension of a given reality and would also be a measure of practices and of ways of intervening in diverse social contexts, especially with the adolescent population, which is the scope of this study.

One of the limitations of this study is the short range of the review (the last two years), however, we highlight that, in a journal specific to the field of evaluation, with a very selective and constantly updated editorial policy, we could find reports representative of the practices and of the studies that have been developed in the international context. For future revisions and updates of the scientific literature, different methodological pathways can be chosen, with the adoption of other index databases of scientific journals, longer periods and using other descriptors, which obviously will make it possible to obtain results that lead to new perspectives regarding the area of evaluation of intervention programs in the context of adolescence.

REFERENCES

1. Aberastury A, Knobel M. Adolescência normal. Porto Alegre: Artes Médicas; 1981.         [ Links ]

2. Brêtas JRS, Moreno RS, Eugenio DS, Sala SCP, Vieira TF, Bruno PR. Os rituais de passagem segundo adolescentes. Acta Paulista de Enfermagem 2008; 21(3): 404-11.         [ Links ]

3. Zagury T. O adolescente por ele mesmo. 10ª ed. Rio de Janeiro: Record; 1999.         [ Links ]

4. Sapienza G, Pedromônico MR. Risco, proteção e resiliência no desenvolvimento da criança e do adolescente. Psicologia em Estudo 2005; 10(2): 209-16.         [ Links ]

5. Mun EY, Windle M, Schainker LS. A model-based cluster analysis approach to adolescent problem behaviors and young adult outcomes. Dev Psychopathol 2008; 20:291-318.         [ Links ]

6. Lacharité C. La dimension formative du bilan d'implantation: des suites d'un programme à l'état des acquis subjectivement perçus par les intervenants au programme. Dans Alain, M. et al., org. Élaborer et évaluer les programmes d'intervention psychosociale: une perspective transdisciplinaire. Québec: PUQ; 2009: 87-102.         [ Links ]

7. Muza GM, Costa MP. Elementos para a elaboração de um projeto de promoção à saúde e desenvolvimento dos adolescentes: o olhar dos adolescentes. Cadernos de Saúde Pública 2002; 18(1): 321-28.         [ Links ]

8. Suárez-Ojeda N, Silber TJ, Munist M. Servicios para la atención de salud del adolescente. In Silber TJ, Munist M, Maddaleno M, Suárez-Ojeda N, org. Manual de Medicina de la Adolescencia. Washington, DC: Organización Panamericana de la Salud; 1992: 601-29.         [ Links ]

9. Bledsoe KL, Graham A. The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation 2005; 26: 302-19.         [ Links ]

10. Faw L, Hogue A. Multidimensional implementation evaluation of a residential treatment program for adolescent substance abuse. American Journal of Evaluation 2005; 26: 77-94.         [ Links ]

11. Weissberg RP, Kumpfer KL, Seligman ME. Prevention that works for children and youth. American Psychologist 2003; 58(6/7): 425-32.         [ Links ]

12. Beyea SC, Nicoll LH. Writing in integrative review. AORN Journal 1998; 67(4): 877-80.         [ Links ]

13. Ganong LH. Integrative reviews of nursing research. Research in Nursing & Health 1987; 10(1): 1-11.         [ Links ]

14. Peres RS, Santos MA. Personalidade e câncer de mama: produção científica em Psico-oncologia. Psicologia: Teoria e Pesquisa 2009; 25(4): 611-20.         [ Links ]

15. Weiss CH, Murphy-Graham E, Petrosino A, Gandhi AG. The fairy godmother and her warts: making the dream of evidence-based policy come true. American Journal of Evaluation 2008; 29(1): 29-47.         [ Links ]

16. Marson SM, Wei G, Wasserman D. A reliability analysis of goal attainment scaling (GAS) weights. American Journal of Evaluation 2009; 30(2): 203-16.         [ Links ]

17. Brandon PR, Singh JM. The strength of the methodological warrants for the findings of research on program evaluation use. American Journal of Evaluation 2009; 30(2): 123-57.         [ Links ]

18. Bisset S, Daniel M, Potvin L. Exploring the intervention context interface: a case from a n-based nutrition intervention. American Journal of Evaluation 2009; 30(4): 554-71.         [ Links ]

19. Savaya R, Spiro S, Elran-Barak R. Sustainability of social programs: a comparative case study analysis. American Journal of Evaluation 2008; 29(4): 478-93.         [ Links ]

20. Urban JB, Trochim W. The role of evaluation in research-pratice integration: working toward the "golden Spike". American Journal of Evaluation 2009; 30(4): 538-53.         [ Links ]