Access Reports and Other Tools
Use the links at the left to access tools on effective evidence-based/informed interventions. Materials range from Evidence Briefs developed by NCEMCH at Georgetown University, detailed Evidence Analysis Reports developed by the Women's and Children's Health Policy Center at Johns Hopkins University (JHU), NPM Title V Transformation Tools developed by the MCH Navigator with the National MCH Workforce Development Center, and webinars hosted by AMCHP with JHU.
Stay tuned for new reports on NPM topic areas as well as integration of state-developed Evidence-based/informed Strategy Measures (ESMs).
About Evidence Analysis Reports
The project uses a continuum of evidence approach that analyzes peer-reviewed literature and promising practices.
Rating the Evidence: The Continuum Methodology
The Women's and Children's Health Policy Center at JHU led the shift from collection of evidence to active analysis and development of Evidence Analysis Reports during the first round of the Strengthen the Evidence Base for MCH Programs initiative (2015-2018). This section describes their approach in developing the initial reports.
Extensive literature searches were undertaken to identify relevant studies for each National Performance Measure (NPM). Examples of databases used include the Cochrane Library, PubMed, ERIC, and PsycINFO. Search results were evaluated systematically for relevant studies using predetermined criteria defined by the STE Team. In addition to peer-reviewed literature, relevant gray literature sources were included in the reviews. Reference lists of relevant review articles also informed the search process. A sequential process was employed:
- Article titles reviewed
- Article abstracts screened
- Full-text reviews of articles/reports conducted
Team members extracted data pertaining to the study characteristics (country, sample, and design); intervention (e.g., components, duration); instruments and measures; and results. Studies were categorized into groups based on target audience for the intervention, such as Population-based Systems; Hospital; Providers; and Patients.
Interventions identified through the assessment process were plotted on a continuum of evidence, which informs the work to guide states in using the most rigorous available MCH science while also encouraging innovation. This continuum is portrayed in the figure below, followed by a rationale for its development.
For each category of evidence, the table below provides a descriptive statement and criteria. In addition to published literature, the process tapped into relevant grey literature (e.g., technical reports, government documents, conference proceedings), particularly when scientifically rigorous evidence is not available and when materials are peer reviewed.
Evidence Ratings (Adapted from RWJ What Works for Health)1
|Scientifically Rigorous||Strategies with this rating are most likely to make a difference. These strategies have been tested in many robust studies with consistently positive results.|
|Moderate Evidence||Strategies with this rating are likely to work, but further research is needed to confirm effects. These strategies have been tested more than once and results trend positive overal.|
|Expert Opinion||Strategies with this rating are recommended by credible, impartial experts and are consistent with accepted theoretical frameworks. However, the strategies have limited research documenting effects; further research, often with stronger designs, is needed to confirm effects.|
|Emerging Evidence||Strategies with this rating have limited research documenting effects. These strategies need further research, often with stronger designs, to confirm effects.|
|Mixed Evidence||Strategies with this rating have been tested more than once and results are iconsistent or trend negative; further research is needed to confirm effects.|
|Evidence Against||Strategies with this rating are not good investments. These strategies have been tested in many robust studies, are not effective and sometimes produce harmful results.|
In addition, consideration needs to be given to public health impact in addition to scientific evidence. As defined by Spencer et al (2013),2 public health impact includes effectiveness (the degree to which the practice achieves the desired outcomes), reach (extent that the practice affects the target population), feasibility (extent to which practice can be implemented), sustainability (degree to which the practice can be maintained with desired outcome over time) and transferability (extent to which the practice can be adapted for various contexts).
Assignment to the continuum required that interventions or intervention categories were:
- Evaluated in 4 or more peer-reviewed studies
- Evaluated in 3 peer-reviewed studies with expert opinion from gray literature
Project members independently assigned ratings to the interventions or intervention categories. Assessments were compared and discrepancies discussed until consensus was reached.
Next, key findings and implications were identified. Drafts of the evidence review documents were reviewed by invited national experts at other academic institutions, the Maternal and Child Health Bureau, the Association of Maternal and Child Health Programs, and at Title V programs. After incorporating expert feedback, completed evidence review reports were posted on the program website and disseminated among stakeholders.
JHU Evidence-Informed Approach
The approach to preparing an evidence continuum for use by State Title V Programs is thus consistent with the McMaster group’s definition of "evidence-informed," which is stated as "the purposeful and systematic use of the best available evidence to inform the assessment of various options and related decision making in practice, program development, and policy making."3 In this formulation, "evidence-based" is considered to be a subset of "evidence-informed."
Therefore, the of "evidence-informed" accounts for the context in which decisions must be made. The need to adapt strategies based on variability in populations and recognized, as well as the need to consider available resources, the timing in which decisions must be made, and the political landscape. Thus, the continuum recognizes that findings from rigorously conducted experimental studies in selected settings may not fully apply in other settings and among other populations. Moreover, in the context of MCH programs and performance measures, there may not in fact be randomized or quasi-experimental studies available. Rather, evidence may only be available from observational studies – with all their inherent constraints – or from unpublished studies.
Using Evidence Analysis Reports
The Evidence Reviews can help state Title V programs address a broad range of policy, communication and educational needs.
Aligning with Current Strategies
Specifically, Title V staff may use these reports to:
- Assess the nature, scope, and quality of interventions used statewide or in local jurisdictions, community organizations, schools or other community settings;
- Inform policy and program discussions at the community and state levels, such as in working committees or advisory boards;
- Work with the media to feature information related to the subject of the reviews; and, to
- Incorporate into trainings for the MCH workforce.
Most important, however, is the opportunity to examine how the findings of the evidence reviews align with current strategies in place as part of the Title V Block Grant Program. If current activities are evidence-informed and appear to be working, no further action may be needed. However, if the current activities are not supported by the evidence reviews or if Title V wants to achieve even greater population level effects, then the evidence reviews may support program leadership in discussions regarding modifying or phasing out current activities. This process may best be undertaken with relevant stakeholders. Broadly, the decision process is described below:
If the MCH program already has implemented strategies to address the National Performance Measure being considered, then the first step is to assess the effectiveness of strategies currently in place. If program objectives are being met or progress is satisfactory, there may be no need to consider alternatives. If change in strategies is being considered, then more information may be needed.
Examples and Key Questions
Title V staff and relevant stakeholders may determine that more robust change in population outcomes is needed. In particular, they may consider implementing one or more additional evidence-based or evidence-informed strategies to complement or replace interventions underway. For example, if the single strategy that schools in the state (or local jurisdiction) are using is promulgation of school rules (emerging evidence) to address bullying, a Title V program might suggest introducing curricular activities to supplement ongoing interventions (i.e., with the intent of enhancing outcomes).4
Another example might be related to Well Women Care. Title V may be working with Title X family planning clinics to implement a strategy to increase women’s use of preventive health services. To enhance outreach to clients using these clinics, they might consider whether a mini-grant to a subset of clinics to use client reminders may improve receipt of an annual well woman visit among their clients.5
In a different scenario, a key staff person in a state legislator’s office has heard about a specific strategy undertaken (successfully) in another state to address Safe Sleep (e.g. baby blanket with back to sleep message on it). Title V staff could examine the full evidence report to learn whether it has been studied and the findings of the research. If findings were positive, then staff can use the set of tables in the reviews to further assess whether or not to begin to pursue this strategy.
Considerations for Adopting New Interventions
In studying the relevant findings in an evidence review, staff could consider:6
- Is the study sample or population similar to mine regarding?
- Demographic characteristics
- Culture, values, and preference
- Health status
- Other characteristics of interest
- Do I have what I need to implement the strategy?
- Workforce capacity
- Does my organization and broader environment support the strategy?
- Political support
- Financial and legal support
- Champions for intervention
- Community norms and partnerships
- Title V priority and jurisdiction
- Favorable environment for change
The Title V program can use these questions to determine if they want to change their strategy.
Considerations only of the evidence, however, may not be sufficient to identify next steps as Title V staff may identify that the proposed intervention(s):
- May not fit local context
- May not be sufficiently rigorous or the evidence is limited
- May offer little information about implementation and what is needed to replicate the strategy
- May not be translated for immediate use or for varied stakeholders
Ongoing Evidence Analysis Process
Processes for identification and analysis of evidence-based/informed resources and programs have been established by many groups, including the Centers for Disease Control and Prevention’s (CDC’s) Continuum of Evidence, Cochrane Reviews, AMCHP Levels of Excellence, the What Works Clearinghouse from the Institute of Education Science, and the What Works/LINKS database from Child Trends.
Read More about the Process
In particular, the Robert Wood Johnson Foundation’s (RWJF’s) What Works for Health provides the six-stage continuum of evidence rating guide currently in use, ranking strategies from (i) evidence against, (ii) mixed evidence, (iii) emerging evidence, (iv) expert opinion, (v) moderate evidence, to (vi) scientifically rigorous evidence. The consortium will use this continuum to inform the foundation of the reports, while taking into account that in public health evidence should be drawn not only from peer-reviewed published literature, but also from grey literature, promising and best practices, and direct experiences of Title V MCH professionals.7
Consortium partners will coordinate with state Title V programs and MCHB to develop and implement criteria to assess the need to update all 15 NPM evidence analysis reports during the five-year project period. The methodology proposed for this goal is based on (1) an in-depth understanding of evidence-based analysis and (2) a consortium-based approach that draws on consortium partners to lead the activity while tapping direct contacts with state Title V MCH programs, the Team of Experts, and MCHB TA Centers.
Methodology 1: Understanding Evidence
To ensure that work related to evidence-based analysis addresses unique characteristics of public health programs and the needs of MCH populations, NCEMCH uses (1) the Institute of Education Sciences’ Procedures and Standards Handbook to define inclusion criteria, review specific parameters, and define search terms and methods and (2) the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) Approach, modified for public health, to identify processes for rating quality of evidence; this process will be used in gathering data for assessing and updating reports.8,9
Methodology 2: Consortium-Based Approach
Each proposed consortium partner has unique strengths that will be leveraged to address components of the report assessment/updating process.
- NCEMCH brings the collected knowledge base of the field and experience in analyzing evidence-based/informed interventions.
- The National MCH Workforce Development Center focuses on evidence-based decision making as one of its core topic areas and has direct TA contacts with state Title V MCH programs.
- AMCHP connects professionals working in state Title V programs.
- CityMatCH has longstanding working arrangements with MCHB TA centers and health departments, with a special emphasis on providing resources that reflect the cultural, racial, linguistic, and geographic diversity of MCH populations.
- The Georgetown University Center for Child and Human Development leads and partners in multiple national TA centers and projects that support state MCH cross-sector work, translating research on evidence-based practices into policy and practice.
Rather than relying on one organization to assess and update the evidence analysis reports, the consortium approach will draw on the strengths of each partner who will be responsible for a particular aspect of the reports. The assessment/updating process will benefit from the combined strengths of consortium partners, while also drawing on subject matter expertise from the Team of Experts and MCHB TA centers.
1 Robert Wood Johnson Foundation. What Works for Health.
2 Spencer LM, Schooley MW, Anderson LA et al. Seeking Best Practices: A Conceptual Framework for Planning and Improving Evidence-Based Practices. Prev Chronic Dis. 2013:10:130186.
3 McMaster University. Health Evidence Glossary.
4 Lai Y, Garcia S, Grason H, Strobino D, Minkovitz C. National Performance Measure 9 Bullying Evidence Review Brief. Strengthen the Evidence Base for Maternal and Child Health Programs. Women's and Children's Health Policy Center, Johns Hopkins University, Baltimore, MD. 2017.
5Garcia S, Martino K, Lai Y, Minkovitz C, Strobino D. National Performance Measure 1 Well-Woman Visit Evidence Review Brief. Strengthen the Evidence Base for Maternal and Child Health Programs. Women’s and Children’s Health Policy Center, Johns Hopkins University, Baltimore, MD. 2017.
6 Jacobs JA, Jones E, Gabella BA, Spring B, Brownson RC. Tools for implementing an evidence-based approach in public health practice. Prev Chronic Dis. 2012;9:110324.
7 Bowen S, Zwi AB. Pathways to “Evidence-Informed” Policy and Practice: A Framework for Action. PLos Medicine. 2005; 2:7:e166.
8 Robert Wood Johnson Foundation. What Works for Health.
9 Rehfuess EA, Akl EA. Current experience with applying the GRADE approach to public health interventions: an empirical study. BMC Public Health 2013; 13:9.