Evidence Tools

• MCHbest Strategies. The database aggregates sample evidence-based/informed strategies that can be used as-is or adapted to develop Evidence-based or informed Strategy Measures (ESMs) for each of the MCH National Performance Measures (NPMs). 
• Established Evidence. Search the peer-reviewed research articles that were reviewed in developing the evidence reports. Links are provided to electronic access.
• Emerging Evidence Searches. The links provided in this database provide you with the latest citations from PubMed based on automated searches of the most current research literature using the same search parameters that the project uses to identify articles for the Evidence Accelerators.
• ESM Database. Find Evidence-based/informed Strategy Measures (ESMs) that are currently in use by states and jurisdictions. You can search by measure, keyword (e.g., WIC, hospital), or state/jurisdiction.

• Evidence Accelerators. Short guides that provide background information and effective evidence-based/informed strategies.
• Evidence Analysis Reports. Access a detailed report for each of the original NPMs.
• Technical Assistance Briefs. These briefs are designed to act as a conversation starter in thinking about programs that can be developed to address issues that affect women, infants, children, adolescents, youth, families, and communities. They are populated from ongoing technical assistance provided by the MCH Evidence Center to Title V agencies related to the evidence base, strategies, and measures related to many topics interconnected to the NPMs and SMs.

• Accelerate Progress with Evidence. Accelerate strategies with evidence to ensure that they are meaningful and have the greatest potential to affect desired change. The strongest strategies rely on research that is unbiased, relevant to public health, and rooted in science, experience, and policy. They should show results that are statistically significant and lead to decisions that bring about change. Use this page to access specific approaches and tools that the Evidence Center uses to identify and adapt evidence-based/informed strategies
• Additional Learning Resources. Access a full range of learning videos on topics such as:
  • Introduction to Literature Reviews.
  • Health Equity and Social Determinants of Health.
  • Using Critical Thinking to Advance MCH through Evidence.
  • Summaries of NPM Evidence and Supplemental Office Hours.
  • Identifying and Using Evidence-Based/Informed Resources to Address MCH Issues Learning Brief.
  • Title V Transformation Tools.

• Think Upstream to Plan: Summary of Results-Based Accountability (RBA). Learn how to plan your programs with RBA, including:
  • Population Tools: summary of how to Develop Stronger ESMs, the Role of Title V in Adapting and Implementing Strategies, the Turn-the-Curve Strategy Tool, the SCALER Tool, and the Public Square Test.
  • Performance Tools: ESM Measurement Tool and the SiLVer Test.
• How to create stronger ESMs: Utilize RBA tools to ensure effective ESMs for the intervention and population of interest.
• How to measure ESM impact: RBA provides four quadrants of measurement to evaluate effort and effect across quantity and quantity in the measurement of ESM impact.

• Role of Title V in Adapting Strategies. When deciding how best to implement a strategy that seems outside the typical role of Title V, it may be helpful to consider activities that support the strategy while aligning with activities that Title V is charged with.
• RBA Population Turn the Curve Tool. Turn the Curve Tool is a quick method to strategically think about your needs assessment data and develop strong measures to assess progress we make in changing the trajectory of your work.
• RBA Performance Measurement Tool. Use to consider seven performance accountability questions once you have set your priorities, identified your NPMs/SMs, and are focused on ESMs.

• Think Upstream to Plan: Summary of Results-Based Accountability (RBA). Plan upstream to ensure that programs/interventions address issues early and are measurable in “turning the curve” on big issues that face MCH populations. To assist us in this process, a system of shifting the evidence into practice is needed. Implementation science uses the foundational building blocks of evidence to translate root causes of population-based issues into responsive programs that bring about change that can be quantified, brought to scale, and replicated across population groups. Use this page to access specific approaches and tools that the Evidence Center uses to advance health equity and address social determinants of health (SDOH)

• Hexagon Tool. This planning and evaluation tool guides the selection of the appropriate, evidence-based/informed strategies through a six-step exploration process. It can be used in collaboration with your partners to better understand how a new or existing program fits into your existing work, context, and health equity priorities. The Center for Implementation has a good summary of the tool.
• Root-Cause Analysis (RCA). A structured facilitated team process to identify root causes of an event that resulted in an undesired outcome and develop corrective actions. The RCA process provides a way to identify breakdowns in processes and systems that contributed to the event and how to prevent future events. The purpose of an RCA is to find out what happened, why it happened, and determine what changes need to be made. A cause and effect diagram, often called a Fishbone Diagram, can help in brainstorming to identify possible causes of a problem and in sorting ideas into useful categories. A fishbone diagram is a visual way to look at cause and effect. Download an Interactive Fishbone Diagram, designed specifically for MCH programs, for use in determining causes. This diagram is in PowerPoint format for easy group work.

The Evidence Center has identified evidence-based/informed tools in work with Title V agencies to ensure that new and ongoing strategies reflect the needs of all populations, advance health equity, and address SDOH.

• Access the Top Tools that the Center recommends for your equity work.

These learning resources developed by the MCH Evidence Center can help ensure effective implementation of movable programs.

• Work Together with Equity Tools. Ensure that programs are movable within the realities of Title V programs and lead to health equity for all people. This strengths-based approach draws on implementation tools designed for MCH populations. Use this page to access specific approaches and tools that the Evidence Center uses to advance health equity and address social determinants of health (SDOH).
• What Title V Agencies are Doing to Advance Health Equity and Address Social Determinants of Health.

The process for reviewing the scientific literature is outlined broadly in the diagram below:

Extensive literature searches are undertaken to identify relevant studies for each National Performance Measure (NPM) and Standardized Measure (SM). Examples of databases used include the Cochrane Library, PubMed, ERIC, and PsycINFO. Search results are evaluated systematically for relevant studies using predetermined criteria defined by the MCH Evidence Team. In addition to peer-reviewed literature, relevant gray literature sources are included in the reviews.  Reference lists of relevant review articles also inform the search process. A sequential process is employed: 

1. Article titles reviewed 
2. Article abstracts screened 
3. Full-text reviews of articles/reports conducted 

Team members extract data pertaining to the study characteristics (country, sample, and design); intervention (e.g., components, duration); instruments and measures; and results. Studies were categorized into groups based on target audience for the intervention, such as Population-based Systems; Hospital; Providers; and Patients.

Interventions identified through the assessment process are plotted on a continuum of evidence, which informs the work to guide states in using the most rigorous available MCH science while also encouraging innovation. This continuum is portrayed in the figure below, followed by a rationale for its development. For each category of evidence, the table below provides a descriptive statement and criteria (adapted from the What Works for Health Database).¹ In addition to published literature, the process taps into relevant grey literature (e.g., technical reports, government documents, conference proceedings), particularly when scientifically rigorous evidence is not available and when materials are peer reviewed. 

In addition, Analogical Evidence refers to those strategies that have proven effective in similar NPM topic areas and are likely to be effective in other settings (e.g., provider training is a strategy that is likely effective to advance all NPM topic areas). It is reasonable to assume that this strategy will also prove effective with a related NPM, but additional research is needed. In adapting this strategy, you may want to start with a pilot group, collect data, and evaluate to ensure impact with this topic area and your population group(s).

Evidence Ratings

In addition, consideration needs to be given to public health impact in addition to scientific evidence. As defined by Spencer et al (2013),² public health impact includes effectiveness (the degree to which the practice achieves the desired outcomes), reach (extent that the practice affects the target population), feasibility (extent to which practice can be implemented), sustainability (degree to which the practice can be maintained with desired outcome over time) and transferability (extent to which the practice can be adapted for various contexts).

Assignment to the continuum required that interventions or intervention categories were: 

• Evaluated in 4 or more peer-reviewed studies
• Evaluated in 3 peer-reviewed studies with expert opinion from gray literature 

Project members independently assign ratings to the interventions or intervention categories. Assessments are compared and discrepancies discussed until consensus is reached.

Next, key findings and implications are identified. Drafts of the evidence review documents are reviewed by invited national experts at other academic institutions, the Maternal and Child Health Bureau, the Association of Maternal and Child Health Programs, and at Title V programs. After incorporating expert feedback, completed evidence review reports are posted on the program website and disseminated among partners.

Ratings for Innovation Hub Field-Based Practices

The Association of Maternal and Child Health Programs (AMCHP) maintains the MCH Innovations Database, a searchable repository of “what’s working” in MCH that includes effective practices, programs, and strategies from the field that are positively impacting MCH populations. Practices in the database are assessed along a Practice Continuum and receive a designation of Cutting-Edge, Emerging, Promising, or Best Practice depending on the level of evidence demonstrating a practice’s impact, among other criteria. AMCHP defines evidence holistically according to the following definition to capture the full-breadth of information and practice-based insights that can support the effectiveness of an intervention.

Ratings for the What Works for Health Database

Funded by the Robert Wood Johnson Foundation, the What Works for Health Database (developed by the University of Wisconsin Population Health Institute as part of its County Health Rankings and Roadmaps project) uses a similar evidence rating system. MCH Evidence staff crosswalk strategies found in this database with our own evidence continuum to assign consistent ratings across all evidence-based/informed strategies.

Evidence-Informed Approach 
​
The approach to preparing an evidence continuum for use by State Title V Programs is thus consistent with the McMaster group’s definition of "evidence-informed," which is stated as "the purposeful and systematic use of the best available evidence to inform the assessment of various options and related decision making in practice, program development, and policy making."³ In this formulation, "evidence-based" is considered to be a subset of "evidence-informed." 

Therefore, the of "evidence-informed" accounts for the context in which decisions must be made. The need to adapt strategies based on variability in populations and recognized, as well as the need to consider available resources, the timing in which decisions must be made, and the political landscape. Thus, the continuum recognizes that findings from rigorously conducted experimental studies in selected settings may not fully apply in other settings and among other populations. Moreover, in the context of MCH programs and performance measures, there may not in fact be randomized or quasi-experimental studies available. Rather, evidence may only be available from observational studies – with all their inherent constraints – or from unpublished studies.

Specifically, Title V staff may use these reports to:

• Assess the nature, scope, and quality of interventions used statewide or in local jurisdictions, community organizations, schools or other community settings;
• Inform policy and program discussions at the community and state levels, such as in working committees or advisory boards;
• Work with the media to feature information related to the subject of the reviews; and, to
• Incorporate into trainings for the MCH workforce.    

Most important, however, is the opportunity to examine how the findings of the evidence reviews align with current strategies in place as part of the Title V Block Grant Program.  If current activities are evidence-informed and appear to be working, no further action may be needed.  However, if the current activities are not supported by the evidence reviews or if Title V wants to achieve even greater population level effects, then the evidence reviews may support program leadership in discussions regarding modifying or phasing out current activities.  This process may best be undertaken with relevant stakeholders.  Broadly, the decision process is described below: 

If the MCH program already has implemented strategies to address the National Performance Measure being considered, then the first step is to assess the effectiveness of strategies currently in place.  If program objectives are being met or progress is satisfactory, there may be no need to consider alternatives. If change in strategies is being considered, then more information may be needed.

Title V staff and relevant partners may determine that more robust change in population outcomes is needed. In particular, they may consider implementing one ​or more additional evidence-based or evidence-informed strategies to complement or replace interventions underway. For example, if the single strategy that schools in the state (or local jurisdiction) are using is promulgation of school rules (emerging evidence) to address bullying, a Title V program might suggest introducing curricular activities to supplement ongoing interventions (i.e., with the intent of enhancing outcomes).^4

Another example might be related to Well Women Care. Title V may be working with Title X family planning clinics to implement a strategy to increase women’s use of preventive health services. To enhance outreach to clients using these clinics, they might consider whether a mini-grant to a subset of clinics to use client reminders may improve receipt of an annual well woman visit among their clients.⁵ ​

In a different scenario, a key staff person in a state legislator’s office has heard about a specific strategy undertaken (successfully) in another state to address Safe Sleep (e.g. baby blanket with back to sleep message on it). Title V staff could examine the full evidence report to learn whether it has been studied and the findings of the research. If findings were positive, then staff can use the set of tables in the reviews to further assess whether or not to begin to pursue this strategy.

ADOPT Strategies to Meet Needs.

In choosing to adopt an existing strategy based on existing science and practice, we should consider:⁶

Is the study sample or population similar to our target audience?

• ​Geography
• Demographic characteristics
• Culture, values, and preference
• Health status
• Other characteristics of interest 

Do we have the resources needed to implement?

• ​Workforce capacity
• Money
• Time
• Leadership 

Does our organization and the broader environment support the strategy? ​

• Political support
• Financial and legal support
• Champions for intervention
• Community norms and partnerships
• Title V priority and jurisdiction
• Favorable environment for change​​

Implementation Tool. The Hexagon Tool (National Implementation Research Network) provides a systematic approach to evaluation effectiveness of a strategy based on six fit and feasibility factors.

Note. Looking at the evidence may not be sufficient to identify next steps if through this process we identify that the proposed strategy(s) does not:

• Fit local context
• Present sufficient evidence
• Offer enough information about implementation and plans for replication
• Have the ability to be modified for immediate use or for varied stakeholders

If this is the case, we recommend using the Science-Based Intervention Approach below to adapt strategies to fit the needs of your populations.

ADAPT Strategies to Meet Needs.

Not all strategies are effective for all populations. To adapt strategies to meet needs, we should consider:⁷

What about the existing strategy works? If we understand the key ingredients of a strategy, we can replicate and/or adapt the effective components. Looking at a strategy through a health behavior theory identifies key ingredients. Here are several to consider:

• Intrapersonal. Theory of Planned Behavior, Health Belief Model, Stages of Change Model.
• Interpersonal. Social Cognitive Theory, Social Norms Theory.
• Community. Diffusion of Innovation, Ecological Models.

How does it work? Being specific about the underlying mechanisms can help us increase the impact. Developing a logic model with program actions, targets, outcomes, and moderators allows you track the process from action to outcome.

For whom does it work, and for whom does it not work? When we know who is and is not responding, we can make targeted adaptations to improve outcomes. Think about the program life cycle:

• Precision. Understand what a strategy entails so you can go beyond “does it work,” to “what about it works” and “for whom does it work.”
• Fast-cycle iteration. Incorporate new ideas as you go – what is working and what is not working.
• Shared learning. Create a mechanism to share learning about success and failures.
• Co-creation. Bring together multiple parties to create a mutually valued outcome.

In what contexts does it work? By evaluating the context in which a strategy is implemented, we can adapt it for other settings. The best way to ensure that a strategy is effective is to conduct a robust evaluation. The MCH Navigator’s Evaluation Spotlight provides trainings and resources related to the steps and standards for effective program evaluation.

In particular, the Robert Wood Johnson Foundation’s (RWJF’s) What Works for Health provides the six-stage continuum of evidence rating guide currently in use, ranking strategies from (i) evidence against, (ii) mixed evidence, (iii) emerging evidence, (iv) expert opinion, (v) moderate evidence, to (vi) scientifically rigorous evidence. The consortium will use this continuum to inform the foundation of the reports, while taking into account that in public health evidence should be drawn not only from peer-reviewed published literature, but also from grey literature, promising and best practices, and direct experiences of Title V MCH professionals.⁸

Consortium partners will coordinate with state Title V programs and MCHB to develop and implement criteria to assess the need to update all 15 NPM evidence analysis reports during the five-year project period. The methodology proposed for this goal is based on (1) an in-depth understanding of evidence-based analysis and (2) a consortium-based approach that draws on consortium partners to lead the activity while tapping direct contacts with state Title V MCH programs, the Team of Experts, and MCHB TA Centers.

To ensure that work related to evidence-based analysis addresses unique characteristics of public health programs and the needs of MCH populations, NCEMCH uses (1) the Institute of Education Sciences’ Procedures and Standards Handbook to define inclusion criteria, review specific parameters, and define search terms and methods and (2) the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) Approach, modified for public health, to identify processes for rating quality of evidence; this process will be used in gathering data for assessing and updating reports.^9,10

MCH Evidence Center staff use an ongoing process of updating the evidence related to each of the NPM topic areas. Updates to the Evidence Analysis Reports are issued through a series of What Works Evidence Accelerators. Staff use the same search strings to perform systematic reviews for each topic area as outlined in the methodology for each Evidence Analysis Report. As new studies are identified, they are added to the Established Evidence database, are linked to strategies in the MCHbest database, and are available through links in the What Works Accelerators.

By tracking the evidence and strategies through a series of databases, we accelerate the pace by which the evidence base is available to incorporate into practice.