CDC - Centers for Disease Control and Prevention  NIDDK - National Institute of Diabetes and Digestive and Kidney Diseases

NDEP is a partnership of the National Institutes of Health, the Centers for Disease Control and Prevention, and more than 200 public and private organizations.

Principles and Limitations of Evidence-Based Decision-Making

The principles of evidence-based decision-making continue to evolve. You may see or hear the term used to describe a wide variety of situations. While many different formats exist, all evidence-based approaches have four common principles:

  1. Use a systematic and explicit method of searching the literature and retrieving the evidence.
  2. Synthesize the body of evidence
    Narrative quantitative (meta-analysis) methods can be used to compile the body of evidence. Evidence is not limited to any one type of literature. While randomized clinical trials may be favored designs for generating high levels of evidence, evidence may also be generated from case studies, pertinent guidelines, high-quality recommendations, authoritative consensus statements, or observational studies such as cohort studies with comparison groups. Much of traditional medical practice has not yet undergone rigorous scientific study and high-quality evidence may not exist to support conventional knowledge or practice.

    The hierarchy of the strength of evidence is often represented as a pyramid (with the strongest evidence towards the top). http://phpartners.org/tutorial/04-ebph/2-keyConcepts/4.2.7.html

    Evidence Pyramid

  3. Rate the quality of the body of evidence
    Many different rating systems have been proposed. Most rating systems will attempt to provide the reader with an assessment of the study design (randomized clinical trial, cohort study, case-control study, consensus panel, etc.) and the study quality (i.e. internal validity, quality of the supporting evidence on a topic).

    Since 1999, the Agency for Healthcare Research and Quality has been mandated to look at "methods or systems to rate the strength of the scientific evidence underlying health care practice, recommendations in the research literature and technology assessments."

    A common rating system is the following ABC rating scale.

    • Level A (randomized controlled trial/ meta-analysis):
      High quality randomized controlled trial that considers all important outcomes. High-quality meta-analysis (quantitative systematic review) using comprehensive search strategies.
    • Level B (other evidence):
      A well-designed, nonrandomized clinical trial. A nonquantitative systematic review with appropriate search strategies and well-substantiated conclusions. Includes lower quality randomized controlled trials, clinical cohort studies, and case-controlled studies with nonbiased selection of study participants and consistent findings. Other evidence, such as high-quality, historical, uncontrolled studies, or well-designed epidemiologic studies with compelling findings, is also included.
    • Level C (consensus/expert opinion):
      Consensus viewpoint or expert opinion. Expert opinion is sometimes the best evidence available.

    Note: Other letter scales may be used. For example, the U.S. Preventive Services Task Force is charged with searching, reviewing, and grading evidence, and making recommendations for prevention practices. They use an ABCDF rating scale with different categories from those above. The American Diabetes Association uses an ABCD rating scale. The Endocrine Society uses a combination of numbers (1 = we recommend or 2 = we suggest) that indicates the strength of the recommendation and symbols (stars) to grade the evidence upon which the recommendation is made.

    The USPSTF GRADE system for evidence is summarized in the tables below:

    What the Grades Mean and Suggestions for Practice

    The USPSTF updated its definitions of the grades it assigns to recommendations and now includes "suggestions for practice" associated with each grade. The USPSTF has also defined levels of certainty regarding net benefit. These definitions apply to USPSTF recommendations voted on after May 2007.

    Grade Definition Suggestions for Practice
    A The USPSTF recommends the service. There is high certainty that the net benefit is substantial. Offer or provide this service.
    B The USPSTF recommends the service. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial. Offer or provide this service.
    C Note: The following statement is undergoing revision.
    Clinicians may provide this service to selected patients depending on individual circumstances. However, for most individuals without signs or symptoms there is likely to be only a small benefit from this service.
    Offer or provide this service only if other considerations support the offering or providing the service in an individual patient.
    D The USPSTF recommends against the service. There is moderate or high certainty that the service has no net benefit or that the harms outweigh the benefits. Discourage the use of this service.
    I Statement The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service. Evidence is lacking, of poor quality, or conflicting, and the balance of benefits and harms cannot be determined. Read the clinical considerations section of USPSTF Recommendation Statement. If the service is offered, patients should understand the uncertainty about the balance of benefits and harms.

    Levels of certainty regarding net benefit
    Level of Certainty Description
    High The available evidence usually includes consistent results from well-designed, well-conducted studies in representative primary care populations. These studies assess the effects of the preventive service on health outcomes. This conclusion is therefore unlikely to be strongly affected by the results of future studies.
    Moderate The available evidence is sufficient to determine the effects of the preventive service on health outcomes, but confidence in the estimate is constrained by such factors as:
    • The number, size, or quality of individual studies.
    • Inconsistency of findings across individual studies.
    • Limited generalizability of findings to routine primary care practice.
    • Lack of coherence in the chain of evidence.
    As more information becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion.
    Low The available evidence is insufficient to assess effects on health outcomes. Evidence is insufficient because of:
    • The limited number or size of studies.
    • Important flaws in study design or methods.
    • Inconsistency of findings across individual studies.
    • Gaps in the chain of evidence.
    • Findings not generalizable to routine primary care practice.
    • Lack of information on important health outcomes.
    More information may allow estimation of effects on health outcomes.
    The USPSTF defines certainty as "likelihood that the USPSTF assessment of the net benefit of a preventive service is correct." The net benefit is defined as benefit minus harm of the preventive service as implemented in a general, primary care population. The USPSTF assigns a certainty level based on the nature of the overall evidence available to assess the net benefit of a preventive service.
  4. Generate recommendations based on the quality of the body of the evidence as well as patient values, preferences and resource constraints.

Section top ^

Limitations of Evidence-Based Decision-Making

Limitations that are universal to science and medicine include shortage of coherent and consistent scientific evidence, difficulties in translating the evidence into individualized clinical practice, inadequate system organization, and financial constraints. Limitations that are unique to the practice of evidence-based decision-making include the need for new skills to conduct literature searches and critically appraise the evidence, as well as time for busy clinicians to digest the literature. These can be overcome, however, through identification and use of pre-appraised resources. A critical point to keep in mind is that evidence is dynamic and needs to be revisited on a regular basis.

The special edition of Health Affairs, sponsored by the Agency for Healthcare Research and Quality, provides a variety of perspectives on the challenges and potential rewards of building and using a stronger evidence base for health care decisions.

Section top ^

Recommendation Examples

The following recommendations come from the American Diabetes Association Position Statement on the Standards of Medical Care for Patients with Diabetes Mellitus, and illustrate the way evidence ratings may appear in an evidence-based report.

A-Level evidence Metformin therapy for prevention of type 2 diabetes may be considered in those with IGT (A), IFG (E), or an A1C of 5.7–6.4% (E), especially for those with BMI >35 kg/m2, those aged <60 years, and those with prior GDM. (A)

B-Level evidence Women with a history of GDM should have lifelong screening for the development of diabetes or prediabetes at least every 3 years. (B)

C- Level evidence The mix of carbohydrate, protein, and fat may be adjusted to meet the metabolic goals and individual preferences of the person with diabetes. (C)

E- Level evidence If ACE inhibitors, ARBs, or diuretics are used, kidney function and serum potassium levels should be monitored. (E)

Level of evidence Description
A

Clear evidence from well-conducted, generalizable, RCTs that are adequately powered, including:

  • Evidence from a well-conducted multicenter trial
  • Evidence from a meta-analysis that incorporated quality ratings in the analysis

Compelling nonexperimental evidence, i.e., “all or none” rule developed by Center for Evidence Based Medicine at Oxford

Supportive evidence from well-conducted randomized controlled trials that are adequately powered, including:

  • Evidence from a well-conducted trial at one or more institutions
  • Evidence from a meta-analysis that incorporated quality ratings in the analysis
B

Supportive evidence from well-conducted cohort studies

  • Evidence from a well-conducted prospective cohort study or registry
  • Evidence from a well-conducted meta-analysis of cohort studies

Supportive evidence from a well-conducted case-control study

C

Supportive evidence from poorly controlled or uncontrolled studies

  • Evidence from RCTs with one or more major or three or more minor methodological flaws that could invalidate the results
  • Evidence from observational studies with high potential for bias (such as case series with comparison with historical controls)
  • Evidence from case series or case reports

Conflicting evidence with the weight of evidence supporting the recommendation

E

Expert consensus or clinical experience