review · Evaluation, Problem Solving · 30–60 min · 1-5 people · low energy

Ruling Out Technical Explanations

This method involves systematically identifying and investigating potential technical limitations that could explain observed results, rather than attributing them solely to causal relationships. By ruling out these technical explanations, you can increase confidence in the validity of your findings.

When to use Ruling Out Technical Explanations

Use this method when you need to ensure that observed results are not due to technical errors or limitations. It's particularly important when evaluating complex interventions or using sophisticated data analysis techniques.

What it solves

Spurious results, invalid conclusions, lack of confidence in findings

How to run Ruling Out Technical Explanations, step by step

  1. Identify potential technical limitations (e.g., measurement error, sampling bias, data analysis errors). (15 min)
  2. Investigate each limitation to determine its potential impact on the results. (Ongoing)
  3. Take steps to mitigate or correct the limitations (e.g., improve measurement techniques, adjust for bias, re-analyze data). (Ongoing)
  4. Document the limitations and the steps taken to address them. (15 min)
  5. Assess the impact of the limitations on the overall conclusions. (15 min)

Materials needed

  • Data
  • Analysis reports
  • Technical expertise

Facilitator tips

  • Involve technical experts in the investigation.
  • Be transparent about the limitations and how they were addressed.

Common pitfalls

  • Overlooking important technical limitations. Conduct a thorough review of the data and analysis methods.
  • Dismissing limitations without proper investigation. Take the time to carefully assess the potential impact of each limitation.

Variations

  • Conduct a sensitivity analysis to assess the impact of different assumptions on the results.
  • Use a checklist to ensure that all potential technical limitations have been considered.

Running it online or hybrid

Share data and analysis reports electronically. Use screen sharing to review technical details together.

What it produces

A report documenting the potential technical limitations and the steps taken to rule them out.

Origin

Attributed to BetterEvaluation.org — source

Build a session around Ruling Out Technical Explanations

METODIC places Ruling Out Technical Explanations inside a complete, timed session plan — agenda, worksheets, facilitator guide and slides — matched to your group and goal.

Generate a session with Ruling Out Technical Explanations · Try METODIC free