A.8 · Data Collection and Graphing

Data reliability and procedural fidelity

Review the difference between consistent measurement and following a procedure as written. Examples show how to report these issues separately so the supervisor can interpret the learner’s progress.

Task A.8: Describe risks of unreliable data collection and poor procedural fidelity

Useful foundations: A.5: Observable and measurable behavior

The rule

Check whether behavior was measured consistently and whether the procedure was implemented as written before interpreting an outcome.

Check measurement agreement, procedural fidelity and the behavior definition separately. Each tells you something specific about the quality of the evidence.

Key terms
Interobserver agreement (IOA)
A comparison of how consistently two observers record the same events.
Procedural fidelity
How closely implementation follows the written procedure.

What to look for

DistinctionLook forHow to decide
Measurement reliabilityObservers recording the same eventsCheck consistency against the defined measurement method.
Procedural fidelityStaff completing prescribed stepsCompare implementation with the actual protocol.
Interpretation limitData quality and implementation evidenceResolve measurement and implementation uncertainty before attributing outcomes to the procedure.

Worked examples

Observer agreement

Matching totals can conceal disagreement

Two observers watch the same four intervals. Observer 1 records yes, yes, no, no. Observer 2 records no, no, yes, yes. Both report two intervals with behavior.

Relevant facts

  • Totals match.
  • No interval-level judgment matches.

The observers disagree on all four interval-level judgments.

0 matching intervals ÷ 4 intervals = 0% interval agreement

Compare the records interval by interval. Review the definition and observation process with the supervisor to understand the disagreement.

Procedural fidelity

Check data accuracy and implementation

Two observers agree on every response in a teaching session. A separate checklist shows the RBT delivered the planned consequence on only 6 of 10 occasions when it was required.

Relevant facts

  • Response measurement agrees.
  • Four required consequence deliveries were omitted.

Report the fidelity problem separately from observer agreement.

6 completed opportunities ÷ 10 required = 60% for that step

Report the accurate response data alongside the implementation errors. The supervisor needs both records when interpreting progress and deciding how to proceed.

What changes the answer?

A learner’s responding decreases. Change the quality check.

  • The outcome data show a decline.

Measurement and implementation checks are sound.

The decline is more interpretable.

Use these quality checks together with the evaluation design when assessing causation.

Observers disagree or required steps were omitted.

Resolve and report those limitations.

The outcome could reflect recording or implementation differences.

Include measurement and implementation checks when reporting the strength of the evidence.

Remember

  • Agreement checks measurement; fidelity checks implementation.
  • Report errors honestly and use the supervisor’s correction process.
Sources for this topic

Reviewed 2026-09-30 · A.8 version 3.