Designing a Funnel-Based Evaluation Strategy for LLM Experiments

Introduction

Evaluating large language models (LLMs) is not a simple pass/fail exercise. A binary approach—like a fork in the road—often misses nuance and leads to misleading conclusions. Instead, a funnel-based strategy allows you to progressively filter and assess model outputs, ensuring that only the most relevant, coherent, and high-quality responses move forward. This guide walks you through building your own evaluation funnel, from defining criteria to running controlled experiments. By the end, you’ll have a repeatable process that reveals genuine performance differences between models.

What You Need

• LLM Outputs: At least two model variants to compare (e.g., baseline vs. fine-tuned).
• Evaluation Dataset: A set of prompts or tasks representative of your use case.
• Automated Judge System: A script or service (e.g., using GPT-4 as a judge or a custom classifier) that can score outputs on criteria like relevance, coherence, and instruction following.
• Scoring Rubric: Clear, numeric scales for each criterion (e.g., 1–5).
• Thresholds: Predefined cutoffs for each stage of the funnel (e.g., only keep outputs with relevance score ≥ 4).
• Statistical Toolkit: Basic knowledge of hypothesis testing (e.g., t-tests, effect size calculation).

Step-by-Step Guide

Step 1: Define Your Evaluation Criteria

Before any funnel can work, you need to know what “good” looks like. List three to five dimensions that matter for your LLM use case. Common criteria include:

• Relevance: Does the response address the prompt?
• Coherence: Is the response logically structured and easy to follow?
• Accuracy: Are facts correct (if applicable)?
• Completeness: Does the response cover all required aspects?
• Safety: Does it avoid harmful or biased content?

For each criterion, decide on a scoring scale (e.g., 1–5) and write anchor descriptions for each score. This ensures consistency when your automated judge applies the rubric.

Step 2: Design a Progressive Funnel

A funnel works by applying increasingly strict filters. Start with a wide net and narrow down. For example:

• Stage 1 – Basic pass: Check that the output is not empty and respects basic formatting.
• Stage 2 – Relevance filter: Only keep outputs scoring ≥ 4 on relevance.
• Stage 3 – Coherence filter: Among retained outputs, keep those scoring ≥ 3.5 on coherence.
• Stage 4 – Accuracy/Completeness filter: Further narrow to outputs scoring ≥ 4 on accuracy.
• Stage 5 – Final review: The top 5% of outputs (by composite score) become the “winning” candidates.

Each stage reduces the dataset, making downstream analysis faster and more focused. You can adjust the number of stages and thresholds based on your precision requirements.

Step 3: Calibrate Your Automated Judge

Automated judges (like using an LLM to evaluate another LLM) can exhibit biases. Calibration ensures your funnel doesn’t unfairly advantage one model. Do the following:

1. Create a gold standard: Have human experts rate a small sample (50–100 outputs) on your criteria.
2. Run the automated judge on the same sample.
3. Compare scores to check for systematic bias (e.g., judge always gives higher scores to longer outputs).
4. Adjust the judge prompt or scoring logic to reduce bias. For instance, explicitly ask the judge to ignore length and focus on content.
5. Re-test until the automated judge’s scores correlate strongly (r > 0.8) with human ratings.

Calibration might take a few iterations, but it’s essential for trustworthiness of the funnel results.

Step 4: Run the Experiment

With your criteria, funnel stages, and calibrated judge ready, now execute the experiment:

1. Generate outputs for each model variant using your evaluation dataset. Ensure all models see identical prompts.
2. Apply each funnel stage sequentially. After Stage 1, record how many outputs pass. Continue to Stage 2, and so on.
3. Track passing rates at each stage for each model. This gives you a granular view: maybe Model A has high relevance but low coherence, while Model B is the opposite.
4. Compute a final “win” score (e.g., percentage of outputs that survive all stages or average composite score of survivors).

Make sure to run the process multiple times or use a large enough dataset to avoid noise. A minimum of 200–500 prompts is recommended.

Step 5: Analyze Funnel Results

Now comes the insight. Don’t just look at the final winner; examine where models drop off:

• Which stage causes the biggest loss for a model? That pinpoints its weakness (e.g., poor coherence).
• Compare the funnel visuals (bar charts of pass rates per stage) across models to see trade-offs.
• Perform statistical tests to determine if differences are significant. Use a paired t-test or Wilcoxon signed-rank test on the per-prompt survival scores.
• Compute effect sizes (Cohen’s d) to understand practical significance beyond p-values.

Document your findings in a report that includes the funnel thresholds, pass rates, and any bias corrections applied.

Tips for Success

• Start simple: Use only two–three criteria and two funnel stages initially. You can always add complexity later.
• Validate your judge regularly: Model updates or prompt changes can shift judge behavior; recalibrate monthly.
• Don’t over-optimize thresholds: Funnel cutoffs should be based on your application’s quality bar, not on making one model look better.
• Combine with human review: Use the funnel to surface borderline cases for manual inspection—this improves both evaluation and your judge training data.
• Share your funnel design: Transparency about criteria and thresholds helps stakeholders trust the results and reproduce your experiments.