How do I reduce LLM hallucinations in a validation pipeline?

Separate generation from judgement: seed candidates deterministically (Scout), optionally have a Critic argue against each one, then let a Judge filter. Making the model justify rejection catches hallucinations a single pass misses.

What is the Scout-Critic-Judge pattern?

A three-stage adversarial architecture: a Scout proposes candidates from deterministic seeds, a Critic attacks them, and a Judge makes the final keep-or-drop decision — so noise is filtered explicitly rather than hoped away.

Building an Adversarial Multi-Stage LLM Validation Pipeline: Scout → Critic → Judge

Target queries: "llm validation pipeline", "adversarial llm", "multi-agent llm verification", "reduce llm hallucination", "filtering llm noise"

The Hallucination Problem: Why Single-LLM Answers Fail at High Stakes

An LLM answers your query. You trust it because the response reads coherently, sounds authoritative, and feels grounded. You ship it. Then production reveals the fabrication: a "fact" that was plausible but false, a correlation that never existed, a reason that was pure confabulation.

This is not a bug in the model—it is the price of fluency. A language model trained to predict the next token will predict a plausible token. Plausibility and truth are not the same. When the cost of noise is high (autonomous observability decisions, patient outcomes, financial audits), a single-LLM pipeline is not a system, it is a roulette wheel.

The trap deepens when you believe the fix is "generate more, then filter." More recipes = more false positives in the upstream, worse precision at the downstream. Noise discrimination—the ability to distinguish signal from static without generating it—beats generating and hoping to filter.

This guide introduces a three-stage adversarial architecture used in infrastructure observability (JuiceAnalyzer, built on EU-hosted Qwen models via JuiceFactory) that inverts the problem: instead of LLM-as-proposer-then-hope-for-filtering, deploy LLM-as-scout-on-deterministic-seeds, then critic-attacks-the-scout, then judge-filters-both. The result: noise discrimination so tight that hallucinations become auditable, not invisible.

Deployment note: The path running in production today is Scout → Judge with a deterministic fallback. The adversarial Critic in the middle is a proposed extension—an architecture pattern we design toward, not a shipped stage. It's called out explicitly wherever it appears below.

Start building: Get a free JuiceFactory API key — no credit card required.

The Three-Stage Architecture: Scout → Critic → Judge

Stage 1: Scout (Propose from Qualified Candidates)

The Scout does not generate from scratch. It receives a curated seed list—candidates already screened by deterministic rules. The Scout's job is to annotate and re-rank, not invent.

Why this matters: An LLM starting with "here are 50 entities with anomalies + evidence summaries" is far more honest than one starting with "tell me what's wrong." Grounding eliminates the largest hallucination vector: making things up from context-free instructions.

In JuiceAnalyzer, the Scout runs every 15 minutes on the top 12 candidates (by severity score) from three deterministic producers:

group_burst       ← entities with synchronized metric anomalies
entity_local      ← single entities with local deviations  
incident_cluster  ← clustered events across timespans

These producers emit JSON-structured candidates with evidence summaries, scores, and scope. The Scout receives this:

# From problem_candidates_refresh.py, lines 735–776
messages = [
    {
        "role": "system",
        "content": (
            "You are a pre-candidate scout for operations analysis. "
            "Review deterministic seeds and return only extra candidates that should remain visible for later RCA. "
            "Do not repeat obvious duplicates unless you add a distinct reason to keep them. "
            "Return strict JSON with key 'suggestions'."
        ),
    },
    {
        "role": "user",
        "content": json.dumps({
            "task": "Find extra candidate-worthy items that should be kept visible before later filtering.",
            "max_suggestions": 8,
            "seeds": ranked,  # Top 12 by score
            "required_shape": {
                "suggestions": [
                    {
                        "seed_ref": "string",
                        "title": "short title",
                        "reason": "why this deserves more analysis",
                        "keep_visible_reason": "why later deterministic filtering may miss it",
                        "confidence": 0.0,
                        "severity_boost": 0.0,
                    }
                ]
            },
        }, default=str),
    },
]

response = await json_completion(
    messages,
    model="qwen3.5-27b",  # Qwen via JuiceFactory
    temperature=0.1,      # Low variance—enforce conformity to structure
    max_tokens=4096,
)

Key constraints:

Scout is not allowed to invent new candidates. It can only re-rank and annotate the 12 seeds.
Temperature = 0.1 (deterministic, not creative).
Response is strictly JSON with suggestions[]—any deviation fails parsing and skips the Scout output (deterministic fallback).
Ranking is capped at 8 suggestions (further noise cut).

Scoring after Scout re-annotation:

# Lines 792–795
confidence = clip01(float(item.get("confidence") or 0.0))
severity_boost = clip01(float(item.get("severity_boost") or 0.0))
base_score = clip01(float(seed.get("candidate_score") or 0.0))
candidate_score = clip01((base_score * 0.55) + (confidence * 0.35) + (severity_boost * 0.10))

The Scout's annotations are weighted at 45% (confidence + severity_boost). The deterministic base is still 55%. The LLM is advisory, not the source of truth.

Stage 2: Critic (Adversarial Review)

The Critic's role is not to validate the Scout—it is to attack it. Where is the Scout wrong? Where does it confabulate a "reason" that doesn't follow? Where does it miss an obvious duplicate?

In the published literature on adversarial multi-agent LLMs (e.g., Minimizing Hallucinations and Communication Costs, MDPI 2025), adversarial debate mechanisms have been reported to outperform consensus voting alone. Treat such figures as illustrative—validate them against your own corpus rather than assuming they transfer. The Critic operationalizes that idea by asking a different question: "Why should we reject this candidate?"

Deployment status: the Critic is a proposed extension, not a shipped production stage. In the JuiceAnalyzer code today, the live path is Scout → Judge with a deterministic fallback; the adversarial Critic described here is an architecture pattern we are designing toward, not yet merged. In the target design it would be gated behind a feature flag (e.g. enable_critic_review) and run asynchronously before promotion:

# Pseudo-code: Critic stage (not yet merged in codebase, but architecture pattern exists)
critic_prompt = {
    "role": "system",
    "content": (
        "You are a skeptical operations critic. Review Scout suggestions and flag: "
        "(1) hallucinated reasons (invented explanations), "
        "(2) duplicates the Scout repeated, "
        "(3) low-signal fluff (e.g., 'something is unusual without clear why'). "
        "Return JSON with 'rejections' and 'concerns' keys. Be adversarial."
    ),
    "user_content": json.dumps({
        "scout_suggestions": scout_output,
        "original_seeds": original_candidates,
    }),
}

critic_response = await json_completion(
    [critic_prompt],
    model="qwen3.5-27b",
    temperature=0.2,  # Slightly higher than Scout to encourage disagreement
    max_tokens=4096,
)

In this proposed design the Critic output is not used to filter directly. Instead, it flags candidates for audit: if the Critic rejects a candidate, the Judge (Stage 3) must override explicitly, and that override is logged.

Why is this powerful? Because hallucination in systems observability often looks like plausible but invented causality. "The Scout said host A is suspect because latency correlates with CPU." The Critic asks: "Does it really correlate, or is the Scout seeing pattern in noise?" If the Critic rejects and the Judge overrides, you have a paper trail of why a human trusted the LLM anyway.

Stage 3: Judge (Deterministic Filter + Override Log)

The Judge is the only stage that commits a candidate to RCA intake. It receives:

The original deterministic candidate (base evidence).
The Scout's re-annotation + confidence score.
(If the Critic extension is enabled) The Critic's rejections.

The Judge applies a final qualification threshold and logs overrides:

# From problem_candidates_refresh.py, lines 802–810
def _candidate_status(
    candidate_type: str,
    candidate_score: float,
    signal_count: int,
    event_count: int,
    metric_anomaly_count: int,
    blast_radius_score: float,
    coverage_score: float,
) -> str:
    if candidate_type == "incident_cluster":
        if (
            candidate_score >= 0.58
            and blast_radius_score >= 0.20
            and coverage_score >= 0.20
            and (signal_count >= 3 or event_count >= 1 or metric_anomaly_count >= 1)
        ):
            return "qualified"
        return "new"
    
    # Entity-local threshold is stricter than Scout override threshold
    if (
        candidate_score >= 0.24
        and coverage_score >= 0.25
        and (signal_count > 0 or event_count > 0 or metric_anomaly_count >= 2)
    ):
        return "qualified"
    return "new"

Thresholds are hardcoded, not LLM-determined. A candidate with Scout confidence = 0.95 but base_score = 0.10 will fail the Judge if it doesn't meet the coverage_score >= 0.25 rule.

If the Critic extension flagged the candidate and the Judge qualifies it anyway, that override is appended to evidence_summary:

override_log = {
    "judge_decision": "qualified",
    "critic_rejections": critic_response.get("rejections", []),
    "override_reason": "base_score and coverage exceed threshold despite critic concerns",
    "timestamp": now().isoformat(),
}

This log is immutable and queryable. Later, when an RCA investigator reviews the candidate, they can see: "The Critic said this was hallucinated, but the Judge overrode because signal_count >= 3. The Critic was [right/wrong]." Over time, this feedback trains your threshold adjustments.

Why This Beats "Generate More, Filter Later"

A naive approach: "Let's use 10 LLMs, run them in parallel, take the majority vote."

Problems:

Correlated hallucination: 10 LLMs trained on overlapping data will hallucinate the same false claim.
False confidence: A 7-out-of-10 vote looks authoritative, even if all 10 are confabulating the same error.
Noisy inputs kill voting: If you feed 10 LLMs bad data + vague instructions, you get 10 plausible bad answers, not 1 rejected one.

The Scout → Critic → Judge pipeline inverts this:

Deterministic baseline (no LLM) produces candidates on strict, auditable rules. This baseline is never wrong in the absence of data bugs—it's just noisy. High false-positive rate by design.
Scout (LLM with constraints) reduces false positives by annotating why a deterministic candidate might matter. It's not the source of truth; it's advisory.
Critic (adversarial LLM, proposed extension) attacks the Scout's reasoning. If both agree, confidence rises. If they disagree, the Judge makes the call explicitly.
Judge (deterministic + log) filters based on hardcoded rules, not LLM consensus. Ambiguous cases are logged so you can tune thresholds.

The result: fewer hallucinations, auditable reasoning, and a feedback loop to improve thresholds.

The Deterministic Fallback: When LLM is Off

One of the most underrated patterns in production LLM systems is the fallback to deterministic behavior when the LLM fails or is disabled.

In JuiceAnalyzer, the Scout is controlled by a feature flag:

# From config.py, line 54
enable_qwen_scout: bool = False

If this flag is False (default), the Scout stage returns [], and the pipeline continues with only the deterministic candidates. No service degradation. No cascading errors.

This is not a hack—it's mandatory resilience. Your infrastructure observability system cannot become dependent on an LLM API's availability. If JuiceFactory is down or your Vault token is stale, candidates should still flow, just noisier.

# From problem_candidates_refresh.py, lines 731–779
if not settings.enable_qwen_scout:
    return []

# ... run Scout ...

except Exception:
    logger.warning("Qwen scout failed; skipping llm_scout candidate generation", exc_info=True)
    return []  # Fallback to deterministic pipeline

Graceful degradation beats heroic recovery.

Embeddings for Deduplication: Prevent Repeated Hallucinations

Once candidates are flowing, a second noise-discrimination problem emerges: duplicates. Did the Scout recommend the same candidate twice with different wording? Did the Critic flag it and the Judge overrode it, creating a duplicate in the log?

Embeddings solve this. JuiceAnalyzer embeds candidates using qwen3-embed (2560-dimensional output) every 5 minutes and deduplicates by cosine similarity:

# From config.py, line 14–15
llm_embed_model: str = "qwen3-embed"
embedding_dim: int = 2560

Why embeddings over exact hashing? Because a duplicate candidate might be worded differently:

Scout candidate: "host A CPU high, correlates with latency"
Next cycle, rephrased: "CPU utilization on host A matches tail latency drift"

A high cosine-similarity threshold (illustrative—start around 0.85, then tune and validate per corpus) flags these as duplicates and merges them, preventing duplicate RCA assignments and reducing alert fatigue.

GDPR + BYOK: Privacy-First Inference

JuiceAnalyzer runs on JuiceFactory, which is built for GDPR compliance:

BYOK (Bring Your Own Key): Your Qwen models run on your infrastructure or Manpro-managed EU workers, never through third-party API endpoints.
Audit trail: Every LLM call (Scout, Judge—and any Critic stage you add) is logged with prompt + response, accessible for DPA audits.
No data leakage: Candidate metadata (entity names, metrics, timestamps) never leaves your network during inference.

When building your own multi-stage pipeline on JuiceFactory (the API is OpenAI-compatible, so most existing clients work with a base-URL swap):

from juicefactory import AsyncLLMClient

client = AsyncLLMClient(
    api_url="https://api.juicefactory.ai/v1",  # Your BYOK endpoint
    api_key=os.getenv("JUICEFACTORY_API_KEY"),  # Stored in Vault
)

# Scout request stays within your VPC
response = await client.json_completion(
    messages=[
        {"role": "system", "content": "You are a scout..."},
        {"role": "user", "content": json.dumps(seeds)},
    ],
    model="qwen3.5-27b",
    temperature=0.1,
)

Practical Implementation Checklist

When building your own Scout → Critic → Judge pipeline:

Deterministic baseline first. Your candidate producers (group_burst, entity_local, incident_cluster in the JA example) should be deterministic and auditable. Every candidate includes an evidence_summary with the rule that triggered it.
Scout on constrained input. Feed the Scout the top N seeds (not all candidates). Require structured JSON output. Use low temperature (0.1). Fail safe (return [] on any exception).
Critic is optional but recommended. If you build the adversarial stage, make it attack the Scout's reasoning, not validate it. Log the disagreement for later audit. (In JuiceAnalyzer this stage is a design pattern, not yet in the deployed path.)
Judge owns the threshold. Qualification thresholds are hardcoded, not LLM-determined. If an LLM crosses the threshold, it's logged. If it's below, it's rejected—no overrides without explicit logging.
Embed for deduplication. After scoring, embed candidate summaries. Merge near-duplicates (a high cosine threshold—illustrative, tuned and validated per corpus). This is cheap (qwen3-embed) and prevents alert fatigue.
Deterministic fallback is non-negotiable. If the LLM fails, the deterministic pipeline continues. Feature-flag all LLM stages. Test the degraded path.
Log overrides; audit them. Any time an LLM-informed decision overrides a deterministic rule, log it. After a few weeks, audit the logs: was the LLM right to override? Use this feedback to adjust thresholds.

The Skeptic's Case: Why Noise Discrimination Matters

In high-stakes systems, an LLM is not a replacement for logic—it is a noise-reduction filter on top of logic. The Scout's job is not to be intelligent; it is to be helpful by annotating candidates that deterministic rules already produced. If the Scout adds a reason that is plausible but false, the Critic (in the target design) catches it, and the Judge logs it.

This architecture assumes LLMs hallucinate and designs around it, rather than hoping they don't.

The alternative—"use a bigger model, hope it's smarter"—is a sinkhole. Bigger models hallucinate bigger, and the correlated error gets worse.

Sources & Research

Start building: Get a free JuiceFactory API key — no credit card required.

Building an Adversarial Multi-Stage LLM Validation Pipeline: Scout → Critic → Judge

The Hallucination Problem: Why Single-LLM Answers Fail at High Stakes

The Three-Stage Architecture: Scout → Critic → Judge

Stage 1: Scout (Propose from Qualified Candidates)

Stage 2: Critic (Adversarial Review)

Stage 3: Judge (Deterministic Filter + Override Log)

Why This Beats "Generate More, Filter Later"

The Deterministic Fallback: When LLM is Off

Embeddings for Deduplication: Prevent Repeated Hallucinations

GDPR + BYOK: Privacy-First Inference

Practical Implementation Checklist

The Skeptic's Case: Why Noise Discrimination Matters

Related Reading

Sources & Research

Related guides

Related Guides

GDPR-Safe AI Inference

RAG in Python: GDPR-Safe Document Search

RAG with Qwen: Private Document Search

Try JuiceFactory — Free API Key in 30 Seconds