Extracting Clean Structured Data from Messy HTML with LLMs

Every scraper knows this pain: you fetch HTML from a vendor's product page, and it's a disaster. Missing prices. Specs buried in image captions. Brand names in different formats. You build regexes, they break with the next site redesign. You hardcode extraction per vendor, then inherit 50 vendor-specific parsers from the previous dev.

LLMs can extract structured data from chaotic HTML reliably—but not if you treat them like magic. This guide shows how to build a production extraction pipeline using JuiceFactory + deterministic temperature + JSON schema validation + explicit error gates. We'll ground this in real code from a Swedish EV-charger catalog (laddbox) that extracts ~100 products/day with zero silent failures.

Start building: Get a free JuiceFactory API key — no credit card required.

The Problem with "Just Ask the LLM"

Asking an LLM to "return JSON" will fail in production in these ways:

1. Markdown code fences — the model wraps output in ```json and ```
2. Thinking tags — reasoning models emit <think>...</think> before the JSON
3. Hallucinated fields — the model adds helpful extra keys not in your schema
4. Type confusion — prices as strings ("7990"), not numbers
5. Truncation — response hits max_tokens mid-JSON, leaving invalid output
6. Schema drift — model renames keys for "clarity" (price_min → minimum_price)
7. Silent null bias — missing data becomes null instead of being absent

Fix these with three layers: schema-first prompting, low temperature for determinism, and strict validation.

Schema-First Prompting

Start with your JSON schema in the prompt, not as an afterthought. Here's the real extraction prompt from laddbox:

EXTRACTION_PROMPT = """\
Du är en produktdata-extraktör för svenska laddboxar (EV chargers).

Analysera HTML-innehållet nedan och extrahera strukturerad produktdata.
Returnera ENBART valid JSON (inget annat) med dessa fält:

{
  "slug": "{slug}",
  "brand": "{brand}",
  "model": "<modellnamn utan brand>",
  "power_kw": <float, laddeffekt i kW, t.ex. 22.0>,
  "ampere": <int, max ström i A, t.ex. 32>,
  "phases": <int, antal faser, 1 eller 3>,
  "has_load_balancing": <bool>,
  "has_app": <bool>,
  "has_wifi": <bool>,
  "has_cable": <bool, om fast kabel medföljer>,
  "cable_length_m": <float eller null>,
  "connector_type": "<Type 2 eller Type 1>",
  "price_min": <int, lägsta pris i SEK, null om ej hittad>,
  "price_max": <int, högsta pris i SEK, null om ej hittad>,
  "pros": ["<fördel 1>", "<fördel 2>", ...],
  "cons": ["<nackdel 1>", ...],
  "suitable_for": ["villa", "radhus", "brf"],
  "description": "<kort beskrivning på svenska, 1-2 meningar>",
  "features": {
    "ip_rating": "<t.ex. IP54>",
    "weight_kg": <float>,
    "warranty_years": <int>
  }
}

Regler:
- Alla priser i SEK (svenska kronor). Ignorera priser i andra valutor.
- Om ett fält inte kan hittas i HTML, sätt null (inte gissa).
- pros/cons ska vara på svenska, korta (3-5 ord per punkt).
- suitable_for: villa, radhus, brf — baserat på produktens egenskaper.
- Svara ENBART med JSON, ingen annan text.

HTML-innehåll:
{html_content}
"""

Notice:

• Type hints inline (<float>, <int>, <bool>). No guessing what field should be.
• Example values (22.0, 32). LLMs copy formats from examples.
• Explicit null rule ("Om ett fält inte kan hittas, sätt null, inte gissa"). Stops hallucinations.
• Language consistency (Swedish prompts, Swedish enum values). Reduces translation mistakes.
• No field descriptions in the schema itself—put rules in a separate "Regler" section. Models get confused by embedded explanations.

Temperature = 0.1 for Determinism

This is non-negotiable for structured extraction. Here's the API call from laddbox:

async with httpx.AsyncClient(timeout=60) as client:
    resp = await client.post(
        f"{settings.juicefactory_api_url}/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {settings.juicefactory_api_key}",
            "Content-Type": "application/json",
        },
        json={
            "model": "qwen3.5-27b",
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.1,  # ← CRITICAL: determinism over creativity
            "max_tokens": 2000,
        },
    )

The qwen3.5-27b model name here comes from our config — confirm the model you ship with. The code is vendor-agnostic; only the URL and model string change.

Why 0.1, not 0?

• temperature=0 enables greedy decoding (always the highest-probability token), which is deterministic but can get stuck in local optima.
• temperature=0.1 adds minimal randomness—enough to escape dead-ends, but low enough that output is repeatable for the same input.
• This matches OpenAI's guidance on deterministic structured extraction.

Test determinism yourself:
Extract the same HTML twice. With temperature=0.1, you should get identical JSON (or a few field switches between semantically equivalent values). With temperature=0.8, prices will bounce around.

HTML Cleaning Before Sending

Don't send raw HTML to the model. Strip noise:

def _clean_html(raw_html: str, max_chars: int = 30_000) -> str:
    """Strip scripts, styles, nav, footer — keep main content text."""
    soup = BeautifulSoup(raw_html, "html.parser")

    for tag in soup(["script", "style", "nav", "footer", "header", "noscript", "svg"]):
        tag.decompose()

    # Keep structural HTML for tables/specs
    main = soup.find("main") or soup.find("article") or soup.find("body")
    if main:
        structural = main.prettify()[:max_chars]
    else:
        text = soup.get_text(separator="\n", strip=True)[:max_chars]

    return structural

Why:

• Removes bloat — ad tags, tracking pixels, CSS. LLMs waste tokens on garbage.
• Keeps structure — tables, <strong>, <h2> signals. These help the model parse specs.
• Max char cap — prevents runaway tokens. 30K chars is ~7.5K tokens, well under most limits.

A 300KB supplier page becomes ~30KB structured content. Your API call costs 1/10th.

JSON Extraction with Error Recovery

Raw JSON from LLMs is malformed. Handle it:

try:
    content = data["choices"][0]["message"]["content"]
    
    # Strip <think>...</think> tags (reasoning models)
    content = re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL)
    
    # Strip markdown code fences if present
    content = content.strip()
    if content.startswith("```"):
        content = content.split("\n", 1)[1]
    if content.endswith("```"):
        content = content.rsplit("```", 1)[0]
    content = content.strip()
    
    # Find JSON object in response (handles leading text)
    json_match = re.search(r"\{[\s\S]*\}", content)
    if json_match:
        content = json_match.group()
    
    product_data = json.loads(content)
    
    # Force slug and brand (AI might return null or wrong values)
    product_data["slug"] = slug
    product_data["brand"] = brand
    
    # Derive model name from slug if AI didn't extract it
    if not product_data.get("model"):
        slug_parts = slug.replace(brand.lower().replace(" ", "-"), "").strip("-").split("-")
        product_data["model"] = " ".join(p.capitalize() for p in slug_parts if p)
    
    # Validate against Pydantic schema
    product = ExtractedProduct(**product_data)
    
    log.info(f"[{slug}] Extracted: {product.brand} {product.model}, "
             f"price={product.price_min}-{product.price_max} SEK")
    return product

except json.JSONDecodeError as e:
    log.error(f"[{slug}] AI returned invalid JSON: {e}")
    log.debug(f"Raw response: {content[:500]}")
    return None
except httpx.HTTPStatusError as e:
    log.error(f"[{slug}] JuiceFactory API error: {e.response.status_code}")
    return None

Each step here catches a real failure mode:

1. Regex strip <think> tags — reasoning models (Qwen 32B+) emit thinking. Strip it.
2. Strip markdown fences — ```json ... ``` happens. Unwrap it.
3. Regex find JSON object — if the model adds text before or after ("Here's the data: {...}"), this extracts just the object.
4. json.loads() — validates syntax. Catches trailing commas, missing quotes.
5. Force slug + brand — override AI in case it misread these from HTML.
6. Derive model from slug — fallback for missing model names.
7. Pydantic validation — ExtractedProduct(**product_data) ensures all types match schema. Catches price_min="7990" (string) vs. expected int.

The except blocks log both the error and the raw response (truncated to 500 chars). Save these logs—they become prompts for retries.

Pydantic Schema as Guardrail

The Pydantic model is not optional boilerplate—it's your quality gate:

class ExtractedProduct(BaseModel):
    """Product data extracted by AI from supplier HTML."""
    
    slug: str
    brand: str
    model: str
    power_kw: float | None = None
    ampere: int | None = None
    phases: int | None = None
    
    has_load_balancing: bool | None = None
    has_app: bool | None = None
    has_wifi: bool | None = None
    has_cable: bool | None = None
    cable_length_m: float | None = None
    connector_type: str | None = None
    
    price_min: int | None = None
    price_max: int | None = None
    
    pros: list[str] = Field(default_factory=list)
    cons: list[str] = Field(default_factory=list)
    suitable_for: list[str] = Field(default_factory=list)
    
    description: str | None = None
    features: dict | None = None

Pydantic catches:

• Missing required fields (slug, brand, model)
• Type mismatches (price_min as string → coerces to int, or raises ValidationError if non-numeric)
• Out-of-range values (custom validators can reject negative prices)
• Schema drift (extra fields from AI are ignored by default, or rejected if extra="forbid")

Business Logic Validation (Sanity Gates)

After Pydantic, apply domain-specific rules. JSON Schema can't express "max_price must be >= min_price" or "80K SEK is unrealistic for a charger":

def ask_ai_price(brand: str, model: str) -> tuple[int | None, int | None]:
    """Call AI to repair bad price data."""
    # ... (AI call omitted)
    
    # Sanity checks
    if pmin and isinstance(pmin, (int, float)) and 2000 <= pmin <= 40000:
        pmin = int(pmin)
        if pmax and isinstance(pmax, (int, float)) and pmax >= pmin and pmax <= 50000:
            pmax = int(pmax)
        else:
            pmax = None
        return pmin, pmax
    
    return None, None

What this catches:

• Price too low (pmin < 2000): probably a partial price or in wrong currency.
• Price too high (pmax > 50000): chargers max out ~45K SEK. Likely hallucination.
• Inverted (pmax < pmin): caught and swapped.
• Suspicious caps (see gotcha below).

From production: AI returned price_max=10000 for five products in a row (likely a training-set artifact or token limit). The sanity gate (pmax <= 50000) passed, but a domain rule (pmax == 10000 is suspicious, ask again) caught it.

AI Price Repair (Retry with Feedback)

When extraction fails, don't discard it—ask the AI again with feedback:

def ask_ai_price(brand: str, model: str) -> tuple[int | None, int | None]:
    prompt = f"""Vad kostar en {brand} {model} laddbox i Sverige just nu (2025-2026)?

Ge mig det lägsta och högsta priset i SEK som denna laddbox säljs för hos svenska återförsäljare.
Viktigt: ge ett realistiskt prisintervall. Min och max bör normalt inte skilja mer än 2000-3000 kr.

Svara ENBART med JSON:
{{"price_min": <lägsta pris i SEK, heltal>, "price_max": <högsta pris i SEK, heltal>}}

Om du inte vet priset, svara: {{"price_min": null, "price_max": null}}"""
    
    # ... (API call, temperature=0.3 for price estimation) ...

Note temperature=0.3 (vs. 0.1 for initial extraction). Prices may vary by retailer, so a bit more variation is ok. But still deterministic.

The retry pattern from laddbox:

for slug, old_min, old_max, notes in PRODUCTS_TO_FIX:
    brand, model = get_product_info(slug)
    
    try:
        new_min, new_max = ask_ai_price(brand, model)
        
        if new_min is None:
            # Fallback: apply deterministic fixes
            if old_max == 10000:
                # Remove the 10000 cap, keep min as-is
                new_min = old_min
                new_max = None
        else:
            # If AI also returns 10000 as max, that's suspicious
            if new_max == 10000:
                log.warning(f"AI returned price_max=10000 again, likely a bias")
                new_max = None
        
        update_prices(slug, new_min, new_max)

Retry strategy:

1. First pass: structured extraction, temperature=0.1.
2. If price fails: AI-driven repair, temperature=0.3.
3. If repair fails: deterministic fallback (remove known-bad caps, swap inverted prices).
4. If all fail: log and skip (don't store bad data).

No infinite retries. Cap at 2–3 attempts. If the LLM can't extract it after three tries, the data is likely outside the training distribution or genuinely ambiguous.

Where JuiceFactory Fits

JuiceFactory is one of several EU-hosted LLM providers offering a GDPR-compliant gateway. For structured extraction:

• OpenAI-compatible API — drop-in replacement for OpenAI clients. Same POST https://api.juicefactory.ai/v1/chat/completions interface.
• Structured-output / JSON support — the API accepts the same JSON-mode and structured-output parameters your OpenAI client already sends.
• Qwen-family models — strong for non-English structured extraction and deterministic output. (Confirm the specific model you ship with; the code here is vendor-agnostic.)
• Temperature control — full support for temperature=0.1 (no abstraction layers hiding it).
• BYOK (Bring Your Own Key) — runs on your account, no data sent through third-party inference APIs.
• EU residency — a stateless, EU-hosted design keeps data in the EU.

For the laddbox case: ~100 products/day, each HTML ~5K tokens, ~2000 completion tokens. Cost per extraction is roughly a cent or two at Qwen pricing (an estimate — confirm current JuiceFactory pricing). Zero data leakage to third-party LLM providers.

# Just works with JuiceFactory
async with httpx.AsyncClient() as client:
    resp = await client.post(
        "https://api.juicefactory.ai/v1/chat/completions",
        headers={"Authorization": f"Bearer {settings.juicefactory_api_key}"},
        json={
            "model": "qwen3.5-27b",
            "messages": [...],
            "temperature": 0.1,
            "max_tokens": 2000,
        },
    )

No vendor lock-in. If you later switch to a different provider, the code doesn't change (only the URL and model name).

Gotchas from Production

1. The 10K Price Ceiling

AI trained on web data learns implicit "nice round numbers." Many products were capped at 10K SEK in training data. The model memorized this.
Fix: Add a domain rule: if price_max == 10000, ask AI again or cap it to price_min + domain_max_spread.

2. Type Confusion on Numeric Fields

LLM returns {"price_min": "7990"} (string). JSON Schema allows it, Pydantic catches it.
Fix: Always run through Pydantic. Don't rely on JSON Schema alone.

3. Markdown Code Fences

Some LLMs (especially Qwen) wrap JSON in ```json ... ```. Older versions of some inference servers propagate this.
Fix: The regex strip shown earlier handles this. Keep it.

4. Thinking Tags from Reasoning Models

If using reasoning variants, the model emits <think>...</think> before JSON.
Fix: Regex strip (re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL)).

5. Max Tokens Cutoff

If max_tokens is too low, the JSON gets truncated mid-object ({"price_min": 7990, "pr...).
Fix: Use max_tokens at least 2x your schema size. Test with the largest products first.

6. Schema Drift (Helpful Hallucinations)

Model adds extra fields not in your schema ("supplier_url", "in_stock_at", "rating"). If you set extra="forbid" in Pydantic, these kill the whole extraction.
Fix: Set extra="ignore" (default) to silently drop unknowns, or collect them in a _extra field for logging.

7. Truncation Without Error

If the HTML is too long, some models silently ignore the end. "Cable not mentioned in this excerpt" becomes cable_length_m: null.
Fix: Trim HTML aggressively (30K chars max). If you lose data, re-run with a smaller subset.

Checklist for Production

Before you ship:

• Schema documented — every field has a type hint and example value
• Temperature set to 0.1 — no randomness in structured extraction
• HTML cleaned — strip scripts, styles, nav before sending
• JSON unpacking bulletproof — strip markdown, thinking tags, find JSON regex
• Pydantic validation — all fields validated, type coerced or rejected
• Sanity gates — domain-specific rules (price ranges, inverted checks)
• Error logging — capture raw response + error for debugging
• Retry strategy — fallback for common failures (10K cap, missing prices)
• Monitoring — track extraction success rate per product, per vendor
• Graceful failure — return None on repeated failures, never store bad data

Measuring Quality

Track these metrics. The targets below are our operating targets, not guarantees — calibrate them to your own catalog and model:

If JSON parse success drops below your target, the model or prompt has likely drifted. Re-test temperature, re-check the prompt for ambiguities, or retry with a larger model.

Next Steps

1. Clone the schema — start with the ExtractedProduct Pydantic model shown here.
2. Set temperature=0.1 — do not skip this.
3. Add sanity gates — domain rules specific to your product catalog.
4. Monitor and iterate — track parse success, collect failure cases, refine the prompt.
5. Self-host or use JuiceFactory — don't send data through public APIs for GDPR/confidentiality.

Further reading:

• Structured outputs best practices (OpenAI)
• Pydantic validation for LLM outputs
• LLM output parsing guide (Tetrate)
• JuiceFactory docs: api.juicefactory.ai

Start building: Get a free JuiceFactory API key — no credit card required.

Related guides

Related guides: EU LLM API Comparison · Adversarial LLM Validation · Multi-Modal AI Assistant · RAG with Qwen