Can I build a RAG pipeline without OpenAI?

Yes. Use an OpenAI-compatible EU provider like JuiceFactory for both embeddings and generation — the official OpenAI SDKs work by changing only the base URL and API key, so no OpenAI account is required.

What do I need to replace OpenAI in RAG?

An embedding model and a chat model. JuiceFactory hosts both Qwen embeddings and Qwen chat behind one OpenAI-compatible endpoint, so switching is a base-URL and key change rather than a rewrite.

Is self-hosted RAG with Qwen production-ready?

Yes. The patterns in this guide — semantic verification gates, caching, retries and concurrency limits — come from live production systems processing thousands of documents.

How to Build a Self-Hosted RAG Pipeline Without OpenAI (Using Qwen + BYOK)

If you're building a RAG system in 2026 and don't want to ship your domain-specific data to OpenAI, the tooling has matured. This guide walks through a real, production-tested pattern for EU-hosted Qwen RAG: semantic decomposition → embedding-verified rendering → indexed retrieval, all on self-hosted infrastructure using Qwen models and BYOK (Bring Your Own Key) licensing.

We'll use two live systems as reference: regelguiden (Swedish regulatory RAG with 6-stage decomposition pipeline) and growth-intelligence (evidence RAG with cross-lingual semantic dedup). Both run on JuiceFactory's self-hosted Qwen GPUs—no external API calls, full data control.

Start building: Get a free JuiceFactory API key — no credit card required. The same BYOK key also works directly inside your editor — see our Cursor BYOK setup guide.

Why Off-OpenAI?

Three reasons teams move:

Data residency: EU regulations (GDPR, NIS2) make external APIs risky. Your regulatory text, proprietary claims, or internal evidence stay on-premise or in your own cloud.
Cost at scale: At 5000+ daily embedding requests or multi-modal inference, OpenAI token costs stack. BYOK shifts cost to fixed compute.
Latency & control: You're not competing for shared API quota during peak hours. Inference predictability matters for search SLAs.

The catch? You own the pipeline complexity. This guide shows how three production teams handle it.

The Pipeline: 6 Stages

Stage 1: Atomic Decomposition

Raw text arrives messy—headers, definitions, boilerplate, legal citations mixed together. The first move: break it into atomic facts.

# regelguiden: decompose raw regulatory text into atomic rules
MODAL_RE = re.compile(r'\b(ska|får|måste|bör|kan|behöver|är skyldig)\b', re.I)
NEGATION_RE = re.compile(r'\b(inte|ej|förbjudet|aldrig)\b', re.I)

class AtomicRule:
    rule_index: int              # position in decomposition
    rule_text: str               # exact substring from original
    byte_start: int              # char position
    byte_end: int
    rule_type: str               # obligation|permission|prohibition|condition|non_rule
    modal: str                   # the obligation verb (ska/får/måste)
    polarity: str                # positive|negative
    conditions: List[str]        # ["om X", "under förutsättning Y"]

Why byte_start/byte_end? Traceability. If your vector search ranks a rule high, you can jump directly to the original regulatory text. No hallucinated reformulation—always cite the source.

Regelguiden does this in two passes:

Fast heuristic (spaCy): Deterministic sentence-splitting via regex + modal-verb detection. Covers ~85% of text without LLM cost.
Complex refinement (Sonnet): Sections flagged as complex (multiple modal verbs, chained conditions) get Sonnet re-decomposition. Runs in background after indexing is live.

The regex tier is cheap and reproducible. The LLM tier is fallback for edge cases—a cost you control.

Stage 2: Structural Validation

At this point, you have ~800 candidate rules. Before rendering, validate:

class StructuralValidator:
    def validate(self, rules, raw_text) -> List[str]:
        errors = []
        
        # 1. rule_text must be an exact substring
        if rule.rule_text not in raw_text:
            errors.append(f"rule_text not a substring: {rule.rule_text[:60]!r}")
        
        # 2. byte_start/byte_end must match
        pos = raw_text.find(rule.rule_text)
        if pos >= 0:
            rule.byte_start = pos
            rule.byte_end = pos + len(rule.rule_text)
        
        # 3. Soft dedup: mark duplicates as non_rule, don't fail hard
        # Reason: scrapers collect boilerplate (nav text) multiple times
        key = rule.rule_text.strip()
        if key in seen_texts:
            rule.rule_type = 'non_rule'
        seen_texts.add(key)
        
        # 4. Modal verb must appear in rule_text
        if rule.modal and rule.rule_type != 'non_rule':
            if rule.modal not in rule.rule_text.lower():
                errors.append(f"modal '{rule.modal}' missing from text")
        
        return errors

The key discipline: No facts beyond what's in raw_text. If the decomposer (Sonnet) invents a modal verb or misses a condition, the validator catches it before rendering.

Stage 3: Plain-Language Rendering

Now render each atomic rule into one clear sentence using Qwen 27B.

RENDER_TEMPLATE = """\
Skriv en klar, enkel mening som uttrycker denna regel:

Regeltext (original): {rule_text}
Typ: {rule_type}
Modal: {modal} ← detta ord MÅSTE finnas i din mening
Polaritet: {polarity}
Villkor: {conditions_text}

Krav:
- Exakt ett modal-verb ({modal})
- Max 2 meningar
- Alla villkor måste nämnas
- Inga egna tillägg
- Håll dig semantiskt nära originalet

Svara med BARA den färdiga meningen."""

# OpenAI-compatible client to JuiceFactory
client = OpenAI(api_key=JUICEFACTORY_API_KEY, 
                base_url='https://api.juicefactory.ai/v1')

response = client.chat.completions.create(
    model='qwen3.5-27b',
    messages=[
        {'role': 'system', 'content': 'Du är en textformaterare för juridisk info...'},
        {'role': 'user', 'content': '/no_think\n' + prompt},
    ],
    temperature=0.1,
    max_completion_tokens=300,
)

The /no_think directive tells Qwen to skip its thinking-tokens—you want speed, not reasoning overhead.

Parallelism matters: Regelguiden renders large rule batches quickly by issuing many concurrent Qwen requests—tune the concurrency to your own quota (confirm current limits against the JuiceFactory docs). Cache is aggressively used:

RENDER_CACHE_BASE = os.path.join('/app/data/glm_cache_v5', 'render', RENDER_VERSION)
# Key: sha256(atomic_rule_json) → rendered text
# When RENDER_VERSION bumps, old cache invalidates; new renders queue

This means re-running the pipeline against the same source is near-instant if no rules changed.

Stage 4: Fidelity Checking (Before Embeddings)

Before you compute an embedding (which is expensive to change later), verify the rendering is faithful.

class FidelityChecker:
    def check(self, rule, plain_swedish) -> Tuple[bool, dict, str]:
        ps_lower = plain_swedish.lower()
        issues = []
        
        # 1. Modal verb must be present
        if rule.modal and rule.rule_type != 'non_rule':
            synonyms = {'ska': ['ska', 'skall'], 'får': ['får'], ...}
            if not any(m in ps_lower for m in synonyms.get(rule.modal, [])):
                issues.append(f"Modal '{rule.modal}' missing—add it")
        
        # 2. Negation must match polarity
        if rule.polarity == 'negative':
            negation_markers = {'inte', 'ej', 'förbjudet', 'aldrig'}
            if not any(m in ps_lower for m in negation_markers):
                issues.append("Prohibition but no negation—add 'inte' or 'ej'")
        else:
            # Don't add negation that wasn't in the original
            orig_has_negation = any(m in rule.rule_text.lower() for m in negation_markers)
            if any(m in ps_lower for m in negation_markers) and not orig_has_negation:
                issues.append("Positive rule but negation added—remove it")
        
        # 3. Conditions coverage
        if rule.conditions:
            for cond in rule.conditions:
                cond_words = set(re.findall(r'\w{4,}', cond.lower()))
                ps_words = set(re.findall(r'\w{4,}', ps_lower))
                if len(cond_words & ps_words) < min(2, len(cond_words)):
                    issues.append(f"Condition '{cond}' not covered")
        
        # 4. Length sanity (rendered ≤ 3x original)
        ratio = len(plain_swedish) / len(rule.rule_text) if rule.rule_text else 1
        if ratio > 3.0:
            issues.append(f"Rendered 3x longer than original ({ratio:.1f}x)—shorten")
        
        return len(issues) == 0, {check: True for check in [...]}, ' | '.join(issues)

If fidelity fails, retry Qwen with specific feedback:

if not all_passed:
    feedback = "Feedback: " + feedback_string
    response = client.chat.completions.create(
        model='qwen3.5-27b',
        messages=[
            {'role': 'user', 'content': f'/no_think\nDin mening hade problem:\n{feedback}\n...försök igen'}
        ],
        temperature=0.1,
        max_completion_tokens=300,
    )

Retry budget: Max 3 attempts per rule. If it still fails, mark it needs_manual_review and skip to the next rule. You don't gate production on LLM stubborness.

Stage 5: The Embedding Verification Gate (The Clever Bit)

Here's where most teams get RAG wrong. They embeddings-in-the-dark: compute e(rule_text) and e(rendered_text), then cross-fingers the rendering was faithful.

Regelguiden does explicit semantic verification:

class CosineVerifier:
    def __init__(self, threshold=0.75):  # domain-tuned for legal docs
        self.threshold = threshold
    
    def verify(self, rule, plain_swedish) -> Tuple[bool, float, str]:
        # Compute embeddings in one batch call (cheaper)
        vecs = embed_texts([rule.rule_text[:500], plain_swedish[:500]])
        
        # cosine_similarity: normalized vectors → dot product
        score = cosine_similarity(vecs[0], vecs[1])
        
        passed = score >= self.threshold
        feedback = (
            f"Semantisk likhet för låg (cosine={score:.3f} < {self.threshold}). "
            f"Håll dig närmre originaltexten."
        ) if not passed else ''
        
        return passed, score, feedback

Why 0.75 for legal? Regulatory text is strict. A prohibition reversed by accident (e.g., "not allowed" → "allowed") is catastrophic. A high threshold catches semantic drift. Treat this as a starting point and validate it against your own corpus.

For narrative RAG (blog posts, internal docs), thresholds drop to 0.6–0.7. For entity-dense content (product specs), 0.8+.

The embedding model matters. Regelguiden uses qwen3-embed (JuiceFactory):

High-dimensional Qwen3 embeddings (confirm the current dimensionality in the model docs), cross-lingual across EN↔SV
Unit-normalized → cosine is just dot product
No external vendor—runs on EU-hosted infrastructure

def embed_texts(texts: List[str]) -> List[List[float]]:
    client = OpenAI(api_key=JUICEFACTORY_API_KEY,
                    base_url='https://api.juicefactory.ai/v1')
    response = client.embeddings.create(model='qwen3-embed', input=texts)
    return [item.embedding for item in sorted(response.data, key=lambda x: x.index)]

If cosine verification fails, feedback loops back to rendering—Qwen tries again, armed with "your semantic drift is too high."

Stage 6: Indexing with Full Traceability

Once a rule passes all gates, index it with full lineage:

raw_section
  ↓ raw_section_id = source_url + section_anchor
  ├─→ atomic_rule
  │    ├─ raw_section_id
  │    ├─ byte_start, byte_end
  │    ├─ rule_text (exact substring)
  │    └─ is_flagged (complexity marker)
  │
  └─→ atomic_render
       ├─ atomic_rule_id
       ├─ plain_swedish
       ├─ render_model ('qwen3.5-27b')
       ├─ cosine_score
       └─ all_checks_passed (bool)
         ↓
       Weaviate (vector DB)
         ├─ embedding: qwen3-embed(plain_swedish)
         ├─ citation_id: sha256(source_url + anchor + v + rule_index)
         └─ full traceability back to raw_section

Why Weaviate? Open-source, no vendor lock, integrates with Python. Growth-intelligence uses it for embedding-similarity search + aggregation:

# Query: retrieve top-10 rules semantically similar to user question
question = "Är det tillåtet för husägare att bygga på sin tomt?"
q_embed = embed_texts([question])[0]

results = weaviate_collection.query.near_vector(
    near_vector=q_embed,
    limit=10,
    return_properties=['citation_id', 'plain_swedish', 'rule_type']
)

# Each result links back to regulatory text via citation_id
for result in results:
    rule_text = fetch_from_postgres_by_citation_id(result['citation_id'])
    print(f"Found: {result['plain_swedish']}")
    print(f"Source: {rule_text}")

The Dedup + Escalation Pattern (Growth Intelligence)

Regelguiden handles one-time text decomposition. Growth-intelligence is different: it ingests claims from multiple sources over time, and semantically deduplicates them.

# embedder.py: semantic dedup of claims
async def run(args):
    conn = await connect()
    
    # 1. Embed missing claims
    missing = await conn.fetch("""
        SELECT id, claim FROM evidence_claims
        WHERE status='active' AND embedding IS NULL
        LIMIT $1
    """, args.max_embed)
    
    # Batch embed via JuiceFactory
    async with aiohttp.ClientSession() as session:
        for i in range(0, len(missing), BATCH):
            chunk = missing[i:i + BATCH]
            vecs = await embed_texts(session, 
                                     [c["claim"][:2000] for c in chunk])
            for c, v in zip(chunk, vecs):
                await conn.execute("UPDATE evidence_claims SET embedding=$1 WHERE id=$2",
                                  v, c["id"])
    
    # 2. Semantic dedup: greedy clustering per claim_type
    rows = await conn.fetch("""
        SELECT id, claim, claim_type, source_trust, confidence, 
               embedding, ingested_at
        FROM evidence_claims
        WHERE status='active' AND embedding IS NOT NULL
    """)
    
    # Group by type; sort by strength (trust → confidence → corroboration → age)
    by_type = {}
    for r in rows:
        by_type.setdefault(r["claim_type"], []).append(r)
    
    for claim_type, claims in by_type.items():
        claims.sort(key=lambda r: (float(r["source_trust"]), 
                                    float(r["confidence"]),
                                    int(r["corroboration_count"]),
                                    -timestamp(r["ingested_at"])), 
                    reverse=True)
        
        kept = []  # (id, normalized_vector)
        for r in claims:
            v = np.asarray(r["embedding"], dtype=np.float32)
            v = v / (np.linalg.norm(v) or 1.0)  # unit-normalize
            
            best_id, best_cosine = None, 0.0
            for kid, kv in kept:
                cosine = float(np.dot(v, kv))
                if cosine > best_cosine:
                    best_id, best_cosine = kid, cosine
            
            # If near-duplicate (cosine ≥ 0.93), mark as superseded
            if best_id is not None and best_cosine >= 0.93:
                await conn.execute("UPDATE evidence_claims SET status='superseded' WHERE id=$1", r["id"])
                await conn.execute("""
                    INSERT INTO evidence_claim_links (from_claim, to_claim, link_type)
                    VALUES ($1, $2, 'duplicate') ON CONFLICT DO NOTHING
                """, r["id"], best_id)
                # Bump corroboration count on canonical claim
                await conn.execute("""
                    UPDATE evidence_claims 
                    SET corroboration_count = COALESCE(corroboration_count, 0) + 1
                    WHERE id=$1
                """, best_id)
            else:
                kept.append((r["id"], v))

Key insight: The dedup threshold (0.93) is higher than the rendering verification threshold (0.75). Why?

Rendering verification: "Is this paraphrase faithful to the original rule?" → Higher tolerance for reformulation.
Dedup: "Is this the exact same claim, just worded differently?" → Much stricter.

This cascading-threshold pattern prevents both false positives (marking different claims as duplicates) and false negatives (letting near-identical claims proliferate).

Confidence Routing: When Qwen Isn't Enough

For high-stakes queries (e.g., regulatory interpretation), Growth Intelligence routes to a stronger GLM model if Qwen's confidence is low:

async def reason_with_escalation(system, user, *, confidence_floor=0.6):
    """Run Qwen; if confidence < floor, escalate to GLM."""
    msgs = [
        {"role": "system", "content": system},
        {"role": "user", "content": user}
    ]
    
    # Qwen's fast answer
    res = await ask_json(msgs, tier="qwen")
    conf = float(res.get("confidence", 1.0))
    
    if conf < confidence_floor and GLM_KEY:
        # Escalate to GLM (more expensive, more capable)
        res_glm = await ask_json(msgs, tier="glm")
        return {
            "result": res_glm,
            "tier": "glm",
            "escalated": True,
            "confidence": float(res_glm.get("confidence", conf)),
            "qwen_confidence": conf
        }
    
    return {
        "result": res,
        "tier": "qwen",
        "escalated": False,
        "confidence": conf
    }

Both Qwen and GLM speak the OpenAI-compatible protocol, so the same httpx client works for both—point the GLM tier at whichever OpenAI-compatible endpoint your provider gives you (confirm the current GLM endpoint in your provider's docs). The prompt asks for a confidence field (0–1), and the orchestration layer decides escalation.

Production Gotchas

1. Cache Invalidation

Regelguiden's two-level cache is critical but dangerous:

glm_cache_v5/decomp/{PIPELINE_VERSION}/{sha256(raw_text)}.json
glm_cache_v5/render/{RENDER_VERSION}/{sha256(atomic_rule_json)}.json

Gotcha: If you fix a decomposition bug and don't bump PIPELINE_VERSION, old corrupted decompositions stay cached forever.

Fix: Version your schema (decomp, render, embedding model) and invalidate aggressively during development. In production, automated versioning:

PIPELINE_VERSION = "3.0.0" (bumped when decomposition logic changes)
RENDER_VERSION = "3.0.0" (bumped separately when rendering templates change)
INGESTION_VERSION_V3 = 4 (bumped for Weaviate schema changes)

2. Embedding Drift

You embed rule_text and rendered_text, then compare cosine. If your embedding model changes (e.g., you upgrade from qwen3-embed to a larger Qwen embedding model), all old embeddings are stale.

Fix: Store embedding_model alongside every embedding. When you upgrade, re-embed in bulk and update DB — a few hundred to a few thousand items re-embed quickly with parallel requests (exact timing depends on your hardware and quota).

3. Chunking Boundaries

Semantic chunking (detecting topic shifts via cosine) works for narrative. Regelguiden uses byte-level precision instead, because regulatory text is unforgiving:

A comma in the wrong place = different rule
A reversed negation = opposite meaning

Instead of computing cosine between consecutive sentences, regelguiden uses regex + modal-verb detection. For your RAG, choose based on domain:

Legal/regulatory: byte-precise, LLM-validated
Tech docs/blog: semantic chunking (cosine ≈ 0.7 threshold)
Product specs: hybrid (semantic + length-based)

4. Concurrency Limits

Regelguiden caps concurrency per tier. These are illustrative values from one deployment—tune them to your own quota:

SONNET_CONCURRENCY = 4      # Sonnet is expensive; serialize decomposition
QWEN_CONCURRENCY = 40       # Qwen is cheap; parallelize rendering
EMBED_CONCURRENCY = 15      # Embeddings are per-API-batch; batch carefully

Fire too many concurrent requests without rate-limiting and you risk 429 (too many requests) responses. Check your plan's current limits in the JuiceFactory docs, and gate with semaphores:

_qwen_sem = threading.Semaphore(QWEN_CONCURRENCY)

with _qwen_sem:
    response = client.chat.completions.create(...)

5. Error Recovery

A single rule failing shouldn't nuke the entire run. Regelguiden retries up to 3 times, then marks as needs_manual_review:

for attempt in range(MAX_RENDER_RETRIES):
    plain_swedish = renderer.render(rule, feedback if attempt > 0 else None)
    passed, checks, feedback = fidelity.check(rule, plain_swedish)
    
    if passed:
        # Passed fidelity; proceed to embedding verification
        verified, cosine_score, verify_feedback = verifier.verify(rule, plain_swedish)
        if verified:
            break  # Success
        feedback = verify_feedback
    
    if attempt == MAX_RENDER_RETRIES - 1:
        # Last attempt failed; mark and skip
        db_mark_rule(rule_id, 'needs_manual_review', feedback)
        break

This keeps the pipeline running even if a small fraction of rules are stubborn.

Where JuiceFactory Fits

JuiceFactory provides the compute layer—no more:

What	Who	Where
Decomposition (Sonnet via claude CLI)	You	Your container/CI
Rendering (Qwen 27B)	JuiceFactory	`https://api.juicefactory.ai/v1/chat/completions`
Embeddings (qwen3-embed)	JuiceFactory	`https://api.juicefactory.ai/v1/embeddings`
Validation & orchestration (Python)	You	Your container/CI
Vector indexing (Weaviate)	You	Your infra or cloud
Citation retrieval (Postgres)	You	Your database

One endpoint (api.juicefactory.ai/v1), OpenAI-compatible client, BYOK token:

client = OpenAI(
    api_key=os.getenv('JUICEFACTORY_API_KEY'),
    base_url='https://api.juicefactory.ai/v1'
)

# Both of these work the same way
response = client.chat.completions.create(model='qwen3.5-27b', ...)
embedding = client.embeddings.create(model='qwen3-embed', input=[...])

The same BYOK token also works directly inside your editor — see our Cursor BYOK setup guide. The rest—decomposition, validation, orchestration, caching, indexing—is your code.

Rough Numbers

These are order-of-magnitude figures from one deployment on specific hardware—treat them as illustrative, not benchmarks. Your throughput depends on GPU, batch sizes, quota, and cache state.

For a corpus of a couple of thousand regulatory sections (roughly 10–15k rules after decomposition), a full re-run lands in the single-digit-minutes range, dominated by decomposition and rendering, with indexing a small tail. Cached re-runs are near-instant when no rules changed.

For incremental evidence ingestion (on the order of dozens of new claims per day), batch embedding and semantic dedup complete in seconds. Re-embedding the whole claim set after a model upgrade is a one-off bulk job measured in tens of seconds for a few thousand claims.

Next Steps

Start with static text (regulatory docs, internal wiki, product spec). Decompose → render → verify → index.
Tune thresholds for your domain. Cosine ≥ 0.75 for legal? 0.65 for blog posts? Measure accuracy on a gold set.
Build the feedback loop. Log cosine scores, fidelity check failures, user-marked errors. Re-train your threshold.
Scale incrementally. Don't embed everything at once. Batch in chunks, monitor rate limits, add circuit breakers.
Monitor semantic drift. If your embedding model changes or your domain evolves, re-embed strategically.