Orchestrating a Multi-Modal AI Assistant (Vision + Embeddings + Chat) That Actually Helps Users

Most AI assistants do one thing: talk. A truly useful assistant does three: see (analyze images), retrieve (find relevant knowledge), and converse (reason over context). This guide shows how to build a real-world multi-modal assistant that knows when to call each mode, how to route data between them, and how to keep latency and costs reasonable.

We'll use a keto meal-planning assistant as the working example — treated here purely as a code pattern, it's simple enough to follow, complex enough to show the orchestration patterns you'll reuse in any domain. The code runs on JuiceFactory's EU-hosted Qwen models (qwen3.5-27b for chat, qwen3-vl for vision, qwen3-embed for embeddings), all served over one OpenAI-compatible endpoint at https://api.juicefactory.ai/v1.

Start building: Get a free JuiceFactory API key — no credit card required.

Why Multi-Modal Orchestration Matters

A single-modality assistant forces workarounds. A text-only assistant can't analyze meal photos; a vision-only model can't plan week-long keto regimens. Integrating all three creates friction:

• Vision is slow: base64-encoding image data + 120s inference time means you can't call it reflexively. You need to decide when to analyze a photo.
• Embeddings are cheap but need context: embedding every possible ingredient is noise; you embed on-demand when the chat model signals it needs data.
• Chat is dense but stateless: it can't fetch live data; you must retrieve, then re-prompt with grounded facts.

The orchestration layer—the "conductor"—decides which model to call and when. Done wrong, you end up with 5+ API round-trips per user query. Done right, one or two.

The Three Modalities and How They Connect

1. Chat: The Conversational Core

The chat model is your primary interface. It's dense, opinionated, and stateless. In the keto assistant, it's trained to recognize when it needs external data:

# From llm_assistant.py
async def answer_with_food_lookup(
    self, request: LLMRequest, db_session
) -> Dict[str, Any]:
    """Fråga med automatisk livsmedelsuppslag vid behov."""
    messages = self._build_messages(request)

    try:
        # Initial response from chat model
        initial_text = await self.client.chat(messages, max_tokens=2048)

        # The model can signal it needs nutrition data
        lookup_request = self._extract_lookup_request(initial_text)

        if lookup_request:
            # Retrieve grounded data
            food_data = await self._lookup_foods(lookup_request, db_session)

            # Re-prompt with facts
            food_info = "Här är näringsinformation:\n\n"
            for name, info in food_data.items():
                food_info += (
                    f"- {name}: {info.get('calories', 0)} kcal, "
                    f"{info.get('fat', 0)}g fett, {info.get('protein', 0)}g protein, "
                    f"{info.get('carbs', 0)}g kolh. per 100g\n"
                )

            messages.append({"role": "assistant", "content": initial_text})
            messages.append({"role": "user", "content": food_info + "\nBesvara min ursprungliga fråga med denna data."})

            # Final response with facts
            final_text = await self.client.chat(messages, max_tokens=2048)
            return self._parse_response(final_text)

The key insight: the model doesn't just ask for data, it signals intent with a structured JSON response:

{"lookup_foods": ["broccoli", "ground beef"]}

The conductor then retrieves and re-prompts. This is a lightweight "partial-RAG" loop—not embedding every query, but grounding the response when needed.

2. Vision: Photo Analysis at the Gateway

When a user uploads a meal photo, vision runs once, not repeatedly. It's expensive (120s timeout), so you invoke it deliberately:

async def analyze_food_photo(self, image_data: bytes, mime_type: str = "image/jpeg") -> Dict[str, Any]:
    """Analysera en matbild med vision-modellen (qwen3-vl)."""
    try:
        text = await self.client.analyze_image(
            image_data, VISION_FOOD_PROMPT, mime_type=mime_type, max_tokens=2048,
        )
        # Vision model returns structured JSON
        parsed = self._extract_json(text)
        if parsed:
            return {"analysis": parsed, "raw_response": text}
        return {"text_response": text}
    except Exception as e:
        logger.error(f"Vision-analysfel: {e}")
        return {"error": str(e), "text_response": "Kunde inte analysera bilden."}

The vision prompt is engineered to return JSON—portion sizes, macros, keto-friendliness:

{
  "foods": [
    {"name": "chicken breast", "portion_g": 200, "kcal": 330, "fat_g": 7, "protein_g": 62, "carbs_g": 0}
  ],
  "total": {"kcal": 330, "fat_g": 7, "protein_g": 62, "carbs_g": 0},
  "keto_friendly": true,
  "comment": "Low-carb meal with excellent protein."
}

Vision output flows into the chat context, not as a separate step. The user sees: "Photo analyzed. Identified: 200g chicken breast (330 kcal, 62g protein). Plan your meal?" Then chat can reason over the identified foods.

3. Embeddings: Ingredient Search Without Roundtrips

Embeddings are the quiet hero. They enable fuzzy ingredient lookup in milliseconds, without forcing the chat model to hallucinate nutrition facts.

When the chat model signals {"lookup_foods": ["...something vague..."]}, the conductor can:

1. Search the database with embeddings (fast, local, cheap):

   # Pseudo-code: embed the food name, search vector DB for nearest neighbors
   embedded_query = await client.embed(["user's food query"])
   similar_foods = vector_db.search(embedded_query, top_k=3)
   

2. Fallback to API if not found locally:

   async def _lookup_single_food(self, food_name: str, db_session) -> tuple:
       db_foods = enhanced_food_crud.search(db=db_session, query=food_name, limit=1)
       
       if db_foods:
           return (food_name, {...nutrition...})
       
       # If not in DB, query Livsmedelsverket API (Sweden's nutrition database)
       api_foods = await livsmedelsverket_api.search_foods(food_name, limit=3)
       if api_foods:
           # Extract details and return
           ...
   

Note: The vector-DB / embedding-search layer and the external nutrition-API fallback (Livsmedelsverket) shown here illustrate the retrieval pattern — confirm both against your current setup before relying on them, since the concrete backend and API status vary per deployment.

The embedding model (qwen3-embed) runs inline, with a 30-second timeout. No round-trip to the user—all happens server-side.

The Orchestration Pattern: Sequential + Parallel Where It Counts

Here's the real flow:

User: "I ate chicken and broccoli. Plan my week."
    ↓
[CHAT] Initial response: "I need nutrition data" + {"lookup_foods": ["chicken", "broccoli"]}
    ↓
[EMBED/LOOKUP] Retrieve facts (parallel DB search + API fallback, <1s)
    ↓
[CHAT] Re-prompt with facts: "Here's the data. Now I'll plan your week..."
    ↓
User sees: Final meal plan with macros, grounded in real nutrition data

Or (alternate path):

User: [Uploads meal photo]
    ↓
[VISION] Analyze image (120s timeout, semaphore = 1 concurrent per user session)
    ↓
[CHAT] "Photo shows 200g chicken. What would you like to do?"
    ↓
User: "Add this to Monday's lunch"
    ↓
[EMBED] Search ingredient DB (if needed)
    ↓
[CHAT] Final response with updated plan

The magic is routing: not every user query needs vision; not every query needs lookup. The chat model signals what's needed, and the conductor dispatches.

Concurrency & Latency: Keep the UX Snappy

Multi-modal doesn't have to mean slow. Here's how the keto assistant manages load:

class JuiceFactoryClient:
    def __init__(self):
        self.semaphore = asyncio.Semaphore(5)  # Max 5 concurrent requests

Each endpoint has a timeout:

• Chat: 90 seconds (dense reasoning)
• Vision: 120 seconds (image processing + feature extraction)
• Embeddings: 30 seconds (vector math)

For a busy multi-tenant system:

1. Semaphore limits runaway concurrency (5 requests, not 500).
2. Timeouts prevent hanging requests from blocking the thread pool.
3. Re-prompting (chat → retrieve → chat) is sequential within one user session, but parallel across users.

A real deployment would:

• Cache embedding results (TTL 3600s, LRU eviction)
• Cache food lookups in-process or Redis
• Pool HTTP connections via httpx
• Monitor p50/p95 latency per modality

food_cache = FoodCache(max_size=1000, ttl=3600)  # 1000 foods, 1hr TTL

Error Handling: What Breaks, and How to Recover

Vision timeouts are common. Embeddings fail when the vector DB is slow. Chat API occasionally errors. Here's the pattern:

async def chat(self, messages, *, temperature=0.7, max_tokens=None, json_mode=False) -> str:
    if not self.api_key:
        return "LLM ej konfigurerad."
    
    try:
        async with self.semaphore:
            async with httpx.AsyncClient(timeout=90.0) as client:
                resp = await client.post(...)
                if resp.status_code >= 400:
                    logger.error(f"LLM API error {resp.status_code}: {resp.text[:500]}")
                    resp.raise_for_status()
                data = resp.json()
                return data["choices"][0]["message"]["content"]
    except Exception as e:
        logger.error(f"LLM-fel: {e}")
        return "Tyvärr kunde jag inte besvara din fråga just nu."

Best practices:

• Log the first 500 chars of errors (enough to debug, not a privacy leak).
• Return a friendly fallback message to the user.
• Let the client retry; don't retry server-side.
• Tag logs by modality (LLM-chat, LLM-vision, LLM-embed) for monitoring.

Where JuiceFactory Fits: BYOK, EU, One Endpoint

The orchestration code above works because JuiceFactory offers:

1. OpenAI-compatible API: one client, three endpoints (/chat/completions, /chat/completions for vision, /embeddings).
2. BYOK (Bring Your Own Key): no vendor lock-in. You provision Qwen models on JuiceFactory infrastructure, not OpenAI's.
3. EU-hosted: data stays in the EU (GDPR compliance). In this pattern, user data never leaves the EU.
4. Model bundling: qwen3-vl (vision), qwen3-embed (embeddings), and qwen3.5-27b (chat) in one dashboard. No juggling three vendor accounts.

The JuiceFactoryClient is lightweight—just HTTP, no SDK overhead:

class JuiceFactoryClient:
    def __init__(self):
        self.api_base = settings.LLM_API_BASE  # e.g., https://api.juicefactory.ai/v1
        self.api_key = settings.LLM_API_KEY
        self.model_chat = settings.LLM_MODEL_CHAT  # qwen3.5-27b
        self.model_vision = settings.LLM_MODEL_VISION  # qwen3-vl
        self.model_embed = settings.LLM_MODEL_EMBED  # qwen3-embed

Three environment variables. One API key. Three models. Done.

Real-World Latency Breakdown

The figures below are typical estimates, not profiled on this deployment — measure your own. For the keto example, a rough user journey might look like:

Treat these as order-of-magnitude expectations, not guarantees: a plain text response tends to feel fast, a lookup-heavy response usually warrants one progress indicator, and a photo analysis is long but users generally expect images to be slow. Profile against your own hardware and models before quoting numbers to stakeholders.

To optimize further:

• Batch embeddings if you have multiple food items (embed 5 at once, not 5 separate calls).
• Stream vision results to the client as they arrive (show "Analyzing..." → "Found chicken, broccoli, olive oil...").
• Warm up caches on app startup (pre-embed common foods).

Lessons: Intent-Driven Retrieval Beats Always-On RAG

The partial-RAG pattern here is different from classical RAG:

• Classical RAG: embed query → search KB → retrieve top-k → pass to LLM. Every query hits the vector DB.
• Intent-driven RAG (keto pattern): LLM says "I need data" → retrieve only if signaled → re-prompt.

Why this matters:

1. Cost: you only embed/retrieve when the model asks. No waste on queries that don't need facts.
2. Latency: fewer round-trips (2–3 round-trips vs. 4+).
3. Hallucination control: the model is trained to ask for data, not to guess.

If your use case does need always-on retrieval, the classical pipeline still applies — see our companion guide on retrieval over EU-hosted Qwen.

The trade-off: the model must be trained or prompted to signal intent. Use structured output (JSON mode), not free-form text, so you can reliably detect when lookup is needed.

Next Steps

1. Start small: pick one modality (e.g., chat-only), ship it, measure latency.
2. Add vision: integrate photo upload. Test timeouts with real images.
3. Add embeddings: when chat signals lookup, embed on-demand. Measure cache hit rate.
4. Monitor: log latency per modality, error rates, cache efficiency. Alert on p95 > 10s.
5. Iterate: watch what queries trigger lookups. Fine-tune the system prompt to avoid spurious retrieval.

For production, consider Haystack as an orchestration framework—it handles the conductor logic (routing, caching, retries) and lets you focus on domain logic.

Related Reading

• Multimodal RAG Best Practices 2026
• Embedding Models Benchmark 2026
• Self-Hosted RAG on JuiceFactory
• Vision Cost Optimization

Start building: Get a free JuiceFactory API key — no credit card required.

Related guides

Related guides: RAG with Qwen · GDPR-Safe Legal Document AI · Vision-LLM Cost Optimization · EU LLM API Comparison