How ChatGeniee Builds Real-Time AI Conversations with RAG

Building a production AI system is deceptively simple in theory: take a user's question, find relevant information, and ask an LLM to answer based on that context. In practice, the engineering challenge is substantial. You need to handle thousands of concurrent users, process documents at scale, retrieve information in milliseconds, and stream responses without blocking the interface.

This is the story of how ChatGeniee solves those problems. Not with hype, not with buzzwords, but with specific technologies, architectural decisions, and optimizations that make the system work in production today.

What is RAG and why it matters

RAG stands for Retrieval-Augmented Generation. The concept is straightforward: instead of relying only on an LLM's training data, you augment it with relevant context retrieved from your own knowledge base.

Without RAG, an LLM answering questions about your company would only know what was in its training data (which could be months or years old). With RAG, the LLM can answer based on your most current documents, product specs, policies, and proprietary information.

The challenge: retrieving the right information fast. If you search based on keyword matching, you miss semantic nuance. If you search by semantic similarity but do it slowly, your users wait. If you don't isolate data between customers, you leak sensitive information.

The RAG Pipeline: From Documents to Vectors

Stage 1: Knowledge Ingestion

When a user uploads a document to ChatGeniee, it's stored temporarily in S3. The system doesn't immediately process it. Instead, it enqueues a message to an AWS SQS FIFO queue. This decoupling allows the upload to return instantly to the user while processing happens asynchronously.

Why FIFO? Because order matters for documents. If a user uploads a 100-page PDF followed by an updated version of the same file, you want them processed in that order, not randomly. FIFO guarantees ordering.

Stage 2: Semantic Chunking

A 100-page PDF is too large to embed as a single vector. You need to break it into meaningful pieces. ChatGeniee chunks documents by paragraph boundaries and semantic coherence, aiming for ~512 tokens per chunk (roughly 400 words).

Why 512? It's a balance: small enough to be precise in retrieval, large enough to maintain context. A single sentence is too granular; a full page loses specificity.

Chunks also include metadata: source file name, upload timestamp, page number. This metadata allows the LLM to cite sources and users to verify where information came from.

Stage 3: Embedding Generation

Each chunk is converted into a vector (a list of 1024 numbers) using Bedrock's Titan embeddings model. These numbers encode the semantic meaning of the text in a way that computers can compare.

The embedding model is trained to ensure that similar texts produce similar vectors. Two chunks about "refund policy" will have vectors close to each other in mathematical space, while one about "refund policy" and one about "shipping rates" will be distant.

Embedding generation scales to thousands of chunks. For a 100-page document, the system might generate 200-300 chunks, meaning 200-300 API calls to Bedrock. These are batched in groups of 10 to reduce latency.

Stage 4: Vector Storage

Vectors are stored in PostgreSQL using the pgvector extension. Each row contains: the vector itself, the original text, the chunk metadata, and crucially, the tenant ID.

Tenant ID is security. Every query to the vector database is filtered by tenant, guaranteeing that a customer never accidentally retrieves another customer's data.

To make search fast, ChatGeniee creates an HNSW (Hierarchical Navigable Small World) index on the vector column. This specialized index structure allows semantic search to complete in under 10 milliseconds for a 10-million-vector database.

From Query to Streaming Response

The magic happens in steps 2-4. The user's question is embedded using the same Titan model. This ensures the embedding space is consistent: questions and documents are in the same mathematical universe.

The vector search is a single query to PostgreSQL: "Find the 5 vectors closest to this query vector, filtered by tenant ID." Because of the HNSW index, this completes in microseconds.

The retrieved chunks are formatted into the system prompt: "You are a helpful assistant. Answer the following question based ONLY on this provided context. Here is the context: [5 chunks]. Here is the question: [user query]."

By telling the LLM to answer "based ONLY on this context," you dramatically reduce hallucinations. The model still has the knowledge from its training data, but it's prompted to prioritize retrieved context.

The Streaming Layer: Why Speed Matters

Once the LLM begins generating the response, ChatGeniee doesn't wait for the complete answer. Instead, it streams each token to the client as it's generated using Server-Sent Events (SSE).

This is not an optimization for bandwidth. It's a perceptual optimization. If a user waits 1 second for the complete response, they perceive it as slow. If they see the first tokens appearing in 450ms and the rest flowing in, they perceive it as fast and responsive (like ChatGPT).

The backend streams events at the HTTP level. The frontend renders each token as it arrives. No batching, no buffering. This requires SSE because HTTP polling would add even more latency.

The Technology Stack: Why These Tools

Vector Embeddings: Bedrock Titan

ChatGeniee uses AWS Bedrock's Titan embeddings model. Why? First, it's available through Bedrock alongside the LLM models, reducing operational complexity. Second, it's language-agnostic, meaning it handles English, Spanish, French, etc. uniformly.

The trade-off: Titan embeddings are 1024 dimensions. Some open-source models are larger (4096 dimensions), offering more precision at the cost of storage and compute. For ChatGeniee's use cases, 1024 dimensions is sufficient; the bottleneck is not embedding quality but retrieval speed.

Vector Database: PostgreSQL + pgvector

ChatGeniee could use specialized vector databases like Pinecone or Weaviate. Instead, it uses PostgreSQL with pgvector. Why?

First, operational simplicity. PostgreSQL is already running for application data (users, bots, conversations). Adding pgvector means one database, one backup strategy, one authentication model.

Second, hybrid search. pgvector supports both vector search and SQL filtering in the same query. You can search semantically AND filter by tenant, date, category, etc. Specialized vector databases require two separate queries.

Third, cost. Pinecone charges per vector per month. PostgreSQL is a fixed cost. At scale, PostgreSQL is dramatically cheaper.

LLM: AWS Bedrock Claude

ChatGeniee uses Claude 3 via AWS Bedrock. The main advantage: tool calling. Claude can decide which tools (knowledge base, web search, calendar integration) to use based on the query. This requires the model to understand context and intent.

Bedrock abstracts away the infrastructure. You don't manage API keys, rate limits, or failover. AWS handles scaling and availability.

Streaming: Server-Sent Events

SSE is not new. It's a browser standard that's been around since 2006. But it's underused in modern applications. Why?

Because WebSockets are sexy. WebSockets are bidirectional, low-latency, and feel "real-time." SSE is unidirectional, slightly higher latency, and feels simpler. But for streaming responses from an LLM to a client, unidirectional is perfect. There's no need for the client to send real-time updates back to the server.

SSE is also cheaper to host. WebSocket connections are stateful and consume server memory. SSE is stateless from the server's perspective; it's just HTTP.

Production Optimization: Making It Fast and Cheap

Embedding Model Drift

Bedrock's Titan embedding model is periodically updated. When a new version releases, old vectors should be regenerated to maintain consistency. ChatGeniee handles this by recording the embedding model version with each vector.

Quarterly, ChatGeniee regenerates all vectors using the latest model. This is done offline, not disrupting service. Without this, semantic search accuracy degrades over months as the model drifts.

Token Counting and Cost Control

AWS Bedrock charges per token (per 1000 tokens). Without visibility, costs can spiral. ChatGeniee counts tokens before making each API call.

If a query would exceed your set budget (e.g., $0.50 per response), it's rejected with a message: "This query is too complex. Please try a shorter question." This prevents runaway bills.

Connection Pooling and Async I/O

Each database query would be slow if done synchronously. ChatGeniee uses async/await throughout the backend. When waiting for the database, the server handles other requests.

Connection pooling ensures the system doesn't open a new database connection for every request. Connections are reused, reducing overhead.

Autonomous Agents: Beyond Single Queries

A simple RAG system answers one question based on retrieved documents. ChatGeniee goes further with agents: the LLM can decide to invoke tools.

The system prompts the LLM with available tools: "You can call the knowledge-base-search tool, the web-search tool, the calculation tool, etc. When should you call them?"

The LLM, if it decides a tool is helpful, returns a structured request: { "tool": "knowledge-base", "query": "what is our refund policy" } (no code, just the concept).

The backend executes the tool and returns results to the LLM. The LLM then formulates the response to the user using both original context and tool results.

Practical: Meeting Booking Integration

This isn't theoretical. ChatGeniee customers who sell services (consulting, software, managed services) use the booking agent to convert conversations into meetings.

The flow is natural. A customer asks a question about pricing or features, and mid-conversation says "I'd like to schedule a demo." The agent recognizes the intent, integrates with Google Calendar or Calendly, and proposes available times. Once booked, both parties receive confirmation.

From the analytics side, every booking is logged with the conversation context. Sales teams can review what questions triggered a booking request, refining messaging over time.

Why This Architecture Wins

If you're building an AI product, you have choices: use a SaaS solution (Zapier, Make, etc.), use an LLM API directly (OpenAI, Anthropic), or build your own system like ChatGeniee.

SaaS solutions are quick but inflexible. LLM APIs are flexible but require you to solve RAG yourself. ChatGeniee's approach is a third path: production-ready RAG with multi-tenancy, autonomous agents, and streaming responses built in.

The technical decisions—pgvector for vectors, SSE for streaming, HNSW indexing for speed, token counting for cost—are optimizations that only matter at scale. But they're baked in from day one, so when your system grows, it doesn't break.

That's the difference between systems built for production and systems that pretend production is an afterthought.