TB
Tino Britty
Back to Log
AI EngineeringLLMsSystem DesignVector Search

Modern AI Engineering: Building Context-Aware Applications

5 min read
AI Engineering

The landscape of software development has fundamentally shifted. We are no longer just writing imperative instructions for computers to execute; we are orchestrating language models to reason over vast datasets. As an AI Engineer, the most common challenge I face isn't training models from scratch, but rather giving existing models the right context to be useful.

This article breaks down the core patterns of modern AI Engineering, specifically focusing on building context-aware applications.

The Illusion of Infinite Context

When developers first build AI applications, they often rely heavily on the prompt. "If the model doesn't know the answer, I'll just paste the entire document into the prompt!"

While context windows (like Gemini 1.5 Pro's 1M+ tokens) are growing rapidly, stuffing the prompt has severe drawbacks:

  1. Cost: You pay per token. Sending 100,000 tokens on every request is financially unsustainable.
  2. Latency: Processing massive prompts takes time, ruining the snappy UX users expect.
  3. Lost in the Middle: Studies show that LLMs can suffer from "lost in the middle" syndrome, where they fail to recall information buried deep within a massive prompt.

The solution is Retrieval-Augmented Generation (RAG).

Building a Robust RAG Pipeline

RAG allows an application to search a private database for relevant information before talking to the LLM. It then injects only the most relevant snippets into the prompt.

Here is the architecture I follow when building AI systems:

1. Data Ingestion & Chunking

You cannot embed a 500-page PDF as a single vector. You must chunk it. The strategy matters immensely.

  • Naive Chunking: Splitting by character count (e.g., every 500 characters). This often splits sentences in half, destroying meaning.
  • Semantic Chunking: Splitting by semantic boundaries, such as paragraphs or markdown headers (##). This preserves context.

2. Vector Embeddings

Once chunked, the text is converted into high-dimensional vectors (arrays of floating-point numbers) representing the semantic meaning of the text. Models like OpenAI's text-embedding-3-small or local open-source models like BGE-m3 are excellent choices.

3. Vector Database Storage

Store the text chunk alongside its vector in a Vector Database. Tools like Pinecone, Weaviate, or pgvector (PostgreSQL extension) are industry standards. I often prefer pgvector because it allows me to keep relational metadata (like the user ID) in the same database as the vectors.

4. The Retrieval Step (Cosine Similarity)

When a user asks a question:

  1. Embed the user's query into a vector.
  2. Perform a similarity search (usually Cosine Similarity or Inner Product) against the Vector DB.
  3. Retrieve the top K (e.g., 5) most relevant chunks.
// Example using Prisma with pgvector
const userQueryVector = await getEmbedding(userQuestion);

const relevantChunks = await prisma.$queryRaw`
  SELECT content 
  FROM "DocumentChunk"
  ORDER BY embedding <=> ${userQueryVector}::vector
  LIMIT 5;
`;

5. Generation

Finally, construct the prompt.

You are an expert assistant. Answer the user's question using ONLY the provided context. 
If the answer is not in the context, say "I don't know."

Context: 
[Insert retrieved chunks here]

User Question: {userQuestion}

Moving Beyond Basic RAG: Agentic Workflows

Basic RAG is great for Q&A, but modern AI engineering is moving towards Agentic Workflows. Instead of just answering questions, the AI has tools it can use.

Using frameworks like LangChain or Vercel AI SDK, you can define tools (e.g., search_web, query_database, send_email). The LLM is prompted to output a specific JSON structure requesting a tool call. The application intercepts this, executes the real code, and feeds the result back to the LLM.

This transforms the LLM from a simple text generator into a reasoning engine that interacts with external systems.

Conclusion

AI Engineering requires a deep understanding of traditional software architecture combined with the probabilistic nature of LLMs. By mastering vector search, semantic chunking, and tool-calling architectures, Full Stack Developers can build the next generation of intelligent, context-aware software.

Share this article