Components of Context Engineering: Part 1

Reintroducing Context Engineering

Up until a few months ago, prompt engineering had most of the attention among AI enthusiasts. That’ll still continue, but increasing attention is now shifting toward context engineering. Context engineering shifts the reasoning behind designing AI agents to include every component in every step of the process. Context engineering is all about optimizing what goes into a context; a context being the state, tools, LLM, and every other component involved in a single iteration of an AI agent’s workflow.

In practice, context refers to the collection of tokens available to the model. During each stage of an agent’s workflow, more tokens are generated, which can influence the outcome of the next stage. Since this is running autonomously and iteratively, you’ll have to carefully design the tokens and instructions such that every token is contextually relevant. Otherwise, you’ll have a poorly designed agent that doesn’t do the job and wastes resources.

Assessing Problems with AI Agents

Imagine you design an agent to find the best components for your car. You carefully design the prompt with the best prompt engineering principles, providing it details of your car, and equipping it with tools, like a search engine, so it can use it to find other relevant information to aid the task. When you start the agent, it puts together the best components based on its LLM’s output, together with extra data about your car supplied using the RAG technique. It then goes online to search for the availability of these components and decides to use results from the first page, assuming they’re the most relevant. But it turns out that the search results had ads at the top. Since these ads aren’t necessarily the most relevant for the query, the output from this stage will reduce the quality of the tokens for the next step.

Considering that agents typically involve multiple stages and that many of these stages are iterative, the quality of the tokens for each stage can degrade quickly. The possibilities are endless. The agent could end up using the wrong tools, feeding on irrelevant outputs from different phases, running potentially long, expensive, yet irrelevant operations, and so on. These are some of the issues with AI agents that context engineering seeks to solve.

Explaining Terminologies

Context Poisoning is when a hallucination or some other error makes it into a context window, where it is repeatedly referenced. This may quickly saturate large portions of the context with highly irrelevant data. These errors may happen not just because an output is irrelevant, but also because there is too much of it.

As the program continues to run, the context window gets larger and larger, causing the agent to lose focus on important information. This can be referred to as Context Rot or context distraction. When this incorrect information is used in subsequent steps, it causes Context Confusion.

These are one of the primary reasons why you can’t just get your prompts into multiple MCP (Model Context Protocol) servers to build one big AI agent. MCP is a standard that allows LLMs to securely and easily connect to external tools. An MCP server is a program that implements the MCP protocol to connect to external applications. You may think that LLMs are smart enough to, for instance, know which tool to use, but multiple tests and research have shown that it’s quite common for them to fail. There’s an LLM leaderboard by Berkeley called Function-Calling which benchmarks LLMs and their tool-use abilities given a prompt. The leaderboard shows that models perform worse when provided with more than one tool.

It’s even possible for tools to return outputs that contain conflicting information which could confuse the LLM. This is also known as Context Clash.

A Context window is the amount of information a model can recall during a session. A larger context window means a larger memory. The contents of the context window could include tools, memory, the system prompt, user prompt, and message history. The message history is everything that has happened since the agent started. One key principle in context engineering is that the context window is limited, and it’s important to keep it as small as possible while maintaining high-quality outputs.

Exploring Techniques in Context Engineering

While LLMs get better, some of these issues could reduce. But, to manage context is also about managing the computing resources and time. This is why context engineering is so important. It provides concepts that allow you to design agents that use the right tokens throughout the process because it keeps everything within context. Note however that much of applied AI is as much about art as it is about science. Many of the techniques used in context engineering are not unique to context engineering. Many are in fact borrowed from concepts applied in RAGs, prompt engineering, and other branches of AI.

These concepts aren’t standardized. In practice, each person or organization’s approach differs. They have evolved from multiple independent research efforts and experiences from engineers, researchers, and others. There are leaderboards for AI papers too, some of which can be found here. In the subsequent sections, you’ll see some of these concepts and how they apply to context engineering.

RAG

One way of solving these problems is by incorporating RAG. With RAG, you can provide the exact information you want your agent to use. You know exactly what’s contained in the additional data and can therefore tune your prompts better to fit within the context window.

Tool Loadout

Tool Loadout is the act of selecting only relevant tool definitions to add to your context. In a bid to ensure your agent has the best tools, you may want to provide it with many options. For instance, you may provide five different credible news websites for reference or thirty Node.js frameworks, allowing it to choose the best. However, it’s better to limit this to a select few trusted tools, typically around ten, depending on the model you’re using. Anything beyond this may result in a higher error rate, where the LLM uses incorrect tools.

Context Quarantine

Context Quarantine is the act of isolating contexts in their own dedicated threads, each used separately by one or more LLMs. You’ll get better results when your contexts are short and focused. Separation of concerns greatly improves efficiency and accuracy. In practice, you may want to use a separate LLM for tool calling and another for reasoning.

Context Pruning

Context Pruning is the act of removing irrelevant or unnecessary information from the context. While this sounds simple, you must be careful not to remove any critical information during the process. There are models specifically designed for pruning. It’s somewhat similar to summarization. A popular technique known as Provence has become widely recognized as effective for context pruning in RAG systems. You can find more about it here.

Below is a simple demonstration of how to use it:

from transformers import AutoModel

provence = AutoModel.from_pretrained("naver/provence-reranker-debertav3-v1", trust_remote_code=True)

# Read an article on climate change
with open('climate_change.md', 'r', encoding='utf-8') as f:
    climate_change_wiki = f.read()

# Use a prompt to prune the article
question = 'What are the biggest causes of climate change?'
provence_output = provence.process(question, climate_change_wiki)

In the next segment, you’ll see a demo showing how to use context pruning with LangChain.