Monitoring Agent Behavior

Once you know your assessment metrics, you must learn how to monitor them. If you have a complex graph with many nodes and LLM calls, it can be hard to track down errors when your agent isn’t behaving as you want it to. You have several tools to help with this, though.

Logging

Printing output messages in the notebook is the most direct way to gauge what’s going on in your AI agent workflow. So far in this module, you’ve been using print statements. However, you can also use the standard Python logging library for more granular control. This lets you set various logging levels or send output to a file instead of the console.

import logging

logging.basicConfig(
  level=logging.DEBUG,
  filename='app.log'
)

logging.debug('debug message')
logging.info('info message')
logging.warning('warning message')
logging.error('error message')

In the code snippet above, the logger will log all levels from DEBUG up and output the logs to a file named app.log.

Step-by-Step Execution

JupyterLab has a debugger that lets you set breakpoints:

You can certainly experiment with this. However, it seems to have some difficulties stepping across nodes when using LangGraph.

Streaming Output

LangGraph natively supports streaming output. This allows you to see what’s happening at each step along the way. Rather than calling app.invoke, you call app.stream:

for output in app.stream(state, thread, stream_mode="values"):
  print(output)

A stream_mode of values, which is the default, means the app will stream the full state at each node. Another option is update, which only streams the state changes. Finally, you also have debug, which will tell you more than you ever wanted to know.

Commercial Options

The makers of LangChain and LangGraph have released those libraries as open-source software. However, they also provide commercial products to help debug your AI agent app.

LangSmith

LangSmith lets you see information about the various nodes your graph traverses during execution in a nice visual format.

LangSmith isn’t difficult to set up. You sign up, get an API key, and then load the key in your project similarly to how you would with an OpenAI API key. After that, the node calls automatically appear in the LangSmith web dashboard. You’ll get to try this out in the demo project later.

LangGraph Studio

LangGraph Studio is still in Beta and doesn’t support all platforms, so this lesson won’t cover it in depth. However, it looks like a promising way to interact with your graph more visually and intuitively. The following is a clip from one of their documentation images:

User Feedback

Low-tech monitoring solutions are just as important or even more important than high-tech ones. You should be collecting feedback from your users about where the pain points are with your agent:

• How natural does this application feel in your language? Does it seem like a native application?
• How good was the support the chatbot gave you? Could it do everything you wanted it to?
• How was it talking to the customer service agent? Did it handle things as naturally as a human would have?

As developers, it’s easy to live in a cave, but it’s important to get human feedback about what you’re building.