OpenAI Agent Monitoring: Custom Instrumentation for Production Agents
Implement a MonitoredAgent wrapper around the OpenAI SDK to capture agent state, tool calls, token usage, and latency for production observability.
Patch your agent code to use the MonitoredAgent wrapper and enable metrics export.
Summary
The article explains how production teams can add deep observability to OpenAI agents beyond the SDK's basic telemetry. It introduces an agent_config.yaml that defines a customer_support_bot using gpt-4-turbo, temperature 0.7, a max of 10 iterations, and three tools with specific timeouts: search_knowledge_base (5000 ms), create_ticket (3000 ms), and retrieve_order (2000 ms). The monitoring section of the config turns on logging at INFO level, enables metric export, and sets a trace sampling rate of 1.0 to capture every request. By wrapping the OpenAI client in a MonitoredAgent class, developers can record start and end times, count iterations, track tool calls, and accumulate token usage per run. The MonitoredAgent’s run method loops until the max_iterations limit, sending messages to the assistant and inspecting the response for usage and tool_use blocks. It appends each tool call with a timestamp, updates the tokens_used counter, and stores the sequence of calls in a metrics dictionary. The wrapper exposes metrics that can be logged or exported to a monitoring backend, giving visibility into state transitions, tool execution patterns, token efficiency, and latency degradation. This approach turns a blind “car dashboard” into a full telemetry suite that can surface infinite loops or silent failures before they reach users.
Key changes
- agent_config.yaml defines gpt-4-turbo, temperature 0.7, max_iterations 10, tools with timeouts
- monitoring.enabled true, log_level INFO, export_metrics true, trace_sampling_rate 1.0
- MonitoredAgent records start_time, end_time, iterations, tool_calls, tokens_used
- run method loops until max_iterations, tracks token usage via response.usage.completion_tokens
- captures tool_use blocks, appends name and timestamp to metrics
- exposes metrics for logging or export, providing visibility into state transitions, tool patterns, token efficiency, latency
- trace sampling rate 1.0 ensures every request is traced
- wrapper turns basic SDK telemetry into full observability