How to Trace and Evaluate a Hermes Agent with NVIDIA NeMo Relay and Phoenix
This tutorial demonstrates how to use NVIDIA NeMo Relay to trace and evaluate the complex execution path of a Hermes AI agent. By capturing the entire workflow—from initial requests and file-reading tool calls to the final response—developers can gain deep visibility into the agent's actions, including which tools were used, the data accessed, tokens consumed, and the overall cost of the run. This detailed tracing is crucial for debugging, evaluation, and improving AI agent reliability.
Key takeaways
-
Comprehensive Agent Tracing
NVIDIA NeMo Relay captures the full execution path of a Hermes Agent, providing visibility into tool usage, data access, and model activity beyond just the final answer.
-
Reviewing the Trace in Phoenix
2:00
The captured trace can be opened in Phoenix, allowing developers to review the session, model activity, tool calls (e.g., reading files, searching the web, writing reports), and the specific data returned at each step.
-
Security and Evaluation
1:10
The process includes a sanitizer that removes personally identifiable information (PII) before the trace leaves the machine, ensuring data privacy while maintaining full debug capability.