# Harness Engineering: What's Actually New (And What Isn't)

## Executive summary

Harness engineering refers to the entire infrastructure surrounding Large Language Models (LLMs)—including tools, context management, guardrails, and telemetry—that ensures agents operate correctly. While the non-deterministic nature of LLMs introduces novel challenges (like model drift and complex evaluation), the core principles of software engineering (security, observability, incremental development, and testing) remain critical fundamentals for building robust AI agents.

## Key takeaways

- Harness Engineering Components: The agent system comprises the LLM (reasoning engine) and the harness, which includes context (RAG, system prompts), tools (APIs), orchestration, guardrails, security, and telemetry. These elements build upon prompt, context, and loop engineering practices.
- Non-Deterministic Challenges: Unlike deterministic software, LLMs are subject to non-determinism, model drift (e.g., changes in phrasing without version updates), and the LLM itself decides whether to invoke a tool, requiring robust telemetry and guardrails to detect failures.
- Evolution of Testing: Traditional Test-Driven Development (TDD) evolves into AI agent evaluations. Due to non-deterministic outputs, dedicated evaluation scorers (using a separate 'judge' model) are necessary to validate quality and relevance, and these must be integrated into the CI/CD pipeline.
- The Persistence of Fundamentals: Core software engineering best practices—such as trunk-based development, small incremental changes, continuous testing, security (OWASP principles), and observability (logs, metrics, traces)—are not obsolete and must be applied to agentic systems.

## Technical details

- Agent Architecture & Context: The context window is expanded by incorporating Retrieval Augmented Generation (RAG) from vector databases (e.g., Elasticsearch), system prompts, and external skills/tools. Data discovery remains crucial, as LLMs can hallucinate due to training cutoffs, biases, or over/under-fitting.
- Tool Definition and OpenAPI: Tools (capabilities) must be defined using specifications like OpenAPI or Zod TypeScript schemas. The description provided to the LLM must be highly engineered to ensure the model correctly decides when and how to invoke the tool.
- Observability and Guardrails: Implementing telemetry (logs, metrics, traces) is vital for diagnosing agent behavior, especially non-deterministic actions. Guardrails must be in place not only to restrict actions (e.g., topic restriction) but also to be monitored via telemetry to detect when the agent attempts to bypass them.
- Security and Authorization: Security principles like the principle of least privilege, data-level entitlements, and authentication/authorization remain paramount. The OWASP Top 10 list has evolved to include specific vulnerabilities for GenAI applications, such as prompt injection.

## Practical implications

- When building agent systems, prioritize utilizing existing, out-of-the-box harnesses first, reserving custom development for situations where fine-grained control or memory limitations are reached.
- Treat LLM tool descriptions with the same rigor as writing an OpenAPI specification, ensuring specificity regarding purpose, limitations, and required inputs.
- Always integrate comprehensive observability (OpenTelemetry signals) to monitor agent execution paths, inputs, outputs, and adherence to guardrails.
- When evaluating agents, do not assume LLM-as-a-judge is always appropriate; consider deterministic logic or established metrics like precision and recall for the specific task.

## Topics

Harness Engineering, LLM Agents, Observability, Software Architecture, Security Engineering, CI/CD, Prompt Engineering, Tessl

Source: https://www.youtube.com/watch?v=mHa0bImgN1o