The Video Signal

Topic

LangSmith LLM Gateway

All digests tagged LangSmith LLM Gateway

Keep model costs in check with LangSmith LLM Gateway thumbnail

· 4:33

Keep model costs in check with LangSmith LLM Gateway

The LangSmith LLM Gateway solves critical governance and cost control issues arising from decentralized usage of multiple Large Language Models (LLMs) across an organization. By acting as a single, centralized endpoint between development teams and various model providers (e.g., OpenAI, Anthropic), the gateway enforces real-time spend limits and rate controls before requests are processed. This allows organizations to maintain a live audit trail of token usage, cost, and model access without requiring code changes in existing applications.

Key takeaways

  1. Centralized Cost Governance 0:15

    The Gateway provides a single point of control for enforcing spend limits and rate limits across all connected LLM providers, preventing unexpected cost spikes that occur when multiple teams use various keys and models.

  2. Non-Invasive Integration 0:40

    Implementing the gateway only requires updating the base URL and API key within existing SDKs (Python, TypeScript), meaning developers do not need to modify their application's core request or response handling logic.

  3. Granular Control and Auditability 1:05

    Cost controls can be applied globally for the organization, or granularly by individual API key, user, or workspace. This provides a live audit trail detailing which model, key, and number of tokens were used, replacing reliance on provider invoices.

Watch on YouTube Full article

The Art of Loop Engineering: How to Build Agents That Improve Over Time thumbnail

· 46:08

The Art of Loop Engineering: How to Build Agents That Improve Over Time

The video introduces 'Loop Engineering,' an emerging design pattern critical for building reliable, production-grade AI agents. Agents are inherently non-deterministic; therefore, they require structured loops—such as the Core Agent Loop, Verification Loop, Event-Driven Loop, and Self-Improvement Loop (Hill Climbing)—to ensure reliability, automate continuous improvement, and integrate seamlessly into existing systems. The LangSmith platform is presented as a key tool for managing this complex agent development lifecycle.

Key takeaways

  1. Core Agent Loop (Level 1) 10:30

    This basic action-taking loop involves the model receiving context, calling tools to complete tasks, and receiving observations until completion. Optimization focuses on selecting the right model intelligence for the task complexity and improving tool descriptions via prompt engineering.

  2. Verification/Goal Loop (Level 2) 15:20

    This loop adds reliability by introducing a 'Grader' or verification step. After the core agent attempts a task, the Grader scores the result against predefined criteria (rubrics). If criteria are not met, the process is fed back into the agent loop for correction.

  3. Event-Driven Loop (Level 3) 20:00

    Agents become powerful when triggered by external systems (e.g., Slack messages, emails). This loop integrates the agent into relevant workflows, making it a system improvement mechanism rather than just an isolated task executor.

  4. Self-Improvement Loop / Hill Climbing (Level 4) 23:20

    This advanced loop automates agent improvement by analyzing traces. A helper agent, like LangSmith Engine, detects failure modes (e.g., improper tool arguments, missed context) and autonomously updates the core harness—including prompts, tools, skills, or memory—to improve future performance.

Watch on YouTube Full article