# Keep model costs in check with LangSmith LLM Gateway

## Executive summary

The LangSmith LLM Gateway solves critical governance and cost control issues arising from decentralized usage of multiple Large Language Models (LLMs) across an organization. By acting as a single, centralized endpoint between development teams and various model providers (e.g., OpenAI, Anthropic), the gateway enforces real-time spend limits and rate controls before requests are processed. This allows organizations to maintain a live audit trail of token usage, cost, and model access without requiring code changes in existing applications.

## Key takeaways

- Centralized Cost Governance: The Gateway provides a single point of control for enforcing spend limits and rate limits across all connected LLM providers, preventing unexpected cost spikes that occur when multiple teams use various keys and models.
- Non-Invasive Integration: Implementing the gateway only requires updating the base URL and API key within existing SDKs (Python, TypeScript), meaning developers do not need to modify their application's core request or response handling logic.
- Granular Control and Auditability: Cost controls can be applied globally for the organization, or granularly by individual API key, user, or workspace. This provides a live audit trail detailing which model, key, and number of tokens were used, replacing reliance on provider invoices.

## Technical details

- LLM Gateway Functionality: The gateway sits between the organization and external model providers (OpenAI, Anthropic, Fireworks, etc.), routing all requests through one endpoint for centralized control over access and spending.
- Implementation Steps: To use the service, an API key must be created within the Gateway. Credentials for upstream providers are then added to authenticate the gateway when authorized requests come in. The process is demonstrated using standard SDKs (Python/TypeScript) by only changing the base URL and API key.
- Spend Limit Enforcement: Spend limits can be set centrally at the organization level or granularly for specific keys/users. When a limit is hit, the gateway blocks the request and returns a clear error message, ensuring the charge never reaches the organization's account.
- Usage Tracking and Auditing: The system provides a usage page that tracks requests by user (e.g., 'Jacob Lee'), showing total requests made against set spend limits, providing a clear audit trail for finance teams.

## Practical implications

- Build teams can implement robust, centralized governance over AI spending without requiring deep changes to application codebases.
- The system provides immediate visibility into token usage and costs, enabling proactive budget management rather than reactive invoice reconciliation.
- It standardizes the access point for all LLM calls across an organization, simplifying security and compliance auditing.

## Topics

LLM Governance, Cost Management, API Gateway, LangChain, AI Infrastructure, LangSmith LLM Gateway

Source: https://www.youtube.com/watch?v=VBZgp1sHosE