# How AI Agents, LLMs & APIs Use Real-Time Data at the US Open

## Executive summary

This video details an AI-infused system at the US Open that analyzes tennis serves by combining specialized APIs with Large Language Models (LLMs) via AI Agents. The core architectural pattern involves specialized backend services handling massive, complex data processing (e.g., biomechanics and joint tracking) and returning structured, actionable scores. The LLM then uses these scores, rather than raw data, to perform high-level reasoning and generate human-readable insights for the user. This approach effectively divides labor between specialized computation and natural language reasoning.

## Key takeaways

- The Limitation of Raw Data for LLMs: LLMs, while excellent at general knowledge, struggle with raw, high-volume numerical crunching, such as processing 21 tracked joints at 50 times per second, which generates millions of position values. Raw data is too large and complex for effective reasoning.
- The Role of Specialized APIs: APIs and specialized services are necessary to process raw data streams into structured, meaningful metrics. For the US Open, this process generates two key scores: **Efficiency** (biomechanics, joint separation) and **Effectiveness** (outcome stats like speed and placement).
- The AI Agent Workflow: An AI Agent acts as an orchestrator, using a defined list of tools (APIs). When prompted, the agent outputs a structured request, which is executed by the API. The API returns the processed data, allowing the LLM to complete the reasoning cycle and answer the user's question.

## Technical details

- Data Scale and Metrics: The system tracks 21 joints on the player's body at 50 times per second, accumulating over one billion data points across the tournament. These raw coordinates are processed into two scores: **Efficiency** (biomechanics, e.g., hip/shoulder separation) and **Effectiveness** (outcome stats, e.g., speed, placement).
- Serve Quality Scoring: The final 'Serve Quality' score is developed using IBM Bob and is grounded in biomechanics and kinetic chain research to weigh the most impactful metrics contributing to the serve's effect.
- Agent Tool-Calling Loop: The agent operates in a continuous loop: it reasons about missing information, requests a tool (API), receives structured data, reasons again, and repeats until it has enough information to write a final, coherent answer.

## Practical implications

- The AI Agent + API pattern is a generalizable architectural pattern, applicable beyond sports analysis, such as an agent investigating a production outage that pulls data from monitoring and log APIs.
- The principle of dividing labor—using specialized services for heavy computation and LLMs for reasoning—can be applied to any complex data system.

## Topics

AI Agents, Large Language Models (LLMs), APIs, Biomechanics, Real-Time Data Processing, System Architecture, IBM Technology, IBM

Source: https://www.youtube.com/watch?v=F3hlZSZc6UI