How a Logistics Giant Keeps AI Data Locked Down
This discussion provides a deep dive into FinOps for AI, detailing how a logistics giant (C.H. Robinson) manages and optimizes AI usage across complex workflows. Key strategies include tagging all AI resources for ownership tracking, utilizing platforms like Azure OpenAI and Vertex AI to maintain data locality, and implementing advanced observability metrics. The speaker emphasizes moving beyond simple total spend to measure efficiency using metrics like 'cost per thought,' 'reasoning ratio,' and 'cache hit rate' to accurately assess ROI.
Key takeaways
-
Resource Tagging is Foundational
10:33
The first step in managing AI spend is tagging every AI resource (at the model or workload level) to establish clear ownership and accountability.
-
Value Over Spend
12:24
When reporting AI value, focusing solely on 'spend' is insufficient. It is more valuable to calculate ROI by measuring 'cost per hour saved' or 'cost per order,' demonstrating efficiency gains.
-
Advanced Observability Metrics
7:01
To accurately track AI value, advanced metrics are necessary. These include 'prompt bloat' (excessive token usage), 'context starvation' (insufficient context leading to retries), 'reasoning ratio,' and 'cache hit rate.'