Channel
Digests from AI Engineer
Sarvam introduced a 3B State-Space Vision Language Model for document AI, achieving SOTA performance across English and 22 Indian languages. The model uses an SSM backbone for efficiency and was trained on 13T tokens.
Read article
Perceptron introduces Embodied Foundation Models, unifying perception, reasoning, and control. By using data sparse Mixture of Experts, they solve context bloat and achieve a new scaling law: 10x video pretraining substitutes for 10x less teleop data.
Read article
The speaker proposes using AI agents and LLMs to automate rotational grazing, arguing that labor is the primary bottleneck preventing more livestock from being raised on pasture. This requires integrating data like GPS, drought conditions, and biomass.
Read article
Stop deploying VLMs directly for real-time CV tasks. Instead, use a toolkit for building specialized detectors (RFDTR) via 'vibe training,' leveraging VLMs as labelers and judges.
Read article
The talk details building the document context layer for AI agents, arguing that RAG requires a three-layer platform: parsing, semantic storage, and repeatable workflows. It addresses the complexity of parsing PDFs and introduces the ParseBench benchmark.
Read article
The talk explores AI's role as a GPT, detailing the shift from simple prompting to complex 'software factories.' Key focus areas include agentic safety, data sovereignty, and advanced orchestration using sandboxes.
Read article
The talk details 'harness engineering,' arguing that advanced AI skills must move beyond simple prompting. Techniques include using blind sub-agents, anti-attractors, and hooks to build robust, deterministic, and multi-model compatible systems.
Read article
Two critical bugs in stateful inference (vLLM/Mamba) were found by starving the GPU and comparing logprobs. The failures were silent, requiring deep kernel debugging and type fixes.
Read article
Agentic inference requires optimizing end-to-end task latency, not single-request latency. FriendliAI's cloud architecture uses Prefix Caching and Cache Aware Routing to make open-weight models economically viable for production agents.
Read article
Small open-source models offer massive cost and latency advantages over managed APIs. The solution requires a decentralized architecture using a shared queue, allowing workers to self-batch and double cluster throughput.
Read article
The talk details the 'big three' of inference engineering: Quantization, KV compaction, and Speculation. Learn about DFlash, TurboQuant, and how training is merging with inference optimization.
Read article
The inference platform supports two models: serverless (pay-per-token) and dedicated (billed per GPU hour). Optimization relies heavily on KV cache awareness and techniques like 4-bit quantization.
Read article
Discover why LLM performance benchmarks are often unreliable. Learn how to use Inference Perf to achieve high-fidelity, reproducible, and production-scale benchmarking for your inference stack.
Read article
Inference workloads are becoming foundational infrastructure, requiring a dedicated control plane. Optimization must shift from cost per token to cost per successful task, managing complex distributed transactions across multiple hardware and software layers.
Read article
Agent harness engineering is the discipline of turning a non-deterministic LLM core into reliable, repeatable output. The architecture focuses on the data layer, memory management, and robust workflow loops.
Read article
Fixed chunking is a flawed assumption in RAG. The optimal chunk size is query-dependent. By implementing multiscale indexing and combining results using RRF, systems can achieve 20-40% higher recall.
Read article
RL is revolutionizing search by creating specialized sub-agents. These agents can adaptively search, leading to results that are 20 times faster and 100 times cheaper than using frontier models.
Read article
TurboPuffer, an object storage-native search engine, solves scaling issues in legal AI search. It achieves physical isolation and massive latency improvements by making the namespace the atomic unit of storage.
Read article
DocuSign and NVIDIA tackle the $2T problem of unstructured agreements. They built Neotron Parse, a purpose-built VLM that extracts complex tables and structured data 20x faster than generic tools.
Read article
Designing AI agents for knowledge work requires mimicking human orchestration. Agents must break down complex, ambiguous problems using specialized sub-agents and multiple search primitives to overcome performance ceilings.
Read article