AI Infrastructure

The most valuable data is process-based data—the reasoning trace or sequence of decisions, rather than state-based data (e.g., rows in an ERP). Type one data (pure capture of real workflows like GitHub commits) offers superior realism compared to type two data (contrived examples manufactured by experts).

A task's ease of training is proportional to its verifiability, which depends on three axes: asymmetry of verification (decomposability into checkable steps), veracity of verification (consensus on what 'correct' means), and proliferation of verification (availability of fresh examples). Coding scored highly because it solved all three.

For data companies, the durable value accrues to the services and application layer of actual work. The true moat for builders is not the raw data itself, but the pipeline into real-world work and the infrastructure required to keep retraining as models improve.

Current methods like RAG are limited because they only provide external, textual context. True intelligence requires embedding knowledge into the model's parameters (weights) to achieve 'intuition,' allowing the model to generalize and extrapolate beyond explicit notes or recipes.

As companies accumulate trillions of tokens of proprietary data, simple context management fails due to 'context rot' (the model becoming less accurate the more context it reads) and extreme token consumption. This necessitates methods that are both highly efficient and scalable.

Engram proposes a multi-faceted approach combining knowledge compression via 'cartridges' (compact capsules of knowledge), gradient-based updates during inference ('test-time training'), and advanced memory layers to achieve superior token efficiency and model accuracy.

The ultimate goal is a system where the model autonomously determines what knowledge should be internalized (in weights) versus what should remain external (in text/RAG), creating personalized models that improve continuously with user interaction, similar to nurturing a Tamagotchi.

Modal is shifting its focus from Developer Experience (DX) to Agent Experience (AX). This shift acknowledges that agents will increasingly operate on infrastructure, making the ability for an agent to self-provision and modify resources via decorators more valuable than manually writing complex YAML files in Kubernetes.

Traditional orchestration tools like Kubernetes were not designed for highly bursty, compute-heavy workloads common in AI. They struggle with the rapid scaling and specialized resource needs (like accelerators) required by modern models.

The platform excels at elastic inference for custom models (e.g., audio, video, robotics). The core challenge is managing unpredictable traffic patterns and scaling rapidly across different regions, a problem Modal addresses through its autoscaling primitives.

To achieve frontier-level inference performance, Modal utilizes techniques like speculative decoding (using a smaller 'draft model' to predict tokens ahead of the main model) and DeFlash (a block-based speculator), achieving significant speedups without quality loss.