Scientific Computing

ERA maps scientific problems into 'scorable tasks,' allowing the system to iteratively propose and mutate code to maximize a defined score. This process moves beyond traditional machine learning by leveraging LLMs' vast prior knowledge and combining it with algorithmic search (Monte Carlo research) to solve problems like estimating CO2 concentrations or predicting wildfire boundaries.

Platt emphasizes that while AI is a powerful tool, human input remains crucial for defining the correct scoring function and maintaining scientific rigor. He warns against 'reward hacking' and the risks associated with overfitting, noting that the ability to distinguish between predictive and descriptive models is a core human task.

AI is being applied to complex, non-stationary problems like climate modeling and contrail warming. For instance, ERA helped solve a counterfactual problem—estimating the effect of contrails on outgoing longwave radiation—by finding a simple model that previously went unnoticed.

The field is undergoing a 'phase change,' moving from specialized, single-problem models to general AI toolchains. The optimal approach for young scientists is to combine deep domain expertise (scientific taste) with the ability to utilize and experiment with multiple advanced tools.

Historically, forecasting relied on physics-based models, which are accurate but computationally demanding and resource-intensive. AI-based models offer a significant advantage by requiring substantially less time and energy for computation, allowing for faster experimentation and iteration.

Weather forecasting models share conceptual similarities with Large Language Models (LLMs). Both utilize an autoregressive paradigm: taking an initial state (like a token or a snapshot of the atmosphere) as input, and predicting the next state (e.g., what happens in the next six hours) sequentially.

The complexity of scientific data is managed through the `EarthMover` marketplace, which allows users to fetch specific variables (e.g., temperature, wind) on demand, rather than downloading all global data at once. Hugging Face provides infrastructure solutions, including Jobs and object storage (`buckets`), to make running these models accessible even without dedicated GPU clusters.

To improve forecast reliability, ensemble models are used, predicting a range of possible outcomes rather than a single deterministic forecast. Evaluation is challenging because ground truth data is often unavailable; thus, reanalysis (e.g., ERA5) or analysis data is used as a proxy for ground truth, and specialized metrics are needed to assess performance on skewed variables like precipitation.

The model is tasked with replicating missing figures from redacted ML/AI papers. This process requires the agent to use tools (like CodeX) and execute code in a simulated environment, moving toward full automation of AI R&D.

Instead of simple verification, the system uses a rubric-based judge (generated by Claude) that assigns fine-grained points for correct reasoning, figure accuracy, and code writing. This process involves averaging judgments across multiple rollouts to prevent reward hacking.

The trained Faraday model demonstrated strong performance, sometimes outperforming much larger models like Claude and GPT-5. Furthermore, the system showed generalization even when given increased compute resources (e.g., scaling up to 8 hours/8 B300s).