# Can you forecast next week's weather?

## Executive summary

This technical discussion explores the evolution of weather forecasting, detailing the shift from resource-intensive physics-based models to more efficient AI/ML approaches. The conversation highlights how modern ML tools, particularly those provided by Hugging Face, are standardizing the workflow for running, evaluating, and fine-tuning these complex scientific models. Key focus areas include using the `EarthMover` marketplace for scientific data, leveraging object storage (`buckets`) for data management, and utilizing Hugging Face Jobs for scalable, accessible computation.

## Key takeaways

- Paradigm Shift in Forecasting Models: Historically, forecasting relied on physics-based models, which are accurate but computationally demanding and resource-intensive. AI-based models offer a significant advantage by requiring substantially less time and energy for computation, allowing for faster experimentation and iteration.
- ML/LLM Parallelism in Weather Prediction: Weather forecasting models share conceptual similarities with Large Language Models (LLMs). Both utilize an autoregressive paradigm: taking an initial state (like a token or a snapshot of the atmosphere) as input, and predicting the next state (e.g., what happens in the next six hours) sequentially.
- Standardizing Data and Compute Infrastructure: The complexity of scientific data is managed through the `EarthMover` marketplace, which allows users to fetch specific variables (e.g., temperature, wind) on demand, rather than downloading all global data at once. Hugging Face provides infrastructure solutions, including Jobs and object storage (`buckets`), to make running these models accessible even without dedicated GPU clusters.
- Ensemble Modeling and Evaluation: To improve forecast reliability, ensemble models are used, predicting a range of possible outcomes rather than a single deterministic forecast. Evaluation is challenging because ground truth data is often unavailable; thus, reanalysis (e.g., ERA5) or analysis data is used as a proxy for ground truth, and specialized metrics are needed to assess performance on skewed variables like precipitation.

## Technical details

- Model Architectures: Forecasting models can use various architectures, including Transformer-based, graph methods, or diffusion models. The model 'Aurora' was cited as an early example of a high-performing AI model for this domain.
- Data Handling and Storage: The `EarthMover` marketplace facilitates scientific data management, allowing users to connect and fetch specific variables (e.g., 2m temperature, wind components) on demand. The data format used is often `Zarr`, which is optimized for streaming scientific data.
- Computational Resources: Running forecasts is significantly faster on a GPU (less than a second per step) compared to a CPU (approximately 100 seconds per step). Hugging Face Jobs provide a scalable way to run these demanding computations.
- Model Fine-Tuning: The goal is to standardize the inference and fine-tuning process, allowing researchers to adapt models (like those predicting air pollution) using tools that integrate with existing LLM frameworks (e.g., integrating `WeatherNext` into the Transformers library).

## Practical implications

- Build engineers can leverage Hugging Face Jobs and object storage (`buckets`) to create scalable, cloud-native pipelines for running complex scientific simulations.
- The standardization of model inference and fine-tuning tools simplifies the deployment of specialized scientific models, reducing the setup time required for new research.
- The use of `Zarr` and `EarthMover` demonstrates best practices for handling large, multi-dimensional scientific datasets in a cloud environment.

## Topics

Weather Forecasting, Machine Learning, Scientific Computing, LLMs, Cloud Infrastructure, Model Deployment, Hugging Face Blog Post, EarthMover Marketplace

Source: https://www.youtube.com/watch?v=1jq7sC6IjMk