# Lecture 117:Building a GPU Compiler from Scratch

## Executive summary

This lecture provides a deep dive into progressive Intermediate Representation (IR) lowering, detailing how high-level ML operations (like those in PyTorch) are systematically transformed into efficient, hardware-specific GPU kernels. The process involves multiple IR stages—from PyTorch to Tensor AR, Loop IR, Tile IR, and finally CUDA—each stage applying optimizations like operator fusion, normalization, and tiling to minimize memory overhead and maximize computational efficiency for modern accelerators.

## Key takeaways

- Progressive IR Lowering Stack: The compiler uses a multi-layered approach: PyTorch $\rightarrow$ Tensor AR $\rightarrow$ Loop IR $\rightarrow$ Tile IR $\rightarrow$ Kernel IR $\rightarrow$ CUDA. This progressive lowering ensures portability and allows optimizations at each abstraction level.
- Operator Fusion and Efficiency: To minimize costly round trips to global memory, the compiler fuses multiple operations into a single kernel using Loop IR. This is achieved by aligning the iteration space and inlining operations.
- Tiling for GPU Scheduling: The Tile IR is the most complex stage, representing the algebra of the operation (e.g., using a 'fold' operation). This allows the compiler to schedule computations efficiently, enabling techniques like register tiling and defining the input pipeline for optimal GPU utilization.
- Compiler Optimization Philosophy: The compiler is designed to fuse everything feasible at the loop level, deferring hardware-specific decisions (like register spillage) until the final kernel generation stage, thus keeping the lower IR layers highly portable.

## Technical details

- IR Layers and Transformation: The process starts at the high-level PyTorch operations, decomposes them into atomic Tensor AR operations (e.g., NumPy-like), converts them to Loop IR (C-like structure), then to Tile IR (algebraic representation), and finally to Kernel IR (abstract CUDA) before generating the final CUDA code.
- Intermediate Representations (IR): The compiler utilizes several IRs: PyTorch IR (high-level), Tensor AR (atomic operations, universal representation), Loop IR (for fusion, enabling single-kernel execution), and Tile IR (for scheduling, defining the algebra of the operation using 'fold' operations).
- Program Acquisition: The compiler acquires the IR via tracing, converting PyTorch's internal operations one-to-one to the compiler's internal representation.
- Optimization Techniques: Key optimizations include: 1) Operator Fusion (combining operations to reduce global memory round trips); 2) Normalization (converting different internal representations of the same math into a canonical loop IR); and 3) Tiling (extracting the minimal set of properties for scheduling, such as the 'fold' operation).

## Practical implications

- The compiler achieves significant performance gains (e.g., 30% speedup in the pre-fill stage of a model) by optimizing the entire model graph, not just individual kernels.
- The architecture allows for hardware-agnostic optimization up to the Tile IR, making the core compiler logic highly portable.
- The process demonstrates how advanced compiler techniques can bridge the gap between high-level ML frameworks (PyTorch) and low-level hardware primitives (CUDA/Tensor Cores).

## Topics

ML Compilers, GPU Architecture, Intermediate Representation (IR), Compiler Optimization, Tensor Operations, CUDA Programming, cloudrift.ai / riftstack.ai, kernelpacabstack.com, Medium

Source: https://www.youtube.com/watch?v=KK6TWf4Pp6A