According to Andrej Karpathy on X, NanoChat now trains a GPT-2 capability model in about 2 hours on a single 8x H100 node, down from roughly 3 hours a month ago, driven primarily by switching the pretraining dataset from FineWeb-edu to NVIDIA ClimbMix and enabling FP8 optimizations (as reported by Karpathy). According to Karpathy, alternative datasets including Olmo, FineWeb, and DCLM produced regressions, while ClimbMix worked out of the box, suggesting immediate gains in data efficiency and reduced tuning overhead for small LLM pipelines. As reported by Karpathy, he also set up autonomous AI agents to iterate on NanoChat, making 110 changes over ~12 hours and improving validation loss from 0.862415 to 0.858039 for a d12 model without adding wall-clock time, indicating a viable pattern for continuous training-ops automation. For practitioners, this points to business opportunities in GPU cost optimization using FP8, higher-quality synthetic or curated corpora like ClimbMix for faster convergence, and agent-driven MLOps that continuously test and merge performance-improving changes.

According to Andrej Karpathy on X, nanochat now trains a GPT-2 capability model in about 2 hours on a single 8x H100 node, improved from ~3 hours a month ago, driven primarily by switching the dataset from FineWeb-edu to NVIDIA ClimbMix alongside FP8 and other tuning features (source: Andrej Karpathy on X, Mar 5, 2026). As reported by Karpathy, alternative datasets including Olmo, FineWeb, and DCLM caused regressions, while ClimbMix worked well out of the box, suggesting immediate gains in data quality and curriculum for smaller models (source: Andrej Karpathy on X). According to Karpathy, an AI agent system now continuously iterates on nanochat, making 110 changes over ~12 hours and reducing validation loss from 0.862415 to 0.858039 for a d12 model without adding wall‑clock time by running on a feature branch and merging effective ideas (source: Andrej Karpathy on X). For practitioners, the cited results highlight business opportunities in faster LLM training cycles on commodity 8x H100 nodes, data curation advantages from ClimbMix, and automation leverage via agent-driven MLOps for continuous training and deployment (source: Andrej Karpathy on X).