According to God of Prompt (@godofprompt), grokking was first discovered by accident in 2022 when OpenAI researchers trained AI models on simple mathematical tasks such as modular addition and permutation groups. Initially, these models exhibited rapid overfitting and poor generalization during standard training. However, when the training was extended far beyond typical convergence—over 10,000 epochs—the models suddenly achieved perfect generalization, a result that defied conventional expectations. This phenomenon, termed 'grokking,' suggests new opportunities for AI practitioners to enhance model robustness and generalization by rethinking training duration and monitoring. The discovery holds significant implications for AI model training strategies, particularly in applications demanding high reliability and transferability. (Source: @godofprompt on Twitter, Jan 6, 2026)

According to Chris Olah on Twitter, introducing a repeated data point, such as p=[1,1,1,0,0,0,0...], into AI transcoder training data leads the model to develop a unique feature specifically for memorizing that point. This insight highlights a key challenge in AI model training: overfitting to repeated or outlier data, which can impact generalization and model robustness (source: Chris Olah, Twitter, August 8, 2025). For businesses deploying AI solutions, understanding how training data structure affects model behavior opens opportunities for optimizing data engineering workflows to prevent memorization and improve real-world performance.