IBM Research Unveils Innovations to Accelerate Enterprise AI Training

IBM Research has unveiled groundbreaking innovations aimed at scaling the data processing pipeline for enterprise AI training, according to IBM Research. These advancements are designed to expedite the creation of powerful AI models, such as IBM’s Granite models, by leveraging the abundant capacity of CPUs.

Optimizing Data Preparation

Before training AI models, vast amounts of data must be prepared. This data often comes from diverse sources like websites, PDFs, and news articles, and must undergo several preprocessing steps. These steps include filtering out irrelevant HTML code, removing duplicates, and screening for abusive content. These tasks, though critical, are not constrained by the availability of GPUs.

Petros Zerfos, IBM Research’s principal research scientist for watsonx data engineering, emphasized the importance of efficient data processing. “A large part of the time and effort that goes into training these models is preparing the data for these models,” Zerfos said. His team has been developing methods to enhance the efficiency of data processing pipelines, drawing expertise from various domains including natural language processing, distributed computing, and storage systems.

Leveraging CPU Capacity

Many steps in the data processing pipeline involve “embarrassingly parallel” computations, allowing each document to be processed independently. This parallel processing can significantly speed up data preparation by distributing tasks across numerous CPUs. However, some steps, such as removing duplicate documents, require access to the entire dataset, which cannot be performed in parallel.

To accelerate IBM’s Granite model development, the team has developed processes to rapidly provision and utilize tens of thousands of CPUs. This approach involves marshalling idle CPU capacity across IBM’s Cloud datacenter network, ensuring high communication bandwidth between CPUs and data storage. Traditional object storage systems often cause CPUs to idle due to low performance; thus, the team employed IBM’s high-performance Storage Scale file system to cache active data efficiently.

Scaling Up AI Training

Over the past year, IBM has scaled up to 100,000 vCPUs in the IBM Cloud, processing 14 petabytes of raw data to produce 40 trillion tokens for AI model training. The team has automated these data pipelines using Kubeflow on IBM Cloud. Their methods have proven to be 24 times faster in processing data from Common Crawl compared to previous techniques.

All of IBM’s open-sourced Granite code and language models have been trained using data prepared through these optimized pipelines. Additionally, IBM has made significant contributions to the AI community by developing the Data Prep Kit, a toolkit hosted on GitHub. This kit streamlines data preparation for large language model applications, supporting pre-training, fine-tuning, and retrieval-augmented generation (RAG) use cases. Built on distributed processing frameworks like Spark and Ray, the kit allows developers to build scalable custom modules.

For more information, visit the official IBM Research blog.