Decoding AI Performance: Analyzing TOPS and Tokens on NVIDIA RTX PCs

The era of the AI PC is here, powered by NVIDIA RTX and GeForce RTX technologies. This shift brings a new way to evaluate performance for AI-accelerated tasks, introducing metrics that can be daunting to decipher when choosing between desktops and laptops, according to the NVIDIA Blog.

Coming Out on TOPS

The first baseline is TOPS, or trillions of operations per second. This metric is akin to an engine’s horsepower rating, with higher numbers indicating better performance. For instance, the Copilot+ PC lineup by Microsoft includes neural processing units (NPUs) capable of performing upwards of 40 TOPS, sufficient for light AI-assisted tasks. However, NVIDIA RTX and GeForce RTX GPUs deliver unprecedented performance, with the GeForce RTX 4090 GPU offering more than 1,300 TOPS, essential for demanding generative AI tasks, such as AI-assisted digital content creation and querying large language models (LLMs).

Insert Tokens to Play

LLM performance is measured in the number of tokens generated by the model. Tokens can be words, punctuation, or whitespace. AI performance can be quantified in “tokens per second.” Another crucial factor is batch size, the number of inputs processed simultaneously. Larger batch sizes enhance performance but require more memory. RTX GPUs excel in this area due to their substantial video random access memory (VRAM), Tensor Cores, and TensorRT-LLM software.

GeForce RTX GPUs offer up to 24GB of high-speed VRAM, and NVIDIA RTX GPUs up to 48GB, enabling higher batch sizes and larger models. Tensor Cores, dedicated AI accelerators, significantly speed up operations required for deep learning and generative AI models. Applications using the NVIDIA TensorRT software development kit (SDK) can unlock maximum performance on over 100 million Windows PCs and workstations powered by RTX GPUs.

Text-to-Image, Faster Than Ever

Measuring image generation speed is another way to evaluate performance. Stable Diffusion, a popular image-based AI model, allows users to convert text descriptions into complex visual representations. With RTX GPUs, these results can be generated faster than on CPUs or NPUs. Performance is further enhanced using the TensorRT extension for the Automatic1111 interface, enabling RTX users to generate images from prompts up to 2x faster with the SDXL Base checkpoint.

ComfyUI, another popular Stable Diffusion interface, recently added TensorRT acceleration, allowing RTX users to generate images from prompts up to 60% faster and convert these images to videos up to 70% faster. The new UL Procyon AI Image Generation benchmark shows a 50% speedup on a GeForce RTX 4080 SUPER GPU compared to the fastest non-TensorRT implementation.

TensorRT acceleration will soon be available for Stable Diffusion 3, Stability AI’s new text-to-image model, boosting performance by 50%. The TensorRT-Model Optimizer further accelerates performance, resulting in a 70% speedup and a 50% reduction in memory consumption.

The true test of these advancements is in real-world use cases. Users can refine image generation by tweaking prompts significantly faster on RTX GPUs, taking seconds per iteration compared to minutes on other systems. This speed and security are achieved with everything running locally on an RTX-powered PC or workstation.

The Results Are in and Open Sourced

The AI researchers behind Jan.ai recently integrated TensorRT-LLM into their local chatbot app and benchmarked these optimizations. They found that TensorRT is “30-70% faster than llama.cpp on the same hardware” and more efficient on consecutive processing runs. The team’s methodology is open for others to measure generative AI performance for themselves.

From gaming to generative AI, speed is crucial. TOPS, images per second, tokens per second, and batch size are all vital metrics in determining performance.