LLM Speed: Pre-Purchase Hardware Benchmarking for Local AI

This tool enables users to predict the speed of local LLMs by selecting specific hardware, models, and quantization settings. It provides essential metrics like tokens per second and latency based on community benchmarks and hardware specs. Ultimately, it helps AI enthusiasts visualize performance before buying or building a local AI workstation.

Key Points

• The tool allows users to simulate LLM performance across a vast range of consumer and professional hardware, including NVIDIA, AMD, and Apple chips.
• Performance metrics focus on tokens per second (tok/s) and Time to First Token (TTFT) to give a realistic sense of model responsiveness.
• Data is sourced from a mix of community benchmarks like r/LocalLLaMA and projections based on hardware specifications.
• The platform supports various quantization levels, which are critical for fitting large models onto local VRAM.
• It serves as a pre-purchase utility for enthusiasts and professionals to 'race' different configurations.

Sentiment

Generally positive regarding the tool's concept and utility, but highly skeptical and critical regarding the accuracy of its specific performance data.

In Agreement

• The tool is highly valuable for users looking to avoid spending thousands on hardware that might not meet their performance needs.
• It provides a helpful way to compare the relative performance jumps between hardware generations, such as Apple's M4 Max versus M5 Max.
• Simulating speeds prevents the 'rude' practice of downloading massive model files and multiple quantizations just to test inference speed.

Opposed

• The performance estimates are frequently inaccurate, with some users reporting real-world speeds significantly higher than the simulator suggests.
• The tool's 'won't run' logic is too restrictive because it doesn't account for hybrid CPU/GPU offloading supported by engines like llama.cpp.
• Renting a cloud GPU for a few cents is arguably a more reliable and cost-effective way to test a specific model's performance before buying hardware.
• The simulator lacks critical variables such as multi-GPU support, PCIe lane speeds, and specific inference engine versions which greatly impact real-world throughput.