LLM Inference

The mechanics of LLM token generation including sampling pipelines, logit processing, temperature scaling, and decoding strategies.

Reading List

Agentic Systems

Forge v0.6.0: Standardizing LLM Sampling and Advanced Reasoning Benchmarks

May 19, 2026685

Forge is a specialized LLM framework for standardizing model orchestration and rigorous performance evaluation across local and cloud backends.

AI Benchmarks LLM Inference LLM Reasoning Developer Tooling

Products & Announcements

Interfaze: A Hybrid Architecture for High-Accuracy Deterministic AI

May 11, 2026164

Interfaze is a hybrid AI model that merges DNN precision with transformer flexibility to outperform generalist LLMs in high-accuracy, deterministic tasks.

AI Architecture AI Benchmarks LLM Inference Structured Output Computer Vision

Products & Announcements

GPT-5.5 Cost Analysis: How Reduced Verbosity Softens the 2x Price Hike

May 8, 2026214

GPT-5.5's 2x price increase is mitigated to a 49-92% actual cost rise because the model produces shorter responses for long prompts.

LLM Inference AI Business Models Token Optimization OpenAI Technology Economics

Damage Control

From Fan to Former User: Why I Canceled Claude

Apr 24, 2026963

A power user cancels their Claude subscription due to declining AI quality, poor support, and inconsistent token management.

Anthropic AI Coding Agents Platform Decay LLM Inference AI & Productivity

Products & Announcements

DeepSeek API Quick Start Guide

Apr 24, 20262066

DeepSeek's API allows developers to access advanced AI models using familiar OpenAI-compatible SDKs and simple configuration changes.

API Integration LLM Inference LLM Reasoning Developer Tooling

Products & Announcements

OpenAI Unveils GPT-5.5: The Next Step in Agentic AI

Apr 23, 20261568

GPT-5.5 is a faster, more efficient, and highly autonomous agentic AI designed to transform professional work and scientific research.

OpenAI AI Agents LLM Inference AI Safety AI Benchmarks

Products & Announcements

Google Launches 8th-Gen TPUs for the Agentic AI Era

Apr 22, 2026451

Google's new 8th-gen TPUs provide specialized, high-efficiency hardware for training and serving the next generation of reasoning AI agents.

AI Hardware AI Infrastructure AI Agents Google LLM Inference

Under the Hood

Claude Opus 4.7 and the Cost of Token Inflation

Apr 20, 2026224

Claude Opus 4.7's new tokenizer increases token counts for the same data, effectively raising costs despite unchanged per-token pricing.

Token Optimization Anthropic AI Business Models LLM Inference Developer Tooling

Damage Control

Opus 4.7: Community Data Shows 39% Increase in Token Costs

Apr 18, 2026613

Upgrading from Opus 4.6 to 4.7 leads to a nearly 40% increase in token usage and API costs.

Token Optimization AI Business Models LLM Inference Anthropic

Products & Announcements

Cloudflare Unifies AI Inference for the Agentic Era

Apr 16, 2026306

Cloudflare’s AI Platform now serves as a unified, high-performance inference layer that simplifies building and scaling AI agents across multiple model providers.

AI Agents AI Infrastructure LLM Inference Cloud Infrastructure API Integration

Products & Announcements

Anthropic Launches Claude Opus 4.7 with Advanced Coding Autonomy

Apr 16, 20261948

Claude Opus 4.7 is a major upgrade focused on autonomous engineering, superior vision, and refined developer controls.

Anthropic AI Coding Agents Multimodal AI AI Agents LLM Inference

Under the Hood

I-DLM: Matching Autoregressive Quality with Parallel Diffusion Speed

Apr 14, 2026267

I-DLM achieves autoregressive-level quality and significantly higher throughput by incorporating a self-verification mechanism into parallel diffusion decoding.

Diffusion Models LLM Inference AI Architecture LLM Reasoning

Damage Control

Anthropic Defends Claude Code Cache TTL Reduction as Cost Optimization

Apr 12, 2026548

Anthropic defended a shift from 1-hour to 5-minute cache TTL in Claude Code as a cost-saving measure, despite user claims that it increased expenses for high-context sessions.

Anthropic AI Business Models AI Coding Agents LLM Inference Developer Tooling

Agentic Systems

Optimizing AI Spend: Moving from Claude Subscriptions to OpenRouter and Zed

Apr 9, 2026344

Switch from fixed AI subscriptions to usage-based credits using Zed and OpenRouter to maximize flexibility and stop wasting money on non-rolling limits.

AI & Productivity Developer Tooling AI Business Models LLM Inference AI Coding Agents

Agentic Systems

Research-Driven Agents: Enhancing AI Code Optimization via Literature Search

Apr 9, 2026177

Coding agents produce superior performance optimizations when they research academic papers and competing implementations to gain domain knowledge before touching code.

AI Coding Agents Autonomous Research Agents LLM Inference Cloud Infrastructure Algorithms & Optimization

Damage Control

Claude Code Quota Crisis: Bugs and Policy Changes Exhaust User Limits

Mar 31, 2026324

Anthropic's Claude Code is facing backlash as a mix of policy changes and technical bugs causes users to hit usage limits prematurely, stalling developer workflows.

Anthropic AI Coding Agents AI Business Models Developer Experience LLM Inference

Agentic Systems

Nullclaw: Building a Code-Aware AI Doorman via IRC

Mar 27, 2026331

A secure, dual-agent AI system using IRC to provide code-aware portfolio insights while protecting private data through a hardened architecture.

AI Agents Self-Hosting Multi-Agent Systems LLM Inference Vendor Lock-in

Agentic Systems

Scaling Local RAG: Lessons from Indexing 451GB of Data

Mar 26, 2026322

Building an enterprise-scale local RAG system requires transitioning from simple scripts to a robust architecture involving data filtering, persistent vector databases, and dedicated GPU hardware.

Retrieval-Augmented Generation Vector Databases Self-Hosting AI Infrastructure LLM Inference

Under the Hood

Quantization: How to Run Massive LLMs on Your Laptop

Mar 25, 2026248

Quantization is a compression technique that makes LLMs significantly smaller and faster for local use with minimal impact on their intelligence.

On-Device AI LLM Inference AI Infrastructure Model Quantization

Under the Hood

MSA: Scaling LLM Context to 100M Tokens via Sparse Latent Memory

Mar 24, 2026

MSA is an end-to-end trainable framework that enables LLMs to process 100 million tokens efficiently using sparse attention and latent memory.

LLM Context Management Retrieval-Augmented Generation AI Architecture LLM Inference Transformer Models

Agentic Systems

Applying Distributed Systems Principles to LLM Teams

Mar 16, 2026104

The research advocates for using distributed systems theory as a formal framework to design and evaluate multi-agent LLM teams more effectively.

Multi-Agent Systems Distributed Systems AI Architecture LLM Inference

Under the Hood

The LLM Architecture Gallery: Mapping the Evolution of Open-Weight Models

Mar 16, 2026383

A comprehensive technical reference gallery documenting the architectural evolution and specifications of modern open-weight large language models.

AI Architecture Foundation Models Mixture of Experts LLM Inference Transformer Models

Products & Announcements

Claude Off-Peak Usage Double Promotion March 2026

Mar 14, 2026243

Claude is doubling usage limits during off-peak hours for most plan types from March 13 to March 27, 2026.

Anthropic AI Business Models AI & Productivity LLM Inference

Under the Hood

Can I Run AI: The Local LLM Hardware Compatibility Guide

Mar 13, 20261404

A hardware compatibility tool that grades the local performance of AI models based on a user's specific GPU and VRAM configuration.

On-Device AI LLM Inference Self-Hosting AI Hardware Developer Tooling

Agentic Systems

Slash Claude API Costs with Automated Prompt Caching

Mar 13, 2026

An open-source MCP tool that automates Anthropic prompt caching to reduce token costs by 90% and provide deep usage observability.

Model Context Protocol Anthropic LLM Inference AI & Productivity Observability

Damage Control

Debunking the $5,000 Claude Code Loss Myth

Mar 10, 2026477

The reported $5,000 loss per Claude Code user is based on retail markups rather than actual compute costs, masking the fact that Anthropic's inference is likely profitable.

Anthropic AI Business Models LLM Inference AI Hype Competitive Moats

Under the Hood

From Sampling to Grammars: Making LLMs Reliably Output Structured Data (Even for Thinking Models)

Sep 23, 2025234

Use efficient sampling plus grammar constraints to guarantee format today, but expect models to natively emit structured outputs tomorrow—especially when you let them think first, then constrain.

Structured Output LLM Inference LLM Reasoning

Agentic Systems

Faster LLMs, Bigger Demands: Why Coding Agents Won’t Stabilize Soon

Sep 22, 2025137

Faster LLMs will reshape coding workflows and productivity, but escalating demand, hardware limits, and pricing pressures mean a bumpy, fast-changing road ahead.

AI Coding Agents AI & Productivity AI Infrastructure LLM Inference AI Business Models

Damage Control

Postmortem: Three Overlapping Infra Bugs Degraded Claude—Fixes Shipped, Evals and Tooling Upgraded

Sep 17, 2025381

Three infrastructure bugs—not load or demand—degraded Claude; rollbacks and a shift to exact top‑k fixed them, and Anthropic is upgrading evaluations and debugging while asking for user feedback.

AI Infrastructure LLM Inference Incident Response Service Reliability Corporate Accountability

Products & Announcements

Qwen3-Next: Hybrid Attention + Ultra-Sparse MoE for 10x Faster Long-Context LLMs

Sep 12, 2025569

Qwen3-Next matches larger models while slashing training cost and delivering order-of-magnitude faster long-context inference via a hybrid attention + ultra-sparse MoE design with native MTP.

AI Architecture Mixture of Experts LLM Inference LLM Context Management

Under the Hood

A Skeptic’s Guide to Running Local LLMs on macOS

Sep 8, 2025388

A pragmatic, privacy-first guide to running and choosing small local LLMs on macOS—what to use, how to pick, and how to stay safe and sane.

On-Device AI LLM Inference Open Source Data Privacy

Under the Hood

Inside a Tiny GPT: A Visual Walkthrough of Autoregressive Prediction

Sep 5, 2025640

A visual, end-to-end demo of a tiny GPT that turns tokens into embeddings, runs them through transformers, and autoregressively predicts the next token to solve a simple sorting task.

Transformer Models LLM Inference Interactive Web Tools AI Interpretability