TimesFM: Google's Foundation Model for Time-Series Forecasting
319
Google Research's TimesFM is a pretrained decoder-only foundation model that brings large-scale transformer efficiency to time-series forecasting.
Transformer Models
Architecture and applications of transformer neural networks, including BERT, GPT, and their variants for various machine learning tasks.
Reading List
MSA: Scaling LLM Context to 100M Tokens via Sparse Latent Memory
MSA is an end-to-end trainable framework that enables LLMs to process 100 million tokens efficiently using sparse attention and latent memory.
The LLM Architecture Gallery: Mapping the Evolution of Open-Weight Models
383
A comprehensive technical reference gallery documenting the architectural evolution and specifications of modern open-weight large language models.
Turning BERT’s MLM Into a Text Diffusion Generator
455
BERT-style MLM is a single-step text diffusion process, and extending it to multiple masking steps turns RoBERTa into a workable text generator.
When ‘Seahorse + Emoji’ Hits an Empty Token: Why LLMs Invent the Seahorse Emoji
734
Models compose “seahorse + emoji,” but with no matching token the unembedding snaps to a nearby emoji, causing confident errors and occasional feedback loops.
SimpleFold: Scalable Flow-Matching Transformers for Protein Folding
471
A large-scale, transformer-only, flow-matching approach makes protein folding simpler while staying competitive and practical.
Why Embeddings Got Bigger—and Where Efficiency Pulls Them Next
113
Embeddings got bigger with Transformers and APIs, but new efficiency techniques and infrastructure mean the future is about smarter—not just larger—dimensions.
Inside a Tiny GPT: A Visual Walkthrough of Autoregressive Prediction
640
A visual, end-to-end demo of a tiny GPT that turns tokens into embeddings, runs them through transformers, and autoregressively predicts the next token to solve a simple sorting task.