From Regression to Transformers

The evolution of machine learning is in three stages: classic regression, then recommender systems, then transformer-based large language models. Each stage crossed a real conceptual boundary that the previous stage couldn't, and the whole arc has a single throughline: machines doing more of the work that humans used to do by hand.

To kick off the story next, here's an attention question. "Do recommender systems used attention already — transformer architecture, yes or no?" The honest answer is both, and the timeline is the whole point. Please read on.

Stage 1 — Classic Regression

The first era of practical ML was built on statistical models with an explicit formula and a cheap way to fit it. At the center sits the generalized linear model family: linear regression predicts a number as a weighted sum of features; logistic regression pushes that sum through a sigmoid to predict a probability; Nelder & Wedderburn unified these under GLMs in 1972.¹ Each coefficient is human-readable — it is the per-feature effect — which is exactly why these models ran the world's prediction systems for decades.

Alongside the linear family grew the tree and margin methods: decision trees (CART, 1984), random forests (Breiman, 2001),² support vector machines (Cortes & Vapnik, 1995),³ and the gradient-boosted tree ensembles that still dominate tabular ML today — XGBoost (Chen & Guestrin, KDD 2016),⁴ LightGBM (2017), CatBoost (2017–18). These capture nonlinearities and feature interactions that linear models can't.

But the decisive ingredient of the whole era wasn't the model — it was manual feature engineering. Because the models are simple, virtually all the task-specific intelligence was injected by humans: log transforms, bucketing, hand-coded cross terms (user × item, age × income), one-hot encodings, rolling-window aggregations. The famous industrial systems — Google's click-prediction pipeline (logistic regression over billions of sparse features)⁵ and Facebook's ad system (boosted trees feeding logistic regression)⁶ — won on features and engineering, not model complexity.

These methods solved tabular prediction superbly — click-through rate, credit scoring, fraud, churn, insurance pricing — and still do. Their limits define the boundary the next stages crossed:

• Linearity / independence assumptions — a linear model can't see an interaction unless a human hand-codes it.
• No sequence or order — nothing in this family natively models "what you did just before."
• High-cardinality categoricals break it — encoding millions of user/item IDs as one-hot columns produces astronomically wide, ~99.99%-sparse matrices, with one isolated weight per ID and no way to share signal between similar entities.
• No representation learning — the model can't learn features from raw data. A human must supply them. This is the gap everything after Stage 1 closes.

Stage 2 — Recommender Systems (how they actually work)

Recommendation looks deceptively like a regression problem ("predict the rating user U gives item I"), but the signal lives somewhere regression can't reach: in the identity of a specific user interacting with a specific item, across millions of each. Recommender systems were the first mainstream technology built to learn from that.

Collaborative filtering: "people like you liked this"

The first generation was collaborative filtering (CF), which uses only the user–item interaction matrix — no hand-built features at all. User-based CF (GroupLens, 1994)⁷ finds users whose rating histories correlate with yours and averages their opinions. Item-based CF (Sarwar et al., 2001; deployed at scale by Amazon, 2003)⁸ instead computes item-to-item similarity — "people who bought this also bought that" — which proved more stable and precomputable, so it scaled.

Matrix factorization: learning the latent space

The conceptual leap came with the Netflix Prize (announced 2006, $1M, won 2009), which pushed the field from neighborhood heuristics to matrix factorization (MF).⁹ The idea is elegant. Take the giant, mostly-empty ratings matrix R (users × items) and approximate it as the product of two thin matrices:

• a user matrix — every user gets a short dense vector (say 50–200 numbers), their embedding;
• an item matrix — every item gets a vector of the same size.

The predicted rating is simply the dot product of the user vector and the item vector (plus a few bias terms). The model is trained — by stochastic gradient descent or alternating least squares — so those dot products reconstruct the ratings you did observe. The magic is what the dimensions become: the model discovers, on its own, latent factors like "how much action," "how serious," "how mainstream." Nobody labels those axes. Two users with similar taste end up with nearby vectors; so do two similar movies. Similarity becomes geometry.

Factorization Machines (Rendle, 2010)¹⁰ generalized this beyond pure user × item: they model all pairwise interactions among arbitrary sparse features (user ID, item ID, time, genre…), where each interaction's strength is again a dot product of learned embeddings — so the model gives sensible estimates even for feature pairs it has barely ever seen, because the embeddings are shared.

Then deep, then attention

The deep-learning wave (2016+) kept the embedding tables but replaced the dot product with neural networks and richer inputs: Google's Wide & Deep (2016), the YouTube deep recommender (2016), DeepFM and Deep & Cross (2017). Then came the attention era you asked about: Deep Interest Network (Alibaba, 2017) learned which of your past behaviors matter for the item being scored, instead of averaging them all equally; and SASRec (2018), BERT4Rec (2019), and Behavior Sequence Transformer (2019) brought full self-attention to model the ordered sequence of your behavior.¹¹

Why recommenders beat plain regression

This is the first comparison you wanted. The difference is not "recsys is a better-tuned regression." It's a different kind of model that learns something regression structurally cannot.

The cleanest way to say it: regression learns one set of weights for the whole population over features you designed; matrix factorization learns a personalized vector for every single user and item, and figures out the meaningful dimensions on its own. That's the first real appearance of representation learning in mainstream ML, and it's why a 10-million-user catalog became tractable.

Stage 3 — Transformers & LLMs

The third stage starts with one paper: "Attention Is All You Need" (Vaswani et al., 2017).¹⁴ It introduced the transformer, built entirely on self-attention, and threw away the step-by-step recurrence of RNNs.

Self-attention is worth understanding intuitively, because it's the same mechanism the attention-based recommenders use. Each token (word) is projected into three vectors:

• a query — "what am I looking for?"
• a key — "what do I offer?"
• a value — "what do I contribute if I'm relevant?"

Every token's query is compared (dot product) against every other token's key, the scores are softmax-normalized into weights, and each token's new representation becomes a relevance-weighted blend of the whole sequence. Two consequences made this revolutionary: it's fully parallel (all positions computed at once, so it trains efficiently on GPUs, unlike sequential RNNs), and it captures long-range dependencies directly, where RNNs forget across distance.

That architecture unlocked the pretraining revolution: train a huge model on mountains of unlabeled text with a self-supervised objective (predict the next token, or fill in masked words), then adapt. BERT and GPT-1 (both 2018), GPT-2 (2019), and then the inflection point — GPT-3 (Brown et al., 2020, 175B parameters),¹⁵ which could do new tasks from a few examples in the prompt, with no retraining. That's in-context learning.

Why did scale keep working? Scaling laws (Kaplan et al., 2020)¹⁶ showed test loss falls as a smooth power law in model size, data, and compute — no saturation in sight. Chinchilla (Hoffmann et al., 2022)¹⁷ corrected the recipe (scale data and parameters together, ~20 tokens per parameter). And past certain scales, emergent abilities appeared — reasoning, instruction-following — that simply weren't present in smaller models (Wei et al., 2022).¹⁸ Stanford named this whole class "foundation models" (2021):¹⁹ train once on broad data, adapt to thousands of tasks.

Why transformers surpass recommenders (and where recsys still wins)

This is your second comparison. The key insight: transformers/LLMs are more powerful mostly in the sense of general, not because they're better at the narrow ranking job.

So an LLM can explain a recommendation, reason about why you might like something, handle a brand-new domain it was never trained on, and generate open-ended output. A classical recommender can only score items from a list it already knows. That generality is the leap.

The real 2024–2026 trajectory is convergence, not replacement. Meta's HSTU / "Actions Speak Louder than Words" (ICML 2024)²⁰ recast recommendation as generative sequential transduction and — the headline result — demonstrated LLM-style scaling laws inside a recommender, scaling to 1.5-trillion-parameter generative recommenders (a first for the field) with up to 65.8% NDCG gains on public data and a 12.4% online A/B lift deployed to surfaces serving billions of users. Google's TIGER (2023)²¹ generates semantic item IDs autoregressively. Meanwhile LLMs increasingly wrap around recommenders — generating content embeddings, solving cold-start, understanding intent, writing explanations. The endpoint isn't "LLMs replace recsys"; it's transformer-based recommenders doing the ranking, with LLMs doing the understanding.

The throughline

Read the three stages as one trend and the pattern — at each step, the machine takes over more of the work humans used to do by hand:

• Stage 1 (regression): humans hand-craft every feature; the model just weights them.
• Stage 2 (recommenders): the model learns its own features (embeddings) for users and items — representation learning arrives, but still one narrow task at a time.
• Stage 3 (transformers/LLMs): the model learns general-purpose representations from raw data at scale, and a single model transfers across countless tasks — and even circles back to make recommenders themselves more powerful.

Less hand-engineering, more learned representation, more generality. Your three-stage framing captures exactly that progression — and the nuance you sensed about attention is real: recommender systems weren't simply "before transformers." They were one of the first places self-attention proved itself, and they're now where transformer-scale ideas are coming full circle.

Sources

The three most load-bearing recent claims — live-verified June 28, 2026: HSTU (ICML 2024) figures, SASRec (ICDM 2018), and DIN (2017/KDD 2018) as the first attention-based recommender. The remaining foundational references are canonical, widely-cited papers cited at their standard arXiv/DOI locations.

1. Nelder & Wedderburn, "Generalized Linear Models," JRSS (1972). jstor.org/stable/2344614
2. Breiman, "Random Forests," Machine Learning (2001). springer
3. Cortes & Vapnik, "Support-Vector Networks," Machine Learning (1995). springer
4. Chen & Guestrin, "XGBoost: A Scalable Tree Boosting System," KDD (2016). arxiv.org/abs/1603.02754
5. McMahan et al., "Ad Click Prediction: a View from the Trenches," KDD (2013). research.google
6. He et al., "Practical Lessons from Predicting Clicks on Ads at Facebook" (2014). research.facebook.com
7. Resnick et al., "GroupLens: An Open Architecture for Collaborative Filtering of Netnews," CSCW (1994). dl.acm.org
8. Linden, Smith & York, "Amazon.com Recommendations: Item-to-Item Collaborative Filtering," IEEE Internet Computing (2003). ieee · Sarwar et al., "Item-Based Collaborative Filtering," WWW (2001). dl.acm.org
9. Koren, Bell & Volinsky, "Matrix Factorization Techniques for Recommender Systems," IEEE Computer (2009). ieee
10. Rendle, "Factorization Machines," ICDM (2010). ieee
11. [verified] Zhou et al., "Deep Interest Network for CTR Prediction" (2017; KDD 2018) 1706.06978; [verified] Kang & McAuley, "Self-Attentive Sequential Recommendation (SASRec)," ICDM (2018) 1808.09781; Sun et al., "BERT4Rec," CIKM (2019) 1904.06690; Chen et al., "Behavior Sequence Transformer" (2019) 1905.06874; Cheng et al., "Wide & Deep" (2016) 1606.07792.
12. Grinsztajn et al., "Why do tree-based models still outperform deep learning on tabular data?" NeurIPS (2022) 2207.08815; Shwartz-Ziv & Armon, "Tabular Data: Deep Learning is Not All You Need" (2021) 2106.03253.
13. Rendle et al., "Neural Collaborative Filtering vs. Matrix Factorization Revisited," RecSys (2020). 2005.09683
14. Vaswani et al., "Attention Is All You Need," NeurIPS (2017). 1706.03762
15. Brown et al., "Language Models are Few-Shot Learners (GPT-3)" (2020). 2005.14165 · Devlin et al., "BERT" (2018) 1810.04805
16. Kaplan et al., "Scaling Laws for Neural Language Models" (2020). 2001.08361
17. Hoffmann et al., "Training Compute-Optimal Large Language Models (Chinchilla)" (2022). 2203.15556
18. Wei et al., "Emergent Abilities of Large Language Models," TMLR (2022). 2206.07682
19. Bommasani et al., "On the Opportunities and Risks of Foundation Models," Stanford CRFM (2021). 2108.07258
20. [verified] Zhai et al., "Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommenders (HSTU)," ICML (2024) — confirmed: up to 65.8% NDCG, 5.3×–15.2× faster than FlashAttention2 at 8192-length, 1.5T params, 12.4% online A/B lift. 2402.17152
21. Rajput et al., "Recommender Systems with Generative Retrieval (TIGER)," NeurIPS (2023). 2305.05065