Claude 3.5 Haiku Counts Characters on Curved Feature Manifolds

This episode explores how Claude 3.5 Haiku decides when a hard-wrapped line of text is full, even though it sees only token IDs and is never told the line width. It covers the paper's finding that the model tracks the character count on a curved, roughly six-dimensional manifold. That manifold is tiled by about ten sparse features with widening tuning curves, a bit like place cells. The discussion connects this to earlier work on circular and helical feature geometry and to the rotary position encoding used in transformers, and it explains attribution graphs and sparse dictionary features along the way. It also covers the experimental setup: newlines reinserted at known widths from 15 to 150 characters. Zero-ablating the character-count subspace raises loss mainly on newline tokens, which gives causal evidence that the model uses this subspace and doesn't just encode the count somewhere a probe can find it. Listeners interested in mechanistic interpretability will find a concrete case of a model implementing a continuous quantity as geometry, not as one direction per concept.

Sources: 1. When Models Manipulate Manifolds: The Geometry of a Counting Task — Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, Joshua Batson, 2026 http://arxiv.org/abs/2601.04480 2. Toy Models of Superposition — Nelson Elhage, Tristan Hume, Catherine Olsson, et al. (Anthropic), 2022 https://scholar.google.com/scholar?q=Toy+Models+of+Superposition 3. Not All Language Model Features Are Linear — Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, Max Tegmark, 2024 https://scholar.google.com/scholar?q=Not+All+Language+Model+Features+Are+Linear 4. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — Trenton Bricken, Adly Templeton, Joshua Batson, et al. (Anthropic), 2023 https://scholar.google.com/scholar?q=Towards+Monosemanticity%3A+Decomposing+Language+Models+With+Dictionary+Learning 5. Language Models Use Trigonometry to Do Addition (and related work on helical number representations) — Subhash Kantamneni, Max Tegmark, 2025 https://scholar.google.com/scholar?q=Language+Models+Use+Trigonometry+to+Do+Addition+%28and+related+work+on+helical+number+representations%29 6. Circuit Tracing: Revealing Computational Graphs in Language Models — Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas Turner, Brian Chen, Craig Citro, et al. (Anthropic), 2025 https://scholar.google.com/scholar?q=Circuit+Tracing%3A+Revealing+Computational+Graphs+in+Language+Models 7. On the Biology of a Large Language Model — Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, et al. (Anthropic), 2025 https://scholar.google.com/scholar?q=On+the+Biology+of+a+Large+Language+Model 8. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small — Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, Jacob Steinhardt, 2022 https://scholar.google.com/scholar?q=Interpretability+in+the+Wild%3A+a+Circuit+for+Indirect+Object+Identification+in+GPT-2+small 9. A Mathematical Framework for Transformer Circuits — Nelson Elhage, Neel Nanda, Catherine Olsson, et al. (Anthropic), 2021 https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits 10. RoFormer: Enhanced Transformer with Rotary Position Embedding — Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu, 2021 https://scholar.google.com/scholar?q=RoFormer%3A+Enhanced+Transformer+with+Rotary+Position+Embedding 11. Quanta of model skills: The Quantization Model of Neural Scaling — Eric J. Michaud, Ziming Liu, Uzay Girit, Max Tegmark, 2023 https://scholar.google.com/scholar?q=Quanta+of+model+skills%3A+The+Quantization+Model+of+Neural+Scaling 12. Space and Time Representations in Language Models (Language Models Represent Space and Time) — Wes Gurnee, Max Tegmark, 2023 https://scholar.google.com/scholar?q=Space+and+Time+Representations+in+Language+Models+%28Language+Models+Represent+Space+and+Time%29 13. Grid cells / place cells literature: The Hippocampus as a Cognitive Map and Microstructure of a spatial map in the entorhinal cortex — John O'Keefe and Lynn Nadel (1978); Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, Edvard Moser (2005), 1978 / 2005 https://scholar.google.com/scholar?q=Grid+cells+%2F+place+cells+literature%3A+The+Hippocampus+as+a+Cognitive+Map+and+Microstructure+of+a+spatial+map+in+the+entorhinal+cortex 14. Not All Language Model Features Are One-Dimensionally Linear — Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, Max Tegmark, 2025 https://scholar.google.com/scholar?q=Not+All+Language+Model+Features+Are+One-Dimensionally+Linear 15. The Origins of Representation Manifolds in Large Language Models — Alexander Modell, Patrick Rubin-Delanchy, Nick Whiteley, 2025 https://scholar.google.com/scholar?q=The+Origins+of+Representation+Manifolds+in+Large+Language+Models 16. Language Models Use Trigonometry to Do Addition — Subhash Kantamneni, Max Tegmark, 2025 https://scholar.google.com/scholar?q=Language+Models+Use+Trigonometry+to+Do+Addition 17. Progress Measures for Grokking via Mechanistic Interpretability — Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt, 2023 https://scholar.google.com/scholar?q=Progress+Measures+for+Grokking+via+Mechanistic+Interpretability 18. Understanding Sparse Autoencoder Scaling in the Presence of Feature Manifolds — Eric J. Michaud, Liv Gorton, Tom McGrath, 2025 https://scholar.google.com/scholar?q=Understanding+Sparse+Autoencoder+Scaling+in+the+Presence+of+Feature+Manifolds 19. Curve Detectors / Curve Detector Manifolds in InceptionV1 — Nick Cammarata et al.; Liv Gorton, 2020 / 2024 https://scholar.google.com/scholar?q=Curve+Detectors+%2F+Curve+Detector+Manifolds+in+InceptionV1 20. Tracing Attention Computation Through Feature Interactions (QK attributions) — Harish Kamath, Emmanuel Ameisen, Isaac Kauvar, et al., 2025 https://scholar.google.com/scholar?q=Tracing+Attention+Computation+Through+Feature+Interactions+%28QK+attributions%29 21. Language Models Need Inductive Biases to Count Inductively — Yingshan Chang, Yonatan Bisk, 2024 https://scholar.google.com/scholar?q=Language+Models+Need+Inductive+Biases+to+Count+Inductively 22. Length Representations in Large Language Models — Sangjun Moon et al., 2025 https://scholar.google.com/scholar?q=Length+Representations+in+Large+Language+Models 23. Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics — Yaniv Nikankin, Anja Reusch, Aaron Mueller, Yonatan Belinkov, 2024 https://scholar.google.com/scholar?q=Arithmetic+Without+Algorithms%3A+Language+Models+Solve+Math+with+a+Bag+of+Heuristics 24. Understanding In-Context Learning of Addition via Activation Subspaces — Xinyan Hu, Kayo Yin, Michael I. Jordan, Jacob Steinhardt, Lijie Chen, 2025 https://scholar.google.com/scholar?q=Understanding+In-Context+Learning+of+Addition+via+Activation+Subspaces 25. Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry — Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, Demba Ba, 2025 https://scholar.google.com/scholar?q=Projecting+Assumptions%3A+The+Duality+Between+Sparse+Autoencoders+and+Concept+Geometry 26. Language Models Represent Space and Time — Wes Gurnee, Max Tegmark, 2024 https://scholar.google.com/scholar?q=Language+Models+Represent+Space+and+Time 27. The Quantization Model of Neural Scaling — Eric J. Michaud, Ziming Liu, Uzay Girit, Max Tegmark, 2023 https://scholar.google.com/scholar?q=The+Quantization+Model+of+Neural+Scaling

Interactive Visualization: Claude 3.5 Haiku Counts Characters on Curved Feature Manifolds