Interpretability methods for language models often seek meaningful concepts in activations while treating features as independent across time. This work shows that language model representations have rich temporal structure, including changing conceptual dimensionality, context-dependent correlations, and non-stationarity, which conflicts with the priors implicit in standard sparse autoencoders. It introduces Temporal SAE, a temporally biased architecture that separates predictable contextual components from residual novel information. Experiments show improved handling of garden path sentences, event boundaries, and slow versus fast-moving information in model activations.