Skip to main content
Scheduled maintenance: Wednesday, October 28, 7:00 to 12:00 EDT (11:00 to 16:00 UTC).Read more on our blog
archive Search
Submit Donate Log in

Computer Science > Machine Learning

(cs)
[Submitted on 22 Oct 2026]

Title:A Fix for Softmax and Cross-Entropy

Authors:Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen
View a PDF of the paper titled A Fix for Softmax and Cross-Entropy, by Jonathan T. Barron and 2 other authors
View PDF HTML (experimental)
Abstract:Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student and teacher logits are equal. We present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs. Our fix eliminates the previously-observed "Slingshot" phenomenon, speeds up grokking with weight decay, prevents accuracy collapse without weight decay, stops student networks from drifting away from the teacher during distillation, and shows that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's ϵ hyperparameter.
Subjects: Machine Learning (cs.LG)
Cite as: arXiv:2610.31642 [cs.LG]
  (or arXiv:2610.31642v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2610.31642
arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jonathan Barron [view email]
[v1] Thu, 22 Oct 2026 17:54:56 UTC (1,136 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled A Fix for Softmax and Cross-Entropy, by Jonathan T. Barron and 2 other authors
  • View PDF
  • HTML (experimental)
  • TeX Source
license icon view license

Additional Features

  • Audio Summary

Current browse context:

cs.LG
< prev   |   next >
new | recent | 2026-10
Change to browse by:
cs

References & Citations

  • SciX
  • Google Scholar
  • Semantic Scholar
Loading...

Bookmark

BibSonomy Reddit

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
We gratefully acknowledge support from our major funders, member institutions, , and all contributors.
About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab)