Skip to main content
archive
Search Submit Donate Log in

Computer Science > Computation and Language

(cs)
[Submitted on 8 Nov 2025 (v1), last revised 4 Sep 2026 (this version, v3)]

Title:Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Authors:Renfei Zhang, Manasa Kaniselvan, Rylan Schaeffer, Niloofar Mireshghallah
View a PDF of the paper titled Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs, by Renfei Zhang and Manasa Kaniselvan and Rylan Schaeffer and Niloofar Mireshghallah
View PDF HTML (experimental)
Abstract:Reinforcement learning (RL) is often credited with improving reasoning at the expense of factual knowledge. We instead find that reasoning models outperform their instruction-tuned versions on factual recall by accessing existing parametric knowledge more effectively. Across five model families, structured prompting, which explicitly guides models through hierarchical traversal, recovers most of this gap, suggesting that much of the missing knowledge is latent rather than absent. Controlled RL experiments further support this: training on unseen, non-extractable facts improves recall of held-out, frequent but previously inaccessible facts, ruling out simple data exposure. Decomposing the training objective further attributes this gain to iterated on-policy exploration. The same mechanism appears behaviorally and internally: the reasoning advantage grows with retrieval depth, while layerwise analysis finds similar factual representations but divergent query representations. Distilled models, in contrast, often imitate self-correction without acquiring the exploration needed for navigation. Together, these findings suggest that improving factual recall in LLMs depends not only on expanding what models know but also on teaching them to navigate it, motivating future post-training methods that optimize traversal.
Comments: `
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2511.05933 [cs.CL]
  (or arXiv:2511.05933v3 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2511.05933
arXiv-issued DOI via DataCite

Submission history

From: Renfei Zhang [view email]
[v1] Sat, 8 Nov 2025 08:56:29 UTC (31,944 KB)
[v2] Wed, 24 Jun 2026 14:20:52 UTC (5,838 KB)
[v3] Fri, 4 Sep 2026 21:27:00 UTC (7,508 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs, by Renfei Zhang and Manasa Kaniselvan and Rylan Schaeffer and Niloofar Mireshghallah
  • View PDF
  • HTML (experimental)
  • TeX Source
view license

Current browse context:

cs.CL
< prev   |   next >
new | recent | 2025-11
Change to browse by:
cs
cs.AI

References & Citations

  • NASA ADS
  • Google Scholar
  • Semantic Scholar
Loading...

Bookmark

BibSonomy Reddit

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
We gratefully acknowledge support from our major funders, member institutions, , and all contributors.
About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab)
Major funding support from
Simons Foundation Simons Foundation International Schmidt Sciences