Computer Science > Computation and Language
[Submitted on 1 Dec 2021 (v1), last revised 9 Dec 2021 (this version, v3)]
Title:A General Language Assistant as a Laboratory for Alignment
View PDF HTML (experimental)Abstract:Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.
Submission history
From: Jared Kaplan [view email][v1] Wed, 1 Dec 2021 22:24:34 UTC (3,014 KB)
[v2] Tue, 7 Dec 2021 20:47:42 UTC (3,014 KB)
[v3] Thu, 9 Dec 2021 21:40:22 UTC (3,031 KB)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.