Between 2019 and 2024 I did a PhD at Bristol with Laurence Aitchison. (Though this includes a year of mandatory classes and a year off in which I started Arb.)
I went in wanting to work on AI safety. True to form, I instead ended up with a grab-bag of fields: approximate Bayesian inference, Covid epidemiology, metascience, the methodology of the social sciences, scientific malpractice, inductive logic, algorithmic fairness and (of course) large language models. Some safety work in there if you squint.
But I published enough, so the resulting thesis is Methods Failing the Data, Data Failing the Methods. Broadly, it’s about epistemics. (Why can’t we learn that much from one study, or many studies?)
In Newell’s typology the piece “contradicts existing knowledge; thoroughly explores an area; provides empirical data; and produces a negative result”.
It looks like a success on the usual bad measures. But I didn’t go into it for poxy numbers or a mere job; I went in become a great scientist. This did not happen. But I did learn how to really read papers, how to write papers, how to present technical ideas clearly, and how to become stubborn and insensitive in the face of latent spaces. Academia is forever demystified for me. My aversion to mathematics has settled down into guarded neutrality. I am unafraid. This was probably worth it.
Undying thanks to Kristi Laurence Jan Jan Juan Tomáš Rian Matthijs Misha Daniel Dandy Dan Nandi Simson Mrinank Sören Kaveh Alexander Nic Charlie Maxime Samir Swapnil Miranda Peter Yarin Tyler. Sine qua non.
Posts about my PhD
Areas
I started my PhD just before Covid. In a strange turn, a bunch of computer scientists invited me to do a little bit of writing on their big Bayesian model of what policies worked against the bug. I had no epidemiology background. 12 months later, we'd produced a series of on important questions which weren't being treated with the proper uncertainty.Yes, this was the least neglected research topic in the world. Yes, it is strange that noobs could do this.
My original project was _Tensorised Probabilistic Programming_.Exact inference is intractable in many realistic latent variable models. Of the available approximations, variational inference is fast, but underestimates the variance; and Markov Chain Monte Carlo estimates the variance well but is far too slow in large models (Bishop 2006, Betancourt, 2020). For policy applications, where the variance must be accurate to prevent large irreversible decisions, we thus need new methods. Extending Aitchison's 2019 work on speeding up variational autoencoders, we seek to generalise the use of tensor products for approximate inference.The end goal is multi-sample inference for any such scheme, and we aim to implement this in a probabilistic programming language (PPL) to maximise usability and impact. There are already ‘tensorised‘ PPLs, in the weak sense of using tensor operations for arbitrary probabilistic programs with one inference scheme (e.g. Bingham et al., 2019, which uses stochastic variational inference for all runs). We seek a further abstraction for any inference scheme. In our project, ‘tensorised’ denotes the tensor products used to achieve the speedup.The original plan has passed to a colleague. Sorry Thomas.
. (For relative sceptics.) I also contributed a couple thousand words to the . I currently work with the , Charles University.At the first AI Safety Camp I worked with a team on inverse reinforcement learning, to probe the limits of such reward learning. Our work was reused by a at Deepmind.I've also about the likely overlap between work on current systems and future systems.
Over Christmas, instead of studying for quals I started all the failed replications in psychology I'd heard of. This ballooned into a list of hundreds, and was taken up by the volunteer org for permanent maintenance.
I took a lot of tangents, for instance into inductive logic, evolutionary game theory, algorithmic fairness.Before starting on probabilistic programming, I played with an odd alternative ML paradigm called __. This led to my first paper, a negative result.I also helped on with a sort of counsel of despair about algorithmic fairness.