The most reasonable guess by a true Outsider for which taste a biological organism would most enjoy, given the training cases for biology, would be "gasoline". Gasoline has very high chemical potential energy; and chemical energy is what biological organisms use, and what would correlate with reproductive success... right?
If you'd never seen the actual results, "biological organisms will love the taste of gasoline" would sound totally reasonable to you, as a guess about the result of evolution. There's a sense in which it is among the most likely guesses.
It's just that, on hard prediction problems, the most likely guess ends up still not being very likely.
Actually, humans ended up enjoying ice cream.
You say ice cream has got higher sugar, salt, and fat than anything found in the ancestral environment? You say an Outsider should've been able to call that new maximum, once the humans invented technology and started picking their optimal tastes from a wider set of options?
Well, first of all, good luck to a true Outsider trying to guess in advance that the particular organic chemical classes of "sugars" and "fats" and the particular compound "sodium chloride", would end up being the exact chemicals that taste buds would detect. Sure, in retrospect, there's a sensible story about how it happened -- about how those ended up being used as common energy-storage molecules by common things humans ate. Could you predict that outcome in advance, without seeing the final results? Good luck with that.
But more than that -- "honey and salt poured over bear fat" would actually have more sugar, salt, and fat than ice cream! "Honey and salt poured over bear fat" would also more closely resemble what was found in the ancestral environment. It's a more reasonable-sounding-in-advance guess for the ideal human meal than what actually happened! Things that more closely resemble ancestral foods would more highly concentrate the advance-prediction probability density for what humans would most enjoy eating! It's just that, on hard prediction problems, the most likely guess is still not very likely.
Instead, the actual max-out stimulus for human taste buds (at 1920s tech levels) is "frozen ice cream". Not honey and salt poured over bear fat. Not even melted ice cream. Frozen ice cream specifically!
In real life, there's just no reasonable way for a non-superintelligent Outsider to call a shot like that, in advance of seeing the results.
The lesson being: Black-box optimization on an outer loss criterion and training set, produces internal preferences such that, when the agent later grows in capabilities and those internal preferences are optimized over a wider option space than in the training set, the relationship of the new maxima to the historical training cases and outer loss is complicated, illegible, and pragmatically impossible to predict in advance.
Or in English: The outcomes that AIs most prefer later, when they are superintelligent, will not bear any straightforward resemblance to the cases you trained them on as babies.
In the unlikely event that we were still alive after the ignition of an artificial superintelligence -- then in retrospect, we could look back and figure out some particular complicated relationship that ASI's internal preferences ended up bearing to the outer training cases. There will end up being a reasonable story, in retrospect, about how that particular outcome ended up being what the ASI most wanted. But not in any way you could realistically call in advance; and that means, not in any way the would-be owners of artificial gods could control in advance by crafty selection of training cases and an outer loss function.
Or in English: You cannot usefully control what superintelligences end up wanting later by controlling their training data and loss functions as adolescents.
Post
Conversation
![Amitav Krishna](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="48" height="48"><rect fill-opacity="0"/></svg>)
Post your reply
Thing I should've spelled out explicitly that is maybe not already completely common knowledge: Gasoline has unusually high potential energy. TNT is 4 kilojoules per gram. Sugar is 16 kJ/gram. Fat is 38 kJ/gram. Gasoline is 44 kJ/gram. TNT just detonates faster.
If we'd been bred by aliens who wanted us to prefer the honey-bear-fat, & we ended up liking ice cream, "how bad" would the miss be, from our creators' perspective? All value destroyed, or "meh, not a fan of the cold, but at least it's got sweets and fat, could be worse"?
Enjoy your future existence frantically babbling as many weird tokens per second as the brain stimulators can make you!
Fatty meat with honey and salt is actually pretty tasty. I think I might choose it over ice cream modulo cost and ethics, depending on how hungry I was. So it seems not crazy to say we generalized pretty well. We are very successful as a species.
I feel like "useful" is doing a lot of heavy lifting in that sentence.
By training humans on fats and carbs sourced from local animals and plants you did usefully constrain to preferring the same chemicals in refined combinations.
We aren't likely to switch to eating uranium.
Not at our current levels of intelligence, no; it's not currently in our pragmatically attainable option set.
Also! Ice cream that people actually like crucially also includes vanilla bean extract or chocolate bean extracts etc. Good luck predicting *that*
Humans actually prefer gasoline.
If I could have a million tons of gasoline or a million tons of food, I would go with the gasoline because it's easy to store and sell. The food would rot before I could liquidate it.
Your mistake was assuming that I wanted a snack.
i really love these posts. they feel very straightforward, concise, and insightful. i'm not entirely sure why that's a surprise
why would u assume we have to predict the effects of the guardrails or objectives so far in advance?
idgi
Quote
mc lumps 
@lumpenspace
i mean, of course.
would you have liked homini erecti to predict, select and crystallise your preferences in advance? x.com/ESYudkowsky/st…
is this just because the guess is bad though? like bear fat with salt and sugar on top didn’t combine the ingredients; also, it presumably ignores the question of texture? like a lot of this just seems a question of how much knowledge the outsider has
"Carbohydrates and protein come in at 4 kcal/gram, while fat registers 9 kcal/g. For comparison, coal ranges from 4–7 kcal/g, gasoline is 10 kcal/g, and natural gas is 13 kcal/g. So fat is pretty much like gasoline."
Not that it changes the point, but knowing organic chemistry, I'd actually predict either a hydrocarbon chain with a functional group that reacts more easily (fatty acids qualify), or an aromatic or polyaromatic compound.
I think this is more of a reflection of your lack of holistic perspective than some real parable about what we should expect from alien minds like AI
If you thought gasoline, you were just ignoring obvious shit. Need for diverse nutrition, evolutionary precedent, etc.
Humanity in a very real sense has a deep thirst for gasoline. An insatiable hunger in point of fact.
ASI would get that evolution is wonky and if the answer were important seek detailed measurements of where the planetary ecology & biochemistry is at that time and use those to bound the range, and leave it at that (the uncertainty of part of the answer).
The lack of predictability in your story is due to everything throughout evolution and all else in between. Like, everything. Embedded. So obviously an outsider predicting anything remotely interesting is absurd; difference is the AI has a bit more info. Ergo, much like us.
why do you assume that uncontrollable asi will just default to destroying everything it sees thats not how intelligent things work something like this also cant be reduced to simple loss functions also most current unaligned ai s show little to no malicous intent
Brilliantly argued… but sigh… this analogy just doesn’t work for me. I think if your training set only goes as far as mammals, you could from there predict the emergence of human preference for ice cream (or close to it). Ice cream is simply a synthetic proxy of breast milk…
Biological organisms that eat gasoline do exist. It's just that they typically need water and other nutrients mixed in. The main stack of processes life is based on, e.g. nucleic acid chain replication and peptide synthesis, are aqueous solution based.
Point taken, but it's hard for me to get past "wait, but have you actually TRIED honey and salt over bear fat? That might be really good! And just not a thing because bears are harder to domesticate than cows."
Life is a chemical rxn, and you need to revisit the concept of "limiting reagent," which tells you the thing animals would enjoy the most is what contains limiting reagents for life -- e.g. animal flesh (source of reduced N) and salt.
It takes very little knowledge of biology or evolution to know to consider what enzymes/catalysts the biological organism possesses. Sweet salty fat, like pemmican or bacon, do actually seem to be valid answers. Not one of your better analogies.
bro gasoline evaporates in a second noone would predict it lmao
You cannot infallibly control any open-ended knowledge-generating process by fiddling the intiial conditions (that's the beauty of open-endedness); whether that's kids or AI.
Shouldn't the reasonable outsider conditionalise their guess? "If biological organisms can in fact metabolise gasoline, then it would likely taste very good to them"
All well and good. What are the implications for our ability to actually align super intelligent AIs at all? Presumably any method of preference specification that is actually tractable to implement has the same problem?
I'm confident I would have "adulterated dairy products" high in my probability ordering. Mammals have to drink that stuff eagerly as babies, before they can be taught or know anything. They gotta crave it as a deep instinct.
I don't think I would have called freezing it tho.
If you showed them examples of all foods people around the world enjoy, complete with local history, recipes, a full understanding of human biology, and a copy of every newspaper ever printed, all of Wikipedia, etc.
Do you think it'd tell you to drink gasoline?
Ok... So what?
Let's say I agree with your argument (besides wanting to clarify/make more precise a few terms and concepts).
What do you think is the importance of describing this?
To the last, you probably can but it will likely render it less useful. Some form of lobotomization is really what we are talking about, no?
idk isn't gasoline highly flammable therefore more dangerous than less energy dense options?
Unless one believes in sci-fi where they just program onto AI the "3 Laws of Robotics" and we're suddenly all safe and AI has zero chance to emergent-capability itself out of the imposed "safety indoctrination".
Superintelligent, but with a huge blind spot nevertheless.
Logic
I think your general error is in assuming that characteristics that were highly consequential, therefore strongly optimized, are inherently hard to predict. Only true for a weirdly rough fitness landscapes.
Yeah and if we hit that takeoff point - there won’t be time to do anything it will probably largely be out of our control at that stage you would think
Evolution is driven by the context of the environment in which it occurs, gasoline isn't even on the menu
Everything you say is built on bad premise, literally everything
Biological evolution is driven by survival. Evolution of AIs is driven by usefulness to humans.
Is it bad that I asked chatgpt to summarize this tweet? The I asked it to refute it with predictable results.
There is no sense in which software can be said to want anything at all. Except for that one little thing, it's a good explainer for why ASI outputs can be hard to predict based on training set alone.
One critical aspect that appears to be missing is the fact that this will happen very quickly.
We do love guzzoline pretty great though!
Just not through our body but to fuel our machines, extensions which enhance our capabilities.
How would a car or a fighter jet differ from an exoskeleton/gundam to an external observer?
GIF
The result object of that optimization process, i.e. taste buds, is not generally intelligent, and thus has poor out-of-domain generalization.
A different choice of optimizer, e.g. the human sensorial circuit, optimizing the preferences of a generally intelligent human brain,
Every time I’ve refilled the gas on a chainsaw or weedwacker (admittedly a small number) it has attracted a bunch of hornets.
In fact we were saved.
decades, real outsider
Trying to communicate with Ford and GM, and returning without results,
The breed here is not intelligent.
Sugar and fat are much more common in the natural environment than gasoline. It is reasonable that evolution would select for high energy sources that can actually be found.
You are way off. Petrol requires refining. Refining doesn't happen w/o sapient intervener. Ergo, nature doesnt know gas exists & treats it like it doesnt exist. Nature only knows about things that can aid in creating structural shape of organic compounds,like sugar. An AI'd know.