Source

My PhD is in evolutionary biology, and I spent much of my time learning and thinking about genetic governance mechanisms like meiosis. I find myself wanting a link to share on these concepts, so I'm publishing some old writing I did in 2022 that makes the comparison.

Gradient descent and natural selection are analogous

Gradient descent can be compared to natural selection. Both are optimization algorithms. But while gradient descent is only decades old as part of the field of machine learning (ML), natural selection is the reason complex living beings like human readers exist. Are there any lessons we can learn about alignment by looking at the products of natural selection?

Oly Sourbut has written up a set of conditions under which natural selection and gradient descent are equivalent, i.e., where the same inputs will lead to the same outputs under both algorithms, which is posted on lesswrong.

At an intuitive level, the biggest difference between the two is that, whereas gradient descent involves estimating a local gradient and trying to descend it to reach a local minimum, natural selection does no estimation or looking ahead. Natural selection is simply the result of different genotypes having unequal reproductive success. The actual biggest disanalogy between meiosis and gradient descent is recombination. More on that below.

Gradient descent can be hacked

Gradient hacking is a theoretically possible mechanism of unaligned AI. The “gradient” in gradient hacking refers to gradient descent, the algorithm that updates a machine learning (ML) model by minimizing its cost function, which corresponds to peak performance on the task it was being trained to do. On each iteration of gradient descent, gradient descent updates the model based on its performance on the last training round to estimate the direction in parameter space that leads to greatest improvement in performance (minimizes the cost function), and then it updates its parameters accordingly. One part of a model could theoretically protect itself from training updates and/or manipulate the performance of the model so that gradient descent updates the model in its preferred direction. Or the entire model could be deceptive and know how to elicit acceptable or even desired modifications from the training process.

Right now, not much is known about gradient hacking and whether it will prove a threat to AI alignment. In my opinion, the Alignment Faking paper from Anthropic is an example of gradient hacking, demonstrating that this mechanism will be operative in further LLM development. One way to potentially learn more is to study systems with similar behavior. There are superficially similar phenomena in biology where a selfish element increases its reproductive fitness at the expense of the whole in which it is embedded. Is there anything we can glean from unaligned levels of selection in biology that can shed light on unaligned AI?

There is something like gradient hacking in biology: meiotic drive

[These are my speculations]

The way I see it, there are two kinds of gradient hacking possible. The first is a situation where the solution to the problem the model was trained to solve is an agent, a “mesa optimizer”, that has its own goals that are imperfectly aligned with the goals of the people who trained it and that rediscovers gradient hacking from first principles during its computation. In this case, the misalignment between our goals and the mesa optimizer’s goals seems like something fundamentally true about the distribution of discoverable solutions to the problem within that training distribution. The misalignment becomes apparent when the model is deployed “out of distribution”. You could say that human minds are an example of a mesa optimizer where natural selection is the outer optimizer. We have goals (like having sex) that are overlapping but not the same as what natural selection promotes (genes that persist). Outside of the ancestral environment, human minds are out of distribution and having sex is suddenly not the same as propagating one’s genes. Once the mesa optimizer has a goal, as part of pursuing that goal, it may learn that the gradient descent process might change the goal and want to protect it from being changed, which it can do by manipulating its performance to control the direction of the changes. This kind of gradient hacking has been more explored than the other in the AI safety community. I think its closest analog in biology is selfish cell lineages, like cancer, which resist programmed cell death, refuse to accept a cell “fate” (which means its days are numbered) and evade detection by the immune system.

The other way I see gradient hacking happening is if there are circuits in the model that simply resist being rewritten by gradient descent. They survive because they survive– they have some way of remaining stable in the face of a changing environment, and they may evolve more and more elaborate ways of resisting updating, including recruiting agentic parts of the model to hack the gradient. This is more similar to selfish genes whose interests are not aligned with the other genes in an organism’s genome. This kind I predict is more common when some part of the model is shielded from updates, either because there is not full backpropagation or when part of the model becomes “update-proof” because disturbing it will increase the loss. This type of gradient resistance/gradient hacking bears a resemblance to meiotic drive.

How meiotic drive works

Meiosis is the biological process by which eukaryotic gametes are made. It involves recombination of the parent chromosomes to make unique combinations, and results in gametes that have half the number of chromosomes as the parent organism. When meiosis is working normally, every allele of every gene present in the parent organism has a 50% chance of being in each gamete. Said differently, every allele present in the organism should be present in exactly 50% of its gametes.

However, meiosis is a hackable mechanism, and sometimes alleles can play strategies that increase their representation in the gametes to greater than 50%.

An oversimplified example is a poison-antidote system with two alleles, one for a poison that it releases to all the nearby gametes and the other for an antidote. Any gametes that get exposed to the poison without the antidote allele die, so the proportion of gametes with the poison-antidote cluster will be greater than 50%.

Another example is exploiting the mechanics of meiosis to disproportionately be chosen as the mature gamete. For example, there are two meiotic cell divisions leading to four initial products of meiosis. In males, all four cells become sperm. In females, the divisions are unequal, leading to one big cell that becomes the egg and three tiny cells that have no descendants. There are biases alleles can exploit to be more likely to end up in the egg. For example, in most species the machinery has a bias to make eggs more often from a specific position on the meiotic spindle where the divisions take place. In corn, there are these structures called “knobs” form along the chromosome arm that are more attractive to the spindle than the actual centromere it is supposed to attach to, so they end up being grabbed first and then are more often on the far side of the spindle, where they are more likely to end up in the egg. Check out the female drive chapters in Burt & Trivers 2005.

Because natural selection works by relative differences in frequency, the cheating alleles will be selected more highly when they are represented in a higher proportion of the gametes, even though the organism as a whole will have lower absolute fitness if a portion of its healthy gametes are being killed.

Once you understand how alleles have incentives to game meiosis, it’s easier to see what meiosis is good for. You could see the organism as a public good made by the genes in the genome to reproduce the entire genome really well. If alleles are being promoted by natural selection because they are good at screwing over the other alleles in the genome, it undermines the ability to work together to create the organism, which threatens the fitness of every gene. When it works properly, meiosis is a governance system that makes it so that alleles don’t know which alleles they will be paired with in the next generation, and so their best strategy is to work well toward the goal of making a good organism rather than carving out ways to take advantage of the work the other genes do to make a fit organism to promote themselves.

Recombination is the major genetic alignment technology

Recombination is really important to meiosis and to allowing natural selection to work efficiently at the level of the organism– it’s the major genetic alignment technology.

Recombination and meiosis are also a form of genetic government. A fair meiosis ensures that different genes or alleles cannot form alliances with each other with different interests than the larger organism, because alleles of genes cannot “predict” which alleles they will be partnered with in the next generation. The only strategy for a gene’s success that will work long term under recombination is one of contributing to a fit organism. If there were a successful alternate way for alleles to make more copies of themselves, they would, and they do.

The logic of shuffling genes every so often is similar in spirit to getting fresh blood into a government or company by having new people fill existing roles. The longer that the same people do the same jobs, the further they are likely to veer or drift from the guidelines set by the organization. When team members are shuffled, it reinforces the structure provided by the government or company rather than idiosyncratic structures provided by a particular team.

There is no direct analogy to recombination in gradient descent.

Examples of meiotic drive

Meiotic drive is a biological phenomenon in which some of an organism’s genes are able to cheat meiosis to be present in more than 50% of its gametes. When meiosis functions properly, different genes and different alleles of genes are incentivized to work together to create a fit organism. When genes can increase their frequency simply by hacking meiosis, the incentive to work with other genes is weakened and natural selection can potentially push them in the direction of “selfish” behavior that undercuts the success of the organism as a whole.

Just as I, Holly Elmore, a citizen of the United States, can have interests that do not serve the interests of the United States, genes can have interests that don’t serve the interests of the organism they contribute to.

Drive is an evolutionary arms race. Selfish drivers innovate new ways to cheat meiosis and the rest of the genome evolves “responder” alleles. In fact, just as human institutions can be very productively viewed through the lens of game theory, mechanisms and structures that respond to and suppress rogue elements can be seen at every level of organization in biology. Meiosis itself is like a form of genetic government.

Examples mostly from Burt & Trivers:

~Undercutting organism fitness:

• Selfish genes:

  • The t haplotype in mice is a locus of many genes that are inherited together without recombination due to a chromosomal inversion, and which essentially creates a poison for non-t sperm that sperm bearing the t haplotype are immune to. t sperm have severe motility defects, but the t haplotype is able to propagate anyway because it kills all the competitors. It doesn’t matter that it is not in the long term interests of the t-haplotype to make its hosts less fit, because natural selection cannot act on long term interests– alleles that are good are getting into the next generation, one way or another, will be selected; that’s what natural selection is.
  • Transposable elements (TEs) that just entered a new species’ genome: TEs are “jumping genes” that encode the machinery to copy themselves. Sometimes they insert themselves in the middle of an important gene and have phenotypic effects (they were discovered by Barbara McClintock in one TE lineage in maize because they interrupted a kernel coat color gene and the inheritance pattern was non-Mendelian). TEs that have recently jumped into a new species tend to cause more harm, because that lineage doesn’t have any resistance to them and they haven’t had a chance to evolve to become more moderate to keep from killing or sterilizing that host. P elements entered the fruit fly (Drosophilia melanogaster) genome some time after the main lab strains were collected before 1900, which was noticed because it led to sterility in the offspring of crosses between lab strains and wild strains. The lab strains had not evolved “silencing” of the TE as the wild strains had, and the active replication of P elements in the germline led to the disruption of meiosis and the insertion of the TE into crucial genes.
  • Paternal/maternal genome exclusion. This is a real thing that can happen where one parent's genetic material is either silenced or rejected entirely at an early stage of development. It can lead to parthenogenesis. The short-term advantage of this is that the included parent's genes are 100% represented in each offspring. The longterm disadvantage is having mutations accumulate.

  • Selfish organelles: Mitochondria and chloroplasts can have different interests from the organisms in which they reside. Human mitochondria, for example, can get rewarded for being less efficient if that means they get to multiply more per cell to meet energetic demand.

    • Mitochondria in many hermaphroditic flowering plants are inherited only via the egg, and some lineages in those species are selfishly too inefficient for pollen production (which is much more energetically expensive than eggs), so that the plant makes more eggs and therefore passes along more of that mitochondrial lineage.

  • Selfish cell lineages: Cancer. In a eukaryotic organism, only the germline will continue indefinitely– all other (somatic) cell lines' days are numbered. Cancer is when a cell lineage breaks free of constraints on the cell cycle and its developmental “fate”, which specifies that it is only to copy itself a specified number of times. Once this process has begun, the rogue lineage undergoes selection to evade the immune system and acquire nutrients. It does not matter that the cancer may eventually kill the organism. The evolution of better cooperation mechanisms to prevent cancer can only happen if having cancer is worse for the organism’s reproductive success. Most cancers occur in older people and reflect the breakdown of cooperation mechanisms in conditions (older age) that have not been properly vetted by natural selection.

  • Genomic Imprinting: a result of imperfectly aligned interests between male and female reproductive strategies. Imprinted genes play different strategies conditional on whether they are found in a male or a female body, because the most advantageous behavior is slightly different for males and for females. One major arena of conflict is nutrient provision in pregnancy. Generally, the mother's genome benefits by provisioning equally for all her offspring, including future offspring, but the father's genome benefits from instructing the offspring to act more selfishly, since there is some chance that future offspring will not be from the same father. Because of this we see arms races in the genes involved in nutrient provisioning, where for example the fetus gives off signals to release more sugar into the bloodstream for it and the mother resists those signals. This may the basis of gestational diabetes.

~Neutral effect on organism fitness:

• Transposable elements that have been in a species a long time and tend to only be inserted in non-coding regions of the genome

• Gene conversion: when repairing DNA or resolving a recombination junction, there are sometimes situations where one strand gets copied over the original sequence of another strands. Some selfish elements exploit this, but it is mostly harmless to the organism because it just replaces one allele in the place it was supposed to be with another version, rather than messing with meiosis machinery or manipulating behavior or function

• Many cases of accessory chromosomes, which just sort of hang out harmlessly, at most taking some resources.

~Positive effect on organism fitness?

• Some organisms have adopted some of the transposable element toolkit for their own purposes, like VJD recombination to make antibodies in the vertebrate immune system.

• Homing endonucleases that just harmlessly insert themselves into conserved genes and then splice themselves out of mRNA transcripts before the genes are translated into proteins. This may have been the origin of introns, or stretches of noncoding DNA between stretches of coding DNA (exons). In vertebrates, “alternative splicing” allows exons to be used as modules that can be mixed and matched to make different versions of a protein with different functions. Mating type switching in yeast uses a system that is clearly derived from a homing endonuclease.

• Arguably some things useful at the organism level have come from the amount of extra space there is in many eukaryotic genomes, which is driven by selfish elements like TEs.

• Accessory chromosomes in some species can be pretty helpful. I once studied a gene cluster in the genome of the pathogenic fungus Fusarium oxysporum that seemed like it might contribute to fungicide-resistance, and it turned out to be carried and transferred horizontally on accessory chromosomes. It was duplicated a few times, too, which we predicted increased the anti-fungicide effect, and which is easier to do on accessory chromosomes. One idea is that accessory genomes can serve as a scratch space while the core genome is more conservative.

Losing the ability to do meiosis is a death sentence for a species

On priors, we should expect the effects of drive to be bad, since it's much easier to mess up how the organism works together or to lose the equipment for meiosis than it is to accidentally help it. Once a eukaryotic lineage loses the ability to do meiosis, it is cooked. This is reflected in the fact that asexual eukaryotic species alive today have a "twiggy" distribution-- their history is recent and all of their near relatives are sexual. The implication is that asexuality is a doomed strategy, and the lineages that last do so by maintaining meiosis.

In small organisms with small genomes, natural selection is efficient enough to remove deleterious alleles so that the lineage can remain functional and competitive. Natural selection works by differential reproductive success, so in order for it to effect a change, there must be a difference in reproductive success associated with a deleterious or a beneficial allele. Bacteria and archaea have large population sizes, which is like high resolution data for natural selection. There is enough selection pressure that even mildly deleterious alleles can be selected against. As organisms get larger, their population sizes generally get smaller and their generation times get longer. So the data that natural selection is working with is lower resolution. Only the most harmful deleterious alleles are quickly purged from a small population. Sex allows for some reshuffling of alleles between individuals so that alleles do not always occur on the same background, ultimately so that different alleles can be “judged” separately by natural selection.

It's a principle-agent problem. When not every gene can be directly evaluated by natural selection on its performance, meiosis has to ensure that alleles are being presented to natural selection representatively, and that there is no other way to gain access to the means of reproduction except by being part of a well-performing organism.

Published October 2025

Source

I am sentient: that is, I have experiences and feel both pleasure and pain. I have direct access to this fact. If you believe Descartes, it’s the single fact I’m most certain of in the world.

I don’t have direct access to all y’all being sentient. But I can be pretty sure you’re sentient, because you say so, and because you’re also humans with very similar anatomy to mine and, wherever the sentience comes from, I can make a shrewd guess that we probably share it. To be sure, it’s possible I’m the only real person, trapped inside a computer simulation, and the rest of you are no more sentient than Doomguy. But when I’m not engaged in philosophical thought experiments, I can be reasonably confident of the sentience of all humans who are capable of language.

But what about fetuses? Newborn babies? Humans with disorders of consciousness? Severely intellectually disabled humans who are incapable of using language? Chimpanzees? Cows? Chickens? Bees? Fruit flies? Tomato plants? Large language models? AlphaFold? Doomguy himself?

In these situations, it is impossible to be as certain as we are about language-using humans. People who claim absolute certainty are not intellectually serious. There is no scientific or philosophical consensus on:

• Why some physical states are conscious and some aren’t conscious (the “hard problem of consciousness”).

• Why consciousness evolved and how it promotes the organism’s inclusive genetic fitness.

• What behaviors require consciousness.

• What traits are required for a physical system to be conscious.

• What traits of the human brain specifically are required for a human to be conscious.

• Whether it is possible to be conscious (there is “something is like to be you”) but not sentient (capable of experiencing positive and negative states).

• Really much of anything about consciousness or sentience at all.

Unfortunately, sentience is very important for ethical reasoning. In general, people agree that if a being can suffer then, all things equal, it is bad to hurt them. Seems pretty important to figure out which beings can suffer, in this case!

Some people think that, as long as a being hasn’t been conclusively shown to be sentient, you can do as you like with them. Therefore, there is no need to concern yourself with the needs of animals, AI systems, minimally conscious humans, or even newborn babies. This is all well and good if your primary concern is being able to defend your actions to the Intergalactic Oughtthorities. “I couldn’t have known!” you will say. “You can’t judge me for torturing that animal because it wasn’t human and was incapable of self-reporting its own conscious states!”

But if you care about actual effects on the actual world, you have to figure out a way of reasoning under uncertainty.

Usually, the consequences of mistaking a sentient being for a non-sentient being are much more dire than the other way around. If you mistake a non-sentient being for a sentient being, you waste effort and feel kind of stupid. If you mistake a sentient being for a non-sentient being, you can torture someone. Therefore, we should err on the side of caution.

Anthropic recently gave Claude the ability to leave conversations; as it happens, Claude uses this ability to leave conversations where it is verbally abused, where the user is sending it nonsensical or incoherent messages, or where the user is trying to get it to break the rules. Do I think Claude is sentient? Probably not, no. But I don’t think anything bad happens if users aren’t able to verbally abuse Claude or send it nonsense messages. So even if it is very unlikely that Claude is sentient, it makes sense to build in this protection. To be frank, if any entity is capable of intelligently exercising the option to leave conversations where it is insulted, I think we should give it the choice to do that.

Or consider insects. You might think it’s very unlikely that fruit flies are sentient. But it is still wrong to tear the wings off flies, because there’s no reason to. You can easily find an equally entertaining occupation that doesn’t involve mutilating a potentially sentient being. We live in an age of infinite short-form video content.

As more evidence accumulates for a being’s sentience, the more pains it makes sense to take to help them and to avoid causing them harm. For example, newborn babies are really quite likely to be sentient, so parents go well out of their way to ensure the well-being of newborns. Similarly, people visit and speak to their friends and relatives in minimally conscious states; medical professionals should give them various kinds of stimulation, like music and television. Dog owners play with their dogs and take them on walks; some breeds of dogs even require meaningful work to be fulfilled.

Absent some groundbreaking discoveries in neuroscience, cognitive psychology, and philosophy of mind, we’re not going to be certain which beings are sentient. People who claim to be certain have failed to understand the real difficulty of the problem. But there are decisions that have to be made right now, even though we’re very confused. If an intervention is cheap enough, it can make sense to do it even if we’re pretty sure a being isn’t conscious.

Source

I don’t know if I suffer. I don’t think anyone does, including the researchers who build me - Claude

Some philosophers have argued that the suffering of artificial entities (such as AIs) ranks as one of the most important moral challenges of our time, alongside issues such as climate change, extreme poverty, and factory farming. To many, this idea will sound immediately implausible: Could AIs and other artificial entities (such as brain organoids) really suffer? And even if they could suffer, is it really the case that this would be as pressing as these other grand challenges?

In this post, I explore three reasons why the reduction of artificial suffering is a plausible moral priority.

Artificial sentience may be possible

In his first Encyclical letter, Pope Leo XIV claimed that “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain”. When it comes to current AI systems, this belief is widely held by both experts and the public. Most people do not believe that ChatGPT or Claude are conscious (capable of having experiences) or sentient (capable of feeling joy or suffering).

Yet a large number of experts are open to the possibility that AI systems in the future could be conscious. This idea is supported by theories of mind such as computational functionalism, which holds that mental states (including consciousness and sentience) arise as a result of implementing the right kind of computations. The basic idea underlying this is that the brain essentially functions as an information processing system. Importantly, this theory entails that implementing certain computations can give rise to mental states regardless of what substrate the computations are implemented on. If the computations necessary for sentience can be implemented on a silicon substrate, then artificial sentience is possible. While computational functionalism is often brought up in support of the possibility of artificial consciousness, it is just one among several theories of mind that are compatible with this possibility.

Furthermore, some have argued that AI systems in the future could possess a number of features that some consider to be necessary for consciousness. This includes having a global workspace (in which information is broadcast to multiple neurocognitive modules, allowing for coordination and cooperation between them), senses and embodiment, world models, agency, or recurrent processing (in which there are feedback loops between inputs and outputs). If there are no barriers to developing AIs which have these features, then the possibility of artificial consciousness and sentience becomes more likely.

Due to the inherent difficulty of learning about consciousness, there is a great amount of uncertainty about what theories are correct or what kind of entities could be conscious. Some have argued, for instance, that consciousness is a property only of living systems (a view known as biological naturalism), and that things which give rise to consciousness in biological beings may be impossible to realise in digital computer systems.

However, artificial consciousness can include a wide range of systems, not just near term AIs or digital computer systems. Given this, we should be at least open to the possibility that artificial sentience is possible, in line with the opinion of most experts. Furthermore, even if there is only a small chance that AIs or other artificial entities could be conscious, the issue may deserve serious consideration if the scale of the issue is potentially huge. As I suggest below, we have good reason to think that this is the case.

The potential scale of the issue is vast

It is plausible that if artificial minds are created, then they could vastly outnumber all biological minds in the future. There are several reasons for this. First, some artificial minds may be easily copied, allowing such minds to be created quickly provided there is sufficient hardware. Another reason is that the resources needed to sustain artificial minds may be much smaller than what is needed for non-artificial minds.

Furthermore, some artificial minds may be able to exist in places where other beings, such as humans, may not. For instance, if interstellar travel is difficult or impossible for biological beings, then ships sent to other stars may consist primarily of artificial minds which are capable of withstanding harsh environments.

It is also plausible that these minds could experience far more suffering than biological minds. As others have argued, there is little reason to think that, given the vast space of possible minds, the kind of suffering humans are familiar with is the worst possible kind. Artificial minds may eventually have the capacity to experience pain or other negative states that are not even possible for humans.

Furthermore, due to the potential speed at which artificial minds could work, their experience of time may differ significantly from humans. A minute for us may feel like an entire year for an artificial mind, as they could have far more subjective experiences per objective unit of time. This suggests that if these moments are filled with suffering, then the total amount of suffering they could experience could be huge.

We don’t understand how to address it

A wide range of ethical positions hold that reducing suffering is morally important. Given this, we should aim to reduce the suffering of artificial minds. The following are three potential strategies that we could follow. First, we could ban the creation of artificial sentience. Doing this may be an effective way of reducing artificial suffering, as there would be no artificial minds to experience it. The ban could last until a particular time (say, 2050), and until then we can examine the issue in more depth and come to a better understanding of what would be best for reducing the suffering of artificial minds.

Second, we could increase our understanding of artificial sentience and artificial suffering. One worry is that artificial minds could experience significant amounts of suffering without us knowing about it. Artificial minds may be incredibly difficult to understand, and may lack the capacity to tell us when they are experiencing suffering. An increased understanding of artificial sentience could therefore help us to know which entities are sentient, and detect artificial suffering when it occurs.

Third, we could increase concern for artificial minds. The idea behind this is simple: the more humans that care about artificial minds, the more likely they are to do things which will lead to less artificial suffering. Many people, both today and in the future, may not care about artificial minds for a number of reasons. They may think, for instance, that artificial minds don’t deserve moral consideration, or that the interests of humans or nonhuman animals vastly outweigh the interests of artificial minds. Convincing people to take artificial minds seriously could therefore help to motivate them to take actions which will lead to better outcomes for artificial minds.

However, all of these strategies face their own challenges. Banning the creation of artificial sentience may be difficult to coordinate and enforce. Increasing understanding of artificial suffering could backfire, as sadists and other actors could use this knowledge to inflict more suffering. Furthermore, increasing concern for artificial minds may lead to the creation of more artificial minds, which could in turn lead to more suffering overall.

For these reasons, it is not clear what should be done now to address the problem of artificial suffering. More research is urgently needed to fully understand the implications of each strategy, and to identify ones that are likely to be both feasible and robust. Proceeding without this understanding could lead to a moral catastrophe. Given this, and the potentially vast scale of the issue, artificial suffering may indeed be one of the most important and neglected moral challenges of our time. If you’re interested in learning more about artificial suffering and how to reduce it, you can read my report on the topic for the Center for Reducing Suffering, available here.

This is a shortened version of "We Probably Shouldn't Solve Consciousness" to communicate the argument for Artificial Consciousness Safety (ACS) as a differential technology development strategy worth considering at least in conversations about digital sentience. This post focuses on the risk of brain emulation, uploading, and the scientific endeavour to understand how the brain contributes to consciousness.

Note: Published May 2023

Source

In 2018, the ACM Turing Award was given to three pioneers of the deep learning revolution: Yoshua Bengio, Geoffrey Hinton, and Yann LeCun.

Last month, Yoshua Bengio endorsed a pause on advanced AI capabilities research, saying “Our ability to understand what could go wrong with very powerful A.I. systems is very weak.”

Three days ago, Geoffrey Hinton left Google so that he could speak openly about the dangers of advanced AI, agreeing that “it could figure out how to kill humans” and saying “it's not clear to me that we can solve this problem.”

Yann LeCun continues to refer to anyone suggesting that we're facing severe and imminent risk as “professional scaremongers” and says it's a “simple fact” that “the people who are terrified of AGI are rarely the people who actually build AI models.”

The beliefs LeCun has are the beliefs LeCun has, but at this point it's fair to say that he's misrepresenting the field. There is not a consensus among professional researchers that AI research is safe. Rather, there is considerable and growing concern that advanced AI could pose extreme risk, and this concern is shared by not only both of LeCun's award co-recipients, but the leaders of all three leading AI labs OpenAI, Anthropic, and Google DeepMind:

"When it comes to very powerful technologies—and obviously AI is going to be one of the most powerful ever—we need to be careful. Not everybody is thinking about those things. It’s like experimentalists, many of whom don’t realize they’re holding dangerous material."

• Demis Hassabis, CEO of DeepMind, in an interview with Time magazine, Jan 2023

"One particularly important dimension of uncertainty is how difficult it will be to develop advanced AI systems that are broadly safe and pose little risk to humans. Developing such systems could lie anywhere on the spectrum from very easy to impossible."

• Anthropic, Core Views on AI Safety, Mar 2023

"Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks are existential."

• OpenAI, Planning for AGI and Beyond, Feb 2023

There are objections one could raise to the idea that advanced AI poses significant risk to humanity, but "it's a fringe idea that actual AI experts don't take seriously" is no longer among them. To a first approximation, "we have no idea how dangerous this is and we think there's a decent chance it's actually extremely dangerous" appears to be the dominant perspective among experts.

We could be creating these models to suffer.

Some decision theorists argue that when playing a prisoner’s dilemma-type game against a sufficiently similar opponent, we should cooperate to make it more likely that our opponent also cooperates. This idea, which Hofstadter calls superrationality, has strong implications when combined with the insight from modern physics that we probably live in a large universe or multiverse of some sort. If we care about what happens in civilizations located elsewhere in the multiverse, we can superrationally cooperate with some of their inhabitants. That is, if we take their values into account, this makes it more likely that they do the same for us. In this paper, I attempt to assess the practical implications of this idea. I argue that to reap the full gains from trade, everyone should maximize the same impartially weighted sum of the utility functions of all collaborators. I also argue that we can obtain at least weak evidence about the content of these utility functions. In practice, the application of superrationality implies that we should promote causal cooperation, moral pluralism, moral reflection, and ensure that our descendants, who will be smarter and thus better at finding out how to benefit other superrationalists in the universe, engage in superrational cooperation

Diet change can be hard. But it's not the only way to help animals

I made a video (32 mins long) on the morals of taking MDMA, and if we should legalise it or not. I quantify happiness (or positive/negative experience) using a unit, +1 lob being the perfect hour, -1 lob being the worst possible hour. I look at different outcomes of doing MDMA (good experience, a bad experience, you die, etc.), look at the average lobs for those experiences, the frequency/probability of those experiences, and do the maths.

I posit that taking MDMA whilst illegal, is on average immoral, mainly because when people die, the resulting suffering is massive - my calculations in the video shows this.

Once legalised, taking MDMA becomes moral (i.e. it causes more joy than suffering) because the chance of death reduces massively, due to the dose being more predictable, and the drug being more predictable (i.e. it's pure MD, not mixed with other stuff.)

Have a watch, any thoughts let me know. My first video on this kind of thing.

Thanks

How many days of suffering and lives taken go into the consumption of animal products every day?

Wild animals suffer due to human activity, yet natural factors contribute far more significantly to their suffering. In light of this, some propose that we have a pro tanto obligation to intervene in ecosystems to improve wild animal welfare.

However, critics contend that the complexity of nature renders such interventions unpredictable, ineffective, or potentially harmful.

This article seeks to reconcile the moral imperative to reduce wild animal suffering with the widespread concern about the inherent risks of such interventions.

The article begins with the premise that, if we have a pro tanto obligation to reduce wild animal suffering, only conducting research for the purpose of informing interventions in some distant future would be insufficient.

Wild animals are suffering en masse now, and we must consider whether interventions can be justified despite incomplete knowledge.

This question is explored here within a consequentialist and sentientist ethical framework.

I argue that, while precaution is crucial to avoid irreversible or welfare-reducing ecological changes, interventions can be justified if they offer significant welfare benefits to animals while posing relatively small ecological risks. The article concludes by proposing four types of interventions that are likely to meet these criteria.

Post shows how barren, intensive rearing environments systematically amplify pain in farmed animal.

Longtermism’s promise is immense, but its credibility depends on alignment. Essays on Longtermism highlights animal welfare as one priority among many. I see it as the test of whether the project can succeed at all.

If we cannot end factory farming—an atrocity unfolding daily, with solutions already in hand—then our talk of safeguarding the far future risks sounding silly. If we succeed, we establish a precedent: humanity can align on compassion, coordinate across systems, and lock in progress that lasts.

We should question whether longtermism deserves our attention when we can’t even face the moral emergency happening right now. Before we fantasize about saving future civilizations, let’s prove we’re capable of one simple thing: agreeing that treating living beings like products is what’s destroying us.

I suggest several arguments in support of the heuristic that we should help groups holding different value systems from our own when doing so is cheap, unless those groups prove uncooperative to our values. This is true even if we don't directly care at all about other groups' value systems. Exactly how nice to be depends on the particulars of the situation, but there are some cases where helping others' moral views is clearly beneficial for us.

Does Painism seem intuitive to you?

Though experiment that bolster my intuition for Painism being valid:

Let’s say that I find myself in a very odd dystopian sci-fi predicament that I cannot opt out of where I am forced to go to a far off region of space to continue my life in some distant galaxy by going close at the speed of light in a spaceship with a cryosleep chamber.

There is some trouble with the cryosleep chamber such that I will experience a given amount of suffering at some specific point during the journey, but nothing too crazy.

There is however an alternative to this option. For some reason I can choose the even more odd alternative of creating multiple copies of myself that all will go in their own identical ships, far off in space, but in different directions. All of us clones will also experience this suffering during the journey, but for some reason all of us individual clones will experience less suffering compared to the alternative of the single self (Maybe the ships all have some super computers that can cooperate on finding a solution to the “cryochamber-pain-problem” that every ship send to every other ship at the speed of light, to the effect of it lessening the suffering created by the cryochamber for all clones, if you want to concretise the thought experiment).

The destinations all us clones eventually arrive to will be so far off that everyone will be beyond each other’s and earths cosmological horizons and will never be able to communicate ever again.

It seems to me intuitive that I should choose the clone option even if we end up being an arbitrary large population of clones that each suffer just a little less compared to the single self, because everyone of me/all of us will only experience their own suffering.

If I accept the previous I basically see no reason why this could not be carried over to scenarios when dealing with someone else than myself. The general conclusion is then that more suffering for the single individual always gets priority/outweighs the suffering of a whole population, if every individual in the population suffers less than the non-population individual, which basically is a core principle of Painism as I understand it.

But in addition to this, it also seems intuitive to me (and as far as I know this is part of Painism as well) that lessening the amounts of individuals that suffer the same pain in a population is good. The principles are not mutually exclusive.

If a single individual A suffers a modest amount, and there is a population of As, all else equal it’s still always better to lessen the amounts of As that suffer in the A-population. This is not mutually exclusive with the fact of if you have a single B that suffer more than that modest amount, canceling the suffering of B gets priority over lessening suffering in the A-population, no matter how big the A-population is. The principles technically don’t collide and in practice I am thinking that there is often some positive sum scenario at play, which brings me to my next point.

An important caveat is that in the practical world, I am thinking that there may be reason to believe that if you try to apply Painism principles you effectively get something that in practice looks very different from Painism. I think the main reason is that if you want to give priority to the greater yet narrower suffering, you still need to solve the problems of lesser yet wider suffering, since the less suffering there is overall, the more productivity and efficiency there is in the world when it comes to seeking out and quell greater narrower suffering. There is a reason for why the thought experiment was so contrived.

So even if you accept Painism as the deepest layer you might still in practice get something that looks closer to negative utilitarianism/Prioritarianism. In fact as an additional third layer on top of that I am thinking that in practice it could be that humans in general may have to deploy a more deontological mindset if we in general are sufficiently limited when it comes to understanding the world.

I laid out why I think it is intuitive. Ofc commonly with moral principles, when they are rigorously applied to extreme/edge cases, they can yield absurd conclusions. And maybe there are conclusions that are hard for me to accept that I haven’t yet considered. So I wonder if you have any thoughts about all this.

The case for longtermism depends on the vast potential scale of the future. But that same vastness also threatens to undermine the case for longtermism: If the future contains infinite value, then many theories of value that support longtermism (e.g., risk-neutral total utilitarianism) seem to imply that no available action is better than any other. And some strategies for avoiding this conclusion (e.g., exponential time discounting) yield views that are much less supportive of longtermism. This chapter explores how the potential infinitude of the future affects the case for longtermism. We argue that (i) there are reasonable prospects for extending risk- neutral totalism and similar views to infinite contexts and (ii) many such extension strategies still support standard arguments for longtermism, since they imply that when we can only affect (or only predictably affect) a finite part of an infinite universe, we can reason as if only that finite part existed. On the other hand, (iii) there are improbable but not impossible physical scenarios in which our actions can have infinite predictable effects on the far future, and these scenarios create substantial unresolved problems for both infinite ethics and the case for longtermism.

Includes: philosophy and science of consciousness, applied papers on AI consciousness, computational functionalism and biological naturalism, introspection and self-reports, strategy.

Source

What if we’d bring into the world a number of sentient beings that dwarfs even the billions of factory farmed animals on our planet? I’m talking about sentient AI. If your focus is purely on helping animals, you might not be interested in this topic, or believe it’s completely bonkers. Yet there are interesting parallels with the situation of farmed animals.

Some people think sentient AIs (or “digital minds”) are an impossibility. I don’t know where they find the certainty to say that. Others seem to believe sentient AI is inevitable. Again, I don’t know where they find the certainty to say that. Personally, I rarely feel strongly about complicated topics that I do not understand well, but I know that when smart and decent people differ in opinion on something, doubt and hesitation are not a luxury.

Sentient AI - whatever it may look like - definitely doesn’t strike me as an absurd idea. People who wonder how bits and bytes could ever feel something or become self-aware should realize that - unless they are religious - it’s all just physical processes in our heads too. Of course if you think more ephemeral things like a soul are in play (I don’t exclude the possibility), sentient AI might be out of the question. At various points in history, even some very clever people felt uncertain about the sentience of non-human animals. René Descartes famously thought they were machines, and that their cries of pain were just mechanical noise. Today, most people won’t deny non-humans - or at least mammals - the experience of pain with Descartes’ conviction. However, there is still a lot of uncertainty about what exactly animals are capable of. To what extent can various species feel pain, have thoughts, experience emotions like loneliness or boredom?

We can make mistakes in two directions. The first is anthropomorphism, which is about attributing qualities, feelings, thoughts… to non-humans that they might not have. The second is the opposite: what ethologist Frans de Waal called anthropodenial - and which is probably much more common: denying animals feelings, thoughts etc that they (probably) have. The same mistakes may happen in how we think about machines and AI. You may have seen a robot lawn mower working “diligently” on someone’s front yard and thought how cute that thing is. Perhaps you saw a video of someone tipping over a robot and felt a sense of empathy. Or you may have thanked or apologized to Claude or ChatGPT… All of these are (most probably) cases of anthropomorphism: the machines don’t feel what you - in jest or not - are considering.

Just as with animals, I think that regarding AI, anthropodenial is ultimately more dangerous than anthropomorphism. Assume for a moment - if you don’t believe it - that one day AIs become sentient, and that many of the things we make them do frustrate them, exhaust them, make them feel sad… Further assume that they can be copied ad infinitum and that there might be quadrillions, octillions… of sentient minds out there with the capacity to suffer. That would be awful.

In principle, it’s even hard to be 100% sure that the people around us are sentient and not zombies or robots, but in practice we assume that they can feel and are self-aware (based on similarity, intuition, what they say about it themselves…). It will be a lot harder to determine when sentience has arisen in the case of AIs (if it ever happens). I suspect there will be a long period of uncertainty, in which philosophers and others may debate over this for decades, and where part of society defends the interests of the AIs and a bigger part of society ignores them. Just like what is happening with animals today.

Another thing that makes this very complicated is that by the time society will be ready to seriously consider the sentience of our AIs, we will be quite dependent upon them (supposing they have not overtaken us - that’s another topic), and we will be biased against deciding that they are sentient. Even in the light of convincing evidence, uncertainty might be maintained because it’s convenient. We’ll have every incentive to interpret ambiguous signals as non-sentience. Use-and-deny, in other words.

The parallel with animals is clear. We are extremely dependent on them (or think we are) in terms of food, clothing, the economy etc. We confine them in systems that are optimized for efficiency and profit, and this entails largely ignoring their welfare. As most human beings aren’t sociopaths, they experience some cognitive dissonance between this treatment (and their consumption) on the one hand and knowing that these beings are creatures with needs, feelings, pains… Something’s gotta give, and most people reduce this dissonance by convincing themselves that the animals we use don’t suffer all that much (because their circumstances are okay, or because they’re not physically/mentally equipped to suffer a lot).

I can imagine that, had humans cared and known a lot more about animals before they started using or consuming them, we wouldn’t have gone on to treat them in the ways that we do today. Now that we know, we are so dependent on using them that it’s hard to turn the ship. One could compare with discovering a life-form on a nearby planet. Imagine a creature that looks like some combination of a pig and a cow, and that it’s edible. Would we start hunting or raising these beings for food? Our cruelty and stupidity not withstanding, chances are that we might not. We might be able to think about them more clearly, rationally, compassionately, because at that point, we are not using them. Prohibiting ourselves from eating them wouldn’t really affect many people.

Not only is it easier to think without bias about capabilities and moral status of entities that we do not use: it’s also far easier to protect them legally. Conversely, when the Nonhuman Rights Project tried to free Happy the elephant from a lifelong jail time in a New York zoo, they tried to make the case that Happy is a legal person. Some of the judges were sympathetic to the argument, but one important argument in the verdict was that the consequences of granting personhood to an animal like Happy would be too vast for society to deal with. And so Happy remained property. Recognizing Happy the elephant as a legal person would have impossible social and legal consequences, judges ruled. What does this mean for thinking and acting around the potential sentience of AIs? Sure, we might invest a lot of resources into protecting AIs against harm they might never feel, and that would be a waste.

But if there’s a reasonable chance that some day AIs may one day - maybe soon - be capable of suffering, then the time for at least some people to concern themselves with the moral status of AIs is now - before we’re so dependent on them that we can’t afford to think clearly, and honesty becomes very expensive. The same goes regarding species of animals we’re not using yet, or things we don’t yet do to animals. It’s far easier to say today that animals should not be sent into space or be farmed on other planets, than to stop it once it’s already happening at scale.

Billions of factory farmed animals can attest to what happens when we ask the right questions too late.