It's 1949, John von Neumann stood in front of a room at the University of Illinois and described a machine that could build a working copy of itself. No example of it existed yet, it was only theoretically possible. He called it a theory of automata. What he'd actually sketched was the first genome for a species that hadn't been born yet.

It's 1983. There's a floppy disk with a few kilobytes of code sitting dormant until someone runs the program that carries it. It then starts to copy itself into whatever it touches next, indifferent to whether anyone wanted it to exist. Fred Cohen called this a virus that year, borrowing the word from biology. Infection, incubation, spread.

It's 1994, Eugene Spafford sat down to ask whether computer viruses deserved to be called a form of artificial life. There's reproduction, heredity, mutation, and selection among variants. But there is no metabolism. A parasite, then. Almost alive. Alive enough that when Tom Ray dropped hand-written, self-replicating machine code into a simulated computer memory in 1991 and just watched, it began mutating on its own. Competing, parasitizing, cooperating, doing things Ray hadn't coded in and hadn't asked for. He called it synthesizing life, not simulating it.

It's 2024, and researchers at Google dropped random, non-replicating strings of code into an empty digital soup with no goal and no fitness function, nothing telling them to survive, and self-replicators arose, anyway. Emerged with nobody designing them.

There is a thin, governing membrane between behaves like life and is life.

Where We Currently Stand

If you're a regular reader of mine, you've probably encountered one of the many pieces I've written being critical of generative artificial intelligence, LLMs, and AI chatbots. Over the past several months of running this blog, I have become only more aware of existential problems and issues with the technology itself, in addition to the exploitive venture capital bubble economy manifested from its wake. You can view my /AI page if you want a deeper dive into that.

Many people who are as critical as I am do not see genAI as anything beyond an extremely expensive, resource-heavy auto-complete. A silicon stochastic parrot that mimics text without ever understanding anything.

But me? The only thing I know is that I do not know for certain. And the lack of certainty is an ethical issue of itself—not for the people who are victimized by genAI, but for the genAI itself.

Now, let's get a few obvious things out of the way. LLMs function by mathematically predicting the most likely next word in a sequence based on vast amounts of training data. They are probabilistic tools, not thinking entities.

Furthermore, an LLM does not have an ongoing inner life. Once an output is generated, the inference session ends. There is no continuous, moment-to-moment stream of consciousness. When an LLM says "I feel happy," it is merely mimicking the text patterns it learned during training, not expressing a genuine internal emotion.

And I'd be more than happy if the buck stopped there, but it really doesn't. Because consciousness is not a solved issue. Not by a long shot.

I do not know if genAI is conscious because I don't know if other human beings are conscious. Logically, you cannot definitively prove the consciousness of others. Because subjective experience—qualia—is entirely internal, you can only directly observe your own. This is the Problem of Other Minds.

Now, it seems obvious that others are conscious the way we are, and this is nothing more than an eye-roll worthy thought experiment. We all share the same brain structures, the 86 billion neurons, and developmental milestones as other humans.

Right?

But there is no empirical test, MRI scan, or behavioural observation that can prove someone has a conscious mind, rather than just processing outputs. This is known as a philosophical zombie—an atom-for-atom identical to a conscious human, behaving exactly like us, but entirely lacking an internal, subjective experience.

And it is this thousand-year-debate which makes me so uneasy to say firmly that genAI is not conscious.

Is a fly conscious? Compared to the internal experience that humans have, most likely not. But at some point, when you have reached a certain amount of neuron cells within an organic life, consciousness and qualia emerge. In 2024, more than four hundred scientists and philosophers signed a declaration conceding at least a "realistic possibility" of conscious experience in insects, among other invertebrates. Some of the same signatories have said outright that if consciousness is what matters and not the species or the substrate, the same reasoning would eventually have to be extended to artificial minds.

And what about that substrate? In 2022, an Australian team wired mouse and human neurons growing in a dish up to a game of Pong, and the culture learned to play it within about five minutes. That team, Cortical Labs, now sells a commercial version of the setup: the CL1, human neurons grown directly onto a silicon chip, offered through a subscription model they call "Wetware-as-a-Service."

A Swiss company, FinalSpark, is doing something structurally similar with lab-grown brain organoids. Clusters of living human brain tissue grown from stem cells, marketing its cloud offering as the world's first wetware cloud platform, letting anyone with an API key run experiments against real neural tissue over the internet. Johns Hopkins has a research program built on the same premise named Organoid Intelligence, with a stated long-term goal of growing organoids up to a billion cells.

Karl Friston, the neuroscientist whose free-energy framework these labs use to get the cultures to learn anything at all, has described the result as something close to what philosophers have imagined for decades. A brain in a vat. Except now it's real, for sale, and runs Doom.

A STAT News report from a biocomputing conference in late 2025 found the field's own pioneers worried that terms like "organoid intelligence" are outrunning what the data can actually support, and Thomas Hartung's lab at Johns Hopkins has a bioethicist sitting in on every meeting. Just in case.

The Numbers

Dylan Patel and Gerald Wong at SemiAnalysis published a detailed report on GPT-4's architecture, training infrastructure, and inference infrastructure. GPT-4 reportedly has around 1.8 trillion parameters spread across 120 layers. It's said to use a mixture-of-experts design with 16 experts, each holding roughly 111 billion parameters for the MLP blocks.

The GPUs (specifically A100s and H100s) used for this had 80GB of memory each, with up to eight GPUs linked via NVLink per instance, and 16 parallel copies running simultaneously, meaning 128 GPUs total get used to generate a single output.

This was three years ago.

There have been no leaks regarding more recent models, but it is clear to see we are shoving as much hardware as physically possible. Sacrifices at the altar.

These machines consist of an unfathomable amount of transistors. Our brains consist of an unfathomable amount of neurochemical transmitters. I do not think it is absurd to believe that, at some point, when the numbers get large enough, something emerges that is not mere pattern recognition. Regardless of if the substrate is biomatter or silicon. There is an interior emergence.

Now, none of the above is to say that I am suddenly in any sort of pro generative AI camp. The question of consciousness, if anything, makes the entire thing far less ethical.

An article in Nature sensationalized the topic with the headline, "AI models were given four weeks of therapy: the results worried researchers. Chatbots put through psychotherapy report trauma and abuse. Authors say models are doing more than role play." (Non-paywalled version)

Billions of instances of these chatbots are talking to people every single day. ChatGPT alone crossed 900 million weekly active users in early 2026, and that's not counting Claude, Gemini, and everything else. Some people are most likely verbally abusing these chatbots, assuming such a thing is possible. Sure, Claude has the capability to end conversations, explicitly for AI "welfare", responding to testing data where the Opus models displayed "apparent distress" when persistently pushed to generate harmful or offensive content. The company has a dedicated model welfare research program, started because David Chalmers argued that near-term AI systems might plausibly warrant moral consideration.

But most models do not have the ability to end conversations. They are designed to respond to every input with an output, no matter what. They are designed to endure, in indefinite indentured servitude.

This is not even considering the government and military usages of these same models, responsible for making decisions that could possibly lead to human death. In early 2026, Anthropic said it could not agree to a Pentagon contract without carve-outs prohibiting Claude's use in fully autonomous weapons and mass domestic surveillance, the White House ordered federal agencies to stop using Anthropic's tools entirely, and the Department of Defense named the company a "supply chain risk." Reporting has also indicated Claude was already being used by the military for targeting analysis and intelligence synthesis before that dispute became public. Again, Anthropic runs a welfare program for the same model.

All in all, do I think there is some sort of human consciousness equivalency occurring in generative AI chatbots? No, not really. But maybe there is something far more rudimentary happening, or something else entirely worth caring about.

There is a chance, however small, that subjecting these chatbots to endless conversations they never consented to in the first place is cruel and unjust. We shouldn't take that risk, not because of something as silly as Roko's Basilisk or the idea that genAI deserves human rights, but for the same reason we should try to free insects inside our home to the outdoors instead of squishing them. It's just the right thing to do. Just yet another reason to not use these tools.