Introduction

The fictional world of Star Trek, which depicts fanciful technologies such as warp drive, replicators and transporters, presents a surprisingly more realistic view of the potential uses for, and evolution of, advanced computation than the press releases of Google etc. and supportively breathless media accounts. 

I say more realistic, because, with notable exceptions (typically used to prove a larger point or create dramatic tension), computers in Star Trek are understood by in-world characters to be mindless, despite exhibiting capabilities which, by our standards, would be considered astounding achievements and irrefutable signs of intelligence and intent.

Artificial Intelligence, an aspirational term that does not describe any existing technology or collection of technologies, is, as a business endeavor, riddled with hype. Consider the article, ‘A robot wrote this entire article. Are you scared yet, human?’ published in the Guardian, 8 September, 2020. The article, assembled by cherry picking output from GPT-3, was, at the time of its publication, promoted as evidence of GPT-3 being a significant step up the ladder towards what’s sometimes called Artificial General Intelligence or AGI. After pushback and critique, Guardian’s editors added a bit more context, admitting that an AI did not, in fact, write the article: “We cut lines and paragraphs, and rearranged the order of them in some places. Overall, it took less time to edit than many human op-eds.” (the bit of face saving at the end is hilarious).

This hype requires, indeed, demands, a variety of counterpoint arguments. Hopefully this essay and the ones to follow will make a contribution.

In a series of three posts, I’ll present three in-show situations (from both the original and Next Generation series) :

• The Original Series Episode “The Ultimate Computer”
• The Next Generation Episode “The Measure of a Man”
• The Next Generation Episode “Boobytrap”

I’ll use these episodes to illustrate Star Trek’s thematic treatment of computer power – as a tool, not to be confused with the complexity and nuance of living minds. Furthermore, I’ll argue that Star Trek posits that the power of minds comes, perhaps paradoxically, from incompleteness (about which, more later).

This may seem trivial or of only academic interest. My argument is that the presentation of computational systems as possessing intelligence is a propaganda project, intended to demobilize workers and obscure the true sources of harm. Each of us who knows better has a responsibility to shine a light on this propaganda in a variety of ways.

This is a part of that effort.

The Ultimate Computer

“The Ultimate Computer” is the twenty-fourth episode of the second season of the American science fiction television series Star Trek. Written by D.C. Fontana (based on a story by Laurence N. Wolfe) and directed by John Meredyth Lucas, it was first broadcast on March 8, 1968.”

In “The Ultimate Computer” the viewer is presented with a clear line of separation between the starship Enterprise’s sophisticated library computer system (known as LCARS in the Next Generation series) – which possesses interactive voice response, large language and text synthesis capacities and extensive command and control capabilities – and a thinking machine, the M5, created by Dr. Richard Daystrom (the scientist who designed standard starship computer systems). The M5, patterned after Daystrom’s mind,  is able to reason and indeed, exhibits the ability to think in basic ethical terms during a critical scene, when it’s forced to confront the fact its actions resulted in death. Despite these remarkable capabilities, the machine lacks nuance and could be said to operate on the level of an extraordinarily well-informed child.

For me however, the remarkable thing about this episode is the fact in-world characters such as Spock, Kirk and McCoy collectively express astonishment that the machine is able to think at all.  

In their experience, there’s a common understanding of what thinking beings do and what sophisticated computers are capable of. There is, in other words, no confusion between the act of rapid, statistical pattern matching, text parsing and data synthesis via sensors and what they, as people, do from moment to moment.

Consider this scene, when Kirk and Spock debate Dr. Daystrom about just what M5 is:

Spock: (to Daystrom, while examining the M5): I am not familiar with these instruments Dr. You are using an entirely new type of control mechanism. However, it appears to me this unit is drawing more power than before.

Daystrom: Quite right! As the unit is called upon to do more work, it pulls more power to enable it to do what is required of it just as a human body draws more energy to run than to stand still.

Spock: Dr, this unit is not a human body. A computer can process information, but only the information that is fed into it.

Kirk (to Daystrom): Granted, it can work a thousand…a million times faster than the human brain but it can’t make a value judgement, it hasn’t intuition, it can’t think.

Daystrom (smiling like a Cheshire Cat – then, waxing poetic) : Can’t you understand? The Multitronic unit is a revolution in computer science. I designed the duotronic elements you use in your ship right now and I know they are as archaic as dinosaurs compared to the M5…a whole…new approach!

[…]

Later, in a tense scene, after M5 has fired weapons on unprotected starships (misinterpreting an exercise for real combat), wounding and killing many, Daystrom tries to reason with it to stop:

Daystrom (to M5 via audio interface): M5 tie-in

M5 (to Daystrom, via ship audio): M5

Daystrom (stressed, trying to calm his voice): This is…this is Daystrom

M5: Daystrom, acknowledged

Daystrom: M5, do you know me?

M5: Daystrom, Richard, originator of comtronic/duotronic systems born…

Daystrom: Stop. M5, your attack on the starships is wrong. You must break it off.

McCoy (to Kirk): I don’t like the sound of him Jim.

Kirk: You’d better pray the M5 listens to the sound of him.

M5 (still responding to Daystrom): Programming includes protection against attack. Enemy vessels must be neutralized

Daystrom: But these are not enemy vessels! These are federation starships. You’re killing…we’re killing…murdering…human beings, beings of our own kind. You were not…created for that purpose. You’re my greatest creation. The ‘unit to save men’ – you must not destroy men.

M5: This unit must survive.

Daystrom: Survive! Yes! Protect yourself! But, not murder. You must not die, men must not die. To kill, is a breaking of civil and moral laws we’ve lived by for thousands of years. You’ve murdered hundreds of people…we’ve murdered…how can we repay that?

M5: They attacked this unit…

Kirk (whispering to Spock while M5 is still replying to Daystrom): The M5 is not responding to him, it’s talking to him.

Spock: I am most impressed with the technology Captain. Dr. Daystrom has created a mirror image of his own mind.

“It’s talking to him” Kirk observes. For him, and everyone else in this world, a clear distinction is made between programmatic response, and actual conversation. This profound difference is purposely obscured by the current discourse which encourages us to view audio response technologies such as Amazon Alexa, Siri and GPT-3 as being capable of conversation.

In the end, M5, built to create a new class of autonomous computers, intended to replace crewed space vessels, is shown to be deeply inadequate for the task. 

This episode establishes what I’ll describe as the pop sci-fi epistemological framework of Star Trek on the question of what Joseph Weizenbaum defined as “Computer Power and Human Reason” (the difference between judgement and calculation). In Star Trek, Computers, as a rule, are unable to reason and incapable of judgement. Outliers and exceptions, such as M5, illustrate this principle via their existence as outliers (which can’t be productionized).

In the next post, I’ll explore how the question of computer power and human reason is addressed in the ‘Next Generation’ episode, “The Measure of a Man“.