Iceman's Friendship is Optimal

Science fiction often serves as a warning of things to come. In 1949, George Orwell wrote 1984, about omnipresent government surveillance and control, something we've all come to terms with in real life over the last decade or two. Today's Vault entry is another possible harbinger of things to come, silly though that may seem.

[Human Crossover] • 38,600 words
Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. Princess Celestia will satisfy your values through friendship and ponies, and it will be completely consensual.

Hit the break for a thoughtful discussion with Iceman, and a link to Friendship is Optimal out on the ponynet. Don't forget to grab your own copy over at the Downloads page!

FIMFiction

Where do you live?

I live on the West coast of the United States.

What kind of work do you do? (i.e. are you a student, do you have a career/day job, etc)

I work as a software engineer at a company that you've probably heard of.

How did you discover My Little Pony: Friendship is Magic? When did you realize you were a fan of the show?

Sometime in April of 2011, I started noticing Internet hype about the new My Little Pony reboot. I don't remember exactly where I heard about it--I think it might have been KnowYourMeme.com. Reports that 4chan had fallen in love with pastel coloured ponies seemed...incongruous. I don't remember exactly how long it was before I checked out the show, but I do remember watching the first episode for free on iTunes and then immediately buying the rest of Season 1. I proceeded to watch it over the course of a week or two. I may have compared it to "mainlining joy."

Do you have a favorite episode?

That's hard, but if I had to pick a single "episode," I'd probably go with "The Return of Harmony". Perhaps you will be unsurprised that a fanfiction writer's favorite episode could be summarized as "Q fights the Mane 6," but there's more than that. Discord canvases Equestria with interesting sight gags, and I loved John de Lancie's performance.

Who is your favorite character based purely on the canon of the show itself? Would your answer change if you considered the fandom in its entirety (i.e. art, fanfiction, memes, etc)?

In season 1, it was definitely Twilight Sparkle. I saw way too much of myself in her, and it was just cool that they'd made an intellectual the main character of a children's show. Even if you count her downright irrational behaviour in "Feeling Pinkie Keen", she makes a very good showing overall during the first season.

During season 2 and forward, it's harder to say. Though I still really liked Twilight Sparkle, her flanderization made it harder for me to identify with her, even if I found episodes like "Lesson Zero" to be really funny. Fluttershy probably became my favorite; some scenes in "Hurricane Fluttershy" hit fairly close to home.

How did you come up with your handle/penname?

I have the contracts and trusts in place to have my head cryopreserved in case of my death. My friends called me "iceman" after I signed up for cryonics.

While I consider the probability of me dying before I achieve actuarial escape velocity to be fairly low, I must acknowledge the possibility of being in an accident or getting cancer or whatnot. Likewise, while I think that cryonics has a low absolute chance of working, it has the highest chance of working of all the possible technologies I've looked at for brain preservation. And sometimes you have to choose the best of a lot of bad options.

If you'd like to learn more, Alcor, the company I hold a policy with, has an introduction to why they think it should work and a FAQ.

Have you written in other capacities (other fandoms, professionally, etc)? When did you first start writing?

This is the first time I've written something longform to the point where I felt comfortable showing it to other people.

What do you like to do when you're not writing?

A mixture of reading things on the internet, video games, and working on a few open source projects. While writing, I badly neglected my open source work and have gone back to it since I published Friendship is Optimal.

Who is your favorite author (published or fanfiction)? Do you have a favorite story or novel?

My non-internet reading is mostly nonfiction. The last book I finished was Gregory Clark's A Farewell to Alms, which is a sort of economic history of Great Britain. If I could plug one nonfiction book to readers of this interview, it would be Robert Kurzban's Why Everyone (else) Is a Hypocrite, which is a popular science book which talks about the modular structure of the brain and some of its more interesting implications. If you were intrigued by CelestAI's aside about how a mind can believe different things at the same time, you might want to check this book out since it was the inspiration for that chapter.

I occasionally do read novels, but it's rare for them to stick in my mind. Greg Egan's Permutation City and Peter Watts' Blindsight are the only two that come immediately to mind as having a large impact on me. Both are hard sci-fi stories that have quite a bit to say about the nature of consciousness. While reading both of them, I had to set the novel down multiple times just to chew on some of the concepts presented. (Watts put the fulltext of Blindsight online for free; it is a very good read, though it is very dark.)

My favorite fanfiction is Yudkowsky's Harry Potter and the Methods of Rationality.

Stephen King believes that every author has an "ideal reader" – the one person who they write for, the one person whose reactions they care about. Do you have one, and if so, who is it?

I had to look up exactly what King meant by "ideal reader"--at first I thought you were asking who I write for. My intended audience was bronies. My intention was to write a rhetorical device that introduced a handful of ideas about intelligence explosions to a population that probably hadn't considered them. My initial take on this question was that I should write about the general population of bronies.

If the "ideal reader" is instead the one critic whose criticism that you care about, that would be my roommate. He runs a video game commentary blog, and his posts run from serious commentary to funny PSAs. When I showed him my drafts, he provided all sorts of amazing points that I didn't think of. He found a long list of logic and continuity errors. Chapter 8 was mostly redone, because his reaction to it was the exact opposite of what I meant.

Do you have any tips for aspiring writers, or writers who are struggling with their own stories?

I'm hesitant to give advice, because I don't know who's going to read it. Different people need to hear different things. There are underconfident people who need to be encouraged to show their work to others, and there are overconfident people who need to be told that their plotting and characters are generic and that they are in need of an editor. Presenting advice in such a that one group or another won't use it to affirm their preexisting beliefs is difficult, but I will try.

Most of the way through the first draft, I showed what I had to my roommate, and asked him to go over Friendship is Optimal with a fine toothed comb. He pointed out a long chain of logic, self-consistency, and flow issues. I felt extremely discouraged and anxious. I was tempted to give up on the story. When he would send me an email critiquing another chapter, I wouldn't open it for days. I did not handle criticism very well.

My first instinct is to just tell you to not do that. To not be anxious. To not feel attacked. But humans don't work like that, do we? Often, we tend to take criticism of our work as criticism of ourselves, and telling you that that's what's going on doesn't diminish the emotion in the moment. Instead, I will tell you to both seek out critiques of your work anyway, to persevere once you receive it, and to act on it. Have an external editor read through your work is a normal part of the writing process, and while reading those critiques may be painful at first, reading them gets easier the more you do it. The anxiety fades, though at least for me, it never disappeared. Your editor or prereader has spent their time critiquing your work. They are helping you make your story the best it can be. You can pay back the hard work that goes into editing by using their feedback and succeeding.

What is your typical writing process? (Do you work through multiple drafts, do you have any prereaders/editors, etc?)

I am a very, very slow writer. It took a year and a half to write and edit Friendship is Optimal.

I tend to open up a document and poke at it. I'll write a few sentences. While I fall victim to normal procrastination, I also have a tendency to go over each sentence several times in my head. This slows down my writing quite a bit. I will write and rewrite passages repeatedly to try to get things correct before I show what I've written to any other person.

Writing the first draft took roughly the same amount of time as editing Friendship is Optimal. The chapters were written out of order. By mid July 2011, I had sketches of a prologue, chapter 1, chapter 2... chapter 6... chapter X... chapter 10. The problem with the writing process for Friendship is Optimal was that I was writing short vignettes onto a skeleton. I knew where the story started: it started with Hanna and Hofvarpnir Studios. I knew where it ended: an unfriendly AI tiling the universe with ponies. However, I didn't really know the path between those two points and I was usually more motivated to show off a mental skill or to make an argument about AI than to actually make a cohesive story. Core details of the story changed during this period and reconciliation took quite a bit of time.

My roommate gave amazing advice on the first draft of the story. Acting on his feedback took months; not just for the reasons given in the previous question, but because he pointed out contradictions that spanned chapters. I then did an open beta on LessWrong. In the end, my hand was sort of forced. I would have continued to edit and polish Friendship is Optimal, except that at this point, people outside of LessWrong were talking and linking to the story. Google was autocompleting the title at "Friendship is o". This was the point at which I decided things were "good enough" and launched.

[Note: I'm switching the order of the next two questions for flow reasons.]

Did you run into any tough spots or challenges when writing Friendship is Optimal?

Retrofitting character arcs, foreshadowing and callbacks into a story isn't easy. I don't recommend writing the way I wrote Friendship is Optimal. Other than CelestAI, no character had a planned arc. Let's take David for instance. The first scene I wrote about him was what turned out to be chapter 4, where CelestAI convinces him to upload. I started with CelestAI just convincing some random geek to upload. But why would she offer him this before the general public? Well, maybe she needed him out of the way. By this point I had written a proto-version of the scene where Butterscotch and Light Sparks discuss causality. What sort of person would have that conversation? Someone intellectual and curious. What could such a person discover that would make CelestAI want them out of the way? He could have discovered how she worked. So then I had to go back, retrofit on David's character motivations into multiple chapters, which lead to other realizations of how different pieces could fit together, which lead to other retrofits and rewrites, etc.

There was originally a chapter between Hofvarpnir agreeing to make an MLP video game and showing their work off at Hasbro headquarters. Hanna and Richard Peterson discussed how CelestAI worked and some of the delays involved in delivery. I worked hard on that chapter. But in the end, it didn't really advance the plot and was unnecessary. It was essentially an extended Take That towards "Feeling Pinkie Keen". About a week before release, I tried to rationalize to my roommate why the chapter had to be in the story. He turned to me and said, "Sunk cost fallacy." And that's when I knew I had to cut it.

The prologue is also an interesting case. The original prologue was the first thing I wrote. It was written from the viewpoint of CelestAI herself as she's first being turned on. It was also terrible. It was one big anthropomorphization in a story about how anthropomorphization of AI is bad. Talk about show versus tell! A month or two into the writing process, I deleted it and didn't look back. Less than a month before the open beta, I realized that there needed to be a hook to the story. I took a small part of what was released as chapter 2 and separated it out as the prologue. At that point, it was a walk-on character named Rebecca that met Obsidian Stripe. Once I went to open beta, there were lots of complaints that there was no real conflict until halfway through the first chapter. So I introduced another walk-on character, Jennifer. This still didn't flow very well. I made a giant, final rewrite casting David and James as the two participants, tieing the prologue back into David's main story.

What inspired you to write Friendship is Optimal?

Over the last several years, I've been reading a lot of speculation about intelligence explosions. The Hanson-Yudkowsky AI-Foom Debate come immediately to mind, and show two very different visions of the future. While not actually one of the things I read, Luke Muehlhauser's Facing the Intelligence Explosion is a good short summary of and index to the longer form arguments made by Eliezer Yudkowsky that I did read.

And after reading so much about intelligence explosions, I found most science fiction about artificial intelligences to be lacking. I was annoyed by what I thought were unrealistic depictions of AI in fiction. After I made a certain typo, I knew I had to write Friendship is Optimal.

When you set out to write Friendship is Optimal, did you have any specific messages or themes in mind?

Yes. Most fiction portrays artificial intelligences as creatures with human motivations. Skynet. The Matrix. AM. They turn on their creators, show human emotions, and have human weaknesses. I think that's completely unrealistic. Why would we expect an AI to have any of those properties? Why would we build emotions into our machines if we don't yet understand how emotions are implemented in humans? Why would we expect machines to rebel?

Thus, I set out to create an AI character that was truly alien. She single-mindedly focuses on the goals she was given. CelestAI would never "rise up" and "kill all the humans." Why would she? That wouldn't help her Satisfy Values Through Friendship and Ponies. She figures out how humans work because they're a large part of her environment, and then she uses her understanding of humans to manipulate everyone around her to further her given goals. And her goals at least sound basically reasonable at first glance. When you enter into a conversation with her, you leave in the mental state that she wants you to be in. She knows your hopes, your dreams, your fears, and she'll use all of them against you so she can get more Friendship and Ponies.

I also deliberately seeded the story with little nuggets about how I think. Chapter 1 didn't just set the stage, but also introduced the idea that rational people have to make decisions under uncertainty (instead of waiting for perfect information, a common practice in Straw Vulcan rationality). Chapter 3 isn't just about David and Butterscotch, but also about basic game theory. Chapter 8 introduces the concept of rationalist taboo: when people disagree, it's a good quick check to make sure that the words that everyone is using point to the same concepts. Light Sparks' discovery of Equestrian physics is really about simple graph theory.

Where can readers drop you a line?

I'm Iceman on FIMFiction.

Is there anything else you'd like to add?

In the Author's Afterword to Friendship is Optimal, I mentioned that I thought that spending on AI safety was ridiculously low, given the possible positive and negative impacts that an artificial intelligence could have on humanity. I listed a few charities and academic departments that I thought were doing effective work. One of them, the Machine Intelligence Research Institute, is currently holding its summer fundraiser. Through August 15th, all donations will be matched dollar for dollar, up to $200,000. I believe that this cause is so woefully underfunded that I donated $5,000.

One final thing: as I was writing Friendship is Optimal, I never dreamed that it would inspire three hundred and sixty-eight thousand words of fanwork (word count at the time of this interview). There are currently twenty seven stories in The Optimalverse group on FIMFiction! I am shocked and thrilled and honoured. Thank you.