Journalists like to punctuate stories about the risks of artificial intelligence—particularly long-term, humanity-threatening risks—with images of the Terminator. The idea is that unchecked robots will rise up and kill us all. But such martial bodings overlook a perhaps more threatening model: Aladdin. Superhuman AI needn’t have it in for us to wreak havoc. Even with our best interests at metallic heart, an AI might misunderstand our intentions and, say, grant our wish of no more cancer by leaving no more people. Here the risk isn’t evil slaughterbots but an overly literal genie.

At...