While much of the world is focused on near-term issues related to Artificial Intelligence — such as regulation, misinformation, IP rights, and labor market impact — less attention has been paid to some fundamental legal and societal challenges we will soon face. The next generation of generally intelligent autonomous systems will raise serious questions about what, when and how they should be permitted to operate.

Should a computer program capable of independent analysis, planning, and action be permitted to enter into contracts? Own or control assets? Serve on boards and committees? Hire, pay, and fire employees? Be licensed to practice law, medicine, and other professional activities? Function as representatives, proxies, agents, executors, brokers, and even arbitrators?

Can your electronic personal assistant refuse to execute any illegal, prohibited, or immoral actions you instruct it to perform or facilitate? Should it be required to report such activities to authorities? Can it lie to promote your interests? Can it break the law when appropriate, for instance, to save a life? And if it does break the law or cause tortious damages, who is responsible?

US jurisprudence has a long history of defining special classes of non-human entities and laws to address such issues, including antebellum slave codes, corporate personhood, and animal rights. Is it time to consider establishing another category of “legal personhood”?

This talk frames these issues around three basic questions:

• Can an AI commit a crime?
• Should an AI ever be permitted to break the law?
• Can an AI be “punished”?

(This lecture is based on material from CS22A, “Social and Economic Impact of Artificial Intelligence”, Winter quarter, 2024.)