A lot of factors can affect how a chatbot behaves — or misbehaves. First, there is the data a model is trained on, which often includes a lot of unsavory stuff from the internet. Then, there is reinforcement learning, that’s when humans give feedback on what's appropriate or helpful. And then, there are system prompts, which are instructions engineers give a model, often once it's already being used out in the world.

But small tweaks can often have big, unpredictable effects, Sayash Kapoor told Marketplace’s Meghan McCarty Carino. He's a PhD candidate at Princeton University and the co-author of “AI Snake Oil," which is both a book and a Substack by the same name.

More on this

“Why xAI’s Grok Went Rogue” from The Wall Street Journal

“How Do You Change a Chatbot’s Mind?” from The New York Times