The new version of GPT-3 performs much better (and should be less toxic)

“This work is an important step in the right direction,” said Douwe Kiela, a researcher at Hugging Face, an AI company working on open source language models. He suggested that the feedback-driven training process could be repeated for multiple rounds to further improve the model. Leike says OpenAI can do this by building on customer feedback.

InstructGPT still makes simple mistakes and sometimes produces irrelevant or meaningless responses. For example, if a prompt is given that contains false information, it will treat that false information as true. And because it’s been trained to do what people ask, if instructed to do so, InstructGPT will produce more toxic language than GPT-3.

Ehud Reiter, who works on text-generating AI at the University of Aberdeen in the UK, welcomes any technique that reduces the amount of misinformation produced by language models. But he noted that for some applications, such as artificial intelligence that provides medical advice, no amount of falsehood is acceptable. Reiter questioned whether large language models based on black-box neural networks could keep users safe. For this reason, he likes to use a mix of neural networks and symbolic AI, with hard-coded rules that limit what the model can and cannot say.

Regardless of the approach, much work remains to be done. “We’re not even close to solving this problem,” Kira said.