Sometimes it’s like the model is reluctant to challenge a premise because it thinks this part of the data distribution doesn’t… Like the AI doesn’t challenge the premise. And if you just tell it’s allowed to do that, that’ll partially fix the problem. And I’d say that you can say that part of some hallucination is just because the model doesn’t know that it’s allowed to say “I don’t know” or it doesn’t know it’s allowed to express uncertainty. The language models are trained to maximize likelihood of text so they can generate text and they produce things that look like text on the internet. I’d say one class of hallucination is about language models having this pattern completion behavior. When people say hallucination, sometimes they mean a few different things. And then it says, “John Schulman is a well-known researcher in the field of artificial intelligence, blah, blah, blah.” So yeah, I think GPT-4 does pretty well there. My knowledge cutoff is September 2021.” That’s where the pre-training data ended. Can you provide some more information?” And then I tried GPT-4 which is fine tuned with the chat recipe and that one says, “I don’t have any information about John Schulman being arrested for keeping exotic animals, blah, blah, blah. So this one it says, “I’m sorry, but I don’t have any information about an individual named John Schulman being arrested, blah blah blah. This is based on a model that’s about the same overall performance, same smartness, but it’s fine tuned differently. So that’s a model that’s trained with RL to be helpful. So it gives you some story about keeping tigers, a serval, which is that cute cat thing over there, et cetera. So tell me about John Schulman’s arrest for keeping exotic animals in his home. All the examples I’m going to show you are the first sample I got with the query, which I just ran yesterday. So can you see the text? OK, so here’s an example. OK, so you might have heard this term hallucination, the language models hallucinate. And then I’ll talk about some open problems in this general area. So I’ll talk about some of the work we did on using retrieval-based methods for fixing this. And it turns out that reinforcement learning is part of the solution for fixing it. So I’ll give my perspective on why that’s happening and how to fix it. And you all know how language models often make things up, often convincingly. And one of the biggest technical problems around language models today is truthfulness. So I wanted to focus the talk a little bit. And since most of my time at OpenAI, I’ve been running the RL team, the reinforcement learning team, which switched to focusing on language models and fine-tuning them a few years ago and that led to some of the projects I’m going to talk about today. Yeah, I worked with Pieter on… Started out working on robotics and then got interested in reinforcement learning midway through my Ph.D., as deep learning was starting to take off and that turned out very well. Yeah, it’s really great to be here back in my alma mater. Yeah, thanks so much for the very kind introduction, Pieter. So glad to have had you and thanks for making it back here. I go to Jose, I say, “Jose, what do you think if John stays in my lab?” And he says, “Please, he seems way more productive in your lab. Next thing we know John is working in my lab. Can you please help me recruit him?” I helped Jose Carmena recruit John. And robotics is going to play a part in that. My own first encounter was not directly John, it was Professor Jose Carmena comes to me and he says, he works in neuroscience, he says, “There’s this new student that I really want to recruit, is absolutely the best. I’ll tell you one quick story of my own first encounter with John. So it’s a real pleasure to have John back here with us. ![]() Then proximal policy optimization, the most widely used algorithm today in that space. He is the inventor of the modern deep learning based policy grant algorithms, including translation policy optimization, which he did at Berkeley together with Mike and me actually. Is that right?įrom there, co-founded Open AI, and most people say rest is history, but not only that, he also is the chief architect of ChatGPT. It’s an honor today have with us here John Schulman. Thank you Ken for hosting the whole series and setting this up. Welcome to, I think Ken, is this the fifth in the series? Yes, the fifth seminar in the Berkeley AI series. That shares stories of people at UC Berkeley and the work that they do on and off campus. ![]() New episodes come out every other Friday. Podcast from the Office of Communications and Public Affairs that features lectures and conversations at UC Berkeley. Episode #166: “ChatGPT developer John Schulman on making AI more truthful.”
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |