FIRSTS Conversation with the First Director of the US Artificial Intelligence Safety Institute

JR: Hello and welcome to First Conversations. This is our podcast and speaker series that puts a spotlight on barrier breakers, glass ceilings, smashers and innovators who helped shape modern life. All of our guests are trailblazers who have cleared a path for others, and you'll get to hear more about just what it took for them to get there. So, my name is Jessica Rosenworcel, and I am the Chairwoman of the Federal Communications Commission.

And today's guest is helping to blaze the trail for the AI revolution. I am talking to Elizabeth Kelly, the inaugural director of the US Artificial Intelligence Safety Institute at the National Institute of Standards and Technology. Now, previously, Elizabeth served as special assistant to the President for economic policy at the White House National Economic Council, and she was one of the principal authors of the President's landmark executive order on artificial intelligence. Now for those efforts, Time magazine recognized Elizabeth as one of the 100 most influential people on AI, which is even cooler when you recognize that the list also includes Scarlett Johansson.

Thank you, Elizabeth, for joining us today. It is great to have you.

EK: Thank you so much. It's a pleasure to be here.

JR: Alright. Before we talk about your current job, I want to roll back. I want to hear the back story. So how did you get interested in working on issues involving technology?

EK: So, I am incredibly lucky to lead a team of incredible technologists, ethicists, anthropologists at the AI Safety Institute. And we are really working to understand the capabilities, the risks of AI and mitigate those so we can enable safe AI innovation. I came to this role, in part, because I had this broad background in economic policy work at the National Economic Council and then in between Obama and Biden administrations actually helped start and sold a company and so got to work with engineers and data scientists and brought that background to my role at the National Endowment Council, where I was phenomenally lucky to lead our work on both technology and on financial services, including AI. As we saw ChatGPT entering the scene, and everything changed.

JR: So, were you a technology geek when you were young or came to it later?

EK: Geek is an interesting phrase.

JR: We embrace it here at the FCC.

EK: Yes, I appreciate that. I think we embrace it here at the federal because yes, I appreciate that. I think I've always been really fascinated by how new innovations can shape our future, can bring about new ways of working, new ways of living, new drug discovery and development. And I think AI is just the quintessential example of that.

JR: Yeah, absolutely. So, what led you exactly to public service?

EK: So, my parents are both social workers, so I grew up in a family where service of some type was always at the forefront. My dad ran a college in New Orleans and worked with homeless teens my mom worked with at risk pre-K kids.
And so, I always knew that I wanted to do some type of service. Pretty early was led to
more public service as opposed to direct service. It was actually Hurricane Katrina hitting my hometown of New Orleans and getting involved in that recovery and response that I think really catapulted a lot of the work I've done since.

JR: Wow. That's a really tremendous back story. And I do think that an instinct for public service runs in families. I know that's true in my own, and if it doesn't run in the family as well, if you're here today, you're probably getting one started. Alright. You were one of the chief architects of the executive order on artificial intelligence. When I counted, it was 63 pages. That is one dense document. So, I want you to tell me a little bit how you got involved with that, and then maybe we could talk a little bit about what's inside of it, as I'm sure you're familiar.

EK: Indeed. So, the President has really been focusing on AI for quite some time. Right.  I think the executive order as a midpoint in those efforts, because you had the AI Bill of Rights coming out of often cited technology policy. You had the risk management framework coming out of NIST, where I now work, and things sort of kicked into high gear with ChatGPT’s advent. We had all of the CEOs of the AI companies in the Roosevelt Room making commitments to the President, and we were instructed after that meeting to use every lever at our disposal to both harness the benefits and mitigate the risks of AI. The President likes to use the phrase “promise and peril”, which I think sums it up pretty well. And, you know, from my perch, the National Economic Council, I was really fortunate to be able to think about how AI affects privacy, affects the workforce, affects decisions around lending and employment, and how do we use AI responsibly across all those areas.

JR: So, I'm still trying to imagine all those people sitting in that room in the aftermath of the issuance of ChatGPT. How did all that lead to what's in the executive order?

EK: I think that it was pretty clear we needed to take a broad view of what his impacts would be, whatever the merits of the AI EO. And I happen to think there are a lot brevity is not among them, as you say. And so, I think it's helpful to think about the executive order in three large chunks. The first, which is actually where I think we've gotten the least attention, is around how do we really harness and spur innovation.

JR: Like the good stuff?

EK: The good stuff. Exactly. How do we put more money to research and development Through a pilot of the National AI Research Resource, or the NAIRR, which provides access to data and compute for academics and entrepreneurs who are using AI for socially beneficial things like climate mitigation and health care. How do we attract the talent to this country that we need through streamlining via processing, interview waivers so we can continue to lead here? How do we actually make sure that we're upskilling our current workforce? That, to me, is a large part of the executive order. The second part is really focused on how do we adopt our current rules, regulations for AI. Khan often said that there's no exceptional laws on the books, which I think is really what the bulk of the EO is focused on, what guidance needs to be issued in terms of how you think about using AI and lending decisions or things like that. And the last piece is really where I think the AI Safety Institute comes into play, which is how do we think about the underlying frontier models? The GPTs, the Geminis that are powering all this innovation and there the EO really sets us on in terms of information gathering mission. There's reporting requirements about the most compute intensive models. There is the work that we're doing at the Safety Institute around voluntary testing, and there's so much more to be done.

JR: Yeah, I mean, I think I get it with the promise, the peril. And how do we prepare?

JR: Yeah.

EK: Very well put. We're going to add that.

JR: Alright. All 20 piece, you're all going to remember it. So, it's been almost a year since that executive order has been released. I'd love it if you could say what you think. It's accomplished, what we've done best. Are there places where you think we still have work to do?

EK: I think one of the most remarkable things is that we have not missed a single deadline in that executive order, which is actually pretty crazy when you think about it. It's not something that I can say is true of all executive orders, but I think it speaks to the huge amount of presidential priority that's been placed on this and how you see that trickle down to every agency, every regulator that's working to implement that. And as a result, there's been some really great things that have happened. Talked about standing up the National AI Research Resource. We at the AI Safety Institute are now about eight months into our efforts. You've seen new best practices for employers who are deploying AI, and there's a lot more to be done. I think that the EO to me is again a midpoint in these efforts. And as the technology continues to evolve, we hopefully see Congress take action. I think that we will be even better positioned to really harness the benefits and mitigate the risks.

JR: Are there any big upcoming deadlines that you think we should be paying attention to?

EK: Well, I've got a couple, on my end. There were two task things that were given to us under the executive order. One was guidance that we put out on how do you mitigate the risk of misuse of what we call dual use foundation models. The idea basically being that a model could be used for very good things like drug discovery and development. But the flip side of that is it can unfortunately be used to facilitate development of chemical or biological weapons. And that guidance really looks at what are the practices you need to have in place across the lifecycle. So, starting from training of the model all the way through incident reporting, once that model is out to make sure that the developers are taking the appropriate steps to mitigate the risk that can arise in everything from cyber threats to nonconsensual intimate imagery.

JR: Is that risk present in every dual use model?

EK: Well, not all models are image generation models, so there are some differences and obviously some dual use models are more powerful than others. And so, it depends in part on what is the level of capability of the models that we're looking at. And these risks are still evolving. So, any answer I gave you today may not be true in a couple of months.

JR: So, the AI Safety Institute that you run was not actually mentioned in the way that it actually works.

EK: That is true. Yes.

JR: So, when did you first learn about this office? How did it get created?

EK: And so, you're right, of course, it is not mentioned in the executive order, but a number of the task things that are outlined in executive order are very much let's say, homework for the AI Safety Institute. I perhaps am the unwise person who wrote the homework and then took it to rather than just leaving it. It came together really that fall, I think we were incredibly aware of the fact that there was so much work being given to NIS, the national security space. We were seeing AI safety institutes popping up across the globe, starting with the UK AI Safety Institute that was founded that summer

JR: When they had that Bletchley Gather.

EK: Exactly. So, it was sort of formally launched at Bletchley.

JR: So, other countries were setting up their own safety institutes?

EK: Japan, Korea, now Canada, the EU has a division. We're up to I think about ten globally.

JR: Wow.

EK: And so, I think it was really important for continued US leadership that we have a safety institute and that we have a place to sort of house these efforts, which are a pretty new for the US government.

JR: Okay, so walk me through what it's like trying to stand up the Safety Institute for Artificial Intelligence. I mean, you're starting from scratch. You're at NIST. So, what is it like on your first day on the job?

EK: So, I have both helped start a startup. I worked in government starting a startup inside the federal government as a whole, new thing. I think we were very fortunate to be, you know, at NIST, which obviously has an incredible scientific legacy. Multiple Nobel laureates, a great tracker record, AI with the AI risk management framework. We're able to harness a lot of that scientific expertise and bureaucratic knowhow. But there's also just the usual things about starting something new. I think either my first day or a couple of days prior, I was actually touring government office space in San Francisco to figure out where we were going to house the engineers, the policy people who were going to be based in the Bay Area, because that's been so essential for our ability to attract really top notch talent, but also to make sure that they have an ear to the ground. They're involved in the conversations, they're at the cutting edge. So, I don't know if there's anything particularly exciting, but touring office space, figuring out how to hire people, identifying a chief of staff, all the usual things that you would prioritize and create something new.

JR: So, for folks who are totally unfamiliar with it, what does the AI Safety Institute now, now that you've got the office space, now that you've, you know, cleaned out some desks at NIST

EK: Just wanted to give you a mental picture of all the glamor of the job,

JR: Right.

EK: So, we are the government entity that is tasked with understanding the capabilities and potential risks of AI models and working to mitigate those risks so that we can truly harness the innovation. We talk about enabling safe AI innovation is really our North Star. That's something you hear a lot from my boss, Secretary Raimondo, and I think that's why we all get up in the morning and we go about doing that through three primary lines of effort. The first is testing. We are building up entirely new US government capacity to actually test the most capable models prior to their deployment across a broad range of risks. And we're working with the developers we just announced to MOUs with open AI and Anthropic and active conversations with other lead developers looking at could this model be misused? For example, perpetuate a more dangerous cyber-attack? What are the risks around synthetic content, and then working with the developer to mitigate those risks?

JR: So, you're working with models that are of a certain size?

EK: Not exactly a certain size. So, I think that there's a specific provision in the executive order that you're probably citing that for models that are trained using 10 to 26 labs. So, that's a certain size, in terms of compute. Those developers have to report to my colleagues, the Bureau of Information Security, about the existence of those models and the red teaming or testing they're doing. We certainly work with our colleagues at BIS, but we have a little bit more of a flexible mandate; we're really focused on the models that show increases in capability and that may or not trip that 10 to 26 threshold. I would think about it in terms of how are these models performing on different benchmarks, like, for example, GBQA, which is a science-based benchmark where we've seen huge improvements in how models are performing even this year. We have about 25% to 80%. And so, we're really focused on those models regardless of where they are in terms of compute.

JR: And are those big companies you mentioned, are they eager to show up at your doorstep and say, here, do some safety testing of what I've produced?

EK: Well, I think we are fortunate in that these companies have already made commitments to the White House. Last July, the White House announced voluntary commitments from, I think, a total of 14 companies between July and September announcements to actually do red teaming and testing of these models, as well as a number of other things.

JR: What is red teaming mean?

EK: Red teaming is sort of a subset of testing. You hear the words testing, evaluation, red teaming, sort of all used interchangeably. Our testing involves both automated capability evaluations. So, things you can just run automatically as well as more expert red teaming where you've got specific individuals prompting the models to see what they're able to elicit. I think we're fortunate that we've already got these commitments to the White House, to the G7. And I think there's also a real understanding of the important expertise that the federal government brings to bear here. Everyone has an interest in making sure that we avoid the worst possible risk of these models so we can continue to see innovation grow. And we're fortunate at the AI Safety Institute to be channeling the expertise across the federal government and cambio risk and cyber risk, radiological nuclear, the DOE and able to bring that to bear in a way that a company wouldn't be able to.

JR: So, all of this testing requires incredibly high levels of technological knowhow. And it must be difficult to bring that knowhow into the government right now when this is, you know, an enormous growth sector in our economy. What is recruiting for the institute and all of this work look like?

EK: Well, this is why I spent my first day touring office space, because getting them to move to D.C. would be even a tougher task. But no, more seriously, I think we've been able to build a fantastic team of computer scientists, ethicists, anthropologists thinking about this from a really holistic view and the person who actually heads up our testing team helped invent a technology called Reinforcement Learning by Human Feedback, which was really key in enabling AI to be both safer and more usable. And we've been recruiting top PhD students from programs like MIT,  Berkeley. I think there's a recognition that this is a pivotal moment in time, and if you care about this technology and its trajectory, there's really no better place to have an impact. And it's been really heartening to see the number of people who were interested in joining us and creating this new capacity inside the federal government.

JR: Well, that's terrific. But they're not moving to Washington, is what you're telling me.

EK: Not really, no.

JR: Alright. So, when you talk about AI, you know, this there's people who fall into the optimistic camp, say, full steam ahead. And then there are the pessimists who insists now is the time to pump the brakes. So, I'm wondering what you think about those two camps and honestly how we balance the desire to innovate here on our shores and also, you know, make a nod to those who talk about existential risk.

JR: All before you, have coffee in the morning.

EK: You know, don't worry, I'm a couple of cups in. So, let me first say that I am incredibly excited about all of the incredible potential use cases of AI: new drug discovery and development that could address previously intractable illnesses, new methods to mitigate a lot of climate risks like carbon capture and storage, individualized education for students in a way that can really help address issues in equity in our educational system. And I think that across the US government there is a tremendous enthusiasm for what we can achieve with AI, and you see that in the fact there's an entire section in the executive order focused on how can the federal government better use AI to serve the American people and a whole AI talent surge across the government to get them into departments so we as a government can better use AI for all of the many things from Social Security claims processing to weather prediction. I think that our job at the Safety Institute is really to enable safe AI innovation. We want all of these future scenarios to happen, which means that we need to have an understanding of and be on top of both the current risks that we're seeing in terms of bias, discrimination, synthetic content, and what that means when you're seeing nonconsensual intimate imagery, but also the risks that could emerge, the risk to public safety and national security, like enabling a more dangerous cyber-attack or providing some type of uplift and development of a chemical or biological weapon. Our job is to sort of be mindful, be watchful beyond the vanguard in order to address those risks as they emerge so that we can make sure to see all the innovation that we're so excited about.

JR: So, has your relationship with the industry producing these innovations been a healthy and cooperative one, or is it sometimes adversarial?  You know, they want to move real fast and you want to pause and ask questions.

EK: So as I said, I think we're fortunate that there's already been a commitment by all of these companies to the White House, to the G7, to do this testing. And there's an acknowledgment they need the expertise that the federal government brings to bear. You know, obviously, our job is to insist on a level of transparency and accountability, which I think is going to be all the more important as we see, you know, continuing increases in the capabilities of these models. And if 2024 was a big year, 2025 will likely be even bigger. So, laying this groundwork this year so we can really be ready to push next year, I think is going to be key.

JR: Yeah, I like what you said there about transparency. I think that's at the center of a lot of this. Something we also hear a lot about is the production of synthetic content, deepfakes, manipulated voices, images, videos. So, I'm wondering how the AI Safety Institute is thinking about all of those things.

EK: So, you asked me about what our homework was. This is actually one of those. We, alongside our colleagues at NIST, are tasked with putting out a report that really provides an overview of what are the existing tools and techniques to detect content that is artificially generated to watermark content that's artificially generated so that you're able to say, okay, this is very visionary. That's not to know what is true content. How do you actually know that? For example, if the President is speaking, that is in fact him? Because there is some type of, you know, chip inside the video camera. These are the types of innovations that we're seeing. And our job as scientists is to help advances innovations, both by providing some clarity about what is really best in class and a field that is evolving so quickly. And whatever answer I give you now is probably not going to be true in six months.

JR: And so, I think that chips or watermarks or a legal obligation to disclose is going to be enough for all of the above.

EK: So, I think that this is moving really quickly and we're going to see  lots of different methods and who knows sort of where it will end up. There are some folks who posit that at  some point it will no longer be that you're looking for a watermark, an artificial generated content, but rather that the true content itself is what is authenticated. And we'll sort of flip the paradigm.

JR: Interesting.

EK: Who knows?

JR: So, the absence of that ID would itself indicate that it could be…

EK: exactly

JR: Produced and synthetic.

EK: That is one possibility. We are still early stage, but our hope is that by really providing some common understanding, some sense of best practices and hopefully helping advance the conversation and lean into those scientific innovations, we can be a pretty positive value out here.

JR: And like you said before, we're not the only country looking at these safety issues. I think you mentioned ten other institutes. So, how are we engaging with those other institutes, those other countries and trying to think about these risks in a collective way?

EK: So, we have a number of bilateral partnerships. We have a MOU with the UK IAC by reference, we have a dialog ongoing with the European Union, which just set up a whole AI office to actually implement,

JR: Is that out of Brussels or

EK: It is out of Brussels. The EU AI Act.

JR: By the way, that's like the European equivalent of Washington.

EK: No comment . . . And will be implement the EU Act which passed last year. But from our view, it's really important to build on these bilateral partnerships and create a multilateral network of AI safety institutes or similarly named institutions across the globe that are actually helping advance this work. We need to stand on each other's shoulders and learn from the work that is happening. There's a fraction of the money that's being spent on AI development that is spent on AI safety. And so, we want to learn from the work that's happening in Singapore around synthetic content, the work happening in Canada around risk mitigations and be able to stand on each other's shoulders.  And more than that, I think we want to start moving towards aligned best practices, interoperable evaluations. This is going to be incredibly important because we don't want there to be blockers to innovation. And the more that we can be aligned globally, not only can we better advance AI safety, but we can better enable  AI innovation. And that's why we've launched this network. We've got 10 countries on it. We're bringing them all together now in San Francisco this fall, and we're really excited to bring the technical folks together and start to cement those ways of working.

JR: Alright. So, you spent so much time thinking about AI now I just want the story. What are you most hopeful about,

EK: AI related or everything.

JR: Well, look, we can use optimism in any form you want to deliver it, so. But we can start with AI.

EK: I think it's how do we use AI to tackle the pressing societal problems that have previously been really tricky. I talked about drug discovery and development. I think that's a huge one, being able to more quickly identify what drugs might actually work and get them to market could be game changing for a lot of people across the globe. I think that it's going to be huge in the climate space, everything from weather prediction to better crop growing techniques. There's going to be a lot that I think could meaningfully improve people's lives. We talked about education. I mean, I think the list goes on, but to my way of thinking, whatever we can do to make sure that we're directing those resources to those most societally beneficial use cases and that we're mitigating the risk so that this can be realized. That is what I'm most hopeful about.

JR: Yeah, we often talk about AI and the problems it creates. We don't always talk about the problems we can solve.

EK: Yes.

JR: Many of which are intractable and costly and, you know, we lack the political scale to really address them. But maybe these tools can change that.

EK: I totally agree. I think it's really interesting if you look at AI favorability, I think in the US a lot of citizens are a lot more skeptical about what AI will mean for their lives, whereas there's a lot more positivity in a lot of the developing world, the global South, because I think there's excitement about how you can use AI to address some of these challenges.

JR: So how do we change that here? A lot of the recent data that I've seen, there's a lot of skepticism in these parts about AI and what it might yield for all of us.

EK: I think a lot of it is really putting resources behind the things that are going to improve people's lives.

JR: Yeah.

EK: I think we're starting to see that. But I think, you know, if my child was receiving a better education because there was a customized education for her, those sorts of things, as they started to show up, I think will be meaningful. I think there also needs to be more competence and how are we mitigating the risks so that we're making AI work for people and that's a lot of our work.

JR: And whether we trust the models that are being developed. 
EK: Exactly.

JR: Not just replicating bias we know in the world today.

EK: Right. Right.

JR: Alright. Well, one of the things we do with this first conversation series is we close with a lot of questions about firsts. And so, we're going to start with this totally mundane. What's the first thing you do in the morning?

EK: A little corny, but I actually start with a gratitude meditation, and it's sort of a nice way to center myself and think about what I'm grateful for before I attack the deluge of emails I've received overnight or chase my two and a half year old around the kitchen.

JR: Both of which take energy. Alright, what's what was the first concert you ever went to?

EK: I grew up in New Orleans.

JR: Oh, you’re going to have a good answer.

EK: Well, the answer is I don't remember because it's so much a part of the culture . . .

JR: Sure. Sure.

EK: That you just go to like weekend festivals with amazing headliners like Kermit Ruffins or John Botay. And you can eat a snow cone at age six and hear these acts.

JR: Wow.

EK: Which is a really amazing way to grow up.

JR: Okay, that's a tremendous answer. What's the first piece of advice you'd give to someone on their first day of a new job?

EK: Read. Stay a student. I think, you know, I've been fortunate to take on roles, White House in the startup ecosystem, where you are constantly having to learn and evolve and carving out that time to make sure that you are getting smarter. Investing in your own education, I think is huge.

JR: Yeah, and these are early days in AI, that opportunity is available for everyone.

EK: It really is.

JR: Alright, I can't resist asking about this first. What's the first thing you had ChatGPT write for you?

EK: I think it was holiday thank you notes.
JR: The gratitude.

EK: Yes. Well, I mean, I grew up in New Orleans. I was taught that every single present gets a thank you note. And ChatGPT made that a lot more efficient.

JR: Alright.

EK: There are restrictions on what I can use for government purposes, but not in my personal correspondence.

JR: Alright. I couldn't resist asking. Alright. So, this podcast and speaker series really celebrates those who paved the way for others or done big things first. So, I want to close by asking, tell us about a mentor or mentors who influenced your life's work.

EK: There are obviously many people I could list here, but one specifically comes to mind,
which is my godfather, David Johnson. He was a 24-year Army veteran, led the Army Strategic Studies group, spent many years at Rand, and there were a couple of things he taught me that really stand out. One is the importance of making sure that your life's work has meaning. It has impact. It lasts beyond you to the importance of speaking truth to power. He was always working to get the army and the entire defense apparatus to think about new ways of better protecting Americans. And especially in government, it's important to be nimble and question the ways of doing things so you can stand up entirely new capacities. And three, just the importance of continuing to mentor and invest in other people. In the final days of his life, he was guest lecturing a classroom in his hospital bed. And one of the real gifts for me was that, you know, a couple of months after he passed away, I started drafting the executive order and working with entirely new folks I'd never worked with, and it turned out that a number of these folks were all former students of his. And so, to be able to see his impact in the work that I'm doing day to day in the EO with the Safety Institute has really been a gift.

JR: Well, that is beautifully said. So, before we go, tell us how people can follow your work and the work of the institute going forward.

EK: It's very straightforward. It's nist.gov/DCAISI. So please follow us. There's a lot happening in the coming months, and we are excited for what's ahead and to partner with all of our colleagues across the Federal government.

JR: Thank you so much for joining us. Appreciate your time and what you do.

EK: Thank you. Pleasure to be here.