Hello, and welcome back to the Cognitive Revolution!
Today, we're kicking off a short series on creative AI governance proposals.
and I'm speaking with Dr. Jillian Hadfield, Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins University, and Andrew Freedman, Co-Founder and Chief Strategy Officer at Fathom, about their proposal to govern AI via private regulatory markets.
AI is – to put it mildly – a hard technology for society to effectively manage. The relentless march of capabilities advances; the radical uncertainty about how powerful AI systems will get, how quickly; the feverish pace of adoption; and the increasingly intense international competition combine to create huge stakes, but little clarity on what should be done, with legitimate worries that even the most tech-savvy policymakers could easily get things wrong.
And yet, while highly proscriptive government regulation, of the sort Europe has attempted with their AI Act, doesn't seem likely to meet the moment, the fact that xAI can credibly claim frontier capabilities while Grok 4 continues to self-identify as Hitler … suggests that a laissez-faire free-for-all won't serve us well for much longer either.
Is there any way to create a governance regime that's agile enough to keep up with AI developments, sophisticated enough to address the most important and extreme risks, yet not so burdensome that I would still be able to have my AI doctor?
It's a hard problem, but Dr. Hadfield and Andrew have a very interesting proposal to harness market mechanisms and hopefully create a "race to the top" in AI safety standards. which has been introduced into California's legislative process as SB 813, and from what I hear seems to be gaining traction in a number of Red States as well.
The core idea is to separate the process of democratic deliberation about the outcomes we want – and want to avoid – from the detailed rule-making process meant to get us there. In concrete terms, a government body – perhaps the California Attorney General, or perhaps a newly created AI Safety Board – would articulate goals, like "AI systems must not enable bioweapon development", or standards, like "autonomous vehicles must be safer than human drivers" – and then create a competitive ecosystem of private certifiers who develop safety standards, engage the companies to make sure they are properly implemented, and report back to the government and public on results.
Companies could choose to work with these approved certifiers and, in exchange for meeting their standards, would receive some degree of liability protection when things still end up going wrong – which, given the unwieldiness of AI systems generally and the unsettled nature of AI liability law, is a serious incentive for that would presumably convince many companies to opt in to the system.
As a lifelong libertarian, I really like the idea of trying to bring market dynamism to AI governance, and I appreciate that while this idea is new to the public now, Dr Hadfield has been developing such concepts for decades, and even worked with Anthropic co-founder and policy lead Jack Clark on related ideas as early as 2019
Andrew, for his part, brings invaluable practical implementation experience to the table, having worked on Colorado's "cannabis czar" while the state was rolling out a new regulatory system for legal marijuana.
Nevertheless, as you'll hear, I pressed them on several concerns:
How do we avoid a race to the bottom where companies simply choose the most permissive certifier?
How would the liability protections interact with existing tort law, and what exactly are people giving up in terms of their ability to sue?
Do we have any organizations that could step up and do a good job in the role of private regulator?
And … who do we really have to trust to do a good job for such a system to work on an ongoing basis?
In the end, there's no silver bullet here – any governance system we might design does ultimately rely on some number of people doing a good job in key roles – but I do come away optimistic that an arrangement of this sort – if it could put the good folks at FarAI, METR, or similarly tech-savvy organizations in a position of authority – could deliver much more responsive regulation than the government could muster on its own, while also making sure that society is not flying entirely blind into the fast-approaching AI future.
Coming up soon, we'll have another episode with Professor Gabriel Weil, who has a very different proposal to address many of the same concerns via liability law – so stay tuned for that, and definitely reach out to let me know which of these ideas you find most promising, or if there are other proposals you think would be better yet.
With that, I hope you enjoy this exploration of a proposal to harness market dynamism to effectively govern AI technology, with Dr. Jillian Hadfield, of Johns Hopkins, and Andrew Freedman, of Fathom.
Nathan Labenz: Dr. Gillian Hadfield, Bloomberg distinguished professor of AI alignment and governance at Johns Hopkins University, and Andrew Friedman, co-founder and chief strategy officer at Fathom. Welcome to The Cognitive Revolution.
Andrew Freedman: Pleasure to be here. Thanks for having us.
Nathan Labenz: I'm excited for the conversation. You are working on some very interesting stuff. I'm always looking for creative solutions to the vexing problems of AI. In the governance space, we're still flying naked through this rapidly cresting technology wave. I think you have a very interesting proposal. I understand there's a one-two punch we'll want to keep in mind for this conversation: one being the general set of ideas, and then...
Andrew Freedman: Yeah.
Nathan Labenz: ...second, but also very important, this is being introduced now into the California state legislative process with an actual bill that will, at some point, either get revised, passed, or hopefully even one day come into law. So for starters, could you tell us what you're up to? Give us the grand landscape of this private governance notion.
Andrew Freedman: I'll start there because so much of the heart of this idea comes from Professor Hadfield. I'll talk about how Fathom got interested in it. Fathom started just over a year ago on the notion that AI was going to, for better or worse, break a lot of things—governance being one of them, as well as the way we interact as a society and as an economy. We as a society were going to have to figure out how to put it back together. Generally, tech policy has been left to tech to figure out. This was a much broader societal issue. So how do we start not just a think tank, but an organization that could help bring ideas up from society that will best fill the needs here and then help build them? That's what Fathom has based its mission around. The very first thing we found when we did tons of polling and qualitative work, meeting with leaders across industry and society, was that everybody thought governance was needed. Almost everyone agreed that the current ways of thinking about heavy-handed, top-down governance probably wouldn't work for AI, but that simply leaving everything to the labs was also not going to solve for society. So that began a journey for us of, if those aren't the solutions, where are the solutions? It turns out that the professor has been thinking about this for a very long time and has some amazing thoughts. There are other thought leaders, Dean Ball being one of them, who thought there was a third way to start thinking about this that was not generally brought up in public. So I'll leave that there as to how Fathom became interested and toss it over to the professor.
Jillian Hadfield: Okay, Gillian. I've been thinking for decades in my career about how well our legal and regulatory systems perform. I worked on access to justice for a long time, and then started thinking about the way our legal and regulatory systems are responding to technology and globalization in the mid-2000s. I recognized that the systems we developed for making law and regulation, starting in the 19th century, were no longer fit for purpose. They didn't keep up with the complexity, speed, and multi-jurisdictional nature of the world we live in. AI just ramps that up several levels. We have this real mismatch between the way we make law and regulation and the speed and complexity with which technology now moves. So I started thinking about how we need to adapt our approaches to building that regulatory infrastructure for a much faster-moving, complex, and now AI-based world. That's when I started thinking about how we get markets more involved in figuring out our regulatory problem. As a starting point, it's really important to recognize that regulation is the thing we build in. It's not something that is just dragging down markets. I started writing about this quite some time ago: the idea that our markets are built on good legal and regulatory infrastructure—contract, property, fraud, antitrust, all that good stuff—that allows people to invest and participate in markets with confidence. So, if we have stuff moving at the speed of very rapidly adapting markets that are producing the technology, how do we get more of that market energy and investment into solving the problem of the best way to build that regulatory infrastructure for technology? That led to this concept of regulatory markets, which is the idea that we still need our governments involved in setting the acceptable risk level for society and making judgments about what we will and won't allow. But for the technical question of how we translate that into what companies and labs actually need to do, we need to get more market activity into that phase. We can talk more in detail about how this all works, but that's really where it came from. Then Jack Clark and I wrote a paper in 2019, when I was on contract and he was policy director at OpenAI, proposing that this was a model for AI safety, and we've just been building on that since then.
Nathan Labenz: That's interesting. I didn't know that tidbit that it goes back to 2019 and that you were working with Jack at that time. I don't know if it was Luigi Zingales who said this, or I forget where the source of the quote was, but I always remember this quote: "Markets are not free or unfree. Markets have rules, and some rules work better than others."
Jillian Hadfield: Perfect. 100%. It's a constant refrain: there's no such thing as a free market. I've been saying this my entire career. My PhD is in economics, which I did jointly with a law degree. The focus there was on all this institutional structure. Economists assumed that markets just existed. This was after the fall of the Soviet Union and the shift from socialist economies to market-based economies. Economists just said, "We'll just get rid of all that government control over industry, and markets will flourish." And they did not, because if you didn't have good legal systems for enforcing contract rights, property rights, intellectual property rights, and good regulation, markets don't thrive. So, fabulous quote, it's exactly the right one. "There's no such thing as a free market." There are healthy markets that are well-structured with good legal underpinnings and the kind of regulation that makes everybody willing to invest and participate. That's what leads to a vibrant market. We know this around the world because the countries that struggle on the development side don't have a good rule of law. They don't have good legal underpinnings, and nobody wants to invest there. So I think that's a really important observation.
Nathan Labenz: Before we get into the details of private governance structure and the way that's instantiated in SB 813, and possibly some variations on that, let's take a minute and cover what you see as the fundamental problem with the top-down approach. You can characterize top-down as you will. Not too long ago, someone like Sam Altman was saying maybe we need licensing for frontier models. Obviously, he's backed off of that. Something like SB 1047 was, in my view, fairly light-touch, but still had elements of top-down in that there were statutory thresholds. I think the critics have been at least partially vindicated in that those thresholds haven't aged super well, and it hasn't been a very long time. I'm interested in the problems in that approach, but also interested in the problems on the other end: why do we need new rules here? Some might say we've got general rules for commerce. Is this really that different? Why should we think of this as being different than any other new product that somebody might bring to market? I don't really buy that, of course, as I spend all my time thinking about this, but I'd like to hear you dismantle that naive notion.
Jillian Hadfield: From the point of view of thinking about regulation as something we're trying to accomplish for healthy, prosperous, and fair societies, top-down governance just becomes more and more untenable. If you're setting detailed rules, it becomes more untenable the more complex technologies are, the more quickly they develop, and the more jurisdictions they're in, because you're setting rules that have to be followed everywhere. I'm an economist and a big fan of markets, not for any ideological reasons, but because they're good information processing and discovery engines. They're down at the ground level, responding to what's happening in the trenches. You need that kind of information to figure out the right way to regulate to capture the benefits of promoting innovation and getting efficient markets, while at the same time establishing those ground rules that make everyone feel confident and willing to invest and participate. The problem with the top-down approach is it's very hard for governments to have access to that kind of information. We do have ways of getting that, which we developed over the 20th century, with chemists and experts in biology, forests, and clean water. But when you need to move at the speed with which technology is advancing, you have this real mismatch. You need this information from the ground level, but you have a process in our legislatures, courts, and agencies that operates on a different time scale, and it's very hard to keep up. One of the things we've observed in that process over the last several decades is that it has, in many ways, gotten more ponderous. The length of our laws is a lot longer than they used to be. Opinions from courts are a lot longer. It takes a lot longer to accomplish things, and that means you don't revisit them. There's just so much sand in the gears, and we don't keep up very well. I think that's the issue with the top-down approach. So what you want to do is find a way to get all that intelligence from the ground level into your regulatory system without abandoning ultimate democratic control, because we as a collective need to be deciding what's okay and what's not. Are we taking this risk with autonomous vehicles, or companion AI, or various algorithmic decision-making? We need to be making those decisions, but can we separate making those risk judgments from the technical process? What do you need to know about how this works, like what data you should train on or what red team tests you should do? Can we get that detail? I think that's the issue with trying to do that all top-down from government.
Andrew Freedman: The only thing I'd love to add to that is about my first job in emerging regulatory systems. I was the cannabis czar in Colorado for the rollout of its regulatory system. In many ways, that's a much simpler policy than AI will ever be, and we thought that touched every area of Colorado law at the time. The strength of the system, despite its muddled success throughout the nation, was that whenever we set up a system that was more iterative, we were able to say, "Here's a problem. We didn't see edibles coming up this way, and now we can change the rules and understand." The idea that we can predict where the AI system is going at any one time and create good guardrails that will make sense even six months later is wrong. The challenges and opportunities will be huge, but our ability to predict the future will be very low. Part of what attracts me so much to this system is that it puts independent subject matter experts upfront and allows them to continue to make decisions over time about what good looks like. The second part is that if the industry understands what good looks like according to independent experts, and understands that it will be universally applied to them, that's the brass ring they'll start reaching for. I think when you create top-down measures, that tends to become a floor that gets thrown to a compliance department. The compliance department then just checks boxes to say they technically meet the standard but doesn't get in the way of the business unit. So that structure is more likely to ensure lawyers have a job in the future of AI than it is to ensure that what's happening is actually in the public's best interest—and Gillian and I are both lawyers, so no shade thrown there.
Nathan Labenz: So let's describe the mechanism. I'll let you do it.
Nathan Labenz: There are three tiers.
Andrew Freedman: Yeah.
Nathan Labenz: But the floor is yours. Tell us what you're proposing.
Andrew Freedman: Let me take a first hack at it and then have Gillian clean everything up. The fullest instantiation of this idea is in Senate Bill 813, but this is a conversation we want to have many times in many different ways, and I think S.B. 813 requires quite a bit of revision, so I don't want to get too in the weeds. But there are two ideas there. One is, how do you set up a regulatory markets mechanism to identify third-party auditors, private-side regulators, verifiers, and certifiers who know what best practices are at all times? They must have the ability to prove to companies and the government that they can verify claims and that companies are meeting these best practices. They would scope where their expertise lies, saying, "We are focused on AI as it pertains to chatbots or automotive vehicles, and these are the safety parameters we're looking at." An AI developer or deployer would come in and say, "We're worried about the risk we're taking on, so we want to be certified that we're meeting best practices." They enter a process to be certified, and if they can show that, Senate Bill 813 would say that should be proof on the back end that they've met a standard of care. It should count as evidence in court that you did the best you could and are meeting that duty of care—a liability shield is probably too strong a word. Meanwhile, the third-party auditors are constantly going back to the California government and saying, "Here's proof that when we certify people, they're doing better in the world. Our certified AI cars are getting in fewer crashes. Our chatbots are causing fewer issues among teenagers. We are outperforming the status quo and other certifiers in this area." Therefore, there's a race to the top to show they should keep their ability to certify against someone who can't prove they're doing as good a job.
Jillian Hadfield: Andrew's given you a version of a particular implementation of it. There are a lot of questions about how you get to this. It's a real transformation in the way we approach regulation. When I first started working on this, I was focusing on AI. A book I released in 2017 discussed that we need to change how we approach regulation and add this tool to our toolkit because most technology is moving very fast, and we have very decentralized supply chains. I'm going to talk about the more abstract version of this and compare it to the top-down version. The cartoon version of regulation is that the government sets rules, regulated companies follow those rules, and then the government monitors for violations and fines them or takes them out of business. It's called command and control or prescriptive regulation. People have been working on designing more agile and adaptive regulation for some time, leading to developments like performance-based regulation. For example, with pollution control, the command and control version is the government saying, "Here are the specific scrubbers you have to install in your smokestacks." It's a very 1970s version of a regulatory problem. A performance-based approach says, "Here's the acceptable level of pollution. You, the factory, figure out the best technology to achieve that." Different factories could use different technologies, and companies could arise to develop more cost-effective ones. To be in compliance, you had to reach those output goals. A regulatory market structure takes that idea of government setting outcomes further. For example: here's the acceptable level of accidents for autonomous vehicles, or we want to ensure any uplift in the capacity to develop bioweapons falls below a threshold. It doesn't have to be a numerical threshold; it could be qualitative, judged by experts. Our tort standards are outcome-based: we want companies to take reasonable precautions to prevent harm. Instead of just saying to the regulated company, "You figure it out," you say, "We want to develop a whole sector of independent firms engaged in figuring out the best way to achieve those outcomes." The government licenses what I'll call regulatory services providers. They are licensed based on demonstrating that their approach achieves the desired outcome. Then the companies you're trying to regulate—the target companies—select a regulator from that market of approved regulators. In the original theoretical framework, this was mandated. In the pollution context, you couldn't come up with your own approach; you had to select an approved company. The key idea here is the market. We need a vibrant market attracting financial and human capital to the project of finding the best way to achieve the outcome. Is it red-teaming tests? Is it a review of the data? Is it embedded officials observing the process? We don't know the best way to regulate to achieve our goals with AI. The basic structure is: government sets outcomes, an independent sector of companies (for-profit or nonprofit) specializes in developing the technology to achieve that goal, and the third component is the target companies you're trying to regulate. As Andrew noted, we're familiar with private actors like certifiers and auditors playing a role in complex regulatory systems. Lawyers and accountants have professional obligations to ensure firms are law-abiding. Part of getting to this ideal version is moving our existing markets toward providing greater oversight and more substantive content on what a lab actually has to do to limit liability or comply with government requirements.
Nathan Labenz: How similar is this to other things that exist today? I'm not aware of any major sectors that have such a market today. I'm also not sure how core the liability protection is to the idea, or if it's one of many carrots or deals that could be made. Then I'm thinking, how similar is this to the auto industry, for example? There's a lot of regulation and standards there. Do we in fact have a similar deal already in place with car companies, even if it's not through this market structure, where as long as they're hitting certain standards prescribed by an agency, they do in fact get liability-
Nathan Labenz: protection from that? So, several compare and contrast points.
Andrew Freedman: The best I can say is this smells a lot like other things, but it is novel in some ways. The novelty is important in a couple of ways, but the one I like to compare it to most is Underwriters Lab, or UL. If you pick up any consumer electric good in your house, it will have a little UL stamp. That company was started during a world's fair in the late 19th century because they didn't want their entire fair to go up in flames. They brought in an independent subject matter expert group to inspect all of these different shows and make sure that there wasn't a fire liability. That group took off and found its way into a number of different arenas, basically saying, "We have a really valuable thing we can add." You don't have to trust every consumer product that comes your way; maybe you should just require those consumer products to get this UL stamp. About 100 years later, UL started to get codified straight into law. I do think there is a need to make sure that the public good is more directly instantiated into these private governance worlds. I don't believe that if we let it go, we will find that there will be enough verifiers, auditors, and an ecosystem out there that all have enough of a true north pointing to societal good to solve this problem for us. Leaving this to the private side without any government accountability will probably do more to create things that cover the butts of the labs and less to ensure what's happening is actually protecting the public good. So, I do think that Gillian's race-to-the-top mechanism, which requires some amount of accountability and sets the goals of these companies via public legislation, is a vital aspect. I also believe we don't have 100 years to wait for these things to organically grow within these systems. We have to take the lessons from UL and ask, "How do we supercharge that? How do we make sure it's accountable? How do we make sure it doesn't get captured by industry? And how do we make sure its true north is pointing in the right direction?"
Jillian Hadfield: Yes, it's exactly right to say this is novel, so you can't point to an existing full-scale model already implemented in another industry. This is a proposal for something we don't have yet that I think we need. It's about being innovative in our legal and regulatory technology in the way that we are innovative in the underlying technology. But there are lots of things that come up to the doorstep. Andrew emphasized entities that perform the function of creating standards and very specific technical requirements, which then get played out into the market. That's a market in the sense that Underwriters Laboratory and other entities—we have lots of standard-setting organizations like ISO and so on—those are non-profit companies in the business of creating standards, sometimes very technical ones. Then they can be adopted because they have a good impact on market share for a company, or they get implemented and picked up by a government that says, "If you want to be in compliance with California regulations on building farm equipment, you have to have followed the standards from this organization." One of the examples I like to give of something that gets close, and is a helpful model, is the regulation of medical devices and quality production. It's a consortium of about five countries, including Canada and the US, that have said each country will choose its own standard for quality control in medical device production. The US has an FDA standard; Canada uses the ISO standard, a non-profit standard-setting body. But collectively, they have produced an approved list of authorized certifiers of compliance with those standards. They have a rule that says if you've been certified by one of these approved certifiers, it's a single audit, and then you can sell your medical devices in all the countries in the consortium. So we've got pieces of this. I always like to emphasize that this is part of the development of regulation. Often the history of regulation in a new area starts in the private sector. Securities regulation started with private organizations like stock exchanges saying, "Here are our rules. If you want to participate on our stock exchange, you have to engage in this kind of disclosure." That disclosure helps bring more people to the exchange, and then that gets picked up by government. It develops on a private basis and gets integrated into government. Today, the regulation of financial transactions has important roles for these private entities. For example, FINRA, the Financial Industry Regulatory Authority, is a private membership organization, but it's overseen by the SEC. The SEC actually approves the rules that FINRA uses to oversee its members. There are lots of examples of this public-private integration in our current regulatory regime. They're much more complex than the cartoon I was giving earlier of command and control, where government sets the rules and companies have to comply. It's a very complex and overlapping system. This takes it to the next level and says let's really lean into that outcome-based role for government and try to get more market activity around the best way to achieve these regulatory objectives. I think your question, Nathan, also went to liability and carrots. It's important to keep these things distinct: what's the regulatory structure we want to get to, and then how do you get there. In the original proposals about this, it was just mandated. Just like we mandate compliance with securities laws or health and safety rules, we could just mandate that you must purchase the services of an approved regulator. Of course, it takes time to get there. We need to evolve this market, to build this market. We don't have lots of players. It would be very hard to turn around tomorrow and say, "You must buy the services of an approved regulator in some of these AI domains," because you just don't have the market. The vision behind something like SB 813 is to ask, "How can we move ourselves towards that?" That's where you start to think, just like there's a market-based incentive to get the UL certification mark on your product, can we create an incentive that says you don't have to participate in this structure, but we're going to create that mark. We're going to create a mechanism for government to say these are approved providers of this service, and we'll create an incentive for that. If you do it, you will have the capacity to demonstrate you've met your compliance requirement. If you get sued, you'll be able to say, "But I was following the program. I was in compliance with the requirements of this approved regulator." The state had said this is an approved oversight body, a certifier. That could just be an argument in your tort case, or it could have a formal role where you get a legal benefit, a presumption that you were in compliance with your tort liability duty. This is because the government started by saying, "We're only going to approve those entities that can actually demonstrate that if you did what they required, you're in compliance with your tort duty."
Andrew Freedman: I will say,
Nathan Labenz: Mm-hmm. Yeah, go ahead.
Andrew Freedman: With SB 813 and some of these things, it's almost like two complex ideas put into one that have interesting ways they play off each other. Some things I like about tort being the backdrop for why you would want to enter this is, one, it does force the market to decide where the risk is. Some players can come in and say, let's say we're certifying for something that the leading labs think, "That is not something we'll ever get sued on, and it's not a problem we think will ever come up." Then that part of the market holds no value. So it forces everyone to get real about where they think the harm will come from. Instead of there being 10,000 edge cases, it forces people to start asking, "Where is the brunt of where we really need to focus?" So I like that part of it. The second elegant thing is that tort as the backdrop has this feature of being national, even when it's not federal. The common law applies everywhere in the United States. For those unfamiliar with tort law, it was brought over from England. It's the idea that makes us all whole in the background of society. It differs from state to state, but there's a lot of commonality across states. So meeting your duty of care in one state can be proof that you're meeting your duty of care in another. At a time when there's a lot of fear that proposals will create a patchwork between states, these two ideas combined could create national, maybe even international, private-side regulators who provide a very real good across state lines and even internationally. There's a carrot there that doesn't complicate compliance; it actually centralizes and focuses where we should be worried and how we can meet compliance goals.
Nathan Labenz: Can you calibrate me on what this would mean? I guess this could obviously be set at various levels. I understand there's a distinction between the more diffuse academic layer of ideas and the specific statutory proposal on the table in California. How much would the public be giving up in terms of its ability to sue if SB 813 were to go into effect? Maybe calibrate me on how that compares to pharma or auto?
Andrew Freedman: Yeah.
Nathan Labenz: I generally have the sense that if I get into a car accident, I can't sue the carmaker, but maybe in some cases I can. I don't know exactly.
Andrew Freedman: Mm-hmm. Yeah.
Nathan Labenz: There's probably always a carve-out for something, right? Similarly, if I have an adverse reaction to a drug, I probably can't sue the drug maker unless maybe they sent me a tainted pill or something. So, I don't quite understand what the limits are, even in the world that I inhabit today.
Andrew Freedman: Yeah.
Nathan Labenz: with products I'm very familiar with.
Andrew Freedman: Yep.
Nathan Labenz: I'm not sure how that translates to AI, where, just to complicate things a little further, we have very familiar concepts like an AI might drive my car, make a medical diagnosis and recommend a treatment, or it might become my romantic partner and then,
Andrew Freedman: Yeah. Totally.
Nathan Labenz: I have no idea how to even think about how one would taxonomize that. But we can maybe leave that for part two.
Andrew Freedman: Yeah.
Nathan Labenz: First question is just,
Nathan Labenz: what is the trade that SB 813 is proposing? How does that compare, and do you think it is the right balance, or how would you maybe revise it, if at all?
Andrew Freedman: Yeah. Great questions. First of all, not only do you not know that, but literally no one knows. We're skating to where we believe the puck is going here. So far in the world of technology, it has been fairly shielded from tort law. It comes up, but in limited situations. I believe, and I think a lot of people would say, and even tech companies are starting to understand, that the agentic nature of AI—AI being more than an algorithm that produces a specific output from a specific input, but in fact being an actor in the world—is going to bring it into a world of liability that I don't think tech has been in before. The nearest proof we have is the Character.AI case, which just survived a motion to dismiss, and is the chatbot example of being open for tort law. My guess is, and I think it's reasonable to say, that the hundreds of state judges across the country are going to find various levels of liability for developers, deployers, and applications. So the full stack of people involved in AI will have to start worrying about liability in a way they didn't before. Consumers, on the other hand, will have options for suing in cases where something bad happens to them and they can say, "It really was all the way back at the model level." Where 813 sits currently is that this is a rebuttable presumption, to get nerdy about it. So,
Nathan Labenz: Please.
Nathan Labenz: It's encouraged.
Andrew Freedman: This counts as evidence, but if you can come in with any other evidence that they didn't meet it... If you really want me to get nerdy, here's the nerdy part. One of the elements of tort law is always going to be, did you meet a duty of care? There are many different ways of defining duty of care. Sometimes there's strict liability, where if it causes harm, it's your fault, versus negligence, gross negligence, or if you clearly did something intentional. There are a lot of different theories of duty of care that come in, and then the question is, whatever category you fall in, did you meet that standard? None of that has really played out in AI at this point. So ours is a fairly light touch as it sits right now. It says one of the ways you can think about it is, was whoever wanted to get the certification acting negligently? Our bill would say this should stand as a good amount of proof that you were not acting negligently. If you can come in with proof that they were acting negligently, that can counteract it in a trial. I would say we're very open about where to move that needle based on stakeholder input. Our initial goal is to determine what's enough to bootstrap this regulatory market system to get people involved and come to the table. I do think tort law offers a lot of what I call potential energy, meaning there are many places where harms could make companies do the right thing. So far, that has not translated into kinetic energy; it all remains theoretical. 10 or 15 years down the road, I do think SB 813 does a lot to say, "You should think about it now, and you should reach for best practices now to be prepared." Let me be the law professor here and think more abstractly about tort law, because my answer to my students when they would ask, "Well, can you sue?" was always, "You can always sue, especially in tort. You can always sue."
Nathan Labenz: It's America. Come on.
Andrew Freedman: Exactly. That's part of the tort system as part of our common law system—the fact that it is entirely court-based and judge-based. It only evolves out of the cases that people have brought. Lots of things can happen once you get there, but you can always sue. You can always get in there and claim, "I was harmed, the defendant is the one that caused my harm, and I was harmed because they didn't take reasonable steps to prevent this harm to me." It's not that complicated. It gets much more nuanced in case law, but that's also part of how we evolved our law through the 19th and 20th centuries. The law evolved because the courts were there and people could file their suits. Regarding your question, Nathan, about the relationship now that we have automobile regulation and pharma regulation, what's the impact on tort there? It varies from place to place, but you don't generally get a barring of the potential to bring and prevail in a tort claim if you're selling an FDA-approved drug.
Jillian Hadfield: You can still file your claim that the company did something that doesn't meet the state standard for what's required. You could definitely sue the manufacturer of the automobile, even though they're in compliance with whatever regulatory requirements exist. What courts do in those cases is take all that into account and say, "Well, we think the reasonable steps to take were getting FDA approval," or, "The reasonable steps to take were complying with federal law." So I think all of that is still available. There's no sense in which you're closing the door to the capacity of courts to participate in structuring this. We do have a few cases—I know of a couple—where the government comes in and says, "No, you cannot file a lawsuit here." For instance, with vaccines, I'm pretty sure you cannot file a tort lawsuit, but we also have a compensation fund for injuries from vaccines. Long ago, I worked on the September 11th Victims' Compensation Fund, created by Congress for those killed or injured in the attacks. That came with the stipulation that you can't sue the airlines or the port authority running the World Trade Center. The project I was working on was about how people felt about not having access to courts and only being able to go through this compensation mechanism. But I don't think we're considering anything like that here. It would change how the tort case works, but it's really important to emphasize that this is about two alternative ways of ensuring people comply with what they're supposed to do in tort, which is to take reasonable steps to prevent harming others. You could litigate that in many different courts, with cases having different ways that courts and juries supply the content to that standard. That happens at the end of a long, expensive process that, by the way, doesn't give everyone access and doesn't work perfectly. Or you could try to pull that back to an earlier stage and say, "We're going to try and establish up front what it means to comply with that duty, and we're going to give you an independent, approved oversight body that will come in and look." We're not going to wait for accidents to happen and for people to get harmed, followed by long litigation. We're going to move that process of deciding what you had to do to do the right thing closer to when we're releasing and observing our products, instead of relying solely on the tort law approach of waiting until something happens and then litigating for those who can afford it.
Andrew Freedman: Can I just say one other thing that I think gets lost in this? Because you end up thinking about the sympathetic person who was harmed and what remedy you have given them. But the overall goal should be fewer people harmed. I would much rather have a system that says, "Can you prove that you're harming fewer people? Then we should reward that behavior." There should be some front-end mechanism. I'd much rather there be fewer cases simply because fewer bad things are happening. But I think it's hard to maintain that as the true north.
Nathan Labenz: Yeah, for certain subcategories, that seems relatively clean. I've seen some of these graphs put out by Waymo and Swiss Re where they're just like, "Here's the level of accidents and injuries with human drivers, and here it is..."
Andrew Freedman: Yep.
Nathan Labenz: "...with Waymo," and it's like, okay, let's all move to Waymo. That seems pretty clear. I guess I wonder how to red team the bill, which has become a meme in this space.
Andrew Freedman: Yeah.
Nathan Labenz: How do we create a race to the top and avoid all sorts of shenanigans when it comes to these legitimately very hard questions? What gets categorized as what? You said a second ago nobody knows in response to some of these fine-grained liability questions. A joke that I've recently made is, what is an AI agent? Nobody knows that either.
Nathan Labenz: Right? It could be anything from a workflow that exists in Zapier to something that is calling senior citizens on the phone and is explicitly instructed not to identify itself as AI, which, by the way, is something I have done on existing commercial platforms. Not actually calling seniors, but demoing that it can be done. So all of that right now is getting swept up into "AI agent." I wonder, with so much in flux and so little clarity on what counts as what, and the space being pre-paradigmatic on taxonomizing itself, how do we create the right incentives to actually have a race to the top? How do we avoid situations where somebody says, "Well, I'm the AI agent regulator," and then they're lumping a lot of things together or doing some sort of weird bundled trade? Another data point on this: I was briefly in the financial services industry in the run-up to the mortgage meltdown, and I saw up close and personal how the credit rating agencies had been totally captured and were basically worthless at that point. So how do we avoid the credit rating agency problem? That was maybe similar in a way too.
Andrew Freedman: Yeah.
Nathan Labenz: There were all these exotic products at the time, and if there's one thing you can say about AI, it's that it's an explosion of exotic products. So what are the key points in terms of creating a race-to-the-top dynamic that is real and durable, as opposed to getting ourselves into the 2007 credit rating scenario?
Jillian Hadfield: Can I take the credit rating point? Because Jack and I actually discussed the credit rating agencies in our 2019 paper, because everyone said it's going to be like that. There are really important points about the credit rating agencies. Their market demand was created by the government because the government said you have to be rated by these credit rating agencies to issue bonds and so on. But at the same time, the government immunized the credit rating agencies from any liability for the ratings they gave. So there was zero government oversight of how well they were doing their job. That's completely different from this regulatory market approach. We tried to move the role of government to having oversight over these regulatory services providers—the ones who are figuring out the best method for making sure we don't get an uplift in bioweapons or that we have safe AI companions. We absolutely need a government role there, and it's government oversight. We're shifting the role of government from the detailed oversight of what the labs are doing, which is almost impossible, to the oversight of what these private regulatory services providers are doing. So the race to the top could come from imposing a standard that the government sets, and then it's peering in, regularly looking, and saying, "Look, we're going to yank your license if you don't meet this standard." You could imagine a standard that required state-of-the-art protection against providing the capacity to build bioweapons to people with nothing more than high school chemistry. So now you have these competitive companies in this business who have a market interest in communicating to the government, "Look at what we figured out. Here's how we can reduce that risk and demonstrate it to you. By the way, our competitors in the regulatory services market are trying to pull the wool over your eyes." Company A has an interest in increasing its market share and demonstrating it can do better. So I think there's a race to the top there. Then there's the race to the top of finding the most effective and cost-effective way. Part of what we're facing with AI governance right now is that we've defaulted to a lot of process-based regulation—check this box, put this oversight process in place—where we didn't actually test whether it works. Do we know that it works to have these logs and these officials in place? So again, the race to the top is to get government oversight that says, "Here's what we want," and have companies competing to achieve that standard. I think it's really important to recognize the difference with the credit rating agencies, which as you pointed out, are a key example of a big failure of this private role. But it wasn't this model because it did not have oversight of those entities. I'm sure there's some regulation of credit rating agencies, so I don't want anyone to say, "Here are all the laws they have to follow." But they were definitely immunized from liability for their ratings.
Andrew Freedman: A couple of other things are important as guardrails for these private side regulators. First, the government has to be very involved in their finances. There are two things. One is to make sure they aren't seeded or funded by the labs themselves, ensuring some independence. Second, they must be able to afford to deny certification and continue existing. This is some stress testing that would fall on the government to ensure this is working correctly. If the mindset is, "Here are some great processes, but if we don't certify four out of five labs, we can't continue to exist, so we're going to figure out how to certify four out of five labs," then that's trouble. That's a deep analysis that has to happen from the government. The second part is that the government has to get real about what outcomes it's expecting. As you mentioned, Nathan, in some places that's easy and in some places that's hard. In some places, you can say we have a very clear human analog to what's going on. So if you can't prove that you're safer than humans in this way, then that's bad. But in some places, it's going to be completely new stuff, like new harms, new ways of thinking about harms, or places where we don't have incident reporting systems. Or places where the first incident is so catastrophic that it doesn't matter that you followed some rules on the way there; that outcome was just totally unacceptable on any level. I do think there are crawl, walk, run parts of this model. One of the reasons I like it being so voluntary at the beginning is that the government sets outcomes in certain places, and the companies get to decide if they think that amount of liability protection or seal of approval is worthwhile to get right now, and the conversation makes sense at the beginning. There are going to be some places where it will have to work out over time. Are there outcomes the government can set for us where this makes sense to bring in this private regulatory model? Or does there have to be a different solution? Is that issue so major and in such a different place that the regulatory markets don't solve for this problem and it needs to be put into a different category?
Jillian Hadfield: Yeah, I can definitely see some bio-risk things in particular being beyond the scope of what any sort of liability framework could handle. I recently was on another episode saying it's hard to go to the Wuhan Institute of Virology—whether or not it ultimately came from there is another question that I'm not taking a position on—but with 10 million plus dead globally, it's hard to go there and sue them for damages. It seems to be an order of magnitude different thing. As the externalities become so big, you may just need a-
Andrew Freedman: Yeah.
Jillian Hadfield: ...totally different regime. Yeah, I think-
Andrew Freedman: I honestly think that is one of the amendments that we would love to see in something like SBA 13, which is that there's a harm so big that it shouldn't fall within this. And that's part of figuring out the exact right landscape for how we start this program or this way of regulating. It doesn't have to accomplish everything, but it's accomplishing some very real things at the beginning, and we can grow it and mature it into a way to accomplish a large portion of things. But certainly, there are always going to be edge cases, especially when attached to really large harms, that have to be handled otherwise.
Jillian Hadfield: Yeah, I think it's really important to emphasize that it's just one tool in the toolbox. I think it's something really important that we're missing in the toolbox and that we will need more in some areas than others. But it is not like it displaces all the other complex ways in which we achieve safe, fair, stable market societies and so on. We already have a very complex set of systems that interweave and overlap—tort law, standard settings, corporate incentives to create safety, the press, oversight, international bodies. There's just tons of stuff. So it's going to be a part of that complex system. And then the other point we maybe haven't emphasized enough is what you had asked earlier, Nathan: what's different about AI? Why isn't it just like any other product? We regulate cars, we regulate drugs; it's another product. Well, AI is not a thing, it's not a product. It's a general purpose technology that I think is going to transform the way we do just about everything. So it's going to end up impacting and requiring regulation on a ton of dimensions. We have a very complex regulatory landscape, and as you mentioned, education, health, justice, logistics, city management, bio-weapons, AI companions—it's just going to be in everything. And so we're going to have different regulatory goals and different regulatory methods in all those places. So we don't want to think, "Oh, we've regulated AI." It's about whether we can build a vibrant, robust, agile mechanism that helps us identify where and how to regulate. This was a point Andrew was getting to earlier. Another way this kind of ecosystem can help is to surface where the problems are. Where is the need to regulate? Where is there demand for regulation? This could be from enterprise purchasers who say, "I can't integrate that chatbot into my customer service because I don't know if it's going to make stuff up." And that could cause them harm by giving bad advice to their customers, or because people will think they're selling something they're not. So, I can imagine that enterprise starts to come in and say, "Here's the kind of protection that we need in order to drive adoption." I think that's a critical part. We talk about risks and harm, but we also want to be talking about why we're building this in the first place. Hopefully, the reason is because it can make everybody better off. So we need to figure out how we can respond to the ways the market can tell us what concerns people have. Parents have this concern about their kids in school using AI. The tort system has surfaced very sad concerns about the way AI companions are impacting children. That's a bottom-up process. We want a way for the market to respond to what it can tell us about what concerns people have, what problems people have found, what stumbles we've identified. That's why the top-down approaches in regulation are so complicated. That's what markets do for you—they sniff it out at the ground level and tell you where an opportunity is, where a harm is, where a risk is, where there's a demand for something different. I think that includes, again, in this very general purpose way, that there's simply no way to sit in a boardroom with a whiteboard and say, "Okay, here's the list of risks we need to worry about from AI, and here's the rules we could put in place." I just think that's a fallacy of lawmaking and regulation.
Nathan Labenz: I want to push a little bit more on this race to the top versus race to the bottom, and then I've got-
Nathan Labenz: a few objections or red teaming from other perspectives besides my own that I want to throw at you and get your reaction to. It seems on the race-to-the-top question, you make a great point about the demand from enterprise. Enterprise in general rightfully wants to use this technology, but also wants to cover their butts, and that seems like a force for good. That also seems like those folks are going to be more oriented toward known unknowns than unknown unknowns, I think. So, putting my AI safety x-risk hat on for a second and focusing on the frontier developers, the ones that are pushing things forward as fast as possible and really getting into uncharted territory, I don't think it's a fair description to say that they are all trying to do the least they can. I think-
Nathan Labenz: on the contrary, we're relatively fortunate compared to counterfactuals I can easily imagine, in terms of the people running these frontier developers and how responsibly they're acting, again, relative to alternatives. But nevertheless, my cynic's, follow-the-incentives sort of analysis would be that the frontier developers are going to want to do the least that they can. So if they have a menu of options in front of them, they're going to choose the most permissive, least costly one. I'm not entirely sure if money is supposed to be flowing from the labs to the regulators, the MROs in SB 813 parlance-
Nathan Labenz: or if the money is coming from some other place. But if there's any correlation between who gets picked—they pick you and that's how you get paid—then there's this incentive to try to be the one that gets picked, which all seems like a race-to-the-bottom type of dynamic. Then it seems, at least in the SB 813 scenario, we're really relying on the attorney general to be doing a great job. They're the ones that have to approve these organizations in the first place and keep a close eye on them. If they take their eye off the ball, everything can race to the bottom probably pretty quickly. And that's a challenge.
Nathan Labenz: We've seen recently in our country how one administration to another can bring about dramatically different attitudes, personnel, and decision-making, right? So from one California AG to the next, I could imagine going from a great scenario where you've got the crack team doing exactly what you'd want them to do, to somebody that's just focused on other things, or even more problematically, prone to being lobbied-
Nathan Labenz: by companies. There's a whole political economy of how people are channeling messages. I don't need to tell you about the complications of the political economy of this. But is it right to say that in the SB 13 world, we're really putting a lot of trust into the AG?
Andrew Freedman: Yes, I think that's not only right, but it probably has to be changed a little bit to put more of a commission structure, some expertise into the government, to be able to say, "Have you rightly scoped your outcomes? Do you actually have the ability to track the outcomes that we say are important?" and so on, on an ongoing basis. I don't think it's right until there's a bit of a fear-of-God moment for these private-side regulators, that their ability to be a certifier can disappear very quickly. If there is a complaint that you bent your rules to make sure something went through, that can be fully investigated, and there's the money and resources to do so. Ideally, this gets passed in a couple of states or a couple of different governments, and there are multiple people looking after and giving the seal of approval. If one drops off, that's a signal. If state Z says, "Hey, something doesn't feel right here, so we are withdrawing the licensing of this private regulator," that should kick off others to start looking into it. And if it becomes an international group, you have countries also watching over and diving into this business. What I like about it is this layer of what the government should be doing: looking into these private-side regulators and really getting up in their business. Are they qualitatively and quantitatively showing that they're making a good difference in the world? If they aren't, there should be enough competition that we can withdraw from one group and give power to another, and we should be held accountable as lawmakers. At some point, it's turtles all the way down. There's a moment where you're trying to create the best scenario for government to hold these groups accountable. If the government is simply not interested in holding them accountable, redundancies aside, there's a problem there. I will point out that this exists with every regulatory structure. There's just a moment. I think what this does is really bring it out in the open and allow the public to see, "Okay, you guys have allowed these people to be certifiers for years. When they put out their numbers about how they create a better world, it is laughable. Meanwhile, look at this other certifier doing this other thing and actually creating a better world." So there's at least some way of maintaining that race to the top that includes public accountability. My last and final point is, I come from a world where I see all ceilings become floors, where the best-intended government regulation just becomes one more way of checking a box and avoiding things. I don't know of another structure that is more set up to do the opposite—to actually create a qualitative race to the top and continue to iterate that system as technology grows.
Jillian Hadfield: I think there's the reality of how you get to this model—the ideal that Jack and I were describing back in 2019 and have been talking about since. There's the sausage-making of doing it this way or that way, in this legislature or this process, and what we think is achievable today. But on the point about the race to the bottom, this system is only as good as the government's capacity to have oversight of these private actors, whatever category we're putting them in: regulatory services providers, independent verification organizations. That government regulation has to have teeth, just as our existing regulation is only as good as our government's capacity—the IRS, the FDA, or the Securities and Exchange Commission—to create good rules and enforce them. I always like to think of this proposal, in the grander scheme of things, as shifting government effort and expertise to the task of overseeing these private regulatory bodies. I think of that as a pretty muscular thing. The model is only as good as your capacity to do that. That's your backstop against a race to the bottom. That's your backstop against, "Come on over here, you're not going to have to do very much to comply with my system." The key design feature is how you address that. Of course, regulatory capture is a problem throughout our regulatory system. The whole term is based on the idea of corporations capturing government. We always need to be comparing this proposal and what we think we could achieve if we put the resources into it and got the design right, relative to what we can achieve in its absence. What's being proposed here are real methods of getting that kind of appropriate and effective government oversight. Going back to the conversation about the various domains here, the expertise you'll need in government to oversee autonomous vehicles will be different than the expertise to oversee companion AI, bio-weapons risk, or financial stability risk. That's the complex regulatory regime we're in. We're just trying to change the role of government in it, but I don't think anybody should think this is a one-and-done, where we just check if you filled out the right forms, have shown us something that looks plausible, and then we'll leave you to it. No, this is a way of actually getting us away from the world we are currently in with AI governance, which is defaulting to corporate oversight—essentially self-governance. We're saying, "We have no idea what you're doing, so the labs will tell us what red teaming tests to do, and we'll have limited visibility into that." Or we're going to kick it over to industry standard-setting bodies, which are corporate-funded with many participants from there. Government has basically been defaulting on its central role in saying, "This is what we want from these domains." A reason for that is because it's technically so challenging. So this is a proposal that's trying to deal with the technical challenge without giving up on—in fact, making more muscular—the democratic role for governments. Governments should be telling us, "Here's how much risk we're willing to take." We're not there right now. At the end of the day, the protection against a race to the bottom is having the government actually regulate. It's just regulating in a different way than it conventionally does and which it's actually not able to do effectively right now. So, as you said, Nathan, are we kind of naked on this? I think that's right. I think that's right.
Nathan Labenz: Yeah. Or another way to put it—and he wasn't talking about this at the time—but friend of the show and research partner of Fathom, Dean Ball, once simply put it to me, "Republics require virtue." I think that is a good reminder that you can always poke a hole and say, "Well, what if the person in that seat is bad, and who's going to monitor the monitors?" But at some point, this is an institution that is going to be populated by people, at least until there are maybe some AIs taking over—
Jillian Hadfield: Yeah, dude.
Nathan Labenz: key roles. We've mentioned—
Jillian Hadfield: Regulated AIs taking—
Nathan Labenz: Future speculation.
Jillian Hadfield: That's right. Sure.
Nathan Labenz: But yeah, somebody's got to actually be trying to do a good job at some point in any given system, or it's going to go to hell. So there's no way—
Jillian Hadfield: Yeah.
Nathan Labenz: ...around that.
Jillian Hadfield: I think so. But republics require virtue, okay? They also require visibility. I think this is one of the things we are in a serious state with. This is pretty much the first time in history we've seen such a massively consequential technology with a general-purpose capacity built almost entirely inside private technology companies. These companies have a legally created, fictional ring around them that says anything that happens inside stays inside and doesn't get out unless they choose to let it out, or the government intervenes and says, "You must let us look." Right now, I think governments are in an impossible position to effectively regulate because they lack visibility. So another feature of this is that we need to increase visibility into what's happening. Who is our partner in that? An independent sector of entities that are in the weeds. We already have some of these companies and nonprofits starting to emerge, providing red teaming services, or checking on and developing technology to check the robustness of systems, or hallucinations in systems. We really want to lean into those startups, that sector. Let's make this a powerful sector. Let's create increased market demand for that. Let's attract investment into this. That's the partner for government that gives increased visibility for government into what is going on, because right now, governments are kind of at the mercy of what the labs have chosen to share with us. I'm not criticizing the labs. If you structure them as corporations, and those are the protections we give corporations, then that's the way they're going to behave. I'm an economist. They will engage in profit-maximizing behavior. That's what gets us this technology in the first place. But it runs headlong into what we need for regulations. So, virtue and visibility.
Nathan Labenz: So that seems like a different law, though, right? Because we couldn't expect people to be... Maybe we could try to incentivize them to, but if we really want visibility, we might just have to mandate it. Do you have thoughts on how visibility should be mandated? There's a connection there to whistleblower protections as well. I was also going to ask about the mechanism of how you think money should flow in this system, so I'm...
Jillian Hadfield: Yeah.
Nathan Labenz: ...rapid-firing questions at you, but...
Jillian Hadfield: Yeah.
Nathan Labenz: Mm-hmm.
Jillian Hadfield: Let me address this, and then, Andrew, I know you have something to say, so I'll come to you. First of all, Jack and I talked about this in his 2019 paper. There's... If you have a private entity that you've contracted with to provide regulatory services, to give you that oversight, then yes, in my vision, dollars are moving. You need to get dollars into the business. I've been chatting with some of the nonprofits that are engaged in doing red teaming under contract for the labs. And, no surprise, they're finding they need more resources. It's a bigger job than a few very virtuous people can do. You need to get dollars into that. So that is part of the flow, and attracting investment in this is one of the top reasons to do it. Then, there's the fact that I actually think you will be able to get much more fine-grained information transfer between two private entities under contract. We see companies engaging in joint ventures and collaborations, and they share detailed information within certain ranges because they have confidence that their confidentiality and IP protections will keep that information private, and they're not sharing it with the government. So I think you will see more visibility going into a private regulator than into the public regulator. But then the public regulator can set whatever standards it wants for its oversight of those private regulatory agencies and say, "Okay, you need to show us your stuff. You have to show us the results of what you've been learning." The government may not end up getting into the weeds on all the information from the labs, but I think you start structuring those information releases. So I think there are ways to improve visibility, and you increase visibility because you've harnessed the incentive of this independent sector to say, "Here is what I need to know, and if you want to be certified by me, you're going to have to share that information. Here's what I need to know to be able to fulfill my duty to the government to demonstrate that my approach, my technology, achieves your government goal, that target for regulation."
Nathan Labenz: Here's a series of questions that I've either gathered or had posed to me by others. I recently had Matt Perault, who's the head of AI policy at A16Z on. He is very focused, and A16Z is very focused, on advocating for little tech and trying to make sure that there's a place for startups. His concern about SBA13 in particular, and I think probably it would abstract to the more general concept,
Jillian Hadfield: That's important.
Nathan Labenz: is what if the rules become so onerous that only the big tech companies can comply? Then the big tech companies get the benefits, and the startups can't enter that beneficial regime. It then becomes very hard for them to compete or raise capital because they're on a disadvantaged legal basis compared to the big tech incumbents. One answer might be, if that's the way it plays out, so be it, or that's a cost worth paying. But I don't know if you want to bite that bullet, or if you think there's a way to
Andrew Freedman: No, I definitely don't want to bite that bullet.
Nathan Labenz: have the best of both worlds.
Andrew Freedman: I honestly don't know of a structure that can scale more to provide solutions that are different for little tech than are available for the frontier labs. Meaning, you could well envision, and perhaps some guiding language within an 813 would be helpful here, that there is a specialty lane for models or applications or deployers that are smaller scale and pose a less immediate risk in these ways. They would have a much lower burden for what it shows to meet best practices in their environment, and therefore, they should still be able to get whatever seal of approval the system ends up creating and being watched after. But it is of a level that makes sense for them, either because they're selling to enterprise or because they believe they introduce some risk into the environment that they need to look after. This is instead of it being, say, like a SOC 2, where it's the same for everyone across and everyone has to comply. There suddenly is some way of actually creating gradation and saying, "This is what the best practices should look like for a 10-developer group looking for a limited application that operates this way." It's just so much different than, "You're going to be in 10,000 vehicles tomorrow, and we need to make sure that you know how to obey traffic signs or stop before a little girl drops a ball in the street." The problem with a top-down approach is there's no way to account for that. You can try to write it into legislation and bifurcate it today, but whatever you've done, whether it's by FLOPS or whatever, it's not going to make any sense tomorrow. Whereas, groups that are specifically looking to meet this moment where it's at can really change and be flexible to that moment. I would also argue for the little tech world that they are in an impossible position with a no-regulation world, because the only people who can prove that they're going to be safe for a fintech to come in are the big players right now. They're the only ones that can go and do the amount of independent certification, work, and long-standing work. A small company deciding they have a way to forever change the banking industry at this moment has no way of proving to the banking industry that their stuff should be trusted. So, I would argue that there's a way of creating this that is actually of massive value to little tech. I'd also argue that the other alternatives are not going to be able to become as bespoke towards the needs of little tech as a solution like this could become.
Jillian Hadfield: This is actually at the very heart of what, for me, has been driving my thinking about how we get more markets into solving our regulatory problems for decades, frankly. Because markets have the capacity to be differentiated. We've got cars for your middle-class worker, and fancy cars for your executives. You get differentiation in markets, so I think that's a key feature of saying, "Let's try and unleash some market effort here." If you have venture money that says, "Hey, we really want to build the little tech world. Let's put money into building the right kind of regulatory infrastructure that serves that need." That's a regulatory puzzle: How do we do that efficiently? So I think that's a key reason for trying to recruit more markets, regulated, overseen markets. Don't lose sight of the fact that we're not just abandoning it to the private sector; it's only with muscular government oversight. But I think the other thing is, when I started thinking about this set of ideas, I was fundamentally driven by the fact that our legal and regulatory systems have become far too expensive, slow, and onerous. We've leaned into a set of techniques for regulation that, on the one hand, are very expensive to comply with, with lots and lots of process-based stuff, front-end process based, with very little demonstration that those process-based protections actually achieve what you're looking for. In some ways, the very problem you're trying to solve is that we've built an incredibly expensive regulatory regime that only our biggest companies can really afford to comply with, which is a massive drag on startup sector innovation. That's a key reason that we need to be adapting and innovating in our regulatory methods. So if you think about the General Data Protection Regulation, for example, GDPR in the EU, lots of process, this definition, that definition, have you got these logs, etcetera. In fact, there's pressure right now in the EU to say, "How can we modify this, because it is too much of a drag on the innovative startup sector?" So I actually think a mechanism like this, its goal is to say, "How do we build more efficient, more effective regulatory regimes and move away from the top-down thing that, frankly, lawyers in a room are going to create?" Or, I criticize lawyers a lot in my book, or at least the way in which our profession has failed to rise to society's need for greater innovation in what we produce. We do not need more words on paper. We need more smart approaches for regulation. How are we going to get there? That's what we're trying to do. That's the path we're trying to set ourselves on. There will be a lot of hiccups, bumps, wrong turns, and dead ends, but I think it's absolutely critical that we make this shift. We should have been making it 10 years ago. The only thing that's happening is AI is ramping up faster and faster, and we are still like we have our shoelaces tied at the starting line. We are not getting there, and we need to get there.
Nathan Labenz: I love that this proposal creates an opportunity for people to come up with new ideas and enter the ecosystem on an ongoing basis. Also, while this may still need to be worked out in SBA 13 and in general, there seems to be an implicit, if not an actual, sunset clause, which is something I always advocate for in law that never seems to happen. There's at least an ongoing renegotiation or reevaluation of many things in this proposal, which I think is great because it gives it a decent chance to age well. That's my constant joke about AI content: it does not age well. AI regulatory proposals generally do not age well, but this meta-structure that allows for new entrants and ongoing revision seems to have a better chance of aging well than anything else I've heard. Here are two more different angles for red-teaming the proposal. One, which is perhaps the most different or almost in the opposite direction, although you may see it differently, is from another law professor, Gabe Weil. As I'm sure you're aware, he has been advocating for an expanded notion of liability. I haven't studied his work in depth yet; I'm going to do an episode with him soon. But the general sketch is that some potential harms—catastrophic or existential in some cases—are so bad that we need a way to deal with them before they happen. His proposal is to expand liability to encompass near misses. So if you were acting negligently and nothing bad happened, but it came close or could have, you could still be sued and held liable for harm, even though you got lucky. Being mindful that I probably haven't described his position quite right because I haven't done the full study yet, do you have any reactions? It seems like we probably can't do both of those things; at a minimum, they seem like two very different directions.
Jillian Hadfield: Going back to the idea that our regulatory ecosystem is a whole bunch of different threads, liability—by which people usually mean litigation-based, court-based regulation—is always after the fact. It's a big process, but it has lots of virtues because it can be bottom-up and reactive. That's a good thing about having a strong litigation regime. I'm not anti-litigation or anti-courts as tools in our toolbox for getting people and companies to do the right thing. So one approach would be to say, 'We don't want to wait for catastrophic harms to happen. We should cover near misses.' Maybe that's a fine amendment to make to tort law. I'm not enough of a tort scholar to know our existing doctrines on how close you have to be. Can you just create a risk, or do you have to cause actual harm? I don't want to go into the details on that. But the fact is, these are the types of domains where we don't generally leave it to litigation. We don't say we're going to rely on the tort system to handle the risk of nuclear facilities blowing up or creating fallout for communities. Think about pharmaceuticals. We started early with pharma. We have an FDA that says you cannot put a drug on the market unless you've demonstrated safety and efficacy and gone through a fairly lengthy regulatory approval process. You still have a backup of liability; you can still sue for harms caused by FDA-approved drugs on the market, but we haven't left the whole thing to litigation. If we're thinking in the domain of catastrophic risk—which I think should include things like collapsing our markets, bungling our financial trading systems, and other economic stability risks we don't pay enough attention to—I wouldn't be heavily focused on dealing with it by making tweaks to the tort law regime. Maybe we make those tweaks as well, but my first line of defense would be to have some oversight on whether there's a bioweapons risk or if these trading agents could cause the equivalent of massive bank runs or stock market crashes. Those are very costly things, and I don't think we want to handle all of that through backend litigation.
Nathan Labenz: Yeah, I...
Andrew Freedman: I'd hate to beat up a straw man because I don't know enough. One of the things I will say is also be careful of the downside of that stuff, which is how much is that just going to mean that near misses are not reported out? How much of that is just going to mean that you have to create your corporation now in such a way that nobody knows the full picture other than a few trusted people and everybody's in their own little silo? I imagine a lot of tech organizations are already trending in that direction. It takes one...
Nathan Labenz: Trending in that direction.
Andrew Freedman: It takes one... Yeah. This is really can push towards much more siloing and if we didn't know then there's nothing you can sue us on because we didn't know that there was a near miss there. And you're asking people to not go do the red teams and do the hard work because the more they know, the more they're potentially liable on the back end. So there's some downside that I do want to be... Now maybe that's thought about in this proposal, so I don't want to... But I'd also argue I really do hope we end up seeing things that allow people to be proactive in this space, and get rewarded for being proactive, and not just like after the world's burned there's a way to go and sue for it. And so I hope that is brought into that strategy as well.
Nathan Labenz: Yeah. Well, stay tuned everybody for another episode where we'll get the full story on that and I can be properly educated and... It does even strike me just in listening to your responses there though that I was maybe too quick to just see liability protection and liability expansion and see those as incompatible because plausibly you could expand but then you could also afford some protection for compliance and it doesn't necessarily in the end seem like those are so diametrically opposed. Okay. A third one is, and this one is maybe closest to your own impulses. Why not require insurance and put everything on the dollar scale? Insurance companies are quite, they're presumably the best organizations we have for calibrating to risk. There would, that would also bring a sort of pricing mechanism to the risk that I don't see quite... Maybe you'd see a way that it emerges from the sort of structure that we've been describing, but it's not... I mean, with insurance it's quite clear how the pricing mechanism works. So yeah, instead of this whole thing...
Andrew Freedman: If...
Nathan Labenz: Why not just say...
Andrew Freedman: Yep.
Nathan Labenz: Everybody's got to have insurance. If you don't have insurance, you can't drive?
Andrew Freedman: So I think, I won't speak for Gillian here, but I will say for me, I've been most bothered in my nerdy capacity by how we've waved a magic wand with insurance and said that it can solve problems without diving into what it is that insurance does in order to solve those problems. And so, for the insurance market to help be a rational market here that does actually properly price risk and then properly charge against that risk, that includes it being able to properly know what risks exist and how to best mitigate risk. And that doesn't exist absent valid third party certifiers going in and doing that work and racing towards it. And so I would actually say the scaffolding... It's easy enough to say "Insurance knows how to price risk." I would argue they probably don't here. Not even that they probably don't. They absolutely do not know how to price risk here. And so you could require them to come into the market and they're like, "Great. We're going to... Everything has to be self-insured and it's an astronomical amount and we don't really know, we're not any better than anybody else right now at coming in and doing that." That there has to be some scaffolding of information, of knowledge base of what does actually decrease risk in this system and how do we actually know risk that comes from a valid source in order for there to be a rational insurance market.
Nathan Labenz: I think people would just...
Jillian Hadfield: Yeah, I...
Nathan Labenz: Argue, though... Oh, sorry.
Jillian Hadfield: Oh, sorry. Let me just add on that. So yeah. So emphasize insurance, you need to price risk, so you need lots of structure that is determining risk and when insurance companies are insuring against liability risk for automobile accidents, or construction work site accidents, or let's go back to pharma, they're doing that against a backdrop of a ton of structure that defines the risk. We've got lots of history of automobile litigation and liability. We have tons of codes that govern how you run a construction site. We've got all this regulation around drugs. So insurance has got a lot of structure to price on the basis of and we don't have that right now. I think there's this idea that insurance companies because they do risk will come in and say, "Oh," and they will magically solve the problem of where is the risk and what should've been done. But notice that we've just reinvented the problem, which is what do the companies need to do? We don't know what that is right now. And I do not think insurance companies are going to become our AI regulators. I mean, that also would... But again, we would not have any oversight of that. That would be insurance companies saying, "How much risk are we going to allow for bio risk or whatever?" They have to build on something. My institute that I ran at University of Toronto up until last year just released a report. We had lots of discussion about insurance. Insurance is a nice complement to building a regulatory market and building regulatory technologies. It's actually another one of the carrots that you can use for, "Hey, if you implement this regulatory method..." then you can get insurance. But it's got to be that kind of partnership, and that's the kind of products that we're starting to see emerge. So Armilla and Lloyd's, Armilla is a company that creates this regulatory technology. It's been one of my go-to examples of a startup in this domain. And through a partnership, they said, "Oh, if you use our approaches, we now have an arrangement through Lloyds of London, I think, or other insurers as well. You can get insurance." But the insurance companies aren't going to become our AI regulators and it wouldn't be appropriate for them to be that. So I think just a basic, just mandate insurance and it'll all work its way out, I think just is not a realistic view of the way insurance markets work, regulation works, or democracy works.
Nathan Labenz: The democracy point is strong. At the end there, though, you were getting to what I think the advocates for the insurance idea would say, which is: first, this market will be massive. This is not a niche corner of the insurance market that insurers would dismiss, saying, "It's not worth our time to figure that out." It seems if you're an insurance company and AI is happening, touching everything, and the risk is massive, you would presumably
Jillian Hadfield: But the risk of what?
Nathan Labenz: ...want to play and then-
Jillian Hadfield: The risk of what? What
Nathan Labenz: Right.
Jillian Hadfield: ...do you insure against? You insure against liability risk, which means you think that courts will be able to impose requirements and standards, or your risk of compliance, regulatory compliance risk. But then that requires government to have a regulatory structure. So we can have bad things happening, but that does not mean risk for the companies unless they have liability attached to that, either tort liability or regulatory compliance liability. So it's not a massive market unless there are requirements they will be held to, either through the courts or through government.
Nathan Labenz: The notion is mostly liability risk, and also that many of these same organizations are ones insurance companies would turn to for help getting a handle on it. So it would be... And I am interested to know, for the future of Fathom, does Fathom envision itself being one of these private regulator entities? What other organizations are you looking at today and saying, "These guys seem like they could step up into this role"? And then, in the insurance-
Jillian Hadfield: We
Andrew Freedman: Yeah.
Nathan Labenz: ...context, those same candidates would perhaps be the ones the insurance companies would go to and say, "Hey, we need... we'll pay you to help us figure this out, and we might insist on companies going through your audit process or whatever if they want to buy the insurance." The hope is actually to end up in a pretty similar spot where experts are
Andrew Freedman: Yeah.
Nathan Labenz: ...defining standards and also conducting audits. But it's less concentrated through one AG or one commission and more concentrated in this global insurance market, which in theory at least has a lot of skin in the game.
Andrew Freedman: I would love to answer the Fathom question first. Fathom is interested in showing proofs of concept here. We think this is a novel enough idea, and the way the marketplace starts will be an important factor for success. So we would love to see some proofs of concept out and then help those who are very interested in doing this work be very successful. We remain a nonprofit. Any interest we have is nonprofit-related and philanthropically funded, and there is no end game here for there being an equity play or any of that. But I do think it is one of those things that if we don't show what good looks like, people are just going to constantly be like, "I don't have time to listen to a two-hour podcast right now on this." So please-
Nathan Labenz: I don't know who has time to listen to us honestly. It's great.
Andrew Freedman: I listen to it. But yeah. We do need to start showing also how complicated it is, and we do want to work out the kinks by having real-world examples. So I think in the near future you will see us trying to show what proofs of concept we are looking for, working with partners who are actually the technical people here. I don't know if I have permission to share the technical people that we are giving grants to to see them do this work. But I think in the near future it will be clear that we think there are some great technical minds working on this right now, and that we are giving small grants to try to see them do this work well. I will just say again, the insurance company is making a bet in this world at the end of the day. If they are making a bet based on no different risks, you could easily imagine a world where they say, "Okay, well, that's such a big catastrophic risk that if it happens, there's no one to sue on the back end. So we'll insure against that risk because the chances that we have to actually be there at the end of the day to pay it out are actually very low." Society will have fallen apart, for example. There just is not, I don't think there's a magic to the governance of how insurance works that they're going to be able to bring in the best third-party validators of risk and have them do the best work. Their focus is going to be on, "What's the greatest tangible next risk that's coming up that we could actually be on the hook for here?" If they can't do it, and they're taking a guess, they're taking the same guess as everybody else. So I get it, it feels like a thing that once you start mandating it, the scaffolding will fall into place. But I actually think thinking through how that scaffolding is actually governed so that we are actually creating the best third-party certifiers is the most important question. And making sure that those people are actually accountable to society rather than to any perverse economic motive on the back end.
Nathan Labenz: Do you want to offer any closing thoughts?
Jillian Hadfield: We need to get moving and be innovative. We want as many people in the conversation as possible, poking at the model and coming up with new ideas. The reason you like markets is because you need lots of different minds, perspectives, and knowledge. We need that conversation to happen, and we need to get going. We can't stand back and say, "Let's design this perfect structure." We need the MVP of new approaches to regulation, so let's get started. Whatever form SB 1044 ends up in, it will continue to evolve. The key thing is we need to get down this pathway. So the key message I would leave you with is that markets can help us address this, but they need to be overseen by governments. Governments should be deciding what's acceptable risk, and we need to get going.
Nathan Labenz: Perfect.
Jillian Hadfield: Okay.
Nathan Labenz: Dr. Gillian Hadfield and Andrew Friedman, thank you both for being part of the Cognitive Revolution.
Andrew Freedman: It was such a pleasure. Thank you for having us.
Jillian Hadfield: It was terrific. Thanks.