Photo-Illustration: Intelligencer; Photo: Getty Images
It has been a momentous week for AI in the military. First, the Pentagon announced it would soon cut its ties with Anthropic and its leading model, Claude — and, in an extreme tactic, label the company a “supply chain risk” — after negotiations broke down over Anthropic’s condition that its systems not be used for autonomous warfare or mass surveillance. Hours later, OpenAI, sensing an opportunity, struck a deal with the Pentagon — though it claimed to retain the same carve outs Anthropic wanted. Then, on Saturday, the U.S. attacked Iran in a broad and ongoing campaign that killed Ayatollah Ali Khamenei and many other top officials. Per The Wall Street Journal, U.S. Central Command has been using Claude during the operation.
To get a better understanding of how the U.S. military is actually employing AI tools in warfare now, I talked with Emelia Probasco, a senior fellow at Georgetown’s Center for Security and Emerging Technology. Probasco, a former surface warfare officer in the U.S. Navy, leads the center’s research team on the application of artificial intelligence and machine learning to national security challenges. We spoke just before the Anthropic/Claude deal fell apart and the morning before bombs began falling in Tehran.
You were quoted in the New York Times as saying that the Pentagon/Anthropic deal “needs” to happen. I was wondering why you were so forceful on this point — what did you think was so essential about this particular partnership?
Anthropic had been the only company with a large language model operating on the classified networks. And most of what the military needs to do on a day-to-day basis is happening at a classified or above level. So removing that one tool right now, when — as anyone can see, there are operations going on around the world — it’s not a great time. It’s never going to be a good time, but people are using it now and it would just be very disruptive to what they’re trying to do.
What do you make of the conflict between Anthropic and the Pentagon? Defense Secretary Pete Hegseth went off on CEO Dario Amodei in harsh and personal terms.
Let me talk about what I think matters on this issue beyond the media and the back and forth. The military has a difficult job to do, and there are lots of operators, good Americans, who are trying to live up to American ideals and what we expect of the military. And they want to use these tools for safe applications. On the other hand, the AI companies have an exquisite knowledge of the technology they have developed, and they know what it’s great at, and they also know that it’s just not perfect. They’re voicing realistic and well-earned concerns about the limitations of the technology. And I think sides want to support national security; they just come at it from different perspectives. And that’s why I wish they would keep talking, keep working together to find where the common ground is and better articulate where there are real differences and how to resolve them. One of the things that has gotten pushed aside in this bigger conversation about the company and the Pentagon is that if you look at the contractual terms, you would say, “Gosh, they’re pretty close.” The Pentagon was saying there would be no unlawful use of the technology, and Anthropic was asking for no mass surveillance and autonomous weapons.
There are laws on the books that deal with mass surveillance and population surveillance and autonomous weapons, but there’s some nuance there that is still unaddressed.
There is a narrow but important gap between the Department’s “all lawful use” stipulation in contracts and Anthropic’s preferred “no autonomous weapons.” On the one hand, you could interpret these two positions as being essentially aligned. But autonomous weapons are not covered in U.S. law in the way you may think. The only requirement in law is that the Department notify Congress if it decides to change the autonomous weapons policy, which currently requires that any autonomous system “allow for appropriate levels of human judgement. So the Department could change its policy back and forth as it desires without the need for Congressional approval or review.
There’s a world in which a conversation would be happening in Congress about whether our laws are still sufficient to the time we are in. Some of that has been getting lost.
We hear a lot about how AI may revolutionize, or perhaps even already is revolutionizing, warfare. In Ukraine, for example, autonomous drones have become a hugely important part of the battlefield. But I read these stories about AI companies in our military, and I don’t really have a good understanding of what they’re actually doing.
That’s kind of the critical question. A lot of people don’t understand this point, and my fear is that this current debate is going to just sort of make people unnecessarily scared about what’s happening. There are lots of different applications of AI. There are also lots of different kinds of AI. Even if you look at Ukraine, there’s actually not that much AI being used — there are drones.
AI is also quite a vague, loosely defined term. But in this context, I think a lot of people take it to mean fully autonomous weapons — like in the Ukraine example, a drone that controls itself.
Here’s some background. When I was in the Navy, I sailed on an Aegis missile destroyer. That ship has two systems that you might call autonomous. One of them is like a little Gatling gun that will, when it’s turned on, shoot down any incoming missile — anything that’s coming at the ship at a certain speed. The rumor used to be that it would shoot at birds if they were coming in at the right speed. So that’s a weapon considered by many to be autonomous. The other one is the Aegis weapon system, which is a little bit like the Patriot missile batteries. And again, it’s a system that is designed for when another missile’s coming in and might hit you.
So how do we control those systems? There are a ton of rules around when you actually turn the thing on. And even more than that, there are technical constraints we put on them. So it has to be a missile that’s incoming; it can’t be something flying in the opposite direction. We have learned how to control technologies that have a certain level of autonomy and place them in applications that make sense. I can’t tell you the number of times I thought, thank God somebody invented this system, because I would otherwise have no way of defending myself against an incoming missile there. That’s a good application, in my mind. So that’s just to say people should understand that there’s nobody with a joystick that’s guiding missiles at the end of the flight. We’ve found ways to put in both process constraints and technical constraints to control systems that have a level of autonomy.
Ok, so the kind of Terminator doomsday scenario people have in mind is quite far from what you just described — even if some AI models do have a habit of recommending nuclear strikes.
It’s so far from that. I think the great thing about science fiction is that it gives us all a picture of what we don’t want. I just think it is very easy to be captured by these fantastic stories. But we’re talking about human beings. When I was a naval officer. I didn’t want a Terminator. I wanted a system that would protect me if I was shot.
But that’s you. Maybe someone out there does want something darker.
Yeah, but think about it. All officers go through a substantial amount of training. Now, this is not to say everybody’s perfect, but there’s a lot of vetting. You don’t just show up on day one, and they’re like, “Now you’re in charge of the weapons.” You have to go through years of general training. You have to go through training about the law. You have to go through training about how ships work, how missiles work, how that particular system works. And then in many cases, you actually need a supervisor, typically the captain of a unit or the commander of a unit, to give you the authority to use the weapon. So there are a lot of controls that are put on the humans that would operate these systems, and those humans have been trained and raised in a system that believes in things like accountability and proportionality.
Boy, I hope you’re right. Sometimes I doubt some of those assertions.
Well, let’s talk about your doubts for a second.
This administration has been carrying out strikes of dubious legality in Venezuela, in Iran, and of supposed drug boats in the Caribbean. Some military leaders have quit over them, but most haven’t. There’s people who either believe in it or want to follow orders. So I’m just accounting for human imperfections here.
Yes, you should account for humanity’s imperfections. But those decisions were political decisions made by political leaders. And this is one thing I think a lot of people miss: they’re like “Don’t obey any illegal order.” But there aren’t that many situations where it’s clearly black and white to a military officer that an order is illegal.
To go back to how AI is actually being deployed in the military now, I had read that Claude was involved somehow in the Maduro operation in January. I was trying to figure out how, which is a murky question.
I don’t know — I’m sure it wasn’t reported because it’s classified. First of all, there are not that many applications of a large language model. Widespread use of LLMs — we’re not quite there yet. We’re still getting comfortable understanding the appropriate deployments. That said, what are people playing with? Well, one thing in the military is that you have to summarize what you did every day. It’s a very standard procedure. I don’t have special knowledge here, but you could imagine that if I had to write a report to my boss every day, AI might help me summarize some of my tests. That sounds super boring, I recognize, but what do people use these tools for? We use it to write emails to our bosses. We use it to fill out standardized forms. We use it to summarize really long reports. The military has to do all those tasks as well.
Palantir used a large language model to do what’s called “foreign disclosure.” Whenever you operate with another country’s military, you can’t share all of the classified information with them, but you can typically share some parts, based on the guidance of your chain of command. You can share how many planes we have, but you can’t share the name of the unit — that’s one of the typical restrictions. So you can train an LLM to take intel reports and strip out the names of the units. And then a human reviews it. That, again, is a pretty non-scary application.
There were reports a couple years ago that Israel was using an AI tool called Lavender, which identified human Hamas targets in Gaza in a much less discriminating way than a human commander probably would, though a human did have to sign off on its choices. Is that something you worry about the U.S. military employing?
I followed the story and it took a while to get what sounded like something that made sense to me technically, so just be careful when you’re trying to characterize it. My understanding is it was gathering data on people and putting it all in one place so that you could more easily connect the dots. That’s very different from you saying “Find me all the terrorists” and it spits out a list. It’s a really hard system to explain except to say, and I think most people would agree here, that the technical capabilities of the system are probably not the thing to focus on. It was more the rules of engagement around what was viewed as acceptable.
The Wall Street Journal reported last week that government agencies had raised alarms about the Grok chatbot, which the Pentagon uses, over concerns that it is overly sycophantic or presenting biased data. You just gave all these case uses about summarizing emails and stuff like that, and it doesn’t sound like a sycophantic chatbot would be so bad for that. So why were those agencies so concerned?
There’s a point at which you just want the most performant model. And if the model seems to be spending more time pleasing you than answering your question, I could imagine that would be pretty annoying. The sycophancy could be problematic if you’re trying to do brainstorming tasks with an AI. So imagine you’re trying to understand a couple of intelligence reports, and you type into your chatbot — Here are my three intelligence reports. Here are my five conclusions. Help me red team these conclusions. This is another pretty standard application of AI ,where you have it help you challenge your idea. If you had an AI that was sycophantic and came back and said, “Oh my gosh, that is just the best idea ever,” and then it allowed you to keep iterating on ideas while it was purely just trying to please you — that’s not helpful. If you’ve ever been in a sycophantic conversation with an AI, it’s pretty annoying. And military people tend to not have a ton of tolerance for things that, as they would say, blow smoke.
These are all interesting, difficult, technical questions that everybody’s working through right now. At the same time, we’re trying to work through really difficult operational questions, and so in some regards, it’s no wonder that this has become a big conversation. It’s just unfortunate that it is not so much about real issues and a real way forward.
This interview has been edited for length and clarity.
