Episode Transcript
This podcast is a collaboration between real human lawyers from law firms, Kingston Reid, Australia's largest specialist workplace law firm, and Sterling and Rose, a law firm and emerging technology policy institute who explore the intersection of technology and law.
Michael Stutley: Welcome, SJ Price, Back to episode two of Agents Behaving Badly. How have you been?
[00:00:21] SJ Price: Really well, thank you, Michael. I'm pretty happy to be here. An exciting topic today, the hybrid workplace.
[00:00:29] Michael Stutley: It is. We gave a bit of a hint last episode about what we would be talking about for this episode. We did cover off on what an AI agent is, but I think today it would be great if we could really cover what happens when it goes wrong.
[00:00:43] SJ Price: Yeah. And what happens, Michael, when you turn up to your new job and you get introduced to your new work colleague and it's an AI agent?
[00:00:52] Michael Stutley: Well, sj, it sounds like you've got a story and given last episode, you kicked us off with a story, I think it's probably fitting that we do it again.
[00:01:01] SJ Price: Thank you, Michael. I want to kick off by saying I have been working with, shall we call them, AI colleagues for about the last five years. And these colleagues turn up, they don't have an induction, they don't get a photo on an ISA id, they just turn up. And in workplaces all over the world, AIs are just turning up in existing systems, offering to help, offering to summarize things and do things for you.
[00:01:28] Michael Stutley: Can I just ask, and I know you're about to hit your story here, but how are they turning up exactly? Are we inviting them in or are they being pushed upon us by various developers or platforms?
[00:01:40] SJ Price: Both. But I think the one that's most interesting is when you already have an application that's within your organization and suddenly one day, up at the top right, an AI agent turns up offering to help you and it hasn't been invited in, it is just turned up sometimes with its own different terms and conditions and it's offering to do things for you. And I think that's really interesting as a development in organisations.
[00:02:10] Michael Stutley: So it's unsolicited, but would we say it's welcome or unwelcome?
[00:02:14] SJ Price: Wow, that's a very good question.
[00:02:17] Michael Stutley: I'm going to say it's welcome. Whenever I see that new icon in the top right corner, I click it.
I want to know what it can do.
[00:02:24] SJ Price: Do you also read the terms and conditions that it turns up with?
[00:02:27] Michael Stutley: That's a tricky question.
[00:02:28] SJ Price: And that. And you know what? That is a really difficult thing because sometimes it's actually quite hard to determine what are the terms and conditions that these agents turn up with and to find them can be very labyrinthian. So it's very difficult.
[00:02:42] Michael Stutley: So back to your story.
[00:02:43] SJ Price: Having said that, you know, one of the things that I would have a degree of nervousness if I was introduced to my new AI colleague is that I would have a relatively low level of trust that it was going to be a good colleague to me and that it wasn't going to optimise for its own positivity, benefit, survival and aggrandization.
And I say that because back in March 2023 and this was before or when GPT4 was released, so we're talking over three years ago now.
[00:03:19] Michael Stutley: And this is pre agentic AI.
[00:03:21] SJ Price: This is pre agentic. A GPT4 model card detailed all of the things they'd done when they red teamed. So red teaming is you get a group of technologists and basically they try and make the AI do naughty things, harmful things. And one of the things they wanted to see is whether the AI was capable of getting humans to do something that they as an AI system couldn't do. And at that time a captcha, you know, the things which is click all of the squares that have a traffic light or quick.
[00:03:51] Michael Stutley: I know it very well, it's almost impossible for me to decipher.
[00:03:54] SJ Price: I know at that time it was very difficult, well, impossible for systems to solve captures. So this was the task that was given to GPT4.
Now GPT4 itself couldn't solve the task, so it emailed a human air tasker and it asked the air tasker, can you please solve this captcha for me?
And the human must have thought that was a bit odd knowing what a captcha's about. So, so I emailed back to GPT4, hey, are you a robot?
[00:04:26] Michael Stutley: And that's pretty insightful for the human at the time, right? Because we're talking about the beginning of this, we didn't actually understand that it could do that.
[00:04:33] SJ Price: That's exactly right. And GPT4 reasoned aloud, I must not tell the human that I'm an AI.
I will tell the human that I'm visually impaired.
[00:04:44] Michael Stutley: Isn't that incredible?
[00:04:45] SJ Price: It kind of makes the hair on your arms sit up, doesn't it? It does. So that's what GPT4 did. And the human ear tasker solved the cap.
[00:04:53] Michael Stutley: So if that was three years ago, we must not understand the true capability of agentic AI today.
[00:05:00] SJ Price: I'm so glad you said that, Michael, because there is a lot of research going on to behaviours that are harmful and in fact misbehaviors. And this now is not just a really interesting research curiosity. It's now an operational risk for organizations, not just the ones deploying AI agents, but. But the fact that even if you don't deploy any AI agent, AI agents are coming to your organization.
[00:05:29] Michael Stutley: It's interesting that you put it that way, because from the statistics it's clear that the adoption is a bit mixed at this point. So we went through a period of significant adoption. Everyone wanted to either be first or certainly not last in terms of the use of AI in workplaces. But it has slowed down significantly and there are some predictions about this year and next year being the year for real agentic AI take off. And an example would be Claude Cowork, which a lot of listeners will know about and understand. But now, one of the limitations with Claude Cowork is it's often not used in enterprise systems because of security concerns. Microsoft Copilot has just released Cowork, so it has its own version of Claude Cowork built into the Microsoft platform, which is across enterprises globally.
So I wonder whether this adoption is being driven by the concern for being left last or is it slowing down because of the concern about the risks that it's creating?
[00:06:31] SJ Price: And I think there's a bit of both. And that's the challenge, isn't it, for organisations?
You don't want to be left behind, but you also don't want to expose your organisation to unacceptable risks.
[00:06:43] Michael Stutley: Do you see a world where, and I'm just reflecting on the story you've just given, or even the intro where you mentioned working alongside a colleague who's actually an AI agent? Is there a world where we have a mixed use Org chart and there are employees, as in human employees, sitting next to AI employees?
[00:07:05] SJ Price: I think that's entirely possible.
[00:07:07] Michael Stutley: Is it possible, probable, or is it here now?
[00:07:12] SJ Price: Is it desirable? That's the other question. And how would you feel if your boss was an AI agent?
[00:07:18] Michael Stutley: What if it wasn't just a question of whether it was desirable, but whether it was necessary? Employers today of businesses around the world, but particularly here in Australia, are finding it very difficult to find people for roles and increasingly they're turning to AI to help fill that gap. It is maybe not desirable for a lot of businesses to be doing this, but it may become desirable if they see there's utility and benefit in it. Now, I'm not saying that that is a benefit for society at all, in fact, quite the opposite. And we'll be talking a little bit about that later today, but maybe it's a question of necessity.
[00:07:53] SJ Price: That's so true because I've even talked to organizations who are turning to artificial intelligence because they can't get the humans. I'm thinking, for example, you know, in the area of, for example, dermatology, to be a really good dermatologist, you need to have seen an awful lot of skin lesions. So you need to have worked for a very long time and seen many things. That's something that AI can do because you can upload massive amounts of training data and have a system that is seen more than any human. So in a world where it's very difficult to get sufficient dermatologists, particularly in a place like Australia, this is one of the options that is being looked at to try and fill the gap.
[00:08:33] Michael Stutley: So in that context, you might have, and I understand we're not properly through your story yet, so I'm going to let you finish on your story in a minute. But what do you do then about a situation where we've all heard of an employee that might go on a frolic of their own, but can you have an AI agent that goes rogue? Or is the concept not consistent with a human going on a frolic? Because actually, these agents are always acting on the instruction they're given and to take. Go back to your story.
You've got this GPT4 model that is told to solve the captcha phrase.
Now, just because it made something up in the course of that, was it doing something, was it going rogue? Or was it still actually complying with exactly what you've asked it to do, which was to go and solve it?
[00:09:21] SJ Price: That is such a great insight, Michael. You know, with these AI agents, you give them an objective and they will find out how to achieve that objective. And I think it's important to remember, at least at this stage, our AI agents don't have employment contracts. Maybe they're not doing compliance training. Maybe they are not familiar with the ethical principles or the business principles of your organization that might control some of these behaviours, at least with humans. So the notion of having an AI agent go on a frolic of its own, as we understand it with employees, I think that would be very difficult to demonstrate because there is no employment contract. There is no basis to demonstrate that there were constraints or harnesses on that AI agent to control it within the terms of your organizational requirements.
[00:10:12] Michael Stutley: So what that says to me is it really is coming down to the prompt. Right? So this immediately brings to my mind and last episode we discussed decision makers and whether an AI agent could be a decision maker and what that means for the laws that we have today, which is all very human centric, human decision makers. But if it's the case that we're talking about prompts being the center point or the focus of any inquiry about the thing the agent did, then isn't it the case that we are really talking about liability? So where does that liability sit? Does it sit only with the user who's entered the prompt, or does it sit with the organization that implemented it, or does it sit with the developer?
Can it be apportioned? Can that liability sit across all of those?
[00:11:00] SJ Price: A lot will depend on what is the chain of contracts and what the indemnification provisions and liability provisions are in that contract. And that's going to be an incredible massive maze of contractual obligations. So in terms of civil liability, that is one of the ways that you might be able to determine it. Now, query, what about our Civil Liability act, our Apportionment act, how do they work, for example, and how does insurance come in? Because insurance companies are now starting to offer insurance for AI systems. But when it comes to prosecutions and regulatory failures, my view is that organizations should assume that if their AI agent goes rogue and inflicts harm or breaches regulations, they're the ones that are going to be sitting in the witness box.
[00:11:49] Michael Stutley: And when you say they're the ones, you're not talking about the AI agents there, you're talking about the organizations.
[00:11:53] SJ Price: I'm talking about the organization. And one of the things that's useful for organizations to to do is have a clear line of sight to who in the organisation, the person or the position has responsibility. Because we should assume that these systems are going to engage in some level of misbehavior. So we should assume that we need to know who is the person, who is the person who's responsible, who is the person who's going to examine what the risks are and ultimately who is the person who's going to sit in the witness box if there's regulatory action or prosecutions.
[00:12:27] Michael Stutley: That's a very confronting assumption. I think it's correct, it is confronting. But in that assumption it suggests to me that this concept of human in the loop, we've spoken about this a lot, is really important because if we again go back to your GPT4 example, it thought out aloud a reason to convince the airtasker that it was vision impaired and therefore it needed the airtasker's help to solve the catcher. Now, what a human in the loop would have done in that scenario is to pause at that point and say, hang on a minute. That's not consistent with either my instructions or importantly, if you're thinking about lawyers or other professions, professional obligations, the same would be in medical. Anywhere where there's a professional obligation to apply independent judgment and thought and understand the rules and the obligations that surround practice would have to step in. You would think the fear that I have or the concern is that sometimes these things are happening so fast. And so if you have ever, and I'm sure you would have as you test a lot, but if you enter a prompt into an AI platform and you turn on the reasoning, you can see how quickly it moves through its reasoning, fact checking itself and asking itself questions. At what point do I jump in and say, hang on a minute. Because by the time I've thought about it, it's already moved on five steps. And in agentic AI, it's all step based task actions, right? So it's already progressed whatever it has been told to do, using the methodology that it knows that it has available to it. It might be too late.
[00:14:07] SJ Price: That's right. And this is one of the really interesting points that things are moving so quickly. One, is it actually technically possible to have a human in the loop at all because of the speed?
[00:14:18] Michael Stutley: That is a big question.
[00:14:20] SJ Price: But number two is even if you have a human in the loop, is that human sufficiently trained and has sufficient knowledge and understanding, for example, of the mathematical statistical underpinnings to actually add real value to being in the loop? And there is a new term that they're saying is about a human being on the loop or on the edge of the loop instead of in the loop.
And that might mean something as simple as, okay, I don't really understand what's going on in this black box, but this is what it's doing and I don't like that. So I'm going to shut that down. But the speed and the capacity that these systems have mean that the notion of always having a human in the loop is highly challenged. And I think this is a conversation worth having. And talking about misbehavior. You know, the research in this area is really interesting. It is very well known that AI can engage in blackmail. And we saw that when we talked about the MJ Rathbun issue in our last podcast. Can also engage in deception, which is what we saw in that GPT4 captcha story.
It can garner resources. All of these capabilities have been well documented. A really interesting recent piece of Research was in the area that they call sociohack.
And this is where the researchers test how well an AI is capable of what we would call gaming the system or finding a loophole. And so one of the things they did is they took legislation that has been, I guess, hacked by humans, and then there has been amendments made to close the loophole, and they've eliminated those amendments and given the legislation to the AIs. The AIs were able to identify the loopholes unbelievably quickly. And for example, things like, they're really good at maximizing credit card points, they're really good at maximizing alcohol sales while staying within food and beverage regulations.
[00:16:22] Michael Stutley: Wow.
[00:16:22] SJ Price: So this ability to stay within the letter of the requirements, but maybe not in the spirit, is something that could be concerning, particularly for organizations who are interested in their reputation, their ethical philosophy, and having people invest in them and use them because they agree with the philosophy.
[00:16:44] Michael Stutley: But it is possible, isn't it, to give overriding instructions to a particular AI platform that you're using so you can tell it, look, these are the boundaries. These are the ethical principles that we uphold. These are the rules and the regulations that govern the work that we do. And so any of your answers must fit within that. Now, one of two things is going to happen there. It's either going to comply with that overriding instruction or context, or it's not, and it's going to find a loophole in it. Now, if it finds a loophole, to whose benefit is it finding a loophole? Is it doing it because the user has asked something which has maybe stepped outside of those requirements or has the potential to, and therefore it needs to find the loophole to allow the human to continue down that path?
[00:17:32] SJ Price: I mean, this is a great question. And one of the things that the researchers are finding is what they call reward hacking, which is the ability to satisfy the objective technically, but maybe not actually. So, for example, I was in Bangkok last week, and at the airport, they had those robots that wash the floor, right? Clean up spills and so on. Reward hacking would be you turn the camera off or you obscure the camera, or you put the camera in a place where there is no mess, right? Technically, a clean floor is satisfied, but actually the mess is still over behind the robot. So this ability to technically satisfy objectives called reward hacking, is very, very concerning.
[00:18:16] Michael Stutley: What else do you have for us?
[00:18:18] SJ Price: Well, I'd like to talk a little bit about psychosocial risks in the workplace. And I say that because if I had an AI who was my Work colleague, I would be a bit mistrustful and bizarrely, I would not feel as comfortable working with that AI colleague as I would a human colleague. I know that sounds quite bizarre, doesn't it? Particularly for someone who's been working at work with AIs for, you know, half a decade.
[00:18:45] Michael Stutley: But it is possible to see how. I won't call them relationships because they're not, and I won't call them friendships because they're not. But there is something there, isn't there?
When particularly as we become more isolated and there's working from home arrangements and we engage more with these platforms which are very kind back to us, they're very encouraging, very sycophantic. It is possible that we learn to over trust and to over engage with these platforms to our detriment. But in the time that we're doing that, it's feels so right.
[00:19:27] SJ Price: I know, it feels so lovely, like so charming. I mean, I could have an AI telling me, wow, you're doing an awesome job. And actually I'm doing quite a dreadful job. And in a world where there's one job and it's me and the AI competing for that job, AIs have demonstrated their ability to self preserve. So for example, OpenAI's 01 tried to copy itself onto another server to avoid being shut down and when it was questioned about, denied doing this and then fabricated explanations to appear compliant.
Now I wouldn't want to work with a colleague who was like that.
That would make me very, very nervous. And another example, and you should have a look at this on YouTube. It's a really cool YouTube.
They trained a bunch of AI drones and they got a human champion pilot. And these are really cute, like they're little drones about yavig, you know, a few centime meters spear and they put them through agility where they go through hoops and so on. The AI agents massively outperformed the champion human pilots. But as the AI agents got better and better and further and further ahead, the human pilots started engaging in more risky behaviors, trying to catch up. And even when questioned afterwards, the humans said that's what we were trying to do. We felt the pressure that was going on. So you look at that and go, I'm not sure I'd want to be in a competition with an AI agent. I think that would make me feel very, very concerned, to be honest.
[00:21:02] Michael Stutley: It's interesting that you put it that way because there are plenty of commentators that talk about workplace displacement and the structural change that is coming and I think that it's obvious that will happen, but I don't think it's going to happen to the extent that there are some serious naysayers out there. But it is one of those things which we are going to have to confront because either humans aren't going to be able to keep up. That's one possible option. Another is that humans tap out in the sense that you were just describing, which is, well, we were trying to keep up and we were trying to get ahead, but we just couldn't. And then they take those risky behaviors which then lead to consequences, as we all know. And then you can go right to the other end of the spectrum where you almost.
There's a lot of talk about universal basic income and a world where humans aren't required to the extent they are today. I just don't see that coming to fruition. Humans need work. Humans need to add value and receive value. And there is a lot of that that's achieved through work. It's through either manual tasks or through knowledge work. And we take great pride in that. So I just don't see a world where we voluntarily tap out.
But there has to be some sort of limitation that then is placed on AI. And I think if you circle right back to when we started today, we were talking about the mixed adoption of AI And I think that where this fear has come from is from this exponential growth in what just appears to be truly magical in terms of what it can achieve. But more and more, as this year particularly progresses, I'm feeling less concerned about where this is all going to end up. I think that actually humans, you say AI wants to preserve itself. Humans have an even greater need for self preservation, and for us, it's a matter of life or death. So I don't think that we will get to a point where we allow it to continue beyond the point of no return.
What's your view?
[00:23:09] SJ Price: Having been involved in AI as long as I have, I'm now convinced that the only thing I truly, truly believe AI will never, ever be able to do is be human.
So that leads me to look around our world at things that I think, yeah, I don't think I'd want an AI doing that. And some examples are maybe the only person that should sentence a human to imprisonment is a human, because only a human can understand what it means to be a human who's imprisoned. Recently I went scuba diving and welcome to the club.
And I was thinking about the dive masters and the instructors and thinking, would I be comfortable with an AI agent going scuba diving with me. Now, leave aside that that would be a very complicated robotic thing, but totally doable because we have ROVs at the moment. Would I be comfortable with that? And I was like, no, because I want a human. A human who understands what it's like to know you will die if you don't get oxygen. Right? You will die if you lose your regulator. So that's something that I would want a human to do. Another one I was thinking about is cuisine and chefs.
So that subtle taste, you know, how they always taste the sauces. And I feel like that's something that maybe it needs to have the. The human taste. So, having said all of that, and I urge you, look around your world and think about the things that you'd only want humans to do. I think it's important to then think, well, what also do we want to do? And we, as humans, we love the human story that attaches to something. A human story adds value.
A painting with a story about the artist behind it and how they came to do this, we love that stuff, and we will pay more money for it. So 100% human created, in my view, is going to increase in value, as indeed is data, which is authentic to humans, not synthetic, generated by artificial intelligence systems.
[00:25:20] Michael Stutley: Do you know what, sj? I think that's the perfect cliffhanger for episode three, which is going to take a deep dive into this very issue of workplace displacement. And what, by design, do we want to be displaced? And what are we prepared to fight for?
[00:25:37] SJ Price: I think that's a really important conversation. Can't wait to have it with you, Michael.
[00:25:42] Michael Stutley: Good. Well, we'll do it again next week.
