Hello, everybody, and welcome to another edition of CNF Talks. It's my great pleasure to have with me today David Rundle, who's a partner at BCLP. David's going to be speaking at the AI Regulation Summit in London on the 1st of October.

David, welcome. 

Hi, thanks Maurice. Thanks for having me. 

Very nice to have you with us. In an interview in the Financial Times on Monday, just before publication of the FCA's AI and Retail Financial Services report, its author Sheldon Mills stated that regulators are in an, quote, arms race to keep up with the use of AI in financial services.

Do you think that the UK's current approach to the regulation of AI, a rather hands-off pro-innovation approach, is sustainable in the long run, particularly given that frontier models are doubling in capability every two months? 

Yeah, thanks. I mean, this comes at such a great time for this conference, doesn't it? I mean, the report itself would suggest to me that the FCA itself is having some doubts about that position. And for some time, as you rightly know, for some time, the FCA has been quite maintained and persistent in its position on AI, that the belief that a principles-based outcomes-focused framework would be adequate and that the existing components of that framework, the senior managers regime and the consumer duty, are also suitable.

But with this report, and also with more recent announcements, particularly a speech given by the CEO of the FCA entitled Rethinking Regulation of the AI in the Age of AI, really suggests that there's an element of reconsideration going on by the regulator. And I think that the prompt for that, at least from the article and from the Mills review, seems to be a fuller appreciation of what agentic AI might mean and look like, particularly where you have more extreme levels of autonomy of agentic models, in circumstances where the human beings within the institutions are effectively operators, rather than approving the output of those models. So I think there's going to be a, I think we are seeing a significant shift.

And I hazard to say that I don't think it's for the FCA to persist in its position that's gone before. I think that the Mills review kind of concludes by suggesting that the existing framework remains sound. However, it clearly goes on to indicate that the framework will become under increasing strain for those higher levels of autonomy, agentic models.

And so I think that it will change. And as with so many things technology based in AI, the constraint really is our imagination and our ability to project forward and understand how financial services will look like in an AI age in the next two, four, five, five years. So I think we will see increasingly, and perhaps incrementally, a rowing back from that original position that the FCA took.

Do you think, David, that the approach of the FCA, where they talk about being technology agnostic, do you think there's a recognition that perhaps AI has moved beyond just being another technology? Particularly, you mentioned agentic AI and where it's going to move forward in the future. Can you elaborate a little bit more about that? Does agentic AI pose specific issues which you think regulators can have to get to grips with beyond what they've had to get to grips with in terms of other technologies that have been introduced?

Yeah, I think that's right. I mean, the FCA has always been, again, clear in its messaging that it's technology agnostic.

But I think it can't ignore the fact that there are technologies that are materially different. And I think one of the key challenges that AI poses, and particularly agentic AI poses, is the question of accountability. In the wake of the financial crisis, there was a significant furore and backlash, both in the media, in the public, in politics, about the lack or perceived lack of senior level accountability for those events.

And what came out of that was the senior managers regime, as you know. And so one of the questions this paper tries to grapple with is whether or not the senior managers regime will truly be fit for purpose in an age of agentic AI. And I think that poses a real problem, because the way these systems will perform in the background will look very, very differently.

And I think it's a concern, not just from a regulator's perspective, but it's also a concern for the industry writ large. Because if you're a senior manager and these agentic models are being deployed within your firm, you will be naturally asking the question what you're expected to do and where you might be at risk of liability. And that's why one of the things that comes out of this paper is that the FCA is undertaking to provide some guidance as to what reasonable steps, which is effectively the requirement for senior managers must undertake, what that will look like in the age of AI.

And then the other aspect of, I think, these models and accountability, which is an old problem, but it feels in the age of agentic AI like it's going to be one which has certainly, at a minimum, has far more dimensions, is the question of accountability externally to the firm. And this is because these models are being developed by third parties, which are typically outside of the perimeter. They may be tested by data from another third party.

And therefore, the models and their output are becoming far more complex than other models that have been used in the past. And so much of the development and ownership of these models sit outside the perimeter. And so regimes like the critical third party regime, I think, will naturally be expanded.

David: The biggest question really is whether or not the FCA has to look beyond that and ask itself whether or not it needs it needs wider powers to deal with third party risks that arise. 

Yes, I mean, it's a wholly new area, really, isn't it? I suppose that that area of that question of liability will be tested in the court sooner or later. And I suppose that that is what will answer that.

And then I guess that there'll be an answer as to what firms, what degree of liability firms have, depending on circumstances. But it's without explicability, the SMCR is really quite difficult. People cannot explain what a genetic AI agent is doing in terms of going out, say, on chain with a digital wallet, making decisions, making money, and learning as it goes.

And with no recourse back to the people who are, as you say, the operators, it's very hard to know how anybody could sign up as having responsibility for that agent, unless they have deep technical knowledge of how it works. So I think you're quite right to highlight that. But where do you think, broadly speaking, that the liability if a self-autonomous agent goes rogue, do you think it's with the company deploying the AI? Or do you think it's with the developer of the software? 

Well, I think that will be the big question.

I mean, there's different ways to tackle that issue, because then we may be talking about regulatory liability, we may talk about liability between customer and product offeror. I mean, I think from a regulatory perspective, I think that is really one of the central questions that the FCA is going to have to consider and potentially recalibrate. I think if you're looking at this from a firm, whilst you're waiting for some more clarity on that answer, I think there are things that you can do.

There are certainly expectations in place, which we would expect to be maintained at a minimum. So if you are deploying and utilising an AI system, irrespective of its qualities, or how much autonomy it might be given, you'll still be expected to interrogate the system to assess the risks that it's going to pose to your business, to test its output, to understand its development, including, for example, if you can, where the data that it's been trained on has come from, the quality of that data, whether or not it's sufficient. And you'll be expected to audit the system and its output and ensure you have controls around it.

So you'll need to do all those things, certainly. And if you don't, then you'll be opening yourself up to some form of regulatory scrutiny. If the worst happens, and this is an entirely rogue decision or singular rogue decision, which could not be stopped by an agentic model, then in those probably quite refined and specific circumstances, I think it remains to be seen how the regulator will respond to that, particularly where the regulator has, in the background, kind of encouraged the use of agentic AI, whether kind of actively or implicitly, then I think it will have to reconsider really what it's to do in that situation from a regulatory output perspective, and also whether it has its hands on the levers of regulation for the providers and developers of those models.

That's very helpful, David. And final question, because I know we're running out of time. There's a lot of discussion about whether there's going to be any regulatory convergence across jurisdictions when it comes to AI.

Obviously, the EU has the AI Act. The US, to the extent it has any regulation here, leaves it to individual states, and the UK has got its, currently at least, technology agnostic approach. Do you think that there is scope really for there to be convergence? And is it desirable? And do you think that leaves open the danger of regulatory arbitrage? 

Well, I don't think there's a, as it currently stands, that I don't see a risk of regulatory arbitrage, because the services are being provided to customers within the jurisdictions.

And in those circumstances, it's hard to imagine or hard to understand at this stage how, especially the major players, will be able to avoid the regulation of any particular country if it has customers there. So I don't see the danger of that. I certainly think that there will be an appetite for convergence and harmonisation, I expect, particularly within the industry.

But it's hard at this stage to see how or when that's going to happen. I mean, particularly, there's a gulf between the EU's position and the position in the United States. And I don't see that gap closing anytime soon.

Yeah, yeah. No, I think that's absolutely right. I think we've run out of time, David.

Very interesting discussion. Thank you so much for sharing those thoughts with us. For our viewers, if you'd like to hear more on this topic, and to hear more from David and the other expert speakers we have at the event, it's the AI Regulations Summit being held in London on 1st October.

Further information is available at our website, www.cityandfinancial.com. So we very much hope to see you there. David, looking forward to seeing you on 1st October. 

Thanks very much.