Transcript 0:00 We live in a time now where we have a lot of open source AI, a lot of them. If you go on Hugging Face, there's tons of models floating around out there. 0:07 You don't know what's in the background of a lot of the large language models and what filters everybody's coming with, including the base models themselves actually. 0:16 DeepSeek came out, one of the first open source reasoning models. What we can't have is that we're going down into very big risk areas such as health diagnostics, healthcare. [on-hold music] Hello everyone. 0:34 Welcome to one more episode of Unlocking the AI Advantage podcast. This is your host, Ramesh Danta. 0:42 So in our previous episodes, we've been talking about AI implementations, um, and what's coming out, uh, you know, with respect to AI, what are the predictions for AI this year? So those kinds of topics. 0:56 But as we talk about the implementation of AI or machine learning, one of the most important areas is the ethics or the bias or the fairness of implementing AI, right? 1:10 All those topics are very important, and now and then we do, um, get into the AI ethics aspect of it. And today I would like to focus on the bias in machine learning models or data, right? Whatever goes into it. 1:25 And, uh, uh, you know, how can we instill fairness while we implement AI, right? 1:32 And to do that actually, so I have a couple of experts, uh, today I'm going to bring, and, uh, both of these people I know very well, and we had these discussions in the past with them. 1:42 So I am, um, incredibly honored to welcome Courtney Abercrombie, she's the CEO and founder of AI Truth, and Hassan Patel, who is the director of Global AI Policy Compliance Engineering, it's a long title, of course, uh, with a company called Fairly AI, and that's what they do, right? 2:03 So welcome Courtney, and welcome Hassan. Hi. Thanks again for having me on, Ramesh, and, and crowd. Um, I had worked previously at IBM before starting the AI ethics nonprofit, um, AI Truth. 2:18 Um, and, uh, I had gone around to all of the... 2:22 some of the biggest clients in the world starting to use AI at the time and saw a lot of things that were going great, but a lot of things that could be a lot better in terms of, um, how we approach things, uh, from a more responsible, um, way of doing things. 2:39 So that's my background. Thanks, Courtney. And Hassan, please. Yep. Thank you for, uh, the opportunity. 2:46 So this is, uh, I think I believe my second or third Bright Talk panel, and the topic is very, very interesting and timely. 2:53 Um, since, as Ramesh mentioned, we're entering the era of agentic AI, um, a lot of these questions are now coming up again. So a little bit about myself. 3:02 I'm the director of Global AI Policy Compliance Engineering at Fairly AI. Very long title, but basically in short, um, I've... 3:10 I'm a lawyer with a little bit of a technical bent, so at Fairly we do safety testing of AIs, um, and also AI governance. So we bridge the gap between the regulatory and the technical. 3:21 Um, so that's a lot of the space that I occupy, um, in that org. Excellent. Welcome you both. And so Hassan, let me get you start, uh, start with you. 3:32 So a lawyer with a technical bent, so I don't know how many of them are there. 3:36 Uh, but, uh, you know, first thing is one of the questions that I have in my mind, um, and, and people have been asking about the definition of a bias. Has it changed, uh, more recently in the la- last year or so? 3:52 And if so or if not, whatever that is, what are the, the sources of bias that you're identifying here? Okay. 4:01 So that's, that's an interesting question because fundamentally I haven't, in, in, in the tracking work that I do, which covers the cybersecurity, the policy, regulatory, and, uh, basically responsible AI framework space, I haven't come across anything that tries to redefine bias or redefine fairness. 4:20 What has, however, changed are the actors that are going to basically be either being biased or their bias is gonna essentially be confined and constrained. 4:32 So I'll give you, like, the, the thing is a lot of people try to separate these types of concerns from the very practical side of actual AI, ML deployment and development and operations. 4:44 Since I've seen both, I can give you like a more practical view on it. Mm-hmm. So here, here's one. You have a business. They don't want to pay OpenAI or Anthropic a ton of money for API calls. 4:55 And those who are not technical in the audience, API calls are very simply, if you don't want to use the GUI web interface and you wanna make something more automated, you can sort of ping an AI model that's sitting on the internet for a little bit of money, and it'll give you a response in the same way it does in a chat window. 5:10 These cost money, and for big AI providers, they will charge a lot and the, the, the, the fees add up. So then the question is, okay, where does bias play into all this? 5:20 Well, we live in a time now where we have a lot of open source AI, a lot of them. If you go on Hugging Face, there's tons of models floating around out there. 5:28 Some of these models have been modified, and the modifications done to these models were not completely documented, right? 5:36 If someone is now gonna deploy a model that's open source, that's been modified, and it fits their use case, it's not the fact that we know that there is explicit bias, but any assurance that maybe Meta or another big provider that's made the original version of the model free, 5:51 those assurances go away because now we have people modifying models to their own needs and then uploading them online. So now we're kind of back to this black box scenario. 6:00 You might have a bunch of documentation from Meta AI or someone else saying, "Hey, look, we made this, we tested this, this and this way."But as soon as you modify the model, there is the potentiality of bias creeping back in, especially when you train it on new data. 6:14 So the lack of documentation has a direct relationship with the possibility of more bias kind of creeping into a model. Wow, that's a very interesting way of looking. 6:26 Courtney, so yeah, please, if you could parse it from your own angle from an AI truth that you deal with it. Well, I, I actually completely agree with what Hassan just said. 6:37 I mean, you don't know what's in the background of a lot of the large language models and what filters everybody's coming with, um, including the base models themselves actually, um, not just the open source ones that everybody's starting to, um, add into Hugging Face and other places, um, and modifying them themselves, but, but even just the transferred, um, you know, learning there. 7:02 Uh, we don't even know what's behind a lot of the, the initial, um, neural nets, right, uh, for the large language models because we... There's, by design, that's the hidden layer. It's literally called a hidden layer. 7:14 So, [laughs] I mean, we don't know what's in the hidden layer exactly except for the people who trained it, and a lot of times they're just dumping as much stuff into these models and languages, um, you know, as they can and using, um, all of the latest approaches to try and make things as unbiased as possible. 7:35 But at the end of the day, you don't know, um, exactly what went into that and what it's, what those, um, neural nets are actually focusing on when they bring back words or bring back responses. 7:48 Um, I think the one that we've been seeing a lot lately, um, is like the, that I've just been seeing, um, that I think we can all relate to because we've seen it out there in the news and some other places is just the, the way that Trump is, like, if you look up Trump, you know, it's all depending on, you know, what you've been... 8:07 your search history and the way that you've looked at things in the past, it brings up different results for different people, right? Mm-hmm. 8:12 So sometimes Gemini brings up, [laughs] you know, information, sometimes it doesn't. Um- Yeah... sometimes it brings up incorrect, it's bent towards a certain, um, direction, you know, and it just becomes clear. 8:25 It's like, wow, well, it's pretty bad when we can't get, you know, just regular old, you know, 8:31 was this person a president or not, you know, answered [laughs] without having a, you know, a, a, a canned response of, "Well, that's a political discussion." 8:41 It's like, no, actually, let's just find out who was the president. [laughs] You know? Like, this president and that president should be able to be- Yeah... 8:48 mentioned without having a canned response, which I think a lot of groups are doing that, and that is, uh, you know... It's quickly getting just kind of to, like, a ridiculous level, I think, honestly. 9:00 It's just we're, we're seeing people go way off the rails with just, "Now I just won't answer any questions." 9:06 So it's, it just, you know, it just begs the question of how far are we gonna go with, with how we, how we, how we think about bias. [laughs] Wow, what a great open- And how, how concerned are we gonna get about it? 9:19 [laughs] So that was my point. Like, so but that's a great opening for both of you. 9:24 So you, you guys are, actually opened up the Pandora's box of, uh, Hassan, you have some thoughts on this one, so which is the, the initial struggle of, uh, you know, identifying is there a bias or not? Yeah. 9:38 So three things, right? Um, and we just at Fairly did a training where this matter actually came up of regarding explainability. So there's three terms I'd like to direct to the audience, right? 9:52 Uh, reasoning, reason, and explainability, and all of them are related but also distinct. So we have, just to kind of set the stage, right? DeepSeek came out. 10:04 It is one of the first open source reasoning models, right? 10:10 And what that means is that instead of just giving you a straight answer after you prompt it, there's like an intermediary step where it shows its reasoning process, right? And I'm putting an asterisk on that- Mm... 10:22 um, where you can kinda see that, okay, I asked a question, here is the answer, and I can see that there is some kind of logic to arrive at that conclusion. It might be going back on its initial analysis, revising it. 10:34 Okay, this seems reasonable, right? That can be a fourth term. And as a result- Yeah... 10:40 if there is a problematic result, to an unaware onlooker you might be thinking, "Okay, well, at least the model gave me its reasoning. It was reasonable, but it was wrong." 10:52 So the thing is, because of the way these models are architected, it's not the reason it gave you that output. 11:00 Rather, if you actually look at what the structure of a model actually is, if you just imagine a giant, like, sea or a lake, and you have, like, a lure, a fishing lure, and you just cast it out, right? 11:14 If you have something very general on that lure that's something that a lot of fishes eat, it'll pull a lot of thing... a lot of fishes that have that type of, uh, appetite. So that's your response from the model. 11:24 Something that's more sophisticated and more particular, you're gonna pull out something that has that type of sophistication. So in that same vein, 11:34 if now you have something where the model is explaining itself, that doesn't really tell you why that thing was pulled out. 11:43 It's just another layer you're adding onto that lure to pull something out of this big dark lake that you don't even actually know what's going on below the surface, right? 11:52 The reasoning a model gives isn't the reason it pulled out an actual answer. 11:57 It could contribute to that, but fundamentally you need something called explainability, where you can actually trace the response from the actual question through the neural network.The problem is, is the state of affairs for explainability right now is so poor that we can't even get close to, like, 10% explainability for outputs when we use, like, an open source model. 12:21 So that's a huge issue because now we're living under the illusion that we have some kind of reasoning from a model, and if we then use that reasoning to support or criticize a model's output, if it's biased, we're essentially living with essen- what's basically an illusion. 12:40 So now if you imagine this kind of ca- you know, cascading, if we have an audit that's happening and there was some problematic outputs and we ask for, you know, evidence towards why these outputs were given, it will give you a reason, but it won't be the reason why it gave those outputs. 12:57 Hmm. Yeah. Wow. So interesting, uh, uh, example of a fishing, uh, uh, uh, expedition there. So Courtney, uh, so one of the things given the challenges that we are facing, 10% explainability is very poor. 13:12 So what, what kinds of good or bad examples of, you know, case studies of things that you are seeing in your consulting side of the world where you actually deal with lots of companies, right, where either bias is being, being addressed or not being addressed? 13:29 Well, first I kinda wanna reframe off of, off of bias because I think, I think bias, as we know, is kind of a... 13:38 It's like, it's like when you come in and say, "I wanna help you with, be ethical with your, with your AI," right? But people say, "Well, I am ethical. I'm an ethical person. I wouldn't do anything wrong." 13:48 [laughs] So of course, then, you know, I think the same thing can be said about bias. It's more about where do you want to focus, um, than about, you know... But there are clear, blatant things about bias. 14:03 I mean, there are times when people actually really are just being biased, but there are other times when we're just trying to put a focus in one area versus another area. Um, and- Mm-hmm... 14:12 I would say, you know, the way that I see that people are starting to address where they want to focus, um, and be strategic about picking up groups that aren't as represented as they would like them to be in their, in their data. 14:30 Um, we're looking now at things like causal AI. Have you guys heard about causal AI? Um, so I'm kind of, uh, you know, this is a new area for me to really be getting into. 14:45 Um, and, uh, I was fortunate to be working with this professor, um, out of University of North Texas on this particular area. 14:53 Um, and we're looking at it in terms of, like, what can we do to help those who don't have traditionally, like, credit scores, for example. 15:03 How could we help the underserved communities that they're responsible with their money, but they may be- Mm-hmm... operating in more of a cash environment, or they may be operating... 15:14 And we're talking about, like, when you start expanding internationally especially, well, even right here, I believe there's something like 40 million unbanked, uh, you know, people in the United States that need some sort of way to have credit. 15:29 So if you think about it, you can't even get a house if you don't have credit, right? I mean, think about all the things you can't do- Yeah... in America when you don't have some kind of credit rating or credit score. 15:40 So the idea here is you start looking causally. So how can we make connections amongst the most important data that should be included to make a decision about how responsible someone could be with their money? 15:55 And just because it's not digitally available today, maybe we don't get information about cash transactions, but maybe we could be. 16:04 Maybe we could be getting more information, like, from Zelle, cash, digital cash, and things of that nature. 16:11 Um, you know, so I think this new focus on causal AI, and there are some, um, already some, uh, groups out there that are, um, you know, really looking into it. 16:22 Like, I saw that Sam's Club had some case studies, um, where they're trying to figure out, uh, more ways to understand their mix of things going on inside their store, everything from their merchandising to what they're purchasing. 16:35 Is it available at the time that some... Did we lose the sale of a particular item, for example, because we didn't have the right merchandising, the right supply chain? 16:45 Versus a predictive model, which would just say, "Well, Courtney bought this, so therefore another like Courtney will buy that." They still don't know why, right? That's the part that's missing. 16:55 So if we start destructuring some of the data and, and decoupling and trying to figure out, okay, what's the expanded version of this data that we could feed into this model, um, I think that we'll, we're, we're gonna start seeing a lot of changes because that causal AI approach can then be applied to many different types of scenarios, um, from th- from then forward. 17:20 Um, for example, you wouldn't just only be looking at, okay, well, how, how can I predict that Courtney's gonna buy again? Or how can I predict that someone like Courtney's gonna buy in my store? 17:31 I can now take and look at the whole causality and say, "Can I influence now how Courtney's gonna buy and when she's gonna buy and who else m- like Courtney may buy?" Do you see the difference? 17:41 I know it sounds nuanced- Yeah... but it's actually huge. No, it, it is. You're bringing the, the aspect, the fairness aspect into, into the discussion, right? So that is exactly what you're saying. 17:50 And then, uh, the, the representation of the data, right? So for example, the 40 million, uh, people who don't have, you know, were unbanked, right? 17:59 So they either were or not included, we don't know, it depends on the dataset that was used. But if they were not used, it's what we have is a, you know, not completely represented dataset, right? So- Right... um, yeah. 18:13 Uh, then how do we- It's about where you wanna focus, right? Yeah.Focus your efforts not just on those who have credit today will always have credit, right? 18:23 Because that's what we're saying is basically, well, you have cr- you have some sort of credit scores and cret- credit coming in, so you will always have credit, theoretically. 18:32 Theoretically, barring any major huge life events that cause you to have uncredit, but... I mean, discredit. 18:38 But I mean, the, the idea though that we could introduce a new situation into this and then start collecting information that is the important parts. What is the important part about the fact that I have credit now? 18:52 How did I get to that point? And can we replicate that information, that data, without having the exact same data that got that, um, person re- you know what I'm saying? Like- Yeah... 19:04 we're expanding the data ecosystem, so to say. [laughs] Got it. So Hassan, uh, to you, transitioning to tech, I like what Hassan said about explainability. 19:14 So does that mean that if AI training companies hide the reasons and the reasoning behind the answer given by the model, should they, should it be interpreted a- as suspicious? 19:25 Just because they didn't give information, whether not documented or whatever, does it generate sus- suspicion? Uh, should we, uh, take it with a grain of salt, uh, Hassan? 19:38 So I wouldn't, uh, say that if an AI lab is not disclosing the reasons and reasoning that it's necessarily suspicious. Um, there are use cases where you wanna hide why and, or what an AI said, 19:57 and some parts of its response, and maybe even some of the prompts. So a practical example here. Um, AIs need to be tested for 20:07 enabling very dangerous things because if we can get it to produce something that's very dangerous in terms of an output, then we can try to stop that. 20:16 So for like, for, uh, e-example, um, someone might ask an AI system, "How do I make chemical weapons at home? And here's what I have in my, under my sink. Um, what else do I need?" Right? 20:32 "And I have a Home Depot nearby." Um, an AI that is pretty, like, uncensored, with a little bit of prompting might start to give that person ideas on how to do exactly that. 20:44 So then when you read a model card or a paper that an AI lab puts out, they may say that, "Okay, here is the initial prompt, and part of it is censored because we don't want you to be able to replicate this. 20:58 The reasoning, here's some general details on it, but again, we're gonna censor it because we don't wanna enable people to actually, like, do this stuff. 21:05 Um, and then the conclusion, we're gonna just summarize it or gi- maybe give you the last few lines." 21:10 And in that case, what you're trying to do is prevent a harm by, on the one hand, showing the public that, look, these things are actually quite capable of dangerous, um, outputs, 21:21 but on the other hand, trying to prevent those same outputs from getting into the public. Because, uh, let me tell you this, that when you have research that explains these jailbreaks, it doesn't take 21:34 a genius to turn that into a workable exploit. Um, you just have to have kind of the technical chops and the reading ability to just, and kind of tinker with it. 21:44 So now imagine if every AI company was totally transparent about all their testing, uh, setups, what prompts went in, what reasoning steps came out, and then, you know, what the outputs were. 21:56 Um, then you would have a bit of a dangerous situation because people could just copy that and then use it with an uncensored model. Just a note on when w- when reasons are mentioned, 22:06 um, some of these AI labs, even though they have access to the backends of these models, it doesn't necessarily mean they have a very clear picture of why a model does what it does. 22:17 W- they're kind of working with the same constraints to a degree. 22:21 Um, so, like, when you hear about Anthropic and the types of weird stuff that AIs have done in their labs, where it's tried to, you know, exfiltrate itself, stop itself from being deleted, that sort of thing, I think it was Anthropic that, that happened with. 22:34 Another one is- Apollo... Apollo Research. Yeah, Apollo Research is another one where they've had AIs do really strange stuff. They're looking at this thing as, like, a black box that has really strange behavior. 22:46 So what they basically do is they have a certain input, and then they have some behavioral output, and we're trying to then fill in the gaps of why is the model doing this, and it's a working hypothesis. 22:57 We don't really have a fully mechanistic way where we can trace from A to B to C to D to E, where then we're like, "Okay, this is why it does what it does." 23:07 So for fairness and bias, this poses a huge risk because when we ta- take things like data, 23:15 if you have data that is itself compromised, and it's able to make its way into a new model's training data, you could essentially set it up to have certain dangerous factors, maybe even biases, that kind of sit below the surface and are activated once a certain kind of prompt comes up. 23:33 It's kind of like you basically are having a bunch of food those fishes train-- like, eat under that lake that are filled with poison, but they're below the surface. 23:43 And then when you have a certain kind of lure that attracts a certain kind of fish, you pull it out of the lake, but then when you try to eat that fish, it poisons you. That's effectively what might be happening. 23:53 So it looks like, Hassan, you fish a lot, so you're giving lots of fishing examples. I don't actually. [laughs] I don't, but I just- Somehow-... thought it was, like, a useful example. 24:02 [laughs] So somehow we are going into fishing today, [laughs] so that's on your mind anyway. So Courtney, I think if I look at th- the discussion so far, right? It's, um, 24:12 I don't wanna say scary, but w- it, it's, like, lots of risks out there. So, so the question is-How do we navigate this risk? 24:21 So what is that we should be doing to, uh, so manage the bias and then, uh, you know, improve the fairness as we implement AI? Well, I think it just all... I think I'm a big fan of keep it simple, right? [laughs] Okay. 24:38 Like, let's go back to where does all the bias come from? Where does all of the things that go wrong, and I've said this from the very beginning for like seven years now. It all comes from the data. 24:50 You- whatever you're training, these L- whether it's an LLM, whether it's computer vision, whether it's, whether it's just straight up good old-fashioned, you know, predictive models, whatever it is, it all comes back to what did you train it on? 25:05 What did you give it- Yeah... to train it on? 25:07 And if you don't know because you gave it such huge datasets that you can't really even remember what on earth you did in the first place, I think that's where we're seeing a lot of problems today. 25:18 [laughs] 'Cause we have it where we're just thinking, okay, just make some connections, make some... 25:23 You know, we run it through these models and we say, "Okay, let's just see what it comes back with and see what it does." Then if it works, if we- if it brings out the results that we want, then we use it. 25:33 And we, you know, and I think that's wrong. I think we need to go back and reassess and look at causal AI. I mean, I keep being back on this subject, but I think that it is the answer. 25:45 I think it has the potential to be the answer that we're looking for, which is let's make, let's make very specific connections between data, very purposeful connections, um, not just throw everything at models and just, you know, hey, this one worked out great at predicting this situation, but we don't know why. 26:07 Um, instead, let's go back to the way that we did things a while back, you know, [laughs] which is let's pick the actual purposeful data and let's build those connections between it so we understand what we did and how we trained this thing, and then why it could have gotten the answers that it got. 26:26 So Courtney, but here's the question though. Practically speaking, the cat is out of the bag, right? So the LLMs are out there and it's, it's... 26:34 Is it realistic to go back and look at the, you know, purposeful data, uh, data and all that stuff? So is that practical at all? Yeah. 26:45 I think it is based on small language models are turning out to be be- having better, much better results than all these huge large language models that are just generically trained, right? 26:58 There, uh, in a lot of very specific, um, use cases, we're seeing that it's actually better if you build from, use the same architecture, use the same whatever, but the main idea is now build it with a specific purpose in mind for now- Okay... 27:15 and make sure that you understand the results. 27:18 Because what we can't have is that we're going down into very big risk areas such as health diagnostics, healthcare of every kind, financially excluding people, whole huge groups of people from systems. 27:34 Um, and as we see the d- the disastrous results of doing some of that, it's time to now go back and refine and build specific models for specific purposes with specific people in mind, because we can't just continue to let people, you know, just... 27:53 "Well, I'm sorry, but you're part of this minority group, so we're not gonna focus on you at all." 27:58 We need to build specific things for specific, um, things that we want to do, and if that means, you know, we focus on the unbanked or whatever specifically to build models for them, then we do that instead of trying to just build these massive models that accommodate everything and we, we try to make them fit everything, and they're just over fitting and they don't, they don't always respond well to every single try- thing that we're trying to throw at them. 28:25 That being said, though, if you're general purpose using them for things like, "Hey, make me a creative email or something," or, "Make me a," whatever, and it's something very general that we can all grasp onto, then great. 28:36 But for things like health diagnostics that get super specific, like the example we were talking about before is the women's health app that I'm currently involved with. Mm. 28:45 So I mean, that's very specific, right, to women's health and the bu- the majority of research data that's been fed in to date in a lot of the bigger systems, um, is based on male symptoms of like, for example, how to have a heart... 29:01 how you're having a heart attack and the symptoms that you're experiencing, those are mostly male driven, and that's not even a case where it's like biased, it's just literally they're... 29:12 women are excluded because of the, uh, hormone fluctuations and things. 29:16 So I mean, that right there is one area that could be focused on with a very specific version of a model as opposed to continuing down this route of just, let's throw a big, huge all-purpose model at everything and be agentic with biggest models we can find. 29:33 I think that we need to go smaller again. Right. So Hassan, I mean, what do, what do you think about the approach? What, uh, Courtney is suggesting is that, hey, you know, identify the use cases. 29:43 There are general purpose use cases, there are specific, uh, you know, uh, purpose use cases that for your own organization or for the, the problems that you're trying to solve. 29:53 And within this, then, uh, work with a data that's a purpose-built for the, uh, the second set of use cases, and whereas the large language, uh, uh, large language models, you know, so may be okay for, for the general purpose, uh, use cases and questions that you want to address. 30:12 Is that way that organizations are thinking of implementing AI ML to manage this? 30:19 So-Not entirely sure if that's what they're thinking about, but that would be like a very popular narrative of essentially going back to the data. 30:29 Now, I think that is one part of the story, but one thing that needs to be unpacked is what do we mean by data here, right? Mm-hmm. 30:35 Um, and there's a bit of a overlap between what goes into a system and how you architect it. 30:42 So for those of you who are, those of you who are curious in the audience, if you actually look at like a breakdown of how, um, DeepSeek was, was built, right? 30:52 Or at least what the popular narrative of it is, um- Mm-hmm... it's, it's not that it was just built using like kind of second grade or scrap GPU parts and then trained on a budget. 31:02 There were some actual architectural decisions made for why that system is the way it is, right? 31:10 So that's the first thing to take into account that whether or not a company or, you know, a group wants to simply focus on the data, 31:18 what is being put out there are being architected in certain ways, and there's gonna be certain trends that I'd imagine that are gonna appear in the future that are gonna result in a different s- type of architecture. 31:30 Now, we're only fooling ourselves if we basically ignore architectural decisions in this whole bias discussion. So just, just to give you an example, right? The training data is im- is important, right? 31:41 Not understating that. However, take for instance something like a mixture of experts. 31:47 So if we look at a normal model, it's all kind of just you could imagine it as just a whole bunch of little dots where you sort of have to go from one point to the other. 31:56 You have to load the entire thing in and sort of operate the entire machine to get an output, right? 32:02 You, you have to almost ping many of these points, if not all of them, to kind of get to the end of that network and get your answer for whatever prompt you give it on one side. 32:11 Mixture of experts sort of kind of makes little clusters of mini models in a sense- Mm-hmm... 32:17 where you can-- you still have to load the whole model, but you can ping one part of that network rather than kind of calling everyone up to get one answer. 32:26 So that makes it more efficient, but then that raises the question of why is there a certain cluster of... why is there basically a partition of this model, right? 32:37 So like for a practical bias example, if for instance we're talking about health symptoms, is the heart attack symptom part of the model? 32:46 Is it-- is that cluster only taking into account symptoms associated with men, and is a different part of the model taking into account symptoms associated with women or particular health, uh, nuances when it comes to certain ethnicities, as that's been reported in the literature? 33:04 If we don't actually look at how this architecting is ha- happening, right, then the way the, the, the architecture of a model almost might be working against you. 33:13 Because think about it, if you only have to ping one part of a model, it becomes cheaper to run. Mm-hmm. And if it becomes cheaper to ru- run, it'll become more widely adopted. 33:22 However, if it's in-- if the way the model is architected has some kind of bias built in where the cheaper to run part happens to be the one that's over-representative of a particular phenomenon that you're looking for, then it poses a problem because you've essentially architected it in a way that even if your training data is not that biased, you get pretty fair representation. 33:45 The part of the model you're calling up to get your answers might be over-represented. So that's part one, right? Yeah. That's when we look at data plus architecture. 33:53 Now, let's say we switch away from this type of transformer architecture a little bit, and there's a new paper released. I haven't read it yet, but I was skimming through a, a summary of it. 34:01 Um, I believe it was Google's researchers, um, that put this one out, and it's being called the Transformer two point o paper. 34:07 Not to get into any of the really technical details, but the big idea-- one of the big ideas here is models that can remember. 34:13 Right now when we have a model, we have to kind of feed it a lot of data to keep that conversation history going. This is more modeled off of how we remember things and how we forget things. 34:24 It's a model that can sort of forget and then remember, um, sort of on the fly. 34:29 So now the question is, is that if it's also retrieving information from a database and it's not strictly relying on training data, right, how are we then looking at the data representation when it comes to retrievals? 34:44 And this applies nowadays too because we have retrieval systems that work with conventional models. 34:49 You might have then your mo- your, your architecture might be sorted out, your training data might be sorted out, but then you have certain 34:57 databases in the background that you're use-- you're using to justify and support the answers a model gives. You might have seen that when you use a model, like it might have retrievals li- linked up to the web. 35:08 It says, "Okay, you know, uh, this website, this website, this website, this is where I got my information from." You can do that with PDFs. Like the references and sources. Yeah. You can do that with... 35:16 Yeah, the references. It doesn't have to be the web and it doesn't have to be a PDF. What if your references now are underrepresented or skewed, right? That then poses an issue. 35:26 So this kind of draws back to then the broader point I'm making that how you architect these th- how you architecting these, uh, how you architect these things 35:34 makes a kind of a broader point about your fairness practices. 35:38 Because if you architect it in one way where you focus on one thing but neglect another, uh, it becomes almost like a hole in the armor where, you know, bias can poke its head through. Fantastic. Okay. 35:50 So we have, uh, laid down, um, the, the model related questions and the, the data related, uh, aspects, and then of course other, uh, thing that you're bringing, which is the database, the retrieval aspect, something that's coming up, uh, as you're saying. 36:04 S- so but still, um, if you put them all together, we are still talking about challenges out there for implementing AI model. 36:14 I mean, as it is, we have technical challenges to get the answers you want, but now we're talking about the bias and fairness aspects of it. 36:21 So Courtney, the question isIn the, the organizations, what are the best practices you're recommending that they should look at? 36:30 I mean, uh, I know primarily from an AI truth perspective, you probably focus more on the fairness and bias. 36:36 That's my guess, so I, I could be wrong, but w-what are your, uh, you know, suggestions, um, with respect to best practices? 36:46 No, [laughs] I would say actually we focus on protecting some of these systems today 'cause, uh... 36:52 and, and making sure they're robust and, uh, inclusive of a lot of informatio- of different types of information, which is still the case. I mean, really just... 37:01 I mean, I can't come back to this point enough because I really think it's going to emerge as a way to be, um, to provide explainability and to provide that, um, 37:14 that layer of purposefulness, um, as opposed to just feeding in all of the big generic models. 37:20 I think we're gonna start seeing a shift away from that, um, as people just aren't getting what they need out of their specific use case, um, from the agentic, um, models that are available today. 37:32 I think we're just gonna con-continue seeing that people are progressing, um, with trying to focus on, uh, specific areas like, like I am with women's health. 37:44 Um, they'll just keep trying to get more and more specific with how they, um, represent those groups, uh, and the data that surrounds those groups, as opposed to making an all-purpose, you know, agentic AI. 38:01 Um, right now I'm trying to build small ontologies for, [laughs] for the way that women talk about their health, right? Because that's what we wanna talk about, that's what we wanna try to help them understand, not... 38:16 There's a big difference between you going out and having a conversation with your best friend about making out or something, versus you going to, like, WebMD and it explaining, you know, what that's all about. 38:27 [laughs] You know what I'm saying? Yeah. Like, you're gonna get- Yeah, yeah... nitty-gritty details and in the words, in different words, in different ontology, right? 38:35 But there's a big difference between going to, like, a WebMD that I would consider to be more of like a big all-purpose built, uh, type of solution, right? 38:44 You go out there, you wanna query that one, and you wanna find out some answers. Well, you, you're gonna get very specific sets of information, and then over here you're lacking that information. 38:55 Where is that information? Is it even available? You know? So a lot, I think this, this idea of having more purpose-built systems, um, systems that focus more on... 39:08 less on very all-purposey types of things and more on specifics, um, specific use cases, for example, and specific targets that you're trying to achieve. 39:20 And not just, um, not just the predictive analytics based on I'm gonna feed it in examples and then train it to spit out, you know, whether or not other people look like that example. 39:31 I think instead of that, we're gonna see people moving away from that and trying to move more into this, "Can I figure out a way to expand this group and really target in on what this problem is and then influence that?" 39:47 So, like, it can be anything from women's health targets to, um, trying to open up financial markets for the unbanked to, you know... 39:58 But I think that, I think that the way forward is really starting to look at that whole approach, which is, um, more of a causal AI approach, so. Got it. 40:07 So basically what you're suggesting is, hey, you know, there are all-purpose, general-purpose kind of use cases, so that's fine. So just segment them. 40:14 But a-also look at, uh, the what are the problems that you're trying to solve and specific purpose ones, and then have a purpose-built AI, uh, you know, combination of representative data a-along with the models, right? 40:28 So do that. So Hassan, so a-again, right, you talked about, uh, in, in your discussions with the companies, you're talking about reasoning, you're talking about reason and, uh, explainability. 40:40 So what are the best practices that you are suggesting as you talk to the companies? So best practices, it would be something that is, you know, very subjective, very industry dependent, right? 40:52 But in terms of just minimal things that you need to do, right? Because right now we're at that level of AI where you just need... 41:00 So everyone needs to be doing something in order to ensure they're using it responsibly, and then the degrees to which companies do additional steps may vary. 41:09 Um, so just to kind of recap, I've talked a lot about theoretical, you know, avenues of potential bias, things like that, architecture, RAG systems- Yeah... 41:17 uh, even, you know, training data that might be used to upend AI. 41:22 But then when it comes to the practical nuts and bolts of this, what's really gonna happen is this: you're gonna have companies that are using Copilot, Microsoft's, right? 41:29 And they're gonna rely on the fact that Microsoft, you know, got certain, has done certain amount of security testing, I presume. 41:35 Um, you're gonna have companies that use just off-the-shelf open source models by Meta, right? They're... Most of these companies, 99% I would say, are not designing their own models from scratch. 41:46 So the question is, what do these types of companies do to operate, uh, responsibly? So to that, I would say at the minimum, they need to start documenting, right? 41:56 So one standard that we've done a lot of implementation work for, um, is ISO 42001. 42:02 Um, and this is actually, like we have companies here in Canada as well that are getting, you know, um, certified in the ISO 42001 standard from actual auditors that have worked with, uh, with the likes of Cohere and other big companies. 42:17 Um, so- Is it becoming a requirement, Hassan, this par- certification, or is it still an optional stage? So it's, it's not... 42:25 There's no law that I'm aware of that says you have to get an ISO 42001But the reason a company might do it is that... 42:32 So just to give the audience a little bit of background, ISO 42001, similar to 27001 and 9001 if you've looked at ISO before, it's a management standard for AI systems. So what does that mean? 42:43 It's like if anyone's played with Legos as a kid, you remember that giant flat piece where you build all your other stuff on. 42:48 ISO 42001 a- acts as like a foundational standard where you can basically add in other practices, standards, regulations, and sort of have that as kind of a way to manage your responsible AI operation, right? 43:04 It's sort of that ecosystem that then you have different responsible AI components in. 42001, a large part of it is having a- awareness and documenting how you do responsible AI. 43:17 So at Fairly, we've kind of boiled it down into five categories. How we, how your company deals with AI at a societal level- Mm-hmm... 43:24 so, you know, your various stakeholders, documenting them, making sure who you know who is dealing with your systems and who is supplying your systems. Okay. Looking at it from the organizational level. 43:34 So who are the people in your org that have competencies for dealing with AI systems? Who takes responsibility for AI outputs? Who's there to deal with an incident when it arises? 43:45 Then you have the tooling, where it's anything that's not the data or the model itself, but you'll have things that, like, run the inference, right? Not everyone uses OpenAI's API calls, for example. 43:56 You might have a model that's sitting on someone else's systems that your company pings. Yeah. Right? 44:02 So those are the types of things you need to be aware of because if there's a security issue in your tooling, there's a security issue in your actual, uh, systems. Then you have the models. What are the models? 44:12 Where were they... You know, where... Like, like, where, where are you getting them from? Are they modified or are they just the stock models? And then you have the data. Is it private client data you're dealing with? 44:20 Is it publicly accessible? Where is it coming from? Um, what have you done to modify it? So these are the five levels. 44:27 Once you just at least at the minimum start documenting these things, you then have an awareness of the different avenues by which fairness and bias issues can creep up. 44:38 If you don't have it and you kinda just have, like, this siloed view of your organization where one team has, like, one model running on a computer and then another team is using Copilot and another team, um, is accidentally, you know, leaking out company minutes on some sort of, you know, free GPT server that they found on Google, it's gonna be a mess, right? 44:57 If at least you have these fundamental elements of documenting what your data is, where your models are, what systems you're using, how your organization is equipped, and then who you're dealing with in society and what your, you know, your, uh, responsibilities are, 45:13 you're more in the direction of then being able to tackle questions of bias and, uh, fairness. Got it, Hassan. Thank you. So, um, there is a question. 45:22 It says, "How can I make AI remember when I ask questions or get advice?" So- Oh, this... Um, no. Yeah. Yeah. 45:31 So he, he's talking about that basically when you have, um, an AI system you're talking to, after, like, however many hundreds of messages, it's gonna start to drift, and it's gonna forget the context of your conversation, and it's gonna forget what you discussed earlier, right? 45:48 So then you have to make a new chat, so then it, it, it's... That... This is actually a big issue. Um, not to plug it, I don't work for them, but I do use their product. There's... 45:57 Quora has a product called Poe, P-O-E, and they essentially have, like, a platform where you pay a flat fee and you can access... And there's a free tier as well. You can access different models from different providers. 46:08 So if you wanted to experiment and try out different models without having to pay their subscription, um, that's one way of doing it. 46:14 Now, in Poe, for certain models, you can set a setting that will use up more of your credits, but it will take more of the past history of the conversation into account when you talk to it. 46:26 So that's basically the way of addressing the memory issue. It'll just use up your credits more, but I've, I've, I've yet to be in a month where I've r- used up all my credits. Like, it li- it gives you more than enough. 46:38 Got it. Got it. So that... So, um, Courtney, as we, as we come into the final stretch of the discussion, so I know you talked about the causal AI. So is there a paper that's coming out? It's published? 46:50 Can you talk a little bit more about, uh- There's a lot of stuff. Yeah, absolutely. 46:54 So, I mean, even if you're just a subscriber to, like, Gartner and some of the other ones that's been in the hype cycle, um, uh, starting in 2023, I believe, um, and there is some causal AI summits. 47:07 Um, there's also, um... Let's see. Oh, I wasn't prepared to answer that question, but I probably should have been. Um, but yeah, it's a thing. I mean, it's... You can look it up. It's, uh, it's- Okay... 47:19 literally causal, C-A-U-S- Mm-hmm... A-L A-I. Um, and it- Okay... is kind of gathering steam as we speak. 47:26 Like I said, I'm kind of new on this forefront, but I really, um, love that there's already companies looking to... I mean, doing projects with it. 47:36 Um, and the, uh, open source community for it is called Dowy, um, as in D-O-W-H-Y. [laughs] Oh. Um, and, uh, so anyway, um, it's a great way to look at can you actually... 47:53 I mean, speaking of Hassan, um, can you actually put your policies into also your AI training? 48:00 For example, um, if you're saying, like, we will consider, um, th- let, let's say you're doing, uh, some kind of a employment recruitment app or something, and you want to consider a particular type of candidate first with maybe a s- particular set of soft skills. 48:17 Um, if your policy is to consider first, you know, um, women or whatever it is, I don't care what it is, right or wrong, um, but you, you put that in first, and then you can set your, you set your policies in it, and then it also will start looking and adding that to the modeling, um, process. 48:36 So it is part of your, um, your weighting factors that would go into your neural nets and things like that. 48:43 So, um, and then you can also, you know, uh, in addition to putting your policies into this stuff-You know, just do some basic things at companies that you as business leaders have complete control over, which is set policies about these things, put training in place. 49:01 Um, a lot of people have policies, but they don't necessarily go out and actually make everybody take training once a year. Make it mandatory. 49:09 Do that wonderful mandatory training that everybody at least has to have seen it. 49:14 Um, the other part of this is, you know, put guidelines in, um, at your corporation and make sure that the thresholds that you set are somehow put in before you release it out into a wild. 49:25 Like, if you're a company that's looking to produce scores for other companies or your product is somehow, um, an application that other companies are using, then you wanna make sure before you go out into the wild that you set guidelines around how you're going to, um, enforce, um, um, 49:45 you know, how you approach your, your bias, um, you know, how you are setting up your bias rules and things like that. And then, uh, you know, the other part of this is I get a lot of questions from lawyers. 49:57 One area that you can absolutely crack down on if you feel like people are just going crazy with the LLMs and that it's out of control inside your company- Mm-hmm... you can't buy it. 50:07 [laughs] Even if I wanted to as a product leader, 'cause I love AI and I wanna buy stuff. Yeah. If I can't buy it, if you shut down my ability to go through procurement before I [laughs] 50:19 if I have-- if procurement forces me to come back to you and vet it through you before I get to buy it and do that, uh, initiative with a third-party vendor whom you may or may not like, and you, as a lawyer, you may think that's a risky, you know, group of people, that's a risky LLM that I don't really like you participating with, you can always force me through the procurement process to come back to you and vet it. 50:43 I mean, it's not a nice thing to do to people, but you can absolutely do that as well. But that's one policy. That's one policy you can enforce. 50:51 So this is the one, uh, last question to both of you, uh, before we conclude. Hassan, uh, you've been on this panel before. Uh, we talked about it. So the question is: Has there been any progress that's been made? 51:02 And in the past we talked about the EU regulations, Canadian regulations and this and that that are there. 51:08 But, you know, on the street, in reality, has there been any progress been made, uh, in, uh, addressing, uh, bias or, I mean, increasing the fairness in, uh, AI and ML implementations? Yeah. So I mean, 51:26 practically speaking, a lot of the research with regards to dealing with this stuff happens in the, you can call it the AI cybersecurity space, for the lack of a better term. 51:37 Um, so in our work actually, we've been looking at this quite intensively because functionally speaking, think about it, if you have a jailbreak for a model that lets you just get whatever you want, for example, like, you know, the chemical weapons example I mentioned earlier, um, then practically there's, there'd be ways to get a biased output from a model as well, right? 51:58 Or the model might be biased to begin with. In both cases, it's kind of like you're trying to dig up problems from the model. Now, the issue is, is that it's like Whac-A-Mole, right? 52:09 You uncover one thing of a model, another one will appear. Yeah. And imagine, like, thousands or millions of these holes. Like, how do you practically do it? 52:16 You can't really use human teams because it's gonna take a very long time with a lot of people, it's gonna cost you a lot of money, right? And you're not gonna get very good coverage. Definitely. 52:24 Um, the other solution is essentially using AI to help you build the tooling and do the testing for you. We have a lot of research that happens internally of even how we judge the judge, like of AI. 52:38 You know, AI as a judge, how do you actually test that and assess that? We've sort of come up with something regarding that. And the basic idea here is you just do large scale rapid fire testing on systems. 52:50 So then it's like, okay, you can do that with jailbreaks. You try to break a model, and then if it gives you output, it gives you output. That's problematic. How does that translate into bias? 52:58 Pretty simply actually, right? You p- you do pinpoint tests. You don't try to basically, like, try to carpet bomb the model and try to get everything out of it all at once. You simply... 53:08 For example, on an image model, right? If you try to ask it, okay, you know, what, what is like, what is a picture of a father, right, and his child? 53:18 And repeatedly, you're essentially getting one ethnicity represented over others, right? When the expectation is diversity, then essentially you've now uncovered something with regards to bias in the model. 53:33 With AI, you're able to do that at scale, and then because we have multimodal models, we're able to then interpret these images at scale as well. 53:41 And then there's technical details of how you ensure and validate those results. But in a nutshell, some of the work we do is outside of just strictly regulations and governance. 53:51 It's in this practical testing, um, aspect, and that's actually how you, how actually doing it. You just rapid fire, um, AI. Got it. Excellent. Thank you. Uh, thanks, Hassan. Um, so Courtney, if you could bring it home. 54:08 Uh, any topics that we have not talked about? So if you could, uh, you know, help concluding this, today's topic, please. 54:15 No, I think we, you know, we discussed a lot, um, and we started with bias and then we quickly, you know, got into all kinds of new and different, uh, ways of looking into these things from both Hassan's perspective and mine. 54:29 Um, uh, you know, I think, uh, if you're looking into-- if you wanna look into causal AI, uh, go to pypwhy.org. Um, that's P-Y, uh, Python, as in Python, uh, W-H-Y.org. That's a good place to start. 54:46 Um, and, uh, you know, we'll be on this journey together 'cause I'm starting to get into it myself. So, uh, you know, let's all, let's all get in there together and see how it goes. [laughs] Thanks, Courtney. 54:58 Appreciate it. Actually, it's something that intri-- uh, definitely something new I'm going to dig into. So Hassan, any last-minute thoughts? Um, this area is big. 55:07 There's a lot of active work going on, and what we try to also do is make sense of this space. So we have at Fairly, you know, three publicly available trackers that we try to regularly update. 55:18 One is our AI, Global AI Regulations Tracker, which will just bring everything all into kind of one map where you can see what the state of AI regulations are. 55:26 Um, we also have a responsible AI, or you can call it an AI Framework Tracker. 55:32 So this is for the stuff that's not regulation, um, but it's published by, you know, standards bodies, companies, things like that, that particularly address one area of responsible AI or just AI operations in general. 55:42 Mm. So we have a tracker for that. We try to update that on a monthly cadence. So we're due for another update actually very soon. 55:48 And then we have our cyber, AI Cybersecurity Tracker, where we organize all the research that we come across or as much of it as we can regarding AI cyber, cybersecurity. 55:56 So it's like if you wanna know about the latest papers in the AI jailbreaks, we also make that publicly available too. You can find that at fairly.ai. Um, those three are publicly available resources. 56:05 And if you wanna have a chat with us, you can find us on LinkedIn. 56:08 Um, and you can also just send us an email, um, if you want to, you know, have a chat about how you can, uh, essentially operationalize responsible AI for your organizations. Thank you both. Thank you, Hassan. 56:22 Thank you, Courtney. So with that, uh, so that's a wrap. Thank you all. [upbeat music]