Kelvin Crosby: 0:06

Welcome to Investing in Accessibility, Samaritan Partners Podcast. We're not waiting for change. We're investing in it. Join us as we speak with entrepreneurs and thought leaders that are focused on creating a more accessible world. It's so good to see you, even though I can't see you. It's another beautiful day in the neighborhood, and I'm so excited that you're here and investing in accessibility. And I got my Chris Maher here. How you doing, man?

Chris Maher: 0:40

Good, Kelvin. It's good to be with you and actually be with you. We're in person today, which is a super special treat. Only second time since we've known each other. Yeah. And our guest is live with us today, or in person with us today. Yes. And it's a repeat guest. It's our good friend Joe Devin. Joe, how are you? Welcome back. I was going to say welcome to the podcast, but welcome back to the podcast.

Joe Devon: 1:00

Thank you so much. It's great to be back.

Chris Maher: 1:02

Yeah, have you been, my friend?

Joe Devon: 1:03

I've been great.

Chris Maher: 1:04

Good.

Kelvin Crosby: 1:05

So I have to ask before we get started, how does Chris smell to you?

Joe Devon: 1:08

How does Chris what?

Kelvin Crosby: 1:10

Smell. That's the key thing about this podcast.

Joe Devon: 1:13

Oh, well,

Kelvin Crosby: 1:13

Because we always have to figure out what Chris smells like. When I first met him, he smelled like lemons. Right now, we're in casinos and doing all this stuff.

Chris Maher: 1:23

It's not a very nice like French bakery bistro, so it might be a little croissant or hot chocolate or something.

Joe Devon: 1:29

I'd say probably hot hot chocolate.

Chris Maher: 1:32

Yeah. I'm not sure Joe is comfortable answering that question.

Joe Devon: 1:37

Probably not, but hey I'll go with it. Yeah. I'll go with it.

Chris Maher: 1:43

All right. So let's we're so the topic we're going to talk about today, which is top of mind for everybody and will be for the foreseeable future, is AI. And specifically AI as it relates to accessibility. And Joe, a quick reminder on his background, he's one of the co-founders of GAAD, Global Accessibility Awareness Day, and he's one of the OGs in accessibility and has been working in and around engineering, code, dev, services, front end, back end. Like Joe is one of the one of the people I met early on in my journey in investing in the accessibility sector and has not only become a good friend, but as someone who is my go-to when I'm looking at things that are super technical and I need someone who can really explain it to me and break it down for me. Is that a fair enough description of you, Joe? Anything you'd like to add?

Joe Devon: 2:26

It is. No, I'll just add that this this this project I've been working on is is often challenging to discuss because there are some technical aspects that you sort of can't get around, so you have to try and explain it to a general audience, but it can it can be challenging sometimes. So bear with me, but we'll we'll try to keep it as light as possible.

Chris Maher: 2:49

And the findings are fascinating, right? You shared a little bit of that. That's part of why we quickly kind of got together with Joe when we were all together in person because some of the findings and insights are are really fascinating. And so before we get into the kind of the audit model you built or processed, talk about kind of like what's what's been good, how has that been happening for these AI models presently, and then you took it to another level around accessibility. Does that make sense?

Joe Devon: 3:18

Sure. So you know the obviously chat GPT is what kicked off the the resurgence in AI in general, and that's what the what most consumers know, but there's been a whole uh different uh different tiers of uh AI coding that have been happening, and that's the first really big uh shift, and it's probably going to uh not only completely change uh every corporation almost, it will also uh impact end users because what we've discovered over time is that when you have AI do coding for you, uh no matter what you want it to do for you, if you ask it to do shopping for you, what what we've discovered is that it will write the code to connect up with the store or go to the website, look it up. It writes code to do that. Everything happens through code. So this is an extremely significant development is that AI coding is changing the world and it's just gonna get bigger.

Chris Maher: 4:19

That's a really interesting point because like I didn't realize that. The fact that anytime you're interacting with one of these platforms, it's coding to get you the response.

Joe Devon: 4:28

I'd say the AI coding platforms do. A lot of a lot of the chats do that sort of behind the scenes, but it's clear that that's what's going to be happening more and more. All of the folks that are building these technologies are saying so outright. So it's very clear. And I was looking at this, enjoying it, playing with it. I hadn't I hadn't actually, you know, I'd run a company of a hundred people, and I didn't I I've gotten out of code for about 15 years, and AI coding has got me back into coding. So now I'm just like when I was 20, I wake up in the morning and I'm coding. I wake up in the middle of the night and I'll be coding. And I bring my computer to bed and I code until I fall asleep. That's how I used to do it in the 20s, and I'm back to doing that now. It is it is exhilarating and fun to code. But as I was doing this, I was like, oh my god, like if the code that is generated is not accessible, it's gonna be just another technology that comes in and makes it worse for people with disabilities.

Kelvin Crosby: 5:28

And I I mean I want to point out something here because what's really interesting about what you're saying is is that you have the the code and uh the other thing from from a like a web developer myself, like reading that code is super important. Like understanding what that means with a screen reader and all that stuff, like this is really important stuff. And like if I've done a little bit of coding on ChatGPT and some of the other platforms, and I get lost. And I'm I'm excited to see like the the result that you're gonna talk about. Cause I think it's gonna be surprising. Yeah, what you're talking about.

Joe Devon: 6:08

So it it is very surprising, but let's let's add another layer here even before we get to AMAC because nobody is talking about this. The the platform that right now is number one that coders love more than anything else, is is called the the command line interface or terminal. You might have heard of it, you might not have, but it's essentially like text messaging for developers. You sit on a screen and you just talk to it. And it's a hundred percent a textual format. However, it is extremely inaccessible. And I only I thought it would be accessible. I built for AMAC, I built a command line interface positive that it would be accessible since it's a textual format, and unfortunately, very deep into it. I finally pulled up a screen reader and nothing, it read nothing, it it saw nothing. There is no accessibility tree, no standards body, nobody working on this, and it is now the most the best, most popular way to code with AI. And that's kind of a disaster. So I don't know what to do about it, but you know, let's bring that up. Maybe someone in your audience has some ideas on on how we can improve that.

Chris Maher: 7:21

Let's shine a light on it.

Joe Devon: 7:22

Yeah. So that that's just the side sort of issue. But coming back to the bigger question, I I thought, okay, we need to we need to make sure that the AI model companies code accessibly. And so I I went around and found the biggest experts on on these fields and got a sense of the pricing because I thought I'd shop it around and try and get funding for it. But you'll be surprised to know that in 2025 AI expertise is really expensive. So I tried a little bit to get some funding, but I didn't bother you with that one with that one, Chris. But there was no way. There was no way. So I'm like, okay, I'm gonna have to do what's called vibe coding, which which was big in 2025, where you have the AI write the code. So AI wrote AMAC.

Chris Maher: 8:15

Tell me what AMAC stands for.

Speaker 2: 8:17

So AMAC stands for AI model accessibility checker. And it's a benchmark, benchmarking how accessible the code is that that AI writes. And and taking a step back, benchmarks are really important because that is how AI model companies compete. They they say, This is how well I did on, for example, there are people benchmarking medical AI. How well did you do on the MCATs? You know, how well did you do on the LSATs for their legal stuff? And they're all competing on how well they did. So until you don't have an accessibility benchmark, they're not going to compete on it. So I thought, all right, we've got to create this this benchmark for them to compete on.

Chris Maher: 8:59

Love it.

Joe Devon: 8:59

Yeah. So I vibe coded something in three days. I came up with a proof of concept, but the difference between production and proof of concept, especially with how how bad, relatively bad the models were back then, was an it was big. And I showed it to my podcast co-host, Eamon Mackerlane, who heads up accessibility for ServiceNow, and he loved it, and he said, you know, you you gotta make this happen. And so we partnered with ServiceNow on it. They they backed us for it. We did it under the the GAAD Foundation and built this. He actually named it. He came up with this AI model accessibility checker, which was which I thought was great. So then, all right, we come out with the results, and I was absolutely shocked by them. I was positive that Google would do really well with their Gemini model. So they just came out with Gemini 3.0 Pro and it killed on all the benchmarks, and we tested 36 models.

Chris Maher: 9:57

And those are the more traditional benchmarks.

Joe Devon: 9:59

Yes, right, yes, the more traditional benchmarks that that they all have been competing on. And we tested 36 models and guess where they landed out of 36.

Chris Maher: 10:10

I'm guessing not as well as you thought.

Joe Devon: 10:13

36.

Chris Maher: 10:14

Oh my god.

Joe Devon: 10:15

All open source models beat them. I mean, it it it it isn't quite on percentages because what we what we do is we we have it generate an HTML page and then we do automated testing and it counts how many the errors there are. And so it had more than anybody else. And and these are a lot of these are open source models that don't have resources. Google literally has a tool, a web developer tool called Lighthouse that tests, uh does automated testing, and all developers use it pretty much. All they had to do was train their models with their own tool and they would get a perfect score. So them coming in last was was a bit of a downer, kind of shocking. And then the other company that I was a pretty shocked at was Anthropic left open AI because they felt like they didn't invest enough money in ethical AI.

Chris Maher: 11:06

they've kind of taken the moral high ground with AI, right?

Joe Devon: 11:11

Yeah, they they they're really saying that you know they're doing it the right way, they care a lot about safety and morality and all of that. And they were super middle of the road.

Chris Maher: 11:20

Wow.

Joe Devon: 11:21

They were some of the more expensive models, but super middle of the road. And then the third shock was open AI killed it on this benchmark. Five of the top ten are open AI models, they have the the top four models. GPT five uh five. I'm sorry, I'm getting confused for a second. 5.2 Pro was number one. They beat 3.0. We have like a particular like uh sort of scoring, but they they they just killed everybody. By far. But they were that model though was very expensive. It cost like 95 bucks to run the benchmark just for that one model. But they they have three more that came in one after the other that were way more uh affordable.

Chris Maher: 12:06

So Joe, with the with these results, and thank you for sharing those with us, uh, and so in your estimate, like what does that mean? Like going forward, like what does that mean for the for not only these platforms, but really for you know our space of accessibility and and disability and inclusion, you know, going forward?

Joe Devon: 12:25

Well, what it means is I'm gonna be a pain in the ass until they fix it and they they do a good job on accessibility. I hate to say it and I hate to be a pain in the ass, but uh it kind of feels like what we what we need is to realize that we're paying attention. Right. But we can't pay attention if we don't have the statistics and the benchmarks. Yeah.

Chris Maher: 12:44

Do you think that we can get to more, you know, standards, industry standards around this?

Joe Devon: 12:49

I don't know if standards are the right word because there are already standards for all of this. We just need their attention on it. Yeah. And you know, I I try to tag the the people that are building this stuff, and they'll respond about other topics, but on accessibility, it's been like 100% zero response. But I have I have heard through friends that it's it's in front of the right people and something is gonna be done. So we're we're gonna see improvement.

Chris Maher: 13:17

Hopefully, conversations like this, and we can post this on our our podcast. I'm sure you're doing that with your own podcast. We just get we got to get the word out, right? And so part of that is how can people, our listeners, and hopefully this gets shared, how can people learn more about this work that you're doing?

Joe Devon: 13:33

Yeah, first obviously just go to the website and check it out, amac.ai, aimac.ai, look around, learn how it works, and and something interesting about it that that regardless who you are or or what you do, should be pretty interesting. Is we have a thousand and eight web pages that were generated by AI. We have a gallery, so you can you can view the HTML that was generated, you can view the prompts that created that page, and you can view what that page looks like. I don't know of any other place where you can compare like how do the different AIs, not only how accessible they are, but what does the design look like? Here you can get a look and pick which which AI model to use on that basis. And even though that's a bit outside of accessibility, I think that being an interesting resource for people in general, it should get the AI model companies to pay more attention and say, hey, you know, we're people are picking their models on the basis of what they see here.

Kelvin Crosby: 14:31

Yeah. When uh well, I think one of the things is as we've been talking about at beginning of 2026 here, even in coming into this year, that we've been talking about we have to get ahead of the game of AI. And if we don't get ahead of the game, we're gonna be left behind like we've done with the internet. And so this is where I'm like, and I think we're so close. Like we're starting to hit we're knocking on doors louder than we've ever knocked in in in all of our lives. And we're starting to get people's attention. And I think what's gonna be interesting, I I'd be interested to see if 2026 is the year that we start making we start seeing AI really become accessible, like we just talked about. Cause I think once Google starts realizing, oh, this is like their accessibility team starts realizing they're gonna start knocking on their door. Like and then the same thing with some of the other platforms and start realizing we gotta address this issue. And and I think this is one of the things that as somebody that is Deaf/Blind and so forth, uh, is like this is this is the the way to our future. And for me to continue to run my businesses, to me for me to do the different things, knowing how to run these things and being able to have access to it is gonna change everything for me as a business owner, uh, run my businesses and so forth. And if I don't know how access to it, it's extremely important that we can get the word out.

Chris Maher: 16:01

And and and I think Joe, to your point, I Calvin, I agree with you 100%. And I think Joe, you made the point earlier that you know if if this is done the right way, AI and these models and code can have a profoundly positive impact for everybody, not just for somebody, but in particular for the disability community. But if it's not done the right way, we're gonna have, again, another piece of very powerful technology that's leaving a lot of people out. Right.

Joe Devon: 16:29

Exactly right. And and I like the way that you said it, Kelvin, that we have to lean in. I'm not I'm not asking for any favors. I think none of us are sitting there asking for favors. We're asking for the basics, the table stakes. And and the truth is we're doing them a favor because when you're when you're an AI model company, what are you trying to do with artificial intelligence? Is you're trying to emulate human beings. So what what is what is it about personalizing technology for human beings is understanding the different abilities that humans have. And and if you want to do great with that, you start focusing on how the edge cases, and and you get the edge cases right, you're gonna get your your models will be way better. Guaranteed. Like anybody that has tried to get to work with with accessibility has found a much better user experience all around. So like this is really doing them a favor by calling it out and saying, you have to pay attention to this. And then let's take the other aspect is demographics. About half of the US population and probably around the world, we're all aging. The populations are aging. So over 50% of the population is over 40. And as soon as you hit 40, you start to reduce you, you know, your vision goes down, your hearing goes down, and you've got a you've got to focus on this group, and if you don't know it now, you're gonna know it tomorrow. So the CFOs that are ignoring it today don't realize what they're doing.

Chris Maher: 17:59

It it's in their best interest. Yeah. For their for their business and their bottom line. Yeah, exactly.

Kelvin Crosby: 18:06

Exactly. I'm curious, like so what are some things like our audience that are listening to this, like what are some things that they can do to start getting other people's attention so that way we can start like really blowing up people's inboxes, getting Congress attention. What what do we need to do and what strategy should we go for to make that happen?

Joe Devon: 18:29

I'm gonna repeat what what how GAAD went so viral to begin with. When I wrote that post, I never thought it would go the way it has, but essentially we we said let's take a day and focus on on growing awareness on accessibility. Write a blog post on the day, write to a journalist, ask a journalist to cover it, and we have gotten an amazing amount of press. This has worked extremely well, and uh all I'd say is you're you're you can't ignore AI. You're talking about it already. Check out the benchmark and talk about the results. Talk about, and and you know, I'm not saying this in a selfish way, we are doing this under a nonprofit anyway, but it doesn't have to be around AMAC. Pull up a screen reader and then try Claude code, try Codex, try the different tools, film yourself using those tools, right? And Kelvin, if you want to do it, that would that could be a good project for us all to sit there and say, hey, let's let's let's try out Claude Code with a screen reader and listen to that to the garbledness because and I only know this because while I'm testing uh with a screen reader and then I'm switching tabs to Claude Code or Codex or any of these, I just see a garb I hear a garbled mess. So I know it's not working, right? So let's let's show people what it's like for you to try and code.

Kelvin Crosby: 19:50

Well, and I think that's so important. Like, I mean, when for me being a web developer, being a CAD designer, being All these in things like I've I've just been able to do all this stuff, but I've had to build my own system around it because none of it's accessible, right? And so this is where like I now that we're like I'm starting to get into AI at a whole different level, and I'm like, alright, cause I want to start building apps, I want to start doing these things, and I'm like, this is the only way I know how to do it. And so this is where I'm like, alright, how do we get to this point so that way I can when there's a problem to be solved, I can quickly get online, get on Chat GPT or something, use the right model, and then explain what I need, write the right prompt, and then troubleshoot it, and be able to keep troubleshooting until it works. And I think that that's so crucial. Like we can it's literally solving for the most simple things that we need access to uh just on a daily basis. And if the people with the disability can actually write their own stuff to get these access to the world, I mean I I just think about a Deaf/Blind person. Like I can visualize if I could teach a handful of Deaf/Blind people how to Chat GPT code or whatever and how to do that, think about what they can create to get access to communication. Think about what they can create just to solve their own daily life struggles because the code is accessible and they they're learning it, they're strategizing it, they're they're getting the concept. I mean we literally could solve for so many problems, but just the only thing we need is just access. And that's super important. Yeah, and and and I love where you went with that, and I meant to go deeper in in that that avenue too, is we're not victims. We're we're not sitting here as victims. We are we are going to demand that the companies do what they what they should be doing, but we're gonna build tools around it. But if you if Open AI is gonna be the one paying attention to accessibility, and it's not that big a surprise when you think about it that they were successful since they they were launch partners with Be My Eyes and created Be My AI, I guess, with them. The they're they're paying attention to uh disability, and that's why they're successful. And we can just decide we're gonna use them instead of somebody else. But if we want to use Claude Code, we're gonna build stuff that will make it accessible. And and whenever I talk about AI, there, you know, I have some friends that have been out of work for a long time in the accessibility space. It it really hurts that that's the case. There's very great people that should be, that should have amazing jobs that that don't, and probably part of it is AI. But but you know, if I was sitting here out of work, which essentially I was, right, just do something to fix these problems and and make the most out of it and lean in. Because if we don't lean in, we're not gonna stop AI from going where it's going. But if we lean in, we're gonna have a say. And so I say lean in, right?

Joe Devon: 23:05

I uh and it's uh for some reason that's controversial,

Chris Maher: 23:08

But Joe, I think and part of like the m a big takeaway for me on this is not aha, gotcha moment on these platforms. This this is a win-win opportunity.

Joe Devon: 23:18

Oh Yeah.

Chris Maher: 23:18

This is an absolute win-win opportunity because by creating awareness around this and shining a light that some are more accessible than others, that's not only good for the community and good for more people if things are more accessible, but it's also good for it's in their best interest, these platforms to make themselves more accessible, it's better for their business. And so it's an absolute win-win here. And this doesn't, I don't think this has to be this confrontational antagonistic thing. Because at the end of the day, everyone wins out.

Joe Devon: 23:44

Yeah, absolutely. And I'm I'm very against the the name and shame for the simple reason that every time I I start something, and it's including AMAC, there's some some purists on the sidelines that will say stuff, and I'm like, this is more true than you can imagine. This this effort is useless. Like, I'm not behind it. And and this effort is going to make a difference in the accessibility of AI models. I know that this is happening. So you know, you can you can sit there and and and try and shame, like we we we did um React Native took the GAAD pledge to make their their open source project more accessible, and then some of the purists came out and they're they were upset because we asked the community for feedback on what was happening, and they're like, You you shouldn't ask the community for feedback, you should be paying in order to do this. And you know, we had so many conversations of well, if we don't involve the community, then people are gonna be upset that we didn't ask the community. But you know, Meta paid or back then Facebook paid for making it more accessible. What what more value can you do than saying, okay, I'm gonna I'm gonna focus on making it accessible? You can't go like go out and name and shame them for doing you know for for attacking this. So, you know, that I'm very conflicted because it's like you had to, I somebody had to go out and say, what's happening here?

Chris Maher: 25:18

Right. And but I think your approach, I think it's the more productive one. I I think the whole shaming thing and making it more antagonistic, I think that ultimately gets people on the defensive. It's counterproductive. I think your approach, and it's the way you've always been, Joe, like like since I've known since I've known you, but also knowing the arc of your career, you've always been incredibly passionate and you've always been engaged, you've always tried to do what what what's the right thing to do, and and it's your leadership, right? And this is another example of of your amazing leadership in this sector of the market and for this community. And just, you know, kudos and and thank you for that for that. And I think your approach of hey, you know, we're not trying to get people in trouble, we're just trying to make people aware of the lack of accessibility here, and let's address that, and then it's a win-win for everybody. And that and that's what we want. And that that's when you get the best outcome in my mind.

Joe Devon: 26:09

Yeah, yeah, exactly right. I mean, Google does some amazing work, and they came in last, they happen to come in last this time. But I I you know, I follow the people building this technology, they're working like 70 hours a week. So by highlighting this, we're just trying to take like a few percent out of that 70 hours, you know, to say you got to include accessibility in it. But it's not like, oh, you're you're bad people or that you're intentionally ignoring this. It's more, it's more like you may not have realized how important it is. So we're just gonna help you sort of see that.

Chris Maher: 26:42

My sense is they're gonna get away, get a get wind of this and they're gonna probably start to address it pretty quickly, is my is my guess.

Joe Devon: 26:48

Absolutely. Absolutely.

Chris Maher: 26:50

All right, Joe, you are the best, my friend. Thank you for spending some time with us. As always, so great to see you in person, and hopefully we'll have you back for a third time very soon.

Joe Devon: 26:58

Yeah, let's hope uh I'll come back and say that they all aced the AMAC benchmark.

Chris Maher: 27:04

That will be that will be a great follow-up episode. Y

Joe Devon: 27:06

Yeah,

Chris Maher: 27:06

And hopefully that will be sooner than later. Thank you, Joe.

Joe Devon: 27:09

I hope so. Thank you.

Kelvin Crosby: 27:10

All right, that wraps up Investing in Accessibility, as I always say, go live beyond your challenges and we'll see you in two weeks. Thank you for listening to Investing in Accessibility, a Samaritan Partners podcast, where we invest in change for accessibility, not wait for change. If you want to follow us, you can find us on YouTube or LinkedIn at @Samaritan Partners. If you would like to invest in Samaritan Partners, email Chris at chris@samaritanpartners.com. If you'd like to learn more about us, go to www.samaritanpartners.com. You can take the first step in investing in change by giving us five stars and sharing this podcast with everybody that you know so we can spread the word so that we can give access to all by Investing in Accessibility.