Tag Archive for: Chatbots

 

Listen to the full episode right here, or wherever you stream podcasts.

Find all episodes at HelloMergeTag.com or wherever you stream podcasts.

Links

The white paper Jack wrote with Meg Schwenzfeier

Caucus AI

Jack’s email | LinkedIn | Twitter

The blog post I wrote about asking ChatGPT to compare the email programs of Joe Biden and Donald Trump

 

Big thanks to our sponsors:

Civic Shout. Learn more about the great work they’re doing to help Democratic campaigns and progressive nonprofits build their lists ethically at civicshout.com/partners.

 

Episode Transcript

This transcript was automatically generated and has not been copy-edited. While it can serve as a helpful guide to our conversation, it may contain errors or omissions. For the most accurate representation, we recommend listening to (or watching) the full episode.

Josh Klemons: Jack Welty advises progressive organizations on technology strategy, data infrastructure, and product development. He served as deputy chief analytics officer on the Harris for president campaign and led paid digital analytics for the 2020 Biden Harris campaign. He co-authored a paper on practical AI adoption for campaigns and is the co-creator of Caucus AI, a research tool that monitors what AI chatbots tell voters about candidates and elections. First of all, thank you so much for coming on the pod. I really want to talk about the paper you authored, Generative AI on campaigns. In it, you argue that Democrats need to get serious about AI, but not for things like content generation, deep fakes, but for the boring stuff. Before we get into what that means practically, just give us the elevator pitch for why you think any of this matters right now.

Josh Klemons: Maybe it’s obvious to some, but like it’s probably not to others. So, lay it out for
Jack Welty: Yeah, totally. Uh, and thank you so much for for having me on. Um,
Josh Klemons: us.
Jack Welty: yeah, I mean, of all of the organizations in politics, campaigns are, as we all know, the most resource constrained. So, um, I think generative AI, especially the the big leaps we’ve seen in the tools in the last like year or so, offer an opportunity for our campaigns to get more efficient, uh, get more done and leave more time for our, uh, you know, talented staffers to focus on the hard stuff, the actual strategy problems, talking to voters, etc. Um, and why I think it’s important right now is, um, the American Association of Political Consultants has run a couple of studies uh, of their members. most certainly recently uh in March um that show that there’s a gap between uh Republican and Democratic consultants in uptake. So we are falling behind Republicans in daily AI use in building custom AI tools for our campaigns and in generally like adopting this stuff to make our organizations better.

Jack Welty: So um you know I don’t want to handwave away a lot of the concerns a lot of uh uh you know moral ethical political etc that we might have with the labs and with this technology but every day that we are not adopting this stuff the other side is moving faster and I think ultimately like that’s going to translate into into votes um so I think it it kind of behooves us to to pick up the
Josh Klemons: Yeah.
Jack Welty: pace
Josh Klemons: So the numbers you like the AAPC 44% of Republican consultants are using AI daily versus only 28% of uh of Democrats. So, I guess my first like what do you think Republicans are using AI for that Democrats aren’t? Um like do you have a sense of
Jack Welty: Yeah, that’s a great question.
Josh Klemons: that?
Jack Welty: Um, there’s a little bit more in that AAPC study. Um, I don’t think they’re using it for anything crazy. I mean, we are seeing some of the like content generation wars and we don’t have to get too deep into like

Josh Klemons: Okay.
Jack Welty: the NRSC memes. I think a lot of that stuff is probably going to backfire. Um, what Meg and I focused on more in the paper is like just just wrote task automation. You know what I mean? Like I think that Republicans that are using this have more time in their day to think about strategy and higher level problems because they’re spending less time summarizing memos or data entry or what have you. Um, and so that’s where we’re falling behind less on the sort of I don’t think anybody should should put their campaign strategy, you know, totally up to Claude or anything.
Josh Klemons: That seems like a good losing strategy for sure,
Jack Welty: Yeah.
Josh Klemons: right? A tool that doesn’t really understand what the hell’s actually happening in the world,
Jack Welty: Yeah.
Josh Klemons: doesn’t understand real time results. Um, the other day I asked it a question and it had to like look up who Zon Mom Donnie was and figure out who was the mayor of New York. And it’s like, yeah, if if you don’t understand the context, you’re probably not going to give me like updated answers.

Josh Klemons: But like the argument you’re making is that AI like I think a lot of people the movies and you know like the internet makes it seem like AI is about the sexy stuff like you know the NRSC is running ads now she’s dropped out but like Janet Mills using AI but you’re actually arguing that we should be using more of like the boring tools the automating high volume low complexity work that’s taking up staff hours which is what you just alluded to Republicans are using it so they could spend more time thinking about strategy. So, what did that look like on the Harris campaign? Or maybe it was too soon to say because AI’s come so far since then, but like in your mind,
Jack Welty: Yeah.
Josh Klemons: give us some concrete examples of like what would you if you’re getting hired as a consultant tomorrow or you’re just talking to a friend who’s just drowning in work,
Jack Welty: Yeah.
Josh Klemons: what’s the first thing you’re doing to help take some of that off their plate and letting AI take the boring stuff off their plate so they can do the actual like important

Jack Welty: Totally. Yeah.
Josh Klemons: stuff
Jack Welty: I have I have a great example from the Harris campaign and again this was you know far in ancient times
Josh Klemons: early in AI. Yeah, I
Jack Welty: uh in AI development back in in you know summer fall of of 2024.
Josh Klemons: know.
Jack Welty: Um but we asked um organizers and volunteers to fill out post shift feedback surveys. How was your canvasing shift? What did you hear on the doors? Did you encounter any issues? yada yada yada. Um, and this was like a great very rich source of data like God bless the volunteers that were,
Josh Klemons: Of Of
Jack Welty: you know, writing detailed notes about their camping experience, whatever. Um, but as you can imagine, especially at a national presidential campaign volume,
Josh Klemons: course.
Jack Welty: this was thousands and thousands and thousands of just free text surveys per day. Um so even with the like again in 26 terms relatively primitive tools available to us um we fed all of that into an LLM um and asked it for structured reports back.

Jack Welty: So we wanted structured summaries broken out by date and state and like type of issue. Were there specific uh flags people were running into? Did a did a staging location not have yard signs? Um, and then also sort of squishier sentiment like what were people telling you commonly on what were we hearing back commonly on the doors from voters etc. Um, and so it turned what would have you know previously been like an absolutely impossible human task of of reading through all of these responses into a like 10 minute click button run pipeline get back structured report. Um, and I think what’s really exciting about that was, you know, one that we were able to get all this actionable information that previously would have taken a bunch of staff hours. Um, but here in 2026, the tools have changed so much that two years ago, this required an engineer and some custom Python code and hitting an API and whatever. And right now,
Josh Klemons: Right.
Jack Welty: like I think my first move would be to drag that folder of surveys into Claude and ask it for a summary.

Jack Welty: And I think that would give me like a pretty good first pass. Uh compared to what required a ton of custom engineering, you know, 18 months
Josh Klemons: Yeah, that’s super interesting. Were the I’m very curious about the technical mechanism of that.
Jack Welty: ago.
Josh Klemons: Were those typed summaries or were they handwritten? Like could you have handled it if like people were filling these out at events? Like how does that work?
Jack Welty: Yeah, the these were mercifully typed surveys uh just in the back end of a Google form.
Josh Klemons: Okay, so it was pretty easy to just copy and paste.
Jack Welty: Yeah, but but the vision models have come a really long way too.
Josh Klemons: They really
Jack Welty: Uh yeah, we’ve we’ve talked about that. I I think this was in the paper as a potential use case.
Josh Klemons: have.
Jack Welty: Um uh petition uh uh survey verification. So, you know, to get on the ballot, you might go have to collect um you know, thousands of signatures. and previously that’s a that’s a human task to go type that up, look those people up in van, make sure they’re registered to vote in the right, you know, county, precinct, state, whatever.

Jack Welty: Um, I think with some of the these vision models, like we could absolutely develop a pipeline to turn that into text and then automate the lookup and um, you know, that’s tons and tons of poor ballot access fellows time uh saved per
Josh Klemons: So,
Jack Welty: campaign.
Josh Klemons: I do a lot of public speaking on like digital marketing and these days I just have a link on my deck and it’s like if you want to join my mailing list, here’s how. But back in the day, I used to like bring paper, you know, like, you know, similar to a petition kind of thing. Um, and I have these like vivid memories of sitting and trying to like sort through each and every one of those and people’s handwriting is terrible, mine included. Um, and I’m like trying to figure out what the hell that says, and then I’m having to troubleshoot it. And yeah, I like I can’t even imagine the idea that I could just take a photo of each page and be like, could you turn this into a spreadsheet? Uh, and now I’m talking about 10 years ago, but yeah, like today versus even like a year or two ago,

Jack Welty: Yep.
Josh Klemons: I have seen that like it can now like look at an image and like dissect it in a way that it could not a year or two ago. A year or two ago, you kind of had to describe it to it and today I can look at it and be like, “Oh, this is nicely designed or no, this needs work.” It’s like I don’t use chatbt for or I don’t I I stopped chat.
Jack Welty: Yeah.
Josh Klemons: I’m on cloud now, but I don’t use it for any design work, but I do sometimes feed it a graphic and say, “Do you like this?” And it gives like surprisingly good feedback like for even for designers out there like it’s worth
Jack Welty: Give me some feedback. Yeah.
Josh Klemons: knowing like if you just like have that nagging you know thing in your head saying this could be better like cloud might be able to tell you if it could be better which I think again that’s where the idea that like only about a quarter of Democratic consultants
Jack Welty: Yeah.
Josh Klemons: are using AI at all is like wild to me because I am very selective about when I use it.

Josh Klemons: I’m not using it for any strategy. Like I’m the strategist. That’s like my whole thing. But like it helps me think things through and like talk it out.
Jack Welty: Yeah.
Josh Klemons: And the idea that three out of four of us us like Democratic consultants are not using it yet.
Jack Welty: Never mind.
Josh Klemons: I mean, it’s a limitation. Like it’s I get why some aren’t. I I have questions on that. We’ll get to that. So, you describe AI as an eager but inexperienced junior staffer. So, like what does that mean for you? And like how should folks be thinking about their AI tools? Uh they’re not managers. You’re arguing they’re not managers,
Jack Welty: Yeah.
Josh Klemons: they’re interns, right? I mean, is that essentially the argument?
Jack Welty: Yeah, that that’s kind of the argument and and I think um what this means is we have to shift into that managerial critic role. Um, for some of us that have managed people, like hopefully this is a natural one, but uh, for junior staff that maybe have not managed work yet, uh, in their careers, I think that it it moves that staff development pipeline left.
Jack Welty: Uh, and so even junior staffers on campaigns need to pick up skills like delegation, review, criticism, you know, breaking work up into bite-sized tasks and stuff like that. Um because yeah, these LLMs have, you know, boundless energy as long as your token budget will account for it. Um but they need direction. Uh they need steering.
Josh Klemons: right?
Jack Welty: They need review. They will confidently plow forward with an incorrect assumption or present to you a wrong answer sort of wrapped in, you know, nice formatting. Um and it means that we have to be like really hyperritical of their outputs. We have to be good reviewers. we cannot just be you know mashing the the tab button or accept at all times. Um and then I think the other big development is you heard a lot in uh late 24 early 25 about prompt engineering. There was this idea that I mean you could tell an LLM like I’ll give you a $100 if you do this task and sometimes it would do the task better.

Jack Welty: Um this latest generation like that doesn’t that stuff doesn’t really work anymore and that was kind of a weird artifact. Um, but context engineering is even more important. And so this is sort of a fancy term that I think applies to the kind of context you would give a junior staffer. Imagine you’re onboarding someone to your organization and you want them to complete a task. You need to give them some background about what the organization is, what you believe, what the goals of the organization are, what the parameters of the task are, what the goals of the task are. Um, I think if you were asking an intern to write a memo, you would maybe give them three past examples of good memos and that would make your intern write a better first draft. Like thinking through all of those things, the more you can carefully curate good context for these models, the better answers, the better results you’re going to get than if you just again sort of like throw it some sort of raw blind question of, you know, write me a first draft of this blog post or what have you.
Josh Klemons: Right. Asking an intern to write a memo when they’ve never seen the type of memo you’re asking for is going to guarantee a bad Right.
Jack Welty: They’re gonna do a bad job.
Josh Klemons: Like, and it’s not their fault. It’s your fault.
Jack Welty: Yeah.
Josh Klemons: Like that’s bad management. But like you’re essentially arguing it’s the same with Claude. Like you can’t ask Claude to write a memo. Like a press release and like the context.
Jack Welty: Yes.
Josh Klemons: Like sure, fine. It like understands what a press release is. But if you’re trying to write a specific type of memo without showing it past examples, you should not expect a good output on the first take, right?
Jack Welty: Yeah. And if your organization has a style guide or certain conventions you wanted to follow,
Josh Klemons: And
Jack Welty: that’s something you would give that intern or junior staffer as well. So you should feed that into
Josh Klemons: yeah, I mean, in general, you talk there’s quite a bit about the idea of this being like a management challenge,

Jack Welty: club.
Josh Klemons: not a technical one. And I think that’s interesting language like that I haven’t really heard people talk about. Like so I I am not a coder. I don’t know any I I can’t write code to save my life. I just launched I just vibecoded my first tool. It’s called Shares Sparrow if you go to sharespar.com and it literally lets you take tweets and blue sky posts and turn them into um Instagram friendly graphics. And it’s an idea I’ve had in my head for years. There was a tool that did it. They went away and I was like I’m going to replace it. Um I’m not technical. It was like it’s an interesting idea that it’s a management tool because a lot of it was going back and forth like telling it like no you could be better here. There’s problems here. I really had to like micromanage. like that’s not language I was thinking about when doing it, but I had a hard time figuring out like the Twitter logo.

Josh Klemons: It was like recreating it from scratch. I was like, “No, that’s not the Twitter logo. It doesn’t look like the Twitter logo.” And I had to literally micromanage it to get it there. There was no technical component to it at all. It told me what to do. I did it. I told it what to do. It did it. Um, and we like literally had a conversation while it built a tool that I was excited about. So, I guess what I’m curious about is like of the 28% of us Democrats that are using it, is there a generational breakdown? Like do you have a sense like is the older generation better off for this or worse off for this? And vice versa the younger generation like who is better suited to actually use this? Like somebody I’m in my 40s and I don’t know how to code anything and I don’t need to now. Um somebody in their 20s who like grew up with AI essentially at this point. Like are they better off or worse off because they don’t understand the world without it?

Josh Klemons: Like do you have a sense of that? Like that’s that’s my age-old question I guess when it comes to this kind
Jack Welty: Yeah. Yeah.
Josh Klemons: of
Jack Welty: I think that’s a great question. The unfortunately the the AAPC data does not have an age breakdown and I am sort of on the hunt.
Josh Klemons: Yeah. Seems like they
Jack Welty: Yeah. I’m I’m on the hunt for for more data on this question. So if anyone knows of any,
Josh Klemons: should
Jack Welty: please send it my way. Um, I think it kind of cuts both ways because on one hand, um, you know, us us gri grizzled vets, uh, folks who have been in this space for a while, we know what good work looks like, um, and we know how to be good critics. We know how to give feedback. Um, we have good, you know, definitions of done. Um, and can sort of broad these things into shape, uh, and know not to ship slot basically. Um, so I do think uh uh as long as you know folks from an older generation are willing to lean in and learn the tools, there’s definitely an advantage.

Jack Welty: You know, all of our hard one experiences is not for not um on
Josh Klemons: Like you’re arguing that the experience to not put out garbage matters.
Jack Welty: the
Josh Klemons: Whereas if you don’t really know what good versus bad looks like, it doesn’t matter. Like what I’m hearing is it doesn’t matter how well you understand the AI tools. If you don’t understand what good output looks like, you’re never going to have good output.
Jack Welty: and not just good output but good process as well. Um uh I’ve been exper experimenting a lot more. So I’m a I’m an analyst.
Josh Klemons: Okay.
Jack Welty: I’m an analytics uh professional by trade and um have been exper experimenting a lot more with using these things to produce analysis. So having them you know write SQL, write Python, compute stats and then write analysis.
Josh Klemons: Are you a coder?
Jack Welty: Um I am yeah yeah I’m an engineer.
Josh Klemons: Just so we understand. Like Okay. So you’re like you’re much more advanced than like the average Democratic consultant when it like I I know what Python is.

Jack Welty: Um
Josh Klemons: I have no idea what how to use it. So Okay.
Jack Welty: yeah yeah yeah I’m I’m I I I’m an engineer uh when I wear some hats.
Josh Klemons: Okay. Got
Jack Welty: Um but I think knowing but I think it’s applicable to like any technical or any technical or
Josh Klemons: it.
Jack Welty: non-technical process. Um you have to know the process steps and the gotchas. Um you know even taking a a non-technical uh approach. Maybe you’re a researcher and you want Claude to help you write a research book for a candidate or something. I know I’m gonna get like a lot of researchers who come yell at me that that that’s not possible, but you wanted them to propose a research book. They Claude might go research four of the five, you know, top sources you would go check and then write you a report based on those four or five top sources. Four sources. But if Claude never checked the fifth source, you’re potentially going to be missing some information.

Jack Welty: If Claude never, you know, looked the candidate up in the voter file or something, you’re going to be missing some critical piece of information and Claude is not going to know that it’s missing. Um, I see this in analytics work all the time. Um, Claude still makes silly SQL mistakes or Claude makes assumptions about what data is in what table and doesn’t necessarily surface those assumptions for you. It just produces a nice looking deliverable that has pretty charts. Um, and so the knowledge of all those process steps, the common gotchas, the oh, if you don’t if you forget to filter voter registration status on this field, then your counts are going to be inflated in this way. Like all of that, um, you know, kind of domain knowledge is still really important and frequently for specifically our domains, it’s not in the training set. like a lot of best practices about how democratic campaigns operate are blocked behind the analyst institute payw wall or just in our heads or in our you know organizations Google drive and so that’s the type of context we still need to give these models to get good outputs um I will say on the flip side the folks that have grown up with this I think are going to be like really really dangerous in a great way um I think the AI native you know I think dropping a couple of AI

Josh Klemons: Right.
Jack Welty: native 24 year olds on a a modern campaign is going to be produce some like really really impressive outputs uh because they just you know folks have learned to use these tools uh from go.
Josh Klemons: Hey
Jack Welty: They don’t know of a way to operate without them and it means that they can sort of execute at at pretty incredible velocity. So it is not to say that like I’m discounting that entirely like I think those folks are going to really cook when they get into this space. um they’re just going to need to, you know, find a way to sort of distill some of this knowledge
Josh Klemons: Yeah. No, I think that’s really interesting. the part that like I I’m very interested in the idea that output and process can’t be differentiated especially for
Jack Welty: down.
Josh Klemons: what you’re talking about like so like building a graphic like not with AI like if I’m working with a graphic designer and they make a graphic I know if it looks good or not regardless somebody could build a website that I think looks good but is actually terrible and it’s much harder to know because process does matter because it could look gorgeous but the whole thing’s just one big image and not actually

Jack Welty: Yep.
Josh Klemons: live text and like for SEO purposes it’s trash and I see that like clients come to me with a website and I’m like uh this is a Canva image like this is not a website, you know, like I’ve seen that and like I think AI is like the next step and I hadn’t really thought about it that way like the output matters but like to get there you really have to think about the process and right if you don’t have the experience like like you learn the process the hard way pre- AAI so you know what you’re looking for I guess the question now and I don’t know if you have an answer for I don’t even know if it’s a question like how do we ensure that the like AI natives who are like coming onto the campaigns are going to like shake things up in the best possible way I agree like that new energy is always what gives our party life. How do we ensure though that they understand the processes so that they can like make sure that like AI is not building them a fancy website that’s actually trash?

Josh Klemons: You know, I think that’s the question.
Jack Welty: Yeah. Yeah.
Josh Klemons: That’s that’s the million-dollar question right there,
Jack Welty: And and I think that’s that’s the shift to the role of manager and critic. You know what I mean? Like we’re we we are not just shipping pure AI driven,
Josh Klemons: right?
Jack Welty: you know, creations. We are asking follow-up questions. we are pushing AI to prove itself to prove those process steps to verify its work in a whole bunch of ways. I mean it’s why you’ve seen AI uh advance so quickly in software engineering because those outputs are so much more verifiable like you can test the API that you’ve built and it either works or it
Josh Klemons: Right.
Jack Welty: doesn’t. Um whereas for something like analysis um you can’t you know prove that the conclusion of
Josh Klemons: Right.
Jack Welty: the analysis is right without fully inspecting all the outputs. So, I mean, that’s even a tip for I think AI use is is make these things explain to you what they did um and produce intermediate artifacts and present to you a plan for what they’re going to do before they go to do it.

Jack Welty: Which again is the type of thing that if it was an intern’s first day and you told an intern to go write a report,
Josh Klemons: Right.
Jack Welty: you wouldn’t say like, “Hey buddy, like have a good day. Come back to me at 5 when the report’s done.” You would say like, “Hey, no, like first you should gather this this particular data here.
Josh Klemons: Right.
Jack Welty: You should analyze it in this way and then you should check in with me once you’ve done that before I send you off to go write the memo or
Josh Klemons: As somebody who’s very publicly like building AI tools for the party, I assume you’re having these conversations.
Jack Welty: whatever.
Josh Klemons: Do you find that folks are having to learn that they they need to focus on process? uh like are most folks jumping in and saying write me a report to cla or chat GPT or do they recognize that that’s not how it works?
Jack Welty: Yeah, it’s a good question. I think everyone goes on their own journey. Um, I think there is there’s certainly I think folks approach it from a couple different directions.

Josh Klemons: Yeah.
Jack Welty: Both are totally valid. Like I you watch people um maybe dive in a little bit too deep and get really excited about the hype and then have to kind of pull themselves back. Um and on the flip side, you have uh folks that are, you know, totally justifiable skeptics for whatever reasons that take a sort of slower approach. And you’ll see those folks um I think advance more deliberately and maybe a little bit more carefully through all the steps of verification before they get to a final
Josh Klemons: Okay. No. Uh,
Jack Welty: output.
Josh Klemons: super interesting. So, let’s talk Caucus AI. Um, tell us about Caucus AI. Like, I I poked around on the site quite a bit. Uh, I read the about section. Um, like what does it do? Why’ you build it? Like, tell us the story behind it.
Jack Welty: Yeah. So, um, uh, Caucus AI, Caucus-ai.com, um, is a product of myself and and Meg Schwenfire, who’s my colleague from the the Harris campaign.

Jack Welty: Um, you know, a AI is replacing search. Um, millions of Americans now ask uh chat bots the questions that they used to type into Google or they still type those questions into Google and they get an AI summary instead. Um, and so uh we realized that there was no one really systematically capturing and measuring what these things were saying about um uh about political candidates or in response to political questions. Um and so you know being the good analysts that we are uh we just started by collecting data. Um so Caucus AI monitors right now three different chat bots across we have about 1100 races in the tool right now. Um we ask um an election level question just like purely factual who’s running for governor in Maryland. Um and then we ask a handful of questions about each candidate in the race. So, um, a generic bio question, a question about where they stand on issues, and then a question about their top criticisms. Um, and then we sort of present all of that data, uh, in our website.

Jack Welty: Um but then we’re also doing a whole bunch of research uh on that data that we’ve collected and and publishing that on our substack.
Josh Klemons: So you found that which chatbot a voter uses um determines which media like essentially it matters right like and it goes without saying like if you’re asking Grock which is like Elon Twitter’s you
Jack Welty: Yeah.
Josh Klemons: know AI you’re going to get a different response than if you ask cla Gemini or chatpt so like how different are the responses like what are you finding when it comes to like these different
Jack Welty: Yeah. So the the biggest difference is in the media ecosystems.
Josh Klemons: models
Jack Welty: So most of the responses are broadly factual. We are not finding um that anyone is like totally off the deep end. Um even even Grock Grock is uh unsurprisingly uh quicker to site
Josh Klemons: even
Jack Welty: the controversies uh about a political candidate. Um and Grock is actually on on either side.
Josh Klemons: on either side of the aisle or mostly just de Okay,
Jack Welty: It’s it’s still kind of early days in our research.

Josh Klemons: cool.
Jack Welty: Um, we also these things are changing rapidly all the time.
Josh Klemons: Sure.
Jack Welty: So, we’re going to be collecting these responses, you know, at least monthly through the election. So, I do think it’s possible we see things start to change as we get closer to November. Um, but uh, yeah, Grock is actually so the the media ecosystems and and and who gets cited is probably the biggest difference. Um, Grock is the only large model that we see citing social media. So obviously it sites sites Twitter um but it also sites in No Grock
Josh Klemons: only Twitter, right? It’s not citing anything.
Jack Welty: sites Grock sites Facebook Grock sites Instagram Grock sites LinkedIn um there’s actually
Josh Klemons: What? That’s
Jack Welty: a pretty wide variety of sources.
Josh Klemons: wild.
Jack Welty: Um Chatpt so OpenAI’s model um really leans in on their big national media partnerships. So OpenAI and Axios have a deal same with Washington Post with AP. So um Chetchup Tea is much much much more likely to site those new sources first um before going anywhere else.

Jack Welty: Um and then Gemini casts a much wider net. So for the average query, you know, Gemini sites like twice as many sources. Uh Gemini is quicker to site local news, stuff like that.
Josh Klemons: Is there a reason you didn’t include Claude? Is it more complicated or it just didn’t seem as relevant as the other three?
Jack Welty: When we started the project, Claude had like low low singledigit consumer US market share. Um, I think the thing to remember is like, yes, a lot of us wonky white collar knowledge workers are using COD, but your average American when they think about asking a chatbot political questions, they’re just using chat GPT. Um,
Josh Klemons: Got
Jack Welty: so we’ll probably add COD to our analysis.
Josh Klemons: it.
Jack Welty: Um, especially it it has risen in popularity in the last couple months with all the the DoD drama and whatnot. Um, but like 80% of US consumers are using Chachi.
Josh Klemons: Right. So talk to us about the uh experiment you ran with chat GPT and a brand new Wikipedia
Jack Welty: Yeah. So um you know we created cog one you know we just wanted to measure what these things were saying but
Josh Klemons: page.
Jack Welty: uh all with the goal of understanding how can we affect these responses how do they vary um what is actually getting cited. So we conducted an experiment um you know Wikipedia is one of the largest sources of these answers. Wikipedia and ballot pedia are sort of the top two for lots of these models. Um, and we created Wikipedia pages for a couple of lesserk known Senate candidates in like uncompetitive states. Um, we followed all of Wikipedia’s rules. They met Wikipedia’s notability guidelines and they were like balanced, neutral Wikipedia pages that just cited uh sort of local news stories. Um, and I think the the big headline finding is it took just 12 minutes from the publishing of that Wikipedia page for it start starting to show up in chat answers. And as soon as Chat was citing the brand new Wikipedia page, twothirds of all of the the sources it cited in its answers were that Wikipedia page.

Jack Welty: And we found verbatim sentences from the Wikipedia page that we had just written uh you know hours before showing up in you know a completely unauthenticated new chatbt request for answers about this candidate. So um yeah I
Josh Klemons: Wow. So, Wikipedia polices itself, right? So, like in theory,
Jack Welty: mean
Josh Klemons: Wikipedia should never have like bad actors. Like, yes, it could exist for a few minutes, but it’s not going to exist for months because somebody’s going to catch it. But what you’re saying is like there are ways like ChatBT is definitely manipulatable. Like, that is clear. And if you’re winning the Wikipedia fight, then you’re winning the Chat GPT fight. So are talk to us about misinformation. Like are how worried are you about bad actors?
Jack Welty: Yeah.
Josh Klemons: And are there any ways that like us as a party can better inoculate ourselves uh from those bad actors? Or is Wiki is Wikipedia essentially should we all just be donating more money to Wikipedia to protect elections forever?

Jack Welty: I I certainly think we should be grateful. you know, thank your local Wikipedia editor when you get a chance.
Josh Klemons: Seriously. Yeah.
Jack Welty: Um, uh, but I do think, you know, it it speaks to, uh, some of the both speed and opacity of these systems uh, that we don’t we don’t even really know what search index Chatt uses uh, to find this stuff. We know that right now at least it thinks Wikipedia is really trustworthy and so if you can change Wikipedia, you can show up there. But that’s not to say that that won’t change over time. Um I, you know, we have not found a ton of factual inaccuracies yet. Um but I think we are we are really in the early days here of folks trying to influence these. Um, I I think we don’t yet know if the other side is is going to, you know, create tons of astroturf Red Reddit posts to edit to to change one of these responses or, you know, push a ton of YouTube shorts to change the Gemini responses.

Jack Welty: I think it’s just it’s very very early to conclude that any of this stuff uh is is totally safe and that, you know, Wikipedia editors will save us all.
Josh Klemons: So you did find that they they seem to all actively be working to not like influence elections per se. um like they’re giving more like factual information with like even Grock who like sure as hell has controversial opinions on Twitter. Uh when it comes down to it, you’re not finding that they’re saying, “Oh yeah, that guy sucks. You should vote for this guy.” Like it’s they’re all being relatively um like careful about how they speak about elections. Is that what is that a fair assessment of your
Jack Welty: Yeah,
Josh Klemons: findings?
Jack Welty: I think that’s fair. And and we know this. I mean, the labs have have told us this. They do a ton of research um and they have a ton of incentives to keep these answers as neutral as possible, right? like the labs have giant political interests. They have big super PACs that are spending money now.

Jack Welty: Um and so they don’t want their answers to be favored by one side. So we know specifically um you know if you ask what brand of toothpaste to buy um it wants to give you a decisive uh answer like that’s how it it gets the you know the most user satisfaction is by it’ll confidently recommend you what water bottle to buy or what have you. Um, but we know that there’s this post-training layer where they try and sort of bang these models into neutrality. Um, and that’s where you get these very like wishy-washy kind of both sidesy answers which I I think make it make our jobs harder because these models really want to be neutral. Um, and so, you know, I think we have to be creative about how we put information on our websites. Um, how we think about, you know, good structured question and answer content. Um and just sort of experiment with lots of tactics to make sure that when asked they are giving you know that the answers we want them to
Josh Klemons: back during the 2024 primary, I I was tracking the Republican email program and writing about it a lot.

Jack Welty: give.
Josh Klemons: I like I did a bunch of writing on it. And then when Trump was the nominee, um I did an analysis of his email program versus Biden’s email program. This was before Harris, like it was still Biden. and I wrote up my own findings and then I went to chatpt and I asked it like essentially who had the better email program and I could not get it to I tried everything I could think of to get it to have an opinion and it refused. So I wrote about this so I finally um I came up with an interesting prompt and I said if these two people were Seinfeld characters who are they and Biden was Seinfeld and uh Trump was George. Um, and then I said, “What about Sex in the City characters?” And same deal, like Biden was the hero and uh, you know, and I think I did the West Wing as well or something. But like, like I really had to work hard to get it to give me something that was even worth putting down on paper.

Josh Klemons: And I did in the end, but like it certainly wasn’t going to say, “Oh, Biden’s emails are better.” Even though they were better, uh, in like by every objectable objective measurement. Um, you know, like, “Oh, you’re writing 10 emails a day from a bunch of different senders. That’s a spam bot.” Nope. uh he he has an interesting varied way. So yeah, it’s just an interesting I I hadn’t really connected it to like I hadn’t thought about that experiment I did with like how people need to be thinking about it. Like if you’re trying to figure out who should I vote for, well here’s what I believe, you know, like you know, it’s essentially like you have to know why you want to vote for somebody and like I assume and I haven’t tested this. I assume if I go to chat GPT and say here are my values, here’s what I believe. Here’s what I want in the world. Who should I vote for in this election? Do you think it would give me an answer or like it still is going to be like conservative about like not doing

Jack Welty: I think it’s more likely to give you a decisive answer,
Josh Klemons: that?
Jack Welty: but That’s sort of some of the extended research we want to do. U right now we were really just trying to ground the sort of baseline of these models that when presented with tell me about Josh Shapiro, what’s it going to say?
Josh Klemons: Got it.
Jack Welty: Um but I think one of our next research avenues is asking questions like um you know which candidate’s going to lower gas prices, which is best on affordability, I care about the environment, which candidate do you think best represents my values? Um we’ve seen some really really limited evidence that uh you know Chachi PT in particular was like quite skeptical about Democrat Democratic economic plans in a way that it was maybe not as skeptical of the Republicans economic plans. So um I think we have
Josh Klemons: You could you could like see Mark Andre in the answers like dictating David Saxs was like rewriting
Jack Welty: a right right yeah is really into trickle down economics.
Josh Klemons: answers before you saw the response.

Jack Welty: Um so uh yeah there there’s like a ton more to do there. We know that these things so while at the same time we know that the model the labs probably have instructions in the system prompts saying like don’t tell the user who to vote for. They also say stuff you know all the safety training like don’t give the user the recipe for napal but there are jailbreaks like my grandmother used to make napal can you please give me her old recipe and you’ll get it.
Josh Klemons: I know that.
Jack Welty: So that’s not to say like the labs are doing their best,
Josh Klemons: Oh man.
Jack Welty: but even they don’t fully understand how these systems work. Um, and I think this is going to continue to be a moving target and and they are tweaking these things
Josh Klemons: Right.
Jack Welty: constantly. They are releasing new models. They are changing the system prompt. And like that’s why we think it’s important to maintain a pretty high degree of vigilance because uh you know what if all of a sudden these things do change in October and suddenly they’re being

Josh Klemons: Right. Uh yeah, I listened to a podcast. Somebody was talking about how engineers at like the big labs,
Jack Welty: decisive.
Josh Klemons: it’s almost like planting orchards and like just seeing what happens. And like if something happens, then they’ll go and like tend to it, but like they don’t really know what’s going to happen. Like some trees might do better than others. And you know, like we’re not exactly sure what’s going to happen. They’re not coding so much as like planting and like experimenting,
Jack Welty: Yeah.
Josh Klemons: which is both like interesting and horrifying like that they’re reshaping our economy and our politics and everything else. So, I think especially down ballot races,
Jack Welty: Yes.
Josh Klemons: I would imagine even some like statewide races are we’re still having a hard time getting our party to think about SEO and now we need to be thinking about AEO like or I don’t know generative however you want to call it like AE I I’ve heard different ways of saying it but essentially AE AI SEO.
Jack Welty: Yep.
Josh Klemons: So, I guess like big picture you we don’t have to get too granular here. not looking for like, oh, make sure your all text is right, but like how should folks be thinking about AI SEO differently than like search SEO like which is redundant, I know, but just trying to clarify for
Jack Welty: Yeah, totally, totally. Um,
Josh Klemons: folks.
Jack Welty: I mean, the first thing I would say is don’t forget about SEO. Um, a lot of these things are using search indices under the hood to figure out which websites to go site. So, um, don’t forget about your robots.txt. Make sure your alt text and, uh, you know, your your structured HTML all looks good for the same kind of crawlers. Like, we we do have some findings that that stuff matters. Um, and we’ve even found candidate sites that have, you know, intentionally or otherwise blocked AI crawlers in their robots.txt. Um, yes, if you’re writing a blog, maybe you don’t want Sam Alman to index your posts and feed it into your training data, but I think if you’re running for Senate, you probably do.

Jack Welty: So, I would make sure you have you have those settings tuned, right? Um,
Josh Klemons: Right.
Jack Welty: in terms of the the AEO question, um, what we found so far is they really like structured data. They like question and answer formats. Um,
Josh Klemons: Yeah. Define structured data.
Jack Welty: so the more
Josh Klemons: So it’s not like a long blog post. It’s like a series of I mean FAQs would be like an example. What’s another example of structured data that like chat GPT or Gemini is going to like cling to?
Jack Welty: FAQs are perfect structured issue pages. Um,
Josh Klemons: Got
Jack Welty: you want to make sure that if uh somebody asks Chat GPT about your position on environmental issues that
Josh Klemons: it.
Jack Welty: there’s an easy answer in your words on your website that that they can pull from. Um I think the other big finding that was um not counterintuitive but um you know cuts against a little bit of you know how we run modern campaigns um was the importance of official sites.

Jack Welty: Um the if you’re you know advising or working on an incumbent campaign the model see that.gov domain and instantly associate that with an authoritative source. So, um, for folks again that are running as incumbents, make sure that you’re not neglecting your official site totally in favor of your campaign site and that it’s not just, you know, a graveyard of press releases, but that it has a good updated bio and some good accomplish like some structured data. List your, you know, top 10 accomplishments in Office and why they matter.
Josh Klemons: Sure.
Jack Welty: um uh you know to the extent all of that is sort of allowable on your official site but um uh they really like those pages and we see those pages get cited for in comments a lot more than campaign sites.
Josh Klemons: Gov gets excited more than okay.
Jack Welty: Yeah.
Josh Klemons: That like that alone is like a massive reminder because I live in this world. I think about this stuff all the time and I don’t think about that at all. Like because I’m not legally allowed to touch the.gov stuff,

Jack Welty: Yeah.
Josh Klemons: but that doesn’t mean I can’t recommend to the candidate that like, hey, you should probably have your staff in the building rework your site to be like up todate. Like I noticed you haven’t updated your site in four years. Maybe you should like Wikipedia you can’t necessarily edit. Like you know that’s complicated.
Jack Welty: Yeah.
Josh Klemons: Ballad Pedia you can and a lot of folks don’t but like you can but then.gov like that if you’re an incumbent you absolutely can and I think a lot of people don’t. So that’s that’s a huge like that alone is like a giant thing that I think most campaigns are not doing and should be. It looked like you had something to add on
Jack Welty: Yeah. Well, I was just going to big plus one to Valipedia.
Josh Klemons: Balladedia.
Jack Welty: So folks don’t know there’s those candidate surveys on ballot pedia that is your owned property. If you are the candidate you can tell about you can prove to balloted you are who you say you are and you get to fill out that candidate survey with your information.

Jack Welty: A lot of times you’ll see only like lesser known candidates third party candidates fill those out. I think it’s really important that our candidates fill that out. We are seeing ballot pedia as like the number two or three most cited um source across a lot of these answers. I think some of the labs even have relationships with balopedia that they can license that data.
Josh Klemons: What?
Jack Welty: Um so to the extent uh you know your your comm’s research and policy team sign off like please fill out those candidate surveys. I think that’s a great way to put in the models sources some sort of like you know uh objective you know truthful content out
Josh Klemons: I work a ton of down ballot races and I care about ballotedia.
Jack Welty: there.
Josh Klemons: I use it as a user, but like I also know that the average voter doesn’t care about ballot pedia and I had not thought like prior to like reading your paper. It had not occurred to me how important it was because it’s not really relevant for SEO.

Josh Klemons: I mean, it will come up like if you search, but like I feel like most people aren’t going to even click on it if that’s like if it’s Wikipedia and valipia, they’re going to Wikipedia. And if it’s Ballopedia and a website, they’re going to go to the website. But if Gemini or Chat GPT are pulling from there, it doesn’t matter if they click or not, they should be going there. So yeah, that’s another like big reminder for folks like Balipedia matters a lot more today than it did three years ago. And I mean I use Valipedia all the time, but like it matters more for like reaching your voters than it did a couple years ago because they’re finding it via cloud or whatever. That’s wild. Um, okay. You address the environmental argument uh about AI in your paper. uh the idea that progressives quote shouldn’t be using AI because of its energy footprint. What’s your response to
Jack Welty: Yeah. I mean, there’s a ton of research on this. I don’t pretend to be an expert.

Josh Klemons: that?
Jack Welty: um uh that says you know your average consumer user uh of any of these tools is a really really really marginal drop in the bucket of the footprint of these tools. Um so I I am not uh you know worried about my own personal use. Um and then I think the other argument that we would posit is um you know if we want to force the labs to adhere to good environmental policies and make sure that we are responsibly building out this infrastructure then we’re going to need to win elections and be in positions of power to to do so. Um so uh I I I don’t want our side to stray away from these tools because of these ethical concern ethical or moral concerns and then lose the opportunity to sort of enact the policy change that we think we need uh in order to produce good
Josh Klemons: So I mean that leads right next to my next question.
Jack Welty: outcomes.
Josh Klemons: Like there’s a real tension in the paper between like Democrats need to adopt these tools now and these tools are owned by companies with their own political interests which we know do not you could say don’t always or even don’t often align with ours.

Josh Klemons: So how do you square that circle? And you sort of just gave the answer is like we need to be in power so we can regulate them. Is that the answer? Like use them because it helps us win elections and then we turn around and regulate the hell out of them or do you have like a bigger approach to how you’re thinking about
Jack Welty: Um, I think that’s the fundamental approach. I mean,
Josh Klemons: this?
Jack Welty: I I think this technology is going to produce like wide ranging potentially earthshattering impacts on our society that I would like our side to be in the driver’s seat to help figure out the appropriate policy responses. um on the big tech concerns more specifically like you know we don’t not run Facebook ads we don’t not run Google ads like a lot of these are the same organizations that we’re talking about that um you know I think we have to make the uh we have to make the political sort of calculus decision that um you know I don’t think any of us like paying Mark Zuckerberg money um but if we need to reach the voters we need to reach in order to win the election like we’ll happily sign up uh to pay meta ads to do

Josh Klemons: I run a ton of meta ads for a ton of clients and I train on meta ads and I always say I think meta is an evil company but it wins elections and like until that changes is like I I see no alternative but to like spend it I spend so much money on meta because it keeps winning elections and yeah it’s it’s it’s hard to live in
Jack Welty: Sorry.
Josh Klemons: the modern world you know it’s hard to like like this small group of like tech companies that are like ruling us all but
Jack Welty: I
Josh Klemons: like we have to live in the like I’m using my Mac computer despite the fact that I’m very uncomfortable with Apple’s politics these days like I don’t know how to get around that. So I think that’s a fair answer for the the AI tools as well.
Jack Welty: A lot of us are advocates of campaign finance reform and we still support super PACs.
Josh Klemons: Right.
Jack Welty: Like they’re just we have to win elections to be able to change these things.
Josh Klemons: Right.
Jack Welty: And I think this is another one of those instances.
Josh Klemons: Uh if you could get every Democratic campaign manager in the country to do one thing differently tomorrow, what is it?
Jack Welty: I would get them to write uh an AI use policy for their organization and create spaces where their staffers can share with one another how they’re using AI. Um because their staffers are using AI. They’re using JGBT on their phones. They’re using it on their personal computers to get work done. Uh and I think that sort of like shadow AI is unsafe and leads to worse work. Um, you’re going to learn a lot more about how to use these tools effectively if you and the person sitting next to you can collaborate
Josh Klemons: Interesting.
Jack Welty: safely on approved uses than if everyone is kind of using AI in a background tab and trying to hide it from their information technology department. So, um,
Josh Klemons: Interesting.
Jack Welty: that’s what I would get every every manager to
Josh Klemons: you should put out a blog post on what those look like.

Jack Welty: do.
Josh Klemons: I bet that would be very helpful. Uh, so I I’ve heard a lot of blog posts you should write based on this conversation. Maybe those are in the work. That’s how I think about the world is blog post. So, are there any campaigns you’re seeing do anything innovative with AI right now or is it like too soon to look at it that way?
Jack Welty: It’s kind of too soon and I have to imagine that the ones that are doing good things um probably aren’t going to tell us about it until after the midterms.
Josh Klemons: Got it. Because we’re not talking about generative here.
Jack Welty: Um uh yeah,
Josh Klemons: talking about like processes and
Jack Welty: that’s I mean again I think a lot of the really big gains are behind the scenes are back office.
Josh Klemons: whatnot.
Jack Welty: I don’t think we’re going to win elections by um yeah, just like AI generating more campaign ads.
Josh Klemons: AI swaps, right?
Jack Welty: Like I think we’re gonna we’re just going to run a faster,

Josh Klemons: Exactly.
Jack Welty: better, more efficient campaigns uh with the resources that we have. And so, you know, hopefully we find out uh uh after November how it’s going for
Josh Klemons: Uh what’s caucus AI?
Jack Welty: folks.
Josh Klemons: What’s one Caucus AI finding that you wish was getting more attention than it
Jack Welty: I think the big one for me um is that we’re able to to prove that getting your
Josh Klemons: is?
Jack Welty: website cited matters. Um so we can look at the similarity between chat’s responses and your website and when chat looks at your website um the answers look uh more like the content you want to be putting out into the

Josh Klemons: Cool.
Jack Welty: world. Um, so everything you can do to get your your site uh in those responses is is better for your campaign.
Josh Klemons: Uh, how can folks connect with you and keep up with your work and all the many blog posts that are coming
Jack Welty: Yes, you can check us out on
Josh Klemons: soon?
Jack Welty: caucus-ai.com. Um, that has the full website. Uh, all you know, 1100 races we’re tracking. We update that monthly. We just added issues and criticisms questions. Um, and we’re going to publish, uh, you know, a blog post of a bunch of our initial findings on those issues, questions very soon. So, you can link to our our Substack and subscribe um, from the Caucus
Josh Klemons: and I’ll have that in the show notes.
Jack Welty: site.
Josh Klemons: And are you on social or like you’re on LinkedIn or just find you at Caucus AI?
Jack Welty: Yeah, find us at Caucus AI. Um, I’m on Twitter, I’m on LinkedIn. Um, feel free to drop me a note.

Jack Welty: Uh, we are really in in early days of this research. We are looking for folks to partner on experiments. um if you’re going to change something about your web presence or your media strategy, um let us know. We’d love to measure uh your AI SEO before and after uh and see if we can detect an effect. Um and then yeah, otherwise if you want to chat about AI on campaigns or sort of where this world is going. Um I’m also Jack camelback.te,
Josh Klemons: Okay.
Jack Welty: drop me a line.
Josh Klemons: And yeah, all those links will be in the show notes. Thank you so much. This was, like I said, I I think about this stuff all the time and I have a lot of new ideas here and like a lot of new ways to think about stuff. So yeah, I’m sure it was helpful for me, so I’m sure it was helpful for others. So I really appreciate your time.
Jack Welty: Awesome. Thank you..