Benjamin Ard00:00 — Welcome back to another episode of Content Amplified. Today I'm joined by Tony. Tony, welcome to the show.
Tony Pataky00:05 — Thanks for having me, excited to be here.
Benjamin Ard00:06 — Yeah, Tony, I'm excited. The subject for this podcast has me thoroughly intrigued. I'm more than excited to dive in. But before we do that, let's get to know you, your background, work history, all that kind of fun stuff so the audience knows who you are.
Tony Pataky00:20 — Yeah, sure. So today, I am the director of SEO over at Procore. I had the SEO team as well as the marketing performance team and marketing performance is basically analytics and CRO as well. Before this, before Procore, I was at Levelset. So I came to Procore through an acquisition when Levelset was acquired. And really my career is comprised of doing SEO. My focus has always been SEO. And I've been doing SEO for companies that are enterprise, startups, everything in between. So yeah, that's my background in a nutshell.
Benjamin Ard00:53 — I love it. And based off of your background, I'm guessing most of the audience is thinking, OK, AI search, SEO, all that kind of stuff. We're going to throw the audience a little bit of a curveball. We're going to talk about why your content team's best quarter might actually not look like a good one in their dashboard. So I'm intrigued. This is amazing. When you sent me this idea, I latched on. This is so cool. Tony, you're saying content teams might be having their best quarter. And they have no idea what's actually going on with attribution right now that makes this really good performance look invisible and non-existent.
Tony Pataky01:28 — Yeah. The traditional search journey would be something that's very measurable, right? Someone goes to Google, comes to your website, and then that person fills out a form. You know exactly where to attribute that, to Google or Bing or whatever it might be. With AI search and LLMs coming into the sphere, basically there are a few things happening. Someone could go to ChatGPT. They put in a prompt, they see your brand being mentioned, and then a couple days later, they either come to your website directly, they go to Google, they type in your brand, and then click on something that's there, whether it's paid or organic, or they see something on social media and they're like, yeah, I remember that brand, and then they go through there. So that attribution gets lost, right? So that's one big thing. Now on top of that, if we look at the traffic that's coming straight from these LLMs like ChatGPT or Gemini, whatever it might be. If you look at your referral traffic in your analytics, that's where you're going to see that traffic coming in. That has been, I would say, quietly decreasing. If we look at Q4 of last year, ChatGPT made a move to decrease the number of links it has in its answers to your questions, right? It's kind of a play to keep people on ChatGPT, especially as they're introducing ads. That's my guess, but you know, it's probably pretty close to the truth. So it's a double whammy in this case, right? Like first is that pre-click journey, what I mentioned of people going to ChatGPT and then coming through other ways. That's almost a black box. Like that's almost impossible to really measure right now. But I mean, there are some techniques. And then the post-click signal is being miscategorized, right? The people who are coming through those areas are going into areas that you might not be looking at. If you're an SEO person, you're looking at the SEO, it could be attributed to those other areas.
Benjamin Ard03:21 — Okay, this is fascinating. The way that we have looked at the analytics, understanding the journey, all of that is completely different with large language models and really how the search behavior has completely changed. I think this is fascinating. So with all of those changes, like how much can we trust our existing dashboards? How much are we losing as far as like understanding what's going on? What does that look like?
Tony Pataky03:45 — Yeah, it's a great question because in the example I was just talking about, what you're paying attention to might be flat, even though you've been putting in all this effort, all this work, and because of how things have changed, the numbers that you've been stuck to is totally flat, right? A lot of people are not aware of this and are kind of going off into different areas in terms of measurement. One would be kind of like, I would say the most popular way of doing it, of tracking LLM performance is going to be something like a Profound or one of those other software tools that shows what is your visibility within ChatGPT, Gemini, whatever it might be. That visibility is based on prompts that you create or the agency you're working with, right? They create prompts and based on that, like how often do you show up within those prompts is basically the main way people are approaching this. This is a challenge that we have today. So yeah, I think that's the most popular.
Benjamin Ard04:47 — Interesting. Okay. So do those tools work? You know, you mentioned some tools for analytics tracking. Are there good solutions out there for actually understanding some of these sources? And in our emails, you mentioned some research by SparkToro. Like, should we be skeptical of these tools? Should we trust them? Like, what does this look like?
Tony Pataky05:09 — Yeah. And I love this topic because I think we're all going through this together, right? I would love to sit here and be like, I'm an expert, I know exactly how all this works, but what I can do is at least give some data and some direction to people to maybe better adjust and tune the way they're actually doing this. Because to be honest, your question of like, does this work? It's not perfect, but it's one of the best things we got right now. Now it also depends on how you approach it, right? For example, there are vendors, agencies out there who are promising number one ChatGPT rankings, for example. And let's go out to that one first, rankings within LLMs. Technically rankings don't exist in LLMs and people might be shocked to hear that, but the fact is there aren't rankings. It's not static. If you ask ChatGPT the same question twice, you're going to get a different answer each time. The only thing ChatGPT and Gemini won't do is tell you that it doesn't know, right? But it'll always give you an answer. And SparkToro, so yeah, Rand Fishkin put this research out there. They did it through SparkToro where they grabbed a bunch of people and they ran 3,000 prompt tests over ChatGPT, Claude, Gemini, Google AI. And I think it was like 600 volunteers, something like that, that they used. And they asked them to put in the same prompt over and over again to see what happens, right? And the finding was that less than one in 100 times did ChatGPT return the same list of brands twice. Less than one in 1,000 times did it show the exact same rankings, so to speak, or the list of the brands in the same order. So this is the dangerous thing, or something that people should just know, that ranking is maybe not the best approach when it comes to looking at that because like I mentioned before, if it's something I was doing the last few days, if you use an industry like cybersecurity, if you open up ChatGPT, five different windows, do not log in. Put in what is the best cybersecurity software in each one and look at the results. For me, every single time it's different companies, different orders. I'm yet to see it duplicate. So your industry might be different, right? Your mileage may vary. If there's only like you and two other competitors, maybe you guys all show up, but in different rankings and stuff like that. So that's the first thing is rankings don't really exist, right? Like that's it, it doesn't. So don't bank on that. The other part I did want to get to, because when I speak to companies about how are they tracking prompts, you know, the first thing I always ask, how do you come up with the prompts? And usually people are doing normally the same thing of like, they took their SEO keywords, the high converting ones, and then they turn them into prompts. And then how do they turn them into prompts? Give them to ChatGPT and be like, turn this into a prompt for me. And sometimes they'll get very, very specific, like, you know, the more specific it gets, your...
Benjamin Ard07:44 — Mm-hmm.
Tony Pataky07:53 — What is the chances that your customer is actually putting that in? It's probably pretty low. So what I like to tell people is, that's fine, you can do that. I mean, you know your business better than anyone, but I would caution people to go towards maybe the more general prompts. Keep it high level, because what you're looking for is if your brand is being named in those prompts that are higher level, for example, what is the best cybersecurity, right? Like something very broad like that. And you keep it broad and regional, maybe go into some persona and stuff like that. But the higher you are, if you're showing up 80 percent of the time for those broad things, you're most likely going to be showing up 80 percent of the time or at least a high number of times for a lot of the prompts within that theme, that category that you're looking for. The thing is like the more deeper you go, chances are so low that anyone's actually searching for that. You might be measuring something where no one will ever put that prompt in there. So those are the two things I would say most about AI rank tracking, quote unquote, visibility tracking, some things to just keep in mind.
Benjamin Ard08:52 — I love that. That's so cool. And I like the way you explained it of what's actually going on, putting yourself in the shoes of the actual searcher. How are they using this? How would you want to work on it? And even the fact of, well, even if you're looking for keywords and the search terms through AI, there's some bias there. Like, you know, we might need to step outside of that box. So what do you do to track everything? How do you look at it? How confident are you in your own system? Like what's your current philosophy to say, here's the best I can do today to kind of know what's moving and shaking across which platforms.
Tony Pataky09:26 — Yeah. And this is a tricky part of, like, you know, what is the best mixture of everything? And I'll say this, like with the SEO team, we've been running tests against LLMs to see what is going to move the needle, what tactics are actually moving the needle. And that gets difficult to really measure well, because you are kind of relying on visibility metrics, as we've mentioned before. When we talk about things like analytics, that's where things get really shaky because if I were to, and so I was at a conference a few weeks ago, met with a bunch of people, different companies. I asked them, my number one question was what percentage of your total traffic is coming from LLMs? 90 percent of people said less than 1%. And I wasn't surprised. With the smaller companies where like, you know, they have smaller traffic, they're going to see bigger numbers. Maybe it's going to be in the teens, but generally it wasn't high, right? So when you rely on analytics, if you were to do a lot of work, you might not be even moving that number because it's not representing the actual percentage. So we have visibility, right, as we're talking about percentage, you putting a prompt, things that you might be targeting, and then you have your referral traffic. But what's missing is that pre-click journey and getting at least close to the actual attribution. And that's where I would say the third layer comes in, which is post-conversion surveys. What this means is that when someone comes to your site and they fill out a form, they buy a product, whatever it might be at the end, you could ask, hey, was AI part of your journey to get here? And actually I heard this from someone else doing this and they found that 50 percent of their customers had AI as part of their journey. Looking at analytics, it's a much smaller, it's nothing, you know, it's nothing to wake up for. But when looking at that number, now that's huge. We saw something very similar as we started doing this as well, of putting in surveys, and people who are using the survey methodology, seeing something of like, okay, they have less than 1 percent of traffic coming from these LLMs, but when it comes to actually people filling out the forms and people who are actually buying, they're seeing 30, 40, 50, or 20, even if it's in the teens, that's a gigantic increase from like a half a percent to like 12. That's gigantic. Right. And that's not 12 percent of your SEO. That's of all channels. And that's the big one. Originally people I spoke to were saying that they were combining organic and direct to kind of measure LLMs, and partly that's right, like looking at the numbers and the numbers that I know, okay, that's the lion's share. But the fact is that paid is also a big one because when someone types in your company name with AI overviews, there's higher branded ad clicks there are just because people don't want to scroll down. It's like, it's right there. I'm not going to go past the AI overviews. So this proxy measurement approach that I'm talking about here, one is LLM visibility tracking, two is going to be your referral traffic, right, coming from these LLMs, and number three, the post-conversion surveys, I would say would be the best direction to take in order to really measure the impact of not just LLMs, but your work with LLMs, right? Because if you keep that survey running, you're going to see, hopefully you're going to see that number increase over time.
Benjamin Ard12:42 — And I love that you brought in the self-reporting attribution. For a while this has been my favorite form of attribution. It's such a cool way to just go out there and ask, how did you really find out about us? It may not be the first or the last or whatever, but it's clearly the most influential way that they were able to discover you. And I think that that's so important. Okay, well, we're almost out of time, Tony. So for every content leader, or someone in charge of their SEO or their website or conversion rate optimization, how can they actually make an improvement in the next little while? And how can they even show the ROI of those efforts and really start to shift things in the right direction to kind of understand where everything's coming from and maybe what the analytical mix actually looks like?
Tony Pataky13:32 — Yeah. What I would say, like first I would sit down with people and say, stop trying to prove AI attribution precisely. Like it's precision. You'll never, you're not going to get there yet. We don't have the tools yet. So I would reframe, I'll tell them to basically reframe the conversation with their CMOs, with the executives within the company of like, hey, this is the infrastructure, this is a measurement infrastructure that we built while others were waiting. And it tells us the best story that we can directionally of how things are working. So the framework I was mentioning earlier is I definitely believe in measuring AI visibility, your brand visibility within AI tools. Really pay attention to the prompts that you are using. Be smart about it. Don't go too deep. I would say stick a little bit higher, have your competitors on there. You're going to be comparing yourself against your competitors for the most part right now. Number two, definitely get your analytics set up. Make sure you have that attribution because if you do look at attribution across, or really just traffic across all the different LLMs, it does line up, at least speaking with a few other companies as well. If you look at how much traffic you're getting from the different LLMs, it does line up with the numbers that are out there. The research that's out there that is showing, okay, Gemini, ChatGPT, no question that's number one, that's leading the pack. And then, you know, perplexity, Gemini number two, three around there and everything else below that. This is for most, but it might be for your company, it might be different, you know? And then the third part, getting that survey in there, because I just told some stories of people who were seeing below 1 percent traffic attribution, yet the surveys were 50 percent or 40 or 30, whatever it might be. But it might be different for you. Like you got to figure out, and like you just said before, ask your customers, reach out to your customers, your customers might be different, it might be even more. So it really depends on your company and what you do. Now with that framework, now you have a story to tell and that survey really tells you how much of an impact LLMs do have for your business. And that's going to help for things like getting headcount, you know, getting things for the SEO team and all the other teams who are working on getting LLM visibility. Those three pillars right now from what I've seen and what I know are going to be your best bet of approaching the conversation within the company around attribution and really how things are going between your efforts and LLMs. And I would say just keep it going. Right now, the fact is when you build that out, you're building a baseline and you're ahead of most companies if you build that out. So you're going to have what's going on now and you're going to have that running throughout the entire year. All the companies who are going to be jumping on these kind of processes six months or 12 months from now, you're way ahead of, because you kind of know the trends and what's going on. And as you test things, you're going to see what's moving the needle, what's not. Hopefully it helps companies from falling into some of the traps that are on LinkedIn right now. A lot of people selling different solutions that I have yet to see any kind of data to actually back them up, so to speak. There's a lot of things being sold that are totally unproven, and hopefully these have the tracking and the work that you're already doing. It lets you know where to actually put in more resources in terms of moving the needle.
Benjamin Ard16:57 — Love it. That's awesome. Well, this was fascinating, I think, for everyone listening. It's a good opportunity to go through and say, OK, well, what does my data show today? How do we maybe add a couple of things into the mix to get a better picture? Here's how we can prove that there's a good opportunity for us to spend the time and money, resources to go and figure this out and give them a direction. Tony, this has been really cool. We're out of time. We want to let people get back to their daily grind and everything that they're doing. But for anyone who wants to reach out and connect with you online, Tony, how and where can they find you?
Tony Pataky17:28 — Yeah, so I'm on LinkedIn. This year I've been jumping into LinkedIn more than in the past. So if you look for Tony Pataky on LinkedIn, look for the person who works for Procore, then yeah, definitely reach out, connect with me. I love to talk shop. So if you have questions, anything on top of what I said, please feel free to reach out and yeah, we can connect there.
Benjamin Ard17:49 — Love it. For everyone listening, scroll down to the show notes. Regardless of what platform you're on, Tony's LinkedIn profile will be linked right there. Click on the link, say hi to Tony. Thank him for coming on the podcast. Tony, this has been incredible. Thank you, thank you, thank you. Have a wonderful rest of your day and thanks for the insights.
Tony Pataky18:07 — Thank you, Ben, really appreciate it.
