Product manager at Ecosia on building AI-powered summaries with search

We spoke with an engineering manager at Ecosia who leads a team using Exa to power AI search summaries for approximately 500,000 daily queries. Our conversation covered their $300,000 monthly investment in Exa's technology, how they measure the 5-10% retention lift from AI search features, and why service quality and flexibility have been more important differentiators than search result quality in this rapidly commoditizing market.

Key points via Sacra AI:

To start, can you tell me a bit about your company? What you do, and what your role is?

I work at Zendesk, which is a customer service company. We're basically providing software for chat, email, phone-based customer service for clients from all industries. I'm an engineering manager in the AI agents team which is responsible for having Agenic AI support customer service.

Which of these products do you use? Exa, Parallel, or both? And roughly how long have you been using them?

In this role at Ecosia, I've been using Exa. We started working with Exa about 1 year ago in February of 2023. Parallel, we didn't use, but we evaluated them as part of the same process where we eventually ended up working with Exa.

Can you walk me through the problem you were trying to solve when you first adopted Exa? What was the need and what were you doing before?

Ecosia is a search engine, and we wanted to add AI functionality on top of the regular web search. We wanted to have AI summaries, and we were looking either for building in-house or having third-party options. We figured we could build faster with an external provider. We looked at different options, and Exa had the best option for us to build a summary of web search results that we could display on top of the regular search results as an AI overview or AI summary.

At a high level, how central is Exa to your workflow today? Is it core infrastructure you rely on daily or more of a nice-to-have that you use occasionally?

It's core infrastructure that we use daily. Around 30 to 40 percent of the web searches are triggering the AI overview. This happens around 500,000 times a day. It's very basic fundamental infrastructure for us.

Let's dig deeper into that. Can you walk me through a specific workflow where Exa is used? Step by step, what does it look like in practice?

What it looks like in practice is that the end user goes to our website ecosia.org and enters a search query that is more complex, like an open-ended question. Then we have intent recognition that says, "This is a complex one that Exa needs to solve." We then contact Exa web search in the backend. They provide us with a summarized answer from multiple sources and also give us the sources as links. We then give it back to the user in the frontend, and they can see the AI overview. Basically, we just have the connection at the backend that decides if we need to ask the Exa product, and then if we do, we have a proxy that connects directly to Exa and provides the results from them.

Who on your team actually interacts with Exa directly? Is it mainly engineers integrating the API, or do analysts or product folks use it as well? And roughly how many people are involved?

The AI team is around 6 people right now. Mostly, it's the engineers interacting with them on API integration. But also me, the engineering manager, and our product manager interact with them quite a lot because they actively involve us in product development and technical decisions. We have a very tight collaboration with them.

How has your usage evolved since you first adopted Exa? Are you using it for more things now, using it more frequently, or has usage stayed fairly flat?

Usage has stayed fairly flat because the usage behavior is mostly driven by the behavior of our end users. The composition of web searches that are conducted via Ecosia hasn't changed a lot, which means that usage of Exa has been pretty stable for us.

How does Exa integrate into your broader stack? Is it plugged into internal tools, AI agents, or pipelines, or is it more of a stand-alone service you call?

It's more of a stand-alone service. It's part of our product, but we're not using it for internal use cases or anything like that.

If Exa disappeared tomorrow, what would you do? How painful would that be to replace? And what would you fall back on?

It would not be super painful to replace because we did evaluate other providers, such as Tavily or Parallel, so we would have other options that we could go to fairly quickly. We would need to work on the different API integration to see if the quality would be the same because the evaluation was, as I said, roughly a year ago. We would have to see what the status is right now, but we have other options that we could fall back to fairly easily.

What does Exa do really well for you today? Specific things that made you choose it and that keep you using it.

It's three different things. First, from the quality standpoint, the quality of the results they provide is really good. From our research, it's up to par with Google or Microsoft Bing. We didn't see a significant difference in the quality of the results. The second thing is the pricing is super competitive. We got a very good price, and they also created a custom pricing scheme to support our product. The third thing is that we have a really good collaboration with their engineers. They saw us more as a partner in this project and not just as a client. We were influential in fixing technical bugs and actually driving product decisions and the product roadmap. We have a really good cooperative relationship with them.

Where have you seen Exa fall short or what have been the biggest frustrations or limitations?

We haven't seen Exa fall short in terms of the quality of the results. There were some smaller edge cases where maybe Exa was a little bit worse, but we could see they're continuously evolving the quality of their search. The main concern we had in the beginning was latency. For some requests, the processing time was really long, which was a bit of a headache for us. But we figured this out during the testing phase, and they were very helpful in explaining what we could do on our side to optimize. They also made implementation changes on their side to optimize the latency for our use case. That worked really well. There's nothing where Exa really failed or had a major flaw that would make us consider changing to another provider.

When you were evaluating solutions originally, what other products or approaches did you consider? What was the shortlist?

The shortlist basically was Tavily, which we talked to very intensively, Parallel AI, and Brave as a third option other than Exa.

What ultimately made you choose Exa over those alternatives?

It was the combination of these three things: the quality of the product and the fit for our use case, the technical support that we got from them, and the pricing scheme that was very competitive. All of these things combined.

You mentioned you evaluated Parallel as well. How did you see Parallel as different from Exa at the time?

From the quality perspective, based on our testing, Parallel was more or less on par with Exa. That wasn't a concern. The main issue we had was with the process. We found it difficult during the testing phase to get meaningful responses from them. Sometimes we had to remind them that we had scheduled a meeting to discuss, and then they had forgotten about it. It wasn't a good experience from our side. They also didn't want to adapt to our product needs. With Exa, it was a much more flexible conversation. Parallel was more like, "You can buy it as it is, and we will point you to the documentation. If you have any problems, they didn't really want to support us." The experience was very distant and dragging on, and that's why we decided against Parallel. From a technical standpoint, there weren't many points against it. It was just not a good experience on our end.

Have you looked at using general purpose search APIs like Google Custom Search, Bing API, Serper, or SerpAPI for this use case?

Yes, we did. There were several problems with that. For Google and Microsoft, we actually have contracts with them for regular web search. They were the first options we thought about, but we quickly discovered that our contractual obligations didn't allow us to build these products with them. For instance, Google has their own AI overviews and AI search functionality now. It would have been very difficult to build a product with them that wouldn't be a competitor and would adhere to all the contractual obligations. It was a risk we didn't want to take.

For plain web search APIs like SerpAPI, we considered this, but it would have meant additional effort on our side to build the AI summaries and infrastructure. We also considered doing this, but at the time we wanted a quick go-to-market and a fast release, at least for our first use case. With a team of 6 people, 4 of them AI engineers, we didn't think we could move fast enough to release it ourselves. So we decided to go with an external provider, and not with one of the big ones because that was the only reasonable choice at the time.

Have you considered building your own search and retrieval pipeline? Crawling or scraping, embedding, and indexing content yourselves? And if so, what made you decide against it?

We actually considered this a lot throughout the company's history. But in the end, as a hundred-person company, it wasn't feasible for us to build the infrastructure and all the staffing around it. That's why we decided to license from a third party.

This has changed a little bit recently. We are collaborating with a French search engine that is actually building their own index, and we're trying to cooperate with them. They are focusing mostly on the French market, and we're trying to provide input for the German market. So there's a bit of a paradigm shift right now—we're moving towards building our own crawling infrastructure. But this is very much in its infancy, so that product isn't ready for a bigger market share right now.

How do you think about the competitive moat for companies like Exa or Parallel? Is the defensibility in the quality of the index, the AI summarization layer, the integrations, or something else? And what would realistically make you switch providers?

At the end of the day, it's really about the quality of the search results. No AI summary or product building on top of search results can work if the index quality is bad. What we've observed is that many competitors right now have more or less the same level of quality, so there's not a differentiating factor or moat there.

The only differentiating factor right now is really the service quality and how fast we can integrate. Currently, Exa has the lead in this area, but if they became less responsive or if they changed their pricing model, that would force us to reconsider. It would be a plain business decision to maybe go to a different vendor. For now, Exa is still the most reasonable choice for us, but there are many competitors in the market doing a great job that we could easily switch to if we needed to.

Are there other companies in this space that you're watching or have heard about? Whether startups or larger players moving into AI native search and research?

Apart from the ones I mentioned before, another option we looked at is Perplexity because their value proposition is very similar to what we wanted to do. We tried to engage with them, but that collaboration never took off. Other than that, we haven't looked intensively at other players at this time because we're pretty happy with the solution we have.

Can you give me a sense of what you pay for Exa today? Roughly, what tier or monthly spend you're at?

Right now, for Exa, we spend around $300,000 US dollars per month.

Just to clarify, you mean around $300,000 per month or per year? And how is that structured in terms of pricing? Is it per query, volume based, a custom enterprise plan?

Yes, it's $300,000 per month, and it's query-based. We pay a fixed rate for a bunch of queries. I can't give you the exact number because that's a business secret, but we pay per thousand queries on a monthly basis.

Do you think about the ROI on that spend? What concrete value are you getting from Exa that makes $300k per month feel worth it?

The thing to understand is that our search engine is free to use for users. The only source of income we have is ads. So when a user uses AI search on our website, they don't pay anything for it, but we can display ads in certain contexts. This is actually enough to justify the extra cost for the AI search.

But the long-term play, which is most relevant for our company strategy, is to increase user retention. In recent months, we've seen users moving away from the plain web search we offered before because they felt it was falling behind competitors like Google, Microsoft, or Perplexity. By adding this functionality, we keep users from moving away and gain new users interested in trying our offering, increasing their lifetime value.

For the majority of queries—around 70 percent—we don't use AI search; we just display ads regularly and make money that way. The idea is to have more users in the long run who create more revenue. The AI search in isolation actually costs us money per user and doesn't directly help with revenue generation.

How do you measure whether Exa is improving retention for you? What metrics or experiments do you rely on internally to decide it's working?

We measure the 7-day retention and 30-day retention for new users. We A/B test this with users who see the AI search versus users who don't. We see a significant uptick in users coming back to our offering. Depending on the market, we have around 5 to 10 percent increase in users coming back after the first 7 days and continuing to use our product. For the 30-day retention, it's around 3 to 5 percent.

We're also currently analyzing user behavior to see if users actually rely more or less on AI search than before, but this analysis is still inconclusive as tests are still running. Our long-term strategy is determining if we should direct more or fewer users to AI search to optimize for user experience versus revenue generation.

When you run those A/B tests, what's the main success criterion that determines whether you show AI overviews more broadly? Retention lift, revenue per user, query satisfaction signals, or something else?

What we're optimizing for is revenue per user. That's what drives our business since we don't have any other income sources. But other factors are important as well. User retention is the easiest thing to measure because you can measure it after a short time. Revenue is more difficult to measure long-term because it depends very much on user behavior—what kind of queries they have and if those queries are commercially valuable.

We try to estimate revenue per user as well as we can, using a mixed model with different user personas. That's what we try to optimize for—finding the sweet spot where users stay on our platform but also have a significant amount of commercially viable searches we can monetize.

Do you feel Exa is priced fairly relative to the value it delivers? And in your view, is there room for them to charge more? Are you already close to what feels reasonable?

For the time being, it's reasonable. It increases our cost per user, which is natural since it's a built-on service. It's not so expensive that it's prohibitive or causes a big cap for us. I think there is room for them to charge more if they wanted to because we're very dependent on them right now. We know we have a special deal that we negotiated, but there's a lot of margin since pricing from competition is fairly comparative. If the pricing became too high, there would be a strong incentive for us to change to a different provider.

How does your usage map to Exa's pricing model in practice? Does paying per query feel aligned with how you create value, or does it introduce friction as you think about when to show AI overviews?

Paying per query definitely creates friction for us because it introduces cost with every user decision. If we had a flat fee where we could operate within a fixed budget, it would be easier for us because the decision for every single user would not be so costly. If they had a different pricing model that was tier-based or had certain other structures, that would give us more flexibility.

I don't think that makes sense for them from a business perspective, though. They have infrastructure costs in the background, and they need to cover those costs. We wouldn't want to switch to another model just to accommodate this issue.

How do you decide in real-time which queries get routed to Exa versus staying on regular web search? What signals or rules drive that gating?

We have some heuristics for analyzing the queries. For navigational instances—if people are searching for LinkedIn, Facebook, or just simple one-word queries—we don't use AI search. So it's not just based on the query itself but also on our historic logs. We analyze what websites users typically go to. If they're navigating very often or just want to buy something, we don't need to show an AI search overview.

We also look at whether similar queries have triggered research or created user behavior that involved multiple site visits. That would be an interesting candidate for AI search. So it's actually a mixture of both analyzing the query as it is, plus relying on historical data to triage where requests should go.

How often do those routing heuristics get updated or retrained? And who owns that process on your side?

This process is owned by my team, the machine learning engineering team. We evaluate this constantly, keeping a close eye on user behavior and logs, trying to figure out if there are any changes in patterns or emerging topics. Search is very interesting because when there's an event or something in the news, people start searching for it, and we see this in the logs. Then we decide whether we want to show an AI overview for this or not.

This doesn't happen every day, but we see these shifts and topics coming up, and we try to determine if we need to react, adjust the model, or make other changes. We monitor this constantly and try to stay on top of current events.

Looking ahead, what's on your wish list for Exa? If you could wave a magic wand and add capabilities that would materially improve your product, what would you want first?

I think the one thing we would want is for Exa to become more language-agnostic. Right now, we're focusing mostly on English and German, but we want to expand into other markets or roll out to markets with less popular languages like Spanish. We're in a proof-of-concept phase for this. It's something we're really interested in, and Exa would be a good partner for us in this effort. We have to see if that would be viable with their current product.

How do you see your usage of Exa evolving over the next year or two? Do you expect to expand significantly? Stay roughly flat, or potentially move off it as you build more in-house or with partners?

I think we'll continue using Exa as we gain more users and improve user retention. We have a good relationship with them for building this product. We might try to negotiate a better deal.

But the future is unpredictable. Pricing could change, or the competitive landscape could shift. We're constantly looking at the market, and while we have a good relationship with Exa, we're open to reevaluating the situation as needed.

Do you think the big foundation model providers—OpenAI, Google, Anthropic—will eventually offer search and research capabilities that make standalone providers like Exa less necessary for you?

These major providers seem to be avoiding direct competition with others because it would cannibalize their own market. But I think for providers like OpenAI, Anthropic, or Meta with their models, offering search as a service could be a viable business model. They have the infrastructure and models to make it happen.

I think if search becomes interesting and relevant to them, they would try to enter that area. But based on their current product direction, I'm not sure if they're truly interested in doing so. They definitely have the capability to do it.

If you were advising an investor looking at this space, what would you tell them? What's exciting about companies like Exa and what risks would you flag?

The exciting part is that Exa has managed to build search infrastructure at much smaller scale than the big players, and that's a good value proposition. That's super interesting, and my advice would be to watch and see if it makes sense to invest.

But I don't really see a big moat. Others can build the same or similar solutions. It's a very open race with many companies in that space doing similar things or capable of doing them. The situation could change fairly quickly. There are many interesting companies in this space, not just one clear winner to bet on.

More broadly, how do you see this market evolving over the next few years? AI native search, AI research agents, and the infrastructure layer for giving access to information.

I think all of this will become much more of a commodity. AI agents that perform web searches will be baked into every conceivable product. The web search infrastructure will also become a commodity, just as search in any knowledge-based job is now something we don't really think about—it's just part of the daily process.

If a company could position itself as a foundation, the same way Google positioned itself as a foundational web search engine, it would be in a very strong position. This will only increase in importance. We already see that a major part of web search is coming from agents, and I think this will only become more relevant.

In your product, are you seeing a meaningful share of traffic already coming from agents or automated tools? And if so, what does that look like compared to human users?

For our own website, we don't actively track this yet, though it's in the works. We have suspicions about some traffic, but it's not something we have actual insight on at this time.

When you say some traffic looks suspicious, what patterns are you seeing that make you think it might be agent-driven or automated?

What we see is that sometimes from the same domain or IP range, there are very frequent queries. These look a bit like DDoS attacks, but they're not as frequent or consistent to actually cause any problems. We just see elevated traffic from certain sources.

We also see queries which obviously don't look like they're made by humans—they're very well-formulated or very crafted. I think these are AI-generated or generated by agents. We can see examples of this with certain sources sending odd or very competent queries that don't look like typical human traffic. But we only have manual observations so far; we haven't fully set up analytics tools for this.

What would you want to do differently in the product once you can reliably detect agent traffic? Would you route it differently, price it differently, block it, or something else?

In the end, that's an open business discussion. Right now, it's not a big problem because it's only a small fraction of our traffic. The issue is that this traffic doesn't create revenue for us because the ads that are interesting to us aren't viewed or clicked by automated agents. It creates infrastructure costs for us but doesn't generate any revenue.

It could be interesting for us to block this traffic to reduce costs. But right now, it's not a significant problem, so we're just monitoring it. If it becomes too much of a problem, blocking would be the main thing to consider.

Is there anything about your experience using Exa, good or bad, that we haven't covered but that you think is important for understanding this market?

To summarize, Exa has a very good product with good value and return on investment for us. They have an excellent engineering team and customer relations.

One thing I maybe wanted to mention is that they've expanded their product offering. They have other things we might want to look at in the future, such as website focus searches for different domains. There's a lot going on with Exa—it's a really interesting, focused company that's definitely worth watching and keeping an eye on.

Before we wrap, I want to dig one level deeper on something you mentioned earlier. You said that, in your view, a lot of providers are converging to similar search quality and that service, pricing, and integration experience are the real differentiators today. If that's true, what would create a durable moat for a company like Exa in your eyes over the next 3-5 years?

It's really difficult to see what could be a durable moat. The web is open. Having good crawling infrastructure and a good set of algorithms for producing search results is just table stakes—it's not really a moat. Google does it, Microsoft does it.

For the AI part—creating summaries and AI products on top of search—it's also difficult to create a moat because there are open models and providers that everyone can use in the market.

I think what would really make a company thrive in this space is becoming better at understanding clients and becoming close collaborators, building solutions that really fit the needs and the products clients want to build. Not just moving to very generic products. I think that's the real differentiating factor.

That's also what I observed when talking to other providers like Parallel—they came across as people who just wanted to sell a product, whereas Exa was more collaborative. I don't know if they can scale this approach to provide customized support for all clients, but I think that's the most differentiating factor. If they keep this up and adapt to clients' needs, that's their best path forward.

How embedded is Exa in your roadmap discussions today? For example, do you ever shape your own product plans around what you know Exa is likely to ship, or is the dependency more one-way where they adapt to you but you don't really plan around them?

We don't really plan around them. We know they want to build certain things, and we consider if those would be interesting for us or fit our product roadmap. But we don't change our priorities based on their input.

We have a very good collaborative relationship with them, but we also don't want to have too much lock-in. We don't want to build power features around Exa that would be difficult to do with another provider if we needed to switch. We're definitely interested in learning what they're doing and if their offerings fit our needs, but we don't actively plan around them.

Given that desire to avoid lock-in, how consciously do you design your architecture to keep the option open to switch providers? What abstractions or safeguards do you put in place to make a change feasible if you ever needed to move off Exa?

We kept our architecture flexible. We're abstracting away a lot of Exa-specific details, which would theoretically allow us to quickly change to another provider. Of course, it would take some work, but we're not deeply integrated in a way that would require several weeks of effort to move away. That's exactly what we designed our infrastructure for from the beginning—to have the flexibility to move to other providers fairly quickly. We know the market is moving very fast, and we don't want to rely on just one provider.

What part of the integration would still be the hardest to swap out if you moved from Exa to another provider? Result quality tuning, latency/performance work, intent gating model, or something else?

I think the most complex part would be latency. We did a lot of tweaks to optimize for latency and ensure everything works well. But that really would depend on the other provider, as we would need to do extensive testing with them. Latency optimization is the part we focused on most, including how we structured our infrastructure. That's the area we would need to reevaluate most carefully if we switched providers.

This transcript is for information purposes only and does not constitute advice of any type or trade recommendation and should not form the basis of any investment decision. Sacra accepts no liability for the transcript or for any errors, omissions or inaccuracies in respect of it. The views of the experts expressed in the transcript are those of the experts and they are not endorsed by, nor do they represent the opinion of Sacra. Sacra reserves all copyright, intellectual property rights in the transcript. Any modification, copying, displaying, distributing, transmitting, publishing, licensing, creating derivative works from, or selling any transcript is strictly prohibited.