What are the technical challenges in creating connectors between apps and data warehouses?

Of course. If you think about these connectors, I think there's a few different categories of challenges that make this really difficult. 

The first one is performance, making sure that the connectors that you have built can scale really well into very high volumes and fast frequencies. 

The next one is obviously reliability. As the different systems are changing—whether it's the API schemas or the protocols to access them—these ETL providers have to think a lot and spend a lot of time making sure there won’t be any downtime.

The final one is edge cases. 

Every API connector and every data warehouse connector is going to have these weird nuances that you see in 1/10 or 1/1,000 connection attempts, and constantly playing whac-a-mole with those challenges is something that Fivetran does a lot. I'm sure all the ETL providers do, and I know at Prequel, we spend a lot of time dealing with this when it comes to data warehouse connectors. 

If I drill down a little bit on the API-specific approach—when Fivetran connects to an API exposed by a SaaS tool, they're heavily reliant on that API without a formal SLA between them and the vendor. 

That creates an entirely new level of complexity because that API is not built for Fivetran to extract data into a data warehouse. That API was built over a number of iterations and dev cycles for their customers to do whatever their customers need to do, and rarely is it built for replication at the scale Fivetran is demanding from it.

When Fivetran decides to go ahead and start building this connector, they're at the mercy of whatever that API looks like to try to figure out how to coerce that data out of the API into something that their customers want in the data warehouse. That's really brittle. It takes a lot of time, and there's a ton of challenges there. 

That’s one of the reasons that, at Prequel, we've decided to focus on providing SaaS vendors the tools to connect to their customers' data warehouses in a more native way. We can get to that 99th percentile of performance a lot quicker than if we're constantly battling APIs on new sources and playing that whac-a-mole game again.