cancel
Showing results for 
Search instead for 
Did you mean: 

4 Questions on the Fivetran Prefect Collection with Emerson Franks

nick-acosta
Lead Developer Advocate
Lead Developer Advocate

The Fivetran Prefect Collection provides prebuilt tasks, flows, and examples to run and control Fivetran connectors in Prefect 2.x. Emerson Franks, the Data Engineering Lead at RecRoom, was the first to try out the collection, so we asked him a few questions on how and why he did.

1. Hi Emerson, thanks for answering some questions today. Could you tell us about RecRoom, data engineering there, and your role within the data engineering team?

RecRoom is a radically cross platform social gaming experience that enables players to build games, communities and environments that bring us closer together in the online world. Given our commitment to enhancing the social aspects of our platform, data and data engineering are hugely important to building and growing our platform. My role is the Engineering Lead of Data & ML Infrastructure. This means that my team owns the entire data architecture that enables our Analytics Engineers, Analysts and Data Scientists to provide insights and metrics about our gaming platform.

2. Could you tell us a little bit about your experience with data orchestration tools?

My experience with orchestration was a bit tented prior to my time here at RecRoom due to my previous roles with Microsoft however, while I was there, we leveraged tools in the Microsoft stack like Azure Data Factory and Synapse Analytics. After joining RecRoom, I took some time to get an understanding of what we had and what else the industry had to offer. This ultimately led to a POC project that I built which compared Prefect, Astronomer/Airflow and Dagster. While all three platforms had very similar aspects, we ultimately went with Prefect due to how it allowed us a little tighter control over our infrastructure and its “out of the box” handling of streaming data scenarios.

3. What Fivetran connectors are you using and why are you bringing them into Prefect?

The primary use case for RecRoom is the ability to control our syncs from our orchestration layer so that we can then trigger other dependencies that we have on that data like jobs in dbt and Databricks. With Fivetran, we are able to sync data from our core Azure SQL Databases and also grab data from other sources like Azure Blob, S3 and even via API calls using webhooks. By having all of this coordinated in one place through Prefect, we can make much more advanced orchestration designs that are coordinated in ways that are not easily possible through the products/services themselves.

With the Fivetran Prefect Collection, even in an early release, I was able to have a flow up triggering our syncs in about 15 minutes. I should note that I had a bit of experience with the Fivetran integration when I did my POC however, that was using Prefect 1.0 and the new collection simplified my experience by only requiring a single task instead of having to coordinate the sync start, polling and completion on my own.

I have now expanded my use of this collection into a provider that will allow me to run N syncs, all from a config file, thus negating the need to make any code updates to add/remove syncs.

4. Have you seen any early results from the combination of Fivetran and Prefect? What are your plans for the two going forward?

Having Prefect and Fivetran working together has made my development process MUCH faster and has led to a great experience in the early goings of this project. I can’t emphasize enough how nice it is to not have to write my own provider for my Fivetran interactions. It’s even better knowing that the collection that I’m using is the “official” Fivetran implementation so I don’t have to spend extraneous amounts of time digging through the code to ensure that there aren’t things in there that shouldn’t be.

I also would like to call out that “out of the box” the Fivetran collection handles my desire to fire many syncs all at once. My current flow will send about 15 syncs out at once that all hit the Fivetran Collection’s flow within milliseconds of each other. What I’ve seen is that there are zero multithreading issues with these calls and whatever is happening behind the scenes (grateful that I don’t have to be concerned about the inner workings!) seems to work well and the syncs all complete as expected.

Thanks again to Emerson and RecRoom! The Fivetran Prefect Collection can be added to your Prefect environment by running pip install prefect-fivetran.

0 REPLIES 0