The Fivetran Prefect Collection provides prebuilt tasks, flows, and examples to run and control Fivetran connectors in Prefect 2.x. There are three Python files that make up the collections, let’s walk through them.
A credentials class is present in most Prefect collections, including Snowflake, dbt, and Databricks. It provides a way for users to easily store and reference their credentials across multiple Prefect flows. In the case of the Fivetran Prefect Collection, the FivetranCredentials class allows the other parts of the collection to access your Fivetran API key and Fivetran API secret in a secure way. More information about this topic can be found in Prefect’s docs.
This portion of the collection contains the code used to interact with the Fivetran API. Functions that Prefect tasks and flows will use to send get, patch, and post requests to Fivetran are contained here. Separating these functions from the rest of the collection will make it easier to build out new tasks and flows. The Prefect dbt Collection uses this architecture for dbt Cloud as well.
The tasks and flows of the Fivetran Prefect Collection and present here. They build on top of the functions in the clients.py to start, monitor, and manage Fivetran connectors in Prefect. The main thing to look into here is the Flow called trigger_fivetran_connector_sync_and_wait_for_completion. The quickest way to add Fivetran to your Prefect environment is to use this flow. With its default settings, the only thing that is needed is to pass a Fivetran connector_id while creating a flow, but it also has options to set or change the connector’s schedule type and the frequency with which Prefect will check to see if the connector has finished syncing. This flow combines 4 different tasks that are also defined in connectors.py: a task to verify the connector, prep the connector, start a connector sync, and wait for the sync to finish. If there is any additional functionality you would like the collection to have please let me know, it will probably end up in this flow. I will be taking a deeper dive into the Tasks and Flows contained in this file on our community at a later date.
Thank you to everyone that has tried out the Fivetran Prefect Collection so far, including Emerson Franks, a Data Engineering Lead at RecRoom (you can read our conversation about the Fivetran Prefect Collection here).
