3 factors to consider when deciding which Fivetran connectors to orchestrate

[nick-acosta]

Whether its Airflow, Dagster, or Prefect… nearly 200 organizations have brought their Fivetran connectors into a data orchestration tool. Many do not run every connector they’ve set in Fivetran with these tools, however, and I’ve identified three things that you might want to consider when deciding which Fivetran connectors to orchestrate. 

Flexibility 

Fivetran connectors that are run with data orchestration tools are triggered to start with code, which can greatly increase their flexibility. When deciding whether or not to orchestrate a Fivetran connector, consider how dynamic that connector is or will be. If there are aspects of the connector’s metadata or schedule that change from sync to sync, or if you’d like a connector to run outside of the scheduling options that Fivetran provides, orchestrating that connector out of Airflow, Dagster or Prefect might be a great option. 

Control

As the number of services that make up the modern data stack continue to grow, dependencies between these services may become an even greater point of concern for organizations, and data orchestration tools provide a single platform to compose all of these services together into cohesive and interoperating data pipelines. Fivetran’s integrated scheduling works well for exactly this reason if you would like to bring together Fivetran data syncs and data transformations with dbt core. If you have many stored procedures in Snowflake or processes in Databricks that are dependent on new data arriving from Fivetran, or would like to tie the ELT of Fivetran with reverse ELT like Census and control all of these components as a pipeline, try bringing these connectors into data orchestration tools.

Efficiency

The gains in flexibility and control that data orchestration tools provide can create a more efficient modern data stack. This is done most effectively by manipulating a connector or set of connectors to maximize a data warehouse’s idle time. For example, Billie is a Berlin-based fintech startup that uses Fivetran and Airflow to save around 20% on their Snowflake costs by triggering Fivetran connectors to sync less frequently when the need for new data is less urgent. If you have a Fivetran connector that consumes a lot of data warehousing resources, running that connector out of a data orchestration tool can go a long way in improving the total cost in moving data to its destination. For this reason, we are seeing a lot of organizations run database and Salesforce connectors in Airflow, Dagster, or Prefect.

More information and new developments on Fivetran's integrations with data orchestration tools will be coming soon here on the Fivetran Community Developer Forum! Let us know if there are another other considerations you are making when identifying which Fivetran connectors and good candidates for orchestration.