Schema change

[emanueol]

I suppose Fivetran stores metadata of souce system, but does it also store metadata of destination table? And does Fivetran regularly pulls destination schema to maintain its own metadata repo and warning of schema changes ?

Or does Fivetran simply uses pushdown method ? Meaning it only stores source table metadata - enough to pull, prepare and store in temporary storage in Fivetran cloud buckets - meaning it blindy executes COPY on destination to load files, assuming table have all required columns ?

I understand Fivetran CREATEs table with good metadata in destination on Initial sync, also in subsequent Incremental syncs it detect new or deleted fields in source, but im curious to whatever Fivetran only pushs down based table on destination was created and not modified by something else ?

[kai]

Hi Emanueol - Thank you for the question!

We store the metadata of the destination table as we provide these information as part of Metadata API which helps you understand the mapping between the source and destination names of schemas, tables, and columns - please refer to the information here: https://fivetran.com/docs/rest-api/metadata.

We capture schema changes through CDC or when the data source features a changelog. These DDL changes identified in changelogs would be propagated to the destination eventually. Take Oracle for example, we will track the DDL changes/events through LogMiner - https://fivetran.com/docs/databases/oracle#logminer

We provide a more detailed write up here to explain how we manage schema drift for databases - https://www.fivetran.com/blog/schema-changes

[emanueol]

Yes, I'm aware of Fivetran pushdown source schema changes to destination, my doubt is the opposite tho.. how often and/or when does fivetran pulls metadata of destination? 

[kai · Accepted solution]

Hi Emanueol - Taking Oracle for example again, the Log Miner would be able to track DDL statements, the order in which they were executed, and who executed them. Therefore we are able to capture these changes and replicate them from source into the destination accordingly. Hence, no we do not pull the metadata of destination to handle the schema drift. Hope that clarifies! 

[Toby-Metcalf]

Thank you for the question @emanueol and for marking @kai 's answer as correct: this makes the knowledge easy for all Community members to find.
Well done @kai 
Best,

Toby