TL;DR: We're seeing a significant percentage of our Monthly MAR made up of updates that shouldn't be happening. We're struggling to get engagement with product team for solutions. Thoughts?
*******************************************************
We've got several tables we're seeing that are being updated in totality each month.
When looking at the Fivetran documentation for the connector in question we note that the PK is solely compromised of an id column. Most other tables are a composite of id + update date + maybe another column (type for example).
When I take a look at the API documentation of the system in question I notice several things.
- GET (Read) and POST (Create) methods only. No PATCH or DELETE methods.
- "These records are created automatically and can not be modified"
The database we're using, Databricks, has built in history and I can go back in time from the current version of data to the previous day to compare any data differentials between those time periods. When I do that and dig in to logs, I see MERGE statements matching on id and updating all matched records. I can also see that no actual data has changed (other than _fivetran_synced).
At one point, we were seeing > 50% of Monthly MAR coming from those kind of tables. We've been able to deselect several of those tables from sync as they don't add business value but several that we do need are making up ~20% Monthly MAR despite the fact that we only see a relatively small number of new records coming in per month.
Maybe not surprisingly, we haven't been able to get a lot of traction through our account representative to connect with the product team responsible for the connector.
Anyone else run into a similar issue? Short of creating a custom connector, how have you dealt with it or how would you work around it? Anything I'm overlooking?
Thanks!
*******************************************************
Destination Database: Databricks
Connector Type: Lightspeed POS
