r/dataengineering • • 19h ago

Discussion Migrating from dbt Cloud to Core

44 Upvotes

Have any of you done this on a mature project, and was it worth it?

We originally picked dbt Cloud for our greenfield DWH project, attracted by the idea of not having to host the infrastructure and being able to focus on the data instead. With the initial high-growth phase nearing its end, I am now looking at optimising what has been built and could not make a good argument to carry on using Cloud.

The repo and CI/CD pipelines are hosted externally. Studio is barely touched as most developers have now switched to VS Code and Claude does all the AI coding we need. Even the semantic layer is managed in an external tool that did not integrate very well with what dbt offers natively.

This pretty much leaves job schedules and monitoring in the Cloud. This looks like terrible value for the money, even though we got a very good deal for the initial year (I imagine the renewal will hit us in the face).

I wonder if anybody ended up in a situation like this? Give me some good reasons not to move to Core.


r/dataengineering • • 11h ago

Blog I tested dbt v2 and how to run multiple engines in a single project

Thumbnail
greybeam.ai
19 Upvotes

r/dataengineering • • 10h ago

Career How should we handle updates in fact tables?

12 Upvotes

We have a table design that is fact_opportunityproducts, which is our Salesforce line fact table that has a row for each product per opportunity. But unlike event data our financial data these rows can be updated (opportunity can be closed, amount and quantity can change etc) such that we ofcourse cannot do normal incremental loading. What are design choices to be made? And we cannot only look at the salesforce last update date because we have created fields that also depend on other system inputs.
Do you then do rcompute the complete view but only merges the records that are different based on a hash? But that can also be very costly.