Escaping 300 Magic ETL flows: a Domo migration case study
A data analytics firm had years of reporting logic locked inside Domo Magic ETL. Here is how that estate got converted to SQL, dbt, and Airflow, with row-level parity signed off before cutover.
A mid-size data analytics firm came to us with a familiar problem. Over several years, its team had built out a Domo estate that had quietly become the backbone of the business: well over 300 Magic ETL dataflows, dozens of connectors, and scheduled jobs feeding dashboards that finance, sales, and operations checked every morning.
None of that logic lived anywhere a human could read it directly. Magic ETL stores its joins, filters, aggregations, and formula tiles inside Domo's own proprietary format. You can see the boxes and arrows in the visual builder, but you cannot export the underlying transformation as SQL, and you cannot check it into version control. The team could describe what their pipelines did. They could not point to the code that did it.
Step one: find out what is actually in there
We started with a two-week Lock-in Assessment, the same free first step we run for every prospective client. Our scanner walked the full Domo estate: every Magic ETL dataflow, every dataset, every connector, every scheduled job. The output was a report that classified each dataflow by migration complexity and mapped every external dependency.
One finding surprised nobody once they saw it in writing: close to a quarter of the estate was dead or duplicate. Dataflows built for a report that no longer existed. Copies made during an old rebrand and never cleaned up. That is typical of a mature estate left to grow organically for years, and it meant the real migration scope was smaller than the raw dataflow count suggested.
Step two: convert, with the original as the answer key
For the roughly 240 dataflows that were still doing real work, we ran our transpiler pipeline: deterministic parsing for the standard joins, filters, and aggregations that make up most of a Magic ETL estate, and AI-assisted conversion for the handful of dataflows with dense, bespoke formula tile logic. The output was SQL running on the client's own Redshift warehouse, orchestrated by Airflow and modeled in dbt.
The step that mattered most to the client's data leadership was not the conversion itself. It was proof that the conversion was correct. Every migrated dataflow ran side by side with its Domo original through our parity harness, which checks row-level and aggregate equivalence and produces a diff report. Nothing moved to production until that report came back clean.
The result
- 300+ Magic ETL dataflows inventoried; roughly 240 carried real business logic worth migrating
- About a quarter of the estate flagged as dead or duplicate and retired instead of migrated
- Full conversion to SQL, dbt models, and Airflow DAGs on the client's existing warehouse
- Row-level and aggregate parity signed off, dataflow by dataflow, before cutover
- Domo consumption pricing eliminated; the BI layer now reads from tables the client owns outright
The client's team now has the thing they didn't have before: pipelines they can read, test, and change without touching a vendor's visual builder. Whatever happens to any platform they depend on next, that part of the business no longer depends on a single company's roadmap.