If we want to move dashboards to Tableau, how should we handle transformation logic that currently resides in Remio, and how would we adjust the ingestion pipeline to load data into the raw zone in Databricks?

Remio’s transformation logic is typically expressed as SQL or Spark jobs. To migrate it to Databricks, I would first extract the SQL scripts and any associated metadata from Remio. In Databricks, I would create notebooks that replicate the logic using PySpark or Spark SQL, leveraging Delta Lake for ACID transactions. The ingestion pipeline would be updated to write raw data into a dedicated raw zone in the Databricks Lakehouse, using structured streaming or batch jobs as appropriate. I would then schedule the transformation notebooks to run after ingestion, writing cleaned data to a curated zone. Finally, I would expose the curated tables as managed Delta tables, which Tableau can connect to via JDBC or the native Databricks connector. This approach preserves the existing business logic while taking advantage of Databricks’ scalability and native integration with Tableau.