Prophecy Launches Data Transformation Copilot For Databricks
Generative AI accelerates data preparation for analytics and AI.
Note: TDWI's editors carefully choose vendor-issued press releases about new or upgraded products and services. We have edited and/or condensed this release to highlight key features but make no claims as to the accuracy of the vendor's statements.
Prophecy, the data copilot company, has released Prophecy Data Transformation Copilot for Databricks that accelerates the preparation of raw data for analytics and AI applications. By leveraging generative AI, Prophecy Data Transformation Copilot expedites the development, deployment, and monitoring of enterprise-grade data pipelines native to the Databricks Data Intelligence Platform, ensuring the delivery of clean, trusted, and timely data for analytics.
Cloud data platforms have become increasingly powerful to support a wide array of use cases, including streaming data pipelines, batch pipelines, and ad hoc SQL analyses. These platforms effectively handle diverse data types, from database tables to APIs to documents, and apply transformations such as cleaning, combining, aggregating, and structuring to prepare the data for utilization in AI and business intelligence.
Enabling a large number of data users to be productive has proven challenging for organizations, because traditional visual ETL products fail to seamlessly integrate with code-based cloud data platforms. Consequently, many enterprises resort to developing code directly for these cloud data platforms which, although powerful, remains inaccessible to most data users. Copilots address this issue by democratizing access to cloud data platforms, thereby increasing productivity across the board.
"The industry's current approach to data transformation is broken. Simplistic tools cater to many users but lack power. Cloud data platforms are powerful but remain accessible only to expert data engineers. Both of these approaches fail to adequately address the problem,” said Raj Bains, CEO and founder of Prophecy. “It's a false dichotomy because copilots make the full capabilities of cloud data platforms accessible to all users; however, to truly evolve, they must move beyond conversational-only interfaces."
Prophecy Data Transformation Copilot uniquely integrates visual interface, generative AI, and compiler technology to provide the most efficient approach to data transformation. Prophecy enables organizations to accelerate every stage of the data pipeline life cycle in the following ways:
- Develop: By suggesting next transforms, completing expressions, and generating equivalent data pipelines based on user input
- Test and deploy: By automating tasks, from suggesting tests to reduce errors to generating documentation and commit messages to streamline deployment
- Observe: By monitoring deployed pipelines and suggesting fixes for errors, the platform enables any data user to fix and redeploy pipelines for improved uptime
Prophecy integrates deeply with Databricks, supporting the full life cycle of Apache Spark- or SQL-based data pipelines. Pipelines developed with Prophecy’s AI-powered visual interface turn into Spark code or into SQL code with dbt core, making the full power of code available to all data users. Integrations with Databricks Unity Catalog and Delta Lake ensure a unified and well governed experience. Additionally, Databricks workflows can be developed visually to orchestrate and schedule multiple pipelines. Finally, Prophecy supports preparing unstructured data for AI, integrating seamlessly with features and models developed in Databricks.
To learn more, read this blog.