With Apache Spark Declarative Pipelines, engineers describe what their pipeline should do using SQL or Python, and Apache Spark handles the execution.

“Data engineering will become great with generative AI, but the Data Copilot will make the business data users much more productive,” Bains said. “Trying to wrestle with a particular SQL or ...