The Apache Spark 4.0 beta release has made significant progress in the past year, with added support for Apache Iceberg and GPU acceleration.
Support for Apache Kyuubi has been added to the Charmed Spark solution, allowing users to deploy an enterprise-grade, fault-tolerant data warehouse on Kubernetes data lake infrastructure.
The Spark 4.0 beta image includes new features such as the Spark Connect API, ANSI SQL, Python-based data source API, and Python-based UDTFs.