Monthly Python Data Engineering, October/November 2025
Monthly news from the Python Data Engineering world.
Hi and welcome to this new issue of the newsletter!
Due to various reasons I had skipped the October and November editions of the newsletter, but those months saw significant updates and improvements to the core libraries for data engineering, so I decided to publish a summary of those months before publishing the December issue. I promise December issue will be published soon, but given the amount of news for October and November I'm sure you will appreciate having an overview of those too
Want to know more about me and why I curate this newsletter?
Check out my personal website at https://alessandro.molina.fyi/
Want to signal interesting libraries and frameworks for the newsletter?
Reply to the newsletter email at alessandromolina@substack.com
Key Highlight
Polars and beyond: Polars continues to push the frontier of DataFrame performance with new group-by optimizations, faster rolling windows, and stable decimal support, Ibis 11.0.0 brings a cleaner API and tighter integration with modern backends like DuckDB and Databricks, marking a maturation point for interoperable data processing in Python. On the execution side, DataFusion Comet’s 0.11.0 release made a strong case for native Spark acceleration, delivering measurable speedups while staying easy to integrate. Combined with Velox’s new shared build model and Narwhals’ growing backend support layer, the Python data ecosystem is rapidly converging toward high-performance, multi-engine compatibility.
News
• Apache Arrow 22.0.0 has been released with three iterations including RC versions. This columnar data format standard continues to be foundational for the Python data ecosystem, enabling efficient data interchange between libraries and systems. Arrow’s standardization of in-memory columnar representation remains critical for performance across data processing pipelines.
• Polars 1.35.2 (Python) and Rust 0.52.0 represent significant performance and stability improvements. The release cycle saw five versions with major optimizations in group-by operations, including native implementations for group-by aggregations and streaming engine support for ewm_mean(). Notable performance work includes optimized rolling windows that no longer recompute entire windows when NaNs leave the window, and improved row-count estimates. The streaming engine received particular attention with support for MergeSorted, rolling_rank(), and complex array expressions. Significant effort went into fixing group-by correctness issues including panic fixes and proper handling of null dtypes in ApplyExpr. Polars 1.35.0 stabilized the decimal type, which is important for financial and precise numerical computations. The Rust implementation (0.52.0) includes extensive performance optimizations for gathering operations, hash operations, and IPC stream reading.
• Narwhals 2.11.0 continues its mission as a dataframe abstraction layer with support for Polars, Pandas, DuckDB, PyArrow, and other backends. This release introduces an experimental plugin system, enabling third-party libraries to provide their own dataframe implementations. Key enhancements include Expr.replace_strict support for lazy backends and better error messages for plugins. The project now supports over 10 distinct dataframe implementations, making it increasingly valuable for library developers who want to support multiple backends transparently.
• Ibis 11.0.0 brought significant breaking changes for API consistency. The project removed deprecated methods like Struct.destructure, String.to_date, String.to_timestamp, and IntegerValue.to_interval in favor of consistent naming conventions (as_* instead of to_*). Support for Databricks streaming tables was added, alongside DuckDB 1.4+ support and performance improvements for Athena with metadata queries. These changes reflect Ibis’ maturation and focus on API stability for downstream users.
• Apache DataFusion Comet 0.11.0 - A significant release of the Spark accelerator delivered 2.2x speedup on TPC-H benchmarks. Major improvements include Parquet Modular Encryption support for secure data handling, substantially improved memory management with configurable memory pools and better disk management, and expanded Apache Spark 4.0.1 support with ANSI mode compatibility. Support for complex types in columnar shuffle and new RangePartitioning for native shuffle expanded the range of workloads that can be accelerated. New SQL functions include weekday, lpad, rpad, reverse, bit_get, and native count(distinct). Performance work included improved BroadcastExchange conversion and use of DataFusion’s native count_udaf. Parquet-independent API improvements for Iceberg integration were also delivered.
• Apache Spark 4.1.0 Preview Release is available for community testing. This preview release is not yet stable but represents the upcoming features and changes coming to Spark. The community is invited to test and provide feedback through mailing lists and JIRA.
• Delta-rs 1.2.1 and 1.2.0 brought major enhancements to the Delta Lake Rust implementation. Version 1.2.1 introduced in-flight streaming PartitionWriter for performance improvements in lazy writes. The 1.2.0 release included kernel log replay support, updated to DataFusion 50 and pyo3 24 for modern async support, and added per-column Parquet encoding configuration. Significant improvements to memory management with EagerSnapshot consolidation and checked arithmetic operations for overflow prevention. Domain metadata read support and deletion vector descriptor methods expand the feature set for advanced Delta Lake operations.
• Lance 0.39.0 and 0.40.0 (beta) represents the vector search index format layer. This release cycle saw 13 versions with continuous improvements to the vector database foundation, supporting efficient approximate nearest neighbor search operations critical for AI/ML workloads.
• LanceDB 0.25.3 (Python) and 0.22.3 (Node/Rust) - The higher-level database interface built on Lance continues to evolve with 14 recent versions. These releases provide SQL query interface and management layers over the Lance format, enabling production vector database capabilities.
• Substrait Python 0.25.0 upgraded protobuf dependencies to 5.29.5 and introduced advanced extensions with configurable registry for SQL conversion. The graceful URI-to-URN migration improves compatibility with evolving standards. Substrait’s standardized representation of database plans enables easier implementation of databases and query engines.
• HoloViews 1.22.0 introduced Narwhals support, enabling transparent support for Polars and DuckDB data backends alongside Pandas. This is a significant architectural improvement that allows users to work with their preferred dataframe library. New features include sizebar support for Points elements and enhanced synthetic legends for ImageStackPlot. Performance improvements include optimized HeatMap rendering and debounce support for Bokeh callbacks. Compatibility updates for Ibis 11.0.0, Python 3.14, xarray 2025.08, and cuDF 25.10 keep the library current.
• Panel 1.8.3 brought targeted UI improvements and better Python and Django compatibility. The release added official Python 3.14 support and introduced export() method for Vega pane. Manual rendering policy for JSComponent and improved CodeEditor prevent text selection issues. Bug fixes addressed FastAPI state retention, Tabulator popup rendering, and pipeline navigation buttons.
• Dash 3.3.0 (RC2) continues as the reactive web framework for building analytical applications. This release candidate version represents the upcoming stable release of Dash 3.3.
• Dask 2025.11.0 and 2025.10.0 continues to evolve as a parallel computing framework. Recent releases brought improvements including numpy 2.2 compatibility for percentile functions, enhanced input validation for dask.dataframe.read_sql_query(), and better handling of mixed HLG/Expr in simplification. The project remains important for distributed computing across Pandas DataFrames, sparse arrays, and bag-like collections.
• PyScript 2025.11.1, 2025.10.x brings Python execution to browsers without errors for unavailable packages in Pyodide. The latest release allows passing File and Blob instances to workers from the main thread, enabling better integration with browser APIs. PyScript continues to evolve as the framework for running Python in web applications.
• Cython 3.2.0 and 3.1.x - Seven recent releases represent ongoing development of the C-extension compiler for Python. Cython remains critical for performance-sensitive data engineering code, enabling C-level performance for compute-heavy operations.
• Great Tables 0.20.0 added grand_summary_rows() method for adding summary rows to tables and expanded Polars expression support in validation functions. This library remains valuable for creating publication-quality tables from Python data structures.
• Velox Shared Library Builds represents a significant infrastructure improvement for the Velox vector database engine. The project solved the challenge of migrating from 300+ static library targets to a monolithic shared library, reducing executable size by 96% (from 18.31GB to 0.76GB in release mode). Debug builds improved from 244GB to 8.8GB. This optimization significantly improves developer experience and CI performance, reducing link time by nearly two hours. The solution uses CMake wrapper functions for transparent monolithic library building while maintaining backward compatibility.