Monthly Python Data Engineering, January 2026
Monthly news from the Python Data Engineering world.
Hi and welcome to this new issue of the newsletter!
First issue of the new year. The amount of projects that are contributing to the rich Python data engineering ecosystem keeps growing, so over the course of the next few issues I’ll try to re-balance the content of the newsletter toward the more important ones, as a long list of changes goes against the purpose of this newsletter and significantly reduces how effective it can be as a source of valuable information.
Want to know more about me and why I curate this newsletter?
Check out my personal website at https://alessandro.molina.fyi/
Want to signal interesting libraries and frameworks for the newsletter?
Reply to the newsletter email at alessandromolina@substack.com
Key Highlight
This month, the Python data ecosystem saw major strides in performance and compatibility. Polars delivered significant speedups in streaming and SQL operations, while Delta-rs and Lance introduced powerful new features for distributed indexing and vector search, setting new standards for scalable data infrastructure. Apache Arrow’s latest release enhances cross-library data sharing, and Substrait Python’s improved type system makes it easier to build interoperable pipelines. Meanwhile, Spark’s new preview and maintenance releases continue to strengthen its SQL engine, and Velox’s blog posts shed light on advanced execution optimizations.
News
Apache Arrow version 23.0.0 was released this month. This release brings several major improvements, including support for new data types like
mapandlist, performance optimizations in IPC streaming, and better integration with Python libraries. The release also introduces a new experimental feature for zero-copy data sharing between processes, which can be a game-changer for high-performance data pipelines. Apache Arrow 23.0.0 Release NotesSubstrait Python reached version v0.26.0. This release drops support for Python 3.9, adds comprehensive type support, and improves validation. It also introduces a Narwhals-compliant dataframe setup, making it easier to integrate with modern Python data tools. Substrait Python v0.26.0 Release Notes
Narwhals released version v2.15.0. The update adds new features such as
{Expr,Series}.sinandlist.sort, along with several bug fixes for GPU operations and documentation improvements. Narwhals v2.15.0 Release NotesPolars released two new versions: 1.37.1 and 1.37.0. These releases focus on performance improvements, especially in the SQL interface with faster UNION and ORDER BY clauses. The new streaming CSV and NDJSON sink pipelines offer better memory usage and speed. There are also several new APIs, such as
pl.PartitionByand improved support for streaming IPC scans. Polars 1.37.1 Release Notes Polars 1.37.0 Release NotesPanel version 1.8.5 is out with fixes for notebook stability and Tabulator behavior. The release also improves handling of ESM/ReactiveComponent and Markdown rendering, along with documentation updates for FastAPI integration and app conversion guides. Panel 1.8.5 Release Notes
Dask released versions 2026.1.1 and 2026.1.0. These updates mainly focus on maintenance, including removing Python 2 support, dependency updates, and bug fixes for task graph processing. Dask 2026.1.1 Release Notes Dask 2026.1.0 Release Notes
Delta-rs released several new versions: python-v1.3.2, python-v1.3.1, rust-v0.30.0, and python-v1.3.0. These releases introduce performance improvements with the new delta_kernel, better read support for deletion vectors and column mapping, and several API changes. The Rust version v0.30.0 includes breaking changes, so users should review the migration guide. Delta-rs python-v1.3.2 Release Notes Delta-rs python-v1.3.1 Release Notes Delta-rs rust-v0.30.0 Release Notes Delta-rs python-v1.3.0 Release Notes
Lance released multiple versions, including v2.0.0-beta.10, v2.0.0-beta.9, v1.0.2, and others. These releases bring major improvements in vector search, distributed indexing, and performance. Key features include support for large minichunk sizes, GEO RTree index, and enhanced cleanup policies. There are also several breaking changes, so users should check the changelog before upgrading. Lance v2.0.0-beta.10 Release Notes Lance v2.0.0-beta.9 Release Notes Lance v1.0.2 Release Notes
Apache Spark released versions 3.5.8, 4.2.0 (preview), and 4.1.1. The 3.5.8 release is a maintenance update, while 4.2.0 introduces new features and optimizations for the Spark SQL engine. The 4.1.1 release includes bug fixes and minor improvements. Spark 3.5.8 Release Notes Spark 4.2.0 Preview Release Notes Spark 4.1.1 Release Notes
Velox released two new blog posts: “Task Barrier: Efficient Task Reuse and Streaming Checkpoints” and “Why Sort is row-based in Velox — A Quantitative Assessment”. These posts discuss new optimizations for task reuse and the rationale behind row-based sorting in Velox. Velox Task Barrier Blog Velox Row-based Sort Blog
Apache DataFusion released a new blog post titled “Extending SQL in DataFusion: from -> to TABLESAMPLE”, which covers recent improvements in SQL syntax and table sampling capabilities. DataFusion Extending SQL Blog