Monthly Python Data Engineering, July 2025

Monthly news from the Python Data Engineering world.

Hi and welcome to this new issue of the newsletter!
This past month was a fairly quiet one, being summer I would bet many people were more willing to enjoy family time, a cold swimming pool or a sea beach instead of writing code and thus projects moved slower than usual. Still there were a lot of interesting updates! Some projects moved forward significantly.

Want to know more about me and why I curate this newsletter?
Check out my personal website at https://alessandro.molina.fyi/

Want to signal interesting libraries and frameworks for the newsletter?
Reply to the newsletter email at alessandromolina@substack.com

Key Highlight

This month was packed with interesting improvements in the dataframe world. Polars enhanced streaming capabilities: top-k and .sort(...maintain_order) benefitted from upgrades to the native streaming engine, improving performance on large datasets during sorting and head/limit operations. Lance and LanceDB continued with rapid development, adding hybrid search in Rust, improved embedding support, and smart ways to infer vector columns at table creation, simplifying high-dimensional ML workflows. Meanwhile, Narwhals introduced useful new APIs like is_close and list.get, and refined deterministic test outputs for polars, addressing edge cases more reliably.

News

• Ibis released versions 10.8.0 and 10.7.0 this month with some nice improvements for the SQL generation ecosystem. Version 10.8.0 added a Schema.from_sqlglot method to convert SQLGlot schemas to Ibis schemas, plus support for dtype(Union[T, None]) patterns. The 10.7.0 release introduced an Expr.to_sql() method that lets you convert individual expressions to SQL, which is handy for debugging and code generation. Both releases include various backend-specific improvements for ClickHouse, DataFusion, Polars, and others.

• Narwhals has been quite active with seven releases ranging from v2.1.2 down to v1.48.0. The major v2.0.0 release brought breaking changes around expression chaining behavior to align with Polars' new aliasing patterns. Key new features across these releases include DType.base_type for working with data types, Series.from_numpy for easier numpy integration, list.contains and list.unique operations, and better support for various backends including improved modin[pyarrow] dtype handling.

• Polars shipped six releases including Python Polars 1.32.3 and Rust Polars 0.50.0. The 1.32 series focused heavily on performance optimizations, with streaming engine improvements for various operations like top-k, extend_constant, and unique_counts. There were significant refactoring efforts around categorical/enum handling and the equivalence system was upgraded for faster planning times. The releases also brought new functionality like is_close method, arr.slice/head/tail methods, and native streaming support for various operations.

• HoloViz ecosystem had several updates. Datashader 0.18.2 fixed a critical segmentation fault bug in quadmesh reduction. Panel 1.7.5 brought security fixes for XSS vulnerabilities in authentication templates, plus improvements to modal behavior and Param integration. hvPlot 0.12.0 was the standout release, bringing a major documentation overhaul, new selector option for sample hover info on datashaded plots, toolbar customization with toolbar and autohide_toolbar options, flexible legends with legend_cols and legend_opts, and a new sampledata module for easy access to sample datasets.

• PyScript released 2025.8.1 with an important update to Pyodide 0.28.1 that changes null/None handling behavior - this may break existing code. The release includes better error messages for fetch operations, improved package caching based on Pyodide's lockFileContents, and new pyscript.ffi.is_none() function to help with the null vs None distinction. WebSockets support was also enhanced with async listener support.

• Apache DataFusion ecosystem saw significant developments. The main project released version 49.0.0 with major performance improvements including dynamic filters and TopK pushdown for LIMIT/ORDER BY queries, an upgraded equivalence system for faster planning, and support for async User-Defined Functions. New features include Parquet modular encryption support, WITHIN GROUP clause for ordered-set aggregates, compressed spill files, and REGEX_INSTR function. Additionally, a comprehensive blog post was published explaining how to accelerate Parquet queries using external indexes, metadata stores, and caches - showing that Parquet can achieve high performance without needing new file formats. The datafusion-table-providers project also upgraded to DataFusion 49 compatibility.

• cuDF released v25.08.00 with breaking changes including dropping CUDA 11 support and moving to C++20. The release brings significant performance improvements with better null-handling, streaming engine optimizations, and enhanced Parquet processing. New features include async User-Defined Functions support, compressed spill files, and improved string processing capabilities. The project also released a nightly v25.10.00 with additional streaming support across various APIs.

• LanceDB and Lance had extensive release activity. Lance released v0.33.0 with format 2.1 breaking changes, support for blob encoding, and various Java transaction operations. LanceDB released Python v0.24.3 and Node/Rust v0.21.3 with SigLIP embedding support, improved vector column inference, hybrid search examples, and better limit/offset support for paginating through search results.

• Delta-rs released python-v1.1.4 and python-v1.1.3 with new keep_versions parameter for vacuum operations, support for non-microsecond timestamp conversion, improved AWS path encoding, and better thread safety for Python operations.

• Dash released v3.2.0 with exposed NoUpdate type, devtool hook support, pattern-matching support in dcc.Loading, and improved slider performance by warning when more than 500 marks are used.

• Pandera released v0.26.0 with Python 3.13 support, improved polars dataframe handling for float columns with NaN values, better duplicate checking, and enhanced parser machinery with a new strict parser implementation.

• Cython released 3.1.3 as a maintenance release with bug fixes and stability improvements. The most important change was probably related to a Lock contention issues in cython.pythread_type_lock under heavy load.