This Python Library Can Run Pandas Workloads Up to 20x Faster

Pandas is usually the first library I reach for when working with tabular data in Python. It is simple to use, works well with the rest of the Python data ecosystem, and can handle most everyday data tasks without much trouble. The problem starts when the dataset gets bigger. Once you are working with millions of rows, operations such as sorting, filtering, grouping, and joining can become noticeably slow. This is where FireDucks comes in. FireDucks provides a pandas-compatible API, so you can continue working with familiar pandas syntax while taking advantage of lazy execution, compiler optimizations, and multithreaded CPU processing behind the scenes. But I wanted to see how much of a difference that actually makes in practice. For this guide, I benchmarked FireDucks against pandas using the same dataset containing 10 million rows and tested seven common data-processing workloads. The results were quite impressive. The biggest difference came from sorting the full dataset, where FireDucks completed the operation 20.77x faster than pandas. Across all seven benchmarks, FireDucks achieved an average speedup of 7.28x. In this guide, we will install FireDucks, use it with familiar pandas-style code, and benchmark both libraries to see where the performance improvements actually come from. What Is FireDucks? FireDucks is a compiler-accelerated DataFrame library developed by NEC. It is designed to work similarly to pandas, which means you can often test your existing pandas code by simply changing the import statement. The main difference is how FireDucks executes your code. Image from Execution Model | FireDucks Pandas usually runs each operation as soon as it is called. FireDucks instead uses lazy execution. It first collects a series of DataFrame operations, creates an execution plan, optimizes it, and then runs the workload across multiple CPU cores. This approach can avoid unnecessary intermediate calculations and significantly improve the performance of common operations such as filtering, grouping, joining, and sorting. The API is highly compatible with pandas, but it is not a complete drop-in replacement. FireDucks DataFrames are different objects internally, and you may still run into compatibility differences with some pandas features or third-party libraries. Getting Started with FireDucks Getting started with FireDucks is straightforward. You can install the latest version using pip: pip install -U fireducks Once installed, the easiest way to try it is to replace your normal pandas import: import fireducks.pandas as pd From there, you can continue writing code using the pandas syntax you are already familiar with. For example: import fireducks.pandas as pd df = pd.read_parquet("transactions.parquet") result = ( df[df["price"] > 100] .groupby("category", as_index=False) .agg( total_sales=("price", "sum"), average_price=("price", "mean"), ) .sort_values("total_sales", ascending=False) ) print(result) If you have used pandas before, there is nothing unusual here. We load a Parquet file, filter rows where the price is greater than 100, group the data by category, calculate a few aggregations, and sort the final result. The important difference is what happens behind the scenes. FireDucks does not necessarily execute every operation immediately. Instead, it can build and optimize the sequence of operations first. When you actually need the result — such as when calling print() — the lazy execution is triggered and FireDucks runs the optimized workload. How We Benchmarked It To make the comparison fair, I used the same datasets and workloads for both pandas and FireDucks. We generated a reproducible 10-million-row dataset along with a separate 2-million-row lookup table. The benchmark covered seven common DataFrame workloads: Parquet reading, filtering, low-cardinality groupby, high-cardinality groupby, sorting, joining, and a more realistic chained data-processing pipeline. Each workload received one warm-up run followed by five measured runs. We alternated the execution order between pandas and FireDucks, reported the median execution time, and verified that both libraries produced equivalent outputs. Since FireDucks uses lazy execution, its results were explicitly materialized using ._evaluate() so that the benchmark captured the actual execution time. The benchmark was CPU-only and ran on Linux with the following environment: CPU: 9 AMD EPYC 9V74 cores RAM: 15.93 GiB Python: 3.12.13 pandas: 2.3.3 FireDucks: 1.4.4 NumPy: 2.5.2 PyArrow: 21.0.0 pandas vs. FireDucks Benchmark Results After running the benchmark, these are the results we got across all seven workloads. Operation pandas Median FireDucks Median Speedup Parquet read 0.3243 seconds 0.1220 seconds 2.66x Filter several columns 0.5159 seconds 0.0444 seconds 11.63x Low-cardinality groupby 0.9061 seconds 0.0587 seconds 15.44x High-cardinality groupby 2.1634 seconds 0.3765 seconds 5.75x Sort full dataset 15.2674 seconds 0.7352 seconds 20.77x Join with 2M-row lookup 1.6886 seconds 0.5271 seconds 3.20x Chained pipeline 1.0418 seconds 0.1754 seconds 5.94x FireDucks was faster in every test. The largest improvement came from sorting, with a 20.77x speedup, followed by low-cardinality groupby at 15.44x and filtering at 11.63x. Even for the more realistic chained pipeline, FireDucks completed the workload 5.94x faster than pandas. Can FireDucks Really Make Pandas 20x Faster? Yes, but not for every workload. In our benchmark, FireDucks was 20.77x faster than pandas when sorting the full 10-million-row dataset. Across all seven tests, we saw speedups ranging from 2.66x to 20.77x, with a geometric mean of 7.28x. And we are not the only ones seeing these improvements. Toyota Technical Development Corporation (TTDC) tested FireDucks in its internal AI framework and reported a 60% reduction in data-analysis time and a 76% decrease in analysis-PC operating time. Data science writer Avi Chawla also tested FireDucks on Google Colab and saw pandas code drop from 12.3 seconds to 3.5 seconds, roughly a 4x speedup. So yes, FireDucks can make pandas workloads significantly faster. Just don't expect 20x every time — the actual improvement depends on your data, operations, hardware, and workload. Final Thoughts What impressed me most about FireDucks was not just the benchmark numbers, but how simple it was to get started. I installed the library, changed the pandas import, and ran almost the same pipeline I was already using. It worked out of the box and was noticeably faster. In our benchmarks, FireDucks was faster across all seven workloads, with speedups ranging from 2.66x to 20.77x. Of course, you are not going to get a 20x improvement every time, but there is a lot of potential here — especially for workloads where FireDucks can take full advantage of lazy execution and multiple CPU cores. Even a small performance improvement can matter when you are working with large datasets. Faster pipelines can reduce compute costs, make experimentation quicker, and let you run the same analysis multiple times a day without spending as much time waiting for results. Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.