Top Technologies for Data Engineering

Data engineering is the work of building the systems and processes that turn raw data into consistent information for analysis and machine learning: storing data, processing it and gluing the steps together. The source article picks nine technologies that it argues changed the field: Pandas, Parquet, Spark, Airflow, object storage, cloud data warehouses, Arrow, Iceberg and DuckDB. The order has no meaning, and dbt gets an honourable mention as a consequence of the shift to ELT rather than a standalone innovation.

Wes McKinney started Pandas in 2008 to bring R's table processing to Python. The article argues it did more than help with data wrangling: it helped make Python dominant for data professionals and paved the way for the DataFrame API, a table you can program against in an imperative language, which is now one of the most used data-processing interfaces.

Row-wise formats such as CSV write all fields of a record together, which wastes reads when analytics touch a few columns of many rows. Columnar formats store each column together, so queries read only the columns they need and similar values compress well. The article says the idea predates Parquet but that Parquet, introduced in 2012, took columnar storage out of the private world of database vendors and made it the de facto format; newer formats such as Lance and Vortex aim to do better for modern and AI workloads, though the author does not expect Parquet to be replaced soon.

Before Spark, MapReduce exchanged data through disks, which was reliable but slow, and its paradigm was rigid. Spark, open-sourced in 2010, processes data in memory and offers friendlier interfaces, first RDDs and then SQL and the DataFrame, in one engine that handles batch, streaming and machine learning. The article calls it the king of distributed processing frameworks.

Maxime Beauchemin created Airflow at Airbnb in 2014 because existing orchestration tools were too rigid or did not scale to dynamic data pipelines. It brought the DAG and task model, a simple way to glue steps together and schedule them in Python, a UI for observability and the ability to backfill. The article notes competition from Dagster and Prefect, which it says were inspired by Airflow, but does not expect Airflow to be replaced soon.

The article says object storage changed the whole software industry: extremely durable, able to store any data, cheap and high-throughput. In data engineering it replaced HDFS, so teams moved from operating storage clusters to renting it with no management overhead, and nearly all lakes and lakehouses sit on object storage; it credits AWS S3 as the first.

Before Snowflake, BigQuery, Databricks and Redshift appeared in the early 2010s, warehouses meant buying servers and licences with storage and compute tightly coupled. Pay-as-you-go cloud warehouses made them accessible in a few clicks. The article says this drove two changes: ETL became ELT, loading raw data first and transforming later, and transformation moved into SQL inside the warehouse, as dbt shows.

Arrow is less visible than the others but changes two things: a standard specification for columnar data in memory, and efficient mechanisms to move and share it. Without a standard, exchanging data between systems means serialising and deserialising it. The article says nearly every modern data system, from Pandas and Polars to BigQuery, Snowflake, Spark, DuckDB and ClickHouse, supports or speaks Arrow, even though you rarely use it directly.

The classic two-tier architecture kept raw data in a lake and served data from a warehouse, because lakes lacked schema management and ACID guarantees. Open table formats add a metadata layer that gives lake data a table abstraction, ACID transactions, schema evolution and time travel, so the lake can behave like warehouse storage while staying open. The article names Iceberg, Delta Lake and Hudi, and picks Iceberg as the representative because it is community driven and widely supported.

The article is unsure DuckDB belongs beside the others in impact, but includes it for its unique vision. While other OLAP systems assume data will be big, DuckDB's creators bet that small and medium data dominates, and built a lightweight embedded OLAP database that runs like a library with one-line setup. It enables single-server analytics, a fast alternative to Pandas for wrangling, and user-facing analytics inside applications, and it pushes other tools to improve local development.

The article leaves dbt off the list because it sees it as a consequence of ELT, not a standalone innovation, but says its impact on how teams use SQL for transformation is undeniable.

Not in the article's nine, but common in modern data stacks.

Neighbouring pages on this site.