Batch, SQL & Interactive Query Engines
7 entriesDistributed engines for large-scale batch processing and SQL over big data. Pick by workload: ETL and ML, interactive SQL or data warehousing.
Streaming & Event Processing
3 entriesEngines and platforms that process data in motion. Combine an event log with a stateful processing engine.
Lakehouse Table Formats
5 entriesOpen table formats that bring ACID transactions and schema evolution to data lakes. These formats let several engines share one copy of the data.
Real-Time OLAP Databases
5 entriesAnalytical stores for low-latency queries over fresh, high-volume data. Suited to user-facing analytics and operational dashboards.
Orchestration & Data Movement
3 entriesTools that schedule pipelines and move data between systems. Reliable orchestration is the backbone of batch analytics.
Managed Cloud Big Data Services
9 entriesManaged services that run Spark, SQL and streaming engines without operating clusters.
Python, DataFrame and Distributed Compute Libraries
8 entriesLibraries that scale analysis beyond one machine, or make one machine go further.
Governance, Metadata and Catalogs
8 entriesCataloguing, lineage and access control for large data estates.
Foundational Papers, Benchmarks and Learning
10 entriesPrimary papers and measurements behind modern big data systems.
Related Pages
5 entriesRelated directories on this site.
Frequently Asked Questions
What are big data analytics tools?
Big data analytics tools store, process and query datasets too large or fast for a single machine. They include batch and SQL engines such as Spark, Hadoop and Trino, streaming engines such as Flink and Kafka, table formats such as Iceberg and Delta Lake, real-time OLAP databases and managed cloud services.
How do I choose a big data analytics tool?
Start with the workload (batch ETL, interactive SQL, streaming, or user-facing analytics), data volume and freshness, team skills, governance needs and cost. Prefer open table formats and standard SQL where possible to avoid lock-in, and test candidates on your own data.
What is the difference between a data lake, a data warehouse and a lakehouse?
A data lake stores raw files cheaply, a warehouse stores structured data optimised for SQL, and a lakehouse adds warehouse features such as transactions and schema enforcement to lake storage using table formats like Apache Iceberg, Delta Lake and Apache Hudi.
Is Hadoop still relevant?
Hadoop's HDFS and YARN remain in many enterprises, but new projects often use object storage with Spark, Trino or Flink and a table format. Hadoop-era components such as Hive and Impala continue to be used, and MapReduce itself is largely replaced by newer engines.
Which tools suit real-time analytics?
Streaming engines such as Flink process events as they arrive, Kafka stores the event log, and real-time OLAP databases such as Apache Pinot, Apache Druid and ClickHouse serve fast queries over fresh data. See the real-time and streaming analytics pages for more.