Data Engineering logopeterindia.net
Knowledge Hub

Data Engineering

A hub of data engineering directories — architectures and platforms, integration and pipelines, databases, data quality, governance and cataloguing, and analytics and AI data — covering the tools that make data reliable and usable.

36 topics•5 categories•Updated October 2026

Data Architecture & Platforms

10 entries

Architectures and platforms that organise how data is stored, shared and served.

01Types of Data ArchitecturesFrom Lambda to the data lakehouse: the foundational data architectures behind modern analytics, AI and enterprise scaling. 02Data Lakehouse PlatformsPlatforms that merge the flexibility and scale of data lakes with the structure, performance and governance of warehouses. 03Medallion ArchitectureBronze, silver and gold layers that progressively improve the structure and quality of lakehouse data. 04Cloud Data WarehousesTop cloud data warehouse platforms compared. 05Data Fabric SolutionsPlatforms that connect and govern data across clouds and systems: virtualization, active metadata and knowledge graphs. 06Data Fabric SoftwarePlatforms that integrate, orchestrate and govern data across distributed environments with a unified architecture. 07Data Mesh SolutionsPlatforms and tools for building a data mesh: data products, federated query, catalogs and governance. 08Data Fabric vs. Data MeshHow to choose between the two enterprise data architectures. 09Translytical Data PlatformsPlatforms combining transactional and analytical processing in a single engine for real-time operational analytics. 10Semantic Layers & Metric StoresSemantic layer and metric store platforms that give analytics and AI a shared business vocabulary.

Data Integration & Pipelines

6 entries

Moving, transforming and streaming data between systems.

11Data Integration, ETL and CDC ToolsETL/ELT, change data capture, data movement, streaming and orchestration platforms. 12Change Data Capture (CDC) PlatformsLog-based replication and streaming: Debezium, Qlik Replicate, GoldenGate, Striim, Fivetran and cloud services. 13Streaming Analytics PlatformsKafka, Flink and other real-time data streaming platforms. 14Streaming DatabasesCurated real-time SQL platforms that maintain results over streaming data. 15DataOps PlatformsPlatforms that apply agile and DevOps principles to data pipeline development, testing and deployment. 16Data Wrangling ToolsTop platforms for cleaning, shaping and preparing data.

Databases

8 entries

Database systems for transactional, analytical, graph, vector and AI workloads.

17Database SystemsRelational, NoSQL, NewSQL and specialised database systems for enterprise data management. 18NoSQL DatabasesSchema-flexible document, key-value, wide-column and graph databases built for scale and speed. 19Distributed SQL DatabasesHorizontally scalable SQL databases with strong consistency. 20Time Series DatabasesDatabases optimised for time-stamped metrics, events and sensor data. 21Vector DatabasesDatabases for embeddings and similarity search. 22Databases for AI Model TrainingDatabase systems for features, embeddings and AI model and agent engineering. 23Knowledge Graphs & Graph TechnologiesGraph databases, query languages and knowledge graph platforms for connected data. 24Graph DatabasesNeo4j, TigerGraph, Neptune, RDF stores and other graph database platforms.

Data Quality, Governance & Cataloguing

7 entries

Keeping data trustworthy, discoverable and compliant.

25Data Quality and Testing FrameworksFrameworks for profiling, validating and monitoring data quality across the pipeline lifecycle. 26Data Quality SolutionsEnd-to-end solutions for measuring, improving and sustaining data quality. 27Data Observability ToolsTools for monitoring pipeline health, anomalies, freshness and schema drift. 28Data Governance PlatformsSystems of record for who owns data, where it lives, who can access it and how it is used. 29Enterprise Data Catalogs for DataOpsCatalog platforms with automated metadata management, lineage tracking and governance. 30Data Modeling ToolsTools for designing, documenting and maintaining logical and physical data models. 31Structured Data Archiving (SDA) SolutionsLong-term retention, compliance archiving and retrieval of structured enterprise data.

Analytics, Visualisation & AI Data

5 entries

Using data for analysis, dashboards and machine learning.

32Data AnalyticsBig data, streaming, log analytics, AIOps, security analytics, visualisation and business intelligence platforms. 33Data Visualization ToolsGuide to charts, dashboards and visual analytics tools. 34Data Science TopicsPlatforms, tools and resources for data science. 35Data Sets for AI Model TrainingPublic and commercial datasets for training, fine-tuning and benchmarking models. 36Data Labeling SoftwarePlatforms for scalable, human-in-the-loop data labeling and annotation.

Frequently Asked Questions

What is data engineering?

Data engineering designs and runs the systems that collect, move, store, clean and serve data, so analysts, data scientists and AI applications can rely on it.

Where should I start?

Start with Types of Data Architectures to understand lakes, warehouses and lakehouses, then Data Integration, ETL and CDC Tools for moving data, and Data Quality and Governance for trust.

What is the difference between a data fabric and a data mesh?

A data fabric connects data across systems using metadata, integration and virtualization, while a data mesh organises ownership around domain-owned data products. The Data Fabric vs. Data Mesh page compares them.

How do lakehouses relate to warehouses?

A lakehouse adds warehouse-style transactions and governance to low-cost lake storage using open table formats such as Iceberg and Delta Lake, so one platform serves BI and machine learning.

Which database type should I choose?

Use relational or distributed SQL for transactions, time series databases for metrics, vector databases for embeddings and graph databases for connected data. The Databases section links to each.