The Eight Patterns at a Glance
8 patternsSummarised from Disha Mukherjee's LinkedIn post; the reference links below cover each in depth.
| Pattern | What it does | When it helps | Watch for |
|---|---|---|---|
| ETL | Extract, transform, then load the target. | Useful when data needs preparation before reaching the target. | Decide which rules must be applied before loading. |
| ELT | Extract, load, then transform within the target platform. | Useful when the destination has suitable processing capabilities. | Loading raw data still requires access controls and ownership. |
| Event-at-a-time | Process an ongoing stream incrementally. | Design around state, event time and backpressure. | Actual latency depends on the full system, including the destination. |
| Micro-batch | Process a stream as a sequence of small batches. | Balance processing overhead against freshness requirements. | Micro-batching is a valid way to implement streaming. |
| Lambda | Maintain batch and low-latency processing paths and combine their outputs for serving. | Gives a fast path and a complete batch path. | Two paths create extra work around logic, testing and consistency. |
| Kappa | Use a streaming path for ongoing processing and reprocess retained events when logic changes. | Avoids a separate batch codebase. | Replay depends on available history, compatible schemas and sufficient capacity. |
| Fan-out | Deliver an event to multiple independent consumers. | Lets analytics, search and notifications react separately. | One consumer's failure should not force every consumer to stop. |
| Event-driven | Start work when a relevant event occurs and route it to the right handler. | Handlers run only when there is something to do. | Make handlers safe to retry when delivery happens more than once. |
ETL and ELT: Where Transformation Happens
4 entriesBatch preparation patterns: transform before loading the target, or load first and transform inside it.
Event-at-a-Time and Micro-Batch Streaming
5 entriesProcessing an unbounded stream incrementally, or as a sequence of small batches.
Lambda and Kappa Architectures
3 entriesCombining batch and low-latency paths, or using a streaming path with replay from a retained log.
Fan-Out and Event-Driven Delivery
4 entriesDelivering one event to many consumers, and starting work when an event occurs.
Source Post
1 entriesThe LinkedIn post this page is based on.
Related Pages
10 entriesRelated directories on this site.
Frequently Asked Questions
What are the main data pipeline patterns?
Eight patterns are commonly compared: ETL, ELT, event-at-a-time streaming, micro-batch, Lambda architecture, Kappa architecture, fan-out and event-driven processing. They answer different design questions, so a single pipeline can combine several of them.
What is the difference between ETL and ELT?
ETL transforms data before loading the target, which suits data that needs preparation first. ELT loads raw data first and transforms it inside the target platform, which suits destinations with strong processing capabilities. Loading raw data still needs access controls and clear ownership.
Is micro-batch streaming?
Yes, micro-batching is one way to implement streaming: it processes a stream as a sequence of small batches. The trade-off is processing overhead against freshness. Event-at-a-time processing handles each event incrementally and has to deal with state, event time and backpressure.
What is the difference between Lambda and Kappa architecture?
Lambda keeps a batch path and a low-latency path and combines their outputs for serving, which creates extra work around logic, testing and consistency. Kappa uses a streaming path and reprocesses retained events when the logic changes. Replay depends on how much history is kept, compatible schemas and enough capacity.
How do fan-out and event-driven processing differ?
Fan-out delivers one event to several independent consumers, such as analytics, search and notifications, so that one consumer failing does not stop the others. Event-driven processing starts work when a relevant event occurs and routes it to the right handler. Handlers should be safe to retry, because delivery can happen more than once.
How do I choose a pipeline pattern?
Start with the freshness you actually need, decide where transformations belong, and plan recovery, replay and downstream delivery. Then combine patterns around the workload. A design is only complete when you can explain how it recovers.
What are common pipeline design traps?
Choosing streaming without a freshness requirement, assuming a retained log keeps history forever, treating retries as safe without checking for duplicate effects, and building two processing paths without budgeting for their maintenance.
How current is this directory?
Links were checked on 23 October 2026. The pattern descriptions follow a LinkedIn post by Disha Mukherjee and are linked to the reference material listed here. Confirm product-specific behaviour in each project's own documentation.