Data Pipeline

Short answer: Reports, dashboards and automations that use data from another system usually depend on a data pipeline to deliver it. The destination might be a data warehouse, a store of raw files, another application, a report or a machine learning model. Pipeline is a broad term. One common type copies data, cleans it and loads it into a central store. Other pipelines move data without changing it, or process it continuously as events happen.

A typical pipeline collects data, checks it, transforms it, loads it and records what happened. Pipelines run in two main modes. Batch pipelines run on a schedule, such as every night or every hour. Each run processes the data collected since the last run, or reloads it in full. Streaming pipelines process each record or event within seconds of it occurring. Batch is simpler and cheaper and is enough for most management reporting. Streaming suits cases such as fraud checks or live stock levels. Two related terms are easy to mix up. Data integration is the goal of combining data from different systems. A pipeline is the running process that does that work.

A pipeline starts with a trigger, such as a schedule, a new file arriving or a message from another system. It extracts data through a direct system connection, a database query or a file transfer. It runs checks, such as whether the expected number of records arrived and whether required fields have values. It also checks whether totals match the source. It then transforms the data as needed and loads it into the destination. A scheduling tool controls the order of steps, so sales data finishes loading before the daily stock report runs. Logs and alerts tell the responsible person when something fails. A well-built pipeline can rerun after a failure without creating duplicate records.

A retailer has 25 stores, and each store system exports a daily sales file at midnight. A pipeline collects the 25 files and checks that each one contains records and matches the store summary. It converts them to one format and loads them into the warehouse before 6 in the morning. One night only 24 files arrive, and the pipeline alerts the data owner. Without that check, the regional report would have understated sales by about 4%, assuming stores of similar size.

If a pipeline fails or loads only part of the data, the reports and automations that depend on it are wrong. When a dashboard figure looks wrong, the cause is often a failed or partial load rather than the report itself. The data owner should decide who receives an alert when a pipeline fails and how quickly someone must fix it. Each report should also show when its data last refreshed.

Pipelines break when source systems change, for example when a vendor renames a field or changes a connection. Each pipeline needs an owner and ongoing maintenance, so the number of pipelines becomes a running cost. Streaming adds expense and complexity and is often unnecessary for decisions made weekly or monthly. Every copy of data in a new location adds security and privacy exposure that someone must manage.