Almost every data and analytics team we work with has tried to run Agile at some point, and almost every one of them describes the same experience: they adopted Scrum because that's what the engineering org next to them uses, ran a few sprints, and found that the ceremonies didn't quite fit. Sprint commitments got missed not because the team was slow, but because a source system changed schema mid-sprint. Story points meant something different for a two-day exploratory analysis than for a well-defined pipeline fix. And "done" meant something different depending on whether the surprise showed up in week one or three weeks after the sprint closed, when a downstream dashboard quietly started showing the wrong number.

This is the pillar guide to a seven-part series on incremental delivery for data and analytics teams - how Agile, Scrum, and Kanban can actually work for data work, not despite its differences from software engineering but by designing around them. This post lays out the whole picture: why data projects need a different agile approach, what a real increment looks like in a data pipeline, how to choose between Scrum and Kanban, how to build a backlog that ships business value instead of technical busywork, how to measure whether any of it is working, and why data deadlines slip more often than software deadlines do. Each section links through to a full deep-dive in the series.

Why "Just Run Scrum" Doesn't Work for Data Teams

Software Agile rests on a set of conditions that mostly hold true for application development: acceptance criteria can be defined clearly up front, outputs are testable, deployment of one team's work is largely independent of another's, and teams aren't tightly coupled to each other's changes. Data engineering only partially satisfies those same conditions, and the gap is structural, not a matter of the team needing better training.

Data work involves infrastructure dependencies that block a team unpredictably - a source system's API changes, a warehouse migration reshuffles a schema underneath a pipeline that was supposed to be untouched this sprint. It involves exploratory analysis whose duration genuinely cannot be bounded in advance, because you don't know how messy the data is until you're inside it. It involves shared platform components - a core dimension table, a central data model - where one team's change affects every downstream consumer simultaneously, unlike most application features. And most distinctively, data quality failures often don't surface in the sprint where the responsible change shipped; they surface three weeks later, when someone finally notices a report has been quietly wrong since a silent upstream change nobody flagged as a defect at the time.

None of this means Agile principles don't apply to data teams. It means the specific mechanics - sprint length, definition of done, what counts as a story, how dependencies get modeled - need to be built for these conditions rather than copied from a software team's playbook. Why Data Projects Need a Different Agile Approach goes deep on this distinction and what to change first.

What Incremental Delivery Actually Means for Data

In software, an increment is a working feature a user can touch. In data, the equivalent increment is a slice of trustworthy, usable data - and the medallion pattern (Bronze → Silver → Gold) is the closest thing data engineering has to shipping working software every sprint. Raw data lands in Bronze exactly as it arrives, untransformed. Silver cleans, deduplicates, and conforms it into something multiple teams can build on. Gold turns it into the business-ready aggregates, metrics, and models that a dashboard or a decision actually consumes.

The mistake most data teams make with this pattern is treating it as an all-or-nothing pipeline build - Bronze, Silver, and Gold planned as one long project with a single go-live date months out. Real incremental delivery means shipping one Gold-layer output that a business stakeholder can use this sprint, even if the underlying Bronze and Silver layers only support that one use case so far and get extended sprint by sprint after. Incremental Delivery: Bronze to Gold Without Waiting Months covers exactly how to slice a medallion build this way without compromising the data quality the layered pattern exists to protect.

Choosing a Framework: Scrum, Kanban, or Both

Scrum's fixed-length sprints and defined roles bring welcome structure and a predictable cadence for stakeholder demos, but they fit least well around the unpredictable, blocking dependencies data teams face constantly. Kanban's continuous flow and work-in-progress limits handle those interruptions far more gracefully, but Kanban deliberately doesn't define roles or sprint-level process, which can leave a team without the planning rhythm stakeholders expect. Most mature data teams land on a hybrid - Kanban-style flow for ongoing pipeline maintenance and unplanned fixes, with Scrum-style time-boxed planning for larger, plannable feature work like a new Gold-layer data product.

Scrum vs Kanban for Data Teams walks through the concrete tradeoffs and how to design the hybrid that fits your team's actual mix of planned feature work and unplanned pipeline fires.

The Backlog Problem Unique to Data Teams

A software backlog is mostly features. A data team's backlog is a mix of new pipelines, data quality fixes, platform maintenance, ad-hoc stakeholder requests, and technical debt - and without a deliberate prioritisation model, the ad-hoc requests and the platform maintenance quietly crowd out the work that actually moves a business metric. The single most common failure mode we see is a backlog with no shared definition of "business value" that both the data team and its stakeholders have agreed to, which means every prioritisation conversation becomes a negotiation from scratch rather than a decision against an existing framework.

How to Build a Data Team Backlog That Delivers Business Value covers how to structure a backlog so that ad-hoc requests, technical debt, and net-new business value compete on the same explicit criteria, rather than whichever stakeholder shouted most recently winning by default.

Bronze to Gold as Your Increment, Not Your Roadmap

It's worth repeating the point from Section 2 in roadmap terms specifically, because it's where most data teams' Agile adoption quietly reverts to waterfall without anyone deciding that on purpose. A twelve-week "build the data platform" project with a single go-live is not incremental delivery, even if the team calls its two-week planning cycles "sprints." Genuine incremental delivery means every sprint ships something a stakeholder can see and use - even a narrow, single-use-case Gold table - rather than every sprint quietly building toward one eventual release. The medallion pattern makes this possible architecturally; the discipline to actually slice work that way sprint by sprint is a delivery practice, not a data architecture decision, and it's the one most teams skip under deadline pressure.

Measuring Whether Delivery Is Actually Working

Software teams have DORA metrics - deployment frequency, lead time for changes, change failure rate, time to restore. Data teams need an equivalent, adapted to a pipeline's actual failure modes: how often a data product ships to a stakeholder, how long it takes from a request landing in the backlog to that data being usable, how often a shipped pipeline breaks or produces a data quality incident after release, and how quickly the team detects and fixes it when it does. Most Heads of Data we talk to are tracking sprint velocity and calling it done - a metric borrowed from software that says almost nothing about whether the business actually trusts and uses what got shipped.

The Metrics Every Head of Data Should Track lays out the specific metric set that replaces velocity with something that actually correlates with stakeholder trust and business impact.

Retrospectives and Why Data Deadlines Slip

A retrospective that only asks "what went well, what didn't, what will we change" tends to surface the same generic answers sprint after sprint for a data team, because the actual causes of missed data deadlines are specific and recurring: undocumented upstream schema changes, source system access delays, scope that expanded the moment stakeholders saw the first working output, and data quality issues that were invisible until three sprints after the responsible change shipped. A retrospective built for data work needs to interrogate those specific failure patterns directly, not just ask the team how they feel about the sprint.

What a Data Sprint Retrospective Should Actually Cover and Common Reasons Data Teams Miss Deadlines-and How to Avoid Them close out the series by turning these recurring failure patterns into a retrospective structure and a set of concrete mitigations you can put in place before the next sprint, not just discuss after this one slips.

Where to Start

If your team is just beginning to formalise how it works, start with Why Data Projects Need a Different Agile Approach and Scrum vs Kanban for Data Teams - the framework decision underneath everything else. If you already run something Agile-shaped but suspect it isn't actually delivering business value, jump straight to the backlog post and the metrics post - those two are usually where the gap between "we do Agile" and "stakeholders trust what we ship" actually lives.

Numlytics helps data leaders redesign how their teams plan, ship, and measure delivery - whether that means standing up a dedicated engineering team to execute against a real incremental roadmap, or a broader data strategy engagement to fix backlog and governance from the ground up. Explore our data strategy consulting practice, or speak with a certified consultant about your team's specific delivery model.