Scrum vs Kanban for Data Teams
Part 2 of the Incremental Delivery series: neither framework is wrong for data work - each is right for a different slice of it, which is exactly why most mature data teams end up running both.
The Scrum-versus-Kanban debate gets framed as a single choice more often than the question deserves. In software engineering, Scrum is the default for most product teams and Kanban fills specific gaps - support queues, ops work, continuous-flow environments. Data teams routinely try to force the same single choice onto work that doesn't behave like a single kind of work, and that's usually where the friction starts.
This is Part 2 of our series on incremental delivery for data and analytics teams. Part 1 covered why data engineering only partially satisfies the conditions that make software Agile workable. This post gets specific: where Scrum genuinely fits data work, where Kanban genuinely fits it, where each breaks down, and the hybrid configuration most data teams that have been through a few painful sprints eventually converge on.
The Real Question Isn't Which Framework Is Better
Scrum and Kanban aren't competing answers to the same question - they're designed for different conditions. Scrum's fixed-length sprints, defined roles, and ceremonies bring structure and a predictable rhythm for planning and stakeholder demos. Kanban's continuous flow and work-in-progress limits are built for exactly the kind of unpredictable, externally-driven interruption that Scrum handles poorly. The mistake isn't picking the "wrong" one - it's assuming a data team's entire body of work is uniform enough for either to be the single right answer.
Where Scrum Fits - and Where It Breaks
Scrum works well in data engineering under specific, fairly narrow conditions: a small team, requirements from a single product owner or stakeholder group, and infrastructure that's genuinely stable underneath the work. Analytics teams building narrowly scoped dashboards for one business unit, where the underlying data model rarely shifts under them, are the most reliable fit - the standard two-week cadence, sprint goals, and story-point estimation produce a predictable rhythm because the work itself is predictable enough to plan.
Scrum breaks down for data work in the same conditions Part 1 covered: cross-team infrastructure dependencies, mixed exploratory and delivery work, and significant infrastructure churn. Story points estimated for infrastructure tasks are notoriously unreliable, because the unknowns that emerge during execution - a migration that uncovers unanticipated complexity, a dependency that wasn't visible at planning - aren't estimation errors so much as evidence that the work never had bounded scope to begin with. Teams that try to force this kind of work into two-week Scrum sprints tend to end up with sprint plans that bear limited resemblance to what actually got worked on, and retrospectives that keep blaming "estimation" for a problem estimation refinement can't fix.
Where Kanban Fits - and Where It Breaks
Kanban's continuous flow model handles unpredictable, blocking dependencies far more gracefully than Scrum does. Infrastructure and platform work - provisioning storage, updating a schema registry, patching a pipeline that broke overnight - arrives on its own schedule, gets prioritised by impact and urgency, and gets pulled by available capacity without waiting for a sprint boundary to open. WIP limits keep the board from silently accumulating more in-flight work than the team can actually absorb, which is precisely the failure mode unplanned data work tends toward.
Kanban's weakness for data teams is almost the mirror image of Scrum's strength: it deliberately doesn't define roles or sprint-level process, which means a team running pure Kanban can lose the planning rhythm that stakeholders - a CFO waiting on a dashboard, an exec team expecting a quarterly roadmap update - reasonably expect from a data function. Kanban optimises for flow, not for predictable, demo-able commitments, and a team that's all flow and no planning cadence tends to struggle communicating progress to anyone outside the team.
The Hybrid Most Mature Data Teams Run
The pattern we see most often in data organisations that have iterated past their first failed Scrum sprint is a deliberate hybrid: domain teams - the people building dashboards, models, and analytics-facing pipelines - run Scrum, with two-week sprints, sprint goals, and the standard ceremonies for feature-shaped work. The platform team - the people who own the shared infrastructure everyone else depends on - runs Kanban, with requests flowing continuously onto a board, prioritised by impact, and pulled by available engineers without sprint boundaries.
The two methodologies are bridged by an explicit service level agreement: the platform team commits to specific provisioning latencies for common request types - a new streaming topic within a defined number of business days, a new storage volume within another - and domain teams plan their sprints with that SLA in mind, requesting infrastructure they'll need one or two sprints ahead rather than discovering the need mid-sprint. This works because it stops forcing two fundamentally different kinds of work - predictable, internally-prioritised feature delivery and unpredictable, externally-driven infrastructure work - into the same methodology and expecting both to behave the same way.
Making the Hybrid Actually Work
How to Decide: A Quick Framework
If your team is small, your stakeholder group is singular, and your underlying infrastructure genuinely doesn't change out from under you sprint to sprint, pure Scrum is a reasonable starting point and probably not worth over-engineering. If your work is dominated by unplanned, externally-driven requests - an ops or platform-heavy function - pure Kanban likely fits better than forcing sprint boundaries onto work that doesn't respect them. If your organisation has both kinds of work happening simultaneously across different teams or different parts of the same team's backlog - which describes most data functions past a certain size - the hybrid is very likely where you'll end up eventually, and starting there deliberately is usually less painful than discovering it after a failed sprint forces the conversation.
Scrum vs Kanban vs Hybrid, Side by Side
| Dimension | Pure Scrum | Pure Kanban | Hybrid (domain + platform) |
|---|---|---|---|
| Best fit | Small team, single stakeholder, stable infrastructure | Ops/platform-heavy, unpredictable request flow | Mixed feature work and infrastructure work |
| Planning rhythm | Strong - fixed sprints, clear cadence | Weak by design - no defined cadence | Domain side keeps Scrum's cadence |
| Handling of unplanned infra work | Poor - breaks sprint commitments | Strong - built for this | Platform side absorbs it via Kanban |
| Stakeholder demo cadence | Predictable, sprint-aligned | Ad hoc, harder to schedule | Predictable on domain team side |
| Coordination overhead | Low - single team, single process | Low - single team, single process | Higher - requires an explicit SLA |
| Common failure mode | Story points shatter against infra unknowns | Stakeholders lose visibility into progress | Platform team silently becomes a bottleneck |
- Scrum and Kanban aren't competing answers for data teams - they fit different kinds of work, and most data functions have both kinds happening at once.
- Pure Scrum fits small teams with a single stakeholder group and genuinely stable infrastructure; it breaks down under cross-team dependencies and infrastructure churn.
- Pure Kanban handles unpredictable, externally-driven work well, but its deliberate lack of defined roles and cadence can leave stakeholders without the planning rhythm they expect.
- The hybrid most mature data teams run puts domain teams on Scrum and the platform team on Kanban, bridged by an explicit SLA - separating predictable feature work from unpredictable infrastructure work rather than forcing both into one methodology.
- The hybrid's main failure mode is the platform team silently becoming a bottleneck - WIP limits, an explicit escalation path, and quarterly dependency mapping are what keep the SLA from quietly slipping unnoticed.
Numlytics helps data leaders design sprint structures that actually fit their mix of feature and infrastructure work. Explore our data strategy consulting practice, or speak with a certified consultant about your team's specific delivery model.