Planning Poker for Data Engineering Teams: Estimating Pipelines, Migrations, and Data Quality Work

Data engineering teams increasingly operate in agile frameworks — two-week sprints, backlog refinement, sprint reviews — but the standard estimation literature was written for teams building user-facing product features. A story about moving data from a raw S3 bucket through a transformation layer into a dimensional model doesn't map onto the "As a user I want..." format, and the complexity of that work doesn't respond well to estimates made without understanding upstream source systems, data volume, and quality characteristics.

The result is a pattern that most data teams will recognize: either estimates are essentially random, made by whoever speaks first, or the team gives up on estimation and operates without sprint commitments. Both make it hard to earn trust from stakeholders or plan work across quarters.

Planning poker is still the right tool. It forces the team to surface what they collectively know and don't know before work begins. But it needs adapting for data engineering reality.

Why Data Engineering Work Is Distinctly Hard to Estimate

Data work carries a specific kind of complexity that product feature work usually doesn't:

The data tells you what the work actually is. You can read a feature spec and produce a reasonable estimate. With a pipeline ticket, you often can't know how complex the transformation is until you've profiled the source data. A "simple join" between two tables turns into a multi-week project when you discover the join key has 15% nulls and a timestamp drift issue.

Upstream systems are outside your control. Third-party API responses change. The source database team alters schemas without notice. Vendor data exports shift format between delivery batches. These changes invalidate estimates that were made against a snapshot of reality.

Data quality problems are discovered, not invented. You cannot estimate "fix data quality issues in the orders table" meaningfully until you've run a profiling analysis. The shape of the problem is unknown until investigation.

Testing is expensive and often manual. Validating that a pipeline produces correct output requires running it against real or representative data, checking row counts, validating aggregates, and comparing against source system totals. This validation work is routinely underestimated — and it often takes longer than the transformation itself.

Technical debt compounds silently. A pipeline built on assumptions from 18 months ago may be technically functional but increasingly fragile. Extending it is much harder than extending a well-structured pipeline, but the ticket description looks the same from the outside.

None of this makes estimation futile. It means your team needs calibration anchors specific to data work, and estimation sessions need to surface the right questions before votes are cast.

Setting Up Calibration Anchors for Data Engineering

Spend one session as a team defining what each point value means for your typical work. These examples are starting points — calibrate them against your own past tickets:

Print these on a card and keep them visible in refinement sessions. The anchors help prevent the common pattern where one engineer has internalized the complexity of a particular source system and votes 13 while everyone else votes 3.

Work Types and How to Approach Each in Estimation

ETL and ELT Pipeline Work

Pipeline tickets range from trivial to genuinely open-ended, and the description often doesn't tell you which. Before the team votes, the facilitator should ask:

Source data questions:

• Has someone profiled this source? Do we know schema, volume, and update frequency?
• Are there known data quality issues?
• Is the source system documented and stable, or changing?

Transformation questions:

• Is the business logic documented and agreed upon?
• Are there edge cases (nulls, duplicates, late-arriving data) that are understood?
• Does the output need to match an existing calculation, or is this net-new?

Infrastructure questions:

• Does this fit our existing orchestration pattern, or does it require new components?
• What are the freshness and SLA requirements?

If the team can't answer most of these before voting, the ticket needs more refinement or a spike. Pipeline tickets that go into a sprint with unanswered source questions routinely take far longer than estimated.

When votes diverge, the outlier almost always reflects one engineer knowing something others don't — a nuance about the source system, a past incident with that vendor's data export, a hidden dependency. Surface it explicitly.

Schema Migrations and Database Changes

Schema migrations have a specific risk profile that estimation needs to account for. The technical change might be small, but the coordination overhead and validation requirements inflate the effort.

A useful pattern: split every non-trivial migration into at least two stories.

Story 1: The migration itself. Write the migration script, test it in staging, verify rollback procedure. This part is often estimated too optimistically.

Story 2: Downstream impact assessment and remediation. Which pipelines, reports, and models consume this table? Which need changes? Who needs to be notified? This story is frequently forgotten in estimation entirely, then surfaces as surprise work mid-sprint.

For production migrations on large tables, add a third story: coordination and execution. Scheduling the maintenance window, running the migration in production, monitoring for issues, confirming everything downstream is healthy — that's real work, and it belongs in estimates.

Data Quality Initiatives

Data quality work is one of the hardest categories to estimate because the scope is almost always discovered, not specified. Trying to size "fix data quality in the customer dimension" before running profiling is guesswork.

The right structure for data quality work:

Phase 1 — Profile and scope (spike). Run data profiling tools, document the issues found, estimate the remediation. Output: a written summary of what's wrong and a list of stories with rough estimates.

Phase 2 — Remediation stories, sized individually. Once you know what's broken, each fix can be estimated on its own. "Deduplicate customer records using email as the canonical key: estimated 5 points" is a real estimate. "Fix customer data quality: estimated 20 points" is not.

Phase 3 — Monitoring and alerting. Every data quality remediation should include work to detect the same issue in the future. This is almost always underestimated and almost always left out of initial scoping.

Teams that follow this structure find their estimates far more accurate because they've separated the discovery cost from the remediation cost.

Exploratory Analysis and Research

Exploratory work — understanding a new data source, building a one-time analysis for a business decision, investigating anomalies in production metrics — is genuinely hard to estimate in story points because the output is open-ended.

Two approaches work for different contexts:

Time-box it. "Spend three days building a prototype report to answer the revenue attribution question" is a legitimate sprint commitment. You are estimating time, not scope. This is appropriate when the business question is well-defined but the analytical path isn't.

Decompose to the first concrete deliverable. "Profile the new event tracking data and produce a schema documentation document" is estimable. "Analyze user behavior" is not. Break exploratory work into concrete, documentable outputs, then estimate each.

Vague exploratory tickets are one of the main sources of velocity instability on data teams. The fix isn't to avoid exploratory work — it's to define what done looks like before the sprint starts.

Backfill Work

Backfilling historical data is a predictable source of underestimation on data teams. The technical work of writing the backfill query or pipeline is often estimated, but several adjacent costs are not:

• Compute time and cost. Running a backfill against 18 months of raw data takes real time and may have real infrastructure cost implications.
• Validation. Confirming the backfill produced correct output requires comparison against source systems, which is often manual and time-consuming.
• Downstream impact. Backfilling a fact table may cause recalculation in dependent models, dashboard cache invalidation, or alert noise.
• Rollback planning. What happens if the backfill produces incorrect data? The rollback plan and its testing belong in the estimate.

A useful rule of thumb: if a ticket involves backfilling more than 90 days of data on a production table, add at least 3 points for validation and coordination overhead, regardless of how simple the logic looks.

Facilitation Tips for Data Team Refinement Sessions

Bring the Data to the Room

Unlike product feature estimation, where a mockup or user story conveys most of what's needed, data engineering estimation benefits from having actual schemas, sample data, and row counts visible during the session. If the team is estimating a pipeline from a new source, screen-share the API documentation or schema definition. If it's a transformation question, show a sample of the input data.

Teams that estimate from abstract descriptions consistently underestimate. Teams that estimate with actual data in front of them produce better-calibrated numbers because they can see the edge cases directly.

Invite Domain Experts for Specific Tickets

Data quality and schema tickets often have a domain expert — a business analyst, a product manager, a data consumer — who knows what the data should represent. Bring them into the refinement session for the specific tickets they can inform. Their presence surfaces business logic requirements that engineers wouldn't think to ask about.

Tools like Planning Poker support anonymous joining without accounts, which makes it practical to invite an analyst or a stakeholder into a single refinement session without requiring them to create a profile or be added to a team workspace.

Surface the "Has Anyone Looked at This Data?" Question Early

Before estimation begins on any pipeline or transformation ticket, ask: "Has someone profiled the source data for this?" If the answer is no, that ticket is a spike candidate. If yes, the person who profiled it should brief the team before votes are cast.

This one habit reduces estimate variance more than any other single change on data engineering teams.

Use Rounds, Not Just One Vote

Data engineering work often has high initial vote dispersion — one engineer knows the source system intimately, another doesn't know it at all, and their estimates reflect that gap. Rather than treating the first round as final, use the first round to surface divergence, discuss it, and revote.

A wide first-round spread on a data ticket is almost always signal: someone has context the rest of the team needs. The second round, after that context is shared, produces a more reliable estimate.

Integrating Linear and Jira with Your Data Team's Estimation Workflow

Most data engineering teams maintain their backlog in Jira, Linear, or similar tools alongside their engineering counterparts. Running estimation sessions that pull directly from those tools eliminates the copy-paste overhead that causes tickets to get lost or outdated between systems.

Planning Poker integrates directly with both Linear and Jira, pulling your data team's backlog into estimation sessions and writing estimates back to the source ticket automatically. For data teams that run joint refinement sessions with product and platform engineers, this means everyone works from the same ticket list with no manual synchronization.

The async voting feature is particularly useful for data teams with members across time zones. A pipeline ticket that needs context from an engineer in a distant timezone can be submitted for async voting — each team member votes when they're ready — and discussion happens in comments rather than requiring a synchronous meeting.

Measuring and Improving Estimation Accuracy

Data teams that track estimation accuracy over time find the same categories misfiring in predictable ways. Pipeline work from unfamiliar source systems comes in late because profiling happened after estimation rather than before. Validation gets underestimated across every ticket type because it doesn't feel like "real" engineering work until it consumes the whole sprint. Backfill jobs consistently balloon from the coordination and rollback overhead that nobody wrote down. Data quality tickets are nearly impossible to size accurately without the spike-then-estimate structure.

The patterns will be specific to your team's context — your source systems, your stack, your business domain — but they will be patterns. Generic industry benchmarks won't tell you that your team consistently underestimates work from your legacy CRM because the schema is undocumented. Your own historical data will. Track accuracy by work type for two or three sprints and you'll have something more useful than any benchmark: a calibration that belongs to your team.

Where to Start

Planning poker works for data engineering teams. The adaptations aren't complicated — calibrated anchors, data profiling before votes, phased stories for migrations and quality work, time-boxed exploratory tickets — but they require the discipline to enforce them consistently.

The discipline of naming what you know and don't know before a sprint begins is exactly what planning poker is designed to build. For data teams, where the gap between what a ticket looks like and what it actually involves can be enormous, that discipline pays off quickly.

Your estimates don't need to be perfect to be useful. They need to be honest, consistent, and improving. Start with your next refinement session, bring the actual data to the room, and let the vote reveal where your team's understanding diverges.