Determine The Original Set Of Data

You're staring at a dashboard. Day to day, the numbers look clean. Here's the thing — the charts trend upward. Someone — maybe your boss, maybe a client — asks a simple question: "Where did this data actually come from?

And suddenly, the room goes quiet.

Determine the original set of data sounds like a technical task. In practice, it's detective work. And most teams are terrible at it Worth keeping that in mind..

What Is Determining the Original Set of Data

At its core, this is about provenance. Not just "what table did this query hit?" but the full chain: who collected it, how, when, under what conditions, and what happened to it between the source and your screen.

Think of it like food labeling. In real terms, frozen? The dish looks the same either way. But was it free-range? Now, you see "chicken" on the menu. In real terms, marinated in something you're allergic to? The difference is in the history you can't see Which is the point..

In data terms, the original set is the rawest version that still exists — before cleaning, before joins, before the analyst "fixed a few outliers" at 2 AM before a board meeting.

It's not the same as a primary source

A primary source is where data originates. Practically speaking, the original set is the earliest accessible version you can verify. Sometimes they're the same. Often they're not. The sensor log got overwritten. The survey platform purged raw responses after 30 days. The CRM sync only keeps the last state.

You work with what survives Simple, but easy to overlook..

Why "raw" is a moving target

Ask three engineers where the raw data lives. You'll get four answers.

• The Kafka topic? Raw.
• The landing zone in S3? Also raw.
• The first dbt model that casts timestamps? Some call that raw too.

The original set isn't a location. It's a commitment — a decision to treat a specific artifact as your ground truth, with documentation to back it up.

Why It Matters / Why People Care

You might think this is academic. It's not Easy to understand, harder to ignore..

Regulatory survival

GDPR's right to explanation. CCPA's data lineage requirements. SOX audit trails. If a regulator asks "show me exactly how you calculated this metric," and your answer is "the dashboard does it," you're in trouble And that's really what it comes down to..

Banks have lost millions because they couldn't prove a risk model's input data hadn't been silently altered. Healthcare companies have faced class actions over "anonymized" datasets that weren't — because no one tracked what the original identifiers looked like.

Debugging the "it worked yesterday" problem

Monday: revenue looks great. Tuesday: it's off by 12%. No code changed.

If you can't determine the original set — the exact rows, the exact filters, the exact timestamp cutoff — you're guessing. And guessing at scale is expensive.

I've seen teams spend two weeks chasing a "bug" that turned out to be a downstream analyst changing a WHERE clause from >= to > on a date column. The original data never changed. The definition did Worth keeping that in mind. Which is the point..

Trust is the real currency

Stakeholders don't trust data because it's pretty. They trust it because they know where it came from.

When a VP asks "can I use this for the board deck?" and you can say "this comes from the 3:00 AM ETL run of the orders table, partitioned by created_at, with test orders filtered by the is_test flag — here's the lineage graph," they nod. When you say "uh, the dashboard team owns that," they hesitate Easy to understand, harder to ignore..

Honestly, this part trips people up more than it should.

That hesitation costs you influence.

How It Works (or How to Do It)

There's no single tool that solves this. It's a practice. Here's what it looks like in the wild Worth keeping that in mind..

Step 1: Define your "original" — explicitly

Don't assume. Write it down Easy to understand, harder to ignore..

Example:

The original set for core.fct_revenue is raw.stripe_charges as of the 06:00 UTC snapshot, filtered to status = 'succeeded' and type = 'charge', with no deduplication applied That's the part that actually makes a difference..

That's a contract. If someone changes the Stripe schema, you know exactly what breaks. If they don't, you've created ambiguity — and ambiguity compounds Simple as that..

Step 2: Capture the transformation chain

Every hop matters. Not just "dbt model X feeds model Y." Capture:

• Filter logic — every WHERE, every JOIN condition
• Aggregation choices — SUM vs COUNT_DISTINCT, window frames, timezone handling
• Business rule versions — "V3 of the churn definition, effective 2023-06-15"
• Data quality gates — "rows with null customer_id dropped; 0.3% loss expected"

Tools help. Worth adding: dbt's docs generate, DataHub, OpenLineage, Monte Carlo — they all surface pieces. But the discipline of writing it down? That's human.

Step 3: Version the original set itself

The source changes. Schemas evolve. Partitions rotate Most people skip this — try not to..

If your original set is "the orders table," which version? Worth adding: the one from last week? The one before the currency column was added?

Treat the original set like code:

• Snapshot it (iceberg, delta, or good old CREATE TABLE AS SELECT with a date suffix)
• Tag it (v2024-01-15_orders_raw)
• Reference the tag in every downstream artifact

Step 4: Make lineage queryable

A PDF diagram is useless at 2 AM during an incident Easy to understand, harder to ignore..

Build a lineage API — even a simple one. Even so, if it takes a Slack thread and three people, you don't have lineage. where_does_this_column_come_from('core.SELECT * FROM lineage.fct_revenue.Plus, net_revenue') should return the full chain in seconds. You have folklore.

Step 5: Test the contract

You defined the original set. Great. Now prove it hasn't drifted.

-- Runs daily. Fails if row count deviates >1% from 7-day avg
-- Fails if null customer_id > 0
-- Fails if max(created_at) < current_date - 1 day

These aren't data quality tests. They're provenance tests. They tell you: "The original set I depend on still looks like the original set I defined And that's really what it comes down to. Nothing fancy..

Common Mistakes / What Most People Get Wrong

Confusing "source system" with "original set"

Salesforce is a source system. The original set is which objects, which fields, which filters, which snapshot Worth keeping that in mind..

"We pull from Salesforce" is not a definition. It's a handwave.

Assuming the warehouse is the source of truth

The warehouse is a copy. Sometimes a transformed copy. Sometimes a copy of a copy Not complicated — just consistent..

The original set lives upstream — in the application DB, the event stream, the vendor's API response. The warehouse is where you reconstruct it. If you treat the warehouse as the original, you lose the ability to detect upstream corruption.

Documenting once and forgetting

Lineage rots faster than code.

A column gets renamed.

Step 6: Automate the Feedback Loop

Lineage isn’t static. When a model breaks, the system should alert stakeholders and surface the impacted original sets, filters, and aggregations. For example:

• A WHERE order_date > '2020-01-01' suddenly returns 20% fewer rows due to a source table schema change.
• A COUNT_DISTINCT aggregation starts double-counting because a source column was deduplicated downstream.
  Automated alerts tied to lineage context turn technical failures into business risks. Tools like dbt’s test framework or Monte Carlo can flag drift, but only if they’re wired to lineage metadata.

Step 7: Audit Trails for Lineage Itself

Who changed the v2023-06-15_churn_definition? Why was the JOIN condition altered from customer_id = customer_id to COALESCE(customer_id, 0) = COALESCE(customer_id, 0)? Track lineage changes like code commits. Tools like Git for data or dbt’s snapshots can version control the logic behind lineage relationships. This ensures accountability: if a model misrepresents revenue because a filter was quietly updated, the audit trail reveals the culprit and the rationale Less friction, more output..

Step 8: Contextualize Lineage with Business Impact

Lineage alone doesn’t explain why a model matters. A net_revenue column might feed into:

• A CFO’s monthly report
• A real-time dashboard for sales reps
• A machine learning model predicting churn
  Tagging models with their purpose (e.g., model: fct_revenue.net_revenue, purpose: "CFO_monthly_report") lets teams prioritize fixes. If net_revenue breaks, the CFO’s team knows to escalate immediately, while the ML team might tolerate a 24-hour delay.

Step 9: The Human Layer

Tools automate discovery, but humans must interpret. A data engineer might notice that a JOIN condition on customer_id fails because the source system started using UUIDs instead of integers. A business analyst might realize a COUNT_DISTINCT on product_id now excludes discontinued items due to a new filter. These insights require cross-team collaboration. Lineage is the map; the analysis is the compass Simple as that..

Conclusion

Original set lineage is the backbone of trustworthy analytics. It transforms opaque dependencies into actionable knowledge, ensuring that every SUM, JOIN, and filter is understood, versioned, and tested. Without it, data teams operate in the dark—reacting to outages, guessing at impacts, and rebuilding models from scratch. With it, they move from folklore to science. The discipline to document lineage rigorously, automate its feedback, and contextualize its purpose separates mature analytics teams from the rest. In a world where data is both a product and a liability, lineage isn’t just a best practice—it’s the foundation of resilience.