This is the third post of six going into building a table of observables for each of the Infonomics metrics. The aim of this set of posts is to help build the management case that will enable capitalisation of a data entity/object rather than expensing it. I'm using Infonomics as a grammar and vocabulary to tie the management case to the accounting and economics cases. For the journey to this point, check back to my blog post on the economic architecture of data.
Key Question: What empirically happens when we use this data?
Level 1 dimensions
- Experimental evidence
- Effect size / Lift
- Robustness / Generalisability
- Scale of impact
Level 2 indicators and observables
- Experimental evidence
- Number of A/B tests or controlled studies run with vs without this dataset.
- Causal inference and analysis; Bayesian inference and analysis.
- Statistical significance of results (p‑values, confidence intervals).
- Effect size / Lift
- % attributable improvement in key KPIs (e.g. conversion, loss ratio, uptime)
- Dollar value per unit time (e.g. $X/month uplift) attributable to data.
- Uplift vs next‑best alternative data source.
- Robustness / Generalisability
- Number of different contexts where lift has been demonstrated (markets, products).
- Stability of uplift over time (does it decay quickly?).
- Sensitivity to model changes (does the data remain useful across model families?).
- Scale of impact
- Size of population affected by the uplift (customers, transactions).
- % of total business volume influenced by processes that use the data.
How to use
- An asset with no controlled experiments is low PVI by definition, even if IVI/BVI are high.
- PVI score should be driven by measured lifts, not perceived value.
PVI scoring rubric
Below is a 1–5 scoring rubric for PVI to drop into a paper or use as a working template. Tweak weights to fit your context. Assume PVI is the weighted sum of four dimensions:
- Experimental evidence - 25%
- KPI lift magnitude - 35%
- Robustness / generalisability - 20%
- Scale of impact - 20%
1. Experimental evidence quality (weight 25%)
Question: How rigorous is the evidence that this data affects performance?
Score | Descriptor |
1 – 2 None / anecdotal | No controlled experiments. Only opinions, case stories, or uncontrolled before/after comparisons. No clear link from data to KPI movement. |
3 – 4 Weak | At least one experiment or pilot, but with serious design limits (no control group, small N, confounders). Results are suggestive but not reliable. |
5 – 6 Adequate | One or more reasonably designed A/B tests or quasi‑experiments with control groups and basic statistical checks, but limited to a single context or short period. |
7 – 8 Strong | Multiple well‑designed experiments in different cohorts/time periods, with consistent results and clear documentation of methodology and data usage. |
9 – 10 Robust / scientific | Programmatic experimental practice (ongoing A/B tests, hold‑outs, back‑testing). Independent reviews or replication confirm the effects. The evidence base would withstand external audit. |
2. KPI lift magnitude (weight 35%)
Question: How big is the effect on key performance indicators when this data is used?
Score | Descriptor |
1 – 2 No lift / negative | No statistically significant improvement in any KPI, or performance worsens when using the data. PVI effectively ≤ 0 for tested use cases. |
3 – 4 Small lift | Statistically detectable but small impact (e.g. <1% relative improvement in a relevant KPI, or effect size small vs noise). Business significance is marginal. |
5 – 6 Moderate lift | Clear, repeatable improvement (e.g. 1–5% relative lift in a key KPI like conversion, default rate, or handling time) in at least one scaled use case. |
7 – 8 High lift | Substantial, recurring improvement (>5–15% relative lift in a key KPI, or clear step‑change in process performance) across more than one use case or segment. |
9 – 10 Transformational | Very large, sustained effect (e.g. >15% relative lift in a core KPI, or enables entirely new, higher‑performing operating models) where data is the decisive differentiator. |
3. Robustness / generalisability (weight 20%)
Question: Does the performance benefit hold across time, segments, and implementations?
Score | Descriptor |
1 – 2 Very limited | Only a tiny portion of activity is affected (e.g. one small pilot channel, <1% of volume or revenue). Little pathway to scale. |
3 – 4 Local | Affects a modest slice (1–10% of relevant volume/revenue), such as a single product line or region, with limited roll‑out plans. |
5 – 6 Meaningful | Impacts 10–40% of relevant volume/revenue (e.g. major channel, multiple products). Scaling is underway but not yet organisation‑wide. |
7 – 8 Broad | Impacts 40–80% of relevant volume/revenue. Embedded into most applicable processes; remaining gaps are due to rollout or legacy constraints. |
9 – 10 Enterprise‑wide | Impacts >80% of relevant volume/revenue or the majority of core decisions. Performance uplift is organisation‑wide where the data is applicable. |
4. Scale of impact (weight 20%)
Question: Over how much of the business does this data‑driven performance gain apply?
Score | Descriptor |
1 – 2 Very limited | Only a tiny portion of activity is affected (e.g. one small pilot channel, <1% of volume or revenue). Little pathway to scale. |
3 – 4 Local | Affects a modest slice (1–10% of relevant volume/revenue), such as a single product line or region, with limited roll‑out plans. |
5 – 6 Meaningful | Impacts 10–40% of relevant volume/revenue (e.g. major channel, multiple products). Scaling is underway but not yet organisation‑wide. |
7 – 8 Broad | Impacts 40–80% of relevant volume/revenue. Embedded into most applicable processes; remaining gaps are due to rollout or legacy constraints. |
9 – 10 Enterprise‑wide | Impacts >80% of relevant volume/revenue or the majority of core decisions. Performance uplift is organisation‑wide where the data is applicable. |
Computing the composite PVI score
- For a given dataset/use case, assign 1–10 for each PVI dimension based on the descriptors.
- Compute a weighted average:
𝑃𝑉𝐼composite=0.25⋅𝐸+0.35⋅𝐾+0.20⋅𝑅+0.20⋅𝑆
Where:
- 𝐸 = Experimental evidence quality score
- 𝐾 = KPI lift magnitude score
- 𝑅 = Robustness / generalisability score
- 𝑆 = Scale of impact score
