Blog · Compliance

Full population testing vs sampling: what a compliance review can and cannot tell you

Full population testing vs sampling in compliance: what a sample can show, what sampling risk is, and when reviewing every case is warranted.

One hundred case tiles, five of them highlighted as the sample and the rest greyed

A sample tells you about the sample. It tells you about the case in front of you only to the extent that the sample was chosen well and is large enough, and a compliance review that rests on a sample carries a risk that its conclusion would differ if every case had been checked. In full population testing vs sampling, the question that matters is not which method is better in general. It is which cases you need to be sure about, and why.

In one sentence: sampling risk is the risk that the conclusion drawn from a sample differs from the conclusion that would have been reached had the entire population been subjected to the same procedure (ISA 530, paragraph 5(c)).

Full population testing vs sampling: the short answer

  • A sample supports a conclusion about the population only when the cases were selected so that each one had a chance of being chosen and the size was enough to bring sampling risk to an acceptably low level.
  • A sample cannot tell you whether the case in front of you is sound. It can tell you how the population behaves, with a stated margin of error.
  • When the expected misstatement is high, the standard itself says that 100% examination, or a large sample, may be appropriate.
  • Reviewing every case removes sampling risk for the procedure you run. It does not remove the risk that the procedure itself is wrong.
  • A missed error has a price. In one public scheme, sanctions since 2015 total EUR 30 million in financial penalties and 6.3 TWhc (the scheme's unit of energy savings) of certificates cancelled (DGEC).

What a sample can and cannot show

A sample is a deliberate trade. You examine fewer than 100% of the items in a population and, in exchange, you accept a conclusion about the whole. ISA 530, the auditing standard on sampling, puts the objective in plain words: to provide a reasonable basis for drawing conclusions about the population from which the sample is selected.

That wording is worth reading twice. The conclusion is about the population. A clean sample supports the statement that the population is probably in order, within a stated tolerance. It does not certify any single case that was not drawn. If one case in your portfolio was never selected, nothing in the sample says it is sound.

The standard is written for auditors of financial statements. The reasoning transfers to any review where a few cases stand in for many: a claims quality check, a loan file review, a monthly control test.

Sampling risk and non-sampling risk

ISA 530 separates two ways a review can reach the wrong conclusion.

Two columns: sampling risk and non-sampling risk, with the ISA 530 definitions

Sampling risk is the risk that the conclusion based on a sample is different from the conclusion if the entire population were subjected to the same procedure. The standard describes two ways it goes wrong. The review concludes that controls are more effective than they are, or that no material misstatement exists when one does. Or it concludes the opposite and triggers extra work. The standard says the first kind is the one auditors worry about most, because it is more likely to lead to an inappropriate opinion.

Non-sampling risk is the risk of reaching an erroneous conclusion for any reason not related to sampling. The standard gives examples: using an inappropriate procedure, misinterpreting evidence, failing to recognise a misstatement or deviation that is in front of you.

The distinction matters for a simple reason. Making a sample bigger lowers sampling risk. It does nothing for non-sampling risk. A reviewer who checks the wrong thing, carefully, in every case, still reaches the wrong conclusion.

When a sample is the right tool

A sample is the right tool more often than the debate suggests. It is the right tool when:

  • the population is large and the cases are alike, so that what you learn from some tells you about the rest;
  • the question is about the rate or size of a problem, not about the status of each case;
  • the expected misstatement is low, so a moderate sample can bring sampling risk to an acceptable level;
  • a wrong conclusion would cost less than reviewing every case.

The standard also says where a sample stops being enough. It states that if the expected misstatement in the population is high, 100% examination or use of a large sample size may be appropriate (ISA 530, application paragraph A7). And when the deviations found in a sample share a common feature, such as a type of transaction, a location, a product line or a period, the standard allows the auditor to identify all items in the population that have that feature and extend the procedures to those items (paragraph A17).

In other words, the honest answer to "sample or everything" starts with what you expect to find, and a first sample is often the way you learn it.

A public example: how little a sample covers, and what it finds

France's energy savings certificate scheme gives a public example of sampling at national scale. The Cour des comptes, the French public auditor, reviewed it in July 2024.

  • On-site controls ordered by the administration represented only 0.9% of the volume of certificates delivered in 2022, and controls by mailed questionnaire 4.2%.
  • Yet the Cour reports that in 2022 one operation in three was not satisfactory after on-site control.

Both numbers matter together. A rate found on the controlled operations is a rate for those operations, and it says little about the share nobody controlled. What the figures show is how thin coverage can be even when the findings are serious, and why a review that relies on a small sample should state its coverage next to its conclusion.

The same report shows why selection matters. For insulation works, the report finds one operation in four technically non-compliant in random controls, against one in two in targeted controls (it notes that the targeted figure is for works begun in 2020). A random sample describes the population. A targeted sample describes the risky end of it. Both are useful, and they answer different questions.

Source: Cour des comptes, Les certificats d'économies d'énergie, July 2024, chapter on controls (pages 12 and 84 of the PDF). ISA 530 text: ISA (Ireland) 530, Audit Sampling, updated October 2022.

Cour des comptes report: Les certificats d'économies d'énergie, July 2024.

What a missed error costs

A sample that misses a problem does not remove it. The case stays in the population, and the conclusion "in order" stands until someone else finds it. This is the first kind of sampling risk the standard describes, the one it says auditors worry about most. What it costs depends on who finds it, and the French energy savings certificate scheme shows the range in public figures. These are examples from that scheme, not general rates. TWhc is the scheme's unit of energy savings.

  • Legal. After a filing, the national control unit can refuse certificates, cancel ones already granted, or sanction. The Cour des comptes describes graduated sanctions, from suspending a dossier until the applicant carries out a control and correction plan, to a temporary ban on obtaining certificates, to financial penalties. These sanctions are published in the Journal officiel (Cour des comptes, July 2024, page 83).
  • Financial. From early 2022 to the end of 2024, sanctions came to 3.2 TWhc of certificates cancelled, about EUR 24 million at exchange-market prices, plus EUR 12 million in financial sanctions. Since 2015 the totals are 6.3 TWhc cancelled and EUR 30 million in financial sanctions, plus four withdrawals of eligibility for six months to three years (DGEC, report to Parliament on the fifth period, page 29). The Cour des comptes counts a different scope: EUR 17.9 million of financial sanctions pronounced since 2015 against obligés and delegates, nearly EUR 14 million of which are unlikely to be recovered (page 85).
  • Operational. About 300 TWhc of whole-renovation certificate requests were suspended by the control unit pending the results of controls when the 2024 report was written (Assemblée nationale, report no. 633, page 39, quoting the IGF, CGE and IGEDD inspection report). And when a lot of operations exceeds the maximum share of unsatisfactory controls among those carried out by the inspection body, only the operations checked and found satisfactory, not verifiable, or corrected before filing can be filed (DGEC CEE information letter, July 2026, pages 18 and 19). Time lost to rework and waiting is real, but no public source puts a figure on it, so we do not.
  • Reputation. Where sanctions are published in the Journal officiel, the applicant's name is public. We found no source measuring the commercial effect, so we do not claim one.

For an insurer or a lender, the consequence sits with its own regulator, auditor or counterparties, and its size is its own. The mechanism is the same, and in our view an error found late costs more than one found before the conclusion was signed.

Sources for these examples: DGEC, report to Parliament on the fifth period; Assemblée nationale, report no. 633; DGEC, CEE information letter, July 2026.

What changes when every case is reviewed

When every case in the population goes through the same procedure, sampling risk for that procedure falls away. By the standard's own definition, sampling risk is the gap between a conclusion drawn from part of a population and the conclusion from all of it. If you reviewed all of it, there is no gap.

Three things change in practice, and each is a method point, not a promise.

  1. The conclusion becomes about each case. You no longer say the population is probably in order. You say which cases were found in order and which were not.
  2. The exceptions become the work. The effort moves from selecting and sizing a sample to deciding what happens to the cases that fail a check, and who decides. That is where human oversight matters.
  3. Non-sampling risk is now the whole risk. Checking everything with the wrong test is still wrong. What counts is the quality of the check and of the evidence it relies on, which is the point of evidence-based verification.

Reviewing everything does not make a review automatically better. It changes where the risk sits.

Five questions to test your own control design

Use these on any control that rests on a sample today, before deciding whether to change it.

Checklist: five questions to test your control design
  1. What decision does the sample support? A conclusion about the rate of a problem needs a different design from a conclusion about each case.
  2. How were the cases chosen? Random selection describes the population. Risk-based selection describes the risky end. Say which one you used.
  3. What share of the population was reviewed? State it next to the result. A finding on a small share is a finding about that share.
  4. What happens to the cases nobody looked at? If the answer is nothing, you hold a conclusion about a population in which part of the cases were never examined.
  5. Who reviews the exceptions, and on what evidence? A flagged case needs a reviewer, a rule for deciding, and the evidence the decision rests on.

If you cannot answer one of them in a sentence, that is the first place to look.

Where this shows up for insurers and lenders

Does your claims quality check, if you run one, rest on a sample of closed claims (see insurance claims)? Does a monthly or quarterly control test stand in for your whole lending book, or other financial services portfolio? The question in both is the same: which cases do you need to be sure about, and what do you do with the ones you did not check? Worked examples are in the case studies.

If your review rests on a sample today and the answers above are uncomfortable, the next step is to look at the evidence behind each decision. Ceertia, our evidence-based compliance verification, is where we start that conversation.

Frequently asked questions

What is the difference between full population testing and sampling?

Sampling examines part of a population and draws a conclusion about the whole. Full population testing applies the same procedure to every item. The standard defines sampling as applying procedures to less than 100% of items in a population so that all sampling units have a chance of selection (ISA 530, paragraph 5(a)).

What is sampling risk in a compliance review?

It is the risk that the conclusion based on a sample is different from the conclusion that would have been reached if the entire population were subjected to the same procedure (ISA 530, paragraph 5(c)).

Does reviewing every case remove all risk?

It removes sampling risk for the procedure you run. It does not remove non-sampling risk, which the standard defines as the risk of reaching an erroneous conclusion for any reason not related to sampling, for example an inappropriate procedure.

When is a sample not enough?

The standard says that if the expected misstatement is high, 100% examination or a large sample size may be appropriate (paragraph A7). It also allows extending procedures to all items that share a feature found in the deviations (paragraph A17).

Is a random sample better than a targeted one?

They answer different questions. A random sample describes the population as a whole. A targeted sample concentrates on the cases you consider riskiest. In the Cour des comptes example, random and targeted controls on insulation works produced different rates of non-compliance, one in four and one in two.

How much of the population does a typical control cover?

There is no general figure. The honest practice is to state the covered share next to the result. In the public example above, on-site controls covered 0.9% of the volume delivered in 2022.

All articles

See it run on your own cases.