Sampling and Extrapolation - "Size Does Matter" - DoctorsManagement Sampling and Extrapolation - "Size Does Matter" - DoctorsManagement

Sampling and Extrapolation

“Size Does Matter”

by Sean Weiss, Partner & VP of Compliance

Sampling is, if not the most critical part of conducting an audit, definitely in the top three. With that said, it is still the area most flawed when I review audit results for clients who are determining whether to appeal a carrier, contractor or commercial payer audit. Recently Jesse Overbay, JD; General Counsel for DoctorsMangement, LLC and I were engaged in an audit appeal in the Northeastern part of the U.S. The carrier agreed to a call with us and our client. During the call, we questioned the SIU Investigator III who performed the review. We asked some very basic questions like:

How did you draw your sample?
What program did you use to draw the sample?
How many times have you performed statistical sampling?
What is your educational background in statistics and advanced mathematics?
How did you determine the error rate that led to the extrapolation?

Again, just very basic questions for which the SIU Investigator III who was very honest with us said, “I have no experience in any of this, the software we used does not perform statistically valid samples even though our letter says it was, and I have no training in statistics.” Upon this admission, General Counsel of this payer who administers on behalf of Medicaid in that state told us they were going to throw out the audit and redo it the correct way. As you can imagine, our response was the heck you are. That would amount to Double Jeopardy in a sense and there is no way we would agree to that.

The payer ignored us and sent a demand to the client who at that point was so scared they complied with the request for 150 records, then a second request for another 300 records and just last week a third request for another 100+ records. We are now working with CMS in the region to intervene and stop this abusive, unethical and borderline harassing behavior. So, this simple story is all in preparation to provide you with the “Musts” according to The CMS. Because the section is so long, I focused on just the two specific portions below but the entire section can be reviewed here: Medicare Program Integrity Manual (PIM) It would actually begin in Section 8.4 Use of Statistical Sampling for Overpayment Estimation.

Section 8.4.2 of the PIM states the following:

“If a particular probability sample design is properly executed, i.e., defining the universe, the frame, the sampling units, using proper randomization, accurately measuring the variables of interest, and using the correct formulas for estimation, then assertions that the sample and its resulting estimates are “not statistically valid” cannot legitimately be made.”

While the above statement is specific to CMS and other government audits (TRICARE Operations Manual 6010.59-M April 1, 2015 Chapter 13, Addendum A; 13.A-5 Statistical Sampling for Overpayment Determinations and Identification of Probable False Claims), it is also generally accepted amongst the statistical community. Accordingly, these criteria must be met for an extrapolation to take place.

Because the accuracy and applicability of an extrapolation is so heavily dependent upon the statistical validity of the sample, the first and perhaps the most important question to be answered is whether the sample is, in fact, statistically valid. In many cases, a sample is assumed valid if there is a belief that the sample units were drawn randomly from the sampling frame or universe. The term ‘random’ is most often defined here as each audit unit of interest (beneficiary, claim, claim line, etc.) has an equal and non-zero chance of being selected. There is, however, an important difference between a sample being random and a sample being representative of the sampling frame from which it was drawn.


According to of the PIM, “The major steps in conducting statistical sampling are:

1) Selecting the provider or supplier;
2) Selecting the period to be reviewed;
3) Defining the universe, the sampling unit, and the sampling frame;
4) Designing the sampling plan and selecting the sample;
5) Reviewing each of the sampling units and determining if there was an overpayment or an underpayment; and, as applicable,
6) Estimating the overpayment. Where an overpayment has been determined to exist, follow applicable instructions for notification and collection of the overpayment.”

In simple terms, we want the sample to be representative of the universe or, more colloquially, to “look like” the universe with regard to its characteristics. These characteristics might include what was paid for the procedure, coding characteristics, such as the differences between the vast array of procedure and service codes included in the HCPCS II coding universe. It may also have to do with specific characteristics found amongst the beneficiaries, which often more accurately define the types of diagnoses and treatments represented within the medical records.

Some characteristics, such as charge, or paid amounts can be visualized by graphing data using histograms for example. Non-valued characteristics, such as those subject to rules, regulations or guidelines, are more difficult to visualize in this way. For value-based characteristics, distribution and variance are two effective ways to look at representativeness of a sample to the universe (or sampling frame).

The above seems simple enough no? However, it is among the biggest areas of error when it comes to payers and their contractors conducting audits. So, the next time you get a letter stating you have been audited and they used as “Statistically Valid” audit sample, you may want to check it because odds are there will be a flaw. Once the sample is fatally flawed then the extrapolation is fatally flawed and as they say… Game Over!

I have been blessed to work with, be trained by and most importantly be great friends with Mr. Frank Cohen, considered to be one of if not the most recognized statistician in health care. Frank created the Compliance Risk Analyzer which should be used by every single health care organization to draw samples, perform audits and track your compliance with government guidelines related to Auditing and Monitoring. If you have time, take a look at the site and all the free tools available to you there.

What to do next…

  1. If you need help with an audit appeal or regulatory compliance concern, contact us at (800) 635-4040 or via email at
  2. Read more about our: Total Compliance Solution

Why do thousands of providers trust DoctorsManagement to help improve their compliance programs and the health of their business?

Experienced compliance professionals. Our compliance services are structured by a chief compliance officer and supported by a team that includes physicians, attorneys and a team of experienced auditors. The team has many decades of combined experience helping protect the interests of physicians and the organizations they serve.

Quality of coders and auditors. Our US-based auditors receive ongoing training and support from our education division, NAMAS (National Alliance of Medical Auditing Specialists). All team members possess over 15 years of experience and hold both the Certified Professional Coder (CPC®) as well as the Certified Professional Medical Auditor (CPMA®) credentials.

Proprietary risk-assessment technology – our auditing team uses ComplianceRiskAnalyzer(CRA)®, a sophisticated analytics solution that assesses critical risk areas. It enables our auditors to precisely select encounters that pose the greatest risk of triggering an audit so that they can be reviewed and the risk can be mitigated.

Synergy – DoctorsManagement is a full-service healthcare consultancy firm. The many departments within our firm work together to help clients rise above the complexities faced by today’s healthcare professionals. As a result, you receive quality solutions from a team of individuals who are current on every aspect of the business of medicine.