Receiver Operator Characteristic Curve (ROC) of the PADDS

ROC Analysis

The ROC curve is a visual representation of the sensitivity (false negative rate) of a test versus its specificity (false positive rate) at each potential cut point for diagnosis. In other words, the ROC provides a comparison of all possible cut-points for diagnostic utility and therefore assists the test developer in selecting and verifying the most appropriate cut point to use to maximize both sensitivity and specificity in determining Positive and Negative Predictive Power (PPP and NPP). From this information, the researcher can then produce likelihood ratios to further improve the diagnostic utility of the test. Table 6.1 shows a graphical representation of the ROC analysis of 725 subjects for the three Target Tests of Executive Functioning.

Receiver Operator Characteristic Curve (ROC) of the PADDS

Table 6.1 ROC for the three Target Tests of Executive Functioning
Area Under the Curve
Test Result Variable(s) Area Std. Error(a) Asymptotic Sig.(b) Asymptotic 95% Confidence Interval
Lower Bound Upper Bound
TR .950 .009 .000 .932 .969
TS .951 .009 .000 .934 .969
TT .921 .013 .000 .896 .946

The test results variable(s): Target Recognition, Target Sequence, Target Tracking has at least one tie between the positive actual state group and the negative actual state group. Statistics may be biased.
a Under the nonparametric Assumption
b Null Hypothesis: true area = 0.5

Using age interval specific referenced cut points, the following decision rule was applied with all 725 subjects: In order to be considered as a classification hit, two of the three Target Tests of Executive Functioning performances must fall within the predicted direction for subjects to remain classified in their initially known group assignment (At least two clinical scores for ADHD classification and at least two non-clinical scores for classification as non-clinical).



Table 6.2 PADDS Overall Psychometric Performance
n=725 ADHD TYP Totals PPV=0.91
Test Positive 347 35 382 NPV=0.86
Test Negative 48 295 343 Sensitivity=0.88
Totals 330 299 725 Specificity=0.89

PADDS Links

Sample Reports(pdf)

Forms and Protocols (pdf)

PADDS Research

Determination of Sensitivity, Specificity, Positive and Negative Predictive Power

Applying the age specific cut scores and decision rule with 725 subjects demonstrated Sensitivity of .88 and Specificity of .89 and Positive Predictive Power of .91 and Negative Predictive Power of .86 indicating exceptional clinical diagnostic utility for screening ADHD and for excluding the over identification of non-ADHD subjects (NPV .86).



Table 6.3 PADDS cut scores, means, standard deviations, standard errors of measurement, and 95% confidence intervals as a function of sample and age groupings
Typical Clinical
AGE PADDS Subtests Cut score M SD SEM 95% CI Cut Score M SD SEM 95% CI
TR >94 103.12 34.23 12.81 78-128 ≤94 65.72 37.31 13.96 38.93
6 yrs TS >22 24.12 10.83 4.05 16-32 ≤22 16.54 9.46 3.54 10-23
TT >6 8.65 3.46 1.29 6-11 ≤6 4.98 3.11 1.16 3-7
TR >102 111.75 24.92 9.32 93-130 ≤102 75.68 33.92 12.69 51-100
7 yrs TS >26 30.29 5.2 1.95 26-34 ≤26 16.15 8.83 3.30 10-23
TT >6 10.13 3.18 1.18 8-12 ≤6 4.5 2.43 0.91 3-6
TR >111 118.41 27.36 10.24 98-138 ≤111 80.91 31.74 11.88 58-104
8 yrs TS >26 31.39 6.62 2.48 27-36 ≤26 18.21 9.14 3.42 11-25
TT >8 11.6 3.53 1.32 9-14 ≤8 5.82 3.66 1.37 3-9
TR >113 130.25 14.77 5.53 119-141 ≤113 83-72 30.93 11.57 61-106
9 yrs TS >28 32.23 6.11 2.29 28-37 ≤28 19.77 8.75 3.27 13-26
TT >8 11.91 3.97 1.49 9-14 ≤8 5.78 3.31 1.24 3-8
TR >125 134.32 12.03 4.50 125-143 ≤125 107.63 18.95 7.09 94-122
10 yrs TS >31 34 5.2 1.95 30-37 ≤31 26.79 6.36 2.38 22-31
TT >11 13.65 3.54 1.32 11-16 ≤11 9.13 4.50 1.68 6-12
TR >128 140.49 8.49 3.18 134-147 ≤128 98.85 34.68 12.98 73-124
11 yrs TS >32 34.87 6.52 2.44 30-40 ≤32 27.1 7.48 2.80 22-33
TT >12 14.8 3.47 1.30 12-17 ≤12 8.95 4.43 1.66 6-12
TR >128 137.77 9.77 3.66 131-145 ≤128 130.07 14.42 5.40 119-141
12 yrs TS >34 36.27 2.49 0.93 34-38 ≤34 29.79 4.08 1.53 27-33
TT >14 16.05 2.77 1.04 14-18 ≤14 10.64 4.80 1.80 7-14

Note. Within typical sample, age 6 n = 25, age 7 n = 32, age 8 n = 52, age 9 n = 64, age 10 n = 79, age 11 n = 53, age 12 n = 25.
Within clinical sample, age 6 n = 72, age 7 n = 80, age 8 n = 95, age 9 n = 67, age 10 n = 44, age 11 n = 22, age 12 n = 15.
SEM = Standard error of measurement.


Table 6.4. Sensitivity, specificity, positive predictive power, and negative predictive power by age grouping
AGE SENS SPEC PPP NPP
6 yrs .89 .84 .94 .72
7 yrs .90 .88 .95 .78
8 yrs .87 .87 .92 .79
9 yrs .91 .92 .92 91
10 yrs .86 .91 .84 .92
11 yrs .86 .92 .83 .94
12 yrs .80 .84 .75 .88

Development of Likelihood Ratios and Evidence-based Application in PADDS

Despite the outstanding classification potential demonstrated by the Target Tests of Executive Functioning with known groups, these metrics, when applied against a base rate of 4% (as with ADHD) will result in significantly lower predictive power than is implied from their ability to separate groups with 100% known assignment. Thus, each potential raw score from the Target Tests of Executive Functioning was analyzed to determine the exact percentile rank for both the ADHD and Typical groups that corresponded to that given raw score. This was done so as to determine the sensitivity and specificity of every possible score for each of the three Target Tests. These sensitivities and specificities were used to develop likelihood ratios from every potential score for all three subtests. These ratios could then be applied incrementally with other data, as judged clinically appropriate, to a Fagan nomogram. These incremental inputs develop a predictive index for or against diagnosis in a given case. This transparent process forces the clinician to evaluate the relative weight of all procedures and to consider the combined evidence accumulated for or against a diagnosis in conjunction with clinical judgment. This is the heart of an evidence-based approach and will constitute a highly standardized approach to ADHD assessment that could help clinicians reduce both over and under identification of ADHD by fine tuning their diagnostic approach over time.

Background Info for Using The Likelihood Ratios

Few, if any, diagnostic tests are accurate enough to "rule in" or "rule out" conditions effectively in all cases. The best approach is to look at test results as altering the probability of an existent condition (ADHD). To do this requires the estimation of a pre-test probability, (base rate), that will then be adjusted up or down by each of the additional measures or tests results.

This process is referred to as the application of Bayesian logic, which uses an adjustment factor called the likelihood ratio (LR), to convert a pre-test probability into a post-test probability.

The upward adjustment of the probability after a positive result is called the LR (+) and is a number >1, while the downward adjustment after a negative result is the LR (−) and is a fraction <1.

The LRs are used to assess how good a diagnostic test is and to help in selecting an appropriate diagnostic test or sequence of tests. They have advantages over sensitivity and specificity alone because they are less likely to change with the prevalence of the disorder, they can be calculated for several levels of the test result, most importantly, they can be used to combine the results of multiple diagnostic tests which then can be used to calculate a post-test probability for a target disorder.

The key feature of the likelihood ratio is that it incorporates both the sensitivity and specificity of a given measure, or multiple measures, into a more useful form for making a clinical decision. Making an evidence-based diagnosis, (or considering subsequent decisions for management or referral) depends on comparing this post-test probability with thresholds for further action based on factors such as severity of impairment and the risks of possible side effects, versus the risk of further delays or no treatment at all.

The factors used by the PADDS scoring paradigm were developed using this 2X2 Matrix: (Sometimes referred to as a "Gold Standard")
· Group a : # of subjects with ADHD, and a positive Test Score.
· Group b : # of subjects without ADHD, and a positive Test Score.
· Group c : # of subjects with ADHD, and a negative Test Score.
· Group d : # of subjects without ADHD, and a negative Test Score.

Table 6.5. Gold Standard 2X2 Matrix
Condition Present Condition Absent Totals
Target Test
Score Result		 (Result +) a b a+b
(Result -) c d c+d
a+c b+d a+b+c+d

Using the 2X2 matrix in Table 6.5 we calculate sensitivity and specificity using these formulas
Sensitivity is the proportion of patients with ADHD who have a positive test.
Sensitivity = a / (a + c)
Specificity is the proportion of patients without ADHD who have a negative test.
Specificity = d / (b + d)
Calculate the Ratios:
Likelihood ratio (LR+) = sensitivity/(1-specificity) = (a/(a+c))/(b/(b+d))
Likelihood ratio (LR-) = (1-sensitivity)/specificity = (c/(a+c))/(d/(b+d))

The reference information provided above was adapted from the following Web resources for EBA:
(http://www.childrensmercy.org/stats/category/DiagnosticTesting.asp), and (Centre for Evidence-based Medicine (nd). Likelihood Ratios. Oxford-Centre for Evidence-based Medicine, http://www.cebm.net/likelihood_ratios.asp