Incremental Validity and Evidence-based Approach of the PADDS
Incremental ValidityGenerally, the process of incremental validation of an assessment involves use of multiple sources of evidence in order to diagnose a disorder, and the use of those multiple sources of evidence must be justified by the fact that each new source of evidence provides additional, non-redundant information.
In their 2003 article, Hunsley and Meyer emphasized the importance of considering incremental validity when developing a new measure, "When new measures are developed or when established tests are revised, there are numerous ethical, clinical, and research factors to be considered (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999)… an important consideration associated with any new measure is its incremental validity over alternative measures available to assess the same construct." (pp. 448-449). Similarly, Johnston and Murray (2003) emphasized the importance of incremental validity in the assessment of children for the presence of psychological disorders. These researchers describe incremental validity as the use of methods that add to the assessment process in such a way as to improve outcomes.
Our development and use of the PADDS is consistent with the spirit and procedures described by Hunsley and Meyer (2003), as well as Johnston and Murray (2003), regarding incremental validity. Consistent with the ethical standards of the American Psychological Association, we have examined the incremental validity of the PADDS as it is used to diagnose attention-deficit/hyperactivity disorder (ADHD). Our presentation of the evidence we have gathered begins by directly addressing a series of issues inherent in the incremental validation of instruments used to diagnose children and adolescents that were outlined by Johnston and Murray (2003). These include: a) developmental and contextual issues, b) problems with defining criterion variables, c) determining base rates, and d) specificity issues involved in establishment of incremental validity, particularly in the case of a newly developed instrument.
Developmental and Contextual Issues
The recommendation of Johnston and Murray (2003) in regard to incremental validation of a newly designed measure, such as the PADDS, is that the incremental validity be evaluated in light of the developmental period of the particular child. This step is imperative to the proper incremental validation of the new instrument. In developing the PADDS, we carefully examined the instrument vis-à-vis the age of the children we tested and determined that at each older age range the children's' raw scores increased necessitating individual cut scores be developed for each age grouping between ages 6 to 12 years.
Defining Criterion Variables
The primary recommendation of Johnston and Murray (2003) in regard to choice of criterion variables is to select highly objective criterion. As they put it, "Demonstrating incremental validity will always be easier when the criterion is objective, widely agreed on, and not confounded with the information used in prediction." (p. 499).
PADDS Links
- Purchase PADDS.
- PADDS Upgrade
- PADDS FAQ
- PADDS Tutorials
- Sample Forms & Reports
- Published Works
- Testimonials.
- About PADDS.
- Informational Handout
- PADDS Brochure(pdf)
- Installation and Use pdf
- PADDS Quick Start Guide
Sample Reports(pdf)
- Cognitive Tests (TTEF)
- SNAP-IV Parent/Teacher
- Case Study Reports
- PADDS Scoring
- Bell Curve Description
- CADI (Diagnostic Interview)
Forms and Protocols (pdf)
PADDS Research
In our process of establishing the incremental validity of the PADDS, the most current information about the underlying cognitive and neuropsychological mechanisms was incorporated into our measure. In addition, we compared the performance of our measure against other established measures of ADHD, particularly those most frequently utilized in the assessment of ADHD.
Determining Base Rates
A particularly challenging task involved in the establishment of incremental validity is determining the appropriate base rate against which to make the comparison. Indeed, the results of a positive test are far less surprising if the client is drawn from a population with a higher incidence of the disorder. Therefore, it is imperative to incorporate base rate information into the process of establishing incremental validity. In our work with the PADDS, we have adopted the most current and conservative estimate of base rate incidence of ADHD as the default value for base rate. In addition, we have, as will be explained in more detail later, established a mechanism by which base rate data from an individual practice can be incorporated into the overall assessment process, thereby not only improving the incremental validity of the PADDS itself, but also contributing to the improvement of the incremental validity for all of the other sources of data that rely on accurate base rate information.
Specificity of Incremental Validity
Incremental validity is highly dependent upon the context in which the source data are collected and applied. Due to the wide variation in approaches to ADHD assessment currently in use we have adopted an evidence-based procedure, as described by Frazier and Youngstrom (2006), to utilize likelihood ratios within a nomogram to improve the appropriateness of decisions based on the multiple sources of data. Frazier and Youngstrom (2006) recommended that nomograms be utilized so that each data source can be allowed to contribute to (or detract from, as the case may be) the prediction of the diagnosis.
Evidence-based Assessment (EBA)
Evidence-based assessment in medical practice has been advocated by members of the medical community of the United States since at least the mid-1990s as supported by a series of articles in the Journal of the American Medical Association (JAMA; Hamilton, 2005). A variety of researchers have advocated the adoption of an evidence-based approach in child and adolescent psychiatry and psychology (Hamiltion, 2005; Youngstrom and Duax, 2005). More recently, an evidence-based approach to the diagnosis of Attention-Deficit/Hyperactivity Disorder has been advocated as well (Frazier and Youngstrom, 2006). Evidence-based assessment involves the practitioners' engagement in scrutinizing the evidence upon which their practices are conducted for its soundness, power of inference, and appropriateness in drawing clinical and diagnostic conclusions (Hamilton, 2005). "Residents are taught to develop an attitude of 'enlightened skepticism' toward the application of diagnostic, therapeutic, and prognostic technologies in their day-to-day management of patients." (p. 495. Hamilton,2005).
