Methods & conventions

What this tool does

Nine working pages: Premorbid Estimate, Score Tables, Change Analysis, Profile Analysis, Performance Validity, Score Graphs, Score Converter, Effect Sizes and Data. Every calculation runs locally in the browser. No patient data is transmitted off-device, and the app works with no network connection.

The auto-fill normative database holds published parameters for seven instrument families: D-KEFS (original and Advanced), WAIS-IV, WMS-IV, WISC-V, CVLT-3, CVLT-C and the RBANS, with the retest sample size N where the publisher reports one. N is required for the Crawford & Garthwaite method and may need entering by hand where it is unavailable. Clinicians should verify every imported parameter against the current manual, and against local service standards, before interpreting it.

Score conversion and classification

Conversions between standard (M 100, SD 15), T (50, 10), scaled (10, 3) and z scores assume an approximately normal reference distribution. Two descriptor schemes are offered, and the one in force is named in the note beneath every exported table: Wechsler bands follow the WAIS-IV/WMS-IV manual conventions, and AACN labels follow Guilmette et al. (2020). Confidence levels throughout are 90% (z = 1.645) and 95% (z = 1.960); intervals round the estimate and the margin separately, so the printed bounds stay symmetric about the printed value.

Confidence intervals on Score Tables

Confidence intervals and standard errors of measurement. The CI column is the obtained score ± z × SEM, where SEM = SD × √(1 − r), centred on the obtained score rather than on an estimated true score.

The standard deviation is the normative SD of the metric the score is reported in (15, 10, 3 or 1), because a coefficient computed on, or corrected to, the normative sample must be paired with that sample's variability. Where a measure's stored statistics are raw, its own standard deviation is used instead, that being the only one in the right units. Four publishers state that rule outright, and the arithmetic confirms it: this pairing reproduces every published standard error of measurement the app is able to check, exactly, at the precision each is printed to: all 300 cells of WAIS-IV Table 4.3, 242 of WISC-V Table 4.4, 240 of WMS-IV Table 3.3, 168 across the D-KEFS SEM tables, 126 of RBANS Update Table 3.7, and all 38 CVLT-3 measures in Tables 3.4 and 3.5.

The reliability is, by default, the retest coefficient held in the normative database (an alternate-form coefficient in the case of the CVLT-3, which publishes no same-form retest), corrected for the normative sample's variability where the publisher reports a corrected value. Retest is the default for two reasons: it keeps a single, stated basis across a table that may mix batteries, and it is the appropriate coefficient for the many timed measures in the database, since split-half and alpha are not valid reliability estimates for speeded tests. The WAIS-IV manual makes that second argument itself for Coding, Symbol Search and Cancellation, describing the split-half coefficient as "not a proper reliability estimate" for a Processing Speed subtest; the values used here for those three are the ones it publishes, in all 38 of the cells its Table 4.1 gives them.

That default is set aside for a measure only where its publisher both reports an internal-consistency coefficient and derives its own published intervals from it. Seven manuals meet that bar:

Which coefficient is right is a question for each manual rather than a policy of this tool, and the manuals genuinely disagree: the two D-KEFS manuals reach opposite conclusions about the same two test names. Each is followed as written, and the Data page names the basis actually in force for every measure in the database.

Age. Where a coefficient is tabulated by age, the interval uses the band for the patient's age, and the age used is named in the note beneath the table. Entering an age is optional. If none is entered, or the age falls outside a measure's normed range, the publisher's all-ages figure is used instead: the published average where a manual prints one, and otherwise the total-sample retest coefficient, which for the D-KEFS is that manual's own second regime rather than a substitute for a missing number. Both paths are therefore the publisher's own figures.

Reliable-change analysis is unaffected by any of the above and always uses the retest coefficient, which measures a different thing.

One consequence is worth bearing in mind when comparing output against a test manual. For the measures that remain on the retest default, where a manual derives its published intervals from internal-consistency reliability (almost always the higher of the two coefficients), the intervals shown here run wider than the manual's. They are therefore the more conservative, and answer the question how much would this score be expected to move on retesting rather than how precisely was it measured on the day. Not every publisher offers that comparison: the CVLT-3 manual declines to report internal-consistency reliability at all, on the grounds that item scores on a word-list task are not independent (recalling one word alters the probability of recalling the others, both within a trial and on later ones), and reports alternate-form coefficients in their place.

Measures with no normative-sample coefficient, and the reliability control

Every interval here multiplies a normative standard deviation by a reliability, and that is only a valid standard error of measurement when the two describe the same group. Most manuals supply a coefficient computed on, or corrected to, their normative sample. Three do not: the D-KEFS, D-KEFS Advanced and CVLT-C manuals report only the correlation observed in their own retest studies, a few dozen people each, and pair it with the normative standard deviation regardless.

The D-KEFS manual states that outright, fixing the standard deviation unit at 3 for all its scaled scores and deriving its test–retest standard errors of measurement "from the total sample of cases". Its Table 2.8 shows the arithmetic: the three Design Fluency all-ages values of 1.94, 1.97 and 2.47 are exactly 3 × √(1 − r) on the uncorrected coefficient. Those measures are therefore scored the way their own manuals score them, and the intervals shown reproduce the published ones. The Data page labels each such measure retest, uncorrected, so which rows rest on that footing can be read off rather than inferred.

The statistical objection to the pairing is nonetheless real, so Score Tables offers a reliability control with two settings. Published, the default, uses each manual's own coefficient and reproduces its printed interval. Corrected applies the standard range-restriction correction of Allen and Yen (1979), r_xx = 1 − (s²_retest ÷ s²_norm)(1 − r), to those measures alone, so that the coefficient describes the same population as the standard deviation it multiplies. A published coefficient is never overwritten in either setting.

The correction is not a guess: across the 267 database entries carrying both an observed and a publisher-corrected coefficient, it reproduces the publisher's own value to a median error of .003. But the resulting figures are not printed in the manuals concerned, which is why the default is Published and why the note beneath a corrected table says so. In practice the control moves 46 of the measures reachable from Score Tables, every one of them D-KEFS or D-KEFS Advanced: at the 95% level 9 intervals widen and 4 narrow, 33 are unchanged after rounding, and the largest single change is 2 scaled-score points. Reliable-change analysis is not affected by the control. Where a corrected reading is taken, the Data page follows it rather than continuing to show the published one.

Change analysis

Five methods, in ascending order of what they model. The Standard Deviation Index is descriptive only: SD Δ = (X₂ − X₁) ÷ SD, with no reliability correction. Simple Reliable Change (Jacobson & Truax, 1991) tests the observed change against measurement error. Practice Effect-Adjusted change (Iverson, 2001) subtracts the mean retest gain observed in the normative sample first. McSweeny Regression-Based change (McSweeny et al., 1993) predicts the retest score from baseline and standardises the residual against the standard error of estimate. Crawford & Garthwaite Regression-Based change (2007) does the same but with a standard error of prediction that accounts for the normative sample size and for the distance of the baseline score from the normative mean.

All p-values are two-tailed. The first four methods use the standard normal distribution; Crawford & Garthwaite uses the Student t distribution with N − 2 degrees of freedom, so a small normative sample raises the threshold: at N = 25 the 95% critical value is 2.069 against 1.960 for z.

These calculations use the retest coefficient paired with the standard deviation of the same retest sample, so that both terms describe one population. Where a publisher reports a coefficient corrected to the normative sample's variability, that value is offered as an option but is not the default, because it describes a differently distributed population from the standard deviation it would be multiplied by, and in the two regression methods the coefficient is a fitted slope, so substituting it changes the predicted score rather than only the interval. Reliability type varies by instrument and is stated in the note beneath each generated table: CVLT-3 coefficients are alternate-form, RBANS Form A coefficients are same-form retest.

Outcomes are reported as significance only ("Reliable change" or "No reliable change"), never as improvement or decline. The database holds many measures on which a higher score is the worse result (intrusions, perseverations, errors, false positives), and carries no score-direction flag, so reading a clinical direction off the sign of the statistic would assert the wrong conclusion for all of them. The signed statistic is displayed alongside, so the direction stays visible without the app interpreting it.

Profile analysis

A battery raises a question no single row answers: this patient has two scores below the 5th percentile, so how unusual is that? By definition 5% of the population falls below the 5th percentile on any one measure, but across several correlated measures a low score somewhere is common. Reading rows independently overcalls impairment: 13.7% of healthy adults show at least one abnormally low score across the four WAIS-IV Indices, and 25.0% across ten core subtests. The Profile Analysis page reports the count and the percentage of the healthy population showing that many or more (Crawford, Garthwaite & Gault, 2007).

Three questions are answered from one simulation: how many scores are abnormally low, how many pairwise differences between measures are abnormally large, and how many measures deviate abnormally from the patient’s own mean across the battery. The criterion is chosen by the clinician from the five the paper tabulates (below 1 SD, or below the 10th, 5th, 2nd or 1st percentile) and governs all three. A difference or deviation is abnormal when larger than the same percentage of the population shows, in either direction. Simulations are seeded, so a figure quoted in a report does not change when the page is reopened, and the sampling error of each percentage is stated beside it.

The only input is the correlation matrix over the measures administered, taken from the publisher’s normative sample: WAIS-IV Technical and Interpretive Manual (GB) Table 5.1, or WMS-IV Technical and Interpretive Manual (GB) Table 4.1 (Adult battery) or 4.2 (Older Adult battery). WAIS-IV and the WMS-IV Adult battery can also be profiled together, on WMS-IV Table 4.12, which correlates the two batteries in their co-normed sample (corrected to the WMS-IV normative sample’s variability); the Older Adult battery cannot, as that table pools both batteries under one Visual Memory Index. RBANS Update (Form A) index and subtest scores are profiled on RBANS Update Manual Table 4.1 (Randolph, 2012), and WISC-V scores on WISC-V Technical and Interpretive Manual Table 5.1. Scores are read from Score Tables rather than entered here, so the two cannot disagree. A profile may not hold both a measure and a part of itself (an index and its own subtest, two indices sharing subtests, or a subtest and its own process scores), so each instrument is profiled one level at a time. Where a publisher’s indices cross, as WMS-IV’s do (auditory and visual against immediate and delayed), the two readings are offered separately. Index scores give the more accurate estimate: the authors note that scaled scores are coarse, one point being a third of a standard deviation, so a subtest-level profile is the rougher reading and says so on screen and in the exported note.

Performance validity

Eight measures, each scored against a published cut-off or base rate. Two are embedded in the RBANS: the Effort Index (Silverberg et al., 2007; accuracy per Shura et al., 2018) and the Effort Scale (Novitski et al., 2012). Two are embedded in the WAIS: Reliable Digit Span (Greiffenstein et al., 1994; cut-offs per Schroeder et al., 2012) and the Digit Span indices (Iverson & Tulsky, 2003; Axelrod et al., 2006). One is embedded in the CVLT-3: the Forced Choice trial, read against the manual’s age-banded base rates (Delis et al., 2017), with CVLT-II cut-offs selectable (Schwartz et al., 2016; Erdodi, Abeare, et al., 2018). One is embedded in the D-KEFS: Trail Making, five age-corrected conditions each read against its own cut-off, failing when at least a chosen number fail (Erdodi, Hurtubise, et al., 2018). Two are stand-alone tests: the Rey 15-Item with recognition trial (Boone et al., 2002) and the TOMM (Tombaugh, 1996; cut-off accuracy per Martin et al., 2020). Every result is a comparison against the published figure only, reported with the published sensitivity and specificity where the source gives a pair; the app issues no verdict on the protocol, because no single indicator supports one.

Aggregation follows Larrabee (2014a): failing two or more independent validity indicators supports probable invalidity. Independence is the load-bearing word: measures derived from the same administration of the same instrument share error and cannot be counted twice, so the two RBANS indices count as one indicator between them, as do the two Digit Span measures. A base-rate index with no published accuracy (the Vocabulary − Digit Span difference, the longest spans, the CVLT-3 critical items) is reported as flagged but not counted as a failure. The Summary tab reports Larrabee's published classification accuracy at each failure count. Positive and negative predictive power appear on the TOMM tab, derived by Bayes' theorem from the meta-analytic sensitivity and specificity and a base rate the clinician selects (Martin et al., 2020, Tables 16–17), a selectable rate rather than a fixed one, because the base rate of invalid performance differs by setting and the choice belongs to the clinician, not the app.

Embedded indices are computed from subtests that also measure genuine ability, so each carries a caution naming the populations in which it over-flags (dementia and severe impairment chief among them), and the caution takes emphasis only once that measure has actually flagged. Cut-offs validated in one edition of an instrument are labelled with that edition, and the clinician should document which edition was administered.

Effect sizes

Every value on the Effect Sizes page is converted from Cohen’s d, and the page prints its formulas in its own Formulas view. Two conversions depend on the group sizes: with group data the correlation uses a = N² / (n₁n₂) in r = d / √(d² + a) (Borenstein et al., 2009) and Cohen’s f uses the group proportions, so unequal groups give a smaller r than equal ones at the same d; with no group sizes both assume equal groups. A t or Z statistic converts as d = 2t / √N, which assumes two independent groups of equal size. Hedges’ g needs the total N and is not printed without it. The size words (Cohen, 1988; Sawilowsky, 2009) are read from the printed value, so a printed 0.800 is large; each measure has its own cut-offs, so the words for d, r and f need not agree.

Premorbid estimation

Premorbid estimates combine ToPF-based and demographic equations with OPIE-4 prorated models, and produce predicted-versus-achieved discrepancy output with confidence intervals, base-rate lookups and APA-formatted export tables. Predicted-versus-achieved significance flagging uses a three-tier scheme at z = 1.645 (*), 1.960 (**) and 2.576 (***); this is separate from the 90%/95% confidence-interval selector.

OPIE-4 is provided for illustration only in a UK context and its output should not be quoted as a concrete premorbid estimate. The regression terms reproduce Holdnack et al. (2013), Table eA5.8, but the published equations also carry US education, ethnicity and region terms that are not applied here, which fixes every prediction at the US reference category (12th-grade high-school graduate, not African-American, not resident in the western US). Those terms are omitted rather than mapped because the education dummies encode how unusual a given attainment level is within the US population the model was fitted on, not years of schooling, and that does not transfer: the US reference category corresponds to A-levels if matched by years but to GCSE/O-level if matched by population position, and UK school-leaving age was raised to 16 only in 1972, so leaving school without qualifications was normative for older cohorts in a way it was not in the US sample. Expect estimates to run high for patients who left school early and low for graduates, by an amount this tool cannot quantify. For a UK demographic estimate, use the Crawford & Allan (1997) model.

Base rates. The ToPF predicted-difference base rates are calculated from a normal model, the proportion of a normal curve with SD equal to the model's standard error of estimate falling at or below that discrepancy. That is how the ToPF-UK manual (Wechsler, 2011) builds its own table, rather than tabulating observed standardisation-sample frequencies: a standardisation sample of about a thousand cases yields no observed frequency at every discrepancy point. On the printed standard errors this reproduces the manual's table to the 0.01% shown on 296 of its 302 cells; the other six (IMI and VWMI, each on a rounding boundary) differ by 0.01 points, the manual having worked from a more precise standard error than it prints. The OPIE-4 discrepancy base rates of ACS Table eA5.12, by contrast, are empirical, sitting on a count grid, and are used as published. The two published tables therefore answer the same question by different methods, and differ by roughly 10% relatively across the decisive −5 to −20 band on models of almost identical standard error of estimate: a discrepancy of −15 gives 3.78% on the ToPF table against about 4.3% on the OPIE-4 one. That difference is a property of the two sources, not an adjustment made here; neither table is modified.