Innumeracy at the Department of Education and the Congressional Committees Overseeing It
On July 21, 2017, preparatory to a July 24 Federalist Society teleforum titled “Are Existing Civil Rights Policies Based on a Statistical Understanding That Is the Opposite of Reality?,” I posted an item here titled “The Government’s Uncertain Path to Numeracy.” The item discussed my July 17, 2017 letter to the heads of the Departments of Education, Health and Human Services, and Justice. Among other things, the letter urged the agencies to explain to school administrators, that, contrary to what the agencies have previously led school administrators and the public to believe, relaxing public school discipline standards will tend to increase, not decrease, (a) relative (percentage) racial and other demographic differences in discipline rates and (b) the proportions racial minorities and other more susceptible groups make up of disciplined students.
[Because of the length of this post, a PDF version is available here.]
That pattern is the simple consequence of the fact that groups that are disproportionately represented among persons who are susceptible to some outcome are generally even more disproportionately represented among persons who are very susceptible to the outcome. I illustrated the matter, both in the letter and in materials used in the teleforum, with data from a Department of Education report showing that giving students reprimands instead of their first suspension would increase the proportions more susceptible groups made up of persons with one or more suspensions.
School discipline was one a number of areas discussed in the teleforum where the government’s understanding of the effects of generally reducing some adverse outcome on standard measures of racial and other demographic differences is not simply incorrect, but is the opposite of reality. The question left unanswered in the teleforum was whether any branch of the government is ever going to figure this out.
On July 26, 2017, two days after the teleforum, 15 Senators and 45 Members of the House of Representatives, including the ranking members of the Senate Committee on Health, Education, Labor, and Pensions and the House Committee on Education and the Workforce, sent a letter to Secretary of Education Betsy DeVos about racial disparities in public school discipline. The letter, however, was not aimed at correcting the Department of Education’s misunderstanding of the effects of generally reducing discipline rates on measures of racial and other disparities that the agency commonly employs. Rather, the letter was premised on that very misunderstanding. The opening paragraphs read (emphasis added):
We are writing to request information about how the Department of Education ("Department") will work with states to reduce exclusionary and aversive discipline in public schools around the country. We are pleased to see the Department open investigations in districts such as Richmond, VA; however, we believe these investigations are limited in scope and do not adequately address the systemic issue of discipline disparities in our nation's schools.
In the 2013-14 school year, approximately 2.8 million students received one or more out of school suspensions from public schools according to the biennial Civil Rights Data Collection (CRDC). The CRDC also showed that black students and students with disabilities were more likely to be subjected to exclusionary discipline measures than their same age peers. Under the previous administration, a growing reversal of the "zero-tolerance" or "no excuses" disciplinary approach helped decrease national rates of exclusionary discipline practices that included out-of-school suspension and expulsions. Despite these national trends, many public schools continue to suspend and expel minority students and students with disabilities at alarmingly disproportionate rates.
In fact, rather than simply continuing, disproportionality in discipline rates has been increasing, and that tends to occur “because of,” not “despite,” general reductions in discipline rates.
While I did not check the party affiliation of each signer of the letter to Secretary DeVos, I assume that each was a Democrat. But if there is one area where Congress is bipartisan, it is innumeracy, at least regarding analyses of demographic differences. It is doubtful that any member of Congress regardless of party is aware that generally reducing adverse outcomes tends to increase, rather than decrease, the principal measures of racial disparity employed by agencies enforcing a variety of civil rights laws.
The reason that is possible for such a prodigious failure of understanding of fundamental statistical patterns to persist in Congress, as well as within the executive and judicial branches, is that the failure is just as pervasive in the social and medical science research communities as it is in the government. See my “Race and Mortality Revisited,” Society (July/Aug. 2014), and my comments of November 14, and November 28, 2016, for the Commission on Evidence-Based Policymaking (CEBP) regarding the way that the aforementioned and related failures of understanding undermine essentially all analyses of demographic differences in adverse or favorable outcomes.
The last item discusses two illustrative issues fairly succinctly. Pages 1-3 address the way that studies of racial disparities in cancer outcomes commonly refer to relative differences in survival and relative differences in mortality interchangeably, often stating that they are measuring the former when in fact they are measuring the latter. Invariably, such studies do so without understanding that improvements in cancer care, while tending to reduce relative differences in surviving cancer, tend to increase relative differences in failing to survive cancer, or that more survivable cancers tend to show smaller relative differences in survival, but larger relative differences in mortality, than less survivable cancers. If you ever read that a study has found that racial differences in cancer survival are greater among a younger group than an older group, you can be virtually certain that the study in fact examined relative differences in mortality rather than survival. You can also be virtually certain that the study actually shows smaller relative differences in survival among the younger group than the older group.
Pages 3-4 of the item discuss a $1 million grant from the Department of Education to the Technical Assistance Center on Positive Behavioral Interventions and Supports (PBIS) that, like the recent letter from members of Congress to Secretary DeVos, is premised on the mistaken belief that programs like PBIS will tend to reduce, rather than increase, the measures of discipline disparities that the agency has commonly employed. The grant is focused on preschool discipline, which has been an area of special concern because relative racial differences in suspensions are higher in preschool than they are in K-12. No one exhibiting that concern has recognized the connection between the size of the relative racial difference in preschool suspensions and the fact that suspensions are fairly rare in preschool (with some districts’ having no preschool suspensions at all). See Table 8 of "Race and Mortality Revisited." Nor does anyone understand that PBIS and like approaches to discipline are likely to make those differences even larger.
That particular $1 million grant, however, represents a minor expenditure compared with the funds the Department of Education devotes to the study of group differences in educational outcomes. Such studies so far have yielded little of value because neither the agency nor its contractors and grantees know how to analyze such differences.
In addition to the illustrations mentioned above, the July 17 letter, using the same table I used here in a February 8, 2017 post titled “Compliance Nightmare Looms for Baltimore Police Department,” showed how lowering a test cutoff, while tending to reduce relative differences in pass rates, tends to increase relative differences in failure rates. I have occasionally noted that the situation is not one where federal agencies enforcing civil rights laws understand this pattern with respect to test score data but believe that, for some reason, the patterns will not be reflected in other contexts. Rather, despite decades of dealing with issues about demographic differences in test outcomes, the Department of Education has yet to show an understanding of these patterns even in the testing context.
The failure to understand the patterns in the testing context undermines a range of the agency’s activities. Substantial Department of Education and state funds are devoted to the monitoring of relative racial differences in assignment to special education programs. But neither the Department and the experts assisting it, nor the states endeavoring to follow Department guidelines, have yet shown an understanding that the lower are the test score cutoffs used for such assignment (or the greater the circumspection otherwise exercised in such assignment), the greater will tend to be relative differences between rates at which different groups are assigned to special education programs.
Nor has the agency or the experts assisting it yet shown any recognition that general improvements in education, while tending to reduce relative differences in rates of achieving subject proficiency, promotion from grade to grade, and graduation, will tend to increase relative differences in rates of failing to achieve proficiency, retention in grade, and failure to graduate.
Demographic differences in subject proficiency are often analyzed in terms of absolute (percentage point) differences between rates of different groups. The absolute difference is the same whether one examines the favorable outcome or the corresponding adverse outcome. But this measure, too, tends to change solely because the frequency of an outcome changes, as explained in "Race and Mortality Revisited" and the November 14, 2016 CEBP comments. Roughly, as an outcome goes from being rare to being fairly common, absolute differences tend to increase; as the outcome goes from being fairly common to being even more common, absolute differences tend to decrease. See Figure 2 (at 31) of the CEBP comments for an illustration of the effects on the absolute difference between pass (or fail) rates of lowering a test cutoff from a point where almost every fails to a point where almost everyone passes.
Thus, for example, general improvements in education tend to increase absolute differences between rates of advanced proficiency (where rates are commonly very low), while decreasing absolute differences of basic proficiency (where rates are often fairly high). But here, too, neither the Department of Education nor the expert entities assisting it understand these issues at all, much less understand them sufficiently to make an informed appraisal as to whether any observed pattern of changes in the absolute difference is other than the consequence of a general change in the frequency of the outcome.
For extensive examples of the way failure to understand the ways measures tend to be affected by the frequency has led to misleading research and misguided activities regarding matters within the purview of the Department of Education, see the subpages to the Discipline Disparities and Educational Disparities pages of jpscanlan.com and my letters to Pyramid Equity Project (Nov. 28, 2016), Boston Lawyers’ Committee (Nov. 12, 2015), Univ. of Oregon Inst. on Violence and Destructive Behavior (July 3, 2016), New York City Center for Innovation through Data Intelligence(June 6, 2016), Texas Appleseed (Apr. 7, 2015, Education Law Center (Aug. 14, 2014), IDEA Data Center (Aug. 11, 2014), Annie E. Casey Foundation (May 13, 2014), and Education Trust (April 30, 2014).
For an example of the nightmare potentially facing the Richmond, Virginia schools as a result of the recently-initiated Department of Education investigation mentioned in the July 26 letter from members of Congress to Secretary DeVos, in conjunction with the agency’s failure to understand that policy changes it commonly requires tend to increase the measures it monitors, see my letter to Oklahoma City School District(Sept. 20, 2016).
The congressional committees overseeing the Department of Education have no better an understanding of these issues than the agency itself. But that holds as well for the committees overseeing every agency that has functions involving analyses of demographic differences. It also holds for every piece of legislation so far passed imposing some burden on a covered entity pertaining to demographic differences. See page 342 of "Race and Mortality Revisited” regarding Individuals with Disabilities Education Improvement Act and my May 6, 2016 post here titled “Is the Disparate Impact Doctrine Unconstitutionally Vague?.”
A second area of congressional bipartisanship involves the belief in the importance of the use of data to promote evidence-based policy. Thus, both parties supported the legislation creating the Commission on Evidence-Based Policymaking. And the Commission is comprised of members appointed by the leadership of both parties. The Commission’s report to the legislative and executive branches is due very shortly.
Were the Commission to follow the recommendations at pages 45-46 in my November 14, 2016 comments, the government’s path to numeracy would suddenly be a much less uncertain one. But my hopes are not high that the Commission will even address the matters where current policies are based on an understanding that is the precise opposite of reality.
A better hope rests in the Department of Education itself. For, while the agency may not have a single employee who understands the issues discussed above, it has hundreds of employees who are capable of understanding the issues if they examine them carefully enough. Possibly the July 26, 2017 letter from members of Congress to Secretary DeVos will assist in prompting such an examination.