It is one of the Supreme Court’s most consequential and controversial decisions, and no one should have been surprised that now-Justice Neil Gorsuch was asked about it during his confirmation hearings. In the 1965 case of Griswold v. Connecticut, the Court held unconstitutional a Connecticut statute that prohibited the use of contraceptives, affirming a “right of privacy” that appears nowhere in the Constitution’s text. Justice William O. Douglas’s majority opinion, which speaks of “penumbras, formed by emanations” from non-textual “guarantees that help give [the guarantees in the Bill of Rights] life and substance” has been ridiculed ever since it was issued. Conservative critics of the Court have long invoked “penumbras” and “emanations” to heap scorn upon the notion that the Constitution protects any rights that are not expressly listed in the Constitution’s text.

Although Senator Richard Blumenthal questioned Gorsuch about whether he thought the Court reached the “right result” in Griswold, Gorsuch declined to expressly say so, averring that it would be “an act of hubris” for him to state “whether [he] agree[d] or disagree[d] with any particular precedent.” Yet it is still worth thinking about what Blumenthal referred to as the “constitutional underpinning[s]” of Griswold, which have informed a line of cases stretching from Roe v. Wade to Lawrence v. Texas to Obergefell v. Hodges. Griswold is now associated with the doctrine of “substantive due process,” which holds that the Constitution’s guarantees of “due process of law” affirm the existence of inherent limits on legislative power.

To borrow a memorable phrase from Justice Potter Stewart, Douglas’s majority opinion was indeed an “uncommonly silly” piece of writing. As I have discussed elsewhere, Douglas was desperately seeking to protect individual liberty that he valued—namely, sexual liberty—without breathing life into a doctrine that could be used to protect liberty which he did not value—namely, economic liberty. But for all that, the answer to Blumenthal’s question to Gorsuch is a resounding “yes.”

In a vigorous and influential dissent from the Court’s holding in Griswold, Justice Hugo Black claimed that that holding could only be defended on the basis of “the same natural law due process philosophy found in Lochner v. New York,” a 1905 decision in which the Court struck down a state law that limited the number of hours bakers could work in a day or a week on the grounds that it deprived them of “liberty of contract as well as of person” and violated the Fourteenth Amendment’s Due Process of Law Clause. That “natural law due process philosophy” held that the individual rights protected by the Constitution could never be comprehensively listed, being as numerous as the peaceful activities that individuals can think to pursue, and that legislative power was therefore inherently limited—mere legislative will was insufficient to justify “meddlesome interferences with the rights of the individual.” On Black’s understanding of the relevant text and history, the Fourteenth Amendment was designed to protect the rights specifically listed in Bill of Rights against the states—and nothing more. States could deprive people of textually unenumerated rights for any reason or no reason at all.

Black was no conservative, and he joined a number of decisions that—whatever might be said on their behalf—cannot said to follow in a straightforward way from the Constitution’s text. Yet conservative originalists have criticized Griswold and subsequent decisions identifying unenumerated “fundamental” rights on similar grounds. Conservative originalist icon Robert Bork urged that burdens on unenumerated rights should not trigger any judicial review at all. Thus, he wrote that because the legislation at issue in Griswold did not “threaten any [textually] guaranteed freedom,” it should have been upheld. According to this view, both Griswold and Lochner—and Roe, Lawrence, and Obergefell—were all wrongly decided.

Are the critics of substantive due process right? It is uncontroversial that the concept of “due process of law” can be traced back to Chapter 39 of Magna Carta—the so-called “law of the land” clause. It is also uncontroversial that “due process of law” and “law of the land” were synonymous during the Founding era. Article VI of the Constitution identifies “this Constitution” as the “supreme Law of the Land.” Forget penumbras and emanations—due process of law guarantees, at the very least, a judicial process in which those who stand to be deprived of life, liberty, or property can contend that the law sought to be applied to them is unconstitutional. A law that is beyond the constitutional powers of the government to enact cannot be considered part of the “supreme Law of the Land.”

Consider again the facts of Griswold. Even if one rejects the proposition that “liberty” in the Fourteenth Amendment’s Due Process of Law Clause encompasses the right to use contraceptives, the statute at issue imposed fines on those who violated it, thus depriving them of property. Accordingly, Estelle Griswold was constitutionally entitled to a judicial proceeding in which she could contend that those statutes were outside the scope of the constitutional powers reserved to the state of Connecticut.

And so we arrive at the crucial question: Do states have the constitutional power to deprive people of what is rightfully theirs on the basis of nothing more or less than a widely-held conviction that a particular act is immoral? If the answer is “yes,” the consequences would be grim indeed. One need not be a moral relativist to recognize that the moral convictions of legislators are governed by no fixed standard and that judges are hardly in a position to deny that a given statute furthers the moral purposes of those who enacted it. The only recourse available to those burdened by such statutes would be to petition the legislature that imposed them in the first place.

Fortunately, the answer is “no.” Under our Constitution, all government power is limited, being delegated by “We the People” to our agents in government, for purposes set forth in the Constitution’s Preamble. Among those purposes is “to secure the Blessings of Liberty”—a direct reference to the natural rights-based political philosophy that the Constitution rests upon and is designed to implement. According to that political philosophy, the legitimate scope of government power is bounded by rights that precede government. Natural rights are referred to as well in the Ninth Amendment, which, as Randy Barnett has shown, is properly understood to affirm the existence of unenumerated natural rights and prohibit government actors, including judges, from “deny[ing] or disparag[ing]” those rights because they are unenumerated.

The Fourteenth Amendment took shape in an ideological context saturated by natural-rights principles—the Republicans who shaped it sought to remove any doubt that the “supreme Law of the Land” was consistent with those principles, and expressly empowered the federal government to prevent states from exercising arbitrary power that undercut those principles. The judicial implementation of the Fourteenth Amendment’s Due Process of Law Clause requires a good-faith effort to ascertain the principles that the clause was designed to effectuate—its original spirit.

Of course, not everything that our agents in government do must have an immediate and obvious connection to our Constitution’s overarching political philosophy. Further, certain governmental activities do not implicate individual rights at all and therefore do not trigger the need for a rights-based justification—requiring states to justify setting particular hours of operation for government buildings in terms of individual rights sounds silly because it is silly. Yet when the government does exercise coercive power over a person’s life, liberty, or property, due process of law guarantees that person an opportunity to challenge that coercion in a court of law for its consistency with the law from which all government actors draw their power—and judges who are duty-bound to give effect to that law must evaluate it accordingly. Such evaluation in turn, requires judicial engagement—impartial judicial inquiry into whether the government’s actions are calculated to achieve a constitutionally proper end, grounded in credible evidence in the record. In the context of state governments that wield powers that are not specifically enumerated in the Constitution but which are inherently limited, that means an end that is connected with protecting our natural rights or preserving such civil privileges as are accorded by the government in order to render the latter more secure.

Griswold, then, as well as much of our case law, gets the proper judicial inquiry backwards. Instead of seeking to determine whether the rights to use contraceptives, or have sex with another consenting adult,  or earn a living, or live with members of one’s own family, or refuse life-saving medical treatment, or try to save one’s own life by procuring investigational medicines are “fundamental” judges should ask, “Are the state’s actions calculated to achieve a constitutionally proper end, or are they arbitrary?” Instead of placing the burden on individuals to justify engaging in conduct that is not inherently injurious to others, judges should require the government to explain why it is regulating that conduct in a particular way or prohibiting it outright, and to justify that regulation or prohibition with credible evidence. The law of our land rests upon and is designed to implement the principle that coercion, not freedom, must be justified. Those whose duty it is to “say what the law is” must ensure that that justification is given.