From Luther Martin, we know that Article VI’s test ban ” was adopted by a great majority of the convention, and without much debate.” The North Carolina delegation opposed the test ban, and the delegations from Maryland and Connecticut were split; it was otherwise widely accepted.
As the Luther Martin selection indicates, the test ban actually became an argument employed by some Anti-Federalists against the proposed Constitution. Without some test, they contended, Jews, Turks, heathens, and, worse, Roman Catholics could hold offices of public trust.
The Federalists countered in three ways. Some argued (fairly incoherently, I think) that the Constitution would allow non-sectarian religious tests, where a candidate was asked to affirm belief in, say, the existence of God, and of an afterlife which included rewards and punishments. Others underscored the necessity of the prohibition to keep any single sect from being excluded by a coalition of the others. Still others, like Tench Coxe, made the more high-minded argument that “The people may employ any wise or good citizen in the execution of the various duties of the government.” (Although Coxe goes on to suggest that a demonstration of religious virtue would be necessary to secure the confidence of one’s fellow citizens.)
It’s worth mentioning that, during the debates over the Constitution, state religious tests were extremely common, and would remain so for decades (and even centuries) afterwards. And, although the test ban was controversial in the debates over ratification, it has been widely accepted ever since. In fact, it has never been litigated; at the federal level, the clause has been entirely self-executing. (The Court extended the prohibition to all public offices at every level in 1961, in Torcaso v. Watkins.)
Where does this leave us? I think the evidence cuts in both directions. You are correct that the arguments centered on the legal prohibition of certain believers from public office. The debates focused on the formal institutions of government–not, as Krauthammer would have it, on the underlying motivations of individual voters. But, in fairness to Krauthammer, there was a sense among some of the leading Federalists that religion should be a peripheral consideration in evaluating candidates for public office.
The best resource on this question is Gerard V. Bradley’s law review article, “The No Religion Test Clause and the Constitution of Religious Liberty: A Machine That Has Gone of Itself,” 37 Case Western Reserve Law Review (1986-87). Also excellent is this post from the Volokh Conspiracy.
Many thanks. I particularly liked this bit from that Volokh link. The writer is Paul Horwitz.