Five Republican congressmen who backed John Boehner in his bid to retain his speakership warned him on Thursday against punishing the GOPers who voted against him in Tuesday’s leadership elections.

“I thought it was a good meeting, I thought we were well-received by John,” says Mick Mulvaney, a South Carolina Republican who joined an attempted coup against Boehner in 2013 but supported him this year. “I’ve been on the other side of this two years ago and he does not seem to me to be a man who is committed to retribution.”

At around 1 p.m. on Thursday, Mulvaney along with Florida’s Ron DeSantis, Ohio’s Jim Jordan, Idaho’s Raul Labrador, and Arizona’s Matt Salmon convened in the speaker’s office to make their case.

“Everybody recognizes that there will be consequences” for voting against the speaker, says Mulvaney, but “we wanted him to know that there was a large group of us that see a difference between consequences and retribution.” Mulvaney says it’s acceptable for Boehner to foreclose committee chairmanships or turn off the spigot of campaign cash to dissenters in the future but that booting them from current committees is a step too far.

The group also made clear there was a larger contingent willing to back them: those who share the sentiments of the five lawmakers who met with Boehner number at least eleven, one Republican lawmaker says. That’s significant because 25 Republicans voted Tuesday to oust Boehner, who received 215 votes of the 408 cast. Eleven additional votes against him would have sent vote to a second ballot, an embarrassing scenario the speaker worked hard to prevent.

Boehner on Tuesday moved swiftly to punish the lawmakers who tried to thwart him, removing Florida’s Daniel Webster and Richard Nugent from the influential Rules Committee. But he has signaled that the reprisals may be temporary, and his aides told Politico that they are having a “family conversation” about how to proceed. Boehner’s allies have in recent days been pushing him to exact retribution: Letting the rebels get off scot-free, they say, encourages squabbling and disunity in the caucus. Others have argued that exacting retribution creates permanent enemies and prevents the caucus from unifying.