Business Minister Kemi Badenoch said Conservative MPs speculating about Rishi Sunak's removal as prime minister should "put a stop to it".
There have been reports in recent days that some Conservative MPs want Leader of the Commons Penny Mordaunt to replace Mr. Sunak.
But Ms Badenoch told BBC Breakfast that this was only a "small minority of MPs".
This comes after a difficult week for the Prime Minister, with former Conservative Deputy Leader Lee Anderson defecting to Reform UK.
Mr. Sunak has also faced criticism for his handling of allegations of racist comments made by a prominent Conservative donor.
Prime Minister under pressure over new funding from donor accused of racism
The mood darkens among Conservative MPs as PM faces questions
"I'm sure if Penny were here she would distance herself from those comments," Ms. Badenoch said of reports that support was rallying around Ms. Mordaunt as a possible replacement for the Prime Minister.
"I have been saying for a long time that the small minority of MPs who think this is something that needs to be talked about need to stop."
He continued: "I don't think there's much point in these rumors... When over 350 MPs have different views, we need to make sure that one or two MPs don't dominate the news narrative."
At present, only two Conservative MPs have publicly called for Mr. Sunak to resign: Sir Simon Clarke and Dame Andrea Jenkyns.
Asked about calls from some Conservatives to resign, the Prime Minister insisted that his party was "united in its desire to secure a brighter future for our country".
"I'm not interested in Westminster politics, that doesn't matter. What matters is the future of our country," she added.
Those close to Ms. Mordaunt believe that responding publicly to rumors of a leadership conspiracy will give them more credit than they deserve.
Ms. Mordaunt, who is popular with the party's grassroots, has previously stood to become party leader in both races in 2022.
As leader of the House of Commons, his public profile rose when he played a key role in the King's Coronation, holding the ceremonial sword for more than an hour.
Former Defense Minister Ben Wallace said it was "too late" to replace Sunak as Prime Minister.
"There comes a moment in the election cycle when you put on your best suit, stand up and walk to the sound of the guns," the Conservative MP told Times Radio.
In an effort to shift the focus to the economy, Downing Street issued a statement on Sunday night in which Mr. Sunak promised that 2024 "will be the year Britain bounces back".
The Prime Minister said he hoped to see "further progress" on inflation when the final official figures are released on Wednesday.
But some Conservatives worry that the budget's cut to National Insurance has failed to boost the party's poll ratings.
