U.S. Defense Secretary Jim Mattis on Tuesday flatly dismissed reports suggesting he may be leaving President Donald Trump’s administration in the coming months, saying flatly: “I wouldn’t take it seriously at all.” “How many times have we been through this, now, just since I’ve been here? It will die down soon, and the people who started the rumor will be allowed to write the next rumor, too,” Mattis told reporters at the Pentagon. “Just the way the town is,” he added. “Keep a sense of humor about it.”

The remarks were the most direct by Mattis to date about intensifying rumors about his future as Trump approaches the half-way mark of his four-year term amid speculation about changes to his cabinet after upcoming November mid-term elections. Mattis has become a focus in media stories in recent weeks about the Trump administration, particularly after the release of a book this month by Watergate reporter Bob Woodward that portrayed Mattis privately disparaging Trump to associates.

Mattis strongly denied making any such remarks. Trump on Sept. 5 said the defense chief would remain in his job, adding: “He’ll stay right there. We’re very happy with him. We’re having a lot of victories.” But a New York Times report on Sept. 15 said Trump had “soured on his defense secretary, weary of unfavorable comparisons to Mattis as the adult in the room.”

It also noted this year’s arrival in the White House of Mira Ricardel, who now has the powerful post of deputy national security adviser and who current and former officials tell Reuters is believed to dislike Mattis. Western officials privately extol Mattis, whose standing among NATO allies has risen as they become increasingly bewildered by Trump’s policies on trade and Iran and disoriented by his outreach to Russian President Vladimir Putin.

Inside-the-beltway journalism

Mattis has a dim view of journalism about inside-the-beltway politics in Washington, using the word “fiction” to describe Woodward’s book and similar reporting about closed-door conversations among U.S. national security leaders. Asked about the recent reports speculating about his departure, Mattis said: “It’s like most of those kinds of things in this town. Somebody cooks up a headline. They then call to a normally chatty class of people. They find a couple of other things to put in. They add the rumors… Next thing you know, you’ve got a story.”

Still, Mattis is not political by nature, and previously made no secret of the fact that he was not looking to become secretary of defense – or even return to Washington – when Trump was elected. The retired Marine general had stepped down from the military in 2013 and taken a job at Stanford University. He told his Senate confirmation hearing last year he was “enjoying a full life west of the Rockies” when the call came about the position.

After answering questions about his future, Mattis was asked whether he never considered life after the Pentagon. Mattis joked: “Of course I don’t think about leaving.” “I love it here,” he said with a smile. “I’m thinking about retiring right here. I’ll get a little place here down on the Potomac.”