Mattis said the Kremlin was treating the 1987 Intermediate Nuclear Forces Treaty (INF) with "cavalier disregard" over a new Russian ground-launched missile system.
The United States has said for almost two years that Russia's 9M729 system breaches the INF treaty, which bans mid-range missiles that had been the subject of a mini-arms race in Europe between the Soviet Union and Washington in the 1980s.
"Russia must return to compliance with the INF treaty or the US will need to respond to its cavalier disregard for the treaty's specific limits," Mattis said after a meeting of NATO defence ministers in Brussels.
"Make no mistake -- the current situation with Russia in blatant violation of this treaty is untenable."
He gave no details of how the US might respond, but earlier this year Mattis told Congress that the Pentagon was working on new low-yield nuclear weapons in a bid to force Russia back into compliance with the INF.
The accord signed between US president Ronald Reagan and Soviet leader Mikhail Gorbachev ended an alarming arms build-up in Europe triggered by Moscow's deployment of SS-20 nuclear missiles targeting Western European capitals.
NATO chief Jens Stoltenberg said Russia was imperilling the agreement, which he has called a "cornerstone" of European security.
"We believe this treaty is in danger because of Russia’s actions. After years of denials, Russia recently acknowledged the existence of a new missile system, called 9M729," Stoltenberg said.
"This system is destabilising. It is a serious risk to our security."
Without credible explanations, Stoltenberg said, the most obvious answer was that Russia was indeed breaching the treaty, though Moscow has denied this.
The US ambassador on disarmament Robert Wood on Thursday suggested Washington may withdraw from the INF treaty if Russia does not end its violations.