The Brazilian authorities have stopped US swimmers Gunnar Bentz and Jack Conger from boarding a flight at Rio de Janeiro airport, the US Olympic Committee has confirmed.
The two men were taken off a plane, bound for the US, for questioning late on Wednesday and released hours later.
Two teammates - Ryan Lochte and James Feigen - were barred from leaving the country, but Lochte had already left.
Police have queried their accounts of a robbery they reported in Rio on Sunday.
The four swimmers said they had been robbed at gunpoint in a taxi in Rio.
But police say their account of when they arrived at the Olympic village does not square with CCTV recordings.
A statement from the US Olympic team said that Bentz and Conger "were released by local authorities with the understanding that they would continue their discussions about the incident on Thursday".
"James Feigen is also communicating with local authorities and intends to make further statements regarding the incident on Thursday as well," said TeamUSA.
Lochte's lawyer told the BBC he got back to the US two days ago.
"He arrived back before the judge issued anything," said Jeffrey Ostrow.
"He was never asked to remain for further investigation or for any other purpose after he met with Brazilian authorities after he gave a statement."
Lochte is one of the most successful swimmers in history, with 12 Olympic medals, and he once had his own reality television show in the US.
In Rio, he swam in two events, winning gold in the 4x200m freestyle relay.
'Inconsistencies'
Feigen, who told the San Antonio Express he was still in Brazil, won gold in the 4x100m freestyle relay.
Bentz competed in the 4x200m preliminaries, but not the final. He still received a gold medal after the US team's win.
Accounts of what happened to the swimmers have been confusing from the beginning.
Lochte gave an initial account of the events to US TV network NBC, but International Olympic Committee spokesman Mark Adams said reports of the robbery were "absolutely not true".
The US Olympic Committee (USOC) later confirmed the swimmers' accounts.
Lochte himself told police that he and team mates Feigen, Bentz and Conger had all attended a party in the French Olympic team's hospitality house on Saturday evening.
He said that during their return drive to the Olympic village, their cab was stopped by men who pulled a gun and robbed him of his money and personal belongings.
But police investigating the case say they have found no evidence of the robbery. They also point to "inconsistencies" in the swimmers' accounts.
Lochte and Feigen told police they and their fellow swimmers had returned to the Olympic village at 04:00 local time (08:00 GMT).
But CCTV recordings appear to show the swimmers returning to the Olympic village at 07:00 local time.
Police say they have not been able to track down the driver who the swimmers said drove them back to the village.
They also said that Feigen and Lochte had given different accounts of how many men robbed them.
Investigators have requested a search warrant for the rooms the swimmers stayed in with a view to examining Feigen's mobile phone.
