The sixth-seeded Williams was beaten 6-1, 6-7 (2/7), 6-3 by 10th seed Keys, who will now face Anastasia Pavlyuchenkova in the quarter-finals
Venus Williams tumbled out in the third round of the Rogers Cup in Montreal on Thursday, suffering an upset loss to Madison Keys in her final tournament before the Olympics.
The sixth-seeded Williams was beaten 6-1, 6-7 (2/7), 6-3 by 10th seed Keys, who will now face Anastasia Pavlyuchenkova in the quarter-finals.
Williams, 36, looked well below her best as Keys raced away with the first set before forging a 3-1 lead in the second.
However, Williams, who reached the final of last week's tournament in Stanford, rallied to level with victory in the second set tiebreak.
Keys raised her game in the decisive third set however and quickly broke Williams to gain the advantage before converting match point with an ace to seal the win.
Pavlyuchenkova earlier scored the biggest upset of the day with a three-set victory over 2014 champion and fourth seed Agnieszka Radwanska of Poland, winning 6-4, 6-7 (4/7), 6-1.
The win comes hard on the heels of her impressive showing at Wimbledon where she reached the quarter-finals for the first time.
"I'm really happy and I knew it'd be a tough match. Every time I play against Agnieszka it's 7-6 in the third, or something," the in-form Russian said.
"She's a really tough opponent, and so I'm really pleased with the win."
Elsewhere Thursday, fifth seed Simona Halep breezed into the last eight, needing just 68 minutes to claim a 6-3, 6-3 win over 14th-ranked Karolina Pliskova of the Czech Republic.
"I'm happy that I won today. It's nice to be in the quarterfinals again, second year in a row," the 24-year-old Romanian said.
Halep will face ninth-ranked Svetlana Kuznetsova of Russia in the next round. Kuznetsova defeated 12th seed Petra Kvitova of the Czech Republic 7-6 (7/2) 6-3.
In another Russian victory on Thursday, Daria Kasatkina knocked out seventh-ranked Italian Roberta Vinci, 7-5, 6-3.