She said she could not stand with an "anarchistic party" that lacked a credible plan to govern and said she would sit in parliament as an independent.
The 42-year-old chemist stormed out of a press conference in which she had sat with her newly-elected colleagues and said she would like to have a "conservative new start".
"I've decided I won't be part of the AfD's group in the German parliament but will initially be an individual member of parliament in the lower house," Petry said as she left the party's news conference.
On Facebook, she criticised party members for their extreme statements.
In a jolt to chancellor Angela Merkel, the anti-immigrant AfD won 12.6 per cent of the votes in Sunday's election making the party the first far-right group to win seats in the Bundestag since the 1950s.
Petry, who served as co-leader of the AfD, had come into conflict with other senior figures in the party in the last few months, saying she understood why some voters were alarmed at some of their radical rhetoric.
She had distanced herself from other top AfD candidates before the election and was also criticised for not supporting Bjoern Hoecke, a senior party official she wanted to have expelled for denying that Adolf Hitler was evil and calling Berlin's Holocaust Memorial a "monument of shame".
Petry was once considered a radical for transforming the AfD from an obscure group opposed to euro zone financial bailouts into Germany's leading anti-immigration party.
Analysts say it is unlikely that Petry’s sudden departure will mean much for the party.
The AfD has long been riven by infighting. Commentators have predicted that its divisions could be amplified by its entry onto the national political stage.
Frauke Petry parted ways with her party, stating she could not stand with an 'anarchistic party' that lacked a credible plan to govern