It coincided with the Jewish Tisha B'av holiday, which typically sees an increase in Jewish visits to the holy site.
In a bid to ease tensions, police barred Jewish visits to the site on Sunday but Muslim worshippers still feared they would be allowed in and protested there. The clashes with police broke out afterwards.
The compound, which includes the Al-Aqsa mosque and the Dome of the Rock, is one of the most sensitive sites in the Israeli-Palestinian conflict.
It is the third-holiest site in Islam and the most sacred for Jews, who revere it as the location of the two biblical-era temples.
It is located in east Jerusalem, occupied by Israel in the 1967 Six-Day War and later annexed in a move never recognised by the international community.
Jews are allowed to visit but not pray there to avoid provoking tensions.
Police fired sound grenades as Palestinian protests intensified at the highly sensitive Al-Aqsa mosque compound, known to Jews as the Temple Mount.