BBC correspondents say protesters are hugging soldiers in the crowd and cheering for them.
One man told the BBC the day was “about a new beginning”.
The rally is supported by the army and members of the ruling Zanu-PF party.
War veterans – who until last year were loyal to the president – are also saying Mr Mugabe should quit.
A BBC correspondent in Harare says this is a watershed moment and there can be no return to power for Mr Mugabe.
A man arriving at the rally said: “As Zimbabweans, we are saying to our army: Thank you very much for the peaceful intervention.
“And it’s time for the masses of Zimbabwe to say: Mugabe must go and must go, like, yesterday. We can’t wait to see his back. For us, it’s about a new beginning.
“For us, it’s about the end of authoritarian rule and we are going to take our Zimbabwe back.”