The fund, called Team London, will give grants worth up to £10,000 to voluntary sector organisations that want to increase the number of volunteers they use.
The cash will also support uniformed youth clubs, including girl guide and scout groups, so that they can recruit more volunteers, and it will fund a new Mayor's Mentors programme that will train volunteers to mentor young people.
The three-year fund is made up of a £2m donation from the Reuben Foundation - a charitable foundation set up in 2002 by the property developers David and Simon Reuben - and £2m from the Greater London Authority.
A statement from the GLA said Johnson would seek additional funds from the private sector and charitable foundations. The body will start looking for partners to deliver the fund in the next month.
In a statement, Johnson said: "London has a rich history of volunteering and civic responsibility, and I am continually humbled by some of the amazing, generous work Londoners do to make it a better place for us all.
"Team London is about what we can achieve when we act together. I want to see the village put back into the city and for every precious hour that Londoners spend, volunteering achieves the greatest possible impact."