Former England Saxons and Northampton Saints player Clarke, who was announced as Gavin Hogg’s successor in April, has left the club without taking charge of a game, due to personal reasons, with Smith brought in to replace him.
The 34-year-old, who made five appearances for England, and featured once for the British and Irish Lions, between 2003 and 2005, has taken over in the hot seat at the Haberden.
Smith made more than 180 appearances for the Leicester Tigers, and also had spells in Australia, with the Brisbane Broncos, and in France, with Montpellier, before a severe knee injury while playing for Harlequins forced him to retire.
Since retiring from playing Smith has had spells coaching at Esher, London Welsh and, most recently, at Barking in London One North.
Terry Sands, performance director at Bury, said: “Jon joined us in the close season but he unfortunately wasn’t with us long enough to leave too much of a footprint.
“Ollie is very experienced and wants to play the game in a way which we felt we should play, and I’m really looking forward to working with him.”