It is not your imagination. My husband has noticed that too.
In the first season, there was Duff and a few fellow cake makers, with Mary Alice covering the desk. Charm City Cakes has added more decorators in each of the subsequent seasons, with interns in the back washing pans and baking the cakes. Duff still works, just not as much as before. He runs the shop, is the front man and does things to make the show successful.