That is absolutely not a silly question. I have heard debates about this since I was a kid. I mean, production shops have to stay on a schedule when doing collision work, so they mud and go pretty quick. Now on the other hand.....my cousin is a pro grahpics/pinstripe guy and once visited Boyd Coddingnton's shop out in California. He noticed a fenced in area where there were many bondo cars just sitting there in the hot sun. When he asked about their status he was told that they were making sure that all the "shrinking" was done before they started working the fillers. Hmmmmmm......
Okay, so I'm just a hobbyist, not in a tearing hurry anyway.... I let my filler cure until cheesgrater state to shape, let it set overnight, come back the next day for a fine tune of my sanding, and finally I like for it to set for 5 to 7 days, one last sanding, then prime. Since adopting that schedule my fillers never move or show signs of shrinkage. We have even adopted a similar schedule when we work those fillers on wood in our restoration shop.