I have seen so much random behavior from puppet runs. It's basically a big fancy wrapper around a bunch of shell commands (much better than the raw shell commands) but subject to all the bizarre race conditions and so on. We had to wait 30 minutes to use a newly created VM so that puppet had run three times, and it was >0.99 likely to be good now. (If it wasn't, it was killed and we retried; 30 minutes was chosen to minimize the expected time; the puppet config was migrated from cfengine and was based on a lot of host-name based regular expressions and very dangerous to debug/refactor).
Puppet can be difficult to get right. Dependencies are _very_ hard to get right, despite the fact that Puppet is virtually designed around the idea of dependencies. I'm a fan of the concept, less a fan of the execution.
Unfortunately, the competition (Salt, Ansible, Chef) aren't really any better here.
These days, I run Kubernetes whenever possible, and keep the base OS light, which makes the configuration management surface extremely small.
After years of pain, I've come to appreciate what was once relayed to me. All configuration management software is broken. They are equally terrible, each in their own merry way. The only thing you get to do is to choose the one that sucks the least for your use-case, and two years down the line hope that you made the right choice.
Which is why I have come to believe that the very concept of host configuration management is broken. We should do it as little as possible, preferably NONE AT ALL. Sure, use something like Ansible to run the image creation steps, and provision the necessary first-boot scripts in place. Only leave the steps in that absolutely can not be done during image pre-bake.
Cycle your hosts without mercy, so that new ones are brought up from fresh pre-baked images, continuously.
And even for the few unavoidable snowflake hosts (eg. those that have to live outside the K8S cluster), follow the same strategy. Make them disposable, so that you can bring up a new one from their own pre-baked images on demand. Try to keep the delta between the snowflake base and your cattle base as small as possible.
Configuring live hosts should be considered an anti-pattern - if you find yourself doing it at all, take a step back and consider how to get rid of the need.
