I have always noticed this myself, and was going to start a thread on this same subject.
All the reasons submitted here are probably relevant, and in addition I've always suspected that Helen had something to do with it. Andy seemed a little more serious around women, and this became more noticable when she came along, and especially as their relationship grew. To be honest, I didn't really like her. She was too serious herself, at least at first, though in the later episodes she seemed to fit in better, even as Andy seemed to grow more apart.
I started a similar thread on the Gomer Pyle guide about Sgt. Carter, whose situation seemed to be the opposite of Andy's.