We never handle our foals here. It is a foal's built in instinct to try to 'escape' from being approached or 'trapped' by any dangerous situation (handling by scarey humans!), so we keep well out of their way. As I have a 'thing' about keeping my stables free from any droppings, we spend quite a lot of time quietly entering the stables to collect the dropping while the mares and foals are in. Mostly this is done on our hands and knees with a small collection bucket, which means that we are 'down' at foal level and not standing up looming over them! Some babies will hide behind their Mommas, some will want to investigate to see who we are, but we still ignore them all, keeping our movements slow and quiet so as not to cause any 'fear'.
Foals are naturally inquisitive and within a couple of weeks they will all approach us to say hello when out in the fields. Within a month they are all leaving their Mommas when we go out to visit and following us around the pasture, and if we stop and sit down we are immediately mobbed, climbed on, sat on and generally 'abused' as they all vie for scratches and attention! We pick up feet as part of this 'game time' and when necessary trim them too, in fact ours dont even see a headcollar until after they are weaned, which never seems to cause a problem as it is just 'one of the games these funny human folk seem to play'!!
Your little lad will be fine, just spend some time sitting in your small paddock with him and his Momma and he will soon come over to investigate and you can then quietly start giving 'scratches' - all babies love scratches!! I do realise that you have had to 'handle' him to get him rugged, but apart from when you have to do his rugs, I would back off a little and give him the chance to approach you in his own time. One word of warning, when 'holding' a foal make sure you have your arm around his chest and not half way up his neck - it is very easy to put pressure on a foal's windpipe and to cut off his breathing!
Receiving cuddles at a couple of weeks/just sitting with them/assembling for foot trimming at a couple of months.