Allow him his time to eat, graze, play, whatever, but sorry to say, he is playing you. IMHO.
They will not break, and contrary to popular belief, a firm slap with a crop against the chest with a sharp NO is not abuse. I will be honest, I had a huge issue pittying my gypsy vanner mare due to her impaired vision (eye injury, lost left eye). I loved on her, and pampered her, got up and personal braiding her long locks, hard to resist on a gypsy mare. Anyhow, she got very up close and personal,and obnoxious when it came to being pushy, whether it was not wanting to stand still or pulling me into the grass to eat. She went out for training and the trainer had no part of it, showed me how to correct it and she has been great ever since, I just need to remember to do my fair share which means not tolerating it at all. That means no hand grazing on a lead in everyday normal life.
The two things that helped most getting her attention be it by stomping towards her while commanding her to back back back, sometimes using a whip on the ground to get her attention, sometimes just by letting the lead smack towards her chest/knees!sometimes a whack on ger chest with the handle of the whip. The other thing that worked, staying at the center point and making her do circles around me by going towards her rump with the lead or whip, sometimes bopping her on the rump with the rubber handle on the whip. Remember, you are the pivoting center, he is to circle around you, not make you chase him or gently walk him in circles. If he slows down or tries to stop, work at focusing at the hip and making him move it out and away from you, you may need to bop him with the lunge whip. Remember, getting his attention like this is not cruel, it is showing him you are gaining your rightful position as the alpha horse. Take a look at horses establishing rank, they kick, they bite, they rear, trust me, a smack to gain attention is nothing compared to what they do to one another.
While at training, my girl did not get to go out and run and play. When she came out, she worked, and worked hard for an hour or two,cooled down, was hosed down, and allowed to relax in her stall. If she was eating hay, she had to leave it to workout.
All big horse training correlates the same to the little guys. I think we as owners, have a bigger issue disciplining the little ones, we see them as fragile, or easily intimidated, or just so darn sweet and fluffy we think we need to get down and talk to them like our dogs or make baby talk to them (I am guilty of this) truth is, their brain is wired the same way as a full sized horse, all we are most likely doing is creating or adding to their behavioral issues. Nothing wrong with loving on them, but when that lead is on, it is time to pay attention.
Wanted to add, once I got past pittying my girl, she has turned out to be one of the best darn riding horses I have ever owned. She was trained six months after she lost her eye, she coped exceptionally well with the loss and was ready to have a job.