We usually start lunging and putting a harness on around 2-2 1/2. We do a lot of ground work lunging, ground driving getting the horse comfortable with the harness etc. I'll start hooking them to a cart and ground driving as well. Depending on how they are doing I only get in the cart when they are ready mentally, training wise and physically. Hercules, my mini, was very easy to train and took everything in stride. He is also a very mature physically. I started getting in the cart with Hercules before he was 3 (he turned 3 end of February).
Long story short, it depends on the horse, but its never to early to do ground work.