The above posts are a great start. A key to remember is to make things fun! Even in the business end of running the club, members must feel like this will all be worth it because they are going to have fun! So regular activities and fun gatherings are a must!
And yes, membership is critical at this stage, but so is having each member WANT to participate as there is a TON of work just to start a club. It will require an investment of time, and some money to get things set up in the safest (monetarily speaking) way.
Yes, this is boring and very easy to get bogged down in, but your members will benefit from knowing they are legal and protected the best that they can be for the future.
Bylaws... You will need these as KayKay stated. This governs how everything is run, how elections are done, how rules are passed, and what happens with the clubs money which even includes where it goes if your club decides to disband.
Check your state laws to see if you have any official state legislation that is required to have a club. For example, do you need to register your club with the state in a simular fashion to registering a business. I am not sure who you would need to contact to check this out, but I would call your Secretary of State as a starting place to find out if you have any laws pertaining to forming/registering a club. They may say no, but at least you will know that you are not breaking any laws.
Next is the treasurer and what is required in your state to open a bank account in the club's name. You do not want to use a personal account as that makes things too questionable. Here in my state, I just went to my local bank that I personally bank with and met with a person and asked what is required to open an account. In my case they have to have an EIN to open the account along with some other things. The EIN is an Employee Identification Number. Yes, I know, you have no employees, but the banks need this for accountability. SO what that ultimately means is that you are telling the government that you exist and that you then have to fill out tax forms and report your income. Yippy. When I found this out, who knew that the club would be seen as an entity by the government.
That leads me to the next topic (Incorporating) for a couple of reasons.
You may wish to incorporate to help protect your club officers and members on an individual basis. This way the club, as an entity, is responsible for accidents and damage to other people who are not participating in equine activites. (For example, spectators at a parade who may get injured due to no fault of their own.) Further, if you are incorporated, then you would be able to obtain equine liability insurance to cover your members and the officers of the club.
I will warn you, it is not cheap to cover the liabilty end of things. But, it is almost necessary if you plan on any outside activites (ie not on a club memeber's property). Only you and your membership can way the risks and costs to determine how far you must go. But you may find that you can get in on an umbrella with another club sponsoring you using their insurance. Talk with your state horse councel for any advice.
And along the topic of incorporating, you would definately want to seek incorporation as a 501 ( c )(7) entity. See this link:
http://www.unclefed.com/IRS-Forms/2001/HTML/p5570404.html. The reason for this is that you want your club to be a non profit club so that you don't have to pay taxes on any money collected! (ie dues, donations, raffles, ect) But again, their are rules and you will need accountability.
I am not sure what else you may be looking at, but from a legal standpoint, I think this is a good start. If you and your membership should decide that this is too much work, then you may decide to just meet as friends in private.
But if you advertise that you have a club, then you should look at taking steps to protect yourselves legally and financially. It will be well worth it in the end.
And a final little tip that could help you tons is this:
Search for other equine clubs in your area and see if you can get a copy of their by-laws and any other ideas to help you get started. You don't need it to be miniature horse related, and it may be better that it isn't so you don't directly compete with them, but you want a well established club to reference and look to for guidance if possible. When you find a club that is willing to help, I would let that club know that your club appreciates any help from them on getting started, and in return, you would be willing to help them at one of their club's events. This is a win-win situation as often horse clubs would love members from other horse clubs to help at shows and what ever so that the club members can participate in their own club's shows.
The barter system at it's best
I by no means have many of the answers, and if the above legal paperwork type of stuff overwhelms you all, then maybe you are just best to have no formal group. But on the other hand, if you make it through all the formalities, then your founding members will really appreciate what they have.
Again, the most important part is to HAVE FUN!