I started out at www.cheesemaking.com but switched over to www.dairyconnection.com and have stayed with them. The first place will sell you small amounts of things, which is less intimidating when you are starting out. I typically make a 4 gallon batch of some cheese every week or so once I start, so the second link provide much cheaper supplies. And useful help over the phone as to which cultures to use depending on which book I was getting the recipe from. The first link is geared towards the owner's book.
I never used a kit, so I can't tell you. Most of New England Cheesemaking Supplies' stuff is geared towards purchased cow's milk it seems.
I dove right in and started making big batches. My favorite book is the 200 cheese recipes book....what is the title??? And the recipe I use the most is a Greek Feta recipe I got online somewhere. I prefer raw cheese and could make a lot of cheese with a small bottle of veal rennet, a mesophillic culture, salt, and that's about it. I made my molds and press and used cotton muslin for my first year but think purchased cheesecloth is worth it. Oh, and if you think you'll delve into waxed cheeses, some cheese wax is in order. I started out with the $6 per pound stuff and ended up getting 10 lbs for less than half that per lb price at Field and Forest (mushroom company.)
Hoeggers goat supply (google them for URL) and www.caprinesupply.com have cheesemaking stuff too, albeit a more limited range than New England Cheesemaking does (but in some cases priced not so much for the yuppie crowd LOL). Especially if you are wanting to order something else goaty from them ANYhow, that can be another option to look into.
I got my original stuff from New England Cheesemaking but now have found a Canadian outfit "Glengarry Cheesemaking" or something like that, that is simpler to deal with (being as I'm in Canada) - I think they also ship to the US and they do carry some slightly more obscure things.
Note that there are some cheesemaking things you can perfectly well make yourself -- some kinds of cheese molds and followers, and for sure a cheese press (which is otherwise tres pricey!)
Another range of products can be found on ebay. Try to find
Jongia cheese accessories
We supply hobby cheese makers world wide. Feel always free to enquire.