I know people talk about rapid cooling, ice baths, etc. We milk, then take it inside and filter, then put it in the fridge. We have never had a problem. Even in the summer when the milk sits for 30-45 min before getting filtered and cooled, no problems. We wouldn't pass a formal dairy inspection, but that doesn't matter to us.
I think the biggest thing is keeping the milk clean during milking (udder wash, no foot dirt in pail, etc). If the taste is off, isolate the milk by individual animals to determine who is the problem. Individual goat health can impact the milk flavor dramatically.