Our baby bunnys are 5 1/2 weeks old now. Our doe had 4 females and one male. The little males is trying to get friskey with his sisters! He can't get them prego at this time of his life, can he?
I've bred bucks at 5 months successfully (New Zealands). I've had them fight at 9 weeks. But with 1 buck and a bunch of does, you've got a few weeks before I'd worry about anything. I wouldn't keep them together past 4 months though, preferably less.