Anytime is fine, as far as my experience goes. They actually seem more calm after dark, much like chickens, and more easily handled. Whenever I did transfers to different cages for breeding, I always did it after dark.
The ones here frequently get fed after dusk, so they're starting to equate a cellphone flashlight with feed. They're usually busy in the morning and evenings, so I dunno if they are overly concerned about night time. We don't usually fuss with them much past dusk, though.