1. In a bowl, whisk egg yolks with granulated and brown sugars until thickened and pale yellow. Set aside.
2. In a medium saucepan over medium-low heat, bring cream, milk and rosemary to a simmer. Simmer, stirring occasionally, for 15 minutes. Discard rosemary. Gradually whisk hot cream mixture into the egg mixture.
3. Return entire mixture to the saucepan. Cook over low heat, stirring constantly, until the mixture is thick enough to coat the back of a wooden spoon. Be careful not to let it boil. Strain into a clean large bowl. Let cool to room temperature. Cover and refrigerate until completely cold or overnight.
4. Stir cream micxture. Transfer to an ice cream maker and freeze according to manufacturer's instrucitons.