Add warm milk and sugar to a large bowl, whisking well. Sprinkle yeast over the top and let sit for 10 minutes, until starting to form a froth on top. Add flour, butter, vanilla and salt, stirring with a sturdy wooden spoon until it starts to leave the side of the bowl. You can also use an electric mixer with a hook attachment as well.
Cover with a dry towel and let rise in a warm spot for one hour, or until about double in bulk.
Punch down the dough and roll it out into a rectangle about 14-inches by 8-inches.
Spread peach all fruit, followed by diced peaches to within an inch from all sides.
In a bowl, blend sugar, brown sugar, cinnamon and ginger. Evenly sprinkle over peaches. Tightly roll up the dough and cut into 12-15 even slices.
Place on parchment-lined baking pan, or well greased pan, cut-side down, at least an inch apart. Allow to rise until they are 50% larger. Bake at 350-degrees F for 16-18 minutes, or until nicely browned.
Meanwhile, make Cream Cheese Glaze by combining all ingredients in a bowl and beat until smooth.
When rolls are done, remove from oven and drizzle glaze over the top before serving hot.