Today’s data center architectures are often built in the form of multi-rooted tree topologies with with less overall bandwidth at higher levels of the tree than at the bottom. In datacenter parlance, this is referred to as the over-subscription factor, and according to literature, many existing datacenters are heavily oversubscribed with ratios of about 240:1 or 80:1. This clearly means that the links at the top of the tree are more scarce resources than at the bottom of the tree. Specifically, the bandwidth that is typically available within a rack (through Top-Of-Rack switches) is significantly higher than the bandwidth available at the core routers.
The current trend of Internet services is to run on distributed nodes in gigantic data centers comprising 100s of thousands of machines spanning multiple continents. Scalable multicast trees and distributed key-value stores form the key building blocks for these applications, relying on a self-organizing overlay among the nodes (Chord, Pastry, Bamboo). However, a straightforward implementation of many existing distributed overlays would pay no heed to this information, and would lead to sub-optimal routes that cross the top of the tree—often many times over.
This paper presents a framework called DC2 (short for data center aware distributed communication) that incorporates the hierarchical information specific to data center environments in its overlay routing. This helps applications to limit communication locally within clusters (e.g. racks) and cross clusters only when necessary. To show the generality of DC2, we build two systems on top of DC2— an overlay multicast system called DC2-Multicast and a key- value object store called DC2-Store, that both minimize cost and increase scalability.
Karthik Nagaraj, Hitesh Khandelwal, Charles Killian, Ramana Rao Kompella. Hierarchy-Aware Distributed Overlays in Data Centers using DC2. In Proceedings of the 4th International Conference on Communication Systems and Networks (COMSNETS 2012). Jan 3-7, Bangalore, India. (to appear)