CS 294-14: Architecture of Internet Datacenters 
(RADLab Research Seminar 2.0)

Course Description

Internet Datacenters have recently emerged as a significant new computing platform, designed to provide high capacity processing for large numbers of web clients. Major web properties like Google have designed their own building-scale computer facilities, integrating processing, storage, internal and external networking, along with integral power and cooling infrastructures. The resulting datacenters typically deploy 100,000 to 1,000,000 computers within a single facility.

In this research seminar, we will read and discuss the very recent literature on the design and implementation of processor clusters, virtual machines, virtual storage, and datacenter networking organization. Architectural approaches to deal with failures, effective sharing of processing/storage/network resources, and efficient management of power across the systems stack will be considered. Some class meetings will be dedicated to meeting with and discussing issues with industrial leaders from Google, IBM, Cisco, and Network Appliances.

Course grading will be based on classroom participation and an independent research project, which can be pursued in groups of two students. In the past, the best of such projects have resulted in publications in leading conferences.

Background in one or more of the following areas is the prerequisite: an undergraduate/senior-level course in operating systems or computer architecture or computer networking (more than one is desirable but not necessary). Willingness to take an interdisciplinary "systems" viewpoint is essential.

First year graduate students are especially encouraged to consider taking this course if they are interested in pursuing research in the general area of modern computing systems.

Course Grading

• Initial Proposal (Week 4): 5%
• Final Proposal (Week 7): 10%
• Midcourse Correction (Week 10): 5%
• Poster Presentation (Week 14): 20%
• Report Write-up (Week 15): 20%

Please join the course facebook group here.

TENTATIVE Course Outline

• Monday, August 27
  1. [Randy] Randy H. Katz, “Internet-scale Computing: The Berkeley RADLab Perspective,” IWQoS 2007, Evanston, IL, (June 2007). [pdf]
  2. [Randy] Stephen Alan Herrod, VMWare, “The Future of Virtualization Technology,” ISCA 2006. [pdf]
  
  
• Wednesday, August 29
  1. [Randy] Raj Yavatkar, Intel, “Platforms Design Challenges with Many Cores,” HPCA-12, 2006. [pdf]
  2. [Randy] Renato Recio, IBM, “System IO Network Evolution: Closing the Requirement Gaps,” HPCA-12, 2006. [pdf]
  3. [Randy] Steve Kleiman, NetApp, “Trends in Managing Data at the Petabyte Scale,” FAST 2007, San Jose, CA, (February 2007). [pdf]

• Monday, September 3: Labor Day Holiday
  
  
• Wednesday, September 5
  2:30-4:00
  1. [Matei] S. Ghemawat, H. Gobioff, S.-T. Leung, “The Google File System,” Proc. SOSP’03, 2003. [pdf] [Notes].
  2. [Kuang] J. Dean, S. Ghemawat, “Mapreduce: Simplified Data Processing on Large Clusters,” Proc. OSDI’04, pages 137 – 150, (December 2004). [pdf] [Notes].
  3. [Michael] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, R. E. Gruber, “Bigtable: A Distributed Storage System for Structured Data,” Proc. OSDI'06, 2006. [pdf]
  
  6:00-7:30
  1. [Randy] Intel and Sun White Papers on Multicore Architectures [Notes]
     • Intel, "Intel Multi-Core Processors: Making the Move to Quad-Core and Beyond." [pdf]
     • Intel, "Inside Intel Core Microarchitecture: Setting New Standards for Energy-Efficient Performance." pdf
     • Intel, "Preparing for Peta-scale." [pdf]
     • Harlan McGhan, "Niagara 2 Opens the Flood Gates," Microprocessor, 11/6/2006. [pdf]
  2. [Ari] L. A. Barroso, J. Dean, U. Holzle, “Web Search for a Planet: The Google Cluster Architecture,” IEEE Micro, 23(2):22–28, March/April 2003. [pdf] [Notes]
  3. [Henry] L. A. Barroso, “The Price of Performance: An Economic Case for Chip Multiprocessing," ACM Queue, 3(7), September 2005. [html] [pdf].

• Monday, September 10
  (Randy traveling this week--Covered previous Wednesday)
  
  
• Wednesday, September 12
  (Randy traveling this week--Covered next Monday)

• Monday, September 17
  1. [Scott] J. Donald, M. Martonosi, “Techniques for Multicore Thermal Management: Classification and New Exploration,” ISCA 2006. [pdf] [Notes]
  2. B. Diniz, D. Guedes, W. Meira Jr., R. Bianchini, “Limiting the Power Consumption of Main Memory,” ISCA 2007. [pdf]
  3. F. J. Mesa-Martinez, J. Nayfach-Battilana, J. Renau, “Power Model Validation Through Thermal Measurements,” ISCA 2007. [pdf]
  4. [Barret] J. Lin, H. Zheng, Z. Zhu, H. David, Z. Zhang, “Thermal Modeling and Management of DRAM Memory Systems,” ISCA 2007. [pdf] [Notes]
  
  
  
• Wednesday, September 19
  Student Initial Idea Presentations (double session, 2:30-4:00 and 6:00-7:30 + Dinner)
  • Special Short Presentation: Mike Howard on RadLab/EECS Available Research Infrastructure [pdf]
  • Special Short Presentation: George Porter on X-Trace Path Tracing Infrastructure

• Monday, September 24
  Randy traveling (hopefully last time this semester!)
  
  
• Wednesday, September 26
  Double Session, 2:30-4:00 and 6:00-7:30 + Dinner
  1. Randy's Storage Notes [pdf]
  2. M. DeBergalis, P. Corbett, S. Kleiman, A. Lent, D. Noveck, T. Talpey, M. Wittle, “The Direct Access File System”, FAST, 2003. [pdf]
  3. [Andy] H. S. Gunawi, N. Agrawal, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, “Deconstructing Commodity Storage Clusters,” ISCA, 2005. [pdf] Conference Presentation [pdf]
  4. [Andrey] M. Eisler, P. Corbett, M. Kazar, D. Nydick, “Data ONTAP GX: A Scalable Storage Cluster”, FAST, 2007. [pdf] Conference Presentation [pdf]
     
     
  5. [Stephen] N. Hutchinson, S. Manley, M. Federwisch, G. Harris, D. Hitz, S. Kleiman, S. O’Malley, “Logical vs Physical File System Backup”, OSDI, 1999. [pdf]
  6. [Andrey] H. Patterson, S. Manley, M. Federwisch, D. Hitz, S. Kleiman, S. Owara, “File-System-Based Asynchronous Mirroring for Disaster Recovery”, FAST, 2002. [pdf]
  7. P. Corbett, R. English, A. Goel, T. Grcanac, S. Kleiman, J. Leong, S. Sankar, “Row-Diagonal Parity for Double Disk Failure Correction (Best Paper), FAST, 2004. [pdf]

• Monday, October 1
  1. J. Gray, “Why Do Computers Stop and What Can Be Done About It?,” Proc. of the 5th Symposium on Reliability in Distributed Software and Database Systems, 1986. [pdf]
  2. [Kurtis] J. Gray, “A Census of Tandem System Availability Between 1985 and 1990,” IEEE Transactions on Reliability, 39(4), 1990. [pdf]
  3. [David] R. K. Sahoo, R. K., A. Sivasubramaniam, M. S. Squillante, Y. Zhang, “Failure Data Analysis of a Large-scale Heterogeneous Server Environment,” Proc. of the 2004 International Conference on Dependable Systems and Networks (DSN’04), 2004. [pdf]
  
  
• Wednesday, October 3
  1. L. Bairavasundaram, G. Goodson, S. Pasupathy, J. Schindler, “An Analysis of Latent Sector Errors in Disk Drives”, SIGMETRICS, 2007 (Best Student Paper). [pdf]
  2. [Mowit] B. Schroeder, G. A. Gibson, “Disk Failures in the Real World: What Does an MTTF of 1,000,000 Hours Mean to You?” FAST 2007, San Jose, CA, (February 2007). [pdf]
  3. [Thomas] E. Pinheiro, W-D Weber, L. A. Barroso, “Failure Trends in a Large Disk Drive Population, FAST 2007, San Jose, CA, (February 2007). [pdf]

• Monday, October 8
  Special Guest Lecturer: Kaladhar Voruganti, NetApps
  1. K. Voruganti, J. Menon, S. Gopisetty, "Land Below a DBMS," SIGMOD Record, V. 33, N. 1, 2004. [pdf]
  2. B. Battles, C. Belleville, S. Grabau, J. Maurier, "Reducing Data Center Power Consumption Through Efficient Storage," NetApp White Paper, (February 2007), WP-7010-0207. [pdf]
  
  
• Wednesday, October 10
  Student Project Proposal Presentations
  • Andy/Matei, "Monitoring Hadoop through Tracing"
  • Ari/Stephen, "Monitoring Infrastructure"
  • Ganesh, "Reducing Power Consumption of Internet Datacenters"
  • Jorge, "Sensing the Datacenter"
  • Junda/Terry/Thomas, "Reducing Server Power Under Application Response Time Constraints"
  • Kuang/Gunho/Byong, "Declarative Distributed Debugging"
  • Kurtis, "Machine Learning for Routing around Congestion Events"
  • Michael/David/Barret, "Scalable Structured Data Storage for Web 2.0"
  • Mohit/Andrey, "A Case for a Fault-Tolerant Virtual Machine"
  • Scott/Henry, "Reducing Memory Power Usage on a CMT System"
  • Tracy/Yanpei, "Energy Efficient Ethernet Encoding"

• Monday, October 15
  Special Guest Lecturer: Dilip Kandlur, IBM
  Presentations on SMART [ppt] and File Systems [ppt]
  Suggested Readings
  • "GPFS: A Shared-Disk File System for Large Computing Clusters," Proc. of the First Conference on File and Storage Technologies (FAST), Jan. 2002. [pdf]
  • "NFSv4 Namespace and pNFS", particularly Section 11 Multi-server Namespace and Sections 13 and 14 on pNFS). [txt]. More details of pNFS at: www.pnfs.com.
  • "Glamour: A Wide-area Filesystem Middleware using NFSv4", IBM Research Report, RJ10368. [pdf]. See also Requirements Draft of Federated Filesystem standard: www.ietf.org/internet-drafts/draft-ellard-federated-fs-01.txt
  • "Nache: Design and Implementation of a Caching Proxy for NFSv4", FAST'07. [pdf]
  • Alvarez, et al., "Minerva: an automated resource provisioning tool for large-scale storage systems", ACM Transactions on Computer Systems 19(4):483-518, Nov. 2001. [pdf]
  • T. Eilam, et al., "Managing the Configuration Complexity of Distributed Applications in Internet Data Centers," IEEE Communications Magazine, March 2006. [pdf
  • L. Yin, et al., "SMART: An Integrated Multi-Action Advisor for Storage Systems" . In Proceedings of the 2006 USENIX Annual Technical Conference, Boston, USA, 2006. [pdf]
  • K. Keeton, et al., "Designing for disasters," Proc. of File and Storage Technologies (FAST'04) San Francisco, CA, March-April 2004. [pdf]
  
  
• Wednesday, October 17
  1. M. Gupta, S. Singh, "The Greening of the Internet," ACM SIGCOMM 2003, Karlsruhe, Germany, (August 2003). [pdf]
  2. [Special Guest Bruce Nordman from LBNL will LEAD the discussion of this paper!] C. Gunaratne, K. Christensen, B. Nordman, "Managing Energy Consumption Costs in Desktop PCs and LAN Switches with Proxying, Split TCP Connections, and Scaling of Link Speed," Intl. J. of Net Mgmt, 2005, 15: 297-310. [pdf] [ppt].
  3. [Yanpei] B. Wang, S. Singh, "Computational Energy Cost of TCP," IEEE Infocomm 2004. [pdf] [ppt]

• Monday, October 22
  1. [Tracy] M. Gupta, S. Singh, "Dynamic Ethernet Link Shutdown for Energy Conservation on Ethernet Links," IEEE ICC'07, Glasgow, Scotland, (June 2007). [pdf]
  2. [Tracy] M. Gupta, S. Singh, "Using Low-power Modes for Energy Conservation in Ethernet LANs," IEEE INFOCOM (Minisymposium) 2007, Anchorage, Alaska, (May 2007). [pdf]
  3. [Ganesh] M. Gupta, S. Grover, S. Singh, "A Feasibility Study for Power Management in LAN Switches," IEEE ICNP 2004, (October 2004), Berlin, Germany. [pdf]
  
  
• Wednesday, October 24
  Special Guest Lecturer: Flavio Bonomi, Cisco

• Monday, October 29
  Revised Student Project Proposal Presentations
  
  
• Wednesday, October 31
  Revised Student Project Proposal Presentations

• Monday, November 5
  1. E. V. Carrera, E. Pinheiro, R. Bianchini, “Conserving Disk Energy in Network Servers,” ICS ’03: Proceedings of the 17th annual international conference on Supercomputing, pages 86–97, 2003. [pdf]
  2. [Gunho] J. Mitchell-Jackson, J. G. Koomey, B. Nordman, M. Blazek, “Data Center Power Requirements: Measurements from Silicon Valley,” Energy (Energy) ISSN 0360-5442, 28(4):837 – 850, 2003. [pdf] [Notes]
  3. [Terry] X. Fan, W-D. Weber, L. A. Barroso, "Power Provisioning for a Warehouse-sized Computer," ISCA'07, San Diego, CA, (August 2007). [pdf]
  4. S. Gurumurthi, A. Sivasubramaniam, V. Natarajan, “Disk Drive Roadmap from the Thermal Perspective: A Case for Dynamic Thermal Management”, ISCA, 2005. [pdf]
  
  
• Wednesday, November 7
  1. [Junda] J. S. Chase, D. C. Anderson, P. N. Thakar, A. M. Vahdat, R. P. Doyle, “Managing Energy and Server Resources in Hosting Centers,” Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles (SOSP ’01), pp. 103–116, (2001). [pdf] [Notes]
  2. P. Bohrer, E. Elnozahy, T. Keller, M. Kistler, C. Lefurgy, C. McDowell, R. Rajamony, “The Case for Power Management in Web Servers,” Power Aware Computing, 2002. [pdf]
  3. [Jorge] J. Choi, Y. Kim, A. Sivasubramaniam, J. Srebric, Q. Wang, J. Lee, “Modeling and Managing Thermal Profiles of Rack-Mounted Servers with ThermoStat". High Performance Computer Architecture, 2007. [pdf]

• Monday, November 12: Veterans Day Holiday
  
  
• Wednesday: November 14
  1. [Kuang] P. Ranganathan, P. Leech, D. Irwin, J. Chase, “Ensemble-level power management for dense blade servers,” ISCA ’06: Proceedings of the 33rd annual International Symposium on Computer Architecture, pages 66–77, 2006. [pdf] [Notes]
  2. [Matei] R. Nathuji, K. Schwan, “VirtualPower: Coordinated Power Management in Virtualized Enterprise Systems,” SOSP 2007. [pdf] [Notes]
  3. W. Felter, K. Rajamani, T. Keller, C. Rusu, “A Performance-Conserving Approach for Reducing Peak Power Consumption in Server Systems,” Proceedings 19th Annual International Conference on Supercomputing (ICS ’05), 2005, pp. 293–302. [pdf]

• Monday, November 19
  1. [Ari] J. Moore, J. Chase, K. Farkas, and P. Ranganathan, “Data Center Workload Monitoring, Analysis, and Emulation,” Eighth Workshop on Computer Architecture Evaluation using Commercial Workloads, February 2005. [pdf]
  2. R. Fonseca, G. Porter, R. H. Katz, S. Shenker, I. Stoica, “X-Trace: A Pervasive Network Tracing Facility,” Fourth USENIX Symposium on Networked Systems Design and Implementation (NSDI 2007), Cambridge, MA, (April 2007). [pdf]
  3. [Scott] J. Moore, J. Chase, P. Ranganathan, R. Sharma, “Making Scheduling ‘Cool’: Temperature-aware Workload Placement in Data Centers,” USENIX Annual Technical Conference, (April 2005), pp. 61 – 75. [pdf] [Notes
  
  
  
• Wednesday, November 21
  Class cancelled for Thanksgivings Holiday!
  

• Monday, November 26
  1. [Henry] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, W. Vogels, “Dynamo: Amazon’s Highly Available Key-Value Store,” Proc. SOSP 2007. [pdf]
  2. [Mohit] M. R. Marty, M. D. Hill, “Virtual Hierarchies to Support Server Consolidation,” ISCA 2007. [pdf]
  3. Q. Zhu, Z. Chen, L. Tan, Y. Zhou, K. Keeton, J. Wilkes, “Hibernator: Helping Disk Arrays Sleep Through the Winter,” SOSP 2005. [pdf]
  4. H. Huang, W. Hung, K. Shin, “FS2: Dynamic Data Replication in Free Disk Space for Improving Disk Performance and Energy-Consumption,” SOSP 2005. [pdf]
  
  
• Wednesday, November 28
  Project Poster Session, 6th Floor Corridor, Soda Hall
  Poster Abstracts

• Monday, December 3
  Final Project Report Format/Course Summary
  
  
• Wednesday, December 5
  Course Summary/Course Evaluation
  

• Monday, December 10
  Class Cancelled -- Project Reports Due Wednesday, 12 December @ 11:59:59 PM
  

Useful Materials

• Amazon's CTO Werner Vogel's Blog Site
• Amazon's Web Services Web Pages
• Google Scalability Conference, June 2007. Some of these talks can be found on Google Video, e.g., http://video.google.com/videoplay?docid=-5525246919548243924&total=36&start=10&num=10&so=0&type=search&plindex=0
• Kunle Olukotun, Lance Hammond, "The Future of Microprocessors," ACM Queue, 2005. [pdf]
• Sun Presentations on Niagara 2 Architecture [pdf], [pdf]
• Energy Efficient Internet Project, Univ. of South Florida [html]
• M. A. Viredaz, L. S. Brakmo, W. R. Hamburgen, "Energy Management on Handheld Devices," ACM Queue, V. 1, N. 7, (October 2003). [html].