DJHell .'s List: The Art of Capacity Planning

Web-based companies live or die by the ability to scale their infrastructure to accommodate increasing demand. This book is a hands-on and practical guide to planning for such growth, with many techniques and considerations to help you plan, deploy, and manage web application infrastructure. Written by the manager of data operations for the world-famous photo-sharing site Flickr.com, the author offers advice based on years of valuable experience.

This comprehensive guide covers the design of software and hardware systems for web applications. Using scores of examples and leading-edge tips, it details proven methods for scaling web applications to millions of users. Topics include application architecture, development practices, technologies, Unicode, and general infrastructure work. Ideal for anyone ready to realize the cost and performance benefits available to web applications today.

As millions of users flood the Internet, websites must be able to handle vast numbers of users. Theo Schlossnagle's Scalable Internet Architecture teaches developers both good and bad design methodologies for building new sites, and how to scale existing websites to robust, high-availability websites. Primarily example-based, the book discusses major topics in Web architectural design, presenting existing solutions and how they fit. Schlossnagle introduces new and innovative concepts to solving traditionally expensive problems without a large technology budget. Using open source and proprietary examples, the book discusses best practice design methodologies for building new sites, as well as appropriately scaling both growing and shrinking sites. Scalable Internet Architecture illustrates the progression from anecdotal conclusions to real-world, practical results and often how the two are askew. Topics include: clustering, databases, high-availability, load-balancing, and caching architectures

RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data.

Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.

Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort.

Using Munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine "what's different today" when a performance problem crops up. It makes it easy to see how you're doing capacity-wise on any resources.

Munin uses the excellent RRDTool (written by Tobi Oetiker) and the framework is written in Perl, while plugins may be written in any language. Munin has a master/node architecture in which the master connects to all the nodes at regular intervals and asks them for data. It then stores the data in RRD files, and (if needed) updates the graphs. One of the main goals has been ease of creating new plugins (graphs).

Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus of httperf is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks. The three distinguishing characteristics of httperf are its robustness, which includes the ability to generate and sustain server overload, support for the HTTP/1.1 and SSL protocols, and its extensibility to new workload generators and performance measurements.

Siege is an http regression testing and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It allows the user hit a web server with a configurable number of concurrent simulated users. Those users place the webserver "under siege."