Davidherron's List: Drupal

• Boost

   

  Boost is a page cache which works in a similar way to aggressive page caching in that it attempts to serve cached data without running the bootstrap. However, it is able to go one step further and avoid PHP being run as the redirect takes place in rewrite rules located in .htaccess. If the static file exists then it is served as the response. The end result is super fast response without running PHP or the bootstrap. This frees up the server to handle more requests from logged in users.

   

  Boost is an attractive module because it is easy to install, has a lot of configuration options which give good control over cache building and invalidation. It is highly recommended.

• Varnish

   

  The Varnish HTTP Accelerator Integration module integrates Drupal with Varnish, a reverse proxy which sits in front of Apache, PHP and Drupal. Varnish stores cached content in RAM and avoids the overhead of Apache and the Drupal bootstrap. As such it offers very high performance for anonymous users on cached pages and is the preferred option for many sites where scaling is paramount.

   

  Varnish requires either a patch to core to add HTTP headers, PressFlow or Drupal 7. Most people are therefore running Varnish in conjunction with PressFlow.

• Varnish vs Boost

   

  A popular thread High Performance Drupal group is for the perfect recipe for page caching – whether to use Varnish or Boost. Varnish has the edge in speed over Apache+Boost as well as Nginx+Boost. Look at results published by brianmercer.

                 

  Setup	Approx requests/s
  Boost with Apache-prefork	500
  Boost with Nginx	2000
  Varnish	2400

   

  Varnish may have the edge in speed but it is more complex to install and requires a patched core. Some people may not want to run a non standard Drupal installation. Varnish also requires RAM to store the cached material – something which may be better spent elsewhere (database or max clients). Boost offers similar performance, is easy to install and has good control of cache invalidation and warmup. Boost is also able to serve files from RAM if the OS has cached them.
• Has anyone tried nginx caching with Drupal?

   

  Drupal Groups thread with lots of benchmarking. Final conclusion seems to be that Varnish vs Boost+Nginx is a pretty close thing.

   

  Project Mercury: A pre-configured Drupal+Varnish EC2 AMI

   

  Josh Konig of Chapter Three claims that Varnish is faster than Boost but gives no numbers.

   

  Quick Drupal Cacherouter and Boost benchmarks

   

  Dylan Tack likes Boost: “Response times are all close enough that it doesn’t really matter what caching backend you choose… The only factor that’s really relevant is how good your system’s cache expiration and regeneration logic is… it seems like Boost is the clear winner here as well.”

   

  What’ recipe should i choose for best performance?

   

  Discussion with participants split between Varnish and Boost depending on circumstances. Nginx+Boost seems pretty equal with Varnish.

   

  Caching: Modules that make Drupal scale

   

  Table of modules, performance gains and features.

• Cache Router

   

  Cache Router is a module which enables you to cache the Drupal cache tables (including views, blocks, menus, variables, and filters) in RAM. Drupal no longer has to hit the database to pull out this content – a big win for logged in users who might not be able to enjoy the advantages of a page cache. Cache Router therefore fills an important niche in your caching strategy.

   

  Cache Router is able to do this via a number of backends including APC or Memcache. APC should be considered if you are running Drupal on a single node and only need the single local store. According to Josh Konig, APC “is less error-prone, more secure, and allegedly as fast (if not faster) than running memcached according to the folks at Facebook.” Memcache can be used to share cached data between or across multiple servers.

   

3 more annotations...

• When running in an unoptimized configuration relatively simple pages can consume a lot of RAM, a fair amount of CPU and issue a large number of database queries. On medium sized sites it can be difficult to run in some shared server environments if the host is concerned about CPU and RAM. It is therefore important to attempt to get the most from your installation given your skill level and the resources available to you.

• Performance refers to the speed of the system. Basically, how long does it take to produce a page? Performance is a result of how long it takes to render a page (i) on the server and (ii) on the client side in the browser. It is important now to forget the client side of the equation as big gains can be made here if all else is running smoothly.

   

  Scalability refers to the ability of the system to handle more requests. ie. as the load increases on a system how is it placed to handle it. What is the throughput of the site under heavy load? For anonymous users and logged in users? How stable is the system? Are there faults or errors? How many resources are consumed?

• YSlow

   

  A Firefox addon which gives you an overview of the request/response cycle for a single web page. It provides a summary of all requests and offers suggestions as to how things can be improved. This tool is mainly used for optimising the rendering time of a page. See Drupal Page Rendering for more details on the client side. However, in terms of server side performance it does report the download time for the actual ‘doc’ requested which indicates how fast a page is being built on the server.

• Devel Module

   

  The Devel module is an essential tool to have on your dev machine. Devel is your first port of call on the server side when troubleshooting a site. You can use it to discover where problems are occurring before diving in to test further and debug. If you are suspicious that things aren’t quite right it can help you out with slow queries, slow pages and RAM consumption.

• MySQL Administrator

   

  MySQL Administrator has a GUI which, amongst other things, shows the health of your server. Having a copy of it running on your dev machine can give you good feedback on:

• mysql client

   

  The mysql command line client is invaluable for getting feedback from the server about what is going on. If a page is stuck, dive in and see what is happening.

   

  mysql> SHOW FULL PROCESSLIST; Lists all currently running queries, how long they have been running, what they are doing and their process id. Are any queries dumping out to disk? Have any been running too long? Any queries locked waiting for another to complete? Whats the PID of that slow query?

• Munin

   

  Munin is a network and system monitoring tool which provides insightful performance graphs over time. If you need to know when a problem hit then Munin will quickly be able to show you the stresses the server has been under.

• Google Analytics

   

  Google Analytics is analytic software which lets you track users and session across time amongst other things. Handy for observing traffic patterns. Implemented as JavaScript so will run for cached pages for users with JS turned on.

• Woopra

   

  Google Analytics competitor for tracking and analyzing web apps. Desktop Client GUI app rather than web interface. Variety of plans. Free plan limited to 30K page views per month.

   

   

• AWStats

   

  AWStats is a server log file analysis tool which provides graphical stats of the requests made on your server. It is generally recommended that the Statistics Module in core be turned off and AWStats run in place. The Statistics Module doesn’t fire if Aggressive caching, Boost or Varnish are in operation and so stats will be skewed.

6 more annotations...

• Characteristic Optimize No problems  Size many nodes few nodes  Activity high traffic low traffic  Users logged in anonymous  Page browsing dispersed concentrated  Contention many writes few writes  Content heavy light  Functionality rich poor  Audience dispersed concentrated

• RAM and CPU throttled

   

  Drupal is a system which can be heavy on system resources, especially RAM, and so running it on shared hosting can be a challenge in some situations. In many cases a relatively minimal Drupal install will exceed maximum limits enforced by your hosting company. In these cases you will receive a polite letter telling you to fix your site or else you can upgrade to a dedicated server. Don’t worry. See Out of the box for some quick and easy wins on shared hosting plans.

   

  Servers set up like this often have oversold the box you are hosted on and you will be at the mercy of other users who might be exceeding their allotment. Your performance will be adversely affected in these cases. YMMV.

• Limited control

   

  The major downside of shared servers is the limited control you have over the hosting environment, leaving you with relatively few options to improve performance. Luckily Drupal has some handy features built in which make it possible to improve speed, memory and CPU. These are dealt with in more detail below.
• scaling drupal – an open-source infrastructure for high-traffic drupal sites

   

  Incremental recipes for scaling a Drupal server out by adding more hardware: basic install on a single box, splitting off the DB, load balanced multi web servers, redundant load balancers with multi web servers, database clustering.
• Comparison of Major VPS Providers

   

  Linode comes out on top again. RAM looks to be the limiting factor here though.

   

  Slicehost vs Linode

   

  Linode comes out ahead in memory/bandwidth/cpu/storage per $.

2 more annotations...

• Drupal comes with a number of in-built caches which store the results of expensive calculations (strings) in the database so that they can be retrieved quickly later on. There are six caches enabled by default: cache, cache_block, cache_menu, cache_filter, cache_form and cache_page. Contributed modules are able to create their own caches for storing data which is handy for module designers. This default cache system provides improved performance across the whole app.

• Aggregate and compress JS and CSS

   

  Drupal’s modular system means that pages can have a large number of CSS and JS includes which results in a lot of client server communication – slowing the page draw time down. The problem can be alleviated by merging the files and then compressing them. This results in less includes and faster downloads. Up to 90% of download time can be attributed to downloading CSS, JS and images so it makes sense to aggregate and compress if possible.

• Block Cache

   

  Enabling the block cache allows finer grained control over cached content. Caching blocks which don’t change frequently will enable speedups for logged in users who need a dynamic page built for them each request.

   
• How to make Drupal run 8.5x faster in 5 minutes…

   

  Page cache provides a 3x speedup. 

  How I Survived a 2300% Traffic Increase With Drupal

   

  Demonstrates that these out of the box techniques can be very effective even for sites on shared hosting.

2 more annotations...

• Server tuning considerations

   

  Drupal documentation covering the basics.

   

  Tuning LAMP systems, Part 1: Understanding the LAMP architecture

   

  Intermediate article covering LAMP.

   

  Tuning LAMP systems, Part 2: Optimizing Apache and PHP

   

  Intermediate article covering Apache and PHP.

   

  Tuning LAMP systems, Part 3: Tuning your MySQL server

   

  Intermediate article covering MySQL.

• Opcode cache

   

  Opcode caches cache the compiled form of a PHP script in shared memory to avoid the overhead of parsing and compiling the code every time the script runs. This saves RAM and reduces script execution time.
• Drupal web server configurations compared

   

  APC gives 2x to 4x increase in throughput under load. PHP5 is around 10% slower.

   

  PHP op-code caches / accelerators: Drupal large site case study

   

  Op-code caches are a must for large sites serving many pages.

   

  Benchmarking APC vs. eAccelerator using Drupal

   

  eAccelerator is faster and smaller than APC. Both offer around 6x – 7x times speedup over PHP.

   

  High PHP execution times for Drupal, and tuning APC for include_once() performance

   

  Make sure apc.shm_size can fit the whole page else there will be no caching.

• Indexes on columns can dramatically speed up queries if the columns are used for filtering, sorting or joining. Generally, Drupal has most of the indexes you need covered, however, there are some areas where standard tables can benefit from an additional index. It is recommended that you profile your queries to see where things are slow before adding indexes in a scattergun approach because adding indexes can harm performance if they are not being used properly. You can use MySQL’s slow query log for queries with no index to identify areas for improvement.

   

  mikeytown2 has come up with a list of tables which could do with an index:

   

   

   
  • All CCK fields that you use in a view. File Field: create an index on the fid; date: index on date; index on value; etc…
   
  • access: type, mask, status
   
  • comments: timestamp
   
  • node_comment_statistics: comment_count
   
  • menu_links: external, updated, customized, depth
   
  • users: pass, status
   
  • menu_custom: title
   
  • date_format_types: title
   
  • filter_formats: roles
   
  • content_group: weight, type_name, group_name
   
  • term_data: name
   
  • system: name
   
  • imagecache_preset: presetname
   
  • blocks: module, delta
   
  • system: status, type
   
  • content_node_field: type, widget_type

• Apache vs Nginx

   

  Other Drupal users have replaced Apache with faster more lightweight (RAM and CPU) web servers such as Nginx and Lighttpd. Nginx is generally preferred over Lighttpd because of memory leaks in the latter. It is currently possible to run Nginx without losing any functionality in Drupal. Boost, a module based on .htaccess rules, now supports Nginx so it is feasible to run Nginx as the main web server. If you are constrained by CPU or have high loads then this certainly is an option worth considering.
• Apache vs Nginx : Web Server Performance Deathmatch

   

  “Nginx seems to compete pretty well with Apache and there doesn’t seem like there is a good reason not to use it especially in CPU usage constrained situations (ie. huge traffic, slow machines and etc).”

   

  In reply to kbahey: apache vs nginx

   

  Discussion and results over the pros and cons of Nginx vs various Apache setups.

   

  How to get Drupal working with Nginx

   

  Simple guide for installing and configuring Nginx on a server with only 256MB RAM. Uses FastCGI which may not be preferred method.

   

  NGINX + PHP-FPM + APC = Awesome

   

  “The following guide will walk you through setting up possibly the fastest way to serve PHP known to man…In this article, we’ll be installing nginx http server, PHP with the PHP-FPM patches, as well as APC.”

   

  PHP-FPM – A simple and robust FastCGI Process Manager for PHP

   

  Preferred way of connecting Nginx with PHP. Currently in PHP core for 5.3.2+ but not yet released. Requires patch to PHP 5.2.

• Apache: Unneeded modules

   

  It is possible to turn off unneeded modules in Apache to reduce memory footprint. The modules you require depends very much on your setup.

   

  The traditional way of controlling modules in Apache has been through the LoadModule directive in httpd.conf. Ubuntu and Debian do it differently with the /etc.apache2/mods-available directory and the a2enmod command. To list all modules to enable try:

   

  
 $ sudo a2enmod

   

  $ sudo /etc/init.d/apache2 force-reload
 

   

  And to see what you have enabled you can do $ sudo a2dismod.

   
• What Apache2 modules can be disabled?

   

  Lists of modules which should be enabled in Apache2.

• he use of mod_php with Apache is the most common setup for calling PHP. mod_php works by embedding PHP into every Apache process. This has the disadvantage of a large memory footprint for each Apache process. FastCGI and mod_fcgid overcomes this problem and reduces resource utilization with no gains in performance.

   

  kbahey lists the disadvantages of mod_php as follows:

   
   
  • All PHP loaded into the process
   
  • Heavy process even if flat file
   
  • Many processes will hog RAM
• Drupal webserver configurations compared

   

  The most common, Apache+mod_php is the slowest. Tests conducted with FastCGI which is faster. NB: FastCGI has subsequently suffered from stability issues.

   

  Apache with fcgid: acceptable performance and better resource utilization

   

  Informative article which comes out in favor of mod_fcgid over FastCGI and mod_php. This is the must read article if you wish to attempt fcgid.

   

  Configure Apache for high performance on drupal 6

   

  Some solid comments from kbahey from 2bits regarding stable setup: Apache, MPM Worker, fcgid, APC (code cache), memcache No SQL.

• Apache MaxClients

   

  The MaxClients parameter controls how many simultaneous clients Apache is able to serve. If it is set to high RAM will be chewed up and the Machine will go into swap. If it is set to low then your site will be unnecessarily limited by the number of clients it can serve. The setting of this value should be determined after consideration of (i) how much spare RAM is available on the server and (ii) how much RAM each Apache process consumes. Obviously you will want to maximize available RAM through frugal allocation of RAM to MySQL, JVM, etc and minimize the size of Apache process through techniques described above.

• 2bits provide the following formula:

   

  MaxClients = (Total Memory - Operating System Memory - MySQL memory) / Size Per Apache process.

10 more annotations...

• Solr

   

  Enter the Apache Lucene project and Apache Solr.

   

   

  Welcome to Lucene!

   

  Lucene “provides Java-based indexing and search technology, as well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities.”

   

  Welcome to Solr

   

  “Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world’s largest internet sites”

• Google Custom Search

   

  Another option for those users looking to switch away from the in-built search is Google Custom Search – a search service offered by Google where Google indexes your data and stores the results on their servers. You are able to search the data via a simple form on your site. Google Custom Search is free for individuals, who don’t mind ads with their results, with a business version starting at $100 pa. The Google Custom Search Module integrates it into Drupal by providing a block with the form.
• Server indigestion: The Drupal contributed modules “open buffet binge” syndrome

   

  Minimal install of Drupal has Apache process around 17MB-33M. A bloated system has 93M bloating to 100MB after a few requests. This means fewer requests can be handled due to RAM consumption.

• CCK design

   

  It’s important to know how to work within the confines of a system to get the most out of it so it is well worth investigating how data is stored in the backend for nodes and CCK fields. Smart, sensible design of CCK fields should ensure that database access is kept to a sensible level. Whilst this guide recommends designing a data model first and then worrying about data access times second, it is worth knowing the consequences of your decisions. 

  In a nutshell, CCK will create another table in the backend for multi fields and fields which are shared between content types. There’s no avoiding the first reason but in the case of shared fields, a separate DB query will need to be run for each shared property. This can be an issue on very large sites on nodes with a lot of properties when you want to keep DB queries to a minimum.

2 more annotations...

• Pressflow

   

  Pressflow is a distribution which attempts to bring many of the improvements discussed above (SQL improvements, Varnish) into a single package. Pressflow is a standard Drupal install which has had its core modified to fix bottlenecks and facilitate the use of advanced caching features. FourKitchens don’t regard Pressflow as a fork since many of the initiatives found in Pressflow are contributed back into the development of the head of Drupal. 

  So long as you haven’t hacked core yourself then using Pressflow is a simple matter of swapping out core drupal and replacing it with Pressflow.

   

  In a nutshell Pressflow allows the following:

   
   
  • Support for database replication
   
  • Support for Squid and Varnish reverse proxy caching
   
  • Optimization for MySQL

• Project Mercury

   

  Project Mercury is an innovative project from Chapter Three which wraps up a tricked out PressFlow installation in to a preconfigured Amazon Machine Image (AMI) for use on Amazon EC2 instances.

   

   

  The Pantheon Project Blazes Ahead

   

  Hot of the press: Mercury will also be available for deployment on other servers, not just EC2. Further there will be a Mercury On Demand service at Rackspace.

   

   

  The goal of this project is to make Drupal as fast as possible for as many people as possible. To that end, we are developing a pre-built Amazon Machine Image (AMI) which will allow anyone with an Amazon Web Services account to spin up an EC2 instance and see how all this works in real-time. The ultimate goal is a production-ready release that can be used for deploying real websites.

• Benchmarking and profiling Drupal

   

  Documentation which covers tools of the trade including Apache Bench (ab) and SIEGE.

• Apache Bench (ab)

   

  ab is the most commonly used benchmarking tool in the community. It shows you have many requests per second your site is capable of serving. Concurrency can be set to 1 to get end to end speed results or increased to get a more realistic load for your site. Look to the “failed requests” and “request per second” results.

   

  In order to test the speed of a single page, turn off page caching and run ab with concurrency of one to get a baseline.

   

  ab -n 1000 -c 1 http://drupal6/node/1
• Drupal Performance Measurement & Benchmarking

   

  Testing with ab and simple changes you can make within Drupal.

   

  On Drupal Performance: Testing with Apache Benchmark

   

  Covers server side tools and walks through ab options and use.

   

  Benchmarking Authenticated Drupal Users with ApacheBench

   

  Demonstrates how to pull out current session id and how to pass that to ab so that authenticated users can be tested.

   

  Has anyone tried nginx caching with Drupal?

   

  Illustrative discussion where different Drupal setups are benchmarked with ab.

• JMeter

   

  JMeter is a Java desktop app designed to test function and performance. It is the preferred testing tool of many administrators.

   

   

  Drupal-Performance-Testing-Suite

   

  Perl script which runs a JMeter test on Drupal and provides graphs.

   

  jmeter scripts to test server performance

   

  Some scripts to get you started testing with JMeter.

2 more annotations...

• Improving Drupal’s page loading performance

   

  Wim Leers covers all the bases on how to improve loading performance.

   

  High Performance Web Sites: Essential Knowledge for Front-End Engineers

   

  Steve Souders, Chief Performance Yahoo! and author of YSlow extension, covers the Yahoo recommedations in this book.

   

  High Even Faster Web Sites: Performance Best Practices for Web Developers

   

  Another Steve Souders book covering Javascript (AJAX), Network (Image compression, chuncked encoding) and browser (CSS selectors, etc).

• Combined Files

   

  The Out of The Box section covered the inbuilt CSS and JS aggregation and file compression. The use of “combined files” is a significant factor in Drupal’s relatively good score in the YSlow tests. Make sure you have this enabled.

• CSS Sprites

   

  CSS Image Sprites are another method of cutting down the number of requests. This approach combines a number of smaller images into one large one which is then selectively displayed to the user through the use of background offset in CSS. It is a useful approach for thing such as small icons which can have a relatively large amount of HTTP overhead for each request. Something for the theme designers to consider.
• S Sprites: Image Slicing’s Kiss of Death

   

  Overview of how CSS sprites work and how they can be used.

   

  A lesson in the usefulness of CSS sprite generators

   

  Covers commonly used spite generators
• CDN integration module

   

  Wim Leers’ fully featured module which integrates with a wide range of CDN servers.

   

  SimpleCDN module

   

  Simple CDN re-writes the URL of certain website elements (which can be extended using plugins) for use with a CDN Mirror service.

   

  Drupal CDN integration: easier, more flexible and faster!

   

  Slides covering advantages of CDNs and possible implementations.

   

  mod_cdn

   

  Apache2 module which shows some promise but not much info available for it with regards to Drupal.

   

  Best Drupal CDN module?

   

  Drupal Groups discussion.

• Add Expires Headers

   

  When a file is served by a web server an “Expires” header can be sent back to the client telling it that the content being sent will expire at a certain date in the future and that the content may be cached until that time. This speeds up page rendering because the client doesn’t have to send a GET request to see if the file has been modified.

   

  By default the .htaccess file in the root of Drupal contains rules which sets a two week expiry for all files (CSS, JS, PNG, JPG, GIF) except for HTML which are considered to be dynamic and therefore not cachable.

4 more annotations...

• All Sites

   
   
  • Get the best server for your budget and requirements.
   
  • Enable CSS and JS optimization in Drupal
   
  • Enable compression in Drupal
   
  • Enable Drupal page cache and consider Boost
   
  • Install APC if available
   
  • Ensure no slow queries from rouge modules
   
  • Tune MySQL for decent query cache and key buffer
   
  • Optimize file size where possible
   
   

  Server: Low resources

   
   
  • Boost stops PHP load and Bootstrap
   
  • Sensible module selection
   
  • Avoid node load in views lists
   
  • Smaller JVMs possibly if running Solr
   
  • Nginx smaller than Apache
   
  • mod_fcgid has smaller footprint over mod_php
   
   

  Server: Farm

   
   
  • Split off Solr 
  • Split off DB server, watch the latency
   
  • With Cache Router select Memcache over APC for shared pools
   
  • Master + slaves for DB
   
  • Load balancing across web servers
   
   

• Size: Many Nodes

   
   
  • Buy more RAM for database indexes
   
  • Index columns, especially for views
   
  • Thoroughly check slow queries
   
  • Warm up database
   
  • Swap in Solr for search
   
  • Solr to handle taxonomy pages
   
   

  Activity: Many requests

   
   
  • Boost or
   
  • Pressflow and Varnish
   
  • Nginx over Apache
   
  • InnoDB on cache tables
   
   

  Users: Mainly logged in

   
   
  • View/Block caching
   
  • CacheRouter (APC or Memcache)
   
   

  Contention: Many Writes

   
   
  • InnoDB
   
  • Watchdog to file
   
   

  Content: Heavy

   
   
  • Optimized files
   
  • Well positioned server
   
  • CDN
   
   

  Functionality: Rich

   
   
  • Well behaved modules
   
  • Not too many modules
   
  • View/Block caching
   
   

  Page browsing: Dispersed

   
   
  • Boost over Varnish if RAM is tight
   
   

  Audience: Dispersed

   
   
  • CDN

• DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data. We hope this will make it easier for the amazing amount of information in Wikipedia to be used in new and interesting ways, and that it might inspire new mechanisms for navigating, linking and improving the encyclopaedia itself.

• Get amped for mobile

   

  In the past it has been easy to not even consider mobile when designing a site or an application. However, there are so many compelling reasons these days to treat your mobile site as a key component of your web strategy. Mobile devices are becoming increasingly popular, more of your users will be accessing your site through a phone or tablet. They will expect a good experience from your site or else will not return. Watching a user attempt to browse your unoptimised desktop app on a mobile device is an embarrassing experience for you and frustrating for the user.

   

   

  Drupal in a tablet world [Dries Buytaert]

   

  Dries opens up a discussion about the future of Drupal for mobile and tablets. “This all begs the question: in a world of tablets and mobile handheld devices, what do we need to do so that Drupal will be a go-to platform? How can Drupal contribute so as to be a player in the ever-expanding ecosystem of tablets and mobile phones?”

   

  Everything you ever wanted to know about mobile, but were afraid to ask [Tomi T Ahonen]

   

  A long article which covers the myths and misconceptions about mobile. Very useful for getting your head around just what a remarkable medium it is.
• Rethinking the Mobile Web by Yiibu

   

  Slideshow covering the old and new ways of doing mobile. “Progressive enhancement” and “Responsive web design” techniques discussed.

• Mobile first

   

  Mobile has traditionally been considered an afterthought, if at all. However the landscape has shifted and it is now vitally important. Luke Wroblewski, a strong proponent of the approach, identifies three key reasons:

   
   
  1. Mobile is exploding
   
  2. Mobile forces you to focus
   
  3. Mobile extends your capabilities
   
   

  Focussing on the mobile platform forces the developer to prioritize the important things. What are the key actions and data for your application? Strip everything away and focus on just that. The end result will be an app which is simpler, easier to understand, just as functional and much better. This brings development back from flashy extraneous eye candy to a functional site which is designed to serve the user.
• Mobile First [Luke Wroblewski]

   

  Three reasons why Web applications should be designed for mobile first instead.

   

  Mobile First [Luke Wroblewski]

   

  “In this presentation, Luke Wroblewski will dig into the three key reasons to consider mobile first: mobile is seeing explosive growth; mobile forces you to focus; and mobile extends your capabilities.”

   

  Barcelona: Mobile First

   

  Eric Schmidt (Google) talks mobile. Google encourages developers to work on mobile first before desktop. “The phone is no longer the phone, it’s your alter ego. It’s the extension of everything we are. It doesn’t think as well as we do, but it has a better memory…. This is the time for us, now is the time for us to get behind this. … We understand that the new rule is mobile first.”

2 more annotations...

• RESTful design of URLS

   

  REST defines the architecture of the World Wide Web. One of the principles of REST is that a single URI represents a resource and that resource is conceptually different from the representations returned to the client.

   

   

  Representational State Transfer

   

  “Representational State Transfer (REST) is a style of software architecture for distributed hypermedia systems such as the World Wide Web. The term Representational State Transfer was introduced and defined in 2000 by Roy Fielding in his doctoral dissertation.[1][2] Fielding is one of the principal authors of the Hypertext Transfer Protocol (HTTP) specification versions 1.0 and 1.1″

• Caching, the fly in the ointment

   

  We’ve just seen that serving different representations from a single URI is a good thing from many perspectives: mobile first, progressive enhancement, REST and SEO. However, there is one reason why we may we may decide to go down the path of using two domains instead of one: caching.

   

  Caching mechanisms, such as Drupal Core and Boost, used the fully qualified domain name of a URI to determine caching keys. This allows the cache to quickly serve content to different clients without knowing the criteria which decides the representation received by the client, ie. the cache just has to know about the URI, it doesn’t need to decipher the user agent. Currently, if different representations are served for the same resource then the cache will likely become populated with a mix of different representations, leading to chaos. For this reason it is generally accepted that having a separate mobile site on a sub domain is a good way to go. ie. we would have two sites:

• Multisite setup

   

  Assume we are using two domains due to caching requirements. How do we serve this content? Drupal does have a multisite feature built in where a single Drupal “platform” can support many different site instances. These sites can share all data, no data or partial data, depending on how they are setup in settings.php. In the case of a mobile site we would want to share all data between the sites.

   

  One possible setup is to create a directory for the desktop and mobile versions under sites/

   

  sites/

   
   
  • all/ 
     
    • modules/ 
       
      • contrib/ 
         
        • cck/ etc
         
         
       
      • custom/ 
         
        • mw_mobile/
         
         
       
       
     
    • themes/ 
       
      • base_desktop/
       
      • base_mobile/
       
      • mw_desktop/
       
      • mw_mobile/
       
       
     
     
   
  • default/ 
     
    • files/
     
     
   
  • example.com/ 
     
    • settings.php
     
     
   
  • m.example.com/ 
     
    • settings.php
     
     
   
   

  The only trick to get this work is to manually set the default theme for the mobile site in the sites/m.example.com/settings.php file. For every page request, the config in settings.php will override the default variables defined in the variables table in the database.

   
  $conf = array(
   ‘theme_default’ => ‘mw_nokia_mobile’,
   );
   

  If you manually set a value like this you won’t be able to change it in the UI, naturally enough. Make sure the theme is active in the GUI.

1 more annotation...

• This article is a followup on how to configure memcache for Drupal, and how multiple bins help with performance.

   

  Each bin in memcached correspond to one or more cache table in Drupal.

   

  To do this, we first setup a start script for memcache that would start each bin with the correct size. This would go into /usr/local/bin/memcache.sh:
• #!/bin/sh case "$1" in  start) /usr/local/bin/memcached -d -u root -m 64 -p 11211  /usr/local/bin/memcached -d -u root -m 320 -p 11212  /usr/local/bin/memcached -d -u root -m 128 -p 11213  ;;  stop) killall memcached  ;; esac

• Then we need to change Drupal's settings.php to match that. You see that the page cache and filter cache each have their own bin, running on a separate instance of memcache. Everything else shares a "default" cache, which includes the menu cache and variables.

   

  $conf = array(   'cache_inc' => './sites/all/modules/memcache/memcache.inc',   'memcache_servers' => array(     'localhost:11211' => 'default',     'localhost:11212' => 'page',     'localhost:11213' => 'filter',   ),   'memcache_bins' => array(     'cache'        => 'default',     'cache_menu'   => 'default',     'cache_page'   => 'page',     'cache_filter' => 'filter',   ), );

1 more annotation...

• Wondering why your views run so slow? Why after switching to InnoDB, MySQL isn't running any better? This module has the answers!

   

  Indexes

   

  It will show what CCK columns get used in a view filter or relationship and give you the option to add an index on it so the views query will run faster. Also can add indexes to core modules that could use one.

• Correctly configure Block Visibility settings

   

  Drupal's block system allows you to place content in specific theme regions. But beware: even if you do not output a block in your theme, it will still be generated by the Drupal stack! That's because any enabled block's generation is determined by the block's visibility settings, not by the theme's region output.

  Read more »

   

    

   

     

    

   

  Disable 'Rebuild theme registry on every page'

   

  This is disabled by default, but double-check to make sure. Often theme developers activate this feature to test their developments. But on live production sites this setting should be turned off at all times. Find the setting via Site Building > Themes > Settings and then click on your Theme to find the setting.

  Read more »

   

    

   

     

    

   

  Disable the statistics module

   

  Drupal's statistics module has a disadvantage: extra MySQL queries per page load. Even when using the Boost module, boost_stats.php can still call these MySQL queries. Of course, it is up to you whether you need these Drupal stats. Most people will use Google Analytics as an alternative.

   

  Other alternatives are server tools like Munin or Cacti.

  Read more »

• Enable core caching features

   

  Often overlooked by Drupal beginners, but still worth mentioning: do not forget to enable core caching at Administer > Site Configuration > Performance (/admin/settings/performance). To avoid issues with Drupal modules, simply select Cache Mode: Normal. And enable Page Compression.

  Read more »

   

    

   

     

    

   

  Enable native module caches (e.g. Views, Panels, Feeds)

   

  Several Drupal modules bring their own caching system and caching tables. For example: Views, Panels and SWF Tools. Each have their own configuration options. On production servers, it is wise to enable them.

  Read more »

   

    

   

     

    

   

  Install ImageCache to generate thumbnails

   

  ImageCache is a module that can generate image thumbnails on the fly, to exactly fit your dimensions. The thumbnails is only generate once, so should not influence performance. The benefit however is to the end user: visitors will download smaller images instead of the full unscaled version.

  Read more »

   

    

   

     

    

   

  Increase minimum cache lifetime on high traffic sites

   

  Drupal's core caching system allows to set minimum cache lifetimes. This means content will not be refreshed until after a certain time. For example, on busy sites, you would set the Minimum Cache Life to 1 hour. That means visitors do not see changes on your site more often than 1 hour. But it greatly helps to improve cache-efficiency and reduces your server load.

  Read more »

• Install Pressflow Drupal

   

  Regular Drupal core 6.x lacks built-in performance enhancers. Pressflow is a Drupal API compatible performance-enhanced distribution of Drupal core. That means it is faster than regular Drupal, but still works with all modules, themes, language packs and installation profiles.

  Read more »

   

    
   
     
    
   
  Install the Block Cache Alter module
   
  Drupal has a built-in Block caching system, which caches the output of any block based on settings provided by module maintainers. (In hook_block() one can specifiy a number of caching settings.) The built-in system has two disadvantage: 1) it does not let you specify per-block caching; 2) it does not work at all when node access modules are in place.
  Read more »
   
    
   
     
    
   
  Install the CSS Gzip module
   
  Drupal handles separate CSS files per module. This way module maintainers can provide their custom CSS. But the disadvantage is you get a lot of CSS files. Each file requires a separate HTTP request. The CSS Gzip module extends Drupal's core CSS handling.
  Read more »
   
    
   
     
    
   
  Install the DB Maintenance module
   
  The DB maintenance module periodically optimizes your MySQL database tables. It runs the SQL statements OPTIMIZE to make sure the integrity of your tables is kept sane.
   
  Advantages
   
   
  • Avoid table corruption, optimize database table structures
   
  • Faster table access

• Install the Global Redirect module

   

  The Global Redirect module is useful when using Clean URLs. The module redirects core paths, such as node/x and taxonomy/term/x, to their appropriate URL aliases. This avoids both duplicate content as well as server caching resource: don't store duplicate pages.

   

    

   

     

    

   

  Install the Javascript Aggregator module

   

  Drupal 6.x has a built-in JavaScript compression tool at Administer > Site Configuration > Performance, near Bandwith Optimizations. However, end-user performance can be taken one step further by minifying that file. The Drupal JavaScript Aggregator module provides that option. Files are minified once, and refreshed when needed.

   

  Uses JSMin.

  Read more »

   

    

   

     

    

   

  Install the Update Status Advanced Settings module

   

  I recommend turning the Update Status module on for production servers. It informs you when to update critical security patches. But the core module does not let you control how often should be checked for updates.

  Read more »

   

    

   

     

    

   

  Replace core cron with Elysia Cron

   

  The cron function should run periodically. But core cron runs all tasks at once. Enter Elysia Cron: this module lets you schedule specific cron tasks. For example, you may want to rebuild the XML Sitemap at midnight, but refresh your Node cache every hour.

  Read more »

• Uninstall unnecessary Drupal modules

   

  Does it really matter how many modules you run? According to 2Bits more modules means more RAM and CPU usage. Each time the Drupal stack is fired it needs to read all module files. Adding more modules means:

  Read more »

   

    

   

     

    

   

  Avoid calls to node_load() where possible, use fields

   

  Drupal's node_load() function fires a great deal of queries and functions. The more module you use, the more node hooks will be involved in this call. Wherever possible it is advised to avoid calls to node_load(). Instead, try to fetch the specific fields you need directly. Best example: Views. Using the Views module you can build a page that lists all nodes.

  Read more »

   

    

   

     

    

   

  Avoid calls to taxonomy_get_tree()

   

  Drupal's taxonomy system, to categorize and tag content, has one bad function taxonomy_get_tree(). It works fine on small sites with few tags. But on large sites with over tens of thousands of taxonomy terms, you should avoid it at all cost. The function namely stores all terms into the PHP RAM memory, quickly blocking further PHP execution.

  Read more »

   

    

   

     

    

   

  Avoid calls to third party web 2.0 services

   

  Digg, Twitter, Facebook Connect, OpenCalais and what not can be configured to submit a new node or comment upon submission. This works for small sites with low volumes, but even then it slows down the form submission: you have to wait for the third party server to process your request and send back an "OK".

  Read more »

• Avoid calls to variable_set() on page loads

   

  The Drupal variable are cached as one string. But upon calling variable_set() on any page requires this variable cache to be refreshed. This leads to a heavy MySQL query, SELECT * FROM variables. Best practice is to only use this call on administrative tasks. Avoid it on any other page.

  Read more »

   

    

   

     

    

   

  Avoid CSS expressions

   

  CSS expressions are an old trick that works only in Internet Explorer browsers, but has no effect in Chrome or Firefox. They are in fact JavaScript function called from CSS. From Yahoo!:

  Read more »

   

    

   

     

    

   

  Avoid using shared CCK fields

   

  Using the Content Construction Kit (CCK) you can create specific fields for Drupal nodes. But when you re-use the same field for multiple content types, then CCK creates a separate shared database table. The consequence is an extra table JOIN to read the data. The solution is to use separate fields per content type.

  Read more »

   

    

   

     

    

   

  Avoid using slow (node access) modules

   

  This is an obvious performance tip: avoid installing modules that are known to drag down Drupal performance. If you do not need them, do not use them. This espcially relates to any kind of node access related modules. Such modules add extra MySQL database "LEFT JOINS" to match each node table item to the node_access table, to check if the user may access the node.

  Read more »

   

    

   

     

    

   

  Avoid using the PHP input filter

   

  Drupal provides a PHP input filter out of the box. It lets you use custom PHP inside nodes and blocks. The downside is that this PHP code is stored in the database. Upon a new page request this code must be retrieved from the database and executed through PHP's eval() function.

  Read more »

• Disable developer modules

   

  A number of modules have no functionality other than to provide an Administation UI. Examples are Views UI, Panels UI, ImageCache UI, Feeds UI and several others. On production servers, once you are done with them, you probably do not need keep them enabled. By just disabling these modules you can save Drupal resources. Do not unintall these modules!

  Read more »

   

    

   

     

    

   

  Disable sessions completely for anonymous users

   

  On large sites with high anonymous traffic the sessions table may get clogged up with anonymous sessions. One solution s to store these sessions into Memcache. However, you may go one step further and decide to eliminate anonymous sessions completely, which includes crawlers and robots.

  Read more »

   

    

   

     

    

   

  Disable the database logging module

   

  For the same reason as disabling the Drupal core statistics module, one can disable the database logging module. It also fire MySQL queries on each page load. For truly optimal performance, for example on shared hosting or on hig traffic sites, you should definitely turn it off. This means you will have to use the server's error logs system.

  Read more »

   

    

   

     

    

   

  Enable .htaccess ExpiresByType directives

   

  Using Google Page Speed tools, one of the recommendations is to enabled Expire headers for Apache. The ExpireByType directives in your Drupal .htaccess file can set the duration which files should be stored locally. For example, if you set ExpiresByType for jpg files to 1 year, that means your file will not be re-downloaded by a user.

  Read more »

   

    

   

     

    

   

  Implement fast 404s for static files

   

  A drawback of dynamic CMS like Drupal is the handling of 404 errors, or Page Not Found errors. Drupal handles every request through index.php. If a file or path does not exist, Drupal has already bootstrapped the entire LAMP and modules stack. 2Bits came up with a solution. Below I show the code I use at the bottom of settings.php.

  Read more »

• Improve theme performance with CSS sprites

   

  CSS sprites are multiple images merged into a single image. This reduces the number of HTTP request to your server. Using CSS style rules, you can position the sprite in such a way that it show different graphics for differenet sections of the site. For example, you can merge the header, logo and footer into one CSS sprite image.

  Read more »

   

    

   

     

    

   

  Install the Cache Expiration module (experimental)

   

  Drupal offers no built-in way to control when caches should expire. The Cache Expiration module offers a better methodoloy than Drupal core. However, the module is experimental and may not sensibly work for your setup. The module offers support for:

  Read more »

   

    

   

     

    

   

  Install the Boost module (on shared hosts)

   

  For shared hosts, the best caching option availabe is the Boost module. Boost creates HTML copies of rendered Drupal pages and stores them in a cache folder. Using .htaccess rules Boost then checks if a file already exists. If so, it will load the static HTML file, while fully avoiding Drupal, PHP and MySQL.

  Read more »

   

    

   

     

    

   

  Install the Job Queue and Queue Mail modules

   

  This is an innovation by the 2Bits company. They figured sending out email for node updates should not occur instantly, but rather once per hour via cron. Job Queue is a "garbage collector" that disposes on its jobs on cron runs.

  Read more »

   

    

   

     

    

   

  Move all JavaScript to the page footer

   

  Part of Yahoo!'s YSlow recommendations is to load JavaScript in the footer of the page. Drupal's theming system usually has two variables, $scripts (in the <head>) and $closure (before the closing </body> tag). According to Wim Leers, the quicket way to move all Drupal's JS to the footer is by moving $scripts just before $closure.

  Read more »

• Move third party libraries outside the modules folder

   

  Every file in the modules folder is read for inclusion in the Drupal stack. If you store large third party libraries, such as teh YUI library or third party WYSIWYG editors, this will slow down Drupal. The modules folder is usually in /sites/all/modules or /sites/all/modules/contrib.

  Read more »

   

    

   

     

    

   

  Simplify your Drupal theme

   

  A number of aspects influence theme performance: its size in kilobytes, number of total files, flash and silverlight, the number of CSS styles and stylesheets, the number of jQuery functions and other JavaScript files included, the size of the images and so on. Your theme may have complex regions and visibility logic.

  Read more »

   

    

   

     

    

   

  Use the Devel module to analyze performance

   

  The Devel module provides a set of developer UI tools and code hooks to analyze performance. It also includes a Performance Logging tool to measure PHP's RAM usage per Drupal page.

  Read more »

   

    

   

     

    

   

  Cache Drupal with Memcached

   

  Memcached is a server caching system used by, for example, Facebook and Digg to boost performance. Memcached stores rendered Drupal pages in the RAM memory. All you need is good RAM. Memcached can work on multiple servers. To use Drupal with Memcached, have a look at the Memcache integration module.

  Read more »

   

    

   

     

    

   

  Configure Alternate PHP Cache (APC) for Drupal

   

  APC is an op-code caching sytem for Apache servers. The more Drupal modules you enable, the greater the need for such PHP pre-processing. Most PHP accelerators work by caching the compiled bytecode of PHP scripts to avoid the overhead of parsing and compiling source code on each request (some or all of which may never even be executed).

• Convert MySQL tables to UTF8 InnoDB

   

  By default, Drupal 6 database tables are installed as MyISAM (Drupal 7 switched to InnoDB). It appears that specific modules already have InnoDB schema's implemented. This results into a mixed-mode database.

  Read more »

   

    

   

     

    

   

  Correctly Set Apache's MaxClients directive

   

  By default, the MaxClients parameter has a compiled in hard limit of 256. 2Bits provides a calculation to find the number you need: (Total Memory - Operating System Memory - MySQL memory) / Size Per Apache process.

   

  To keep things working smoothly, you probably want to check your apache error log regularly to make sure MaxClients is not reached.

  Read more »

   

    

   

     

    

   

  Disable unneeded Apache modules

   

  For a fast performing server, you want to reduce the "bloat" - disable any server feature you do not need. For example you would not want Plesk or cPanel running on your server. But Apache itself has a bloat of modules that are hardly used. It requires an expert's opinion to see what is really needed.

  Read more »

   

    

   

     

    

   

  Enable Apache mod_deflate

   

  Apache's mod_deflate module compresses static HTML files before sending them to the end-user. You must enable it on your server.

   

  Example .htaccess snippet

   

  <ifmodule mod_deflate.c>
     AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript
   </ifmodule>

• Enable MySQL query caching

   

  Add another caching layer to your system: make sure to properly cache MySQL queries. This relies on settings you find (or should insert) in your my.cnf configuration file on your server. According to Nadeau software, the MySQL query caching can boost performance by 5-15%.

  Read more »

   

    

   

     

    

   

  Fix slow Views queries by adding indexes

   

  The Views module is a powerful tool to build custom database queries and style the results. But the Views UI does not always produce the most optimal query. Especially, it does not add appropriate table indexes (at the MySQL schema level) to improve performance.

  Read more »

   

    

   

     

    

   

  Handle users and their sessions via Memcached

   

  For busy sites the sessions table will be read very frequently and can cause performance locking problems. This is not an issue for small and medium sites. But to take Drupal performance a step further, let APC or Memcache handle the sessions.

  Read more »

   

    

   

     

    

   

  Install ImageMagick to replace PHP's GD image library

   

  ImageMagick is an image processing tool for PHP, and Drupal's Image toolkit can use it. ImageMagick is more resource-efficient than PHP's often built-in GD libraries. ImageMagick is also a solution to those who have PHP not compiled with the correct GD libraries.

  Read more »

   

    

   

     

    

   

  Move Drupal's .htaccess rules to vhost.conf

   

  When Apache's AllowOverride is set to allow the use of .htaccess files, Apache will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! Also, the .htaccess file is loaded every time a document is requested.

  Read more »

• Run the MySQL tuning primer to optimize my.cnf

   

  The Drupal MySQL database is often a performance bottleneck. Poorly configured MySQL limits the speed at which Drupal retrieves and stores data. If you have access to your server, then the best way to measure your MySQL performance is by running the MySQL Tuning Primer bash script for Linux. The configuration options are stored in a my.cnf file on your server.

  Read more »

   

    

   

     

    

   

  Turn Apache KeepAlive On

   

  From Ebizontek:

  Read more »

   

    

   

     

    

   

  Install Varnish HTTP Accelerator integration

   

  The Varnish HTTP Accelerator Integration module integrates Drupal with Varnish, a reverse proxy which sits in front of Apache, PHP and Drupal. Varnish stores cached content in RAM and avoids the overhead of Apache and the Drupal bootstrap.

  Read more »

   

    

   

     

    

   

  Get Pantheon / Drupal Mercury

   

  Pantheon is Drupal on a supercharged cloud stack with the community's best practices built in from the beginning. Stability. Security. Sheer speed. So you can focus on your Drupal sites. With Pantheon, you stand on the shoulders of giants.

   

  Drupal Mercury is a stack of:

  Read more »

   

    

   

     

    

   

  Replace core search with Apache Solr

   

  For big sites, say tens of thousands of nodes, the Drupal core search module is not suitable any longer. You could tweak it to handle hundreds of thousands of nodes, but there is a limit. To handle high search volumes, one should look to external third party search solutions. The most popular -open source- tool is Apache Solr.

  Read more »

• Run Drupal 6 on Solid State Disks

   

  SSDs do provide apples-to-apples performance improvement over HDDs (even 10K RPM drives). One challenge with AMP-based packages such as Drupal is that they are not often
   I/O-bound unless precisely tuned for particular workloads. As such, 'out of the box' performance gains might not justify the increased cost.

  Read more »

   

    

   

     

    

   

  Run Drupal on Zend Server with Page Cache

   

  Acquia tested Drupal performance using PHP only, PHP with APC and Zend Server with Page Cache. They find that Zend Server with Page Cache enabled increased Drupal performance by 2605% (in terms of page requests, valid for Acquia's testing environment).

  Read more »

   

    

   

     

    

   

  Use a Content Delivery Network (CDN)

   

  Drupal's best Content Delivery Network integration module is written by Wim Leers as part of his bachelor  thesis. It is compatible with push and pull setups. A content delivery network does two things.

  Read more »

11 more annotations...

• Omega has the most features of any of these HTML5 base themes and is one of the most popular. My experience with Omega is that it is an extremely well put together theme. There is a bit of a learning curve due to the vast number of configuration options, but once you have figured out how all of the settings relate to each other, most of the layout can be configured in the browser. It is my opinion that Omega is great for new themers, design shops that can systematize their process based on it, and anyone who would rather configure in the browser instead of writing CSS & PHP code.
• I think Omega will likely become the dominant player in Drupal 7 HTML5 base themes. It is in active development and has its own microsite with detailed documentation. The unique features that really set it apart are the Context integration, Responsive layout (can adapt to different size displays), and Drush support. On top of that there is a UI sub-module in development that will allow drag & drop configuration of Omega layouts.