Secrets Of High-Traffic WordPress Blogs

Advertisement

We all know that WordPress is awesome — but being awesome isn’t always enough. Does it perform well under pressure? Can it deal with traffic from millions of visitors every month? There’s no question that WordPress can be used for your or my blog, but what about multi-author blogs with thousands of comments? How do developers make it scale and perform?

I spoke with the developers behind some of the biggest WordPress blogs on the planet and asked them to tell me their secrets. Now I get to share them with you.

The Blogs

Website Monthly uniques Monthly page views
Digital Trends571 10 – 12 million 33 million
iPhoneclub.nl592 2.5 million 5.4 million
The Next Web603 4 million 8 million
Neatorama614 2.5 million 4.5 million
Slashgear635 6 million 10 million
Hot Air666 2 – 3 million 35 – 45 million
Laughing Squid677 Undisclosed Undisclosed

In The Beginning

The first thing I asked the developers was whether they prepared for the heavy traffic that now flows through their website. In almost all cases, the answer was a resounding no. From The Next Web, CTO Arjen Schat and lead developer Pablo Roman said they had planned for growth but didn’t expect growth to happen on such a large scale. “There were few large WordPress sites at the time, so we learned as we went along.”

Neatorama started out in late 2005 on cheap shared hosting until it got kicked out. It moved to a VPS and got kicked out again. In 2007, it moved to a dedicated server with a CDN, which eventually was insufficient, until finally it moved onto load-balanced servers with a CDN. Similar stories are echoed by iPhoneClub.nl and Laughing Squid.

Hot Air’s developer, Mark Jaquith, also a lead developer at WordPress, had to migrate the website to a new server within 48 hours of launching. Only SlashGear planned for a 30% increase in traffic per year.

Digital Trends8

Digital Trends started out with around a million uniques a month and has since risen to 10 to 12 million uniques. Tom Willmot, of Human Made9, the development shop behind Digital Trends, said this of starting out:

When I started work on the website, there were some pretty big performance sinks in the code base that needed ironing out. A clean code base takes you only so far, however. I don’t think it’s something that should be too heavily focused on at the outset as you don’t know what specific features will need performance considerations. Coding well plus some persistent object caching are enough to begin with.

Server Shred And Burn

High-traffic blogs have to deal with things that regular bloggers only dream of: loads and loads of visitors, eager to read all of the latest news. Neatorama held up when it was featured on the front page of Digg, but was shredded within a few minutes of being featured on Yahoo’s front page — over 2 million visits in the space of a few hours. To deal with that, it had to make a static page on a CDN and redirect traffic there.

Other blogs face challenges related to major events. Visitors flock to their websites to follow what’s happening and hear the news. iPhoneClub.nl and SlashGear are particularly affected during Apple’s live broadcasts. SlashGear experiences traffic of more than 4 million uniques within the two hours of a broadcast’s duration; to deal with this, it adds Amazon EC2 to its normal infrastructure.

iPhoneclub.nl10

Jean-Paul Horn, whose iPhoneclub.nl started out on WordPress 2.0.5, had to learn to deal with these problems on the fly:

We used to run a standard LAMP stack with WP Super-Cache, since the site was already growing rapidly when the iPhone started to get really popular. Our main problem was that the server ran perfectly fine on normal news days, but almost literally burned whenever Apple had a keynote or another big iPhone-related announcement. We tried preparing for these short avalanches by temporarily adding more cores and RAM, but it was never really enough, and I wasn’t really keen on investing in a big-ass server just for two to three days of insane amounts of Web traffic a year.

While moving to a Web server and database server with MySQL stored on a separate server helped in its continuing growth, it was still dealing with the same issues when Apple held a press conference. Enter Frederick Townes of W3 Total Cache11 and Mashable fame. Jean-Paul met Frederick at WordCamp Netherlands, and Frederick went on to help them set up their current configuration:

  • The app server hosts the Varnish front end.
  • They use two nginx Web servers: one with PHP-FPM, the other with static content, like a CDN.
  • The database server runs a highly tuned MySQL server, in which some of the WordPress tables, such as wp_posts, have been transitioned from MyISAM to InnoDB. Because InnoDEV doesn’t have full text search, Sphinx Search was implemented.
  • A performance analysis was carried out using the cachegrind files that xdebug generates to easily identify execution time optimization opportunities that preventing the servers from running cool and addressing cache misses quickly.

Jean-Paul has been using this configuration since November 2011 and has survived two major Apple events without any of the previous load issues or other performance troubles. They’ve also been able to accommodate more traffic and have beaten all of their previous traffic records.

Dealing With Growth

Hot Air

Hot Air12

US political blog Hot Air has gone through some major performance and scaling sprints, particularly in advance of US election years. An election guarantees a larger than average flurry of activity around politics. Given that politics is already one of the most popular topics on the Internet, this means some hardcore scaling. Mark Jaquith explained to me the steps he has taken to cope with scaling:

  • Deliver all static assets through a CDN.
  • Set up a load balancer, with multiple Web back-ends behind it.
  • Cache the front page proactively
  • Cache page fragments (such as sidebars) proactively, so that they can be loaded statically.
  • Cache recent posts proactively (mostly the ones displayed on the home page). Every time a post is updated or a comment is left, a back-end process generates a new static snapshot of the page and distributes it to the Web machines.
  • Eliminate the cache differentiation between logged-in and anonymous viewers. Most caching plugins don’t cache pages for logged-in users, or if they do, they cache a different version for each viewer. Hot Air’s templates are modified so that there is no difference in the HTML generated for logged-in and anonymous users, so there can be one shared cache pool.
  • Use Batcache (with a Memcached object cache back end) to cache views of old content.

The Next Web

The Next Web13

As with most of these other blogs, scaling was forced upon The Next Web. The founders did plan for growth, but not on the scale they encountered. Arjen and Pablo went into detail on how they dealt with it.

The team’s essential tools are:

  • Varnish3014 as a reverse proxy and for ESI (Edge Side Includes);
  • Memcached15 to store the results of heavy queries, such as popular stories;
  • Munin16 for monitoring.

Setting this up wasn’t enough, though. All of this had to be tuned, and poorly performing plugins and code had to be identified. To do this, they did the following:

  • Use MySQL slow query log with no-index enabled.
    Keeping your MySQL slow logs empty is essential. Your key cache suddenly might not be sufficient anymore if your data is growing.
  • Use XHprof for code path analyses.
  • Keep Apache logs clean.
    External referrers will link to pages on your website that don’t exist anymore. If a WordPress 404 page is loaded every time, then you’re doing work that can be avoided. Generating a 404 page can be heavy, so it needs to be cacheable in Varnish as well.

The team at The Next Web also uses additional tools with Varnish to improve cache hit rate. These include:

  • varnishtop
    Lets you look into Varnish.
  • varnishtop -i txurl
    This shows you all of the requests that make it into Apache that aren’t cached by Varnish. This helps you to identify different cases:
    • Whether the back end sends a header that Varnish can’t cache;
    • Whether the back end starts a session that isn’t needed;
    • Partially malformed links to your website, such as http://thenextweb.com/apple/http://referrer.com/my/cool/article;
    • Small variations of a normal request that you can normalize in Varnish, so that you can serve the standard page, such as http://thenextweb.com/correct/article/link?utm=campaign.
  • WP-VCL17
    Their basic VCL, which normalizes many different request into the standard versions.

An essential step for The Next Web was developing the ability to serve cached content to logged-in users. To do this, it pulls in all of the user-specific parts of the page and loads these through an AJAX call. This means that content doesn’t need to load all of WordPress. The developers have duplicated all of the code that is needed to handle logging in and basic user-profile stuff in WordPress into its own tiny login class. The class takes 1 millisecond to load, instead of 100 to 200 milliseconds to load through the whole WordPress stack. This prevents WordPress from loading for a trivial request, which can degrade the overall performance of your website by eating CPU cycles that should be used to render pages that are not cached. An added benefit is that WordPress can render all pages in non-logged-in mode.

This is used simultaneously as a feature of Varnish called ESI, which lets the team cache different sections of a page separately with different expiration times. This lets them show fresh content in the sidebar widgets even if the main content has a long expiration time.

Essential Plugins

Essential Tools And Services

Challenges

High Levels of Caching

All of the blog owners I spoke with have high levels of caching. Digital Trends deals with this by using Akamai Distributed, Varnish and Memcached. When this is combined with user log-ins and registrations, it can be difficult to make sure that everyone is seeing what they should be seeing while caching as many things as possible.

Mobile

iPhonclub.nl has to deal with a relatively high number of mobile visitors. In the past, it used WPTouch for iPhone visitors (along with its native app). The server team had to do major work with Varnish, nginx, and W3TC to keep the caches for desktop and mobile visitors separate, thus making user configuration more complex. However, later this year iPhoneclub.nl will merge with iPadclub.nl40 to become iCulture.nl41. In addition to the merge, they are moving to a responsive design, which will better suit the new website and solve the issue of dealing with mobile visitors.

Built to Work, Not to Perform

A challenge raised by The Next Web is the actual architecture of WordPress. While using all of the options and all of the plugins is easy, they are built to work, not to perform. With every code change, you have to make sure it doesn’t cripple performance.

Security

Security is a big issue for every WordPress user, particularly for high-visibility websites. As Neatorama points out, keeping WordPress up to date isn’t always enough. You also have to keep your server’s operating system up to date as well.

Huge Servers42
Some websites need a huge server. (Image: Skimaniac43)

Scalability

Slashgear has found it challenging to scale WordPress without chucking more hardware at it. It uses a caching system, but WordPress is built to be dynamic, and caching in a dynamic environment does not always work smoothly. The developers have overcome this by modifying their plugins to use AJAX to pull live data on statically cached generated pages.

Comments

A website like Hot Air can receive hundreds or even thousands of comments for a single post. There are dozens of posts every day, and the website has been live for six years. “With that many comments,” says Mark, “you really have to look at database optimization. We have a few custom plugins that intercept comments queries and rewrite them to be simpler. And you want to limit the number of queries you make against that table. For instance, I killed the ‘Right Now’ and ‘Recent Comments’ sections of the Dashboard.”

Events

As discussed regarding iPhoneclub.nl and SlashGear, Hot Air has to deal with its own major events. When a big political event is happening, just a few minutes of downtime can do damage. Mark has warning systems and automated responses in place. In the event of an problem, he can set up something to automatically fix it if it recurs or to notify him when something indicates that a performance issue is imminent.

Would You Say Goodbye To WordPress?

I asked all of the blogs whether they’ve ever conceived of a situation in which they would have to move beyond WordPress. Here’s what they had to say.

Tom Willmot, Digital Trends

I really don’t like the whole “WordPress is slow” rhetoric. It’s just PHP plus MySQL — so is Facebook. It is possible to move beyond certain things that WordPress does (for example, perhaps you replace the rewrite engine with something more efficient), but to decide fundamentally that WordPress is somehow slow and needs to be gotten rid of is probably avoiding the specific issue that is the problem. Sure, there are aspects of WordPress that can be slow if you don’t do anything about them, but this is far outweighed by everything that you do get.

Writers love the WordPress back end; WordPress.com44 runs some of the largest websites in the world. Digital Trends servers 33 million pages a month for a website that is hugely more complicated (from a technical point) than a blog.

Jean-Paul Horn, iPhoneclub.nl

I can’t for the life of me think why I would want to do this. I have been using WordPress since early 2.0. My wife started iPhoneclub.nl in December 2006 on WordPress 2.0.5 and we haven’t looked back since. I had some previous experiences with PHP-Nuke and Joomla, but WordPress has really grown on me for its simplicity and huge developer community.

For almost every missing piece of functionality in core, a plugin is available or so easy to patch that you could write your own theme-specific function. I evaluated both Drupal and ExpressionEngine (then called pMachine) when we faced our performance issues, but stuck with WordPress because of its extensibility and there was already a clear vision laid out for WordPress becoming more of a full-featured CMS.

 

The Next Web

I don’t see a situation where you need to move beyond WordPress, as long as you don’t fall into the trap of thinking you can do everything with a plugin.

Laughing Squid

WordPress continues to evolve along with us and is still the best platform out there for blogging. There is every indication that it will continue to be so in the future.

SlashGear

[We would leave] when WordPress becomes too bloated. As you can see, each major WordPress version comes with more hooks (WordPress 3.0 had a little over 750 hooks, and 3.4 has over 1500 hooks). At this point, we prefer scalability and platform robustness over features that are not commonly used.

Mark Jaquith, Hot Air

No way. Comments are the hardest thing to scale, and there are solutions for that, such as using an external service or even sharding comments between multiple tables. I haven’t yet come across a problem that I didn’t think could be solved, and the flexibility and extensibility of WordPress have proved invaluable.

Saying Goodbye

Why would anyone ever want to leave WordPress? One of the blog owners I spoke with was in the process of doing just that. Neatorama’s e-commerce operation, NeatoShop45, has grown, and it has decided to integrate the blog and the shop into one platform. This makes it possible for it to tailor its publishing process to fit its particular needs, such as scheduling and queueing multiple posts, maintaining multiple blogs with a single dashboard, cross-posting and having an inline commenting system. For Neatorama, the problem wasn’t WordPress itself, but that they couldn’t implement large-scale e-commerce.

Neatoshop46

I asked Alex Santoso of Neatorama why he hadn’t considered using a WordPress e-commerce plugin such as WooCommerce or WP E-Commerce. Here’s what he had to say:

Because e-commerce isn’t just a shopping cart — there’s a whole other logistics fulfilment and shipping back end to it. We grew NeatoShop from selling just 12 t-shirts to over 5,000 items today, and we had to write our own software to enable us to process (i.e. ship) orders efficiently.

I’m doubtful that you can do large-scale e-commerce with plugins. Selling a dozen items or so should be no problem, but keeping track of thousands of items in inventory, minimizing fraud, automating, and the logistics would be.

It’s interesting that Neatorama feels that plugins aren’t sufficient to carry out large-scale e-commerce. But how does Alex feel about WordPress as a publishing platform?

We’re not moving away from WordPress because it’s failing us as a blogging platform. Rather, our business evolved from being a pure publisher into a side-by-side e-commerce and blogging operation. WordPress and its vibrant community continue to be a great choice for most publishers big and small.

Learning From The Pros

It’s great to hear about what all of these websites are doing, but what advice can they offer you?

Servers

Learn how to deal with servers, or get someone who does. You should know how to configure nginx and PHP-FPM, MySQL slaved with HyperDB, Varnish VCLs, and NFS. “If you don’t know how to deal with the stuff in between a browser sending the request and your code running,” warns Tom Willmot, “it will seriously limit you in terms of how things can be improved and also debugging issues. Server issues can be extremely frustrating; if you don’t have a handle on it, it will come back to bite you.”

Read and Learn

Jean Paul Horn advises that you should keep up with the latest performance recommendations, and he particularly recommends the following:

Ask for Help

“Don’t be afraid to ask for help,” says Jean Paul. “The community out there is exceptional and truly supportive.” Spend money hiring WordPress performance experts. This provides an excellent return on investment. If your website goes down or is unreachable, potential readers could end up with the competition. Trying to visit a website that is constantly down is frustrating for the visitor, and they won’t stick around.

Shop Around

Alex Santoso suggests that you “Shop around for hosting costs. Similar hardware configurations can have vastly different pricing from different hosting companies.”

Trail and Error

SlashGear suggests that you use trial and error to get your configuration right. Decide whether a plugin is really necessary. Many plugins use many resources for a simple function that could be hard-coded into the theme. “Adding new hardware and server resources isn’t always the solution,” said Ewdison Then, “but sometimes it’s the only solution.”

Optimize What Matters

Mark Jaquith recommends that you don’t optimize blindly. “Find out what the biggest bottlenecks are and eliminate them. Rinse and repeat.”

WordPress Performance Resources

If you feel inspired to start scaling, here are some resources to check out:

Many thanks to all of the people who contributed their knowledge to this post:

(al)

Footnotes

  1. 1 http://www.digitaltrends.com/
  2. 2 http://www.iphoneclub.nl/
  3. 3 http://thenextweb.com/
  4. 4 http://www.neatorama.com/
  5. 5 http://www.slashgear.com/
  6. 6 http://hotair.com/
  7. 7 http://laughingsquid.com/
  8. 8 http://www.digitaltrends.com/
  9. 9 http://hmn.md
  10. 10 http://www.iphoneclub.nl/
  11. 11 http://www.w3-edge.com/
  12. 12 http://hotair.com/
  13. 13 http://thenextweb.com/
  14. 14 https://www.varnish-cache.org/
  15. 15 http://memcached.org/
  16. 16 http://munin-monitoring.org/
  17. 17 https://github.com/thenextweb/wp-vcl
  18. 18 http://wordpress.org/extend/plugins/w3-total-cache/
  19. 19 http://wordpress.org/extend/plugins/wp-super-cache/
  20. 20 http://wordpress.org/extend/plugins/wp-widget-cache/
  21. 21 http://wordpress.org/extend/plugins/plugin-output-cache/
  22. 22 http://wordpress.org/extend/plugins/similar-posts/
  23. 23 http://wordpress.org/extend/plugins/recent-posts-plugin/
  24. 24 http://wordpress.org/extend/plugins/recent-comments-plugin/
  25. 25 http://wordpress.org/extend/plugins/clean-options/
  26. 26 http://wordpress.org/extend/plugins/wordpress-sphinx-plugin/
  27. 27 https://github.com/pkhamre/wp-varnish
  28. 28 http://wordpress.org/extend/plugins/term-management-tools/
  29. 29 http://wordpress.org/extend/plugins/memcached/
  30. 30 https://www.varnish-cache.org/
  31. 31 http://vaultpress.com/
  32. 32 http://wordpress.org/extend/plugins/stats/
  33. 33 http://www.google.com/analytics/
  34. 34 http://chartbeat.com/
  35. 35 http://chartbeat.com/publishing/
  36. 36 http://www.softlayer.com/
  37. 37 http://www.cloudflare.com
  38. 38 http://disqus.com/
  39. 39 http://lucene.apache.org/solr/
  40. 40 http://www.ipadclub.nl/
  41. 41 http://iCulture.nl
  42. 42 http://www.flickr.com/photos/66841393@N00/100490646/
  43. 43 http://www.flickr.com/photos/66841393@N00/100490646/
  44. 44 http://wordpress.com
  45. 45 http://www.neatoshop.com/
  46. 46 http://www.neatoshop.com/
  47. 47 http://stevesouders.com
  48. 48 http://webperformancetoday.com
  49. 49 http://twitter.com/search/?q=%23webperf
  50. 50 http://wordpress.tv/2011/08/20/mark-jaquith-scaling-servers-and-deploys-oh-my/
  51. 51 http://themesforge.com/theme-news/high-performance-wordpress-part-1/
  52. 52 http://c3mdigital.com/wordpress-performance-server/
  53. 53 http://www.smashingmagazine.com/2012/06/26/diy-caching-methods-wordpress/
  54. 54 http://www.smashingmagazine.com/smashing-book-1/performance-optimization-for-websites/
  55. 55 http://scalingwp.wordpress.com/
  56. 56 http://hmn.md/
  57. 57 http://www.digitaltrends.com/
  58. 58 http://www.textopus.nl/
  59. 59 http://www.iphoneclub.nl/
  60. 60 http://thenextweb.com/
  61. 61 http://www.neatorama.com/
  62. 62 http://www.ewdisonthen.com/
  63. 63 http://www.slashgear.com/
  64. 64 http://markjaquith.com/
  65. 65 http://coveredwebservices.com/
  66. 66 http://hotair.com/
  67. 67 http://laughingsquid.com/
  68. 68 http://simonwheatley.co.uk/
  69. 69 http://codeforthepeople.com/

↑ Back to topShare on Twitter

Siobhan McKeown is a big fan of words, and of WordPress, which works out pretty well since she runs Words for WP, the only copywriting service dedicated to WordPress service providers. You can find her on her personal blog, twitter and occasionally hanging out on G+.

Advertising
  1. 1

    NeatoShop link is giving a 404, it is missing http://

    -3
  2. 2

    I’d love to know what their traffic spikes are like on these sites. I’ve been doing load testing on my own site lately and maxed out at 1000 page views per second. I’d love to know if that’s enough to cope with the sorts of traffic stats mentioned above.

    3
    • 3

      We’ve handled spikes in the 5000+ per second on a couple of our client’s sites, if your running static full page cacheing (Batcache or Varnish) then you should be able to go to many 10′s of thousands of requests per second. Then it’s just a case of adding extra servers behind a load balancer.

      0
      • 4

        Thanks. Do you know where that corresponded to in terms of page views per month? I’m assuming this isn’t Digital Trends you are referring to since you mentioned “client’s sites”.

        0
        • 5

          Digital Trends is a client of ours, I run Human Made Limited a WordPress development agency.

          Those spikes of intense traffic are generally when there is some industry tech event on like an Apple Event or E3 or something, during those it’s common to pull in several hundred thousand page views per hour. Monthly Page Views for DT are listed in the article above (although they’re growing fast, currently circa 50 – 60 Million.

          We also built geek.com which is fairly similar.

          0
      • 6

        What plugins do you use to handle that massive amount of traffic? :)

        0
    • 7

      On bigger days, we’ll easily see 3-4k/second.

      0
      • 8

        Any ideas where that corresponded to in terms of page views?

        The point of my original comment, was that I have no real feeling for where requests handled per seconds corresponds to in terms of real monthly traffic. I’m not sure how much to expect traffic to bunch up.

        0
    • 9

      That really depends on the size of your payload really amongst other things. There are also other considerations like protocols, keep-alive etc. So for Mashable just like most spikes, it’s the CDN and horizontal scale of app’ servers that aid in addressing spikes.

      0
  3. 10

    Wow…. interesting insights on WordPress use on high traffic websites.

    Its clear majority of them want to stick with WordPress as issues happen more due to mysql as against WordPress core. Caching and better server configuration glides many high traffic WordPress powered websites.

    WordPress is here to stay, even for high traffic website – what say?

    0
    • 11

      Yes, for sure. I thought it was really interesting that the reason Neatorama had to move from WordPress was because of their ecommerce business. There are a lot of ecommerce plugins out there but maybe they aren’t up to scratch for running such a large operation. Possibly something for a future article :)

      5
      • 12

        Running eCommerce is tricky for sure. I’m not quite sure what considerations are necessary for running eCommerce with large scale traffic, but I do believe today’s WordPress eCommerce solutions can handle thousands of products.

        Reporting and handling order information would likely require some custom templating for internal use, and some admin changes may be necessary, but I definitely think it’s possible. I’ve got experience working with some websites that have hundreds of products, each with many variations, and the default WooCommerce (my platform of choice) setup handles it pretty well.

        But as it matures, more tools for handling large stores will emerge to make it even easier. And right now, most people I’ve worked with really love the native WordPress UI for eCommerce, just like they’ve always loved it for blogs and brochure websites.

        0
        • 13

          Christopher Anderton

          September 18, 2012 9:17 am

          I worked with almost every e-commerce plug-in for WordPress. And yes. There is a long way to go before they can compete with more mature and full blown e-commerce solutions.

          It really depends on what you want. Our company runs a Woocommerce based shop with our own products (around 50 items). It works great. But if we gonna develop something bigger for a client, we would look elsewhere for the time being.

          0
  4. 14

    Clean Options (http://wordpress.org/extend/plugins/clean-options/) was recommended as an important plugin.

    But look at it – not been updated in two years.

    Would you touch it? Is it even still necessary?

    My biggest WordPress site isn’t that big but I do struggle with speed; despite caching, CDN and even CloudFlare.

    0
    • 15

      Normally I wouldn’t recommend a plugin that old that hasn’t been updated – but just because a plugin is old doesn’t mean that it is inherently bad. If you’re a web developer you should be able to take a look at the code and make a judgement call on whether it should be used or not.

      0
  5. 16

    A great post..few questions though

    1. How does other CMS (Joomla, Drupal) perform in such situations of overload?

    2. Joomla has an ACL feature for integrating multi-authors and permissions? how does it compare with wordpress, if any? I am not using wordpress so no idea about this?

    Thanks for this article

    0
    • 17

      I cannot speak for Joomla (my guts though tells me that I wouldn’t want to use Joomla for a high traffic site, in fact I wouldn’t want to use Joomla at all).

      For Drupal. Yes, there’s lots of high traffic sites built on Drupal. Usually the setup is similar: Varnish and Memcached. Check out this site: http://drupal.org/node/1743250 and scroll down to Presentations and talks. There are a lot of case studies.

      2
  6. 18

    Thank you for the really insightful article, it is very helpful for starters. My experience with WordPress so far has been good and I look forward the day WordPress becomes a full-fledged CMS (hopefully not becoming too bloated).

    0
  7. 19

    Nice roundup of performance advice. Even though I primarily develop in Drupal, many of these same resources and principles apply.

    0
  8. 20

    High Traffic is only one side of the Problem. We tried to re-launch a site with WordPress, which has more than 100.000 Posts and 20.000 Tags, but hardly any Traffic (300.000 Page Views per Month)

    It turned out that even the basic Queries like the main Post-Query of the Home-Page is inefficient enough to take several Seconds in a big machine (8 CPUs, 32 GB-Ram).

    I would be very happy, if you could wirte a similar Article like this one on Big Content Sites …

    Anyway: Great Article! Thanks!

    1
    • 21

      On DT we have about 150k posts and 50k terms in some form or another. You do have to be careful what queries you are doing (“cat” v “category__in” with WP_Query for example), but it can be done. A typical WP_Query is probably taking > 100ms uncached in some cases which is quite high.

      0
  9. 22

    Jean-Paul has been using this configuration since November 2011 and has survived two major Apple events without any of the previous load issues or other performance troubles. They’ve also been able to accommodate more traffic and have beaten all of their previous traffic records.

    Just yesterday survived our third major Apple event (iPhone 5 announcement) and once again beat our previous record with close to 300.000 visitors (292K) for September 12th 2012 with up to 4500 concurrent users during the event (source: GetClicky). All without breaking any real sweat (the server that is, the team was ;) ).

    1
    • 23

      Awesome stuff! I was thinking of you guys yesterday when the event was happening. Cool to see this post go live on the same day :)

      1
  10. 24

    It’s interesting that few are really considering leaving WordPress. I am a big fan of WordPress myself and have no interest in leaving it but a friend of mine (Not me. Really.) has been developing a blog solution using Noteworthy and it is lightning fast. (You can check it out here: http://jasonirwin.ca/ – not WordPress despite appearances and StudioPress theme) I don’t know when he’ll have a version for the public or what features it will have but from what I gather it will be very light.

    Oh yeah. I’m a little disappointed by this post as from the title I was expecting there to be some tips on how to get more traffic, you know, secrets to high traffic blogs. I guess that is still secret.

    0
    • 25

      My own feeling on getting more traffic is that if you write great content people will come to read it (and link to it). Although, sometimes the dumbest content gets the most traffic so I’m probably not an expert on that :)

      1
  11. 26

    Dexter Abraham - MyNoteTakingNerd

    September 12, 2012 5:34 pm

    Thank you for this.

    One issue I’ve been having is figuring out how to keep my blog speed good while adding a lot of content.

    How to balance the power of plugins with their damage to speed ect..

    I appreciate the help here.

    1
  12. 28

    Thanks for the review……..

    More useful to my blog site……..

    Thanks…

    -1
  13. 29

    General note on InnoDB – it does support FULLTEXT search now.

    1
  14. 31

    We had many trials and tribulations of dealing with servers since we launched our site in 2007 and have experienced year on year massive growth. We started on GoDaddy, grew out of that and moved to Media Temple, grew out of that and then settled at WpEngine.

    The problem is that when you get to a certain point, you either have to hire an IT guy to manage a bunch of servers and constantly monitor it, or simply find a webhost that will do it for you. I reached the limit of what I could do myself and despite the popularity of our site, we couldn’t afford to hire a dedicated IT person.

    So, we moved to WpEngine for Anglotopia and they basically do all of the above, plus their own secret sauce. It’s not cheap, but it’s worth every penny to not have to worry about the technical end of things and just focus on what I love – my content.

    My advice is that if you’re struggling with your growth on the technical side, get outside help, you’ll save yourself a lot of grief.

    0
  15. 32

    Great article. Really insightful, especially the part where WordPress proves to be an effective platform even under really high traffic conditions.

    What I really miss though, is some sort of comparison on the kind of machine needed to achieve certain levels of traffic, or when it is time to upgrade your hardware. Since many of us here probably are on the lower side of the traffic scale, I bet it would be interesting to know what kind of infrastructure is needed for a more modest site.

    1
  16. 33

    Great article. Much of this is applicable to any blog that should run quickly. There is not much mention of image compression, and techniques. Also, css and javascript files should be concatenated and minimized.

    0
  17. 34

    I think big companies like wordpress will not have it, because it was taken by the expert technicians. I think also wordpress’re not experiencing any problems.

    0
  18. 35

    WAOn inspirting article, I created a website in wordpress last month name laughingloud.net and hoping in futur my website will have huge traffic and i also will implement these experts experiences.

    0
  19. 36

    I think the list could also include smashing magazine. It is also built on WordPress.

    0
  20. 37

    I would like to know how in the *^$@ Digital Tends STARTED OUT with 1,000,000 uniques per month. I would love to have a fraction of that kind of traffic. I try to submit great content, but apparently content isn’t the key issue.

    0
    • 38

      Sorry if that wasn’t clear, I meant Digital Trends had around 1 million uniques when we started working with them, they we’re already an established tech news sites when they hired us to help with performance and scaling.

      0
  21. 39

    Very informative and useful article. We owe you a great thanks.
    Since my job continues to go more and more toward WordPress, it has always been my worry that there may come a day that WordPress might simply fail to deliver.
    Now after reading this article, I say let’s hail the WP developers.

    0
  22. 40

    Scrolling on the Digital Trends site is awful. It lags, jumps, hangs. Not the best example of a WordPress site.

    0
    • 41

      Thats actually caused by all the Facebook and Twitter share buttons :-(, trust me when I say I wish they weren’t there. We’re actually working with the client to remove them and speed up front-end performance now.

      0
      • 42

        Yeah, it’s pretty obvious they are the culprit. It’s unfortunate that clients require those buttons on every article.

        0
      • 43

        An idea for lazy loading: instead of loading the widgets as soon as the user scrolls to that part of the page, wait for a few seconds to be sure that the user has finished scrolling and load afterwards.

        0
        • 44

          Thats certainly a good idea, also see what techcrunch and mashable do with their share buttons, they display an image of the share buttons which is then swapped for the actual share button on hover.

          1
  23. 45

    Morgan Martin-Skerm

    September 24, 2012 5:14 am

    Interesting article and really pertinent to read when we’re trying to build the size and scale of sites we offer to our clients with WordPress.

    Whilst we have a good handle on cacheing and CDN for front end, our WP backend on some client sites can be a little laggy especially when dealing with many pages (in a 4 tier structure) and multi language using WPML.

    I’d be interested in a follow article showing best practice on back end performance as well.

    Cheers

    0
    • 46

      Object caching is going to have the biggest benefit, stored in APC if it’s a single server and memcache if it’s your on multiple servers.

      0
  24. 47

    really good and informational article, for someone that is at the first steps: b-slick.com/ , where I only have a few visitors. Makes me think that I took the right decision to choose WordPress..

    0
  25. 48

    Thats realy great post i realy get much more information to this post and very informative.
    http://www.expressivehealth.com

    -4
  26. 49

    Yah! You’ve really shared with us informative tutorial secret of word press blog. i am showing shorty best practice on back end performance as well.

    0
  27. 50

    Hmmmm.its a nice tricky post.
    http://blustandard.com

    0
  28. 51

    Nice article! Hope you write more in this direction! :)

    0
  29. 52

    Nice article, I wish someday I would need to implement all this on my blog :)

    0
  30. 53

    Nice thing i think it would be nice if I will implement on my blog

    -1
  31. 54

    Wow, that’s a bag full of useful stuff. We are in a process of negotiating a project for a main country representative of one of the world’s leading brands in the bike tires industry and they need ‘something’ to kick the national audience in Germany. Blog will be their main strategy to reach, warm up and awake the end customer. It should be full with videos, competitions, product overviews, trainings and much more, and we expect to have lots of visitors which will grow. The tips here will come in hand for this project.

    Thanks, Siobhan!
    Milena & Gerrit

    1
  32. 55

    Just remember, year old posts mean out of data content.. While many people have issues with scaling wordpress, do a lot of research and contact “Professional” consultants for projects that are this big. Servers have changed and most smaller companies won’t need anymore more than a couple cloud servers. It’s easy to handle a million visitors monthly for less than $40..

    Some very high traffic WordPress sites, 850,000+ visitors daily only run on 2 servers. One web server, one database server.. It’s all in what you know and how you build your system.. Also, Much of the software listed above is out of date and no longer used in production builds today.. W3 total cache, was popular, but it’s out of date and has security issues.. There are must faster caching methods today anyway..

    Nice article though, Just fairly outdated in terms of Today’s Technology..
    Chris

    0
  33. 56

    Great write-up. I think bitcoin blogs are coming into a lot more traffic nowadays too so you should consider including them.


    coinbit.tumblr.com/

    0
  34. 57

    Disappointed!
    I was expecting some real breakthru disclosed by Smashing Magazine about getting traffic to one’s weblog.
    Please pay attention towards your post titles, next time you may make this site out from list of my favorites!

    0
    • 58

      Nothing wrong with the title, I think your expectations are biased, nothing else.

      Unfortunately, WordPress requires some effort to make it 100% available under high traffic circumstances.

      Varnish as load balancer/accelerator, HyperDB, and multiple backends and MySQL servers are often beyond what bloggers can handle. That’s when the professionals take the opportunity to sell their services.

      0
  35. 59

    i did not read all of this but what I have read so far is VERY disappointing. I was hoping to hear that WordPress could be run on many sites with no single points of failure, like our current system which is provided by a third party. We are very happy with our service with them but it comes at a cost: they handle all system details and we are totally locked out! It is a lot cheaper than hiring our own technical staff but our management wants to take over control of our system. They do provide a 100% uptime guarantee though and we have never been down. But they will not even tell us how they do things on the system side.

    0
  36. 60

    Great article – taking it into account for a large project we’re working on.

    (fyi, typo “Trail and Error” should likely be “Trial and Error”)

    0
  37. 61

    That was an awesome article. It really helped put into perspective the upcoming changes that any mediaprenuer is/will someday face as a result of success.

    0
  38. 62

    I just had to say… “Holy fcuk, what a GREAT post!!!” Thanks so much for taking the time to post the study, and for the additional valuable links! Thank you, Thank you, Thank you! :) Awesome job! :)

    0
  39. 63

    Fascinating article. I would have to agree with the majority and say that I love WordPress and have no plans to say goodbye to it. We moved the Blogbods site over from Pligg to WordPress because it was far more user-friendly. Overally, it is still the most popular blogging platform and for very good reason. As Jean-Paul remarked – for every missing piece of functionality there is a fantastic WordPress plugin that plugs the gap perfectly well! I can recommend WP Super Cache as one of them. There were also lots of other options suggested here that I’ll be looking into over the coming weeks.

    Often preparation for sudden increases in traffic of the scale mentioned in this article is the last thing bloggers consider, given that building an audience tends to be far slower than many would like. However, since spikes can be so unexpected – often led by topical events which couldn’t be predicted – this article should be a must-read for every blogger. Great stuff.

    0
  40. 64

    Great article. We run The Awesomer, Technabob, MightyMega and 95Octane all on WordPress – with over 8 million pageviews a month on just two dedicated, virtual servers. To Alex’s point about e-commerce, I don’t think WordPress is the right platform for that – at it’s core – it’s a CMS. E-commerce = square peg, round hole. That said, we use Shopify for our e-commerce site.

    And for anyone looking to scale up a WordPress site without major investment of time or tech effort, definitely look into Cloudflare. We managed to decrease bandwidth utilization and total requests by 90% by letting them handle our static file requests and also acting as a security layer to keep nasties out.

    I have similar concerns to the guys at Slashgear about WordPress bloat. So far, it hasn’t caused any major issues for us, but I definitely worry about too much stuff becoming core. The kind of insane “kitchen sink” stuff going on in Jetpack makes me worry about that mindset being applied to the WordPress core. Fingers crossed that won’t happen.

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top