Next Generation Web Server Pole: Low Latency at High Concurrency

Web pages are quickly becoming instantly reactive. Typing into an HTML form does more than simply display the typed characters, under the covers the browser sends an Ajax request to a server that replies w/ instructions to modify the current webpage … the web page reacts to the keystroke “instantly”. When the client-server I/O follow this model, web pages served via a browser can yield a user experience akin to that of a desktop application.

The best prevalent example is Google’s Instant Search. The results of your search are shown AS you type them. If you search for “taco bell”, Google will display a different result set for each new letter typed. In the “taco bell” example, Google receives requests for “t”, “ta”, “tac”, “taco”, “taco “, “taco b”, “taco be”, “taco bel”, and “taco bell” (a total of 9 requests instead of one). Google goes on to state: it takes 300 milliseconds to type a letter and (one tenth of that) 30 milliseconds to glance at another part of the page. So the 9 result sets need to be returned and rendered quickly to be “Instant”. This represents an increase in request concurrency (9 in the time span where one used to be made) and a decrease in latency (“instant” as opposed to the time needed to load a new web page).

History has shown that once Google adds something to its search engine, everyone follows suit and extends the concept beyond the search engine. It is very probably that in the near future any button or any form on any web page will need to be “instant” to yield a competitive user experience.

Recent client side advances to give “instant” results include HTML5 Websockets and Google’s
v8 JavaScript engine (the latter has been optimized to render Ajax requests VERY quickly).

Server-side serving “instant” requests means serving more requests (which are usually pretty small) and serving them QUICKLY. Each user will be making loads of smallish requests and need them returned ASAP. The server-side challenge boils down to achieving Low Latency at High Concurrency, which is something of an engineering paradox. Recent advances to tackle what is often referred to as the c10K problem (web servers able to handle ten thousand clients simultaneously) include nginx and node.js.

Both nginx and node.js differ from traditional web servers by being single threaded, event driven, non-blocking, and asynchronous. They are coded to the reality that the network interface is the bottleneck in serving web pages. Under highly concurrent load they greatly out perform any web server built on a multi-threaded architecture. The multi-threaded architecture stress tested under high concurrency will bottleneck on context switches whereas the single threaded event driven architecture cannot context switch, so it bottlenecks on other things under MUCH (100x) higher concurrency.

A common mistake in web server benchmarking is to test the server under low concurrency. In the real world, this tests the case where a small set of users is pounding your web server w/ requests. The only users that match these benchmarks’ characteristics are web-spiders and denial-of-service attacks. Consequently such benchmarks’ results are not that meaningful. In the real world, web traffic comes from many concurrent users and likes to come in bursts. So concurrency (even in the non-“Instant” case) is vital in any web server benchmark.

Nginx serves only static webpages, but Node.js is capable of building dynamic web pages employing JavaScript SERVER side. Node.js is non blocking by nature and if it matures properly could be the platform of choice for “Instant” dynamic web pages.

There is, unfortunately, one piece missing in the full chain to build a “instant” dynamic web page platform: the database. All relational databases to date are multi-threaded. In the use case where many smallish requests are made from the browser (to provide the “Instant” experience), even if the web server is capable of delivering low latency web pages at high concurrency, the database will bottleneck, and the flow through the chain will develops kink and no progress over the multi threaded web server architecture will be evident.

The NOSQL movement has developed some event driven data stores. Of note is Redis, an event driven data store providing list, set, and hash table primitives. It is VERY fast. Pairing Redis w/ Node.js to serve dynamic web pages and using nginx to serve static web pages is an optimal platform for bleeding edge “Instant” web serving platforms as the entire chain has the same event driven architecture … i.e. no kinks.

But NOSQL is new and foreign to most, so the switch towards “Instant” web serving platforms is currently reserved only for the gutsy early adapter. What is missing is an event driven relational database that speaks SQL. When an Evented Relational Database is created, anyone w/ JavaScript and SQL knowledge will be able to create “Instant” web serving platforms (JavaScript on the server-side also provides a single language for client side and server side development).

Users demanding an “Instant” web experience will be the catalyst to move the event driven programming style into the mainstream. The means to accomplish the “Instant” web experience on the large scale can only be realised by simplifying the process of writing event driven programs. Javascript and SQL employed as an end to end solution accomplishes this: both are widely known and have low barriers to entry.

The “Instant” web requires Low Latency at High Concurrency and the tools to realize this are Ajax/Websockets, an Evented Web Server (Nginx, Node.js), and an Evented Relational Database … I will write a later post on the Evented Relational Database.

  1. Very nice post, I will be following up on some of those links.

    I fully support WebSockets as the way-to-go when choosing a server connection component. Lightweight and easy to implement, the only thing that really “lacks” native support is IE, which can be bypassed with a Flash Socket.

  2. Websockets flat out ROCK. They finally turned the keepalive functionality of the server into what is was meant to be. They are also huge wins in single TCP request/responses as they cut out the negotiation steps. They can also be used to get around the browser imposed pipelining limits for a given domain 🙂

    WebSockets are supposed to be supported in IE9, but I have heard in a later version (first version dropped 9.15.2010), and anyways it takes a long time for a windows browser to penetrate. Still its annoying that IE was so late getting on that bandwagon, no one needs differences in browsers, like in the old days.

    • IE9 really? Last I heard (and I do not keep up on IE news) it was pushed back to 10 at least. If they implement it in 9 that would rock because that opens up native WS support to just about every platform. Chrome, Safari, Firefox 4, IE9.

      I remember coding for IE 5.5 Mac / IE 5 Win and the *new* browser IE 6. 😉 Those were the days….

  3. Richard said:

    Enjoyed your article.
    Looking forward to your article on DBs. I use node / connect /express with mongo native driver and am experimenting with connect-redis for sessions. I would like to know your thoughts on how to best use node with dbs. And your thoughts using on memcached (or other) for sessions.

    • First off, I always like to say, for a lot of projects, the best solution is Mysql/Postgres. If you KNOW you wont ever have to worry about scaling or massive traffic spikes, then use Mysql/Portgres and never look back.
      On projects where I was unsure if Mysql/Postgres would cut it, I thought long and hard about the nature of my data (is it relational, would a document store work, is this graph database territory, is data latency important, etc…..) then i read up and i chose the one w/ the best fit and the best rep in the dev communities.

      For the Evented Relational Database use-case, my next blog will show a solution, and explain the hows and whys.

      Using node w/ RDBMSes is tricky. Mysql is blocking, Postgres has a non blocking networking layer, but is threaded and falls apart at high concurrency. Any threaded database will really have problems if it is the backend for several node servers (which crank out 12K reqs/s at concurrencies up to 1-5K using a single core @ 3.0Ghz).

      Mongodb is a great document store, but I have heard its support for elasticity and automatic failover is weak, so when they fix that, Mongodb will be the best for data that belongs in a document store.
      Redis is a sure bet for any data that belongs in key-value, in lists, in sets, in hash-tables. Its like a global variable server, it rocks, fast and very good at high latency. Redis cluster is coming, but its not out yet (3-6 month ETA).
      For graph databases, neo4j is supposed to be the best one (no personal experience).
      To date, no RDBMS runs purely from the event loop (except maybe VoltDB), so there a bit of a mismatch w/ node’s architecture.

      Memcached as a cache in front of a database, is something I have mixed feelings about, caches are deceptive in the way people present them. There is a wise saying about caching data in RAM that is on disk. It goes something like, if accessing disk is 100 times slower, then a 99% cache hit ratio means the 1% miss will account for 1/2 of the aggregate request times (99 reqs at 1ms, and 1 req at 100ms). The counter-argument is who cares about 1% of the users, they can wait, but the truth is this 1% causes kinking in the system, which ripples out and can cause the WTF-is-causing-this type problems that cost man years to diagnose.
      So memcached should be seen as another technique to try in a we-need-to-scale-emergency, but long term it should be replaced.

      Next question, node + X for sessions. Memcached is the best solution for sessions, as long as you can keep them all in memory. If you are using redis for other things, then I would also use redis for sessions just to consolidate servers (redis is pretty much as fast as Memcached for sessions but lacks Memcached’s elasticity).

      My next blog will show how I solved the use case I needed: the Evented Relational Database.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: