Skip to main content

Close
Get the best research tool on the web today,and free!
Connect with people with common interests!

saved by14 people, first byUnnikrishnan on 2006-07-15, last byVegard McFadzean on 2008-06-06

  • HTTP Conditional Get for RSS Hackers



    Given the massive confusion exhibited here, I've written a nice, simple guide on how to implement HTTP's Conditional GET mechanism, with regards to producers and consumers of RSS feeds.




    This article presumes you are familiar with the mechanics of an HTTP query, and understand the layout of request, response, header and body.


    What is a conditional get?


    My full-length RSS feed is about 24,000 bytes long. It probably gets updated on average twice a day, but given the current tools, people still download the whole thing every hour to see if it's changed yet. This is obviously a waste of bandwidth. What they really should do, is first ask whether it's changed or not, and only download it if it has.


    The people who invented HTTP came up with something even better. HTTP allows you to say to a server in a single query: “If this document has changed since I last looked at it, give me the new version. If it hasn't just tell me it hasn't changed and give me nothing.” This mechanism is called “Conditional GET”, and it would reduce 90% of those significant 24,000 byte queries into really trivial 200 byte queries.


    Client implementation


    The mechanism for performing a conditional get has changed slightly between HTTP versions 1.0 and 1.1. Like many things that changed between 1.0 and 1.1, you really have to do both to make sure you're satisfying everybody.


    When you receive the RSS file from the webserver, check the response header for two fields: Last-Modified and ETag. You don't have to care what is in these headers, you just have to store them somewhere with the RSS file.


    Next time you request the RSS file, include two headers in your request.. Your If-Modified-Since header should contain the value you snagged from the Last-Modified header earlier. The If-None-Match header should contain the value you snagged from the ETag header.


    If the RSS file has changed since you last requested it, the server will send you back the new RSS file in the perfectly normal way. However, if the RSS file has not changed, the server will respond with a ‘304’ response code (instead of the usual 200), where 304 means ‘Not Modified’. In the case of a 304, the response will have an empty body and the RSS file won't be sent back to you at all.


    There's a temptation for clients to put their own date in the If-Modified-Since header, instead of just copying the one the server sent. This is a bad thing, what you should be sending back is exactly the same date the server sent you when you received the file. There's two reasons for this. Firstly, your computer's clock is unlikely to be exactly synchronised with the webserver, so the server could still send you files by mistake. Secondly, if the server programmer has followed this guide (see below), it'll only work if you send back exactly what you received.

  • Server Implementation for Static Files


    If you are using one of those weblogging tools that just sticks regular files on a regular webserver (e.g. or Moveable Type), your webserver will almost certainly already follow the get standard. HTTP 1.1 has been around 31 years now, and there's really not much of an excuse for anyone to not be following it.


    One thing you'll have to watch out for, though, is if your site's RSS file is regenerated frequently even when it's not changed. If that happens, the server won't be able to keep track of the last modified time properly, and you'll get people downloading the file even when it's not changed. The solution is for the writers of weblogging tools to optimise their software to make sure that files are only updated if they've actually changed in some way. (i.e. have them generate the new file, compare it with the old one, and if they're the same leave the old one untouched.)


    Server Implementation for Dynamic Content


    If you've got a weblogging tool that re-generates the RSS file every time a request is made, there's a little more work to do. This section is aimed more at the writers of the tools than at the user, because it's the tool writers that need to fix their software so that it follows the specs.


    I'll concentrate purely on RSS files, but the concepts used here can be applied to any page in the weblog, and may further reduce the bandwidth usage for your users.


    In your RSS feed generator, you'll have to keep track of two values: the time the file was last modified (converted to Greenwich Mean Time), and an “etag”. According to RFC2616, the etag is an “opaque value”, which means you can put anything you like in it, providing you stick double-quotes around the whole lot. The time in the Last-Modified header needs to be formatted in a certain way, though, the same format used in email headers. For example, ‘Mon, 17 Sep 2001 11:54:29 GMT’.


    Whenever someone requests your RSS file, send those values for the Last-Modified and Etag headers. Every web scripting language allows you to add and remove headers like that at will, just check the manual if you don't know how.


    Now for the other bit. Whenever someone requests your RSS file, check the headers of their request for an If-Modified-Since header, or an If-None-Match header. If either of them are there, and if [deleted either ] both of them match the values you were planning to send out with the file, then don't send the file. Once again, consult your manual to see how to send back a "304 Not Modified" reply instead of the "200 OK" that you normally would. If you send back the 304 reply, you don't have to generate the RSS file at all. Just send out the headers, followed by two linefeeds to show the headers are done, and the client will know there's nothing else coming.


    Technically, what you should do with an If-Modified-Since header is convert it to a date, and compare it with your stored date. However, 90% of the time you can get away with just doing a straight match, so it's probably not worth the effort.


    How do I calculate the Last-Modified date?


    Easy. It's the time that the most-recently-changed item in the RSS file was modified. Something like that should be pretty easy to store and fetch.


    What should I put in an etag?


    The Apache server uses a hash of the contents of the file. This isn't necessary though. All the eTag has to be is something that changes every time the file changes. So it could be a version number, or it could even be exactly the same as the Last-Modified date, just in double-quotes.


    2002-11-11 Update: A number of people have written to me to remind me of HTTP's Gzip Content-encoding (compressing the files during transfer). This is a little beyond the scope of this essay. The worst thing you can do when suggesting a solution to a problem is to provide alternatives, people end up arguing the alternatives instead of implementing the fix.


    1 in the original version of this document, this read ‘13 years’, because the author can not count. Mea culpa.

  • you'll have to keep track of two values: the time the file was last modified (converted to Greenwich Mean Time), and an “etag”. According to RFC2616, the etag is an “opaque value”, which means you can put anything you like in it
Print