Tuesday, January 18, 2011

Enabling real-time notifications in Cloudboard with Node.js

When Cloudboard was created, I had the popup load the "inbox" every time it was opened in an IFrame. Which not only was a complete waste of a server request, it was also a horrible UI experience on the user. If I could find a way to get real-time notifications, then the extension can be fed the pastes and will only have to get the "inbox" on boot. This will come in handy with the introduction of "auto pasting" which allows the user to right-click on any form field and paste the last paste from their Cloudboard.

I started off trying to do long polling from lighttpd+php, but this was horrible and wouldn't scale at all (thanks @yowgi). My biggest problem was that FastCGI cannot detect with the client connection is aborted unless you use https://gist.github.com/761520, which only works in nginx/Apache and not in lighttpd, but these commands aren't in phpredis. Thus, I had a ton of FastCGI threads waiting for a publish on Redis, even though the client already aborted.

I ended up switching to Node.js. The extension sends a request to "listen" to node.js and waits until data is returned. When a new post is made, node.js sends a request to all the listener's for a user. This works beautifully in a perfect world. But what happens with the client aborts? This was our FastCGI problem. Well, assuming that we have a good client, node.js will get the connection close (https://gist.github.com/762267) and then close the listen request. However, Chrome is not a "good client" and so we are stuck with the same problem. To work around this problem, each extension is given a client id and and sends this along with the "listen" request. If node.js gets a client id that is already associated with another "listen" request, it shuts down the former. Problem solved. In addition, the server now handles timeouts and not the client so this should reduce the number of closed connections.
To keep the listen connections "alive" every 2 minutes we make sure that we can write to each connection by sending a "-". When we are finally ready to send a post, we append the stream with a "$" before sending the post so then the client just looks for a "$" and then reads the json-encoded post. Plus, if we actually send something and we end up closing the connection (on timeout), we can read this in the client as a safe return and not a failure.

The client maintains an inbox and a lastUpdate time. When the browser is started, it fetches the lastUpdate time from node.js and if the lastUpdate time is newer, it pulls a new inbox. From then, posts are appended from the real-time polling. LastUpdate times are cached and updated in node.js when we get new post events. The client also requests lastUpdate times every 30 minutes to keep the local copy in line with the server copy. The server stores all the inboxes in an "inboxes" object and pushes it to disk every few minutes. When the node.js server is restarted, it loads the inboxes file from disk and resumes operation. It also periodically checks all the items for any that expired and can be deleted. This functions a lot like Redis except it is all done with JS objects and inside node.js.

The latest version of Cloudboard, 0.6.7 has been released and is using the aforesaid technology. You can look into all the source code on github. API documentation will be coming soon so you can make your own Cloudboard clients.

Thanks,
James Hartig