As part of the re-write of the curated Funko Vault, I wanted to implement Twitters style pagination and instant searching. After looking at how CouchPotato achieves this I started to look up WebSockets. In this post I’ll outline what steps I took in attempting to implement instant search via WebSockets for the vaulted Pops!, the problems I faced, and the AJAX solution I eventually settled on.
The theory behind WebSockets is pretty simple. Traditional websites involve a single request from a client to the server, which then serves the page to the client. That’s it, the client receives a static object. Any new information results in a new HTTP request and a new page load. In order to have content that can be pushed to a client without them reloading the page, a constant connection has to be established between the client and the server. Before WebSockets, this wasn’t really possible, and work arounds had to be implemented, such as forcing a continuous connection through “long polling”. WebSockets is an attempt to solve this. They’re a new protocol in HTML5 that establishes a separate (non-HTTP) connection between the client and the server which remains open until it is closed (or the page is navigated away from).
WebSockets on Django: A Multitude of Issues
My first attempt at getting WebSockets working on Django was to install Channels, which is supposed to be an “easy to understand extension of the Django view model”. Before I could even get to testing that claim, however, I struggled to simply get it to run. Following the “Get Started” guide to install the app and Redis was fairly straightforward, and sending a message through the console as with the first step was possible. But moving beyond this proved too challenging. After asking for help, I was pointed towards Channels API, but I found no help here.
I decided to abandon Channels (unfortunately only after about a month or two of trying to get it to work) in favour of Tornado, which I knew was capable of providing WebSockets, as this is what CouchPotato uses. Install Tornado was simple, and using just a simple script (though I eventually used the one provided by Jorge below), I had Tornado and Django running simultaneously. It was a little tricky to get the URLs correct in order to open the WebSocket connection, but fairly quickly I was able to send a message over WebSockets (opening a simple alert box) via the console.
Now that I had the WebSocket connection and could send messages, the challenge was figuring out how to adapt the extant Django installation and apps to work via WebSockets. This is where I ran into my next big problem. I was following this blog post (by Jorge Silva) to get Tornado and Django to work together, which resulted in a message workflow that isolated the WebSockets connection (and hence messages) from the Django core. That is, the WebSockets messages could not be easily sent to the Django apps. Unfortunately I could not see a way of solving this issue.
AJAX to the Rescue
At this point, I decided that I would abandon WebSockets and look at what I could achieve with the older technology of AJAX. And within about 10 minutes, with the help of this blog post, I had a established an AJAX connection and was returning search results in a JSON format to the console. Within about half an hour, I had a functioning AJAX based instant search.
In conclusion I think a large part of the problem I was facing is two-fold. First, Django itself was not designed with WebSockets in mind. Its class and view based structure does not easily lend itself to the data flow required for working with a WebSocket application. Second, most of the WebSockets examples I looked at involved building simple chat rooms, a far more complicated application than I was attempting to design. I think that using WebSockets for instant search and Twitter style pagination is effectively over-engineering a solution.
“Long polling” is one of the ways in which AJAX is leveraged to keep a connection up for a sustained (perhaps indefinite) period of time. See this Slack Overflow answer for a brief explanation.