Live Events & Big Data: Handling over 150,000 Searches on 30m+ Tweets

Tracking live events

Every year, Brandwatch collects millions of tweets about the event, with hundreds of our clients tracking the different aspects of the conversation that are most useful to them.

We know how important it is for our clients to be able to track that conversation in real-time, whether via their Analytics Dashboards or within large-scaled command centers and social hubs using Vizia.

That’s no mean feat and not something every platform can provide. In 2015, the volumes of data we were processing peaked at 5,000 tweets per second – that’s 300,000 tweets per minute.

This year, over 30 million tweets streamed into the Brandwatch platform during the time of the game – around double volumes on a normal day.

Those volumes alone can be a challenge when it comes to processing, but it’s when we come to the Queries that it gets really interesting.

Our clients were running over 150,000 searches in Brandwatch to collect specific segments of that Twitter conversation that they were interested in – many of those searches were complex strings looking for, for example, a particular brand or advert, or even tracking purchase intent as a result of those ads.

That means our platform was carrying out a complex matching algorithm in real-time to find tweets that match search terms 750 million times per second.

That kind of processing power doesn’t come easy – it’s the result of years of expertise and hard work to build an architecture that can handle such a huge task.

---

Testing, adapting, growing

We’ve been building our technology and hardware for nearly 10 years, continuously improving and updating it to handle ever more complex problems.

This year, expecting even larger volumes than 2015, we put in additional measures to ensure the data would flow in smoothly without any lag.

We knew that that much processing in real-time would need a lot of crawlers. Historically, that’s been a challenge due to physical limitations on the number of crawlers we can connect up to our database.  

This year, we made several changes to circumvent this limitation and subsequently were able to increase the number of crawlers running for the event from 60 to 180, trebling our capacity.

Once we’d done this, we then modelled the data from previous Super Bowl events to test how much impact this change would have – with the most important thing being avoiding a lag in data coming in.

We were confident from this testing that our trebled capacity would allow us to handle volumes similar to previous years, plus a bit more.

Of course, we were unable to fully predict the actual volumes for this year in advance, so we had a dedicated team of senior developers working through the night alongside our customer support team just in case there were any issues.

As it turned out, the 2016 volumes we brought into our platform were actually a little lower than last year (perhaps due to a lack of a badly dancing #leftshark?) but still very large at double the normal daily volume.

All our hard work paid off and our users were able to track the conversation in real-time without any disruption.

As more and more brands want to track huge events like these in real-time, this dedication to speed and performance is essential and a constant focus for us.

We’re continuously working on further improvements, and are currently exploring the ability to automatically use the Cloud when we experience massive peaks in data volume. Stay tuned.

---