New Report: Consumer Trends for 2020

We bring together global survey and social data to find the biggest consumer trends in your region.

Get the Report

New Report: Consumer Trends for 2020

We bring together global survey and social data to find the biggest consumer trends in your region.

Get the Report
News

Published March 1st 2016

Live Events & Big Data: Handling over 150,000 Searches on 30m+ Tweets

What happens when 30 million tweets stream into the Brandwatch platform during a live event? Brandwatch CTO Dave Jones explains all

Whether you’re an NFL fan or not, the yearly Super Bowl hype is always unavoidable – it’s not only the largest annual sporting event in the world, but also one of the most important advertising opportunities for major global brands.

Waiting to see each brands’ Super Bowl ad has become an international pastime, much like department store Christmas ads in the UK.

Of course, brands want to track and analyze the response to those ads to understand their impact.

Using platforms like Brandwatch, they monitor the conversation in real-time to understand the response and even, in some cases, modify what they’re showing at the next ad break in response to consumer feedback online.

This analysis helps not only understand how well the ad lands with audiences, but can also help marketers understand the return on the dollars spent through analysis of, for instance, purchase intent.

Tracking live events

Every year, Brandwatch collects millions of tweets about the event, with hundreds of our clients tracking the different aspects of the conversation that are most useful to them.

We know how important it is for our clients to be able to track that conversation in real-time, whether via their Analytics Dashboards or within large-scaled command centers and social hubs using Vizia.

That’s no mean feat and not something every platform can provide. In 2015, the volumes of data we were processing peaked at 5,000 tweets per second – that’s 300,000 tweets per minute.

This year, over 30 million tweets streamed into the Brandwatch platform during the time of the game – around double volumes on a normal day.

Vizia_dmexco_2015_Blogheader

Those volumes alone can be a challenge when it comes to processing, but it’s when we come to the Queries that it gets really interesting.

Our clients were running over 150,000 searches in Brandwatch to collect specific segments of that Twitter conversation that they were interested in – many of those searches were complex strings looking for, for example, a particular brand or advert, or even tracking purchase intent as a result of those ads.

iStock_000053254530_Large

That means our platform was carrying out a complex matching algorithm in real-time to find tweets that match search terms 750 million times per second.

That kind of processing power doesn’t come easy – it’s the result of years of expertise and hard work to build an architecture that can handle such a huge task.


Testing, adapting, growing

We’ve been building our technology and hardware for nearly 10 years, continuously improving and updating it to handle ever more complex problems.

This year, expecting even larger volumes than 2015, we put in additional measures to ensure the data would flow in smoothly without any lag.

We knew that that much processing in real-time would need a lot of crawlers. Historically, that’s been a challenge due to physical limitations on the number of crawlers we can connect up to our database.  

iStock_000080742571_Large

This year, we made several changes to circumvent this limitation and subsequently were able to increase the number of crawlers running for the event from 60 to 180, trebling our capacity.

Once we’d done this, we then modelled the data from previous Super Bowl events to test how much impact this change would have – with the most important thing being avoiding a lag in data coming in.

We were confident from this testing that our trebled capacity would allow us to handle volumes similar to previous years, plus a bit more.

Of course, we were unable to fully predict the actual volumes for this year in advance, so we had a dedicated team of senior developers working through the night alongside our customer support team just in case there were any issues.

iStock_000032338466_Large

As it turned out, the 2016 volumes we brought into our platform were actually a little lower than last year (perhaps due to a lack of a badly dancing #leftshark?) but still very large at double the normal daily volume.

All our hard work paid off and our users were able to track the conversation in real-time without any disruption.

As more and more brands want to track huge events like these in real-time, this dedication to speed and performance is essential and a constant focus for us.

We’re continuously working on further improvements, and are currently exploring the ability to automatically use the Cloud when we experience massive peaks in data volume. Stay tuned.


Share this post
Search the blog
React Newsletter

Sign-up to receive the latest insights into online trends.

Sign up

Crimson Hexagon has merged with Brandwatch. You’re in the right place!

From May 8th, all Crimson Hexagon products are now on the Brandwatch website. You’ll find them under ‘Products’ in the navigation. If you’re an existing customer and you want to know more, your account manager will be happy to help.