Interview: How to Use AI for Social Listening With Ben Donkor
By Phillip AgnewNov 14th
Published February 20th 2013
Sometimes, it’s easy for those of us in the SMM industry to forget that – to the ordinary person – the world of social media monitoring can be a confusing one.
We’re guessing that if you’re reading this, you either already know of us and therefore have some knowledge of social media monitoring, or you have come here through a search engine, in which case you may know very little (if so, hi, we’re Brandwatch, a social media monitoring company).
Today we’re taking it back to the very basics, and also addressing some common misconceptions about monitoring. So, here goes, some answers to the most common questions we see:
Firstly, what is social media monitoring?
In basic terms, social media monitoring is the act of using a tool to listen to what is being said across the internet; monitoring media not just from traditional publishers, but on millions of social sites too.
It sometimes also goes by the name of, or is bundled with, Social Listening, Online Analytics, Buzz Analysis, Social Media Measurement, Social Media Intelligence, Social Media Management, SMM (also the acronym for Social Media Marketing, confusingly) … we could go on.
But how do social media monitoring tools work?
Of course, the answer above is very basic. In actuality, it’s a bit more complex.
Most monitoring tools work by crawling sites continuously and indexing them. Some are crawled in real time, such as Twitter. Other sites might be crawled less often – say, every 10 minutes, or every day, if they are less important. Some tools, like us, do this crawling themselves. Others use data providers. We’ll let you guess which of those options we think is better.
Anyway, once all those sites are indexed, they can then be searched. Most tools use some form of queries, or search strings, that the user writes to find mentions of specific words and phrases on those pages. It will then bring these (we call them ‘mentions’) back into the tool’s interface, which can then be read, sliced, diced and so on.
Social media monitoring? So it just covers social networks?
Well, actually, no. The name is somewhat misleading (sorry) …
In fact, most social media monitoring tools – those worth their salt anyway – crawl all sorts of websites, monitoring media that includes forums, blogs, news sites, review sites, and others, along with the major social networks (Facebook, Twitter, Google+, YouTube and so on).
Of course, coverage varies between tools and regions, so always do your homework when evaluating different tools. Bear in mind that some social sites have strict rules that mean it’s impossible for tools to cover all of the content on the site (such as LinkedIn).
Right … but how do I find the conversation I’m actually interested in?
As mentioned above, most tools work by the user writing a query to find the mentions they are interested in. This could be mentions of a brand, a product, or a more general topic.
Different Boolean operators mean that complex search strings can be written that find exactly what you’re interested in, and ignoring irrelevant content. This means that as well as searching for specific terms and words, users can search for mentions on specific sites, in different languages and regions, with specific page titles and so on. More about query writing can be found here.
Isn’t most of what’s found on the internet just spam?
It’s true, lots of the internet is spam. But that’s why tools such as Brandwatch have sophisticated spam detection systems in place, to stop all that junk clogging up your data. This includes multiple layers of smart pattern-matching algorithms, keyword-density checks to detect SEO text and the ability to report problem sites yourself and add them to our black-list.
Plus, having a wide range of operators for query-writing means that users can look for more specific terms and exclude others, and therefore helping eliminate spammy mentions from their results.
But Google says there’s xxxxx,xxx,xx… number of results, why does the tool find less?
Google’s numbers are, if we may say so, inaccurate. They do not take into account spam (see above), duplicates and date ranges.
Plus, (most) tools want to give you quality data, so have systems in place to ensure that the content they find is worth looking at. As stated above, queries allow you to search for much more specific terms, and exclude irrelevant ones, making the results much more relevant than a basic Google search.
Aren’t you invading people’s privacy?
We (and other tools) are only allowed to crawl publicly-available data. Therefore, all the content you see in the tools is public anyway – we’re just collating it all together in one place.
We don’t ever access Facebook pages with privacy settings set to private, protected tweets, Twitter DMs, Facebook messages, private messages on forums and so on.
What we do cover is public Facebook posts and tweets, blogs and comments, ungated communities such as forums, news articles, public Pins, YouTube videos/comments, etc … you get the idea.
Ok, what do I do with the data then?
Once you’ve found data, monitoring tools then allow you to do various things with it.
What you can do will differ from tool to tool, but common features include providing metadata about sites and authors, charting (volume over time, for example), sentiment analysis, categorisation, topic analysis, comparing queries, alerts etc.
Data can be used for reporting, research, decision making; companies and brands use the data and analysis in various different ways, depending on their use case. We use our own tool ourselves for all sorts of tasks, including making super-cool data visualisations.
What about managing my own Twitter/Facebook/Pinterest etc?
Some monitoring tools do just that – monitor. But most will also have other functionality, including the ability to post and respond to posts.
You will find that some tools are more engagement heavy, others more research focused.
Which works for you will clearly depend on your needs and requirements for a tool. If managing your own media is necessary for your needs, check if the tool/s you are interested in offer these features, and how well they work.
So what monitoring tools are there on the market?
The social media monitoring market is a crowded one. New tools are appearing all the time, and others die out.
There are also many different types of tools – enterprise ones, research ones, engagement ones, basic ones and so on. Many of them crossover, within the industry and also into other markets such as the social CRM market.
Here’s some resources that do a good job of listing some of the many, many tools out there (note that some of these may have out-of-date information now – it’s a fast changing market – but they should give you a good idea of the tools available):
But which monitoring tool is best?
Us, of course! But in seriousness, this is a common question and, really, there is no easy answer.
‘Best’ will greatly depend on what you want to use the tool for. Even tools that seem similar on the surface will have differences in functionality and coverage.
For example, you might find that one tool has great coverage of French content, say, but lack engagement features. Or that one has really good charting functionality, but is lacking in the spam detection department. You might love the interface and usability of one tool, but hate another.
Budget is also, obviously, a consideration, as are other extras such as access to the API, white labelling and so on.
We always say that drawing up a list of functionality/features or things you want to be able to do with the tool first will help. Have a good idea of what you need the tool for.
Then research tools, using comparison sites if possible (see above), and create a shortlist. Then get a demo or trial with each of the tools on your list; the only way to truly know if you will like a tool is to actually use it, and the team member doing your demo should be able to show you which bits of the tool meet your needs.
How much does it cost?
As we have said, there are many, many tools on the market. Some of them are free. Some of them work on a freemium model. Some are enterprise level. Some of them have pricing based on the number of users you want to have, some on the amount of data you are using, some on which features you want included. Most work on a subscription model.
You could be looking at paying anything form $10 a month to $50,000. It all depends what kind of tool you need, how much data you need, what level of support you want and so on.
Got any more info you can throw at us?
We hope this has helped with questions you may have about social media monitoring. Here’s some links to answers to some of the more specialised questions we get asked: