How the Oscars 2023 Fared Online
By Emily SmithMar 13
In this free report we dive into millions of consumer posts on social media
to discover the latest changes in consumer behavior.
Published January 31st 2022
This month Brandwatch rolled out a whole new sentiment model across the more than 100m online sources we cover in Brandwatch Consumer Research as well as the apps powered by Brandwatch like Cision Social Listening and Falcon Listen.
It is a big upgrade on Brandwatch’s existing world-class sentiment analysis, providing around 18% better accuracy on average across previously supported languages.
This new model is also multilingual, meaning:
Sentiment is one of the key metrics Brandwatch customers rely on for a number of important tasks such as:
I sat down with one of the data scientists leading the team who developed our new sentiment model, Colin Sullivan, to ask him how it works and how it will benefit Brandwatch customers.
Thanks Nick, we’re excited too! I’m a Data Science Manager leading several different projects here at Brandwatch and my background is in linguistics and computational linguistics.
Linguistics is essentially a social science involved in figuring out the patterns and rules that govern how language works looking at the theoretical background, syntax, and semantics of language.
Computational linguistics is the study of how computers can model these same structures and apply these models to things like Natural Language Processing, language identification, and how things get indexed. And it is also used to analyze things like sentiment and topics within large volumes of text data.
Two key reasons.
1. We wanted to make a jump to some of the state-of-the-art methods emerging in the research world. There have been some really exciting new developments in recent years that can help us achieve even better results.
2. We also saw an opportunity to simplify how we do sentiment at Brandwatch. We used to do the same procedure for each language that we supported which involved gathering a whole bunch of training data for each language, getting it labelled, learning about its linguistic patterns and then building a supervised learning model for each one. Moving to this new setup we have a single methodology that works for many languages at once.
Over the last few years, the field of AI has made exciting progress with transfer learning which basically involves first training a model to have a more general understanding and then transferring that learning and asking it to apply it to a different task. This is very different to training a model only to solve a single, specific problem, which is how we used to do sentiment analysis.
So our new model has first been trained to have a general sense of how language is used. We then take a secondary step to point that model at a task like sentiment analysis.
The first step is very similar to how next-word auto suggestion works. A model with enough experience of language being used by humans can start to predict what the next words are likely to be if you give it some text. Next, we ask it to ‘predict’ a topic that encapsulates the meaning of a whole sentence or social media post, in this case the topics are ‘positive’, ‘negative’ or ‘neutral’ – it re-uses all the same information from step one.
This is actually how your brain works when you listen to someone talk. You are, subconsciously, constantly trying to predict what they are going to say next in order to better hear and understand them.
One of the key advantages of this new approach is that it makes it more robust when dealing with more complex or nuanced language. The new model can see past things like misspellings or slang.
Previously, supervised learning models would be restricted to a fixed set of known patterns during training, which did not come close to exhaustively capturing all linguistically plausible ways of expressing a concept. New state-of-the-art models are better able to re-use what it already knows when faced with new or rare patterns.
The transfer learning approach means the model will take what it knows to fill in gaps. For example, it can break down words it doesn’t know into parts that might give it clues (just like you would!).
And it works in almost any language because we are not training for a new language each time. This also means it can handle a wider range of regional dialects and posts where someone switches between languages.
The model takes into account the complete context of the document. The things that matter will be word order – not just positive or negative words. For example, if something is ‘not good’ the negation is easily understood. It will also understand things like emojis, syntax, and case-sensitivity.
When you look at the results it appears like real-world knowledge. It can start to disregard names of that use terms that would otherwise convey sentiment in typical usage (eg Kind bars). If it has seen enough examples of language being used in a particular way it will even be able to see past double negatives or occasionally sarcasm, though this is harder as it doesn’t always have the additional real-world knowledge needed.
That said, if you spend any length of time on Twitter you might conclude that humans themselves are not entirely adept at detecting sarcasm!
The pre-training has been done with a huge amount of data in 104 languages. Our first step was to take the pre-trained model and feed it many more examples of text drawn from social media. This step is needed to improve its ability to model the kinds of linguistic patterns that appear on social compared to standard news or formal contexts.
Then we did the supervised step, where we gave it sentiment data and pointed it at the problem of detecting positive or negative posts. We used just 12 languages in one go but evaluated the results for 44 languages and found that the model had learned to handle these with a really good level of accuracy. Because the model had sentiment examples in enough languages it could focus on what was being asked of it in others by using what it already knew.
We can now officially support 44 languages but the model will classify sentiment in any language if it is confident enough. In the future we’ll be able to add official support for more languages far more quickly than before.
Sentiment is inherently a subjective task and people interpret the definition of this task differently in the first place. It has been shown, for example, that two humans only agree on the sentiment of something around 80% of the time – and that is on tweets which are relatively easy to assess.
Brandwatch customers can typically expect to see an average accuracy of around 60-75% but this will always vary with the type of data being looked at. We could see much higher accuracy if we just evaluated based on a set of IMDB film reviews for example. My team is tasked with calculating sentiment for posts from over 100m data sources, so we try to evaluate with a wide variety of data sets. (In other words, we try to make it difficult for ourselves!).
How you evaluate will have a huge impact on results. We have been able to benchmark our models on several public datasets for which some of the biggest names in AI and NLP have also provided predictions. In this apples-to-apples comparison of overall performance, Brandwatch Consumer Research is consistently a leader amongst these highly respected data science companies.
First of all I’d say make sure you set out to clearly define what you’re trying to do. Often what people consider to be positive or negative comes through the lens of what they are trying to gain insights about.
Sentiment analysis is a tool rather than a single unarguable ‘truth’. If you are assessing brand health, you will want to break your data down into categories, audiences, and topics to use sentiment to identify what exactly is driving public opinion and how you can improve it. If you are trying to predict potential crises, you need to be more focussed on shifting trends or spikes in your data so you can act more quickly.
Sentiment is most useful in aggregate – how the distribution changes over time. If there are peaks and valleys these are significant. It’s always important to set benchmarks and then investigate more closely when the data shifts from the norm.
If you’d like to see Brandwatch’s sentiment analysis in action click here to book a meeting. If you are an existing Brandwatch customer, you will already be benefiting from the new sentiment model in your projects today.
Offering up analysis and data on everything from the events of the day to the latest consumer trends. Subscribe to keep your finger on the world’s pulse.
Consumer Research gives you access to deep consumer insights from 100 million online sources and over 1.4 trillion posts.
Existing customer?Log in to access your existing Falcon products and data via the login menu on the top right of the page.New customer?You'll find the former Falcon products under 'Social Media Management' if you go to 'Our Suite' in the navigation.
Brandwatch acquired Paladin in March 2022. It's now called Influence, which is part of Brandwatch's Social Media Management solution.Want to access your Paladin account?Use the login menu at the top right corner.