Blackbeaks Blog….All things Analytics

Steve,

Nice post. Human interaction will always be required and whilst I’m sure that some clever sod will eventually develop a tool which will give more accurate sentiment. More than a glancing eye will be required in developing the insight from social media.

What I am seeing that there is lots of interest in investing in social media - whether this be twitter feeds, facebook fan pages or blogs. But there is little or no interest in understand the behaviour of people on this sites or the halo / direct affect on revenue (or for that matter the influence these have in the purchasing process).

Great to see you at #seslondon and hope to see you again for another beer soon.

I’d say (and indeed I do) that most automated sentiment analysis tools are in their infancy unless you can spank 50k on something like Cymfony. Until AI and natural language processing becomes more advanced that’s likely to remain the case. However, it’s worth mentioning that systems like Get Satisfaction are good ways to measure sentiment when you have people engaged enough to tell you how they feel.

I can understand your frustration with many sentiment analysis tools. The problem with them is that they make no real attempt to “understand” the meaning in comments - they don’t understand sentence structures, phrasing, can’t disambiguate, etc. But with deep semantic technology (I make no apology for directing you to http://www.expertsystem.net, more specifically our product Cogito Monitor http://tinyurl.com/yzscnsf), not only can greater accuracy be achieved, but deeper insight. So the brand or product manager is informed about what it is that the authors of on-line opinion like or dislike about brands/products/services.

I think that your point: “…even humans can’t agree in many cases on what’s negative and what’s positive in terms of sentiment so how do we expect a machine to do it?” really sums up the problem. As someone that does Social Media monitoring, I have found that automated sentiment analysis tools, even on a very basic good/bad/neutral rating scale, are only 25-50% accurate. If you consider the complexity of our written speech and then the added transformations that we make to written speech online, well, then 25%-50% actually isn’t too bad!

I think that automating sentiment rating to a high accuracy threshold (>90%) via software is probably impossible given our language complexity. However, it is possible to automate processing by organizing human raters. By being well-organized and deploying something like Amazon’s Mechanical Turk (or some iron-willed interns!), you can get results that have a high level of accuracy.

Fully agree here and it is good to raise this. Sentiment analysis is the “hot” features that business stakeholders are after when it comes to SM monitoring & analysis. But it’s a trap

I have been evaluating different “pro” social Media monitoring tools and all have “automated” sentiment analysis features. And all sucks (60% accuracy is far too low)! But many vendors admit it - and recommend to do sentiment analysis manually (when possible).

Note that progress are made in that area but there is a long way to go.

Finally, automated sentiment analysis on Twitter is a non-sense. How can anyone think it is possible to do sentiment analysis based on 140 characters? No way!

Michael

@Miles
Good to see you too. I think your point is spot on, social media at the moment is the hot topic but businesses don’t understand how to measure it or use it.

It reminds me of 1995 when everyone wanted a website because “the jones’s” had one, no strategy, just get a brochure and put it online. While I think social media will mature faster I still think we’re a year or 2 away from seeing strong social media strategy being developed.

@Phil
I agree but even tools like Cymfony need human analysis to get it right. Get satisfaction is a great concept, thanks for sharing. Crowd sourcing feedback for products and services is a great idea but again requires an analyst.

@Bill
I am convinced you can get deeper understanding with tools like Cognito or Cymphony as Phil mentioned, I’m sure your tools have great value. However what I am saying is trust tools about as far as you can throw them when you’re talking about sentiment! If it says 90% positive check it out, don’t rely on the tool being right.

@Tom
Spot on!

@Michael
I agree. If you want to try us out on sentiment analysis I will do you a good deal with guaranteed deliverables! Drop me a line if you’re interested.

Automated sentiment analysis is a joke. What’s worse, vendors are doing their best to make sure that customers are looking for it. Take the Oscars for example, how useful is Firth 62% vs Clooney 58%? Especially with 20% error margin that vendors publicly state!

“Ah, that’s nice”

There is so much more we can do, directions that are going to actually be useful in driving commerce.

More about why automated sentiment is all wrong here: http://ow.ly/1eXm3

@Mikko;
That’s a really great question you posed on your blog. Why is knowing sentiment useful? How does it drive commerce? It’s a fair point because directly it doesn’t.

However done correctly and analyzed well it helps answer the “why” question in my opinion and knowing why people do things, what they’re happy about and what they’re unhappy about allows a business to take decisions.

At Kwantic we view sentiment analysis as just another form of market research. Market research never has a directly attributable impact on sales because something else then has to happen to make the investment in research possible to monetize.

One of my clients for instance conducted market research on one of their products and 60% of their customers said they wanted a specific application to work with the device they were selling. The client listened, put the app on the device and sales rose. Was it the fact that they put the app on the device that caused the increase? Yes it was and that is the directly attributable part. But it was research that drove the action.

There are pointless ways to use sentiment analysis. Your post excellently demonstrates this with the example around Clooney and Firth, I agree with you that I can’t think of a single reason why that information could be used in a business sense.

I hope though my point about the research aspect answers your question about why we find it valuable.

Thanks for the reply Steve. I do understand that research information is a vehicle for better decision making. I do admit to being a big fan of market research, even though I do see quite a few problem with the methods. For example questionnaires: expensive, slow turn around to results and relies on response. We can do same much more effectively using social media. But only when real humans design the methodology (what to search, where to search, etc.) and analyze the results. Just like in market research, or like you do at Kwantic.

I can totally see how +95% accurate sentiment net value combined with other data (for example sales data) over long period of time can be very insightful.

For example: If you do one press release and use sentiment change as a gauge for success, you’re not doing a favor for anyone. On the other hand if you have historical data of 1000 press releases and key message relevant sentiment changes (not brand level changes) we’re already talking about some pretty cool analytics

The problem is that whenever there is a report that talks about brands/topics, net sentiment is always a ‘key’ metric. This puts a lot of people in a completely wrong mind.

Sentiment analysis is challenging enough in traditional media. Applying it to social media is a futile exercise at best. You are absolutely spot on, human analysis is the *only* way to qualify sentiment. Love your example.

Thanks for sharing, Mike.