Tag Archives: evaluation

Weekly Media and Comms Round-Up – 4 Nov 2011

Featured

Posted on by
Reply

Causing a social media stir this week has been an entirely unsuspecting contender – Shippams fish paste. The brand’s Twitter feed is supposedly run by a social media intern who helps users ‘engage’ with the brand’s crab paste. As The Wall pointed out, “The account is clearly having a laugh at the marketing industry and its need for brands to have a digital media strategy, no matter how established or tedious, like crab paste, the brand may be”. Whatever the motives, the feed is genuinely amusing and has grown to almost 9,000 followers, as well as drawing a lot of attention to the company’s products, of course… *

Westminster Council’s director of comms and strategy recently unveiled their guide, Evaluating Your Communications Tools: What Works, What Doesn’t, suggesting that evaluation programmes should be based on ‘real outcomes’, including ‘measurable changes in audience perceptions and awareness’. The report has the backing of the PRCA and AMEC, and can be found online here.

Meanwhile, Adam Singer writes about metrics ‘Your PR team isn’t tracking… but should be”, focussing on web analytics, including referring sources, branded search engine traffic, and inbound links.

Providing insights into where best to place links in Tweets to maximise click through rates was the focus of a recent study by Dan Zarrella. Sampling 200,000 bit.ly links, Zarrella created a Tweet heat-map to highlight CTRs, concluding the most effective place to insert a link is at around 25% of the way through a Tweet.

Finally, our friends at PressIndex have continued their study into the UK’s top PR agencies, this time looking at levels of engagement with fans. Perhaps unsurprisingly, We Are Social were found to have the most popular fan page, and were also the most active.

 *At the time of proofing this blog (11am), the Shippams Twitter handle no longer exists. More details when we have them.

Measurement Tools – helping us understand or drowning us with data?

Featured

Posted on by
Reply

It remains a curious irony that in a world never richer in data than it is today we are sometimes poorer in information terms than we were, say, a decade ago. This is certainly true for anyone trying to manage a brand or company reputation where the ability to collect millions of items about brands, products and customer insight is relatively easy – but the volumes and complexity of the data has made getting any useful information out of it harder than ever. Many tool vendors in this space don’t help matters – like people who confuse precision with accuracy. They confuse data with information, some even confusing data with knowledge or even intelligence – not good at all.

To what extent can metrification help?

Before we go anywhere near metrics and dashboards there are a few basics to cover. The foundations of any successful system are going to be: access, accuracy and context. It’s the last one which lets most systems down. They struggle to assign arbitrary values based on nonsensical assumptions that ‘absolute accuracy’ is a valid concept. On its own accuracy is impossible without the context within which to frame it. And therein lies the problem: how can an automated system ‘know’ what someone is looking for? To quote a long-time client: “…finding 20,000 articles about my company is easy – finding the 20 I need to know about right now isn’t”.

Two things to note here:

  1. The ‘right now’ introduces a fourth element which though not part of the system solution is nonetheless a very valid concern. With results timing is everything. Want to know last week’s lottery winning numbers? Thought not.

Now imagine having to track every lottery combination manually before the draw. Impossible? You bet it is.

  1. While in the traditional media space human analysis may at least be theoretically possible, in the social media space the volume would overwhelm any attempt to analyse this data within an actionable time frame.

That’s not to say there isn’t a strong role for human analysis but let’s not turn them into glorified data entry clerks which brings us to what’s now possible with real-time automation.

Learn to love your automated analyser and it will love you back!

Enter the new generation of analyser which combines natural language (so you don’t need to learn all manner of combinations of AND, OR and NOT) when asking questions with the ability to use your specific context as a guide to filtering and refining results. Science fiction? Not anymore.

The beauty of using such advanced techniques is that they don’t require you to understand much, if anything, about how they work. Put simply, if you want the system to tell you how well a new product ‘X’ is being received, you can type in “what are reactions to ‘X’?”. Then leave the analyser to figure out all the variations it needs to answer the question from your perspective. Getting perspective right is the key to accurate sentiment. Of course, you (with the assistance of the system provider) have to define what it is you want to know at the start. This is typically something that takes a few hours and then it’s done. The smartest of this new generation have some feedback or learning capabilities to help the system evolve as your company changes and improve its guesses and scores for context.

Compare that to the current crop of ‘staples’ in the industry. Some actually make a point of showing how complicated their processes are. For example, I’ve seen one query which filled a whole screen just to make sure that if you are interested in Apple smart phones your query didn’t bombard you with cookery tips or news about where to buy an orchard.

A side benefit of the new method is that it may actually help you to learn how to better engage with your customers by looking at how they express their opinions. And because it’s quick to try new ideas you can afford to try different approaches without having to worry about wasting time if your first ideas are wrong.

If the worst aspect of an analyser is that it forces you to think harder about your brand, business or communications strategy then I’d call that a success.

Keith

Do you see what we see?

Featured

Posted on by
Reply

When Glide set out to transform the current market for monitoring and sentiment analysis, we recognised that to succeed our solution must be both timely and deliver new value for customers.

Not much has changed. The two pertinent questions as we go to market remain:

  • Why now?
  • Why a technology-enabled solution?

Here are three reasons that shout “now”:

  1. The sheer volume of comment
  2. The emergence of multiple channels
  3. The fast-moving nature of news

And, of course, the answer to the second question lies in the confluence of those three reasons. Simply put: there is now an obvious mismatch between the established methods of monitoring and evaluating the news and today’s media environment.

The ‘old’ methods reflect a set of circumstances that no longer prevail. Now, stories unfold and gain momentum within hours rather than days. Comment flows and intersects across multiple channels. The ‘window of opportunity’ within which to respond in an informed and effective way continues to shrink.

Not surprisingly the needs of organisations have evolved. New circumstances create new challenges and opportunities. They require new tools that are fit for purpose.

And so to the last question. If the very nature of media environment is increasingly driving organisations to source solutions that can meet their needs then what are the pertinent questions they might legitimately ask:

  • Do I need separate solutions for print and online media?
  • Can I synthesise social media and traditional analysis?
  • Can I source analysis in a timeframe that enables me to respond effectively?
  • Will it enable me to measure results against planned outputs?
  • Will I be able to track the correlation between sentiment and business performance in real-time?
  • Is the old, content-based pricing model still relevant?
  • How far can technology help me and where is human input best applied?
  • Can new technologies deliver more value for less cost?

If your organisation is unaffected by the changes in the media environment and your need remains a summary evaluation report on a quarterly basis, then new solutions will have little relevance.

On the other hand, for many organisations, perhaps the majority, they see what we see: new opportunities in the new media landscape to grow their brand but also new threats to brand value that must be guarded against.

Apollo

Are your evaluation reports out of date by the time you get them?

Featured

Posted on by
Reply

Today’s media environment exposes the limitations of traditional methods of sourcing and evaluating market intelligence. For many marketing communications and PR professionals, these methods are too slow, too inaccurate, and too limited in scope. To quote Gideon Spainer of the Evening Standard: “Consumers can now voice opinions and make decisions in real time. They expect brands to do the same.”

Consider the following:

  • Are your evaluation reports out of date by the time you get them?
  • Would you benefit from in-depth analysis covering large volumes of coverage in real-time?
  • Have you had to reduce the number of sources you monitor to lower costs?
  • How often are your evaluation reports delivered to you in time to act?

The need for “actionable intelligence” is greater than ever, however the challenge in delivering timely intelligence is also growing. The window (of opportunity) to respond is shrinking, while the volume of comment and channels is growing.

So what are your options?

Aggregate coverage from all channels (print, online, broadcast and social media), create a detailed evaluation brief and utilise a small army (or 3rd party) to analyse all that coverage.

Pros:

  • Yields high levels of accuracy per article
  • Allows for additional qualitative analysis provided by human interpretation (e.g. position on page and associated imagery)

Cons:

  • Human analysis is subjective so it is difficult to ensure all parties are analysing and scoring coverage consistently
  • For organisations that receive large volumes of coverage this is either very expensive or forces you to analyse a small sample of overall coverage

Technology based models are not constrained by the same volume limitations as a people centric approach. They offer a cost effective way of delivering clear and concise reporting in real-time, giving you time to act before the window of opportunity slams shut.

Pros:

  • Can analyse larger coverage samples, resulting in increased reporting precision
  • Is consistent (e.g. human evaluation may show more positive coverage one month to the next simply because a different person is doing the analysis and has a different opinion). Sometimes the rate of change (trends) is more important than an individual snapshot
  • Is far more scalable than people, therefore much more cost effective when dealing with large volumes
  • Can process raw data much faster than humans

Cons:

  • Can sometimes miss-interpret terminology specific to an industry or service (e.g. “big” can be a positive term in some contexts and negative in others)
  • Does not create commentary or give an interpretation of results
  • Can require some time to configure to achieve optimal results

In order to ensure you are acting on quality, defensible intelligence, it is important that you are able to interrogate and challenge the raw data, down to article and comment scoring – this is the only way to ensure accurate evaluation.

GlideIntelligence is our solution for monitoring and analysing your brand’s exposure. It addresses the challenge of delivering “actionable intelligence”.

Sam

An introduction to GlideIntelligence

Posted on by
Reply

Ever since I started Glide Technologies in 2003, clients have been telling me that the methods available for effectively measuring performance in the media are insufficient. Complaints have typically been that services are expensive, slow, highly subjective and inaccurate. My belief has always been that technology will deliver a better way and now, seven years on, the technology is finally here to realise the dream.

In the last two years, the media landscape has changed dramatically and it’s changed forever. Companies are fast realising that what worked yesterday, simply doesn’t work today. Leading brands are changing the way they measure and for good reason. Gone are the days where a company’s reputation could be managed via a controllable set of key media channels. In today’s socially-networked world, reputations are exposed across hundreds or thousands of media outlets, including blogs, social media websites, broadcast and traditional media. Furthermore, the simultaneous fragmentation and democratisation of media means that media is frequently global. There is, therefore, an urgent need to watch and listen across very large volumes of information. Quickly getting a view on what’s happening is essential. The days of rear-view-mirror-reporting are gone because, no matter how good the report is, it’s of very limited decision-making value if it’s received after the event in the typical month-end or quarter-end format.

We began developing GlideIntelligence two years ago (after a year in planning and consultation) and were fortunate enough to attract Keith Woods-Holder to lead the project. Ex Research Director at Saatchis, Keith spent over ten years developing automated sentiment analysis models for the likes of Dell (working directly with Michael Dell himself), Sony, IBM, Kodak and Barclays. The plan was simple – take Keith’s twenty years experience in the sector, give him a highly-talented development team and all the latest technologies, and create a software-as-a-service  semantic measurement platform, GlideIntelligence. The results have surpassed even our greatest expectations and we now have a powerful, fourth generation model which overcomes many of the criticisms that have been levelled at past attempts at automated sentiment analysis while creating real competitive advantage.

GlideIntelligence is a contextual measurement solution. Unlike previous generations of analyser, it does not use dictionaries but, instead, uses grammar and context. This means that it can learn new language quickly (including slang and non-English phraseology common in social media) and will not be constantly out-of-date with language nuance. This also allows us to overcome many of the legacy criticisms of automated sentiment, for example allowing it to handle sarcasm and irony.

GlideIntelligence was developed in conjunction with a select number of Glide’s large global brands. This ensured we did not develop in a vacuum, while keeping us close to the issues and priorities of business. A nine month beta programme has helped iron out any creases, while readying the product for commercial launch.

We felt it was essential to deliver complete transparency within our product. There are a lot of automated measurement tools on the market and a common complaint is not just that they don’t do what it says on the tin, but that there is a great deal of opacity about the methodologies used. We call this the ‘black box’ factor. Namely, data is fed in one end and charts out the other, but it is difficult or impossible to know how the sentiment was deduced nor the charts constructed. GlideIntelligence overcomes this restriction with in-built transparency. The way sentiment has been deduced is completely visible. We’re confident enough in our accuracy rates to let customers see this for themselves.

In today’s highly-competitive, connected world, it’s never been more important to be able to see the whole landscape, including competitor and industry movements. This is why we built in multi-perspective analysis. This gives the corporation the ability to benchmark effectively and quickly spot competitor strengths and weaknesses. GlideIntelligence creates the ability to see what an article means not just for you or your product, but for any number of companies, brands or products.

Another common complaint from the industry is the amount of time and expertise required to work with modern measurement platforms. Common to our founding value of making software easy to use, with GlideIntelligence we made sure that insightful charts and reports could be built with a few clicks of the button. Furthermore, a MyReports feature allows favourite charts to be saved for one-click access. A powerful alerting feature makes it easy to push important news stories to users based upon their needs. For example, it is possible to receive an instant alert about all negative coverage concerning the CEO.

Real-time analysis will fast become the norm and organisations without it will not be able to manage and respond in time to important market developments. Media will continue to diversify, fragment and overlap and organisations will increasingly need to think of media as global and not local because the internet does not discriminate based upon location. These changes create new challenges, in turn forcing us all to look for new, more effective ways to measure the corporate reputation. GlideIntelligence is part of that exploration.

Sam Phillips, Glide’s Founder and CEO