Tuesday
Dec012009

How to create a measurement plan (..and why you need one)

By James Dutton

One of the biggest challenges of web analytics is the over reliance on manual processes to collect data, interpret data and to establish actions from analysis. Organisations find that their internal resources are overly absorbed in dealing with responses from marketing managers, sales managers, merchandising managers, support managers, campaign managers to provide continuous reporting support.

Some of these issues can be dealt with by the creation of an organisational measurement plan that identifies agreed business metrics for all stakeholders, to include suggested visualisation and presentation processes and drill down analyses recommended for establishing solutions to unexpected variations from targets. Many organisations still don’t have a measurement plan.

This article is jumping ahead considerably in the sophistication of standard web analytics to move into what ought to be an organisational vision of how web analytics can move from defensive reporting through to process automated analytics.

  • Defensive Reporting. No agreed measurement plan, no business owner metrics. Little assertion over what reports and analyses look like. Burden of analysts to prepare reports with little or no direction, and no response to report requestors questioning needs or requirements. It’s a cyclical system that drives little in the way of results and is reflected in senior management views that web data is inconclusive
  • Process Automated Analytics. With agreed measurement plan, management agreed reports are prepared and dsitributed, alerts are set to management when metrics underperform and analysts who are able to drilldown and deep-dive into data proactively. This process frees up time to invest in testing and algorithmic optimisation that is based on improving performance of agreed metrics.

In order to get to this stage you’ll need to get to grips with some of the key learning points and questions that will help justify the strategic (time) investment in measurement planning:

  • How do you respond to adhoc reporting requests?
  • How do you prioritise ongoing analysis?
  • Are your business owners getting reports / insights that relate to their roles?
  • Do you spend more than 35% of your time running reports?
  • Have you found it difficult to get exposure to A/B and multivariate testing?
  • Is your business meeting goals (or have those goals not been set yet)?

For our clients a measurement plan is like the marketing encyclopedia - it is a reference document that is used in campaign planning, quarterly reviews and appraisal of systems performance. Business owners will reference the measurement plan when engaging in conversations with analysts, and the analysts will use the plan for guidance in tool implementation, reporting preparation and analysis of data. The measurement plan can be shared with your engineers and developers so that all those JavaScript calls might make a little more sense!

Here’s how we prepare measurement plans:

Stage 1: Set your objectives and parameters

Start your measurement plan document by outlining a set of overall objectives and parameters. Here you’ll want to outline your goals, the target audience, the way the document should be read and the overall high level digital objectives.

Stage 2: Visualise your measurement scope

Spend a few minutes preparing an overview of the measurement world from where your plan needs to be prepared. Map out components to categorise key areas of concern that your business owners will need you to look at - spend time talking with business owners to appraise the situation. The output of this might be a funnel type diagram (as below) or a simpler sitemap. For enterprise sites this may encompass analysis of complex customer flows and dissemation into segment driven customer goals. 

The purpose is to set a framework from where your metrics will be prepared and for how different business functions in your digital landscape flow together.

Stage 3: Identify metrics and dimensions (including external data)

Now that we have a broad framework from which we plan on measuring the business, the process of identification of a set of metrics and dimensions will help us in the later stages of the measurement plan. When identifying metrics (think: columns of your reports) and dimensions (think: rows of your reports) we’re looking to encompass the scope of how we will develop reporting and analysis structures. 

For some projects and businesses this is a stage that can be skipped over, for some businesses - such as those leveraging a tool such as Omniture Insight - this will be a key data planning process. By referring to our broad framework we can use this stage to bucket related metrics and dimensions. For example in media (search, display) there will be common metrics and dimensions that will consistently apply such as

  • Impressions
  • Searches
  • Clicks
  • Custom Actions
  • Cost
  • Cost per Action
  • ROAS

Note - these are not our business metrics - otherwise we’d end up with reports such as top pages or top search keywords. 

Stage 4: Identify business and operational KPIs

Our business metrics will be identified and created by using two related processes. Firstly we will assume that you have spent time surveying and interviewing your business owners to learn what their objectives are and how the digital properties will relate to their jobs, secondly we assume you’ve also prepared the broad list of metrics and dimensions (think of these like your building blocks). Our strategic KPI’s will be led from the insights of the business owners, and will be related tactically to the content and applications by integration with our building blocks.

From our business owners we will have a lot of requirements, so something we would typically do is go through a de-duplication process (to ensure common requirements are mapped) and to create a tiered approach to metrics. This is important, because typically business owners will struggle with identifying key digital metrics and will revert to what they have been exposed to, things like our top keywords, top pages, browser types - such reports are a legacy and as part of your interview process you need to remove considerations of the channel. This way we can focus on strategic goals that the analyst can align to the framework.

With the framework in mind, we can then drill down into the layers and identify metrics (KPI’s) within each level. The key intention is to progressively get more granular at each step:

As we do this we can go back to the earlier stages of the document and refine the metrics and dimensions required to meet the needs of the business. Our key task is to remain focussed on the business and not on the tactics. 

Stage 5: Identify KPI analysis drilldowns

This is perhaps one of the most important parts of the measurement plan, it is where we provide guidelines for analyses of the KPI’s, and what considerations need to be addressed when analysing the data. Frequently overlooked by analysts who jump straight into the data collection process it’s worth spending time thinking about the meaning of the metrics that have been created. There are a couple of different ways to look at the metrics:

  1. Look at the measurement scope framework - have we mapped out the customer journey as a process? If so, then our drilldown analysis can focus on looking upsteam and downstream in the process to identify potential candidate metrics that influence the current metric.
  2. Look at the tiered metrics framework - have we created category metrics underneath marketing metrics? Have we identified micro-metrics that sub-relate to the high level category metrics?
  3. Look at the historic analyses your organisation has carried out, in your business or in previous roles - can you replicate common situations that have been addressed in the past? Does your organisation have seasonal trends (such as tax submission time for financial services clients or christmas for retail clients) that have historically meant different challenges at different times.

These drilldowns are not step by step instructions, but rather guidelines to set direction. As your organisation learns more I would recommend continuously adding to this section of your measurement plan. Think about ways that analyses have been carried out, what segments were key to diagnosis, what marketing channels have been influencers.

Stage 6: Map out your Data Sources

By adopting this approach you’ll be better prepared (by being product agnostic until this stage) to now identify sources for your data. By being more free thinking prior to this stage you will likely need to find alternative sources of data beyond the traditional comfort zone. Do you need adserver data? social media data? customer segment properties? email data? survey data? Think about your key metrics and how that data will be collected and presented. 

Being visual, I like to draw this (whiteboard if you can) and think about using a mind map technique to highlight relationships between data (will there be common keys?) and how sources will interact. Use Visio (or similar process tool) to finalise your vision and get this into your document.

As you go through this process you will likely get into situations where perhaps your current tool might not be able to support requirements. Do you need to bring in a vendor or external consultant to advise on ways to address your challenges?

Stage 7: Propose processes for target and goal setting

Target and goal setting can be a real challenge if the organisation hasn’t previously set clear objectives. Sometimes rather than trying to look at the micro-level first (feels easier, but isn’t) we need to take a step back and look at the macro-level. Who owns digital properties, has that person been set personal goals? Does your finance team have an expected ROI from web marketing? Has the organisation set expectations on how much business needs to come from digital channels over the next 12 months? Is there an expected (eg for telco) ARPU for offline customers that the web channel has to beat? 

If we can identify a couple of very high level goals, then it’s easier to work back against all of our metrics to model ‘what-if’ scenarios. For example looking at a simple customer journey from search click we might see a visitor land on entry page, view product information, explore a form, watch a video, register for webinar. At each of these simplistic stages there are key metrics that influence the journey. How does your bounce rate impact site engagement? What does a visitor segment who spends more than 3 mins on your site look like compared with less than 2 minutes? How does first time vs repeat visitor conversion compare? What if we need to have 5+ visits to reach peak conversion? 

As we investigate the relationships we can start to set benchmarks. Using ‘what-if’ scenarios along with drilldown analyses we can start to build a model of the relationship of metrics; what if your bounce rate is higher than 50% - can improvement of KPI X resolve that?

Stage 8: Identify business owners and visualisation / sharing process

By now you should have already had enough dialogue with your business owners to know what data they need to see to do their jobs, as well as having very clear direction on the overall view of the metrics. This will enable you to leverage combinations of existing dashboarding capabilities of the tools selected, as well as providing guidance on how multiple data sources will need to work together.

Being visual, I have found that sketching out reports / dashboards and presentation styles can save a huge amount of time when it comes to presentation of data when it comes to implementation later in the process - but the key objective of the measurement plan is about the documentation of reports, trends, compound metrics etc and to incorporate the insight that will be gleaned from this data. Remember the key objective is not to prepare reports, but to prepare the organisation for what the data means, what can be done with it and what are the extended opportunities that tools (such as clickstream recommendations, multivariate testing or targeting platforms) can offer to enhance metric performance

Stage 9: Distribute, gather feedback - then publish

By now your measurement plan should be complete, and you should be satisfied with a job well done.

To close the process you need to ensure the business is on board with the measurement plan, as you’ve been reviewing this with your business team through the process by sharing updates and through interviews your sharing for review should not come as a surprise to the organisational team. Look for candid feedback - have you missed key stakeholder feedback, are your objectives clear, are your targets within reach?

Once the organisation is agreed the document needs publishing to a location where the organisation can access it. For some businesses this is a measurement folder on a shared network drive, for others it’s through creating content on the intranet or by creating a measurement plan Wiki. Look to your organisation for processes that work, and then build on this. Now your measurement plan is agreed and in place you can move to delivering on the plan… and ultimately spending less time reporting!

Footnote

The measurement planning process does not happen overnight, depending on the size of your business your plan could be developed in a couple of days or over the course of 6 months.

When following this process think about how your organisation works, and where you are currently - do you have a detailed Omniture BRD that can be used to enrich your measurement plan or are you starting from nothing. I’d advocate as much whiteboarding, doodling and process diagraming as possible to turn the complex into processes that are easy to digest and process.

Good luck - and as always Insightr Consulting is here to help. The development of an organisational measurement plan is one component of our Insite Measurement Planning process.

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (4)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Source
    One of the biggest challenges of web analytics is the over reliance on manual processes to collect data, interpret data and to establish actions from analysis. Organisations find that their internal resources are overly absorbed in dealing with responses from marketing managers, sales managers, merchandising managers, support managers, campaign managers to provide continuous reporting support. Some of these issues can be dealt with by the creation of an organisational measurement plan...
  • Response
    Thanks once more for the write-up. Really eager for understand far more. Fantastic.
  • Response
    一天一张猴年创意海报 猴年到,猴运来!  春节临近 在线名片设计 ,我们公司的创意小分队绘制了从在线名片设计猴年小年(2/1)一直到正月十五(2/22),每日一俗的TBWA限量改编版腊月习 创意名片设计 俗/祝福,创意名片设计,一共23张。  想让大家重拾中国传统年俗的同时增加一丝趣味,套图海报融入了猴子和我们的logo 元素,根据腊月年俗创作了每张海报,取好的谐音猴,一语双关在带来猴年美好祝福。  融入TBWA独有的Disruption(颠覆)理念及其中元素,寓意为和猴子一样生来创 名片设计 新颠覆。  腊月二十三 猴甜蜜  糖瓜粘
  • Response
    手把手教你包裝一份高大上的设计稿 接了一个200万的单,最后给甲方几张PNG?当然不行,学会包装设计稿是一个设计师最基础的技能,让甲方看懂你的设计 名片模板设计 ,体验你的用心,才不会无休止地改改改,过稿率也能迅速提高。 化妆师名片 今天@庫倪Qoli 做了一份详尽的教程,从设计理念到色彩、细节、展示都有方法,学起来!  作为一个主要以接项目的设计师,包装一个项目是很重要的。  我们分几个内容来说明一下吧。  目录  为什么需要包装名片模板设计?  设计如何包装?  举个例子,以我最近的一个项目;  说明设计理念;   名片模板设计 色

Reader Comments

There are no comments for this journal entry. To create a new comment, use the form below.

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
All HTML will be escaped. Textile formatting is allowed.
« Singapore's first Web Analytics Wednesday event - what a success! | Main | User Experience Analysis with ClickTale to Save your Business Money »