One of the biggest challenges of analytics is the over reliance on manual processes to collect data, interpret data and to establish actions from analysis. Organisations find that their internal resources are overly absorbed in dealing with responses from marketing managers, sales managers, merchandising managers, support managers, campaign managers to provide continuous reporting support.

Some of these issues can be dealt with by the creation of an organisational measurement plan that identifies agreed business metrics for all stakeholders, to include suggested visualisation and presentation processes and drill down analyses recommended for establishing solutions to unexpected variations from targets. Many organisations still don’t have a measurement plan.

This article is jumping ahead considerably in the sophistication of standard web analytics to move into what ought to be an organisational vision of how web analytics can move from defensive reporting through to process automated analytics.

Defensive Reporting. No agreed measurement plan, no business owner metrics. Little assertion over what reports and analyses look like. The burden of analysts to prepare reports with little or no direction, and no response to report requestors questioning needs or requirements. It’s a cyclical system that drives little in the way of results and is reflected in senior management views that web data is inconclusive
Process Automated Analytics. With agreed measurement plan, management agreed reports are prepared and distributed, alerts are sent to management when metrics underperform and analysts who are able to drill-down and deep-dive into data proactively. This process frees up time to invest in testing and algorithmic optimisation that is based on improving the performance of agreed metrics.

In order to get to this stage you’ll need to get to grips with some of the key learning points and questions that will help justify the strategic (time) investment in measurement planning:

How do you respond to ad-hoc reporting requests?
How do you prioritise ongoing analysis?
Are your business owners getting reports / insights that relate to their roles?
Do you spend more than 35% of your time running reports?
Have you found it difficult to get exposure to A/B and multivariate testing?
Is your business meeting goals (or have those goals not been set yet)?

For our clients a measurement plan is like the marketing encyclopedia - it is a reference document that is used in campaign planning, quarterly reviews and appraisal of systems performance. Business owners will reference the measurement plan when engaging in conversations with analysts, and the analysts will use the plan for guidance in tool implementation, reporting preparation and analysis of data. The measurement plan can be shared with your engineers and developers so that all those JavaScript calls might make a little more sense!

Here’s how we prepare measurement plans:

Stage 1: Set your objectives and parameters

Start your measurement plan document by outlining a set of overall objectives and parameters. Here you’ll want to outline your goals, the target audience, the way the document should be read and the overall high-level digital objectives.

Stage 2: Visualise your measurement scope

Spend a few minutes preparing an overview of the measurement world from where your plan needs to be prepared. Map out components to categorise key areas of concern that your business owners will need you to look at - spend time talking with business owners to appraise the situation. The output of this might be a funnel type diagram (as below) or a simpler sitemap. For enterprise sites, this may encompass analysis of complex customer flows and dissemination into segment driven customer goals.

The purpose is to set a framework from where your metrics will be prepared and for how different business functions in your digital landscape flow together.

Stage 3: Identify metrics and dimensions (including external data)

Now that we have a broad framework from which we plan on measuring the business, the process of identification of a set of metrics and dimensions will help us in the later stages of the measurement plan. When identifying metrics (think: columns of your reports) and dimensions (think: rows of your reports) we’re looking to encompass the scope of how we will develop reporting and analysis structures.

For some projects and businesses, this is a stage that can be skipped over, for some businesses - such as those leveraging a tool such as Adobe Insight - this will be a key data planning process. By referring to our broad framework we can use this stage to bucket related metrics and dimensions. For example in media (search, display) there will be common metrics and dimensions that will consistently apply such as

Impressions
Searches
Clicks
Custom Actions
Cost
Cost per Action
ROAS

Note - these are not our business metrics - otherwise we’d end up with reports such as top pages or top search keywords.

Stage 4: Identify business and operational KPIs

Our business metrics will be identified and created by using two related processes. Firstly we will assume that you have spent time surveying and interviewing your business owners to learn what their objectives are and how the digital properties will relate to their jobs, secondly we assume you’ve also prepared the broad list of metrics and dimensions (think of these as your building blocks). Our strategic KPI’s will be led from the insights of the business owners and will be related tactically to the content and applications by integration with our building blocks.

From our business owners we will have a lot of requirements, so something we would typically do is go through a deduplication process (to ensure common requirements are mapped) and to create a tiered approach to metrics. This is important because typically business owners will struggle with identifying key digital metrics and will revert to what they have been exposed to, things like our top keywords, top pages, browser types - such reports are a legacy and as part of your interview process you need to remove considerations of the channel. This way we can focus on strategic goals that the analyst can align to the framework.

With the framework in mind, we can then drill down into the layers and identify metrics (KPI’s) within each level. The key intention is to progressively get more granular at each step:

As we do this we can go back to the earlier stages of the document and refine the metrics and dimensions required to meet the needs of the business. Our key task is to remain focussed on the business and not on the tactics.

Stage 5: Identify KPI analysis drill-downs

This is perhaps one of the most important parts of the measurement plan, it is where we provide guidelines for analyses of the KPI’s, and what considerations need to be addressed when analysing the data. Frequently overlooked by analysts who jump straight into the data collection process it’s worth spending time thinking about the meaning of the metrics that have been created. There are a couple of different ways to look at the metrics:

Look at the measurement scope framework - have we mapped out the customer journey as a process? If so, then our drill-down analysis can focus on looking upstream and downstream in the process to identify potential candidate metrics that influence the current metric.
Look at the tiered metrics framework - have we created category metrics underneath marketing metrics? Have we identified micro-metrics that sub-relate to the high-level category metrics?
Look at the historic analyses your organisation has carried out, in your business or in previous roles - can you replicate common situations that have been addressed in the past? Does your organisation have seasonal trends (such as tax submission time for financial services clients or Christmas for retail clients) that have historically meant different challenges at different times.

These drill-downs are not step by step instructions, but rather guidelines to set direction. As your organisation learns more I would recommend continuously adding to this section of your measurement plan. Think about ways that analyses have been carried out, what segments were key to diagnosis, what marketing channels have been influencers.

Stage 6: Map out your Data Sources

By adopting this approach you’ll be better prepared (by being product agnostic until this stage) to now identify sources for your data. By being more free thinking prior to this stage you will likely need to find alternative sources of data beyond the traditional comfort zone. Do you need ad-server data? social media data? customer segment properties? email data? survey data? Think about your key metrics and how that data will be collected and presented.

Being visual, I like to draw this (whiteboard if you can) and think about using a mind map technique to highlight relationships between data (will there be common keys?) and how sources will interact. Use Visio (or similar process tool) to finalise your vision and get this into your document.

As you go through this process you will likely get into situations where perhaps your current tool might not be able to support requirements. Do you need to bring in a vendor or external consultant to advise on ways to address your challenges?

Stage 7: Propose processes for target and goal setting

Target and goal setting can be a real challenge if the organisation hasn’t previously set clear objectives. Sometimes rather than trying to look at the micro-level first (feels easier, but isn’t) we need to take a step back and look at the macro-level. Who owns digital properties, has that person been set personal goals? Does your finance team have an expected ROI from web marketing? Has the organisation set expectations on how much business needs to come from digital channels over the next 12 months? Is there an expected (eg for a telco) ARPU for offline customers that the web channel has to beat?

If we can identify a couple of very high-level goals, then it’s easier to work back against all of our metrics to model ‘what-if’ scenarios. For example looking at a simple customer journey from search click, we might see a visitor land on the entry page, view product information, explore a form, watch a video, register for a webinar. At each of these simplistic stages, there are key metrics that influence the journey. How does your bounce rate impact site engagement? What does a visitor segment who spends more than 3 mins on your site look like compared with less than 2 minutes? How does first time vs repeat visitor conversion compare? What if we need to have 5+ visits to reach peak conversion?

As we investigate the relationships we can start to set benchmarks. Using ‘what-if’ scenarios along with drill-down analyses we can start to build a model of the relationship of metrics; what if your bounce rate is higher than 50% - can improvement of KPI X resolve that?

Stage 8: Identify business owners and visualisation/sharing process

By now you should have already had enough dialogue with your business owners to know what data they need to see to do their jobs, as well as having very clear direction on the overall view of the metrics. This will enable you to leverage combinations of existing dashboarding capabilities of the tools selected, as well as providing guidance on how multiple data sources will need to work together.

Being visual, I have found that sketching out reports / dashboards and presentation styles can save a huge amount of time when it comes to presentation of data when it comes to implementation later in the process - but the key objective of the measurement plan is about the documentation of reports, trends, compound metrics etc and to incorporate the insight that will be gleaned from this data.

Remember the key objective is not to prepare reports, but to prepare the organisation for what the data means, what can be done with it and what are the extended opportunities that tools (such as clickstream recommendations, multivariate testing or targeting platforms) can offer to enhance metric performance

Stage 9: Distribute, gather feedback - then publish

By now your measurement plan should be complete, and you should be satisfied with a job well done.

To close the process you need to ensure the business is on board with the measurement plan, as you’ve been reviewing this with your business team through the process by sharing updates and through interviews, your sharing for review should not come as a surprise to the organisational team. Look for candid feedback - have you missed key stakeholder feedback, are your objectives clear, are your targets within reach?

Once the organisation is agreed the document needs publishing to a location where the organisation can access it. For some businesses this is a measurement folder on a shared network drive, for others, it’s through creating content on the intranet or by creating a measurement plan Wiki. Look to your organisation for processes that work, and then build on this. Now your measurement plan is agreed and in place, you can move to delivering on the plan… and ultimately spending less time reporting!

Footnote

The measurement planning process does not happen overnight, depending on the size of your business your plan could be developed in a couple of days or over the course of 6 months.

When following this process think about how your organisation works, and where you are currently. I’d advocate as much white-boarding, doodling and process diagraming as possible to turn the complex into processes that are easy to digest and process.