Actionable E-Metrics

An actionable online analytics framework is a key ingredient in any intelligent enterprise (By Allen Crane - Reprinted from Intelligent Enterprise, Feb 2003)

Many of the reasons offered to explain the dot-com meltdown of 2000-2001 involved more wishful thinking than reality of the business case models used — models that would supposedly have become profitable with more venture capitalist spending.

While overhyped, and poorly-executed business plans were certainly the root cause for a number of now-defunct companies, such blanket explanations don't fully describe a true cause and effect. Many successful dot-coms — such as Yahoo, Amazon, and eBay — continue to do business, despite their "if we spend first, they will come" business approach.

Given similar spend-first strategies, why do some e-businesses fail, while others prosper? In essence, some fail because they are unable to develop a new set of metrics that differs fundamentally from traditional finance-based metrics, yet is intrinsic to e-commerce. Too few companies understand that the Internet is not simply a "channel" for business; it redefines the business itself. The nontraditional business frontier demands nontraditional metrics to manage it.

In this article, I won't attempt to redefine the industry of e-commerce measurement tools, nor will I claim to offer a silver-bullet cure for struggling e-businesses. Rather, I'll present an original framework for how to measure and analyze real-world, relevant, e-commerce metrics, and how to present them in an actionable, executable format. Ultimately, this framework gives you the ability to:

• Rapidly diagnose site problems
• Understand the elements of site conversion
• Understand the relationships among units, revenue, and margin, and the behaviors that drive them.

Traditional and "Blue Sky" Metrics

Traditional profit and loss (P&L)-based metrics fall far short of painting a complete customer behavior picture. Such metrics are essentially transaction-based — they're gathered only at the time of sale. In contrast, metrics for e-commerce (hereafter referred to as e-metrics or clickstream metrics) offer the opportunity to view more than the final transaction of a purchase. The beauty of online data is that it captures every click visitors make and every image they see. Clickstream data offers a rich picture of all the customer-behavior events that lead to a purchase (or, perhaps more important, the events that result in a nonpurchase).

During the dot-com explosion, several e-metrics buzzwords infiltrated the analyst vernacular: page leakage, stickiness, slipperiness, velocity, shopping cart abandonment, convergence, and perhaps the most seductive of them all, path analysis (the ability to find the "magic path to purchase").

These concepts are interesting, but they tend to be more of an academic exercise than anything else because they're based on anecdotal scenarios. A former director of mine perhaps said it best when he asked me, "So what if a page is sticky? Is that a factor of the page confusing customers who don't know where to go, or are they investigating our products, and their time spent is proportional to their interest? How can we tell the difference?"

To get to an actionable decision, the most effective approach lies somewhere between P&L metrics (which are vague and not properly integrated) and the academic metrics I've described (which can be interesting but nonactionable). The Holy Grail is a suite of quantifiable, actionable e-metrics that capture behavior patterns and accurately relate them to the key transactional business levers of units, revenue, and margin. Ideally, the reporting of all such data should be managed and developed together, and accessed by flexible reporting tools that can accommodate technical as well as nontechnical users. Ideally, the entire process should be managed by an ultra-lean, experienced support staff that balances tactical and strategic visions.

Where to Begin

Consider the questions that you're trying to answer about your Web site visitors and you'll find that they range widely. The first step toward building a successful analysis toolset is to broadly categorize these types of analyses. While each analysis is unique, most analytic questions can be framed in a two-by-two matrix. I'll examine each part of this matrix in more detail in later posts.

Quality e-metrics. These basic traffic metrics are known for their short-term/low integration effort. They rely only on clickstream data and report on a predefined set of data. At the minimum, such reports must be available on a weekly basis, but the code must also be readily available to query the data directly as needed, if the project can't wait until the week's data is ready. Examples include:

• Traffic by page and site area (groups of pages): the raw traffic data in terms of requests (clicks), visits (per session-based logic), and users (per cookie or other individual or machine-specific identifier).
• Page leakage percentage: the percentage of the number of visits to any particular page where that page was the last page in a visit, divided by the number of visits to that page. By identifying pages that result in a high percentage of visit termination, you can better address site design, either by intuitive means or qualitative usability testing. In either case, before and after results can be compared in order to identify acceptable limits for leakage on certain pages.
• Next click: a component of path analysis, the pages that most frequently followed in direct sequence from a page or a set of pages. (Because of the number of enumerations made possible by Web site design, you should first try this metric on a small subset of pages.) Next click is critical for site design, when quantifying navigation patterns from certain main pages that contain a multitude of links. Even more importantly, it helps you quantify of one of the most elusive metrics of Web design: site real estate value.
• Previous click: similar to next click, a previous click report will yield the pages that most frequently occurred directly prior to a page or a set of pages. (My previous advice to try this metric on a small subset of pages applies here as well.) Navigation patterns and site real estate are key insights from this metric. On a related note, referring domain and referring URL reporting can be extremely useful, especially when presented in a format that illustrates trending effects over time.

It's important to note that quality e-metrics are fully automated and published regularly. Minimal effort should be spent in maintaining such regular reporting metrics.

Project-centric metrics. Although they rely solely on clickstream data, these metrics are designed to be more long-term in scope — long-term meaning that these analyses are designed to facilitate online application processes that will result in measurable technological changes. Such metrics are designed to facilitate the transition to better processes (strategic in nature) by measuring the before and after cases.

For example, if a project manager wants to improve a part of the Web site, the Web analytics consultant would work with the project manager to create a set of metrics around the specific pages that need to be improved. These metrics could include frequency of visit — or, if the set of Web pages is systemic and follows a process (a registration or application process, for example) — the metrics could include step-by-step fallout detail, also known as a waterfall report.

Waterfall reports are invaluable for analyzing the effectiveness of the sales funnel, or the ratio of the number of customers in one stage of the sales process, to the number of customers at another stage in the sales process. Say, for example, that the project manager wants to move more customers online in order to reduce phone calls to the customer service center. The Web analytics consultant would create a metrics package that the project manager could run through the course of the implementation to see if more customers used the new and improved Web application as a percentage of all customers (Web and phone) who interact with the company.

Such metrics are necessary to sustain results and substantiate quantifiable financial benefit. When the project is completed and the business process improvement is booked to the financials, it may no longer be necessary to run such reporting — therefore the term disposable metrics. However, while the metrics may no longer be used in running the day-to-day business, it's important to note that the metrics were critical to proving the value of the project and lowering the overall cost-to-serve.

These metrics are essentially queries that are developed in close consultation with business users, specifically project managers. When the queries are developed, the project manager should be given the database access and reporting tools necessary to run the specific-use reports as needed.

Although these metrics are designed to facilitate the strategic implementation of technical projects, these queries are often disposable, in that they may become short-lived after a project's implementation. However, if after an implementation these metrics are deemed to be business imperatives, they're considered for regular publishing as quality e-metrics.

Deep dive metrics. These metrics are known for their short-term/high integration effort. They are 100-percent customized and thus among the most time-consuming metrics to develop, mainly because they're designed to answer any of a large range of questions. These questions may include root-cause analysis, hypothesis testing, developing confidence intervals, online customer cluster analysis, shopping cart (market basket) analysis, sequence and association analysis, and other data mining applications.

Deep dive metrics can be very difficult to obtain, for at least three reasons:

• They require integration among multiple databases, which is almost never as straightforward as it sounds.
• The complex data manipulation required to perform the relevant statistical analysis is often complicated by huge amounts of data typical of online clickstream analysis.
• The statistical analysis is often a first of its kind, requiring technical "acid-tests" by peer or managerial reviewers.

It's a rare company indeed that contains clickstream, financial, customer, and support databases in a single, seamless repository on a single database platform. The reality is almost always complicated by multiple database instances running on multiple platforms, requiring extracts of data to other databases and matching on key fields to get additional data, which is then extracted to additional databases and further matched before final reporting.

Because of their complexity, deep dive metrics are often prototypes for technology projects. You must carefully balance these requests so that the result analysis is actionable in its final scope, yet deliverable in the required time frame. Deep dive analyses frequently drive significant business changes because of their ability to quantify formerly invisible cause and effect behaviors.

An example here would be to determine specific opportunities for your customers by developing a report that shows the number of purchases for a specific product or stock keeping unit, divided by the number of visits to that product, and then plotting the results:

• Low visits/low conversion: low interest or difficulty in finding these products on the site. If the path to purchase is easy to navigate and awareness is there, low interest could be a candidate for removal from the site.
• Low visits/high conversion: the "surgical shoppers" who know what they're looking for. Consider bundling these items with other top-selling products.
• High visits/high conversion: hot products, hot sellers. No need to waste promotion dollars here, because everything is selling so well already.
• High visits/low conversion: opportunities galore. Better merchandising, competitive price position, and aggressive promotions should be seriously considered here.

Highly integrated and automated reporting. These metrics are known for their long-term/high integration effort. They derive from deep dives as well as the result of complex integrated reporting, which essentially says that the data would otherwise represent an incomplete picture unless reported in the context of multiple types of data (for example, clickstream data combined with financial and customer data such as online product conversion — units sold online as a ratio of online visits).

The development of highly integrated and automated reports is a collective effort among online analysts, business users, and the IT infrastructure necessary to deploy them. (See the sidebar "The Online Analytics Team.") Because these highly integrated and automated reports take so much effort, build as much data into the them as you can in order to answer as many questions as possible.

As I stated earlier, the ideal reporting of such data should be managed and developed together, and accessed by flexible reporting tools that can accommodate technical and nontechnical users. To that goal, flexible reporting tools such as Excel Pivot Tables and online analytic processing can accommodate both high- and low-level decision reporting. These tools make drill-down reporting easy, allowing significant insight to be distributed widely on a regular basis, while complex data can be synthesized into a presentable format that's as readily available as quality e-metrics.

With the basic framework for online analytics in place, you now have a method for measuring the critical business drivers of online behavior.

Survival Instinct

A robust online analytics framework must be in place to provide quality analytic solutions to problems that would otherwise be invisible and unquantifiable: the integrated analysis of online and transactional data.

Inventing and building the framework is only the first step, however. Sustaining the process via quantitative management in the form of education, delegation, and evangelization is key to doing more with less, by improving processes, increasing analytic efficiencies, and leveraging resources wisely.

The ability to make visible and measurable that which is of utmost importance is crucial to e-commerce success and a critical component of e-business survival.

---

THE ONLINE ANALYTICS TEAM

Now that we've identified the various types of analyses, we can focus on the processes for getting the most out of the online analytics team.

The quantitative management approach is best implemented in phases, focusing first on inventing and building processes for getting the right metrics to the right people at the right time, and then on sustaining them:

Phase I — Inventing and Building

• Consulting: Engaging with the business users to identify opportunities for analysis that can answer, "What do we know to be of utmost importance, but are unable to measure?" Such engagements lead to ...
• Development: Creating a systematic, reliable, and repeatable process that will deliver the desired results of the consulting engagement, which will eventually need ...
• Automation: This is the number one ingredient for doing more effective work with fewer people, in order to sustain world-caliber metrics with an ultra-lean staff. As much as possible, every process should be automated to minimize human intervention, process dependencies, and make increased use of communication and distribution channels, so as to prepare the way for ...
• Education: Now begins the process of educating users on the data, where to find the results of the analysis, and how to approach online analytics in general.

Now that we've delivered on our promise of better data, we're ready to exploit the second phase of quantitative management to maximize individual efficiency and increase overall process speed.

Phase II — Sustaining

• Education: As just stated, you begin with the education of users on the how, what, when, and where of the data. When you give the business users the tools to access the data, you increase their responsibility through ...
• Delegation: At this time, users are familiar enough with the basic processes to get at all necessary quality e-metrics and reports: some may even run their own projectcentric metrics. These users understand how to find, access, and navigate reports and are versed in the process by which they can charter and prioritize (for the online analytics team) future deep dives and integrated reports. The best business users may even become mentors to others in their own groups on how to use the data, taking ownership of the process themselves to fetch and analyze the data and contributing to ...
• Evangelization: The work to date has become so thoroughly instilled in the company that the business uses the data on a regular basis. At this point the self-serve nature of analyses and reporting is clearly understood in the organization, and the online analytics team can focus on further enhancements and next-generation analyses.