Allen Crane Analytics

#5 The Collaborator - Ten Faces of eBiz Analytics Innovation

The Collaborator

Despite my best intentions, it's been several months since my last post...but somehow quite appropriate, for in that time, I've found myself reflecting on my colleague's comments on a tendency to be a bit self-postulating in my writing style, finding out I was anonomously "discovered" by my hero, re-org'ed into an overly matrixed leadership role, and frankly just too damn tardy in following up on my own posts (I should probably blame the house building or my new son Diego--now 6 months--but nonetheless, I've simply been a little lax).

So it's probably timely (and were it not for this post, even a bit cliche) that the topic of this Web Analytics post addresses the Collaborator. As an Analyst, I show up in every personality, leadership, and insights test as being in the "upper right" quadrant. "Blue" to those who are familiar with the Leadership Insights, "Analytical" (duh) for the Senn-Delaney-ites, and ENTJ for the Meyers-Briggs folks. In short, I and my team are your best partners in data, analytics, root cause, and driving for results. But I'm not so great (so far as I think) at the charismatic qualities of inspiration and support. So this is my segue into the topic.

When I think of great Collaborators, I think of leaders that could garner support from various areas--often with conflicting agendas of their own--and somehow help them to align on the common qualities that can focus on their similarities, in support of the mission. Teams and organizations that trive on "turf issues" get stuck in their routines and don't progress. Without the litmus test of Collaboration, you have no way of knowing how efective you can be in a truly flat and cross-functional environment.

In my days at Dell, I was put into a role that required advocating an Online Controlled Experimentation program to a Sales and Marketing organization. Imagine the look of 100 top B-schoolers looking back at me as if I could help be more efficient - who the hell was I anyway, an engineer, who may as well have been from another planet than tell a Harvard MBA they had an opportunity in their supply chain(!)

#4 The Hurdler - Ten Faces of eBiz Analytics Innovation

The Hurdler

“We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one we are willing to accept, one we are unwilling to postpone, and one we intend to win.”

--John F. Kennedy 5/25/1961

As technology becomes more complex across organizations, there is a greater need than ever for the role of Hurdlers within IT and Web Analytics. From the analyst that works across multiple database teams to make sure that newly implemented capabilities can capture and store the right data elements, to the project manger who builds a programmatic process for IT to design their applications to be quantifiable, Hurdlers are the ones that can get things done because they understand what the end result needs to be, and they challenge us to see through every step. They know that the faster our interconnected teams can figure out these steps, the faster we can execute the solution so we can know if our efforts have proven successful. Hurdlers take ownership and are not held back by org charts to get things done. And they communicate well.

Elements of an Effective Hurdler

• They quickly bottom out the issues in detail.
• They identify the requisite tasks and drive for results regardless of org structure.
• They communicate the value proposition across multiple levels, ensuring that what they’ve helped to change, sticks.

The Analyst as Hurdler

Diana is a Web Analyst who came from the IT development organization. Because of her background, she knows the back end system and how it collects data. One day, she received a request for some analysis on page traffic, but could not find any instance of the pages in our Web Analytics tool. Instead of reporting back to the customer that the pages simply could not be found, she let the customer know that we were looking into the request and would likely have to do some further work to get them what they needed. What she discovered was the application was writing an extremely long concatenated value into the query string, before being fed into the database. Further investigation uncovered that the field in the database was only one-fifth the size it needed to be. Worse yet, she discovered that the data was not even getting truncated, but it was getting dropped completely! A simple change to the database to increase the field size was required. Diana not only followed through on getting the cross-functional partner to task and fixing the database, but she also worked upstream to identify ways the application itself could be more efficient and not require so much data to be appended in the first place. Finally, her tenacity resulted in building a stronger relationship with the customer.

The Project Manager as Hurdler

Every week, there seems to be a new online application or capability that our team is asked to measure. These days, we are involved in the design and development process, so we usually know how to quantify these results. This wasn’t always the case. For us, we had to overcome a culture of “measurement adds too much scope, so reporting gets thrown out first”. It took the creation and adoption of a whole new project management concept called Design for Measurability to transcend this gap.

Brenda was a new Web Analyst that came to my team after many years in project management. We had already identified that a key issue for us was the lack of our ability to measure the capabilities that were being created—specifically, there was a lack of precisely instrumented page names and query string parameters—which hampered our ability to do analysis about specific online behaviors. Frustrated product owners could not tell if the new capability was working, and the analytics team could not get the attention of developers since the measurement requirements were now adding "scope". Brenda started with the issue (capability exists, but no data to know how we’re doing) and worked backwards – What kind of business questions were we trying to answer? What was limiting us from measuring? What types of people and technologies would need to be brought together to solve this? If asynchronous data was the issue, what business events would need to be created and captured in order to understand the desired behavior?

After she bottomed out the issues, she worked with the technology architects to pull together our internal standards for coding query string parameters, page names, AJAX, Flash, and other forms of data calls that result in an activity that we might want to measure for a  business purpose. She then packaged this into a type of “umbrella” document and incorporated this as a mandatory step in the first phase for all project kickoffs. She then led a massive communication effort to sell the process, not taking for granted that the existence of the new process itself would be sufficient for adoption.  Which brings me to my final point…

Communication – Visio alone won’t make a process stick

My team resides in what I refer to as a “process heavy” organization. And while many processes do serve their purpose and provide consistency and quality, there are enough examples otherwise that the word “process” is sometimes viewed as a bad word, perceived as something that adds a lot of overhead with little or no value added. So indeed, adding yet another process—even for the best of reasons—was a challenge. The Hurdler realizes they must do more than create a Visio diagram for a process to take hold, which means that any good Hurdler must also be a good communicator. In Diana’s case, it was listing out the tasks required from key individuals across a disparate technology organization, then holding them all to task and giving frequent, group-wide updates. For Brenda, this took the form of copious meetings across all levels – from one-on-ones with senior management, to project requirements auditors, to road shows with project management and development teams. In Brenda's case, she added a separate value proposition just for the development teams, pitching the process as a way they could know how much their new capability was contributing to the organization.

In short, Hurdlers are successful because they understand exactly what their hurdles are, and they adjust their approach, speed, and intensity accordingly to overcome each one. And just like the physical athlete, they do it in such a way that they still maintain their forward momentum after each successful completion.

Concepts in this series are based on the book, The Ten Faces of Innovation, by Tom Kelley, 2007.

#3 The Cross-Pollinator - Ten Faces of eBiz Analytics Innovation

The Cross-Pollinator

Early in my managerial career, I had the mistaken impression that if I was responsible for any effort, that it could only be done by members of my team. As my analysts and project managers began to take on more and more work, the team would tire under the pressure, or simply be less efficient if the work requested was outside their specialization. In an effort to be more efficient, managers and other analysts would find others in the organization with similar specialized skills and start going to others for their requests. The resulting perception of redundancy was frustrating, not only for myself as a manger, but also for my team. We had to be careful of not being repetitive, but still viewed our area as responsible for the requested analyses and projects. In my rush to provide services for the company, I inadvertently caused confusion and added bureaucracy to the process. In short, I lacked Cross-Pollination skills.

Great Cross-Pollinators intuitively seek out others who are already experts. They find ways to partner across groups, thus allowing their own teams to stay focused while extending their reach into other areas. They take advantage of managerial inefficiencies, and find ways to engage and reward other cross-functional resources who may be underutilized because they are not operating at maximum capacity.

I first met the Cross-Pollinator when I worked for Kelly Truax at Dell. She was a newly minted executive straight from the outside. Almost instantly, she developed a loyal base of cross-functional contributors that extended beyond her core team. These contributors seemed to relish in taking on the extra efforts, given the executive exposure they received (sometimes more than they had received from their own managers). Additionally, these role players cycled in and out with a predictable cadence, like task force agents on special missions. Kelly held them to deliver and rewarded them as much as any of us, and when their focus returned to their original teams, they were now evangelists for her vision, largely because they had an ownership in it! Her effectiveness was magnified because of her ability to Cross-Pollinate her vision throughout the organization. She viewed her organization as interdependent and without boundaries. 

To be honest, I initially resisted the idea of Cross-Pollination, mainly because I perceived the skill to be a political one, rather than subject-matter-expert driven. Today, I view Cross-Pollination as a key asset to my overall productivity. I constantly look for ways to expand the reach of my team through others. For example, my current team of 10 analysts is largely dependent on the efforts of multiple teams, and as a result I have Cross-Pollinating analysts and project managers across the company, including I/T, customer research, finance, forecasting, online marketing, demand generation, and data warehousing. I use Cross-Pollination heavily to bring these special agents together to solve tactical efforts, which drives results and enables my core team to focus on their primary web and multi-channel analytics. I give credit liberally, even if I'm heavily involved.

Keys for Cross-Pollination

• Don't think in terms of your organization only, great talent is everywhere
• Focus the primary team on core efforts, outsource everything else to special agents
• Give credit liberally, both internally to your team, and externally to your Cross-Pollinators
• Don't delegate the management of Cross-Pollinators, if they feel like they work for you, your vision will be what they take back to their teams
• Cross-Pollination is efficient - if you can do more without hiring, you're bound to be productive and make friends with your CFO!

The saying is true, "It's amazing what you can accomplish if you don't care who gets the credit." The irony is that when you give credit away, it comes back to you.

Concepts in this series are based on the book, The Ten Faces of Innovation, by Tom Kelley, 2007.

#2 The Experimenter - Ten Faces of eBiz Analytics Innovation

The Experimenter

Shortly after Thomas Edison went through ten thousand types of filament before landing on Tungsten as the best choice for his light bulb, a reporter asked him if he felt bad for “failing” ten thousand times. Edison simply responded that he had not failed at anything, but in fact had learned ten thousand things that did not work. The difference between failure and learning, he said, was that “Experience is failure with learning applied.”

When deciding between site changes—whether a design decision or a business process—decisions are often made by opinions or feelings, rather than data. This isn’t always a bad thing, especially when an inspired leader who knows their business can really connect with customers in the online world. The rest of us need a mechanism to gauge whether or not a new lever is moving us forward, and provide enough of a fact-base to say with certainty if something is working.

Prototyping and Controlled Experimentation are two of the best methods to see if competing ideas work better. Examples from my client base include the following:

When Dell wanted to increase the sale of extended warranties on their personal computers, we used an A/B experiment to see which default setting (1yr, 2yr, 3yr) on which model (low end, medium range, high end) would produce the highest overall margin. The test yielded the optimal margin warranty mix.

A “refer a friend” email campaign used a prototype naming convention for the email links to describe the type of offers and their locations within the body of the email.

eLuxury, an online luxury retailer specializing in high end women’s apparel desired to know if the high cost of using models to display handbags resulted in enough incremental sales and fewer returns than shooting the handbags without models, therefore offsetting the increased marketing cost.

Wells Fargo wanted to test the clickthrough rates of several offer placements for their redesigned home page. A multivariate test with different design treatments (rollover, flash, text) was used to determine which treatments made the home page offers more compelling.

A large financial services company used controlled experimentation to determine whether images on the bottom of the page had a higher clickthrough rate than the same images on the right hand side of the page. This straightforward A/B test resulted in a clear distinction between treatments and helped establish design standards for the site.

(Click the above image to open in larger window)

Controlled Experimentation Lifecycle

• Exploratory Data Analysis – Something has to point you in the direction of what to test, right? Whether data-driven or from an executive, ideas can come from anywhere.
• Create Hypothesis – The most important part of the experiment, defining what you want to measure and what your success metrics will be.
• Pre-Experimentation Setup – After you set up the hypothesis, you need to determine your sample size in order to detect a statistically significant change in your results.
• Experiment Implementation – Configure the filter with the ability to separate your audience into groups, and keep their treatments separated through the duration of the experiment.
• Execute Experiment – Launch the pages and wait. Wait until you get sufficient sample size before drawing conclusions. Guard against publishing premature results.
• Evaluate Experiment – Now that the experiment is over, put the results into your favorite statistics tool and evaluate the difference between treatments. If the groups are dimensionalized in a web analytics platform, you may also want to look at additional cross-effects with other data.
• Summarize Results – Report the results (even if the learning was not as planned)
• Take Action – Make production-level changes, if necessary.
• Monitor Actions – Validation, Control, and Dashboard metrics.
• Exploratory Data Analysis – What do we want to test next…?

Prototyping

Sometimes we simply cannot wait to set up and run a controlled experiment. Prototyping several low-fidelity models can cut your development time quickly and build a fact-base of feedback in minutes, rather than days or weeks.

Author Tom Kelley cites two guiding principles for effective prototyping: Real User Feedback and Multiple Options.

Feedback will be more valid and convincing if received from someone that has a personal stake in the process—getting feedback from a set of “typical” customers, and not just from your roommate who dreams in code—will likely yield results that are better for your sales. For best results, use as many real customers as you can.

Presenting multiple options to real stakeholders—actual users of the site—takes your personal attachment out of any one option. Users will tend to give more reserved feedback if they feel they are being guided toward a specific conclusion, as when being asked to comment on a single option. Conversely, multiple prototypes can often inspire users to be more open with their feedback and even suggest additional options by combining the best elements that each design has to offer.

Having an Experimenter on your team pushes everyone to think creatively, and the discipline of framing success metrics as hypotheses brings rigor and critical thinking. It can also be a great source of fun for the team – even redefining what it means to “fail wisely,” and perhaps most important, to dispel the fear of taking the first step – asking WHY?

Concepts in this series are based on the book, The Ten Faces of Innovation, by Tom Kelley, 2007.

#1 The Anthropologist - Ten Faces of eBiz Analytics Innovation

The Anthropologist

If done well, web and cross-channel analytics are disruptive technologies. I say this because the insights gleaned by tools that combine new ways to measure data and show customer behavior across many action types should be breakthrough. The result is actionable opportunities for improving customer experience, increasing revenues, and exploring the right ways to drive an optimal channel strategy. Analysts who have the chance to bring such technology to organizations usually find themselves between two camps: those that feel an enormous challenge and excitement about cracking open new discoveries, and those unwilling to change.

Anthropology is the mental foundation of a good analyst. Like Paco Underhill and his department store assistants, secretly jotting every step of a shopper’s experience, web and cross-channel analysts can see where customers are going, the pages they are on, the links they are selecting, where they are hitting dead-ends, the browsers they are using, where they abandon, when they choose to call, how long they stayed, and the time since their previous contact. 

Using Anthropology to Increase Conversion

For years, retail and web companies alike have had customer and sales metrics, but it wasn’t until secret shoppers started doing data-driven anthropology on shopping behavior that opened up the field of customer analytics. For example, Underhill demonstrated that men are three times more likely to purchase if they go in a dressing room than women. Looking back, it would be insane to “average” the two experiences, but that’s what traditional metrics were doing (sales per visitor, products per sale, etc).

We apply this concept every day in our web and channel analytics team. We see where customers are abandoning and look for clues for behavior. Sometimes it’s unclear as to why customers are not completing a process, so we have to look deeper. A recent analysis of a multi-step application process showed that only 20% of customers that began a process completed it. We identified the pages that had the highest exit rates (the lowest clickthroughs to the next page in the process). It turned out that each of these were form field pages, where the application was collecting information. The usability team identified several design flaws that needed to be addressed: below-the-fold required fields, form labels written in technical jargon, and errors for missing information, which confirmed for us that these form fields were not easy for our typical customer to navigate through. We zoned in on the error messages and found that these errors were not being captured by the web analytics tool. So we worked with the developers to capture the errors in a query string parameter.

By making what was previously invisible known, we made recommendations for the page design, and reduced the number of errors on our highest abandoned pages by 60%, which resulted in an overall conversion improvement of 40%.

Applying the mindset of the Anthropologist was key to making the mental change from “what we think” to “what we know” and “what we can do about it”. And data-driven Anthropology creates more than a fact-base, it creates a dialogue that even those who are resistant to change can begin to identify with.

Concepts in this series are based on the book, The Ten Faces of Innovation, by Tom Kelley, 2007.

The 10 Faces of eBiz Analytics Innovation

The 10 Faces of Innovation is my favorite book this year so far. Author Tom Kelley describes 10 innoovation personas in your organization--think of them as the kinetic and potential energy on your team to tap into--to overcome devil's advocates and naysayers. As analysts, our team faces such naysayers constantly:

Isn't it enough if we just build the site?

Why do we need to measure customer behaviors?

How do I get the XYZ team to understand that adding a querystring parameter is a win-win for us, and won't add scope?

Does anyone need cross-channel analytics really?

How do I make sure our Design for Measurability concept gets approved?

I'm starting an Analytics Innovation series based on the 10 personas with examples from the world of Internet and Customer Analytics. I welcome your feedback and I hope you enjoy it.

1. The Anthropologist

2. The Experimenter

3. The Cross-Pollinator

4. The Hurdler

5. The Collaborator

6. The Director

7. The Experience Architect

8. The Set Designer

9. The Caregiver

10. The Storyteller

Getting Actionable – Driving Operational KPI’s from Rear-View Reporting

In standing up analytics platforms and teams for measuring web activity, I am amazed at the number of companies that still use performance measures that do not connect to the fundamental success measures of the company. I am further shocked at the length of time companies will use an imperfect metric, rather than create the necessary data elements that make corporate metrics accurate, repeatable, and durable. I have even seen companies use an imperfect metric for so long, it becomes so accepted and goes unquestioned. It’s like knowing you have a leak in your walls, knowing there is a problem, and yet being too timid to find out what it will take to fix it completely.

For example, a client of mine had created an Online Sales metric using a monthly batch cycle that matched back all customer logins per month to a master product file for all products closed that month. If the login occurred within an agreed-upon timeframe of the product closing, the product would get attributed to the online channel. Also, product counting rules allowed certain products to be counted immediately while other products could not be counted until they had been used by the customer. The monthly lag created a “driving through the rear-view mirror” approach – and added time to the decision making process.

1. Discrete Channelling

The first problem is the messy matchback methodology. While business rules can help in deciding which channel to attribute a product to, there is a clear benefit to capturing a channel flag at point of purchase and at points of contact and consideration along the way. Not only does this provide clarity in terms of channel performance, it provides the essential markers necessary to build the gold standard of customer analysis: Multichannel analytics.

2. Too Many Rules to Count

The second issue is the product counting rules – sometimes products get counted, and sometimes they do not. Depends on customer eligibility? Sometimes. How about the level of product? Maintaining a minimum balance? Maybe. What about Frequency of service? Perhaps. Product counting rules, by definition, are incomplete views of your product exposures. Managinig product counts by defining the parameters for which they are counted is like having a cupboard of glassware, but only counting the wine glasses. Not only does this create a kind of cryptic language where everyone has to constantly ask “Now are those customer numbers everyone, or just ABC-eligible? Are the products listed those we can count or not?” Product performance is not “managed” by counting rules.

3. Measuring Inertia is Not Managing Performance

When a company measures itself on its own intertia and calls it “performance”, it is simply being lazy, resting on its enviable (though temporary) position of customers coming to them. Having an actionable fact-base means that you move beyond measuring just the outputs. Measuring the inputs is key as well, because you can then form the connections between the inputs and outputs, the levers between cause and effect. I have seen "unsophisticated" companies do this very well, because they understood the relationship between the few but critical "simple" variables to the outcomes of their business. I have also seen "sophisticated" companies over-engineer this, complicating the analysis with multivariate factors and levels so numerous that it was impossible to tell what was driving results.    

4. A 3-Second Late Start

Giving a 100-meter competitor a 3-second head start is a bad idea, even if you’re in the junior leagues. You might even still beat them if they are terrible, but with world-class athletes, that kind of lead is an unrecoverable position to be in. But that’s exactly what you are doing when you measure at the monthly level – you’re driving through the rear-view mirror, wondering what happened in month 1 and how you can put into place something in month 2 that will hopefully be in production by month 3, just so you can wait around until month 4 to see if those changes made a difference. Meanwhile your customers knew their key issues in the first week of the year, and have been making course corrections since then. By the time you're out of the blocks, your competition is a third of the way down the field.   

That said, it is still possible to create an actionable metric from rear-view reporting. While it may be ideal to have a channel indicator and a different approach to counting products and customers, the reality is you may not be there yet. Driving online performance really requires just two essential components, what I call “Minding the Gap” and “Finding Common Denominators”.

Minding the Gap

Building a true Online Sales metric introduces a new challenge to performance measurement – connecting database actuals with customer behaviors. No matter how precise you measure them, behaviors are directional, and should be seen as the collection of events that lead to an outcome, whether desirable or not. Nevertheless, ratios can be useful to connect the behaviors and actions in meaningful ways that can be useful over time. Trending these ratios is the basis for modeling overall growth and channel penetration. When combined with channel markers, can be an essential asset for planning and managing channel shift over time. Here’s an example:

Online Close Rates: (# of products purchased online) / (# of applications started online)

Finding Common Denominators

Sometimes the concept of Demand and Net can be useful to represent Operational and Financial performance measures, respectively. They are different measures because they are used for different purposes. Demand or Operational numbers are used as indicators of company performance NOW, and point to the business levers that leaders need to pull TODAY. Net or Financial metrics on the other hand are the outcome of operations. They are the result of the cumulative data and decisions that lead to the booked orders and RUM – Revenue, Units, Margin.

That said, it should then be expected that the numerators for both numbers are different. Denominators, on the other hand, should be common to both. The denominator in both cases should be the same number of initial opportunities for a given product sale. And whether that is a measure of Visitors of Sessions, they should be consistent with each other.

Online Conversion: (# of applications completed online) / (# of applications started online)

Online Close Rates: (# of products purchased online) / (# of applications started online)

In the end, there may not be an appetite for changing the legacy metric (no matter how bad), in the near term. You may have to run two parallel metrics for some time, in order to gain buy-in and adoption. Bad thinking—however entrenched—needs to be driven out. When a metric’s credibility is based on its historicity rather than its accuracy, remove everything that prevents you from delivering precision, and focus on driving measurable performance through metrics that are grounded in reality.

Actionable E-Metrics

An actionable online analytics framework is a key ingredient in any intelligent enterprise (By Allen Crane - Reprinted from Intelligent Enterprise, Feb 2003)

Many of the reasons offered to explain the dot-com meltdown of 2000-2001 involved more wishful thinking than reality of the business case models used — models that would supposedly have become profitable with more venture capitalist spending.

While overhyped, and poorly-executed business plans were certainly the root cause for a number of now-defunct companies, such blanket explanations don't fully describe a true cause and effect. Many successful dot-coms — such as Yahoo, Amazon, and eBay — continue to do business, despite their "if we spend first, they will come" business approach.

Given similar spend-first strategies, why do some e-businesses fail, while others prosper? In essence, some fail because they are unable to develop a new set of metrics that differs fundamentally from traditional finance-based metrics, yet is intrinsic to e-commerce. Too few companies understand that the Internet is not simply a "channel" for business; it redefines the business itself. The nontraditional business frontier demands nontraditional metrics to manage it.

In this article, I won't attempt to redefine the industry of e-commerce measurement tools, nor will I claim to offer a silver-bullet cure for struggling e-businesses. Rather, I'll present an original framework for how to measure and analyze real-world, relevant, e-commerce metrics, and how to present them in an actionable, executable format. Ultimately, this framework gives you the ability to:

• Rapidly diagnose site problems
• Understand the elements of site conversion
• Understand the relationships among units, revenue, and margin, and the behaviors that drive them.

Traditional and "Blue Sky" Metrics

Traditional profit and loss (P&L)-based metrics fall far short of painting a complete customer behavior picture. Such metrics are essentially transaction-based — they're gathered only at the time of sale. In contrast, metrics for e-commerce (hereafter referred to as e-metrics or clickstream metrics) offer the opportunity to view more than the final transaction of a purchase. The beauty of online data is that it captures every click visitors make and every image they see. Clickstream data offers a rich picture of all the customer-behavior events that lead to a purchase (or, perhaps more important, the events that result in a nonpurchase).

During the dot-com explosion, several e-metrics buzzwords infiltrated the analyst vernacular: page leakage, stickiness, slipperiness, velocity, shopping cart abandonment, convergence, and perhaps the most seductive of them all, path analysis (the ability to find the "magic path to purchase").

These concepts are interesting, but they tend to be more of an academic exercise than anything else because they're based on anecdotal scenarios. A former director of mine perhaps said it best when he asked me, "So what if a page is sticky? Is that a factor of the page confusing customers who don't know where to go, or are they investigating our products, and their time spent is proportional to their interest? How can we tell the difference?"

To get to an actionable decision, the most effective approach lies somewhere between P&L metrics (which are vague and not properly integrated) and the academic metrics I've described (which can be interesting but nonactionable). The Holy Grail is a suite of quantifiable, actionable e-metrics that capture behavior patterns and accurately relate them to the key transactional business levers of units, revenue, and margin. Ideally, the reporting of all such data should be managed and developed together, and accessed by flexible reporting tools that can accommodate technical as well as nontechnical users. Ideally, the entire process should be managed by an ultra-lean, experienced support staff that balances tactical and strategic visions.

Where to Begin

Consider the questions that you're trying to answer about your Web site visitors and you'll find that they range widely. The first step toward building a successful analysis toolset is to broadly categorize these types of analyses. While each analysis is unique, most analytic questions can be framed in a two-by-two matrix. I'll examine each part of this matrix in more detail in later posts.

Quality e-metrics. These basic traffic metrics are known for their short-term/low integration effort. They rely only on clickstream data and report on a predefined set of data. At the minimum, such reports must be available on a weekly basis, but the code must also be readily available to query the data directly as needed, if the project can't wait until the week's data is ready. Examples include:

• Traffic by page and site area (groups of pages): the raw traffic data in terms of requests (clicks), visits (per session-based logic), and users (per cookie or other individual or machine-specific identifier).
• Page leakage percentage: the percentage of the number of visits to any particular page where that page was the last page in a visit, divided by the number of visits to that page. By identifying pages that result in a high percentage of visit termination, you can better address site design, either by intuitive means or qualitative usability testing. In either case, before and after results can be compared in order to identify acceptable limits for leakage on certain pages.
• Next click: a component of path analysis, the pages that most frequently followed in direct sequence from a page or a set of pages. (Because of the number of enumerations made possible by Web site design, you should first try this metric on a small subset of pages.) Next click is critical for site design, when quantifying navigation patterns from certain main pages that contain a multitude of links. Even more importantly, it helps you quantify of one of the most elusive metrics of Web design: site real estate value.
• Previous click: similar to next click, a previous click report will yield the pages that most frequently occurred directly prior to a page or a set of pages. (My previous advice to try this metric on a small subset of pages applies here as well.) Navigation patterns and site real estate are key insights from this metric. On a related note, referring domain and referring URL reporting can be extremely useful, especially when presented in a format that illustrates trending effects over time.

It's important to note that quality e-metrics are fully automated and published regularly. Minimal effort should be spent in maintaining such regular reporting metrics.

Project-centric metrics. Although they rely solely on clickstream data, these metrics are designed to be more long-term in scope — long-term meaning that these analyses are designed to facilitate online application processes that will result in measurable technological changes. Such metrics are designed to facilitate the transition to better processes (strategic in nature) by measuring the before and after cases.

For example, if a project manager wants to improve a part of the Web site, the Web analytics consultant would work with the project manager to create a set of metrics around the specific pages that need to be improved. These metrics could include frequency of visit — or, if the set of Web pages is systemic and follows a process (a registration or application process, for example) — the metrics could include step-by-step fallout detail, also known as a waterfall report.

Waterfall reports are invaluable for analyzing the effectiveness of the sales funnel, or the ratio of the number of customers in one stage of the sales process, to the number of customers at another stage in the sales process. Say, for example, that the project manager wants to move more customers online in order to reduce phone calls to the customer service center. The Web analytics consultant would create a metrics package that the project manager could run through the course of the implementation to see if more customers used the new and improved Web application as a percentage of all customers (Web and phone) who interact with the company.

Such metrics are necessary to sustain results and substantiate quantifiable financial benefit. When the project is completed and the business process improvement is booked to the financials, it may no longer be necessary to run such reporting — therefore the term disposable metrics. However, while the metrics may no longer be used in running the day-to-day business, it's important to note that the metrics were critical to proving the value of the project and lowering the overall cost-to-serve.

These metrics are essentially queries that are developed in close consultation with business users, specifically project managers. When the queries are developed, the project manager should be given the database access and reporting tools necessary to run the specific-use reports as needed.

Although these metrics are designed to facilitate the strategic implementation of technical projects, these queries are often disposable, in that they may become short-lived after a project's implementation. However, if after an implementation these metrics are deemed to be business imperatives, they're considered for regular publishing as quality e-metrics.

Deep dive metrics. These metrics are known for their short-term/high integration effort. They are 100-percent customized and thus among the most time-consuming metrics to develop, mainly because they're designed to answer any of a large range of questions. These questions may include root-cause analysis, hypothesis testing, developing confidence intervals, online customer cluster analysis, shopping cart (market basket) analysis, sequence and association analysis, and other data mining applications.

Deep dive metrics can be very difficult to obtain, for at least three reasons:

• They require integration among multiple databases, which is almost never as straightforward as it sounds.
• The complex data manipulation required to perform the relevant statistical analysis is often complicated by huge amounts of data typical of online clickstream analysis.
• The statistical analysis is often a first of its kind, requiring technical "acid-tests" by peer or managerial reviewers.

It's a rare company indeed that contains clickstream, financial, customer, and support databases in a single, seamless repository on a single database platform. The reality is almost always complicated by multiple database instances running on multiple platforms, requiring extracts of data to other databases and matching on key fields to get additional data, which is then extracted to additional databases and further matched before final reporting.

Because of their complexity, deep dive metrics are often prototypes for technology projects. You must carefully balance these requests so that the result analysis is actionable in its final scope, yet deliverable in the required time frame. Deep dive analyses frequently drive significant business changes because of their ability to quantify formerly invisible cause and effect behaviors.

An example here would be to determine specific opportunities for your customers by developing a report that shows the number of purchases for a specific product or stock keeping unit, divided by the number of visits to that product, and then plotting the results:

• Low visits/low conversion: low interest or difficulty in finding these products on the site. If the path to purchase is easy to navigate and awareness is there, low interest could be a candidate for removal from the site.
• Low visits/high conversion: the "surgical shoppers" who know what they're looking for. Consider bundling these items with other top-selling products.
• High visits/high conversion: hot products, hot sellers. No need to waste promotion dollars here, because everything is selling so well already.
• High visits/low conversion: opportunities galore. Better merchandising, competitive price position, and aggressive promotions should be seriously considered here.

Highly integrated and automated reporting. These metrics are known for their long-term/high integration effort. They derive from deep dives as well as the result of complex integrated reporting, which essentially says that the data would otherwise represent an incomplete picture unless reported in the context of multiple types of data (for example, clickstream data combined with financial and customer data such as online product conversion — units sold online as a ratio of online visits).

The development of highly integrated and automated reports is a collective effort among online analysts, business users, and the IT infrastructure necessary to deploy them. (See the sidebar "The Online Analytics Team.") Because these highly integrated and automated reports take so much effort, build as much data into the them as you can in order to answer as many questions as possible.

As I stated earlier, the ideal reporting of such data should be managed and developed together, and accessed by flexible reporting tools that can accommodate technical and nontechnical users. To that goal, flexible reporting tools such as Excel Pivot Tables and online analytic processing can accommodate both high- and low-level decision reporting. These tools make drill-down reporting easy, allowing significant insight to be distributed widely on a regular basis, while complex data can be synthesized into a presentable format that's as readily available as quality e-metrics.

With the basic framework for online analytics in place, you now have a method for measuring the critical business drivers of online behavior.

Survival Instinct

A robust online analytics framework must be in place to provide quality analytic solutions to problems that would otherwise be invisible and unquantifiable: the integrated analysis of online and transactional data.

Inventing and building the framework is only the first step, however. Sustaining the process via quantitative management in the form of education, delegation, and evangelization is key to doing more with less, by improving processes, increasing analytic efficiencies, and leveraging resources wisely.

The ability to make visible and measurable that which is of utmost importance is crucial to e-commerce success and a critical component of e-business survival.

---

THE ONLINE ANALYTICS TEAM

Now that we've identified the various types of analyses, we can focus on the processes for getting the most out of the online analytics team.

The quantitative management approach is best implemented in phases, focusing first on inventing and building processes for getting the right metrics to the right people at the right time, and then on sustaining them:

Phase I — Inventing and Building

• Consulting: Engaging with the business users to identify opportunities for analysis that can answer, "What do we know to be of utmost importance, but are unable to measure?" Such engagements lead to ...
• Development: Creating a systematic, reliable, and repeatable process that will deliver the desired results of the consulting engagement, which will eventually need ...
• Automation: This is the number one ingredient for doing more effective work with fewer people, in order to sustain world-caliber metrics with an ultra-lean staff. As much as possible, every process should be automated to minimize human intervention, process dependencies, and make increased use of communication and distribution channels, so as to prepare the way for ...
• Education: Now begins the process of educating users on the data, where to find the results of the analysis, and how to approach online analytics in general.

Now that we've delivered on our promise of better data, we're ready to exploit the second phase of quantitative management to maximize individual efficiency and increase overall process speed.

Phase II — Sustaining

• Education: As just stated, you begin with the education of users on the how, what, when, and where of the data. When you give the business users the tools to access the data, you increase their responsibility through ...
• Delegation: At this time, users are familiar enough with the basic processes to get at all necessary quality e-metrics and reports: some may even run their own projectcentric metrics. These users understand how to find, access, and navigate reports and are versed in the process by which they can charter and prioritize (for the online analytics team) future deep dives and integrated reports. The best business users may even become mentors to others in their own groups on how to use the data, taking ownership of the process themselves to fetch and analyze the data and contributing to ...
• Evangelization: The work to date has become so thoroughly instilled in the company that the business uses the data on a regular basis. At this point the self-serve nature of analyses and reporting is clearly understood in the organization, and the online analytics team can focus on further enhancements and next-generation analyses.