My Metrics Wishlist
This post is long overdue. Sorry, Clint. The following are a few examples [untainted by any sort of experience with the subject matter, and irrespective of what is currently regarded as possible] of metrics I would find useful for gauging and improving the effectiveness of an instructional website:
Order and Timing: My primary frustration with the rudimentary analytics I have encountered so far has been a lack of correlation between behaviors. It seems that every minute piece of behavior is excised from the holistic experience and examined under a microscope independent of any of the surrounding behaviors. Not only that, but significant decisions are reached based on these dissected data. I want to be able to track navigation patterns with time spent per page and so forth.
Complex Behavior Visualization: The gist of this one is the creation of a three-dimensional map of the site allowing the designer to follow the path of a user [or an aggregate of users] through the site; streams of color that indicate the speed of transitions between pages, the number of times a page was hit, search terms, results and which one was chosen; size differences indicative of relative popularity/traffic; etc.
Comparative Paths: I'd love to be able to filter and compare the behavior of different target populations. Even just knowing how users referred from Google interact with the material differently than those who type in the address directly could be insightful. In an ideal world, I would be able to sort users according to target profiles [could be hardware, could be navigation style, could be socio-economic status] and examine the patterns of their interactions with the site; which links were clicked in what order, how much time was spent on which pages, what "conversion" goals were reached by each group, where and when did they bail?
Diverting the Stream of Consciousness: Having experienced first-hand the sometimes dramatically counter-intuitive insights provided by eye-tracking and think-aloud usability testing protocols, I would love to allow users the opportunity to opt into a remote usability lab environment, where the machine would capture eye motion and any verbal feedback the user cared to offer, even offering occasional prompts and feed it into a real-time database. Emerging themes would be automatically flagged and the designer could set up filters according to demographics or any other standard metric; entry point, referring site, time on page etc. to uncover patterns and monitor the effects of changes.
They're all probably crazy--but you asked.
