SROI : In Search of a Verb

The concept of “social return on investment” is absolutely core to balancing the proverbial double (or triple) bottom line of social enterprise. It’s therefore no surprise that the need to accurately and consistently evaluate and express that value has been a topic of much discussion and hot debate. It’s a critical dialog–but I think the current conversation has a verb problem.

Much of the time, these conversations refer to SROI measurement. First off, only things that exist on an ratio scale can even BE measured. And I think we can all agree that there is no “absolute zero” on the scale of social good and that the “units” are hardly regular or continuous. (Seems to me we’d be lucky to even agree on an ordinal scale for something as context-dependent as social good.) So, in the strictest sense, measuring SROI is not even an option.

Organizations that acknowledge the stickiness of the measurement issue often claim to calculate SROI instead…It sounds less concrete perhaps, but often ends up just as arbitrary. One well-known (and arguably quite effective) US foundation literally uses a multiplier termed the “(Foundation Name) Factor” to calculate how much of the “measured” social change is attributable to their programs. Most SROI calculation schema I’ve encountered have produced this same unidimensional, artificial, even misleading oversimplification–though the amount of time and effort required to arrive there varies widely.

I’m in no way suggesting we stop looking for ways to wrap our heads around the effects of our efforts, but I think the obsession with quantification does not serve us well. So…

Should SROI be measured? Good luck with that.
Should it be calculated? Perhaps, when it fits.
Should it be demonstrated? Whenever possible.
Should it be explored? Always.

The Trouble with Stakes

Last evening, during the President’s health-care speech, I found myself frustrated. Why can’t someone just talk to me straight!? Why can’t anyone simply compare the perspectives, analyze the arguments, and explore the implications free from rhetoric, empassioned mantras, scare tactics, and tear-jerking stories. Why can’t we have some kind of genuinely objective perspective?

The answer’s pretty simple: the genuinely objective observers don’t CARE enough to do the careful analysis.

The people who care, the ones who invest time and energy and resources, are the one who have something on the line. They have a stake.

The connection from there was at once natural and surprising. So often in the non-profit and social entrepreneurship worlds, we extoll the virtues of (and even decry the absence of) objective, third-party impact assessments and evaluations. We proclaim (often quite correctly) that it is impossible for those at the heart of a venture, doing the day-to-day work, pouring their blood, sweat and tears into their programs to accurately assess their own impact and effectiveness.

The problem, of course, with these stakeholders (and any stakeholder) is that they CARE.

Essentially, we’re saying that in order to provide a reliable assessment, you must not be a stakeholder in the venture. You must not care.

Admittedly, this is a bit of a hyperbole. But it seems worth looking at. If what we want from non-profit and social entrepreneurship evaluation is thorough exploration, careful analysis and strategic recommendations, can we truly rely on evaluators without a stake?

“Ye Have Need of Patience”

I’ve been vexed all week. Really, vexed.

Here’s a selection of the blog posts I didn’t write this week: “Why Open Education Won’t Save the World,” “Lurking and Ignorance in Qualitative Research” and “The Malignant Delusion of Educational Assessment.”

Like I said… vexed.

I don’t know how to take the mass of largely useless lecture notes that is open education today and turn it into something that will create intrinsic value for universities AND actually contribute to the self-actualization of a micro-entrepreneur in Mozambique. I don’t know how to get useful instructional direction from formulaic “objective” statements or how to write a test item that actually taps higher-order thinking (heck, the textbook can’t even do it!) Let alone how to change the morally and logically bankrupt system that says standardized tests somehow indicate the worth and quality of schools, teachers, and children. All the problems just seem too complex, too convoluted, too entrenched, too intractable, too freaking HUGE.

David told us a story this week about a time when everything got to be too much and Stephen Downes just sort of disappeared for 6 months. He got choked up talking about how it changed things, how he needed that foil, that critique. I haven’t struggled with these problems long enough, let alone come up with any ideas or opinions significant enough to be needed or missed, but I was vexed this week. And sad. And already tired.

Last night, I thought of that story, and I listened to this. I still don’t have any answers. Nothing is any clearer, brighter, or easier. But “we are not of them that draw back,” are we?


What Execs Want…

Best line I’ve heard in a while, from Ben Robertson (senior consultant in the High-Tech division at Omniture…yeah, I think I’m older than he is 🙂 about how to deal with the common resistance to tracking new metrics in web analytics. You know, metrics besides “unique visitors” and “pageviews” which most executives (and most analysts for that matter) simply call “traffic.”

Company IT Manager: We can’t! That’s what our executives want to see!
Omniture Consultant: Not it’s not! That’s just what you’ve been giving them.

His advice: go ahead and give them what they’re asking for, then give them what they should be asking for.

More Analytics Musings…

This has really got me thinking. Take, for example the opening page of BYU’s Math 110 independent study course, entitled Special Instructions. Time-on-page for these users seemed fairly cleanly [though not evenly] split between those who simply skipped or made a quickly-abandoned attempt at scanning the page [spending 30 seconds or less] and those who put forth the rather ponderous 4 minute [an eternity online] effort to read the entire page. If 75% of our users really are “scanners” [then again, we only think we know this] then perhaps they [and, for that matter, we] would get more out of our site if we designed the content to be scanned.

Continue reading

Does Data Belong in the Driver’s Seat?

I’m struggling with this one. Our web analytics class is looking at a couple months’ worth of data from BYU’s most popular independent study course, Math 110 [not sure what the definition of “popular” is in this case, by the way] and making some recommendations, both about their tracking suite and about the course itself. Clint explained, and I understand, that analytics is not meant for examining a handful of people—it’s for looking at trends, types, aggregates. But, for me, that aggregation leads to serious questions.

Even as we look at beautifully-created Excel graphs that expertly bring home the enormity of the first-lesson-drop-off effect, I hear echoes of Andre the Giant in the Princess Bride; “Are you sure that means what you think it means?”

When we aggregate, in our particular case, “shoppers” with “students,” don’t we run the risk that a site adjusted to the needs of their amalgam might actually be worse for both groups of individual users? Then again, in our case, we don’t actually know who it is we’re aggregating, and though an aggregation of unknowns is surely better than a single unknown case, I still don’t know if it’s something I want to be basing decisions off.

I’ve seen data-driven decisions based on landing page A/B testing result in huge traffic and conversion gains, and I have been particularly pleased when “the data” have backed me up and finally tipped the scales with a difficult boss.

Those successes notwithstanding, making “data-driven” decisions before we can be certain what the data actually represent seems to me a foolish, and potentially counter-productive, proposition.

PS: I’m also forming a hypothesis that the answers instructional designers beg from analytics data are going to be, on the whole, a good deal more complex than those petitioned by sellers of bivy sacks or purveyors of celebrity gossip. More on this later…at the rate I’ve been going lately, much later.

My Metrics Wishlist

This post is long overdue. Sorry, Clint.

The following are a few examples [untainted by any sort of experience with the subject matter, and irrespective of what is currently regarded as possible] of metrics I would find useful for gauging and improving the effectiveness of an instructional website:

Order and Timing: My primary frustration with the rudimentary analytics I have encountered so far has been a lack of correlation between behaviors. It seems that every minute piece of behavior is excised from the holistic experience and examined under a microscope independent of any of the surrounding behaviors. Not only that, but significant decisions are reached based on these dissected data. I want to be able to track navigation patterns with time spent per page and so forth.

Complex Behavior Visualization: The gist of this one is the creation of a three-dimensional map of the site allowing the designer to follow the path of a user [or an aggregate of users] through the site; streams of color that indicate the speed of transitions between pages, the number of times a page was hit, search terms, results and which one was chosen; size differences indicative of relative popularity/traffic; etc.

Comparative Paths: I’d love to be able to filter and compare the behavior of different target populations. Even just knowing how users referred from Google interact with the material differently than those who type in the address directly could be insightful. In an ideal world, I would be able to sort users according to target profiles [could be hardware, could be navigation style, could be socio-economic status] and examine the patterns of their interactions with the site; which links were clicked in what order, how much time was spent on which pages, what “conversion” goals were reached by each group, where and when did they bail?

Diverting the Stream of Consciousness: Having experienced first-hand the sometimes dramatically counter-intuitive insights provided by eye-tracking and think-aloud usability testing protocols, I would love to allow users the opportunity to opt into a remote usability lab environment, where the machine would capture eye motion and any verbal feedback the user cared to offer, even offering occasional prompts and feed it into a real-time database. Emerging themes would be automatically flagged and the designer could set up filters according to demographics or any other standard metric; entry point, referring site, time on page etc. to uncover patterns and monitor the effects of changes.

They’re all probably crazy–but you asked.