Archive for Rigour:

September 28, 2011

By jleffron



Posted In

Learning, Rigour

Critically Flawed

We have met the enemy and he is us.

Multiple Working Hypotheses vs Pet Theories

It was “smackdown” time in my brain last week. I fell into the situation quite innocently.

When I’m working to create change, or to build something new, it’s easy to become so busy being the ‘champion’ for my project (or ‘evangelist’, if you like) that I forget the need for Multiple Working Hypotheses.*  This is understandable.  If I decide to work very hard to develop “B” in response to flaws in “A”; my (non-objective) gut feeling is that I’m fixing things. I start to feel like “A” and “B” are the only games in town; I’ve done something good here by replacing erroneous “A” with meritorious “B”. But it could well be that “B” is as flawed as “A”. If “A” is ‘wrong’; and “B” is different from “A”, it does not necessarily follow that “B” is ‘right’, only that “B” is not “A”. To do my work well, I really need to consider (create, if necessary) “C”, “D”, and possibly “E”.

This reality hit me smack between the eyes at about 7:00 on Saturday morning. Right up until that moment, I’d seen the my pet project of several months as this lovely shiny solution. But as I stood in the kitchen, waiting for the coffee to finish brewing, every weakness of my approach became clear to me.  The fact that another approach was problematic in some areas did not make mine right, even if I felt I was addressing a particular set of flaws in the old approach.  I’d become my own worst enemy – lack of critical thought left my project critically flawed.

It’s tough to not become enamoured with a particular theory or approach, and that’s where Multiple Working Hypotheses come in handy. Having more than one working hypothesis not only cuts back on things like confirmation bias, it makes for stronger research.  Entertaining multiple theories leaves my mind open to new connections often by pushing me to examine a wider body of data or evidence. If I don’t have a pet theory I’m nurturing along, I’m more open to new insights and surprising (but true) conclusions.

I’m back to the drawing board with my project. It’s not a blank slate, by any means.   The work I did before still has merit but it was only a partial view of the whole picture.  So I have a bigger drawing board now, with lots of room to examine a wider set of ideas. A little dose of objectivity diverted my work; the end result will be much stronger for that diversion.





* It wasn’t until I started writing this that it occurred to me that the concept of Multiple Working Hypotheses, coming from the work of a 19th century geologist, was commonly known in Geology departments, but not something I’ve heard much discussed in other disciplines.

September 26, 2011

By jleffron


No Comments

Posted In

Blogs and Presentations, Learning, Rigour

In Praise of Critiques

I don’t like criticism, which probably puts me in pretty good company. But I like critiques very much.

Yesterday there was some lively discussion on Twitter  about the use of public critiques as a learning tool. The phrase “learning tool” is the key – critiques are about learning, not correction.

Critiques are not a new tool for learning, many university Music and Art Departments post their guidelines for effective critiquing online. It’s not a tool restricted to the world of performing arts; at it’s best public critiquing is a civilized discussion of a person’s work, looking at what’s good, what’s lacking, what other avenues could be pursued. It’s a blend of offering needed objectivity and providing additional germane resources. It’s discourse, it’s learning, and it can be a bit unnerving to the uninitiated; but that last bit doesn’t have to be the case.

Consider the popular site where there are loads of people actively seeking critiques of their novels and short stories. Why would someone do this? If you’ve written a piece of fiction, even if you (rightly) suspect that it’s absolute dreck, you’ll be irrationally fond of it, protective of it; you love it like your firstborn child, so why seek out critiques? The answer seems to be simply this: for writers, improving in their craft matters more than their fear of hearing something negative about their work; they know they need that objective outside voice.

Common Objections

Supposing you’re sold on the notion of critiques. It’s still not all sunshine and daffodils, because there a few hurdles to clear before most people are going to jump on the “Critiquing is Awesome” band wagon. And they’re big ones.

If critiques are going to be effective as a learning tool, there are some logistics to be addressed, in the form of trust, culture (individual and organizational), and personality. Some of these are clearly not in your control; but you can usually address enough of them to be successful. Being aware of obstacles allows you to address them up front in your learning design.

As an example in some organizations the nature of the work is such that “failure is not an option” (things like nuclear power plants and neurosurgery come to mind) but that’s the big picture view the performance of the job. During the learning process, it has to be clear that critiques are not indicators of failure but stepping stones to success – or at least are akin to bumper bowling, keeping you from drifting too far off target.

Of course, there may be some cultures where critiques are a bad match. In a company with an up-or-out policy critiques may turn to daggers pretty fast.

Making it Work

In most circumstances it’s probably not the best plan to jump in with full throttle critiquing.

The whole notion of critiques is a bit contrary to our nature, or at least to our habits. It’s a bit like going for a swim in very cold lake – most of us like to ease in a little at a time; jumping in right off the bat is likely shock all but the most hearty folks right back out onto the shore. So, you need to help people want to swim in the (metaphorical) water, and then make it as painless as possible for them to ease into it.

Let’s look at back at those writers who willingly seek critiques – what’s going on there? In general it boils down to a sense that “the success of my project; the excellence of my work has risen to a higher priority than my fears or feelings”. We might call this “engagement” in the learning world. There has to be a sense that what is being achieved worth the investment.

So, let’s say you’ve done your up-front work and you have group that says “we’re so committed to excellence in understanding that what we want mutual public critiquing here so we can maximize our learning curve”. You still have real people with real fears and feelings, so how do you ease them into critiquing?

The first step will be tough: as an instructor you need to minimize the teaching and leave a void for people to fill. Cover the principles and potential pitfalls of the subject, but then turn folks loose. Critiquing is participatory by it’s nature. Too much noise from the instructor tends to silence the other voices; provide guideposts as needed, but let people wander around a bit and find their way.

Imagine the following scenario: you’re an instructor trying to build critiquing in your class. What might you do to get the conversation started?  One approach might look like this:

  1. Step away from spoon-feeding information and then fill that gap with brain-work. You have your learners “Read 5 of these 8 sources (and others if you like), then write an opinion/commentary on 2 of them. Leave comments on at least 4 other posts; your comments should be substantial. Bring in relevant information from other sources to support your comments as needed.” Repeat every week and pretty quickly comments evolve into meaningful, complex discussions. The learners realize that critiques aren’t red marks on paper, they are conversation; invitation to discourse.
  2. Build up to the big time. When there is a high comfort level, you might move onto oral assessments, taken collectively where students can amplify or refute comments made – shared marks on this kind of assessment can lead to a group of learners really working to build the best possible responses to questions. In this setting the only failure is lack of preparation, thought or constructive contribution. I’ve seen this work in real life – the students all agreed it was the most they’d learned in a class, and the most excited they’d been about preparing for classes.

Make no mistake: the instructor does not get off easily in this format. Sure there’s less lecture on information, but the workload shifts to preparation and mediation Curating a list of readings and references, creating the discussion formats, addressing group culture and checks/balances that will work with the learners at hand will all take some serious time on the front end. Even more important will be knowing the learners, knowing what’s going to convince them that (insert topic here)  is vital to them.

A lot of time especially early on, may be spent providing mediation in conversations in order to keep balance, allow voices to be heard, and redirect non-constructive approaches. This requires a bit of a deft hand, and more than a dash of diplomacy in some cases. But it’s really a big chunk of the learning – learning how to exchange ideas (even differing ones) rationally and respectfully; recognizing that you just might be wrong about some things; and it may be the others are also quite wrong, but collectively you might all hash through the wrong and get to something approaching “right”*. In particular, if this is the first time learners have walked through this model, there may be a learning curve from violent disagreement (or violent agreement) to actual discourse.

When done well, critiques a bit like Beta testing for ideas. Ideally in the course of discussion, terms are defined, assertions are supported and a spectrum of opposing opinions are allowed, all of which refines the original work.  And like Beta testing, ] the learning comes from hashing through the “Whys” in the critiques as much as from the “Whats”.

November 16, 2010

By jleffron


No Comments

Posted In

Learning, Rigour

Artificial Rigour – Searching for a Field Guide

booksRigour is a popular term in learning and training environments.  It gets trotted out a lot in marketing materials as well.  But the problem is that a lot of what get posited as “rigourous” is actually not.  In an elementary school textbook, a work place learning module, or a keynote presentation, you’ll find things that look like rigour, but that doesn’t guarantee that they are.

Someone who really knows a topic will spot false rigour in an instant – much as adults may chuckle indudgently (or cringe) at adolescents who attempt to pose as being much older. So, maybe the first question is ‘how do I spot an expert?’, because they are the quickest, easiest path to spotting false rigour. A real expert is often easily identified  by their ability to accurately reduce a complex concept into layman’s terms without losing the fundamental meaning.  Of course it might take a real expert to recognize that  was done properly – so that’s getting you into a worthless ‘infinite loop.’

Leaving us with a conundrum of the first order: it is very difficult to accurately call out artificial rigour without sufficient expertise. So, what’s a non-expert to do?

My first instinct was to look at the problem from the perspective fields like math and science (simply due to my own background).

In classroom texts it is not uncommon to find a sort of artificial rigor that was created to meet a list of criteria, as opposed to lessons rooted in true fundamental understanding and applicability.  The focus is not on a meaningful “why”, a reason we want students to learn something; it is rooted in lists and box-checking, which are themselves rooted in standards that have as much basis in perception and political agendas as they do in actual learning.

Box-checking driven learning has a high probability of being guilty of false rigour.    So that’s one warning signal, easily found, but it’s only a starting point.

What else comes into play?

We may not be experts on a given topic, but we can take what we know about expertise and use it as a guide.

A while back I wrote:

chalkboard equation“If I’m really, really good at, let’s say, math, then I may not have to stop and think about quadratic equations because I intuitively grasp them; but if asked, I clearly explain (in simple terms) why they have the solutions they have. If I am merely good at arithmetic, I can show you how to solve the equations (just by plugging numbers into the formulas) which might look like expertise to a novice, but is really just mechanics; in that case I know it works but don’t fully grasp why or how.  A lot of false rigour works the same way.

Simon Bostock countered these thoughts with the insight that being able to break concepts down into their component parts may (will) not work for all domains:

“I’m not sure true experts can always unpick and unpick. I think it depends, rather, on the domain.

Maths and physics are inherently unpickable, and the reputation of Feynman as a teacher, therefore, shines. Science depends on the principles of proof and peer-review so being a teacher (ie explaining stuff and testing that it’s been understood) is essentially the same as science. [Warning: massive over-simplification!!!]

But things like medicine, art and computer programming just have to work. We don’t necessarily care how the surgeon genius or the does-the-work-of-a-hundred programmer work. And we certainly don’t trouble them to explain themselves. In many cases, they probably couldn’t because it’s doubtful they’re aware of how they do it themselves – my feeling is that they’re drawing from as-yet-unnamed disciplines, and you can’t unpick things you can’t name” And he’s absolutely right about this…

Different fields having differing degrees of inherent “unpickability”. I can see in the case of, say, a violinist – they can ‘unpick’ the details of technique and tone production, but as far as (for lack of a better word) artistry – well that’s a personal thing, that’s not so readily broken down.  But then again, in that case, I would put the expectations of instructional rigor on the technical aspects, and not assign it to the area of personal expression or artistry.   But we still do need to look at what constitutes rigour (or at least expertise) in topics that are not inherently disectable.

The Role of Narratives

I was helping someone with a technical problem which they were grinding through it rather mechanically, without any real understanding (I could recognize this as I’ve been in the same situation).  I took a comparable problem and broke it down into logical components, but did so within the context of a narrative about the physical reality which the equations were describing.  The same person later was able to discuss another problem with me in terms of meaning, rather than mere mechanics.  They had crossed a threshold, perhaps not into expertise, but at least onto the path that leads there.

Expertise goes beyond merely breaking down a problem into component parts, it’s deeply tied into a narrative.  Real rigour has a narrative rooted in truth;  artificial rigour’s narrative is not entirely so – it looks almost like the truth, but on closer examination the narrative of artificial rigour is either rooted in superficial function, not understanding; or is rooted in fallacy.

We see artificial rigour in this guise in a lot of modern math curricula where elementary texts proclaim sub-sections to be “Algebra” when, in fact, the students do not have sufficient intuitive grasp of numeric relations for there to be any meaning to the work.    It looks like 8 year olds are ‘grokking’ algebraic concepts, but they do not truly do so because their mind is so filled with painful, tedious mechanics so they haven’t the mental energy left to grasp the intuitive connections.

The real narrative is one that shows mastery (and rigour), describing not “what is done” but “what it means”.

For non-techinical areas like Simon’s examples of music or surgery there are two layers.  There is the mechanical aspect of the work, and then there is, for lack of a better word, “artistry’.    If I am a reasonably capable technical musician, I can follow along and imitate styles and variations by talented musicians, but I don’t have the internal grasp to create my own riffs.  To an outsider on the right day from the right angle I might look like I know a bit, but really i’m just a reflection of those who do.  An expert would know that pretty much right off, for someone else it might take some closer scrutiny over a bit of time to realize I can’t really improv like a pro.  A non-expert might not be sufficiently interested to notice.

violinFor ‘unpickables’ (to use SImon’s term), expertise and rigour reveal themselves not imitation but in creation.  The expert surgeon does not exactly mimic his peers, nor does Itzhak Perlman imitate other violinists; they may learn and absorb what other experts do on a technical level, but from that understanding, they can create.  So the teacher of these subjects does not provide rigour through mere mechanics, but through fostering the learner’s innate understanding, challenging it, stretching it. Where Do We Go From Here?

It seems virtually impossible to separate a discussion of rigour from a discussion of expertise.  But it is possible for a non-expert in the field to keep a weather eye out for warning signs.

Artificial rigour tends to lean on the smoke and mirrors of a quick grind through the mechanical motions; this is a common feature of learning based on box-checking agendas.

Box-checking as a concept provides a bit of a compass star to to another indicator – the antithesis to box checking is  understanding, and understanding often reveals itself in meaningful narratives (as opposed to snake-oil style narratives; substance rather than a sales pitch).  Real  experts can create meaningful illustrations, applications, and narrative; false rigour  can only ape what it has heard or seen.

Another measure of artificial rigor is that it tends to make one “feel” good (accomplished, affirmed….).   It is very appealing. Real rigour requires hard work. A bit like climbing a mountain: it may be pleasing in a deep gut level, but it doesn’t come easily or quickly.

I would love to have found a simple check-list (you know, like the box-checking discussed above) to help a novice identify real rigour when they see it.   But then again maybe that’s the point: if you are a novice it’s time to start asking around and finding experts.

The best I can offer is an invitation to continue the discussion.  I still have a lot to learn.

October 13, 2010

By jleffron


No Comments

Posted In

Learning, Rigour

Surely You’re Joking: Feynman on Learning, Textbooks and (un)Common Sense

“First figure out why you want the students to learn the subject and what you want them to know, and the method will result more or less by common sense.” -Richard Feynman, 1952

The concept of real learning can be easy to describe but difficult to achieve.  The work of Richard Feynman provides an interesting case study of the value starting with ‘Why’, and where to take things from there.

Why’, ‘What’…. then ‘How’ The name Caltech tends to conjure the image of highly talented, motivated students, but in 1960 it was clear there was a problem. The standard two-year introductory physics course offered to freshmen and sophomores was actually dampening their enthusiasm.  The classes offered the usual necessary foundational topics in physics, but for students who walked the university in with visions of quantum mechanics, it was more than a slight let-down.  There was not a lot of connection made between what was presented in classroom lectures and where modern day physics was heading.

Feynman recognized that what was missing was the ‘Why’ – the meaning, the reasons, the endgame, if you like, that stemmed from these foundations.  Without a sufficient ‘Why’ the ‘What’ and the ‘How’ are destined to go astray.  So from 1960-1962 he delivered what are now known as his Lectures on Physics.   They were presented to the entire introductory physics class, but the content was geared to spark the curiosity of the most advanced students (with the intent being that practice problems within their recitation sections would shore up practical understanding for others).  The lectures often presented, if sometimes only in a summary manner,  concepts beyond the students’ current understanding, giving them a window into where their studies could take them.  It was the kind of window that a standard, linearly presented course in physics did not provide.

The  Lectures were not (at the time) an unqualified success.  Feynman recognized that the somewhat spontaneous nature of the lectures meant that there was not time for sufficient front-end preparation of practice problems that were to be provided by the instructors of the recitation sections.  Advance preparation is always a key “cost” to consider when looking at non-linear, inquiry-driven learning; it is also key to its success.  Despite the need for better preparation to allow for more effective practice problems, those students who ‘got’ the concepts were inspired and motivated in ways they would not have been otherwise (as were the many graduate students and professors who attended the lectures).   In principle Feynman’s Lectures were on the right track, in practice, he was aware of the improvements and changes needed to make his approach effective (better opportunities for practice and support).

Surely You’re Joking… Evaluating Textbooks

In the book Surely You’re Joking, Mr. Feynman, there is a memorable chapter regarding a time in the 1960s when Feynman was asked to help review math textbooks for the State of California’s school system.  The whole story is worth reading (being both disturbing and entertaining) and can be found online.   The experience proved to be a bit of a shock for Feynman as he went through book after book. All the texts tried to embody the kind of real learning that Feynman himself strove to provide, but each one was guilty of a serious shortcomings: inaccuracy, poor terminology, and ridiculous problems.

The root of the issue for all the texts was a sort of artificial rigor that was created to meet a list of criteria, as opposed to lessons rooted in true fundamental understanding and applicability.  Additionally, despite the names of ‘experts’ listed as authors of textbooks, the actual mass assembly process used by publishers tended to involve many authors of limited knowledge and skill; the names experts were every bit as much window-dressings as were the aspirations to suggest the course was rigorous and correct.  It is an example of expediency and costs driving content.  Creating meaningful learning takes time, effort, and deep understanding of a concept.  (Sadly, having been on committees that review Math and Science texts, I can vouch for the fact that little has changed in the intervening decades.)

Feynman immediately recognized the lack of both substance and of meaningful practice in the books.  In this case, the ‘Why’ for learning was: to meet standards generated by state bureaucracy.  This made it unlikely that the ‘what’ or ‘how’ of learning were going to to be any more meaningful; if there is no real goal, it’s unlikely there will be meaningful practice; expedient checking off the boxes becomes the priority goal, taking precedent over deep learning.

Three Easy Pieces

When comes down to it, providing the opportunity for real learning is quite simple, at least in principle:

Remember the Why Context matters.  ‘Why’ you are learning something drives everything else from motivation (it’s your ‘elevator pitch’), to sense-making, to methods.

Front End Preparation Good learning requires sufficient front-end preparation that students have worthwhile opportunities to practice and learn.  This is how they make understanding their own.  Lack of advance preparation leads to a lot more box checking and micromanagement.  Yes, it’s more work at the beginning, but if the goal is learning, not just ticking things off the list… Rigor Needs to Be Real, Not Just Window Dressing Rigor for rigor’s sake can lose track of the ‘why’ and become another form of “box-checking”.  When you know the ‘why’ you have the chance for meaningful teaching based on deep knowledge.

August 17, 2010

By jleffron


No Comments

Posted In

Knowledge Management, Learning, Rigour

Information Filtering: the adaptive capacity of the human mind

Each time I’ve read something about “information overload” and how we need better external filters, it’s left me with a vague sense that we are over-looking the obvious.  And it really is obvious, once you stop and think about it:

Human minds are born to filter.

More than that, they are born to build very sophisticated, constantly evolving filters.  You know this intuitively; if you’ve ever taken a walk with a one year old child, they will stop and see every detail – every variation in grasses, or tree leaves; they stop to evaluate every sound, to admire every insect.  You don’t do this; your brain has learned to filter.

There have been a number of studies on infant language development.  A recent study looks at the filtering strategies employed by 18-24 month olds as they work to distinguish and learn individual words from the ambient noise of conversation. A study back in the 1990s looked at how very young children (around age 2) had already learned to filter the normal variations of pronunciations within their native language, but would respond to extremely subtle variations of pronunciation in sounds that were not present in their native language.

Those are some pretty complex filters developing in very young minds.  And our filtering abilities grow and develop throughout our lives; we filter staggering amounts of information every day.  Don’t believe me?  Step outside and turn off your filters.  Try to catch how many ambient sounds, scents and visual details your brain has routinely learned to dismiss because they do not require action – they are background noise, safe, uninteresting.  Thousands of inputs filtered out every second.

You really notice the amount of daily filtering you do if you move to a new environment.  Your brain doesn’t know what sounds or smells it can safely ignore, nor what normal weather patterns look and like, nor which insects it can allow you to simply overlook, what social cues are relevant.  So your senses are bombarded with a much higher level of detail; it can be overwhelming (and exhausting).   But over time your brain builds new filters for the new environment.

To some extent we are starting to see this evolution in our increasingly information saturated 2.0 world.  On first exposure our mental filters are as overwhelmed as they would be if one moved from Duluth to Mumbai.  Over time our filtering ability evolves; we do, after all, have minds capable of the complex  and unceasing inputs of the natural world.  But as we deveolop our media filters, we do have to take care that we are filtering well.**

**e.g.  Simon Bostock discussed an important article, Six Views of Embodied Cognition which, among other things, looks at the cognitive strategies (often short-cuts) we use when time pressured.  If those pressure driven strategies are employed consistently over time, it seems possible they may lead to over-filtering so that it becomes habitual to merely skim the online information stream, instead of selectively reading certain items with the same level of attention one would give to reading a good book.