Learning through formative comment
Alexander Coward, a math lecturer at Berkeley, shares an interesting piece of education theory on the impact of grades vs. feedback on learning:
The research entails dividing students into three groups. The first group has their work graded in a summative way, for example with a letter grade or percentage score. The second group receives a formative comment, which is feedback on ways their work is good and should be built on, as well as feedback on areas where their work needs to be improved. The final group receives both the formative comment and the summative grade or score. These three groups of students may then be tracked over a period of time and their learning measured. The results of such an experiment are startling. The students who learn the best are the ones in the second group, who receive feedback but no score. There is then a big gap to the students in the final group, who receive both feedback and a grade. Finally there is a smaller gap to the students who receive only a grade.
This observation is made as part of his teaching statement, which encourages insight through reflection.
First, I wondered how much my learning in school was stunted because I cared too much about grades.
Then, I thought, how does this apply now that I’m no longer in school?
If you are in a job rather than in school, you probably no longer get grades. There may be numbers appended to your performance in the form of compensation changes, revenue targets, and the like. But if you take improving your work seriously, you know it has little to do with maximizing any number.
So if we know feedback is important for learning, how do we find ways to receive enough “formative comment”?
Coward explains how he finds ways to give formative assessment to students, even if it’s a large lecture class and it is logistically impractical to give individual feedback. For instance, as he goes through solutions in class, he encourages students to compare their solving methods with his. He also goes through poor, medium and perfect solutions and explains the differences between them. In effect, he is getting students to practice assessing their own work. He provides them with guidance on what is better and what is worse. They take that information and self evaluate.
How does this apply to learning situations outside the classroom?
First, improving overall performance is about learning to do specific things well. This echoes what is understood about deliberate practice, where one has to break down great performance into the mastery of specific skills and then work repeatedly at those skills.
Second, reflection is critical to learning. You cannot just rely on performance reviews or external advice for feedback. You also need to use observation and reflection to gain formative comment. You must observe bad execution and good execution and reflect on what separates the two. There usually won’t be a teacher to point out the difference to you, so you have to rely on honing your tastes.
If you read Coward’s full teaching statement, you’ll see that this is someone who has reflected deeply on what excellent teaching entails. He has processed it from a philosophical level, he has methods and experience, he even has practical tips. When I learn about someone and think, “Oh there’s a professional,” I usually mean I admire their work ethic. But there is also this depth of reflection that comes with the territory.
So if you had to give yourself formative comment every day, what would you say?
Image from The People Equation.
Transitions and discoveries
I’ve gone quiet on the blogging lately because first I was bored and then I became too busy.
I crossed the 2 year mark at work a few months ago without saying much about it, but it’s a huge milestone for me. I have spent longer at this startup job than any job I’ve had before. Many of my b school classmates are on their 2nd or 3rd jobs since graduating. Most picked much larger, more established organizations. I picked the uncertain startup job. It’s been an amazing ride.
I’d like to say that the lesson here is that you should follow your passion and stuff, but I don’t really believe that. I do think you should choose and not drift into a job just because some companies are recruiting aggressively or because everyone else thinks some jobs are more desirable. But I think the real lesson is to make the most of the opportunity at each step. If you have a chance to choose, choose well. Once you have chosen, do the job well. The people who seem most unhappy seem to be the ones who never really chose and then never really got out of choosing.
I recently transitioned into my third role here. You could look at the third role as an extension of the second. But when the job scope more than doubles, I have to rethink even the way I did my earlier job.
I get to feel like a beginner again. In some ways, what I have to deal with now is much harder than anything I’ve had to handle before. In other ways, it just feels like the next step. If you scale enough steep learning curves, they no longer look as daunting.
People come to me with all kinds of problems. The range is both a source of stress and of fascination. By the time it gets to me, it’s probably because the standard process can’t handle it or something has failed or it’s extremely important that we not screw this thing up. For the first few weeks, when I got pulled into these situations, my first thought was usually, I have no idea what’s going on. Other people have thought about it for weeks, I’ve just learned about it, and I’m supposed to fix it? Okayyy.
So you learn to ask helpful questions to tease out signal from noise. You learn to assess risk. You marshal resources. You use whatever influence you have to lean on the right pressure points. You prioritize ruthlessly. You hope that the time you’ve spent digging into the details has given you enough intuition to make good calls quickly.
That’s the entire ask really - make good decisions quickly.
I cannot seem to get used to telling people “no”. Prioritization is an inherent part of the job. I say “no” all the time. In fact, I’ve had to say “no” a lot more lately. Often to pretty good ideas. But I cannot get comfortable with it. Fortunately, my comfort is not a requirement for getting the job done. Maybe it’s like what they say about courage - that it is not the absence of fear, but action despite fear.
It has totally surprised me that I can now take a lot of back to back meetings and not lose focus. I used to look at my half-packed schedule and wonder how people with fully packed schedules handle it. How do you give your 5pm meeting the same attention and clarity as your 10am. I don’t do it every day, but I now believe that you can develop the capacity for it. You have to want to get good at it, but it can be learned.
A generous empathy
"The episode really tried hard to hold all of the perspectives at once, but they were incompatible. I just liked that… It requires empathy, it requires a sense of objectivity, it requires a sense of being emotionally open, but also being emotionally distant and analytical, so that you can balance. Somehow, on the deeply human part of what we do as storytellers, it all came together in that story."
- Jad Abumrad, on his favorite episode of Radiolab
When I read that part, I thought, this. This is what it’s all about.
I feel like, holding multiple perspectives and extending your empathy to each one of them, that’s like the holy grail of good management. I know, the Radiolab guys are talking about great storytelling and this incredibly rich, creative work. I’m talking about, well, how you run a good meeting.
But it’s the same thing, isn’t it? If you can get yourself to listen to each person and listen to all the multiple intents at work and the many layers of voiced and implied conversation, if you can listen to all that with the broadest empathy you can muster, then I think you get close to the reality of the conversation. Because that’s what you’re always trying to get to. You’re trying to silence your biases and just listen. Just see what’s going on objectively. Just see reality. Because that’s when you can get the best conversation out of everyone and get to the best decisions.
The whole thing is so hard. We fail more often than we succeed. But I think we just have to keep trying in that direction. Get closer to reality. Keep holding open your empathy.
On letting your work change you
"I am using this job to become a better human being."
My friend has known all along that his primary mode of operating is “bulldozer”. He doesn’t just lean in, he charges ahead. By sheer willpower, he can execute anything into being. It has served him well, but he was also discovering its limitations. The universe was giving him feedback in the form of growing frustrations. Bulldozing was no longer enough.
So he is starting to explore the way that other people operate. How do you motivate the employee who just wants a 9-to-5 job. How do you give feedback to someone who can only hear that you are upset and cannot hear that you are trying to help them grow.
At such a juncture, it seems to me that one can choose to get out of this line of work and go find one that better suits his ambitions. Go find a company that only hires A players. Go find some private equity firm where that aggression is rewarded.
There is a time and place for changing one’s environment rather than changing oneself. He chooses to change himself.
So he learns empathy. Even though it doesn’t come naturally at all. He learns to understand why people might carry different motivations. He is trying to check his raw ambition with compassion.
I think it’s wonderful.
I think we don’t talk about this enough. We talk about skills, like number-crunching, analytical skills, and they are important. We talk about “soft skills”, like clear communication and working well in teams, and they are even more important.
But we don’t talk about character.
(Even writing that here feels awkward.)
I love this idea about how to make work humanizing.
This is the other side of the same coin: can you use work to practice becoming a better human being?
Can you use it to exercise compassion if you are a bulldozer, to practice putting yourself out there if you naturally withdraw, to display calm if you are hotheaded, to reveal vulnerability if you are stoic.
I believe work at its highest, should you choose to take it on, has that potential. You can let it sculpt you.
Atul Gawande on his career failures and rescue
When I graduated from college, I went abroad to study philosophy. I hoped to become a philosopher, but I proved to be profoundly mediocre in the field. I tried starting a rock band. You don’t want to know how awful the songs I wrote were. I wrote one song, for example, comparing my love for a girl to the decline of Marxism. After this, I worked in government on health-care legislation that not only went nowhere, it set the prospect of health reform back almost two decades.
But the only failure is the failure to rescue something. I took away ideas and experiences and relationships with people that profoundly changed what I was able to do when I finally found the place that was for me, which was in medicine.
-Atul Gawande, who has an extraordinarily storied career in medicine and writing, on Failure and Rescue
What he doesn’t say in his speech is that he was a Rhodes Scholar and that he went to Harvard Medical School, so it’s not like his career was ever seriously going nowhere. But still, I would like to think that it was not a foregone conclusion that he would become as influential as he is today.
Creating a sense of self-propulsion
I received an email that made me very sad. It gets to the heart of what I think we don’t answer in the Singapore Scholarship Guide: what are one’s options if you don’t take a scholarship?
This Singaporean student is deciding whether or not to take a scholarship, and cannot afford to go overseas for college without one. This person wrote:
"I think very few of us have any idea what taking charge of our own destiny means… Even if I do decide to study locally, I don’t see how it can be much different, i.e. choose degree - study - graduate - choose job in related field. Just an extension of the education system so far. There isn’t really much opportunity to ‘find myself’ in the sense that an overseas liberal arts education allows. So, in perhaps an over simplistic way, am I trapped?"
When I think back to my high school self, I understand how this person feels. Up to that point, I don’t think I did anything that I would consider vaguely original or self-initiated in my life. I didn’t know how to start. (My high school self would probably be appalled that one day I would write a public blog that has become a platform for helping strangers break their bonds. !!)
It was in college that I was exposed to a dazzling range of opportunities. That is where my ability to invent possibilities for myself was born. But really, it was the bondbreaking that sealed it. I made one big decision to choose something I really wanted, that nobody else really wanted for me. From then on, the only perspective that made sense was that I can choose for myself.
This got me thinking: can you achieve a similar effect by other means?
I don’t think the question is about whether local Singaporean universities are as good as overseas ones. I don’t even think the question is about the value of living overseas and all the changes in perspective it can bring. All students - from any country - should try to find a way to spend time abroad, and students in Singaporean universities have no shortage of such opportunities.
I think it’s about creating a sense of self-propulsion. That you can take charge of your own destiny, as the student puts it.
But how does one create a sense of possibilities? What are things you can do if you have no idea where to start?
This got me thinking about the idea of a Self-Propulsion Project.
You need to do something that you want to do. You need to get used to the idea that you can craft your own outcomes. Choose a project that will give you a sense of doing something original for yourself. Don’t choose something that the system tells you is good to do, or that your parents think you should do, or that all your peers are doing. Pick something that is for you.
So. What could you do? Ideas for starters:
- Learn to code. There are so many resources for this. Build something.
- Take an online course in something that intrigues you. There are lots of MOOCs out there. Don’t pick something that you should know. Pick something you want to know.
- Go on a round the world trip. Again, so much info out there! People have done it on anything from $12K to $30K (and of course, much more). But it can be done for less than 10% of the S$500K your bond would be worth.
- Run a project that will earn you $1000. See SVA’s Entrepreneurial Design syllabus on their $1K project.
- Fundraise to sponsor a water project for charity:water
- Get inspiration from the things you could do to get yourself an arts education if you didn’t want to pay for art school.
"But all of that is so hard!" you say. Of course, it is. Leading an interesting, fulfilling life, in general, is hard. All this takes an incredible amount of discipline. It takes saying “no” to a lot of things that your peers or that the system might want you to do. It even takes saying “no” to things that you want to do that are easier - like wasting time on Facebook.
I have to admit that it never occurred to my high school self to attempt any of that. I lacked the knowledge that there might be other options. I couldn’t imagine the possibilities.
But things change, and one day very far down the line it became perfectly reasonable to do things like run a Kickstarter and teach a class and publish an ebook. On a larger scale, it becomes possible to go after interesting jobs that you believe would be engaging and fulfilling, even if from the outset you feel completely unqualified to do them.
There is this quote that life is an exploration of one’s appetites. I think that’s the idea that Singaporean students (and Singaporeans and all other people) need to get used to.
So here is my offer to you:
If you are a Singaporean student or unhappy scholar who wants to try out a Self-Propulsion Project, email me your project plan. I will work on it with you and help hold you accountable. Set yourself a deadline to email me - a week from today whenever you are reading this post - and do it.
I wrote an “about” page for the blog
I know, finally. Read about me.
On startups and having an impact
If you ask people why they want to join startups before they join one, one of the most common answers is, “I want to have an impact”. You can have a larger impact on a small growing company, they believe. I believed. My experience has shown me this is true.
When I first started tracking my product’s revenue more closely, many quarters ago now, I remember comparing it to the numbers I used to see in my investment banking days. I once worked on a deal that raised $200 million in equity and took just 48 hours of work. The numbers we come across in a startup are a tiny fraction of what I used to see. But you know what? I remember thinking that you can take my product’s revenue numbers and draw a fairly direct line to the team - at the time, just 4 engineers and me. Of course, the product’s success depends heavily on the support of the rest of the company, but the feeling of direct ownership was real. I felt like I could say, I drove that. I was not doing it as a cog in a gigantic machine with a long, white-shoe history and well defined processes that had been performed by tens of thousands of employees. I felt very much that if I did a better job, things - not just for me but for the entire company - would be better, and if I did a worse job I could actually put a lot more at risk.
Everyone wants to have an impact. I used to think you had to be doing some kind of save the world work, and for some people that’s what it takes. But for me, I have come to appreciate that impact really means feeling that you have unique influence over outcomes.
As for what generates that feeling, I believe it has to do with creativity.
Early on, I remember having to prepare slides for a checkpoint meeting and asking, “Is there a template for this type of deck?” The response was, “You could create one?” We now have all kinds of templates, but back then, I was constantly inventing new ways to do things because they just hadn’t been done before.
These days the most creative part of my job is probably defining what it should be. As a Product Manager, I go to work each day and there are things to respond to - analyses for why something happened, questions about how something should work, custom requests for whether something can be done - and also, of course, getting features to ship. Responding to things has a way of expanding to fill whatever time is available. The challenge is to create space for planning bigger, thinking forward. What should the product be doing? What should the team be doing? What am I not doing now that I could be doing that would kick things up a notch? Lately, it’s been about reaching out to people in functions I don’t usually work with to better understand how different products and functions should interact.
If you feel that you can shape your job to reflect your thoughts and beliefs, if you feel you can infuse it with your originality, and you see it pay off in results - products ship, revenue increases, morale rises, satisfaction increases - that feels like impact. We come back to one of my favorite ideas - how can we do what we are.
Image by gomattolson.
Borrowing a page from the nomad’s book
I am not the kind of person to sell all my possessions and go traipsing around the world. I also have no wish to go totally minimalist and live in a 400 sq ft studio with a bed that folds down. I am no packrat, but I like employment and fancy meals and a comfortable bed with nice bedsheets.
But when I read this post by Vanessa Runs about leading a nomadic life, a lightbulb went off. She wrote:
At home, there were chores like dishes and laundry and dinner and cleanup… Moving into a 22-foot RV, 97 percent of my previous To Do list was eradicated. There was no office to spend my day in. There was no house to maintain.
Again, it did not make me want to move into an RV, but…
I had ordered an iPhone car mount from Amazon. This past weekend, I spent hours fretting over the fact that did not show up. I had ordered on Prime. The delivery tracking said it was delivered, but it was nowhere to be found. I checked the mailbox. The management office didn’t have the package either. So I searched on Amazon.com for what to do. I got to a page that said “contact your carrier”. So I called Fedex, which said it’s “smartpost”, so it’s USPS’s fault. Great. So I called USPS and couldn’t get through (of course) and tried to email them.
Eventually I found a way to send a note to Amazon. They got in touch with me and offered to send me a replacement at no charge.
By then I had spent hours fussing over the whole ordeal. Except it is not an ordeal, I know. I made it an ordeal in my head.
The item was worth just $20.
In the same weekend, I had to return a dress I had ordered online that didn’t fit. I had to resolve a car insurance issue. I had to pick up a couple of things from the mall, which resulted in 20 minutes of frustrating circling in a busy parking lot.
I don’t even shop much. But there is just so much stuff to maintain.
Reading about the nomadic life made me realize that if I simply hadn’t bought the iPhone car mount, I would not have had to deal with all this. Of course, you could argue that Amazon package delivery should simply work, and that the convenience of having the mount so I can use my phone as a GPS outweighs the cost.
But if I did sell everything and move into an RV, what would that be like? All the bills I’d no longer have to pay. All the things that I’d no longer have to repair or replace, because I just wouldn’t have those things.
Granted, there would be some real inconveniences. I imagine laundry will be more inconvenient in the RV world than in the washer/dryer in-unit world. But on the flip side, maybe I’d have just a few sets of clothes. Maybe I’d stop wearing clothes that require ironing or dry cleaning. I’m sure there are many things I would miss, but…
All I’m saying is that when we acquire things, there is a real peripheral cost. Unknown acquisition cost. Unconsidered maintenance cost. Unaccounted brain space cost.
If you have less stuff to maintain, you open up time for other things. Vanessa Runs talks about the Project List she was able to make time for:
These were not tasks that repeated on a daily, weekly, or monthly basis. These were big ideas and huge projects that would take months—years, even—to see through.
That sounds nice, but it’s not clear to me that you need to get rid of that much stuff to make space for projects.
What is much more clearly appealing to me is what she writes about in Why We Need Nomads:
These days I do nothing just as often as I do something.
That is oddly appealing. Making space to appreciate nothing.
Image by Matt Lemmon.
On experiencing art in NYC
On this trip to New York, our days revolved unexpectedly around a lot of art.
We walked the full length of the High Line.
We saw the James Turrell exhibit at the Guggenheim.
We spent time wandering around The Cloisters.
We took the quickest zip through The Met ever with only stops to the Temple of Dendur and the rooftop garden.
It’s only when I shift to a very different pace that I realize that so many of the things that occupy me - the self-optimizing, the constant information consuming, the bits and bytes - all these things are so abstract.
But here we were engaged in walking and seeing and sensing.
At James Turrell’s Iltar, that rectangle of chalkboard grayness is not a solid wall but actually a cutout. You are staring into a space on the other side over 10 feet deep, but your eyes tell you you see flatness.
At Aten Reign, those ovals of changing color we see staring up into the rotunda of the Guggenheim are lit by changing artificial lights. But did you know that at some point the lights were off and you were staring at pure skylight?
The city is represented differently when viewed from the High Line. It feels a little quieter, slower, more like summer. You are both more distant from the intensity of New York and further immersed in it.
At the Temple of Dendur at The Met, it is dazzling to think that 2000 years of history separates you from the civilization that carved this stone that sits here.
I’m reading Michael Pollan’s A Place of My Own, where he describes how he fell in love with the idea of building his own house. He is someone who “works with words and symbols every day” and wanted to immerse himself in something more tangible. He talks about:
…this sense of living at too great a remove from the things of this world and the life of the senses.
I get that. It was refreshing to experience through the senses, instead of just through backlit screens.