Note: You can read this post on my website, where you can also read my other thoughts and subscribe via RSS. You can also talk to my chatbot about my posts.
As a K12 practitioner, I'm wondering if your group spent any time thinking about the same problem, but with foundational skills. This affects the first consensual question of who the course is for. Do you see what I mean? What can you tell us about the evolution of students profile/skills when they first get to your course? Is it too early still?
Really interesting. I do think it's too early with respect to foundational skills affected by AI (I'd say we're just now starting to get students who spent at least one year of their college time leveraging AI extensively). But a taxonomy for those foundational skills would be very helpful. Pre-AI when talking about who our students were coming in we'd talk about: exposure to and comfort with probability, ability co compute statistical quantities like medians and percentiles, and basic math not being a barrier (this is just off the cuff so sort of superficial). But I suspect we're going to have to get into more foundational, and more abstract, skills coming into the course as people start diverging because of differential uses of AI.
The question of whether we can exercise judgement without doing is really interesting.
The sociologist Harry Collins has an interesting take on this when he separates "interactional expertise" from "contributory expertise." Contributory expertise is the ability to do a task (which entails judgements that become somewhat taken-for-granted for the expert, a matter of tacit knowledge) and interactional expertise is the ability to talk about the task to an expert in ways that the expert will appreciate and will contribute to the discussion but without the ability to actually do the task.
The relationship between contributory and interactional expertise is complicated. Tennis coaching requires interactional expertise; but tennis coaches often were players before they became coaches (probably not great players but still; that said, not every good player can be a good coach). A science journalist immersed in the science they cover is probably an interactional expert but not a contributory one (though often we see science reporters have a background in doing science). Research managers often start off as researchers but once they become managers, they lose the ability to actually do an experiment.
In statistics, of course, there are people who can actually do statistics; but a lot of researchers are able to talk about statistical results with fluency.
I think part of getting AI to do certain tasks is that we are going to have to think carefully about what expertise we want to develop in people: the boundary between contributory and interactional expertise will have to shift. But for that, I think, we're going to have to decide what's important. So in that sense, this was a very enlightening read because it gave me the sense that even statisticians are figuring out these boundaries.
Thank you for the comment! If I may ask you instead of researching it myself: does Collins have strong views on whether interactional expertise can be developed in a way that is different (perhaps less “hands on”) than contributory expertise? At this convening I think there was a fair amount of skepticism from several attendees that the two can be taught in different ways. Would love any examples you’re aware of!
He does. He thinks it's possible. That's the dispute he is having with Dreyfus in the link I put in. Dreyfus thinks that an AI program cannot have human-like capacities because it doesn't have a body to have experiences. Collins thinks that if you think all expertise is contributory, then that is true; but there is interactional expertise and humans can achieve interactional expertise without contributory expertise (a science journalist among scientists; a sociologist like Collins himself who was able to talk to gravitational wave scientists on their own terms and read their correspondence and papers without being able to do an experiment) and so there is no reason that an AI program can display knowledge without having an embodied experience.
And I think ChatGPT has ultimately proved Collins right. I ask it all sorts of questions about cities it has never visited and it gives me perfectly good answers.
But that said, there's no clear-cut way to apply all of this to the teaching of statistics. I think it has to start from the kinds of tacit knowledge that being a competent statistics practitioner presumes and where that tacit knowledge comes into play when someone is asking Claude to do statistics for you.
Thanks for this level of detail! I have generally been on the side of interactional expertise being teachable without having to be able to do it all yourself (as a teacher of policy students, I focus mostly on interpretation and critique of statistical analyses for my students, not the production of analyses), I also realize that the judgment that I find so essential in my work with AI tools came from a lot of really hard-won intuition that took years to develop.
It may be that the development of that judgment is just a happy side effect of all the analyses I learned to do, and I can instead focus on the judgment as the main thing I’m developing, but I have been struggling to find models of this in other disciplines that I can point to. Have you encountered any?
As a K12 practitioner, I'm wondering if your group spent any time thinking about the same problem, but with foundational skills. This affects the first consensual question of who the course is for. Do you see what I mean? What can you tell us about the evolution of students profile/skills when they first get to your course? Is it too early still?
Really interesting. I do think it's too early with respect to foundational skills affected by AI (I'd say we're just now starting to get students who spent at least one year of their college time leveraging AI extensively). But a taxonomy for those foundational skills would be very helpful. Pre-AI when talking about who our students were coming in we'd talk about: exposure to and comfort with probability, ability co compute statistical quantities like medians and percentiles, and basic math not being a barrier (this is just off the cuff so sort of superficial). But I suspect we're going to have to get into more foundational, and more abstract, skills coming into the course as people start diverging because of differential uses of AI.
The question of whether we can exercise judgement without doing is really interesting.
The sociologist Harry Collins has an interesting take on this when he separates "interactional expertise" from "contributory expertise." Contributory expertise is the ability to do a task (which entails judgements that become somewhat taken-for-granted for the expert, a matter of tacit knowledge) and interactional expertise is the ability to talk about the task to an expert in ways that the expert will appreciate and will contribute to the discussion but without the ability to actually do the task.
The relationship between contributory and interactional expertise is complicated. Tennis coaching requires interactional expertise; but tennis coaches often were players before they became coaches (probably not great players but still; that said, not every good player can be a good coach). A science journalist immersed in the science they cover is probably an interactional expert but not a contributory one (though often we see science reporters have a background in doing science). Research managers often start off as researchers but once they become managers, they lose the ability to actually do an experiment.
In statistics, of course, there are people who can actually do statistics; but a lot of researchers are able to talk about statistical results with fluency.
I think part of getting AI to do certain tasks is that we are going to have to think carefully about what expertise we want to develop in people: the boundary between contributory and interactional expertise will have to shift. But for that, I think, we're going to have to decide what's important. So in that sense, this was a very enlightening read because it gave me the sense that even statisticians are figuring out these boundaries.
Collins has many publications but this is my favorite one: it's a response to Hubert Dreyfus' analysis of what AI (the old-school one) can and can't do. https://www.sciencedirect.com/science/article/pii/0004370296000836
Thank you for the comment! If I may ask you instead of researching it myself: does Collins have strong views on whether interactional expertise can be developed in a way that is different (perhaps less “hands on”) than contributory expertise? At this convening I think there was a fair amount of skepticism from several attendees that the two can be taught in different ways. Would love any examples you’re aware of!
He does. He thinks it's possible. That's the dispute he is having with Dreyfus in the link I put in. Dreyfus thinks that an AI program cannot have human-like capacities because it doesn't have a body to have experiences. Collins thinks that if you think all expertise is contributory, then that is true; but there is interactional expertise and humans can achieve interactional expertise without contributory expertise (a science journalist among scientists; a sociologist like Collins himself who was able to talk to gravitational wave scientists on their own terms and read their correspondence and papers without being able to do an experiment) and so there is no reason that an AI program can display knowledge without having an embodied experience.
And I think ChatGPT has ultimately proved Collins right. I ask it all sorts of questions about cities it has never visited and it gives me perfectly good answers.
But that said, there's no clear-cut way to apply all of this to the teaching of statistics. I think it has to start from the kinds of tacit knowledge that being a competent statistics practitioner presumes and where that tacit knowledge comes into play when someone is asking Claude to do statistics for you.
Thanks for this level of detail! I have generally been on the side of interactional expertise being teachable without having to be able to do it all yourself (as a teacher of policy students, I focus mostly on interpretation and critique of statistical analyses for my students, not the production of analyses), I also realize that the judgment that I find so essential in my work with AI tools came from a lot of really hard-won intuition that took years to develop.
It may be that the development of that judgment is just a happy side effect of all the analyses I learned to do, and I can instead focus on the judgment as the main thing I’m developing, but I have been struggling to find models of this in other disciplines that I can point to. Have you encountered any?