Through the ongoing debate about assessment via Purposed I came across this presentation from last year by Derek Robertson at the eAssessment Conference in Scotland.
There are a few points that I wanted to highlight that are worth drawing out and discussing further. Just to say that, if you didn’t already know, I am an advocate for games based learning and how it can positively impact on the work we do in school, so it was great to see Derek sharing some of what he has learned regarding assessment.
Derek outlines in his presentation what we can learn from games and what they are very good at:
- Giving dynamic and ongoing feedback
- Presenting incentivised learning experiences
- Using meaningful profiles and reports
- Trusting in the ability of the player/learner
- Nurturing growth mindsets
- Maximising the potential of peer assessment
- Presenting purposeful and relevant learning intentions
- Ensuring assessment is not “done to” learners
- Giving the players the best chance of success
If you notice from Derek’s points he uses the terms player and learner interchangeably as we have to learn to be successful and progress through a game. So it is natural extension that just about all game mechanics pivot around a player being a learner.
The first of Derek’s points: “Giving dynamic and ongoing feedback” is what in my opinion refers to formative assessment. It is the “ongoing” assessment that takes place. Lots of the examples he shared in the presentation were in fact summative assessments, goals scored, points in total, notes correct. You may even argue that unlocking badges or bonus material is summative as it is the result of a set of actions within the game; on the other hand it signals progress and is provided on the course to an overall goal. Perhaps here is where the definition becomes a little blurred.
During his Slash-like demonstration on Guitar Hero Derek referred to the summary score sheet including notes completed correctly, but it is the feedback during gameplay that interests me the most. The types of “dynamic and ongoing feedback” that help a player improve at the point of learning – the summary sheets help us to reflect on how we scored but this is the same as what grade did I get.
Guitar Hero gives all sorts of feedback during gameplay that encourages a player/learner to adjust their play – this comes through visual cues such as simple traffic light dials, auditory signals from the sound of the correct or incorrect notes being played and of course points and mini-goals that further enhance what can be seen.
These are all straight forward and can be seen throughout many games – perhaps it is the timing and overall strategy of ongoing feedback that would reveal something inherently more valuable to teachers. Not just seeing the individual method of feedback in isolation but placing it within the whole picture, the whole plan for supporting new players and helping them to be successful.
Incentives are also important with regard to learning experiences and Derek makes this point in his summary. This is illustrated in more detail by Girlie Delacruz’s work on” Games as Formative Assessment Environments” in which she conducted some studies with regard to how formative assessment and feedback affected maths and game performance. They used a purpose built game to learn about fractions and various parts of the game feedback were altered and presented to different groups (see the study detail in the presentation below).
Delacruz summarised the outcomes, explaining that: “Incentive + Scoring Information is superior to minimal scoring information, with better performance on:
- Math achievement measures
- Game play”
In the game their is a simple structure to work within and normally a game “currency” that can be used to incentivise a player – in the work from Delacruz it was simply points (see Slide 18) but what would that be in the classroom? Perhaps something meaningful within the topic or project? In a previous post the debate shifted to these short term incentives and Oliver Quinlan pointed out in a number of comments:
Unfortunately points scoring and rewards are in the short term ‘easy’ ways for teachers to motivate pupils to do what the are told. Look at the number of ‘team points’ and ‘star charts’ that exist in primary schools. This may get them to behave in the required way, but it teaches pupils that only things that are worth doing are things that get them a number score…
I think it is worth bearing in mind always whether we are rewarding children or just bribing them. That is just behaviour, let alone the potential implications for motivation and dispositions to learning that happen if children are trained to only value tangible and quantifiable outcomes like rewards and grades. Dylan William’s work has shown the impact that losing rewards and grades can have for intrinsic motivation, and focusing attention on learning rather than just the outcomes.
Once more it seems we need to strike a balance and create a system that makes best use of incentives for learning as per game design but perhaps addresses what Oliver points out, making them more meaningful and broad so that they do not remain Pavlovian nor isolated within that context.
It would seem that the role of the incentive is crucial in game mechanics and how a player progresses and indeed learns using a game. The question for me would be how do I use these ideas within my own teaching? Do we try and design an incentivised curriculum project? What practical ways can I implement such a system with not only one player but potentially 30?
Thank you for discussing the importance of time management and productivity when juggling multiple money-making endeavors. It’s a valuable skill set for maximizing efficiency and effectiveness. click here for more details.
Your post highlights the potential of creating and monetizing content in the digital age. It’s a powerful way to share knowledge and generate income. For more insights, click here.
(oops, sorry, i was copying and pasting and it looks like a bunch of garbage got embedded into the html). My apologies.
Hi Derek and Tom, if you don’t mind my jumping into this discussion–but you guys are touching upon some things I have been focusing on a lot lately. I agree that the formative/summative distinction is quite murky. I sort of like Taras’ take on formative assessment (which is actually closer to that of Scriven (1967) and Stabler (1989;1998):
@font-face {
font-family: “Cambria”;
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: “Times New Roman”; }div.Section1 { page: Section1; }“the process of formative assessment can only
be said to have taken place when feedback has been used to improve the work;
whether this work is used for accreditation or within a less formal arena does
not affect the definition.” Taras (2005, 2007, 2010)
@font-face {
font-family: “Cambria”;
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: “Times New Roman”; }div.Section1 { page: Section1; }
tries to make a distinction between “intent” (i.e., the designer’s intent of the uses of the assessment or as she puts it the assessment’s “function”) and how it is actually used (i.e., the assessment “process”). I think Popham makes a good point about this also in one of his papers–that an assessment is only formative when a change in behavior occurs. So, achievements or badges have the potential to serve both either summary or formative purposes–depending on how the user uses that feedback. If the user walks away and says, okay that’s that–I suck at Guitar Hero–that it’s the “end” or summary of his/her performance. However, if the user chooses to use that information to decide to figure out what to do next (i.e., practice, ask for help, etc.,) then that information can be said to be formative. I think what might be more useful is looking at how the assessment criteria or parameters are implicitly or explicitly communicated to the student. With implicit parameters, any external person to the designer of the assessment (i.e., student or other ‘assessor) has to try to figure out what points are salient for the assessment. In contrast, explicit parameters make the criteria transparent and clear and start to open up shared meaning @font-face {
font-family: “Times”;
}@font-face {
font-family: “Cambria”;
}p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: “Times New Roman”; }div.Section1 { page: Section1; }(for a discussion of
‘fuzzy criteria’ see Sadler, 1989). That is why the other part of my study was looking at how making the scoring rules transparent in a learning game (especially if they were being evaluated based on performance directly tied to the academic domain) could be used to increase performance. Similar to the work in classrooms of student use of rubrics to guide performance or peer assessment. Anyhow, thanks you guys for indulging my ramblings. Looking forward to more discussions.
Hi, Girlie Delacruz here. Came across your blog and wanted to address a few of your points (if you don’t mind)? Regarding incentives–the research has shown that incentives do not have negative repurcussions if they are used for tasks that originally have perceived low value. In fact, the results of my study seem to suggest that there may be several related underlying causes for the increased math achievement performance for the students who were given an incentive to use provided feedback: (1) The incentive communicated to the students that using the feedback (or “help-seeking”) was something of value which historically may have had some stigma attached to it before in the past, especially for traditionally underperforming students; (2) this signal led to deeper processing of the information provided in the feedback, which meant they were “learning” more; and (3) given that the students who were offered the incentive actually solved the game level more quickly and with fewer errors actually gave the students a feeling of success which highlighted the utility and value of seeking the feedback. What was interesting is that the students who received the incentive were the only ones (with a few isolated exceptions in the other treatment conditions) that accessed relevant information in the help menu when a similar mistake was made at a later game level (even when NOT incentivized to do so). So, in terms of viewing the incentives as bribery–I understand the concern regarding the short-term incentives. However, for things that have the potential to quickly demonstrate utility and value, such as using provided feedback, incentives may be one beneficial way for some students than just simply trying to convince them through reason.
There seems to be much we can explore about the processes that drive gameplay. It would be interesting to work with game developers to see the structure they use and how feedback is utilised within the overall strategy.
Thanks for bringing this to my attention Daniel – the nearest school is in Lincolnshire here in the UK, certainly something to consider exploring more as I agree with your comments in your “Superwoman” guest blog post. We are doing what we can to help schools and districts redefine their vision for learning using Design Thinking principles.
Tom,
Everything in your bullet list above happens in good Montessori schools on a regular basis and has taken place for several decades . Have you ever visited a Montessori school? I think you would find it very interesting. There is a lot of game based learning. I was a bit surprised that a search of Montessori on your website came up with no hits.
Where are you based? I would be happy to recommend good Montessori schools in your area to visit.
Daniel
Tom,
Everything in your bullet list above happens in good Montessori schools on a regular basis and has taken place for several decades . Have you ever visited a Montessori school? I think you would find it very interesting. There is a lot of game based learning. I was a bit surprised that a search of Montessori on your website came up with no hits.
Where are you based? I would be happy to recommend good Montessori schools in your area to visit.
Daniel
That’s what I am still sorting out. You can see some of the work I did with a summer camp for middle school students here.
http://summergaming.yolasite.com/
We focused on design elements, but — it was summer — less academic vigor than I could expect in the classroom.
Kevin
‘Missing the point’ mmm, poor choice of phrase by me in terms of it’s connotations!
The important aspect of the examples that I shared was the intrinsic desire of players/learners to reach high standards and to continually improve in games – without adult intervention. I feel that that point, the important one for me, was missing BUT this is your post with your take 🙂
The formative/summative aspect of assessment and how this manifests itself within games is fuzzy as you suggest. As a result of this post I have thought about feedback during the gameplay and considered whether it is formative assessment or not. I suppose that it is. For me, and what I felt was what those in education would connect with was the feedback mechanisms in Guitar Hero but more specifically in the FIFA11 examples. I see them as formative and not summative because they are used as part of the ongoing development of the player/learner and more than a ‘what grade did I get’ resource.Incentivised curriculum that can exploit the best of assessment mechanics in games is a real goal. I’d love to see it develop…
Thanks Derek for taking the time to post your thoughts.
I don’t think I am missing the point with regard the summative / formative assessment details of Guitar Hero. I was simply extracting some of the common practices of games and relating them to my own interest in the role of formative assessment in the classroom. I have been focusing specifically on that type of assessment recently. Nor was I seeing them out of context, simply highlighting the details and mechanisms within that particular game that acts as feedback to the player.
I do accept your point that forcing the summative / formative distinction on games and gaming may well cause us to have too narrow a focus, nonetheless it has been a useful lens to view games through.
As for Delacruz’s work, you question, “Will learners go home and play that fraction game or get on their XBox 360 etc?” which seems to me to be an empty point. No doubt they will choose which ever motivates them the most. I don’t see any reason why I shouldn’t compare the two types of games. Are you suggesting the two domains should forever remain exclusive and not learn from each other?
On that note – I don’t think motivation or engagement is only possible through off the shelf gaming, nor solely reliant on context. I recall watching a pair of Year 5 boys play an iPad fractions game all the way through their breaktime one day – this had no context other than acting as an extension to some work and was the first time they had played the game. Nor would they choose to play it – probably unlikely they would download it from the app store. Despite this they were fully engaged and the incentive system was clearly meaningful to them. My point is that I DO think we can learn from both types of games and comparing them offers us an interesting perspective we shouldn’t preclude from our desire to understand the topic.
No doubt there is some difference in the motivations to design a game for entertainment and one for education – this could have an impact on how incentive systems are structured.
You know that I agree with you in regard to the contextual framing of games and learning and how important this can be, but I do think there is a sliding scale and it doesn’t always need to be a grand context stretching over many weeks – I have seen gaming work on a much smaller level. And as I outline above with a bare bones of a context.
The key thing for me is how educators and learning professionals can better understand the feedback systems and structures that are, as you point out “dynamic and ongoing” within different games.
Tom, thanks for your continuing attention to this topic. To me, it seems clear that video game dynamics are entirely about pedagogy. The underlying issue of game-oriented assessment is that the return is not, “That was right” or “That was wrong.”. It becomes, “That worked,” or “That didn’t work.” Regardless of the return, the ‘player/learner’ walks away with a new piece of knowledge.
— Dave
The process of game design is certainly a rich curriculum area given the right context – how were you thinking of exploring it?
The process of game design is certainly a rich curriculum area given the right context – how were you thinking of exploring it?
Intriguing to think of gaming as a way into other online tools and open learning methodologies – suppose it is all very obvious considering what is intrinsically part of a good game. I suppose as we get older we begin to make choices about playing or not playing games – this in my opinion manifests later in life as “oh I am not into that sort of thing.”
Do you think it could be true that we all love games? It is just that some of us have weened ourselves off of them and so superficially baulk against them – even though the home film footage of when we were toddlers would say otherwise.
There was quite a lot about assessment, good assessment practice, in that presentation but the screen recording shows too much of me! Although your Slash like observation was welcome this was a thoughtful post that made me revisit my own thinking. Anyway, my tuppence worth in response…
There is much to discuss about this subject but I’d like to focus on a couple of things that you say in this post that I am not sure that I agree with. You discuss whether the feedback from Guitar Hero is summative or formative. This is missing the point in my opinion. My argument that I tried to establish through that presentation was that children are using the feedback mechanisms that are built in to games (Guitar Hero in this case) in order to improve their performance, to give them the best chance of success. Players/learners in this setting are using the mechanisms built in to the game to self-improve, to make their own judgements about where their strengths and development needs are and to address those – all without adult intervention. I offered other examples such as those from our Nintendogs work and with an interview with my friend’s son who is a FIFA11 expert that shows the very same thing. As a result I argued that we continue to underestimate just how able our learners are and how self-determination is part of their make-up and that we in education almost lazily/blindly accept that children only learn when the ‘authoritative adult’ is there in place to affirm or deny learning has taken place.
The work of Girlie Delacruz that you refer to is of interest also however I feel that again comparing this example (interesting though it may be) with incentives in games and game environments that learners choose to situate themselves in is not comparing like with like. Will learners go home and play that fraction game or get on their XBox 360 etc? Just look at the power of Xbox live (I share this in my talk) It is so about context again and in your/our collective attempts to bring the best of games based learning mechanics into the realm of ‘education’ we must not lose sight of the power of context and why this help makes the game mechanic work. The points that follow on from this that are raised by Oliver Quinlan also need to be considered in the light of the context of the game and game play. Let’s not forget that points, high scores, achievements etc. etc. in games are all delivered within a context that has cultural resonance with learners, they have value, meaning, they are desired! AS for points in school such as Best behaved table points…you can keep them compared with the points/achievements that really mean something to learners who play games?
In closing Tom, it’s not just about game mechanics – context is key
I am thinking these days more from the student as designer of game as opposed to player of the game, and trying to figure out the best ways to “flip” that agency, and work in design element. Thanks for the post, Tom.
Nice work. I had the pleasure in doing an interview with Derek around classroom innovations though games recently. What I learned, despite my fascination with massive multi-players games is that what holds true of them, is equally true of stand alone games that Derek is using such as the DS and Wii. What was most interesting is that what emerges from games – things that can be formative or summative – and quite traditional non-game tasks – is that teacher’s do change their pedagogic strategy, as a result of playing (with kids), and are more open to trying things such as IWBs, blogs etc. To me, games offer the potential of opening up new possibilities. By that I mean, that teachers who play games (and probably avoid Web2.0) are far more likely to try them as a result … which leads to new potential in assessment. Simply putting teachers in game spaces – using devices that play games, has an effect with powerpointing them about Web2.0 hasn’t achieved. If you then try this with games of their own youth (older consoles and arcade platforms), they sort of re-capture something … and this is great if you then introduce a kid to play with them. It achieves some kind of irrational parity between them and they share in the mastery. That to me, doesn’t happen outside of games too often.