Keith Lay, Composer - Week Four Publishing

Dear colleagues,

This is a simplified version of the keynote that I plan to use as a presentation conduit for my AR project.

Professional media education is facing two major challenges: 1) how can education best meet both student and teacher needs for rich media assessments, and 2) how can teachers help students prepare to enter the media profession after graduation?

Affordable media production tools are enabling educators to assign audio, music, 2D image, 3D, animation and video based projects. But, do educators have enough time to grade them appropriately, and, more importantly, are they the best party to do the job?

The process of becoming a member of a profession includes authentic participation in the activities of the community at a level appropriate to the learner's competency. As a member's understanding increases, he or she is more able to participate in a more central practice until they become an acculturated member of the community.

A 'revealed' peer grading system [RPG] is my term for a media assessment system that answers these challenges in which peers anonymously grade each other. The unique aspect of this approach lies in the later step of the process wherein the names of the peers are privately revealed to both the assessed and the assessors. My action research set out to explore and measure the viability of RPG.

The system works as follows:

1. The student creates a media piece to fulfill the requirements of an assignment that includes instructions and a grading rubric

2. The student securely uploads their media piece onto a private server, called a 'workspace'. The workspace closes when the assignment deadline is reached.

3. Then, each workspace is opened to a teacher-selected number of peers that have also turned in the same assignment. The names of the assessed and the peer assessors are hidden or replaced with anonymous names.

4. Each peer-grader uses the rubric to grade their assigned peers and comments on the media, as well as uploads a grade broken down into rubric-based score into each workspace. 5. After all peer assessment is accomplished, the workspace is again opened to its owner. The real names of the graders is revealed.

Most of the available research on peer grading is focused upon revealing the lurkers and shirkers from the contributors in group work measurements. The best research on the use of technology for self and peer grading comes from a group of Australian professors led by Dr. Keith K. Willey at the University of South Wales who created SPARK (Self and Peer Assessment Resource Kit) in 1996 to facilitate group work assessments by group members. This body of research is very useful in the development of the RPG concept and application. It provides strong evidence that peer grading combined with collaborative peer learning activities yields many positive, measurable benefits to students and successfully encourages engagement with the class materials and their learning community. Willey concluded that self and peer assessment required both instruction on teamwork skills and opportunities to practice those skills. Doing so increased the development of teamwork skills and was deemed a crucial element in a successful move to the engineering profession.

Christine Greenhow found positive results when students were supported in assuming positions of authority; redistributing authority to students with the instructor taking a coaching role.

A culture-shift is needed in the classroom for the coming generations of media literate students. There is evidence to support that revealed peer grading could provide these students with an excellent environment for developing media literacy, critical judgment and professional dialogue skills. It can also increase a sense of connectedness among class peers. By employing SPARK-like metrics, the data collected by the peer collective can provide very useful assessment data to both students and teachers. Research in collaborative and professional education practices indicate that the processes in RPG will also encourage better judgment skills, professional ethics and values.

To test the RPG system, I invited ten audio professionals and two non-professionals who are all Full Sail faculty to take part in this research. Though all were open and interested in the RPG process, only four took part and followed through with the process. I made a video of the first 3+ minutes of one of my performances which was recorded in February. I mixed the tracks down to stereo without reverb and made it available for download by each professional.

Here is their assignment given to the professionals who contributed to this project along with a rubric chart:

Your Assignment: "String Orchestra and Soloist - final mix/pre-master" The owner of this recording would like you to prepare it for commercial release as a DVD and CD. Though the performance was excellent, the hall acoustics were not. Your goal is to produce a final mix that enhances the enjoyment of the music by allowing the buyer to believe that the acoustics of the hall were, instead, amazing. Decide upon the right kind and amount of reverb to achieve this goal. You may optionally employ any other processes that you think will help achieve this goal. Your plug-ins, levels and settings will be captured in a screenshot and made available to your peers to display your technique.

PART ONE: DOWNLOAD ASSETS BEGIN:
1.Answer the Dropbox invitation from keithl@me.com to get Dropbox access to the video for mixing. If you have not received it, please Email keithlay@fullsail.com. This will require that you download and install the Dropbox application. 
2.Read the assignment above. This assignment assumes Audio Workstation familiarity. 

DOWNLOAD VIDEO:
1.Open our shared Dropbox/K Lay Cycle One 
2.Copy the file Joyful Play_48-24_h264.mov to your hard drive 

PART TWO: MIX:
1.Open the movie file in your AWS of choice, keeping the video associated with the audio 
2.Read the assignment and rubric (above) 
3.Mix 
4.Make a screencapture (command-shift 3 or 4) of your open project. Make sure your screenshot captures all opened plug-ins to show settings and mixer levels. Name this file "Screenshot". If more than one screenshot if necessary to capture all of your plug-ins, name them "Screenshot 1", "Screenshot 2", etc. ConceptShare accepts common image file type. 
5.Save your audio mix with the movie as a .mov file. ConceptShare cannot upload audio files.  
PART THREE: UPLOAD:
1.Email keithlay@fullsail.com to receive an anonymous name 
2.You will receive an email from me with your ConceptShare workspace. You will only have permission to launch your assigned workspace. 
3.Rename your mix "Joyful Play [anonymous name] 
4.Open your workspace by hovering over the icon - press the "Launch" button  5.Upload your mix with the "Add A New Concept" button on the left side of the window  6.The site will convert your video to flash. Use these settings: 
7.After this download is complete, upload your screenshot(s) using the same method. Each item in your workspace will be a different 'Concept' within your workspace.  PART FOUR: GRADE YOUR PEERS
1.I will receive a notice from ConceptShare that you have uploaded your mix video. I will then email you a peer whom you are to grade according to the rubric. Your 'instructor' might also anonymously submit a mix.  
2.When grading your peer, use the green "Add Comment" at the bottom of the page at any point in the video. This will greatly enhance the value of your assessment. Be positive and constructive. Be prepared to defend your grade. 
1.After the Assessment period has closed, the anonymous names of the class will be revealed.  
2.If you feel that you have not been graded fairly, contact the peer who graded you and communicate your defense. A professional demeanor is critical in these discussions. If either party (Graded or Grader) behaves unprofessionally, contact your instructor for help.  
3.Grades cannot be changed without a case by case synchronous appeal meeting between both parties and the instructor. Non-professional behavior will be reflected in GPS accordingly.

The surprise - the scores were lower than expected. No one liked the others mix very much, apparently, at least using the rubric. I explored the differences between grades given. If the grader had a lower bar of judgement, then the graph might show the grades they gave as higher. Similarly, if their bar of quality was lower, then it might show up in their own work becoming lesser quality work and the grades they received might be lower than the median. The graph "Correlation between Grades Given...." chart displays the opposite! This will have to be studied further with a more sizeable test group.

The purpose of this research was to explore the validity of RPG in the following areas:

1) would RPG develop better judgment skills in its students?

2) would RPG teach professionalism?

3) Would RPG generate valid grades thus alleviate the burden of grading on the instructor?

A prominent feature of constructivist education theory is active learning where students are engaged, or 'active' rather than passively listening in the classroom. Problem based learning refers to a constructivist principled system used in medical schools that teaches through self-discovery through small collaborative groups. Savery and Duffy identified eight instructional principles which constitute a constructionist learning domain and suggested how problem based learning follows these principles. RPG employed five out of eight of these principles and through engagement and repetition, by:

1) anchor all activities to a larger task or problem. 2) develop ownership of the overall problem or tasks

3) design an authentic task

4) design the task to reflect the complexity of the environment

5) design the learning environment to challenge the learner's thinking

In teaching professionalism, a key challenge is the creation of curricula that employs the technology in a manner congruent with the profession for which it prepares. Revealed peer grading can create a student who can think critically and communicate professional judgment, thus advancing an objective of producing employees equipped for the 21st century. Though the grades were lower than those given by the instructors, through grading and carefully targeted rubrics, the system provides a wealth of assessment data without the teacher's direct involvement. This allows the teacher the ability to elect a role of their choice in the RPG process.

A culture-shift is needed in the classroom for today's media literate student. Revealed peer grading can provide today's students with an environment for developing media production literacy, critical judgment and professional ethics, values and dialogue skills. RPG can also increase a sense of connectedness among class peers.