Pathfinder UoB

Activities and Progress

During the reporting period, activities have been focused on the following project objectives:
• Develop systems for CAA
• Working with academics in target schools to lay foundations for use of online assessment
• Training Sessions for instructors (individual and group)
• Identify optimum use of learning spaces for CAA
• Gathering student feedback on perceptions of online assessment.
• Developing shared bank of online assessment materials.
• Preparation for Dissemination (ie conferences and submissions)

At this stage, progress has been according to the project plan, and it is not deemed necessary to make any changes to the objectives regarding CAA. The team is confident that it will be able to meet its objectives and deliver on time.

Outputs and Outcomes

1. Supporting online assessment:
- during exam period Dec 07 – Jan 08. Collaboration with examinations office, technical services, academics, invigilators to ensure effective support for online exams in Bradford and partner institutions in Delhi, Muscat, Leeds and Halifax.
- ongoing formative assessment during semester 2
- ongoing use of QMP for questionnaires (in Schools of Health, Management, Social and International Studies, Lifelong Education and Development, Corporate Services and Learner Support Services)
- supporting School of Pharmacy (liaison with academic staff and consultant in creating online formative and summative materials)

2. Developing systems for CAA. Work has continued with IT technical and customer services further to improve the set-up for Questionmark Perception at UoB. This includes:
- creating a student exam image
- further experimentation with secure web pages for dealing with large numbers of students (ie where using the VLE might pose a risk). We are still working on the optimum configuration for the second (high stakes) Questionmark server.
- a way of matching QMP usernames with student ID numbers to ease admin

3.  Staff Development: delivered group training sessions for staff;
- Come and create online assessments with Questionmark Perception (14th and 15th January),
- UoB Rules and regulations for CAA (14th January)
- Writing high quality MCQs (14th January).
- CAA briefing for invigilators (7th Dec)
- co-delivered short online distance course for staff into use of CAA (Jan-Feb)
- ongoing F2F support for individuals and small groups for staff starting out with QMP but unable to attend scheduled sessions.

4. Exploring and evaluating the options for item banking at the University of Bradford. The available options include
- JORUM
- SWAP (ie Questionmark’s ‘Sharing with Professionals’ repository)
- Questionmark Perception Shared Repository. Currently, the third appears to be the most likely option (see issues and challenges below)

5. Identify optimum use of learning spaces for CAA: work closely with JISC Institutional Exemplar project into thin client technology for CAA as part of project team. This also links in with developments within the University’s own e-strategy, which has a specific project devoted to CAA (to pick up on many of the initiatives developed under the Pathfinder project)

6. Project Sustainability: Working to ensure continued support for online assessment as part of the University’s e-strategy “Project C” and JISC funded Institutional exemplar project (see above). This has now been approved (as of Dec 07)

7. Student perception survey: Start work on questionnaire on student perceptions on CAA. For delivery to students on modules where CAA is being used either formatively or summatively.

8. Networking within the CAA/QMP community: Constant liaison with fellow QMP HE users nationwide, through email and discussion groups, and now a Facebook community. I am also in constant contact with Questionmark Computing technical support to resolve ongoing issues.

9. Dissemination and preparation for dissemination
- 12/13 Feb 2008 Pathfinder Progress Meeting, York Park Inn Hotel
- 26 Feb 2008, Presentation at JISC RSC YH event, University of Sheffield, 26/02/08, “Implementing Online Assessment: Finding the Right Path for an HEI”
- Proposal accepted for Bb World Europe (VLE User conference, Manchester, 12-14 May 2008) “Formative and Summative Assessment: Integrating Blackboard and Questionmark Perception”
- Submitted proposals for the following conferences (currently awaiting confirmation) “Secure, reliable and effective institution-wide e-assessment”; EARLI/Northumbria Assessment Conference, Berlin Potsdam, 27-29 August 2008, ALT-C 2008: Rethinking the Digital Divide, Leeds 9-11 Sept 2008, 12th International CAA Conference, Loughborough University, 8-9 July 2008

Issues and Challenges

1.Item banking: it had been hoped that JORUM or SWAP might have offered viable ways of sharing item banks. However, due to a number of issues, including limited uptake and IPR, it is unlikely that these will be viable options in the immediate future. However, due to the flexibility of Questionmark perception’s shared repository (especially when integrated with the VLE to manage permissions), we can establish a flexible and reliable set-up, at least in-house, without needing to rely on these external services.

2.Staff changes within the learning technology team did pose a potential risk, in that human resources were temporarily stretched when one staff member left the institution. However, we were able to plan ahead, seek alternative support internally and quickly replace the missing team member.

Activities and Progress

During the reporting period, activities have been focused on the following project objectives:
• Develop systems for CAA – increased security, functionality, customisation
• Working with academics in target schools to lay foundations for use of online assessment
• Training Sessions for instructors (face-to-face and group)
• CAA induction and training sessions for existing and new invigilators
• Identify optimum use of learning spaces for CAA
• Approved University Guidelines for online assessment
• Baseline study and Needs analysis (SWOT) report on use of CAA at UoB

At this stage, progress has been according to the project plan, and it is not deemed necessary to make any changes to the objectives regarding CAA. The team is confident that it will be able to meet its objectives and deliver on time.

Outputs and Outcomes
• Develop systems for CAA
Work has continued with technical services further to improve the set-up for Questionmark Perception at UoB, particularly in the following areas:.
• Security: try out QM Secure Server and purchase under e-strategy
• Working with IT technical and customer services to set up alternative secure ways of delivering assessments to students on and off campus. This includes creating a student exam image or using secure webpages.
• Expanded functionality of integration with Blackboard
• Customised login screens for UoB
• Exploring new features of QMP – Moodle connector, improved feedback in coaching reports, delayed results in new version QM 4.4 (due out this month)
• Exploring new question types (esp. including confidence ratings)

Training Sessions
Deliver individual F2F training sessions on QMP and group session on rules and regulations. Also for the first time, specific CAA training sessions have been set up for invigilators.

Identify optimum use of learning spaces for CAA
Work closely with JISC project into thin client technology for CAA as part of project team. Also investigate new solutions (with technical services) for improved delivery of assessments within existing learning spaces, using special exam image. This also links in with developments within the University’s own e-strategy, which has a specific project devoted to CAA (to pick up on many of the initiatives developed under the Pathfinder project).

Ongoing needs analysis
Gather more data from the Exams office concerning the use of optically marked scansheets in MCQs in School of Life Sciences. This provides further evidence of the need to work closely with this school.

Rules and Regulations for CAA
Ongoing consultation with the Exams office to further develop official University Rules and Regulations Guidelines for summative assessment. Our document has currently reached a “final” format and has been approved by APC and Senate, and is now official policy. The exams office will be publishing the document on the university website. We are now in a position to start to work on expanding the documentation to include formative assessment and external partner institutions.

Making contacts within schools
Now that the academic year is well underway, it has been possible to start making contact with academics, especially in the school of Life Sciences and School of Health. Already projects are underway to support formative assessment using QMP in 2 modules on the MPharm programme in years 1 and 2 (“The human body in health and disease” and “Professional practice 2”), in the Department of Optometry, and to deliver summative assessment on one module of the Foundation Degree in Health and Social Care.

Networking within the CAA/QMP community
Constant liaison with fellow QMP HE users nationwide (e.g. Lancaster, Glamorgan, Leeds, Salford, Southampton, Dundee) through email and discussion groups. I am also in constant contact with Questionmark Computing technical support to resolve ongoing issues.

Subject specific examples
Repeatedly, discussion with colleagues in HE CAA has stressed the importance of having subject specific examples of materials. So far it has been possible to produce 3 example assessments in the area of Pharmacy and Biomedical sciences, demonstrating a range of question types. These have already been very effective in demonstrations to academics in these subject areas.

CAA project webpage
A project webpage was created in order to keep interested parties informed about the overall project, without needing to point them to the extra detail of the public blog.

Online solutions for online unit / stage evaluations

I have been involved in a working group (including ASSU, ADTLs and school administrators among others) investigating the possible future conversion of the current paper-based module evaluations to online versions. QMP has been considered as one possibility.

Issues and Challenges

New issues that have arisen during this period have largely related to expanding the use of CAA to its external partner institutions, both within the UK and overseas. Communication, administration and security have emerged as issues for consideration. This is especially the case where exams are being delivered online simultaneously in more than one institution, in different cities and even different continents. This has implications for communication between technical departments, exams offices as well as instructors and e-learning experts.
However, coming out of this we have made huge steps forward in moving towards systems that will certainly be useful in future. At the moment it is small-scale, tentative and on a trial basis. At all times we have been careful to create a model that is secure and scalable. One of the main challenges has in fact been reining in the enthusiasm of our partners to make sure we do not rush into anything.
In looking into the rules and regulations of e-assessment some other interesting issues have come to light. For example, we need to ensure that Health and Safety issues need to be built in.

John Dermo, December 2007

Expanding and embedding support processes for e-assessment

Develop systems for CAA
Work has continued with technical services further to improve the set-up for Questionmark Perception at UoB, particularly in the following areas:.
• Security: arrange trial license for QM Secure Server and find funding for purchase under e-strategy
• Expanding functionality of integration with Blackboard possibly to include student numbers as well as usernames
• Customising login screens for UoB
• Exploring new features of QMP – Moodle connector, improved feedback in coaching reports, delayed results in new version QM 4.4 (due out this month)
• Exploring new question types (esp. including confidence ratings)

Training Sessions
Deliver individual F2F training sessions on QMP and group session on rules and regulations.

Identify optimum use of learning spaces for CAA
Work closely with JISC project into thin client technology for CAA as part of project team. Also investigate new solutions (with technical services) for improved delivery of assessments within existing learning spaces, using special exam image.

Ongoing needs analysis
Gather more data from the Exams office concerning the use of optically marked scansheets in MCQs in School of Life Sciences. This provides further evidence of the need to work closely with this school.
Rules and Regulations for CAA
Meet again with Exams office to further develop official University Rules and Regulations Guidelines for summative assessment (after consultation with Universities of Southampton and Dundee). Our document is currently on draft 8 and is ready to start its way progressing through the relevant committees.

Making contacts within schools
Now that the academic year has started, it has been possible to start making contact with academics, especially in the school of Life Sciences. Already 2 projects are underway to support formative assessment using QMP in 2 modules on the MPharm programme in years 1 and 2 (“The human body in health and disease” and “Professional practice 2”).

Networking within the CAA/QMP community
Constant liaison with fellow QMP HE users nationwide (e.g. Lancaster, Glamorgan, Leeds, Salford, Southampton, Dundee) through email and discussion groups.

Subject specific examples
Repeatedly discussion with colleagues in HE CAA has stressed the importance of having subject specific examples of materials. So far it has been possible to produce 3 example assessments in the area of Pharmacy and Biomedical sciences, demonstrating a range of question types. In future these can be used to promote the use of QMP in these departments.

Online solutions for online unit / stage evaluations
I have been involved in a working group (including ASSU, ADTLs and school administrators among others) investigating the possible future conversion of the current paper-based module evaluations to online versions. QMP has been considered as one possibility.

John Dermo, October 2007