Embedding ePortfolios in programme-level assessment in Business disciplines: Reflections on a pilot project

This practice report reflects on a pilot project which explored the embedding of ePortfolios in programme-level assessment in a number of Business disciplines across the College of Business in the Technological University Dublin. The pilot was important for a trio of reasons; firstly, for the university, positioning of an ePortfolio programme level strategy required an understanding of the importance of professional development and support structures for staff and students, and guidance on scaling from a pilot project to a full-scale implementation. Secondly, for staff, highlighting the purpose of the ePortfolio and particular effective teaching, learning and programme level assessment strategies was crucial e.g. personalization for increasing student engagement and reflective practice. Finally, for students, an ePortfolio can enable learning beyond the scope of a particular module to highlight holistic learning across a programme, and can be utilized for showcasing of work and in procuring future employment where it is suitable for capturing practical, critical and creative thinking. This article explores implications for practice for staff seeking clarity on establishing the purpose for assessment, reflection and audience (UG and PG students) for a programme-based ePortfolio. Reflections include the professional development support that paralleled the adoption of the pilot, strategies that can be put in place to move from an initial project to a scalable implementation across a college, and the challenges of engaging critically in programme-level assessment. Recommendations are for educational developers, those working and leading in learning development and learning technologists who support academic staff in decision-making around ePortfolio integration to programme-level assessment strategies. Irish Journal of Technology Enhanced Learning (2021)


Introduction
Higher education has been called upon to respond to the need for educational innovations promoting graduate employability and lifelong learning (European Higher Education Area, 2015). To achieve this, D' Andrea & Gosling (2005) have previously argued that students must progressively become able to reflect both on their learning and their potential to improve and plan their own professional development accordingly. Song (2021) has called for further research into determining the relationship between training and support and personalization of student ePortfolios and their outcome factors or purposes i.e. showing an ePortfolio to potential employers, and understanding the required items to fulfil the summative assessment. Over the past decade in Irish higher education, ePortfolios have been recognised as a digital technology that has the capacity to support student growth and development, and studies have emerged slowly on its pedagogic potential (Donnelly & O'Keeffe, 2013). However, despite academic interest and considerable investment in technology and standards, Weller (2018) argues that ePortfolios did not become the standard form of assessment as proposed back in 2008. Since then, a survey (eportfolio hub, 2016) explored the extent to which employers review ePortfolios as part of their recruitment and selection process. Findings from the small study indicate an opportunity for higher education institutions to familiarise graduate employers with ePortfolios by demonstrating and showcasing samples of student ePortfolios to employers. With the surge of interest and use of learning technologies post-Covid, ePortfolios may yet find their place, with ALT (2020) reporting e-Assessment as one of the top priorities for 2021, and 32% of the 214 respondents identifying ePortfolios as important or very important to their work in the past year, with 42% indicating that ePortfolios would be significant for their practice for the subsequent year.
Within the College of Business (CoB) in the Technological University Dublin (TU Dublin), an aim was to expand the use of ePortfolios at programme level, and this practice report discusses the process in a pilot project and reflects on its outputs for a potential wider adoption across Business programmes. Torre's (2019) literature review indicates that an ePortfolio's effectiveness during the teaching-learning and assessment processes is dependent on the ability of teachers to master this tool. It follows that, in order to facilitate the effective implementation and use of the portfolio in universities, lecturers need to receive suitable professional development.

Context and Rationale
In early 2020, channeled through the CoB Senior College Leadership Team, a call to action was submitted around the Collegein which programme(s) are you interested in embedding an ePortfolio delivery and assessment strategy? With the full support of senior management, four Schools each nominated a programme to engage with the pilot project of embedding of a holistic, curated ePortfolio into the programme assessment strategy for Business students. The origin of this pilot study came from recent work in the College (2019-20) on refining Programme Learning Outcomes (PLOs) and programme level assessment strategies (to avoid the issue of over-assessment of students).
An underpinning dimension of the initiative was the signature pedagogy of the College of Business, shown in Figure 1. The signature pedagogy is the characteristic forms of teaching and learning in business disciplines, and the forms of instruction used when preparing students for particular professions. Pedagogical signatures can show much about the dispositions and cultures of professions and they implicitly define what counts as knowledge in a field and how things become known. In the College of Business, case study teaching, live industry projects, critical thinking and reflective practice all formed the signature pedagogy that influenced the ePortfolio pilot project. Coupled with programmes being practice-based, research-informed and close to businesses and the associated professions, an integrated pedagogy has evolved over time and for the ePortfolio, how that plays out in everyday terms for students and lecturers is important; this involved consideration of the links to strategic positioning of the college and TU Dublin, interaction with local, national and global industry, the lived experience of the students, core intellectual outputs of staff, competitions and showcasingthese all featured in the context for the pilot project.

Reflecting on the ePortfolio Pilot Project
The conceptual framework for a programme-wide approach from Griffith University outlined in White (2019) was informative for the pilot project; it provided a number of useful insights e.g. on how to design and integrate learning activities for ePortfolios, how to support the learning activities with authentic assessment that then contributes as evidence for collection and inclusion in ePortfolios, and also how to assess ePortfolios using rubrics. White (2019) suggests that ePortfolios offer students the opportunity to create a well-organised visually appealing record of their academic and professional knowledge, skills and attributes as well as practical achievements. To reach this, the mapping approach from White (2019)   The intention of the pilot project was to explore how to: • challenge students to revisit their work on each module within their programme and the feedback they received across modules; • help develop a holistic, analytic picture of the 'ground covered' on the programmes; • cultivate a stronger sense of the discipline and the ways in which themes relate to one another and to the student's own identify and values; • enable students to articulate explicitly their own identity, along with perspectives and skills underpinning the range of disciplinary work showcased; • create spaces across a programme where students select, curate and gather their best work in line with programme level learning outcomesso that their output will be a presentation of the practical application of their ideas and discussion of their practice and process 'live' to an audienceand they can share their 'showcase' ePortfolio which brings in theoretical For some of the Programme Chairs, the pilot project offered an opportunity to consider the ePortfolio as part of larger curricular revisions, and were considering how to embed the use of an ePortfolio at the core of the UG or PG experience. The addition of the ePortfolio was intended to be an integral component in supporting students as they develop the reflective skills of discovering themes and patterns in their own work, analyzing their experiences, and making connections across the artefacts they develop in modules and Business contexts. Andre et al.
(2017) report that the process of considering each artefact supports critical thinking and reasoning, enabling students to project forward to future practice fostering the development of a professional identity.

Pilot Timeframe, Stages of Support and Delivery Plan
The timeline for the proposed pilot stages of planning/development/implementation/evaluation was negotiated and agreed from January-June 2020 as follows: January 2020 • Induction to the pilot project and ePortfolio pedagogies.
• A Planning Template sent out to Programme Chairs to do some initial thinking and move them forward in integrating an ePortfolio to either a module in the first instance, or a programme.

February-May 2020
• Development consultations/Brightspace ePortfolio demo and PD-training for staff.
• Preparing for implementation with the full programme team.
June 2020: Evaluation of pilot project; student training; preparation for going live in new semester.
September 2020: Presentation of findings from the Pilot to College Plenary and Senior Leadership Committee. Figure 3 depicts the four stages of professional development training and support that was put in place for the pilot project. This was informed by the College identification of addressing coherent assessment at the programme level, resources available for provision of support for lecturers on technical and pedagogical skills, and fostering reflective practice among College of Business staff.  to their own programme, and had a particular influence on the rubrics they planned to design: • What is the purpose of the ePortfolio in your programme? How does it relate the existing programme/module assessment design? What is your pedagogical approach? • Is an ePortfolio the right assessment for this module? What will it assess?
• How does it align with the programme/module learning outcomes? • How best to embed the ePortfolio assessment in the curriculum and map it to the assessment criteria; ensure the ePortfolio aligned to Graduate Attributes so that students are collecting evidence and reflections as they progress. This requires thinking through how does it fit in with your existing approach to assessment and how to integrate the use of the ePortfolio within your assessment plan, not just use it as an add-on. • What are you assessingis it the product or the process (or both)? How will it be assessed validly and reliably?
A further question that Programme Chairs had was how to use ePortfolios with larger classes?
Suggestions included piloting it with smaller groups initially to try it out and learn from that, and that doctoral students would also be a useful group with whom to evaluate the pilot.
Between workshops, an ePortfolio Planning Template was given to the Chairs to complete in discussion with their programme team; it was key to highlight that not all the items in the template needed to be covered in detail as it was important not to overwhelm busy programme teams; instead they could act as a launchpad for discussion. Table 1 shows the relationship between the issues pinpointed by the Progrmame Chairs as needing a focus in the pilot project, and the 'planning for delivery' template provided to address these. The template is offered as a resource for others considering ePortfolio implementation at programme level.

Review the PLOs
• What is the module/programme context for integration? What is the specific audience for the ePortfolio? • What is the purpose of the ePortfolio in relation to your PLOs/MLOs? What role does the ePortfolio play in the module/programme as a whole? How might students use an ePortfolio to demonstrate what they have learned? • Are there existing programme/module learning outcomes that the ePortfolio can enhance and support? Identify the PLOs/MLOs into which the ePortfolio will be aligned; Write a number of learning outcomes for an ePortfolio assessment in your module. How will you integrate the ePortfolio into your curriculum and your instruction? • What will it be worth? What will the students have to do? Will you give a single grade to the ePortfolio as a whole, or will you assess its components separately? • How will it be assessed? Who will assess it? How will the students submit?

Stakeholders
• Do you need buy-in from your students, colleagues, programme team, Head of School for this initiative? How will you get it? • Who will train and support the students to use the ePortfolio platform? • Who will help academics to use the platform? How will you roll-out support? Identifying what supports are needed for UG and PG students to use the ePortfolio: A College Philosophy and Guide are needed on Reflective Practice/Writing; this emanated from wanting to develop a deeper understanding of ePortfolio use for reflective learning.

Structure
• Who will design the structure of the ePortfolio (number and layout of pages, elements, artifacts, etc.)? You? Your students? Both together? • Will you require certain items in the ePortfolio? Or will students self-select some or all of the items? • What elements in the ePortfolio are essential to its success? • How will it be used for reflection? What model of reflection will you adopt? Who will support and train the students to write in a reflective style? What reflective prompts will you use?

Support
• Will the external examiner need access?
• Are there accreditor requirements from the Business Schools?
• Private or public ePortfolio?
• Do you want students to work together or peer review?
• How will you ensure students do not alter the work after submission? Across the College, ePortfolios need to be properly understood by staff and students, invested in, and supported.

Evaluation
• How will you evaluate the organization of the ePortfolio and its visual design?
• What does a successful ePortfolio programme integration look like? What next?

Organizational layout (navigation):
• How should it be laid out and structured? What are the natural divisions of accompanying text? • How big/how many entries/artefacts will it feature? What will be the nature of the multimedia tools used? How long should audio/video be? • What kind of information will be needed to orient a reader to the purpose and content of the ePortfolio? The institutional plagiarism software needs to be fit-forpurpose.

Platform
• What platform will I use? Free online tools or the Brightspace institutional platform?

There should be examples available of how the ePortfolio works with different types of programmes and modules in the College.
What materials do you need • ePortfolio Guidebook for students/Support Space on your BS module.
• What to consider: will students be reviewing one another's ePortfolios? Is the ePortfolio intended to be public on the web (at some point?) • Exemplars of good practice needed -would include a curated collection of outputs from across the programme which are framed within a substantive intellectual, analytical, connective narrative. This analytical, yet personal narrative would have a disciplinary spirit in contente.g. using a substantive new theoretical or professional framing in e.g. Business Administration. It is to be based on PLOsand the design, preparation, criteria and guidance would be carefully crafted to ensure quality submissions. Students are also learning and demonstrating development of digital literacy skillsproducing academic/industry blogs, multimedia presentations, podcasts or video documentaries. for the ePortfolio and provide feedback and feed forward on the selection; the benefits of providing students with self-assessment opportunities; for fairness and equity, scheduling class time in a computer lab for students to contribute and establish their ePortfolio; provide guides to the students at the outset (e.g. artefact log sheet, reflective practice guide, rubrics (preferably codesigned with students) and marking guidelines); clarify all aspects of the ePortfolio submission for assessment (due dates, expectations, content, selection, reflection); inclusion of opportunities for students to reflect thoughtfully and critically on their learning and business knowledge construction (goal-setting and identifying their ePortfolios strengths and areas for development); consider the lifelong dimension of the ePortfolio post-assessment.
A tip-sheet (Figure 4) was also provided by the leader of the pilot project at this point to the Programme Chairs for use in their subsequent discussions with their full teams: TIP on 'purpose': ePortfolios have different purposes. The type of ePortfolio you choose will depend in part on your targeted learning outcomes. The purpose of the ePortfolio in your module/programme should be clearly defined at the start and clearly communicated in your course materials.
Is it: Showcase ePortfolio | Progress ePortfolio | Process/Product ePortfolio | Reflective ePortfolio TIP on 'goals': Be sure to design an ePortfolio assessment that will draw out evidence of student learning related to your PLOs. Let students know early on if they should collect certain artifacts/kinds of artifacts to illustrate their knowledge and skills. As you plan your module/programme integration, think about designing class activities and/or assignments so that students will create or collect essential artifacts and develop the skills they need to create a strong, persuasive ePortfolio.
TIP on 'structure': In its most basic form, an ePortfolio is simply a collection of work, usually accompanied by a reflective commentary that explains the purpose of the collection and the reason for including particular items. The particular artifacts to be included in the ePortfolio may be designated by you, or selected by students; many ePortfolios are a combination of lecturer-selected and student-selected work.
TIP on 'teaching': Help students to see the connection between what you are doing in class and what they are being asked to do in their ePortfolios. Some lecturers ask that students complete a page in their ePortfolios before a specific class as preparation for discussion. Others may use an ePortfolio assignment as a way to extend classroom activities, or to reflect on learning over the module. Where appropriate, make mention of the ePortfolio assignment throughout the programme/module, and engage students in discussion about the content (not merely the requirements) of their ePortfolios as they develop them.
TIP on 'assessing': ePortfolios can be excellent tools for both formative and summative assessment, and they provide an opportunity for students to reflect on their own learning. Think about how you will assess the ePortfolio as you are writing the assignment instructions. Consider how each component of the ePortfolio -artifacts and commentary, organization and visual design -will provide evidence of student achievement in relation to your MLOs/PLOs.

Figure 4: ePortfolio Tip Sheet for Programme Teams
Stage 4 was based on strategies which Krause (2006) recommended for the successful integration of ePortfolios; these are predicated on the claim that while ePortfolios have the potential to transform pedagogy, this potential cannot be realised without real curriculum change. These principles apply across contexts, whether the ePortfolio is for UG, PG or academic staff use: • Start small. Plan a slow implementation; explain everything clearly to your students as you go. Integrate other electronic components into your module before you introduce the concept of an ePortfolio to students.
• Build towards a programme-wide strategy, aiming for a new culture in assessment. This takes time. ePortfolios can transform pedagogy; you cannot implement them without real curriculum change.
• Develop a strong resource plan for technical rollout. Through the pilot project spend time developing technical proficiency among the programme team.
• Review with the full programme team that the ePortfolio is aligned with the learning outcomes, clearly discussing and defining the purpose of the ePortfolio for students and staff.
• Make the ePortfolio a sustainable assessment tool. Use ePortfolios to track and gather resources for students from Year 1 onwards.
• Set up professional development and skill-based workshops for the programme team and students.
• Prepare the evaluation so that you support ePortfolio enhancement, the exclusion of extraneous material and content improvements early, and continue this support throughout the programme.
In terms of preparing for the ePortfolio for the new semester, plenary discussion involved thinking about ways to keep an ePortfolio vibrant in a programme (being well-rounded; finding things to write about; advertising your portfolio/blog using pings, Twitter, Facebook; subscribing to and commenting on others' blogs; additional resources, including the ePortfolio and blog spaces; encouraging 'failure' and learning from mistakes). We also talked through: "What" -is needed next in terms of support that you anticipate needing?
"So What" -consider how will the use of ePortfolios change what you currently do?
"Now What" -what do you plan to do with your ePortfolio exploration moving forward?
Feedback from the Programme Chairs after the pilot project was generally positive (Figure 5), the only caveat being lack of time to undertake the planning and implementation effectively:

Reflecting on moving from Pilot to Wider Implementation
The pilot project came to a conclusion in September 2020, when findings were presented to the Programme Chairs Forum, the College Plenary and Senior Leadership Committee. Going forward, strategic recommendations that emerge from the pilot project can inform a wider implementation/adoption of ePortfolios in the College as part of the continuing focus on programme-level assessment:

MSc International Business seminar series:
Purpose of the ePortfoliohow to capture and assess student reflections over time?
As a pilot it has been successful, both in encouraging timely reflection and elevating the role of reflection in terms of learning outcomes.

PG Diploma in Product Management:
Purpose of the ePortfoliohow to achieve a balance in delivery and assessment? How to knit together all the modules on the programme? How to get meaningful feedback on students' progress at key points on the programme?
• Platform for engagement and enhanced learning

UG Teaching Irish Law and Legal System to non-law students:
Purpose of the ePortfolio -what is the learner journey from the lecturer perspective?
The ePortfolio allows a multidimensional identity for students: • as a content collector (identify-read-consume-understand) • as a content curator (select-discern-classify-contextualise-categorise) • as a content producer (compare-critique-contrast-evaluate-reflect) • as a content prosumer (students takes ownership of their learning journey through autonomous engagement outside class) • It is an excellent companion for ongoing assessment as a process and a product; rubrics outline key criteria; develops universal research skills, higher order thinking (summarising-evaluatingreflecting-creating)

MSc Finance:
Purpose of the ePortfolio -what alternatives are there for students as well as a CV?
• Key to have buy-in from Programme Chairs which is better achieved through having ePortfolio exemplars from the pilot to share across the Business disciplines.
• Articulate a clear, compelling, and communicable purpose for the ePortfolio initiative under development or expansion; highlight the benefits of a holistic, curated ePortfolio for Business Programmes especially that it can encourage students to revisit their work on each module and on the feedback they received (findings from Farrell (2019) indicate that learning with an ePortfolio can enhance the nature of the learning experience by providing learners with a personal space to evaluate their own learning, to process their thoughts and experiences and to document their lives and learning in an authentic and meaningful way. In addition, the findings suggest that learning with an ePortfolio can enhance the development of critical thinking skills and dispositions within a disciplinary context); develops a holistic, analytic picture of the ground covered on the programme; develops a stronger sense overall of the discipline and the ways in which themes relate to one another and to the student's own identify and values; articulates explicitly students' own identity, along with perspectives and skills underpinning the range of work presented).
• Emphasise evidence-based, theory-to-practice models that connect research and scholarship with effective approaches to developing and utilizing ePortfolios for high-impact learning, programmatic assessment, and/or professional development.
• Identify collaborators and form key partnerships with those who can broaden the argument for resources, support, and adoption of ePortfolios.
• Utilize evidence-based diffusion of innovation and leadership strategies to increase the probability of successfully scaling the ePortfolio initiative.
• Create an ePortfolio action plan for implementation, along with measurable benchmarks for charting success.
• Dedicated support for the initiative is very important; useful to investigate rolling out smallscale funding initiatives to encourage enhancements to teaching and learning more widely, and include ePortfolio support.
• Identify what can be learnt from how it is done in other disciplines; Share ePortfolio assessment rubrics.

Conclusion
As a collaborating team of ePortfolio developers, we presented the pilot work at a University Business School webinar in September 2020. Continual updates were provided on the pilot to the Senior College Leadership Team and to School Executives and College Plenaries.
Reflecting on the effectiveness of the 'planning for delivery' approach used in the pilot resulted in a number of questions emerging from our discussions that can be used to move the initiative forward. Firstly, consideration of what staff and student PD supports are needed? Do students have the necessary digital skills to produce an ePortfolio? Support is needed for students in demonstrating development of their digital literacy skills (producing academic/industry blogs, disciplinary multimedia presentations, podcasts and video documentaries); How will staff know if the digital evidence is authentic?
Secondly, a focus is needed on whether systems are sufficiently robust to handle a large number of online submissions of ePortfolios. How will the ePortfolio be stored once submitted, for the purposes of official record keeping? Will it remain at the university after graduation? Who will own it -the university, the staff or the student? Who is allowed access to the ePortfolio? (security, data, copyright, legal); will all the Business professional accreditation awarding bodies accept ePortfolios?
Finally, it will be important to consider promotion of the initiative -how will the College promote ePortfolios? Learning from what has been successful elsewhere will continue to be important and a number of positive initiatives could be undertaken to engage staff in this work -a formal launch for all staff with the Heads of School buy-in; offering PD training for staff using the ePortfolio platform in Brightspace; drop-in clinics for students; mentoring for staff (more experienced ePortfolio users matched with those who are new to using it; a wide range of support resources in Brightspace; factoring in a dedicated ePortfolio champion/support staff for the College; regular inprogramme student showcases, with an annual ePortfolio student showcase (this can involve creating spaces where students select, curate and gather their best work in line with progamme-level learning outcomes and present the practical application of their ideas and discussion of their practice and process 'live' to an audience and it also brings in theoretical dimensions to their work).
Further work is needed on staff professional development support to bring the findings from the pilot project to a wider implementation strategy. Emphasis needs to be on the design, preparation, criteria and guidance that is required to be carefully crafted by the Programme Chair and team to ensure quality student ePortfolio submissions across the Business disciplines for their programmelevel assessment strategy.