Participant Engagement with Eportfolio on the PACT Digital Badge: An Explanatory Sequential Mixed- Methods Analysis

This paper explores participants’ perceptions of eportfolio on the National Forum for the Enhancement of Teaching and Learning’s PACT digital badge in which eportfolio is a central component of the teaching and learning experience. The PACT badge was launched in 2018 by the National Forum to promote engagement with their National Professional Development Framework for All Those Who Teach in Higher Education (National Forum, 2016). A small but growing body of studies indicates that the pairing of eportfolio with digital badges increases participant engagement with course content, facilitates the open evidencing of skill acquisition as well the transfer of knowledge beyond the life span of the course (Kehoe and Goudzwaard, 2015; Ambrose and Anthony 2016; Nilson and Dewiyanti, 2019). In January 2021, an explanatory, sequential mixed methods approach comprising a survey and focus group was employed to obtain and analyse feedback from graduates of three iterations of the PACT digital badge in relation to their perceptions of the eportfolio component. Data indicated a range of benefits including enhanced digital literacy, an enhanced ability to reflect, a more engaging learning experience, more robust evidencing of professional development and increased engagement with the National Professional Development Framework for all Those Who Teach in Higher Education. Seventy-Eight percent of respondents stated that eportfolio was an important part of the PACT course. A key recommendation of the study is the additional front loading of support materials in relation to eportfolio engagement in Week 0 of the course in response to feedback in which PACT participants indicated that eportfolio could be technically challenging and time consuming. The findings of this exploratory study could be tested at larger scale across a wider range of National Forum digital badges that incorporate eportfolio.


1.Introduction
This paper explores participants' perceptions of eportfolio on the National Forum for the Enhancement of Teaching and Learning's PACT digital badge in which the construction of an eportfolio is a central component of the teaching and learning experience. The PACT digital badge promotes engagement with the National Forum for the Enhancement of Teaching National Professional Development Framework for All Those Who Teach in Higher Education (National Forum, 2016). A small but growing body of studies indicates that the pairing of eportfolio and digital badges into the design and delivery of courses increases participant engagement with course content, the open evidencing of skill acquisition as well the transfer of knowledge beyond the life span of the course. This study seeks to augment the body of literature on eportfolio use in the Irish context with specific reference to digital badges and the National Professional Development Framework for all Those Who Teach in Higher Education (National Forum, 2016).

Background
The National Forum for the Enhancement of Teaching and Learning's PACT short course was launched in 2018, with the term 'pact' symbolising professionals' personal pact or commitment to professional development. The badge promotes engagement with the National Forum's National Professional Development Framework for all Those Who Teach in Higher Education (National Forum, 2016)  The PACT course which culminates in the awarding of a digital badge, is a part of a suite of open courses developed by Ireland's National Forum which recognises engagement through a "peer triad digital badge ecosystem" (Donnelly and Maguire, 2020, p.1). Donnelly and Maguire (2020, p.6

) observe that:
The open courses with digital badges were developed for a national rather than an institutional context and for recognition rather than formal accreditation, and is an important initiative for Irish higher education. The courses are short (25 hours of learner effort), accessible (allowing participant choice across three modes: face-to-face/blended; self-study; online) and provide recognition to participants who have meet agreed criteria by awarding a National Forum digital badge. This national endorsement supports staff employment mobility by giving credibility to these badges.
Referencing the work of Hickey et al (2015); Donnelly and Maguire (2020) state that stronger connections between digital badges, eportfolios and credit for prior learning, seem like promising directions for increasing the perceived value of badges. Participants of the PACT badge must complete an eportfolio to evidence learning on the programme. The construction of an eportfolio during the course is also designed to promote and scaffold participants' on-going engagement with the National Forum's Professional Development Framework for All Those Who Teach beyond the lifespan of the course. This paper seeks to explore the perceptions of participants on the PACT digital badge course in relation to the eportfolio component. The study will explore whether the eportfolio component of PACT enhanced the learning experience and supported on-going engagement with the National Forum for the Enhancement of Teaching and Learning digital badge upon cessation of the course.

Literature Review
Digital badges are electronic symbols used to document performance and achievement. Open badges are intended to provide additional information via metadata in order for viewers of badges to verify issuer details, evaluation criteria, and evidence such as the actual work product used to earn the badge (Parker, 2015). Parker also describes digital badges as an effective form of alternative assessment which support collaboration and engagement; enables students to reflect upon their progress as well as evidence skill attainment to employers. Farrell (2018, p.154) describes eportfolio as fulfilling a range of functions as "a tool or technology, a practice, a pedagogical model, an assessment method and a framework for learning. " Farrell, (2020, para. 45) in her exploration of the history of eportfolio identifies two key areas in relation to recent developments in higher and further education. The first relates to the impact of eportfolio on student learning and the second on "pairing eportfolios with emerging edtech such as digital badges, block chains and massive open online courses." Eynon and Gambino (2017) identify the pairing of eportfolio and digital badges as an important area for future eportfolio research. This study also explores the convergence of digital badges and eportfolio and its impact on the learning experience and professional development of participants. Four principle themes have emerged in the small but growing body of literature on the pairing of eportfolio and digital badges. Articles explore the role of digital badges and eportfolio in underpinning the open evidence of learning; in promoting student motivation and persistence; in synthesising and transferring learning beyond the campus environment and the pedagogical challenge of integrating eportfolios into digital badges. Ambrose and Anthony (2016, para. 24) indicates that the integration of eportfolio with digital badges makes "learning visible" and "brings "MOOC learners alive with a face and a story that shows proof of performance competency." A 2019 case study by Nilson and Dewiyanti highlights how digital badge metadata can be used to evidence achievement which can in turn be "uploaded and displayed in the students' PeddlePad eportfolios demonstrating the students..core skills." Weller (2018) explores the role of blockchain with "the aim to provide an individual, portable record of educational achievement." A 2015 study by Kehoe and Goudzwaard explores how the pairing of digital badges with eportfolios promotes integrative learning and identity development so that students graduate from "making simple connections among ideas and experiences to synthesizing and transferring learning to new, complex situations within and beyond the campus." This is a particularly powerful finding in the context of this study which also seeks to explore whether use of eportfolio on the PACT badge promotes engagement with the National Forum's National Professional Development Framework beyond the life cycle of the PACT badge.
Scully, O Leary and Brown in their 2018 review on learning portfolios describe some of the challenges of pairing digital badges with eportfolio. They highlight how the use of digital badges act as an incentive representing a form of extrinsic motivation to learn "but one of the ultimate goals of learning portfolio practice is to foster intrinsic motivation to learn." (p.18) Buchem 2016 cited in Scully, O'Leary and Brown's article, (2018, p.349) cautions against shifting the focus from learning to "badge-collecting". Gibson et al (2016) also cited in Scully, O'Leary and Brown, (2018, p.349) advocates for the use of "Inward facing" badges which are less formal and "can be awarded by peers or even by the learner themselves, and do not necessarily adhere to formal open badging infrastructures. It may be that this format is a better match for the pedagogy of the learning portfolio." Cheng and Chau (2013) recommend that learners have dual goal orientation whereby they focus on both the process and product elements of their eportfolios which result in higher levels of persistence and reflective activity.
The challenge of using eportfolio to openly evidence learning in a digital badge context versus engaging with eportfolio as a more introspective and personal exercise without outside incentive or pressure is also explored in this study. Gibson's description of the peer awarding of badges as a solution to this challenge is particularly interesting given the fact that National Forum open courses are awarded by peer triads.

Methodology
Exploring the perceptions of PACT participants on eportfolio was deemed by the authors to be best served by a mixed methods research approach. Creswell and Plano Clark, (2011) outline the numerous benefits of a mixed methods approach comprising qualitative and quantitative research methodologies, chief of which is the synergistic utilisation of data which will provide more indepth insights than the employment of one research method alone. This study employs an explanatory sequential design with an initial quantitative phase followed by a qualitative phase in which the qualitative data is used to elucidate quantitative findings (Creswell and Plano Clark, 2018). Mixed methods studies provide a breath of perspective and understanding with data grounded in individuals' lived experiences. They also increase the validity and rigor of findings through the triangulation of data. In a mixed methods sequential analysis, meta-inference is obtained through a narrative reporting process in which both qualitative and quantitative findings are explored concurrently on a theme-by theme basis (Guetterman, Fetters and Creswell, 2015). The concurrent exploration of quantitative and qualitative data according to emergent themes in the data is executed in this study. Bryant and Chittum (2013) in their review of eportfolio research address the limitations of research focusing on student perceptions of eportfolio highlighting how students do not always prefer the instructional methods that result in the greatest learning gains. They state that that it is difficult to know whether those that had negative perceptions of an eportfolio's impact on their learning felt that way because the tool itself was flawed, or because there were flaws in its implementation." (p. 194). This insight was also considered when analyzing data emanating from the study.

Data Collection and Analysis
One hundred and six individuals completed the first three iterations of the National Forum's digital badge between 2018 and 2020. To ascertain the perceptions of PACT participants of the portfolio component of the badge, an anonymous survey using Survey Monkey comprising closed and open questions was distributed to all graduates of the three iterations eliciting a response rate of 30%. The consent process was integrated into the survey tool, incorporating background information to the study and information on the use and storage of data. A focus group was subsequently conducted comprising a range of higher and further education personnel who were recruited using purposive sampling to ensure a cross section of insight and opinion. A consent form was disseminated to focus group participants prior to the execution of the focus group outlining the nature of the research and guaranteeing the anonymisation of data. Focus group data was coded as part of a grounded theory approach to identify concepts and construct theory (Corbin and Strauss, 2008). Grounded theory is inductive in its approach, attempting to identify and understand how and why people behave the way they do. Ethics approval was obtained for the paper from the Research Committees of CCT College Dublin and Waterford Institute of Technology respectively prior to the commencement of the study which took place between January and May of 2021.

Demographics
Feedback to the survey emanated from a wide range of higher and further education institutions. The two largest groups were the university and the institute of technology sectors representing 27% and 33% of responses respectively. See Fig.3.

Fig. 3 What type of academic institution are you working in?
Fifty-Six percent of respondents were lecturers followed by librarians at 16%. Responses were also obtained from educational developers, academic managers and learning technologists.  Focus group members comprised professionals from the university, private higher education and ETBI sectors in a range of roles. The authors were not successful in recruiting participants to the focus group from the IOT sector however. Please see Table 1 for details pertaining to the composition of the group.

Focus group member
Role 1. Student Success Manager and ICT Lecturer 2.
Senior Academic Manager 4.
Lecturer in Library Science 5.
Professional Learning and Development Coordinator 9.
Senior Academic Librarian Table 1

: Focus group members
The largest proportion of survey responses were from the science and engineering disciplines followed by social science, technology, business and technology. See Fig. 5.

Themes
This study employs an explanatory sequential design which comprises an initial quantitative phase followed by a qualitative phase. Qualitative data is used to elucidate quantitative findings (Creswell and Plano Clark, 2018). Meta-inference has been obtained through a narrative reporting process in which both qualitative and quantitative findings are explored concurrently on a side by side and theme-by theme basis (Guetterman, Fetters and Creswell, 2015). The following themes emerged from this methodological approach.

The underutilisation of eportfolio in higher education in Ireland
Farrell, (2018) discusses how "eportfolio has not emerged at scale in Irish Higher Education" (p.154), adding that this trend is "out of alignment with the ubiquitous nature of eportfolio in higher education worldwide". Farrell describes drivers of eportfolio use globally as concerted and includes government policy and flourishing communities of practice. Farrell (2018) also describes a lack of "explicit government policy" (p.154) in relation to the adoption of eportfolio in higher education in Ireland, "compared to the government policy drivers in the UK, U.S., Australia and New Zealand" (p.156). She describes how a rapidly expanding and energetic community of practice in Ireland, along with the work of National Forum for the Enhancement of Teaching and Learning is progressing eportfolio adoption in higher education in Ireland.
Survey findings in this study reinforce Farrell's article. Fifty-four respondents had not used eportfolio prior to starting the PACT digital badge. Of those who had used eportfolio, 59% had been using it for a year or less. Only 15% stated that they had been using eportfolio for more than three years. Additionally, 55% stated that their institution did not provide support for eportfolio use. Twenty-Four percent stated that they did receive support for training in relation to eportfolio; 18% in relation to hosting and 12% in relation to technical support. See   Only 24% of PACT participants used an existing eportfolio during the course. The remainder created a new one. WordPress was the most popular platform for those creating a new eportfolio with 45% of respondents choosing it compared to 21% of participants who chose to use the National Forum's Mahara instance. See Fig. 7. In response to the question what influenced your choice of eportfolio platform (participants were requested to select one or more reasons). Seventy-five percent of respondents cited ease of use as the reason for their choice, followed by cost at 25%. Twenty-Two percent stated that they availed of the National Forum's Mahara instance; 3% because the platform they chose integrated with an institutional license and 3% because of institutional support.

Fig. 7 Please indicate which eportfolio platform you used during the PACT course?
The majority of focus group members had no prior experience of eportfolio prior to the start of the course. Focus group members 1 and 2 were required to complete eportfolios as part of Masters programmes that they undertook. Having completed PACT, they regretted not maintaining these earlier portfolios however. The majority of focus group members stated that eportfolio adoption within their institutions for teaching and learning and professional development purposes remained ad-hoc. Focus group member 9 suggested that a strategic approach to eportfolio adoption was required nationally and institutionally. Focus group member 2 stated that institutions involved in blended learning have been proactively using eportfolio for teaching and learning purposes for many years. Its use within these institutions has also grown during the Covid-19 pandemic. She was less aware of eportfolio being used as extensively by staff for professional development purposes.

The complexity of Mahara as an eportfolio platform
The National Forum's PACT digital badge provides access to course participants to the Mahara platform in perpetuity. The Mahara instance drew mixed responses however with just 22% stating that they managed it from the outset. Fifty-Six percent of respondents stated that they found the Mahara platform confusing initially but worked their way through it, whilst 11% found it difficult to use throughout. Finally, 11% of respondents stated that they did not like the platform and had abandoned it. Open ended survey responses in relation to Mahara reinforced these findings. Responses included the following statements: The majority of focus group members also found Mahara difficult to use. Focus group member 3 stated that she reviewed Mahara in relation to possible institution wide adoption but feedback from staff and students, indicated that they found it clunky and difficult to use. Google Sites was chosen instead. She had also found Mahara difficult to use as a student on the PACT badge to the point that it occasionally undermined her intention to complete the course. Focus group member 6 stated that she didn't like the aesthetics of the Mahara platform and found the customisation aspect burdensome. By contrast she has maintained a Wordpress eportfolio which she established on an alternative programme of study. She also used this eportfolio during PACT.

Time management issues
A dominant theme in the survey and focus group findings was the difficulty of finding time to engage with eportfolio during the PACT digital badge course. Both survey respondents and focus group members felt that the time spent on eportfolio detracted from their engagement with course content on the PACT digital badge. Open ended survey comments also indicated a frustration with the time spent on constructing an eportfolio. One participant described how 'a lot of time' was spent 'trying to learn how to use Wordpress,' which was "disappointing as I had invested so much time and energy into the course with the intention of having my eportfolio for professional development ready to go." Another described how "Eportfolio set up can be very time consuming as a novice. It can detract from reflection on the actual content" All focus group members found engagement with eportfolio on the PACT course time-consuming. Focus group member 8 stated that if you were a novice who gave serious consideration to your professional persona and identity, the amount of time it took to configure the portfolio appropriately, compromised engagement with course content. Focus group member 2 also felt that the time that she devoted to the eportfolio aspect of the programme eroded her ability to engage with course content. She undertook the course "to reflect" and "not to take a digital badge in eportfolio". Focus group member 2 used a Wordpress eportfolio during the PACT digital badge and didn't maintain it upon completion of the course. She now uses a spreadsheet to maintain a list of her professional development activities in which she maps entries to the National Professional Development Framework. Focus group member 2 stated that it would have been helpful to have ready-made templates for those who do did not have the time to be too creative with the eportfolio aspect of the badge. These templates would support the PACT participant to create an appropriate eportfolio for the course and for on-going professional development in an efficient and time effective manner.

Sustainability
Sixty percent of respondents agreed or strongly agreed that they planned to use eportfolio going forward to engage with the National Professional Development Framework for All Those Who Teach in Higher Education. Survey data revealed the reasons why the remaining PACT participants stated that they wouldn't. Reasons included lack of time and technical expertise. See Fig.8.

Open ended survey comments included:
"More scaffolding of support at the start and end of the course to ensure effective engagement during the badge but also beyond the lifetime of the badge. Perhaps where a NF course has an eportfolio dimension students should have to complete a badge on eportfolio first." Focus group member 9 highlighted the growing importance of LinkedIn for capturing and evidencing professional development activity as a potential threat to eportfolio. She uses OneNote for her eportfolio but has struggled to maintain the platform due to time constraints. Focus group member 2 now tries to reflect on themes as opposed to individual conferences and events which she finds a more time effective approach. Stating that she "loves a deadline", she wonders if the National Forum would prompt PACT participants to "write a reflection on a theme every quarter". This would support the sustainability of eportfolios beyond the life span of the course.
Focus group member 3 found the process of maintaining a portfolio and aligning entries to the National Forum's Professional Development Framework helpful in terms of reflecting on her professional development but she has struggled to maintain the eportfolio over the last year. Focus group member 6 stated that the COVID-19 pandemic eroded her motivation to maintain her eportfolio with the extra demands placed on staff professionally during this period.
Focus group member 8 stated that the pandemic made staff much more strategic about what they maintain and what they don't. She felt that she struggled with the eportfolio aspect of the digital badge and that she wanted to get more comfortable with eportfolio before she rolls it out to staff. She added that the pandemic has made "people more experimental but also more focused".

Earlier provision of support for eportfolio on the PACT digital badge
In response to open ended questions on how interaction with the eportfolio component of the course could be improved, survey respondents and focus group members stated that the front loading of training prior to the commencement of the badge was critical. Open-ended comments included: "More instruction on eportfolio before the course starts. " Focus group members also advocated the front loading of additional training and resources in relation to eportfolio use on the PACT digital badge. In comparison to survey respondents, several focus group members highlighted the educational and support role of eportfolio communities of practice in Ireland. Focus group member 6 observed that there was a much higher prevalence of literature and events pertaining to the embedding of eportfolio into teaching and learning practice. Much of this insight and guidance is disseminated by experts on social media platforms such as Twitter. All focus group members were aware of eportfolio Ireland as an example. Focus group member 4 highlighted the importance of institutional champions to increase engagement with and support the adoption of eportfolio. Focus group member 1 described how the eportfolio aspect of the PACT digital badge commences in week two or three advocating that engagement with eportfolio should commence in "Week 0". Focus group member 3 facilitated an iteration of the PACT badge and promoted Google sites due to its ease of use, producing an accompanying video which was provided to participants prior to the commencement of the programme. Consequently, complaints about the eportfolio component of the course were less prevalent on this iteration of the badge.

Eportfolio promotes engagement with course content and with the National Framework for All Those Who Teach in Higher Education
Seventy-Two percent of survey respondents said that they better understood the National Forum's National Professional Development for All Those Who Teach in Higher Education as a result of the eportfolio component of the course. 63% of survey respondents agreed or strongly agreed that eportfolio helped them to capture and document course work for the peer triad discussions. Less positive scores were elicited for the role of eportfolio in promoting collaboration. Forty-four percent agreed or strongly agreed that eportfolio facilitated collaboration with other learners. Fifty-four percent agreed or strongly agreed that eportfolio facilitated the submission of assessment tasks. Forty-four percent agreed or strongly agreed that eportfolio underpinned feedback from facilitators. Thirty-six percent of participants strongly agreed and 42% agreed that eportfolio was an important part of the PACT digital badge. Overall, 62% of respondents stated that the learning experience on the PACT course was enhanced by use of eportfolio. Eight-one percent agreed or strongly agreed that eportfolio promoted reflection. Open ended survey comments included: The majority of focus group members felt that eportfolio promoted reflection. Focus group member 6 felt that the triads were a more effective mechanism for reflection however. Focus group member 5 felt that having a pre-existing and established eportfolio prior to the commencement of the PACT badge made the learning process far more beneficial and rewarding.

The inclusion of eportfolio on the PACT badge promotes engagement with eportfolio in teaching and learning
Sixty-nine percent of survey respondents agreed that their knowledge of eportfolio had increased as a result of their participation on the PACT course whilst 60% agreed or strongly agreed that their confidence with eportfolio had increased as a result of being on the PACT course. Sixty percent of survey respondents agreed or strongly agreed that they planned to use eportfolio going forward to engage with the National Professional Development Framework for All Those Who Teach in Higher Education. Sixty-two percent agreed or strongly agreed that the learning experience has enhanced use of eportfolio in their own teaching practice. Sixty-five percent of survey respondents agreed or strongly agreed that they planned to take knowledge they had acquired as a result of participating on the PACT digital badge back to their institutions. Seventyeight percent of survey respondents stated that eportfolio should remain a central component of the PACT digital badge. Eighty-four percent of survey participants felt that integration of eportfolio into professional practice courses is an effective way to promote engagement with eportfolio by staff in education. Open comments include: Focus group members 4 and 9 are working with the National Forum and the Library Association of Ireland to produce a version of the PACT digital badge to promote uptake amongst library personnel at all levels sector wide. Focus group member 3 stated that she encourages all new programmes within her institution to incorporate eportfolios. Having the facilitator badge for PACT made her more aware of promoting eportfolio generally.

Conclusions
An explanatory sequential mixed methods design has served a number of functions in this paper providing a more in-depth analysis of themes from the survey data as well as the triangulation of findings for additional rigor and validity. In this study, focus group findings and the literature review corroborate and expand upon a number of key concepts emanating from survey data supporting the generalisation and transferability of findings which is particularly important given the relatively small survey sample size. This study has found that the pairing of eportfolio with the PACT digital badge enhances the learning experience, promotes reflection and additional skill acquisition as well as on-going engagement with the National Professional Development Framework for All Those Who Teach in Higher Education. These findings could be tested on a larger scale across National Forum open courses that have an eportfolio component. Bryant and Chittum (2015) in their review of the eportfolio literature indicates that learners do not always appreciate a teaching approach or tool even if it is beneficial for them. In the context of this paper, this was not the case. This may be in part because of the nature of the learner on the PACT course which is a higher or further education professional. Seventy-Eight percent of participants deemed the eportfolio component of the PACT badge as central to the delivery of the course and the learning experience.
Data emanating from both the survey and focus group did not indicate a tension between the extrinsic motivation of a digital badge and the more intrinsic, introspective and reflective elements of an eportfolio.

Recommendations
A key finding of this study is that eportfolio on the PACT badge enhances the learning experience, develops other key skills such as reflection and digital literacy and promotes engagement with the National Framework for Professional Development for All Those Who Teach in Higher Education. Survey and focus group participants indicated that the time and technical knowledge required for platforms such as Mahara in the context of a short six-week course are key challenges however.
A key recommendation of this study pertains to the additional front loading and scaffolding of information and training on eportfolio prior to the commencement of the programme in Week 0.
Another key recommendation is the provision of eportfolio templates or components as per focus group feedback to underpin more effective and efficient eportfolio engagement on the PACT course. Whilst 60% of PACT participants indicated that they would maintain engagement with their eportfolio upon cessation of the course, consideration could also be given to the provision of additional support to scaffold post-PACT eportfolio engagement to increase this percentage. This model of eportfolio support could be extrapolated to all digital badges offering eportfolio.