A Higher Education Action Research Study on the Effectiveness of an ePortfolio as a Learning Tool to Promote Reflective Professional Development

Within higher education, ePortfolios have been demonstrated to enhance professional development and to promote critical thinking. In this context, one higher education level nine module, namely, the Professional Development Framework Portfolio module, was analysed. The module was piloted during the 2019-2020 academic year in one Irish Institute of Technology. It was intentionally designed to align with the Professional Development Framework promoted by the Irish National Forum for the Enhancement for Teaching and Learning. The primary aim of this study was to explore the effectiveness of an ePortfolio as a learning tool for professional development, in the context of the module’s delivery. An action research methodology was employed. Data collection tools and processes included a learnerquestionnaire, an ePortfolio mapping analysis, and a dialogical reflection on the part of two lecturer-researchers. Salient findings were that the use of ePortfolios for the purpose of professional development in higher education was evidenced, and the effectiveness of ePortfolios to facilitate the mapping of professional development was established. The main recommendations to enhance this module for future action research iterations included: 1) to facilitate deeper reflection on personal professional development goals, 2) to support reflective practice and reflective writing skills, and 3) to provide a pre-module workshop on ePortfolio web development prior to module commencement. More broadly, it was also recommended to revisit the time demands of the module and to research the extent to which the module might effectively promote a sustainable professional development community of practice framed around ePortfolio development.


Introduction
In Ireland, the adoption of ePortfolios has been slow (Farrell, 2019). However, in recent years, Eportfolio Ireland and the National Forum for the Enhancement of Teaching and Learning (NF) have advanced an understanding of the relationship between ePortfolios and professional development (PD) in higher education (HE) in Ireland (Donnelly & Maguire, 2018;Farrell, 2019;NF, 2019). PD is an all-embracing term for a multitude of developmental processes, experiences, and opportunities-formal and informal (Desimone, 2010;NF 2019). It refers to experiences of educational practice, in a broad sense, with a focus on enhanced performance and outcomes (Darling, Hammond & McLaughlin, 2011). Ownership of learning, self-directed learning, and the application of learning in the real world of practice (otherwise referred to as 'authentic learning') are significant aspects of PD (Hargreaves and Fullan, 2012, p. 132). The NF Professional Development Framework (PDF) aligns with the Irish National Strategy for Higher Education to 2030, which proposes that, in HE "increased attention must be paid to core skills such as quantitative reasoning, critical thinking, communication skills, team-working skills and the effective use of information technology" (Department of Education & Skills, 2011, p.35). The PDF refers to five domains of learning, namely, 1) the self, 2) professional identify, values and development in teaching and learning, 3) professional communication and dialogue, 4) professional knowledge and skills, and 5) personal and professional digital capacity.
One vehicle to achieve this strategic goal is ePortfolio-based learning and assessment. While the ePortfolio can act as a showcase platform or a repository, it is potentially much more than that (Farrell, 2019, p.13). Building upon Philippa Butler's literature analysis of ePortfolios (2006), Helen Barrett defines an ePortfolio as "an electronic collection of evidence that shows your learning journey over time" and involves a balance between process and product (Barrett, 2010, p.6).
In light of the above context, the research question addressed in this study was: 'how effective is an ePortfolio as a learning tool to promote reflective professional development? The study focused on a level nine staff development module, entitled, Professional Development Framework Portfolio (PDFP), which was piloted in a Higher Education Institute of Technology during the 2019-2020 academic year. Participants of the pilot consisted of academic and support staff. The module learning outcomes mapped against the NFs PDF (NF, 2019) with a PDF teaching and learning ePortfolio used to display the module learning processes and outputs.
In line with the above research question, the primary aim of this study was to investigate the effectiveness of an ePortfolio as a learning tool to promote reflective professional development. The objectives were to: • Provide a concise thematic literature analysis on the use of ePortfolios in higher education, in the context of PD. • Conduct primary research on the pilot of the PDFP module, with an emphasis on evidence-based PD, critical thinking, and technology-related learning.
Methodologically, this was an action research (AR) study, and the data collection methods included a learner Microsoft (MS) Forms questionnaire, an ePortfolio mapping analysis process, using Microsoft Excel, and a videoed lecturer-researcher dialogical reflection.
Structurally, the article begins with a concise analysis of literature relating to ePortfolios in HE, most especially in the context of PD. It then outlines the AR pilot study and ends with a literature-informed discussion.

Literature Analysis
The following literature analysis explores three dominant themes that emerged from the literature, in the context of ePortfolios in HE, namely, evidence-based PD, critical thinking, and technology-related learning.

Evidence-based PD
The ePortfolio is an effective vehicle to document evidence-based continuous PD (Hampe & Lewis, 2013). Where used in HE, ePortfolios are presented as both learning tools and as an electronic collection of evidence of a learning journey in the form of a showcase that can enhance learning, PD and critical thinking (Farrell, 2019). The NF refers to the importance of "evidence based, flexible, inclusive professional development for all those who teach" in their key strategic priorities (NF, 2019, pg.4). It is generally accepted that examples of PD evidence that might be included in an ePortfolio include professional qualifications and credentials, continuous professional development, research or scholarship, membership of professional bodies and academic awards. McLean and Bullard (2000, pg. 95) list other forms of evidence as lesson plans, teaching aids, accounts of being observed, minutes of meetings, extracts from a teaching journal and assessment criteria. As part of the NF PDF the need for the individual to carefully choose evidence representative of their PD is highlighted, as is the need to store that evidence (NF, 2019). Critical thinking on evidence gathered is key to understanding where one is currently on the PD journey.

Critical thinking and ePortfolio
Developing an ePortfolio can facilitate a "process of reflection, of organizing, prioritizing, analysing, and communicating one's work and its value, which may prompt insights and goals" (Zubizaretta, 2009, p.124). As such, it is rooted in pedagogy (Chen & Black, 2010, p.2;Eynon & Gambino, 2017) and can support developmental pedagogical reflection in Personal Learning Environments (PLEs) (Oakley, Pegrum & Johnston, 2013). Critical thinking frameworks, such as the 'critical folio thinking' framework, is associated with ePortfolios in the literature (Bryant & Chittum, 2013;Farrell, 2019) and is aligned with authentic, student-centred and meaningful learning (Baird et. al., 2016). Farrell states that "effective learning portfolio practice can stimulate critical thinking…(and)… provide space for students to experiment and reflect on their learning journeys (Farrell 2019, p.13). As such, they are spaces "focused on learning and growth" Eynon & Gambino's (2017, p.1). While this learning and growth can be individual and independentas in the case of professional academic development and promotion (Winberg & Pallitt, 2016) -ePortfolio learning and growth is commonly aligned with the formation of learning communities, involving collaboration with peers, establishment of communities of practice and a structuring of peer support (Farrell, 2019, p.96;Shepherd & Bolliger, 2014;Tucker, 2012).

Technology-related learning
The relationship between digital learning and ePortfolio development is also examined in the literature. Rowley and Bennett, exploring technical advantages of ePortfolio development in the context of creative arts in one Australian HE Institute, write that "many students see ePortfolios as part of wider and increasingly developing educational and personal technoscapes", and educators see advantages to ePortfolio learning platforms, such as, easy access to materials for assessment, and collaborative learning (Rowley and Bennett, 2016, p.8

Research Methodology, Methods and Analysis
As indicated above, the research question addressed in this study was: how effective is an ePortfolio as a learning tool to promote reflective professional development? To address this research question, the methodological framework employed in this study was AR. This approach was deemed appropriate since it relates to the practice of teaching: the researcherlecturers were applying AR to a specific teaching and learning context, seeking to transform teaching and learning practice in subsequent cyclical iterations. Furthermore, it was an intentionally collaborative AR process, of dialogical action research, where the reflective process was situated in the context of dialogue or reflective conversations between the researcher-lecturers. The researcher-lecturers acted as mutual professional 'critical friends' and engaged in 'Dialogical Reflections' as adapted from Costello et al. (2015).
With respect to sampling, purposeful sampling was used, participants being the candidates who signed up as the first pilot iteration of the PDFP module. In this first pilot iteration of the module, funding allowed for a relatively small cohort (n=10), with a view to future expansion in future AR cycles, following the pilot (hence the small sample size in this case).
Mirroring the literature discussion above, data gathering was centred around three dominant themes-evidence-based PD, critical thinking, and technology related learning. In the interests of triangulation, three data gathering methods, were employed: 1) learner questionnaire, 2) ePortfolio mapping analysis, and 3) lecturer-researcher dialogical reflection, structured as a SWOT (strengths, weaknesses, opportunities and threats) analysis. The data gathering and analysis processes were made up of three phases, each of which involved a cyclical process of data gathering and analysis. Each phase in the sequence was informed by the preceding one. See Figure 1.

Figure 1: Data gathering and analysis phases
The questionnaire (see Appendix A) explored the three literature themes from the perspective of process and product (Barrett, 2010, p.6). A mixed method approach was taken including quantitative and qualitative questions. An analysis of the questionnaire results led to the identification of sub-themes. Those sub themes were then used to guide the gathering of PD evidence from the research participants' ePortfolios using a mapping analysis template (see Appendix B). Subsequently, a quantitative frequency analysis and a qualitative content analysis were conducted on the evidence gathered. Phase three involved a dialogical reflection by the researchers based on a SWOT analysis of the findings from phase 1 and 2 using a SWOT analysis template (see Appendix C). Ethical approval was obtained for the research, as per institute policy.
As indicated, the three dominant themes that emerged from the investigation were evidencebased PD, critical thinking and technology-related learning. In the case of evidence-based PD, the three main sub-themes identified were: PD action plan, reflective practice and the philosophy statement. For the second dominant theme -critical thinkingthe sub-themes were critical reflections on the NF five domains, multi-media reflections on teaching and learning and NF evidence based reflection model. The third dominant theme-technologyrelated learning -had as its sub-themes digital content, usability and design. To facilitate weighting, the data was coded according to a Likert quality 3-point scale system, based on poor evidence, fair evidence, and good evidence of the presence of the sub-themes in the participants' e-Portfolios (see Table 1). Analysis standardisation and validity -across all e-Portfolios -were provided for by means of a double cross-check involving both researchers. Finally, a frequency analysis was carried out on these three categories of evidence, based on the incidence of evidence.

Results
This section presents the results of the data gathering and analysis process addressing each dominant theme in turn.

Evidence-based PD
The MS Forms questionnaire was electronically distributed to the module candidates (n=10) with a response rate of 8. This questionnaire, in part, explored the respondents' general learning experience of building an ePortfolio. The first three statements, as show in Figure 2, indicate strong agreement that building an ePortfolio gave them a clear understanding of the strengths and gaps in their PD, a clear plan for future PD aligned to the NFs PDF, and the PDF assisted the respondents to reflect on their own PD. One of the respondent's comments on the process leading up to, and including, building an PD ePortfolio was that "(t)he structure and motivation helped in reflection & research in areas I should as a professional educator have done anyway but have not due to lack of focus or urgency". The fifth statement in the question also shows strong agreement that PD evidence covers both accredited and non-accredited PD learning. A respondent confirms that "the process helped me make things more clear [sic] and value not just academic achievement but industry and personal achievements as well".

Figure 2: PD ePortfolio building experience results
Learning activities within the PDFP module that supported an evidence-based approach to PD were considered in question five of the questionnaire. The results of this question are seen in Figure 3. There was strong take-up of learning activities: reflecting on your PD to-date, individual journaling, developing a PD plan, and module submission formative reviews were identified as being very effective. On the other hand, in the case of learning activity 7creating an audio reflection and learning activity 8 -a key learning video -the scores were lower, with one respondent indicating it was very ineffective for an evidence-based PD ePortfolio.

Figure 3: Effectiveness of module learning activity results
A summary position of the ePortfolio mapping analysis, based on the dominant themes, using the above ePortfolio mapping analysis key, is outlined in Table 2 below.

Table 2: Evidence-based PD ePortfolio mapping analysis
As can be seen above, there was evidence of PD action plans in the ePortfolios reviewed. All nine ePortfolios had an action plan that aligned with the NF Five Domains, and all respondants had included short, medium and long term goals. Emphasis was predominantly placed on short term goals. There was less evidence of medium term goals and very limited evidence of long term goals. A shortfall was found in the evidence of reflective practice, from the perspective of quality and depth. Only one respondent demonstrated deep (or higher order) reflection on the NF 'Self' domain, and six of the ePortfolios analysed had poor evidence of reflections on their T&L practices. In terms of the philosophy statement sub theme, the majority of participants (n=6) were found to only have fair evidence of T&L theoretical underpinnings and all nine participants had little explicit evidence of the application of the philosophy theory to their T&L practice.
The third and final data gathering method was a SWOT analysis that emerged from the lecturer-researcher dialogical reflections (see Figure 4).

Figure 4: Evidence-based PD SWOT analysis
Strengths included PD five domain reflections, Pair/Triad engagements, facilitator instructions and feedback. Some weakness emerged including engagement in deep selfreflection, link between philosophies and T&L evidence, and medium and long-term action plans. Opportunities identified were the provision of samples as criteria for success, the establishment and fostering of a community of practice and dissemination opportunities. Lastly, threats included time-constraints, technical challenges to setting up an ePortfolio and barriers to reflective practice. Building on the evidence-based PD results the critical thinking domain results are presented next.

Critical thinking
Research participants provided insights into their general experience of using critical thinking during the PDFP module, in question 12 (see Figure 5). A 'strongly agree' response was selected by n=5 of the participants for the statement that the module enabled them to critically reflect on, plan and contribute to evidence-based enhancement and transformation of their teaching and learning practice. The same response was found when participants indicated their self-perception of their level of reflective practice, as a result of undertaking this module. The support of the PDF to enable the participants to critically self-reflect and self-evaluate their current teaching and learning practices with a view to developing professionally was evident, with a 50-50 spilt between strongly agree and agree responses.

Figure 5: Critical thinking skills development results
Participants corroborated the above findings with one stating "I feel I am now better at applying critical thinking and reflection to aspects of my teaching and exploring other eportfolio samples provided more material of which to consider". A number of participants also indicated the challenges that they encountered when trying to engage in critical thinking and reflective dialogue with their peers. For example "(t)his is one area where I felt I lacked a bit as work was so busy at the end of semester (it was no ordinary semester) and the opportunities to really devote a chunk of time to the module were hard to come by". And a second participant states "(w)orking to the last minute also detracts from deep reflection and learning as there is little time to reflect, digest and learn from what others may be doing / creating".
Findings pertaining to the impact of the PDFP module on critical thinking on the PDF domains showed a very positive response with either strongly agree or agree selected by the participants for all five domains (see Figure 6). The 'Self' and 'Professional Identify, Values and Development' domains received particularly high strongly agree responses.

Figure 6: PDF domain critical thinking results
Results from the second data gathering tool, namely the ePortfolio mapping analysis, are illustrated in Table 3. ePortfolio evidence of critical reflection on the PDF 5 domains was found to be generally of high quality, with n=5 for good evidence, and n=4 for fair evidence. Reply posts to Pair/Triad posts about the 5 domains were found to be lacking in higher order thinking. Only two of the ePortfolios reviewed demonstrated good evidence with the majority in the fair category. Multi-media reflections in the form of audio and blogs proved challenging for the participants with n=6 participants opting not to include an audio reflection, and n=5 opting not to include a blog reflection. Those included in their ePortfolio were generally found to be of poor quality. The third sub-theme under critical reflection analysed the application of the NF Evidence Based Reflection Model (EBR) in the participants ePortfolios to be evidenced through a video and the participants T&L practice. Seven participants did not include a video in their ePortfolio, and five were identified as not applying the EBR model to their practice. It was particularly difficult to find any evidence of the latter with evidence implied rather than being explicit, and indeed absent in some places.
The third and final data gathering method was a SWOT analysis that emerged from the lecturer-researcher dialogical reflections (see Figure 7).
In relation to the theme of critical reflection, a number of strengths were identified, including the quality of critical reflection posts, ePortfolio development aligned to critical thinking, and critical thinking aligned to the PD five domains . Some weaknesses that emerged during the dialogical reflection included the timing of the module introduction to the use of media for reflective evidence, inclusion of multi-media in the ePortfolio and academic rigour of 'reply posts' to critical reflection. Opportunities included facilitator expertise, engagement with Institute learning technologists, and the development of a community of practice. Lastly, threats included time constraints, clear communication of T&L focus to prospective students, and moderation requirements to develop critical thinkers.

Figure 7. Critical thinking SWOT analysis
Findings of the third and final domain, technology-related learning are presented next.

Technology-related learning
The technology-related learning questionnaire identified findings around how ePortfolio building problems were solved, as illustrated in Figure 8. Pair/Triads emerged as the most popular support for problem solving. The was followed closely by the use of a step-by-step guide provided by the module, YouTube/Internet and module facilitator support.
Interestingly, the services of the Institute learning technologist and the IT Services were not availed of by any of the respondents when problem-solving.

Figure 8: Technology-related learning SWOT analysis
One participant commented on the value of the Pair/Triad for problem solving, saying "I felt the pair/triad was a great place to start if I didn't know something as there was someone there who directly understood what I was going through or was there to give help, ask questions of directly… who you had built up a working relationship with throught the course". Another respondent reported that they "(f)ound Weebly a very straight forward platform to engage with". When the participants were asked if they thought that an ePortfolio is an effective learning tool to promote PD, n=6 responded 'yes' (see Figure 9).

Figure 9: Effectiveness of an ePortfolio as a learning tool to promote PD results
One participant reflects on the effectivness of using the ePortfolio saying For me building an e-portfolio clearly shows to me and anyone looking at it what I have achieved as a professional both academic and non academic which giving me short medium and long term goals which will change as I reach each mile stone. I now feel more clarity in the direction I am going while better reflecting critical thinking and actioning on my goals and aspirations in both professional and non professional capacities.
Participant ePortfolio mapping analysis results under the 'technology' theme are presenting in Table 4. Frequencies associated with the sub themes of digital content, usability and design can be seen.

Table 4: Technology-related learning ePortfolio mapping analysis
In terms of digital content the majority (n=5) were found to have good evidence of content with high readability. A small number had display and conciseness issues. With respect to 'useful content', n=4 participant ePortfolios demonstrated good evidence and n=5 had fair evidence. On the sub theme of 'usability of the ePortfolios', n=4 had good evidence with a number of very comprehensive ePortfolios identifed. However, there were some navigation issues with the remaining ePortfolios. A finding of concern was the poor evidence of source referencing for content in the majority of the ePortfolios: evidence for all ePortfolios fell either into the poor or no evidence categories. The design aesthetics applied by some (n=5) of the participants were found to be good, with consistent themes applied. The remainder, however, lacked the professionalism expected of a PD ePortfolio at level 9. Lastly, there was a difference between those that strategically identified as a PD ePortfolio (n=6) and those that did not (n=3).
The third and final data gathering method was a SWOT analysis that emerged from the lecturer-researcher dialogical reflections (see Figure 10).
Technology-related learning strengths included strong design aesthetics with consistent themes throughout, effectiveness of ePortfolio development process steps, and the development of PD digital literacy domain skills. Weaknesses under this theme consisted of a focus on technology rather than on PD content itself, lack of content variety and data access, and issues relating to security and privacy. Institute technical support, internal expertise in ePortfolio design and development, and ePortfolio development workshop prior to module opportunities emerged as suggestions for a future iteration from the dialogical reflections. Lastly, the threats identified included time, student technical abilities and institute provision of an appropriate ePortfolio platform.

Figure 10: Technology-related learning SWOT analysis
Having completed the data analysis, let us now turn to a discussion of the findings.

Discussion
Acknowledging the wider context, this study addressed the research question: 'how effective is an ePortfolio as a learning tool to promote reflective professional development?'. The study focused specifically on the role of ePortfolios in fostering evidence-based PD, critical thinking, and technology-related learnings. Let us discuss each of these dominant themes in turn.

Evidence-based PD
Literature-research on PD modules testifies to a number of professional advantages for learners (Gormley, O'Keefe & Bruce Ferguson p. 2017, p.104), not least that accredited PD effects transformation (Maguire, Harding, Noonan and O'Connor, 2017, pp. 8-14). This study indicated that the accredited PDFP module contributed effectively to PD collation, development and future planning. On the the effectiveness of the ePortfolio as a learning tool to promote PD, one participant writes: "(i)t gathers all your information in one central location. A single source of truth!". The questionnaire, in part, explored the respondents' general experience of building an ePortfolio. In line with Hampe & Lewis (2013) there was strong agreement that the ePortfolio is an effective vehicle to document evidence-based continuous PD. Moreover, building an ePortfolio gave learners a clear understanding of the strengths and gaps in their PD and assisted future PD planning. The lecturer-researchers identified "very detailed action plans, particularly around the area of their short-term action… maybe not so detailed when it came to the medium and long-term plans". There was substantial evidence of PD action plans in the ePortfolios reviewed, with the majority including short, medium and long term goals. All ePortfolios reviewed had an action plan that aligned with the NF 5 Domains. Emphasis, however, was found to be placed disproportionately on short term goals, indicating the need for more focused exploration on medium and long-term timeframes, in future implementations of the module.

Critical thinking
Critical thinking on the PD evidence gathered is key to understanding where one is on the PD journey. Farrell (2019) argues that "effective learning portfolio practice can stimulate critical thinking…(and)… provide space for students to experiment and reflect on their learning journeys (Farrell 2019, p.13). This study found that participants value the opportunities provided to critically reflect, and there was evidence of critical reflection aligned with personal philosophy statements. One participant writes: " I reflected on how my teaching philosophy has evolved over my career and how my values on teaching and learning impact on my approaches to teaching and student engagement." Barriers to critical reflection, including collaborative reflection, was an issue that emerged over the course of the study. One participant said that "(t)iming and circumstances have prevented … dialogue with peers … but the intention would be to visit this regularly in the future". One respondent demonstrated deep and profoundly philosophical personal/profession critical reflections on the 'Self' domain, however, others strugggled to achieve and/or articulate a depth of critical reflection, particularly in relation to the'Self'domain. The lecturer-researchers observed a discipline-related pattern in relation to perceptions of, and the practice of critical selfreflection, and identfied the need for specific supports for discipline-specific cohorts in the future.

Technology-related Learning
International studies have identified key factors for fostering ePortfolio learning in HEincluding technical standards (Hallam & Creagh, 2010). This study identified varying competencies in the use of technology and the need for technological support in the context of ePortfolio development, mirroring the statement by Oakley, Pegrum and Johnston (2014, pg. 47) that "since technology avoidance is not an option the participants undertaking an ePortfolio need to be supported through 'how to' and 'troubleshooting' workshops". While the skills obtained and the expansion of digital learning and "personal technoscapes" is valued by learners (Rowley and Bennett, 2016), the process is highly time consuming and pressurized. Significant supports are required. Furthermore, excessive focus on technical skills development can detract from deeper PD learning. One participant struggled with a focus "on the acrobatics and performance and frills" of techology and suggests that "there may be less focus on process and [more] perhaps on content." Another commented: "sometimes there was a risk … technology … become more important than the actual content. Reflecting on this concern, researcher-lecturers proposed, as a solution, a pre-module web development workshop.
It is to be noted that, apart from the three dominant themes focused on above, a further emerging theme was the potential of ePortfolio development to foster an institute (and wider) ePortfolio community of practice. ePortfolio learning and growth is commonly aligned with the formation of learning communities (Farrell, 2019, p.96;Moran et. al, 2014;Shepherd & Bolliger, 2014;Tucker, 2012). A potential follow-on area of research investigation is the role played by ePortfolios in forming learning communities or communities of practice (CoPs) within the PDFP module, and beyond.

Conclusions & Recommendation
This research set out to investigate the effectiveness of an ePortfolio to promote professional development. Three dominant themes emerged from the investigation: evidence-based PD, critical thinking, and technology-related learning -each with corresponding sub-themes. Salient conclusions -from both literature and primary research -are that ePortfolios are effective in promoting PD, critical thinking and technology-related learning in HE contexts.
With respect to this AR study, challenges included web development skills, medium and long-term PD planning, limited self-reflection skills, and a competing demanding workload for module participants.
The recommendations that arose from this study, in relation to module delivery, included: 1) to provide a pre-module workshop on ePortfolio web development and design prior to future delivery of the PDFP module, 2) to facilitate a deeper PD reflection on medium and longterm PD action planning, and 3) to provide increased supports on personal and professional reflection, in the context of the NF 'Self' domain, 4) to re-visit assessment demands in light of module workload, and 5) to investigate the extent to which the module might effectively promote a sustainable PD community of practice framed around ePortfolio development.