Irish Journal of Technology Enhanced Learning, Vol 7, Issue 2
Special Issue: The Games People Play: Exploring Technology Enhanced
Learning Scholarship & Generative Artificial Intelligence
https://doi.org/10.22554/ijtel.v7i2.122
Kristen Peña
Jonathan McMichael
Medha Dalal
Arizona State University
We explore whether Generative AI can serve as a valuable tool for educational and scholarly contexts. The current iteration of Large Language Models (LLMs) exhibits several critical shortcomings for reliably producing an article fit for publication, including imprecise information, overt hallucinations, problematic oversights, and vague language that would typically render it unsuitable for rigorous academic discussion. However, Generative AI can present us with viable, impactful ideas for improving learning. It is incumbent on us as educators to recognize, evaluate, and develop those ideas for our campuses and communities.
This article presents our accounts of testing the use of a Large Language Model (LLM) to write a short report on the topic of a technology-enhanced student success plan to improve first-year, first-generation college students' experiences at a large, public university.
In our staff roles at an engineering teaching and learning center in a large, public university, we, the authors, support and collaborate with engineering faculty and course instructors to create environments and experiences that foster student learning. Our relationship with Generative AI (GenAI) is characterized as positive in this context. We welcome GenAI as a tool to help students learn (within faculty guidelines), encourage educators to enhance their teaching with GenAI, and recognize the opportunity for its use by administrators as a transformative tool to address complex educational challenges. To further establish transparency, the first and the third authors have a basic familiarity with GenAI tools whereas the second author has more experience with GenAI serving on university-wide committees, leading professional development sessions, writing, and participating in panel discussions on the technology’s use in the higher education classroom. At the moment in our university, the use of GenAI by students and faculty is still being discovered and GenAI use by staff and administrators has not been the focus of conversation.
We chose the topic of student success programs because of two primary reasons:
Approximately 56% of students nationally are first-generation college students (Hamilton, 2023) and 34% of our university’s students are first-generation (Facts and Figures, n.d.), meaning that their parents did not complete a four-year college degree. The first and second authors are first-generation college students. We also recognized that retention during the first two years of college is critical (Burgette & Magun-Jackson, 2008; Clark, 2005; Graham et al., 2013). Creating or improving technology-enhanced student success programs for first-year, first-generation college students would be especially valuable for us to recruit and retain a diverse engineering student body.
Phase 1: Topic and LLM Selection
In the initial meeting for this short report, we selected the above-mentioned topic and a strategy for engaging with a LLM in a responsible manner. With ample practitioner and scholarly experience on the topic, the authors felt confident in evaluating the LLM's output. We also noted that being from a large public university in the United States and adding this context in the prompt would align well with the LLM's training data. If we had chosen a somewhat obscure context or less common topic, we would expect lower-quality results (Wang et al., 2023). Due to the inability of the LLMs to produce the required 2000-word report in its entirety, our strategy involved using an initial prompt, hereby referred to as a starter prompt, to generate an outline, followed by subsequent prompts to compose each subsection. The final article would be a compilation of this interaction, edited for formatting to meet submission criteria.
We began the process by drafting a starter prompt that would effectively capture the topic and we used the prompt to test the viability of various LLMs. Examining the responses, we concluded that ChatGPT-4, which is only available to paid subscribers, offered the highest quality writing and coverage of the topic. However, the prompt was going to require further refinement to produce a usable draft.
Phase 2: Interactions and Revisions
With each revised starter prompt, the second author had a series of interactions with ChatGPT-4 to test and attempt to further refine the final product. These interactions lasted a few hours of multiple sessions and included minor tweaks to the language of the starter prompt, changes in the types of feedback offered to the LLM, and experiments with the product generation process.
This also involved adding an outlining phase to the series of prompts to create the draft, adding and removing references to specific pieces of literature or terminology, prompting examples for concepts, and emulating the structure of target articles. For example, our initial prompt included references to van der Zanden’s Domain of First Year Success (2019) but was removed as ChatGPT-4 was not aware of this specific article. There were three major revisions to the starter prompt. Additionally, the transcript of our final interaction with ChatGPT-4 shows that we needed to prompt it for real-world applications (see Examples under 2.4 Proposed Plan heading) of the approaches presented in its plan.
Ultimately, the starter prompt used to create this draft was:
Write an outline for a short report that proposes a critically engaged practice approach to develop a technology-enhanced student success plan aimed to improve first-year first-generation college students' success at a large, public university campus. The short report should be 2,000 to 2,500 words. Include citations and references. Sections of the short report should be an introduction, background, the main body with a plan to respond to these challenges that are informed by the relevant research on this subject, and a conclusion that offers advice to other colleges for implementation. The report should have an academic or scholarly tone and be from the perspective of a scholarly practitioner.
Phase 3: Compiling LLM outputs
Because ChatGPT-4 was not capable of creating a full draft of 2000 to 2500 words, subsequent prompts were used to ask ChatGPT to fill in each section of the outline. Those prompts included: “Great, for each of these sections let's create a draft. Let's start with the Introduction. Follow the outline above and write a draft” and “Great! Let's move on to the next section. Create a draft of the Background section. Follow the outline above and write a draft.”
We then took the transcript of the interaction and compiled it into one cohesive draft included in the next section. It is lightly edited to adhere to APA and journal formatting guidelines and to meet the length restrictions. The full transcript of this interaction can be found in the reference section (OpenAI, 2023). The title and headings were generated by Chat-GPT. Notably, several sections contained redundant material (i.e., Thesis Statement, Implementation Strategy, Challenges and Limitations, and Conclusion) and were cut completely or combined in the Advice for Implementation section (2.5) to better adhere to the short report word limit. Real-world examples were also added to the proposed plan section (2.4) from a separate interaction that was removed. Ideally, these sections would have been opportunities to expound on ideas introduced in previous sections, but the LLM instead reiterated points without any additional depth.
ChatGPT titled this short report, “Leveraging Technology-Enhanced Strategies for Elevating Success among First-Year, First-Generation College Students.” It is 2556 words in length, with references generated by ChaGPT-4.
Definition of First-Generation College Students
First-generation college students are typically defined as those for whom both parents have not attained a baccalaureate degree, navigating through the unfamiliar terrains of tertiary education without a direct parental blueprint (Pascarella et al., 2004). These students, often hailing from diverse socio-economic and cultural backgrounds, encounter unique challenges as they embark on their higher education journey, negotiating through academic, financial, and socio-cultural landscapes that might be foreign to their familial experiences.
Significance of the Report
The significance of addressing first-generation students' unique needs cannot be overstated, particularly in a large, public university setting where diverse student populations converge. The challenges faced by these students are multifaceted, encompassing academic, social, and emotional domains, which often influence their persistence and success in higher education (Tinto, 2017). This report elucidates a technology-enhanced plan tailored to uphold the success of first-generation students, underpinning their journey from admission to graduation, and potentially influencing their post-collegiate pathways.
Scope of the Report
This report critically engages with the development and implementation of a student success plan, with a keen focus on leveraging technology to facilitate supportive ecosystems for first-year, first-generation college students. Recognizing the intersectionality that often accompanies the first-generation status - such as socio-economic, racial, and ethnic aspects - the plan aims to be inclusive and adaptable, thus enabling a wide spectrum of students to benefit from the envisaged interventions.
Key Challenges Faced by First-Generation College Students
The unique challenges confronted by first-generation students range from navigating the academic rigor, understanding and accessing institutional resources, to forming social connections within the university community (Stebleton & Soria, 2012). Furthermore, first-generation students often grapple with “imposter syndrome”, financial strain, and balancing familial responsibilities alongside their academic commitments. These challenges, if unaddressed, could potentially impede their academic progression and overall well-being during their collegiate tenure.
The Demographic Profile of First-Generation College Students
First-generation college students represent a heterogeneous group, often encompassing varied ethnic, socio-economic, and cultural demographics. Predominantly, these students are likely to emanate from low-income families, and statistically, they exhibit higher representations from minority ethnic groups (Covarrubias & Fryberg, 2015). Their divergence in experiences and backgrounds requires a nuanced understanding and approach to developing supportive infrastructures that honor their individual and collective needs within the higher education context.
The Importance of Supporting First-Generation Students
Supporting first-generation students is not merely a benevolent act but an imperative that aligns with the principles of equity and inclusiveness in higher education. Their success holds socioeconomic implications, often providing upward mobility and enabling them to contribute positively to their communities and the broader society (Yee, 2016). Furthermore, their presence and success within universities enrich the academic and social environment, fostering a diverse and inclusive campus culture that is reflective of the multifaceted society beyond the university walls.
Current Challenges and Gaps in Existing Support Mechanisms
Despite the increased recognition of the unique challenges faced by first-generation students, there exist evident gaps in existing support mechanisms, especially in large public universities where individualized support might be diluted due to the vast student population. The challenges include:
Importance of Technology in Higher Education
The advent and advancement of technology have reshaped the landscape of higher education, offering novel pathways to enhance student learning, engagement, and success. Technology, when thoughtfully integrated, has the potential to democratize access to resources, facilitate personalized learning experiences, and bridge the informational and resource gap that is often experienced by first-generation students (Bowers & Kumar, 2015).
Definition and Importance of a Critically Engaged Practice Approach
A critically engaged practice approach underscores the symbiotic interplay between theory and practice, aiming to embed scholarly insights into pragmatic, context-specific strategies to enhance first-generation student success. It involves a meticulous understanding, evaluation, and application of existing research to inform the development and implementation of initiatives, ensuring they are grounded in empirical evidence and tailored to the specific needs and challenges of the target demographic (Tinto, 2017). This approach is pivotal in enabling interventions to be not only well-informed but also adaptive and responsive to the evolving needs and challenges encountered by first-generation students throughout their academic journey.
Previous Implementations and Outcomes
Historically, the critically engaged practice approach has proven effective in various contexts, particularly in enhancing student retention, engagement, and success in higher education. For instance, initiatives such as learning communities, peer mentorship programs, and culturally responsive pedagogy, informed by a critically engaged practice, have demonstrated notable impacts on student outcomes (Yee, 2016). However, the integration of technology to augment these initiatives, especially for first-generation students, warrants further exploration and contextualization within the specific milieu of a large, public university.
Relevance to First-Generation Students’ Success
The application of a critically engaged practice approach to first-generation student success is particularly salient given the multifaceted challenges these students encounter. By intertwining rigorous academic research with practical, on-the-ground insights, initiatives can be developed to address the academic, socio-economic, and psychological challenges peculiar to first-generation students, thereby fostering an environment that not only supports their immediate needs but also fortifies their capacity to navigate through subsequent challenges in their academic trajectory (Tinto, 2017). Further, this approach ensures that the developed interventions are not static but are continuously refined based on ongoing research and feedback, ensuring sustained relevance and impact.
Incorporating Adaptive Learning Technologies
Development and Utilization of Online Mentoring and Counseling Platforms
Implementing a Virtual Community-Building Framework
Employing a Technology-Based Early Alert System
Facilitating Digital Literacy and Competency Workshops
Engage with Stakeholders Authentically
Involving all pertinent stakeholders in the implementation of the technology-enhanced student success plan is paramount to its efficacy and sustainability. Engaging faculty, students, IT staff, and administrators ensures that the plan is not only robust and comprehensive but also adaptive to the specific needs and dynamics of the university environment.
Ensure Equity in Access and Engagement
Ensuring that all students, particularly those from marginalized or underserved demographics, have equitable access to and engagement with the technological resources and platforms is paramount. This involves providing necessary resources, such as devices and internet access, and ensuring that the platforms are accessible and inclusive, catering to the diverse needs and contexts of the students (Hsieh, Sullivan, & Guerra, 2007).
The variance in technological proficiency among faculty and students could pose a significant barrier to the effective implementation and utilization of the technology-enhanced interventions. Ensuring that all users can adeptly navigate and maximize the technological platforms and resources requires not only comprehensive training but also ongoing support to address challenges and issues that may arise during the implementation phase. Moreover, the digital divide is not merely access-oriented but also encompasses disparities in digital literacy and the quality of digital engagement, which must be meticulously addressed to ensure true equity in technology-enhanced initiatives (Bowers & Kumar, 2015).
Continuous Assessment and Refinement
Implement continuous assessment and feedback mechanisms to ensure that the plan is not static but is continuously refined and adapted based on empirical data and user feedback. Engage with both quantitative and qualitative data to glean comprehensive insights into the impact and areas for improvement within the plan.This ensures that the plan is not static but is continuously refined and adapted based on empirical data and user feedback to enhance its relevance and impact over time (Covarrubias & Fryberg, 2015).
Be Mindful of Potential Resistance
Resistance to technological adoption, especially among faculty, should be approached with empathy and support. Providing comprehensive training, ongoing support, and platforms for feedback and engagement will be crucial in navigating through potential resistance and ensuring effective integration within academic practices.
Collaborate Across Departments
Interdepartmental collaboration enhances the holistic impact and sustainability of the plan. Ensuring that the plan is not siloed within specific departments but is collaboratively engaged with across various academic and administrative units enhances its reach, impact, and alignment with broader institutional goals and priorities.
Adapt to Institutional Contexts
While the core principles and strategies of the plan are transferable, it is crucial to adapt them to the specific contexts, needs, and dynamics of your institution. Engage critically with the plan and ensure that its implementation is tailored to the unique characteristics and challenges of your institution.
Advocate for Financial and Administrative Support
Securing and sustaining financial and administrative support for the plan will be pivotal for its sustained impact and success. Ensure that the plan is aligned with institutional priorities and advocate for its importance and potential impact on student success and institutional effectiveness to garner necessary support. From procuring and maintaining technological platforms to training stakeholders and ensuring equitable access to resources among students, financial constraints could potentially impede the full realization of the plan’s envisioned impact. For instance, providing devices and internet access to all first-generation students who may lack these resources could strain limited university budgets.
Closing Note and Reflection
While the challenges and limitations outlined could potentially impede the seamless implementation and efficacy of the plan, they also provide valuable insights that can inform the refinement and adaptation of strategies to mitigate these challenges. Thus, acknowledging and critically engaging with these challenges and limitations is not merely a procedural necessity but an imperative to ensure that the plan is not only robust and comprehensive but also adaptable and resilient amidst the dynamic and multifaceted challenges encountered in its implementation.
Implementing a technology-enhanced student success plan in different college contexts, while holding the potential to significantly enhance first-generation student outcomes, necessitates a critical, adaptive, and collaborative approach. While the strategies and insights provided herein provide a foundational framework, it is the meticulous adaptation, implementation, and continuous refinement of these strategies within your specific institutional context that will determine their impact and success.
References
Engaging in this process with GenAI by capturing and documenting our experience showed that even early generations of LLMs have the capacity to facilitate, but not replace, the critical discourse between scholarly practitioners. We found it useful that the tool recognized the conventions of academic writing and articulated the reasoning for design choices. ChatGPT-4 also made some notable connections from the literature. Present was a robust list of ideas that are beneficial across student groups including, a multi-domain view supporting academic achievement with social-emotional well-being and critical thinking, the need for digital literacy support, and connecting individual efforts with resources and expertise (Bates & Poole, 2003; van der Zanden, 2019). Interestingly, the report also referenced existing programs (i.e., Purdue’s Course Signals) to provide real-world applications of the student success plan interventions.
We chose the paid version of ChatGPT due to its superior capacity for academic prose, ability to articulate complex ideas, and apparent facilitation of more profound analysis and interconnections. However, these advantages were still found to be insufficient. The report was awash with obvious fabrication, under-explored content, unexplained concepts, poorly cited references, and assertions that do not align with the evidence. While the report referenced existing programs, ChatGPT mischaracterized them by not delineating first-generation students in their study population. It also completely fabricated some of its applications (e.g., the Digital Natives program). Though the ideas generated were adequate to share with stakeholders as a starting point, they lacked the depth and critical lens to provide actionable practitioner insights.
Absent also were the connections to literature and knowledge regarding critically engaged practice (CEP). ChatGPT did not describe how CEP is grounded in critical theories (Winkle-Wagner et al., 2018), nor did it recognize the influence of social structures, legal frameworks, and economic policies on practice (Muruthi et al., 2023; Terrazas et al., 2020). While ChatGPT did acknowledge the intersectionality of other identities to first-generation student status, it did not further explore how those identities impacted the proposed technology-enhanced student success plan. Rather than recognizing the diversity that first-generation students bring to their education as assets, including culture, language, disability, and socio-economic status, the generated report was framed with a deficit lens using negative language and stereotypes (Wildhagen, 2015). This is possibly because past research on first-generation college students, likely present in the LLM’s training data, has taken this perspective.
Additionally, we hoped for more critical reflection from ChatGPT-4 on how the interventions connected students to their faculty and peers, which positively impacts student success (Astin, 1993; Chen et al., 2008). Several ideas were reliant on access to technology, time and financial resources, and institutional support mechanisms. When a challenge was presented (e.g., addressing resistance to technology), little was said to address the challenge with viable practitioner solutions (e.g., getting buy-in, positive student outcomes). Despite our prompt specifying a report in a large, public university context, the output failed to incorporate references to the large university setting or provide insights into the contextual relevance of certain strategies.
It took us approximately 20 total hours through this process to produce the short report. At first thought, the use of GenAI seemed to reduce the time spent to create such a report. However, as we think this report is insufficient, we would likely have spent more time refining this draft before submission. This would have included connecting concepts and discussion of ideas to specific (non-hallucinated) literature and adding practitioner-based examples and insights not accessible to the LLM, but only through our experience in the field. Additionally, there are ways we would have modified our report process, but those would have taken us out of the guidelines for this submission. These include uploading specific texts to inform the LLM’s approach, distinguishing aspects of the draft that would be better written by the authors themselves, word count edits, and adjusting its framing on certain subjects.
This process of creating a prompt and critical reflection was an opportunity for us to explore meaningful ways to employ GenAI tools and to enhance our skills to leverage GenAI. There are several implications for higher education practitioners looking to utilize GenAI to help design initiatives like student success programs. First and foremost, it is crucial to recognize that GenAI can be used as a valuable tool to formulate a wide range of program development or teaching and learning ideas. However, at this moment GenAI cannot reliably apply a deep critical lens. It cannot provide the real-world experience, perception, or discernment required for creatively designing solutions to a unique and evolving campus environment. Staff and administrators should consider utilizing GenAI partnered with their human expertise to create learning environments and holistic experiences that help students learn and thrive. It is still incumbent on us as professionals to leverage our expertise to recognize, evaluate, and develop those ideas for our campuses and communities.
While this experiment produced an insufficient result in terms of fabrication, under-explored content and contextualization, poorly cited references, and assertions that do not align with the evidence, there is substantial value in the use of technology in academic writing. LLM's ability to rapidly produce academic-sounding prose and structure, its ability to refine and adapt with pointed feedback, and its ability to represent patterns in a particular discourse could make it an effective companion for academic writing. Contrasting this experience with a typical academic writing experience, the authors spent more time critically analyzing what it produced as subject matter experts. We were more capable of noticing patterns in the writing, making connections to existing literature, thinking of ways to improve the piece, and recognizing the appropriate insight, context, or scope in the short report. If used correctly, it can seemingly place a scholar in a more critically evaluative position. However, if used without proper attention and context, we also see the potential for uncritically accepting the framing, language, descriptions, and conclusions of GenAI to potentially introduce harmful discourse, reproduce inherent bias from existing literature, and miss opportunities for novel insight that could hamper academic discourse.
Teaching and learning thrive when expert practitioners share their observational insights and critical perspectives. If LLMs homogenize and obscure the diverse experiences of learners and those that support learners, our practice and collective knowledge will atrophy. This work has helped us recognize an opportunity to utilize GenAI to complement our experiences, knowledge, and expertise and become even more impactful as scholars and educators.
Astin, A. (1993). What matters in college?: four critical years revisited (1st ed.). Jossey-Bass.
Bates, T., & Poole, G. (2003). Effective teaching with technology in higher education: Foundations for success. Jossey-Bass.
Burgette, J. E., & Magun-Jackson, S. (2008). Freshman orientation, persistence, and achievement: A longitudinal analysis. Journal of College Student Retention: Research, Theory & Practice, 10(3), 235–263. https://doi.org/10.2190/CS.10.3.a
Chen, H. L., Lattuca, L. R., & Hamilton, E. R. (2008). Conceptualizing engagement: Contributions of faculty to student engagement in engineering. Journal of Engineering Education, 97(3), 339-353. https://doi.org/10.1002/j.2168-9830.2008.tb00983.x
Clark, M. R. (2005). Negotiating the freshman year: Challenges and strategies among first-year college students. Journal of College Student Development, 46(3), 296-316. https://doi.org/10.1353/csd.2005.0022
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Wright, R., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A. S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., & Buhalis, D. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
Facts and figures. (n.d.). Arizona State University. Retrieved October 13, 2023, from https://www.asu.edu/about/facts-and-figures
Graham, M. J., Frederick, J., Byars-Winston, A., Hunter, A. B., & Handelsman, J. (2013). Increasing persistence of college students in STEM. Science, 341(6153), 1455-1456. https://doi.org/10.1126/science.1240487
Hamilton, I. (2023, June 13). 56% of all undergraduates are first-generation college students. Forbes. https://www.forbes.com/advisor/education/first-generation-college-students-by-state/
Muruthi, B. A., Thompson Cañas, R. E., Zárate, J., Lahoti, A., Romero, L., & Stafford McRell, A. (2023). Policies during COVID-19 and their impact on immigrant families’ psychological well-being: A critically engaged practice. Families, Systems, & Health, 41(2), 265. https://doi.org/0.1037/fsh0000808
OpenAI. (2023). ChatGPT (October 2023 version) [Large language model]. https://chat.openai.com/share/6956f83c-9260-4559-ba0d-1df499cfca16
Qadir, J. (2023, May). Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. In 2023 IEEE Global Engineering Education Conference (EDUCON) (pp. 1-9). IEEE. https://doi.org/10.36227/techrxiv.21789434.v1
Ray, P. P. (2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems. https://doi.org/10.1016/j.iotcps.2023.04.003
van der Zanden, P. J., Denessen, E., Cillessen, A. H., & Meijer, P. C. (2019). Patterns of success: first-year student success in multiple domains. Studies in Higher Education, 44(11), 2081-2095, https://doi.org/10.1080/03075079.2018.1493097
Wang, C., Liu, X., Yue, Y., Tang, X., Zhang, T., Jiayang, C., Yao, Y., Gao, W., Hu, X., Qi, Z. & Wang, Y. (2023). Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. https://doi.org/10.48550/arXiv.2310.07521
Wildhagen, T. (2015). “Not your typical student”: The social construction of the “first-generation” college student. Qualitative Sociology, 38, 285-303. https://doi.org/10.1007/s11133-015-9308-1
Winkle-Wagner, R., Lee-Johnson, J., & Gaskew, A. N. (Eds.). (2018). Critical theory and qualitative data analysis in education. Taylor & Francis.