Can Law Students Develop the Expertise Necessary to Supervise Legal Chatbots … by Using Legal Chatbots?

Casetext is marketing their new GPT-based tool CoCounsel as the “first AI legal assistant” and “the AI legal assistant for every practitioner.” Attorneys are required by rule 5.3 of the ABA Model Rules of Professional Conduct to supervise “non-lawyer assistants” which includes AI-based technology like CoCounsel. See, e.g., Amy B. Cyphert, A Human Being Wrote This Law Review Article: GPT-3 and the Practice of Law, 55 UC Davis Law Review 401 (2021). It makes sense that experienced attorneys adopting a tool like CoCounsel can review AI-generated output for errors and quality and make adjustments. Such AI technologies very likely will prove to be useful time-savers in legal practice in building the basic skeletons of routine legal documents.

But what about law students and early career lawyers? Can they learn their craft and gain the ability to competently supervise an AI legal assistant by skipping the old-fashioned research and writing of memos and briefs and relying on an AI-generated draft to speed up the process? This question goes directly to the process of how human beings develop domain expertise.

Expertise is the ability to make an appropriate response in an unpredictable situation. Expertise also requires innovation, adaptability, and flexibility. Deliberate, thoughtful practice has been considered to be the prerequisite of expertise for 30 years though recent research has suggested that other factors are involved that may be domain-dependent. Expertise, however, involves developing more complex and nuanced mental models and pushing oneself past a basic level of understanding. Yujin Kim, The Changing Concepts of Expertise and Expertise Development, in Expertise at Work (2021).

“The journey to truly superior performance is neither for the faint of heart nor for the impatient. The development of genuine expertise requires struggle, sacrifice, and honest, often painful self-assessment. There are no shortcuts.” Ericsson et al., The Making of an Expert, Harvard Business Review (2007). 

A recent study of the use of technology in teaching music composition to high school students in the UK found that teachers often used the technology as a “short cut” to make up for time pressures in the curriculum and to help students who they believed had “low ability.” The teachers encouraged students to begin their compositions on an instrument to avoid and avoid “over-reliance” on the technology in order to avoid producing “unimaginative and quite un-human” work. Another teacher expressed their concern that some students “can become controlled and driven by the computer program.” Kirsty Devaney, ‘Waiting for the Wow Factor’: Perspectives on Computer Technology in Classroom Composing, 12 Journal of Music, Technology & Education 121 (2020). 

To become legal experts (and good writers), law students and new attorneys need to go beyond fact-checking AI-generated briefs. But that doesn’t mean generative legal AI should have no role in teaching legal research and writing, but that role must be carefully thought out. Law students who have access will probably try it out on their own anyway. Using examples of excellent writing and research is a fabulous pedagogical tool, but AI-generated content does not count as “excellent writing,” at least not yet, and maybe not ever. It will never have real heart—just see the recent blowup at Vanderbilt over a ChatGPT-drafted email in response to a shooting.

During a recent visit to our law school, legal technology expert and all-around awesome person Bob Ambrogi said that he is concerned that generative AI may stifle lawyer creativity.  He provided the example of an attorney drafting an off-the-wall brief as a creative exercise to test new legal arguments and theories before refining that brief into the actual document to be filed with the court. An attorney who starts with AI instead will be “controlled by the computer program” and may miss a creative, winning argument and perhaps an opportunity to develop the law in a new way that will benefit society.

Law students want to make a difference. Let’s help them develop skills and creativity, not turn them into zombies.

Posted in Artificial intelligence, Legal Research Instruction, Legal Writing, Uncategorized | Tagged , , , | Leave a comment

Oh No, Another ChatGPT Post: Incorporating AI-Powered Chatbots into Legal Research Exercises and Assignments

Since it was launched at the end of November 2022, the discourse around ChatGPT and AI search tools has been unrelenting. What impact will AI-powered chatbots have on education? Will students submit ChatGPT-written essays and homework assignments? Will AI make lawyers obsolete? Look, this chatbot just passed the bar exam! Wait a minute—is this thing. . . sentient?

Despite its release barely four months ago, a flurry of scholarship on ChatGPT and legal education has already appeared on SSRN: Jonathan Choi, Kristin Hickman, Amy Monahan, and Daniel Schwarcz’s ChatGPT Goes to Law School (Jan. 25, 2023); Lea Bishop’s A Computer Wrote this Paper: What ChatGPT Means for Education, Research, and Writing (Jan. 26, 2023); Tammy Pettinato Oltz’s ChatGPT, Professor of Law (Feb. 4, 2023); Jennifer Murphy Romig’s The Ethics of ChatGPT: A Legal Writing and Ethics Professor’s Perspective (Feb. 18, 2023); and Joseph Regalia’s ChatGPT and Legal Writing: The Perfect Union? (Feb. 26, 2023), to name a few.

Law librarians on the RIPS Blog and elsewhere have joined the conversation, highlighting the need to teach about this technology in legal research courses as an extension of critical information literacy and algorithmic skepticism instruction.

In addition to class discussions about ChatGPT and other AI chatbots, it seems to be time to incorporate AI-assisted research and drafting into our courses. But how?

Right now, chatbot AI isn’t necessarily the best when it comes to facts: when asked to provide court opinions as part of a legal research prompt, ChatGPT sometimes provides citations to non-existent—made-up—cases. Sometimes it is easy enough to catch ChatGPT in a lie, like when it told SCOTUSBlog that Justice Ruth Bader Ginsburg wrote the dissent in Obergefell v. Hodges. But when researching more discrete, less well-known topics, a chatbot user might not be aware of any inaccuracies. This can be leveraged in a legal research exercise or assignment by asking students to edit and confirm the accuracy of a ChatGPT-generated legal analysis or document (a la history professor Jonathan S. Jones’s exercise discussed on Twitter, and Legal writing professor Dyane O’Leary’s similarly structured exercise where students compare their own research and writing to that of the chatbot).

Professor Andrew M. Perlman’s The Implications of ChatGPT for Legal Services and Society article from late last year includes a ChatGPT-drafted legal complaint seeking damages after a car accident and a ChatGPT-written brief to submit to the Supreme Court regarding same-sex marriage, among other ChatGPT-written examples. Note that none of the AI-written language include legal citations. This is where a legal research exercise could begin: read the complaint or brief and research both to confirm accuracy and to provide citations to legal authority. This exercise gives students the chance to practice researching while learning how ChatGPT functions by interacting with it directly.

I’m considering an exercise or assignment like this over the summer intersession, and I’m curious if others have or are planning to do something similar. Are there other ways to incorporate chatbot AI into the course materials of legal research classes?  

Posted in Information Literacy, Legal Research, Legal Research Instruction, Legal Technology, Legal Writing | Tagged , , , , , , , , | 2 Comments

Chat GPT-4 Aced the Bar Exam and the Bing Chat Waitlist is No More: Did the Future of Legal Research Just Arrive Today to Make Us Fact Checkers?

By Sarah Gotschall and ChatGPT-4*

Does today mark an important milestone in the world of AI legal research as Bing Chat, powered by GPT-4 technology, just became available to all without the need to languish for weeks on a waiting list? This powerful tool combines the large language model technology of ChatGPT-4 with the ability to access current information on the internet, having the potential to quickly transform the legal research landscape and the way we approach legal research and writing.

Insights from the ALWD Writing Professors from Tuesday

On Tuesday, March 14, which was two days ago, I watched an ALWD writing professors presentation about the future of AI and writing. Many on the panel shared their view that, as AI continues to improve, legal professionals will eventually transition from being writers to editors and fact-checkers. Even as AI-generated text is increasingly well-structured and coherent, it will still require human oversight to verify accuracy and ensure it aligns with the context and specific circumstances of each case. The presenters theorized that continual improvement of these systems will likely lead to a greater emphasis on teaching critical thinking, analytical skills, and research skills to fact check and find additional legal materials missed by AI. This thought had occurred to me, though I hadn’t exactly thought of legal research in the AI era as factchecking, and I wondered how far in the future this might be. Just weeks ago, I wrote a blog post welcoming the day that our AI overlords ease the research and writing burdens on humans to make law less of a jealous mistress…

Significant Leap Forward That Same Day…

Well, it arrived sooner than I anticipated, because on that same day, I discovered that the new version of ChatGPT, version 4, was now available for premium ChatGPT subscribers (of which I am one). Also, I read that ChatGPT-4 had leapfrogged the legal analysis and writing capabilities of GPT-3.5 by acing the LSAT and bar exam. During my 1L legal research class two weeks ago, I introduced my students to ChatGPT and had them sign up to answer some questions on their research exercise. At that time, ChatGPT produced a fairly good answer to a fact pattern, but all the cases were hallucinated and didn’t actually exist. Now, a mere fortnight later, GPT-4 demonstrated a significant improvement by accurately citing actual cases relevant to the fact pattern. Bing Chat is also powered by GPT-4, and I had already been impressed with its legal research capabilities. Having eagerly joined the waiting list for Bing when I first heard about its potential, I have had access to the platform for weeks now. However, two weeks ago, during class, the students couldn’t access it due to the waiting list. Consequently, I couldn’t incorporate Bing-specific questions in their exercise, so instead, I demonstrated its capabilities in class, highlighting its potential to make legal research easier and stressing the importance of staying current with emerging technologies. I was excited about GPT-4, but didn’t expect if to become widely available so soon.

The Future of Legal Assignments

The next day, for the students’ final assignment, I told them they could use any resources they desired, with the exception of seeking help from other people. Just as they would use Google in the past, I told them to feel free to use ChatGPT or Bing, if they had access to it. During the dinosaur days of the previous day, I didn’t think to ChatGPT-4 or Bing-proof the assignment, as I didn’t expect the Bing platform to suddenly be available to all, and I assumed most students wouldn’t sign up for the premium version of ChatGPT, because they didn’t seem that impressed two weeks ago.

Now today, with Bing’s release, I can’t help but feel a little apprehensive about whether the assignment has become significantly easier for the students. It will be interesting to see the impact of Bing’s accessibility on their performance and the quality of their work!


In conclusion, the rapid advancement and accessibility of AI-powered tools like Bing Chat, powered by GPT-4 technology, has the potential to revolutionize the field of legal research and writing. With insights from the ALWD writing professors suggesting a shift towards fact-checking and editing roles, the recent progress demonstrated by GPT-4 and Bing Chat is a testament to how quickly AI can evolve and improve. As educators and legal professionals, we must stay current with emerging technologies and adapt our methods accordingly. The future of legal assignments may indeed change, but the importance of analytical skills and adaptability will remain a cornerstone of the profession. As Bing becomes more widely available and integrated into legal research, we must remain vigilant in our pursuit of accuracy, ensuring AI-generated content aligns with the specific context and circumstances of each case. Only time will tell how these advancements will impact students’ performance and the quality of their work, but one thing is certain: the future of legal research and writing is evolving, and we must be prepared to evolve with it.

*Thanks to ChatGPT for contributing to my blog post!

Posted in Uncategorized | Tagged , , , | 1 Comment

Strategies for Dealing with Difficult People

Laughter is one strategy for dealing with difficult people. 

For instance, it was humor that probably saved me the time a patron came to the desk, stood in front of me, and asked, “Can I speak to someone smarter than you?”

Granted, some individuals might feel insulted by such a question, but I was amused.  Waving my hands in the general direction of people walking by, I said, “Take your pick.  It’s a university.” 

The patron had the grace to appear embarrassed standing at the reference desk, in search of a smarter librarian, not me, but the reference desk is where you go for help.  “Just ask.” It’s our policy, and in all our library marketing material.

The patron was looking for someone he trusted. Rather quickly, I realized that it was my female-powered brain that was interfering with his ability to get legal research help.  So, I set him up with a male colleague.  And voilà, problem patron no more. 

Why didn’t I react in anger?  Because my ego is not attached to what strangers think about my intelligence.  I was free to go with the flow.  He wanted a man’s help.  I knew a few.

Sometimes as a public service librarian, I deal with difficult people.  Perhaps you have also.  Often the problems are minor, like the story I just told you, occasionally the problem person is complicated, and their issue can’t be resolved easily.

Recently, I had an unpleasant experience with an unhoused community member who complained because I allowed children to run around the library. Unsupervised, in front of law students!   

A week earlier, the woman complained about my work performance, after sending an email to the reference email service, asking staff to pull a list of cases and books for her.  Her email said she would pick the material up at nine in the morning. I replied that I would be happy to help her find the cases, but she would have to pull and copy them.  She was so affronted that she then sent a strongly worded email to my supervisor about my lack of professionalism.  She was sorry that she had to report me, but it was for my own good, so my supervisor could correctly teach me how to respond promptly to requests. It was a hilariously harrumph-y email.  Again, my sense of humor gave me perspective.

 Defining a problem or difficult person is a challenge because emotional reactions can be influenced by a variety of factors such as past experiences, familiarity with the individual, training and education, where we grew up, even the support we have from our community. Unfortunately, studies show that even one aggressive encounter at a service desk affects performance and work productivity. Having a strategy for dealing with a difficult person may protect our emotional wellbeing, increase our ability to perform work tasks, as well as give us additional health benefits.

About that study on work productivity and aggressive behavior, the research suggests that the brain processes the perceived threat first, while work tasks become secondary.  I love these studies. If someone approached the reference desk with a baseball bat, finishing up that powerpoint you are working on is probably not going to be your initial reaction. You will address the threat.

The study also suggested that if the exchange triggers a stress response in an employee, it can also lead to burnout, depression, and absenteeism.   Even in less emotionally charged responses, conflict with others can cause rumination, focusing our energy on thoughts about what we could have done better, or what should we do next time. The time spend wondering about our feelings and reactions to the situation becomes lost productivity and wasted energy.  As Elsa sings, “Let it go.” 

Strategies for Dealing with Difficult People

First, let’s acknowledge something happening to librarians across the country. Many in our profession are exhausted and when it comes to dealing with others, exhausted people have less emotional reserve to deal with a problem, and the problem after that.  So, my first suggested strategy is to take a break.  I realize this is a cliche, but as the stewards in the airplane tell us, put your mask on first, before helping the person next to you.


Take a break. Breathe deep the gathering gloom. I believe I aged myself with those lyrics. Anyway, give yourself a breather. I used to chase lawyers out of my office if I realized they were too tired to speak.

Change your perspective: According to the author of Zen and the Art of Dealing with the Difficult Patron, Louisa Toot, “Half the battle of dealing with a difficult patron is one’s own perspective.” Even if you don’t practice Zen Buddhism, changing how you perceive a person is useful.  Is the person “difficult” or “Do they have a problem?”   In another article I read, the author compared library literature to nursing literature. The author used the health literature to create a new framework for thinking about the problem patron.  Nurses frame the issue of dealing with people differently and the author suggested librarians could benefit from doing the same to change the dynamics of how we perceive patron challenges.

Listen: Ask questions and rephrase what you’ve heard, tell the person you are trying to understand if they appear frustrated.  Listening is especially useful when someone isn’t clear. I once worked at a nursing home, one of the residents was a woman with aphasia, the loss of the ability to speak. I learned how to communicate with her through a series of Q &A, because I realized although she might not be able to speak as clearly as others, she could nod and shake her head, move her body. We used the tools that were under her control, and it saved both of us frustration. Her nurse and family asked me to teach them my trick. I listened, and I watched her body language for clues to how things were going.

Trust: Developing trust is an important factor in communicating with others.  I once calmed a patron who threatened another librarian by using his trust in me to help him gain control of his emotions. The patron was shouting, waving his arms, scaring everyone else in the library. I reminded him that he knew me and that I had helped before and asked, “Could I help you again?”  It worked.  He stopped ranting and eventually calmed down.  I don’t know why he lost control, but I knew that under normal circumstances, he was a nice man.

Respect: Show others respect. We all deserve to be respected. I say please, thank you, and address people formally when appropriate. But respect goes both ways, and that means you deserve it too.  You don’t have to accept verbal abuse because you are in a service profession.  You always have the right to walk away from a situation, to tell someone no or to ask them to leave unless they can control their behavior.  The customer isn’t always right.

Stay Calm:  Sometimes, when someone is angry, I think about how a toddler behaves when having a meltdown.  I don’t blame the toddler or myself for the loss of emotional control.  Aim for calm and if your emotions get the best of you, that’s fine too.  It takes practice.  Once an attorney sat in my office squeezing a stress ball as he gave me a request, and the ball broke in his hand.  We both stared at it, amazed. I believe he may have been stressed. Emotional control is hard work. 

Humor: I told you laughter is one of my favorite strategies.  Studies show humor may be effective in emotional regulation and it has been my sense of humor that has influenced my interactions with students, faculty, and my community. I am a big fan of laughter. I believe that it is healing. I admire comedians and I give funny books to friends and family when they are sick or injured. I act as if laughter is part of the recovery process.  If you are stressed by an angry patron, and going for a jog or a walk isn’t possible, try lessening the tension in your body with laughter. Can’t find anything funny about the interaction even after it is over and the problem gone? Still feeling shaken? If you are in that dark place, try lightening the mood, even if it is only to spend a few distracting minutes watching a quick video on whatever amuses you, before you return to your normal tasks. It might slow productivity down, but it will also slow your heartbeat, and that makes the time well spent.

List of articles on the topic of dealing with difficult people.

James Gross is an emotional researcher who has published books and articles on the Model Process of Emotional Regulation. Emotional regulation is the study of what influences our emotions. Gross, J. J. (1998a). The emerging field of emotion regulation: an integrative review. Review of General Psychology, 2(3), 271-99. 

Rafaeli, Anat, Amir Erez, Shy Ravid, Rellie Derfler-Rozin, Dorit Efrat Treister, and Ravit Scheyer. 2012. When Customers Exhibit Verbal Aggression, Employees Pay Cognitive Costs. Journal of Applied Psychology 97 (5): 931–50. doi:10.1037/a0028559.

Toot, Louisa. 2002. Zen and the Art of Dealing with the Difficult Person, The Reference Librarian, 36, 75-76, 217-233.

Ferrell, Shelley, Who Says There’s a Problem? A New Way to Approach the Issue of “Problem Patrons,” Reference & User Services Quarterly, 2010. vol. 50, no 2, pp. 141-51.

Posted in Uncategorized | Tagged , , , , , | 4 Comments

Legal Research for the People

As research and reference librarians, we all need to take our audience into consideration when delivering written information. Academic librarians may need to provide more context and explanation when writing a reference email to a first-year law student than they would in an email to a faculty member. Likewise, law firm librarians must tailor their communications depending on whether they are instructing a summer associate or assisting a seasoned attorney.

Providing information to the general public poses an even greater communication challenge. In fact, AALL has an entire special interest section dedicated to it, LISP-SIS. I have never worked in a law library that is open to the public, so this mode of communication was new to me when I joined the Social Responsibilities Committee of my local AALL chapter, the Greater Philadelphia Law Library Association (GPLLA).

Legal research can be daunting for those who are unfamiliar with the legal system. Pennsylvania law in particular contains many quirks and traps for the unwary. So GPLLA’s Social Responsibilities Committee perceived a need for a guide to Pennsylvania legal research that would be accessible to laypersons and to generalist public librarians who receive questions about legal information.

If you are considering a similar project, you should start by surveying the resources that are already available. LISP-SIS created How to Research a Legal Problem: A Guide for Non-Lawyers. There is also Starting Points for Researching a Legal Issue Using Print or Digital Resources, a pamphlet geared towards incarcerated library patrons created by AALL’s Social Responsibilities Special Interest Section (SR-SIS). These resources are instructive examples of what kinds of resources to include and how to present information in an accessible manner.

To break down legal jargon, we peppered our guide with links to Cornell Legal Information Institute’s free legal dictionary, Wex. We linked, wherever possible, to free sources of legal information, especially from official government sources. We also tried not to assume any familiarity with the structure of the federal or state government, so we included an overview of the United States legal system to introduce readers to the basic concepts of federalism and separation of powers that are necessary to understand where to find different types of primary legal sources. Members of the Social Responsibilities Committee each volunteered to write one chapter of the guide, then we used Google Docs to comment on each others’ drafts.

After many months of planning, drafting, and editing, we launched Pennsylvania Legal Research 101: About the Law and How to Find It. To promote it and gather feedback to improve the guide, we held a free webinar that was open to anyone who wanted to attend. (A recording of our presentation is available here.) My co-authors and I also appreciate any feedback from RIPS Law Librarian readers, so please sound off in the comments or email us at

In a perfect world, everyone would have access to competent counsel to answer their legal questions. But in our imperfect legal system, there will probably always be those who must resort to self-help. Making the law more accessible to everyone also has the benefit of educating Americans about the working of our democracy. As law librarians, we should all consider how we can help further the cause of access to justice. There is much work to be done.

Posted in Access to Justice | Tagged , , , , | Leave a comment