by Christine Anne George
I do not often compare myself to Charlie Brown, but when it comes to citation metrics, I am Charlie Brown with the football. Why is that? Because there is no perfect way to collect citation metrics. Honestly, I have sincere doubts whether traditional citation metrics should be the focus at all, but that’s not the football I’ll opine upon today. Tomorrow, maybe, but today it’s about the latest Lucy—Hein.
There is a lot to be said for the Hein Author Profiles. They provide metrics within the Hein universe. There’s no set standard of what number of citations is a “good” number, so it seems to work. The downside of the Profiles (beyond being limited to the Hein universe) is that the onus is on the author—or the author’s librarian—to do the necessary clean up. Name variations require multiple profiles to be combined. The unfortunate circumstance of authors having the same or similar names just mean that the author/librarian had to parse through the Profile to ensure that what was listed was accurate. Overall I thought that the Profiles were worthwhile to prepare for my faculty, and it was something that stayed on my to do list to get to…you know, one of these days. Then on February 13, 2019 my priorities list experienced a massive reshuffle when U.S. News announced that they were going to be evaluating scholarly impact using Hein. If we were in a Peanuts strip, there would have been a very abrupt record scratch at that moment.
Straightening up the Profiles seemed like a straightforward enough task. (Sidebar: I only dealt with Hein. I was not the point of contact for U.S. News.) Hein had a spreadsheet. I had to provide the name variations and a few other pieces of information (faculty profile links, twitter handles, etc.) to enhance the profile. I noticed that there were three separate name listings for my institution, so I asked Hein to take care of that as well.
Theoretically, that’s where it should have ended. I received confirmation from Hein that the profiles were updated and that they were all listed under a single institution. Done. Except that I wanted to be sure. Really, really sure. So I coordinated with the Dean’s Office to get updated CVs for all our faculty. Then my amazing research assistant, Johnny, took each of those CVs and went through the faculty’s newly enhanced Profiles. I asked him to let me know of any publications that appeared on Hein but not the CV, and any law review/journal articles that appeared on the CV but not the Profile. I hadn’t thought that there’d be many of the latter, so I asked him that if there was an article that was on the CV, but not the Profile, to check on Hein to see if it was available at all. And that, dear readers, is the first time the football was pulled away.
There were articles on Hein that weren’t linked to the faculty’s Profile because of a typo in either their first or last name on Hein. There were articles from symposia that didn’t have traditional bylines, and instead either listed the authors in a footnote or partway through the article. There were articles and book reviews that weren’t indexed by Hein at all, though they were available if you navigated to the starting page within the publication. We also found a few profiles that had been enhanced, but were empty of the articles and therefore cites.
There were emails back and forth to Hein and all of the edits, corrections, whatever you’d like to call them, were addressed. I have to give Hein credit for being extremely responsive to any issue that arose. All was right with the world. It was. Truly. Everything was fine. Except I decided that I needed to check one more time. Just to make sure. I probably wasn’t going to find anything—and that’s when I saw it. I was on one faculty member’s Profile and noticed that an article with 65 cites on Hein had just been added. But then, when I looked at her total citation count, I noticed that it was 27. That couldn’t be right. I chose a few other profiles that had been edited in the last go-around and saw that many had the same problem. I sent an email to Hein about the issue and then started yet another spreadsheet. This time, I compared the total number of citations from three places—the CSV file that can be downloaded for the entire institution, the Profile total listed at the top of the Profile, and the sum of all of the citations listed for each article within the Profile. The results were…troubling. In some instances, I had three different numbers for a faculty member. I heard back from Hein that ScholarCheck numbers are reprocessed each month so it could take a few weeks for them to reflect changes. When I asked about the three different numbers (because if there’s a delay in updating, then at least two of the numbers should be the same), I was told that Hein had been experimenting with updating the CSV more often. An update just went through the end of March, so, once again, I created another spreadsheet so I could make sure that this time the football really and truly was teed up and ready to go.
If I were a glass-half-full person, I would happily report that the update corrected 12 Profiles with inconsistencies. But my metaphorical glass was smashed on the floor because there were still 21 Profiles with inconsistencies, and I couldn’t understand why. I sent another email to Hein and received a response thanking me for pointing out the inconsistencies and explaining that they would work on it. That was last week. Since then, there has been a bit of a discussion on the ALL-SIS listserv talking about determining whether the citation counts themselves are correct. As I typed that last sentence, my eye began twitching as it is wont to do whenever discussion of the Profiles arises. Through the spasms, I see many, many more spreadsheets and emails to Hein in my future. More checking, double checking, and just-one-more-final checking in the hopes that any errors can be found and corrected before the numbers are pulled.
While I do think that using Hein for citation counting is an improvement over other methods, the panic I felt at the U.S. News announcement wasn’t unfounded. Having had some previous experience with the Profiles, I knew that there was going to be work involved in ensuring their accuracy. I had no idea how much work, but I went into it with that expectation. I don’t know that every other law school did, particularly those who just followed the instructions from U.S. News. No rollout of a system—or forced opt-in of an existing product (as was the case in this instance)—is without slight hiccups. The important thing is to address the hiccups and keep moving forward. Some of these fixes appear to be ad hoc. Only if you notice the error will it be addressed.
Previous paragraph aside, not all the fault for this lies with Hein. There are other Lucys. Faculty publishing under name variations and not communicating new publication information (e.g. listing it on a CV or however it is communicated in an institution) complicates the process. Law reviews and journals not properly listing authorship and being less than exact with footnotes also throws a wrench into the mix. I haven’t addressed it here, but after having spent a summer manually counting citations I. Have. Thoughts. about the consequences of a misspelled name in a footnote and what it can do to citation counts.
Moving forward it would be great to see Hein incorporate more online companions to law review and journals (it’s a bit hit or miss right now) and online publications like blogs, beyond JOTWELL. If the Author Profiles could be expanded to allow for a full CV listing of publications—books, chapters, and all—it could give a more accurate citation count. While Hein may never be able to pull citation counts from books themselves, surely those books may be cited within articles within the Hein universe. It’s not perfect, but it would be an improvement. The precedent is set because there are publications that Hein lists in the Profiles that users can’t access, but still have citations counts listed.
Alas it’s time for me to get back into the thick of it. Hein’s got the football, telling me that there was another round of updates and I should check to see if there are any more inconsistencies. Here’s hoping that, as I’m going for my next attempted kick, they hold the ball steady this time. Charlie Brown prevails in the end, right?