Could One Tired Cliché Be Preventing People From Having Their Minds Blown by AI?

Two new podcasts debunk a common misperception about large language models

According to some recently published data, lawyers have been unimpressed with GenAI tools. But I’m starting to think that it’s a fundamental misunderstanding of how LLMs work that has lawyers pooh-pooing when they should be woo-wooing.¹ In fact, I’m convinced the culprit is an oft repeated cliché that’s both tired and inaccurate.

Hey there, I’m Zach Abramowitz and I’m Legally Disrupted!

In today’s newsletter, I’ve got

1. Two new podcasts for you; and

2. A reframe that should help people — especially the heretofore skeptics — see the light on AI.

Here’s the twist: the podcasts, although recorded independently, collectively debunk a myth that’s spoiling AI for lawyers.

Links to the New Podcasts

As Legally Disrupted regulars know, I record two regular podcasts each month: Zach Abramowitz is Legally Disrupted, which I do solo, and The AAAi Podcast, which I cohost with American Arbitration Association President & CEO (and former Chief Justice of the Michigan State Supreme Court) Bridget McCormack.

Bridget and I recently sat down with Adam Unikowsky, a partner in the Washington D.C. office of Jenner Block, to discuss part II of his article “In AI We Trust.”² Here’ are the links to the the interview with Adam on Apple Podcasts and on Spotify

Separately, I sat down with LegalMation CEO James Lee before we both headed to Nashville for ILTACON. We discussed his recent comments to Isha Marathe from Legaltech News about why ‘GPT-Wrappers’ miss the mark, and how he has modeled LegalMation on his experience in high school selling Snap-On Tools at a motorcycle store.³ Here are the links to my conversation with James on Apple Podcasts and Spotify.

Now, I didn’t know it when I recorded these two conversations, but I realized afterwards that, together, they debunk a myth about LLMs.

Meet the Cliché That’s Ruining AI For You

Last week I spoke in Nashville on a panel at the G100 and G200 about why LLMs may be experiencing a setback in legal circles.⁴ Someone in the room made a point that I have heard so many times: an LLM is good for repeat, mundane tasks but for more sophisticated work it falls short. I’ve encountered this narrative before, but for some reason, it never struck me how off the mark it is until I heard it formulated last week. Here’s why it’s wrong:

1. LLMs often struggle with high volume, repeat tasks.

2. LLMs are next level amazing at highly creative tasks.

Point #1 was my takeaway from the podcast with James, and point #2 was my takeaway from the podcast with Adam.

Maybe the reason it finally occurred to me how misleading this narrative can be is that the person who repeated it used the word “LLMs,” not “AI.” Here’s why that matters: this cliché is kind of true if you’re talking about older AI models not built using the transformer architecture (the backbone of all LLMs). Some of the best AI tools of the last five years (take LegalMation as an example) are quite good at performing what CEO James Lee refers to as “scut work,” e.g. responding to high volume lawsuits, discovery requests, subpoenas etc. But the fact that an LLM thinks more like a human than it does a one dimensional calculator can often lead to different outputs, when a lawyer automating these workflows needs to have the same result 100% of the time. Because the transformer model is so powerful you just assume it can handle the same kind of tasks that an older model could automate. But, that’s like concluding that a space shuttle can cross the street as effectively as a pogo stick.

Where the LLM excels is being a space shuttle. It’s not that it can’t perform repeat tasks — it can. It’s just much better at being creative. It understands language at a conceptual level plus it has familiarity with more topics/disciplines than any human.⁵ Up till now, I understood this intuitively because of the way in which I interact with AI models in my own work. I’m not always looking for AI to save time. Instead, I’m using it to up-skill on the fly, focus test new ideas, brainstorm, steel-man, strategize, issue spot and catch insights I missed, among other things.

A great example of LLMs being leveraged for creative legal work in particular appears in Adam Unikowsky’s article, which was the starting point for Bridget’s and my conversation with him. Adam recently showcased the fact that Claude (out of the box) was able to do the work of a Supreme Court clerk better, faster, cheaper. Now, it’s one thing for me to say that, but Adam clerked for Scalia and has argued multiple cases in front of SCOTUS. Adam even shows — and this is the part that really blew my mind — that AI is awesome at generating novel legal theory. He confirmed to Bridget and me that he is using AI on a daily basis in similarly creative ways.

So, if you’ve been underwhelmed by AI, maybe it’s because you were looking through the wrong pair of goggles. Perhaps you asked it to draft an NDA for you or perform some other low level task, and it acted a little too smart — more like a human and less like a robot. It would make sense that you’d find flaws in the final product. You are, after all a lawyer — you’re paid to find every little nit possible. So here’s the reframe: if you’re trying to automate mindless work, try first with older (e.g. Amazon Extract). Use LLMs like Claude and ChatGPT, on the other hand, to help you with the fun, challenging and creative stuff. Pogo stick across the street; rocket ship when you’re shooting for the moon.

1. I totally stole this from someone who attended my session at ILTACON last week. My apologies in advance.

2. Link to Adam’s article here

3. Link to Isha’s article here

4. The G100/200 are Chatham House Rules so I’m being intentionally vague. That said, if you ever have a chance to participate in this invite-only track at ILTACON (either as an audience member of a speaker), do it. Hands down, the most unique, dynamic and informative session I have attended at a legal industry conference. Totally worth the trip.

5. If you haven’t seen my webinar on how LLMs work (a fan favorite), you can register for it here and see it on demand.