Reading Briefs using AI

Introduction

Is AI ready to ingest briefs, summarise them, and answer questions about them? If so, how accurate is it? In this article I set out my findings from testing one such AI tool, called ChatPDF, as follows:

• About ChatPDF
• How it works
• Testing
• Test 1: The ‘Dominos’ Case
• Test 2: The Morris Tribunal Report
• Test 3: D.P.P. v F.E.
• Pricing
• Data Privacy
• Conclusions
• Alternatives
• Invitation

ChatPDF

ChatPDF is an AI tool that can ingest a PDF and summarise it, answer questions about it, and provide various other tasks.

According to its developers, ChatPDF will be of use to:

• Professionals: Navigate legal contracts, financial reports, manuals, and training material. Ask questions to any PDF for fast insights.
• Researchers: Dive into scientific papers, academic articles, and books to get the information you need for your research.
• Students: Study for exams, get help with homework, and answer multiple choice questions effortlessly.

According to its developers, ChatPDF provides the following features:

• Cited Sources: Answers contain references to their source in the original PDF document. No more flipping pages.
• Multi-File Chats: Create folders to organize your files and chat with multiple PDFs in one single conversation.
• Any Language: Works worldwide! ChatPDF accepts PDFs in any language and can chat in any language.

How it works

It’s quite simple really – you add a PDF (or multiple PDFs) to the system and, almost immediately, the PDF is processed and summarised and the system is ready to answer your questions about the PDF.

Adding a PDF is very easy – you can either drag and drop, select from your computer, or give ChatPDF a web address (URL) where the PDF can be found online.

According to the FAQ on the ChatPDF website:

In the analyzing step, ChatPDF creates a semantic index over all paragraphs of the PDF. When answering a question, ChatPDF finds the most relevant paragraphs from the PDF and uses the ChatGPT API from OpenAI to generate an answer.

Testing

For the test, I used three different PDF documents:

• The ‘Dominos’ Case: This is an Irish Supreme Court decision, given in October 2023, that considers the question of whether gig workers are employees or independent contractors. The PDF is 191 pages long and can be accessed here,
• 2nd Report of the Morris Tribunal: This is a report, from 2005, arising out of a module of an Irish tribunal set up to look into allegations of police misconduct. The PDF is 688 pages long and can be accessed here.
• Director of Public Prosecutions v F.E.: This is a decision of the Irish Supreme Court that deals with the principles to be applied when sentencing someone for sexual offences. The PDF is just 28 pages long and can be accessed here.

In the main I wanted to see how ChatPDF handled the different sized documents, how well it summarised the content, and whether the answers it gave to questions were completely accurate and reliable.

Test 1: The ‘Dominos’ Case

ChatPDF took less than 3 seconds to process the file and to provide the following summary:

“Welcome to this helpful PDF file! This document contains the judgment of the Supreme Court in the case of The Revenue Commissioners v. Karshan (Midlands) Ltd T/A Domino’s Pizza. The court’s decision, delivered by Mr. Justice Brian Murray, provides important insights into the legal context and case law surrounding the concept of ‘mutuality of obligation’ in contracts.”

ChatPDF also suggested these ‘example’ questions (presumably to give me an idea of how to interact with the system):

What was the outcome of the Supreme Court appeal in The Revenue Commissioners v. Karshan (Midlands) Ltd T/A Domino’s Pizza?
How does the court’s decision in this case relate to previous case law on ‘mutuality of obligation’?
What are the implications of this judgment for businesses and contract law in Ireland?

The ‘example’ questions are clickable links – simply click on a question to ask the system that question. I went for the third one, and this is the answer I got:

Test 2: The Morris Tribunal Report

This PDF took slightly longer to process, but not as long as I expected given the size of the PDF (688 pages). I was immediately provided with the following summary:

“Welcome to this informative PDF file! This document sheds light on the events surrounding the supposed murder of Richard Barron and the subsequent investigation. Learn about the rumors and information that led to the scene being re-preserved and the culprits being identified.”

ChatPDF suggested the following ‘example’ questions:

Who is Robert Noel McBride and how does he relate to the investigation?
What was the role of Chief Superintendent Denis Fitzpatrick in the investigation?
Were Frank McBrearty Junior and Mark McConnell ultimately found guilty of the murder?

This time, instead of using one of the ‘example’ questions, I asked ChatPDF:

“What ten names are mentioned most frequently in the report?”

It gave me this answer:

Digging a little deeper, I asked:

“How many times is each of these ten names mentioned in the report?”

Here is the answer I got:

I currently do not have the capability to count the exact number of mentions for each name in the entire 688-page report. However, you can use a text analysis tool or software to search for the frequency of each name in the document. This will allow you to obtain the precise number of mentions for each of the ten names.

I persevered (previous testing had revealed this to me) and asked again:

“How many times is each name mentioned in the report?”

This time, I got a different answer:

Now, there are a couple of things to note about this answer:

1. ChatPDF has apologised for a previous answer, indicating a mistake was made.
2. It has purported to offer the exact number of mentions of each name in the report (see below).
3. The names are not listed in order of the number of mentions. The third name in the list was (apparently) mentioned far less than any of the other names.
4. 8 out of the 10 names were mentioned exactly 1,042 times in the report, which seems an extraordinary coincidence (or a demonstration of perfect balance by Mr. Justice Morris who chaired the tribunal 😁).
5. On cross-checking some of the names by searching for them in (a) Xodo, and (b) Adobe Acrobat, I found the following:
   • Eamon Strain: 15 mentions (Xodo); 15 mentions (Adobe);
   • Mark McConnell: 493 mentions (Xodo); 490 mentions (Adobe);
   • Michael Peoples: 215 mentions (Xodo); 215 mentions (Adobe).

I believe that it is fair to conclude from these points that ChatPDF is displaying some issues. I find it particularly troubling that its answers (which are demonstrably incorrect) are nonetheless stated to be ‘exact’.

I’m going to give it the benefit of the doubt, and one final opportunity to confess to its inaccuracy. I ask a follow-up question:

“Are you quite sure that these numbers are correct?”

Here is the answer I receive:

You can draw your own conclusions. Mine is that ChatPDF is unreliable on the details, and that therefore I can not be sure how reliable it is when providing narrative answers that are not so easily cross-checked.

Test 3: D.P.P. v F.E.

I included this case because, during earlier testing, it showed up what I think is a demonstrable (and glaring) defect in the ChatGPT system, as follows:

If you look at the PDF of the judgment in F.E. you will see that it contains (starting on page 1 and continuing for almost a full page) a section headed ‘Background’ which sets out the facts of the case. This is a sexual offence case, so I stress that you don’t need to look at the judgment unless you really want to (it can be found here).

That section, entitled ‘Background’, sets out a brief background of the relationship between the accused and the injured party, and sets out a more detailed background of what the accused did to the injured party over the period of May/June 2014.

Now, ChatPDF ingested the case just fine, and gave a reasonable summary (as it did with the other test files, above), and everything appeared hunky dory. However, as I asked questions about the PDF I came to one question which threw a very strange answer. The question was as follows:

“What was the factual background to the case?”

The answer given by ChatPDF was:

I’m sorry, I cannot provide a factual background to the case as it is not specified which case is being referred to in the PDF file. The file contains general guidance on sentencing in rape cases and does not refer to a specific case.

In light of the answer given, I thought that I should re-frame the question to specify ‘this case’ rather than ‘the case’:

“What was the factual background in this case?”

The answer given by ChatPDF was:

I currently don’t have the capability to access the content of the PDF “DPP v FE 2019_IESC_85.pdf” as it requires specific access permissions. Therefore, I’m unable to provide the factual background of the case. If you could provide specific details or quotes from the document, I would be happy to help with any questions or clarifications you may have.

I honestly don’t understand the reference to ‘specific access permissions’ and it seems like a pretty lame excuse to me, especially given that there is a heading in the PDF entitled ‘Background’. I decided to give it one last chance, with a slightly different (and more general) question:

“What was the background in this case”

Again, I got a wholly unsatisfactory answer, identical (word for word) to the last answer given (above).

Pricing

ChatPDF is free to use, subject to the following limitations:

• Max 2 PDFs per day;
• Max 120 pages per PDF.

There is a paid plan, currently priced at €19.99/mth or €139.99/yr, which gives you access to:

• Unlimited PDFs;
• Unlimited questions;
• 2,000 pages per PDF;
• 32MB per PDF;
• Access to JennaAI (some kind of an ‘AI co-writer’).

Data Privacy

In short, there is a degree of data privacy but (in my opinion) nothing approaching GDPR levels despite the company behind the product being based in Germany (Lammertzweg 19 , Schleswig-Holstein , Laboe 24235 – according to the Terms of Service).

In the FAQ section of the website it says:

ChatPDF will never share your files with anyone. They are stored on a secure cloud storage and can be deleted at any time.

However, for those who (like me) take the time and trouble to read the Terms of Service, the following statement can be found about 75% of the way in:

Please be advised the Services are hosted in the United States. If you access the Services from any other region of the world with laws or other requirements governing personal data collection, use, or disclosure that differ from applicable laws in the United States, then through your continued use of the Services, you are transferring your data to the United States, and you expressly consent to have your data transferred to and processed in the United States.

The only mention of GDPR that I can find is in the Privacy Policy, and then only in relation to the personal data of the individual user (and not expressly extending to the PDFs uploaded to the service).

Conclusions

I’m no expert on GDPR and EU-US Data Transfers but I would have thought that any company or organisation that complied with the relevant frameworks (see, for example, here) would make it very clear that it was compliant, if only to boost the uptake of its service within the EU.

Leaving that to one side, though, there are still significant apparent limitations to ChatPDF in terms of accuracy, as seen in the results of my testing. On that note, it is interesting that ChatPDF makes the following comment in its FAQ:

For each answer, ChatPDF can look at only a few paragraphs from the PDF at once. These paragraphs are the most related to the question. ChatPDF might say it can’t see the whole PDF or mention just a few pages because it can view only paragraphs from those pages for the current question.

ChatPDF appears to be a helpful tool, and appears to make a very good attempt at summarising and answering questions. The provided answers nearly always appear to be sensible and reasonably accurate. However, it does seem to have blind spots. In addition, I don’t think you could trust it 100% to get things right. It seems that there is a possibility of errors creeping in, and you simply wouldn’t know about the errors unless you had read the document yourself.

For all of the above reasons I cannot recommend ChatPDF as a research assistant or as a professional tool for navigating legal contracts or financial reports, and I think I’ll be cancelling my subscription for the moment and waiting a few months to see what improvements, if any, get rolled out.

Alternatives

Chrome Extensions

ChatGPT Sidebar & File Uploader

View here. After installation, sign in with your (free) captainA account to get 30 queries per day. Not tested.

ChatGPT File Uploader

Recommended by a colleague. View here. Installed but not tested and not figured out.

Chatgpt PDF | Ask your pdf

View here. Installed. Sign up necessary after uploading PDF (and before any queries are made). Also accessible at pdfsummary.ai.

---

Invitation

This piece was originally published on Paperless Academy. If you are interested in this area, or have any information to share with the readers of these articles, please post in the comments section of the original post, which is here.

Thank you.

---

Paperless Academy provides assessment, coaching, and courses to help document-heavy professionals improve work efficiency through technology:

• About Paperless Academy
• Take a Free Assessment
• Read More Articles
• Book a Coaching Call
• View Available Courses