Behind LAGDA: How lawyers verify and keep legal AI accurate

In this interview, Atty. Sandro Perez, Legal Intelligence Lead at Anycase, explains why lawyers have always been central to how we develop and evaluate our AI system. We discuss the common failures we've observed in legal AI, why human oversight matters, and how our proprietary LAGDA system allows lawyers to guide AI toward more accurate legal answers.

Tyra Delos Reyes: Hello, Attorney Sandro. Thank you for taking the time to meet with me. I guess let's start with an introduction. Let me know what you do at Anycase.

Atty. Sandro Perez: I'm Attorney Sandro Perez. I currently lead the Legal Intelligence team at Anycase.ai. What we primarily do is make sure that all information in Anycase is accurate, no? That means checking actual legal documents, verifying the source documents behind our answers, and ultimately making sure that the answers our users receive are accurate.

Tyra Delos Reyes: Awesome. A big role of your team has been evaluating our AI system, and part of that is using the LAGDA system.

Could you briefly describe what LAGDA is and what it does?

Atty. Sandro Perez: LAGDA is an acronym. It's short for Lawyer-Annotated and Governed Data Architecture. It's an internal tool that lets our Legal Intelligence team help maintain the quality of our legal data.

There are really two key parts to it, no? The first is adding and vetting legal documents, and the second is lawyer annotations and verification.

For the first part, whenever we run evaluations and find that an answer is incorrect because a relevant legal source is missing from our database, our team looks for that document. For example, let's say there's a government issuance that isn't in the library yet. We verify the missing document first. We make sure we have the right source, and then we add it to the system with the correct title, citations, and all the relevant metadata.

But I think the more interesting part is lawyer annotations and verification. LAGDA allows lawyers at Anycase to review the documents in our system and provide annotations where necessary.

For example, an answer given by Anycase might not reference another important document. Through LAGDA's annotations, our lawyers can guide Anycase toward understanding the relationship between those two documents. In turn, the system can give users a more accurate answer because of that annotation. So yeah, that's what LAGDA is for.

Tyra Delos Reyes: I know you've been working with the Anycase system since 2024, when you first started, and you've essentially been at the helm of making sure its answers are accurate.You've been evaluating Anycase alongside Copilot, ChatGPT, and Gemini. Claude as well, right?

Atty. Sandro Perez: Yeah.

Yeah. So I'm curious: in those evaluations, why do you think it's important for an AI system to have humans in the loop?

Atty. Sandro Perez: Right. That's right. Our guiding philosophy at Anycase is "Trust starts before the answer." What does that mean exactly?

Before an AI answer even reaches the user, we want to know that the system has been tested on the kinds of legal questions it's supposed to answer, no?

For instance, before we made documents from our Banking and Finance cluster—BSP (Bangko Sentral ng Pilipinas), CIC (Credit Information Corporation), IC (Insurance Commission), and AMLC (Anti-Money Laundering Council)—available in the system, our team of lawyers had already run evaluations using actual banking and finance questions.

We looked at things like: Can the AI answer this correctly? Is it finding the right legal basis to ground its answer, no? And is it actually applying that source properly?

For us, it's not enough to put documents into the system and assume the AI will already know what to do with them. Even if the right document is there, the AI can still miss it, prioritize the wrong source, or use it incorrectly.

That's where the human-in-the-loop part comes in. In the Legal Intelligence team, we test the answers, look at where the system gets things wrong, and make sure those sources are actually translating into accurate and useful legal answers.

Tyra Delos Reyes: You've talked about how your team has tested Anycase against ChatGPT and Copilot, using questions from the Philippine Bar exams. Am I getting that right?

Atty. Sandro Perez: Yes.

What's the difference between an AI system that has humans in the loop and something that doesn't have that level of involvement?

Atty. Sandro Perez: Like I mentioned a while ago, Tyra, evaluations are really non-negotiable for us at Anycase. When we run these evaluations, we find that, time and again, Anycase can outperform general-purpose AI tools like ChatGPT and Claude Sonnet. From what we've seen, that really boils down to two things: first, the data; and second, the fact that AI systems still need human evaluators, or humans in the loop.

I'll speak more about the latter. Even when you have the data and the available sources, AI can still get things wrong, no? There's always a chance it will give an incorrect answer. That's a reality we embrace at Anycase: AI simply isn't more capable than lawyers at legal tasks. You really need lawyers vetting the answers before they reach users.

I can speak about some of the common issues we find, no? For example, there have been instances where the AI prioritizes an outdated ruling. When you search for a case, it might cite a case that isn't actually controlling for that particular doctrine, or it might not cite the correct references at all, no?

That's where we, the lawyers, come in. Instead of just letting the AI dictate the answer, we evaluate it and say, "Wait, that isn't the case you should be prioritizing," or, "When you retrieve these cases, this is the governing doctrine you need to consider."

Sometimes, as I've experienced, just one correction like that can completely change the quality of the answer.

Could you give me an example of LAGDA's impact on an AI system? Maybe a situation where you made an annotation or a change and it affected the quality or accuracy of the answer?

Atty. Sandro Perez: Right, yeah. We actually found one recently during our evaluations, involving the query: "When is preliminary investigation mandatory?"

Previously, there were instances where the system answered "four years, two months, and one day," because it was relying on Rule 112 of the Revised Rules of Criminal Procedure.

When our team reviewed that answer, we flagged it because the system should already have been citing the 2024 DOJ (Department of Justice) rules on preliminary investigations and inquest proceedings, no? Those rules were released in 2024.

So we basically told the AI, "For this query, you should be referring to the 2024 DOJ issuance to answer the question correctly."

After that, we ran the query again. This time, it correctly answered the question about when preliminary investigation is mandatory. It gave the answer "six years and one day of imprisonment, regardless of the fine."

On that note, another thing that's important to us is transparency, no? We don't want the answer to suddenly change without any context.

For this example, the system also explains that there was a previous threshold under Rule 112 and that it was subsequently changed by the newer DOJ issuance. The user can see why the answer is different and open the relevant documents. The same principle applies to other documents whose treatment has been modified through LAGDA annotations.

And if you look at the annotation itself, no? We're not creating a new rule or stating anything that isn't already in the legal sources. We're simply guiding how the AI understands and uses those sources.

There's obviously a lot of judgment involved in legal research. When your team reviews an answer, how do you determine whether the AI needs an annotation? What principles guide you in making that kind of correction?

Atty. Sandro Perez: Right, yeah. I think that's a really important discussion, Tyra.

First and foremost, an annotation doesn't mean we're rewriting the law or changing the contents of cases, no? As lawyers, we treat our canonical documents as sacred. We don't alter what the law, jurisprudence, or any legal document says. Annotation is really about helping the system understand the legal material properly, no?

For example, we might flag that a source the AI is prioritizing is already outdated, or point it toward a controlling authority or landmark case that it might have missed.

Sometimes it's also about the relationship between two documents. Maybe there's a newer issuance that amended an older rule, or a later case that clarified an earlier ruling by the Supreme Court. Those are things a lawyer would naturally look for when doing legal research, no? The challenge is that an AI system might not always prioritize them correctly on its own.

We're not deciding what the AI should or shouldn't say based on our personal opinions or what we think it should prioritize. We stay completely neutral in that process. We're applying the same standards you would expect from proper, ethical legal research: What is the current law? What is controlling? What is actually relevant to the question?

So really, the annotation follows the legal sources.

Could you tell me a little bit about our legal data infrastructure? How does Anycase go from source to answer?

Atty. Sandro Perez: The first step is identifying the documents we need to add to the Anycase library and then actually getting the files.

Sometimes we can retrieve them digitally from publicly available sources. But sometimes, hindi talaga maiiwasan, you have to get data runners, and that alone can already be a challenge.

For example, there was one time we were quoted ₱500 per page for a legal document. That's really expensive, considering how many pages there are and how many documents you need to get. So access itself can already be difficult.

Once we have those documents, the next step is vetting them. Our team checks each document to make sure the title, citation, date, and contents are complete and accurate. We add one source at a time.

For instance, if we're adding BIR (Bureau of Internal Revenue), we'll first look at what's available on the BIR website. We'll gather that information and check those documents. If we identify missing issuances or decisions from the BIR, that's when we send letters through our data runners to request those documents.

The process repeats itself, no? We get the documents, verify them, and make sure everything is complete, no?

Once we've finished collecting the documents we need from an agency, we turn them over to our Data Engineering team. They clean and structure the documents before making them available to users. That includes making sure the title, citation, and other metadata are accurate, based on the information provided by the Legal Intelligence team.

After all of that, our team evaluates the actual answers. Again, having the documents in the system is one thing, but making sure the AI can retrieve and use them correctly is another.

We run evaluations, and only once the system meets our criteria for accuracy do we push that new source to the library and make it available to users, no? So when a user sees a document on Anycase, they can rest easy knowing it's already gone through several human-led evaluation loops.

You mentioned criteria. Can you speak more about the criteria for approving a release? Specifically, how do you determine that an evaluation round is complete and an update is ready to be released to users?

Atty. Sandro Perez: We don't expect Anycase to be perfect during the evaluation period. But before we release an update to users, we need to be able to identify where the system is getting answers wrong. There should be no manifestly incorrect answers that could materially affect a user's professional life if they relied on them.

So, to sum it up, if the system performs below a certain level, we don't release it to users. We investigate where the problem is coming from. Is there a missing document? Is retrieval failing? Is the system classifying the wrong material? And then we fix those identified issues before the documents go live.

Tyra Delos Reyes: I know we have an internal goal of adding a new government source within one to two months.

I'm curious if there have been times when that's been delayed. What causes that?

Atty. Sandro Perez: Yeah, for sure, no? Like I mentioned a while ago, "Trust starts before the answer," and that's really our CEO's philosophy.

I'll give you a specific example of a time when ingesting new sources was delayed because we wanted to prioritize evaluation, no?

When we were adding documents from BOC (Bureau of Customs), during testing pa lang, no? We already saw that Anycase's accuracy for queries involving BOC documents was only around 60%. For us, that was unacceptable.

So the Legal Intelligence team, which I lead, had to investigate why we were performing at only 60%. And I think that's really the lawyer-in-the-loop part of the system. It's not enough to say that we have an answer. We also want to understand why an answer is wrong.

Through the evaluation process, we guide Anycase toward the correct legal basis para the next time a user asks that question for their own legal matter, the system can support the lawyer's actual query, for their actual case, and do it correctly.

So yeah, sometimes that means a source takes longer to release, no? We have a lot of data requests from our users, but we'd rather hold something back and investigate the errors first than release it when the answers aren't meeting our lawyers' standards.

FAQ and interview recap

Why are lawyers involved in developing Anycase?

Through rigorous evaluations, we've observed that AI systems can still make significant errors, even when they have access to the right legal sources. Common failure modes include citing outdated rulings, missing controlling authorities, and incorrectly applying legal provisions. We've documented these patterns in our evaluations.

These findings reinforce a philosophy we've held from the beginning: lawyers need to be directly involved in developing, testing, and improving legal AI. This approach also reflects the principles of human oversight and accountability emphasized in the Supreme Court's Governance Framework on the Use of Human-Centered Augmented Intelligence in the Judiciary (A.M. No. 25-11-28-SC).

What is LAGDA, and how does it work?

LAGDA (Lawyer-Annotated and Governed Data Architecture) is Anycase's proprietary system that allows real lawyers to guide how AI understands and uses Philippine legal sources.

We've been doing this through our Data Engineering team but operationalized it as early as May 2026 so Anycase's in-house lawyers could easily integrate into the data verification process.

Through LAGDA, our Legal Intelligence team can verify legal documents, annotate relationships between authorities, and flag outdated or non-controlling sources. These annotations help the system retrieve and apply the appropriate legal basis without changing the underlying laws or jurisprudence.

How does Anycase use human-in-the-loop evaluations to improve accuracy?

Our lawyers regularly test Anycase using actual legal questions, review its answers, and investigate where the system makes mistakes. They identify whether errors come from missing documents, incorrect retrieval, or the misapplication of legal authorities.

These findings inform improvements to the system, which undergoes further evaluation before new legal sources are released to users. As Sandro explains, our guiding philosophy is simple: "Trust starts before the answer."