Project Background
A legal tech platform provides online legal consulting services for enterprises and individuals, with an average daily consultation volume exceeding 3,000. The platform previously used general-purpose large language models to answer legal questions, but due to the highly specialized nature and dense terminology of the legal domain, the general model achieved only 71% accuracy in legal consulting scenarios, with a hallucination rate as high as 28%. It frequently gave plausible but incorrect or even erroneous suggestions, severely undermining the platform's professionalism and user trust. The platform urgently needed a dedicated model with genuine legal understanding.
Core Pain Points
Solution
Legal Domain LoRA Fine-Tuning
LoRA (Low-Rank Adaptation) fine-tuning was performed based on ChatGLM-6B for the legal domain. A high-quality dataset of 2,000 annotated legal Q&A pairs was carefully constructed, covering core legal areas such as contract disputes, labor disputes, intellectual property, and corporate law. After fine-tuning, model accuracy increased from 71% to 95%, and the hallucination rate dropped from 28% to 4%.
Legal Knowledge Enhancement
A legal knowledge base was built as a RAG supplement, incorporating authoritative sources such as laws and regulations, judicial interpretations, and leading cases. When generating responses, the model automatically retrieves relevant legal provisions and case precedents as supporting evidence, ensuring every answer is traceable to legal authority and further enhancing credibility and professionalism.
Quality Assessment and Continuous Iteration
A legal response quality evaluation system was established, automatically assessing model output across three dimensions: accuracy, completeness, and compliance. Training data is continuously supplemented based on issues identified during evaluation, creating a data flywheel that ensures ongoing model capability improvement.
Effect Data
| Metric | Before | After | Improvement |
|---|---|---|---|
| Legal consultation accuracy | 71% | 95% | 34% |
| Hallucination rate | 28% | 4% | 86% |
| Legal provision citation accuracy | 55% | 92% | 67% |
| User satisfaction | 62% | 91% | 47% |
Tech Stack
ChatGLM-6B, LoRA fine-tuning, PEFT, Legal Knowledge Base, RAG, Python, PyTorch, Hugging Face Transformers
