.png)
Problem
A supply chain analytics company based in Silicon Valley wanted to enhance its investment decision-making by accurately extracting organization names from financial documents - a task known as Named Entity Recognition (NER). However, financial data poses a unique challenge for NER models, as it contains a high density of multi-word organization entities compared to the more common person or location entities found in general text.
Approach
Our team employed a two-step approach to speed up the development process, reduce its cost, and increase the AI accuracy.:
- AI-Assisted Label Correction: We used an AI-powered process to quickly identify and correct any labeling discrepancies in the training data, ensuring high-quality annotations.
- Fine-Tuning a Smaller Language Model: We fine-tuned the Roberta-base model, a more compact language model, on the cleansed dataset.
Before
Before working with our team, the client had tried using several off-the-shelf AI solutions, including the popular ChatGPT 3.5 model. But the results of these models only achieved F1 sores (a measure of accuracy) between 50-60%.
After
The fine-tuned Roberta-base model achieved 87.15% F1-score, which is 27% better than off-the-shelf ChatGPT 3.5. It also slightly outperformed ChatGPT 3.5 fine-tuned on the same data, which achieved 87% F1-score.
Result
Our approach yielded remarkable 27% accuracy improvement in financial NER performance.
Not only that, but our Roberta-base models in production processed 100,000 articles at 1% of the cost of GPT3.5 model - a staggering 99% reduction in ongoing operational expenses. While GPT3.5 costs around $1,000, Roberta-base costs only $13.09!
To learn more about this case study, how we fine-tuned both the winning Roberta-base model and GPT3.5, and how we computed the ongoing cost of each model, contact our team here.
