ModelsArchitectures & capability
Fintan Labs releases Sealed News dataset, reporting that Qwen2.5-7B scored 91.4% on the next day's news with nothing added to the prompt
Fintan Labs has published a 266-question dataset on news from 13 October 2026. It reports that its unreleased Fintan method lets small open models answer next-day questions from their weights alone, beating both LoRA fine-tuning and putting the facts in the prompt.
huggingface.co (article, date unverified): In the company's own runs on all 266 questions, Qwen2.5-7B-Instruct rose from 7.9% as shipped to 91.4% with Fintan. [1]
SOURCES[1] fintanlabs/sealed-news-2026-10-13… huggingface.co
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- In the company's own runs on all 266 questions, Qwen2.5-7B-Instruct rose from 7.9% as shipped to 91.4% with Fintan. [1]
- Original source
- huggingface.co article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- Original publication date unverified
- Covered
- Oct 18 edition
- Evidence
- 1 source across 1 site
Reach: not measured yet
In the company's own runs on all 266 questions, Qwen2.5-7B-Instruct rose from 7.9% as shipped to 91.4% with Fintan. LoRA fine-tuning reached 13.2–16.2%, and putting the facts in the prompt reached 76.7%. With Fintan, Llama-3.1-8B and Gemma-2-9B scored 94.0% and Mistral-7B scored 90.2%. [1]
The CC-BY-4.0 dataset includes fact triples, source URLs, fairness flags and each model's answers, so the outputs can be checked. The method, weights and code are not released, so the Fintan scores cannot be reproduced, and the fine-tuning baselines are only partly specified. [1]
Some design choices weaken the headline score. It includes 35 questions the judges marked unfair. Some questions can only be answered with the source article. Fintan answers often repeat the stored fact word for word. And some of the 'news' facts date from 2024–2025. [1]
Earlier studies found fine-tuning a weak way to add new facts. If this result holds up, cheap nightly weight updates could replace some retrieval pipelines for fast-changing or per-user facts.
Read the full assessment
For now, the evidence is one day of news and a closed method.
Executive brief
Fintan Labs reports that its unreleased "Fintan" system lifted Qwen2.5-7B-Instruct from 7.9% to 91.4% on questions about the news of 13 October 2026, with nothing added to the prompt. The same card puts the facts-in-prompt baseline at 76.7%, below Fintan's score (dataset card). The company says it is "not publishing how Fintan works yet." What is public is a 266-question dataset with every answer. No independent reproduction or press coverage has appeared. Readers should treat it as an interesting, auditable claim that has not been verified.
What changed and event timeline
Fine-tuning shown weak for new facts
Ovadia et al. found that RAG beat unsupervised fine-tuning at injecting new knowledge. They also found that showing the model many paraphrases of a fact helped ().
Fine-tuning on new knowledge linked to hallucination
Gekhman et al. reported that models learn new facts slowly during fine-tuning, and that learning them raises the rate of hallucination ().
Before Oct 13, 2026: Settings pre-registered
Fintan Labs says it wrote down all settings and expected scores before that day's news existed. The card does not give the date, a hash or where the record is stored ().
Twelve feeds crawled, first run
Articles seen while the system was being built were thrown out. Five open models were run as shipped and with Fintan ().
- Also
"Next morning": Questions written and judged
A different model wrote one question per fact. Three judges, blind to which system answered, set aside 35 of 266 questions as unfair ().
Capabilities and access
- Reported capability: the model learns from a day's news overnight and answers the next day with no retrieval or prompt context.
- Models in the dataset: Qwen2.5-7B and 14B, Llama-3.1-8B, Gemma-2-9B and Mistral-7B, each as shipped and with Fintan.
- Personalisation claim: 40 simulated users with 12 facts each, 87.5% correct, about one cent per person per night.
Read the full section
- Reported capability: the model learns from a day's news overnight and answers the next day with no retrieval or prompt context.
- Models in the dataset: Qwen2.5-7B and 14B, Llama-3.1-8B, Gemma-2-9B and Mistral-7B, each as shipped and with Fintan. A Qwen2.5-72B score (95.0%) is reported but has no answer columns (dataset).
- Personalisation claim: 40 simulated users with 12 facts each, 87.5% correct, about one cent per person per night. Serving 256 users ran at "98% of normal speed" in vLLM.
- Access: the dataset is CC-BY-4.0. The method, weights and code are not released.
Technical analysis for researchers and developers
- Data schema: each row holds a fact triple (subject, relation, value, as_of), 1–3 source URLs, a
judged_fairflag, and per-model answers with correct/incorrect booleans. - Scoring: an answer counts as right if it states the fact's value. The card says automatic and human scoring agreed on 99% of answers.
- Reproducibility: the base-model and Fintan outputs can be audited, but the Fintan results cannot be reproduced without the method.
Read the full section
- Data schema: each row holds a fact triple (subject, relation, value, as_of), 1–3 source URLs, a
judged_fairflag, and per-model answers with correct/incorrect booleans. Qwen rows also carry the judge's RIGHT/WRONG verdict. - Scoring: an answer counts as right if it states the fact's value. The card says automatic and human scoring agreed on 99% of answers.
- Reproducibility: the base-model and Fintan outputs can be audited, but the Fintan results cannot be reproduced without the method. The fine-tuning baselines (LoRA, lr 2e-4 and 1e-3) are only partly specified.
- Dataset observations (inference): many Fintan answers repeat the stored value word for word, such as "Guardian as defence and security editor." Some questions, such as "By whom were preparations for another art heist uncovered?", cannot be answered without the source article. That design favours a system that has memorised the triples over a general-knowledge baseline.
Claims and evidence
- Qwen2.5-7B on all 266 questions (card):
- Shipped model: 7.9%
- LoRA fine-tuning: 13.2% at lr 2e-4, 16.2% at lr 1e-3
Read the full section
- Qwen2.5-7B on all 266 questions (card):
- Shipped model: 7.9%
- LoRA fine-tuning: 13.2% at lr 2e-4, 16.2% at lr 1e-3
- Facts in prompt: 76.7%
- Fintan: 91.4%
- Other models with Fintan: Llama-3.1-8B 94.0%, Gemma-2-9B 94.0%, Mistral-7B 90.2%.
- Who reported it: every figure comes from the vendor. No outside evaluation, paper or news coverage was found.
- Which subset is headlined: the headline numbers are over all 266 questions, including the 35 the judges set aside. Per-system scores on the 231 fair questions are not reported, but can be computed from the dataset.
Context and prior work
Earlier research found fine-tuning a poor way to add facts: RAG outperformed it (Ovadia et al.), and new facts learned through fine-tuning raised hallucination (Gekhman et al.). Fintan's 13–16% LoRA baselines are consistent with those findings. Its claimed edge is beating both fine-tuning and in-prompt context on fresh facts. The "sealed," pre-registered design targets a common weakness of continual-learning demos: training-data contamination.
Limitations, safety and contested findings
- Self-stated limits: the test covers one day of news with one question per fact. The personalisation test used simulated people (card).
- Not all "news" is new: some facts date from 2024–2025, from WHO pages.
- Degraded output: on row 16, Fintan-Qwen2.5-7B produced garbled mixed-language text.
Read the full section
- Self-stated limits: the test covers one day of news with one question per fact. Only 45 of 54 source articles could be re-fetched to train the baselines. The personalisation test used simulated people (card).
- Not all "news" is new: some facts date from 2024–2025, from WHO pages. On row 7 (a 2024 WHO fact), every Fintan variant answered wrongly.
- Degraded output: on row 16, Fintan-Qwen2.5-7B produced garbled mixed-language text.
- The in-prompt anomaly: Fintan scoring above facts-in-prompt runs against the prior work and needs explaining. The card offers no explanation.
Business and practitioner implications
- If the result is reproduced, nightly weight updates could replace some RAG pipelines for fast-changing facts and per-user memory.
- For now, nothing can be deployed. The method is closed, and the evidence is one vendor-run day.
- Practical step: download the 266 rows, score the 231 fair questions, and read the failures before engaging with the vendor.
Read the full section
- If the result is reproduced, nightly weight updates could replace some RAG pipelines for fast-changing facts and per-user memory. The claimed cost of about one cent per user per night would make that cheap.
- For now, nothing can be deployed. The method is closed, and the evidence is one vendor-run day.
- Practical step: download the 266 rows, score the 231 fair questions, and read the failures before engaging with the vendor.
- What to ask before relying on it:
- Results over several days.
- Knowledge retention after repeated nightly updates.
- Standard benchmark scores before and after Fintan, to check for regressions.