← All posts

Human vs. model: how we read the news with neural nets

It is now common to hand a language model any task that can be phrased in words: "Determine which aspect an outlet frames the event through" or "Find the persuasion or manipulation techniques in the text".

We tried to do both on a real news stream, then checked the result from a human's point of view. And we decided to show readers neither. Below is why, with numbers. Yes, this is a report on a negative result, the kind rarely published, though they are more useful than many positive ones.

01Out of the frame

There is a well-established dictionary of 14 "frames" — economy, security, legality, morality, quality of life, cultural identity, public opinion, politics and so on. It came from political science and has been used in media research for a quarter of a century.

The idea is clear: two outlets can name exactly the same set of facts about a fire and present it differently — one leads with the cost in money, the other with the threat to residents. If you can measure this, under each story you could write: "18 outlets out of 31 framed it as a security matter, 7 as an economic one".

We gave the same instruction and the same hundred news texts to three models from different developers — call them A, B and C, where C is noticeably more expensive and more powerful than the others.

At first the result looked encouraging. A and B agreed on 76 texts out of a hundred; Cohen's kappa — a statistic that subtracts chance agreement — was 0.71 against a chance agreement of 0.17. For a task with fourteen options, that is a lot.

Then we added the third model, and the picture changed.

The reason is visible in how the models use the dictionary. On this hundred texts, A's two most frequent frames cover 56%, B's — 52%, C's — 38%. The cheaper models dump almost everything into two big bins ("politics" and "security"), the expensive one spreads it wider. And the largest divergence is systematic, not random: what A called politics nine times, C saw as international reputation.

02The human as reference

So we did what such projects usually do not: we asked a human to label forty texts blind — without the outlet's name and without the models' answers. That is the reference.

The intervals overlap completely: on forty texts, "the expensive model is better" is not established. Money did not buy accuracy.

And the hypothesis "C spreads the dictionary wider, so it is probably right" did not hold either: on these forty texts the human's two most frequent frames cover 55% — right between A's 68% and C's 40%. The cheap model collapses, the expensive one fragments, the human does neither.

03When it is better to say "no"

Here is where it gets most interesting. We split the error into two parts.

And the instruction explicitly allowed an empty answer. The models simply did not use it. Hence a decomposition that explains everything:

• on texts where there is a frame, all three models are right in 67% of cases;
• on texts where there is no frame, they are almost never right.

04We tried to fix it

The hypothesis was: the model can tell frames apart but cannot abstain, and that is cured by wording. We rewrote the instruction two ways: the first puts abstention up front and names it an ordinary answer, the second adds an explicit two-step check.

The instruction does not calibrate the model, it throws it to the other extreme. The share of abstentions follows the slant of the instruction's wording, not the content of the news. And accuracy on texts that do have a frame falls in step: the model starts abstaining exactly where it should not. There is no setting in this range that gives you both.

There is a final touch. We ran the same original instruction twice on the same model with randomness turned off. First run: zero abstentions. Second: five. The quantity we were going to show the reader wanders between runs.

05In search of the "straw man"

We looked for other techniques in news reports too. There is a dictionary of techniques from propaganda research — loaded language, name-calling, appeal to fear, false dilemma, "whataboutism", the straw man (a substitution: the opponent is credited with a deliberately weak or distorted argument and triumphantly refuted, as if that had been their position). Finding them in text is hard for everyone: even narrowly specialized systems, built precisely for it and competing on scientific benchmarks, score only F1 ≈ 0.35–0.40.

F1 is a general score for "finding things in text" on a scale from 0 to 1: it folds into one number precision (how much of what was found is real, not a false alarm) and recall (how many of the real techniques were caught). 0.35–0.40 is low: even the best systems miss about as often as they hit. In other words, the ceiling is low for the task itself, not just for our models.

Best systems (ceiling)~0.38

A0.33

B0.23

C (expensive)0.21

"Loaded language": the human marked it in 6 texts out of forty, the models — in 16, 26 and 18. A threefold-to-fourfold overshoot. The model considers almost any lively language loaded.

"Doubt": the human marked it 6 times, the models — 0, 3 and 1 times, and zero overlap. Under the same word the human and the model mean different things.

It is worth saying separately why this is more dangerous than an error in a frame. The claim "the outlet did not name fact number four" a reader can check themselves in a minute. The claim "the outlet used an appeal to fear" is a value judgement about an organization, and there is no way to check it other than to take it on faith. Putting that next to an outlet's name at F1 0.2–0.33 means passing noise off as an accusation.

06Burn after reading

We did not show the audience the framing markers and persuasion techniques, though they were computed, measured and turned off a day later. What remained in the product are only the metrics a reader can check themselves: who wrote it, how many outlets, which facts are independently confirmed, which fact is missing from whom.

We consider this the right outcome, and publishing it more important than publishing a pretty one. Negative results in this area are almost never published: a product that tried to measure framing with a model and failed usually just ships the metric and stays silent about its quality.

07Three takeaways that carry over to other tasks

• Models agreeing is not truth. Two models can agree because they share one error. You have to check with a human, blind, and preferably early, before half a year is spent.
• Models are bad at saying "nothing". A classifier with a list of options will pick an option. If in your task "none of these fits" is a legitimate and frequent answer, measure separately how often the model uses it. Perhaps never.
• An expensive model is no rescue. The difference between a model that costs pennies and one twenty times more expensive turned out to be inside the confidence interval. When a task is ill-defined, money does not help — a reference does.

That is why digesta has neither a "framing lens" nor a "manipulation index" — only what can be checked: how many outlets, which facts are independently confirmed, and which fact is missing from whom.

---