The Mechanism Survives, the Magnitude Doesn’t

Auditing seven months of a news sentiment pipeline, including two of our own published claims

Tommi Johnsen and Svetlana Shasharina

Abstract

Here is the thesis, stated before the evidence: when we re-measured seven months of work on a pipeline that reads financial headlines and asks whether each one should move a stock, the mechanisms we had found held up. The magnitudes almost never did, including, twice, the magnitudes we ourselves had computed and believed.

The pipeline’s job is narrow. For roughly 850 tickers a night, it reads headlines and asks one question of each: is this new, material, and about this specific company? We have written about the results as we went. This document re-measures all of it on a rebuilt data basis and a fresh window, and grades every prior claim by whether it survived.

Three things broke under us this summer, and each broke in a direction that made the measurement better, not worse. A relevance gate now removes articles attributed to the wrong company before anything gets labeled. An audit of our stored data found that the next-day return column had silently covered only about half the days we thought it did. And a placebo test, in which we fed the pipeline deliberately scrambled sentiment, caught a defect in our own spread metric and dissolved the finding we had considered our strongest: the hybrid cascade’s apparent return inversion was an artifact of how our code broke ties, not a fact about markets.

What survived is simple to state. News sentiment does not predict tomorrow’s returns; on the post-gate window, most scrambled placebo draws beat the real signal at the next day. Whatever price move accompanies the news happens the same day the news arrives, and that concentration is real: not one scrambled draw came close to it. The cheap tone model buries good news about twice as often as bad, a bias so stable it survived every re-measurement, and the cost-saving cascade built on top of that model provably filtered out the very catalysts the system exists to find; it is retired. Two Claude prompts, identical except for how skeptical they are told to be, now label every article side by side, with returns as referee, and returns have not yet separated them.

We make no strategy claim. The honest summary of seven months is that mechanisms survive re-measurement and magnitudes do not, including, twice now, our own.

Part 1. The problem

Every day, financial outlets publish thousands of articles about public companies. The practical question hidden in that flood is narrow: which of today’s headlines actually matter for tomorrow’s price?

Most sentiment research answers a broader question, and the breadth is the error. It takes a corpus of articles, labels each positive, negative, or neutral, averages those labels into a daily score per company, and correlates the score with returns. The flaw is that the articles being averaged are not the same kind of thing. An earnings beat, a $5B acquisition, an analyst upgrade, and a Seeking Alpha opinion piece all enter the same average and count equally. Three of them carry information; one is noise dressed as signal. Averaging them together throws away most of what made the news informative.

The deeper flaw is attribution. A sentiment model trained on text answers “does this text sound positive?” An investor needs a different answer: “is this positive for this specific company?” The two diverge constantly. “Tech stocks flat, investors cautious, Nvidia shares rally” carries cautious macro language and a positive Nvidia signal in the same sentence; a tone reader sees the caution and calls it negative, confidently, while the investor-relevant answer is positive, about Nvidia, specifically.

This pipeline was built to handle both problems at once: read news about a target company, judge whether it carries new, material, directional information, and then test, without flattering itself, whether that judgment predicts returns. We have published what we found as we went. This document re-measures all of it: the return data was rebuilt this month, every claim in the series was recomputed on the corrected basis and on a fresh window, and each is graded below by whether it survived.

In plain terms: we built a system that reads financial headlines and asks one question about each: does this news actually matter for this stock’s price tomorrow? Most research averages all news together; this paper is us testing, in public, whether our more careful version works, and being honest about the parts of our own past work that didn’t hold up.

Part 2. How the news is collected, and the two windows

Articles come from Google News RSS feeds, gathered for each ticker through three tiers of search terms: the ticker symbol, the canonical company name, and a small vetted set of synonyms and abbreviations, so that “Meta,” “Facebook,” and “Meta Platforms” all land in the same net. For each article we keep the headline, source, timestamp, and summary; the full article body is left aside.

Why headlines only. Two reasons. First, noise: a full article about one company’s earnings usually carries several paragraphs of macro context (rate anxiety, sector headwinds) that contaminate the sentiment read and leave the model accurate about the text and wrong about the company. A headline is already an act of editorial compression: the editor has decided what the story is about, which is most of the work we would otherwise ask the model to do. Second, and more plainly: scoring full bodies across tens of thousands of articles per cycle is slow and expensive, for no gain we could measure.

The relevance gate. The loose tiered matching occasionally over-reaches, tagging an article to a company it only happened to name in passing. Since June 26 a relevance gate has run between collection and labeling: it drops articles whose headline does not actually concern the attributed ticker, re-routes mis-attributed articles to the company they are really about, and screens out options-chain noise. The gate is not a footnote; it changed the measured population, measurably. Five name-collision tickers (a salad chain that shares its symbol with “SG,” a biotech supplier trading as “TECH,” and three like them) accounted for 6.2% of all pre-gate articles; after the gate, their combined volume fell by roughly 89%, and by more than 95% for four of the five. Overall, Claude’s directional rate fell from 24% of ticker-days to 20%, and FinBERT-Claude agreement fell from 74.6% to 67.8%. The gate removes exactly the low-information articles both models used to wave through as neutral, so what remains is harder and more contested. Any number computed across the gate boundary mixes two populations. The gate itself needs no return-side justification and we claim none: it removes articles that are objectively about the wrong company, and it stays regardless of what returns do, because measuring the right inputs precedes any question about signal.

The two windows. That break dictates the design of this document. Everything below is computed on one of two windows, labeled accordingly:

• Window 1 (pre-gate): May 15 to June 28, 2026. 863 tickers, 24,329 ticker-days, three models (FinBERT, Claude, and the since-retired hybrid).

• Window 2 (post-gate): June 26 to July 12, 2026. 846 tickers, 8,454 ticker-days per model, a single clean article population, two Claude prompt variants labeling in parallel. Ten trading days of realized next-day returns. That is short, and we treat it accordingly: Window 2 is the baseline measurement for the new regime, not a source of findings in its own right. Where a Window 1 result fails to show up in Window 2, we say “did not reproduce,” not “reversed”; ten days can fail to confirm a magnitude, but cannot establish its opposite.

(The windows overlap by three days; the overlap is immaterial to every number reported.)

The corrected return basis. As part of the same rebuild we audited the stored return data and found two defects. The nightly job writes its summary before the next market close exists, so the stored next-day return column was filled only for runs regenerated later, covering roughly half of the signal-days, and weekend rows carried artificial zero returns from forward-filled prices. We also traced and repaired a small price corruption (ten tickers across two runs had stored a neighboring ticker’s quote after a data-vendor mix-up under request throttling) and manually verified every remaining daily move beyond ±30% in the analysis windows as a real market event. Every number in this document is computed on complete, consistently fetched returns, with no winsorization and no filtering; weekend news enters through the next trading day’s actual return, never through a forward-filled price. The correction is not cosmetic: it shrinks most return magnitudes several-fold, and part of what this document does is retire numbers this series has quoted that were computed on the incomplete basis.

A known limitation. Google News RSS over-represents aggregators (Seeking Alpha, Benzinga, Motley Fool) and under-represents primary sources. This shapes the news mix, and it is part of why roughly two-thirds of coverage that sounds non-neutral turns out, on inspection, to be immaterial.

A note on cadence. Early on we ran the pipeline intermittently, with lookbacks reaching thirty days. Once the analysis showed that whatever signal a headline carries dissipates within a day or two, those long lookbacks stopped making sense, and we switched to running daily. Both windows analyzed here are drawn from that daily-cadence period.

In plain terms: we collect headlines every night for about 850 companies. Since late June a filter throws out articles that are about the wrong company or about nobody in particular. We also found and fixed a defect in our own stored return data, which had made our earlier numbers look bigger than they were. The data splits into a before-the-filter period and an after-the-filter period, and we are careful about comparing across that seam.

Part 3. The models

The production pipeline now runs three labelers over every article, but they are not the three we started with.

The shared output schema

Every model labels each article positive, negative, or neutral for the target company. From those labels, for each ticker on each day, we compute three numbers:

• net_count = (positive articles) − (negative articles).

• total_articles = all labeled articles that day.

• sentiment_ratio = net_count ÷ total_articles: the daily signal, running from −1 on an all-negative day to +1 on an all-positive one, with 0 meaning balanced or, far more often, no directional news at all.

The Claude prompts add one thing: each non-neutral article is tagged by category, the kind of information it carries. quantitative: a specific reported number set against expectation (earnings, guidance, deliveries). event: a discrete corporate action (M&A, an FDA decision, a contract, an executive change). analyst: a price-target or rating change with a stated direction. none: everything else (opinion, sector roundups, generic commentary), and by rule, none means neutral.

Model 1. FinBERT, the retained baseline

ProsusAI/finbert is a BERT transformer fine-tuned on financial text: a fast, cheap, specialized reader that scores the emotional tone of financial language at about 400 articles a second, for nothing, on local hardware. It is the field-standard sentiment classifier, and that is precisely its job here: it keeps us measured against the literature, and it supplies the tone-reading foil against which the impact-reading models are defined. It labels 41% of ticker-days directional on Window 2 (42% on Window 1, one of the most stable numbers in the project), far above the share that is genuinely material, because it reads tone, not impact. It is not a signal, it feeds no return analysis, and as of July 5 it no longer appears in delivered dashboards; it remains in the stored data for auditability and comparison.

Model 2. Claude with the standard prompt, the production signal

Claude Sonnet is prompted as a target-aware reasoning agent: for each article it answers a single scoped question, is this new, specific, verifiable information likely to move this company’s stock, judging the target company only?, and defaults to “no” until the article earns a “yes.” The prompt anchors that default with an explicit base rate (roughly 80% of financial headlines are neutral) and an exhaustive list of what stays neutral: opinion, routine partnerships, CEO commentary restating known positions, descriptions of stock moves, press releases without dollar amounts, analyst reiterations, sector commentary. It labels 20% of ticker-days directional on the post-gate population.

Model 3. Claude with the strict prompt, the controlled experiment

The third labeler is the same model, the same rubric, the same output schema, with one dial turned: the strict variant anchors on a roughly 90% neutral base rate instead of 80%, and changes nothing else. It exists because of a calibration result described in Part 7: on an eight-ticker development panel, the stricter anchor improved the signal-only return correlation. Nightly paired labeling began July 1, with June 26–30 backfilled that same day using the identical prompt and model version, so both prompts cover the full post-gate window and accumulate a paired out-of-sample record on news neither has seen. The strict variant labels 17% of ticker-days directional; at the article level the two prompts agree 97.2% of the time, which is why telling them apart on returns is slow work.

The two prompts, in plain English. Both ask Claude the same single question about every headline: is this new, specific, verifiable information likely to move this company’s stock, judging the target company only? Both carry the same list of things that never count: opinion pieces, routine partnerships, executives repeating known positions, descriptions of yesterday’s price move, press releases without dollar amounts, analyst reiterations, sector commentary. The only difference is the stated expectation of how rare real news is: standard says roughly eight in ten headlines end up neutral; strict says roughly nine in ten. Same question, same rules, one dial turned toward more skepticism. In practice, standard calls 20% of ticker-days directional and strict calls 17%; they agree on 97.2% of articles; and the experiment in Part 7 exists to learn whether the extra skepticism buys anything in returns.

What happened to the hybrid

The original third model was a FinBERT-to-Claude escalation cascade, built to save money when Claude-on-everything was unaffordable. It was retired from production on June 29 after we established that it can invert the very signal it was built to refine. That result was the strongest architectural finding of the project, and it is presented in full, along with its own correction, as a closed case study in Part 6.

The current status of every labeler, in one place:

Model

Status

Claude, standard prompt

Production signal

Claude, strict prompt

Challenger; paired trial since July 1; first checkpoint July 29

FinBERT

Baseline only; feeds no return analysis; removed from dashboards July 5

Hybrid cascade

Retired June 29; case-study only

Figure 1. One real headline read three ways: FinBERT keys on the tone words and calls it negative; Claude with the standard prompt reads the regulatory catalyst and calls it positive; Claude with the strict prompt reaches the same call through a higher bar.

In plain terms: three readers score every headline. A fast, cheap model reads emotional tone. Two versions of Claude ask whether the news would actually move the stock, one ordinary and one extra-skeptical. A fourth arrangement, a money-saving combination of the cheap model and Claude, was shut down in June because it did more harm than good.

Part 4. What holds in both windows

The measurements below hold on the corrected basis, in both windows, across a changed article population. In this document, that is the highest compliment available.

4.1 The pooled correlation is zero, everywhere, always

Across the whole universe, daily sentiment does not predict next-day returns in any linear way, in any window, for any model, on any basis we tried. Window 1: pooled correlation +0.006 across the 24,143 Claude ticker-days with a computable next-day return (p = 0.34); restricted to news published on trading days only, +0.005 (p = 0.49). Window 2: +0.002 for Claude standard, −0.003 for Claude strict, −0.001 for FinBERT, all equally indistinguishable from zero on the trading-day-only basis. (Weekend news enters through the next trading day’s actual return, per Part 2; excluding weekend cells entirely changes nothing.) Dropping the zero-sentiment days does not rescue it either: the signal-only correlation is +0.016 on Window 1 (p = 0.23) and +0.005 on Window 2 (p = 0.85). Draw a straight line through “how positive was the news” against “what the stock did the next day,” and the line is flat. This is now the most replicated result in the project.

This is not the pipeline failing; it is the pipeline being honest about something most sentiment vendors never measure. Most ticker-days carry no directional news at all: 76% of Claude’s daily cells were exactly zero on Window 1, 80% on Window 2. No averaging scheme conjures signal out of a distribution that is mostly zeros. Whatever information exists must be looked for in the tails, in the slices, in the structure, and, as Part 5 reports, most of what we thought we had found there did not survive re-measurement either.

4.2 The asymmetry: the cheap model swallows good news twice as fast as bad

The single most durable empirical result of the project is a bias. When FinBERT defaults an article to neutral, that article turns out to be one Claude reads as a positive catalyst far more often than a negative one:

• Window 1 (155,603 unique ticker-article pairs, deduplicated by ticker and URL): 5,236 Claude-positives against 3,001 Claude-negatives in FinBERT’s neutral row, a ratio of 1.74:1, 95% CI [1.67, 1.82].

• Window 2 (30,905 pairs, post-gate population): 1,218 against 626, a ratio of 1.95:1, 95% CI [1.77, 2.14].

The two intervals nearly coincide at their boundary. A different window, an article population reshaped by the gate, and the direction and the roughly two-to-one magnitude came back intact; whether the post-gate ratio is genuinely a touch higher is a difference at the edge of statistical resolution, and we do not claim it. What replicated is the asymmetry itself, and that is exactly what makes this the strongest result we have. Press releases and corporate announcements are written in deliberately measured language, so the good news they carry reads as no news to a tone model; bad news, by contrast, tends to arrive with its vocabulary showing. This asymmetry is the engine of the cascade inversion in Part 6, and it is the one number in this document we would defend without hedging.

Figure 2. FinBERT’s neutral row, both windows. The good-news-to-bad-news swallow ratio: 1.74:1 pre-gate, 1.95:1 post-gate, confidence intervals nearly coinciding. The bias is a property of the model, not of the window.

4.3 The anatomy of disagreement reproduces, category for category

The claim that FinBERT reads tone while Claude reads impact is easy to assert; what makes it credible is that the disagreements between them fall into the same six clean categories in every window, down to the flavor of the examples. The Window 2 confusion matrix (30,905 articles, 67.8% overall agreement, almost all of it the giant neutral-neutral cell):

Claude: negative

Claude: neutral

Claude: positive

FinBERT: negative

1,067

2,904

156

FinBERT: neutral

626

18,799

1,218

FinBERT: positive

93

4,963

1,079

The six categories, with fresh post-gate examples:

• FinBERT negative, Claude positive (156): good news wearing negative vocabulary. Replimune: “FDA accepts Replimune filing; Merck KGaA drops $11B on life sciences services.” The word “drops” belongs to another company’s spending; the news is a regulatory acceptance. Rocket Lab: NASA awards it three Electron launch missions, reported as a stock “pop.”

• FinBERT positive, Claude negative (93): positive vocabulary, bad for this ticker. Bloom Energy: a headline about FuelCell Energy “rocketing” sits beside Bloom “tumbling”; the rocket belongs to the competitor. Apple: seeking approval to buy chips from a blacklisted Chinese supplier reads like initiative and is really regulatory exposure.

• FinBERT positive, Claude neutral (4,963, the largest disagreement): upbeat words about things that already happened. Ameren hitting an all-time high is a description of the past, not a catalyst. Advance Auto Parts’ “turnaround improving, but still too early to buy” is an opinion.

• FinBERT negative, Claude neutral (2,904): descriptions of past price drops. “Why is Alibaba stock sliding today?” “Bloom Energy shares are sliding Friday: what’s going on?”

• FinBERT neutral, Claude positive (1,218): restrained-language new catalysts. Agilent finalizing the Biocare Medical takeover, reported in flat corporate prose. Bernstein SocGen resuming Agilent coverage with an outperform rating.

• FinBERT neutral, Claude negative (626): dryly worded bad news. Biogen scaling back Apellis programs and trimming research roles after a $5.6 billion buyout. FAA directives on Boeing jets over Canada 5G interference.

Categories three and four are tone the market has already priced, the reason FinBERT over-calls directional news. Categories five and six are real catalysts its tone-reading misses entirely, and their imbalance is the 1.95:1 asymmetry of the previous section, sitting in plain view in the middle row of the matrix.

4.4 Per-ticker instability, a caution that keeps confirming itself

In this series we once wrote Nvidia off as un-signalable: saturated with generic AI coverage, its sentiment-return correlation near zero or dragged negative. Then the pre-gate window read +0.37 on the incomplete return column (the 21 days that happened to have returns), −0.04 on the corrected basis (44 days), and −0.27 on the post-gate window (17 days). Four measurements, four materially different numbers. The durable claim is the caution, not any of the values: per-ticker results are window-unstable, and any analysis that leans on one ticker’s correlation is leaning on noise. The full-universe view is not a stylistic preference; it is the only view the data supports.

4.5 What spread exists sits on the same day, and dies within two

Wherever we measure it, the return spread concentrates on the day the news arrives, not the day after. On the sign spread (mean return on positive-sentiment ticker-days minus negative ones, the tie-free metric adopted in Part 5): Window 1 reads +1.98% same-day [+1.70%, +2.26%] against +0.20% next-day [−0.03%, +0.44%]; Window 2 reads +1.20% same-day [+0.70%, +1.71%] against −0.11% next-day. The same-day interval excludes zero in both windows and for every model, including the tone reader; no model-level next-day interval excludes zero anywhere in this document, on either metric. By two days out the point estimates are mildly negative in both windows and statistically nothing. A permutation check sharpens the next-day reading further: shuffling each ticker’s sentiment across its own dates, destroying timing while preserving composition, reproduces the Window 1 next-day estimate in full, so even that small positive residual reflects which companies had good news over the window, not when the news arrived.

This concentration matters more than any other number here, because a same-day concentration is consistent with an uncomfortable hypothesis this project has circled before: that much of what the pipeline scores as “sentiment” is coverage of the move rather than news preceding it. Retrospective journalism arrives the same day as the return it describes. That even the tone reader separates same-day returns points the same way: descriptions of a move correlate with the move. The hypothesis is testable with data we already collect: the category tags distinguish datable catalysts (earnings, corporate events, rating changes) from commentary, and splitting the same-day and next-day spreads by category is queued for the longer window. If the same-day concentration lives in the commentary and the next-day residue lives in the catalysts, the pipeline’s directional content is real but smaller than measured; if the catalysts are also same-day, the market prices them before our nightly collection sees them.

Figure 3. Sign spread by return lag, both windows, corrected return basis, bootstrap intervals for all points. The spread sits on the same day in both windows, where it excludes zero, and is gone within two.

In plain terms: four things held up every way we measured them. Averaged over everything, news sentiment does not predict tomorrow’s returns. The cheap model buries good news about twice as often as bad news. The two kinds of readers disagree in the same recognizable ways every time. And whatever price move exists happens the same day as the news, not the day after.

Part 5. The published claims, re-measured

This section is the reason the document exists. Over the past months this series has made claims, some of them quotable, and the rebuilt basis of Part 2 plus the second window are the graders. Two caveats stand over everything below. First, several of our claims were computed on the incomplete return column, so part of what follows is correction rather than new evidence: the number changes because the basis does. Second, Window 2 is ten trading days on a gate-changed population, so where it disagrees with Window 1 we say “not confirmed,” nothing stronger; the difference could be time, could be population, could be noise, and the mid-August window fixes only the length problem.

5.1 The magnitudes: smaller than we knew, on a metric we had to replace, and no model wins

Midway through this audit, the placebo study of Part 5.6 exposed a defect in the spread metric itself. Roughly a third of signal cells carry sentiment ratios of exactly +1 or −1, a decile ranking is ill-defined on ties, and a defensible tie-order choice moved the Window 1 next-day value from +0.098% to +0.354%. A headline statistic should not move by a quarter point on an implementation choice, so the decile spread is retired from this document. The primary metric is now the sign spread: mean return on positive-sentiment ticker-days minus negative ones, no ranking step, no tie sensitivity (the same data reads +0.200% [−0.03%, +0.44%]). A conviction spread, the same construction restricted to strong signals (|ratio| ≥ 2/3; roughly a fifth of signal-days for the Claude prompts, a quarter for FinBERT, a seventh for the hybrid), covers what the deciles were reaching for: whether stronger signals separate more. They do not. The full scoreboard on the corrected basis:

Window 1 (pre-gate):

Model

Pooled r (p)

Signal-only r (p)

Sign spread, next-day [95% CI]

Sign spread, same-day [95% CI]

Conviction, next-day [95% CI]

Claude standard

+0.006 (0.34)

+0.016 (0.23)

+0.200% [−0.03%, +0.44%]

+1.98% [+1.70%, +2.26%]

+0.05% [−0.48%, +0.57%]

FinBERT

+0.004 (0.56)

+0.008 (0.44)

+0.050% [−0.12%, +0.22%]

+1.60% [+1.42%, +1.79%]

−0.01% [−0.30%, +0.30%]

Hybrid

+0.005 (0.47)

+0.016 (0.31)

+0.259% [−0.03%, +0.56%]

+2.83% [+2.45%, +3.21%]

−0.04% [−0.64%, +0.60%]

Window 2 (post-gate):

Model

Pooled r (p)

Signal-only r (p)

Sign spread, next-day [95% CI]

Sign spread, same-day [95% CI]

Conviction, next-day [95% CI]

Claude standard

+0.002 (0.88)

+0.005 (0.85)

−0.108% [−0.54%, +0.32%]

+1.20% [+0.70%, +1.71%]

+0.58% [−0.53%, +1.93%]

Claude strict

−0.003 (0.79)

−0.002 (0.93)

−0.281% [−0.76%, +0.21%]

+1.12% [+0.59%, +1.65%]

+0.69% [−0.60%, +2.13%]

FinBERT

−0.001 (0.90)

−0.001 (0.94)

−0.165% [−0.45%, +0.14%]

+1.33% [+1.02%, +1.64%]

+0.19% [−0.29%, +0.73%]

The reading is uniform. Every correlation is statistically zero. Every next-day interval straddles zero, in both windows, on both spread metrics. We tested every pairwise next-day difference between models directly, with bootstrap intervals: none excludes zero, in either window. And the ordering of the models is not merely unstable across windows; it is unstable across defensible implementations of the same metric, which is its own verdict (under the retired deciles the hybrid looked worst; under the sign spread its point estimate is highest; both readings are noise). There is no winner. Nothing this series has quoted survives at the next day as a demonstrated magnitude, with one graded exception taken up in 5.4.

Figure 4. Pearson correlations between sentiment and next-day return, pooled and signal-only, with 95% confidence intervals. Every value, for every model, in both windows, is statistically zero. Window 1’s intervals are narrower only because its sample is larger; the two windows do not differ from each other (Fisher z and bootstrap tests, all p > 0.6).

5.2 “Wrong 83% of the time on positive headlines”: the pattern stands, the anti-signal narrows

The published figure was exact: of 268 headlines FinBERT called positive on a 991-headline Nvidia and AMD panel, 45 were genuine catalysts by the impact reading, with Claude as referee. That makes it a label-level claim about two independent models, the kind this document finds durable, and unlike the hybrid’s accuracy numbers (5.3) there is nothing circular in it. That measure travels better than we expected. Generalized to the full universe on the corrected data: roughly 74% of FinBERT’s positive calls pre-gate, and 82% post-gate, are not confirmed as positive by the impact reading. The number moves with the window and the universe; the pattern it summarizes does not. The stronger version we published, that FinBERT’s positive-labeled days average −0.38% the next day, a genuine anti-signal, does not stand as a constant: it was true in the February semiconductor window and has not been true since; on the current data FinBERT’s positive days are merely weak (its scoreboard row above is indistinguishable from zero, not inverted). The standing claim is narrower and better: weak, and in some regimes inverted. Regime matters, which is itself a finding.

5.3 “The hybrid improves on FinBERT”: withdrawn as undemonstrated

In April we wrote that a 35% improvement in the next-day return spread over FinBERT alone confirmed the cascade’s filtering was the right call. The corrected basis does not support that claim, and after the metric replacement it does not support its opposite either. On the tie-free sign spread the three Window 1 models read +0.20% (Claude), +0.05% (FinBERT), and +0.26% (hybrid), every pairwise difference straddling zero; under the retired decile metric the same data had made the hybrid look inverted below both of its inputs. When the ordering of three models depends on the tie handling of the metric, the only defensible statement is that no ordering is established. The improvement claim is withdrawn as undemonstrated, not reversed. We also deliberately decline to lean on the classification-level results published alongside it: those were measured on a pre-gate, two-ticker panel, and for a cascade whose non-neutral labels are Claude’s labels, grading accuracy against Claude as referee is circular by construction. The hybrid’s retirement rests entirely on current-era, in-production measurements: the asymmetric discard of Part 6, present in both windows including the gated population; the 100.00% design fidelity on production runs; and the absence of any demonstrated return benefit on the corrected basis. None of it depends on anything measured before the current collection.

5.4 “Markets process a headline within one trading day”: graded like for like, at last

The April claim had two parts, and we were careful at the time to separate them: a cross-sector positive-minus-negative day spread of +0.28 percentage points at the next day, gone by two days out, with the same-day number set aside as untradeable. The sign spread is that same statistic, so for the first time the claim can be graded against its own metric on the corrected basis. Pre-gate, the published number was in the right range: +0.200pp with an interval of [−0.03, +0.44], against the published +0.28pp. Post-gate it is not reproduced: −0.108pp, well within noise. And the pre-gate number does not survive as a timing signal even where it is consistent: its interval includes zero, and the permutation check of Part 4.5 reproduces it in full from composition alone. The decay half of the claim stands everywhere: nothing persists past two trading days in any window. What we published as “news today, move tomorrow” is, on corrected data and the honest metric, “news and move arrive together, and the residue at the next day is which companies had good news, not when they had it.” The same-day concentration, precisely the part set aside as untradeable, is now the only spread that excludes zero, in both windows, for every model (Part 4.5).

5.5 The per-ticker and sector claims: narrowed

Nvidia (Part 4.4) is the emblem: four measurements, four materially different correlations, and a standing rule that no single ticker’s number means anything. Our sector rankings, the best and worst sectors for the signal, fall under the same statute. The sector-level evidence did not survive the corrected basis (details in the next section), and until the September re-test treats them kindly, sector rankings should be read as window artifacts, not properties of sectors.

5.6 The claims you never saw, and the study that killed them twice

Not everything the analysis produced reached this series, and the re-measurement is why. Inside the pre-gate window, three return patterns once looked sturdier than any single significant cell, on the argument that a monotone ordering across bins is hard to manufacture by chance. On the corrected basis and the tie-free metric, all three are gone. Volume, the last partial survivor, dies on the sign spread: Window 1’s bins read −0.14%, −0.09%, +0.52%, +0.07%: not monotone, and the one interval that excludes zero belongs to the unpredicted middle bin, the multiple-comparisons problem introducing itself yet again. Momentum was already dead on the corrected basis. Continuation against reversal, which had briefly survived in an unpredicted “reversals anti-predict” form, evaporates entirely on the tie-free metric (−0.02% [−0.80%, +0.81%]); the watch-list entry is withdrawn.

Behind these verdicts sits a study worth describing, because it caught more than the slices. We ran a placebo test: take the real sentiment column, destroy its information by permutation (once across tickers within each day, killing target-specificity; once across dates within each ticker, killing timing), a thousand draws each, and recompute every statistic in this paper on the known garbage. Three things came out. First, the same-day spread is genuinely company-specific: the real +1.98% clears the scrambled-ticker null band, whose edge sits at +0.38%, by about 1.6 points; the remaining fraction is market-mood loading that survives reassignment. Second, the next-day spread is not a timing signal: the scrambled-timing null reproduces it in full (Part 4.5). On the post-gate window the comparison is blunter still, under both schemes: roughly four in five draws with the sentiment scrambled across tickers, and seven in ten with the timing scrambled, produced a better next-day spread than the real production signal. Third, and least expected, the placebo caught the tie defect in our own decile metric, because real and null had to run through identical code, and the tie-order arbitrariness surfaced as irreproducible spreads. The metric was replaced (Part 5.1); the placebo study earned its permanent place in the pipeline.

The same pre-gate window had produced six slice cells whose bootstrap intervals excluded zero, out of roughly seventy cells tested across thirteen dimensions, registered internally as pass-or-fail predictions for a September re-test. On the sign spread, their real values now range from −0.04% to −1.52% (one cell, “both models agree negative,” is undefined under a sign spread because it is one-signed by construction, and the September protocol needs a stated grading rule for such cells). None comes close to the placebo yardstick: across a thousand garbage draws, the largest cell in the grid reaches a median of about 2% and a 95th percentile of 3.51%. The September pass threshold is therefore 3.51% in absolute value, recomputed on the September window at test time, not zero. A footnote of poetic justice: Window 2’s spectacular energy cell, +10.4% under the deciles and one small-cap’s crash-and-rebound under the autopsy, deflates to unremarkable on the sign spread without any manual exclusion; a mean over signed days is naturally harder for a single name to hijack.

The methodological lesson, in its amended form: a monotone ordering across bins, computed on one window, one return basis, and one implementation of one metric, is not in fact hard to manufacture by chance. We were wrong about what constituted a stability argument, twice. These findings died before publication, which is the process working; we list them because the graveyard is part of the ledger.

In plain terms: we checked our own published claims against the corrected data. The claim that the cheap model mislabels most of its “positive” headlines held up. The claim that our money-saving combination improved returns was never demonstrated, and we withdraw it. The claim that news predicts the next day’s move did not survive: the news and the move arrive together, and what little shows up the next day reflects which companies had good news, not when. Several patterns we almost published died in re-checking, which is what the re-checking is for, and it also caught a defect in one of our own measuring sticks.

Part 6. The cascade case study, closed twice

For its first six months the pipeline ran the hybrid cascade: FinBERT read every article; articles it called non-neutral went to Claude, whose label won; articles it called neutral were accepted unseen. The gate was a cost measure (FinBERT’s neutral pass-through, roughly 60% of articles on the corpora of that era, cut the Claude bill by more than half), never a quality claim, and when batch pricing removed the cost constraint the only question left was whether the gate was costing signal.

At the article level, the answer is proven and stands. The cascade executed its design with 100.00% fidelity over 263,793 articles across 45 production runs; there was no bug. And because FinBERT’s neutral default swallows Claude-positive catalysts roughly twice as often as Claude-negative ones (Part 4), the cascade silently zeroed good news at twice the rate of bad. That is not a hypothesis about the design; it is an arithmetic consequence of the asymmetry, visible in the stored labels. A gate with directionally asymmetric errors, placed in front of a better model in an AND-cascade, structurally distorts what the better model is allowed to say, and it distorts it with a sign.

The return-side half of this case study is where we correct ourselves twice. On the decile metric this document originally used, the hybrid’s Window 1 next-day spread came out negative and below both of its inputs, and we promoted that inversion to the strongest architectural finding of the project. The tie-free metric dissolved it. On the sign spread the hybrid reads +0.26% against Claude’s +0.20% and FinBERT’s +0.05%, all three statistically indistinguishable from each other and from zero; the dramatic sign flip was a tie-order artifact of our own decile implementation, exposed by the placebo study of Part 5.6. The label-stream distortion is proven; whether it surfaces in next-day returns is not something our data can demonstrate in either direction, because nothing surfaces in next-day returns.

The retirement of June 29 stands, on grounds that were always sufficient: the cost rationale is gone, no return-side benefit was ever demonstrated, and the gate provably discards the exact catalysts the system was built to detect. The general lesson survives in amended form, and gained a clause: when a cheap gating model’s errors are directionally asymmetric, an AND-style cascade inherits the bias into everything downstream, so measure the gate’s error asymmetry before trusting the cascade’s output; and before promoting any return-side consequence to a finding, check that it survives more than one implementation of your own metric. Our strongest architectural claim did not, which makes it a cleaner illustration of this document’s thesis than the inversion ever was: the mechanism replicated, the magnitude was ours.

Figure 5. The cascade schematic. The gate’s asymmetric discard of Claude-positives (roughly 2:1) is proven at the article level; the apparent return-side inversion did not survive the tie-free metric, under which all three next-day spreads are statistically indistinguishable.

In plain terms: the money-saving shortcut let the cheap model decide which headlines the smart model got to see, and it provably threw away good news twice as fast as bad news. We also thought we had measured its returns turning upside down; that turned out to be a defect in our own yardstick, caught when we fed the system deliberately scrambled data. The shortcut stays retired: it filters out exactly what the system was built to find, and it no longer saves money. The lesson doubled: check what your filter throws away, and check your yardstick.

Part 7. Calibrating the prompt against returns

With the cascade retired and FinBERT demoted to baseline, the live modeling question changed shape: not which model, but which prompt. Everything here is one experiment, run in three acts.

Act one: humans disagree with the rubric. A natural challenge to our labeling: the “impact, not tone” rubric is our own invention, so how do we know it is right, and not merely strict? We ran a blind study. Two people labeled the same news independently: one of us, and a professional options analyst. Both humans read far more articles as directional than Claude did, and on the contested cases the analyst sided with the looser human threshold about two to one. Two independent readers, one asymmetry, both against the model. Taken at face value: the rubric is too strict.

Act two: returns disagree with the humans. We operationalized the looser human threshold as an alternative prompt and re-scored every article on the eight most news-rich tickers. The strict rubric produced a signal-only correlation of about +0.23 and a spread of +3.9% (the latter on the since-retired decile metric; the correlations carry the argument and have no ranking step); the human-calibrated rubric inverted both, to roughly −0.06 and −2.4%, converging almost exactly on FinBERT’s numbers. Relaxing Claude toward how humans read news pulled the opinion pieces, hype language, and past-move explainers back into the “directional” pile, the very categories the strict rubric was built to exclude. The p-value on the strict rubric’s correlation grazes 0.05 on this small panel and we do not lean on it; what survives every resampling is the sign flip. Moving toward the human threshold does not weaken the return signal; it reverses it. Markets price news for catalysts; humans read news for tone and narrative; those are different questions with different correct answers. (It rhymes with Part 6: there, an asymmetric gate provably distorted what the better model was allowed to say; here, relaxing one prompt toward tone reverses the sign of its correlation with returns. Different mechanisms, one moral: the tone reading is not a weaker version of the impact reading, it is a different reading.)

Act three: the base-rate experiment, still undecided. If relaxing the rubric inverts the signal, does tightening it further improve it? A sweep of stricter variants on the eight-ticker calibration panel said yes, modestly: raising the prompt’s neutral base-rate anchor from roughly 80% to roughly 90%, changing nothing else, improved the signal-only correlation from +0.24 to +0.29. That variant, the strict prompt of Part 3, was promoted to parallel production labeling on July 1. The comparison below excludes the eight calibration tickers permanently; these numbers therefore differ, by design, from the full-panel table in Part 5. The out-of-sample verdict so far, on the post-gate window:

Prompt

Signal-only correlation

Next-day sign spread [95% CI]

standard (80% anchor)

+0.002

−0.185% [−0.61%, +0.29%]

strict (90% anchor)

−0.004

−0.335% [−0.83%, +0.13%]

Both intervals straddle zero, and their pairwise difference (+0.16% [−0.52%, +0.85%]) straddles zero as well. Statistically indistinguishable, from each other and from zero. The calibration edge has not yet shown up on live data, and given Part 5’s lessons about in-sample magnitudes, we hold open the possibility that it never will. The two prompts disagree on only about 3% of articles, so the paired record separates them slowly; the first scheduled checkpoint is at twenty trading days (around July 29), and the clean six-week post-gate window closes in mid-August. Until then the experiment’s honest status is: running, paired, and undecided, which is what an experiment is supposed to be before its data arrives.

In plain terms: we now run the ordinary and the extra-skeptical versions of Claude side by side on every headline. So far they perform the same. We will declare a winner only when returns separate them, and so far returns do not.

Part 8. What we can and cannot claim

Can claim (replicated across windows, populations, and the corrected return basis):

• The pipeline operates at production scale: roughly 850 tickers, nightly, two parallel prompt variants, honest returns alignment, a measured relevance gate.

• The pooled daily sentiment-return correlation is zero. Any claimed sentiment signal that pools all days should be presumed noise.

• FinBERT reads tone, not impact; its errors are directionally asymmetric (roughly 1.7 to 2.0 to 1, good news swallowed faster than bad, confidence intervals nearly coinciding across windows); and the disagreement structure between tone-reading and impact-reading reproduces category for category.

• The cascade’s asymmetric discard: an AND-cascade behind a gate with directionally asymmetric errors provably distorts the label stream, zeroing good news at roughly twice the rate of bad (100.00% design fidelity over 263,793 articles across 45 runs; the asymmetry itself is the best-replicated number here).

• Prompt calibration toward human judgment inverts the return signal; the strict rubric is the one returns reward.

• Per-ticker correlations are window-unstable and individually meaningless.

• What return spread exists is concentrated on the same day as the news, where it excludes zero in both windows for every model, and is gone within two days. The next-day residue is attributable to ticker composition, not timing.

• Real sentiment beats scrambled sentiment, but only on the same day: in a thousand-draw placebo test, no permuted assignment of the real sentiment values came close to the same-day spread, while at the next day the real signal sits inside the garbage distribution.

Cannot claim (not supported, not yet supported, or corrected away):

• Any statistically significant next-day return magnitude, anywhere, in any window, on any metric. Every next-day spread interval straddles zero, and magnitudes computed on the incomplete return basis shrink several-fold on the corrected one.

• The model ordering as a stable fact, on any metric. It is unstable across windows and across defensible implementations of the same spread statistic.

• The return-side cascade inversion. It was an artifact of the retired tie-sensitive metric and is withdrawn; only the article-level discard is claimed.

• Any spread magnitude computed by ranking this label distribution, where a third of the values are tied at the extremes.

• The monotone patterns (volume, momentum, continuation) or the six pre-registered slices, most of which weaken or flip on the corrected basis before the new window is even consulted.

• That the strict prompt beats the standard prompt out-of-sample. Undecided.

• That the signal leads the price. The same-day concentration raises the live possibility that some of what we measure is coverage chasing the move; the category-split test is queued.

• Anything tradeable after costs. We make no strategy claim.

Two earlier claims are deliberately absent from both lists. The first is that the structure of a company’s news cycle (discrete binary events against diluted or macro-driven coverage) predicts signal quality better than sector membership. Nothing in the new data contradicts it, but nothing retested it either; it requires per-ticker coverage profiling that we did not re-run. It is deferred, not dropped, and returns to the docket with the longer window. The second is the pairing of this pipeline with the earnings-tone model, the subject of an earlier post in this series: nothing here re-examined that pairing, and the one lesson that carries over to it is prospective: that any agreement-based ensemble should have its error asymmetry measured before its agreement is trusted.

A note on instrument. Everything above measures next-day stock returns, and the objections that sink a stock strategy (turnover, borrow costs, a magnitude below transaction costs) are specific to that instrument. Options may sidestep them structurally: a one-day decay that kills a stock hold is irrelevant to an option held for weeks, where the signal selects the name rather than predicting drift; and a materiality detector that flags that something happened better than which way it cuts is exactly what a direction-agnostic structure like a straddle monetizes. The honest caveat is that a catalyst readable in public headlines is one the options market has likely repriced, so any edge would live in the narrow cell where the news is real but undramatic enough that the volatility surface lagged. Establishing that would take a separate study on options data. It remains the one test that could answer whether any of this pays.

In plain terms: what we can say: the system runs at scale, the cheap model’s bias is real and repeatable, and news and price move together on the same day. What we cannot say: that any model reliably predicts the next day’s returns, or that anything here is tradeable as it stands.

Part 9. Moving forward

The forward agenda is organized around dates rather than ideas, because the ideas’ tests are already scheduled.

• July 29, the twenty-trading-day checkpoint. First adequately-powered read on standard against strict, on the paired out-of-sample record.

• Mid-August, the clean six-week post-gate window. The full analysis battery is scripted and reusable; re-running it when the window closes takes one command and no new decisions. Every number in Parts 4 and 5 gets its third measurement, on a single population, on the corrected return basis. This is the first window that can confirm rather than merely fail to refute.

• September, the registered slice re-test. The six cells of Part 5 are graded on the sign spread against the placebo yardstick, not against zero: a cell passes only if it beats the placebo max-cell 95th percentile (3.51% on the current grid, recomputed on the September window at test time), meets the minimum cell size, and survives removal of its largest single-ticker contributor. A stated grading rule for one-signed cells is required before the test runs. Current expectation: most fail.

• Split the lag structure by category. The same-day concentration demands the retrospective-coverage test: datable catalysts (quantitative, event, analyst) should lead the price if anything does; commentary should coincide with it. The category tags were built for exactly this question.

• Directional error asymmetry in the prompts. Split the paired panel by call direction: are positive calls less accurate than negative ones? If bullish opinion follows up-moves more than bearish opinion follows down-moves, the residual noise should live disproportionately on the positive side, and the next prompt variant should apply asymmetric skepticism rather than moving the base rate symmetrically.

• Split the day at the opening bell. The pipeline stores article timestamps but aggregates by date, which is exactly what makes same-day and next-day ambiguous. Splitting articles into overnight (published after the prior close, before the open) and intraday buckets turns the coverage question into a clean test: overnight catalysts have a genuinely tradeable next session, and if any timing signal exists, that is where it must live. The study is running.

• Insider transactions as a time series. SEC Form 4 filings are collected continuously but not yet analyzed as a signal in their own right. Build the time series properly and test whether insider flow leads, lags, or confirms the news signal.

• A second reasoning model, not a better tone reader. The frontier is not a cheaper classifier but an independent reasoner: replace “what is the tone?” with “does a second judge agree on the impact?”, turning disagreement into a robustness measure. The cascade lesson of Part 6 applies in full: any such ensemble must be measured for error asymmetry before its output is trusted.

• Calendar structure. Weekday against weekend against holiday news mixes, once a longer collection exists to slice.

This document set out to re-measure everything the series has claimed, and found more than it bargained for: a return column that quietly covered half the data, price glitches worth repairing, magnitudes that shrank on contact, and mechanisms that did not. The September window will grade this document the way this document has graded the series. On present evidence we expect the mechanisms to survive and the magnitudes to keep humbling us, and we would rather publish that expectation now, where it can be checked, than explain it afterward.

In plain terms: next comes a longer, clean measurement window through mid-August, a first verdict on the skeptical prompt in late July, a September re-test of six predictions we wrote down in advance, graded against how well deliberately scrambled data performs rather than against zero, and a check on whether news that arrives before the market opens behaves differently from news that arrives during the trading day.