How We Tried to Teach Claude to Read Like an Analyst, and the Market Said No

This is the story behind one section of the technical post “Testing a News Sentiment Pipeline” (forthcoming) but you can read it on its own. It is about first fully trusting Claude, then testing it ag

Here is a thing that turns out to be true: two thoughtful humans can look at the same piece of financial news, both decide it is good news, agree with each other, and both be wrong in exactly the same way. Not wrong about the news. Wrong about what the stock was going to do. And the machine we had spent six months half-trusting and half-arguing with was the one that got it right, precisely because it refused to read the news the way we did.

We did not expect to write this. We expected to write a small triumphant note about how we caught our AI being too strict and fixed it. Instead, we get to tell you how we caught ourselves, which is more useful and much funnier.

Let us back up.

The pipeline, in one breath

For half a year the two of us, Tommi Johnsen, an emeritus professor of finance and former director of a School of Finance, now consulting on quantitative strategies, and Svetlana Shasharina, a physicist far more at home with C++ and Python than in front of a trading screen, have run a system that reads the daily flood of financial news and scores it: is this article about a company good, bad, or nothing, for that company’s stock? We use three models to do the scoring. One is FinBERT, a small fast specialist that reads the tone of the language. One is Claude, a large model we ask to read the impact, not “does this sound cheerful” but “is this a real, new thing that could move the stock tomorrow.” The third stitches the two together.

The big technical post (”Testing a News Sentiment Pipeline: Findings and Next Steps”) is about what held up over those six months. The short version of that post: the signal is faint, it lives only in the extremes, it lasts about a day, and out of the three models Claude reads it best. But there was a hole in the story, one we had circled for a while without ever quite addressing. We had started from two comfortable assumptions: that humans are the best judges of the news, and that Claude was our stand-in for a human, after all, we trust it to help choose our medications, our hotels, and the poems we send our loved ones. Our unhappiness with the results made us suspect the first assumption was wrong: maybe Claude was not as good as we were, even though we had written its instructions ourselves. And so we finally decided to check Claude against a human, in the hope of making Claude better, and of finding trading signals that would let us become millionaires.

That is where this story starts.

We hand-labeled the news, like it was 1995

The plan was simple and a little tedious. Take a batch of real headlines, and have humans read each one cold, headline only, no peeking at what the models said, and label it good, bad, or nothing for the stock. Then compare the humans to Claude and see where they part ways.

Sveta went first, labeling 560 headlines across eight companies, chosen because they generate the kind of messy, contested news where humans and machines are most likely to disagree: Nvidia, Micron, Kohl’s, Best Buy, and a few others.

Before Tommi looked at anything, Sveta wrote down a confession about her own likely bias, in one sentence: I probably treat strong partnerships and technology advances as good news, and Claude probably shrugs at them as nothing. This matters more than it looks. Writing the guess down first turns the whole thing into a real test. If you look at the results and then invent an explanation, you can always find one; humans are pattern-matching machines and we will happily see a face in a cloud. Calling your shot in advance is the difference between a test and a horoscope.

Then Tommi, the finance authority, our tie-breaker, labeled a harder subset: 192 headlines, all of them cases where Claude and Sveta had disagreed, plus some controls where they agreed, all shuffled so she couldn’t tell which was which. She was blind to Sveta’s labels and to Claude’s.

Drum roll…

The humans ganged up on the machine

Here is what came back, and it was not subtle.

Sveta called 69% of those articles positive. Tommi called 57% of them positive. Claude called 12% positive, and 78% of them nothing at all. On this measure Claude isn’t in the same room as the humans. It is the outlier by a mile, sitting in the corner insisting that almost all news is noise while the two humans nod at each other about how much good news there is.

When you measure how often two labelers simply agree, Tommi and Sveta agreed about 68% of the time, a normal amount for two humans; people are not clones. Tommi agreed with FinBERT 66% of the time. And Tommi agreed with Claude… 36% of the time. Claude was the one everybody disagreed with.

And on the headlines where Claude and Sveta had split, Tommi came in precisely to break the tie, and she broke it in Sveta’s favor, about two to one. Sixty-five percent of the time the finance professor sided with the human read, not Claude’s. Sit with that pairing for a moment: a physicist who lives in data and a professor of finance, two people who are not casual about numbers, both reading the news more eventfully than the machine, and both, as it turned out, wrong about the stock.

Then the confession came true. On the specific articles where Sveta had said “positive” and Claude had said “nothing”, the partnerships and tech-advance stories she had flagged as her weakness, Tommi also said positive, about 71% of the time. The predicted bias was real, and the finance professor shared it. Two humans, the same tilt, in the same direction, away from Claude.

At this point the story writes itself, and at this stage, it seemed to be embarrassing for the machine. Two independent humans, one of them a pro, both read financial news as far more eventful than Claude does, both lean the same way, and the tie-breaker sides with the looser human threshold. The obvious conclusion, the one we fully believed on the afternoon we saw these numbers: our AI is too strict. It’s throwing away real signal because it insists on a stricter bar than a professional analyst uses. We don’t need a new model. We just need to loosen the rules we gave it.

So we did.

We built a looser Claude, and it worked

You tune a model like Claude not by retraining it, that’s the expensive kind of change, but by rewriting its instructions, the paragraph of rules we hand it with every article. Our production instructions, the ones running every day, we’ll call v2. It is worth spelling out what v2 actually tells Claude, because the whole story turns on it. v2 is deliberately strict. It starts from the assumption that most news is nothing, and defaults to “neutral” unless a headline clears a high bar: to be called positive, an article has to carry a specific, new, checkable catalyst, an earnings beat with actual numbers, a deal with a dollar figure attached, an upgrade from a named Wall Street firm, a regulatory approval that unlocks real revenue. And it comes with a long, explicit list of what does not count, no matter how cheerful it sounds: opinion pieces, articles explaining yesterday’s move, restated old news, routine partnerships with no money named. That strictness is the thing everyone, including us, was about to decide was a bug. We wrote a looser version, v4, designed to read more the way the two of us read.

Two changes did most of the work. First, we lowered the default, the rate at which Claude falls back on “nothing” when it’s unsure. v2 defaulted to “nothing” about 80% of the time; v4, about 50%. Second, we broadened what counts as a real event. Where v2 wanted that dollar figure or named firm before it would call anything directional, v4 let in the softer stuff: a named partnership, an executive making a strategic move, “signs of recovery in the latest earnings.” The kind of news a person reads as meaningful even without a number attached.

(We had actually tried a narrower fix first, a version whose details we’ll spare you, which failed in an instructive way: we tried to loosen only the partnership rule, and instead accidentally made Claude stricter across the board. Adding rules to an instruction sheet is like adding clauses to a contract, they interact, and not always the way you meant. We threw it out and changed the two big dials instead.)

And v4 worked, exactly as designed. Claude’s agreement with Tommi rose from 36% to 43%. Its positive rate doubled. It started catching the Kohl’s-recovery and Nvidia-partnership stories it used to wave off. Only a handful of its new calls looked like overreach. By every measure we’d set out to improve, v4 was better.

This is the point in the story where, on the original schedule, we close the laptop and write the triumphant note. We found the machine was too strict, we loosened it, agreement went up (on our way to wealth!).

Not quite, though.

The test that ruined everything (correctly)

Everything above measures v4 against humans and Claude. But we are not the thing the system is ultimately trying to predict. The thing it’s trying to predict is the stock price the next day. And we had never once checked v4 against that.

So we did the only test that actually settles the argument. We took the same eight companies, six weeks of them, 4,296 articles. We scored every article two ways, with strict old v2, and with loosened new v4. And then, instead of comparing to a human, we compared to what each stock actually did the next day.

The number that matters here is called a correlation, specifically a Pearson correlation, named after the statistician Karl Pearson, which is just a single number between −1 and +1 that measures how tightly two things move together. +1 means they rise and fall in perfect lockstep. Zero means they have nothing to do with each other. −1 means they move in perfect opposition, when one goes up, the other reliably goes down. So a positive correlation between our “good news” scores and next-day returns is what we want: good news scored high, stock goes up. A negative correlation is the nightmare: the thing we labeled good news reliably went down. It means our signal is not just weak, it’s backwards.

Here is what came back.

Strict old v2: correlation with next-day returns of about +0.24. Positive. Modest, but pointing the right way, the news v2 called good tended, faintly, to precede stocks going up.

Loosened new v4, the one that agreed with the humans, the improvement, the thing we were about to ship: −0.06. It didn’t just get weaker. It flipped sign. It went backwards.

Figure 1. The same signal, scored two ways. Loosening the prompt toward how humans read the news (v4) flips the decile spread from positive to negative, landing on the same backwards signal as the tone-reading model.

It is worth sitting on that for a second, because it genuinely stopped us. We took a signal that pointed, however faintly, in the right direction. We “improved” it by teaching it to read the news the way two intelligent humans read the news. And in doing so we turned it around to point the wrong way. The very changes that made Claude agree with us made it point the wrong way at the only job that pays.

We ran it a second way to be sure, a measure called the decile spread, which is less scary than it sounds. You take all the days, rank them from the most negative news to the most positive, chop them into ten equal buckets (those are the “deciles”), and then ask: what’s the difference in next-day return between the top bucket and the bottom bucket? If your signal is any good, the “best news” bucket should beat the “worst news” bucket, and the spread is positive. Strict v2’s spread: +3.9%. Loosened v4’s spread: −2.4%. Same story. Flipped sign.

And here is the detail that turned our stomachs a little, in a good way. When v4 flipped, its numbers didn’t land on some random wrong answer. They landed almost exactly on FinBERT’s numbers, the little tone-reading model that the whole first post is about not trusting. By loosening Claude toward the human way of reading, we had, without meaning to, rebuilt FinBERT inside of Claude. We turned our impact-reader back into a tone-reader, and tone-reading is the thing that fails.

A necessary, unglamorous confession about the +0.24

We have to stop and be modest about that +0.24. It is borderline. Six weeks and eight stocks is not a lot of data. When you have a small sample, you can’t just report a number, you have to ask how much it would wobble if you’d happened to catch a slightly different six weeks. The tool for that is called a bootstrap: you take your data, resample it at random thousands of times, and watch how much the answer jitters. When we do that to v2’s +0.24, the honest result is that it sits right on the edge of what statisticians call significant. Depending on the exact random seed you use for the resampling, its plausible range either barely clears zero or barely grazes it. The p-value, the standard “could this be a fluke?” number, where below 0.05 is the conventional line for “probably not a fluke”, comes in at almost exactly 0.05. On the line. Not comfortably past it.

So we are not going to stand here and tell you v2’s signal is proven.

What we can tell you is the thing that doesn’t wobble, no matter how you resample it: the sign flip. The gap between v2 pointing forward and v4 pointing backward is large and it is stable. Every single time we resampled, v2 came out ahead of v4, and v4 came out negative. You can doubt the exact size of v2’s edge. You cannot, on this data, doubt its direction. And the direction is the whole point: loosening Claude toward the humans didn’t help. It inverted the thing.

This is a small lesson worth keeping. When your data is thin and your question is “which of these two is better,” trust the comparison, not the absolute score of either one alone. A sign flip between two conditions is a much sturdier fact than any single condition’s significance. Comparisons are robust where absolutes are fragile.

What just happened? Why would the market disagree with a professional?

Two humans, one of them a working professional, read the news one way. The market, the next morning, prices it the other way, closer to strict Claude. Whoever is setting these prices at the open is reading the news less like a person and more like our strictest machine. So who, or what, is doing the pricing?

Sveta immediately asked: is all trading done by LLMs? This is such a great headline (could be analyzed by our pipeline☺): the market is being priced by machines now, and our machine is just reading the same way its cousins on the trading desks read. Is the world being quietly priced by AI?

Maybe. But we’ve been burned once already in this very post by grabbing the exciting explanation, so let us be disciplined. The finding is real. The reason for it has at least four suspects, and we genuinely cannot pick a winner.

Suspect one: this has been true since before the machines. Event-driven trading, funds that pounce specifically on earnings, drug approvals, deal announcements, executive changes, has existed since the 1980s. Elliott, Pershing Square, the classic merger-arbitrage desks. For forty years these specialists have trained the market to move on specific datable catalysts and to yawn at everything else. Claude’s strict rubric lines up with the market not because Claude is trading, but because Claude was trained on decades of financial writing that already encodes what those traders taught the market to care about. This is the boring explanation. It is also probably the right one.

Suspect two: the market already read the real news before the article was written. A professional trader gets the earnings number off a Bloomberg terminal in seconds. The rest of us read a Yahoo Finance article about it hours later. Claude and Tommi are both reading the slow, after-the-fact Yahoo headline, but Claude’s strict rules happen to fire on the same underlying events the fast money already traded, while Tommi’s looser threshold picks up more of the leisurely commentary that only exists because the price already moved. The alignment isn’t Claude-versus-humans; it’s that Claude and the fast traders happen to care about the same events.

Suspect three: Tommi and the market are answering different questions. A finance professor reads a headline to ask “does this change the story of this company, its fundamentals, its thesis?” The market’s next-day price asks a narrower and cruder question: “will people buy or sell tomorrow morning?” Those are genuinely different questions, and someone who has spent a career being excellent at the first can produce answers that simply don’t map to the second. This isn’t Tommi being wrong, it’s Tommi being right about her own question, which is not the question the stock price is answering. If you want to understand a company, ask the professor. If you want to trade tomorrow’s move, ask the strict machine. Both are correct; they are correct about different things.

Suspect four, conspiratorial one: cheerful news arrives late, on purpose. “Why Micron Stock Is Up Today” is published after Micron is up. Bullish opinion pieces cluster after good days like seagulls after a fishing boat. So if you treat that cheerful, after-the-fact tone as a signal about tomorrow, you are systematically buying things right after they’ve already risen, buying the top. The apparent “positive tone leads to negative next-day return” isn’t a quirk of the reader; it’s baked into when financial news gets written. Retrospective news is out of phase with the future by construction. FinBERT walks straight into this trap. v4 walks into it more politely.

Notice that all four suspects produce the exact same crime scene, strict beats loose, catalysts beat tone, and six weeks of data on eight stocks cannot tell them apart. To actually convict one, you’d need different evidence: does the effect get stronger after 2020, when the language models arrived? Does it hold in tiny thinly-traded stocks where the fast money doesn’t bother? Does the signal show up first in the big liquid names or the small retail ones? We haven’t run those. So we won’t pretend to know.

But, and this is why the finding matters regardless of which suspect did it, the practical lesson is the same no matter what. Cheerful news is a lagging indicator, not a leading one. The news that actually predicts tomorrow is narrow, specific, and datable: a real number against expectations, a named binary event, a named analyst making a named move. Everything else, the hype, the “here’s why it’s a buy,” the retrospective explainers, is the past wearing the costume of the future. Whatever is doing the pricing, it has already figured that out. We’re just late to it.

What this story teaches, beyond stocks

Strip away the finance and there’s a lesson here for anyone using one of these models for real work, medicine, law, summarizing science, anything.

Figure 2. The dividing line was not human against machine. Everyone who read the news generously, the two of us included, landed on the losing side; only the skeptical, catalyst-only readers predicted returns. This is a grouping by approach, not a shared numeric scale: the humans and the models are measured on different samples, so the figure sorts them by which side of the line they fall on, not by exact magnitude.

Test the machine against the thing you actually want, not against your own opinion of a good answer. This is the whole story in one sentence. We spent real effort making Claude agree with two smart humans, and that effort actively made it worse at its job, because “agrees with a smart human” and “predicts the stock” turned out to be different, even opposite, targets. Human agreement felt like the right measure. It was a decoy. The real judge was the returns, and we almost never asked it.

Call your shots before you look. Writing the bias down before Tommi labeled anything is the only reason the confirmation means something. A guess made after the fact is not a finding, it’s a story.

Change one dial at a time (like in grad school). Our first failed attempt tried to be clever and adjust one narrow rule, and the rules interacted and made a mess. The version that worked moved two big obvious dials and measured each. Cleverness in prompt-writing is usually just complexity you’ll have to debug later.

When the data is thin, trust the comparison, not the trophy. We can’t prove v2 is good. We can absolutely prove v2 beats v4. Lean on the sturdy comparative fact, not the fragile absolute one.

And the one we’ll actually carry around: when two smart people agree with each other and both disagree with reality, that’s not a fact about the two people. It’s a fact about what reality is rewarding, and it’s usually rewarding something narrower and less flattering to human intuition than we’d like. The market doesn’t care that the news sounds good. It has already priced the only part that was ever going to matter, and moved on.

What we are sure about and what needs more work

Can say: on these eight stocks over these six weeks, the strict prompt beat the loose one at predicting the next day, and it wasn’t close on direction. Loosening Claude toward the human way of reading turned it into FinBERT, same backwards signal. Two humans read the news more eventfully than Claude, and they were consistent about it.

Can’t say: that the strict prompt’s edge is proven, it’s borderline on this little data. That it’s the best possible prompt, we tested a couple of alternatives, not all of them. That any of this survives a longer window, a different market, or a different question until we go and check. And we especially can’t say Tommi was wrong, she was answering a different, perfectly good question. Just not the one the stock price was asking.

One more test: is the strict prompt the peak, or just a way-station?

There was one loose end we could not leave alone. If loosening the prompt broke the signal, does tightening it further help? Is our strict v2 the best we can do, or merely the strictest thing we happened to have written? So we swept three more variants, each a different flavor of “even stricter than v2,” and ran them against the same returns.

The first, call it the quantitative-only version, allowed a directional label only when a headline carried a specific reported number against expectation. A real earnings figure counted; a partnership, an executive move, an analyst opinion, all became “nothing.” The second restricted directional labels to a short allow-list of professional wire services (Reuters, Bloomberg, and the like), treating everything from the aggregators and opinion sites as noise by rule. The third simply turned the “default to nothing” dial even higher, from about 80% up to 90%.

Here is what came back, and it is the most useful kind of result: it points clearly in one direction without crowning a clean winner.

The wire-service version failed, and it failed informatively. This variant only lets Claude call a headline directional if it comes from a professional wire like Reuters or Bloomberg; anything from Yahoo Finance, Seeking Alpha, or a press release is automatically neutral. The correlation with returns collapsed to zero. The lesson: v2 already handles sources gracefully — it treats a raw press release skeptically but reads a Reuters story on its merits — and that was doing the useful filtering all along. A blunt source ban threw away too many real catalysts hat happened to be reported by the “wrong” outlet.

The other two both improved on v2’s correlation, though in different ways and with different degrees of trust. The quantitative-only version produced the single sharpest point estimate of the whole project, a correlation of about +0.34, but it is also the least trustworthy number here, and it is worth being blunt about why: it rests on only seventeen signal-bearing days, and on a sample that thin, a correlation that size could still be luck (its confidence interval, the range of values consistent with the data, just barely includes zero, and the odds of seeing it by chance are about one in six). It goes silent on roughly 89% of days, saying “nothing” to almost everything. So we read it less as a production tool than as a proof of a point, that a specific number measured against expectation is the purest catalyst there is, and not as our strongest result. The sharpest estimate and the weakest evidence can be the same number. The higher-default version was the more practical win, and on this sample the cleanest result we have: a correlation of about +0.29 with a confidence interval sitting a little more comfortably above zero than v2’s borderline one did, bought with nothing more than changing a single number in a single line of instructions. We still treat it as promising rather than proven, because fifty-nine days over six weeks is not many, but if any single number in this project earns cautious optimism, it is this one.

But before we crown it, the honest complication: on the decile spread, the tail-separation measure, v2 still leads. Correlation rewards a tidy relationship across the whole range; decile spread rewards clean separation at the extremes, where the tradeable signal actually lives. The stricter variants have a slightly cleaner middle; v2 has slightly better tails. They are not really competing for one throne. They are sitting together on a small plateau of strict prompts, each a bit better at one thing and a bit worse at another.

Which is the real finding, and it is bigger than any single number. We now have six versions of the reader tested against the same returns. Every version that moved toward the way humans read the news, looser, more generous, more tone-sensitive, got worse. Every version that moved toward more skepticism held up or improved. Six independent attempts agree on the direction, and only bicker about the exact address: the way to make one of these models better at reading financial news is to make it more skeptical, not more human. Our original strict prompt was not a lonely summit after all. It was the first house we built on the right street.

Figure 3. Six versions of the reader, ordered from most human to most skeptical. Every variant on the human end anti-predicts returns; every variant on the skeptical end predicts them. Starred points clear statistical significance; v6a’s higher estimate rests on too few days to qualify.

Where this goes next

More data, mostly, and we have already started collecting it, along three lines that each close a different gap.

First, we are now running both prompts live and in parallel: v2, the current standard, and v6c, the stricter cousin that edged it on correlation. The six weeks we have cannot cleanly say which of the two is better, so rather than guess we let them both score the news every day and wait for the accumulating record to settle it. A tie you cannot break on today’s data is a tie you break with tomorrow’s.

Second, the sweep that produced all of this ran on only eight news-rich tickers, which is why the samples were so thin (a few dozen signal-bearing days apiece). We are re-labeling the full six weeks across all 800-plus tickers with the strict prompt, to see whether its edge survives outside the handful of noisy, heavily-covered names, or whether it was a property of those eight. That fixes the breadth of the test, though not its length.

Because length is the third gap, and the only real cure is time. Six weeks becomes ninety days in the autumn re-run, which is the test that actually matters: it will either firm up the strict prompt’s edge or expose it as a fluke, and finally say whether v6c genuinely beats v2 or merely tied it on too little data. We have also only looked one day ahead so far; looking five and ten days out would tell us whether we have found a real signal or a very-short-lived twitch. And the honest way to test the “is it the machines?” question is to run the same thing on a market the fast money ignores and see if the pattern still holds.

As of publication, we have one week of parallel-labeled data. v2 and v6c are running essentially tied on Pearson r on the newest days, and the “does v6c actually beat v2 going forward” question remains genuinely open. What is not open, on any of our data so far, is that both beat the tone-reader — FinBERT and v4 have never predicted returns on any window we’ve measured. Around late July, at the 20-trading-day checkpoint, we’ll have the first meaningful read.

The technical companion to this post, “Testing a News Sentiment Pipeline: Findings and Next Steps” (forthcoming) has all the machinery, the confusion matrices, the confidence intervals, the model retirement decisions, for anyone who wants to check our arithmetic. This post was the human story. That one is the receipts.