What I learned tagging 600,000 events in SEC filings with an LLM
Notes from building and evaluating an LLM pipeline that tags SEC 8-K filings, including why the quality score only became useful once a separate call did the grading.
I've just got back from ICAIF '26 in Milan, where I presented a poster on a project from work. At Massive we use an LLM to tag SEC 8-K filings (i.e. the reports US public companies file when something material happens) with the events they describe, and the paper covers how we built and evaluated it. This post is the less formal version, with the parts I found most interesting and a few things that didn't work.
My first version asked the model to score each tag from 1 to 5 as it produced it, and 74% of tags came back as a 5. With three quarters of the tags at the top score, filtering on the score removes very little. Moving the scoring into a separate call, which sees only the evidence for the tag, spread the scores out. Precision (i.e. the share of tags a stronger model judged correct) then rose from 12% at score 1 to 96% at score 5.
The problem with item codes
When something material happens at a US public company (a CEO leaving, a merger closing, a data breach), the company has four business days to file a Form 8-K with the SEC. Our corpus has 292,984 of them, from January 2022 to June 2026. Each filing carries one or more of 32 item codes, and for most datasets those codes are the only machine-readable label.
An item code identifies which paragraph of the SEC's rules required the filing. Item 5.02 covers every officer and director change, so a CEO being fired and a routine board retirement get the same code. Item 8.01, "Other Events", is a voluntary catch-all, and a lot of market-moving news ends up there.
One way to see how much varies within a single code is to measure how much the company's stock moves around each filing, on the filing day or the next, in either direction, relative to its typical daily move (i.e. its daily variation over 60 trading days ending about two weeks before the filing). The figure below does this for each event type in our taxonomy (i.e. finer labels such as CEO departure or clinical trial results, described in the next section), grouped by the item code each type is most often filed under.
One item code, very different events Each dot is an event type, placed at the stock's average move on the filing day or the next, as a multiple of its typical day (1.0× means an ordinary day's move). Grey dots are event types usually filed under other items.
So the aim was a tag for what actually happened, fine-grained enough to filter on, with evidence attached to each tag.
The pipeline
The tags come from a taxonomy of 119 event types, grouped into 29 groups and 8 broad domains such as leadership and governance or strategic transactions. Specific types include ceo_departure, merger_completion and cybersecurity_incident. Each type has a one or two sentence definition, and the prompt includes the definitions as well as the names. A sibling type below means another type in the same group, such as a merger agreement and a merger completion.
For each filing, one call to a small, cheap model (the extractor) gets the whole taxonomy plus the filing text and returns a list of tags. Each tag is an event type plus a quote from the filing that supports it. Two checks run on the output, and if either fails, the error goes back to the model and it tries again (up to three times).
- The type must be a real taxonomy entry. No invented categories and no near-miss spellings.
- The quote must actually be in the filing. The quote check splits the quote into overlapping four-word runs and accepts it if at least 40% of them appear word for word in the filing. The threshold is loose on purpose, because models lightly tidy quotes even when the prompt tells them not to.
Below is that check run on a real filing, with the names swapped out. Each button picks a candidate quote, and the highlighting shows which four-word runs were found.
The quote check Four-word runs of the quote found in the filing; the quote passes at 40%
The quote check makes it much harder to cite text that isn't in the filing, but it doesn't test whether the quote establishes the event, which is what the last case shows.
Grading in a separate call
My first version asked the extractor to give each tag a score from 1 to 5 as part of the same call, which I'll call the inline score. It seemed like the obvious design, because the extractor already has the whole filing in front of it.
Across all 601,088 tags, 74% came back as a 5 and almost all the rest as a 4, with about 2% at 3 or below. That makes the score close to a yes or no flag. You can keep everything, or keep the 5s and still have three quarters of the data.
So we moved scoring into a separate call, which I'll call the separate grader. For each tag, it gets only the quote and the definition of the event type, without the filing or the reasoning that produced the tag. It then rates how well the quote on its own establishes that the event happened. A 5 means the quote states the event outright, and a 1 means it doesn't establish the event or points to a different one. The separate grader uses the same small model as the extractor. So the extra cost is one short call per tag, at about two tags per filing and a sentence or two of input each, rather than a whole filing.
To test whether either score tracks correctness, I had a stronger model act as a judge. For each tag it reads the full filing, the tag and the event type's definition, and returns one of three verdicts: correct, wrong category, or event not in filing. Precision is the share of tags it marks correct.
The main evaluation covers 5,125 tags, sampled so that every event type and every separate-grader score is equally represented (so rare scores still get plenty of examples). That sample is balanced on the separate grader's scores, which says little about the inline score, so I compared the two scores on a separate set of 3,065 judged tags balanced on the inline score. The figure below shows that comparison, and dragging the target line shows how many tags each score can keep at that precision.
Precision against share of tags kept Precision against the share of tags kept, as the keep-threshold rises from ≥1 to ≥5
Drag the target line (or focus it and use the arrow keys).
On the main sample, the separate grader's precision for scores 1 to 5 is 12%, 47%, 78%, 87% and 96%. In practice we don't serve tags with low scores. Keeping only the 5s keeps 34% of tags at 96% precision, and keeping 4s and 5s keeps 55% at 93%. The share of tags judged event not in filing (i.e. outright hallucinations) falls from 8% at score 1 to none of the 1,039 score-5 tags in the sample.
The CFO tag from the quote check is a concrete example, with the same quote given both scores:
The Company has appointed [CFO], the Company's Chief Financial Officer, as the Company's Interim Principal Executive Officer
The separate grader isn't better at every threshold. At the inline score's one useful threshold, which keeps about three quarters of tags, the two give roughly the same precision (about 88%). The difference is that with the separate grader you can keep going, keeping fewer tags in exchange for higher precision, while the inline score has no stricter threshold to move to.
I don't know why the separate grader gives a more useful score, and the paper doesn't claim a reason. Perhaps scoring in the same call that produced the tag pushes the score up, and a call that didn't produce the tag avoids that. Perhaps it's the narrower question, since with the rest of the filing out of view the separate grader can't fill gaps with what the filing seems to imply.
What tripped me up
The first evaluation was too generous. The judge originally had a "close enough" verdict for a real event filed under a sibling category (say, a merger agreement tagged as a merger completion), and we counted those as correct. That gave 99% precision at score 4 and above. But the judge only ever saw the definition of the one category it was checking, so it couldn't judge whether a sibling fit better. We dropped that verdict, re-ran the judge with the strict three-way scheme, and precision at score 4 and above fell to 93%.
The comparison between the two scores changed too. Under the generous scheme the separate grader looked better at every threshold. Under the strict one the inline score looks like a near-binary flag and the separate grader gives a range of thresholds to choose from.
Real quotes can still support made-up events. Reading through the tags that had the top inline score but were judged wrong, the biggest group fired on a reference to an event rather than on the event happening. Examples include a rating-downgrade tag on "if a downgrade occurs" risk language, and a deal-withdrawal tag on a letter that discussed a deal without withdrawing it. The quotes themselves are real. The separate grader's prompt says that hypothetical, conditional or procedural language rarely establishes that something happened, which is aimed at exactly this kind of tag.
Out of curiosity, I re-ran 48 of these failures through the exact production extractor, with and without a prompt patch saying that the event must actually have occurred and a reference isn't enough. Without the patch, 43 of the 48 (90%) came back with the same mistake. For event-not-in-filing tags, 27% came back wrong with the patch, against 64% with a stronger model and no patch. For wrong-category tags, 66% came back wrong with the patch, against 41% with the stronger model. So in this small sample, the event-not-in-filing errors responded more to clearer instructions, and the wrong-category errors responded more to a stronger model.
The extractor can inherit item-code noise. Somewhat ironically, a shareholder director-nomination tag kept firing on routine deadline notices, because the filing's own item heading (Item 5.08, "Shareholder Director Nominations") contains the phrase. It ended up among the least precise types in our evaluation. The separate grader scores 83% of its tags below 4, so they mostly drop out once you filter.
There are a lot of ways to get price reactions wrong. When I first measured them, some JPMorgan filings were being measured on exchange-traded notes instead of the common stock, and same-day filings by one company were double-counting a single price move. An independent audit of the analysis code caught both. Neither changed the conclusions, but both would have been embarrassing to find later.
Checking against market data
Apart from the first figure, everything so far is LLMs checking LLMs, so the last check goes back to market data. That is what the first figure measures, across 182,174 filing events. The figure itself uses the 125,726 of these filings with a single item code apart from the exhibits item, so that each price move belongs to one kind of event. Event types differ a lot on it, even within one item code.
Re-running the market analysis on only the tags the separate grader scored 4 or above leaves the ranking of event types almost unchanged, with a rank correlation of 0.94 (where 1 would mean an identical ordering). What drops out includes the imprecise tags the judge flagged, like those director-nomination notices. So the separate grader's score is also useful in the market analysis, as a way to set aside the least reliable tags.
What I'd take away
- Make every output checkable. With a fixed list of event types and a quote that has to appear in the source, anyone can check a tag against the filing without rerunning the model.
- Score in a separate call. Asking the extractor to score its own tags gave me something close to a yes or no flag. A narrow second call that sees only the evidence gave a score I could filter on. It's cheap, and I'd try it in other LLM extraction pipelines.
- Check against something that isn't an LLM. The judge is still a model with its own biases. Market data gives an independent check that the tags carry real information.
We didn't measure recall, so we know how often a served tag is right but not how many events the pipeline misses. More detail on all of this is in the paper, and the high-scoring tags are served on the Massive API, updated as new filings come in.