Super interesting direction. I've been pretty skeptical of “AI for stock picking” for the same reasons you mention. Curious how you handle the challenge of companies pivoting into new business areas that don't have historical precedent? For example, Apple's shift into services or Amazon's AWS dominance weren't really predictable from their earlier financials.
One interesting finding in Stockfisher data is that a lot of these business pivots are actually planned by managers years in advance, in their 10-K and 10-Q filings.
Yes, managers are not good forecasters. But they do get certain things right. And if you figure out the patterns of what types of manager promises tend to play out, and assess them individually for their reliability, you can reason about these business model changes decently well.
This seems like a really interesting usecase of LLMs. I’m curious how you’ve validated the outputs and what gives you confidence that the forecasts are good and not affected by hallucinations or incomplete information?
The most interesting aspect of this is backtesting. Quant models get run on past data to see if their predictions work.
When you use LLMs agents, though, you run into their memorized knowledge of the world. And then there's the fact that they do their research on the open internet. It makes backtesting hard - but not impossible.
Super interesting direction. I've been pretty skeptical of “AI for stock picking” for the same reasons you mention. Curious how you handle the challenge of companies pivoting into new business areas that don't have historical precedent? For example, Apple's shift into services or Amazon's AWS dominance weren't really predictable from their earlier financials.
One interesting finding in Stockfisher data is that a lot of these business pivots are actually planned by managers years in advance, in their 10-K and 10-Q filings.
Yes, managers are not good forecasters. But they do get certain things right. And if you figure out the patterns of what types of manager promises tend to play out, and assess them individually for their reliability, you can reason about these business model changes decently well.
This seems like a really interesting usecase of LLMs. I’m curious how you’ve validated the outputs and what gives you confidence that the forecasts are good and not affected by hallucinations or incomplete information?
Constant iteration, mostly!
The most interesting aspect of this is backtesting. Quant models get run on past data to see if their predictions work.
When you use LLMs agents, though, you run into their memorized knowledge of the world. And then there's the fact that they do their research on the open internet. It makes backtesting hard - but not impossible.
We wrote about how we do our pastcasting validation here: https://stockfisher.app/backtesting-forecasts-that-use-llms