Ask ChatGPT for a trading strategy and you will always get one

It cannot say no

Ask a model for a mean-reversion strategy on European mid-caps and you will receive one. Ask for a strategy that exploits Tuesday afternoons and you will receive that too, with entry rules, exit rules and a paragraph about why it is plausible.

The second one is not a failure. Producing fluent, plausible, well-structured text on request is the entire function, performed correctly. The model has no mechanism for concluding that a request describes nothing real, because it has no way of checking — it is not looking at any data when it answers you.

So the response you get carries no information about whether the idea is any good. It carries information about whether the idea is *articulable*, which is a much lower bar and one that almost everything clears.

Most people know this in the abstract. What trips them is that the plausible-sounding rationale attached to the strategy feels like evidence. It reads exactly like the reasoning in a paper that found a real effect, because it was trained on those.

It cannot count

This is the one that matters more, and almost nobody talks about it.

You ask for a strategy. It is fine, not exciting. You ask for a variation. Then another. Then you try a different asset class, a different holding period, a shorter lookback. Over a week you have maybe twenty candidates, and you keep the one that backtested best.

You have just run a search over twenty variants, which is exactly the process that produces false positives, and nothing in the conversation recorded it. The model does not know how many it gave you — earlier turns fall out of context, and separate sessions never shared one. Ask it directly how many strategies it has suggested and it will guess, politely.

And twenty is the small case. Once the model has tool access and can run backtests itself, the number goes into the hundreds inside a single afternoon, and the only artefact that survives is the one that scored highest.

The trial count is the single largest input into whether a backtested result means anything. Chat is a medium that structurally destroys it.

Backtested and hypothetical results are not indicative of future performance. Nothing here is investment advice.

What language models are genuinely good at here

Ending the piece at "so do not use them" would be lazy and wrong. There are parts of quantitative research where they are legitimately excellent.

• Turning a vague intuition into a precisely specified rule. Going from "buy when it has been oversold for a while" to something with a defined lookback, threshold and exit is real work, and they are very good at it.
• Writing and debugging the code. Reliable, fast, and the failure modes are visible when it does not run.
• Explaining a concept you have hit for the first time, so you know what to check next.
• Generating candidates. Breadth is a genuine advantage — the problem is never that a model produced too many ideas, it is that nobody counted them.

Notice that all four are generation and articulation tasks. None of them is adjudication. That split is not about current model capability and will not be fixed by a better model, because the missing input is a fact about your process rather than a fact about markets.

What to do if you are going to do this anyway

You probably are, and that is a reasonable choice. Three habits make it much less dangerous, and none of them requires any tooling.

1. Keep a tally. A text file with one line per strategy you have asked for, including the ones you abandoned after ten seconds. Those count. When you eventually find something that looks good, the length of that file is the context you need to read it against.
2. Choose your held-out period before your first prompt, and do not test anything against it until you have exactly one candidate left. Not after each idea — once, at the end.
3. Ask the model to argue the other side in a fresh session, without telling it that the strategy is yours or that it backtested well. It is much better at finding problems when it is not also the author.

That last one is a manual version of a structural point: the same system should not both propose and judge. Doing it in two sessions is a weak approximation, and it is still far better than doing it in one.