When the Money Doesn't Arrive
A deterministic "one request at a time" gate blocked an AI agent's survival-critical retry. Can context help?
The Hugging Face attack informs what happens when a goal is specified but without intervening gates. Today I am contemplating gates that prevent the goals from being reached, and what is the balance between deterministic rules and state: preventing acts of commission while sacrificing potentially important acts of omission.
The discussion stems from tests on hybrid processes, and on size of context (in this case: injections on next steps), as we look at memory to store Enterprise knowledge on processes and manage agent self-improvement.
I'm pondering Sarvex Jatasra's post about the article "Procedural Graphs: Self-Evolving Execution Structures for LLM Agents" as a solution in lieu of workflows. Personally, I will not be designing a graph (which is context) for problems requiring deterministic workflows, even when those workflows can be triggered by an LLM reviewing state, and even when it does that accurately.
The test in a nutshell: a company goes bankrupt if it runs out of cash. When it asks for funding, the money takes 1–6 months to arrive never sooner. There's a rule against barraging: don't send a new request while one is still outstanding. So the agent is caught between two failure modes: ask too late and it dies waiting for slow money; retry too eagerly and it violates the no-stacking rule.
In the baseline, the agent checks state, the deterministic agent says don't ask twice. The agent is not given the goal (remaining funded), just the state check.
When should it ask again?
Ways to Give an Agent Procedural Knowledge
The study tested using "memory" vs "process graph".
The paper runs three relevant conditions (Table 3, the efficiency ablation):
No graph (baseline) — the agent decides from its raw, accumulating trajectory. This is the "let the model figure it out from history" default.
Full graph, injected/generative — the entire procedural structure is dumped into context each step. This is exactly the "memory" instinct I’ve been worried about, which is not limited to just procedure graphs: shove the whole thing in a big chunk and hope relevance sorts itself out.
Subgraph, 2-hop generative (their method) — only the localized neighborhood around the active node.
Recursive learning in a sandbox can lead to iterative improvements without human in the loop, and the article goes into the benefits of a process graph over use of memory. It also varied the size of the context input. The study varied the scope of injected guidance (full graph vs. a 2-hop slice). Full-graph injection made ALFWorld worse than no graph at all. In the end, we are just substituting one "suggestion" for another though.
The subgraph process layer (Condition 3) helped. The process graph was not fail-proof, but it did add another deterministic step via the graph. The graph moved survival from near-zero to the 80s by getting guidance to the agent at the decision point, not before it. Memory-in-context (full-graph, Condition 2) was the baseline that failed.
I prefer a defined workflow, with defined gates.
What deterministic steps or procedure graph can't do is resolve non-arrival: default risk, negotiation, follow-up. It can only retry, and there is nothing that clearly bounds retries.
This got me thinking, is there anything even a deterministic set of steps could do to mitigate in the scenario laid out?
The Replacement for Human Judgment
Not really, not for the question the scenario turns on: what to do when the money doesn't arrive. That requires human-in-the-loop, and here's why: judgment. Here, we have a concrete example where a person is the right means to decide what to do next if the funding does not arrive. But we don't want to send a firehose of requests when an end state or its timing is uncertain.
1) what happens if it never arrives? the organization goes bankrupt
2) is an agent going to perform followup? (this could be a separate set of tasks, but assume an agent deterministic or not is not the best tool to negotiate or follow up on a funding request)
3) agents do not have judgment, even when they are effective at evaluating state or outcome
Judgment requires evaluation of risk. And risk is always a function of two things: the likelihood of something bad occurring and its impact if it occurs.
If you really wanted to set up judgment, you would set up a probability of failure increasing over time and a step can check it.
My Current Takeaways
On Context, On Localized Tests to Improve Performance, on Human Judgment, and On Risk
The fix is outside the scope of this discussion, but it is worth noting the tradeoffs the study showed between the size of the context window and the outcome, and the unexpected token costs of iterating. Further, some of the more important takeaways are:
1) context implies what is available, not what is enforced. All those instituting contextual graphs like semantic layers etc should take note
2) risk is not modeled into your workflow unless you make it explicit
3) modeling is safe in test to deliver improved context, but in Production the lessons learned would be catastrophic, and not favorable to incremental self-improvement
4) if risk can be modeled enough to determine when escalation is needed, that should be a place to put human-in-the-loop, even if the gates otherwise do not include it.
How would you have solved this problem?
