Skip to content

[WIP] askrene-getroutes adaptive bounds - #9380

Draft
Lagrang3 wants to merge 8 commits into
ElementsProject:masterfrom
Lagrang3:xpay-adaptive-bounds
Draft

[WIP] askrene-getroutes adaptive bounds#9380
Lagrang3 wants to merge 8 commits into
ElementsProject:masterfrom
Lagrang3:xpay-adaptive-bounds

Conversation

@Lagrang3

@Lagrang3 Lagrang3 commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

After #9150, some flaws in askrene came to light. I've documented as much as I saw appropriate
in the code and commit messages, but for more details there was a thought process explained in #9282.

TLDR:

  • we update fake channeld to simulate an evolving network, liquidity is not just assigned but it moves when a payment succeeds,
  • in askrene's probability cost function we drop the assumption that the cost of X is not far from the cost of X+fees, we instead pessimistically assume a 1% fee attached to every amount,
  • min/max bounds are adaptively relaxed when there is evidence of our knowledge being wrong,
  • we spice a little bit the min/max bound computation with an exponential time evolution operation,
  • added unit tests,
  • re-enable fake channeld tests.

@Lagrang3 Lagrang3 added this to the v26.09 milestone Aug 5, 2026
@Lagrang3
Lagrang3 marked this pull request as draft August 5, 2026 07:07
Improved the fakenet simulation by moving funds after a payment
succeeds.

This triggers a bug in xpay. At PAYING Node ElementsProject#8 xpay fails after trying
many 1msat routes:
```
error: {'code': 209, 'message': "Timed out after after 842 attempts.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102199msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102197msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102196msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102195msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102194msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102193msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102192msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102191msat.
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198102190msat
...
We got temporary_channel_failure for 45210x2134x44171/0, assuming it can't carry 198101779msat.
Payment of 9996982msat reached destination, but timed out before the rest arrived..
Payment of 998000msat reached destination, but timed out before the rest arrived..
Payment of 3000000msat reached destination, but timed out before the rest arrived..
Payment of 197000000msat reached destination, but timed out before the rest arrived..
Payment of 91005018msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived..
...
Payment of 1msat reached destination, but timed out before the rest arrived..
Payment of 1msat reached destination, but timed out before the rest arrived.. "}
```

A lower bound learned for channel 45210x2134x44171 at a previous payment
is to blame. xpay thinks it has enough liquidity for ~198ksat 100% sure
and it keeps trying and failing, while refine produces 1msat alternative
payment routes after every MCF computation because fees don't fit even
for our current knowledge. As a matter of fact the computed routes are
attributed a very low probability of success at around ~0.4%. This is a
case for which our assumption that "the cost of X+fees is not much
greater than the cost of X" breaks down.

To fix this:
-  we need xpay to relax that lower bound. The multiple failures are
empirical evidence that our knowledge is wrong and we should correct it,
- the probability cost function should add a buffer for hypothetical
  fees, ie. New_cost(x) = Old_cost(x + fees).

Changelog-None

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
When the known max and known min are about the same value, the
probability cost of sending x=value is zero and MCF will try it.
It doesn't take into account the possibility that when we hit that
channel the actual flow is x+fees. We adjust the min/max bounds
by a factor of 1/1.01 like if we had a 1% reserve for fees.

Changelog-Fixed: askrene-getroutes: Account for a worst case 1% fee in the flow amount when computing probability costs.

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
@Lagrang3
Lagrang3 force-pushed the xpay-adaptive-bounds branch from 8ac9c19 to 7bca9fb Compare August 5, 2026 11:27
Changelog-None

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-None

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
@Lagrang3
Lagrang3 force-pushed the xpay-adaptive-bounds branch 3 times, most recently from d1727d5 to 6d2ff19 Compare August 6, 2026 21:11
@Lagrang3 Lagrang3 changed the title askrene-getroutes adaptive bounds [WIP] askrene-getroutes adaptive bounds Aug 6, 2026
@Lagrang3
Lagrang3 marked this pull request as ready for review August 6, 2026 21:19
@Lagrang3
Lagrang3 force-pushed the xpay-adaptive-bounds branch 2 times, most recently from 4009272 to 5860bf2 Compare August 7, 2026 07:10
Liquidity bounds are estimated from the observations gathered from
askrene-inform-channel. When combined, these intel entries are relaxed
base on the likelyhood of the observation with the prior knowledge, eg.
observing a channel failure when Pickhardt-Richer probability of success
is 99% indicates that it is likely that our prior knowledge was wrong.

We also use observations corresponding to the opposite direction of the
channel.

Changelog-Fixed: askrene-getroutes: liquidity bounds change with the evidence gathered from askrene-inform-channel

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
@Lagrang3
Lagrang3 force-pushed the xpay-adaptive-bounds branch 2 times, most recently from 2323e5b to c6d596a Compare August 7, 2026 11:30
Use an exponential function to relax the liquidity bounds in
askrene-getroutes. The choice of the time scale, currently lifetime=24h
is arbitrary, the choice of the function is arbitrary as well.
For whatever decay function we use it is important for consistency that
the semi-group property of time evolution be satisfied.

This time relaxation is useful because it allow us to smoothly discard
old information, eg. we can try routes that have failed on us in the past
and we don't get too optimistic about lower bounds we have learned a
while ago.

Another benefit is that we can combine channel intels giving older
entries "less importance" by applying the time evolution between intel
time gaps.

Changelog-None

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
We have set an exponential decay time for liquidity bounds with a
lifetime of 1 day, it halves every 16 hours. This parameter can be
tuned. But I think throwing away entries after 1 hour is too aggresive.
Better to remove them after 1 week. In that time it will be halved 10
times.

Changelog-None

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
@Lagrang3
Lagrang3 force-pushed the xpay-adaptive-bounds branch 3 times, most recently from a2357db to 6505622 Compare August 8, 2026 11:04
The 3 channel intel types, constraint_min, constraint_max and
impressions, are basically the same type. Instead of doing pointer magic
define a single channel intel type to gather these 3 cases for the
internal use.
The JSON API still makes a distinction between them.

Changelog-None

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
@Lagrang3
Lagrang3 force-pushed the xpay-adaptive-bounds branch from 6505622 to 1524935 Compare August 10, 2026 07:43
@Lagrang3 Lagrang3 modified the milestones: v26.09, v26.12 Aug 10, 2026
@Lagrang3
Lagrang3 marked this pull request as draft August 10, 2026 08:41

@ksedgwic ksedgwic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a concept-level review: for each of the PR's main ideas I
offer a concept ack or nack — a position on the idea itself,
while explicitly not having done the line-level work yet. That
detail pass can follow once the direction settles. Context: I run
a production rebalancer (clboss + the xrebalance plugin) built on
askrene's layer API, so this PR lands on semantics it depends on
daily. Overall the direction aligns with problems I have hit in
production — fee headroom and native knowledge-fading have both
been on my wish list. Below is a per-concept summary; each row has a pinned
comment thread for discussion so topics can be resolved
independently.

# concept where concept verdict
1 fakenet moves liquidity on payment success channeld_fakenet.c concept ack
2 fee headroom in MCF bounds mcf.c concept ack, with a suggested refinement
3 bounds from timestamp-ordered event replay route_query.c concept ack for observations, concern for assertions
4 surprise-weighted relaxation on failures route_query.c concept ack for observations, same assertion concern
5 exponential time decay of bounds route_query.c concept ack, two conditions
6 only our half of local channels in local layer askrene.c concept nack as implemented, alternatives suggested

The one design distinction all the concerns below reduce to: layer entries
today carry two different kinds of content. Observations
experiment outcomes ("an HTLC of X failed here at time T") — which
the replay/relax/decay machinery genuinely improves. And
assertions — facts the caller states that askrene cannot know
("this channel's deliverable amount is exactly X, I just read it
from listpeerchannels"; "do not plan more than X through this
channel") — which reinterpretation can only degrade, because the
caller is the authority. The PR treats everything in constraint
space as an observation. Notably, channel_update fields keep their
assertion semantics under this PR, so consumers can smuggle facts
through policy fields — that asymmetry is the clearest sign that
constraints lost a class they used to have.

To make the discussion concrete, here is every way xrebalance
uses the layer API today, and what each usage would have to become
under this PR as-is. ("Mirror" = the fake inbound-copy channels the
plugin creates to route to a phantom destination for circular
rebalances — the layers-only node split discussed in #9286.)
Lifecycle: per-request = rebuilt into a transient layer each
request; persist+age = persistent layer, trimmed via askrene-age;
memory cache = kept in plugin memory and re-projected each request
(this bucket exists only because updates/disabled-nodes never age
server-side).

goal today (mechanism) lifecycle under this PR as-is
route only via chosen drain and fill channels update enabled=false on the rest per-request unchanged
limit amount drained per drain channel inform constrained @ cap+1 per-request update htlc_maximum_msat=cap
limit amount filled per fill channel cap the mirror's capacity per-request unchanged (rides mirror)
share exact fill-channel liquidity (local truth) inform unconstr+constr @ known per-request same + set capacity=known
enforce minimum part size (no dust slivers) update htlc_minimum_msat on mirror per-request unchanged
record failure-learned liquidity ceilings (obs) inform constrained @ failed amount persist+age native decay (as designed)
record success-learned liquidity floors (obs) inform unconstrained @ carried amt persist+age native decay (as designed)
never route channels proven gone (0x400a) inform constrained @ 1msat persist+age update enabled=false
avoid positive-inbound-fee peers (unpriceable) inform constrained @ 1msat persist+age update enabled=false
correct stale gossip fees/cltv (from failures) update fee/cltv override memory cache unchanged
avoid nodes blamed by node-level fails (0x2000) askrene-disable-node memory cache unchanged

Two patterns in that table worth pausing on. First: every row that
survives unchanged already lives in update/disable space; every row
that must move lives in constraint space. Second: the two
exclusion rows are pushed out of the only server-aged store and
into client-side bookkeeping — the opposite direction from where
the decay work says the design wants to go.

The goal is to reduce the number of interface bends in this
table over time, and several rows look like shared needs with xpay
(stale-gossip correction, node disables, gone-channel learning) —
input from other layer-API consumers would be valuable here; cc
@daywalker90 in case sling's usage rhymes.

One structural note: the "adaptive liquidity bounds" commit
currently carries three separable decisions (the replay engine, the
relaxation rule, and the local-channel half-drop); splitting them
would let the uncontested parts land while the rest iterates.

tal_free(mp);
}

static void move_funds(struct info *info,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fakenet moves liquidity on payment success — concept ack

Looks good; don't see any pitfalls and no difficulty adapting to it
on the xrebalance side.

* */
/* FIXME: This could prevent us from finding flows that fit tightly through
* channel capacities. */
static const double FLOW_FEE_ADJUSTMENT = 1.01;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fee headroom in MCF bounds — concept ack, with a suggested
refinement

Strong ack on the motivation, with production data to back it: MCF
packing flows to exactly known_max with fees added afterwards is
the mechanism behind the increase_flows BROKEN-line floods I
reported — a single circular getroutes emitted 103,887 BROKEN
lines in 26 seconds before failing, because saturating an arc is
the designed regime for rebalance requests. Headroom at the
solver level attacks the real cause.

Two reservations about the flat 1.01:

  • It is too big for the common case (typical path fees are
    0.01–0.1%), and the cost is paid precisely by tight-fit flows —
    the "drain this channel fully" requests rebalancers make. The
    test adjustments in this PR (20k -> 21k sat in test_max_htlc)
    show the effect.
  • It is too small for small parts on local channels: the per-HTLC
    additional cost (commitment-tx fee per HTLC) was ~7.4k msat/part
    on my node, which exceeds 1% for parts under ~740k msat, so the
    overflow regime survives there.

A refinement that avoids both without new API: the request already
states its fee budget, so g = (amount + maxfee) / amount is a
guaranteed-sufficient per-request factor — for the 300 ppm budgets
my rebalancer uses that is 1.0003 instead of 1.01, reclaiming almost
all of the sacrificed capacity; xpay-style budgets get more
headroom, as they should. The per-HTLC cost for local channels is
exactly known via additional_costs and could be included rather
than approximated. Related regardless of the constant chosen:
refine_flows still ignores increase_flows' false return and
logs per-flow-per-iteration — shrink-to-fit plus a rate-limited
log would close the remaining gap.

* However it would be nice to have a theoretically sound adjustment, eg.
* Maximum Likelyhood, if applicable.
* FIXME: unit test it */
static void get_bounds_adaptively(struct channel_intel *intelarr,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bounds from timestamp-ordered event replay — concept ack for
observations, concern for assertions

For askrene's own learned knowledge this is a genuine improvement:
under intersection semantics one wrong pessimistic bound poisons a
channel until aged out, while recency-ordered replay resolves
contradictions the way a learner should, and folding events across
the direction pair is correct modeling. Concept ack there.

The concern: the replay treats every constraint as an
observation, but user layers also carry assertions — facts the
caller knows and askrene cannot (see the review summary). Concrete
consequence, reproducible test available: a user-layer cap on a
local channel (inform constrained below spendable, combined with
auto.localchans) is silently erased. On master the route is
refused citing the cap; on this branch the full amount routes
through the capped channel at probability_ppm: 1000000.
Mechanism: the local layer's exactness constraint carries timestamp
UINT64_MAX, sorts last in the replay, and
bounds_by_unconstrained raises max back to spendable — a
freshness race the caller cannot win by construction, however
recently the cap was informed.

Suggested direction: give constraints an assertion class. The
minimal version is one behavior change — internal
(UINT64_MAX-stamped) constraints clamp (intersect) rather than
override, which restores the master behavior for the case above.
The fuller version lets a user layer mark a constraint as an
assertion, exempt from relaxation and decay. Under the PR as-is,
consumers can still express facts through channel_update fields
(htlc_maximum as a flow cap, enabled=false as exclusion), which the
replay leaves alone — but that works only as long as updates stay
outside the replay, and the same staleness logic motivating this PR
applies to them equally, so it is a dodge rather than a design.

Two implementation notes while here: get_constraints now
allocates, merges, sorts, and calls exp() per channel×direction
inside the solver's hot loop even when no layer has intels for the
scid — an early-out for the empty case looks cheap and worthwhile
on ~80k-channel gossmaps; and same-second events replay in
asort's unstable order, so tie-heavy layers can produce
nondeterministic bounds.

/* It could be any number between 0 and 1. It represents the fraction of lower
* liquidity bound that we adjust when we find a failure. The smaller it is the
* more we trust previous knowledge. Similar to a "learning velocity" for AI. */
#define ASKRENE_FAILURE_RELAX_FRACTION 0.5

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

surprise-weighted relaxation on failures — concept ack for
observations, same assertion concern

Grading the surrender of optimism by how surprising the failure was
(via the Pickhardt-Richter probability) is a principled learning
rate, and matches what I converged on operationally from the
outside: wrong optimism causes repeated failures and should be
given up quickly; expected failures should not move beliefs. Ack
for observation-derived constraints.

One trap worth documenting (or fixing via the assertion class from
the replay thread): an exact assertion — min=max=X on a channel
whose capacity exceeds X, which is precisely how a caller states
"deliverable amount is exactly X" — enters the surprising-failure
branch and gets its min halved immediately, although no failure
ever occurred. The constraint was testimony, not an experiment.

Agree with the FIXME that a theoretically grounded update rule
would be preferable to the hand-tuned 0.5 eventually; either way
this function composed with the decay operator badly wants unit
tests — the interaction of relaxation, decay, and the direction
fold is where surprises will hide.

/* Lifetime of liquidity bounds is one day. "Lifetime" in the sense of the
* exponential time decay: the time it takes for the liquidity lower bound to be
* reduced by half is "lifetime" times ln(2) ~ 16 hours. */
#define ASKRENE_RELAX_TIME_SECS 86400

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exponential time decay of bounds — concept ack, two conditions

Native knowledge-fading is something I have wanted for a long
time: today xrebalance ages its persistent layer aggressively
client-side
(3h cutoff before every request) precisely because bounds otherwise
live forever. The asymmetric shape is right — stale pessimism
(ceilings) is the expensive kind, silently pricing away corridors
that have refilled, and it heals toward capacity here; stale
optimism fades harmlessly. The semigroup construction is tidy.

Two conditions for this to work for layer-API consumers:

  • Configurable lifetime, per layer or per request (0 = off).
    86400s is one policy among many: my operational tuning landed on
    ~3h for failure-learned ceilings; notably this PR itself moves
    xpay's aging horizon from 1h to 1 week — consumers keep choosing
    different horizons; a
    conservative operator may want days. Per-layer configuration
    composes naturally with consumers already splitting layers by
    knowledge class.
  • Assertions exempt (see the replay thread) — an assertion that
    cannot be relaxed but fades to nothing in hours is still not an
    assertion. A durable exclusion ("this channel is closed; gossip
    is stale") regains ~12% of its capacity within 3h under the
    current constant, resurrecting known-dead routes.

Also worth a sentence of guidance in the docs: how decay composes
with askrene-age (fading versus deletion, on independent clocks),
since consumers now hold both levers.

Comment thread plugins/askrene/askrene.c
* (not 95k) because the other side has 0 sat spendable.
* We avoid these troubles by submitting only our half of the
* channel to local_layer. */
return;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only our half of local channels in local layer — concept nack as
implemented, alternatives suggested

The incoherence being dodged is real: with reserves, spendable +
peer-spendable < capacity, so feeding both directions' exact
constraints into a model that couples directions via the capacity
complement produces contradictions. But the early return discards
more than the incoherent constraint — it also skips
layer_add_update_channel for the peer's direction, so an
unannounced local channel's inbound half has no channel_update at
all and becomes unroutable. Reproducible A/B test available:
unannounced channel, peer given balance, getroutes from the peer
to us with auto.localchans; master finds the route, this branch
fails with "Missing gossip for source: only known 0/1 channels".
That breaks any into-us routing over unannounced channels,
including the circular use case. The lost constraint was also
doing correct work on master (with an unfunded peer, master
correctly refuses citing "source has maximum capacity 0msat" —
that exactness came from the discarded half), and the fix now
depends silently on gossmods_from_listpeerchannels calling the
LOCAL direction first.

Suggested alternatives, in increasing order of ambition:

  • Narrow: keep the peer-direction channel_update, drop only the
    liquidity constraint. Un-breaks inbound routing; costs only the
    reserve-sized optimism on the inbound bound.
  • Better: keep both directions' constraints but treat
    local-channel exactness as per-direction assertions that are
    exempt from complement-coupling — reserves are exactly why the
    complement inference is invalid for these entries, and this
    dissolves the incoherence without discarding data.

This is also the piece that most deserves to be its own commit,
since it is a user-visible behavior change independent of the
replay machinery.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants