AgentXray Blog Editorial standards

How we decide

Every threshold below is read out of the code that enforces it. If an engineer changes a rule and forgets this page, a test goes red — because the page has no numbers of its own to forget. Rules we do not enforce are not listed, however comparable they would make us look.

How long do we watch before judging a change?

14 days after, against the same window before

Amazon's fee events settle in roughly 7.5 to 13 days. A shorter window is not just noisier — it is measuring an incomplete picture, because some of the money has not been deducted yet.

Enforced in app/services/drafts/proof_loop.py

When is a verdict readable at all?

16 days after the change was applied

Settlement maturity plus two days: one because the day of the change is excluded, one because Amazon's report for a day lands the following morning.

Enforced in app/services/drafts/proof_loop.py (matured_on)

What does a change get judged ON?

Net contribution — orders × contribution per unit − spend

Never ACoS. ACoS improves when you cut a profitable arm, so a loop that optimised toward it would report a win for a change that cost the seller money. When the contribution cannot be priced, the verdict is withheld rather than falling back to sales-minus-spend.

Enforced in app/services/drafts/proof_loop.py (_net_contribution)

How much data before we call a conversion change real?

3 percentage points, and at least 50 sessions

Below that a ratio is a coincidence with a decimal point. The verdict is allowed to say inconclusive, and does.

Enforced in app/services/drafts/proof_loop.py

How many clicks before we suggest a negative keyword?

10 clicks, or spend past one order

At a 15% conversion rate, seeing zero orders in two clicks happens about 72% of the time. Acting on that is not pruning waste, it is turning off keywords at random.

Enforced in app/services/drafts/ppc_draft.py

How fresh is too fresh to read?

The last 2 days are excluded

Attribution lags 12 to 48 hours. Spend is instant and orders are not, so looking at the freshest days and concluding an arm dropped is a measurement error, not a finding.

Enforced in app/services/amazon/ppc_arm_metrics.py

When do we refuse to report a ratio?

Below 2 orders

One order makes any ratio look decisive. We would rather show nothing than a number that cannot survive its second data point.

Enforced in app/services/amazon/ppc_arm_metrics.py

When do we say a fee rate has moved?

At least 20 settled units, and a split-half gap over 15%

Below that it is anecdote, not economics. Above it, the window is split in half and the two halves compared — which is how a referral rate that changed mid-month is reported as a change instead of hidden inside an average.

Enforced in app/services/amazon/economics_from_settlement.py

What can block a change from reaching Amazon?

Only what Amazon itself rejects — a bid outside 0.02–1000.00, a zero budget, a state that does not exist

The economics warn; they never block. Sellers launch products, clear stock and buy market share for reasons this software cannot see, and a tool that overrode them would be wrong more often than it was right. A test pins this so it cannot be quietly 'improved' into a guardrail.

Enforced in app/services/drafts/ppc_draft.py

Can the assistant apply its own suggestion?

No — and not as a policy, as a structure

The connector token that drafts a change cannot authenticate the endpoint that applies one. Presenting it there returns 401: at that endpoint the token is not an identity at all. A test exists whose only job is to go red if someone widens that guard for convenience.

Enforced in tests/test_ppc_airlock.py

How far back can you read my settlement history?

89 days

Amazon's Reports API refuses to list settlement reports created more than 90 days ago: asking for 730 days returns 400 InvalidInput — "RequestedFromDate is more than 90 days old" — while 89 days returned 14 reports. So a deeper settlement history is not something any tool can offer from this endpoint, ours included. We would rather have 89 days of what Amazon charged than two years of what it estimated.

Not our rule — measured against Amazon SP-API getReports, measured against a live seller account on 2026-08-21.