Skip to content

Add durable execution conformance test harness (.NET, step suite) - #2517

Open
GarrettBeatty wants to merge 4 commits into
devfrom
feature/durable-conformance-tests
Open

Add durable execution conformance test harness (.NET, step suite)#2517
GarrettBeatty wants to merge 4 commits into
devfrom
feature/durable-conformance-tests

Conversation

@GarrettBeatty

@GarrettBeatty GarrettBeatty commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Wires the .NET Durable Execution SDK into the language-neutral aws-durable-execution-conformance-tests runner. The runner deploys a SAM template, invokes each mapped Lambda, and validates the durable execution result and event history against language-agnostic requirement specs.

All nine conformance suites are implemented and passing. Handlers are ported from the prior internal .NET implementation in DurableExecutionsSDKTestingFramework/examples/dotnet, adapted so each handler project references the in-repo SDK directly (no SDK copy/pack step).

Verified end-to-end against real AWS (us-east-1)

Every suite deployed, invoked, and validated — 122 passed, 0 failed:

Suite Passed NotImplemented Total
step 20 0 20
wait 5 0 5
child 17 1 18
callback 19 0 19
invoke 15 1 16
wait_for_condition 13 0 13
wait_for_callback 15 0 15
parallel 21 1 22
map 17 3 20

The 6 NotImplemented requirements (3-14, 5-16, 8-15, 9-14, 9-19, 9-20) are all custom-serdes cases: the .NET SDK has no per-operation serializer slot — all payloads flow through the one registered ILambdaSerializer (matching the Java approach). They're declared under TestingMetadata.NotImplemented so the runner reports them as non-blocking gaps rather than silent omissions.

What's included

  • Conformance/ under Amazon.Lambda.DurableExecution.IntegrationTests:
    • template_<suite>.yaml per suite — maps each function to its requirement id(s) via TestingMetadata.TestDescription; deploys on the dotnet8 managed runtime with the makefile build method.
    • <suite>/<Handler>/ — one executable handler project per requirement (144 handlers total).
    • scripts/build_examples.sh (publish handlers → publish/<Fn>/, incl. the InvokeEchoTargetInvokeEchoTargetTenant tenancy alias), discover_suites.py (CI matrix), inject_execution_role.py (CI role injection).
    • README.md, .gitignore.
  • .github/workflows/conformance-tests.yml — per-suite matrix job: publish handlers → pip-install runner → OIDC creds → inject execution role → run suite → upload JUnit report.
  • Excludes Conformance/** from the parent test project's compile glob so the standalone handler types don't collide.

Notable handler details

  • Retry-across-invocation tests (step 1-11/1-13/1-14/1-15/1-18, child 3-7/3-12) count attempts across invocations, which the replay model can't hold in memory, so they use the AttemptsTable DynamoDB table.
  • invoke deploys two callee target functions (InvokeEchoTarget, InvokeFailTarget) invoked via AWSSDK.Lambda; ARNs wired via env vars + Fn::GetAtt. The tenancy test (5-8) reuses the echo binary under a second logical id (InvokeEchoTargetTenant, PER_TENANT isolation) produced by build-time aliasing.
  • child 3-11/3-17 and step 1-17/1-18 log via the durable context logger (records carry durableExecutionArn, which the runner's CloudWatch filter keys on — plain Console.WriteLine is dropped). 3-11 disables replay-aware filtering so the ReplayChildren re-execution is also observed (expects the line logged twice).

Before CI can run

Set the repo secrets CONFORMANCE_DEPLOY_ROLE_ARN (SAM-capable deploy role) and CONFORMANCE_LAMBDA_EXECUTION_ROLE_ARN, plus (optionally) the CONFORMANCE_AWS_REGION variable (defaults to us-east-1).

Notes

  • Test-only infrastructure — no shipping library change, so this carries the Release Not Needed label.
  • Upstream nit: on Windows the runner crashes printing under cp1252; PYTHONUTF8=1 works around it (does not affect the Linux CI runner).

Wire the .NET Durable Execution SDK into the language-neutral
aws-durable-execution-conformance-tests runner. The runner deploys a SAM
template, invokes each mapped Lambda, and validates the durable execution
result and event history against language-agnostic requirement specs.

- Conformance/ under the durable integration tests: per-suite
  template_step.yaml mapping functions to requirement ids via
  TestingMetadata.TestDescription, plus one executable handler project per
  requirement referencing the in-repo SDK directly.
- Full step suite implemented (1-1 .. 1-20); retry-across-invocation tests
  use a DynamoDB AttemptsTable. Verified 20/20 PASSED against real AWS.
- scripts/: build_examples.sh (dotnet publish -> publish/<Fn>/),
  discover_suites.py (CI matrix), inject_execution_role.py (CI role).
- conformance-tests.yml workflow: per-suite matrix, OIDC creds, pip-install
  runner, inject role, JUnit upload.
- Exclude Conformance/** from the parent test project's compile glob so the
  standalone handler types don't collide.
@GarrettBeatty GarrettBeatty added the Release Not Needed Add this label if a PR does not need to be released. label Aug 6, 2026
Add handlers + SAM templates for wait, child, callback, invoke, parallel,
map, wait_for_callback, and wait_for_condition, completing every durable
execution conformance suite for .NET. Handlers ported from the internal
DurableExecutionsSDKTestingFramework reference, with ProjectReferences
retargeted to the in-repo SDK.

Suite coverage (verified end-to-end against real AWS, us-east-1):
- step 20, wait 5, child 18, callback 19, invoke 16, wait_for_condition 13,
  wait_for_callback 15, parallel 22, map 20.
- 122 passed, 0 failed. 6 requirements declared NotImplemented (custom-serdes
  gaps: 3-14, 5-16, 8-15, 9-14, 9-19, 9-20) — the .NET SDK has no per-operation
  serdes slot; all payloads use the one registered ILambdaSerializer.

Notes:
- Retry-across-invocation tests (child 3-7, 3-12) use the AttemptsTable
  DynamoDB table, like the step suite.
- invoke deploys two callee target functions (InvokeEchoTarget, InvokeFailTarget)
  wired via AWSSDK.Lambda; the tenancy test reuses the echo binary under a
  second logical id (InvokeEchoTargetTenant) produced by build_examples.sh
  aliasing.
- child 3-11/3-17 log through the durable context logger (records carry
  durableExecutionArn, which the runner filters on); 3-11 disables replay-aware
  filtering so the ReplayChildren re-execution is also observed.
@GarrettBeatty
GarrettBeatty marked this pull request as ready for review August 6, 2026 19:08
@GarrettBeatty
GarrettBeatty requested review from a team as code owners August 6, 2026 19:08
@GarrettBeatty
GarrettBeatty requested a review from normj August 6, 2026 19:08
@@ -0,0 +1,137 @@
name: Durable Execution Conformance Tests

# Full-integration conformance run for the .NET Durable Execution SDK: publishes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will configure all of this after this pr is merged. i ran the conformance test locally and it passes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think i may need to modify this slightly to work with the existing buildspec etc but ill do that in a follow up pr

@@ -0,0 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this code was all approved already in the internal git farm repo by alex

The aws-dotnet-ci test-runner accounts are us-west-2, and durable execution is
available there, so align the conformance workflow's default region (still
overridable via the CONFORMANCE_AWS_REGION repo variable).
The runner + its test-requirements are pinned to @main, so new upstream
requirements are pulled automatically. With the default --fail-on (failed),
a new requirement with no .NET handler reports UNCOVERED and the run stays
green — silently missing coverage. Switch to failed+uncovered so a new
requirement turns CI red, prompting a handler (or a NotImplemented
declaration). Declared gaps report NOT_IMPLEMENTED and never block.

Also fix stale secret/variable names in the README CI section
(CONFORMANCE_* not TEST_ROLE_ARN/AWS_REGION) and document the coverage gate
+ what to do when it fires.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Release Not Needed Add this label if a PR does not need to be released.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant