Add durable execution conformance test harness (.NET, step suite) - #2517
Open
GarrettBeatty wants to merge 4 commits into
Open
Add durable execution conformance test harness (.NET, step suite)#2517GarrettBeatty wants to merge 4 commits into
GarrettBeatty wants to merge 4 commits into
Conversation
Wire the .NET Durable Execution SDK into the language-neutral aws-durable-execution-conformance-tests runner. The runner deploys a SAM template, invokes each mapped Lambda, and validates the durable execution result and event history against language-agnostic requirement specs. - Conformance/ under the durable integration tests: per-suite template_step.yaml mapping functions to requirement ids via TestingMetadata.TestDescription, plus one executable handler project per requirement referencing the in-repo SDK directly. - Full step suite implemented (1-1 .. 1-20); retry-across-invocation tests use a DynamoDB AttemptsTable. Verified 20/20 PASSED against real AWS. - scripts/: build_examples.sh (dotnet publish -> publish/<Fn>/), discover_suites.py (CI matrix), inject_execution_role.py (CI role). - conformance-tests.yml workflow: per-suite matrix, OIDC creds, pip-install runner, inject role, JUnit upload. - Exclude Conformance/** from the parent test project's compile glob so the standalone handler types don't collide.
Add handlers + SAM templates for wait, child, callback, invoke, parallel, map, wait_for_callback, and wait_for_condition, completing every durable execution conformance suite for .NET. Handlers ported from the internal DurableExecutionsSDKTestingFramework reference, with ProjectReferences retargeted to the in-repo SDK. Suite coverage (verified end-to-end against real AWS, us-east-1): - step 20, wait 5, child 18, callback 19, invoke 16, wait_for_condition 13, wait_for_callback 15, parallel 22, map 20. - 122 passed, 0 failed. 6 requirements declared NotImplemented (custom-serdes gaps: 3-14, 5-16, 8-15, 9-14, 9-19, 9-20) — the .NET SDK has no per-operation serdes slot; all payloads use the one registered ILambdaSerializer. Notes: - Retry-across-invocation tests (child 3-7, 3-12) use the AttemptsTable DynamoDB table, like the step suite. - invoke deploys two callee target functions (InvokeEchoTarget, InvokeFailTarget) wired via AWSSDK.Lambda; the tenancy test reuses the echo binary under a second logical id (InvokeEchoTargetTenant) produced by build_examples.sh aliasing. - child 3-11/3-17 log through the durable context logger (records carry durableExecutionArn, which the runner filters on); 3-11 disables replay-aware filtering so the ReplayChildren re-execution is also observed.
GarrettBeatty
commented
Aug 6, 2026
| @@ -0,0 +1,137 @@ | |||
| name: Durable Execution Conformance Tests | |||
|
|
|||
| # Full-integration conformance run for the .NET Durable Execution SDK: publishes | |||
Contributor
Author
There was a problem hiding this comment.
will configure all of this after this pr is merged. i ran the conformance test locally and it passes
Contributor
Author
There was a problem hiding this comment.
i think i may need to modify this slightly to work with the existing buildspec etc but ill do that in a follow up pr
GarrettBeatty
commented
Aug 6, 2026
| @@ -0,0 +1,19 @@ | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
Contributor
Author
There was a problem hiding this comment.
this code was all approved already in the internal git farm repo by alex
The aws-dotnet-ci test-runner accounts are us-west-2, and durable execution is available there, so align the conformance workflow's default region (still overridable via the CONFORMANCE_AWS_REGION repo variable).
The runner + its test-requirements are pinned to @main, so new upstream requirements are pulled automatically. With the default --fail-on (failed), a new requirement with no .NET handler reports UNCOVERED and the run stays green — silently missing coverage. Switch to failed+uncovered so a new requirement turns CI red, prompting a handler (or a NotImplemented declaration). Declared gaps report NOT_IMPLEMENTED and never block. Also fix stale secret/variable names in the README CI section (CONFORMANCE_* not TEST_ROLE_ARN/AWS_REGION) and document the coverage gate + what to do when it fires.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the .NET Durable Execution SDK into the language-neutral
aws-durable-execution-conformance-testsrunner. The runner deploys a SAM template, invokes each mapped Lambda, and validates the durable execution result and event history against language-agnostic requirement specs.All nine conformance suites are implemented and passing. Handlers are ported from the prior internal
.NETimplementation inDurableExecutionsSDKTestingFramework/examples/dotnet, adapted so each handler project references the in-repo SDK directly (no SDK copy/pack step).Verified end-to-end against real AWS (us-east-1)
Every suite deployed, invoked, and validated — 122 passed, 0 failed:
The 6
NotImplementedrequirements (3-14, 5-16, 8-15, 9-14, 9-19, 9-20) are all custom-serdes cases: the .NET SDK has no per-operation serializer slot — all payloads flow through the one registeredILambdaSerializer(matching the Java approach). They're declared underTestingMetadata.NotImplementedso the runner reports them as non-blocking gaps rather than silent omissions.What's included
Conformance/underAmazon.Lambda.DurableExecution.IntegrationTests:template_<suite>.yamlper suite — maps each function to its requirement id(s) viaTestingMetadata.TestDescription; deploys on thedotnet8managed runtime with the makefile build method.<suite>/<Handler>/— one executable handler project per requirement (144 handlers total).scripts/build_examples.sh(publish handlers →publish/<Fn>/, incl. theInvokeEchoTarget→InvokeEchoTargetTenanttenancy alias),discover_suites.py(CI matrix),inject_execution_role.py(CI role injection).README.md,.gitignore..github/workflows/conformance-tests.yml— per-suite matrix job: publish handlers → pip-install runner → OIDC creds → inject execution role → run suite → upload JUnit report.Conformance/**from the parent test project's compile glob so the standalone handler types don't collide.Notable handler details
AttemptsTableDynamoDB table.InvokeEchoTarget,InvokeFailTarget) invoked viaAWSSDK.Lambda; ARNs wired via env vars +Fn::GetAtt. The tenancy test (5-8) reuses the echo binary under a second logical id (InvokeEchoTargetTenant,PER_TENANTisolation) produced by build-time aliasing.durableExecutionArn, which the runner's CloudWatch filter keys on — plainConsole.WriteLineis dropped). 3-11 disables replay-aware filtering so the ReplayChildren re-execution is also observed (expects the line logged twice).Before CI can run
Set the repo secrets
CONFORMANCE_DEPLOY_ROLE_ARN(SAM-capable deploy role) andCONFORMANCE_LAMBDA_EXECUTION_ROLE_ARN, plus (optionally) theCONFORMANCE_AWS_REGIONvariable (defaults tous-east-1).Notes
✅under cp1252;PYTHONUTF8=1works around it (does not affect the Linux CI runner).