Skip to content

#452 Data Generator private package to generate test data from Dictionaries and Schemas - #454

Open
joneubank wants to merge 10 commits into
feat/iterative-validatorsfrom
feat/data-generator
Open

#452 Data Generator private package to generate test data from Dictionaries and Schemas#454
joneubank wants to merge 10 commits into
feat/iterative-validatorsfrom
feat/data-generator

Conversation

@joneubank

Copy link
Copy Markdown
Contributor

Summary

Adds a private package for Lectern data-generator which can generate data that conforms to a Lectern dictionary or schema.

  • Field generators produce typed values respecting field restrictions (codeList, regex, required, range)
  • Record and schema generators handle uniqueness constraints and field dependencies
  • Dictionary generator coordinates cross-schema foreign key relationships
  • File writer streams generated data to TSV/CSV files

Issues

Fully implement typed value generation for boolean, integer, number, and
string schema fields, with deterministic seeding, conditional restriction
resolution, and conflict reporting when restrictions cannot be reconciled.

- Add `FieldGeneratorResult<T>`, `FieldGeneratorFailureData<T>`, `FieldGeneratorOptions`, and `FieldGenerator<TField>` types
- Export `testConditionalRestriction` from `@overture-stack/lectern-validation`
- Add `@overture-stack/lectern-validation` as a dependency of `data-generator`
- Add test suites for `fieldGenerators`, `resolveRestrictions`, and `restrictionReducers` (93 tests passing)
- reorganizes generation files into /fields and /records dirs
- includes field generation order based on conditional restriction relationships
- includes ability to provide values from a parent entity to help satisfy FK restrictions
…is not required

Based on the presence of a `required:true` restriction. If required, always return a value. If not required, return an empty (undefined) value at a default 25% rate.

- the rate can be specified in the generator options. rate will be between 0 and 1, with 0 being never empty and 1 being always empty.
- record generator also gets the emptyRate option and this is threaded into the field generators
- all tests are updated to have NO_EMPTY (emptyRate: 0) added to their generators to ensure they are testing a generated value and not a potentially empty result
- empty results are determinsitic based on seed
- Fix Set mutation in extractFieldDependencies: build filtered set instead of deleting from Set during iteration
- Fix cycle detection in resolveGenerationOrder: only lump genuinely cyclic fields in fallback tier; independent zero-in-degree fields now correctly get their own tier
- Optimize resolveGenerationOrder to O(N+E) using a ready queue (Kahn's algorithm as intended)
- Replace fc.sample with lightweight xorshift32 seeded draw in resolveForeignKeyOverrides
- Rename mergedRange/effectiveRange to validatedRange/fallbackRange in resolveNumericConstraints
- Extract buildRequiredEmptyConflict helper and apply required+empty conflict check to all field generators (was silently ignored for numeric and string fields)
- Guard resolveArrayLength against min>max when integer exclusive bounds produce an impossible range
- Narrow reduceRegex return type from RestrictionRegex|undefined to string|undefined; simplify filterCodeListByRegex accordingly
- Add tests for cycle+independent field interaction and impossible array length exclusive bounds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant