diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index e0775b351..4998ccfbf 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -356,6 +356,24 @@ cases for these three BSON types — they fell to `default: return nil`. | An unquoted negative number in an XPath constraint fails to parse: `where [Amount > -7]` → `Parse error: extraneous input '7' expecting {',', ')'}`. Reported as "negative numeric literals truncate (`-7` becomes `-`)" | `xpathWord` — the name-part rule inside XPath — is a **negated token set** that did not exclude `MINUS`, so the sign was consumed as a name word and the digits were left stranded (hence the truncation appearance). The lexer deliberately keeps `-` out of `NUMBER_LITERAL` (a leading sign there mis-tokenises `$x -2`), leaving negation to the parser; the general grammar has `unaryExpression` for this and the XPath grammar simply never got the equivalent | `mdl/grammar/domains/MDLPage.g4` (`xpathValueExpr` gains `MINUS xpathValueExpr`; `MINUS` added to the `xpathWord` exclusion set), `mdl/visitor/visitor_xpath.go` (`buildXPathValueExpr`), `mdl/visitor/visitor_page_v3.go` (`xpathExprToString` emits `-7`, not `- 7`) | **The grammar fix alone is worse than the bug.** With the parser accepting `-7` but the XPath AST builder having no case for the new alternative, the constraint parses and silently serializes to `[Amount > ]` — a dropped operand instead of a loud parse error. Caught only because the visitor has a round-trip helper; the microflow write path uses `GetText()` and looked fine. **When adding a grammar alternative, check every consumer of that rule, not just the one your repro exercises.** **Scope correction**: the finding's own example (`addDays([%CurrentDateTime%], -7)`) still fails — `addDays` is a *microflow expression* function, not an XPath one, and it fails `CE0161` with a POSITIVE argument too, so the sign was never its problem. Repro `mdl-examples/bug-tests/it-18-xpath-negative-literal.mdl`; A/B on Mendix 11.12.1: pre-fix the script does not parse, fixed binary writes it and `mx check` reports 0 errors. issuetracker #18 | | Text painted by an **Atlas topbar widget is invisible in a dark theme** — the language selector measures ~1.13:1 contrast, glyph pixels spanning 4 luminance values out of 255. A theme override exists and *names the right element*, so it looks handled | Two separate mistakes stacked. (1) **Specificity**: Atlas's own rule is `.navbar-brand .widget-language-selector .current-language-text` at (0,3,0); a bare `.current-language-text` at (0,1,0) never wins, and only appears to on layouts that do not nest the selector under `.navbar-brand`. (2) **Wrong value**: `color: inherit` inherits *body ink*, which is dark, while the rail is dark in both palettes — so even at the winning specificity it measures 1.00:1. Atlas paints from `--bg-color-secondary` with a `#fff` fallback because it assumes a dark rail | `cmd/mxcli/theme/assets/*/files/theme/web/_mxcli-atlas-map.scss` (the "Atlas fixups" block) | Re-declare **Atlas's own selector shape** so the weights match and source order decides, and resolve the colour through the rail token (`var(--mxt-rail-ink-active, var(--mxt-rail-ink))`) rather than `inherit`. List the bare and the `.navbar-brand`-nested selectors together — each is matched at its own specificity, so one rule covers both layouts. **Generalisable — the shape to look for**: a guard that names the right element is not evidence it applies. Read the *winning* declaration (`CSS.getMatchedStylesForNode` in DevTools, or the computed value) instead of the one you wrote. **Measure contrast, not colour**: reading `getComputedStyle(el).color` once and seeing a plausible value proves nothing — compute the WCAG ratio against the first non-transparent ancestor background, which is what turns "looks fine" into 1.13 vs 19.47. Reported from the RssReader test build; tests in `cmd/mxcli/theme/theme_test.go`, verified in a browser at 17.79:1 light / 19.47:1 dark | +| A loop's variable used after `end loop;` passes `mxcli check`, then `mx check` fails `[error] [CE0108] "Variable 'item' is defined but not in scope at this location."` at the referencing activity. Applies to the **iterator** and to anything the body introduces (a `retrieve`, a `$X = create …`, a call output) | Nothing tracked loop-variable *visibility*. MDL052 already covered the sibling rule — names are unique across the whole microflow (CE0111) — and the wording of that rule ("scoped to the WHOLE microflow") reads as if the variable is readable flow-wide. Uniqueness and visibility are different: the name is reserved everywhere, readable only inside the loop body | `mdl/executor/validate_microflow_loop_scope.go` (new `MDL053`, wired from `microflowValidator.validate`), skill `.claude/skills/mendix/write-microflows.md` | Map each loop-scoped name to the loop whose **own** body introduces it (nested loops keep their own names), then walk the flow with the set of enclosing loops and flag any reference from outside the owner. **A name claimed by two loops is dropped, not reported** — that is the MDL052/CE0111 case, and without the guard the MDL052 negative example started failing for the wrong reason: the first loop's own use of `$R` was blamed on the second loop's claim. **Generalisable**: a rule keyed by variable *name* needs an ambiguity escape hatch whenever another rule exists precisely because names can collide. Both flavours verified against mxbuild 11.12.1 (2 × CE0108 in one probe). Repro `mdl-examples/bug-tests/loop-variable-out-of-scope.fail.mdl`; tests `mdl/executor/validate_microflow_loop_scope_test.go`. Found while working the sudoku findings, but **not** one of them — the numbered finding it was filed under is an app bug in that project, not an mxcli defect | +| `mxcli oql` silently omits a whole column: a `select A, B …` renders only `A`, and the JSON output has no `B` key at all — no error, no empty column. Reproduces whenever the value of `B` is **null in the first row** | The runtime omits a null-valued column from a row's JSON object entirely, and `parseOQLFeedback` took the column list from `extractColumnOrder(rows[0])` — one row, chosen for its key *order*. Every later row was then projected onto that short list, so the column vanished from the result rather than showing NULLs | `cmd/mxcli/docker/oql.go` (`parseOQLFeedback`, `mergeColumnOrder`, `hasOnlyKnownKeys`) | Union the keys of **all** rows, inserting each new key directly after the last key already known rather than appending — merging `[A, C]` with `[A, B, C]` must give `[A, B, C]`, not `[A, C, B]`, or a column that is null early in the result set jumps to the end of the table. The per-row re-scan is skipped when a row carries no unseen key, so the uniform case still costs one length check. **Generalisable**: any "take the shape from the first element" over a sparse encoding is a silent-wrong-answer bug, not a formatting bug — the output looks complete. Tests `TestParseOQLFeedback_ColumnUnionAcrossRows`, `TestMergeColumnOrder`; proven by stubbing the union back to first-row-only and watching the reported symptom return. sudoku #39, first half | +| `mxcli test` cannot run at all in a container without a Docker daemon — parsing, runner generation, model injection and the **entire mxbuild build** succeed natively, then `docker up` fails with "failed to connect to the docker API at unix:///var/run/docker.sock". So microflow tests are unavailable in exactly the environment mxcli targets (Claude Code web containers ship `/usr/bin/docker` with no daemon) | Only one step of the run was containerised — start the runtime and read its log — but it was wired directly to `docker compose`, with no seam for another way to run the app. `run --local` had all three pieces already (boot a standalone runtime, tee its log, restore) | `cmd/mxcli/docker/localapp.go` (new `StartLocalApp`), `cmd/mxcli/testrunner/runner_local.go` (new), `cmd/mxcli/testrunner/runner.go` (docker path extracted to `runDockerAndCapture`), `cmd/mxcli/cmd_test_run.go` + `main.go` (`--local`) | Give the run a seam — `runLocalAndCapture` / `runDockerAndCapture` — so both modes share parse/inject/parse-results/cleanup and differ only in how the app is started. **Two traps, both found by running it rather than reasoning about it.** (a) The runner reports via an **after-startup** microflow, so its LOG output happens DURING the start action, before the runtime's log subscriber attaches; registering the subscriber early is not possible (the runtime answers `LoggingException` pre-start). What actually carries the output is the JVM console tee, live from spawn — verified by A/B running with the early attach removed. (b) A **failing** test makes the runner return false, which makes the after-startup action fail, which makes `start` return an error — the first version reported that as a broken run and dumped a stack trace instead of the test report. A failed boot whose log shows a verdict is a normal outcome. Local runs use their own ports (8081/8091) and a `_test` database so a `run --local` dev loop can keep serving. Verified end-to-end in a daemon-less container: 2 passed; then 1 passed / 1 failed with exit 1, project restored. sudoku #41 | +| A solution's apps are served on one host with different ports, so they share a cookie jar (cookies key on host name and **ignore the port**) — logging into one silently replaces the other's `XASSESSIONID`. Giving each app a host name in **App Settings -> Configurations -> Application root URL** appears to do nothing under `run --local` | `runtimeConfigParams` builds the entire boot `update_configuration` payload from `LocalRuntimeOptions`, and `ApplicationRootUrl` was only ever populated from a `--hub` registration. The model's own value had no path into the payload — and since the admin action REPLACES rather than merges, nothing else could supply it either | `cmd/mxcli/docker/runlocal.go` (`configuredApplicationRootURL`, `applicationRootURLFrom`, `customHostRootURL`) | Read the setting off the project at boot and use it when no hub URL was assigned (hub wins — that URL is the one actually serving the app). **The trap is that a blank Mendix app already ships `ApplicationRootUrl = http://localhost:8080/`**, so "is set" does not mean "was chosen": honouring every value would change behaviour for every existing project and, under `--app-port`, advertise a port the app is not serving on. Only a **non-loopback host** is passed through, and a port that disagrees with `--app-port` warns. Serving under the host name needs no flag at all — the runtime accepts any `Host` and the client uses relative URLs (verified: 200 via `/etc/hosts`, nip.io and localtest.me); the setting matters only for the **absolute** URLs Mendix generates. **Generalisable**: before defaulting from a model setting, check what a blank project already has in it — a non-empty default makes "fall back to the model" a behaviour change, not a no-op. Verified end-to-end on 11.12.1: with `backend.local` configured, boot prints `Application root URL from configuration "Default"` and the app answers 200 on both the host name and the listen address. Tests `TestApplicationRootURLFrom`, `TestCustomHostRootURL` | +| A page bound to an attribute the entity **inherits** (`Person extends Administration.Account`, page binds `FullName`) passes `mxcli check --references` AND `mxcli lint`, then the real MxBuild fails `[error] [CE1613] "The selected attribute 'TaskBoard.Person.FullName' no longer exists."` The message reads as a deletion; it never existed there | Mendix stores a page's attribute reference against the entity that **declares** the attribute. `resolveAttributePath` qualified a bare name with `pb.entityContext` unconditionally, so a specialization that merely inherits the attribute got a dangling reference. Two independent resolvers had the same bug: the direct binding, and the final attribute of an association path (`resolveAssociationAttributePath`) | `mdl/executor/cmd_pages_builder_input.go` (`declaringEntityFor`, `entityAttributeOwners`), `mdl/executor/cmd_pages_builder_v3.go` (final attribute of the association path) | Walk the generalization chain and qualify with the first entity that declares the name. **Fix both resolvers** — a probe that only tested the direct case would have shipped half a fix; the reporter's own table already showed the associated case failing, and it did still fail after the first patch. Unknown names keep today's context qualification rather than being re-pointed, and a cyclic chain terminates. **Watch the new dependency**: attribute resolution now consults the domain models, and `getDomainModels` panics on a nil backend — several unit tests build a `pageBuilder` with neither backend nor cache, so the lookup bails out early instead. A/B on Mendix 11.12.1: pre-fix binary → CE1613 for the inherited column and 0 errors for the own column; fixed binary → 0 errors for both shapes. Repro `mdl-examples/bug-tests/todo-12-inherited-attribute-on-page.mdl`; tests `mdl/executor/cmd_pages_builder_inheritance_test.go`. mxcli-todo #12 | +| The SessionStart hook `mxcli init` writes cannot survive an idle reap: it is guarded on `test -x ./mxcli`, and `.gitignore` excludes that binary (~85 MB) on purpose. The container is reclaimed, the repo re-cloned without it, the guard fails and the hook no-ops through `\|\| true` — the next session has no mxcli, no MxBuild cache, no database, and no message saying so | The hook inlined the whole bring-up in one shell line, so "binary missing" could only be expressed as "skip". A hook line cannot reasonably do OS/arch detection and a download; something committed has to | `cmd/mxcli/init_hook.go` (`bootstrapScriptTemplate`, `writeBootstrapScript`, `sessionStartHookCommand`, `sessionStartHookMarkers`) | Emit a committed `.claude/bootstrap-mxcli.sh` that resolves OS/arch, fetches the binary when absent (`MXCLI_TAG` to pin), then runs the setup; the hook becomes `sh .claude/bootstrap-mxcli.sh \|\| true`. **Changing the hook command breaks dedupe**, which matched on the old command string — so `addSessionStartHook` now recognises *any* known marker and **rewrites the entry in place**, migrating an old project instead of leaving it with two hooks that both run. **Generalisable**: a guard whose condition is something you deliberately do not commit is a silent no-op waiting for a fresh clone — make the guard able to satisfy itself. Verified by reproducing the reap: moved `./mxcli` out of the project, ran the hook command verbatim, watched it re-download (88 MB, new mtime) and finish with "Setup complete … database ready". Tests `TestAddSessionStartHook_MigratesLegacyCommand`, `TestEnsureSessionStartHook_WritesFile`. mxcli-todo #2 | +| `mxcli exec -p app.mpr - <<'EOF' … EOF` fails with `Error reading file: open -: no such file or directory` — `-` is taken literally as a filename, so MDL cannot be piped or written as a heredoc and every ad-hoc script needs a temp file first | `exec` (and `check`) called `os.ReadFile(path)` directly, with no case for the conventional stdin spelling | `cmd/mxcli/mdlsource.go` (new `readMDLSource`, `mdlSourceLabel`), `cmd/mxcli/cmd_exec.go`, `cmd/mxcli/cmd_check.go` | One helper both commands share, so `check` gained the same spelling rather than only the reported one; `check` reports the source as `` instead of a bare `-`. Verified live: a heredoc through `exec` and a pipe through `check` both run. Tests `cmd/mxcli/mdlsource_test.go`. mxcli-todo #5 | +| `mxcli syntax` documents spellings the parser rejects, so an agent following the reference writes MDL that fails — `TEXTBOX … (Binds: Attr)` ("'Binds:' is no longer supported, use 'Attribute:' instead") and `DataSource: MICROFLOW Module.MF()` (a zero-arg microflow datasource takes NO parens, unlike RETRIEVE/CALL) | Nothing checks the `syntax` corpus against the parser. `make check-skill-mdl` validates MDL blocks in the skills and the docs site, but the `Syntax`/`Example` strings in `cmd/mxcli/syntax/*.go` are not covered, so a retired spelling can sit there indefinitely | `cmd/mxcli/syntax/features_page.go` (10 × `Binds:` → `Attribute:`, the datasource parens), `cmd/mxcli/syntax/retired_spellings_test.go` (new guard) | Fix the text **and** pin it: a table-driven test fails if a retired spelling reappears in any topic's Syntax or Example. It is a spelling guard rather than a parse — the snippets are fragments (a DATAVIEW body, a property line) that do not stand alone as statements, so they cannot just be fed to the parser. Proven by reintroducing `Binds:` and watching the test name the topic and field. **A third claim in the same report did not reproduce**: `CONTAINER (OnClick: SHOW_PAGE M.P(Param: $currentObject))` parses fine on current main, so only the two verified ones were changed. mxcli-todo #8 | +| `mxcli test … --local` (or any other `StartLocalApp` caller) fails with MxBuild's `the project file path should be an absolute path`, followed by a page of Windows sample requests, whenever `-p` is given a **relative** path | `ServeServer.Build` forwarded `ProjectFilePath` verbatim. `mxcli run` had learned to absolutize at the CLI layer (findings #17), but that fix lived in `cmd_run.go`, not in the code that talks to MxBuild — so the next caller re-hit it | `cmd/mxcli/docker/mxserve.go` (`ServeServer.Build`) | Absolutize `req.ProjectFilePath` in `Build` itself, the single place that talks to MxBuild, so no future caller can miss it; also resolve `LocalAppOptions.ProjectPath` in `applyDefaults` so `DeployDir` and the runtime log path are not derived from a relative value. Test by pointing a `ServeServer` at an `httptest` fake and asserting on the request body — the CLI-layer fix cannot be tested that way, which is part of why it did not generalise | +| `mxcli test --attach` fails with `reload_model failed: Authentication failed.` — after the test microflows have already been injected into the project | The M2EE admin API and the test endpoint are **different secrets**. `attach` built its `RuntimeController` with `M2EEOptions{Token: hs.Token}` — the endpoint token — instead of the runtime's admin password | `cmd/mxcli/testrunner/runner_attach.go` (`attach`), `cmd/mxcli/testrunner/handshake.go` (`Handshake`) | Carry `AdminPass` in the handshake alongside `Token` and pass that to `M2EEOptions`. The hosting `run --local` publishes it via `docker.LocalAppInfo` (the resolved value, not the package default, so a `--admin-pass` override still works). Whenever one process drives another's M2EE API, check which credential is being passed — `defaultLocalAdminPass` and any app-level token are unrelated | +| `alter page … set Editable = [expr]` (or `set Visible`) writes a project Studio Pro refuses to open: `StorageLoadException: Conditional editability settings has an invalid value '' for property Attribute`. `mxcli check` ✓ and `mx check` ✓ — neither inspects the stored value. The identical settings written by `create page` load fine | The ALTER path builds the `Forms$Conditional{Visibility,Editability}Settings` node by hand and wrote `Attribute: null`. `Attribute` is a **BY_NAME** `AttributeIdentifier`, so its unset value is the empty string, not null — exactly what the CREATE path already encodes via `codec.RegisterTypeDefaults(..., EmptyStringFields: []string{"Attribute"})`, whose comment records this same StorageLoadException from #627. Only the hand-built ALTER node missed it | `mdl/backend/pagemutator/mutator.go` (`setWidgetConditionalSettingMut`) | Write `{Key: "Attribute", Value: ""}`, not `nil`. **General rule: when one path hand-builds BSON that another path builds through the codec, diff the two encodings rather than eyeballing the hand-built one** — `mxcli bson dump --type page --object M.P` on a CREATE-authored and an ALTER-authored widget makes the divergence a one-line diff (key sets and values were otherwise identical). `SourceVariable` stays `nil`: it is BY_ID, where null *is* the absent value, so "null is wrong" is per-field, not a blanket rule. Test `TestSetWidgetConditionalSetting_AttributeIsEmptyString`; repro `mdl-examples/bug-tests/851-alter-page-conditional-attribute.mdl`. Issue #851 | +| A widget conditional using a function whose name is also an MDL lexer keyword — `visible: [trim($currentObject/Slug) != '']`, `[length(…) > 0]`, `empty`/`count`/`find` — **silently drops the whole property**; `mxcli check` ✓, `mx check` ✓, and the widget renders unconditionally visible. `toUpperCase`/`isMatch`/`contains` in the same position work | `xpathFunctionName` (MDLPage.g4) enumerated only `IDENTIFIER \| HYPHENATED_ID \| NOT \| TRUE \| FALSE \| CONTAINS`, so `trim(` never matched `xpathFunctionCall`. The enclosing `[...]` then failed to parse as an `xpathConstraint` and matched the generic `propertyValueV3` alternative instead, so the visitor set `Visible` (an array) rather than `VisibleIf`, and the builder's `else if pages.StaticVisibleExpression(...)` — which reads only bool/string — never fired | `mdl/grammar/domains/MDLPage.g4` (`xpathFunctionName`) + `mdl/executor/validate_widgets.go` (`validateConsumableConditional`) | Define `xpathFunctionName : xpathWord \| NOT` — `xpathWord` is a negated token set, so it self-maintains as the lexer gains keywords; an enumerated list reacquires this bug with the next promoted function name. Safe because `xpathFunctionCall` requires a following LPAREN and no `xpathStepValue` may be followed by one, so bare `empty` still parses as a path word. `NOT` is spelled out (xpathWord excludes it). **Also add the general guard**: MDL-WIDGET19 errors when `Visible`/`Editable` holds a value that is neither routed to `VisibleIf`/`EditableIf` nor a bool/string — that is the residue signature of any conditional the visitor could not build, so the next one fails loudly instead of vanishing. `make grammar` regenerates the parser (not committed). **Verify in a browser, not at `mx check`** — a dropped property is still a valid model, so `mx check` reports 0 errors before AND after; the symptom only exists at render time (see `verify-in-runtime.md`). The repro script carries a `Bug852.Verify` page for this: `Slug` is three spaces, so `trim()` changes the outcome and a dropped `Visible` renders (Mendix defaults to visible). Pre-fix all 5 markers render; post-fix only the 3 that should. **One rule, two contexts**: `xpathConstraint` serves both `Visible:`/`Editable:` (a Mendix *client expression* — trim/length/toUpperCase/find) and a datasource `where` (real *XPath* — contains/starts-with/ends-with/string-length/not, `length()` = list length, aggregates Java-only, and `empty`/`NULL` are KEYWORDS not calls). The sets differ, so the grammar must not enumerate either; mxbuild adjudicates. Regression-test the XPath side when touching this rule — `[Name = empty]`, `[Name = NULL]`, `not()`, `contains()`, `starts-with()`, `string-length()` all still parse and `mx check` clean. Tests `TestConditionalVisibility_KeywordFunctionNames`, `TestValidateStaticWidget_UnconsumableConditional`; repro `mdl-examples/bug-tests/852-conditional-keyword-functions.mdl`. Issue #852 | +| `download file $Doc;` is accepted by `mxcli check` and `mxcli exec` ("Created microflow") but the activity lands with **no action at all** — `describe` renders `-- Empty action` and `mx check` fails `[CE0008] "No action defined."`. Same for `download file $Doc show in browser;` | `microflowActionToGen` (the modelsdk write path) had no `*microflows.DownloadFileAction` case, so it hit `default: return nil` and the enclosing ActionActivity was serialized with a nil Action. Grammar, visitor, flow builder, read path and DESCRIBE formatter were all already in place, so the statement passed every stage that reports anything and vanished at the one that does not | `mdl/backend/modelsdk/microflow_write.go` (`microflowActionToGen`) | Add the case, setting `FileDocumentVariableName`, `ShowFileInBrowser` and `ErrorHandlingType` (Rollback default). **The storage key is `ShowFileInBrowser`, not `ShowInBrowser`** — the gen setter binds the right one; legacy's `parseDownloadFileAction` reads the wrong key. **Test at the round trip, not the reader**: a reader-only test starts from BSON the writer never had to produce, so `TestActionFromGen_DownloadFile` was green throughout. `roundTripMicroflow` (model→gen→codec→model) is the harness; assert the ActionActivity's `Action` is non-nil, which is the CE0008 shape itself. This is the same silent-drop mechanism as the `microflowObjectToGen` default branch (#791) — when auditing, diff the write switch's cases against `sdk/mpr/writer_microflow_actions.go`. Test `TestMicroflowRoundTrip_DownloadFile`; repro `mdl-examples/bug-tests/850-download-file-action.mdl`. Issue #850 | +| A decision written with **uppercase** keywords — `IF $T/Status != M.Status.Done AND $T/CompletedOn != empty` — passes `mxcli check` and then fails the build with `[error] [CE0117] "Error(s) in expression."`, quoting the expression back with `AND` still uppercase. The same condition written with `=` builds fine, which makes it look like `!=` cannot be an operand of `AND` | Mendix requires its word operators lowercase. A rebuilt `BinaryExpr` gets `strings.ToLower(e.Operator)`; a condition kept as an `ast.SourceExpr` (original text **plus** the parsed tree) returned `e.Source` verbatim and skipped it. The `=` form parses to a BinaryExpr and the `!=` form to a SourceExpr — hence the operator-shaped illusion | `mdl/executor/cmd_microflows_helpers.go` (`normalizeMendixOperatorCase`, applied in the `SourceExpr` branch) | Lowercase `and/or/not/div/mod` in preserved source, leaving everything else byte-identical: a scanner that tracks single-quoted literals (with `''` escapes) and skips any word preceded by `.`, `/` or `$`, so `'AND'`, `M.Enum.And`, `$Task/Mod` and `$Android` are untouched. **The reporter's own probe table is the cautionary bit** — nine builds established a rule ("any `!=` inside `AND` fails") that was real in every observation and wrong about the cause, because every failing probe was uppercase and the control was not. When a table's rule tracks a token, check what ELSE differs between the rows. Reproduced and fixed against mxbuild 11.12.1: stored `AND` → CE0117, stored `and` → 0 errors. Repro `mdl-examples/bug-tests/todo-14b-uppercase-and-operator.mdl`; tests `TestNormalizeMendixOperatorCase`. mxcli-todo #14b | +| `ALTER ENTITY X ADD EVENT HANDLER …` errors when the handler exists and `DROP EVENT HANDLER …` errors when it does not, so a script containing either cannot be re-run — and a defensive drop-then-add fails on whichever half does not match. `ADD ATTRIBUTE` has `IF NOT EXISTS`; event handlers had no equivalent | The idempotency guards added for attributes (findings #10) were never extended to the event-handler clauses, which are the one member with no other re-run route | `mdl/grammar/domains/MDLDomainModel.g4` (`ifNotExists?` / `ifExists?` on the event-handler clauses), `mdl/visitor/visitor_entity.go`, `mdl/executor/cmd_entities.go` | Reuse the existing `ifNotExists`/`ifExists` grammar rules rather than inventing a second spelling, so the guard reads the same everywhere. The error messages now name the flag, so the fix is discoverable from the failure. Verified by running the same script twice: first run drops (skipping, absent) then adds; second run skips both, exit 0, and the project still builds 0 errors. mxcli-todo #18 | +| `CREATE DEMO USER` reports success and `SHOW PROJECT SECURITY` reports `Demo Users Enabled: true`, yet the running app has **zero** accounts — `SELECT Name FROM Administration.Account` returns 0 rows, there is no login page, and none of the row-level XPath rules are enforced | A blank mxcli template ships with **Security Level: Off**, and with it off the runtime creates no accounts at all. The demo users are written to the model correctly; nothing connected the two facts, so the model said yes and the app said nothing | `mdl/executor/cmd_security_write.go` (`warnDemoUsersInert`, called after a successful create) | Say it at the moment the user would otherwise believe it worked, and name the one statement that fixes it (`alter project security level prototype`). A *warning*, not a refusal: authoring demo users before raising the level is legitimate ordering. **Generalisable**: when a write succeeds but a project-level setting makes it inert, the write path is the only place with both facts in hand. Tests `TestWarnDemoUsersInert`. mxcli-todo #15 | +| Wiring a microflow that takes parameters to a **BEFORE CREATE** event handler passes `mxcli check` (and `--references`), and the build then fails `[error] [CE7247] "Microflow should not have parameters" at Event handler of entity …` | Mendix passes no object to a before-create handler — the object does not exist yet — so the handler is called with no arguments. Nothing compared the handler's moment against the microflow's signature; the pairing is only invalid for this one moment/event combination | `mdl/executor/cmd_entities.go` (`checkBeforeCreateHandlerHasNoParameters`, called from `buildEventHandlers`) | Guard where the two paths converge — `buildEventHandlers` is shared by `CREATE ENTITY`'s inline handlers and `ALTER ENTITY ADD EVENT HANDLER`, so one check covers both. It refuses **before the model is written**, and the message carries the build code plus the way out (AFTER CREATE, which does receive the object). **A microflow created earlier in the same script is not readable back yet, so an unreadable microflow is skipped rather than refused** — mxbuild still catches the real case, and failing on the read would break legitimate scripts. Note this is an exec-time guard: `mxcli check` without a project cannot see the microflow's signature at all. A/B on 11.12.1: pre-fix binary writes it and mxbuild reports CE7247; fixed binary refuses, and both AFTER CREATE and a no-parameter BEFORE CREATE still work. Tests `mdl/executor/cmd_entities_before_create_test.go`. mxcli-todo #14a | +| "Contrast is low and not everything uses the dark theme" — and switching theme does not help, because `signal`, `ledger` and `console` all render the same defects. The worst of it is the **login page**: Atlas's stock photograph fills half the viewport on a dark app, and the Sign in button is green while the app's primary button is the brand colour | The login page is served from `theme/web/login.html`, which loads the SAME compiled theme CSS (`{{themecss}}`) — so it IS themeable, but nothing themed it. Two Atlas rules do the damage: `.loginpage-image` layers a brand-tinted gradient over `url("./resources/work-do-more.jpeg")`, and the submit button is `.btn-success`, so it follows the **success** colour rather than the brand. Separately, `--link-color` was mapped straight to the brand, and console's light-variant teal is 3.74:1 on white — under AA for body text | `cmd/mxcli/theme/assets/*/files/theme/web/_mxcli-atlas-map.scss` (login block + `--mxt-link` indirection, all three copies stay byte-identical), `cmd/mxcli/theme/assets/console/.../_mxcli-console.scss` (light `--mxt-link`) | Replace the photo with a token-built gradient and point the login button at the brand; give link TEXT its own `--mxt-link` token defaulting to the brand, because a link needs 4.5:1 as text while a brand used as a button FILL only needs 3:1 plus contrast against its own ink — darkening the brand everywhere would have been the wrong lever. Console light gets `#0f766e` (5.47 / 5.10 / 4.92 against surface / ground / surface-alt). **The project's own `theme/web/logo.png` is deliberately left alone** — it is the app's asset to replace, and hiding it would strip a real logo from apps that have one. **Verification boundary, stated plainly**: verified at the compiled-CSS layer (the overriding `.loginpage-image` is last and carries no photo; the button and link declarations resolve), NOT in a browser — raising the scratch project's security level to serve a login page surfaced pre-existing model errors there that block deploy. The `#999` empty-state label could not be reproduced: every `#999` in the compiled CSS is a Bootstrap default (popover arrows, modal border, print styles), so that one needs the reporter's app. mxcli-todo #19 | **Key insight:** `microflows$ListRange` stores offset/limit inside a nested `CustomRange` map — must cast `raw["CustomRange"].(map[string]any)` before @@ -369,3 +387,18 @@ extracting `OffsetExpression`/`LimitExpression`. - [ ] `go test ./mdl/executor/... ./sdk/mpr/...` passes - [ ] New symptom row added to the table above (if not already covered) - [ ] PR title: `fix: ` +| A login that is known to be correct starts failing on `run --local` — the page says only **"Sign in failed"**, as if the password were wrong. `--screenshot-user` silently produces screenshots of the login page instead of the requested page | The local runtime is unlicensed, and an unlicensed runtime caps concurrent sessions. Past the cap it refuses the sign-in and logs `Maximum number of sessions exceeded! (You are currently using a trial license)` to `.mxcli/runtime.log` only; the page reports the refusal identically to a bad password. mxcli's own login helper never noticed either — it filled the form, waited, and saved whatever session it had | `cmd/mxcli/docker/screenshot_login.go` (`loginScript` failure detection, `loginFailureHint`, `readLogTail`), `cmd/mxcli/docker/runlocal.go` (passes `RuntimeLogPath`), `.claude/skills/mendix/run-local.md` | Not an mxcli defect in the model layer — but mxcli held both halves and joined neither. Mendix answers a rejected sign-in by re-rendering the same form, so **the username field still being present after the click is the signal** that login did not complete; on that signal read the tail of the runtime log and name the cap. **Generalisable**: when a browser-driven helper "succeeds" with a degraded result, look for a server-side log that already says why — the page is often the least informative witness. Sessions are released by restarting `run --local`; a browser-driving script should sign out at the end, or the fifth or sixth run is the one that fails and it looks like a regression. Tests `TestLoginFailureHint`, `TestReadLogTail`. mxcli-todo #16 | +| A fresh clone of a project created by `mxcli new` goes dirty the first time anyone builds it: ~50 **tracked** files modified that nobody edited — every `javascriptsource/*/actions/*.js` gains a banner, `import { Big } from "big.js"` and `export async function`, plus the matching `javasource` stubs. In a cloud session with a stop-hook git check it reads as "uncommitted changes" at the end of clean work | The template ships the generated action stubs in a slightly older shape and MxBuild rewrites them all on the first build. `mx check` does **not** — only a build does — so nothing before the first `run --local` could reveal it, which is after the user has already committed | `cmd/mxcli/docker/settle.go` (`SettleGeneratedSources`), `cmd/mxcli/cmd_new.go` (step 5/6, `--skip-build`), `cmd/mxcli/init.go` (`/theme-cache/` in the generated ignore list) | Fix the *timing*, not the content: run the build while the project is still being created, so the settled form lands in the first commit. Do **not** reimplement the rewrite — it is mxbuild's generator and version-specific; run the real thing. Best-effort by contract (no JDK, no mxbuild, failed build → warning, never a failed creation), because a settled tree is a nicety and a usable project is the deliverable. The other half is gitignore: `theme-cache/` is a cache and says so. A/B on 11.12.1, both git-init'd then built: `--skip-build` → 50 dirty files, default → **0**. Tests `cmd/mxcli/docker/settle_test.go`. mxcli-todo #7 | +| `mxcli check` reports `✓ Syntax OK` / `Check passed!`, then `mxcli exec` on the same script fails partway through with `failed to resolve page: page not found: Module.Page` — a button targeting a page the script creates further down. `exec` is not transactional, so the statements before the failure are already written to the .mpr | Page references are resolved in statement order at exec time. `check --references` already had an ordered pass (`validateForwardPageRefs`), but it needs `-p`; plain `check` had nothing, and plain `check` is what gets run | `mdl/executor/validate_page_order.go` (`ValidateScriptPageOrder`, MDL-PAGE01), wired in `cmd/mxcli/cmd_check.go` beside the other project-free validators | The soundness argument is what makes this work without a project: a **plain** CREATE later in the script would fail if the page already existed, so the script itself asserts the page does not exist yet and the earlier reference cannot resolve against the project either. `CREATE OR MODIFY`/`OR REPLACE` assert nothing, so they stay with `--references`, which can look. **Generalisable**: an ordering rule that seems to need project state often does not, once you find the statement that already asserts what you were going to look up. Two things the diagnostic must say and does: a cycle cannot be fixed by ordering (create one page without the linking widget, add it with `ALTER PAGE … INSERT`), and commit before executing a large script. Verified against all `mdl-examples/**/*.mdl` for false positives (0). Tests `mdl/executor/validate_page_order_test.go`, example `mdl-examples/bug-tests/todo-9-forward-page-reference.fail.mdl`. mxcli-todo #9 | +| A sidebar navigation label renders truncated — "All tasks" shows as "All task" — under every theme (`signal`, `ledger`, `console`) and both variants, so switching theme does not help. Measured on the live DOM as `scrollWidth=56` inside `clientWidth=48` | Atlas's **closed** sidebar is an icon rail: `--navsidebar-width-closed: 48px` in Atlas's own `themesource/atlas_core/web/themes/_theme-default.scss`. The label is wider than the rail, and the closed rail expects an *icon*, not text. No mxcli theme sets any navigation width — the themes map colours — which is exactly why every theme reproduces it | `.claude/skills/mendix/theme-styling.md` (documented; no code change) | **A fix was written, verified, and reverted** — record the reason: adding `text-overflow: ellipsis` to the nav item helps only where Atlas also sets `white-space: nowrap`; elsewhere the label wraps to two readable lines, and the rule turns `All / tasks` into `All / t…`. Screenshotted both ways against the real compiled CSS. The answer belongs to the app (give nav items icons — what the rail is for — or keep the sidebar open), not to a theme that would impose it on every app. **Generalisable**: when a reported symptom traces to an upstream layout constant, reproduce the geometry against the real compiled CSS (48px container, real class names, Playwright measurement) — it takes minutes, tells you whose constant it is, and shows when the "obvious" CSS fix is a regression. mxcli-todo #19d | +| A published OData service created purely from MDL passes `mxcli check` and then fails the build — `[CE0729] "The service name should not be empty."` and `[CE7375] "Attribute ID for entity 'X' must be published and be the key when associations are exposed as an associated object id."` — the second firing even with no associations exposed at all | Two defaults `CREATE ODATA SERVICE` never set. (1) `Name` (the document) and `ServiceName` (the name in the OData metadata document) are different properties and only the first was set; the CONSUMED path had defaulted this for CE0339 all along. (2) `PublishAssociations` defaults to false = "associations as an associated object id", which Mendix only allows when the system `ID` is published as the key — but MDL's `expose (Attr (KEY))` publishes an ordinary attribute | `mdl/executor/cmd_odata.go` (`serviceName` fallback + heal on create-or-modify; `publishAssociationsFor`; `nonPersistablePublishedEntities` warning), `mdl/ast/ast_odata.go` + `mdl/visitor/visitor_odata.go` (`PublishAssociationsSet`) | **Wider than reported**: the finding framed CE7375 as a non-persistable-entity problem. Measured on 11.12.1, the identical service with a PERSISTENT entity and a unique key builds 0 errors with `true` and CE7375 with `false` — so the default broke *every* published service, and non-persistable was just where it could not be worked around. Defaulting to true does not pick a preference; it picks the only value that can build from the MDL people write. Needs tri-state (`PublishAssociationsSet`) so an explicit `false` is still honoured, and `create or modify` no longer flips a stored value the author did not mention. **Why nothing caught it**: `mdl-examples/doctype-tests/10-odata-examples.mdl` sets both properties explicitly, so the repo's own example worked around both defaults. Tests `cmd_odata_service_name_test.go`, `cmd_odata_publish_associations_test.go`; examples `f1-10.1-…`, `f1-10.4-…`. mxcli-formula1 #10.1/#10.4 | +| A typo in an OData property — `ReadMicroflow:` for `ReadMode:`, `ServiceNam:` for `ServiceName:` — passes `mxcli check` and `exec` reports success, but the model does not have the property. Hours can go into wondering why a published resource ignores its read microflow | The grammar accepts any `name: value` pair inside an OData property list, and the visitor's `switch` had no `default` — so an unrecognised name was dropped between parse and AST. The ALTER path has always answered `"unknown OData service property: %s"`; CREATE, PUBLISH ENTITY, the client and the external entity had nothing | `mdl/ast/ast_odata.go` (`UnknownProperties` on four statements), `mdl/visitor/visitor_odata.go` (four `default:` arms), `mdl/executor/validate_odata_properties.go` (`ValidateODataProperties`, MDL-ODATA01), wired in `cmd/mxcli/cmd_check.go` | The visitor is where the name is lost, so the visitor is where it must be recorded — a validator over the AST alone cannot see a key that was already discarded. Carry it as `UnknownProperties` and report at check time, before anything is written. The message names the property AND guesses the intended one (prefix/substring, then one edit), because a bare known-property list still leaves the reader diffing two spellings by eye. **Correction to the report**: `Pagesize:` is *not* silently dropped — the visitor lowercases before matching, so casing is never a typo, and the test pins that. Verified against every `mdl-examples/**/*.mdl` for false positives (0). Tests `mdl/executor/validate_odata_properties_test.go`. mxcli-formula1 suggested issue 8 | +| A read-microflow-backed OData resource must declare a `System.ODataResponse` parameter and compute a count, even when the count is expensive (a full CSV scan) and nobody asked for it — with no MDL to say otherwise. Same for `$skip`/`$top` support | `Countable`, `SkipSupported` and `TopSupported` were written as literal `true` in the BSON writer's `ODataPublish$QueryOptions`; nothing above the writer could express them | `mdl/ast/ast_odata.go` + `mdl/visitor/visitor_odata.go` (`*bool` on `PublishedEntityDef`, `odataBoolPtr`), `model/types.go`, `mdl/executor/cmd_odata.go` (`astEntityDefToModel`), `mdl/backend/modelsdk/odata_write.go` (`boolOrDefault`), `odata_read_detail.go` (`falseOnly`) | Tri-state (`*bool`) is load-bearing: these default to **true**, so "unset" and "false" cannot share a representation or every existing script would silently turn them off. The reader maps a stored `true` back to nil (`falseOnly`) so DESCRIBE prints only what the author wrote instead of three defaults on every resource. Verified end to end on 11.12.1: `Countable: No` + a read microflow with **no** `$Response` parameter builds 0 errors, which is exactly the combination that was impossible before. Tests `cmd_odata_query_options_test.go`, `odata_write_test.go`. mxcli-formula1 #10.3 | +| `describe odata service Module.Api` emits MDL that will not parse — `ReadMode: CallMicroflow:Module.Read` matches no value, `expose (Module.Entity.Attr ...)` where the clause takes a bare member name — and quietly renames the entity set, printing the entity TYPE's exposed name in the `as '…'` position where the entity SET's belongs | Three independent slips in one emit block. The backend stores a microflow-backed mode as `CallMicroflow:` and a member fully qualified; DESCRIBE printed the stored forms verbatim. The set/type name confusion is invisible in a single-word case and only shows when the two differ (Studio Pro's convention is singular type, plural set) | `mdl/executor/cmd_odata.go` (`odataModeToMDL`, `bareMemberName`, entity-set exposed name, `KEY` over `IsPartOfKey`) | Storage form ≠ input form: anywhere DESCRIBE prints a value read back from the backend, ask whether the *parser* accepts that spelling — `mx check` and the linter never see DESCRIBE output, so nothing else can catch it. Proved by round trip rather than by eye: describe → check (parses) → drop → exec the output → describe again → **byte-identical**, and the rebuilt model reports 0 errors from mxbuild. Tests `cmd_odata_describe_roundtrip_test.go`. mxcli-formula1 #10.5 | +| A `create database connection … type 'Redshift'` (or `'SQLServer'`) executes, builds **0 errors**, and the connection does not work. The skill's own table listed both, and omitted the one value that matters for an unsupported driver | mxcli passes the type string straight to BSON (`addStr(e,"DatabaseType",…)`) and **mxbuild does not validate it either** — verified on 11.12.1 — so nothing between the author and the runtime says the type is not real. Studio Pro's picker (read from `modeler/ide-client/database-connector-editor/`, identical on 11.10.0/11.12.1/11.13.0) is MSSQL, MySQL, Oracle, PostgreSQL, Snowflake, **BYOD** ("Other") — no Redshift, no SQLServer | `mdl/executor/validate_database_type.go` (`ValidateDatabaseConnectionType`, MDL-DB01), wired in `cmd/mxcli/cmd_check.go`; `.claude/skills/mendix/database-connections.md` | **A warning, not an error**: the value set is version-specific and mxcli cannot prove a string wrong on a Mendix version it has not seen — but silence is worse when the build is green and the connection is dead. **`BYOD` is the discovery worth keeping**: it forces connection-string config and *skips the driver-presence check*, so any JDBC driver Mendix has no entry for (DuckDB, SQLite, ClickHouse) works by dropping the JAR in `userlib/`. **Generalisable**: when a doc lists enum values, the shipped Studio Pro editor bundle is the authority — grep `id:"…",label:"…"` out of `ide-client/`, and diff across cached versions to see whether the set moved. Tests `validate_database_type_test.go`. mxcli-formula1 #6 | +| `mxcli init` run from a solution root (several app folders, no `.mpr` at the root) reports success and writes tooling that points at `project.mpr` — a file that does not exist. Nobody is told which project it picked, because it did not pick one | `findMprFile` looks only in the target directory; an empty result fell through to a hardcoded `"project.mpr"` default and initialisation continued as if that were a real project | `cmd/mxcli/init.go` (`findMprFilesInSubdirs` + the three-way branch on the candidate count) | Look one level down, then branch on the **count**: 0 → warn that generated paths will be placeholders; 1 → announce the project and initialise **that** directory; 2+ → refuse, list them, and print the exact command naming one. One level only — a Mendix app keeps its `.mpr` at its own root, and walking deeper starts finding deployment copies and backups. Candidates are sorted so the refusal and its suggested command are stable rather than directory-order dependent. **Generalisable**: a "sensible default" that names a file which does not exist is not a default, it is a silent wrong answer — count the candidates and let the count choose the behaviour. Tests `cmd/mxcli/init_discover_test.go`. mxcli-formula1 #3 | +| `alter settings configuration 'Default' …` is the write form, but `describe settings configuration 'Default'` is a **parse error** — and `show settings configurations` summarises the configuration without `ApplicationRootUrl`, so the obvious command for "did my root URL land?" cannot answer it (and renders an empty DatabaseUrl as a bare `, ,`) | The grammar's `DESCRIBE SETTINGS` alternative took no object, so the read form of a write statement simply did not exist; the summary builder listed database/port fields and never gained the root URL when that property was added | `mdl/grammar/domains/MDLCatalog.g4` (`DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)?`), `mdl/visitor/visitor_query.go` (reuses `DescribeStmt.Qualifier`), `mdl/executor/cmd_settings.go` (`writeSettingsConfiguration`, `describeSettingsConfiguration`, summary) | **Read forms should mirror write forms** — where MDL has `alter X `, `describe X ` should parse, and reaching for it and getting a parse error teaches the wrong lesson. Factor the emit into one helper so the whole-settings dump and the single-configuration dump cannot drift. An unknown name lists the ones that exist, or the user is guessing. **Watch for**: changing `describeSettings`'s signature broke three existing callers, and the same commit's `IsPartOfKey`→`KEY` change broke an OData round-trip test that only the FULL suite caught — run `go test ./mdl/...`, not just the new test. Tests `cmd_settings_configuration_test.go`. mxcli-formula1 #8 | +| `ALTER MODULE X ADD JAR DEPENDENCY (…)` succeeds, `list jar dependencies` reports it, the build is **green** — and the runtime throws `SQLException: No JDBC driver found in app for URL`. `deployment/build.gradle` has no dependencies block and `find deployment -iname '**'` returns nothing | Not a bad write. Declaring and resolving are **separate steps**: the model records the coordinate, and `mx sync-java-dependencies ` is what downloads it into `vendorlib/`. Studio Pro runs that when you edit Module Settings; nothing headless was running it. Confirmed on 11.12.1 — a full `mxbuild --target=deploy` resolves nothing, and the sync command then fetches the jar | `cmd/mxcli/docker/javadeps.go` (`SyncJavaDependencies`, `UnvendoredJarDependencies`), `cmd/mxcli/cmd_sync_java_deps.go` (`mxcli sync-java-deps [--check]`), `cmd/mxcli/docker/runlocal.go` (vendors before boot), `mdl/executor/cmd_modules.go` (`warnUnvendoredJarDependencies`) | **How to find the missing step**: the reporter's open question was "does mxbuild skip Maven resolution, or does mxcli write it somewhere MxBuild cannot read?" — neither. `strings mx.dll | grep -i dependenc` surfaced `ISyncJavaDependenciesRunner`/`SkipManagedDependencySync`, and `mx --help` listed `sync-java-dependencies`. When a model-level write "works" but the artefact never appears, check whether the **toolset** has a separate command for it before suspecting the write. Wired at three levels so the gap cannot stay silent: the executor says so the moment it writes an unvendored coordinate, `run --local` resolves it before boot, and `--check` exits non-zero as a build gate. Resolution needs network, so every call site is best-effort with an actionable message. Tests `cmd/mxcli/docker/javadeps_test.go`. mxcli-formula1 #12 | +| `$Total = 5;` does not parse — `no viable alternative at input '$Total=5'` — while `DECLARE $Total Integer = 0;` does, and so do `$X = HEAD($List)`, `$X = create M.E (…)` and `$X = execute database query …`. The error names the token, not the missing keyword | Assignment existed only as a **prefix** on specific activity statements (`(VARIABLE EQUALS)?` on CALL/CREATE/RETRIEVE/…), plus a `SET $Var = expression` statement. A plain value therefore required `SET`, which nothing in the error or the surrounding syntax suggested | `mdl/grammar/domains/MDLMicroflow.g4` (`setStatement : SET? …`), `cmd/mxcli/syntax/features_microflow.go` | Make the guessable form work rather than improve the error: `SET` is now optional and both spellings produce the same `MfSetStmt`/`ChangeVariableAction`. **Prove a grammar relaxation causes no regressions with a control binary, not by reading**: `git stash` the `.g4`, `make grammar`, build `bin/mxcli-control`, sweep every `mdl-examples/**/*.mdl` with both — 13 scripts fail, the *same* 13, all pre-existing. ANTLR's adaptive prediction picks the activity-prefixed alternatives over `setStatement` on its own; no ordering change was needed. Executed against a real .mpr, mxbuild reports 0 errors. Tests `mdl/visitor/visitor_microflow_bare_assign_test.go` (bare and keyword forms must agree on the AST, not merely both parse). mxcli-formula1 #13 | +| `mxcli test tests/ -p app/App.mpr` fails with "no such file or directory" for a `tests/` that sits right next to the `.mpr` | Test paths resolved against the process CWD only. Defensible in isolation, but mxcli otherwise encourages naming the project (`-p`) rather than standing in its directory, and project auto-discovery searches outward — so the two conventions collide and the failure looks like a missing directory | `cmd/mxcli/cmd_test_run.go` (`resolveTestPaths`) | Fall back to project-relative **only when the CWD-relative path does not exist**: a `tests/` in both places must resolve to the one the user is standing in, since silently preferring the project's copy would run the wrong suite. A path that exists in neither is passed through unchanged so the error names what was typed, not a rewritten path the user never mentioned. Tests `cmd/mxcli/cmd_test_run_paths_test.go`. mxcli-formula1 #13 | +| After `SET` became optional, `mx check` on a project built from `02-microflow-examples.mdl` reports six errors that no MDL change caused: `[CE0109] "Undefined variable 'ProductList.Price'."` at four Aggregate list activities, and `[CE0015] "Aggregate function must specify a valid attribute."` at the expression-based one. Identical on both engines. `mxcli check` on the same script is silent | Two conversions existed for one syntax. `$Sum = sum($List.Price)` used to reach the dedicated `aggregateListStatement` rule; making `SET` optional put `setStatement` — alternative 5 of ~50 — in front of it, so ANTLR matched the lower-numbered alternative and the statement fell through to `buildSetStatement`'s fallback conversion, which joined list and attribute into one name and dropped the per-item expression entirely. Underneath, `buildListAggregateAsFunction` never appended the expression argument, so the SET path could not have seen it either | `mdl/grammar/domains/MDLMicroflow.g4` (`setStatement` moved LAST in `microflowStatement`), `mdl/visitor/visitor_microflow_statements.go` (`buildSetAggregate` replaces `extractVariableAndAttribute`), `mdl/visitor/visitor_microflow_expression.go` (`buildListAggregateAsFunction` appends the expression argument) | **A permissive alternative belongs last.** `$X = ` overlaps every `VARIABLE EQUALS ` statement in the rule — aggregates, list operations, RANGE — and ANTLR's ALL(*) picks the lowest-numbered alternative that matches, so a new general form silently steals from every specific one above it. **The measurement trap that let this ship**: the `SET?` change was swept with a control binary over every `mdl-examples/**/*.mdl` and found no difference — but with `mxcli check`, which parses and validates and never serializes. This defect lives between the AST and the BSON, where only `exec` + `mx check` can see it. Sweeping with `check` proves the grammar still *parses*; it proves nothing about what the visitor *builds*. For a grammar change, the control sweep must run the integration gate (`go test -tags integration -run TestMxCheck_DoctypeScripts`), not `check`. Fix proven by reverting both halves and watching the new tests fail with the reported symptom. Tests `mdl/visitor/visitor_microflow_aggregate_test.go`. Upstream CI on the ako→mendixlabs sync PR | diff --git a/.claude/skills/mendix/cheatsheet-errors.md b/.claude/skills/mendix/cheatsheet-errors.md index 18b1aebf5..b23b1875b 100644 --- a/.claude/skills/mendix/cheatsheet-errors.md +++ b/.claude/skills/mendix/cheatsheet-errors.md @@ -217,6 +217,19 @@ set $count = 1; 1. Check microflow exists: `show microflows in ModuleName` 2. Use fully qualified name: `Module.MicroflowName` +### "page not found" for a page the script creates further down (MDL-PAGE01) + +**Problem**: A widget action targets a page created by a LATER statement in the +same script. Page references resolve in statement order, and `exec` is not +transactional — the statements before the failure are already written. + +**Fix**: +1. Move the `create page` for the target above the page that links to it. +2. If two pages link to each other, no ordering works: create one without the + linking widget, then add it with `alter page ... insert`. +3. Commit before executing a large script — recovery from a partial run is + `git checkout -- App.mpr mprcontents/`. + ## Studio Pro Error Code Reference | Code | Message | Common Cause | diff --git a/.claude/skills/mendix/create-page.md b/.claude/skills/mendix/create-page.md index d366f6b4c..fbde3e364 100644 --- a/.claude/skills/mendix/create-page.md +++ b/.claude/skills/mendix/create-page.md @@ -1082,8 +1082,38 @@ textbox txtHidden (label: 'Hidden', attribute: Name, visible: false) -- A quoted-string expression is also accepted (CREATE and ALTER). Unlike the -- bracket form, it is NOT auto-rooted — write $currentObject/ yourself. dynamictext ovChip (content: 'chip', visible: '$currentObject/Name != empty') + +-- Function calls work in the bracket form, including functions whose name is +-- also an MDL keyword (trim, length, find). Arguments are rooted like any other +-- reference. +dynamictext tTrim (content: 'x', visible: [trim($currentObject/Slug) != '']) +textbox txtSlug (label: 'Slug', attribute: Slug, editable: [length(Slug) > 0]) ``` +> **`visible:`/`editable:` is a Mendix *expression*, not XPath** — a different +> function set from a datasource `where` clause, even though both use `[ ... ]`: +> +> | | `visible:` / `editable:` (client expression) | `where [ … ]` (XPath) | +> |---|---|---| +> | String tests | `trim()`, `length()`, `toUpperCase()`, `find()`, `contains()` | `contains()`, `starts-with()`, `ends-with()`, `string-length()` | +> | `length()` | character count | number of elements in a list | +> | Emptiness | `$currentObject/X != ''` / `!= empty` | `[X = empty]` or `[X = NULL]` — a **keyword**, never `empty(…)` | +> | Aggregates | not available | `count()`/`avg()`/`min()`/`max()`/`sum()` are Java-API-only | +> +> mxcli's grammar accepts any function name in both and lets MxBuild adjudicate, +> so a wrong-context call surfaces as **CE0117** "Error(s) in expression" at +> build rather than as a parse error. See the Mendix reference guide: +> [XPath constraint functions](https://docs.mendix.com/refguide/xpath-constraint-functions/), +> [XPath keywords](https://docs.mendix.com/refguide/xpath-keywords-and-system-variables/). + +> **An unparseable conditional is an error, not a silent drop.** If the +> expression inside `visible: [ ... ]` / `editable: [ ... ]` can't be parsed, the +> property has nowhere to go and would vanish on write — leaving the widget +> unconditionally visible/editable, which looks identical to a specificity bug in +> the running app. `mxcli check` reports this as **MDL-WIDGET19** and fails the +> command instead. Until v0.16.x, `trim(…)` and `length(…)` hit exactly this path +> and disappeared without a word (issue #852). + > **Attribute rooting is automatic** — a bare attribute in a widget > visibility/editability expression (`[Name != '']`, `[IsActive]`) is rooted in the > widget data context as `$currentObject/Name != ''` for you, so it no longer diff --git a/.claude/skills/mendix/database-connections.md b/.claude/skills/mendix/database-connections.md index 8aedad889..50f9a0f55 100644 --- a/.claude/skills/mendix/database-connections.md +++ b/.claude/skills/mendix/database-connections.md @@ -76,14 +76,60 @@ end; ### Supported Database Types -| Database | TYPE Value | -|----------|------------| -| Oracle | `'Oracle'` | -| PostgreSQL | `'PostgreSQL'` | -| MySQL | `'MySQL'` | -| SQL Server | `'MSSQL'` or `'SQLServer'` | -| Snowflake | `'Snowflake'` | -| Amazon Redshift | `'Redshift'` | +These are the values Studio Pro's own connector editor offers — read out of the +shipped bundle at `modeler/ide-client/database-connector-editor/`, identical on +11.10.0, 11.12.1 and 11.13.0. + +| Database | TYPE Value | Studio Pro label | +|----------|------------|------------------| +| SQL Server | `'MSSQL'` | Microsoft SQL | +| MySQL | `'MySQL'` | MySQL | +| Oracle | `'Oracle'` | Oracle | +| PostgreSQL | `'PostgreSQL'` | PostgreSQL | +| Snowflake | `'Snowflake'` | Snowflake | +| *anything else* | `'BYOD'` | Other | + +**`'BYOD'` — bring your own driver.** Selecting it forces connection-string +configuration and **skips the driver-presence check**; its only validation is +that the connection string is non-empty. That is the hook for any JDBC driver +Mendix ships no picker entry for (DuckDB, SQLite, ClickHouse, …). Verified end to +end on Mendix 11.13: a booted runtime opened `jdbc:duckdb:` through a `BYOD` +connection and returned real rows — the runtime accepts it, not just the editor. + +### Getting the driver onto the classpath + +The driver JAR has to be *resolved*, and declaring it is not resolving it: + +```sql +ALTER MODULE MyModule ADD JAR DEPENDENCY ( + group = 'org.duckdb', artifact = 'duckdb_jdbc', version = '1.5.5.1', included = true +); +``` + +writes the coordinate to the model — `list jar dependencies` will report it — and +downloads **nothing**. MxBuild does not resolve it either: a full +`mxbuild --target=deploy` emits a `build.gradle` with no dependencies block. The +first symptom is a runtime `SQLException: No JDBC driver found in app for URL`, +from a connection that looks correctly configured. + +Studio Pro runs the resolution when you edit Module Settings. Headless, ask for it: + +```bash +mxcli sync-java-deps -p app.mpr # download into vendorlib/ +mxcli sync-java-deps -p app.mpr --check # report what is missing, exit 1 (build gate) +``` + +`mxcli run --local` does this automatically for anything not already in +`vendorlib/`, so the warm loop works from a fresh clone. Dropping the jar into +`userlib/` by hand works too — it is the same classpath — but then the model and +the file system disagree about where the dependency comes from. + +**`'Redshift'` and `'SQLServer'` are not real values.** Both appeared in an +earlier version of this table and neither is in the picker on any version +checked. mxcli writes the type string through unchanged and **mxbuild does not +validate it** — `type 'Redshift'` builds 0 errors and simply does not connect — +so `mxcli check` warns about an unrecognised type (MDL-DB01) rather than letting +a green build hide it. ## Query Definition Syntax @@ -315,6 +361,30 @@ $ResultList = execute database query Module.Connection.QueryName dynamic 'SELECT id, name FROM employees WHERE active = true LIMIT 10'; ``` +**A dynamic override still requires a value for every declared parameter** — +including the ones the replacement SQL does not use. The parameter list belongs +to the query *definition*, not to the SQL string, so Mendix asks for all of them +whatever you substitute. Pass a placeholder for the unused ones: + +```sql +-- The definition declares $driverId; this SQL ignores it, and the call still +-- has to supply it. +$Count = execute database query F1.DuckDB.CountAllDrivers + dynamic 'SELECT count(*) AS n FROM read_csv(''/data/f1db-drivers.csv'')' + ( driverId = 'unused' ); +``` + +**A `{param}` placeholder can be concatenated into a path**, which is what keeps +absolute paths out of the model — bind the data directory as one constant and +build the file name around it: + +```sql +-- read_csv({dataDir} || '/f1db-drivers.csv') +``` + +Verified against DuckDB through the connector on Mendix 11.13, and against a +standalone JDBC harness before that. + ### Parameterized Queries Pass values for query parameters defined with `parameter` in the query definition: diff --git a/.claude/skills/mendix/odata-data-sharing.md b/.claude/skills/mendix/odata-data-sharing.md index acefc6a29..a61793172 100644 --- a/.claude/skills/mendix/odata-data-sharing.md +++ b/.claude/skills/mendix/odata-data-sharing.md @@ -223,8 +223,12 @@ create odata service ProductApi.ProductDataApi ( ODataVersion: OData4, namespace: 'DefaultNamespace', ServiceName: 'ProductDataApi', - Summary: 'Product and customer data API', - PublishAssociations: No + Summary: 'Product and customer data API' + -- PublishAssociations is left at its default (Yes = associations as links). + -- Setting it to No means "associations as an associated object id", which + -- Mendix only allows when the system ID is published as the key — publishing + -- an ordinary attribute as the key then fails the build with CE7375, even + -- when no associations are exposed at all. ) authentication basic { @@ -412,6 +416,64 @@ alter entity ShopClient.Product set allow_create_change_locally = true; alter entity ShopClient.Product set allow_create_change_locally = false; ``` +## Publishing a Non-Persistable Entity (no copy of the data) + +A published entity does **not** have to be persistable. Back it with a read +microflow and the rows are produced per request — nothing is stored, and there +is no refresh job to keep a copy in step with the source. This is the shape to +use when the data lives outside Mendix (an external database, a CSV, an API). + +```sql +create non-persistent entity Api.Lap ( + LapKey: string(60), + Driver: string(120), + LapTime: decimal +); + +-- While Countable is Yes (the default), the read microflow MUST take a +-- $Response: System.ODataResponse parameter — Mendix asks it for the count. +CREATE MICROFLOW Api.Read_Laps ($Response: System.ODataResponse) + RETURNS List of Api.Lap AS $Laps +BEGIN + -- retrieve from wherever the data actually lives, e.g. EXECUTE DATABASE QUERY + $Laps = CREATE LIST OF Api.Lap; + RETURN $Laps; +END; + +create odata service Api.LapApi ( + path: 'odata/laps/', + version: '1.0.0', + ODataVersion: OData4, + namespace: 'Api.Laps' +) +authentication basic +{ + publish entity Api.Lap as 'Laps' ( + ReadMode: microflow Api.Read_Laps, + InsertMode: not_supported, + UpdateMode: not_supported, + DeleteMode: not_supported + ) + expose ( + LapKey as 'lapKey' (KEY, Filterable, Sortable), + Driver (Filterable, Sortable), + LapTime (Sortable) + ); +}; +``` + +Two things worth knowing before you write this: + +- **`ReadMode: microflow Module.MF`** is the whole feature. `InsertMode`, + `UpdateMode` and `DeleteMode` take the same form for a read-write resource. +- **Counting is not free.** If the count means a full scan of the underlying + source, set `Countable: No` on the published entity — the read microflow then + takes no parameters at all. `SkipSupported: No` and `TopSupported: No` turn + off `$skip` and `$top` the same way. All three default to Yes. + +`PublishAssociations` must stay at its default (Yes) here: a non-persistable +entity cannot publish its ID, so object-id mode can never build for it. + ## Step-by-Step: Read-Write API with Microflow Handlers For write operations (insert, update, delete), the OData service delegates to microflows that map between the view entity and the underlying persistent entities. diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 4b9bd7d5b..81e8f4eb6 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -133,6 +133,7 @@ Launch `run --local` as the **sole** command in its invocation (don't chain a tr | `--screenshot-path` / `--screenshot-url` | `.mxcli/run-local.png` / app root | Screenshot output / page (URL or `/path`) | | `--screenshot-user` / `--screenshot-password` | — | Log in once, reuse session (pages behind login) | | `--runtime-log` | `.mxcli/runtime.log` | Runtime log file: JVM stdout/stderr **and** the application log (microflow `LOG` output + server stack traces, via an attached file log subscriber). `-` disables. | +| `--test-endpoint` | off | Host mxcli's token-guarded test endpoint so `mxcli test … --attach` can run a suite against this app with no boot of its own. Installed **before** the boot (the handler registers from after-startup), your own after-startup microflow is chained not displaced, and both are removed on exit. See `test-microflows.md`. | | `--debug` | off | Enable the microflow debugger at boot + start a session, so `mxcli debug break/paused/…` works from another terminal (see `debug-microflows.md`). No breakpoints = no behaviour change; disabled on shutdown. | | `--debug-pass` | `mxdebug` | Debugger password when `--debug` is set | | `--metrics` | off | Register a Prometheus meter registry at boot; the runtime serves metrics at `http://127.0.0.1:/prometheus` | @@ -248,6 +249,27 @@ marker); the subscriber is re-attached on every restart and never rotates the fi the JVM tee's handle stays valid). Override the path with `--runtime-log `, or pass `--runtime-log -` to disable the file (and the subscriber) entirely. +## "Sign in failed" that is not about the password + +The local runtime is **unlicensed**, and an unlicensed runtime caps concurrent +sessions at a handful. Past the cap it refuses the sign-in, and the login page +reports that as a plain **"Sign in failed"** — exactly what a wrong password +looks like. The real reason is written only to the runtime log: + +``` +Maximum number of sessions exceeded! (You are currently using a trial license) +``` + +So: when a login you know is correct starts failing, `grep -c "Maximum number of +sessions" .mxcli/runtime.log` before touching the credentials or the user's +password in the model. `--screenshot-user` does this for you — a rejected sign-in +now reads the log and says so instead of quietly screenshotting the login page. + +Sessions are held until they expire; restarting `run --local` clears them all. +A script that drives the app through a browser should **sign out at the end**, +otherwise each run leaks a slot and the fifth or sixth run is the one that fails — +which makes it look like a change you just made broke authentication. + ## External browser preview (`--hub`) `--hub ` exposes the running app in a **browser at a public URL** without the app diff --git a/.claude/skills/mendix/test-microflows.md b/.claude/skills/mendix/test-microflows.md index 90420179a..8d1534bab 100644 --- a/.claude/skills/mendix/test-microflows.md +++ b/.claude/skills/mendix/test-microflows.md @@ -16,8 +16,22 @@ For **UI/page testing** (widget rendering, form interactions, browser tests), se ## Prerequisites - Mendix project with microflows to test -- Docker stack initialized: `mxcli docker init -p app.mpr` -- App buildable: `mxcli docker build -p app.mpr` +- A way to run the app — **either** of: + - `--local` (no Docker): mxcli boots the runtime itself, the same way + `mxcli run --local` does. This is the only option in a container without a + Docker daemon, which includes Claude Code web sessions. + - Docker: stack initialized (`mxcli docker init -p app.mpr`) and the app + buildable (`mxcli docker build -p app.mpr`). + +```bash +mxcli test tests/ -p app.mpr --local # no daemon needed +mxcli test tests/ -p app.mpr # Docker +``` + +`--local` uses its own ports (app 8081, admin 8091) and its own +`_test` database, so a `mxcli run --local` dev loop can keep serving the +same project while the tests run — the tests never write into the database you +are looking at in the browser. The database is created on first use. --- @@ -107,20 +121,132 @@ mxcli test tests/ -p app.mpr --verbose ## How It Works -The test runner uses the **after-startup microflow** pattern: +There are two mechanisms. `--local` uses the **test endpoint**; Docker uses the +older **after-startup microflow** pattern. + +### `--local`: the test endpoint + +1. Parses test files and extracts test blocks with annotations +2. Records the project's current after-startup microflow, and whether an `MxTest` + module already exists +3. Generates **one `MxTest.Test_` microflow per test**, plus a Java action + that registers an HTTP endpoint, and points after-startup at a microflow whose + only job is to call it — **no test runs during startup** +4. Builds and boots the app once +5. Invokes each test by name over HTTP; each returns its own verdict in the + response +6. Restores the original after-startup setting and removes everything generated +7. Outputs results (console, JUnit XML) + +Two consequences worth knowing when reading a failing run: + +- **A test that throws fails only itself.** It is reported as `ERROR` with the + root-cause message, and the next test still runs. Under the after-startup + mechanism an uncaught error ends the whole flow — and because that flow *is* + the startup action, it also fails the boot. +- **Results are returned, not scraped**, so a test cannot be lost to log + buffering or a runtime that stopped echoing to the console. + +Each test is a separate microflow with its own variable scope, so `$result` in +one test never collides with `$result` in another. + +#### `--watch`: keep the runtime warm + +```bash +mxcli test tests/ -p app.mpr --local --watch +``` + +The first run pays the cold boot; after that the runtime and the build server +stay up, and the suite re-runs on every change — to a test file **or** to the +project's model. Measured on an 11.13.0 app: + +| | | +|---|---| +| First run (cold boot) | ~30s | +| Edit a test → verdict on screen | **~2s** | +| Edit a microflow → verdict on screen | **~2s** | +| The tests themselves | 20–70ms | + +Editing a microflow and seeing straight away whether it still passes is the loop +this exists for. Ctrl-C stops watching and restores the project — the shutdown +prints `project restored` when it has. + +Adding, editing and deleting tests all work mid-session: the suite is re-parsed +on every change, and a deleted test's microflow is dropped rather than left +behind reporting a stale pass. + +`--watch` requires `--local`. The Docker and `--legacy-runner` paths can only +re-run tests by restarting, which is the thing being avoided. + +#### `--attach`: no boot at all + +If you already have the app running, tests can skip the boot entirely. The dev +loop has to opt into hosting the endpoint, because the handler is registered by +the after-startup microflow and so cannot be added to an app that is already up: + +```bash +# terminal 1 — the app you are working in +mxcli run --local --test-endpoint -p app.mpr + +# terminal 2 — runs in ~2s, no boot, repeatable +mxcli test tests/ -p app.mpr --attach +mxcli test tests/ -p app.mpr --attach --watch # ...and re-run on every change +``` + +The hosting app chains your project's own after-startup microflow rather than +displacing it, so it still boots normally. The endpoint and the handshake file +(`.mxcli/test-endpoint.json`, mode 0600) are removed when the app stops. + +Three things to know before reaching for it: + +- **Tests run against the running app's database**, not a scratch one, so they + can leave data behind in the app you are looking at. `--local` uses a separate + `_test` database; `--attach` does not. +- **An attach only owns its own test microflows.** The endpoint and the + after-startup setting belong to the app hosting them, and cleanup never + touches them. +- **A change needing a runtime restart is refused** — a new entity or + association. That runtime belongs to the other process. Restart it, or drop + `--attach`. + +| | Boot | Database | Owns the runtime | +|---|---|---|---| +| `--local` | ~30s each run | `_test` | yes | +| `--local --watch` | ~30s once, then ~2s | `_test` | yes | +| `--attach` | none | the running app's | no | + +#### Security of the endpoint + +It executes microflows under a system context, so it is gated four ways: + +| Guard | Behaviour | +|---|---| +| No `MXCLI_TEST_TOKEN` in the runtime's environment | The handler is **not registered at all** (404) | +| Missing or wrong `X-MxTest-Token` header | 401, compared in constant time | +| Non-loopback caller | 403 | +| `mf` outside `MxTest.Test_*` | 403 — it is not a general microflow-invocation API | + +The token is generated per run and reaches the runtime through its **environment**, +never written into the project. Combined with fail-closed registration, that means +a project which kept the `MxTest` module through a failed cleanup exposes nothing +when deployed anywhere else. + +### Docker: the after-startup microflow 1. Parses test files and extracts test blocks with annotations 2. Records the project's current after-startup microflow, and whether an `MxTest` module already exists -3. Generates a `MxTest.TestRunner` microflow with assertion logic and points - after-startup at it -4. Builds the project and restarts the Docker runtime +3. Generates a single `MxTest.TestRunner` microflow containing every test, and + points after-startup at it +4. Builds the project and restarts the container 5. Captures structured `MXTEST:` log lines for pass/fail 6. Restores the original after-startup setting and removes the generated runner — the whole `MxTest` module when the runner created it, otherwise just the `TestRunner` microflow 7. Outputs results (console, JUnit XML) +### Both mechanisms + The project's **Security Level is not modified**. The after-startup microflow runs in an administrative context and is not subject to it, and forcing it off breaks projects whose published REST/OData services use custom authentication. If a diff --git a/.claude/skills/mendix/theme-styling.md b/.claude/skills/mendix/theme-styling.md index 6f8f96fe0..218fe8aff 100644 --- a/.claude/skills/mendix/theme-styling.md +++ b/.claude/skills/mendix/theme-styling.md @@ -172,6 +172,30 @@ container ctn (style: 'color: red;') { This also applies to `alter styling` and `alter page set style` — never target a DYNAMICTEXT widget with Style. +### Clipped navigation labels are the CLOSED sidebar, not the theme + +A sidebar item reading `All task` instead of `All tasks` is Atlas's **closed** +sidebar, which is an icon rail: `--navsidebar-width-closed: 48px`, set in Atlas's +own `themesource/atlas_core/web/themes/_theme-default.scss`. Measured against a +real compiled theme in a browser, the `` for "All tasks" is **57px wide inside +a 48px rail** — the same overflow reported from a live app (56 in 48). + +No mxcli theme sets any navigation *width*; the themes map colours only. So this +reproduces identically under `signal`, `ledger` and `console`, in both variants — +a layout constant, not a palette. + +The fix is in the app, not the theme: + +- **Give each nav item an icon.** That is what the closed rail is for; the icon is + what stays visible when the sidebar is closed. +- **Or keep the sidebar open**, where the label has room. +- Shorter labels help, but only until the next one is too long. + +Do **not** reach for `text-overflow: ellipsis` on the nav item as a blanket fix. +Tried and rejected: where Atlas does not also set `white-space: nowrap`, the label +wraps to two lines and reads fine — and the ellipsis rule turns that readable +`All / tasks` into `All / t…`. It trades one truncation for a worse one. + ### DataGrid2 Renders ARIA `
`s, Not a `` — and `Size` Is a Flex Weight Two surprises when styling a **DataGrid2** matrix/pivot (ledger finding #46): diff --git a/.claude/skills/mendix/write-microflows.md b/.claude/skills/mendix/write-microflows.md index 524532123..efdc57f08 100644 --- a/.claude/skills/mendix/write-microflows.md +++ b/.claude/skills/mendix/write-microflows.md @@ -491,6 +491,33 @@ end loop; - The loop variable type is **automatically derived** from the list type (e.g., `list of Test.Product` → `Test.Product`) - CHANGE statements inside loops use the derived type to resolve attribute names +> **Nothing a loop defines survives past `end loop;`.** The iterator *and* +> anything the body creates (a `retrieve`, a `$X = create …`, a call output) are +> visible only inside the body; using one afterwards is +> `CE0108 "Variable 'X' is defined but not in scope at this location."` +> (`mxcli check` flags it as **MDL053**). +> +> ```mdl +> -- WRONG: $Last is created inside the loop, read outside it +> loop $Item in $Items +> begin +> $Last = create Test.Product (Name = $Item/Name); +> end loop; +> commit $Last; -- MDL053 / CE0108 +> +> -- RIGHT: declare before the loop, assign inside, read after +> declare $LastName string = ''; +> loop $Item in $Items +> begin +> set $LastName = $Item/Name; +> end loop; +> log info node 'Test' $LastName; +> ``` +> +> Visibility and *naming* are separate rules: names must also be unique across +> the **whole** microflow, so two loops cannot share an iterator name either +> (`CE0111`, flagged as **MDL052**). + ### Performance: Batch Commit After Loop **CRITICAL**: Do NOT commit inside a loop. Each `commit` inside a loop issues a separate database transaction, which causes N round-trips for N records and degrades performance significantly. diff --git a/CLAUDE.md b/CLAUDE.md index 071b69322..088cd61be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -496,7 +496,7 @@ go build -o bin/mxcli ./cmd/mxcli | **Full-text search** | `search 'keyword'` | Search across all strings and source | | **Linting** | `mxcli lint -p app.mpr [--format json\|sarif]` | 15 built-in rules + 27 Starlark rules (MDL, SEC, QUAL, ARCH, DESIGN, CONV) | | **Report** | `mxcli report -p app.mpr [--format markdown\|json\|html]` | Scored best practices report with category breakdown | -| **Testing** | `mxcli test tests/ -p app.mpr` | `.test.mdl` / `.test.md` files, requires Docker | +| **Testing** | `mxcli test tests/ -p app.mpr [--local] [--watch] [--attach]` | `.test.mdl` / `.test.md` files. `--local` runs on mxcli's own runtime (no Docker daemon), on its own ports + `_test` database, driving a **token-guarded test endpoint** (one microflow per test, invoked over HTTP — a throwing test fails only itself, results are returned not log-scraped). `--watch` keeps the runtime warm (~30s first run, then ~2s). `--attach` runs against an app already up under `run --local --test-endpoint` (no boot; uses **that app's** database) | | **Diff** | `mxcli diff -p app.mpr changes.mdl` | Compare script against project state | | **Diff local** | `mxcli diff-local -p app.mpr --ref head` | Git diff for MPR v2 projects | | **Diff revisions** | `mxcli diff-local -p app.mpr --ref main..feature` | Compare two arbitrary git revisions | @@ -520,7 +520,7 @@ mxcli new MyApp --version 11.8.0 mxcli new MyApp --version 10.24.0 --output-dir ./projects/my-app ``` -Steps performed: downloads MxBuild → `mx create-project` → `mxcli theme apply` → `mxcli init` → downloads correct Linux mxcli binary for devcontainer. The result is a ready-to-open project with `.devcontainer/`, AI tooling, mxcli's default styling, and a working `./mxcli` binary. Pass `--theme none` for plain Atlas. +Steps performed: downloads MxBuild → `mx create-project` → `mxcli theme apply` → `mxcli init` → one `mxbuild --target=deploy` run (`--skip-build` to skip) → downloads correct Linux mxcli binary for devcontainer. That build settles the JS/Java action stubs MxBuild rewrites on first build (48 tracked files in a blank 11.12 app), so a fresh clone does not go dirty the first time anyone builds it. The result is a ready-to-open project with `.devcontainer/`, AI tooling, mxcli's default styling, and a working `./mxcli` binary. Pass `--theme none` for plain Atlas. ### Slash Command Namespaces @@ -610,6 +610,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding - VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) - Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) +- Warm test loop (`mxcli test --local [--watch]`, `--attach`, `run --local --test-endpoint`): local test runs go through a **token-guarded HTTP endpoint** registered by a generated Java custom request handler, instead of compiling the suite into the project's after-startup microflow. One microflow per test, resolved by name at request time from `Core.getMicroflowNames()` and invoked with `Core.microflowCall(...).execute(...)` — so a throwing test fails only itself (not the boot), results are returned rather than scraped from the runtime log, and each test has its own variable scope. The handler **survives `reload_model`** (after-startup does not re-run, the JVM is unchanged), which is what makes `--watch` possible: ~30s first run, then ~2s from an edit — to a test *or* to the microflow under test — to a verdict. `--attach` skips even that boot by running against an app already up under `run --local --test-endpoint`, driving that process's serve + admin APIs over loopback; it uses **that app's database**, only ever adds/removes its own test microflows, and refuses a change needing a restart. Security: the handler is **not registered at all** without `MXCLI_TEST_TOKEN` in the runtime env (so a project that kept the `MxTest` module through a failed cleanup is inert in production), the token is constant-time compared, non-loopback callers are refused, `/list` is clamped to the test namespace, and only `MxTest.Test_*` may be invoked. The token reaches the runtime via its environment and is never written into the project. Docker keeps the after-startup runner (`--legacy-runner` selects it locally). Packages: `cmd/mxcli/testrunner/` (`endpoint.go`, `client.go`, `watch.go`, `host.go`, `handshake.go`). See `docs/15-testing/SPIKE_test_endpoint_request_handler.md` - Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` - External browser preview (`mxcli run --hub ` + `mxcli tunnel-hub`): the app stays local and reverse-tunnels out over a single 443 connection (embedded chisel) to a static relay, so it is reachable in a browser at a public URL — works from egress-only environments (Claude Code web), verified live through the session's MITM egress proxy. `run --hub` implies `--local`, boots the runtime with `ApplicationRootUrl` set to the assigned URL (so the SPA/`originURI` work under the public origin), resolves the control proxy honouring `NO_PROXY`, and retries forever. `mxcli tunnel-hub --domain ` is the **multi-tenant** relay: a registry keyed by prefix/project/solution/branch/worktree (stable URLs on reconnect) fronts many previews at per-subdomain hosts (`[prefix-]project[-branch].`; main collapses to the project) over one 443 with per-subdomain autocert, a registration API (`/api/register|status|deregister|backends|sessions`), and an availability overview at `hub./` **grouped by Claude Code session** (`/api/sessions`): each session lists the endpoints it exposed and links back to its `claude.ai/code` conversation. Client identity flags: `--hub-prefix`/`--hub-project`/`--hub-solution`/`--hub-branch`/`--hub-worktree` (project + branch auto-detected); `--hub-session` groups a session's endpoints (auto-detected from `CLAUDE_CODE_REMOTE_SESSION_ID`). Past sessions are retained: a durable per-session endpoint history (`--sessions-file`, default `~/.mxcli/hub-sessions.json`) survives restarts and reaping, and is pruned after `--session-retention` (default 30d) — so the overview shows offline sessions too (`SessionLog` in `cmd/mxcli/tunnelhub/sessions.go`). Package: `cmd/mxcli/tunnelhub/`. See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` (slices 3–4) - Tunnel-hub GitHub authentication (opt-in, gated on `--github-oauth-client-id`; absent = today's open hub): **viewer plane** — GitHub OAuth web flow + HMAC-signed SSO session cookie (`Domain=.`), owner-checked previews (`--require-auth` default on → 302 to login / 403 non-owner; soft mode filters the listing only), `/api/backends` filtered to the viewer (unauthenticated → 401), admin "signed in as" via `/api/whoami`. **Registration plane** — durable, hashed hub API keys (`--keys-file`, default `~/.mxcli/hub-keys.json`, survive restarts) presented as `X-Hub-Key` → stamps `Backend.Owner`; shared `X-Hub-Secret` still works as an owner-less fallback. **Key issuance** — the hub's `/cli` browser page mints a key from the session cookie (no PAT; the device flow was removed as Claude Code containers block GitHub's device endpoints), rotate-by-default + count + revoke-all; `mxcli auth hub login --token ` is the headless path; `run --hub` reads `MXCLI_HUB_KEY` (env → `~/.mxcli/auth.json`) and degrades to local-only if registration fails. Append-only JSONL audit trail (`--audit-log`, no secrets). Packages: `cmd/mxcli/tunnelhub/` (+`audit/`), `cmd/mxcli/hubauth/`. See `docs/11-proposals/PROPOSAL_hub_authentication.md` diff --git a/README.md b/README.md index 8e750c9c2..5a96ca3d3 100644 --- a/README.md +++ b/README.md @@ -212,10 +212,18 @@ Or build from source (Go + Make — `make build` runs the ANTLR parser generatio ```bash git clone https://github.com/mendixlabs/mxcli.git cd mxcli +make -C mdl/grammar bootstrap # one-time: installs the pinned ANTLR generator +export ANTLR4_TOOLS_ANTLR_VERSION=4.13.2 make build # binary is at ./bin/mxcli ``` +The generator version is pinned and load-bearing: it must match the ANTLR +runtime in `go.mod`, or the generated parser will not compile. `antlr4-tools` +downloads the ANTLR jar on first run, so the build needs **network and a JVM**, +not only Go. Skip the bootstrap if you already have an `antlr4` launcher on +PATH (e.g. `brew install antlr4`). + > `go install …@latest` is not supported: the generated ANTLR parser isn't committed, so a module-source build fails. Use a pre-built binary or `make build`. ## Core Features @@ -498,7 +506,7 @@ mxcli add-tool cursor - `.ai-context/examples/` - Example MDL scripts **Tool-Specific:** -- **Claude Code**: `.claude/settings.json`, `CLAUDE.md`, commands, lint-rules, skills +- **Claude Code**: `.claude/settings.json`, `CLAUDE.md`, commands, lint-rules, `lint-config.yaml` (System module excluded from lint), skills - **Cursor**: `.cursorrules` - Compact MDL reference - **Continue.dev**: `.continue/config.json` - Custom commands and slash commands - **Windsurf**: `.windsurfrules` - MDL rules for Codeium diff --git a/cmd/mxcli/cmd_check.go b/cmd/mxcli/cmd_check.go index eacfb5931..89705bbb7 100644 --- a/cmd/mxcli/cmd_check.go +++ b/cmd/mxcli/cmd_check.go @@ -15,7 +15,7 @@ import ( ) var checkCmd = &cobra.Command{ - Use: "check ", + Use: "check ", Short: "Check an MDL script for errors without executing it", Long: `Check an MDL script file for syntax errors and optionally validate references. @@ -46,6 +46,9 @@ Examples: # Output as JSON or SARIF mxcli check script.mdl --format json mxcli check script.mdl --format sarif + + # Read the script from stdin + cat script.mdl | mxcli check - `, Args: cobra.ExactArgs(1), Run: func(cmd *cobra.Command, args []string) { @@ -59,8 +62,8 @@ Examples: outputFormat := linter.OutputFormat(format) formatter := linter.GetFormatter(outputFormat, !isStructured) - // Read the file - content, err := os.ReadFile(filePath) + // Read the script (a path, or "-" for stdin) + content, err := readMDLSource(filePath) if err != nil { fmt.Fprintf(os.Stderr, "Error reading file: %v\n", err) os.Exit(1) @@ -68,7 +71,7 @@ Examples: // Parse the script if !isStructured { - fmt.Printf("Checking syntax: %s\n", filePath) + fmt.Printf("Checking syntax: %s\n", mdlSourceLabel(filePath)) } prog, errs := visitor.Build(string(content)) if len(errs) > 0 { @@ -170,6 +173,23 @@ Examples: // not row-scoped, so the argument is unbound (CE1571) at build time. violations = append(violations, executor.ValidatePageButtonContext(prog)...) + // Flag a database-connection TYPE Studio Pro does not offer. mxcli writes + // the string through and mxbuild does not check it, so a wrong value + // builds green and simply does not connect. + violations = append(violations, executor.ValidateDatabaseConnectionType(prog)...) + + // Flag OData property names nothing below will act on. The grammar takes + // any `name: value` pair, so a typo used to be discarded in silence and + // the model quietly lacked what the author asked for. + violations = append(violations, executor.ValidateODataProperties(prog)...) + + // Flag a page whose widgets point at a page created further down the same + // script. `exec` resolves page references in statement order and is not + // transactional, so this fails after earlier statements are already + // written. --references catches it too, but the ordering needs no project + // when the target is created by a plain CREATE (#9). + violations = append(violations, executor.ValidateScriptPageOrder(prog)...) + // Flag a document-access GRANT naming a role from another module — Mendix // rejects it with CE0148. Needs no project, so it runs here rather than // under --references, where it would only fire with -p (#836). diff --git a/cmd/mxcli/cmd_exec.go b/cmd/mxcli/cmd_exec.go index 4fbe54c9a..b40c15d22 100644 --- a/cmd/mxcli/cmd_exec.go +++ b/cmd/mxcli/cmd_exec.go @@ -13,7 +13,7 @@ import ( ) var execCmd = &cobra.Command{ - Use: "exec ", + Use: "exec ", Short: "Execute an MDL script file", Long: `Execute an MDL script file containing MDL commands. @@ -24,10 +24,16 @@ makes a partially-applied domain script re-runnable — the already-applied statements (e.g. "attribute already exists") error individually while the not- yet-applied ones still run — without a failure masking later work. +Pass "-" as the file to read the script from standard input, so MDL can be +piped or written inline as a heredoc without a temporary file. + Example: mxcli exec setup.mdl mxcli exec -p app.mpr script.mdl mxcli exec -p app.mpr script.mdl --continue-on-error + mxcli exec -p app.mpr - <<'EOF' + SHOW STRUCTURE DEPTH 1; + EOF `, Args: cobra.ExactArgs(1), Run: func(cmd *cobra.Command, args []string) { @@ -35,8 +41,8 @@ Example: projectPath, _ := cmd.Flags().GetString("project") continueOnError, _ := cmd.Flags().GetBool("continue-on-error") - // Read the file - content, err := os.ReadFile(filePath) + // Read the script (a path, or "-" for stdin) + content, err := readMDLSource(filePath) if err != nil { fmt.Fprintf(os.Stderr, "Error reading file: %v\n", err) os.Exit(1) diff --git a/cmd/mxcli/cmd_lint.go b/cmd/mxcli/cmd_lint.go index ac9229c0e..b984463e1 100644 --- a/cmd/mxcli/cmd_lint.go +++ b/cmd/mxcli/cmd_lint.go @@ -7,6 +7,7 @@ import ( "fmt" "os" "path/filepath" + "strings" "github.com/mendixlabs/mxcli/mdl/linter" "github.com/mendixlabs/mxcli/mdl/linter/rules" @@ -186,6 +187,20 @@ Examples: if len(cfg.ExcludeModules) > 0 { merged := append(excludeModules, cfg.ExcludeModules...) ctx.SetExcludedModules(merged) + // An exclude always wins over --modules (LintContext.IsExcluded + // checks the exclude set first), so asking for a module the + // config excludes yields zero findings with no explanation. + // `mxcli init` now ships a config excluding System, which makes + // `lint -m System` exactly that trap — say so rather than + // returning a silent empty result. + if shadowed := intersect(moduleFilter, cfg.ExcludeModules); len(shadowed) > 0 { + fmt.Fprintf(os.Stderr, + "Warning: --modules names %s, but %s excluded by %s — no findings will be reported for %s. Remove it from excludeModules to lint it.\n", + strings.Join(shadowed, ", "), + pluralIsAre(len(shadowed)), + configPath, + pluralItThem(len(shadowed))) + } } cfg.ApplyConfig(lint) } else { @@ -252,3 +267,38 @@ func catalogRefreshCommand(mode linter.CatalogMode) string { return "REFRESH CATALOG" } } + +// intersect returns the values of want that appear in have, preserving want's +// order and dropping duplicates. +func intersect(want, have []string) []string { + if len(want) == 0 || len(have) == 0 { + return nil + } + inHave := make(map[string]bool, len(have)) + for _, h := range have { + inHave[h] = true + } + seen := make(map[string]bool, len(want)) + var out []string + for _, w := range want { + if inHave[w] && !seen[w] { + seen[w] = true + out = append(out, w) + } + } + return out +} + +func pluralIsAre(n int) string { + if n == 1 { + return "it is" + } + return "they are" +} + +func pluralItThem(n int) string { + if n == 1 { + return "it" + } + return "them" +} diff --git a/cmd/mxcli/cmd_new.go b/cmd/mxcli/cmd_new.go index ce026db00..f2a060e3d 100644 --- a/cmd/mxcli/cmd_new.go +++ b/cmd/mxcli/cmd_new.go @@ -25,7 +25,8 @@ This command performs the following steps: 2. Creates a blank Mendix project using mx create-project 3. Applies mxcli's default styling (--theme, see 'mxcli theme list') 4. Initializes AI tooling and devcontainer configuration (mxcli init) - 5. Downloads the correct mxcli binary for the devcontainer (linux) + 5. Runs one build so generated sources are settled (--skip-build to skip) + 6. Links this mxcli into the project (or downloads a Linux build on macOS/Windows) Examples: mxcli new MyApp @@ -40,6 +41,7 @@ Examples: outputDir, _ := cmd.Flags().GetString("output-dir") skipInit, _ := cmd.Flags().GetBool("skip-init") themeName, _ := cmd.Flags().GetString("theme") + skipBuild, _ := cmd.Flags().GetBool("skip-build") if mendixVersion == "" { fmt.Fprintln(os.Stderr, "Error: --version is required (e.g., --version 11.8.0)") @@ -75,7 +77,7 @@ Examples: // On Windows and macOS, Studio Pro ships a native mx binary — prefer it. // CDN downloads contain Linux ELF binaries that cannot run on those platforms. // On Linux (CI, devcontainers), download mxbuild from CDN and derive mx. - fmt.Printf("Step 1/5: Resolving MxBuild %s...\n", mendixVersion) + fmt.Printf("Step 1/6: Resolving MxBuild %s...\n", mendixVersion) mxPath, err := docker.ResolveMxForNewProject(mendixVersion, os.Stdout) if err != nil { fmt.Fprintf(os.Stderr, "Error: could not find mx binary for version %s: %v\n", mendixVersion, err) @@ -86,7 +88,7 @@ Examples: } // Step 2: Create project - fmt.Printf("\nStep 2/5: Creating Mendix project '%s'...\n", appName) + fmt.Printf("\nStep 2/6: Creating Mendix project '%s'...\n", appName) if err := os.MkdirAll(absDir, 0755); err != nil { fmt.Fprintf(os.Stderr, "Error creating directory: %v\n", err) os.Exit(1) @@ -156,7 +158,7 @@ Examples: // writes files under theme/ only — the model is untouched, so the theme // can be re-applied, swapped or removed at any point. if themeName != theme.NoneName { - fmt.Printf("\nStep 3/5: Applying '%s' styling...\n", themeName) + fmt.Printf("\nStep 3/6: Applying '%s' styling...\n", themeName) res, err := theme.Apply(absDir, themeName, theme.Options{}) if err != nil { fmt.Fprintf(os.Stderr, "Error applying theme: %v\n", err) @@ -166,19 +168,38 @@ Examples: fmt.Printf(" %-9s %s\n", f.Action, f.Path) } } else { - fmt.Printf("\nStep 3/5: Skipped styling (--theme none)\n") + fmt.Printf("\nStep 3/6: Skipped styling (--theme none)\n") } // Step 4: Initialize tooling if !skipInit { - fmt.Printf("\nStep 4/5: Initializing AI tooling...\n") + fmt.Printf("\nStep 4/6: Initializing AI tooling...\n") initCmd.Run(initCmd, []string{absDir}) } else { - fmt.Printf("\nStep 4/5: Skipped (--skip-init)\n") + fmt.Printf("\nStep 4/6: Skipped (--skip-init)\n") } - // Step 5: Ensure correct mxcli binary for devcontainer - fmt.Printf("\nStep 5/5: Setting up mxcli binary...\n") + // Step 5: Settle the sources MxBuild generates. The template ships the JS + // and Java action stubs in a slightly older shape and the first build + // rewrites all of them — 48 tracked files in a blank Mendix 11.12 app — so + // without this the project goes dirty the first time anyone builds it, with + // changes nobody wrote (mxcli-todo #7). Doing it here puts the settled form + // in the first commit. Best-effort: this is a nicety, not a precondition + // for a usable project, so a missing JDK or a build failure is a warning. + if skipBuild { + fmt.Printf("\nStep 5/6: Skipped first build (--skip-build)\n") + } else { + fmt.Printf("\nStep 5/6: Running the first build (settles generated sources)...\n") + if err := docker.SettleGeneratedSources(mprPath, mxPath, mendixVersion, os.Stdout); err != nil { + fmt.Fprintf(os.Stderr, " Warning: could not run the first build: %v\n", err) + fmt.Fprintln(os.Stderr, " The project is usable. Note that the first build will rewrite the") + fmt.Fprintln(os.Stderr, " generated action stubs under javascriptsource/ and javasource/ —") + fmt.Fprintln(os.Stderr, " that diff is build output, so commit it and move on.") + } + } + + // Step 6: Ensure correct mxcli binary for devcontainer + fmt.Printf("\nStep 6/6: Setting up mxcli binary...\n") mxcliBinPath := filepath.Join(absDir, "mxcli") if runtime.GOOS != "linux" { // Running on Windows/macOS — download the Linux binary for devcontainer @@ -275,6 +296,8 @@ func init() { newCmd.Flags().String("version", "", "Mendix version (e.g., 11.8.0) — required") newCmd.Flags().String("output-dir", "", "Output directory (default: ./)") newCmd.Flags().Bool("skip-init", false, "Skip AI tooling initialization (mxcli init)") + newCmd.Flags().Bool("skip-build", false, + "Skip the first build (leaves generated action stubs to be rewritten by the next build)") newCmd.Flags().String("theme", theme.DefaultName, "Default styling to apply ('none' to keep plain Atlas; see 'mxcli theme list')") diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index c2df3b6d7..6f28d7418 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -9,6 +9,7 @@ import ( "github.com/mendixlabs/mxcli/cmd/mxcli/docker" "github.com/mendixlabs/mxcli/cmd/mxcli/hubauth" + "github.com/mendixlabs/mxcli/cmd/mxcli/testrunner" "github.com/spf13/cobra" ) @@ -47,6 +48,20 @@ subscriber after start, so the application log lands there too (a standalone runtime attaches no subscriber by default). The path is printed at boot; override with --runtime-log , or "-" to disable. +With --test-endpoint, the app hosts mxcli's token-guarded test endpoint, so +'mxcli test -p --attach' runs a suite against this already-warm +app instead of booting a runtime of its own — a couple of seconds instead of ~30. +The endpoint has to be installed before the boot (its handler is registered by +the after-startup microflow, which only runs at startup), so it cannot be added +to an app that is already up. Your project's own after-startup microflow is +chained, not displaced, so the app still boots the way you expect. The endpoint +is removed and the project restored when the app stops. + +Two things to know: tests then run against THIS app's database, not a scratch +one, and while the app is up its model carries a microflow-executing endpoint — +guarded by a per-run token, loopback-only, and limited to the generated +MxTest.Test_* microflows, but present. Leave the flag off for a normal dev loop. + With --debug, the microflow debugger is enabled at boot and a session is started, so 'mxcli debug break/paused/step/continue' works from another terminal (use the same -p). No breakpoints exist until you set one, so --debug alone does not change @@ -66,6 +81,7 @@ custom OpenTelemetry span filters. Examples: mxcli run --local -p app.mpr mxcli run --local -p app.mpr --watch + mxcli run --local -p app.mpr --test-endpoint # then: mxcli test tests/ -p app.mpr --attach mxcli run --local -p app.mpr --debug # then: mxcli debug break … -p app.mpr mxcli run --local -p app.mpr --app-port 8081 --db-name myapp mxcli run --hub https://hub.example.com -p app.mpr # browser preview @@ -109,6 +125,7 @@ Examples: } watch, _ := cmd.Flags().GetBool("watch") + testEndpoint, _ := cmd.Flags().GetBool("test-endpoint") ensureDB, _ := cmd.Flags().GetBool("ensure-db") setupOnly, _ := cmd.Flags().GetBool("setup") appPort, _ := cmd.Flags().GetInt("app-port") @@ -175,8 +192,40 @@ Examples: Stderr: os.Stderr, } + // --test-endpoint installs the token-guarded test endpoint into the project + // so `mxcli test --attach` can run tests against this app without booting + // its own runtime. It must be installed before the boot (the handler is + // registered by the after-startup microflow, which only runs at startup) + // and removed on the way out. + var hosted *testrunner.HostedEndpoint + if testEndpoint { + if setupOnly { + fmt.Fprintln(os.Stderr, "Error: --test-endpoint has nothing to do with --setup (which never boots the app)") + os.Exit(1) + } + var err error + hosted, err = testrunner.InstallHostedEndpoint(projectPath, os.Stdout) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + // Ctrl-C is the normal way to stop a dev loop, and it must not leave the + // endpoint in the project. RunLocal returns on SIGINT, so the deferred + // removal runs — but os.Exit below would skip it, hence the explicit + // removal on the error path too. Remove is idempotent. + defer hosted.Remove() + opts.Env = append(opts.Env, hosted.Env...) + opts.OnReady = func(info docker.LocalAppInfo) { + if err := hosted.Publish(info); err != nil { + fmt.Fprintf(os.Stderr, "Warning: could not publish the test-endpoint handshake: %v\n", err) + } + } + } + if err := docker.RunLocal(opts); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) + // os.Exit skips deferred calls, so remove explicitly here. + hosted.Remove() os.Exit(1) } }, @@ -193,6 +242,7 @@ func init() { runCmd.Flags().String("hub-worktree", "", "Worktree label to distinguish multiple worktrees of one branch") runCmd.Flags().String("hub-session", "", "Session id to group this preview under in the hub overview (default: CLAUDE_CODE_REMOTE_SESSION_ID / MXCLI_HUB_SESSION)") runCmd.Flags().Bool("watch", false, "Rebuild and hot-apply on every project change") + runCmd.Flags().Bool("test-endpoint", false, "Host mxcli's token-guarded test endpoint so 'mxcli test --attach' can run tests against this app without booting its own runtime (removed on exit)") runCmd.Flags().Bool("ensure-db", false, "Provision the local Postgres + app database if missing (fresh-session bootstrap)") runCmd.Flags().Bool("setup", false, "Prepare prerequisites (cache MxBuild+runtime, ensure DB) and exit without booting — for a SessionStart hook") runCmd.Flags().Int("app-port", 0, "HTTP port for the app (default 8080)") diff --git a/cmd/mxcli/cmd_sync_java_deps.go b/cmd/mxcli/cmd_sync_java_deps.go new file mode 100644 index 000000000..5f96d1ce5 --- /dev/null +++ b/cmd/mxcli/cmd_sync_java_deps.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "os" + "path/filepath" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/mendixlabs/mxcli/sdk/mpr" + "github.com/spf13/cobra" +) + +var syncJavaDepsCmd = &cobra.Command{ + Use: "sync-java-deps", + Short: "Download the project's managed Java (JAR) dependencies into vendorlib/", + Long: `Resolve every managed Java dependency the model declares and download it +into the project's vendorlib/ directory. + +Declaring a dependency and resolving it are separate steps. MDL's +'ALTER MODULE X ADD JAR DEPENDENCY (…)' records the coordinate in the model — +'list jar dependencies' will report it — but nothing downloads the jar, and +MxBuild does not resolve it either: a full build produces a build.gradle with no +dependencies block. Studio Pro runs the resolution for you when you edit Module +Settings; headless, this command is that step. + +Without it the failure is silent until runtime, as a missing-driver exception +from code that looks correctly configured. + +Requires network access (Maven) and the mx binary for the project's version; +'mxcli setup mxbuild' fetches the latter. + +Examples: + mxcli sync-java-deps -p app.mpr + mxcli sync-java-deps -p app.mpr --check # report what is missing, download nothing +`, + RunE: func(cmd *cobra.Command, args []string) error { + projectPath, _ := cmd.Flags().GetString("project") + checkOnly, _ := cmd.Flags().GetBool("check") + if projectPath == "" { + return fmt.Errorf("--project (-p) is required") + } + + deps, version, err := declaredJarDependencies(projectPath) + if err != nil { + return err + } + if len(deps) == 0 { + fmt.Println("No managed Java dependencies declared.") + return nil + } + + missing := docker.UnvendoredJarDependencies(filepath.Dir(projectPath), deps) + fmt.Printf("Declared: %d managed Java dependency/dependencies; %d not in vendorlib/\n", + len(deps), len(missing)) + for _, m := range missing { + fmt.Printf(" missing %s\n", m) + } + if checkOnly { + if len(missing) > 0 { + // A non-zero exit makes this usable as a build-gate. + os.Exit(1) + } + return nil + } + if len(missing) == 0 { + return nil + } + + fmt.Printf("Syncing against Mendix %s...\n", version) + if err := docker.SyncJavaDependencies(projectPath, "", version, os.Stdout); err != nil { + return err + } + if still := docker.UnvendoredJarDependencies(filepath.Dir(projectPath), deps); len(still) > 0 { + // mx reports success even when a coordinate resolves to nothing, so + // check the postcondition rather than trusting the exit code. + return fmt.Errorf("sync finished but %d dependency/dependencies are still missing from vendorlib/: %v", len(still), still) + } + fmt.Println("All declared dependencies are in vendorlib/.") + return nil + }, +} + +// declaredJarDependencies reads the model's managed Java dependencies and the +// project's Mendix version. +func declaredJarDependencies(projectPath string) ([]docker.JarDependencyRef, string, error) { + reader, err := mpr.Open(projectPath) + if err != nil { + return nil, "", fmt.Errorf("opening project: %w", err) + } + defer reader.Close() + + version := reader.ProjectVersion().ProductVersion + + // ListModuleSettings covers every module in one read; the dependency's owner + // does not matter here, only whether the jar is on the classpath. + all, err := reader.ListModuleSettings() + if err != nil { + return nil, version, fmt.Errorf("reading module settings: %w", err) + } + var out []docker.JarDependencyRef + for _, ms := range all { + if ms == nil { + continue + } + for _, d := range ms.JarDependencies { + out = append(out, docker.JarDependencyRef{ + Group: d.GroupID, Artifact: d.ArtifactID, Version: d.Version, + }) + } + } + return out, version, nil +} + +func init() { + syncJavaDepsCmd.Flags().Bool("check", false, "Report missing dependencies and exit non-zero; download nothing") + rootCmd.AddCommand(syncJavaDepsCmd) +} diff --git a/cmd/mxcli/cmd_test_run.go b/cmd/mxcli/cmd_test_run.go index ac30574b2..56f059b1c 100644 --- a/cmd/mxcli/cmd_test_run.go +++ b/cmd/mxcli/cmd_test_run.go @@ -5,6 +5,7 @@ package main import ( "fmt" "os" + "path/filepath" "time" "github.com/mendixlabs/mxcli/cmd/mxcli/testrunner" @@ -27,13 +28,53 @@ Tests use MDL syntax with javadoc-style annotations for expectations: ); / -The test runner: -1. Parses test files and extracts test blocks with @test/@expect annotations -2. Generates a TestRunner microflow -3. Injects it into the project as after-startup microflow -4. Builds and restarts the Mendix runtime in Docker -5. Captures structured log output to determine pass/fail -6. Restores original project settings +With --local the app runs on mxcli's own runtime instead of a container — the +same boot as 'mxcli run --local', so no Docker daemon is needed. It uses its own +ports (8081/8091) and its own '_test' database, so a warm 'run --local' +loop can keep serving the same project while tests run. + +--local also uses a different, better mechanism to run the tests: + + 1. Parses test files and extracts test blocks with @test/@expect annotations + 2. Generates one microflow per test, plus a Java action that registers a + token-guarded HTTP endpoint + 3. Boots the app once — startup only registers the endpoint, it runs no tests + 4. Invokes each test by name over HTTP; the verdict comes back in the response + 5. Restores original project settings + +Because each test is its own microflow invoked on its own, a test that throws +fails only itself instead of ending the run, and results are returned rather +than recovered from the runtime log. + +The endpoint is only reachable from loopback, only with a per-run token passed +to the runtime through its environment (never written into your project), and +will only ever invoke the generated MxTest.Test_* microflows. With no token in +the environment it is not registered at all, so a project that kept the MxTest +module through a failed cleanup exposes nothing when deployed elsewhere. + +--local --watch keeps the runtime and the build server up between runs and +re-runs the suite on every change — to a test file, or to the project's model. +The first run pays the cold boot (~30s); each one after it is a warm rebuild +(~1-4s) plus the tests themselves (milliseconds). Editing a microflow and seeing +whether it still passes is the loop this exists for. Ctrl-C stops watching and +restores the project. + +--attach skips the boot entirely and runs against an app already started with +'mxcli run --local --test-endpoint'. That app's runtime is already warm, so a run +costs only the test-microflow injection, a warm rebuild, and the tests: about two +seconds, with no cold boot at all. It combines with --watch. + +The trade is deliberate and worth knowing: the tests run against the running +app's database, not a scratch one, so they can leave data behind in the app you +are looking at. An attach only ever adds and removes its own test microflows — +the endpoint and the after-startup setting belong to the app hosting them. A +change that needs a runtime restart (a new entity or association) is refused, +since that runtime belongs to the other process. + +Without --local the Docker path is used instead: the suite is compiled into a +single after-startup microflow, the container is restarted, and results are +parsed out of its log. Pass --legacy-runner to use that mechanism on a local run +too, if the endpoint ever misbehaves. Supports two file formats: .test.mdl — Pure MDL test blocks separated by / @@ -52,6 +93,18 @@ Examples: # List tests without executing mxcli test tests/ -p app.mpr --list + # Run without Docker, on mxcli's own local runtime + mxcli test tests/ -p app.mpr --local + + # Keep the runtime warm and re-run on every change + mxcli test tests/ -p app.mpr --local --watch + + # Run against an app already up (mxcli run --local --test-endpoint) — no boot + mxcli test tests/ -p app.mpr --attach + + # ...and re-run on every change, still without owning the runtime + mxcli test tests/ -p app.mpr --attach --watch + # Skip build (reuse existing deployment) mxcli test tests/ -p app.mpr --skip-build @@ -64,6 +117,10 @@ Examples: list, _ := cmd.Flags().GetBool("list") junitOutput, _ := cmd.Flags().GetString("junit") skipBuild, _ := cmd.Flags().GetBool("skip-build") + local, _ := cmd.Flags().GetBool("local") + legacyRunner, _ := cmd.Flags().GetBool("legacy-runner") + watch, _ := cmd.Flags().GetBool("watch") + attach, _ := cmd.Flags().GetBool("attach") verbose, _ := cmd.Flags().GetBool("verbose") color, _ := cmd.Flags().GetBool("color") timeoutStr, _ := cmd.Flags().GetString("timeout") @@ -90,15 +147,19 @@ Examples: } opts := testrunner.RunOptions{ - ProjectPath: projectPath, - TestFiles: args, - SkipBuild: skipBuild, - Timeout: timeout, - JUnitOutput: junitOutput, - Verbose: verbose, - Color: color, - Stdout: os.Stdout, - Stderr: os.Stderr, + ProjectPath: projectPath, + TestFiles: resolveTestPaths(args, projectPath), + SkipBuild: skipBuild, + Local: local, + LegacyRunner: legacyRunner, + Watch: watch, + Attach: attach, + Timeout: timeout, + JUnitOutput: junitOutput, + Verbose: verbose, + Color: color, + Stdout: os.Stdout, + Stderr: os.Stderr, } result, err := testrunner.Run(opts) @@ -107,8 +168,51 @@ Examples: os.Exit(1) } + // A --watch session interrupted before any run completed has no result to + // report. Exiting 0 is right: nothing failed, the user just stopped watching. + if result == nil { + return + } if !result.AllPassed() { os.Exit(1) } }, } + +// resolveTestPaths lets a relative test path be relative to the PROJECT as well +// as to the working directory. +// +// `mxcli test tests/ -p app/App.mpr` used to fail with "no such file or +// directory" for a tests/ that sits right next to the .mpr — the path resolved +// against the process CWD only. That is defensible on its own, but mxcli +// otherwise encourages naming the project rather than standing in its +// directory, so the two conventions collide (mxcli-formula1 findings #13). +// +// The working directory still wins: a tests/ in both places resolves to the one +// the user is standing in, which is what every other tool does. +func resolveTestPaths(paths []string, projectPath string) []string { + if projectPath == "" || len(paths) == 0 { + return paths + } + projectDir := filepath.Dir(projectPath) + out := make([]string, 0, len(paths)) + for _, p := range paths { + if filepath.IsAbs(p) { + out = append(out, p) + continue + } + if _, err := os.Stat(p); err == nil { + out = append(out, p) + continue + } + if alt := filepath.Join(projectDir, p); alt != p { + if _, err := os.Stat(alt); err == nil { + out = append(out, alt) + continue + } + } + // Neither exists: keep the original so the error names what was typed. + out = append(out, p) + } + return out +} diff --git a/cmd/mxcli/cmd_test_run_paths_test.go b/cmd/mxcli/cmd_test_run_paths_test.go new file mode 100644 index 000000000..462638a3a --- /dev/null +++ b/cmd/mxcli/cmd_test_run_paths_test.go @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "testing" +) + +// mxcli-formula1 findings #13: `mxcli test tests/ -p app/App.mpr` failed with +// "no such file or directory" for a tests/ sitting right next to the .mpr, +// because the path resolved against the process CWD only. Defensible alone, but +// mxcli otherwise encourages naming the project instead of standing in its +// directory, so the two conventions collided. +func TestResolveTestPaths(t *testing.T) { + root := t.TempDir() + projectDir := filepath.Join(root, "app") + if err := os.MkdirAll(filepath.Join(projectDir, "tests"), 0o755); err != nil { + t.Fatal(err) + } + projectPath := filepath.Join(projectDir, "App.mpr") + if err := os.WriteFile(projectPath, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + + t.Run("falls back to the project directory", func(t *testing.T) { + got := resolveTestPaths([]string{"tests"}, projectPath) + want := filepath.Join(projectDir, "tests") + if len(got) != 1 || got[0] != want { + t.Errorf("got %v, want [%s]", got, want) + } + }) + + t.Run("the working directory still wins", func(t *testing.T) { + // A tests/ in both places must resolve to the one the user is standing + // in — that is what every other tool does, and silently preferring the + // project's copy would run the wrong suite. + cwdTests := filepath.Join(root, "tests") + if err := os.MkdirAll(cwdTests, 0o755); err != nil { + t.Fatal(err) + } + t.Chdir(root) + + got := resolveTestPaths([]string{"tests"}, projectPath) + if len(got) != 1 || got[0] != "tests" { + t.Errorf("got %v, want the CWD-relative [tests]", got) + } + }) + + t.Run("an absolute path is untouched", func(t *testing.T) { + abs := filepath.Join(projectDir, "tests") + got := resolveTestPaths([]string{abs}, projectPath) + if len(got) != 1 || got[0] != abs { + t.Errorf("got %v, want [%s]", got, abs) + } + }) + + t.Run("a path that exists nowhere keeps what was typed", func(t *testing.T) { + // The error must name what the user wrote, not a rewritten path they + // never mentioned. + got := resolveTestPaths([]string{"nope"}, projectPath) + if len(got) != 1 || got[0] != "nope" { + t.Errorf("got %v, want [nope]", got) + } + }) + + t.Run("no project means no rewriting", func(t *testing.T) { + got := resolveTestPaths([]string{"tests"}, "") + if len(got) != 1 || got[0] != "tests" { + t.Errorf("got %v, want [tests]", got) + } + }) +} diff --git a/cmd/mxcli/docker/javadeps.go b/cmd/mxcli/docker/javadeps.go new file mode 100644 index 000000000..35aa3eb11 --- /dev/null +++ b/cmd/mxcli/docker/javadeps.go @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strings" +) + +// javadeps.go resolves a project's managed Java (JAR) dependencies. +// +// mxcli-formula1 findings #12: `ALTER MODULE X ADD JAR DEPENDENCY (…)` writes the +// coordinate to the model and `list jar dependencies` reports it — but the jar +// never reaches the classpath, and the only symptom is a runtime SQLException +// about a missing driver, long after the model looked right. +// +// The cause is not a bad write. Measured on 11.12.1: a full `mxbuild +// --target=deploy` produces a `deployment/build.gradle` with no dependencies +// block and downloads nothing, while `mx sync-java-dependencies ` +// fetches the jar into `vendorlib/`. Resolution is a separate step that Studio +// Pro runs for you when you edit Module Settings, and that nothing was running +// headless. + +// SyncJavaDependencies runs `mx sync-java-dependencies` against projectPath, +// which downloads every declared managed dependency into the project's +// vendorlib/. It needs network access (Maven) and the mx binary. +// +// mxPath may be empty, in which case mx is resolved from the version's cache. +func SyncJavaDependencies(projectPath, mxPath, version string, w io.Writer) error { + if mxPath == "" { + mxPath = CachedMxPath(version) + } + if mxPath == "" { + // No mx for this exact version. Fall back to any available one — a + // newer mx reads an older project fine — but say so, because a + // version mismatch is otherwise invisible until mx complains about + // the mpr format and the message reads as a corrupt project. + if resolved, err := ResolveMxForVersion("", version); err == nil && resolved != "" { + mxPath = resolved + if w != nil { + fmt.Fprintf(w, " Note: no mx for Mendix %s; using %s. Run 'mxcli setup mxbuild --version %s' if this misbehaves.\n", + version, mxPath, version) + } + } + } + if mxPath == "" { + return fmt.Errorf("mx not found for Mendix %s (needed to resolve managed Java dependencies); run 'mxcli setup mxbuild --version %s'", version, version) + } + + cmd := exec.Command(mxPath, "sync-java-dependencies", projectPath) + cmd.Dir = filepath.Dir(projectPath) + PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround + + out := &syncBuffer{} + cmd.Stdout = out + cmd.Stderr = out + if err := cmd.Run(); err != nil { + return fmt.Errorf("mx sync-java-dependencies failed: %w\n%s", err, lastLines(out.String(), 20)) + } + if w != nil { + if s := strings.TrimSpace(out.String()); s != "" { + fmt.Fprintln(w, " "+strings.ReplaceAll(s, "\n", "\n ")) + } + } + return nil +} + +// UnvendoredJarDependencies returns the coordinates ("group:artifact:version") +// declared by the model that have no matching jar in the project's vendorlib/, +// so a caller can sync only when something is actually missing. +// +// The vendored file name is `-.jar`, which is what +// `mx sync-java-dependencies` writes. +func UnvendoredJarDependencies(projectDir string, deps []JarDependencyRef) []string { + var missing []string + for _, d := range deps { + if d.Artifact == "" || d.Version == "" { + continue + } + jar := filepath.Join(projectDir, "vendorlib", fmt.Sprintf("%s-%s.jar", d.Artifact, d.Version)) + if _, err := os.Stat(jar); err != nil { + missing = append(missing, fmt.Sprintf("%s:%s:%s", d.Group, d.Artifact, d.Version)) + } + } + return missing +} + +// JarDependencyRef is the coordinate of a managed Java dependency, decoupled +// from the model types so this package does not depend on the executor. +type JarDependencyRef struct { + Group string + Artifact string + Version string +} diff --git a/cmd/mxcli/docker/javadeps_test.go b/cmd/mxcli/docker/javadeps_test.go new file mode 100644 index 000000000..c8592c138 --- /dev/null +++ b/cmd/mxcli/docker/javadeps_test.go @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// mxcli-formula1 findings #12: `ALTER MODULE X ADD JAR DEPENDENCY (…)` writes the +// coordinate and `list jar dependencies` reports it, but nothing puts the jar on +// the classpath. Measured on 11.12.1: a full `mxbuild --target=deploy` emits a +// build.gradle with no dependencies block and downloads nothing, while +// `mx sync-java-dependencies` fetches it into vendorlib/ — resolution is a +// separate step. Knowing which coordinates are unvendored is what lets a caller +// run that step only when it is needed. +func TestUnvendoredJarDependencies(t *testing.T) { + dir := t.TempDir() + vendorlib := filepath.Join(dir, "vendorlib") + if err := os.MkdirAll(vendorlib, 0o755); err != nil { + t.Fatal(err) + } + // mx writes -.jar. + if err := os.WriteFile(filepath.Join(vendorlib, "commons-lang3-3.14.0.jar"), []byte("jar"), 0o644); err != nil { + t.Fatal(err) + } + + deps := []JarDependencyRef{ + {Group: "org.apache.commons", Artifact: "commons-lang3", Version: "3.14.0"}, // present + {Group: "org.duckdb", Artifact: "duckdb_jdbc", Version: "1.5.5.1"}, // missing + // A different version of a vendored artifact is still missing: the file + // name carries the version, and the wrong jar is not the right jar. + {Group: "org.apache.commons", Artifact: "commons-lang3", Version: "3.12.0"}, + } + + got := UnvendoredJarDependencies(dir, deps) + if len(got) != 2 { + t.Fatalf("got %d missing, want 2: %v", len(got), got) + } + for _, want := range []string{"org.duckdb:duckdb_jdbc:1.5.5.1", "org.apache.commons:commons-lang3:3.12.0"} { + found := false + for _, g := range got { + if g == want { + found = true + } + } + if !found { + t.Errorf("expected %q among the missing, got: %v", want, got) + } + } +} + +// An incomplete coordinate cannot be checked or resolved, and must not be +// reported as missing — that would send the caller into a sync that cannot help. +func TestUnvendoredJarDependencies_SkipsIncompleteCoordinates(t *testing.T) { + dir := t.TempDir() + deps := []JarDependencyRef{ + {Group: "g", Artifact: "", Version: "1.0"}, + {Group: "g", Artifact: "a", Version: ""}, + } + if got := UnvendoredJarDependencies(dir, deps); len(got) != 0 { + t.Errorf("expected incomplete coordinates to be skipped, got: %v", got) + } +} + +// No vendorlib/ at all is the fresh-clone case: everything declared is missing. +func TestUnvendoredJarDependencies_NoVendorlib(t *testing.T) { + deps := []JarDependencyRef{{Group: "g", Artifact: "a", Version: "1.0"}} + if got := UnvendoredJarDependencies(t.TempDir(), deps); len(got) != 1 { + t.Errorf("got %v, want the one declared dependency", got) + } +} + +// mx's own failure text has to reach the caller. This whole finding was a +// silent failure, so a sync that does not work must not look like one that did. +func TestSyncJavaDependencies_SurfacesMxFailure(t *testing.T) { + missing := filepath.Join(t.TempDir(), "NoSuch.mpr") + err := SyncJavaDependencies(missing, "", "0.0.0-not-a-version", nil) + if err == nil { + t.Fatal("expected an error for a project file that does not exist") + } + // Either mx is unavailable (named as such) or mx ran and complained; both + // are errors that say what happened, which is the contract under test. + msg := err.Error() + if !strings.Contains(msg, "mx not found") && !strings.Contains(msg, "sync-java-dependencies failed") { + t.Errorf("error should name the failure, got: %v", err) + } +} diff --git a/cmd/mxcli/docker/localapp.go b/cmd/mxcli/docker/localapp.go new file mode 100644 index 000000000..d68874178 --- /dev/null +++ b/cmd/mxcli/docker/localapp.go @@ -0,0 +1,272 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os" + "path/filepath" + "time" + + "github.com/mendixlabs/mxcli/sdk/mpr" +) + +// LocalAppOptions configures StartLocalApp. +type LocalAppOptions struct { + // ProjectPath is the .mpr file. + ProjectPath string + // DeployDir is the deployment directory (default /deployment). + DeployDir string + // AppPort / AdminPort / ServePort default to 8080 / 8090 / 6543. + AppPort int + AdminPort int + ServePort int + // AdminPass is the M2EE admin password (defaults to the local-run password). + AdminPass string + // DB is the database to connect to; empty fields take the run --local + // defaults (PostgreSQL at 127.0.0.1:5432, user/password mendix, database + // name derived from the project file name). + DB DBConfig + // EnsureDB provisions the local Postgres + database when missing instead of + // only checking reachability. + EnsureDB bool + // SkipBuild boots against whatever is already in DeployDir. + SkipBuild bool + // RuntimeLogPath tees the runtime JVM output and the runtime's own + // application log to this file. + RuntimeLogPath string + // Env are extra "KEY=value" entries for the runtime JVM (see + // LocalRuntimeOptions.Env) — how a secret reaches the runtime without being + // written to disk. + Env []string + // Stdout/Stderr receive progress messages. + Stdout io.Writer + Stderr io.Writer +} + +// LocalApp is a booted local app: an mxbuild serve server plus the standalone +// runtime it deployed to. It is the Docker-free equivalent of `docker compose +// up` for callers that need an app running and then stopped again. +type LocalApp struct { + Runtime *LocalRuntime + // Version is the project's Mendix version. + Version string + // RuntimeLogPath is where the runtime log is being written (may be empty). + RuntimeLogPath string + + serve *ServeServer +} + +func (o *LocalAppOptions) applyDefaults() { + // Resolve the project path before anything is derived from it: DeployDir + // below, and the runtime's own working directory, both hang off it, and a + // relative value would leave them relative to whatever cwd the caller + // happened to have. ServeServer.Build absolutizes too — that is the backstop + // for MxBuild's own requirement; this is so the paths around it agree. + if o.ProjectPath != "" && !filepath.IsAbs(o.ProjectPath) { + if abs, err := filepath.Abs(o.ProjectPath); err == nil { + o.ProjectPath = abs + } + } + if o.DeployDir == "" { + o.DeployDir = filepath.Join(filepath.Dir(o.ProjectPath), "deployment") + } + if o.AppPort == 0 { + o.AppPort = 8080 + } + if o.AdminPort == 0 { + o.AdminPort = 8090 + } + if o.ServePort == 0 { + o.ServePort = 6543 + } + if o.AdminPass == "" { + o.AdminPass = defaultLocalAdminPass + } + if o.DB.Type == "" { + o.DB.Type = "PostgreSQL" + } + if o.DB.Host == "" { + o.DB.Host = "127.0.0.1:5432" + } + if o.DB.User == "" { + o.DB.User = "mendix" + } + if o.DB.Password == "" { + o.DB.Password = "mendix" + } + if o.DB.Name == "" { + o.DB.Name = deriveDBName(o.ProjectPath) + } + if o.Stdout == nil { + o.Stdout = os.Stdout + } + if o.Stderr == nil { + o.Stderr = os.Stderr + } +} + +// StartLocalApp builds the project with mxbuild and boots the standalone +// runtime against the result — the same sequence as `mxcli run --local`, minus +// everything a headless caller does not need (no web client bundle, no hub, no +// watch loop, no screenshots). +// +// The caller owns the returned app and must Stop it. +func StartLocalApp(opts LocalAppOptions) (*LocalApp, error) { + opts.applyDefaults() + w := opts.Stdout + + if err := checkLocalAppPortsFree(opts); err != nil { + return nil, err + } + + // 1. Project version → which mxbuild and runtime to use. + reader, err := mpr.Open(opts.ProjectPath) + if err != nil { + return nil, fmt.Errorf("opening project: %w", err) + } + version := reader.ProjectVersion().ProductVersion + reader.Close() + + // 2. Cache mxbuild + runtime (no-ops when already present). + if _, err := DownloadMxBuild(version, w); err != nil { + return nil, fmt.Errorf("setting up mxbuild: %w", err) + } + installPath, err := resolveRuntimeInstall(version, w) + if err != nil { + return nil, fmt.Errorf("setting up runtime: %w", err) + } + + // 3. Database. + if opts.EnsureDB { + if err := EnsureDatabase(opts.DB, w); err != nil { + return nil, fmt.Errorf("ensuring database: %w", err) + } + } else if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { + return nil, fmt.Errorf("database not reachable at %s: %w\n"+ + " Pass --ensure-db to provision it, or start Postgres and create the %q database (user %q).", + opts.DB.Host, err, opts.DB.Name, opts.DB.User) + } + + app := &LocalApp{Version: version, RuntimeLogPath: opts.RuntimeLogPath} + + // 4. Build, unless the caller is reusing an existing deployment. + if !opts.SkipBuild { + fmt.Fprintln(w, "Building project (mxbuild --serve)...") + serve, err := StartServe(ServeOptions{Version: version, Host: "127.0.0.1", Port: opts.ServePort}) + if err != nil { + return nil, fmt.Errorf("starting mxbuild serve: %w", err) + } + app.serve = serve + + build, err := serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: opts.ProjectPath}) + if err != nil { + app.Stop() + return nil, fmt.Errorf("build: %w", err) + } + if !build.OK() { + app.Stop() + return nil, fmt.Errorf("build failed: %s\n%s", build.Message, string(build.Raw)) + } + } + + // 5. Boot the runtime against the deployment. + rt, err := StartLocalRuntime(LocalRuntimeOptions{ + DeployDir: opts.DeployDir, + InstallPath: installPath, + AppPort: opts.AppPort, + AdminPort: opts.AdminPort, + AdminPass: opts.AdminPass, + DB: opts.DB, + RuntimeLogPath: opts.RuntimeLogPath, + Env: opts.Env, + Stdout: opts.Stdout, + Stderr: opts.Stderr, + }) + if err != nil { + app.Stop() + return nil, err + } + app.Runtime = rt + return app, nil +} + +// Rebuild rebuilds the project through the warm serve server and applies the +// result to the running runtime, returning whether that was a hot reload or a +// restart. This is the warm loop: the serve server keeps the model loaded, so a +// rebuild is ~1s instead of the ~15s cold build, and the runtime is only +// restarted when the build says the metamodel changed. +// +// A restart re-spawns the JVM from the same options, so anything passed via Env +// — notably the test runner's endpoint token — survives it. +// +// Returns an error if the app was started with SkipBuild: there is no serve +// server to rebuild through. +func (a *LocalApp) Rebuild(projectPath string) (ApplyAction, *BuildResult, error) { + if a.serve == nil { + return ActionReload, nil, fmt.Errorf("this app was started without a build server (SkipBuild); nothing to rebuild through") + } + if a.Runtime == nil { + return ActionReload, nil, fmt.Errorf("the runtime is not running") + } + build, err := a.serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: projectPath}) + if err != nil { + return ActionReload, nil, err + } + if !build.OK() { + return ActionReload, build, fmt.Errorf("build failed: %s", build.Message) + } + action, err := a.Runtime.Controller().ApplyBuild(build, a.Runtime.Restart) + return action, build, err +} + +// ProjectSourceMTime is the newest modification time across a project's model +// source — the change signal a warm loop polls. Exported for callers outside +// this package that run their own watch loop (the test runner). +func ProjectSourceMTime(projectPath string) time.Time { return projectSourceMTime(projectPath) } + +// Stop shuts down the runtime and the build server. Safe to call more than once +// and on a partially-started app. +func (a *LocalApp) Stop() error { + var firstErr error + if a.Runtime != nil { + if err := a.Runtime.Stop(); err != nil { + firstErr = err + } + a.Runtime = nil + } + if a.serve != nil { + if err := a.serve.Stop(); err != nil && firstErr == nil { + firstErr = err + } + a.serve = nil + } + return firstErr +} + +// checkLocalAppPortsFree refuses to boot onto a port something is already +// serving — otherwise a stale runtime is silently adopted and the caller reads +// results from an app it did not build. +func checkLocalAppPortsFree(o LocalAppOptions) error { + ports := []struct { + port int + what string + }{ + {o.AppPort, "app"}, + {o.AdminPort, "runtime admin API"}, + } + if !o.SkipBuild { + ports = append(ports, struct { + port int + what string + }{o.ServePort, "mxbuild serve"}) + } + for _, p := range ports { + if err := pingTCP(fmt.Sprintf("127.0.0.1:%d", p.port), 300*time.Millisecond); err == nil { + return fmt.Errorf("port %d (%s) is already in use — stop the running instance first, "+ + "or pass a different port", p.port, p.what) + } + } + return nil +} diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 08cfa3f9c..ded08b3a2 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -88,6 +88,12 @@ type LocalRuntimeOptions struct { // ReadyTimeout bounds how long StartLocalRuntime waits for the admin API // (default 90s). ReadyTimeout time.Duration + // Env are extra "KEY=value" entries layered onto the runtime JVM's + // environment, last-wins over the inherited process environment. Used to hand + // the runtime a secret that must not be written to disk — the test runner + // passes its per-run endpoint token this way rather than baking it into the + // generated Java source, which would land in the user's javasource/ tree. + Env []string // Stdout/Stderr receive progress messages (default os.Stdout/os.Stderr). Stdout io.Writer Stderr io.Writer @@ -161,14 +167,17 @@ func (o *LocalRuntimeOptions) jvmArgs() []string { // localRuntimeEnv builds the environment for the runtime JVM, layered on the // current process environment. PrepareMxCommand later adds the FreeType fix. +// o.Env is appended last so a caller-supplied value wins over both the inherited +// environment and these defaults. func localRuntimeEnv(o LocalRuntimeOptions) []string { - return append(os.Environ(), + env := append(os.Environ(), "M2EE_ADMIN_PASS="+o.AdminPass, fmt.Sprintf("M2EE_ADMIN_PORT=%d", o.AdminPort), "M2EE_ADMIN_LISTEN_ADDRESSES="+o.ListenAddr, "MX_INSTALL_PATH="+o.InstallPath, "MX_LOG_LEVEL=i", ) + return append(env, o.Env...) } // otelAgentJar locates the OpenTelemetry Java agent bundled with the runtime diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go index b19557eda..af78aeabb 100644 --- a/cmd/mxcli/docker/mxserve.go +++ b/cmd/mxcli/docker/mxserve.go @@ -184,10 +184,26 @@ func (s *ServeServer) waitReady(timeout time.Duration) error { // Build sends a build request to the warm server and parses the result. The // caller inspects RestartRequired to decide reload_model vs restart. +// +// A relative ProjectFilePath is resolved here rather than by each caller. +// MxBuild rejects one outright ("the project file path should be an absolute +// path") and answers with a page of Windows sample requests, which tells a user +// who typed `-p app.mpr` nothing about what to do. `mxcli run` learned to +// absolutize at the CLI layer (findings #17), but that left the requirement +// unenforced for every other caller — and `mxcli test --local` then hit exactly +// the same error through StartLocalApp. Doing it at the one place that talks to +// MxBuild is what stops a third caller finding it again. func (s *ServeServer) Build(req BuildRequest) (*BuildResult, error) { if req.Target == "" { req.Target = TargetDeploy } + if req.ProjectFilePath != "" && !filepath.IsAbs(req.ProjectFilePath) { + abs, err := filepath.Abs(req.ProjectFilePath) + if err != nil { + return nil, fmt.Errorf("resolving project path %q: %w", req.ProjectFilePath, err) + } + req.ProjectFilePath = abs + } bodyBytes, err := json.Marshal(req) if err != nil { return nil, fmt.Errorf("marshaling build request: %w", err) diff --git a/cmd/mxcli/docker/mxserve_test.go b/cmd/mxcli/docker/mxserve_test.go index 3c0f774a6..f991569a1 100644 --- a/cmd/mxcli/docker/mxserve_test.go +++ b/cmd/mxcli/docker/mxserve_test.go @@ -4,7 +4,8 @@ package docker import ( "encoding/json" - "net" + "fmt" + "io" "net/http" "net/http/httptest" "net/url" @@ -14,138 +15,113 @@ import ( "testing" ) -// newTestServe returns a ServeServer wired to an httptest server, so the HTTP -// client (Build) can be tested without spawning mxbuild. -func newTestServe(t *testing.T, handler http.HandlerFunc) *ServeServer { +// fakeServe stands in for `mxbuild --serve`, recording the build request it was +// sent so a test can assert on what actually went over the wire. +func fakeServe(t *testing.T) (*ServeServer, *BuildRequest) { t.Helper() - ts := httptest.NewServer(handler) - t.Cleanup(ts.Close) - u, err := url.Parse(ts.URL) + var got BuildRequest + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(body, &got) + w.Header().Set("Content-Type", "application/json") + fmt.Fprint(w, `{"status":"Success"}`) + })) + t.Cleanup(srv.Close) + + u, err := url.Parse(srv.URL) if err != nil { - t.Fatalf("parse test URL: %v", err) + t.Fatalf("parsing test server URL: %v", err) } - host, portStr, err := net.SplitHostPort(u.Host) + port, err := strconv.Atoi(u.Port()) if err != nil { - t.Fatalf("split host:port: %v", err) + t.Fatalf("parsing test server port: %v", err) } - port, _ := strconv.Atoi(portStr) - return &ServeServer{Host: host, Port: port} + return &ServeServer{Host: u.Hostname(), Port: port}, &got } -func TestServeBuild_DeployRestartRequired(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/build" { - t.Errorf("path = %q, want /build", r.URL.Path) - } - if r.Method != http.MethodPost { - t.Errorf("method = %q, want POST", r.Method) - } - var req BuildRequest - if err := json.NewDecoder(r.Body).Decode(&req); err != nil { - t.Errorf("decode request: %v", err) - } - if req.Target != TargetDeploy { - t.Errorf("target = %q, want Deploy (default)", req.Target) - } - if req.ProjectFilePath != "/x/App.mpr" { - t.Errorf("projectFilePath = %q", req.ProjectFilePath) - } - _, _ = w.Write([]byte(`{"restartRequired": true, "status": "Success"}`)) - }) +// TestBuildAbsolutizesProjectPath pins the fix for MxBuild's "the project file +// path should be an absolute path" rejection. `mxcli run` used to absolutize at +// the CLI layer, which left `mxcli test --local` hitting the raw error through +// StartLocalApp; doing it in Build covers every caller. +func TestBuildAbsolutizesProjectPath(t *testing.T) { + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(mpr, []byte("x"), 0o600); err != nil { + t.Fatalf("writing fixture: %v", err) + } - res, err := s.Build(BuildRequest{ProjectFilePath: "/x/App.mpr"}) // Target empty -> Deploy + // Run from the project directory so "App.mpr" is a valid relative path. + cwd, err := os.Getwd() if err != nil { - t.Fatalf("Build: %v", err) - } - if !res.OK() { - t.Errorf("OK() = false, status = %q", res.Status) + t.Fatalf("getwd: %v", err) } - if !res.RestartRequired { - t.Error("RestartRequired = false, want true (domain/view-entity change)") + if err := os.Chdir(dir); err != nil { + t.Fatalf("chdir: %v", err) } -} + t.Cleanup(func() { os.Chdir(cwd) }) -func TestServeBuild_HotReloadable(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - _, _ = w.Write([]byte(`{"restartRequired": false, "status": "Success"}`)) - }) - res, err := s.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: "/x/App.mpr"}) - if err != nil { + srv, got := fakeServe(t) + if _, err := srv.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: "App.mpr"}); err != nil { t.Fatalf("Build: %v", err) } - if !res.OK() { - t.Errorf("OK() = false, status = %q", res.Status) + + if !filepath.IsAbs(got.ProjectFilePath) { + t.Fatalf("MxBuild was sent a relative path %q; it rejects those", got.ProjectFilePath) } - if res.RestartRequired { - t.Error("RestartRequired = true, want false (microflow/page change -> reload_model)") + // EvalSymlinks because macOS /tmp is a symlink to /private/tmp. + wantResolved, _ := filepath.EvalSymlinks(mpr) + gotResolved, _ := filepath.EvalSymlinks(got.ProjectFilePath) + if gotResolved != wantResolved { + t.Errorf("sent %q, want %q", gotResolved, wantResolved) } } -func TestServeBuild_Failure(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - w.WriteHeader(http.StatusBadRequest) - _, _ = w.Write([]byte(`{"status": "Failure", "message": "Specified 'target' is invalid."}`)) - }) - res, err := s.Build(BuildRequest{Target: "Bogus", ProjectFilePath: "/x/App.mpr"}) - if err != nil { - t.Fatalf("Build should parse the failure envelope, got transport error: %v", err) - } - if res.OK() { - t.Error("OK() = true, want false") +// TestBuildLeavesAnAbsolutePathAlone guards against the resolution mangling a +// path that was already correct. +func TestBuildLeavesAnAbsolutePathAlone(t *testing.T) { + abs := filepath.Join(t.TempDir(), "App.mpr") + srv, got := fakeServe(t) + if _, err := srv.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: abs}); err != nil { + t.Fatalf("Build: %v", err) } - if res.Message == "" { - t.Error("Message empty, want the failure message") + if got.ProjectFilePath != abs { + t.Errorf("absolute path was rewritten: got %q, want %q", got.ProjectFilePath, abs) } } -func TestServeBuild_PackageTargetSendsMdaPath(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - var req BuildRequest - _ = json.NewDecoder(r.Body).Decode(&req) - if req.Target != TargetPackage { - t.Errorf("target = %q, want Package", req.Target) - } - if req.MdaFilePath != "/out/app.mda" { - t.Errorf("mdaFilePath = %q, want /out/app.mda", req.MdaFilePath) - } - _, _ = w.Write([]byte(`{"restartRequired": true, "status": "Success"}`)) - }) - if _, err := s.Build(BuildRequest{Target: TargetPackage, ProjectFilePath: "/x/App.mpr", MdaFilePath: "/out/app.mda"}); err != nil { +// TestBuildDefaultsTargetToDeploy pins the pre-existing default, which the +// absolutization now sits next to. +func TestBuildDefaultsTargetToDeploy(t *testing.T) { + srv, got := fakeServe(t) + if _, err := srv.Build(BuildRequest{ProjectFilePath: filepath.Join(t.TempDir(), "App.mpr")}); err != nil { t.Fatalf("Build: %v", err) } + if got.Target != TargetDeploy { + t.Errorf("Target = %q, want %q", got.Target, TargetDeploy) + } } -func TestVerifyMxBuildCache(t *testing.T) { - // layout: /modeler/mxbuild and /runtime - cache := t.TempDir() - modeler := filepath.Join(cache, "modeler") - if err := os.MkdirAll(modeler, 0o755); err != nil { - t.Fatal(err) +// TestLocalAppOptionsAbsolutizeProjectPath pins that DeployDir is not derived +// from a relative project path. +func TestLocalAppOptionsAbsolutizeProjectPath(t *testing.T) { + dir := t.TempDir() + cwd, err := os.Getwd() + if err != nil { + t.Fatalf("getwd: %v", err) } - mxbuild := filepath.Join(modeler, "mxbuild") - if err := os.WriteFile(mxbuild, []byte("#!/bin/sh\n"), 0o755); err != nil { - t.Fatal(err) + if err := os.Chdir(dir); err != nil { + t.Fatalf("chdir: %v", err) } + t.Cleanup(func() { os.Chdir(cwd) }) - // runtime/ missing -> error - if err := verifyMxBuildCache(mxbuild); err == nil { - t.Error("expected error when runtime/ dir is missing") - } + o := LocalAppOptions{ProjectPath: "App.mpr"} + o.applyDefaults() - // runtime/ present -> ok - if err := os.MkdirAll(filepath.Join(cache, "runtime"), 0o755); err != nil { - t.Fatal(err) - } - if err := verifyMxBuildCache(mxbuild); err != nil { - t.Errorf("expected no error when runtime/ present, got %v", err) + if !filepath.IsAbs(o.ProjectPath) { + t.Errorf("ProjectPath = %q, want an absolute path", o.ProjectPath) } -} - -func TestServeBuild_BadJSONBody(t *testing.T) { - s := newTestServe(t, func(w http.ResponseWriter, r *http.Request) { - _, _ = w.Write([]byte(`not json`)) - }) - if _, err := s.Build(BuildRequest{ProjectFilePath: "/x/App.mpr"}); err == nil { - t.Error("expected an error decoding a non-JSON body") + if !filepath.IsAbs(o.DeployDir) { + t.Errorf("DeployDir = %q, want an absolute path", o.DeployDir) } } diff --git a/cmd/mxcli/docker/oql.go b/cmd/mxcli/docker/oql.go index 6e5d71e0d..b41b10395 100644 --- a/cmd/mxcli/docker/oql.go +++ b/cmd/mxcli/docker/oql.go @@ -169,20 +169,40 @@ func parseOQLFeedback(rawFeedback json.RawMessage) (*OQLResult, error) { return result, nil } - // Extract column order from the first row using json.Decoder Token() method - columns, err := extractColumnOrder(rows[0]) - if err != nil { - return nil, fmt.Errorf("extracting columns: %w", err) - } - result.Columns = columns - - // Parse each row preserving column order + // The column set is the union of the keys of every row, not the keys of the + // first one: the runtime omits a column from a row's JSON object when its + // value is null, so a column that happens to be null in row 1 is absent + // there and would otherwise be dropped from the whole result — silently + // answering a different query than the one that was asked. + var columns []string + known := make(map[string]bool) + rowMaps := make([]map[string]any, 0, len(rows)) for _, rawRow := range rows { var rowMap map[string]any if err := json.Unmarshal(rawRow, &rowMap); err != nil { return nil, fmt.Errorf("parsing row: %w", err) } + rowMaps = append(rowMaps, rowMap) + + // Re-scanning a row for key order is only needed when it carries a + // column not seen yet; the common case (every row has the same keys) + // costs one length check. + if !hasOnlyKnownKeys(rowMap, known) { + keys, err := extractColumnOrder(rawRow) + if err != nil { + return nil, fmt.Errorf("extracting columns: %w", err) + } + columns = mergeColumnOrder(columns, keys) + for _, col := range columns { + known[col] = true + } + } + } + result.Columns = columns + // Project each row onto the merged column order. A column missing from a + // row is a null value, which formats as NULL. + for _, rowMap := range rowMaps { row := make([]any, len(columns)) for i, col := range columns { row[i] = rowMap[col] @@ -193,6 +213,48 @@ func parseOQLFeedback(rawFeedback json.RawMessage) (*OQLResult, error) { return result, nil } +// hasOnlyKnownKeys reports whether every key of rowMap is already a known column. +func hasOnlyKnownKeys(rowMap map[string]any, known map[string]bool) bool { + if len(rowMap) > len(known) { + return false + } + for key := range rowMap { + if !known[key] { + return false + } + } + return true +} + +// mergeColumnOrder folds one row's key order into the accumulated column list. +// +// New keys are inserted directly after the last key that was already known, +// rather than appended, so a column absent from earlier rows still lands in its +// SELECT position: merging [A, C] with [A, B, C] yields [A, B, C], not +// [A, C, B]. +func mergeColumnOrder(columns []string, rowKeys []string) []string { + index := make(map[string]int, len(columns)) + for i, col := range columns { + index[col] = i + } + + insertAt := 0 // just past the last key of this row found in columns + for _, key := range rowKeys { + if pos, ok := index[key]; ok { + insertAt = pos + 1 + continue + } + columns = append(columns, "") + copy(columns[insertAt+1:], columns[insertAt:]) + columns[insertAt] = key + for i := insertAt; i < len(columns); i++ { + index[columns[i]] = i + } + insertAt++ + } + return columns +} + // extractColumnOrder uses json.Decoder to preserve key order from a JSON object. func extractColumnOrder(raw json.RawMessage) ([]string, error) { dec := json.NewDecoder(bytes.NewReader(raw)) diff --git a/cmd/mxcli/docker/oql_test.go b/cmd/mxcli/docker/oql_test.go index 84aa1ef89..7d12bb90e 100644 --- a/cmd/mxcli/docker/oql_test.go +++ b/cmd/mxcli/docker/oql_test.go @@ -316,6 +316,85 @@ func TestExecuteOQL_ColumnOrder(t *testing.T) { } } +// TestParseOQLFeedback_ColumnUnionAcrossRows covers the runtime's habit of +// omitting a column from a row's JSON object when its value is null. Taking the +// column set from row 1 alone dropped such a column from the entire result, so +// the table silently answered a narrower query than the one that was asked. +func TestParseOQLFeedback_ColumnUnionAcrossRows(t *testing.T) { + tests := []struct { + name string + feedback string + want []string + wantRows [][]any + }{ + { + name: "column null in first row survives", + feedback: `{"data":[{"Name":"Alice"},{"Name":"Bob","Nickname":"Bobby"}]}`, + want: []string{"Name", "Nickname"}, + wantRows: [][]any{{"Alice", nil}, {"Bob", "Bobby"}}, + }, + { + name: "missing middle column keeps its SELECT position", + feedback: `{"data":[{"A":"a1","C":"c1"},{"A":"a2","B":"b2","C":"c2"}]}`, + want: []string{"A", "B", "C"}, + wantRows: [][]any{{"a1", nil, "c1"}, {"a2", "b2", "c2"}}, + }, + { + name: "column null in every row but one is still reported", + feedback: `{"data":[{"A":"a1"},{"A":"a2"},{"A":"a3","B":"b3"}]}`, + want: []string{"A", "B"}, + wantRows: [][]any{{"a1", nil}, {"a2", nil}, {"a3", "b3"}}, + }, + { + name: "uniform rows keep first-row order", + feedback: `{"data":[{"Zebra":"z","Alpha":"a"},{"Zebra":"z2","Alpha":"a2"}]}`, + want: []string{"Zebra", "Alpha"}, + wantRows: [][]any{{"z", "a"}, {"z2", "a2"}}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + result, err := parseOQLFeedback(json.RawMessage(tt.feedback)) + if err != nil { + t.Fatalf("parseOQLFeedback: %v", err) + } + if got := fmt.Sprintf("%v", result.Columns); got != fmt.Sprintf("%v", tt.want) { + t.Errorf("columns: got %v, want %v", result.Columns, tt.want) + } + if got, want := fmt.Sprintf("%v", result.Rows), fmt.Sprintf("%v", tt.wantRows); got != want { + t.Errorf("rows: got %s, want %s", got, want) + } + }) + } +} + +func TestMergeColumnOrder(t *testing.T) { + tests := []struct { + name string + columns []string + keys []string + want string + }{ + {"first row seeds the order", nil, []string{"A", "B"}, "[A B]"}, + {"known keys change nothing", []string{"A", "B"}, []string{"A", "B"}, "[A B]"}, + {"new key inserted in position", []string{"A", "C"}, []string{"A", "B", "C"}, "[A B C]"}, + {"new leading key goes first", []string{"B"}, []string{"A", "B"}, "[A B]"}, + {"new trailing key goes last", []string{"A"}, []string{"A", "B"}, "[A B]"}, + {"two new keys keep their order", []string{"A", "D"}, []string{"A", "B", "C", "D"}, "[A B C D]"}, + {"row with only unknown keys appends", []string{"A"}, []string{"B", "C"}, "[B C A]"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := fmt.Sprintf("%v", mergeColumnOrder(tt.columns, tt.keys)) + if got != tt.want { + t.Errorf("mergeColumnOrder(%v, %v) = %s, want %s", tt.columns, tt.keys, got, tt.want) + } + }) + } +} + // parseTestServerAddr extracts host and port from an httptest server URL. func parseTestServerAddr(t *testing.T, rawURL string) (string, int) { t.Helper() diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 1353a1658..96023d7a2 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -9,6 +9,7 @@ import ( "io/fs" "net" "net/http" + "net/url" "os" "os/signal" "path/filepath" @@ -17,6 +18,7 @@ import ( "syscall" "time" + "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/mpr" ) @@ -27,6 +29,17 @@ import ( // serve build's restartRequired flag decides which. See // docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md. +// LocalAppInfo is what a running local app exposes to another process: the +// loopback ports it serves on, and the admin password needed to drive its M2EE +// API. The admin password is separate from any application-level credential — +// conflating the two is an authentication failure at the first admin call. +type LocalAppInfo struct { + AppPort int + AdminPort int + ServePort int + AdminPass string +} + // LocalRunOptions configures RunLocal. type LocalRunOptions struct { // ProjectPath is the .mpr file. @@ -118,6 +131,15 @@ type LocalRunOptions struct { // charts, since the console exporter omits timestamps/parent span IDs. // Implies Trace. TraceOTLP string + // Env are extra "KEY=value" entries for the runtime JVM (see + // LocalRuntimeOptions.Env) — how a secret reaches the runtime without being + // written to disk. `--test-endpoint` passes the test-endpoint token this way. + Env []string + // OnReady, when set, is called once the app is serving, with everything a + // second process needs to drive it. Used by `--test-endpoint` to publish its + // handshake only after there is something for `mxcli test --attach` to + // connect to. + OnReady func(LocalAppInfo) // RuntimeSettings are raw "Key=Value" runtime settings merged into the boot // update_configuration payload (Value is parsed as JSON, else a string), e.g. // 'Metrics.Registries=[{"type":"otlp"}]' or @@ -235,6 +257,86 @@ func parseRuntimeSetting(s string) (string, any, error) { // deriveDBName turns a project file name into a safe Postgres database name: // lowercased, non-alphanumerics collapsed to underscores, leading digit prefixed. +// configuredApplicationRootURL returns the ApplicationRootUrl set on the +// project's server configuration, plus the name of the configuration it came +// from. Empty when no configuration sets one, which is the default. +// +// The model has no "active configuration" marker — Studio Pro remembers the +// selection per developer — so the one named "Default" wins, falling back to +// the first configuration that actually sets a URL. A settings read failure is +// not fatal: the run simply proceeds without a root URL, exactly as before. +func configuredApplicationRootURL(reader *mpr.Reader) (rootURL, configName string) { + settings, err := reader.GetProjectSettings() + if err != nil { + return "", "" + } + return applicationRootURLFrom(settings) +} + +// applicationRootURLFrom implements the selection rule over already-read +// settings, so it is testable without a project on disk. +func applicationRootURLFrom(settings *model.ProjectSettings) (rootURL, configName string) { + if settings == nil || settings.Configuration == nil { + return "", "" + } + first, firstName := "", "" + for _, cfg := range settings.Configuration.Configurations { + if cfg == nil || cfg.ApplicationRootUrl == "" { + continue + } + if strings.EqualFold(cfg.Name, "Default") { + return cfg.ApplicationRootUrl, cfg.Name + } + if first == "" { + first, firstName = cfg.ApplicationRootUrl, cfg.Name + } + } + return first, firstName +} + +// customHostRootURL reports whether an ApplicationRootUrl names a host worth +// telling the runtime about. +// +// A blank Mendix app already ships `http://localhost:8080/`, so "set" does not +// mean "chosen". Passing that back would be a behaviour change for every +// existing project — and an actively wrong one under --app-port, where the +// stock value names a port the app is not serving on. A loopback host also adds +// nothing: with no ApplicationRootUrl the runtime derives one from the listen +// address, which is the same thing. So only a real host name — the case this +// exists for, giving each app in a solution its own name — is honoured. +func customHostRootURL(rootURL string) bool { + if rootURL == "" { + return false + } + u, err := url.Parse(rootURL) + if err != nil || u.Host == "" { + return false + } + host := strings.ToLower(u.Hostname()) + if host == "localhost" || host == "::1" { + return false + } + if ip := net.ParseIP(host); ip != nil && ip.IsLoopback() { + return false + } + return true +} + +// urlPort returns the explicit port of a URL, or "" when it has none. +func urlPort(rawURL string) string { + u, err := url.Parse(rawURL) + if err != nil { + return "" + } + return u.Port() +} + +// DeriveDBName is the local-run database name for a project: the .mpr file name +// lowercased and sanitised to a legal identifier. Exported so callers that boot +// their own local app (e.g. the test runner, which appends a suffix to keep test +// data out of the dev database) derive the same base name. +func DeriveDBName(projectPath string) string { return deriveDBName(projectPath) } + func deriveDBName(projectPath string) string { base := strings.TrimSuffix(filepath.Base(projectPath), filepath.Ext(projectPath)) var b strings.Builder @@ -451,6 +553,15 @@ func RunLocal(opts LocalRunOptions) error { return fmt.Errorf("opening project: %w", err) } pv := reader.ProjectVersion() + // Read the configured application root URL while the project is open — the + // app's own configuration is where a custom host name belongs (App Settings -> + // Configurations in Studio Pro, `alter settings` in MDL), versioned with the + // app rather than repeated on every command line. + modelRootURL, modelRootConfig := configuredApplicationRootURL(reader) + // Managed Java dependencies are declared in the model but resolved by a + // separate step; collect them here so the boot can vendor any that are + // missing (mxcli-formula1 findings #12). + declaredJars := declaredJarDependencies(reader) reader.Close() version := pv.ProductVersion fmt.Fprintf(w, " Mendix version: %s\n", version) @@ -466,7 +577,22 @@ func RunLocal(opts LocalRunOptions) error { return fmt.Errorf("setting up runtime: %w", err) } - // 3. Ensure the database is available. With --ensure-db, provision it (start + // 3. Vendor any declared Java dependency that is not in vendorlib/. MxBuild + // does not resolve these (a full build emits no dependencies block), so + // without this the app builds green and throws "no driver found" at runtime. + // Best-effort: it needs network, and a project whose jars are already + // vendored — the common case — does no work at all. + if missing := UnvendoredJarDependencies(filepath.Dir(opts.ProjectPath), declaredJars); len(missing) > 0 { + fmt.Fprintf(w, "Resolving %d managed Java dependency/dependencies (%s)...\n", + len(missing), strings.Join(missing, ", ")) + if err := SyncJavaDependencies(opts.ProjectPath, "", version, w); err != nil { + fmt.Fprintf(stderr, " Warning: could not resolve Java dependencies: %v\n", err) + fmt.Fprintln(stderr, " The app will build, but code needing those jars fails at runtime.") + fmt.Fprintf(stderr, " Retry with: mxcli sync-java-deps -p %s\n", opts.ProjectPath) + } + } + + // 4. Ensure the database is available. With --ensure-db, provision it (start // local Postgres + create the role/db if missing); otherwise just check // reachability and point the user at --ensure-db. if opts.EnsureDB { @@ -488,7 +614,7 @@ func RunLocal(opts LocalRunOptions) error { return nil } - // 4. Start the warm build server. + // 5. Start the warm build server. fmt.Fprintln(w, "Starting mxbuild --serve...") serve, err := StartServe(ServeOptions{ Version: version, @@ -560,6 +686,22 @@ func RunLocal(opts LocalRunOptions) error { } } + // No hub URL: fall back to the one configured in the project. This is what + // makes "give each app its own host name" work for a local run — Mendix needs + // to know the URL it is reached at to generate absolute URLs (OIDC/SAML + // redirect URIs, deep links) that point at the host name rather than the + // listen address. A hub assignment wins, since that URL is the one actually + // serving the app. + if appRootURL == "" && customHostRootURL(modelRootURL) { + appRootURL = modelRootURL + fmt.Fprintf(w, "Application root URL from configuration %q: %s\n", modelRootConfig, appRootURL) + if port := urlPort(appRootURL); port != "" && port != fmt.Sprint(opts.AppPort) { + fmt.Fprintf(stderr, "Warning: configuration %q says port %s but the app is serving on %d — "+ + "absolute URLs will point at %s. Update the configuration or pass --app-port %s.\n", + modelRootConfig, port, opts.AppPort, appRootURL, port) + } + } + // 6. Boot the runtime against the fresh deployment. Tee the runtime's own // stdout/stderr to a log file so server-side errors are debuggable ("-" // disables). (findings #25) @@ -589,6 +731,7 @@ func RunLocal(opts LocalRunOptions) error { Trace: opts.Trace, TraceServiceName: traceService, TraceOTLPEndpoint: opts.TraceOTLP, + Env: opts.Env, Stdout: w, Stderr: stderr, }) @@ -597,6 +740,15 @@ func RunLocal(opts LocalRunOptions) error { } defer rt.Stop() + if opts.OnReady != nil { + opts.OnReady(LocalAppInfo{ + AppPort: opts.AppPort, + AdminPort: opts.AdminPort, + ServePort: opts.ServePort, + AdminPass: opts.AdminPass, + }) + } + fmt.Fprintf(w, "\nApp is running at %s\n", rt.AppURL()) // The local runtime boots with the live-preview dev flags (see // LocalRuntimeOptions.jvmArgs), so `mxcli oql` can query it directly — and it @@ -698,7 +850,7 @@ func RunLocal(opts LocalRunOptions) error { fmt.Fprintf(w, "Logging in as %q for authenticated screenshots...\n", opts.ScreenshotUser) if err := LoginAndSaveStorage(LoginOptions{ AppURL: rt.AppURL(), Username: opts.ScreenshotUser, Password: opts.ScreenshotPassword, - StoragePath: storage, MxBuildPath: mxbuildPath, + StoragePath: storage, MxBuildPath: mxbuildPath, RuntimeLogPath: runtimeLog, }); err != nil { fmt.Fprintf(stderr, " screenshot login failed (continuing unauthenticated): %v\n", err) } else { @@ -1019,3 +1171,24 @@ func watchAndApply(opts LocalRunOptions, serve *ServeServer, rt *LocalRuntime, w } } } + +// declaredJarDependencies collects the managed Java dependency coordinates the +// model declares, across every module. +func declaredJarDependencies(reader *mpr.Reader) []JarDependencyRef { + all, err := reader.ListModuleSettings() + if err != nil { + return nil + } + var out []JarDependencyRef + for _, ms := range all { + if ms == nil { + continue + } + for _, d := range ms.JarDependencies { + out = append(out, JarDependencyRef{ + Group: d.GroupID, Artifact: d.ArtifactID, Version: d.Version, + }) + } + } + return out +} diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go index 06eb1cd3b..15c2e6d4f 100644 --- a/cmd/mxcli/docker/runlocal_test.go +++ b/cmd/mxcli/docker/runlocal_test.go @@ -11,6 +11,8 @@ import ( "strings" "testing" "time" + + "github.com/mendixlabs/mxcli/model" ) func TestDeriveDBName(t *testing.T) { @@ -489,3 +491,91 @@ func TestCheckTargetPortsFree(t *testing.T) { t.Errorf("error should explain the port is in use; got: %v", err) } } + +// The app's host name belongs in the project's configuration (App Settings -> +// Configurations), not on the command line. These pin the selection rule. +func TestApplicationRootURLFrom(t *testing.T) { + cfg := func(name, url string) *model.ServerConfiguration { + return &model.ServerConfiguration{Name: name, ApplicationRootUrl: url} + } + settings := func(cfgs ...*model.ServerConfiguration) *model.ProjectSettings { + return &model.ProjectSettings{Configuration: &model.ConfigurationSettings{Configurations: cfgs}} + } + + tests := []struct { + name string + in *model.ProjectSettings + wantURL string + wantConfig string + }{ + {"no settings at all", nil, "", ""}, + {"no configuration part", &model.ProjectSettings{}, "", ""}, + {"no configurations", settings(), "", ""}, + {"configuration without a URL", settings(cfg("Default", "")), "", ""}, + { + "single configuration", + settings(cfg("Default", "http://backend.local:8080/")), + "http://backend.local:8080/", "Default", + }, + { + // No "active configuration" marker exists in the model, so Default wins + // wherever it sits in the list. + "Default wins over others", + settings(cfg("Acceptance", "https://acc.example.com/"), cfg("Default", "http://backend.local:8080/")), + "http://backend.local:8080/", "Default", + }, + {"Default match is case-insensitive", settings(cfg("default", "http://x/")), "http://x/", "default"}, + { + "first one that sets a URL when there is no Default", + settings(cfg("Local", ""), cfg("Test", "https://test.example.com/"), cfg("Acc", "https://acc.example.com/")), + "https://test.example.com/", "Test", + }, + {"nil entries are skipped", settings(nil, cfg("Default", "http://y/")), "http://y/", "Default"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + url, name := applicationRootURLFrom(tt.in) + if url != tt.wantURL || name != tt.wantConfig { + t.Errorf("got (%q, %q), want (%q, %q)", url, name, tt.wantURL, tt.wantConfig) + } + }) + } +} + +// A blank Mendix app already sets ApplicationRootUrl to http://localhost:8080/, +// so honouring every configured value would change behaviour for every existing +// project — and name the wrong port under --app-port. Only a real host name is +// worth passing to the runtime. +func TestCustomHostRootURL(t *testing.T) { + tests := map[string]bool{ + "": false, + "http://localhost:8080/": false, // the stock value in a blank app + "http://LOCALHOST:8080/": false, + "http://127.0.0.1:8080/": false, + "http://127.0.0.2:8080/": false, + "http://[::1]:8080/": false, + "not a url": false, + "http://backend.local:8080/": true, + "https://app.example.com/": true, + "http://app.127.0.0.1.nip.io:8080/": true, // a name, even if it resolves to loopback + } + for in, want := range tests { + if got := customHostRootURL(in); got != want { + t.Errorf("customHostRootURL(%q) = %v, want %v", in, got, want) + } + } +} + +func TestURLPort(t *testing.T) { + tests := map[string]string{ + "http://backend.local:8080/": "8080", + "https://app.example.com/": "", + "": "", + } + for in, want := range tests { + if got := urlPort(in); got != want { + t.Errorf("urlPort(%q) = %q, want %q", in, got, want) + } + } +} diff --git a/cmd/mxcli/docker/screenshot_login.go b/cmd/mxcli/docker/screenshot_login.go index 56c44114b..a9d42ca64 100644 --- a/cmd/mxcli/docker/screenshot_login.go +++ b/cmd/mxcli/docker/screenshot_login.go @@ -4,9 +4,11 @@ package docker import ( "fmt" + "io" "os" "os/exec" "path/filepath" + "strings" "time" ) @@ -65,8 +67,11 @@ const { chromium } = require(require.resolve("playwright-core", { paths: [pkgDir const ctx = await b.newContext(); const p = await ctx.newPage(); await p.goto(appURL, { waitUntil: "load", timeout: 30000 }); + let sawForm = false; + let failure = ""; try { await p.waitForSelector("#usernameInput", { timeout: 8000 }); + sawForm = true; await p.fill("#usernameInput", username); await p.fill("#passwordInput", password); await Promise.all([ @@ -78,8 +83,21 @@ const { chromium } = require(require.resolve("playwright-core", { paths: [pkgDir // No login form within the timeout: anonymous or already authenticated. process.stderr.write("login: no login form detected (" + e.message.split("\n")[0] + ")\n"); } + // Mendix answers a rejected sign-in by re-rendering the same form, so the + // username field still being there is the signal that login did not complete. + // Without this the script saved an anonymous session and every later + // screenshot silently showed the login page instead of the requested page. + if (sawForm && (await p.locator("#usernameInput").count()) > 0) { + const alert = ((await p.locator(".alert").first().textContent().catch(() => "")) || "") + .replace(/\s+/g, " ").trim().slice(0, 200); + failure = alert || "still on the login page after submitting"; + } await ctx.storageState({ path: storagePath }); await b.close(); + if (failure) { + process.stderr.write("login: sign-in did not complete: " + failure + "\n"); + process.exit(2); + } })().catch((e) => { process.stderr.write(String(e) + "\n"); process.exit(1); }); ` @@ -88,9 +106,58 @@ type LoginOptions struct { AppURL string Username string Password string - StoragePath string // where to write the Playwright storage state JSON - MxBuildPath string // fallback node source - Timeout time.Duration // default 60s + StoragePath string // where to write the Playwright storage state JSON + MxBuildPath string // fallback node source + // RuntimeLogPath is consulted when a sign-in is rejected: the runtime + // records why, and the page does not. + RuntimeLogPath string + Timeout time.Duration // default 60s +} + +// sessionCapMarker is what the unlicensed runtime logs when it refuses a sign-in +// because every session slot is taken. The browser only ever says "Sign in +// failed", which points at the credentials — so the cause is invisible unless +// someone thinks to open the runtime log. +const sessionCapMarker = "Maximum number of sessions exceeded" + +// loginFailureHint returns an explanation to append to a failed sign-in, read +// from the tail of the runtime log. Empty when the log says nothing useful. +func loginFailureHint(runtimeLogPath string) string { + if runtimeLogPath == "" || runtimeLogPath == "-" { + return "" + } + tail, err := readLogTail(runtimeLogPath, 64*1024) + if err != nil || !strings.Contains(tail, sessionCapMarker) { + return "" + } + return "\nThe runtime log reports: " + sessionCapMarker + " — the unlicensed local\n" + + "runtime allows only a handful of concurrent sessions, and the login page reports\n" + + "that as \"Sign in failed\" as if the password were wrong. Sessions are released by\n" + + "restarting `mxcli run --local`; a script that drives the app through a browser\n" + + "should sign out when it finishes so it does not consume a slot per run." +} + +// readLogTail reads at most the last n bytes of a file. +func readLogTail(path string, n int64) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + info, err := f.Stat() + if err != nil { + return "", err + } + if off := info.Size() - n; off > 0 { + if _, err := f.Seek(off, io.SeekStart); err != nil { + return "", err + } + } + b, err := io.ReadAll(f) + if err != nil { + return "", err + } + return string(b), nil } // LoginAndSaveStorage logs into the app and writes a Playwright storage-state @@ -137,7 +204,7 @@ func LoginAndSaveStorage(opts LoginOptions) error { select { case err := <-done: if err != nil { - return fmt.Errorf("login failed: %w\n%s", err, out.String()) + return fmt.Errorf("login failed: %w\n%s%s", err, out.String(), loginFailureHint(opts.RuntimeLogPath)) } case <-time.After(timeout): _ = cmd.Process.Kill() diff --git a/cmd/mxcli/docker/screenshot_login_test.go b/cmd/mxcli/docker/screenshot_login_test.go new file mode 100644 index 000000000..89a0bb8d7 --- /dev/null +++ b/cmd/mxcli/docker/screenshot_login_test.go @@ -0,0 +1,82 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// mxcli-todo findings #16: the unlicensed local runtime caps concurrent +// sessions, and the login page reports that as "Sign in failed" — pointing at +// the credentials. The real reason is only in the runtime log, so a failed +// sign-in has to go and read it. +func TestLoginFailureHint(t *testing.T) { + dir := t.TempDir() + + capped := filepath.Join(dir, "capped.log") + if err := os.WriteFile(capped, []byte( + "INFO - Core: Starting\n"+ + "ERROR - Security: Maximum number of sessions exceeded! (You are currently using a trial license)\n", + ), 0o644); err != nil { + t.Fatal(err) + } + hint := loginFailureHint(capped) + if hint == "" { + t.Fatal("expected a hint when the log shows the session cap") + } + // The hint is only worth printing if it names the cause and the way out. + for _, want := range []string{sessionCapMarker, "Sign in failed", "run --local"} { + if !strings.Contains(hint, want) { + t.Errorf("hint should mention %q, got:\n%s", want, hint) + } + } + + quiet := filepath.Join(dir, "quiet.log") + if err := os.WriteFile(quiet, []byte("INFO - Core: Starting\n"), 0o644); err != nil { + t.Fatal(err) + } + if got := loginFailureHint(quiet); got != "" { + t.Errorf("expected no hint for an ordinary log, got: %s", got) + } + + // A disabled (`-`), unset, or missing log must not turn into an error path: + // the login failure itself is what the caller reports. + for _, path := range []string{"", "-", filepath.Join(dir, "absent.log")} { + if got := loginFailureHint(path); got != "" { + t.Errorf("loginFailureHint(%q) = %q, want empty", path, got) + } + } +} + +// The marker can sit far back in a long-running log; only the tail is read, so +// check the tail window actually holds recent lines. +func TestReadLogTail(t *testing.T) { + path := filepath.Join(t.TempDir(), "runtime.log") + body := strings.Repeat("filler line that is here only to push the file past the window\n", 2000) + if err := os.WriteFile(path, []byte(body+"the last line\n"), 0o644); err != nil { + t.Fatal(err) + } + + tail, err := readLogTail(path, 1024) + if err != nil { + t.Fatal(err) + } + if len(tail) > 1024 { + t.Errorf("read %d bytes, want at most 1024", len(tail)) + } + if !strings.Contains(tail, "the last line") { + t.Error("tail should contain the end of the file") + } + + // A file smaller than the window is read whole, not skipped. + small := filepath.Join(t.TempDir(), "small.log") + if err := os.WriteFile(small, []byte("short\n"), 0o644); err != nil { + t.Fatal(err) + } + if got, err := readLogTail(small, 1024); err != nil || got != "short\n" { + t.Errorf("readLogTail(small) = %q, %v", got, err) + } +} diff --git a/cmd/mxcli/docker/settle.go b/cmd/mxcli/docker/settle.go new file mode 100644 index 000000000..60331528c --- /dev/null +++ b/cmd/mxcli/docker/settle.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os/exec" + "path/filepath" + "strings" +) + +// settle.go runs one deploy build against a freshly created project so that the +// sources MxBuild generates are already in their post-build shape. +// +// mxcli-todo findings #7: the template ships `javascriptsource/*/actions/*.js` +// (and the matching Java stubs) in a slightly older shape, and MxBuild rewrites +// every one of them on the first build — banner, `import { Big } from "big.js"`, +// `async function X()` -> `export async function X()`. They are *tracked* files, +// so a fresh clone goes dirty on the first build (48 of them in a blank Mendix +// 11.12 app) and stays dirty until someone commits build output they did not +// write. `mx check` does not do this; only a build does. Doing that build while +// the project is still being created means the first commit already holds the +// settled form. + +// SettleGeneratedSources runs `mxbuild --target=deploy` once against +// projectPath. It is best-effort by contract: every failure is returned for the +// caller to report as a warning, never as a reason to fail project creation. +// +// mxPath is the `mx` binary already resolved by the caller — mxbuild lives +// beside it — and version is used to find a cached download when it does not. +func SettleGeneratedSources(projectPath, mxPath, version string, w io.Writer) error { + mxbuildPath := resolveMxBuildForSettle(mxPath, version) + if mxbuildPath == "" { + return fmt.Errorf("mxbuild not found next to %s or in the cache for %s", mxPath, version) + } + javaHome, err := resolveJDK21() + if err != nil { + return fmt.Errorf("no JDK 21 available: %w", err) + } + + cmd := exec.Command(mxbuildPath, + "--target=deploy", + fmt.Sprintf("--java-home=%s", javaHome), + fmt.Sprintf("--java-exe-path=%s", filepath.Join(javaHome, "bin", "java")), + projectPath, + ) + cmd.Dir = filepath.Dir(projectPath) + PrepareMxCommand(cmd) // FreeType LD_PRELOAD workaround + + // MxBuild is very chatty and this build is incidental to what the user asked + // for, so its output is only worth showing when it fails. + out := &syncBuffer{} + cmd.Stdout = out + cmd.Stderr = out + if err := cmd.Run(); err != nil { + return fmt.Errorf("mxbuild failed: %w\n%s", err, lastLines(out.String(), 20)) + } + fmt.Fprintln(w, " Generated sources are in their post-build form.") + return nil +} + +// resolveMxBuildForSettle finds mxbuild beside the resolved mx binary (the +// Studio Pro / CDN layout puts them in the same directory), falling back to the +// version's download cache. +func resolveMxBuildForSettle(mxPath, version string) string { + if mxPath != "" { + if found := findMxBuildInDir(filepath.Dir(mxPath)); found != "" { + return found + } + } + return CachedMxBuildPath(version) +} + +// lastLines returns the final n lines of s — enough to show why a build failed +// without pasting the whole log into a creation summary. +func lastLines(s string, n int) string { + lines := strings.Split(strings.TrimRight(s, "\n"), "\n") + if len(lines) > n { + lines = lines[len(lines)-n:] + } + return strings.Join(lines, "\n") +} diff --git a/cmd/mxcli/docker/settle_test.go b/cmd/mxcli/docker/settle_test.go new file mode 100644 index 000000000..e98c0c978 --- /dev/null +++ b/cmd/mxcli/docker/settle_test.go @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "os" + "path/filepath" + "runtime" + "strings" + "testing" +) + +// mxcli-todo findings #7: mxbuild sits beside the mx binary `mxcli new` already +// resolved, so the settle build should not need a second download — including on +// macOS/Windows, where the resolved mx comes from a Studio Pro install that no +// version cache knows about. +func TestResolveMxBuildForSettle_NextToMx(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("binary names differ on Windows; the Linux/macOS layout is what this checks") + } + dir := t.TempDir() + mxPath := filepath.Join(dir, "mx") + mxbuildPath := filepath.Join(dir, "mxbuild") + for _, p := range []string{mxPath, mxbuildPath} { + if err := os.WriteFile(p, []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + } + + if got := resolveMxBuildForSettle(mxPath, "11.12.1"); got != mxbuildPath { + t.Errorf("got %q, want %q", got, mxbuildPath) + } +} + +// With no mxbuild beside mx and nothing cached for the version, there is nothing +// to run — the caller must get "" and report a warning, not run a random binary. +func TestResolveMxBuildForSettle_NotFound(t *testing.T) { + dir := t.TempDir() + mxPath := filepath.Join(dir, "mx") + if err := os.WriteFile(mxPath, []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + // A version string no cache directory can match. + if got := resolveMxBuildForSettle(mxPath, "0.0.0-not-a-version"); got != "" { + t.Errorf("got %q, want empty", got) + } +} + +func TestLastLines(t *testing.T) { + tests := []struct { + name string + in string + n int + want string + }{ + {"fewer lines than asked for", "a\nb\n", 5, "a\nb"}, + {"truncates to the tail", "a\nb\nc\nd\n", 2, "c\nd"}, + {"empty", "", 3, ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := lastLines(tt.in, tt.n); got != tt.want { + t.Errorf("lastLines(%q, %d) = %q, want %q", tt.in, tt.n, got, tt.want) + } + }) + } +} + +// A settle build that cannot run must return an error the caller can print as a +// warning — never panic, and never take the project creation down with it. +func TestSettleGeneratedSources_NoMxBuild(t *testing.T) { + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(mpr, []byte("not a real project"), 0o644); err != nil { + t.Fatal(err) + } + err := SettleGeneratedSources(mpr, filepath.Join(dir, "mx"), "0.0.0-not-a-version", os.Stdout) + if err == nil { + t.Fatal("expected an error when mxbuild cannot be found") + } + if !strings.Contains(err.Error(), "mxbuild") { + t.Errorf("error should name what is missing, got: %v", err) + } +} diff --git a/cmd/mxcli/init.go b/cmd/mxcli/init.go index df8c83281..ca43f9a5c 100644 --- a/cmd/mxcli/init.go +++ b/cmd/mxcli/init.go @@ -10,8 +10,10 @@ import ( "path/filepath" "runtime" "slices" + "sort" "strings" + "github.com/mendixlabs/mxcli/mdl/linter" "github.com/spf13/cobra" ) @@ -37,6 +39,9 @@ const mendixGitignore = `# Mendix project /packages/ /project-settings.user.json /releases/ +# Compiled theme output. MxBuild regenerates it on every build, so tracking it +# means a fresh clone goes dirty the first time anyone builds. (mxcli-todo #7) +/theme-cache/ *.mpr.lock *.mpr.bak /vendorlib/temp/ @@ -130,10 +135,33 @@ Container Runtime: os.Exit(1) } - // Find .mpr file + // Find .mpr file. With none here, look one level down: a solution repo + // keeps each app in its own folder, and running `mxcli init` from the + // root used to write everything at the root against an invented + // `project.mpr` that does not exist — silently wrong, and in a two-app + // repo the odds of it being what you meant are zero. + // (mxcli-formula1 findings #3.) mprFile := findMprFile(absDir) if mprFile == "" { - mprFile = "project.mpr" // Default if not found + candidates := findMprFilesInSubdirs(absDir) + switch len(candidates) { + case 0: + fmt.Fprintf(os.Stderr, "Warning: no .mpr file found in %s.\n", absDir) + fmt.Fprintln(os.Stderr, " Generated files will refer to 'project.mpr'; run this from the app folder to get real paths.") + mprFile = "project.mpr" + case 1: + absDir = filepath.Dir(candidates[0]) + mprFile = filepath.Base(candidates[0]) + fmt.Printf("No .mpr here; initializing the project found below: %s\n", candidates[0]) + default: + fmt.Fprintf(os.Stderr, "Error: %d Mendix projects found below %s:\n", len(candidates), absDir) + for _, c := range candidates { + fmt.Fprintf(os.Stderr, " %s\n", c) + } + fmt.Fprintln(os.Stderr, "\nName the one you mean, so a solution repo does not get one app's tooling by coin flip:") + fmt.Fprintf(os.Stderr, " mxcli init %s\n", filepath.Dir(candidates[0])) + os.Exit(1) + } } projectName := filepath.Base(absDir) @@ -213,6 +241,12 @@ Container Runtime: } } + // Seed the lint config. `mxcli lint` reads this regardless of which AI + // tool was selected, so it is written outside the per-tool branches. + if path, created := writeDefaultLintConfig(absDir); created { + fmt.Printf(" Created %s (System module excluded from lint)\n", filepath.Base(path)) + } + // Write universal skills to .ai-context/skills/ skillCount := 0 err = fs.WalkDir(skillsFS, "skills", func(path string, d fs.DirEntry, err error) error { @@ -579,6 +613,63 @@ Container Runtime: }, } +// defaultLintConfig is the lint configuration written into a freshly +// initialised project. System is excluded because its contents are Mendix's, +// not the developer's: you cannot document its entities, give them access +// rules, or rename their members. Linting it produced ~100 un-actionable +// findings on a blank app, which buried the handful about the developer's own +// code (issuetracker finding #9). +const defaultLintConfig = `# mxcli lint configuration. +# Docs: mxcli lint --help + +# Modules that 'mxcli lint' skips entirely. +# +# System is Mendix's own platform module — its entities, members and access +# rules are not yours to change, so findings against it are noise. On a blank +# app it accounts for the large majority of all issues. +# +# Marketplace modules (Atlas_Core, Atlas_Web_Content, Administration, …) are +# equally read-only in practice; add them here if their findings distract you. +# +# NOTE: this list always wins. It is merged with '--exclude', and a module +# listed here is skipped even if you ask for it with '--modules'. To lint +# System, remove it from this list (or delete this file). +excludeModules: + - System + +# Per-rule overrides. Examples: +# +# rules: +# QUAL002: # missing documentation +# enabled: false +# CONV009: # max microflow objects +# severity: warning +# options: +# maxObjects: 20 +rules: {} +` + +// writeDefaultLintConfig creates .claude/lint-config.yaml unless the project +// already has a lint config in any of the locations linter.FindConfigFile +// searches. Never overwrites: init is re-runnable, and the config is meant to +// be edited. +func writeDefaultLintConfig(projectDir string) (string, bool) { + if existing := linter.FindConfigFile(projectDir); existing != "" { + return existing, false + } + claudeDir := filepath.Join(projectDir, ".claude") + if err := os.MkdirAll(claudeDir, 0755); err != nil { + fmt.Fprintf(os.Stderr, " Error creating .claude directory for lint config: %v\n", err) + return "", false + } + path := filepath.Join(claudeDir, "lint-config.yaml") + if err := os.WriteFile(path, []byte(defaultLintConfig), 0644); err != nil { + fmt.Fprintf(os.Stderr, " Error writing lint config: %v\n", err) + return "", false + } + return path, true +} + func findMprFile(dir string) string { entries, err := os.ReadDir(dir) if err != nil { @@ -606,3 +697,26 @@ func init() { initCmd.Flags().BoolVar(&initListTools, "list-tools", false, "List supported AI tools and exit") initCmd.Flags().StringVar(&initContainerRuntime, "container-runtime", "docker", "Container runtime for devcontainer (docker or podman)") } + +// findMprFilesInSubdirs returns the .mpr files one level below dir, sorted, so +// the choice is deterministic and the error message lists them in a stable +// order. One level only: a Mendix app keeps its .mpr at its root, and walking +// deeper would find deployment copies and backups. +func findMprFilesInSubdirs(dir string) []string { + entries, err := os.ReadDir(dir) + if err != nil { + return nil + } + var out []string + for _, e := range entries { + if !e.IsDir() || strings.HasPrefix(e.Name(), ".") { + continue + } + sub := filepath.Join(dir, e.Name()) + if mpr := findMprFile(sub); mpr != "" { + out = append(out, filepath.Join(sub, mpr)) + } + } + sort.Strings(out) + return out +} diff --git a/cmd/mxcli/init_discover_test.go b/cmd/mxcli/init_discover_test.go new file mode 100644 index 000000000..f21bd3a33 --- /dev/null +++ b/cmd/mxcli/init_discover_test.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "testing" +) + +// mxcli-formula1 findings #3: with no .mpr at the target, `mxcli init` used to +// carry on against an invented `project.mpr`, writing tooling that points at a +// file which does not exist. In a solution repo — one app folder per app — that +// is silently the wrong answer, and with two apps there is no right guess. +func TestFindMprFilesInSubdirs(t *testing.T) { + t.Run("finds one project per subdirectory, sorted", func(t *testing.T) { + root := t.TempDir() + for _, name := range []string{"Zeta", "Alpha"} { + dir := filepath.Join(root, name) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, name+".mpr"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + } + got := findMprFilesInSubdirs(root) + if len(got) != 2 { + t.Fatalf("got %d candidates, want 2: %v", len(got), got) + } + // Sorted, so the refusal message and the suggested command are stable + // rather than dependent on directory order. + if filepath.Base(got[0]) != "Alpha.mpr" || filepath.Base(got[1]) != "Zeta.mpr" { + t.Errorf("candidates are not sorted: %v", got) + } + }) + + t.Run("ignores dot directories", func(t *testing.T) { + root := t.TempDir() + hidden := filepath.Join(root, ".mendix-cache") + if err := os.MkdirAll(hidden, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(hidden, "stale.mpr"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + if got := findMprFilesInSubdirs(root); len(got) != 0 { + t.Errorf("a dot directory is not a candidate project, got: %v", got) + } + }) + + t.Run("one level only", func(t *testing.T) { + root := t.TempDir() + deep := filepath.Join(root, "apps", "Nested") + if err := os.MkdirAll(deep, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(deep, "Nested.mpr"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + // A Mendix app keeps its .mpr at its own root; walking deeper would + // start finding deployment copies and backups. + if got := findMprFilesInSubdirs(root); len(got) != 0 { + t.Errorf("expected no candidates two levels down, got: %v", got) + } + }) + + t.Run("empty directory", func(t *testing.T) { + if got := findMprFilesInSubdirs(t.TempDir()); len(got) != 0 { + t.Errorf("got %v, want none", got) + } + }) +} diff --git a/cmd/mxcli/init_hook.go b/cmd/mxcli/init_hook.go index aa4304422..007f740a6 100644 --- a/cmd/mxcli/init_hook.go +++ b/cmd/mxcli/init_hook.go @@ -3,6 +3,7 @@ package main import ( + "bytes" "encoding/json" "fmt" "os" @@ -19,14 +20,88 @@ import ( // The hook is setup-only (non-blocking): it must return, so it prepares // prerequisites and exits rather than booting the long-lived warm loop. -// sessionStartHookMarker identifies our hook command so re-running init is -// idempotent and we never clobber a user's own SessionStart hooks. -const sessionStartHookMarker = "run --local --setup" +// bootstrapScriptName is the committed script the SessionStart hook runs, +// relative to the project root. +const bootstrapScriptName = ".claude/bootstrap-mxcli.sh" -// sessionStartHookCommand is the shell command the hook runs. It is guarded so a -// missing ./mxcli (or a setup hiccup) never blocks the session from starting. -func sessionStartHookCommand(mprFile string) string { - return fmt.Sprintf("test -x ./mxcli && ./mxcli run --local --setup --ensure-db -p %s || true", mprFile) +// bootstrapScriptTemplate is written to bootstrapScriptName. %s is the .mpr file +// name. It is POSIX sh (no bashisms) and safe to re-run: every step is a no-op +// once satisfied. +// +// It is COMMITTED on purpose — the opposite of the mxcli binary, which +// .gitignore excludes for its size. After an idle reap the container is +// reclaimed and the repo re-cloned, and this file is then the only thing in the +// tree that can bring the binary back. +const bootstrapScriptTemplate = `#!/bin/sh +# Generated by 'mxcli init'. COMMIT THIS FILE. +# +# Run by the Claude Code SessionStart hook to make a fresh (or reaped and +# re-cloned) session ready to work: fetch mxcli if the binary is missing, then +# cache MxBuild + the runtime and provision the local database. +# +# The mxcli binary is git-ignored (~85 MB), so after an idle reap it is NOT in +# the fresh clone — which is exactly why this script fetches it rather than +# skipping when it is absent. +# +# Pin a specific mxcli with MXCLI_TAG=vX.Y.Z (default: nightly). +set -e + +MPR='%s' +TAG="${MXCLI_TAG:-nightly}" + +if [ ! -x ./mxcli ]; then + os=$(uname -s | tr 'A-Z' 'a-z') + case "$(uname -m)" in + x86_64|amd64) arch=amd64 ;; + arm64|aarch64) arch=arm64 ;; + *) arch=$(uname -m) ;; + esac + url="https://github.com/mendixlabs/mxcli/releases/download/${TAG}/mxcli-${os}-${arch}" + echo "mxcli not found — downloading ${TAG} for ${os}/${arch}..." + if ! curl -fsSL -o ./mxcli "$url"; then + echo "Could not download mxcli from ${url}." >&2 + echo "Fetch it manually, or set MXCLI_TAG to a released version." >&2 + exit 1 + fi + chmod +x ./mxcli +fi + +exec ./mxcli run --local --setup --ensure-db -p "$MPR" +` + +// sessionStartHookMarkers identify OUR hook command — so re-running init updates +// it instead of duplicating it, and a user's own SessionStart hooks are never +// touched. The first is the current form; the rest are older spellings kept so +// existing projects migrate rather than accumulate a second hook. +var sessionStartHookMarkers = []string{ + bootstrapScriptName, // current: the committed bootstrap script + "run --local --setup", // pre-bootstrap-script: the command inlined in the hook +} + +// isMxcliSessionStartHook reports whether a hook command is one mxcli wrote. +func isMxcliSessionStartHook(command string) bool { + for _, m := range sessionStartHookMarkers { + if strings.Contains(command, m) { + return true + } + } + return false +} + +// sessionStartHookCommand is the shell command the hook runs. It delegates to a +// committed script rather than inlining the work, for two reasons: +// +// 1. The inlined form was guarded on `test -x ./mxcli`, and the binary it +// guards on is git-ignored (~85 MB, deliberately). In an ephemeral container +// the repo is re-cloned after an idle reap, `./mxcli` is not in the clone, +// the guard fails and the hook silently no-ops through `|| true` — leaving +// exactly the unprepared session the hook exists to prevent, with no error +// saying so. The script can fetch the binary back; a hook line cannot +// reasonably do OS/arch detection. (mxcli-todo findings #2) +// 2. The command string stays constant, so re-running `mxcli init` recognises +// its own hook even when the project is renamed. +func sessionStartHookCommand() string { + return fmt.Sprintf("sh %s || true", bootstrapScriptName) } // ensureSessionStartHook adds (idempotently) the mxcli bring-up to @@ -36,15 +111,22 @@ func sessionStartHookCommand(mprFile string) string { func ensureSessionStartHook(claudeDir, mprFile string) (changed bool, err error) { path := filepath.Join(claudeDir, "settings.json") + // The script is rewritten every time so a renamed .mpr (or an improvement to + // the script itself) is picked up; it holds no user content. + scriptChanged, err := writeBootstrapScript(claudeDir, mprFile) + if err != nil { + return false, err + } + settings := map[string]any{} if data, readErr := os.ReadFile(path); readErr == nil { if json.Unmarshal(data, &settings) != nil { - return false, fmt.Errorf("%s exists but is not valid JSON; leaving it untouched — add a SessionStart hook manually", path) + return scriptChanged, fmt.Errorf("%s exists but is not valid JSON; leaving it untouched — add a SessionStart hook manually", path) } } - if updated := addSessionStartHook(settings, sessionStartHookCommand(mprFile)); !updated { - return false, nil // already present + if updated := addSessionStartHook(settings, sessionStartHookCommand()); !updated { + return scriptChanged, nil // hook already current } out, err := json.MarshalIndent(settings, "", " ") @@ -58,11 +140,31 @@ func ensureSessionStartHook(claudeDir, mprFile string) (changed bool, err error) return true, nil } +// writeBootstrapScript writes (or refreshes) the committed bootstrap script the +// SessionStart hook runs. Reports whether the content changed. +func writeBootstrapScript(claudeDir, mprFile string) (changed bool, err error) { + // claudeDir is /.claude and bootstrapScriptName is project-relative, + // so take the base name to land beside settings.json. + path := filepath.Join(claudeDir, filepath.Base(bootstrapScriptName)) + want := []byte(fmt.Sprintf(bootstrapScriptTemplate, mprFile)) + if existing, readErr := os.ReadFile(path); readErr == nil && bytes.Equal(existing, want) { + return false, nil + } + if err := os.MkdirAll(claudeDir, 0o755); err != nil { + return false, err + } + if err := os.WriteFile(path, want, 0o755); err != nil { + return false, fmt.Errorf("writing %s: %w", path, err) + } + return true, nil +} + // addSessionStartHook inserts a SessionStart command hook into a parsed settings -// map, preserving existing keys and hooks. It returns false if a SessionStart -// hook whose command contains sessionStartHookMarker already exists (idempotent). -// Exported-for-test via the package; operates on the generic JSON shape so it -// never drops unknown settings. +// map, preserving existing keys and hooks. An entry matching any known marker is +// UPDATED in place rather than duplicated, so a project written by an older +// mxcli (which inlined the whole command) migrates to the current one instead of +// ending up with two hooks that both run. Returns whether anything changed. +// Operates on the generic JSON shape so it never drops unknown settings. func addSessionStartHook(settings map[string]any, command string) bool { hooks, _ := settings["hooks"].(map[string]any) if hooks == nil { @@ -81,9 +183,16 @@ func addSessionStartHook(settings map[string]any, command string) bool { if !ok { continue } - if c, _ := hm["command"].(string); strings.Contains(c, sessionStartHookMarker) { - return false // already configured + c, _ := hm["command"].(string) + if !isMxcliSessionStartHook(c) { + continue + } + if c == command { + return false // already current } + hm["command"] = command + settings["hooks"] = hooks + return true } } diff --git a/cmd/mxcli/init_hook_test.go b/cmd/mxcli/init_hook_test.go index 305e20d4d..087f99087 100644 --- a/cmd/mxcli/init_hook_test.go +++ b/cmd/mxcli/init_hook_test.go @@ -24,10 +24,11 @@ func TestAddSessionStartHook_Empty(t *testing.T) { func TestAddSessionStartHook_Idempotent(t *testing.T) { s := map[string]any{} - addSessionStartHook(s, "x run --local --setup y") - // A second add with the marker present must be a no-op. - if addSessionStartHook(s, "run --local --setup --ensure-db -p App.mpr") { - t.Error("expected no change when the marker is already present") + cmd := sessionStartHookCommand() + addSessionStartHook(s, cmd) + // Re-adding the identical command is a no-op. + if addSessionStartHook(s, cmd) { + t.Error("expected no change when the current command is already present") } ss := s["hooks"].(map[string]any)["SessionStart"].([]any) if len(ss) != 1 { @@ -35,6 +36,27 @@ func TestAddSessionStartHook_Idempotent(t *testing.T) { } } +// A project written by an older mxcli inlined the whole command in the hook. +// Re-running init must REWRITE that entry, not add a second one that runs the +// same bring-up twice. (mxcli-todo findings #2) +func TestAddSessionStartHook_MigratesLegacyCommand(t *testing.T) { + s := map[string]any{} + addSessionStartHook(s, "test -x ./mxcli && ./mxcli run --local --setup --ensure-db -p App.mpr || true") + + want := sessionStartHookCommand() + if !addSessionStartHook(s, want) { + t.Fatal("expected the legacy hook to be migrated") + } + ss := s["hooks"].(map[string]any)["SessionStart"].([]any) + if len(ss) != 1 { + t.Fatalf("SessionStart len = %d, want 1 (migrated in place, not duplicated)", len(ss)) + } + inner := ss[0].(map[string]any)["hooks"].([]any) + if got := inner[0].(map[string]any)["command"].(string); got != want { + t.Errorf("command = %q, want %q", got, want) + } +} + func TestAddSessionStartHook_PreservesExisting(t *testing.T) { // Existing unrelated settings + a different SessionStart hook must survive. s := map[string]any{ @@ -72,9 +94,25 @@ func TestEnsureSessionStartHook_WritesFile(t *testing.T) { if err != nil { t.Fatal(err) } - if !strings.Contains(string(data), "run --local --setup --ensure-db -p App.mpr") { + if !strings.Contains(string(data), sessionStartHookCommand()) { t.Errorf("settings.json missing the hook command:\n%s", data) } + // The hook delegates to a committed script, which must exist, name the + // project's .mpr, and be executable. + scriptPath := filepath.Join(dir, filepath.Base(bootstrapScriptName)) + script, err := os.ReadFile(scriptPath) + if err != nil { + t.Fatalf("bootstrap script not written: %v", err) + } + if !strings.Contains(string(script), "MPR='App.mpr'") { + t.Errorf("bootstrap script does not name the project:\n%s", script) + } + if !strings.Contains(string(script), "releases/download/") { + t.Error("bootstrap script must be able to fetch mxcli after a reap") + } + if info, err := os.Stat(scriptPath); err == nil && info.Mode().Perm()&0o100 == 0 { + t.Errorf("bootstrap script is not executable: %v", info.Mode()) + } // Valid JSON round-trips. var check map[string]any if err := json.Unmarshal(data, &check); err != nil { @@ -95,9 +133,7 @@ func TestEnsureSessionStartHook_InvalidJSONUntouched(t *testing.T) { if err == nil { t.Error("expected an error for invalid existing settings.json") } - if changed { - t.Error("must not report a change when leaving invalid JSON untouched") - } + _ = changed // the script may be (re)written even when settings.json is not // The original content is preserved. data, _ := os.ReadFile(path) if string(data) != "{ not json" { diff --git a/cmd/mxcli/init_lint_config_test.go b/cmd/mxcli/init_lint_config_test.go new file mode 100644 index 000000000..60705bdcd --- /dev/null +++ b/cmd/mxcli/init_lint_config_test.go @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// `mxcli init` seeds a lint config excluding System. On a blank app the System +// module accounted for ~96% of all lint findings — entities you cannot +// document, give access rules to, or rename — which buried the findings about +// the developer's own code (issuetracker finding #9). +func TestWriteDefaultLintConfig(t *testing.T) { + t.Run("creates the config and excludes System", func(t *testing.T) { + dir := t.TempDir() + + path, created := writeDefaultLintConfig(dir) + if !created { + t.Fatal("expected the config to be created in an empty project") + } + if want := filepath.Join(dir, ".claude", "lint-config.yaml"); path != want { + t.Errorf("path = %q, want %q", path, want) + } + + // It must parse as a real config, not just look right. + cfg, err := linter.LoadConfig(path) + if err != nil { + t.Fatalf("the seeded config does not load: %v", err) + } + if len(cfg.ExcludeModules) != 1 || cfg.ExcludeModules[0] != "System" { + t.Errorf("ExcludeModules = %v, want [System]", cfg.ExcludeModules) + } + + // And lint must actually find it where it looks. + if found := linter.FindConfigFile(dir); found != path { + t.Errorf("FindConfigFile = %q, want %q — lint would not pick it up", found, path) + } + }) + + t.Run("never overwrites an existing config", func(t *testing.T) { + // init is re-runnable and the config is meant to be edited, so a second + // run must not discard the developer's changes. + for _, existing := range []string{ + filepath.Join(".claude", "lint-config.yaml"), + "lint-config.yaml", + ".lint-config.yaml", + } { + t.Run(existing, func(t *testing.T) { + dir := t.TempDir() + full := filepath.Join(dir, existing) + if err := os.MkdirAll(filepath.Dir(full), 0755); err != nil { + t.Fatal(err) + } + const mine = "excludeModules: [MyOwnModule]\n" + if err := os.WriteFile(full, []byte(mine), 0644); err != nil { + t.Fatal(err) + } + + if _, created := writeDefaultLintConfig(dir); created { + t.Error("reported creating a config when one already existed") + } + got, err := os.ReadFile(full) + if err != nil { + t.Fatal(err) + } + if string(got) != mine { + t.Errorf("existing config was modified:\n%s", got) + } + // It must not have written a competing config elsewhere either. + other := filepath.Join(dir, ".claude", "lint-config.yaml") + if other != full { + if _, err := os.Stat(other); err == nil { + t.Error("wrote a second, competing config at .claude/lint-config.yaml") + } + } + }) + } + }) +} + +// An exclude beats --modules (LintContext.IsExcluded checks the exclude set +// first), so once init ships a config excluding System, `lint -m System` +// returns nothing. intersect drives the warning that explains why. +func TestIntersect(t *testing.T) { + tests := []struct { + name string + want []string + have []string + out []string + }{ + {name: "shadowed module detected", want: []string{"System"}, have: []string{"System"}, out: []string{"System"}}, + {name: "unshadowed module ignored", want: []string{"MyModule"}, have: []string{"System"}, out: nil}, + { + name: "only the overlap, in the caller's order", + want: []string{"MyModule", "System", "Atlas_Core"}, + have: []string{"Atlas_Core", "System"}, + out: []string{"System", "Atlas_Core"}, + }, + {name: "duplicates collapse", want: []string{"System", "System"}, have: []string{"System"}, out: []string{"System"}}, + {name: "no filter", want: nil, have: []string{"System"}, out: nil}, + {name: "no excludes", want: []string{"System"}, have: nil, out: nil}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := intersect(tc.want, tc.have) + if strings.Join(got, ",") != strings.Join(tc.out, ",") { + t.Errorf("intersect(%v, %v) = %v, want %v", tc.want, tc.have, got, tc.out) + } + }) + } +} diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 27cbf146e..be87beadd 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -370,6 +370,10 @@ func init() { testRunCmd.Flags().BoolP("list", "l", false, "List tests without executing") testRunCmd.Flags().StringP("junit", "j", "", "Write JUnit XML results to file") testRunCmd.Flags().BoolP("skip-build", "s", false, "Skip build step (reuse existing deployment)") + testRunCmd.Flags().Bool("local", false, "Run on mxcli's local runtime instead of Docker (no daemon needed)") + testRunCmd.Flags().Bool("legacy-runner", false, "With --local, run tests from the after-startup microflow and parse the log, instead of over the test endpoint") + testRunCmd.Flags().BoolP("watch", "w", false, "With --local, keep the runtime warm and re-run the suite on every test or model change (Ctrl-C to stop)") + testRunCmd.Flags().Bool("attach", false, "Run against an app already started with 'mxcli run --local --test-endpoint' instead of booting one (tests hit that app's database)") testRunCmd.Flags().BoolP("verbose", "v", false, "Show all runtime log output") testRunCmd.Flags().BoolP("color", "", false, "Use colored output") testRunCmd.Flags().StringP("timeout", "t", "5m", "Timeout for runtime startup and test execution") diff --git a/cmd/mxcli/mdlsource.go b/cmd/mxcli/mdlsource.go new file mode 100644 index 000000000..e2fb5c854 --- /dev/null +++ b/cmd/mxcli/mdlsource.go @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "os" +) + +// stdinPath is the conventional spelling for "read from standard input". +const stdinPath = "-" + +// readMDLSource reads an MDL script from a file, or from standard input when the +// path is "-". +// +// A heredoc is the natural way to drive MDL from an agent or a shell script, and +// `-` is how every other Unix tool spells it — without this the dash was taken +// literally and the command failed with "open -: no such file or directory", +// forcing a temp file. (mxcli-todo findings #5) +func readMDLSource(path string) ([]byte, error) { + if path == stdinPath { + content, err := io.ReadAll(os.Stdin) + if err != nil { + return nil, fmt.Errorf("reading MDL from stdin: %w", err) + } + return content, nil + } + return os.ReadFile(path) +} + +// mdlSourceLabel names the source in messages: a real path, or "". +func mdlSourceLabel(path string) string { + if path == stdinPath { + return "" + } + return path +} diff --git a/cmd/mxcli/mdlsource_test.go b/cmd/mxcli/mdlsource_test.go new file mode 100644 index 000000000..6d723839f --- /dev/null +++ b/cmd/mxcli/mdlsource_test.go @@ -0,0 +1,66 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "path/filepath" + "testing" +) + +// mxcli-todo findings #5: `-` was taken literally as a filename, so a heredoc — +// the natural way to drive MDL from an agent or a shell script — failed with +// "open -: no such file or directory" and forced a temp file. +func TestReadMDLSource_Stdin(t *testing.T) { + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + const script = "SHOW STRUCTURE DEPTH 1;\n" + go func() { + _, _ = w.WriteString(script) + w.Close() + }() + + orig := os.Stdin + os.Stdin = r + defer func() { os.Stdin = orig }() + + got, err := readMDLSource(stdinPath) + if err != nil { + t.Fatalf("readMDLSource(%q): %v", stdinPath, err) + } + if string(got) != script { + t.Errorf("got %q, want %q", got, script) + } +} + +func TestReadMDLSource_File(t *testing.T) { + path := filepath.Join(t.TempDir(), "script.mdl") + const script = "create module M;\n" + if err := os.WriteFile(path, []byte(script), 0o644); err != nil { + t.Fatal(err) + } + got, err := readMDLSource(path) + if err != nil { + t.Fatalf("readMDLSource(%q): %v", path, err) + } + if string(got) != script { + t.Errorf("got %q, want %q", got, script) + } +} + +func TestReadMDLSource_MissingFile(t *testing.T) { + if _, err := readMDLSource(filepath.Join(t.TempDir(), "absent.mdl")); err == nil { + t.Error("expected an error for a missing file") + } +} + +func TestMDLSourceLabel(t *testing.T) { + if got := mdlSourceLabel(stdinPath); got != "" { + t.Errorf("label for stdin = %q, want ", got) + } + if got := mdlSourceLabel("script.mdl"); got != "script.mdl" { + t.Errorf("label for a path = %q, want the path", got) + } +} diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index 764b8d465..506a6e8a2 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -71,23 +71,30 @@ func init() { Keywords: []string{ "create odata service", "publish entity", "publish odata", "expose", "key", "navigation property", "association exposure", - "authentication", "page size", + "authentication", "page size", "servicename", "publishassociations", + "readmode microflow", "non-persistable", "countable", "skipsupported", + "topsupported", }, Syntax: "CREATE [OR MODIFY] ODATA SERVICE Module.Name (\n" + " path: 'odata/customers/', -- no leading slash; trailing slash required\n" + " version: '1.0.0',\n" + " ODataVersion: OData4,\n" + - " namespace: 'Module.Customers'\n" + + " namespace: 'Module.Customers',\n" + + " ServiceName: 'CustomerApi', -- optional; defaults to the document name\n" + + " PublishAssociations: Yes -- optional; default Yes (associations as links)\n" + ")\n" + "authentication basic, session\n" + "{\n" + " publish entity Module.Entity as 'EntitySet' (\n" + - " ReadMode: source,\n" + - " InsertMode: source | not_supported,\n" + - " UpdateMode: source | not_supported,\n" + - " DeleteMode: source | not_supported,\n" + + " ReadMode: source | microflow Module.Read_X,\n" + + " InsertMode: source | not_supported | microflow Module.Insert_X,\n" + + " UpdateMode: source | not_supported | microflow Module.Update_X,\n" + + " DeleteMode: source | not_supported | microflow Module.Delete_X,\n" + " UsePaging: Yes,\n" + - " PageSize: 100\n" + + " PageSize: 100,\n" + + " Countable: No, -- default Yes; No drops the $Response requirement\n" + + " SkipSupported: No, -- default Yes ($skip)\n" + + " TopSupported: No -- default Yes ($top)\n" + " )\n" + " expose (\n" + " KeyAttr as 'ExposedKey' (KEY, Filterable, Sortable),\n" + @@ -96,7 +103,15 @@ func init() { " );\n" + "};\n" + "\n" + - "GRANT ACCESS ON ODATA SERVICE Module.Name TO Module.Role;", + "GRANT ACCESS ON ODATA SERVICE Module.Name TO Module.Role;\n" + + "\n" + + "-- A NON-PERSISTABLE entity can be published: back it with a read\n" + + "-- microflow returning a list of that entity. Nothing is stored, so\n" + + "-- there is no copy of the data in the database.\n" + + "--\n" + + "-- While Countable is Yes (the default) the read microflow must take a\n" + + "-- $Response: System.ODataResponse parameter and set its Count; with\n" + + "-- Countable: No it takes no parameters at all.", Example: "create persistent entity Shop.Customer (\n" + " Email: string(200) unique error 'unique' required error 'required',\n" + " Name: string(200)\n" + @@ -156,7 +171,7 @@ func init() { "body", "response", "mapping", "authentication", "json structure", "import mapping", "export mapping", }, - Syntax: "CREATE [OR MODIFY] REST CLIENT Module.Name (\n BaseUrl: 'https://...',\n Authentication: NONE | BASIC (...)\n)\n{\n OPERATION Name {\n Method: GET|POST|PUT|DELETE|PATCH,\n Path: '/path/{param}',\n Parameters: ($param: Type),\n Query: ($param: Type),\n Headers: ('Key' = 'Value'),\n Timeout: 30,\n Body: JSON FROM $var | MAPPING Entity { jsonField = Attribute, ... },\n Response: JSON AS $var | MAPPING Entity { Attribute = jsonField, ... }\n }\n};\n\n-- MAPPING takes a target ENTITY plus a body listing the JSON fields; Mendix\n-- stores it inline on the operation. An existing import/export mapping\n-- document cannot be referenced here (rejected as MDL-REST01).", + Syntax: "CREATE [OR MODIFY] REST CLIENT Module.Name (\n BaseUrl: 'https://...',\n Authentication: NONE | BASIC (...)\n)\n{\n OPERATION Name {\n Method: GET|POST|PUT|DELETE|PATCH,\n Path: '/path/{param}',\n Parameters: ($param: Type),\n Query: ($param: Type),\n Headers: ('Key' = 'Value'),\n Timeout: 30,\n Body: JSON FROM $var | MAPPING Entity { jsonField = Attribute, ... },\n Response: JSON AS $var | MAPPING Entity { Attribute = jsonField, ... }\n }\n};\n\n-- MAPPING takes a target ENTITY plus a body listing the JSON fields; Mendix\n-- stores it inline on the operation. An existing import/export mapping\n-- document cannot be referenced here (rejected as MDL-REST01).", Example: "CREATE REST CLIENT Module.PetStore (\n BaseUrl: 'https://petstore.example.com/api',\n Authentication: NONE\n)\n{\n OPERATION GetPet {\n Method: GET,\n Path: '/pets/{id}',\n Parameters: ($id: String),\n Query: ($verbose: String),\n Response: MAPPING Module.Pet {\n Name = name,\n Status = status\n }\n }\n};", SeeAlso: []string{"rest", "rest.published"}, }) diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index 77e497a57..e81afb481 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -34,8 +34,12 @@ func init() { "declare", "variable", "set", "assign", "change", "attribute", "expression", }, - Syntax: "DECLARE $Var Type;\nDECLARE $Var Type = expression;\nSET $Var = expression;\nSET $Var/Attribute = expression;", - Example: "DECLARE $Count Integer = 0;\nDECLARE $Name String;\nSET $Name = 'Hello';\nSET $Order/Status = 'Pending';", + Syntax: "DECLARE $Var Type; -- a variable must be declared before it is assigned\n" + + "DECLARE $Var Type = expression;\n" + + "$Var = expression; -- assign; SET is optional\n" + + "$Var/Attribute = expression;\n" + + "SET $Var = expression; -- same statement, explicit form", + Example: "DECLARE $Count Integer = 0;\nDECLARE $Name String;\n$Count = $Count + 1;\n$Name = 'Hello';\nSET $Order/Status = 'Pending';", SeeAlso: []string{"microflow.object-operations"}, }) diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 131a885de..85560be1e 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -125,7 +125,7 @@ DISCONNECT;`, "settings", "project settings", "configuration", "startup", "shutdown", "hash algorithm", "java version", }, - Syntax: "SHOW SETTINGS;\nDESCRIBE SETTINGS;\nALTER SETTINGS MODEL = ;\nALTER SETTINGS CONFIGURATION '' = ;", + Syntax: "SHOW SETTINGS;\nDESCRIBE SETTINGS;\nDESCRIBE SETTINGS CONFIGURATION ''; -- just one configuration\nALTER SETTINGS MODEL = ;\nALTER SETTINGS CONFIGURATION '' = ;", Example: "SHOW SETTINGS;\nALTER SETTINGS MODEL AfterStartupMicroflow = 'Module.MF_Startup';", SeeAlso: []string{"settings.show", "settings.alter"}, }) @@ -136,8 +136,8 @@ DISCONNECT;`, Keywords: []string{ "show settings", "describe settings", "list settings", }, - Syntax: "SHOW SETTINGS;\nDESCRIBE SETTINGS;", - Example: "SHOW SETTINGS;\nDESCRIBE SETTINGS;", + Syntax: "SHOW SETTINGS;\nDESCRIBE SETTINGS;\nDESCRIBE SETTINGS CONFIGURATION '';", + Example: "SHOW SETTINGS;\nDESCRIBE SETTINGS;\nDESCRIBE SETTINGS CONFIGURATION 'Default';", }) Register(SyntaxFeature{ @@ -265,18 +265,25 @@ SEARCH 'word*';`, Register(SyntaxFeature{ Path: "test", - Summary: "Microflow testing — run .test.mdl or .test.md files against a Mendix project in Docker", + Summary: "Microflow testing — run .test.mdl or .test.md files against a Mendix project (local warm loop, or Docker)", Keywords: []string{ "test", "testing", "microflow test", "nanoflow test", "test.mdl", "test.md", "junit", "docker", "@test", "@expect", "@throws", "@cleanup", + "watch", "attach", "test endpoint", "warm", }, Syntax: `mxcli test -p app.mpr [flags] Flags: -l, --list List tests without executing -j, --junit FILE Write JUnit XML results - -s, --skip-build Skip Docker build (reuse existing) + -s, --skip-build Skip the build (reuse existing deployment) + --local Run on mxcli's own runtime — no Docker daemon needed + -w, --watch With --local: keep the runtime warm and re-run on + every test or model change (Ctrl-C to stop) + --attach Run against an app already started with + 'mxcli run --local --test-endpoint' — no boot at all + --legacy-runner With --local: use the old after-startup runner -v, --verbose Show runtime log lines -t, --timeout DUR Runtime startup timeout (default: 5m) @@ -285,7 +292,17 @@ Annotations: @expect $var = value Assert variable equals value @expect $obj/Attr = val Assert entity attribute @throws 'message' Expect error - @cleanup rollback|none Cleanup strategy (default: rollback)`, + @cleanup rollback|none Cleanup strategy (default: rollback) + +How --local runs tests: one microflow per test, invoked by name over a +token-guarded HTTP endpoint the app registers at boot. A test that throws +fails only itself, and results are returned rather than scraped from the log. +Docker still uses the older after-startup runner. + +Cost of a run: + cold (--local) ~30s boots a runtime on its own ports + DB + warm (--local --watch) ~2s runtime stays up between runs + attached (--attach) ~2s no boot; uses the running app's database`, Example: `-- .test.mdl file format /** * @test String concatenation @@ -297,8 +314,14 @@ $result = CALL MICROFLOW MyModule.ConcatNames( / -- Run tests -mxcli test tests/ -p app.mpr -mxcli test tests/ -p app.mpr --junit results.xml`, +mxcli test tests/ -p app.mpr -- Docker +mxcli test tests/ -p app.mpr --local -- no Docker daemon +mxcli test tests/ -p app.mpr --local --watch -- warm loop, re-runs on change +mxcli test tests/ -p app.mpr --junit results.xml + +-- Or attach to an app you already have running: +mxcli run --local --test-endpoint -p app.mpr -- terminal 1 +mxcli test tests/ -p app.mpr --attach -- terminal 2`, }) // ── Errors ────────────────────────────────────────────────────────── diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index bf95bd9b0..197d1f503 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -13,7 +13,7 @@ func init() { "widget", "layout", "screen", }, Syntax: "CREATE PAGE Module.Name\n (\n Title: 'Page Title',\n Layout: Module.LayoutName\n [, Params: { $Param: Module.Entity }]\n [, Url: 'page-url']\n [, Folder: 'FolderPath']\n [, Variables: { $var: Boolean = 'true' }]\n [, PopupWidth: 800, PopupHeight: 480, PopupResizable: true]\n [, Class: 'css-class', Style: 'css: rule']\n )\n {\n -- widgets\n }", - Example: "CREATE PAGE MyModule.EditCustomer\n (\n Params: { $Customer: MyModule.Customer },\n Title: 'Edit Customer',\n Layout: Atlas_Core.PopupLayout,\n Class: 'container-fluid'\n )\n {\n DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Binds: Name)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n }\n }", + Example: "CREATE PAGE MyModule.EditCustomer\n (\n Params: { $Customer: MyModule.Customer },\n Title: 'Edit Customer',\n Layout: Atlas_Core.PopupLayout,\n Class: 'container-fluid'\n )\n {\n DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n }\n }", SeeAlso: []string{"page.create", "page.widgets", "page.alter", "snippet"}, }) @@ -39,8 +39,8 @@ func init() { "dynamictext", "snippetcall", "navigationlist", "column", "row", "footer", "header", "controlbar", }, - Syntax: "-- Containers\nLAYOUTGRID name { ROW r { COLUMN c (DesktopWidth: 6) { ... } } }\nCONTAINER name (Class: 'cls') { ... }\nCONTAINER name (OnClick: MICROFLOW Module.MF) { ... } -- clickable container\n\n-- Data widgets\nDATAVIEW name (DataSource: $Param) { ... FOOTER f { ... } }\nDATAGRID name (DataSource: DATABASE Module.Entity) { COLUMN c (Attribute: A) }\nGALLERY name (DataSource: DATABASE Module.Entity, DesktopColumns: 3) { ... }\nLISTVIEW name (DataSource: DATABASE Module.Entity) { ... }\n\n-- Inputs\nTEXTBOX name (Label: 'L', Binds: Attr)\nTEXTAREA | DATEPICKER | COMBOBOX | CHECKBOX | RADIOBUTTONS\n\n-- Actions\nACTIONBUTTON name (Caption: 'C', Action: SAVE_CHANGES, ButtonStyle: Primary)\n\n-- Display\nDYNAMICTEXT name (Content: 'Hello, {1}!', ContentParams: [{1} = Name])", - Example: "DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Binds: Name)\n COMBOBOX cbStatus (Label: 'Status', Binds: Status)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n}", + Syntax: "-- Containers\nLAYOUTGRID name { ROW r { COLUMN c (DesktopWidth: 6) { ... } } }\nCONTAINER name (Class: 'cls') { ... }\nCONTAINER name (OnClick: MICROFLOW Module.MF) { ... } -- clickable container\n\n-- Data widgets\nDATAVIEW name (DataSource: $Param) { ... FOOTER f { ... } }\nDATAGRID name (DataSource: DATABASE Module.Entity) { COLUMN c (Attribute: A) }\nGALLERY name (DataSource: DATABASE Module.Entity, DesktopColumns: 3) { ... }\nLISTVIEW name (DataSource: DATABASE Module.Entity) { ... }\n\n-- Inputs\nTEXTBOX name (Label: 'L', Attribute: Attr)\nTEXTAREA | DATEPICKER | COMBOBOX | CHECKBOX | RADIOBUTTONS\n\n-- Actions\nACTIONBUTTON name (Caption: 'C', Action: SAVE_CHANGES, ButtonStyle: Primary)\n\n-- Display\nDYNAMICTEXT name (Content: 'Hello, {1}!', ContentParams: [{1} = Name])", + Example: "DATAVIEW dvCustomer (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n COMBOBOX cbStatus (Label: 'Status', Attribute: Status)\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n}", SeeAlso: []string{"page.create", "page.datasource"}, }) @@ -51,7 +51,7 @@ func init() { "datasource", "data source", "database", "microflow", "selection", "variable", "binding", "binds", "association", "data from context", }, - Syntax: "DataSource: $Variable -- Parameter/variable binding\nDataSource: DATABASE Module.Entity -- Database query\nDataSource: MICROFLOW Module.MF() -- Microflow datasource\nDataSource: SELECTION widgetName -- Selection from another widget\nDataSource: $currentObject/Module.Assoc -- Over an association (\"data from context\")\n -- list widget → to-many collection\n -- nested DATAVIEW → the to-one referenced object\nBinds: AttributeName -- Attribute binding (inputs)", + Syntax: "DataSource: $Variable -- Parameter/variable binding\nDataSource: DATABASE Module.Entity -- Database query\nDataSource: MICROFLOW Module.MF -- Microflow datasource (no parens when it takes no arguments)\nDataSource: SELECTION widgetName -- Selection from another widget\nDataSource: $currentObject/Module.Assoc -- Over an association (\"data from context\")\n -- list widget → to-many collection\n -- nested DATAVIEW → the to-one referenced object\nAttribute: AttributeName -- Attribute binding (inputs)", Example: "-- Database datasource with grid\nDATAGRID grid (DataSource: DATABASE Module.Customer) {\n COLUMN colName (Attribute: Name, Caption: 'Name')\n}\n\n-- Microflow datasource\nDATAVIEW dv (DataSource: MICROFLOW Module.GetData()) { ... }\n\n-- Over an association: a nested DataView shows the referenced (to-one) object\nDATAVIEW dvOrder (DataSource: $Order) {\n DATAVIEW dvCustomer (DataSource: $currentObject/Order_Customer) {\n TEXTBOX (Label: 'Name', Attribute: Name)\n }\n}\n\n-- Over an association: a list widget shows the (to-many) collection\nLISTVIEW lvLines (DataSource: $currentObject/Order_OrderLine) { ... }", SeeAlso: []string{"page.widgets", "page.create"}, }) @@ -90,7 +90,7 @@ func init() { "popup width", "popup height", "popup resizable", }, Syntax: "ALTER PAGE Module.Name {\n SET property = value ON widgetName;\n SET (prop1 = val1, prop2 = val2) ON widgetName;\n SET Title = 'New Title'; -- page-level (case-sensitive)\n SET Class = 'css-class'; -- page-level CSS class / style\n SET Style = 'css: rule';\n SET PopupWidth = 800; -- page-level pop-up dimensions\n SET PopupHeight = 480;\n SET PopupResizable = true;\n INSERT AFTER widgetName { };\n INSERT BEFORE widgetName { };\n INSERT INTO containerName { };\n DROP WIDGET name1, name2;\n REPLACE widgetName WITH { };\n};", - Example: "ALTER PAGE Module.EditPage {\n SET (Caption = 'Save & Close', ButtonStyle = Success) ON btnSave;\n INSERT AFTER txtName {\n TEXTBOX txtMiddleName (Label: 'Middle Name', Binds: MiddleName)\n };\n DROP WIDGET txtUnused;\n};", + Example: "ALTER PAGE Module.EditPage {\n SET (Caption = 'Save & Close', ButtonStyle = Success) ON btnSave;\n INSERT AFTER txtName {\n TEXTBOX txtMiddleName (Label: 'Middle Name', Attribute: MiddleName)\n };\n DROP WIDGET txtUnused;\n};", SeeAlso: []string{"page.create", "page.show", "snippet.alter"}, }) @@ -153,7 +153,7 @@ func init() { "alter snippet", "modify snippet", "update snippet", }, Syntax: "ALTER SNIPPET Module.Name {\n SET property = value ON widgetName;\n INSERT AFTER widgetName { };\n INSERT BEFORE widgetName { };\n INSERT INTO containerName { };\n DROP WIDGET name1, name2;\n REPLACE widgetName WITH { };\n};", - Example: "ALTER SNIPPET Module.NavSnippet {\n REPLACE navItem1 WITH {\n ACTIONBUTTON btnHome (Caption: 'Home', Action: SHOW_PAGE Module.HomePage)\n };\n DROP WIDGET txtOldField;\n INSERT AFTER txtName {\n TEXTBOX txtNewField (Label: 'New Field', Binds: NewAttr)\n };\n};", + Example: "ALTER SNIPPET Module.NavSnippet {\n REPLACE navItem1 WITH {\n ACTIONBUTTON btnHome (Caption: 'Home', Action: SHOW_PAGE Module.HomePage)\n };\n DROP WIDGET txtOldField;\n INSERT AFTER txtName {\n TEXTBOX txtNewField (Label: 'New Field', Attribute: NewAttr)\n };\n};", SeeAlso: []string{"snippet", "page.alter"}, }) @@ -192,7 +192,7 @@ func init() { "use fragment", "template", "script scope", }, Syntax: "DEFINE FRAGMENT Name AS { };\nDEFINE FRAGMENT Name AS { SLOT [name] };\nDEFINE FRAGMENT Name ($d: datasource, $a: action) AS { };\nUSE FRAGMENT Name [(args)] [AS prefix_];\nUSE FRAGMENT Name [(args)] [AS prefix_] { };\nSHOW FRAGMENTS;\nDESCRIBE FRAGMENT Name;\nDESCRIBE FRAGMENT FROM PAGE Module.Page WIDGET widgetName;", - Example: "DEFINE FRAGMENT SaveCancelFooter AS {\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n};\n\nCREATE PAGE Module.EditPage (...) {\n DATAVIEW dv (DataSource: $Param) {\n TEXTBOX txtName (Label: 'Name', Binds: Name)\n USE FRAGMENT SaveCancelFooter\n }\n};", + Example: "DEFINE FRAGMENT SaveCancelFooter AS {\n FOOTER footer1 {\n ACTIONBUTTON btnSave (Caption: 'Save', Action: SAVE_CHANGES, ButtonStyle: Primary)\n ACTIONBUTTON btnCancel (Caption: 'Cancel', Action: CANCEL_CHANGES)\n }\n};\n\nCREATE PAGE Module.EditPage (...) {\n DATAVIEW dv (DataSource: $Param) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n USE FRAGMENT SaveCancelFooter\n }\n};", SeeAlso: []string{"fragment.define", "fragment.use", "fragment.slot", "fragment.params", "snippet"}, }) @@ -203,7 +203,7 @@ func init() { "define fragment", "declare fragment", "create fragment", }, Syntax: "DEFINE FRAGMENT Name AS {\n \n};", - Example: "DEFINE FRAGMENT FormFields AS {\n TEXTBOX txtName (Label: 'Name', Binds: Name)\n TEXTBOX txtEmail (Label: 'Email', Binds: Email)\n};", + Example: "DEFINE FRAGMENT FormFields AS {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n TEXTBOX txtEmail (Label: 'Email', Attribute: Email)\n};", SeeAlso: []string{"fragment", "fragment.use"}, }) diff --git a/cmd/mxcli/syntax/retired_spellings_test.go b/cmd/mxcli/syntax/retired_spellings_test.go new file mode 100644 index 000000000..514f2ce4e --- /dev/null +++ b/cmd/mxcli/syntax/retired_spellings_test.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package syntax + +import ( + "strings" + "testing" +) + +// `mxcli syntax` is the reference an agent reads before writing MDL, and nothing +// checks it against the parser — so a spelling the parser has dropped can sit in +// it indefinitely. Two did, and both cost a build round-trip to discover +// (mxcli-todo findings #8). +// +// This pins the corrections. It is a spelling guard, not a parse: the snippets +// are fragments (a DATAVIEW body, a property line) that do not stand alone as +// statements, so they cannot simply be fed to the parser. +func TestSyntaxDocs_NoRetiredSpellings(t *testing.T) { + retired := []struct { + text string + reason string + }{ + { + "Binds:", + "the parser rejects it: \"'Binds:' is no longer supported, use 'Attribute:' instead\"", + }, + { + "MICROFLOW Module.MF()", + "a zero-argument microflow DATASOURCE takes no parentheses (unlike RETRIEVE/CALL, where they are normal)", + }, + } + + for _, f := range All() { + for _, r := range retired { + for field, text := range map[string]string{"Syntax": f.Syntax, "Example": f.Example} { + if strings.Contains(text, r.text) { + t.Errorf("syntax topic %q, %s field, still shows %q — %s", f.Path, field, r.text, r.reason) + } + } + } + } +} diff --git a/cmd/mxcli/testrunner/client.go b/cmd/mxcli/testrunner/client.go new file mode 100644 index 000000000..27253f415 --- /dev/null +++ b/cmd/mxcli/testrunner/client.go @@ -0,0 +1,187 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "net/url" + "strings" + "time" +) + +// endpointClient talks to the test endpoint registered inside the running app. +type endpointClient struct { + baseURL string + token string + http *http.Client +} + +// newEndpointClient returns a client for the app serving on port. +// +// The transport deliberately takes no proxy: the address is always loopback, and +// an HTTP_PROXY in the environment (this is common in container and CI images) +// would otherwise send the token to the proxy. +func newEndpointClient(port int, token string) *endpointClient { + return &endpointClient{ + baseURL: fmt.Sprintf("http://127.0.0.1:%d/%s", port, endpointPath), + token: token, + http: &http.Client{ + Timeout: 10 * time.Minute, + Transport: &http.Transport{ + Proxy: nil, + DialContext: (&net.Dialer{Timeout: 5 * time.Second}).DialContext, + TLSHandshakeTimeout: 5 * time.Second, + }, + }, + } +} + +// runResponse is the endpoint's reply to a run request. +type runResponse struct { + MF string `json:"mf"` + OK bool `json:"ok"` + DurationMicros int64 `json:"durationMicros"` + Result string `json:"result"` + Error string `json:"error"` +} + +// listResponse is the endpoint's reply to a list request. +type listResponse struct { + Microflows []string `json:"microflows"` + Error string `json:"error"` +} + +// get performs one authenticated GET and decodes the JSON body into out. +func (c *endpointClient) get(route string, params url.Values, out any) error { + u := c.baseURL + route + if len(params) > 0 { + u += "?" + params.Encode() + } + req, err := http.NewRequest(http.MethodGet, u, nil) + if err != nil { + return err + } + req.Header.Set(endpointTokenHeader, c.token) + + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + body, err := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + if err != nil { + return fmt.Errorf("reading response: %w", err) + } + + // 401/403 mean the gate rejected us, which is a bug in how mxcli passed the + // token rather than anything to do with the tests. Say so plainly instead of + // letting it surface as an unmarshalling error. + switch resp.StatusCode { + case http.StatusUnauthorized: + return fmt.Errorf("test endpoint rejected the token (is another app serving port %s?)", portOf(c.baseURL)) + case http.StatusForbidden: + return fmt.Errorf("test endpoint refused the request: %s", strings.TrimSpace(string(body))) + } + + if err := json.Unmarshal(body, out); err != nil { + return fmt.Errorf("decoding response (HTTP %d): %w: %s", resp.StatusCode, err, truncate(string(body), 200)) + } + return nil +} + +// ping reports whether the endpoint is up and accepting our token. +func (c *endpointClient) ping() error { + var lr listResponse + return c.get("list", nil, &lr) +} + +// list returns the test microflows the running app knows about. +func (c *endpointClient) list() ([]string, error) { + var lr listResponse + if err := c.get("list", url.Values{"prefix": {testFlowPrefix}}, &lr); err != nil { + return nil, err + } + if lr.Error != "" { + return nil, fmt.Errorf("%s", lr.Error) + } + return lr.Microflows, nil +} + +// run executes one test microflow and returns the endpoint's reply. +func (c *endpointClient) run(mf string) (*runResponse, error) { + var rr runResponse + if err := c.get("run", url.Values{"mf": {mf}}, &rr); err != nil { + return nil, err + } + return &rr, nil +} + +// waitReady polls until the endpoint answers or the deadline passes. The runtime +// reports itself started before the after-startup action has necessarily +// finished registering the handler, so a first call can legitimately 404. +func (c *endpointClient) waitReady(timeout time.Duration) error { + deadline := time.Now().Add(timeout) + var last error + for { + if err := c.ping(); err == nil { + return nil + } else { + last = err + } + if time.Now().After(deadline) { + return fmt.Errorf("test endpoint did not come up within %s: %w", timeout, last) + } + time.Sleep(200 * time.Millisecond) + } +} + +// toResult maps an endpoint reply onto the test's result. +// +// Three outcomes are distinguished, and the distinction matters when reading a +// failing run: the test decided it failed (an assertion), the microflow threw +// (StatusError — the test did not reach a verdict), or the verdict came back in +// a shape this runner does not recognise. +func toResult(tc TestCase, rr *runResponse) TestResult { + res := TestResult{ + ID: tc.ID, + Name: tc.Name, + Duration: time.Duration(rr.DurationMicros) * time.Microsecond, + } + switch { + case !rr.OK: + res.Status = StatusError + res.Message = rr.Error + if res.Message == "" { + res.Message = "microflow threw, but the runtime reported no message" + } + case rr.Result == verdictPass: + res.Status = StatusPass + case strings.HasPrefix(rr.Result, verdictFailPrefix): + res.Status = StatusFail + res.Message = strings.TrimPrefix(rr.Result, verdictFailPrefix) + default: + res.Status = StatusError + res.Message = fmt.Sprintf("unrecognised verdict from the test microflow: %q", truncate(rr.Result, 200)) + } + return res +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "…" +} + +// portOf pulls the port back out of a base URL for error messages. +func portOf(baseURL string) string { + u, err := url.Parse(baseURL) + if err != nil { + return "?" + } + return u.Port() +} diff --git a/cmd/mxcli/testrunner/client_test.go b/cmd/mxcli/testrunner/client_test.go new file mode 100644 index 000000000..a95b660f1 --- /dev/null +++ b/cmd/mxcli/testrunner/client_test.go @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +// fakeEndpoint stands in for the handler the Java action registers, enforcing +// the same token gate. It lets the Go side of the contract — header name, +// routes, response shape, status codes — be tested without a Mendix runtime. +type fakeEndpoint struct { + token string + flows map[string]runResponse + // seenTokens records what each request presented, so a test can assert the + // client actually sends the token rather than the server merely allowing it. + seenTokens []string +} + +func (f *fakeEndpoint) handler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + presented := r.Header.Get(endpointTokenHeader) + f.seenTokens = append(f.seenTokens, presented) + w.Header().Set("Content-Type", "application/json") + + if presented != f.token { + w.WriteHeader(http.StatusUnauthorized) + fmt.Fprint(w, `{"error":"unauthorized"}`) + return + } + + switch { + case strings.HasSuffix(r.URL.Path, "/list"): + names := []string{} + for name := range f.flows { + if p := r.URL.Query().Get("prefix"); p == "" || strings.HasPrefix(name, p) { + names = append(names, name) + } + } + json.NewEncoder(w).Encode(listResponse{Microflows: names}) + case strings.HasSuffix(r.URL.Path, "/run"): + mf := r.URL.Query().Get("mf") + resp, ok := f.flows[mf] + if !ok { + w.WriteHeader(http.StatusNotFound) + fmt.Fprintf(w, `{"error":"unknown microflow","mf":%q}`, mf) + return + } + json.NewEncoder(w).Encode(resp) + default: + w.WriteHeader(http.StatusNotFound) + fmt.Fprint(w, `{"error":"no such route"}`) + } + }) +} + +// newFakeEndpoint starts the fake and returns a client pointed at it. +func newFakeEndpoint(t *testing.T, token string, flows map[string]runResponse) (*fakeEndpoint, *endpointClient) { + t.Helper() + fake := &fakeEndpoint{token: token, flows: flows} + srv := httptest.NewServer(fake.handler()) + t.Cleanup(srv.Close) + + c := newEndpointClient(0, token) + c.baseURL = srv.URL + "/" + endpointPath + return fake, c +} + +func TestClientSendsTheToken(t *testing.T) { + fake, c := newFakeEndpoint(t, "s3cret", map[string]runResponse{ + testFlowPrefix + "test_1": {OK: true, Result: verdictPass}, + }) + + if _, err := c.list(); err != nil { + t.Fatalf("list: %v", err) + } + if len(fake.seenTokens) != 1 || fake.seenTokens[0] != "s3cret" { + t.Errorf("server saw tokens %q, want one request presenting %q", fake.seenTokens, "s3cret") + } +} + +// TestClientReportsAnUnauthorizedGateClearly pins that a rejected token is +// reported as a token problem, not as a JSON decoding failure — the gate +// rejecting mxcli is a bug in how the token was passed, and the message has to +// say so. +func TestClientReportsAnUnauthorizedGateClearly(t *testing.T) { + _, c := newFakeEndpoint(t, "the-real-token", nil) + c.token = "the-wrong-token" + + _, err := c.list() + if err == nil { + t.Fatal("list with a wrong token succeeded") + } + if !strings.Contains(err.Error(), "rejected the token") { + t.Errorf("error %q does not explain that the token was rejected", err) + } +} + +func TestClientListFiltersToTestFlows(t *testing.T) { + _, c := newFakeEndpoint(t, "t", map[string]runResponse{ + testFlowPrefix + "test_1": {}, + "MyModule.SomethingElse": {}, + testFlowPrefix + "test_222": {}, + }) + + names, err := c.list() + if err != nil { + t.Fatalf("list: %v", err) + } + for _, n := range names { + if !strings.HasPrefix(n, testFlowPrefix) { + t.Errorf("list returned a non-test microflow: %q", n) + } + } + if len(names) != 2 { + t.Errorf("got %d test microflows, want 2: %q", len(names), names) + } +} + +func TestToResult(t *testing.T) { + tc := TestCase{ID: "test_1", Name: "a test"} + + tests := []struct { + name string + resp runResponse + wantStatus TestStatus + wantMsg string + }{ + { + name: "pass", + resp: runResponse{OK: true, Result: verdictPass, DurationMicros: 1500}, + wantStatus: StatusPass, + }, + { + name: "assertion failure is a FAIL", + resp: runResponse{OK: true, Result: verdictFailPrefix + "expected $r = 'x'"}, + wantStatus: StatusFail, + wantMsg: "expected $r = 'x'", + }, + { + name: "a thrown microflow is an ERROR, not a FAIL", + resp: runResponse{OK: false, Error: "NullPointerException"}, + wantStatus: StatusError, + wantMsg: "NullPointerException", + }, + { + name: "a throw with no message still says something", + resp: runResponse{OK: false}, + wantStatus: StatusError, + wantMsg: "microflow threw, but the runtime reported no message", + }, + { + name: "an unrecognised verdict is an ERROR", + resp: runResponse{OK: true, Result: "who knows"}, + wantStatus: StatusError, + wantMsg: `unrecognised verdict from the test microflow: "who knows"`, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := toResult(tc, &tt.resp) + if got.Status != tt.wantStatus { + t.Errorf("status = %v, want %v", got.Status, tt.wantStatus) + } + if got.Message != tt.wantMsg { + t.Errorf("message = %q, want %q", got.Message, tt.wantMsg) + } + if got.ID != tc.ID || got.Name != tc.Name { + t.Errorf("identity not carried over: got %q/%q", got.ID, got.Name) + } + }) + } +} + +func TestToResultCarriesDuration(t *testing.T) { + got := toResult(TestCase{ID: "test_1"}, &runResponse{OK: true, Result: verdictPass, DurationMicros: 2500}) + if got.Duration != 2500*time.Microsecond { + t.Errorf("duration = %v, want 2.5ms", got.Duration) + } +} + +// TestClientUsesNoProxy pins that the loopback call cannot be diverted. An +// HTTP_PROXY in the environment is common in container and CI images, and would +// otherwise send the token to the proxy. +func TestClientUsesNoProxy(t *testing.T) { + c := newEndpointClient(8081, "tok") + tr, ok := c.http.Transport.(*http.Transport) + if !ok { + t.Fatalf("transport is %T, want *http.Transport", c.http.Transport) + } + if tr.Proxy != nil { + t.Error("the client honours a proxy; the token could leave the machine") + } +} + +func TestWaitReadyGivesUp(t *testing.T) { + // Port 1 on loopback: nothing listens, and connections fail fast. + c := newEndpointClient(1, "tok") + start := time.Now() + err := c.waitReady(600 * time.Millisecond) + if err == nil { + t.Fatal("waitReady succeeded against a dead port") + } + if elapsed := time.Since(start); elapsed > 10*time.Second { + t.Errorf("waitReady took %v; it should honour its timeout", elapsed) + } + if !strings.Contains(err.Error(), "did not come up") { + t.Errorf("error %q does not explain the endpoint never came up", err) + } +} diff --git a/cmd/mxcli/testrunner/endpoint.go b/cmd/mxcli/testrunner/endpoint.go new file mode 100644 index 000000000..86a2fea14 --- /dev/null +++ b/cmd/mxcli/testrunner/endpoint.go @@ -0,0 +1,265 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "crypto/rand" + "encoding/hex" + "fmt" + "strings" +) + +const ( + // endpointPath is the path the request handler is registered at. Mendix + // matches on the leading segment, so the trailing slash is part of the name. + endpointPath = "mxtest/" + // endpointTokenEnv is the environment variable the runtime JVM reads its + // per-run token from. It is passed via the process environment and never + // written into the project, so a cleanup that fails cannot leave a working + // credential behind in javasource/. + endpointTokenEnv = "MXCLI_TEST_TOKEN" + // endpointTokenHeader carries the token on each request. + endpointTokenHeader = "X-MxTest-Token" + + // endpointRegisterAction is the Java action that registers the handler. + endpointRegisterAction = mxTestModule + ".RegisterTestEndpoint" + // endpointStartupFlow is the after-startup microflow that calls it. It only + // registers the endpoint — unlike the log-scraping runner it replaces, no test + // ever executes during startup. + endpointStartupFlow = mxTestModule + ".RegisterEndpoint" + // testFlowPrefix prefixes every generated per-test microflow. + testFlowPrefix = mxTestModule + ".Test_" +) + +// newEndpointToken returns a fresh 256-bit token as hex. +func newEndpointToken() (string, error) { + var b [32]byte + if _, err := rand.Read(b[:]); err != nil { + return "", fmt.Errorf("generating endpoint token: %w", err) + } + return hex.EncodeToString(b[:]), nil +} + +// testFlowName is the microflow generated for a test case. +func testFlowName(tc TestCase) string { return testFlowPrefix + tc.ID } + +// GenerateEndpointMDL returns the MDL that installs the test endpoint: a Java +// action registering a request handler, plus the after-startup microflow that +// calls it once at boot. +// +// The handler is generic — it resolves microflows by name from +// Core.getMicroflowNames() at request time — so this MDL does not mention any +// test and never has to be regenerated when tests change. That is what lets a +// re-run be an HTTP call instead of a restart. +// +// Three properties make an endpoint that executes arbitrary microflows under a +// system context safe enough to install in a developer's project: +// +// 1. It fails closed. With no token in the environment the handler is not +// registered at all, so a project whose cleanup failed and still carries the +// MxTest module exposes nothing when deployed anywhere else. +// 2. Every request must present the token, compared with a length-independent +// constant-time equality so a wrong guess leaks no timing signal. +// 3. Non-loopback callers are refused outright. mxcli always talks to +// 127.0.0.1; nothing legitimate reaches this handler from off-box. +// +// chainAfterStartup, when non-empty, is a microflow the generated startup flow +// calls after registering the endpoint — the project's own after-startup +// microflow, which this one displaces. +// +// The test runner passes "" : a test run wants a known starting state, and the +// suite is the only thing that should execute. A dev loop hosting the endpoint +// (`run --local --test-endpoint`) passes the real one, because the developer's +// app must still seed its data and do whatever else it does at boot. +func GenerateEndpointMDL(chainAfterStartup string) string { + var b strings.Builder + + b.WriteString("CREATE MODULE " + mxTestModule + ";\n\n") + b.WriteString("/** Registers the mxcli test endpoint. Called once at startup. */\n") + b.WriteString("CREATE OR REPLACE JAVA ACTION " + endpointRegisterAction + "() RETURNS Boolean\n") + b.WriteString("AS $$\n") + b.WriteString(endpointJava) + b.WriteString("\n$$;\n/\n\n") + + b.WriteString("/** Registers the mxcli test endpoint at boot. Runs no tests. */\n") + b.WriteString("CREATE OR REPLACE MICROFLOW " + endpointStartupFlow + " ()\n") + b.WriteString("RETURNS Boolean AS $Registered\n") + b.WriteString("BEGIN\n") + b.WriteString(" $Registered = CALL JAVA ACTION " + endpointRegisterAction + "();\n") + if chainAfterStartup != "" { + // Register first, then hand over: if the project's own startup microflow + // fails, the endpoint is already up and the failure is diagnosable over + // HTTP instead of only in the log. + b.WriteString(" $Chained = CALL MICROFLOW " + chainAfterStartup + "();\n") + } + b.WriteString(" RETURN $Registered;\n") + b.WriteString("END;\n") + b.WriteString("/\n") + + return b.String() +} + +// endpointJava is the body of the registration Java action. It is a constant, +// not a template: nothing about a particular run is interpolated into it, and in +// particular the token is read from the environment rather than baked in. +// +// Fully-qualified type names throughout — the generated .java file's import list +// is fixed by the Java-action scaffold and cannot be extended from MDL. +const endpointJava = `final com.mendix.logging.ILogNode log = com.mendix.core.Core.getLogger("MxTest"); + +// Fail closed: no token in the environment means this is not an mxcli test run, +// so the endpoint is never exposed. A project that kept the MxTest module +// through a failed cleanup is inert everywhere else, including production. +final String expectedToken = System.getenv("` + endpointTokenEnv + `"); +if (expectedToken == null || expectedToken.isEmpty()) { + log.info("MxTest: no ` + endpointTokenEnv + ` in the environment; test endpoint NOT registered"); + return true; +} + +com.mendix.core.Core.addRequestHandler("` + endpointPath + `", new com.mendix.externalinterface.connector.RequestHandler() { + + private String esc(String s) { + if (s == null) return "null"; + StringBuilder b = new StringBuilder("\""); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + switch (c) { + case '"': b.append("\\\""); break; + case '\\': b.append("\\\\"); break; + case '\n': b.append("\\n"); break; + case '\r': b.append("\\r"); break; + case '\t': b.append("\\t"); break; + default: + if (c < 0x20) b.append(String.format("\\u%04x", (int) c)); + else b.append(c); + } + } + return b.append('"').toString(); + } + + // Constant-time comparison. String.equals returns early on the first + // differing byte; MessageDigest.isEqual does not, and is also safe when the + // lengths differ. + private boolean tokenOK(String presented) { + if (presented == null) return false; + return java.security.MessageDigest.isEqual( + presented.getBytes(java.nio.charset.StandardCharsets.UTF_8), + expectedToken.getBytes(java.nio.charset.StandardCharsets.UTF_8)); + } + + private boolean isLoopback(String addr) { + if (addr == null || addr.isEmpty()) return false; + try { + return java.net.InetAddress.getByName(addr).isLoopbackAddress(); + } catch (java.net.UnknownHostException e) { + return false; + } + } + + @Override + protected void processRequest(com.mendix.m2ee.api.IMxRuntimeRequest request, + com.mendix.m2ee.api.IMxRuntimeResponse response, + String path) throws Exception { + response.setContentType("application/json"); + java.io.Writer out = response.getWriter(); + + // mxcli always calls 127.0.0.1. Anything else is not a test run. + if (!isLoopback(request.getRemoteAddr())) { + log.warn("MxTest: refused non-loopback request from " + request.getRemoteAddr()); + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.FORBIDDEN); + out.write("{\"error\":\"forbidden\"}"); + out.flush(); + return; + } + if (!tokenOK(request.getHeader("` + endpointTokenHeader + `"))) { + log.warn("MxTest: refused request with a missing or incorrect token"); + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.UNAUTHORIZED); + out.write("{\"error\":\"unauthorized\"}"); + out.flush(); + return; + } + + java.util.Set known = com.mendix.core.Core.getMicroflowNames(); + + if ("list".equals(path)) { + // Never widen past the test namespace. An unfiltered list would hand + // back the app's entire microflow inventory, which this endpoint has + // no business disclosing — it will not run those microflows either. + // A caller-supplied prefix can only narrow further. + String prefix = request.getParameter("prefix"); + if (prefix == null || !prefix.startsWith("` + testFlowPrefix + `")) { + prefix = "` + testFlowPrefix + `"; + } + java.util.List names = new java.util.ArrayList(); + for (String n : known) { + if (n.startsWith(prefix)) names.add(n); + } + java.util.Collections.sort(names); + StringBuilder b = new StringBuilder("{\"microflows\":["); + for (int i = 0; i < names.size(); i++) { + if (i > 0) b.append(','); + b.append(esc(names.get(i))); + } + b.append("]}"); + out.write(b.toString()); + out.flush(); + return; + } + + if (!"run".equals(path)) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.NOT_FOUND); + out.write("{\"error\":\"no such route\",\"path\":" + esc(path) + "}"); + out.flush(); + return; + } + + String mf = request.getParameter("mf"); + if (mf == null || mf.isEmpty()) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.BAD_REQUEST); + out.write("{\"error\":\"missing mf parameter\"}"); + out.flush(); + return; + } + // Only ever run a microflow this runner generated. Even behind the token + // this handler should not be a way to invoke the rest of the app. + if (!mf.startsWith("` + testFlowPrefix + `")) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.FORBIDDEN); + out.write("{\"error\":\"not a test microflow\",\"mf\":" + esc(mf) + "}"); + out.flush(); + return; + } + if (!known.contains(mf)) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.NOT_FOUND); + out.write("{\"error\":\"unknown microflow\",\"mf\":" + esc(mf) + "}"); + out.flush(); + return; + } + + long t0 = System.nanoTime(); + com.mendix.systemwideinterfaces.core.IContext ctx = com.mendix.core.Core.createSystemContext(); + Object result = null; + String error = null; + try { + result = com.mendix.core.Core.microflowCall(mf).execute(ctx); + } catch (Throwable t) { + Throwable root = t; + while (root.getCause() != null && root.getCause() != root) root = root.getCause(); + String msg = root.getMessage(); + error = (msg == null || msg.isEmpty()) ? root.getClass().getName() : msg; + } + long micros = (System.nanoTime() - t0) / 1000L; + + StringBuilder b = new StringBuilder("{"); + b.append("\"mf\":").append(esc(mf)); + b.append(",\"ok\":").append(error == null); + b.append(",\"durationMicros\":").append(micros); + b.append(",\"result\":").append(result == null ? "null" : esc(String.valueOf(result))); + if (error != null) b.append(",\"error\":").append(esc(error)); + b.append('}'); + out.write(b.toString()); + out.flush(); + } +}); + +log.info("MxTest: test endpoint registered at /` + endpointPath + `"); +return true;` diff --git a/cmd/mxcli/testrunner/endpoint_test.go b/cmd/mxcli/testrunner/endpoint_test.go new file mode 100644 index 000000000..7e8f3ff8e --- /dev/null +++ b/cmd/mxcli/testrunner/endpoint_test.go @@ -0,0 +1,167 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" +) + +func TestNewEndpointTokenIsUniqueAndLongEnough(t *testing.T) { + seen := make(map[string]bool) + for i := 0; i < 100; i++ { + tok, err := newEndpointToken() + if err != nil { + t.Fatalf("newEndpointToken: %v", err) + } + if len(tok) != 64 { + t.Fatalf("token %q is %d hex chars, want 64 (256 bits)", tok, len(tok)) + } + if seen[tok] { + t.Fatalf("newEndpointToken returned a duplicate: %q", tok) + } + seen[tok] = true + } +} + +// TestEndpointJavaFailsClosed pins the property that makes it safe to leave the +// MxTest module in a project: with no token in the environment the handler is +// never registered, so there is nothing to reach. +func TestEndpointJavaFailsClosed(t *testing.T) { + guard := strings.Index(endpointJava, `System.getenv("`+endpointTokenEnv+`")`) + if guard < 0 { + t.Fatal("the handler does not read the token from the environment") + } + register := strings.Index(endpointJava, "Core.addRequestHandler") + if register < 0 { + t.Fatal("the handler is never registered") + } + if guard > register { + t.Error("the token is read after the handler is registered; it must gate registration") + } + + // The early return between them is what makes the guard load-bearing. + between := endpointJava[guard:register] + if !strings.Contains(between, "return true;") { + t.Error("no early return between reading the token and registering: an empty token would still register the handler") + } +} + +// TestEndpointJavaNeverEmbedsASecret pins that the token reaches the runtime +// through the environment only. Interpolating it into the generated Java would +// write a live credential into the user's javasource/ tree, where a failed +// cleanup leaves it behind. +func TestEndpointJavaNeverEmbedsASecret(t *testing.T) { + tok, err := newEndpointToken() + if err != nil { + t.Fatalf("newEndpointToken: %v", err) + } + mdl := GenerateEndpointMDL("") + if strings.Contains(mdl, tok) { + t.Fatal("the generated MDL contains the token") + } + // GenerateEndpointMDL takes no token argument at all, so the only way one + // could appear is via the environment read. + if !strings.Contains(mdl, endpointTokenEnv) { + t.Errorf("the generated MDL does not reference %s", endpointTokenEnv) + } +} + +func TestEndpointJavaChecksTheToken(t *testing.T) { + // Match the return statement, not the word: an earlier version of this test + // looked for "MessageDigest.isEqual" anywhere in the source and was satisfied + // by the comment above the method, so it stayed green when the body was + // swapped for String.equals. + if !strings.Contains(endpointJava, "return java.security.MessageDigest.isEqual(") { + t.Error("token comparison is not constant-time (use MessageDigest.isEqual, not String.equals)") + } + if strings.Contains(endpointJava, "presented.equals(") { + t.Error("the token is compared with String.equals, which returns early on the first differing byte") + } + if !strings.Contains(endpointJava, endpointTokenHeader) { + t.Errorf("the handler does not read the %s header", endpointTokenHeader) + } + if !strings.Contains(endpointJava, "return java.net.InetAddress.getByName(addr).isLoopbackAddress();") { + t.Error("the handler does not refuse non-loopback callers") + } +} + +// TestEndpointJavaOnlyRunsTestMicroflows pins that the endpoint is not a general +// microflow-invocation API even for a caller holding the token. +func TestEndpointJavaOnlyRunsTestMicroflows(t *testing.T) { + if !strings.Contains(endpointJava, `mf.startsWith("`+testFlowPrefix+`")`) { + t.Errorf("the handler does not restrict execution to %s* microflows", testFlowPrefix) + } +} + +// TestEndpointListCannotEnumerateTheApp pins that /list is clamped to the test +// namespace. Found by probing the live runtime: an absent prefix returned every +// microflow in the app, Administration.* included. The endpoint will not run +// those, so it must not disclose them either. +func TestEndpointListCannotEnumerateTheApp(t *testing.T) { + if !strings.Contains(endpointJava, `!prefix.startsWith("`+testFlowPrefix+`")`) { + t.Error("a caller-supplied prefix is not clamped to the test namespace") + } + if strings.Contains(endpointJava, "if (prefix == null || n.startsWith(prefix))") { + t.Error("a null prefix still lists every microflow in the app") + } +} + +// TestEndpointRejectsBeforeItActs pins the ordering of the two gates: both the +// loopback check and the token check must precede any use of the request. +func TestEndpointRejectsBeforeItActs(t *testing.T) { + loopback := strings.Index(endpointJava, "if (!isLoopback(") + token := strings.Index(endpointJava, "if (!tokenOK(") + execute := strings.Index(endpointJava, "Core.microflowCall(mf).execute") + names := strings.Index(endpointJava, "Core.getMicroflowNames()") + + for _, tc := range []struct { + name string + gate, work int + }{ + {"loopback check precedes listing microflow names", loopback, names}, + {"token check precedes listing microflow names", token, names}, + {"loopback check precedes execution", loopback, execute}, + {"token check precedes execution", token, execute}, + } { + if tc.gate < 0 || tc.work < 0 { + t.Fatalf("%s: a landmark is missing (gate=%d work=%d)", tc.name, tc.gate, tc.work) + } + if tc.gate > tc.work { + t.Errorf("%s: gate at %d comes after the work at %d", tc.name, tc.gate, tc.work) + } + } +} + +func TestGenerateEndpointMDLShape(t *testing.T) { + mdl := GenerateEndpointMDL("") + for _, want := range []string{ + "CREATE MODULE " + mxTestModule + ";", + "CREATE OR REPLACE JAVA ACTION " + endpointRegisterAction + "() RETURNS Boolean", + "CREATE OR REPLACE MICROFLOW " + endpointStartupFlow + " ()", + "RETURNS Boolean AS $Registered", + } { + if !strings.Contains(mdl, want) { + t.Errorf("generated MDL is missing %q", want) + } + } + // The startup microflow must return Boolean or Mendix fails the build with + // CE0142 on a void after-startup microflow. + if !strings.Contains(mdl, "RETURN $Registered;") { + t.Error("the startup microflow does not return a Boolean (CE0142)") + } +} + +// TestGenerateEndpointMDLIsTestIndependent pins the property the whole design +// rests on: the endpoint MDL does not mention any test, so it never has to be +// regenerated when tests change. +func TestGenerateEndpointMDLIsTestIndependent(t *testing.T) { + a := GenerateEndpointMDL("") + b := GenerateEndpointMDL("") + if a != b { + t.Fatal("GenerateEndpointMDL is not deterministic") + } + if strings.Contains(a, "test_1") { + t.Error("the endpoint MDL references a specific test") + } +} diff --git a/cmd/mxcli/testrunner/generator_endpoint.go b/cmd/mxcli/testrunner/generator_endpoint.go new file mode 100644 index 000000000..92e649ca3 --- /dev/null +++ b/cmd/mxcli/testrunner/generator_endpoint.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "strings" +) + +// Verdict protocol. A test microflow returns one string: either verdictPass, or +// verdictFailPrefix followed by the reason. The endpoint hands that string back +// as the HTTP response's "result" field, so a result is a returned value rather +// than something recovered from the runtime log. +const ( + verdictPass = "PASS" + verdictFailPrefix = "FAIL:" +) + +// GenerateTestFlows returns the MDL declaring one microflow per test case. +// +// This is the endpoint path's counterpart to GenerateTestRunner, which compiles +// the whole suite into a single after-startup microflow. One microflow per test +// buys three things that the monolith cannot give: +// +// - Each test can be invoked, re-invoked, or skipped on its own, so --filter +// and single-test runs are a matter of which URL is called. +// - A test that throws fails only itself. In the monolith an uncaught error +// ends the whole flow, and because that flow is the after-startup action it +// also fails the boot. +// - Every test gets its own variable scope, so the suffix-renaming the +// monolith needs to keep `$result` in test 1 from colliding with `$result` +// in test 2 is simply not required here. +func GenerateTestFlows(suite *TestSuite) string { + var b strings.Builder + b.WriteString("CREATE MODULE " + mxTestModule + ";\n\n") + for _, tc := range suite.Tests { + writeTestFlow(&b, tc) + b.WriteString("\n") + } + return b.String() +} + +// writeTestFlow writes one test's microflow. +func writeTestFlow(b *strings.Builder, tc TestCase) { + fmt.Fprintf(b, "/** %s */\n", escapeMDLComment(tc.Name)) + fmt.Fprintf(b, "CREATE OR REPLACE MICROFLOW %s ()\n", testFlowName(tc)) + b.WriteString("RETURNS String AS $Verdict\n") + b.WriteString("BEGIN\n") + fmt.Fprintf(b, " DECLARE $Verdict String = '%s';\n", verdictPass) + + if tc.Throws != "" { + writeThrowsFlowBody(b, tc) + } else { + writeExpectFlowBody(b, tc) + } + + b.WriteString(" RETURN $Verdict;\n") + b.WriteString("END;\n") + b.WriteString("/\n") +} + +// writeExpectFlowBody writes the body of a normal test: run the MDL, then check +// each @expect. An error during the body short-circuits to a FAIL verdict. +func writeExpectFlowBody(b *strings.Builder, tc TestCase) { + for _, line := range rewriteBodyForVerdict(strings.Split(tc.MDL, "\n"), tc) { + b.WriteString(" ") + b.WriteString(line) + b.WriteString("\n") + } + for _, exp := range tc.Expects { + writeExpectCheck(b, exp) + } +} + +// writeThrowsFlowBody writes the body of an @throws test: the verdict starts as +// a failure and only the error handler can clear it, so a body that completes +// without throwing fails — which is the point of the annotation. +func writeThrowsFlowBody(b *strings.Builder, tc TestCase) { + fmt.Fprintf(b, " SET $Verdict = '%s';\n", + escapeMDLString(verdictFailPrefix+"expected an exception but none was thrown")) + for _, line := range rewriteBodyForThrows(strings.Split(tc.MDL, "\n")) { + b.WriteString(" ") + b.WriteString(line) + b.WriteString("\n") + } +} + +// writeExpectCheck writes one @expect assertion. +// +// Only the pass condition is expressed with `=`; a `<>` expectation is compiled +// as the same equality with the branches swapped. That is deliberate and +// inherited from the monolithic generator: `<>` in a generated Mendix expression +// produced expression errors, so the operator never reaches the model. +func writeExpectCheck(b *strings.Builder, exp Expect) { + equal := fmt.Sprintf("%s = %s", exp.Variable, exp.Value) + failMsg := escapeMDLString(fmt.Sprintf("%sexpected %s %s %s", + verdictFailPrefix, exp.Variable, exp.Operator, exp.Value)) + + // An earlier statement may already have failed the test; never overwrite an + // existing failure with a later assertion's result. + fmt.Fprintf(b, " IF $Verdict = '%s' THEN\n", verdictPass) + if exp.Operator == "<>" { + fmt.Fprintf(b, " IF %s THEN\n", equal) + fmt.Fprintf(b, " SET $Verdict = '%s';\n", failMsg) + b.WriteString(" END IF;\n") + } else { + fmt.Fprintf(b, " IF %s THEN\n", equal) + b.WriteString(" ELSE\n") + fmt.Fprintf(b, " SET $Verdict = '%s';\n", failMsg) + b.WriteString(" END IF;\n") + } + b.WriteString(" END IF;\n") +} + +// rewriteBodyForVerdict attaches an ON ERROR handler to every CALL in the test +// body, turning a thrown error into a FAIL verdict and an early return. +func rewriteBodyForVerdict(lines []string, tc TestCase) []string { + handler := []string{ + fmt.Sprintf(" SET $Verdict = '%s';", + escapeMDLString(verdictFailPrefix+"exception during execution")), + " RETURN $Verdict;", + } + return attachOnError(lines, handler) +} + +// rewriteBodyForThrows attaches an ON ERROR handler that clears the pre-set +// failure verdict — the error is the expected outcome. +func rewriteBodyForThrows(lines []string) []string { + handler := []string{fmt.Sprintf(" SET $Verdict = '%s';", verdictPass)} + return attachOnError(lines, handler) +} + +// attachOnError appends `ON ERROR { ... }` to each CALL statement in the body, +// joining a statement that spans several lines first. +func attachOnError(lines, handler []string) []string { + var out []string + for i := 0; i < len(lines); i++ { + trimmed := strings.TrimSpace(lines[i]) + if !containsCallMicroflow(trimmed) { + out = append(out, lines[i]) + continue + } + + stmt := lines[i] + for !strings.HasSuffix(strings.TrimSpace(stmt), ";") && i+1 < len(lines) { + i++ + stmt += "\n" + lines[i] + } + stmt = strings.TrimSuffix(strings.TrimSpace(stmt), ";") + + out = append(out, stmt+" ON ERROR {") + out = append(out, handler...) + out = append(out, "};") + } + return out +} + +// escapeMDLComment keeps a test name from closing the javadoc block it sits in. +func escapeMDLComment(s string) string { + return strings.ReplaceAll(s, "*/", "* /") +} diff --git a/cmd/mxcli/testrunner/generator_endpoint_test.go b/cmd/mxcli/testrunner/generator_endpoint_test.go new file mode 100644 index 000000000..dedfd9ba7 --- /dev/null +++ b/cmd/mxcli/testrunner/generator_endpoint_test.go @@ -0,0 +1,225 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "strings" + "testing" +) + +func TestGenerateTestFlowsOneMicroflowPerTest(t *testing.T) { + suite := &TestSuite{ + Name: "suite", + Tests: []TestCase{ + {ID: "test_1", Name: "first", MDL: "$r = CALL MICROFLOW Mod.A();"}, + {ID: "test_2", Name: "second", MDL: "$r = CALL MICROFLOW Mod.B();"}, + }, + } + mdl := GenerateTestFlows(suite) + + for _, want := range []string{ + "CREATE OR REPLACE MICROFLOW MxTest.Test_test_1 ()", + "CREATE OR REPLACE MICROFLOW MxTest.Test_test_2 ()", + } { + if !strings.Contains(mdl, want) { + t.Errorf("generated MDL is missing %q", want) + } + } + if n := strings.Count(mdl, "CREATE OR REPLACE MICROFLOW"); n != 2 { + t.Errorf("got %d microflows, want one per test (2)", n) + } +} + +// TestGenerateTestFlowsNoVariableRenaming pins the simplification that per-test +// microflows buy. The monolithic runner has to suffix every variable to keep +// test 1's $result apart from test 2's; separate microflows have separate +// scopes, so the same name in two tests must survive unmangled. +func TestGenerateTestFlowsNoVariableRenaming(t *testing.T) { + suite := &TestSuite{ + Tests: []TestCase{ + {ID: "test_1", Name: "a", MDL: "$result = CALL MICROFLOW Mod.A();", + Expects: []Expect{{Variable: "$result", Operator: "=", Value: "'x'"}}}, + {ID: "test_2", Name: "b", MDL: "$result = CALL MICROFLOW Mod.B();", + Expects: []Expect{{Variable: "$result", Operator: "=", Value: "'y'"}}}, + }, + } + mdl := GenerateTestFlows(suite) + + if strings.Contains(mdl, "$result_1") || strings.Contains(mdl, "$result_2") { + t.Error("variables were suffix-renamed; per-test microflows have their own scope") + } + if n := strings.Count(mdl, "$result"); n < 4 { + t.Errorf("expected $result to survive in both tests, found %d references", n) + } +} + +func TestGenerateTestFlowsExpectAssertion(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "equality", MDL: "$r = CALL MICROFLOW Mod.A();", + Expects: []Expect{{Variable: "$r", Operator: "=", Value: "'John'"}}, + }}} + mdl := GenerateTestFlows(suite) + + if !strings.Contains(mdl, "IF $r = 'John' THEN") { + t.Errorf("missing the equality check:\n%s", mdl) + } + if !strings.Contains(mdl, verdictFailPrefix+"expected $r = ''John''") { + t.Errorf("missing the failure verdict with the expected value:\n%s", mdl) + } +} + +// TestGenerateTestFlowsNotEqualIsCompiledAsEquality pins the inherited +// constraint: `<>` produced Mendix expression errors, so it must never reach the +// model — a <> expectation is the same equality with the branches swapped. +func TestGenerateTestFlowsNotEqualIsCompiledAsEquality(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "inequality", MDL: "$r = CALL MICROFLOW Mod.A();", + Expects: []Expect{{Variable: "$r", Operator: "<>", Value: "'John'"}}, + }}} + mdl := GenerateTestFlows(suite) + + if strings.Contains(mdl, "$r <> 'John'") { + t.Error("the <> operator reached the generated Mendix expression") + } + if !strings.Contains(mdl, "IF $r = 'John' THEN") { + t.Errorf("<> was not compiled as a swapped equality:\n%s", mdl) + } +} + +func TestGenerateTestFlowsWrapsCallsWithErrorHandling(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "throwing", MDL: "$r = CALL MICROFLOW Mod.A();", + }}} + mdl := GenerateTestFlows(suite) + + if !strings.Contains(mdl, "ON ERROR {") { + t.Errorf("the CALL was not wrapped in ON ERROR:\n%s", mdl) + } + if !strings.Contains(mdl, verdictFailPrefix+"exception during execution") { + t.Errorf("the error handler does not set a FAIL verdict:\n%s", mdl) + } +} + +// TestGenerateTestFlowsThrowsTestStartsFailed pins that an @throws test whose +// body completes normally fails: the verdict is pre-set to a failure and only +// the error handler clears it. +func TestGenerateTestFlowsThrowsTestStartsFailed(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "expects a throw", MDL: "$r = CALL MICROFLOW Mod.A();", + Throws: "boom", + }}} + mdl := GenerateTestFlows(suite) + + failIdx := strings.Index(mdl, verdictFailPrefix+"expected an exception") + handlerIdx := strings.Index(mdl, "ON ERROR {") + if failIdx < 0 { + t.Fatalf("no pre-set failure verdict:\n%s", mdl) + } + if handlerIdx < 0 { + t.Fatalf("no error handler:\n%s", mdl) + } + if failIdx > handlerIdx { + t.Error("the failure verdict is set after the handler; a non-throwing body would pass") + } + if !strings.Contains(mdl[handlerIdx:], "SET $Verdict = '"+verdictPass+"';") { + t.Error("the error handler does not clear the failure verdict") + } +} + +func TestGenerateTestFlowsMultiLineCall(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "multiline", + MDL: "$r = CALL MICROFLOW Mod.A(\n FirstName = 'John',\n LastName = 'Doe'\n);", + }}} + mdl := GenerateTestFlows(suite) + + if !strings.Contains(mdl, ") ON ERROR {") { + t.Errorf("a statement spanning lines was not joined before ON ERROR was attached:\n%s", mdl) + } + if strings.Count(mdl, "ON ERROR {") != 1 { + t.Errorf("expected exactly one handler for one call:\n%s", mdl) + } +} + +// TestGenerateTestFlowsEscapesNameInComment pins that a test name cannot close +// the javadoc block it is written into and break the generated MDL. +func TestGenerateTestFlowsEscapesNameInComment(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ + ID: "test_1", Name: "ends the comment */ CREATE MODULE Evil;", MDL: "", + }}} + mdl := GenerateTestFlows(suite) + + head := mdl[:strings.Index(mdl, "CREATE OR REPLACE MICROFLOW")] + if strings.Count(head, "*/") != 1 { + t.Errorf("the test name closed the javadoc block early:\n%s", head) + } +} + +func TestEndpointCleanupCommands(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + + tests := []struct { + name string + state projectState + present bool + want []string + }{ + { + name: "drops the whole module when the runner created it", + state: projectState{afterStartup: "Mod.ASU", createdMxTest: true}, + present: true, + want: []string{ + "ALTER SETTINGS MODEL AfterStartupMicroflow = 'Mod.ASU'", + "DROP MODULE MxTest", + }, + }, + { + name: "drops only the generated documents from a user's module", + state: projectState{createdMxTest: false}, + present: true, + want: []string{ + "ALTER SETTINGS MODEL AfterStartupMicroflow = ''", + "DROP MICROFLOW MxTest.Test_test_1", + "DROP MICROFLOW MxTest.Test_test_2", + "DROP MICROFLOW " + endpointStartupFlow, + "DROP JAVA ACTION " + endpointRegisterAction, + }, + }, + { + name: "drops nothing when the module never landed", + state: projectState{afterStartup: "Mod.ASU", createdMxTest: true}, + present: false, + want: []string{"ALTER SETTINGS MODEL AfterStartupMicroflow = 'Mod.ASU'"}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := endpointCleanupCommands(tc.state, suite, tc.present) + if len(got) != len(tc.want) { + t.Fatalf("got %d commands %q, want %d %q", len(got), got, len(tc.want), tc.want) + } + for i := range got { + if got[i] != tc.want[i] { + t.Errorf("command %d: got %q, want %q", i, got[i], tc.want[i]) + } + } + }) + } +} + +// TestEndpointCleanupRestoreIsAlwaysFirst pins the ordering: after-startup must +// stop pointing at the startup microflow before that microflow is dropped. +func TestEndpointCleanupRestoreIsAlwaysFirst(t *testing.T) { + suite := &TestSuite{Tests: []TestCase{{ID: "test_1"}}} + for _, st := range []projectState{ + {createdMxTest: true}, + {createdMxTest: false}, + {afterStartup: "Mod.ASU", createdMxTest: true}, + } { + cmds := endpointCleanupCommands(st, suite, true) + if !strings.HasPrefix(cmds[0], "ALTER SETTINGS MODEL AfterStartupMicroflow") { + t.Errorf("state %+v: first command is %q, want the after-startup restore", st, cmds[0]) + } + } +} diff --git a/cmd/mxcli/testrunner/handshake.go b/cmd/mxcli/testrunner/handshake.go new file mode 100644 index 000000000..e46f68683 --- /dev/null +++ b/cmd/mxcli/testrunner/handshake.go @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "syscall" + "time" +) + +// syscallSignalZero is signal 0: delivered to no one, but still performs the +// process-exists and permission checks. The idiom for "is this pid alive?". +const syscallSignalZero = syscall.Signal(0) + +// HandshakeFile is where a `run --local --test-endpoint` session publishes what +// `mxcli test --attach` needs to reach it. It lives beside the project rather +// than in a shared temp directory so two projects cannot collide. +const handshakeName = "test-endpoint.json" + +// Handshake is the contract between a dev loop hosting the test endpoint and a +// test run attaching to it. +// +// It carries a live credential, so it is written 0600 and removed when the dev +// loop exits. It is not a secret store: the token it holds only works against a +// loopback endpoint on this machine, and only until that runtime stops. +type Handshake struct { + // Project is the .mpr the dev loop is serving, so an attach can refuse a + // handshake left behind by a different project. + Project string `json:"project"` + // PID of the hosting `mxcli run --local` process, used to detect a stale file. + PID int `json:"pid"` + // AppPort is where the test endpoint is reachable. + AppPort int `json:"appPort"` + // AdminPort is the M2EE admin API, used to reload the model after injecting + // test microflows. + AdminPort int `json:"adminPort"` + // AdminPass authenticates against that admin API. It is NOT the endpoint + // token: the two are different secrets, and using one for the other fails + // with "Authentication failed" at the first reload. + AdminPass string `json:"adminPass"` + // ServePort is the mxbuild serve API, used to rebuild after injecting. + ServePort int `json:"servePort"` + // Token authenticates against the endpoint. + Token string `json:"token"` + // Started is when the dev loop published this, for a clearer stale message. + Started time.Time `json:"started"` +} + +// HandshakePath is the handshake file's location for a project. +func HandshakePath(projectPath string) string { + return filepath.Join(filepath.Dir(projectPath), ".mxcli", handshakeName) +} + +// WriteHandshake publishes the handshake, replacing any existing one. +// +// Written via a temp file and renamed so an attach can never read a +// half-written file, and created 0600 because it carries the endpoint token. +func WriteHandshake(projectPath string, h Handshake) error { + path := HandshakePath(projectPath) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return fmt.Errorf("creating %s: %w", filepath.Dir(path), err) + } + body, err := json.MarshalIndent(h, "", " ") + if err != nil { + return err + } + tmp := path + ".tmp" + if err := os.WriteFile(tmp, body, 0o600); err != nil { + return fmt.Errorf("writing %s: %w", tmp, err) + } + if err := os.Rename(tmp, path); err != nil { + os.Remove(tmp) + return fmt.Errorf("publishing %s: %w", path, err) + } + return nil +} + +// RemoveHandshake deletes the handshake. Safe when it is not there. +func RemoveHandshake(projectPath string) { + os.Remove(HandshakePath(projectPath)) +} + +// ReadHandshake loads the handshake for a project and rejects a stale one. +// +// Staleness matters more than it looks: a dev loop killed with SIGKILL leaves +// the file behind, and attaching to a dead runtime would fail with a confusing +// connection error several steps later. Checking the recorded PID turns that +// into one clear message at the start. +func ReadHandshake(projectPath string) (*Handshake, error) { + path := HandshakePath(projectPath) + body, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil, fmt.Errorf("no test endpoint is being hosted for this project\n"+ + " --attach needs an app already running with the endpoint. Start one with:\n"+ + " mxcli run --local --test-endpoint -p %s\n"+ + " (expected the handshake at %s)", projectPath, path) + } + return nil, fmt.Errorf("reading %s: %w", path, err) + } + + var h Handshake + if err := json.Unmarshal(body, &h); err != nil { + return nil, fmt.Errorf("%s is not readable as a handshake: %w", path, err) + } + if h.Token == "" || h.AppPort == 0 { + return nil, fmt.Errorf("%s is incomplete; stop and restart the hosting 'mxcli run --local --test-endpoint'", path) + } + if !processAlive(h.PID) { + return nil, fmt.Errorf("the app that published %s (pid %d, started %s) is no longer running\n"+ + " Start one with: mxcli run --local --test-endpoint -p %s", + path, h.PID, h.Started.Format(time.RFC3339), projectPath) + } + return &h, nil +} + +// processAlive reports whether a pid names a live process. Signal 0 performs the +// existence and permission checks without delivering anything. +func processAlive(pid int) bool { + if pid <= 0 { + return false + } + proc, err := os.FindProcess(pid) + if err != nil { + return false + } + return proc.Signal(syscallSignalZero) == nil +} + +// nowFunc is time.Now, indirected so a test can pin the timestamp. +var nowFunc = time.Now diff --git a/cmd/mxcli/testrunner/handshake_test.go b/cmd/mxcli/testrunner/handshake_test.go new file mode 100644 index 000000000..26d5ad503 --- /dev/null +++ b/cmd/mxcli/testrunner/handshake_test.go @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func tempProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + mpr := filepath.Join(dir, "App.mpr") + if err := os.WriteFile(mpr, []byte("x"), 0o600); err != nil { + t.Fatalf("writing project fixture: %v", err) + } + return mpr +} + +func TestHandshakeRoundTrip(t *testing.T) { + mpr := tempProject(t) + want := Handshake{ + Project: mpr, PID: os.Getpid(), + AppPort: 8080, AdminPort: 8090, ServePort: 6543, + Token: "tok", Started: time.Now(), + } + if err := WriteHandshake(mpr, want); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + + got, err := ReadHandshake(mpr) + if err != nil { + t.Fatalf("ReadHandshake: %v", err) + } + if got.Token != want.Token || got.AppPort != want.AppPort || + got.AdminPort != want.AdminPort || got.ServePort != want.ServePort { + t.Errorf("round trip lost data: got %+v", got) + } +} + +// TestHandshakeIsNotWorldReadable pins the file mode: the handshake carries a +// live token for an endpoint that executes microflows. +func TestHandshakeIsNotWorldReadable(t *testing.T) { + mpr := tempProject(t) + if err := WriteHandshake(mpr, Handshake{PID: os.Getpid(), AppPort: 8080, Token: "tok"}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + info, err := os.Stat(HandshakePath(mpr)) + if err != nil { + t.Fatalf("stat: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o600 { + t.Errorf("handshake mode is %o, want 600 — it holds a live token", perm) + } +} + +// TestHandshakeLeavesNoTempFile pins that the write-then-rename never leaves the +// intermediate behind, which would also be a token on disk nobody cleans up. +func TestHandshakeLeavesNoTempFile(t *testing.T) { + mpr := tempProject(t) + if err := WriteHandshake(mpr, Handshake{PID: os.Getpid(), AppPort: 8080, Token: "tok"}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + if _, err := os.Stat(HandshakePath(mpr) + ".tmp"); !os.IsNotExist(err) { + t.Error("the temp file used for the atomic write was left behind") + } +} + +func TestReadHandshakeMissingExplainsHowToStartOne(t *testing.T) { + mpr := tempProject(t) + _, err := ReadHandshake(mpr) + if err == nil { + t.Fatal("reading a missing handshake succeeded") + } + if !strings.Contains(err.Error(), "--test-endpoint") { + t.Errorf("error %q does not say how to start a hosting app", err) + } +} + +// TestReadHandshakeRejectsADeadHost pins the staleness check. A dev loop killed +// with SIGKILL leaves the file behind; without this the attach would fail much +// later with a confusing connection error. +func TestReadHandshakeRejectsADeadHost(t *testing.T) { + mpr := tempProject(t) + // PID 0x7FFFFFFF is above any real pid_max, so it cannot be live. + if err := WriteHandshake(mpr, Handshake{PID: 0x7FFFFFFF, AppPort: 8080, Token: "tok", Started: time.Now()}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + _, err := ReadHandshake(mpr) + if err == nil { + t.Fatal("a handshake naming a dead process was accepted") + } + if !strings.Contains(err.Error(), "no longer running") { + t.Errorf("error %q does not identify the host as dead", err) + } +} + +func TestReadHandshakeRejectsIncomplete(t *testing.T) { + mpr := tempProject(t) + body, _ := json.Marshal(Handshake{PID: os.Getpid(), AppPort: 8080}) // no token + path := HandshakePath(mpr) + os.MkdirAll(filepath.Dir(path), 0o755) + if err := os.WriteFile(path, body, 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if _, err := ReadHandshake(mpr); err == nil { + t.Fatal("a handshake with no token was accepted") + } +} + +func TestRemoveHandshake(t *testing.T) { + mpr := tempProject(t) + if err := WriteHandshake(mpr, Handshake{PID: os.Getpid(), AppPort: 8080, Token: "tok"}); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + RemoveHandshake(mpr) + if _, err := os.Stat(HandshakePath(mpr)); !os.IsNotExist(err) { + t.Error("the handshake survived RemoveHandshake") + } + RemoveHandshake(mpr) // must not panic when already gone +} + +// TestGenerateEndpointMDLChainsAfterStartup pins that hosting the endpoint in a +// dev app does not silently drop the app's own startup logic — seed data, for +// instance — which displacing after-startup would. +func TestGenerateEndpointMDLChainsAfterStartup(t *testing.T) { + mdl := GenerateEndpointMDL("MyModule.ASU_Startup") + if !strings.Contains(mdl, "CALL MICROFLOW MyModule.ASU_Startup()") { + t.Errorf("the project's own after-startup is not chained:\n%s", mdl) + } + + register := strings.Index(mdl, "CALL JAVA ACTION "+endpointRegisterAction) + chained := strings.Index(mdl, "CALL MICROFLOW MyModule.ASU_Startup()") + if register > chained { + t.Error("the endpoint is registered after the chained microflow; a failure in that microflow would then leave no endpoint to diagnose it with") + } +} + +func TestGenerateEndpointMDLNoChainWhenNone(t *testing.T) { + mdl := GenerateEndpointMDL("") + if strings.Contains(mdl, "$Chained") { + t.Errorf("a chained call was emitted with no microflow to chain:\n%s", mdl) + } +} + +func TestDropTestFlows(t *testing.T) { + got := dropTestFlows(&TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}}) + want := []string{"DROP MICROFLOW MxTest.Test_test_1", "DROP MICROFLOW MxTest.Test_test_2"} + if len(got) != len(want) { + t.Fatalf("got %q, want %q", got, want) + } + for i := range got { + if got[i] != want[i] { + t.Errorf("command %d: got %q, want %q", i, got[i], want[i]) + } + } +} + +// TestDropTestFlowsNeverTouchesTheEndpoint pins the ownership boundary: an +// attach adds only test microflows, so it must remove only those. The endpoint +// and the after-startup setting belong to the dev loop hosting them. +func TestDropTestFlowsNeverTouchesTheEndpoint(t *testing.T) { + for _, cmd := range dropTestFlows(&TestSuite{Tests: []TestCase{{ID: "test_1"}}}) { + for _, forbidden := range []string{"DROP MODULE", endpointStartupFlow, endpointRegisterAction, "AfterStartupMicroflow"} { + if strings.Contains(cmd, forbidden) { + t.Errorf("attach cleanup would remove %q, which the hosting dev loop owns: %q", forbidden, cmd) + } + } + } +} + +func TestValidateOptionsAttach(t *testing.T) { + tests := []struct { + name string + opts RunOptions + wantErr string + }{ + {name: "attach alone is fine", opts: RunOptions{Attach: true}}, + {name: "attach with watch is fine", opts: RunOptions{Attach: true, Watch: true}}, + { + name: "attach with the legacy runner", + opts: RunOptions{Attach: true, LegacyRunner: true}, + wantErr: "--attach cannot be combined with --legacy-runner", + }, + { + name: "attach with skip-build", + opts: RunOptions{Attach: true, SkipBuild: true}, + wantErr: "--attach cannot be combined with --skip-build", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateOptions(tt.opts) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + return + } + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("error = %v, want it to contain %q", err, tt.wantErr) + } + }) + } +} + +// TestAttachDoesNotRequireLocal pins that --attach implies a local app: needing +// --local as well would be noise, since there is nothing else to attach to. +func TestAttachDoesNotRequireLocal(t *testing.T) { + if err := validateOptions(RunOptions{Attach: true, Local: false}); err != nil { + t.Errorf("--attach without --local was rejected: %v", err) + } +} + +// TestAttachUsesTheAdminPasswordNotTheEndpointToken pins a bug found by running +// --attach against a live app: the M2EE admin API and the test endpoint use +// different secrets, and passing the endpoint token to the admin API fails with +// "Authentication failed" at the first reload — after the test microflows have +// already been injected. +func TestAttachUsesTheAdminPasswordNotTheEndpointToken(t *testing.T) { + mpr := tempProject(t) + hs := Handshake{ + Project: mpr, PID: os.Getpid(), + AppPort: 8080, AdminPort: 8090, ServePort: 6543, + AdminPass: "the-admin-password", + Token: "the-endpoint-token", + } + if err := WriteHandshake(mpr, hs); err != nil { + t.Fatalf("WriteHandshake: %v", err) + } + got, err := ReadHandshake(mpr) + if err != nil { + t.Fatalf("ReadHandshake: %v", err) + } + if got.AdminPass != "the-admin-password" { + t.Fatalf("the handshake does not carry the admin password (got %q)", got.AdminPass) + } + if got.AdminPass == got.Token { + t.Error("the admin password and the endpoint token are the same value; they are different secrets") + } +} diff --git a/cmd/mxcli/testrunner/host.go b/cmd/mxcli/testrunner/host.go new file mode 100644 index 000000000..b70e0812a --- /dev/null +++ b/cmd/mxcli/testrunner/host.go @@ -0,0 +1,119 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "os" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// HostedEndpoint is a test endpoint installed into a project for the lifetime of +// a dev loop, so `mxcli test --attach` can run tests against it without booting +// its own runtime. +// +// It exists because the endpoint cannot be added to a running app: the handler +// is registered by the after-startup microflow, which only runs at boot, and its +// token comes from the runtime's environment. Whoever boots the app therefore +// has to opt in — which is also the right place for the decision, since hosting +// it means the developer's own app carries a microflow-executing endpoint and +// runs tests against the database they are looking at. +type HostedEndpoint struct { + // Token the runtime must be given as MXCLI_TEST_TOKEN. + Token string + // Env is the entry to add to the runtime process environment. + Env []string + + projectPath string + state projectState + out io.Writer + removed bool +} + +// InstallHostedEndpoint injects the test endpoint into the project and returns +// what the caller needs to boot with it. The caller must Remove it on shutdown. +// +// The project's own after-startup microflow is chained rather than displaced, so +// the app still boots the way the developer expects. +func InstallHostedEndpoint(projectPath string, w io.Writer) (*HostedEndpoint, error) { + token, err := newEndpointToken() + if err != nil { + return nil, err + } + + state, err := captureProjectState(projectPath) + if err != nil { + return nil, fmt.Errorf("capturing project state: %w", err) + } + + h := &HostedEndpoint{ + Token: token, + Env: []string{endpointTokenEnv + "=" + token}, + projectPath: projectPath, + state: state, + out: w, + } + + fmt.Fprintln(w, "Installing the mxcli test endpoint (for 'mxcli test --attach')...") + if err := execMDLScript(projectPath, GenerateEndpointMDL(state.afterStartup), "mxtest-endpoint-*.mdl"); err != nil { + h.Remove() + return nil, fmt.Errorf("injecting the test endpoint: %w", err) + } + if err := execMxcliCmd(projectPath, "ALTER SETTINGS MODEL AfterStartupMicroflow = "+quoteMDLString(endpointStartupFlow)); err != nil { + h.Remove() + return nil, fmt.Errorf("pointing after-startup at the endpoint: %w", err) + } + if state.afterStartup != "" { + fmt.Fprintf(w, " after-startup chained: %s runs, then %s\n", endpointStartupFlow, state.afterStartup) + } + return h, nil +} + +// Publish writes the handshake that `mxcli test --attach` reads. Called once the +// app is actually serving, so an attach never finds a handshake for a runtime +// that has not come up. +func (h *HostedEndpoint) Publish(info docker.LocalAppInfo) error { + if h == nil { + return nil + } + err := WriteHandshake(h.projectPath, Handshake{ + Project: h.projectPath, + PID: os.Getpid(), + AppPort: info.AppPort, + AdminPort: info.AdminPort, + ServePort: info.ServePort, + AdminPass: info.AdminPass, + Token: h.Token, + Started: nowFunc(), + }) + if err != nil { + return err + } + fmt.Fprintf(h.out, " test endpoint ready — run 'mxcli test -p %s --attach' from another terminal\n", h.projectPath) + return nil +} + +// Remove withdraws the handshake and restores the project. +// +// Best-effort by design: this runs on the dev loop's shutdown path, where the +// user is trying to stop the app, and a hard failure there helps nobody. What it +// must not do is stay silent — a project left carrying the endpoint is something +// the developer has to know about before they commit. +func (h *HostedEndpoint) Remove() { + // Idempotent and nil-safe: the caller both defers this and calls it on the + // os.Exit path (which skips defers), so it must tolerate running twice. + if h == nil || h.removed { + return + } + h.removed = true + RemoveHandshake(h.projectPath) + if err := cleanupEndpoint(h.projectPath, h.state, &TestSuite{}, h.out); err != nil { + fmt.Fprintf(h.out, "\nERROR: could not remove the test endpoint — the project has been left modified:\n%v\n", err) + fmt.Fprintf(h.out, "Check the after-startup microflow and the %s module before committing.\n", mxTestModule) + return + } + removeGeneratedJavaSource(h.projectPath, h.out) + fmt.Fprintln(h.out, " test endpoint removed; project restored") +} diff --git a/cmd/mxcli/testrunner/runner.go b/cmd/mxcli/testrunner/runner.go index f0d35c25e..ffc8bf459 100644 --- a/cmd/mxcli/testrunner/runner.go +++ b/cmd/mxcli/testrunner/runner.go @@ -37,6 +37,27 @@ type RunOptions struct { // SkipBuild skips the MxBuild step (reuse existing deployment). SkipBuild bool + // Local runs the app with mxcli's own local runtime (`run --local`) instead + // of a Docker container, and drives the tests over the test endpoint. + Local bool + + // LegacyRunner forces the after-startup mechanism on a local run — the suite + // compiled into one startup microflow, results read back from the runtime + // log. An escape hatch for the case where the endpoint misbehaves; the + // Docker path uses this mechanism regardless. + LegacyRunner bool + + // Watch keeps the runtime and the build server up and re-runs the suite on + // every change to a test file or to the project's model, until interrupted. + // Requires the test endpoint, so it is incompatible with LegacyRunner and + // with the Docker path — both of which can only re-run by restarting. + Watch bool + + // Attach runs against an app already started with + // `mxcli run --local --test-endpoint`, skipping the boot entirely. The tests + // then run against that app's database rather than a scratch one. + Attach bool + // Timeout for runtime startup and test execution. Timeout time.Duration @@ -56,14 +77,22 @@ type RunOptions struct { Stderr io.Writer } -// Run executes the test suite using the after-startup pattern: -// 1. Parse test files -// 2. Generate TestRunner microflow -// 3. Inject into project, set as after-startup -// 4. Build and restart runtime -// 5. Parse logs for results -// 6. Cleanup (restore original settings) -// 7. Output results +// Run executes the test suite. +// +// There are two mechanisms, and which one is used follows from opts.Local: +// +// - Local runs go through the test endpoint (runEndpoint). Boot registers an +// HTTP handler and nothing else; each test is then invoked by name against a +// runtime that stays up, and returns its verdict in the response. +// - Docker runs go through the after-startup runner (runAfterStartup), which +// compiles the suite into the project's after-startup microflow, restarts the +// container, and recovers results from its log. +// +// The endpoint is the better mechanism — a re-run is an HTTP call rather than a +// restart, a failing test is a result rather than a failed boot, and results are +// returned rather than scraped. It is confined to the local path because it +// needs to hand the runtime a secret through its environment and to reach it on +// loopback, neither of which is wired through docker-compose yet. func Run(opts RunOptions) (*SuiteResult, error) { w := opts.Stdout if w == nil { @@ -74,12 +103,24 @@ func Run(opts RunOptions) (*SuiteResult, error) { stderr = os.Stderr } + if err := validateOptions(opts); err != nil { + return nil, err + } + timeout := opts.Timeout if timeout == 0 { timeout = 5 * time.Minute } - // Step 1: Parse test files + // Resolve the project path up front so everything derived from it — the + // runtime log path, the deployment directory, the paths named in error + // messages — is absolute and agrees. + if opts.ProjectPath != "" && !filepath.IsAbs(opts.ProjectPath) { + if abs, err := filepath.Abs(opts.ProjectPath); err == nil { + opts.ProjectPath = abs + } + } + fmt.Fprintln(w, "Parsing test files...") suite, err := parseTestFiles(opts.TestFiles) if err != nil { @@ -91,108 +132,188 @@ func Run(opts RunOptions) (*SuiteResult, error) { return nil, fmt.Errorf("no tests found in the provided files") } - // Step 2: Generate TestRunner microflow MDL - fmt.Fprintln(w, "Generating test runner microflow...") - runnerMDL := GenerateTestRunner(suite) + if opts.Attach { + return runAttached(opts, suite, timeout, w) + } + if opts.Local && !opts.LegacyRunner { + return runEndpoint(opts, suite, timeout, w) + } + return runAfterStartup(opts, suite, timeout, w) +} + +// validateOptions rejects combinations that cannot work, with a message that +// says what to do instead. Watching depends on re-invoking tests without a +// restart, which only the test endpoint can do. +func validateOptions(opts RunOptions) error { + if opts.Attach { + if opts.LegacyRunner { + return fmt.Errorf("--attach cannot be combined with --legacy-runner: the after-startup runner can only run tests by restarting, which is what attaching avoids") + } + if opts.SkipBuild { + return fmt.Errorf("--attach cannot be combined with --skip-build: the attached app must be rebuilt to pick up the test microflows") + } + // --attach implies a local app; requiring --local as well would be noise. + return nil + } + if !opts.Watch { + return nil + } + if !opts.Local { + return fmt.Errorf("--watch requires --local: the Docker path can only re-run tests by restarting the container") + } + if opts.LegacyRunner { + return fmt.Errorf("--watch cannot be combined with --legacy-runner: the after-startup runner can only re-run tests by restarting the runtime") + } + if opts.SkipBuild { + return fmt.Errorf("--watch cannot be combined with --skip-build: watching exists to rebuild on every change") + } + return nil +} + +// runEndpoint injects the test endpoint plus one microflow per test, boots the +// app once, and drives the suite over HTTP. +func runEndpoint(opts RunOptions, suite *TestSuite, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + token, err := newEndpointToken() + if err != nil { + return nil, err + } + + fmt.Fprintln(w, "Generating test endpoint and test microflows...") + // "" : a test run wants a known starting state, so the project's own + // after-startup is not chained here (a hosted endpoint does chain it). + endpointMDL := GenerateEndpointMDL("") + flowsMDL := GenerateTestFlows(suite) if opts.Verbose { fmt.Fprintln(w, "--- Generated MDL ---") - fmt.Fprintln(w, runnerMDL) + fmt.Fprintln(w, endpointMDL) + fmt.Fprintln(w, flowsMDL) fmt.Fprintln(w, "--- End MDL ---") } - // Step 3: Save original settings and inject test runner - fmt.Fprintln(w, "Injecting test runner into project...") // Capture what cleanup will need to restore, before touching anything. This // must succeed: without it cleanup cannot tell an existing MxTest module from // the one it is about to create, nor restore the original after-startup. + fmt.Fprintln(w, "Injecting test endpoint into project...") state, err := captureProjectState(opts.ProjectPath) if err != nil { return nil, fmt.Errorf("capturing project state: %w", err) } - // Write the runner MDL to a temp file and execute it - tmpFile, err := os.CreateTemp("", "mxtest-runner-*.mdl") + // From here on the project is modified, so every exit runs cleanup. + // + // cleanupSuite, not the suite captured here: --watch re-parses on every + // change, so by the time cleanup runs the set of injected test microflows may + // differ from the one injected at boot. Dropping the wrong list would leave + // generated microflows in the user's project. + cleanupSuite := suite + finish := func(result *SuiteResult, runErr error) (*SuiteResult, error) { + fmt.Fprintln(w, "Cleaning up...") + cleanupErr := cleanupEndpoint(opts.ProjectPath, state, cleanupSuite, w) + removeGeneratedJavaSource(opts.ProjectPath, w) + reportCleanup(w, cleanupErr) + if cleanupErr == nil { + fmt.Fprintln(w, " project restored") + } + if runErr != nil { + return nil, runErr + } + if cleanupErr != nil { + return result, fmt.Errorf("cleanup failed, project left modified: %w", cleanupErr) + } + return result, nil + } + + if err := execMDLScript(opts.ProjectPath, endpointMDL, "mxtest-endpoint-*.mdl"); err != nil { + return finish(nil, fmt.Errorf("injecting test endpoint: %w", err)) + } + if err := execMDLScript(opts.ProjectPath, flowsMDL, "mxtest-flows-*.mdl"); err != nil { + return finish(nil, fmt.Errorf("injecting test microflows: %w", err)) + } + for _, cmd := range setupCommands(endpointStartupFlow) { + if err := execMxcliCmd(opts.ProjectPath, cmd); err != nil { + return finish(nil, fmt.Errorf("preparing project for the test run (%s): %w", cmd, err)) + } + } + fmt.Fprintf(w, " After-startup set to %s (registers the endpoint; runs no tests)\n", endpointStartupFlow) + + // --watch keeps the runtime and the build server up and re-runs on every + // change, so it owns the loop — including printing each run's results, which + // it must do before cleanup rather than after. It reports each re-injected + // suite back so cleanup drops what is actually in the project. + if opts.Watch { + return runEndpointWatch(opts, suite, token, timeout, w, finish, func(s *TestSuite) { cleanupSuite = s }) + } + + result, err := runViaEndpoint(opts, suite, token, timeout, w) if err != nil { - return nil, fmt.Errorf("creating temp file: %w", err) + return finish(nil, err) + } + + result, err = finish(result, nil) + if result != nil { + PrintResults(w, result, opts.Color) + if jerr := writeJUnit(opts, result, w); jerr != nil && err == nil { + err = jerr + } + } + return result, err +} + +// runAfterStartup is the original mechanism: compile the suite into the +// after-startup microflow, restart the runtime, and read results out of the log. +// Always the Docker path, and the local path under --legacy-runner. +func runAfterStartup(opts RunOptions, suite *TestSuite, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + fmt.Fprintln(w, "Generating test runner microflow...") + runnerMDL := GenerateTestRunner(suite) + + if opts.Verbose { + fmt.Fprintln(w, "--- Generated MDL ---") + fmt.Fprintln(w, runnerMDL) + fmt.Fprintln(w, "--- End MDL ---") } - tmpPath := tmpFile.Name() - defer os.Remove(tmpPath) - if _, err := tmpFile.WriteString(runnerMDL); err != nil { - tmpFile.Close() - return nil, fmt.Errorf("writing runner MDL: %w", err) + // Save original settings and inject the test runner. + fmt.Fprintln(w, "Injecting test runner into project...") + state, err := captureProjectState(opts.ProjectPath) + if err != nil { + return nil, fmt.Errorf("capturing project state: %w", err) } - tmpFile.Close() - // Execute the MDL to create the TestRunner microflow - if err := execMxcli(opts.ProjectPath, "exec", tmpPath, "-p", opts.ProjectPath); err != nil { + if err := execMDLScript(opts.ProjectPath, runnerMDL, "mxtest-runner-*.mdl"); err != nil { return nil, fmt.Errorf("injecting test runner: %w", err) } // Set after-startup microflow - for _, cmd := range setupCommands() { + for _, cmd := range setupCommands(mxTestRunner) { if err := execMxcliCmd(opts.ProjectPath, cmd); err != nil { return nil, fmt.Errorf("preparing project for the test run (%s): %w", cmd, err) } } fmt.Fprintf(w, " After-startup set to %s\n", mxTestRunner) - // Step 4: Build and restart - dockerDir := filepath.Join(filepath.Dir(opts.ProjectPath), ".docker") - if err := ensureDockerStack(opts.ProjectPath, dockerDir, w); err != nil { - reportCleanup(w, cleanup(opts.ProjectPath, state, w)) - return nil, fmt.Errorf("docker init: %w", err) - } - - if !opts.SkipBuild { - fmt.Fprintln(w, "Building project...") - if err := execMxcli(opts.ProjectPath, "docker", "build", "-p", opts.ProjectPath, "--skip-check"); err != nil { - reportCleanup(w, cleanup(opts.ProjectPath, state, w)) - return nil, fmt.Errorf("docker build: %w", err) - } - } - - fmt.Fprintln(w, "Restarting runtime...") - // Stop existing containers - runCompose(dockerDir, "down") - // Start fresh - if err := runCompose(dockerDir, "up", "--detach", "--force-recreate"); err != nil { - reportCleanup(w, cleanup(opts.ProjectPath, state, w)) - return nil, fmt.Errorf("docker up: %w", err) + var logOutput string + if opts.Local { + logOutput, err = runLocalAndCapture(opts, timeout, w) + } else { + logOutput, err = runDockerAndCapture(opts, timeout, w) } - - // Step 5: Wait for runtime and capture logs - fmt.Fprintf(w, "Waiting for test execution (timeout: %s)...\n", timeout) - logOutput, err := captureRuntimeLogs(dockerDir, timeout, w, opts.Verbose) if err != nil { reportCleanup(w, cleanup(opts.ProjectPath, state, w)) - return nil, fmt.Errorf("runtime execution: %w", err) + return nil, err } - // Step 6: Parse results from logs fmt.Fprintln(w, "Parsing test results...") result := ParseLogResults(strings.NewReader(logOutput), suite) - // Step 7: Cleanup fmt.Fprintln(w, "Cleaning up...") cleanupErr := cleanup(opts.ProjectPath, state, w) reportCleanup(w, cleanupErr) - // Step 8: Output results PrintResults(w, result, opts.Color) - // Write JUnit XML if requested - if opts.JUnitOutput != "" { - f, err := os.Create(opts.JUnitOutput) - if err != nil { - return result, fmt.Errorf("creating JUnit output: %w", err) - } - defer f.Close() - if err := WriteJUnitXML(f, result); err != nil { - return result, fmt.Errorf("writing JUnit XML: %w", err) - } - fmt.Fprintf(w, "JUnit XML written to: %s\n", opts.JUnitOutput) + if err := writeJUnit(opts, result, w); err != nil { + return result, err } // A failed cleanup leaves the project modified, so the run must not be @@ -203,6 +324,23 @@ func Run(opts RunOptions) (*SuiteResult, error) { return result, nil } +// writeJUnit writes the JUnit XML report when one was asked for. +func writeJUnit(opts RunOptions, result *SuiteResult, w io.Writer) error { + if opts.JUnitOutput == "" { + return nil + } + f, err := os.Create(opts.JUnitOutput) + if err != nil { + return fmt.Errorf("creating JUnit output: %w", err) + } + defer f.Close() + if err := WriteJUnitXML(f, result); err != nil { + return fmt.Errorf("writing JUnit XML: %w", err) + } + fmt.Fprintf(w, "JUnit XML written to: %s\n", opts.JUnitOutput) + return nil +} + // ListTests parses test files and prints the test names without executing. func ListTests(files []string, w io.Writer) error { suite, err := parseTestFiles(files) @@ -374,19 +512,55 @@ func moduleExists(projectPath, name string) (bool, error) { } // setupCommands returns the MDL statements Run issues to put the project into its -// testing state. The project's Security Level is deliberately absent: the -// after-startup microflow runs in an administrative context and is not subject to -// it, so forcing it OFF bought nothing — while breaking any project with a -// published REST/OData service using custom authentication ("App security is off, -// but custom authentication is enabled for this service"), and the restore -// hardcoded PRODUCTION, silently changing projects that run at another level -// (mendixlabs/mxcli#802). -func setupCommands() []string { +// testing state, pointing after-startup at startupFlow. The project's Security +// Level is deliberately absent: the after-startup microflow runs in an +// administrative context and is not subject to it, so forcing it OFF bought +// nothing — while breaking any project with a published REST/OData service using +// custom authentication ("App security is off, but custom authentication is +// enabled for this service"), and the restore hardcoded PRODUCTION, silently +// changing projects that run at another level (mendixlabs/mxcli#802). +func setupCommands(startupFlow string) []string { return []string{ - "ALTER SETTINGS MODEL AfterStartupMicroflow = " + quoteMDLString(mxTestRunner), + "ALTER SETTINGS MODEL AfterStartupMicroflow = " + quoteMDLString(startupFlow), } } +// runMDLCommands executes each statement, attempting all of them even after one +// fails, and joins the failures. Restores must not stop at the first error: a +// half-restored project is worse than a fully failed one, because it looks fine +// (#803). +func runMDLCommands(projectPath string, cmds []string) error { + var errs []error + for _, cmd := range cmds { + if err := execMxcliCmd(projectPath, cmd); err != nil { + errs = append(errs, fmt.Errorf("%s: %w", cmd, err)) + } + } + return errors.Join(errs...) +} + +// lowerModule maps a module name to the javasource/ directory Mendix generates +// for it, which is always lowercased. +func lowerModule(name string) string { return strings.ToLower(name) } + +// execMDLScript writes MDL to a temp file and executes it against the project. +func execMDLScript(projectPath, mdl, namePattern string) error { + f, err := os.CreateTemp("", namePattern) + if err != nil { + return fmt.Errorf("creating temp file: %w", err) + } + path := f.Name() + defer os.Remove(path) + + if _, err := f.WriteString(mdl); err != nil { + f.Close() + return fmt.Errorf("writing MDL: %w", err) + } + f.Close() + + return execMxcli(projectPath, "exec", path, "-p", projectPath) +} + // cleanupCommands returns the MDL statements that put the project back the way it // was, in order. Kept separate from execution so the restore can be tested without // a project. mxTestPresent says whether the generated module is still there — @@ -432,16 +606,7 @@ func cleanup(projectPath string, st projectState, w io.Writer) error { fmt.Fprintf(w, " %s module already existed; dropping only %s\n", mxTestModule, mxTestRunner) } - var errs []error - for _, cmd := range cleanupCommands(st, mxTestPresent) { - if err := execMxcliCmd(projectPath, cmd); err != nil { - errs = append(errs, fmt.Errorf("%s: %w", cmd, err)) - } - } - if len(errs) > 0 { - return errors.Join(errs...) - } - return nil + return runMDLCommands(projectPath, cleanupCommands(st, mxTestPresent)) } // reportCleanup prints a cleanup failure prominently. The project is left mutated, @@ -454,6 +619,37 @@ func reportCleanup(w io.Writer, err error) { fmt.Fprintf(w, "Check the after-startup microflow and the %s module before committing.\n", mxTestModule) } +// runDockerAndCapture builds the project, restarts the compose stack, and reads +// the test runner's output from the container log. +func runDockerAndCapture(opts RunOptions, timeout time.Duration, w io.Writer) (string, error) { + dockerDir := filepath.Join(filepath.Dir(opts.ProjectPath), ".docker") + if err := ensureDockerStack(opts.ProjectPath, dockerDir, w); err != nil { + return "", fmt.Errorf("docker init: %w", err) + } + + if !opts.SkipBuild { + fmt.Fprintln(w, "Building project...") + if err := execMxcli(opts.ProjectPath, "docker", "build", "-p", opts.ProjectPath, "--skip-check"); err != nil { + return "", fmt.Errorf("docker build: %w", err) + } + } + + fmt.Fprintln(w, "Restarting runtime...") + runCompose(dockerDir, "down") + if err := runCompose(dockerDir, "up", "--detach", "--force-recreate"); err != nil { + return "", fmt.Errorf("docker up: %w\n"+ + " hint: this environment has no Docker daemon. Pass --local to run the tests "+ + "against mxcli's own runtime instead — no container needed.", err) + } + + fmt.Fprintf(w, "Waiting for test execution (timeout: %s)...\n", timeout) + logOutput, err := captureRuntimeLogs(dockerDir, timeout, w, opts.Verbose) + if err != nil { + return logOutput, fmt.Errorf("runtime execution: %w", err) + } + return logOutput, nil +} + // captureRuntimeLogs tails the docker compose logs, waiting for MXTEST:END or timeout. // Returns the captured log output. func captureRuntimeLogs(dockerDir string, timeout time.Duration, w io.Writer, verbose bool) (string, error) { diff --git a/cmd/mxcli/testrunner/runner_attach.go b/cmd/mxcli/testrunner/runner_attach.go new file mode 100644 index 000000000..53565477c --- /dev/null +++ b/cmd/mxcli/testrunner/runner_attach.go @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// attachedApp is a running app someone else owns, reached through its handshake. +// +// It satisfies the same two needs bootForTests does — a client for the endpoint, +// and a way to apply a model change — but owns neither the runtime nor the build +// server, so it must never stop them. +type attachedApp struct { + client *endpointClient + serve *docker.ServeServer + ctrl *docker.RuntimeController + hs *Handshake +} + +// attach connects to an app already hosting the test endpoint. +func attach(opts RunOptions, w io.Writer) (*attachedApp, error) { + hs, err := ReadHandshake(opts.ProjectPath) + if err != nil { + return nil, err + } + + client := newEndpointClient(hs.AppPort, hs.Token) + if err := client.ping(); err != nil { + return nil, fmt.Errorf("the app on port %d is not answering the test endpoint: %w\n"+ + " The handshake looks live, so the app may still be starting. Retry in a moment.", hs.AppPort, err) + } + + fmt.Fprintf(w, "Attached to the app on port %d (pid %d) — no boot needed.\n", hs.AppPort, hs.PID) + // The dev app's database is the developer's, and these tests are about to + // write to it. That is the whole trade --attach makes, so say it every time + // rather than burying it in the docs. + fmt.Fprintln(w, " NOTE: tests run against the running app's database, not a scratch one.") + + return &attachedApp{ + client: client, + serve: &docker.ServeServer{Host: "127.0.0.1", Port: hs.ServePort}, + // The admin password, not the endpoint token — different secrets. + ctrl: docker.NewRuntimeController(docker.M2EEOptions{ + Host: "127.0.0.1", + Port: hs.AdminPort, + Token: hs.AdminPass, + }), + hs: hs, + }, nil +} + +// applyModelChange rebuilds the project through the dev loop's serve server and +// applies the result through its admin API. +// +// Driving the other process's services rather than requiring it to be in +// --watch: both are plain loopback APIs, so an attach can apply its own +// injections deterministically instead of waiting to see whether someone else's +// watcher noticed. A dev loop that *is* watching may also rebuild — harmless, +// since both produce the same deployment from the same source. +func (a *attachedApp) endpoint() *endpointClient { return a.client } + +func (a *attachedApp) applyModelChange(projectPath string) (string, error) { + build, err := a.serve.Build(docker.BuildRequest{Target: docker.TargetDeploy, ProjectFilePath: projectPath}) + if err != nil { + return "", fmt.Errorf("rebuilding through the attached app's build server on port %d: %w", a.hs.ServePort, err) + } + if !build.OK() { + return "", fmt.Errorf("build failed: %s", build.Message) + } + // No restart callback: the runtime belongs to the other process. A structural + // change is refused rather than half-applied — see the error below. + action, err := a.ctrl.ApplyBuild(build, nil) + if err != nil { + return action.String(), err + } + if action == docker.ActionRestart { + return action.String(), fmt.Errorf("this change needs a runtime restart (an entity or association changed), " + + "which --attach cannot do — the runtime belongs to the 'mxcli run --local' process.\n" + + " Restart that process, or drop --attach to run against a scratch runtime.") + } + return action.String(), nil +} + +// runAttached injects the test microflows into the already-running app, runs the +// suite, and removes them again. +// +// It deliberately does not touch the endpoint or the after-startup setting: the +// dev loop installed those and will remove them when it exits. An attach only +// owns the test microflows it adds. +func runAttached(opts RunOptions, suite *TestSuite, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + app, err := attach(opts, w) + if err != nil { + return nil, err + } + + // Only the generated test microflows are ours to remove. + injected := suite + finish := func(result *SuiteResult, runErr error) (*SuiteResult, error) { + fmt.Fprintln(w, "Cleaning up...") + cleanupErr := runMDLCommands(opts.ProjectPath, dropTestFlows(injected)) + if cleanupErr == nil { + // Leave the app serving a model that matches the project on disk; + // otherwise the developer's next page load still runs the test flows. + if _, err := app.applyModelChange(opts.ProjectPath); err != nil { + fmt.Fprintf(w, " note: the app is still serving the test microflows until its next rebuild: %v\n", err) + } + fmt.Fprintln(w, " test microflows removed") + } else { + reportCleanup(w, cleanupErr) + } + if runErr != nil { + return nil, runErr + } + if cleanupErr != nil { + return result, fmt.Errorf("cleanup failed, project left modified: %w", cleanupErr) + } + return result, nil + } + + fmt.Fprintln(w, "Injecting test microflows...") + if err := execMDLScript(opts.ProjectPath, GenerateTestFlows(suite), "mxtest-flows-*.mdl"); err != nil { + return finish(nil, fmt.Errorf("injecting test microflows: %w", err)) + } + if _, err := app.applyModelChange(opts.ProjectPath); err != nil { + return finish(nil, err) + } + + if opts.Watch { + return runAttachedWatch(opts, app, suite, timeout, w, finish, func(s *TestSuite) { injected = s }) + } + + result, err := runSuite(app.client, suite, opts, w) + if err != nil { + return finish(nil, err) + } + result, err = finish(result, nil) + if result != nil { + PrintResults(w, result, opts.Color) + if jerr := writeJUnit(opts, result, w); jerr != nil && err == nil { + err = jerr + } + } + return result, err +} + +// dropTestFlows returns the DROP statements for a suite's generated microflows. +func dropTestFlows(suite *TestSuite) []string { + if suite == nil { + return nil + } + cmds := make([]string, 0, len(suite.Tests)) + for _, tc := range suite.Tests { + cmds = append(cmds, "DROP MICROFLOW "+testFlowName(tc)) + } + return cmds +} diff --git a/cmd/mxcli/testrunner/runner_cleanup_test.go b/cmd/mxcli/testrunner/runner_cleanup_test.go index 4809c0c04..0c1f0fbf5 100644 --- a/cmd/mxcli/testrunner/runner_cleanup_test.go +++ b/cmd/mxcli/testrunner/runner_cleanup_test.go @@ -107,10 +107,14 @@ func TestQuoteMDLString(t *testing.T) { } // TestNoSecurityLevelManipulation pins #802: the Security Level is the project's -// business. Neither setup nor cleanup may touch it. +// business. Neither setup nor cleanup may touch it — on either mechanism. func TestNoSecurityLevelManipulation(t *testing.T) { - all := append(setupCommands(), cleanupCommands(projectState{}, true)...) + suite := &TestSuite{Tests: []TestCase{{ID: "test_1"}}} + all := append(setupCommands(mxTestRunner), setupCommands(endpointStartupFlow)...) + all = append(all, cleanupCommands(projectState{}, true)...) all = append(all, cleanupCommands(projectState{afterStartup: "Mod.Flow", createdMxTest: true}, true)...) + all = append(all, endpointCleanupCommands(projectState{}, suite, true)...) + all = append(all, endpointCleanupCommands(projectState{afterStartup: "Mod.Flow", createdMxTest: true}, suite, true)...) for _, cmd := range all { if strings.Contains(strings.ToUpper(cmd), "SECURITY LEVEL") { t.Errorf("the runner still alters the project Security Level: %q (#802)", cmd) diff --git a/cmd/mxcli/testrunner/runner_endpoint.go b/cmd/mxcli/testrunner/runner_endpoint.go new file mode 100644 index 000000000..c3d67c0de --- /dev/null +++ b/cmd/mxcli/testrunner/runner_endpoint.go @@ -0,0 +1,241 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "os" + "path/filepath" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// testAppSession is a booted app plus a client for its test endpoint. It is what +// the warm loop keeps alive between runs. +type testAppSession struct { + app *docker.LocalApp + client *endpointClient + logPath string +} + +// bootForTests boots the app and waits for the test endpoint to register. +func bootForTests(opts RunOptions, token string, timeout time.Duration, w io.Writer) (*testAppSession, error) { + logPath := filepath.Join(filepath.Dir(opts.ProjectPath), ".mxcli", "test-runtime.log") + + fmt.Fprintln(w, "Starting local runtime (no Docker)...") + app, err := docker.StartLocalApp(docker.LocalAppOptions{ + ProjectPath: opts.ProjectPath, + AppPort: localTestAppPort, + AdminPort: localTestAdminPort, + ServePort: localTestServePort, + DB: docker.DBConfig{ + Name: docker.DeriveDBName(opts.ProjectPath) + localTestDBSuffix, + }, + EnsureDB: true, + SkipBuild: opts.SkipBuild, + // The token reaches the runtime through its environment and is never + // written to the project. See endpointTokenEnv. + Env: []string{endpointTokenEnv + "=" + token}, + RuntimeLogPath: logPath, + Stdout: w, + Stderr: w, + }) + if err != nil { + // Unlike the after-startup path, a boot failure here is never a test + // result — no test has run yet. It is always a real error. + return nil, fmt.Errorf("local runtime: %w", err) + } + + client := newEndpointClient(localTestAppPort, token) + if err := client.waitReady(endpointReadyTimeout(timeout)); err != nil { + app.Stop() + return nil, fmt.Errorf("%w\n hint: check %s for a registration failure", err, logPath) + } + return &testAppSession{app: app, client: client, logPath: logPath}, nil +} + +func (s *testAppSession) stop() { + if s != nil && s.app != nil { + s.app.Stop() + } +} + +// testTarget is an app the runner can run tests against and push model changes +// into. Two things satisfy it: a runtime the runner booted itself, and one +// already running that it attached to. The watch loop is written against this +// so it does not care which. +type testTarget interface { + // endpoint is the client for the app's test endpoint. + endpoint() *endpointClient + // applyModelChange rebuilds the project and applies it, returning a label for + // what it took ("reload"/"restart"). It returns only once the endpoint is + // reachable again, so the caller can invoke a test straight after. + applyModelChange(projectPath string) (string, error) +} + +func (s *testAppSession) endpoint() *endpointClient { return s.client } + +// applyModelChange rebuilds through the serve server this session owns. +// +// A restart is fine here — the session owns the runtime — but it re-runs +// after-startup, so the endpoint has to be waited for before the next test. +func (s *testAppSession) applyModelChange(projectPath string) (string, error) { + action, _, err := s.app.Rebuild(projectPath) + if err != nil { + return action.String(), err + } + if action == docker.ActionRestart { + if err := s.client.waitReady(endpointReadyTimeout(0)); err != nil { + return action.String(), fmt.Errorf("the test endpoint did not come back after a restart: %w", err) + } + } + return action.String(), nil +} + +// runViaEndpoint boots the app once and drives the suite over HTTP. +// +// The contrast with the after-startup path is the whole point: there, tests run +// during boot, so every re-run is a restart and a result is something recovered +// from the log. Here boot only registers the endpoint, and each test is a +// request against a runtime that stays up. +func runViaEndpoint(opts RunOptions, suite *TestSuite, token string, timeout time.Duration, w io.Writer) (*SuiteResult, error) { + sess, err := bootForTests(opts, token, timeout, w) + if err != nil { + return nil, err + } + defer sess.stop() + return runSuite(sess.client, suite, opts, w) +} + +// runSuite invokes every test in the suite against a booted app and collects the +// verdicts. It never returns an error for a test-level problem — a missing +// microflow or a failed request is that test's result, so one bad test cannot +// hide every other. +func runSuite(client *endpointClient, suite *TestSuite, opts RunOptions, w io.Writer) (*SuiteResult, error) { + // Ask the app which test microflows it actually has. A test whose microflow + // is missing is reported as an error against that test rather than failing + // the run. + present := make(map[string]bool) + names, err := client.list() + if err != nil { + return nil, fmt.Errorf("listing test microflows: %w", err) + } + for _, n := range names { + present[n] = true + } + + result := &SuiteResult{Name: suite.Name, Started: time.Now()} + fmt.Fprintf(w, "Running %d test(s) over the test endpoint...\n", len(suite.Tests)) + + for _, tc := range suite.Tests { + flow := testFlowName(tc) + if !present[flow] { + result.Tests = append(result.Tests, TestResult{ + ID: tc.ID, + Name: tc.Name, + Status: StatusError, + Message: fmt.Sprintf("microflow %s was not created — the test body may not have compiled", flow), + }) + continue + } + + rr, err := client.run(flow) + if err != nil { + // A transport failure is not a verdict. Report it against this test + // and keep going; if the runtime died the rest will say so too. + result.Tests = append(result.Tests, TestResult{ + ID: tc.ID, + Name: tc.Name, + Status: StatusError, + Message: fmt.Sprintf("calling the test endpoint: %v", err), + }) + continue + } + + res := toResult(tc, rr) + result.Tests = append(result.Tests, res) + if opts.Verbose { + fmt.Fprintf(w, " %s %s (%s)\n", res.Status, res.Name, res.Duration.Round(time.Millisecond)) + } + } + + result.Duration = time.Since(result.Started) + return result, nil +} + +// endpointReadyTimeout bounds the wait for the endpoint to register. It is +// capped well below the suite timeout: the handler is registered during the +// start action, so if it is not up shortly after the runtime reports started, it +// is not coming. +func endpointReadyTimeout(suiteTimeout time.Duration) time.Duration { + const cap = 60 * time.Second + if suiteTimeout > 0 && suiteTimeout < cap { + return suiteTimeout + } + return cap +} + +// endpointCleanupCommands returns the MDL that removes what the endpoint path +// injected, in order. +// +// It mirrors cleanupCommands but has more to take out: the registration Java +// action and startup microflow, plus one microflow per test. When Run created +// the MxTest module, dropping the module removes all of it in one statement; +// when the module was already the user's, each generated document is named +// explicitly so nothing of theirs is touched. +func endpointCleanupCommands(st projectState, suite *TestSuite, mxTestPresent bool) []string { + restore := "ALTER SETTINGS MODEL AfterStartupMicroflow = ''" + if st.afterStartup != "" { + restore = "ALTER SETTINGS MODEL AfterStartupMicroflow = " + quoteMDLString(st.afterStartup) + } + cmds := []string{restore} + if !mxTestPresent { + return cmds + } + if st.createdMxTest { + return append(cmds, "DROP MODULE "+mxTestModule) + } + for _, tc := range suite.Tests { + cmds = append(cmds, "DROP MICROFLOW "+testFlowName(tc)) + } + return append(cmds, + "DROP MICROFLOW "+endpointStartupFlow, + "DROP JAVA ACTION "+endpointRegisterAction, + ) +} + +// cleanupEndpoint restores the project after an endpoint run. +// +// As with cleanup, every statement is attempted even after one fails and the +// failures are returned rather than warned about: a half-restored project still +// carries a test endpoint and an after-startup pointing at it. +func cleanupEndpoint(projectPath string, st projectState, suite *TestSuite, w io.Writer) error { + mxTestPresent := true + if exists, err := moduleExists(projectPath, mxTestModule); err == nil { + mxTestPresent = exists + } + if mxTestPresent && !st.createdMxTest { + fmt.Fprintf(w, " %s module already existed; dropping only the generated documents\n", mxTestModule) + } + return runMDLCommands(projectPath, endpointCleanupCommands(st, suite, mxTestPresent)) +} + +// removeGeneratedJavaSource deletes the .java file the Java action generated. +// +// DROP JAVA ACTION removes the model document; the source file it wrote into +// javasource/ is not the model's to delete, so it is left behind. For a +// generated per-run artifact that is litter in the user's tree — and litter that +// still contains a request handler, which is exactly what should not be left +// lying around. Failure is not fatal: the file is inert without the model +// document, and a cleanup error here would mask the real ones. +func removeGeneratedJavaSource(projectPath string, w io.Writer) { + dir := filepath.Join(filepath.Dir(projectPath), "javasource", lowerModule(mxTestModule)) + if _, err := os.Stat(dir); err != nil { + return + } + if err := os.RemoveAll(dir); err != nil { + fmt.Fprintf(w, " note: could not remove generated Java source at %s: %v\n", dir, err) + } +} diff --git a/cmd/mxcli/testrunner/runner_local.go b/cmd/mxcli/testrunner/runner_local.go new file mode 100644 index 000000000..9092c6d62 --- /dev/null +++ b/cmd/mxcli/testrunner/runner_local.go @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "bufio" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// Local test runs deliberately do not share the dev loop's ports or database: +// `mxcli run --local` may well be serving the same project (that is the point of +// the warm loop), and a test run must neither refuse to start because of it nor +// write its fixtures into the database the developer is looking at. +const ( + localTestAppPort = 8081 + localTestAdminPort = 8091 + localTestServePort = 6544 + // localTestDBSuffix is appended to the project's local database name. + localTestDBSuffix = "_test" +) + +// runLocalAndCapture builds the project and boots mxcli's own runtime — no +// Docker — then reads the test runner's output out of the runtime log. +func runLocalAndCapture(opts RunOptions, timeout time.Duration, w io.Writer) (string, error) { + logPath := filepath.Join(filepath.Dir(opts.ProjectPath), ".mxcli", "test-runtime.log") + // The log is appended across runs, so remember where this run starts. + offset := fileSize(logPath) + + fmt.Fprintln(w, "Starting local runtime (no Docker)...") + app, err := docker.StartLocalApp(docker.LocalAppOptions{ + ProjectPath: opts.ProjectPath, + AppPort: localTestAppPort, + AdminPort: localTestAdminPort, + ServePort: localTestServePort, + DB: docker.DBConfig{ + Name: docker.DeriveDBName(opts.ProjectPath) + localTestDBSuffix, + }, + EnsureDB: true, + SkipBuild: opts.SkipBuild, + // The runner reports through an after-startup microflow, so its LOG output + // is produced DURING the start action — before the runtime's own log + // subscriber is attached. What carries it is the JVM console tee, which is + // live from spawn. Verified on 11.12.1; registering the subscriber early + // instead is not an option, the runtime rejects it pre-start with a + // LoggingException. If a future runtime stops echoing to the console the + // failure is loud, not silent: unseen tests are reported as errors. + RuntimeLogPath: logPath, + Stdout: w, + Stderr: w, + }) + if err != nil { + // A failing test IS a failed boot: the generated runner returns false, so + // the runtime's after-startup action fails and `start` reports an error. + // That is a normal test outcome, not a broken run — if the log shows the + // runner reached a verdict, hand it back and let the results speak. (The + // Docker path gets this for free by only ever reading the container log.) + tail := readFrom(logPath, offset) + if runnerReportedVerdict(tail) { + return tail, nil + } + if tail != "" { + return tail, fmt.Errorf("local runtime: %w", err) + } + return "", fmt.Errorf("local runtime: %w", err) + } + defer app.Stop() + + fmt.Fprintf(w, "Waiting for test execution (timeout: %s)...\n", timeout) + return waitForTestLog(logPath, offset, timeout, w, opts.Verbose) +} + +// waitForTestLog polls the runtime log from offset until the run reports a +// terminal marker or the timeout expires, returning everything this run wrote. +// +// Polling rather than following: the after-startup microflow normally completes +// inside the start action, so the output is usually already on disk by the time +// this is called — the loop exists for the case where it is not. +func waitForTestLog(path string, offset int64, timeout time.Duration, w io.Writer, verbose bool) (string, error) { + deadline := time.Now().Add(timeout) + echoed := 0 + + for { + content := readFrom(path, offset) + + if verbose { + lines := splitLines(content) + for ; echoed < len(lines); echoed++ { + fmt.Fprintln(w, lines[echoed]) + } + } + + if done, failMsg := scanTestLog(content); done { + if failMsg != "" { + return content, fmt.Errorf("runtime failed: %s", failMsg) + } + return content, nil + } + + if time.Now().After(deadline) { + return content, fmt.Errorf("timeout after %s waiting for test completion", timeout) + } + time.Sleep(250 * time.Millisecond) + } +} + +// scanTestLog reports whether the log has reached a terminal state, and the +// failure line when the runtime failed rather than the tests completing. The +// markers match the Docker path's, so both modes stop on the same conditions. +func scanTestLog(content string) (done bool, failMsg string) { + for _, line := range splitLines(content) { + switch { + case strings.Contains(line, "Error starting runtime"), + strings.Contains(line, "Critical error"), + strings.Contains(line, "After startup microflow should return a boolean"): + return true, line + case strings.Contains(line, "Successfully ran after-startup-action"), + runnerReportedVerdict(line): + return true, "" + } + } + return false, "" +} + +// runnerReportedVerdict reports whether the test runner got far enough to +// produce results — either it finished, or the runtime said the after-startup +// action failed, which is what a failing test looks like from outside. +func runnerReportedVerdict(content string) bool { + return strings.Contains(content, "MXTEST:END:") || + strings.Contains(content, "after-startup-action failed") || + strings.Contains(content, "After-startup action failed") +} + +func splitLines(s string) []string { + if s == "" { + return nil + } + return strings.Split(strings.TrimRight(s, "\n"), "\n") +} + +// fileSize returns the file's current size, or 0 when it does not exist yet. +func fileSize(path string) int64 { + info, err := os.Stat(path) + if err != nil { + return 0 + } + return info.Size() +} + +// readFrom returns the file's content from offset onward, or "" if unreadable. +func readFrom(path string, offset int64) string { + f, err := os.Open(path) + if err != nil { + return "" + } + defer f.Close() + if _, err := f.Seek(offset, io.SeekStart); err != nil { + return "" + } + var b strings.Builder + sc := bufio.NewScanner(f) + sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) + for sc.Scan() { + b.WriteString(sc.Text()) + b.WriteByte('\n') + } + return b.String() +} diff --git a/cmd/mxcli/testrunner/runner_local_test.go b/cmd/mxcli/testrunner/runner_local_test.go new file mode 100644 index 000000000..e791a3fe4 --- /dev/null +++ b/cmd/mxcli/testrunner/runner_local_test.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// A failing test makes the runner return false, which makes the runtime's +// after-startup action fail, which makes `start` report an error. That is a test +// verdict, not a broken run — the local path must recognise it and let the +// results be parsed rather than aborting with a runtime error. +func TestRunnerReportedVerdict(t *testing.T) { + tests := []struct { + name string + log string + want bool + }{ + {"clean finish", "INFO - MXTEST: MXTEST:END:tests\n", true}, + {"m2ee wording", "The after-startup-action failed with an exception or returned false.\n", true}, + {"runtime log wording", "2026-01-01 ERROR - Core: After-startup action failed.\n", true}, + {"still running", "INFO - MXTEST: MXTEST:RUN:test_1:adds\n", false}, + {"boot died early", "java.lang.OutOfMemoryError\n", false}, + {"empty", "", false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := runnerReportedVerdict(tt.log); got != tt.want { + t.Errorf("runnerReportedVerdict(%q) = %v, want %v", tt.log, got, tt.want) + } + }) + } +} + +func TestScanTestLog(t *testing.T) { + tests := []struct { + name string + log string + wantDone bool + wantFailure bool + }{ + {"end marker", "MXTEST:END:tests\n", true, false}, + {"after-startup success", "Successfully ran after-startup-action\n", true, false}, + {"failing test", "Core: After-startup action failed.\n", true, false}, + {"runtime failure", "Error starting runtime: boom\n", true, true}, + {"non-boolean runner", "After startup microflow should return a boolean\n", true, true}, + {"mid-run", "MXTEST:RUN:test_1:adds\n", false, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + done, failMsg := scanTestLog(tt.log) + if done != tt.wantDone { + t.Errorf("done = %v, want %v", done, tt.wantDone) + } + if (failMsg != "") != tt.wantFailure { + t.Errorf("failMsg = %q, want failure=%v", failMsg, tt.wantFailure) + } + }) + } +} + +// The runtime log is appended across runs, so a run must read only what it +// wrote — otherwise the previous run's verdict is reported as this one's. +func TestReadFrom_SkipsEarlierRuns(t *testing.T) { + path := filepath.Join(t.TempDir(), "runtime.log") + previous := "MXTEST:END:old run\n" + if err := os.WriteFile(path, []byte(previous), 0o644); err != nil { + t.Fatal(err) + } + + offset := fileSize(path) + if offset != int64(len(previous)) { + t.Fatalf("offset = %d, want %d", offset, len(previous)) + } + + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + f.WriteString("MXTEST:START:new run\n") + f.Close() + + got := readFrom(path, offset) + if strings.Contains(got, "old run") { + t.Errorf("read included the previous run: %q", got) + } + if !strings.Contains(got, "new run") { + t.Errorf("read missed this run's output: %q", got) + } +} + +func TestFileSize_MissingFileIsZero(t *testing.T) { + if got := fileSize(filepath.Join(t.TempDir(), "absent.log")); got != 0 { + t.Errorf("fileSize of a missing file = %d, want 0", got) + } +} + +func TestWaitForTestLog_ReturnsOnTerminalMarker(t *testing.T) { + path := filepath.Join(t.TempDir(), "runtime.log") + content := "MXTEST:START:s\nMXTEST:PASS:test_1\nMXTEST:END:s\n" + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + + var out bytes.Buffer + got, err := waitForTestLog(path, 0, 2*time.Second, &out, false) + if err != nil { + t.Fatalf("waitForTestLog: %v", err) + } + if !strings.Contains(got, "MXTEST:PASS:test_1") { + t.Errorf("returned log missing the result line: %q", got) + } +} + +func TestWaitForTestLog_TimesOutWithoutMarker(t *testing.T) { + path := filepath.Join(t.TempDir(), "runtime.log") + if err := os.WriteFile(path, []byte("still booting\n"), 0o644); err != nil { + t.Fatal(err) + } + + var out bytes.Buffer + got, err := waitForTestLog(path, 0, 300*time.Millisecond, &out, false) + if err == nil { + t.Fatal("expected a timeout error") + } + // The partial log still comes back, so the caller can show what happened. + if !strings.Contains(got, "still booting") { + t.Errorf("timeout dropped the partial log: %q", got) + } +} diff --git a/cmd/mxcli/testrunner/watch.go b/cmd/mxcli/testrunner/watch.go new file mode 100644 index 000000000..2457f4aab --- /dev/null +++ b/cmd/mxcli/testrunner/watch.go @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "fmt" + "io" + "os" + "os/signal" + "path/filepath" + "syscall" + "time" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// watchPollInterval is how often the loop checks for a change. Polling rather +// than inotify for the same reason `run --local --watch` polls: container +// filesystems do not reliably deliver inotify events for host-mounted paths. +const watchPollInterval = 1 * time.Second + +// runEndpointWatch keeps the runtime and the build server up across runs, +// re-running the suite on every change to a test file or to the app's model. +// +// This is what the endpoint was for. A cold boot is ~30s and the one-shot runner +// pays it on every invocation; here it is paid once and each subsequent run is a +// warm rebuild plus a few HTTP calls. +// +// The loop has one hazard the dev loop does not: **the runner writes to the +// project it is watching**. Injecting the test microflows changes model source, +// which is the very signal being polled, so every baseline is taken *after* the +// injection and rebuild have settled. Getting that wrong is an infinite rebuild +// loop, not a subtle bug. +func runEndpointWatch(opts RunOptions, suite *TestSuite, token string, timeout time.Duration, w io.Writer, finish finishFunc, onInject func(*TestSuite)) (*SuiteResult, error) { + sess, err := bootForTests(opts, token, timeout, w) + if err != nil { + return finish(nil, err) + } + // Belt and braces: the exits below all stop the app explicitly, before + // cleanup rewrites the project out from under it. This catches a panic. + defer sess.stop() + + // shutdown stops the app and then restores the project, in that order — + // nothing should still be serving a model that is about to have the test + // endpoint removed from it. + shutdown := func(result *SuiteResult, runErr error) (*SuiteResult, error) { + fmt.Fprintln(w, "Stopping the runtime...") + sess.stop() + return finish(result, runErr) + } + return watchLoop(opts, sess, suite, w, shutdown, onInject) +} + +// runAttachedWatch is the same loop against an app someone else owns. Nothing is +// stopped on the way out — only the injected test microflows are removed. +func runAttachedWatch(opts RunOptions, app *attachedApp, suite *TestSuite, timeout time.Duration, w io.Writer, finish finishFunc, onInject func(*TestSuite)) (*SuiteResult, error) { + return watchLoop(opts, app, suite, w, finish, onInject) +} + +// watchLoop re-runs the suite on every change until interrupted. +func watchLoop(opts RunOptions, target testTarget, suite *TestSuite, w io.Writer, shutdown finishFunc, onInject func(*TestSuite)) (*SuiteResult, error) { + // Ctrl-C has to reach the cleanup, not kill the process with the project + // still carrying the test endpoint and an after-startup pointing at it. + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + defer signal.Stop(sigCh) + + ticker := time.NewTicker(watchPollInterval) + defer ticker.Stop() + + injected := suite + var last *SuiteResult + gen := 0 + + for { + gen++ + result, err := runSuite(target.endpoint(), injected, opts, w) + if err != nil { + // The endpoint stopped answering — the runtime is probably gone, and + // nothing further will work. Bail rather than spin. + return shutdown(last, err) + } + last = result + PrintResults(w, result, opts.Color) + if err := writeJUnit(opts, result, w); err != nil { + fmt.Fprintf(w, " %v\n", err) + } + + // Baseline AFTER the run, so an edit made while tests were executing is + // still caught on the next tick. + baseline := watchMTime(opts) + fmt.Fprintf(w, "\nWatching tests + model for changes (run #%d; Ctrl-C to stop)...\n", gen) + + changed := false + for !changed { + select { + case <-sigCh: + fmt.Fprintln(w, "\nShutting down...") + return shutdown(last, nil) + case <-ticker.C: + if now := watchMTime(opts); now.After(baseline) { + changed = true + } + } + } + + fmt.Fprintln(w, "Change detected, rebuilding...") + start := time.Now() + + // Re-parse: a test may have been added, edited, or deleted. + reparsed, err := parseTestFiles(opts.TestFiles) + if err != nil { + fmt.Fprintf(w, " test files do not parse: %v\n", err) + continue + } + + // Report the new set BEFORE injecting: if the injection fails partway, + // cleanup must still know about the microflows that did land. Dropping a + // microflow that was never created is harmless; leaving one behind is not. + onInject(reparsed) + if err := reinjectTests(opts, injected, reparsed, w); err != nil { + fmt.Fprintf(w, " injecting tests: %v\n", err) + injected = reparsed + continue + } + injected = reparsed + + action, err := target.applyModelChange(opts.ProjectPath) + if err != nil { + // Not fatal: a build error is usually the edit that just happened, and + // the next save is likely to fix it. Report and keep watching. + fmt.Fprintf(w, " %v\n", err) + continue + } + fmt.Fprintf(w, " rebuilt and applied via %s in %s\n", action, time.Since(start).Round(time.Millisecond)) + } +} + +// finishFunc restores the project and decides the final result. runEndpointWatch +// takes it rather than owning cleanup so that every exit — a clean Ctrl-C, a +// dead runtime, a boot failure — goes through the same restore as the one-shot +// path. +type finishFunc func(result *SuiteResult, runErr error) (*SuiteResult, error) + +// watchMTime is the change signal: the newer of the test files' and the model's +// modification times. +// +// Both matter, and for different reasons. A test file changing means the +// assertions changed. The model changing means the code under test changed — +// which is the case a developer actually cares about, editing a microflow and +// wanting to know immediately whether it still passes. +func watchMTime(opts RunOptions) time.Time { + newest := docker.ProjectSourceMTime(opts.ProjectPath) + if t := testFilesMTime(opts.TestFiles); t.After(newest) { + newest = t + } + return newest +} + +// testFilesMTime is the newest modification time across the test paths, which +// may be individual files or directories. +// +// A directory is walked rather than stat'ed: on Linux a directory's own mtime +// changes when an entry is created or removed but *not* when an existing file is +// edited in place, which is the common case this loop exists to catch. +func testFilesMTime(paths []string) time.Time { + var newest time.Time + for _, p := range paths { + info, err := os.Stat(p) + if err != nil { + continue + } + if !info.IsDir() { + if info.ModTime().After(newest) { + newest = info.ModTime() + } + continue + } + filepath.WalkDir(p, func(path string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() || !isTestFile(d.Name()) { + return nil //nolint:nilerr // an unreadable entry is not a change + } + if fi, err := d.Info(); err == nil && fi.ModTime().After(newest) { + newest = fi.ModTime() + } + return nil + }) + // Catch a deletion too: the directory's own mtime moves when an entry + // goes away, and the walk above cannot see a file that is no longer there. + if info.ModTime().After(newest) { + newest = info.ModTime() + } + } + return newest +} + +// reinjectTests updates the project's generated test microflows to match a +// re-parsed suite. +// +// CREATE OR REPLACE covers a test that was added or edited, but says nothing +// about one that was deleted: its microflow would linger and keep being invoked, +// reporting a stale pass for a test that no longer exists. So the flows for tests +// that are gone are dropped explicitly. +func reinjectTests(opts RunOptions, old, new *TestSuite, w io.Writer) error { + if drops := staleTestFlows(old, new); len(drops) > 0 { + fmt.Fprintf(w, " dropping %d removed test microflow(s)\n", len(drops)) + if err := runMDLCommands(opts.ProjectPath, drops); err != nil { + return err + } + } + return execMDLScript(opts.ProjectPath, GenerateTestFlows(new), "mxtest-flows-*.mdl") +} + +// staleTestFlows returns DROP statements for test microflows in old that no +// longer have a counterpart in new. +func staleTestFlows(old, new *TestSuite) []string { + if old == nil { + return nil + } + keep := make(map[string]bool, len(new.Tests)) + for _, tc := range new.Tests { + keep[testFlowName(tc)] = true + } + var drops []string + for _, tc := range old.Tests { + if flow := testFlowName(tc); !keep[flow] { + drops = append(drops, "DROP MICROFLOW "+flow) + } + } + return drops +} diff --git a/cmd/mxcli/testrunner/watch_test.go b/cmd/mxcli/testrunner/watch_test.go new file mode 100644 index 000000000..4566dc329 --- /dev/null +++ b/cmd/mxcli/testrunner/watch_test.go @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestValidateOptions(t *testing.T) { + tests := []struct { + name string + opts RunOptions + wantErr string + }{ + {name: "no watch is always fine", opts: RunOptions{}}, + {name: "watch with local", opts: RunOptions{Watch: true, Local: true}}, + { + name: "watch without local", + opts: RunOptions{Watch: true}, + wantErr: "--watch requires --local", + }, + { + name: "watch with the legacy runner", + opts: RunOptions{Watch: true, Local: true, LegacyRunner: true}, + wantErr: "--watch cannot be combined with --legacy-runner", + }, + { + name: "watch with skip-build", + opts: RunOptions{Watch: true, Local: true, SkipBuild: true}, + wantErr: "--watch cannot be combined with --skip-build", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateOptions(tt.opts) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + return + } + if err == nil { + t.Fatalf("expected an error containing %q", tt.wantErr) + } + if !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("error = %q, want it to contain %q", err, tt.wantErr) + } + }) + } +} + +// writeTestFile writes a test file with a controlled mtime. +func writeTestFile(t *testing.T, path string, mtime time.Time) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + if err := os.WriteFile(path, []byte("/** @test x */\n"), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if err := os.Chtimes(path, mtime, mtime); err != nil { + t.Fatalf("chtimes: %v", err) + } +} + +// TestTestFilesMTimeSeesAnEditInsideADirectory pins the reason the directory is +// walked rather than stat'ed: on Linux a directory's own mtime does not move +// when an existing entry is edited in place, which is the common case. +func TestTestFilesMTimeSeesAnEditInsideADirectory(t *testing.T) { + dir := t.TempDir() + old := time.Now().Add(-2 * time.Hour) + f := filepath.Join(dir, "a.test.mdl") + writeTestFile(t, f, old) + // Pin the directory itself to the old time, so only the file can move. + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + + before := testFilesMTime([]string{dir}) + + edited := time.Now() + writeTestFile(t, f, edited) + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + + after := testFilesMTime([]string{dir}) + if !after.After(before) { + t.Errorf("editing a file in a watched directory produced no change signal (before=%v after=%v)", before, after) + } +} + +func TestTestFilesMTimeAcceptsAFilePath(t *testing.T) { + dir := t.TempDir() + f := filepath.Join(dir, "a.test.mdl") + want := time.Now().Add(-time.Hour).Truncate(time.Second) + writeTestFile(t, f, want) + + if got := testFilesMTime([]string{f}); !got.Truncate(time.Second).Equal(want) { + t.Errorf("mtime = %v, want %v", got, want) + } +} + +// TestTestFilesMTimeIgnoresNonTestFiles keeps an unrelated file in the tests +// directory — a README, an editor swap file — from re-triggering the loop. +func TestTestFilesMTimeIgnoresNonTestFiles(t *testing.T) { + dir := t.TempDir() + old := time.Now().Add(-2 * time.Hour) + writeTestFile(t, filepath.Join(dir, "a.test.mdl"), old) + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + before := testFilesMTime([]string{dir}) + + noise := filepath.Join(dir, "notes.md") + if err := os.WriteFile(noise, []byte("scratch"), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + now := time.Now() + if err := os.Chtimes(noise, now, now); err != nil { + t.Fatalf("chtimes: %v", err) + } + // Hold the directory back so only the noise file could move the signal. + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + + if after := testFilesMTime([]string{dir}); after.After(before) { + t.Error("a non-test file in the tests directory moved the change signal") + } +} + +// TestTestFilesMTimeSeesADeletion pins that removing a test is a change. The +// walk cannot see a file that is gone, so the directory's own mtime — which does +// move on a deletion — has to be folded in. +func TestTestFilesMTimeSeesADeletion(t *testing.T) { + dir := t.TempDir() + old := time.Now().Add(-2 * time.Hour) + keep := filepath.Join(dir, "a.test.mdl") + gone := filepath.Join(dir, "b.test.mdl") + writeTestFile(t, keep, old) + writeTestFile(t, gone, old) + if err := os.Chtimes(dir, old, old); err != nil { + t.Fatalf("chtimes dir: %v", err) + } + before := testFilesMTime([]string{dir}) + + if err := os.Remove(gone); err != nil { + t.Fatalf("remove: %v", err) + } + + if after := testFilesMTime([]string{dir}); !after.After(before) { + t.Errorf("deleting a test file produced no change signal (before=%v after=%v)", before, after) + } +} + +// TestStaleTestFlowsDropsRemovedTests pins the correctness hazard of re-running: +// CREATE OR REPLACE updates a test that changed but says nothing about one that +// was deleted, whose microflow would otherwise linger and keep reporting a pass. +func TestStaleTestFlowsDropsRemovedTests(t *testing.T) { + old := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}, {ID: "test_3"}}} + new := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + + got := staleTestFlows(old, new) + want := []string{"DROP MICROFLOW MxTest.Test_test_3"} + if len(got) != len(want) || (len(got) > 0 && got[0] != want[0]) { + t.Errorf("got %q, want %q", got, want) + } +} + +func TestStaleTestFlowsNoneWhenSuiteGrew(t *testing.T) { + old := &TestSuite{Tests: []TestCase{{ID: "test_1"}}} + new := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + + if got := staleTestFlows(old, new); len(got) != 0 { + t.Errorf("got %q, want no drops when tests were only added", got) + } +} + +// TestStaleTestFlowsDropsAllWhenEmptied covers deleting the last test in a file: +// every previously-injected flow has to come out. +func TestStaleTestFlowsDropsAllWhenEmptied(t *testing.T) { + old := &TestSuite{Tests: []TestCase{{ID: "test_1"}, {ID: "test_2"}}} + new := &TestSuite{} + + if got := staleTestFlows(old, new); len(got) != 2 { + t.Errorf("got %d drops %q, want 2", len(got), got) + } +} + +func TestStaleTestFlowsHandlesNoPriorSuite(t *testing.T) { + if got := staleTestFlows(nil, &TestSuite{Tests: []TestCase{{ID: "test_1"}}}); got != nil { + t.Errorf("got %q, want nil for a first injection", got) + } +} diff --git a/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-atlas-map.scss b/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-atlas-map.scss index 17ed472eb..fdc03a81a 100644 --- a/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-atlas-map.scss +++ b/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-atlas-map.scss @@ -35,8 +35,12 @@ // brand fill — so it tracks the rail, which is the surface it actually // sits on. Text on a brand-filled button comes from --btn-*-color below. --font-color-contrast: var(--mxt-rail-ink-active); - --link-color: var(--mxt-brand); - --link-hover-color: var(--mxt-brand-hover); + // Link TEXT needs 4.5:1 against the page; a brand used as a button FILL + // only needs 3:1 plus contrast against its own ink. A theme whose brand is + // too light to read as text sets --mxt-link rather than darkening the + // brand everywhere. (mxcli-todo findings #19c) + --link-color: var(--mxt-link, var(--mxt-brand)); + --link-hover-color: var(--mxt-link-hover, var(--mxt-brand-hover)); --border-color-default: var(--mxt-line); --border-radius-s: var(--mxt-radius); @@ -203,3 +207,45 @@ .navbar-brand .widget-language-selector .language-arrow { color: var(--mxt-rail-ink-active, var(--mxt-rail-ink)); } + +// --------------------------------------------------------------------------- +// The login page — the first screen a user sees, and the one Atlas themes least. +// +// It is served from theme/web/login.html, which loads the SAME compiled theme +// CSS ({{themecss}}), so it can be themed here — but Atlas's own rules leave it +// off-brand in two visible ways (mxcli-todo findings #19): +// +// 1. `.loginpage-image` layers a brand-tinted gradient over Atlas's stock +// photograph (`url("./resources/work-do-more.jpeg")`), so a dark app opens +// on a large, bright, unrelated picture. +// 2. The submit button is `.btn-success`, so it follows the SUCCESS colour, +// not the brand — on console that is a green button two screens away from +// a teal one. +// +// The project's own logo (theme/web/logo.png) is deliberately left alone: it is +// the app's asset to replace, and hiding it would remove a real logo from apps +// that have one. +// --------------------------------------------------------------------------- +.loginpage-image { + background: + left / cover no-repeat + linear-gradient( + 160deg, + var(--mxt-brand) 0%, + color-mix(in srgb, var(--mxt-brand) 55%, var(--mxt-rail)) 100% + ); +} + +.loginpage-form .btn-success { + background-color: var(--mxt-brand); + border-color: var(--mxt-brand); + color: var(--mxt-brand-ink); +} + +.loginpage-form .btn-success:hover, +.loginpage-form .btn-success:focus, +.loginpage-form .btn-success:active { + background-color: var(--mxt-brand-hover); + border-color: var(--mxt-brand-hover); + color: var(--mxt-brand-ink); +} diff --git a/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-console.scss b/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-console.scss index f4c0abd9d..5847bb6e4 100644 --- a/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-console.scss +++ b/cmd/mxcli/theme/assets/console/files/theme/web/_mxcli-console.scss @@ -19,6 +19,15 @@ $mxcli-theme-variant: {{VARIANT}} !default; --mxt-brand: #0d9488; --mxt-brand-hover: #0b7f75; --mxt-brand-ink: #ffffff; + + // Console is dark-first, and its teal reads well on a dark ground. As LINK + // TEXT on the light palette the same teal is 3.74:1 against white — under the + // 4.5:1 AA needs for body text — so links get a darker teal of their own + // rather than the whole brand being darkened (which would also change every + // button fill). #0f766e measures 5.47:1 on surface, 5.10:1 on ground and + // 4.92:1 on surface-alt. (mxcli-todo findings #19c) + --mxt-link: #0f766e; + --mxt-link-hover: #0c5f58; --mxt-accent: #7c5ce0; --mxt-success: #2a8a3c; diff --git a/cmd/mxcli/theme/assets/ledger/files/theme/web/_mxcli-atlas-map.scss b/cmd/mxcli/theme/assets/ledger/files/theme/web/_mxcli-atlas-map.scss index 17ed472eb..fdc03a81a 100644 --- a/cmd/mxcli/theme/assets/ledger/files/theme/web/_mxcli-atlas-map.scss +++ b/cmd/mxcli/theme/assets/ledger/files/theme/web/_mxcli-atlas-map.scss @@ -35,8 +35,12 @@ // brand fill — so it tracks the rail, which is the surface it actually // sits on. Text on a brand-filled button comes from --btn-*-color below. --font-color-contrast: var(--mxt-rail-ink-active); - --link-color: var(--mxt-brand); - --link-hover-color: var(--mxt-brand-hover); + // Link TEXT needs 4.5:1 against the page; a brand used as a button FILL + // only needs 3:1 plus contrast against its own ink. A theme whose brand is + // too light to read as text sets --mxt-link rather than darkening the + // brand everywhere. (mxcli-todo findings #19c) + --link-color: var(--mxt-link, var(--mxt-brand)); + --link-hover-color: var(--mxt-link-hover, var(--mxt-brand-hover)); --border-color-default: var(--mxt-line); --border-radius-s: var(--mxt-radius); @@ -203,3 +207,45 @@ .navbar-brand .widget-language-selector .language-arrow { color: var(--mxt-rail-ink-active, var(--mxt-rail-ink)); } + +// --------------------------------------------------------------------------- +// The login page — the first screen a user sees, and the one Atlas themes least. +// +// It is served from theme/web/login.html, which loads the SAME compiled theme +// CSS ({{themecss}}), so it can be themed here — but Atlas's own rules leave it +// off-brand in two visible ways (mxcli-todo findings #19): +// +// 1. `.loginpage-image` layers a brand-tinted gradient over Atlas's stock +// photograph (`url("./resources/work-do-more.jpeg")`), so a dark app opens +// on a large, bright, unrelated picture. +// 2. The submit button is `.btn-success`, so it follows the SUCCESS colour, +// not the brand — on console that is a green button two screens away from +// a teal one. +// +// The project's own logo (theme/web/logo.png) is deliberately left alone: it is +// the app's asset to replace, and hiding it would remove a real logo from apps +// that have one. +// --------------------------------------------------------------------------- +.loginpage-image { + background: + left / cover no-repeat + linear-gradient( + 160deg, + var(--mxt-brand) 0%, + color-mix(in srgb, var(--mxt-brand) 55%, var(--mxt-rail)) 100% + ); +} + +.loginpage-form .btn-success { + background-color: var(--mxt-brand); + border-color: var(--mxt-brand); + color: var(--mxt-brand-ink); +} + +.loginpage-form .btn-success:hover, +.loginpage-form .btn-success:focus, +.loginpage-form .btn-success:active { + background-color: var(--mxt-brand-hover); + border-color: var(--mxt-brand-hover); + color: var(--mxt-brand-ink); +} diff --git a/cmd/mxcli/theme/assets/signal/files/theme/web/_mxcli-atlas-map.scss b/cmd/mxcli/theme/assets/signal/files/theme/web/_mxcli-atlas-map.scss index 17ed472eb..fdc03a81a 100644 --- a/cmd/mxcli/theme/assets/signal/files/theme/web/_mxcli-atlas-map.scss +++ b/cmd/mxcli/theme/assets/signal/files/theme/web/_mxcli-atlas-map.scss @@ -35,8 +35,12 @@ // brand fill — so it tracks the rail, which is the surface it actually // sits on. Text on a brand-filled button comes from --btn-*-color below. --font-color-contrast: var(--mxt-rail-ink-active); - --link-color: var(--mxt-brand); - --link-hover-color: var(--mxt-brand-hover); + // Link TEXT needs 4.5:1 against the page; a brand used as a button FILL + // only needs 3:1 plus contrast against its own ink. A theme whose brand is + // too light to read as text sets --mxt-link rather than darkening the + // brand everywhere. (mxcli-todo findings #19c) + --link-color: var(--mxt-link, var(--mxt-brand)); + --link-hover-color: var(--mxt-link-hover, var(--mxt-brand-hover)); --border-color-default: var(--mxt-line); --border-radius-s: var(--mxt-radius); @@ -203,3 +207,45 @@ .navbar-brand .widget-language-selector .language-arrow { color: var(--mxt-rail-ink-active, var(--mxt-rail-ink)); } + +// --------------------------------------------------------------------------- +// The login page — the first screen a user sees, and the one Atlas themes least. +// +// It is served from theme/web/login.html, which loads the SAME compiled theme +// CSS ({{themecss}}), so it can be themed here — but Atlas's own rules leave it +// off-brand in two visible ways (mxcli-todo findings #19): +// +// 1. `.loginpage-image` layers a brand-tinted gradient over Atlas's stock +// photograph (`url("./resources/work-do-more.jpeg")`), so a dark app opens +// on a large, bright, unrelated picture. +// 2. The submit button is `.btn-success`, so it follows the SUCCESS colour, +// not the brand — on console that is a green button two screens away from +// a teal one. +// +// The project's own logo (theme/web/logo.png) is deliberately left alone: it is +// the app's asset to replace, and hiding it would remove a real logo from apps +// that have one. +// --------------------------------------------------------------------------- +.loginpage-image { + background: + left / cover no-repeat + linear-gradient( + 160deg, + var(--mxt-brand) 0%, + color-mix(in srgb, var(--mxt-brand) 55%, var(--mxt-rail)) 100% + ); +} + +.loginpage-form .btn-success { + background-color: var(--mxt-brand); + border-color: var(--mxt-brand); + color: var(--mxt-brand-ink); +} + +.loginpage-form .btn-success:hover, +.loginpage-form .btn-success:focus, +.loginpage-form .btn-success:active { + background-color: var(--mxt-brand-hover); + border-color: var(--mxt-brand-hover); + color: var(--mxt-brand-ink); +} diff --git a/docs-site/src/ide/init-output.md b/docs-site/src/ide/init-output.md index 513bbde24..d26baaa54 100644 --- a/docs-site/src/ide/init-output.md +++ b/docs-site/src/ide/init-output.md @@ -42,10 +42,29 @@ your-mendix-project/ ├── settings.json # Claude Code project settings ├── commands/ # Slash commands for Claude │ └── mendix/ # Mendix-specific commands -└── lint-rules/ # Starlark lint rules +├── lint-rules/ # Starlark lint rules +└── lint-config.yaml # Lint settings (excluded modules, rule overrides) CLAUDE.md # Project context for Claude ``` +#### `lint-config.yaml` + +Seeded with the **System module excluded**. System is Mendix's own platform +module — you cannot document its entities, give them access rules, or rename +their members — so linting it produces findings you can never action. On a +blank app that was the large majority of all issues. + +`mxcli init` writes this file only when the project has no lint config yet +(`.claude/lint-config.yaml`, `lint-config.yaml`, or `.lint-config.yaml`), so +re-running init never discards your edits. + +Add Marketplace modules (`Atlas_Core`, `Administration`, …) to `excludeModules` +if their findings distract you — they are equally read-only in practice. + +To lint System after all, remove it from `excludeModules`. Note the list always +wins: it merges with `--exclude`, and a module listed there stays excluded even +if you name it with `--modules` (lint warns when you try). + ### Cursor ``` diff --git a/docs-site/src/migration/validation.md b/docs-site/src/migration/validation.md index 2cec7163b..2df058ccf 100644 --- a/docs-site/src/migration/validation.md +++ b/docs-site/src/migration/validation.md @@ -71,7 +71,7 @@ CALL MICROFLOW Sales.ACT_Order_CalculateTotal ($Order = $Order); ``` ```bash -# Run tests (requires Docker) +# Run tests (Docker, or --local for mxcli's own runtime) mxcli test tests/ -p app.mpr ``` diff --git a/docs-site/src/reference/capabilities.md b/docs-site/src/reference/capabilities.md index 1f6262931..34367323d 100644 --- a/docs-site/src/reference/capabilities.md +++ b/docs-site/src/reference/capabilities.md @@ -126,7 +126,7 @@ Everything mxcli can do, organized by use case. | Stdin piping | `echo "CMD" \| mxcli -p app.mpr` | Quiet mode, no prompts | | Docker build | `mxcli docker build -p app.mpr` | Build MDA in container | | Docker check | `mxcli docker check -p app.mpr` | Validate in container | -| Testing | `mxcli test tests/ -p app.mpr` | `.test.mdl` / `.test.md` | +| Testing | `mxcli test tests/ -p app.mpr [--local] [--watch] [--attach]` | `.test.mdl` / `.test.md`; `--local` needs no Docker, `--watch` keeps the runtime warm (~2s per run), `--attach` runs against an app already up | | SARIF output | `mxcli lint --format sarif` | For CI integration | | New project | `mxcli new --version X.Y.Z` | Create project from scratch with all tooling | | Init project | `mxcli init` | Set up `.claude/` with skills | diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index 5d85e75d1..738cfac3a 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -2,8 +2,15 @@ The **primary** way to start a Mendix + mxcli project from the web or an iPad — no local CLI, no GitHub template to pick from a (short) mobile list. Open an **empty -repo** in Claude Code Web and paste the prompt below; the agent provisions -everything and commits the result so future sessions self-bootstrap. +repo** in Claude Code Web and paste the prompt below; the agent asks you what the app +is, then provisions everything and commits the result so future sessions +self-bootstrap. + +The interview comes first for a reason: the app name becomes the `.mpr` file name, the +Studio Pro app name and the path baked into the SessionStart hook, so it is far cheaper +to ask than to rename afterwards. The rest of the answers are the brief — they get +written into the repo, so the session that resumes after an idle reap knows what it is +building. Why a prompt instead of a GitHub template repo: the mobile "New repository" template dropdown shows only a small subset of templates, and a template repo needs per-Mendix- @@ -13,7 +20,43 @@ can seed the model from a design prototype in the same session — nothing to ma ## The prompt ````text -This is an empty repo. Provision it as a Mendix app developed with mxcli: +This is an empty repo. You are going to provision it as a Mendix app developed with +mxcli — but first find out what the app is. + +## Step 0 — interview me, and WAIT for my answers before running anything + +Ask all of these in ONE message, numbered, each with the default you would pick, so I +can reply "defaults" or answer only the ones I care about. Do not start provisioning +until I have replied. + +1. **One app, or a solution of several?** One Mendix app is the default. Say + "solution" if this is several apps in one repo — e.g. a backend that owns the data + and publishes OData/REST, and a frontend that consumes it. If so, ask for each + app's name and one line on what it owns, and follow the multi-app deltas below. +2. **App name.** Becomes the `.mpr` file name, the app name in Studio Pro, and the + path in the session hook, so it is awkward to change later. One PascalCase word, + letters and digits only — `OrderPortal`, `FieldService`, `ClubAdmin`. Propose one + from my answer to Q3. +3. **What is the app for?** One or two sentences: who uses it, and what it lets them + do. If my answer is vague ("a tool for work"), ask one follow-up — everything below + is derived from this. +4. **What does it keep track of?** Three to six nouns that will become entities, and a + word on how they relate (e.g. "a Job has many Visits; each Visit has Photos"). For + a solution, also ask which app owns each noun. +5. **Who logs in?** The user roles, and roughly what each may do (e.g. "Requester + creates and sees their own; Approver sees everything and approves"). +6. **Look and feel.** One of the bundled themes: `signal` (light, high contrast), + `ledger` (light, dense, data-heavy), `console` (dark), or `none` for stock Atlas. + Default `signal`. +7. **Mendix version.** Default `11.13.0`. + +If I say "defaults" or ignore a question, choose something sensible for it, tell me +what you chose in one line, and keep going — do not block on me twice. + +## Then provision + +Substitute my answers for ``, `` and `` throughout. For a +solution, do steps 2–4 once per app and read "If this is a solution" first. 1. Ensure `mxcli` is available. It should be pre-installed by the environment; if not, download a prebuilt binary for your OS/arch and put it at `./mxcli`, e.g.: @@ -30,28 +73,130 @@ This is an empty repo. Provision it as a Mendix app developed with mxcli: release instead (`.../releases/download/vX.Y.Z/mxcli--`). Note: `go install …@latest` does **not** work — the generated ANTLR parser isn't committed, so use the prebuilt binary (a from-source build needs `make grammar`). -2. Create the app at the repo root: `mxcli new App --version 11.6.3` - (or `mxcli init` if an .mpr already exists). -3. Ensure the Claude tooling is set up: `mxcli init --tool claude`. This adds a - SessionStart hook to `.claude/settings.json` that self-bootstraps future sessions. -4. Bring prerequisites up: `./mxcli run --local --setup --ensure-db -p App.mpr` +2. Create the app, and put it at the **repo root** — that is where `.claude/` and the + `./mxcli` binary have to live for future sessions to self-bootstrap. `mxcli new` + refuses to write into a directory that is not empty, and a git repo always has + `.git`, so create it in a subfolder and move it up: + + ```bash + ./mxcli new --version --theme + rm -f /mxcli # a hardlink to the ./mxcli you just ran; mv would + # refuse it as "the same file" + shopt -s dotglob && mv /* . && rmdir + ``` + + (Use `mxcli init` instead if an `.mpr` already exists.) `mxcli new` also runs + `mxcli init`, which writes `.claude/settings.json` with a SessionStart hook plus + the `.claude/bootstrap-mxcli.sh` it runs — check that the `.mpr` named in the + script is right after the move. +3. Confirm the Claude tooling: `./mxcli init --tool claude` (idempotent — it is what + step 2 already ran, and re-running it is the cheapest way to be sure the hook, + skills and commands are in place). +4. Bring prerequisites up: `./mxcli run --local --setup --ensure-db -p .mpr` (caches MxBuild + runtime, starts Postgres, creates the app database). -5. Create a `FINDINGS.md` at the repo root and keep appending to it as you work. +5. Write the brief to `README.md` at the repo root: the app name(s), my answers to + Q3–Q5 in my words, and the theme and Mendix version you used. For a solution, say + which app owns what and how they talk to each other. This is what tells the next + session — after an idle reap, with none of this conversation — what it is building. + Keep it short enough that it stays true. +6. Create a `FINDINGS.md` at the repo root and keep appending to it as you work. Log anything surprising or broken: an mxcli command that errored, a workaround you applied, a `mxcli check` that passed but a real `mx check` later flagged. Note the Mendix + mxcli versions and how each finding was verified. This is durable context for the next session, and the most useful thing to share back to improve mxcli. -6. COMMIT everything now — `App.mpr`, `.devcontainer/`, `.claude/` (including the - SessionStart hook), and `FINDINGS.md` — so that after idle reaping the next session - bootstraps from files, not from re-running this prompt. -7. Boot and verify: `./mxcli run --local -p App.mpr` in the background, then confirm - the app answers HTTP 200 at http://localhost:8080/ and report. -8. (Optional) For a browser preview from this cloud session, run - `./mxcli run --hub https://hub.mxcli.org -p App.mpr` and report the preview URL it - prints. This needs `MXCLI_HUB_KEY` set on the environment (see the workflow page); - without it, continue as a normal local run. - -(Optional) Seed the domain model, pages, and microflows from this prototype: . +7. COMMIT everything now — `.mpr`, `.devcontainer/`, `.claude/` (the + SessionStart hook **and** `.claude/bootstrap-mxcli.sh`), `README.md` and + `FINDINGS.md` — so that after idle reaping the next session bootstraps from files, + not from re-running this prompt. The `mxcli` binary itself stays git-ignored (~85 + MB); the bootstrap script is what fetches it back into a fresh clone, so committing + the script is what makes the hook survive a reap. +8. Boot and verify: `./mxcli run --local -p .mpr` in the background, then + confirm the app answers HTTP 200 at http://localhost:8080/ and report. +9. (Optional) For a browser preview from this cloud session, run + `./mxcli run --hub https://hub.mxcli.org -p .mpr` and report the preview + URL it prints. This needs `MXCLI_HUB_KEY` set on the environment (see the workflow + page); without it, continue as a normal local run. + +## If this is a solution (several apps in one repo) + +Each app is a full Mendix project — one `.mpr`, one runtime, one database. Same steps, +with these deltas: + +- **Layout.** One subfolder per app, nothing at the repo root but `README.md`, + `FINDINGS.md` and `.claude/`. Run `mxcli new --version --theme + ` once per app and leave each where it lands; do not move anything up. +- **Ports.** Every app defaults to 8080/8090/6543 and they will collide. Give the + first app the defaults and the second `--app-port 8180 --admin-port 8190 + --serve-port 6643`. Avoid 8081/8091/6544 — `mxcli test --local` uses those. +- **Give each app its own hostname**, not just its own port. Cookies are keyed on + host name and **ignore the port**, so two apps on `localhost:8080` and + `localhost:8180` share one cookie jar: logging into one can silently replace the + other's `XASSESSIONID`. Two hostnames give two jars, and the differing ports do no + harm. Add them to `/etc/hosts` — + + ``` + 127.0.0.1 backend.local frontend.local + ``` + + — and browse `http://backend.local:8080/` and `http://frontend.local:8180/`. The + runtime binds `127.0.0.1` and serves any `Host` you send it, and the client uses + relative URLs, so it works under any name that resolves to loopback. (`*.nip.io` + works too if you would rather not touch `/etc/hosts`; prefer `/etc/hosts` in a + locked-down container, where public wildcard DNS may not resolve — `localtest.me` + resolves to `::1` in some of them.) + + Then record the name in each app's own configuration, so the runtime knows the URL + it is reached at and generates absolute URLs — OIDC/SAML redirect URIs, deep links + — against the host name rather than the listen address: + + ```sql + alter settings configuration 'Default' + ApplicationRootUrl = 'http://backend.local:8080/'; + ``` + + `run --local` picks that up at boot and prints which configuration it came from. + A blank app ships `http://localhost:8080/` there, and that stock loopback value is + deliberately ignored — otherwise every project would start advertising a URL, and + the wrong port under `--app-port`. Only a real host name is passed through. +- **Databases** need no action: the name is derived from the `.mpr` file name, so + differently-named apps get different databases. +- **The session hook.** `mxcli init` writes `.claude/settings.json` inside each app + folder, but Claude Code reads the one at the **repo root** — and it will not add a + second entry for you (it dedupes on the command, not on the project). Write the root + one yourself, one line per app, e.g. + `test -x backend/mxcli && (cd backend && ./mxcli run --local --setup --ensure-db -p Backend.mpr) || true`. + Verify it by checking that a fresh shell can boot each app. +- **Previews.** Pass `--hub-solution ` to every `run --hub` so the apps + appear grouped in the hub overview instead of as unrelated previews. + +**Wire the integration in dependency order — the producer must be running first.** +`CREATE ODATA CLIENT` fetches the `$metadata` at the moment you create it and caches +it in the model; if the URL is unreachable it warns and leaves the client unvalidated, +with no external entities to import. So: publish on the producer +(`CREATE ODATA SERVICE … publish entity …`), boot it (`run --local`), and only then, +on the consumer, `CREATE ODATA CLIENT … MetadataUrl: 'http://backend.local:8080/odata/…/$metadata'` +followed by `CREATE EXTERNAL ENTITIES FROM …`. Use the hostname here too, so the +cached contract and the constant below agree with what the browser sees. Point `ServiceUrl` at a **constant** +(`ServiceUrl: @Module.SvcUrl`) so the address can be changed per environment without +touching the model — it will not stay `localhost`. `mxcli syntax odata.publish` and +`mxcli syntax odata.consume` have the full syntax; business events +(`mxcli syntax business-events`) are the alternative when the link should be +asynchronous. + +## Then propose the model — do not build it yet + +The blank template ships a `MyFirstModule`; the app's own work belongs in a module +named after it. From the brief, propose in chat: + +- a module name, and the entities from Q4 with their attributes and associations +- the user roles from Q5 and what each may read/write +- the handful of pages that make it usable +- for a solution: which app owns each entity, and what crosses the boundary — publish + only what the other app actually needs + +Show me that as MDL I can read, and wait for my go-ahead before executing it. If I +gave you a design to work from, use it as the source of truth for the model and the +pages: . ```` ## Which mxcli version gets installed @@ -80,13 +225,29 @@ tag (latest is v0.16.0) **and** as a rolling `nightly` pre-release, with assets `make grammar` first). Enabling `go install` would require committing the generated parser (or generating it during module build) — a maintainer decision. +## Which Mendix version to ask for + +The prompt defaults to the newest version that has a published MxBuild — everything +mxcli does starts with downloading it, so "supported" means "on the CDN". Check before +bumping the default: + +```bash +curl -sI -o /dev/null -w '%{http_code}\n' https://cdn.mendix.com/runtime/mxbuild-11.13.0.tar.gz # 200 +curl -sI -o /dev/null -w '%{http_code}\n' https://cdn.mendix.com/runtime/mendix-11.13.0.tar.gz # 200 (runtime) +``` + +Both have to answer `200` — `run --local` needs the runtime tarball as well as +MxBuild. In a solution, give every app the **same** version: they share the +`~/.mxcli/mxbuild` cache, and a mismatch means a second multi-hundred-MB download and +two runtimes to keep straight. + ## Two rules that make this robust -- **Committing the config (step 5) is mandatory.** The prompt is a *one-time seed*. - Its output — `.mpr` + `.devcontainer/` + `.claude/` with the SessionStart hook — must - be committed so the steady state is file-driven and deterministic. After that, every - new session runs the hook (`run --local --setup --ensure-db`) automatically; you - never re-paste the prompt. +- **Committing the config (step 7) is mandatory.** The prompt is a *one-time seed*. + Its output — `.mpr` + `.devcontainer/` + `.claude/` with the SessionStart hook and + `bootstrap-mxcli.sh` — must be committed so the steady state is file-driven and + deterministic. After that, every new session runs the hook automatically; you never + re-paste the prompt. Miss the script and the hook has nothing to run after a reap. - **mxcli delivery is an environment concern, not the prompt's.** Step 1 is the fragile part in a gated web session (a GitHub release `curl` may be blocked). The robust fix is for the Claude Code Web **environment image / setup script to pre-install mxcli** @@ -96,9 +257,23 @@ tag (latest is v0.16.0) **and** as a rolling `nightly` pre-release, with assets ## After bootstrap — the inner loop ```bash -./mxcli run --local -p App.mpr --watch --screenshot # warm dev loop + screenshots -./mxcli exec change.mdl -p App.mpr # edit the model; the loop hot-applies +./mxcli run --local -p .mpr --watch --screenshot # warm dev loop + screenshots +./mxcli exec change.mdl -p .mpr # edit the model; the loop hot-applies +``` + +In a solution, run one loop per app from its own folder, with the second app on the +alternate ports, and start the producer first so the consumer's external entities +resolve: + +```bash +(cd backend && ./mxcli run --local -p Backend.mpr --watch) +(cd frontend && ./mxcli run --local -p Frontend.mpr --watch \ + --app-port 8180 --admin-port 8190 --serve-port 6643) ``` +With `127.0.0.1 backend.local frontend.local` in `/etc/hosts`, browse them at +`http://backend.local:8080/` and `http://frontend.local:8180/` so each app gets its +own cookie jar. + See [mxcli run --local](run-local.md) for the warm loop, `--watch`, `--ensure-db`, and the screenshot flags. diff --git a/docs-site/src/tools/run-local.md b/docs-site/src/tools/run-local.md index 43649d900..d5de0f008 100644 --- a/docs-site/src/tools/run-local.md +++ b/docs-site/src/tools/run-local.md @@ -79,6 +79,7 @@ so structural changes need a restart; behavioural changes do not. | `--screenshot-url` | app root | Page to shoot: full URL, or a path relative to the app root (e.g. `/p/customers`). Repeat for a multi-page set. | | `--screenshot-user` / `--screenshot-password` | — | Log in once (Mendix form auth) and reuse the session, so pages behind login render authenticated | | `--runtime-log` | `/.mxcli/runtime.log` | Runtime log file — JVM stdout/stderr **and** the application log (server stack traces + microflow `LOG` output); `-` disables | +| `--test-endpoint` | off | Host mxcli's token-guarded test endpoint so [`mxcli test … --attach`](running-tests.md) runs a suite against this app with no boot of its own. Installed before the boot (the handler registers from after-startup, so it cannot be added to a running app); your project's own after-startup microflow is chained, not displaced. Removed on exit. Tests then use **this app's database** | | `--debug` | off | Enable the microflow debugger at boot; then use [`mxcli debug`](debug-microflows.md) from another terminal. Behaviour-neutral until a breakpoint is set | | `--debug-pass` | `mxdebug` | Debugger password when `--debug` is set | | `--metrics` | off | Register a Prometheus meter registry; metrics served at `http://127.0.0.1:/prometheus` | diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index 14eeb1a4f..4d45072d2 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -4,24 +4,27 @@ The `mxcli test` command executes test files and reports results. ## Prerequisites -Running tests requires **Docker** for Mendix runtime validation. The test runner uses: +Tests need a Mendix runtime to execute against. There are two ways to get one, +and **Docker is only needed for the first**: -- `mx create-project` to create a fresh blank Mendix project -- `mx check` to validate the project after applying MDL changes +- **Docker** — the container path. Requires a running Docker daemon. +- **`--local`** — mxcli's own runtime, no daemon involved. It uses its own ports + (8081/8091) and its own `_test` database, so a `mxcli run --local` + dev loop can keep serving the same project while tests run. -The `mx` binary is located at: +`--local` also downloads what it needs on first use. To pre-cache it: + +```bash +mxcli setup mxbuild -p app.mpr +``` + +The `mx` binary, when you need it directly: | Environment | Path | |-------------|------| | Dev container | `~/.mxcli/mxbuild/{version}/modeler/mx` | | Repository | `reference/mxbuild/modeler/mx` | -To auto-download mxbuild for the project's Mendix version: - -```bash -mxcli setup mxbuild -p app.mpr -``` - ## Basic Usage ```bash @@ -35,12 +38,61 @@ mxcli test tests/sales.test.mdl -p app.mpr mxcli test tests/integration.test.md -p app.mpr ``` -## Test Execution Flow +## Choosing a mode + +| | Boot cost per run | Database | Needs Docker | +|---|---|---|---| +| `mxcli test …` | container restart | the container's | yes | +| `--local` | ~30s | `_test` | no | +| `--local --watch` | ~30s once, then ~2s | `_test` | no | +| `--attach` | none | **the running app's** | no | + +```bash +# No Docker daemon needed +mxcli test tests/ -p app.mpr --local + +# Keep the runtime warm; re-runs on every test or model change (Ctrl-C to stop) +mxcli test tests/ -p app.mpr --local --watch + +# Attach to an app you already have running — no boot at all +mxcli run --local --test-endpoint -p app.mpr # terminal 1 +mxcli test tests/ -p app.mpr --attach # terminal 2 +``` + +`--watch` is the everyday loop: edit a test *or* the microflow under test, and +the verdict lands in about two seconds. + +`--attach` skips even the first boot, at one cost worth knowing: the tests run +against the running app's database rather than a scratch one, so they can leave +data behind in the app you are looking at. It needs the dev loop to have been +started with `--test-endpoint`, because the endpoint's handler is registered by +the after-startup microflow and cannot be added to an app that is already up. + +## How tests execute + +**`--local` — the test endpoint.** One microflow is generated per test, plus a +Java action that registers a token-guarded HTTP endpoint. The app boots once; +startup only registers the endpoint and runs no tests. Each test is then invoked +by name over HTTP and returns its verdict in the response. + +Two consequences when reading a failing run: + +- A test that throws fails **only itself** and is reported as an error with the + root-cause message; the next test still runs. +- Results are **returned**, not recovered from the runtime log. + +The endpoint executes microflows under a system context, so it is gated: it is +not registered at all without a per-run token in the runtime's environment, +every request must present that token, non-loopback callers are refused, and it +will only ever invoke the generated `MxTest.Test_*` microflows. The token is +never written into your project. + +**Docker — the after-startup runner.** The whole suite is compiled into the +project's after-startup microflow, the container is restarted, and results are +parsed out of its log. `--legacy-runner` selects this on a local run too. -1. **Create project** -- A fresh Mendix project is created in a temporary directory using `mx create-project` -2. **Execute MDL** -- The test script is executed against the fresh project using `mxcli exec` -3. **Validate** -- `mx check` validates the resulting project for errors -4. **Report** -- Results are reported with pass/fail status per test case +Both paths restore the project when they finish, and both report loudly if that +restore fails — a modified project must never read as a clean pass. ## Isolated Testing Pattern diff --git a/docs-site/src/tools/testing.md b/docs-site/src/tools/testing.md index 338ff805b..65772d97a 100644 --- a/docs-site/src/tools/testing.md +++ b/docs-site/src/tools/testing.md @@ -13,7 +13,7 @@ The testing framework supports two test file formats: ## Prerequisites -Running tests requires **Docker** for Mendix runtime validation. The test runner: +Tests execute against a real Mendix runtime. **Docker is one way to get one, not the only one** — `--local` uses mxcli's own runtime with no daemon, and is the faster path (see [Running Tests](running-tests.md) for the mode comparison and the `--watch` / `--attach` loops). On the Docker path the test runner: 1. Creates a fresh Mendix project using `mx create-project` 2. Executes the MDL test script against the project diff --git a/docs-site/src/tutorial/installation.md b/docs-site/src/tutorial/installation.md index 25657bfdf..402d3974c 100644 --- a/docs-site/src/tutorial/installation.md +++ b/docs-site/src/tutorial/installation.md @@ -75,14 +75,17 @@ This single command: 2. Creates a blank Mendix project (`App.mpr`) 3. Sets up AI tooling (`.claude/`, skills, `AGENTS.md`) 4. Configures a Dev Container (`.devcontainer/`) -5. Downloads the correct Linux mxcli binary for the container +5. Runs one build, so the action stubs MxBuild regenerates are already settled +6. Puts an mxcli binary in the project — a hard link to the one you ran on Linux, + a downloaded Linux build on macOS/Windows (the container needs a Linux ELF) Open the resulting `MyApp/` folder in VS Code and click **"Reopen in Container"** — you're ready to go. Options: ```bash mxcli new MyApp --version 10.24.0 --output-dir ./projects/my-app -mxcli new MyApp --version 11.8.0 --skip-init # Skip AI tooling setup +mxcli new MyApp --version 11.8.0 --skip-init # Skip AI tooling setup +mxcli new MyApp --version 11.8.0 --skip-build # Skip the first build ``` ## Dev Container for existing projects diff --git a/docs/15-testing/SPIKE_test_endpoint_request_handler.md b/docs/15-testing/SPIKE_test_endpoint_request_handler.md new file mode 100644 index 000000000..c54ce9140 --- /dev/null +++ b/docs/15-testing/SPIKE_test_endpoint_request_handler.md @@ -0,0 +1,228 @@ +# Spike: custom request handler as a re-invokable test entry point + +**Status**: **shipped** for `mxcli test --local` — see `cmd/mxcli/testrunner/endpoint.go`. +Measured end-to-end on Mendix **11.13.0**. +**Reproduce the bare spike with**: [`mdl-examples/spikes/test-endpoint-request-handler.mdl`](../../mdl-examples/spikes/test-endpoint-request-handler.mdl) + +> The "open issues" at the bottom were what stood between the spike and the +> implementation. The token gate, the loopback check, the test-namespace +> restriction and the javasource cleanup are all now in `endpoint.go`; the +> remaining unresolved items are called out as still open. + +## Question + +`mxcli test --local` triggers tests from the project's **after-startup microflow**. +That is a boot hook, so re-running a suite requires a full runtime restart by +construction. Can a Java custom request handler give us a *re-invokable* entry +point instead, so a re-run costs an HTTP round trip rather than a restart? + +## Answer + +Yes, and it is better than the `check_health` idea it replaces — because +`Core.getMicroflowNames()` lets the handler resolve microflows **by name at +request time**. The Java is written once and never regenerated; tests can be +added, edited, and removed with no change to it. + +## What was built + +A single Java action, `MxTest.RegisterTestEndpoint`, authored inline from MDL +(`CREATE JAVA ACTION … AS $$ … $$`), registering one handler: + +```java +Core.addRequestHandler("mxtest/", new RequestHandler() { … }); +``` + +It is called once from a two-line after-startup microflow. After-startup is +still used — but only to *register the endpoint*, never to run tests. + +| Route | Behaviour | +|---|---| +| `GET /mxtest/list?prefix=` | Test discovery from `Core.getMicroflowNames()` | +| `GET /mxtest/run?mf=Module.Flow` | `Core.microflowCall(mf).execute(Core.createSystemContext())` | + +Results come back as JSON — return value, wall time, and on failure the +**root-cause** exception message. + +## Measurements + +All on the same machine, same project, Mendix 11.13.0, Postgres local. + +| Operation | Time | +|---|---| +| Cold boot to first test invocable (**today's cost per re-run**) | **30.55s** | +| Re-run whole suite, no model change (4 tests, 4 HTTP calls) | **0.084s** | +| Edit a test → watcher rebuild → hot reload → new result over HTTP | **4.29s** | +| Single test invocation, in-process | 0.6–26ms | + +So an unchanged re-run goes from ~30s to **0.08s (~360×)**, and an +edit-then-re-run from ~30s to **~4.3s (~7×)** — the 4.3s being almost entirely +the existing `--watch` rebuild, not the test mechanism. + +## The load-bearing finding: the handler survives `reload_model` + +This is what makes the warm loop actually work, and it was not obvious. + +- After-startup does **not** re-run on `reload_model`. Verified by grepping the + runtime log: exactly two `MxTest request handler registered` lines across two + *boots*, and none after the reload. +- The runtime **JVM PID is unchanged** across the reload (31312 before and + after), and `mxcli run --watch` reported `build #2 applied via reload`, not a + restart. +- The handler object registered by the *old* model resolves the *new* model's + microflows correctly. Proven with two probes in one reload: an **edited** + test returned its new value (`2+2=4` → `40+2=42`), and a test **created after + boot** appeared in `/mxtest/list` and ran — with no restart. + +## Consequences for the test runner + +Beyond the speed, three structural problems in `cmd/mxcli/testrunner/` dissolve: + +1. **The monolithic runner microflow goes away.** `generator.go` currently + compiles every test into one microflow, regex-renaming variables with `_N` + suffixes to avoid collisions. With per-name invocation each test is its own + microflow, so `--filter` and single-test runs are free and one throwing test + can no longer end the run. +2. **Results stop being scraped from JVM stdout.** They are the HTTP response + body. `results.go`'s log parsing is no longer on the critical path. +3. **A failing test stops being a failed boot.** Verified: a test throwing + `MendixRuntimeException` returns HTTP 200 with `ok:false` and the root-cause + message, and the runtime stays up and serves the next test. Compare + `runner_local.go:58-71`, which has to special-case boot failure today. + +## Issues found by the spike, and how they were resolved + +- **The endpoint was unauthenticated.** Verified in the spike: `curl` with no + cookies and no session executed a microflow. **Resolved** — four gates, each + verified against a live 11.13.0 runtime: + + | Guard | Verified behaviour | + |---|---| + | No `MXCLI_TEST_TOKEN` in the environment | Handler not registered; `/mxtest/list` → 404 | + | Missing / wrong `X-MxTest-Token` | 401 (constant-time compare) | + | Non-loopback caller | 403 | + | `mf` outside `MxTest.Test_*` | 403 | + + The token is generated per run and passed through the runtime's **environment** + (`LocalRuntimeOptions.Env`), never written into the project — so a failed + cleanup cannot leave a live credential in `javasource/`. + +- **`/list` disclosed the whole app.** Found only by probing the live runtime: + with no `prefix` the handler returned every microflow in the app, + `Administration.*` included. **Resolved** — the prefix is clamped to + `MxTest.Test_`; a caller-supplied prefix can only narrow further. The endpoint + will not run those microflows, so it must not enumerate them either. + +- **Path dispatch was loose** — anything that was not `list` was treated as + `run`. **Resolved**: exact matches, anything else 404. + +- **`DROP JAVA ACTION` leaves the `.java` file behind.** The model document goes; + the generated source in `javasource/mxtest/` is not the model's to delete. + **Resolved** — `removeGeneratedJavaSource` removes it, non-fatally. + +### Still open + +- **There is no `Core.removeRequestHandler`.** The API only offers + `addRequestHandler`, so the handler cannot be unregistered for the life of the + JVM. This is why registration is gated rather than reversed. Re-registering + the same path is still unexercised — after-startup does not re-run on + `reload_model`, so nothing in the current design hits it. +- **Test parameters and setup/teardown.** Only no-argument microflows are + invoked. `MicroflowCallBuilder.withParams(Map)` exists, and + `inTransaction(boolean)` looks directly relevant to rolling back a test's + database writes — the `@cleanup rollback` annotation is parsed but not yet + honoured by either mechanism. +- **The Docker path still uses after-startup.** The endpoint needs to hand the + runtime a secret through its environment and to be reached on loopback, + neither of which is wired through docker-compose. No Docker daemon was + available to verify a change there, so it was left alone rather than shipped + untested. +- ~~`--attach` to an already-running `run --local`~~ — **shipped**, see below. + +## The warm loop, realised (`--watch`) + +`mxcli test --local --watch` keeps the runtime and the build server up and +re-runs the suite on every change to a test file or to the model. Measured on +the same 11.13.0 app: + +| | | +|---|---| +| First run (cold boot) | ~30s | +| Edit a test → verdict on screen | **~2.0s** | +| Edit a microflow under test → verdict on screen | **~2.1s** | +| The tests themselves | 20–70ms | + +Every re-run in the session applied via **reload**, not restart — the property +the spike established. Verified live across a session that edited a test, +deleted a test, added a test, and changed the microflow under test. + +Two hazards this loop has that the `run --local` dev loop does not: + +1. **The runner writes to the project it is watching.** Injecting the test + microflows moves the very mtime being polled, so the baseline is taken after + the injection and rebuild settle. Getting it wrong is an infinite rebuild + loop; verified by idling a session and confirming the run counter does not + advance. +2. **The injected set changes during the session.** Cleanup drops what is + *currently* injected, not what was injected at boot — otherwise a test added + mid-session is left in the user's project. A deleted test's microflow is + dropped explicitly, since `CREATE OR REPLACE` says nothing about removal and + a lingering flow would keep reporting a stale pass. + +## `--attach`: no boot at all + +`mxcli test --attach` runs against an app already up, skipping the boot entirely. +Measured on the same 11.13.0 app: **2.83s** for the first attached run and +**2.30s** for a repeat, against ~30s cold — with the dev app still serving +throughout. + +### Why it has to be cooperative + +The obvious reading of "attach to a running app" does not work, and the reason is +worth recording because it constrains the design completely: + +- The handler is registered by the **after-startup microflow**, which runs only + at boot. It cannot be added to an app that is already up. +- Its token comes from the **runtime's environment**, which a second process + cannot change either. + +So the app has to opt in *before* it boots: `mxcli run --local --test-endpoint`. +That is also the right place for the decision, since hosting the endpoint means +the developer's own app carries a microflow-executing endpoint and tests will +write to the database they are looking at. + +What a second process *can* do, and does, is drive the dev loop's **serve server +and admin API** over loopback — both are plain HTTP. So an attach applies its own +injections deterministically instead of waiting to see whether someone else's +`--watch` noticed; `--attach` does not require the dev loop to be watching. + +### The handshake + +`run --local --test-endpoint` publishes `/.mxcli/test-endpoint.json` +(mode 0600, written-then-renamed) carrying the app/admin/serve ports, the +endpoint token, the admin password, and its own PID. `--attach` reads it and +refuses a stale one by checking the PID — a dev loop killed with SIGKILL leaves +the file behind, and without the check that surfaces much later as a confusing +connection error. + +The project's own after-startup microflow is **chained**, not displaced, so the +dev app still seeds its data and does whatever else it does at boot. + +### Two bugs the live run caught that review had not + +1. **The admin API and the endpoint use different secrets.** The first attempt + passed the endpoint token to the M2EE admin API, which failed with + `Authentication failed` — *after* the test microflows had already been + injected. The handshake now carries the admin password separately. +2. **`DROP JAVA ACTION` leaves the generated `.java` behind** (found earlier, in + the same family): the model document is the model's, the source file is not. + +### Ownership boundary + +An attach adds and removes only its own test microflows. The endpoint, the +after-startup setting and the `MxTest` module belong to the hosting dev loop and +are removed when *it* exits. Verified live: after an attached run the test +microflows were gone, `MxTest.RegisterEndpoint` was still installed, and the app +was still serving HTTP 200. + +A change needing a runtime restart (a new entity or association) is refused +rather than half-applied — that runtime belongs to the other process. diff --git a/mdl-examples/bug-tests/850-download-file-action.mdl b/mdl-examples/bug-tests/850-download-file-action.mdl new file mode 100644 index 000000000..73931c3f3 --- /dev/null +++ b/mdl-examples/bug-tests/850-download-file-action.mdl @@ -0,0 +1,49 @@ +-- ============================================================================ +-- Issue #850 — `download file` parsed but never written to the project +-- ============================================================================ +-- +-- `microflowActionToGen` (mdl/backend/modelsdk/microflow_write.go) switches on +-- the semantic action type and had no `*microflows.DownloadFileAction` case, so +-- the action fell through to `default: return nil` and the enclosing +-- ActionActivity was written with no Action at all. +-- +-- Everything either side of the writer was already in place — grammar, visitor, +-- flow builder, read path and DESCRIBE formatter — so the statement was accepted +-- at every stage that reports anything and vanished at the one stage that does +-- not. `mxcli check` passed, `mxcli exec` printed "Created microflow", and only +-- `mx check` noticed: +-- +-- [error] [CE0008] "No action defined." at Action activity 'Activity' +-- +-- To verify: +-- 1. Run this script. +-- 2. `mxcli describe microflow Bug850.ACT_Download` — the body must render +-- `download file $Doc;`, NOT `-- Empty action`. Likewise +-- `Bug850.ACT_DownloadInBrowser` must render `… show in browser;`. +-- 3. mx check should report 0 errors (CE0008 before the fix). +-- +-- Note the storage key is ShowFileInBrowser, not ShowInBrowser. +-- ============================================================================ + +CREATE MODULE Bug850; + +CREATE OR MODIFY PERSISTENT ENTITY Bug850.Doc EXTENDS System.FileDocument ( +); + +CREATE OR MODIFY MICROFLOW Bug850.ACT_Download ( + $Doc: Bug850.Doc +) +RETURNS Boolean +BEGIN + download file $Doc; + return true; +END; + +CREATE OR MODIFY MICROFLOW Bug850.ACT_DownloadInBrowser ( + $Doc: Bug850.Doc +) +RETURNS Boolean +BEGIN + download file $Doc show in browser; + return true; +END; diff --git a/mdl-examples/bug-tests/851-alter-page-conditional-attribute.mdl b/mdl-examples/bug-tests/851-alter-page-conditional-attribute.mdl new file mode 100644 index 000000000..b746ce019 --- /dev/null +++ b/mdl-examples/bug-tests/851-alter-page-conditional-attribute.mdl @@ -0,0 +1,63 @@ +-- ============================================================================ +-- Issue #851 — `alter page … set Editable = [expr]` produced an unopenable project +-- ============================================================================ +-- +-- The ALTER path built the Forms$Conditional{Visibility,Editability}Settings node +-- by hand and wrote `Attribute: null`. Attribute is a BY_NAME AttributeIdentifier, +-- so its unset value is the empty string; a null fails the reader with +-- +-- StorageLoadException: Conditional editability settings has an invalid value '' +-- for property Attribute +-- +-- The CREATE path already encoded it as "" (via the TypeDefaults in +-- mdl/backend/modelsdk/widget_write.go), which is why a widget authored with +-- CREATE loaded and the same widget authored with ALTER did not. Neither +-- `mxcli check` nor `mx check` inspects the stored value, so both reported +-- success on the broken project. +-- +-- To verify in Studio Pro: +-- 1. Run this script. +-- 2. Open Bug851.PageAltered — it must open without a StorageLoadException, +-- and txtAltered must show "Editable: conditional" with the expression. +-- 3. Compare against Bug851.PageCreated (same settings via CREATE): the two +-- widgets' conditional settings must be structurally identical. +-- 4. mx check should report 0 errors. +-- ============================================================================ + +CREATE MODULE Bug851; + +CREATE OR MODIFY PERSISTENT ENTITY Bug851.Thing ( + Slug: String(200) +); + +-- Reference: the CREATE path, which was already correct. +CREATE OR REPLACE PAGE Bug851.PageCreated ( + Params: { $currentObject: Bug851.Thing }, + Title: 'Conditional settings via CREATE', + Layout: Atlas_Core.Atlas_Default +) { + dataview dv (datasource: $currentObject) { + textbox txtCreated ( + label: 'Slug', + attribute: Slug, + editable: [$currentObject/Slug != ''], + visible: [$currentObject/Slug != ''] + ) + } +} + +-- The regression: the same two settings applied through ALTER. +CREATE OR REPLACE PAGE Bug851.PageAltered ( + Params: { $currentObject: Bug851.Thing }, + Title: 'Conditional settings via ALTER', + Layout: Atlas_Core.Atlas_Default +) { + dataview dv (datasource: $currentObject) { + textbox txtAltered (label: 'Slug', attribute: Slug) + } +} + +ALTER PAGE Bug851.PageAltered { + set Editable = [$currentObject/Slug != ''] on txtAltered; + set Visible = [$currentObject/Slug != ''] on txtAltered; +}; diff --git a/mdl-examples/bug-tests/852-conditional-keyword-functions.mdl b/mdl-examples/bug-tests/852-conditional-keyword-functions.mdl new file mode 100644 index 000000000..6e5e9791c --- /dev/null +++ b/mdl-examples/bug-tests/852-conditional-keyword-functions.mdl @@ -0,0 +1,103 @@ +-- ============================================================================ +-- Issue #852 — `trim()` / `length()` in a widget conditional silently dropped it +-- ============================================================================ +-- +-- `xpathFunctionName` admitted only IDENTIFIER, HYPHENATED_ID, NOT, TRUE, FALSE +-- and CONTAINS, so a call to a function whose name is also an MDL lexer keyword +-- never matched `xpathFunctionCall`. The enclosing `Visible: [...]` then failed +-- to parse as an xpathConstraint, fell through to the generic property-value +-- alternative, and the widget's whole conditional property was dropped — with no +-- diagnostic from `mxcli check` or `mx check`. +-- +-- A dropped Visible defaults to "always visible", so the only symptom was a +-- widget that should have been hidden showing up in the running app. +-- +-- toUpperCase / contains were never affected (plain IDENTIFIERs, and CONTAINS was +-- listed) — they are included below so the two classes can be compared directly. +-- +-- To verify: +-- 1. Run this script. +-- 2. `mxcli describe page Bug852.PageConditionals` — EVERY dynamictext must +-- carry its Visible expression. Before the fix, tKeyword* lost theirs while +-- tIdentifier* kept theirs. +-- 3. mx check should report 0 errors. +-- +-- Only Mendix-valid client-expression functions appear below. `count(…)` and +-- `empty(…)` are ALSO keyword tokens and now parse, but they are not client +-- expression functions — `empty` is a literal (`$x != empty`), not a call — so +-- mxbuild rejects them with CE0117. Before this fix they were silently dropped; +-- now they fail loudly at `mx check`, which is the intended direction but is a +-- visible behaviour change for anyone who had written one. +-- ============================================================================ + +CREATE MODULE Bug852; + +CREATE OR MODIFY PERSISTENT ENTITY Bug852.Thing ( + Slug: String(200) +); + +CREATE OR REPLACE PAGE Bug852.PageConditionals ( + Params: { $currentObject: Bug852.Thing }, + Title: 'Conditionals using keyword-named functions', + Layout: Atlas_Core.Atlas_Default +) { + dataview dv (datasource: $currentObject) { + -- Keyword-named functions: these were the dropped ones. + dynamictext tKeywordTrim (content: 'trim', visible: [trim($currentObject/Slug) != '']) + dynamictext tKeywordLength (content: 'length', visible: [length($currentObject/Slug) > 0]) + dynamictext tKeywordFind (content: 'find', visible: [find($currentObject/Slug, 'x') >= 0]) + + -- Identifier-named functions: these always worked. Guard against regressing them. + dynamictext tIdentifierUpper (content: 'upper', visible: [toUpperCase($currentObject/Slug) != '']) + dynamictext tIdentifierContains (content: 'contains', visible: [contains($currentObject/Slug, 'x')]) + + -- A bare attribute inside a keyword-named call is still rooted in the data + -- context, exactly as it is outside one. + textbox txtEditable (label: 'Slug', attribute: Slug, editable: [trim(Slug) != '']) + } +} + +-- ============================================================================ +-- Runtime verification page (issue #852) +-- ============================================================================ +-- +-- Slug is three spaces: non-empty as a raw string, empty once trimmed. That is +-- what makes trim() observable rather than incidental — a widget whose Visible +-- expression was dropped renders, because the Mendix default is "visible". +-- +-- Boot with `mxcli run --local -p .mpr` and open /p/verify852: +-- TRIM_HIDDEN_MARKER must NOT render (trim(' ') = '' -> false) +-- TRIM_VISIBLE_MARKER must render (negation of the above) +-- LEN_HIDDEN_MARKER must NOT render (length(trim(' ')) = 0) +-- NOTRIM_VISIBLE_MARKER must render (control: ' ' != '' is true, so +-- trim() genuinely changes the outcome) +-- PAGE_RENDERED_BEACON must render (proves the page rendered at all) +-- +-- Before the fix all five rendered. `mx check` reported 0 errors either way — +-- a dropped property is still a valid model, which is why this needs a browser. +-- ============================================================================ + +CREATE OR MODIFY NON-PERSISTENT ENTITY Bug852.Probe ( + Slug: String(200) +); + +CREATE OR MODIFY MICROFLOW Bug852.DS_Probe () +RETURNS Bug852.Probe +BEGIN + $p = create Bug852.Probe (Slug = ' '); + return $p; +END; + +CREATE OR REPLACE PAGE Bug852.Verify ( + Title: 'Verify 852', + Layout: Atlas_Core.Atlas_Default, + Url: 'verify852' +) { + dataview dv (datasource: microflow Bug852.DS_Probe) { + dynamictext tTrimHidden (content: 'TRIM_HIDDEN_MARKER', visible: [trim($currentObject/Slug) != '']) + dynamictext tTrimVisible (content: 'TRIM_VISIBLE_MARKER', visible: [trim($currentObject/Slug) = '']) + dynamictext tLenHidden (content: 'LEN_HIDDEN_MARKER', visible: [length(trim($currentObject/Slug)) > 0]) + dynamictext tNoTrimVisible (content: 'NOTRIM_VISIBLE_MARKER', visible: [$currentObject/Slug != '']) + dynamictext tBeacon (content: 'PAGE_RENDERED_BEACON') + } +} diff --git a/mdl-examples/bug-tests/aggregate-after-optional-set.mdl b/mdl-examples/bug-tests/aggregate-after-optional-set.mdl new file mode 100644 index 000000000..3960099bd --- /dev/null +++ b/mdl-examples/bug-tests/aggregate-after-optional-set.mdl @@ -0,0 +1,62 @@ +-- Aggregates must survive SET becoming optional. +-- +-- `$Sum = sum($List.Price)` used to reach the dedicated aggregateListStatement +-- rule. Making SET optional put `$X = ` ahead of it in the grammar, so the +-- statement fell through to the generic SET conversion, which joined the list +-- and the attribute into one name and dropped the per-item expression: +-- +-- [CE0109] "Undefined variable 'ProductList.Price'." +-- [CE0015] "Aggregate function must specify a valid attribute." +-- +-- Every aggregate below must build with 0 errors, in both spellings. + +create entity Bug.Product ( + Name: string(200), + Price: decimal +); + +-- Bare form (no SET keyword) — this is what regressed. +create microflow Bug.AggregatesBare ( + $ProductList: list of Bug.Product +) +returns decimal as $Total +begin + $Count = count($ProductList); + $Total = sum($ProductList.Price); + $Average = average($ProductList.Price); + $Lowest = minimum($ProductList.Price); + $Highest = maximum($ProductList.Price); + -- Aggregate over a value computed per item. + $Tax = sum($ProductList, $currentObject/Price * 0.21); + return $Total; +end; +/ + +-- The SET keyword must produce exactly the same activities. It routes through a +-- different conversion in the visitor, and that one was wrong even before the +-- bare form ever reached it. +create microflow Bug.AggregatesWithSetKeyword ( + $ProductList: list of Bug.Product +) +returns decimal as $Total +begin + set $Count = count($ProductList); + set $Total = sum($ProductList.Price); + set $Average = average($ProductList.Price); + set $Lowest = minimum($ProductList.Price); + set $Highest = maximum($ProductList.Price); + set $Tax = sum($ProductList, $currentObject/Price * 0.21); + return $Total; +end; +/ + +-- A plain value assignment must still be a Change Variable, not an aggregate. +create microflow Bug.PlainAssignment () +returns integer as $N +begin + declare $N integer = 0; + $N = 5; + $N = $N + 1; + return $N; +end; +/ diff --git a/mdl-examples/bug-tests/f1-10.1-odata-service-name.mdl b/mdl-examples/bug-tests/f1-10.1-odata-service-name.mdl new file mode 100644 index 000000000..c52af4ec1 --- /dev/null +++ b/mdl-examples/bug-tests/f1-10.1-odata-service-name.mdl @@ -0,0 +1,55 @@ +-- ============================================================================ +-- mxcli-formula1 finding #10.1: every published OData service failed to build +-- ============================================================================ +-- +-- Symptom (before fix): a service created purely from MDL passed `mxcli check` +-- and then failed the build: +-- +-- [error] [CE0729] "The service name should not be empty." +-- at Published OData service 'ProbeOData.ProbeApi' +-- +-- `Name` (the document name) and `ServiceName` (the name in the OData metadata +-- document) are different properties, and CREATE ODATA SERVICE only set the +-- first. So the failure hit EVERY published service, not just the exotic ones +-- — and it was invisible until a build ran, because check does not resolve it. +-- +-- The consumed path had defaulted the same field to the document name for +-- CE0339 all along; the published path just never got the same line. +-- +-- After fix: ServiceName falls back to the document name. An explicit +-- `ServiceName: 'X'` still wins. `create or modify` on a service written before +-- the fix heals an empty one, so re-running a script repairs the model. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/f1-10.1-odata-service-name.mdl -p app.mpr +-- then a real build must report 0 errors: +-- scripts/mx-check.sh -p app.mpr +-- ============================================================================ + +create module F1SVCNAME; + +create persistent entity F1SVCNAME.Driver ( + -- The published key needs a unique validation rule, or Mendix answers + -- CE6624 "Add a unique validation rule to attribute 'Code' … to be able to + -- use it as the key". + Code: string(10) unique error 'Code must be unique', + Name: string(120) +); + +-- No ServiceName: — the shape that used to fail CE0729. +create odata service F1SVCNAME.DriverApi ( + path: 'odata/f1svcname/', + version: '1.0.0', + ODataVersion: OData4, + namespace: 'F1SVCNAME.Drivers' +) +authentication basic +{ + publish entity F1SVCNAME.Driver as 'Drivers' ( + ReadMode: source + ) + expose ( + Code as 'code' (KEY, Filterable, Sortable), + Name (Filterable, Sortable) + ); +}; diff --git a/mdl-examples/bug-tests/f1-10.4-odata-publish-associations.mdl b/mdl-examples/bug-tests/f1-10.4-odata-publish-associations.mdl new file mode 100644 index 000000000..24d86d6e1 --- /dev/null +++ b/mdl-examples/bug-tests/f1-10.4-odata-publish-associations.mdl @@ -0,0 +1,74 @@ +-- ============================================================================ +-- mxcli-formula1 finding #10.4: PublishAssociations defaulted to unbuildable +-- ============================================================================ +-- +-- Symptom (before fix): a published service failed the build with +-- +-- [error] [CE7375] "Attribute ID for entity 'X' must be published and be the +-- key when associations are exposed as an associated object id." +-- +-- even with NO associations exposed at all. PublishAssociations=false means +-- "expose associations as an associated object id", and Mendix then requires +-- the system ID attribute to be published as the entity key. MDL's +-- `expose (Attr (KEY))` publishes an ordinary attribute, so the old default of +-- false could not build — and for a non-persistable entity nothing could fix +-- it, because publishing the ID of a non-persistable entity is forbidden. +-- +-- Wider than first reported: this was never specific to non-persistable +-- entities. Measured on 11.12.1 with a PERSISTENT entity and a unique key, the +-- identical service builds 0 errors with true and CE7375 with false. +-- +-- After fix: an unspecified PublishAssociations defaults to true (links). An +-- explicit `PublishAssociations: false` is still honoured — and warned about +-- when a published entity is non-persistable, where it can never build. +-- +-- This script is also the reporter's architecture in miniature: a +-- non-persistable entity fed by a read microflow, published over OData v4, with +-- no copy of the data into the database. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/f1-10.4-odata-publish-associations.mdl -p app.mpr +-- then a real build must report 0 errors: +-- scripts/mx-check.sh -p app.mpr +-- ============================================================================ + +create module F1PUBASSOC; + +-- Non-persistable: the rows are produced per request, never stored. +create non-persistent entity F1PUBASSOC.Lap ( + LapKey: string(60), + Driver: string(120), + LapTime: decimal +); + +-- A read microflow backs the published entity set. The $Response parameter is +-- required because the published resource is Countable — Mendix asks the +-- microflow for the count. +CREATE MICROFLOW F1PUBASSOC.Read_Laps ($Response: System.ODataResponse) + RETURNS List of F1PUBASSOC.Lap AS $Laps +BEGIN + $Laps = CREATE LIST OF F1PUBASSOC.Lap; + RETURN $Laps; +END; + +-- No ServiceName:, no PublishAssociations:, no follow-up ALTER — one statement. +create odata service F1PUBASSOC.LapApi ( + path: 'odata/f1laps/', + version: '1.0.0', + ODataVersion: OData4, + namespace: 'F1PUBASSOC.Laps' +) +authentication basic +{ + publish entity F1PUBASSOC.Lap as 'Laps' ( + ReadMode: microflow F1PUBASSOC.Read_Laps, + InsertMode: not_supported, + UpdateMode: not_supported, + DeleteMode: not_supported + ) + expose ( + LapKey as 'lapKey' (KEY, Filterable, Sortable), + Driver (Filterable, Sortable), + LapTime (Sortable) + ); +}; diff --git a/mdl-examples/bug-tests/f1-13-bare-variable-assignment.mdl b/mdl-examples/bug-tests/f1-13-bare-variable-assignment.mdl new file mode 100644 index 000000000..2907f677c --- /dev/null +++ b/mdl-examples/bug-tests/f1-13-bare-variable-assignment.mdl @@ -0,0 +1,41 @@ +-- ============================================================================ +-- mxcli-formula1 finding #13: `$N = 0;` did not parse +-- ============================================================================ +-- +-- Symptom (before fix): +-- +-- $Total = 5; +-- -> no viable alternative at input '$Total=5' +-- +-- while `DECLARE $Total Integer = 0;` worked, and so did every other assignment +-- form in MDL: `$X = HEAD($List)`, `$X = create M.E (...)`, +-- `$X = execute database query ...`. Assignment was only a PREFIX on specific +-- activity statements; a plain value needed the `SET` keyword, and the error +-- named the token rather than the missing keyword — so the rule was not +-- guessable from the message. +-- +-- After fix: `SET` is optional. `$Total = 5;` and `set $Total = 5;` are the same +-- statement and produce the same ChangeVariableAction. The keyword form is +-- unchanged; existing scripts are full of it. +-- +-- Usage: +-- mxcli check mdl-examples/bug-tests/f1-13-bare-variable-assignment.mdl +-- mxcli exec mdl-examples/bug-tests/f1-13-bare-variable-assignment.mdl -p app.mpr +-- then a real build must report 0 errors: +-- scripts/mx-check.sh -p app.mpr +-- ============================================================================ + +create module F1ASSIGN; +create persistent entity F1ASSIGN.Item (Name: string(50), Qty: integer); +create microflow F1ASSIGN.MF ($Item: F1ASSIGN.Item) +begin + declare $Total integer = 0; + $Total = 5; + $Total = -1; + $Total = $Total + 1; + set $Total = 7; + $Item/Qty = 3; + $New = create F1ASSIGN.Item (Name = 'x'); + retrieve $Items from F1ASSIGN.Item; + $First = HEAD($Items); +end; diff --git a/mdl-examples/bug-tests/loop-variable-out-of-scope.fail.mdl b/mdl-examples/bug-tests/loop-variable-out-of-scope.fail.mdl new file mode 100644 index 000000000..10110ad63 --- /dev/null +++ b/mdl-examples/bug-tests/loop-variable-out-of-scope.fail.mdl @@ -0,0 +1,56 @@ +-- ============================================================================ +-- Loop variable used after the loop passed mxcli check (CE0108) +-- ============================================================================ +-- +-- Symptom (before fix): referencing a loop's variable AFTER `end loop;` passed +-- `mxcli check` cleanly, and `mx check` then failed with +-- +-- [error] [CE0108] "Variable 'item' is defined but not in scope at this +-- location." at Change object activity +-- +-- Two flavours, both reproduced against mxbuild 11.12.1: +-- 1. the loop ITERATOR itself ($item below) +-- 2. anything the loop BODY introduces ($Inner below) — a retrieve, a create, +-- a call output +-- +-- Note the sibling rule: MDL052 rejects two loops REUSING an iterator name, +-- because Mendix requires variable names to be unique across the whole +-- microflow (CE0111). Uniqueness and visibility are different things — the name +-- is reserved flow-wide, but it is only readable inside the loop body. +-- +-- After fix: MDL053 rejects both flavours at check time and points at the +-- carry-out idiom (declare before the loop, assign inside, read after). +-- +-- Usage (expected to FAIL check): +-- mxcli check mdl-examples/bug-tests/loop-variable-out-of-scope.fail.mdl +-- ============================================================================ + +create module S40; + +create entity S40.Thing ( + Label : string(200) +); + +-- (1) The loop iterator, used after the loop → MDL053 / CE0108. +create microflow S40.MF_IteratorAfterLoop ( + $Things: list of S40.Thing +) +begin + loop $item in $Things + begin + change $item (Label = 'in loop'); + end loop; + change $item (Label = 'after loop'); +end; + +-- (2) A variable created inside the loop body, used after the loop → same. +create microflow S40.MF_BodyVarAfterLoop ( + $Things: list of S40.Thing +) +begin + loop $t in $Things + begin + $Inner = create S40.Thing (Label = 'x'); + end loop; + change $Inner (Label = 'after loop'); +end; diff --git a/mdl-examples/bug-tests/todo-12-inherited-attribute-on-page.mdl b/mdl-examples/bug-tests/todo-12-inherited-attribute-on-page.mdl new file mode 100644 index 000000000..f79622686 --- /dev/null +++ b/mdl-examples/bug-tests/todo-12-inherited-attribute-on-page.mdl @@ -0,0 +1,68 @@ +-- ============================================================================ +-- mxcli-todo finding #12: a page bound to an INHERITED attribute failed the build +-- ============================================================================ +-- +-- Symptom (before fix): a widget bound to an attribute the context entity +-- inherits rather than declares passed `mxcli check --references` AND `mxcli +-- lint`, and then the real MxBuild rejected the project: +-- +-- [error] [CE1613] "The selected attribute 'TaskBoard.Person.FullName' no +-- longer exists." at Columns (n/n) of data grid 2 'gridTasks' +-- +-- The message reads as if the attribute had been deleted. It never existed +-- there: Mendix stores the reference against the entity that DECLARES the +-- attribute, and mxcli qualified it with the entity in context. +-- +-- Both shapes were affected, and they go through different resolvers: +-- 1. a direct binding on a grid over the specialization +-- 2. the final attribute of an association path ending at the specialization +-- +-- Entity access rules were never affected — `grant … on TaskBoard.Person` +-- resolves inherited members correctly — so this was the page layer alone. +-- +-- After fix: the reference is qualified with the declaring entity found by +-- walking the generalization chain. An own attribute is unaffected, and an +-- unknown name keeps its previous (context) qualification rather than being +-- silently re-pointed. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/todo-12-inherited-attribute-on-page.mdl -p app.mpr +-- then a real build must report 0 errors: +-- scripts/mx-check.sh -p app.mpr +-- ============================================================================ + +create module TODO12; + +-- Person inherits FullName/Email from Administration.Account and adds one of +-- its own, so a single page can bind both kinds. +create persistent entity TODO12.Person extends Administration.Account ( + IsAvailable : boolean +); + +create persistent entity TODO12.Task ( + Title : string(200) +); + +create association TODO12.Task_Assignee + from TODO12.Task to TODO12.Person; + +-- (1) Direct binding: the grid's data source IS the specialization. +create or replace page TODO12.People +( Title: 'People', Layout: Atlas_Core.Atlas_Default ) +{ + datagrid gridPeople (datasource: database from TODO12.Person) { + column colOwn (caption: 'Available', attribute: IsAvailable) + column colInherited (caption: 'Full name', attribute: FullName) + } +} + +-- (2) Association path: the final hop lands on the specialization. +create or replace page TODO12.Tasks +( Title: 'Tasks', Layout: Atlas_Core.Atlas_Default ) +{ + datagrid gridTasks (datasource: database from TODO12.Task) { + column colTitle (caption: 'Title', attribute: Title) + column colOwnHop (caption: 'Available', attribute: Task_Assignee/IsAvailable) + column colInhHop (caption: 'Full name', attribute: Task_Assignee/FullName) + } +} diff --git a/mdl-examples/bug-tests/todo-14b-uppercase-and-operator.mdl b/mdl-examples/bug-tests/todo-14b-uppercase-and-operator.mdl new file mode 100644 index 000000000..6f486e686 --- /dev/null +++ b/mdl-examples/bug-tests/todo-14b-uppercase-and-operator.mdl @@ -0,0 +1,66 @@ +-- ============================================================================ +-- mxcli-todo finding #14b: an uppercase AND reached the model verbatim (CE0117) +-- ============================================================================ +-- +-- Symptom (before fix): a decision written with uppercase keywords — +-- +-- IF $Task/Status != M.Status.Done AND $Task/CompletedOn != empty THEN +-- +-- passed `mxcli check` and then failed the build: +-- +-- [error] [CE0117] "Error(s) in expression." at Decision +-- '$Task/Status != NEQ.Status.Done AND $Task/CompletedOn != empty' +-- +-- Mendix requires its word operators in lowercase, and the stored expression +-- kept the author's `AND`. +-- +-- Why it looked like something else: the reporter's probe table concluded that +-- "any != appearing as an operand of AND fails", because the `=` form of the +-- same condition builds cleanly. It does — but for an unrelated reason. A `=` +-- condition is rebuilt from the AST, and that path already lowercases the +-- operator; the `!=` form is kept as preserved source text, which skipped it. +-- The trigger is the CASING, not the operator, and it is invisible in a +-- lowercase script. +-- +-- After fix: preserved source has its word operators (and, or, not, div, mod) +-- lowercased on the way to the model, leaving string literals and member names +-- byte-identical. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/todo-14b-uppercase-and-operator.mdl -p app.mpr +-- then a real build must report 0 errors: +-- scripts/mx-check.sh -p app.mpr +-- ============================================================================ + +create module TODO14B; + +create enumeration TODO14B.Status ( ToDo, Done ); + +create persistent entity TODO14B.Task ( + Status : TODO14B.Status, + CompletedOn : datetime +); + +-- Uppercase keywords throughout — the shape that used to fail. +create or modify microflow TODO14B.MF_UppercaseAnd ( $Task: TODO14B.Task ) +begin + IF $Task/Status != TODO14B.Status.Done AND $Task/CompletedOn != empty THEN + change $Task (CompletedOn = empty); + END IF; +end; + +-- The lowercase spelling of the same condition, which always worked. +create or modify microflow TODO14B.MF_LowercaseAnd ( $Task: TODO14B.Task ) +begin + if $Task/Status != TODO14B.Status.Done and $Task/CompletedOn != empty then + change $Task (CompletedOn = empty); + end if; +end; + +-- A string literal containing the word AND must survive untouched. +create or modify microflow TODO14B.MF_LiteralAnd ( $Task: TODO14B.Task ) +begin + IF $Task/Status != TODO14B.Status.Done AND $Task/CompletedOn != empty THEN + log info node 'TODO14B' 'AND OR NOT stay as written'; + END IF; +end; diff --git a/mdl-examples/bug-tests/todo-9-forward-page-reference.fail.mdl b/mdl-examples/bug-tests/todo-9-forward-page-reference.fail.mdl new file mode 100644 index 000000000..b69792aeb --- /dev/null +++ b/mdl-examples/bug-tests/todo-9-forward-page-reference.fail.mdl @@ -0,0 +1,60 @@ +-- ============================================================================ +-- mxcli-todo finding #9: check passed, exec then rejected a forward page ref +-- ============================================================================ +-- +-- Symptom (before fix): `mxcli check` — with no project — answered +-- +-- ✓ Syntax OK (3 statements) +-- Check passed! +-- +-- and `mxcli exec` on the same script then failed partway through: +-- +-- Error: failed to build page: failed to build widget: failed to build +-- action: failed to resolve page: page not found: FWD9.TaskEdit +-- hint: FWD9.TaskEdit is defined later in this script — move its create +-- statement before this one +-- +-- The hint is precise, but it arrives after earlier statements have already +-- been written: `exec` is not transactional, so recovery is a git checkout of +-- the .mpr back to a checkpoint commit. Commit before executing a large script. +-- +-- After fix: plain `mxcli check` (no project needed) reports MDL-PAGE01 and +-- exits 1. Soundness without a project comes from the target being created by +-- a PLAIN create: that create would fail if the page already existed, so the +-- earlier reference cannot resolve against the project either. A later +-- `create or modify page` says nothing of the kind and is left to +-- `check --references`, which can look at the project. +-- +-- A CYCLE cannot be fixed by ordering — if two pages link to each other, no +-- order satisfies both. Create one without the linking widget and add it +-- afterwards with `alter page … insert`. +-- +-- Usage (this script is expected to FAIL the check): +-- mxcli check mdl-examples/bug-tests/todo-9-forward-page-reference.fail.mdl +-- -> MDL-PAGE01, exit 1 +-- ============================================================================ + +create module FWD9; + +-- References FWD9.TaskEdit, which is created below. +create page FWD9.Board +( + title: 'Board', + layout: Atlas_Core.Atlas_Default +) +{ + container ctnMain { + linkbutton btnNew (caption: 'New', action: SHOW_PAGE FWD9.TaskEdit) + } +} + +create page FWD9.TaskEdit +( + title: 'Edit', + layout: Atlas_Core.Atlas_Default +) +{ + container ctnEdit { + dynamictext txtEdit (content: 'edit') + } +} diff --git a/mdl-examples/spikes/test-endpoint-request-handler.mdl b/mdl-examples/spikes/test-endpoint-request-handler.mdl new file mode 100644 index 000000000..726f345c2 --- /dev/null +++ b/mdl-examples/spikes/test-endpoint-request-handler.mdl @@ -0,0 +1,185 @@ +-- ============================================================================ +-- SPIKE: custom request handler as a re-invokable test entry point +-- ============================================================================ +-- Goal: prove that a Java custom request handler registered once at boot can +-- invoke ANY microflow by name over HTTP, so re-running tests costs one HTTP +-- round trip instead of a full runtime restart. +-- ============================================================================ + +create module MxTest; + +-- ---------------------------------------------------------------------------- +-- The handler. Registered once, at boot, from the after-startup microflow. +-- Generic: it resolves the microflow by name from Core.getMicroflowNames(), +-- so adding/renaming/removing tests never requires touching this Java code. +-- ---------------------------------------------------------------------------- +/** Registers the /mxtest/ request handler. Called once from after-startup. */ +create java action MxTest.RegisterTestEndpoint() returns boolean +as $$ +final com.mendix.logging.ILogNode log = com.mendix.core.Core.getLogger("MxTest"); + +com.mendix.core.Core.addRequestHandler("mxtest/", new com.mendix.externalinterface.connector.RequestHandler() { + + private String esc(String s) { + if (s == null) return "null"; + StringBuilder b = new StringBuilder("\""); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + switch (c) { + case '"': b.append("\\\""); break; + case '\\': b.append("\\\\"); break; + case '\n': b.append("\\n"); break; + case '\r': b.append("\\r"); break; + case '\t': b.append("\\t"); break; + default: + if (c < 0x20) b.append(String.format("\\u%04x", (int) c)); + else b.append(c); + } + } + return b.append('"').toString(); + } + + @Override + protected void processRequest(com.mendix.m2ee.api.IMxRuntimeRequest request, + com.mendix.m2ee.api.IMxRuntimeResponse response, + String path) throws Exception { + response.setContentType("application/json"); + java.io.Writer out = response.getWriter(); + + java.util.Set known = com.mendix.core.Core.getMicroflowNames(); + + // GET /mxtest/list?prefix=MxTest.Test_ -> discover tests at runtime + if (path != null && path.startsWith("list")) { + String prefix = request.getParameter("prefix"); + java.util.List names = new java.util.ArrayList(); + for (String n : known) { + if (prefix == null || n.startsWith(prefix)) names.add(n); + } + java.util.Collections.sort(names); + StringBuilder b = new StringBuilder("{\"microflows\":["); + for (int i = 0; i < names.size(); i++) { + if (i > 0) b.append(','); + b.append(esc(names.get(i))); + } + b.append("]}"); + out.write(b.toString()); + out.flush(); + return; + } + + // GET /mxtest/run?mf=Module.Microflow + String mf = request.getParameter("mf"); + if (mf == null || mf.isEmpty()) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.BAD_REQUEST); + out.write("{\"error\":\"missing mf parameter\"}"); + out.flush(); + return; + } + if (!known.contains(mf)) { + response.setStatus(com.mendix.m2ee.api.IMxRuntimeResponse.NOT_FOUND); + out.write("{\"error\":\"unknown microflow\",\"mf\":" + esc(mf) + "}"); + out.flush(); + return; + } + + long t0 = System.nanoTime(); + com.mendix.systemwideinterfaces.core.IContext ctx = com.mendix.core.Core.createSystemContext(); + Object result = null; + String error = null; + try { + result = com.mendix.core.Core.microflowCall(mf).execute(ctx); + } catch (Throwable t) { + Throwable root = t; + while (root.getCause() != null && root.getCause() != root) root = root.getCause(); + error = root.getClass().getName() + ": " + root.getMessage(); + log.warn("test microflow " + mf + " threw: " + error); + } + long micros = (System.nanoTime() - t0) / 1000L; + + StringBuilder b = new StringBuilder("{"); + b.append("\"mf\":").append(esc(mf)); + b.append(",\"ok\":").append(error == null); + b.append(",\"durationMicros\":").append(micros); + b.append(",\"result\":").append(result == null ? "null" : esc(String.valueOf(result))); + if (error != null) b.append(",\"error\":").append(esc(error)); + b.append('}'); + out.write(b.toString()); + out.flush(); + } +}); + +log.info("MxTest request handler registered at /mxtest/"); +return true; +$$; +/ + +-- ---------------------------------------------------------------------------- +-- Boot hook: the ONLY thing that runs at startup. Registration, not tests. +-- ---------------------------------------------------------------------------- +/** Registers the test endpoint at boot. */ +create microflow MxTest.AfterStartup () +returns boolean as $Registered +begin + $Registered = call java action MxTest.RegisterTestEndpoint(); + return $Registered; +end; +/ + +alter settings model AfterStartupMicroflow = 'MxTest.AfterStartup'; + +-- ---------------------------------------------------------------------------- +-- Sample tests. Each is its own microflow -> per-test invocation and filtering +-- come for free, and a throwing test is an HTTP 200 with ok:false, not a +-- failed runtime boot. +-- ---------------------------------------------------------------------------- +create persistent entity MxTest.Widget ( + Label: string(100), + Amount: integer +); +/ + +/** Passing test: creates and reads back an object. */ +create microflow MxTest.Test_CreateWidget () +returns string as $Outcome +begin + declare $Outcome String = 'FAIL amount was not 41'; + $W = create MxTest.Widget (Label = 'alpha', Amount = 41); + commit $W; + if $W/Amount = 41 then + set $Outcome = 'PASS created widget with amount 41'; + end if; + return $Outcome; +end; +/ + +/** Passing test: pure arithmetic, no database. */ +create microflow MxTest.Test_Arithmetic () +returns string as $Outcome +begin + declare $Outcome String = ''; + declare $Sum Integer = 0; + set $Sum = 2 + 2; + if $Sum = 4 then + set $Outcome = 'PASS 2+2=4'; + else + set $Outcome = 'FAIL arithmetic broken'; + end if; + return $Outcome; +end; +/ + +/** Throws, to prove a failing test is reported not fatal. */ +create java action MxTest.Boom() returns string +as $$ +throw new com.mendix.systemwideinterfaces.MendixRuntimeException("deliberate failure from Test_Failing"); +$$; +/ + +/** Failing test: propagates a Java exception out of the microflow. */ +create microflow MxTest.Test_Failing () +returns string as $Outcome +begin + $Outcome = call java action MxTest.Boom(); + return $Outcome; +end; +/ diff --git a/mdl/ast/ast_odata.go b/mdl/ast/ast_odata.go index 36bde4eae..729a8a24a 100644 --- a/mdl/ast/ast_odata.go +++ b/mdl/ast/ast_odata.go @@ -43,6 +43,9 @@ type CreateODataClientStmt struct { // Custom HTTP headers Headers []HeaderDef + + // UnknownProperties: see CreateODataServiceStmt.UnknownProperties. + UnknownProperties []string } // HeaderDef represents a custom HTTP header entry. @@ -70,20 +73,32 @@ func (s *DropODataClientStmt) isStatement() {} // CreateODataServiceStmt represents: CREATE ODATA SERVICE Module.Name (...) AUTHENTICATION ... { ... } type CreateODataServiceStmt struct { - Name QualifiedName - Path string - Version string - ODataVersion string - Namespace string - ServiceName string - Summary string - Description string - Documentation string - Folder string // Folder path within module (e.g., "Integration/APIs") - PublishAssociations bool - AuthenticationTypes []string - Entities []*PublishedEntityDef - CreateOrModify bool // True if CREATE OR MODIFY was used + Name QualifiedName + Path string + Version string + ODataVersion string + Namespace string + ServiceName string + Summary string + Description string + Documentation string + Folder string // Folder path within module (e.g., "Integration/APIs") + // PublishAssociations selects how associations appear in the metadata: + // true = as links, false = as an associated object id. The executor + // defaults an unspecified value to true, so PublishAssociationsSet records + // whether the author said anything at all — an explicit false is the + // author's choice and is written as given. + PublishAssociations bool + PublishAssociationsSet bool + AuthenticationTypes []string + Entities []*PublishedEntityDef + CreateOrModify bool // True if CREATE OR MODIFY was used + + // UnknownProperties holds property names the visitor did not recognise, in + // source order. The parser accepts any `name: value` pair, so without this + // a typo is discarded in silence and the model is quietly missing what the + // author asked for. + UnknownProperties []string } func (s *CreateODataServiceStmt) isStatement() {} @@ -99,6 +114,17 @@ type PublishedEntityDef struct { UsePaging bool PageSize int Members []*PublishedMemberDef + + // Query options. nil means "not specified" and stores Mendix's own default + // of true. Countable in particular is not free: it forces the read + // microflow to take a System.ODataResponse parameter and to compute a count + // the caller may never ask for. + Countable *bool + SkipSupported *bool + TopSupported *bool + + // UnknownProperties: see CreateODataServiceStmt.UnknownProperties. + UnknownProperties []string } // PublishedMemberDef represents an EXPOSE member within a PUBLISH ENTITY block. @@ -144,6 +170,9 @@ type CreateExternalEntityStmt struct { Attributes []Attribute // reuse from ast_entity.go Documentation string CreateOrModify bool + + // UnknownProperties: see CreateODataServiceStmt.UnknownProperties. + UnknownProperties []string } func (s *CreateExternalEntityStmt) isStatement() {} diff --git a/mdl/backend/modelsdk/microflow_downloadfile_test.go b/mdl/backend/modelsdk/microflow_downloadfile_test.go index a5765560c..2c194614e 100644 --- a/mdl/backend/modelsdk/microflow_downloadfile_test.go +++ b/mdl/backend/modelsdk/microflow_downloadfile_test.go @@ -5,6 +5,7 @@ package modelsdkbackend import ( "testing" + "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/microflows" "go.mongodb.org/mongo-driver/v2/bson" ) @@ -35,3 +36,71 @@ func TestActionFromGen_DownloadFile(t *testing.T) { t.Errorf("ErrorHandlingType = %q, want Rollback default", df.ErrorHandlingType) } } + +// TestMicroflowRoundTrip_DownloadFile is the write-side counterpart of +// TestActionFromGen_DownloadFile, and the test that would have caught issue #850. +// +// microflowActionToGen had no *microflows.DownloadFileAction case, so the action +// fell through to `default: return nil` and the enclosing ActionActivity was +// written with no Action at all. `download file $Doc;` was accepted by the +// grammar, the visitor, the flow builder and the DESCRIBE formatter, and only +// vanished in the one stage that reports nothing — `mxcli exec` printed "Created +// microflow" and `mx check` then failed with CE0008 "No action defined." +// +// A reader-only test cannot catch this class: it starts from BSON the writer +// never had to produce. The round trip closes that gap. +func TestMicroflowRoundTrip_DownloadFile(t *testing.T) { + for _, showInBrowser := range []bool{true, false} { + name := "ShowInBrowser=false" + if showInBrowser { + name = "ShowInBrowser=true" + } + t.Run(name, func(t *testing.T) { + act := µflows.DownloadFileAction{ + FileDocument: "Doc", + ShowInBrowser: showInBrowser, + ErrorHandlingType: microflows.ErrorHandlingTypeRollback, + } + act.ID = model.ID("df-1") + activity := µflows.ActionActivity{Action: act} + activity.ID = model.ID("act-1") + + mf := µflows.Microflow{ + Name: "ACT_Download", + ObjectCollection: µflows.MicroflowObjectCollection{ + Objects: []microflows.MicroflowObject{activity}, + }, + } + mf.ID = model.ID("mf-1") + + got := roundTripMicroflow(t, mf) + + var found *microflows.DownloadFileAction + for _, obj := range got.ObjectCollection.Objects { + aa, ok := obj.(*microflows.ActionActivity) + if !ok { + continue + } + if aa.Action == nil { + t.Fatal("ActionActivity round-tripped with a nil Action — " + + "this is the CE0008 \"No action defined.\" shape from #850") + } + if df, ok := aa.Action.(*microflows.DownloadFileAction); ok { + found = df + } + } + if found == nil { + t.Fatal("no DownloadFileAction survived the round trip") + } + if found.FileDocument != "Doc" { + t.Errorf("FileDocument = %q, want Doc", found.FileDocument) + } + if found.ShowInBrowser != showInBrowser { + t.Errorf("ShowInBrowser = %v, want %v", found.ShowInBrowser, showInBrowser) + } + if found.ErrorHandlingType != microflows.ErrorHandlingTypeRollback { + t.Errorf("ErrorHandlingType = %q, want Rollback", found.ErrorHandlingType) + } + }) + } +} diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index 51878c559..05db0d046 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -543,6 +543,21 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { } addPartList(g, "ParameterMappings", mappings) return g + case *microflows.DownloadFileAction: + // DOWNLOAD FILE. Without this case the action fell through to + // `default: return nil` and the enclosing ActionActivity was written with + // no Action at all — `mxcli exec` reported "Created microflow" and only + // `mx check` noticed, as CE0008 "No action defined." (issue #850). + // + // The storage key is ShowFileInBrowser, not ShowInBrowser; the gen setter + // binds the right one (legacy's parseDownloadFileAction reads the wrong + // key — see TestActionFromGen_DownloadFile). + g := genMf.NewDownloadFileAction() + g.SetID(element.ID(a.ID)) + g.SetErrorHandlingType(orDefault(string(a.ErrorHandlingType), "Rollback")) + g.SetFileDocumentVariableName(a.FileDocument) + g.SetShowFileInBrowser(a.ShowInBrowser) + return g case *microflows.LogMessageAction: g := genMf.NewLogMessageAction() g.SetID(element.ID(a.ID)) diff --git a/mdl/backend/modelsdk/odata_read_detail.go b/mdl/backend/modelsdk/odata_read_detail.go index 953245ef3..95a5c4894 100644 --- a/mdl/backend/modelsdk/odata_read_detail.go +++ b/mdl/backend/modelsdk/odata_read_detail.go @@ -127,6 +127,13 @@ func publishedEntitySetFromRaw(raw map[string]any, byID map[string]*model.Publis UpdateMode: parseODataModeRaw(raw["UpdateMode"]), DeleteMode: parseODataModeRaw(raw["DeleteMode"]), } + // Query options round-trip so DESCRIBE can print a turned-off one. Absent + // (or true, the default) is left nil so DESCRIBE stays quiet about it. + if qo := jsToMap(raw["QueryOptions"]); qo != nil { + es.Countable = falseOnly(qo["Countable"]) + es.SkipSupported = falseOnly(qo["SkipSupported"]) + es.TopSupported = falseOnly(qo["TopSupported"]) + } if et, ok := byID[jsExtractBsonID(raw["EntityTypePointer"])]; ok { es.EntityTypeName = et.Entity } @@ -172,3 +179,14 @@ func jsExtractInt(v any) int { } return 0 } + +// falseOnly returns a pointer to false when v is present and false, else nil. +// A stored true is the default, and printing every default back would make +// DESCRIBE output noisier than what the author wrote. +func falseOnly(v any) *bool { + if v == nil || jsExtractBool(v) { + return nil + } + f := false + return &f +} diff --git a/mdl/backend/modelsdk/odata_write.go b/mdl/backend/modelsdk/odata_write.go index 34c1cc6b1..7a153da21 100644 --- a/mdl/backend/modelsdk/odata_write.go +++ b/mdl/backend/modelsdk/odata_write.go @@ -287,10 +287,14 @@ func publishedEntitySetToGen(es *model.PublishedEntitySet, entityTypeID string) addStr(g, "AlternativeExposedName", "") addBool(g, "UsePaging", es.UsePaging) addInt64(g, "PageSize", int64(es.PageSize)) + // Query options default to true (Mendix's own defaults) and are only turned + // off when the author says so. Countable is not free: it forces the read + // microflow to take a System.ODataResponse parameter and compute a count. + // (mxcli-formula1 findings #10.3.) qo := newElem("ODataPublish$QueryOptions", "") - addBool(qo, "Countable", true) - addBool(qo, "SkipSupported", true) - addBool(qo, "TopSupported", true) + addBool(qo, "Countable", boolOrDefault(es.Countable, true)) + addBool(qo, "SkipSupported", boolOrDefault(es.SkipSupported, true)) + addBool(qo, "TopSupported", boolOrDefault(es.TopSupported, true)) addPart(g, "QueryOptions", qo) if entityTypeID != "" { addIDRef(g, "EntityTypePointer", model.ID(entityTypeID)) @@ -425,3 +429,11 @@ func addByNameRefListV3(b *element.Base, name string, qnames []string) { p.Append(qn) } } + +// boolOrDefault resolves an optional bool: nil means "not specified". +func boolOrDefault(v *bool, def bool) bool { + if v == nil { + return def + } + return *v +} diff --git a/mdl/backend/modelsdk/odata_write_test.go b/mdl/backend/modelsdk/odata_write_test.go index ac86033dd..54b1bf372 100644 --- a/mdl/backend/modelsdk/odata_write_test.go +++ b/mdl/backend/modelsdk/odata_write_test.go @@ -119,6 +119,10 @@ func TestCreatePublishedODataService_RoundTrip(t *testing.T) { EntitySets: []*model.PublishedEntitySet{{ ExposedName: "Things", EntityTypeName: "MyFirstModule.Thing", ReadMode: "source", UsePaging: true, PageSize: 100, + // mxcli-formula1 #10.3: these were hardcoded true in the writer. + // An explicit false must survive the BSON round trip; SkipSupported + // is left unset here so the default still applies to it. + Countable: boolPtrLocal(false), TopSupported: boolPtrLocal(false), }}, } if err := b.CreatePublishedODataService(svc); err != nil { @@ -179,8 +183,21 @@ func TestCreatePublishedODataService_RoundTrip(t *testing.T) { if !es.UsePaging || es.PageSize != 100 { t.Errorf("entity set paging not round-tripped: %+v", es) } + if es.Countable == nil || *es.Countable { + t.Errorf("Countable=false not round-tripped: %v", es.Countable) + } + if es.TopSupported == nil || *es.TopSupported { + t.Errorf("TopSupported=false not round-tripped: %v", es.TopSupported) + } + // An unset option is stored as Mendix's default of true, and reads back as + // nil so DESCRIBE does not print a default nobody wrote. + if es.SkipSupported != nil { + t.Errorf("SkipSupported should read back nil when defaulted: %v", *es.SkipSupported) + } } +func boolPtrLocal(b bool) *bool { return &b } + // TestConsumedODataServiceToGen_ConfigMicroflowKey guards issue #728: the config // microflow must be serialized under the version-appropriate BSON key. On // 11.10+ that is ConfigurationEntityMicroflow (writing the pre-11.10 diff --git a/mdl/backend/pagemutator/mutator.go b/mdl/backend/pagemutator/mutator.go index 8613c2456..a320725f6 100644 --- a/mdl/backend/pagemutator/mutator.go +++ b/mdl/backend/pagemutator/mutator.go @@ -2139,7 +2139,14 @@ func setWidgetConditionalSettingMut(widget bson.D, field, typeName, expression s doc := bson.D{ {Key: "$ID", Value: bsonutil.NewIDBsonBinary()}, {Key: "$Type", Value: typeName}, - {Key: "Attribute", Value: nil}, + // Attribute is a BY_NAME AttributeIdentifier: unset is the empty string, + // NOT null. A null fails the reader with StorageLoadException "…has an + // invalid value '' for property Attribute" and the project will not open, + // while `mx check` still passes. The CREATE path encodes it the same way + // via the Forms$Conditional{Visibility,Editability}Settings TypeDefaults + // (mdl/backend/modelsdk/widget_write.go, EmptyStringFields); this ALTER + // path builds the node by hand and has to match. Issue #851. + {Key: "Attribute", Value: ""}, {Key: "Conditions", Value: bson.A{int32(3)}}, {Key: "Expression", Value: expression}, } diff --git a/mdl/backend/pagemutator/mutator_test.go b/mdl/backend/pagemutator/mutator_test.go index 45da2b295..1e667b9c4 100644 --- a/mdl/backend/pagemutator/mutator_test.go +++ b/mdl/backend/pagemutator/mutator_test.go @@ -346,6 +346,80 @@ func TestSetWidgetProperty_VisibleExpression(t *testing.T) { }) } +// lookupBsonKey reports a key's value AND whether the key is present, which +// bsonnav.DGet cannot distinguish (an absent key and a stored null both read +// back as nil). Presence is the thing under test here: Mendix fills an omitted +// optional property on load, so "absent" and "null" fail differently. +func lookupBsonKey(doc bson.D, key string) (any, bool) { + for _, e := range doc { + if e.Key == key { + return e.Value, true + } + } + return nil, false +} + +// makeEditabilityWidget builds an input widget with null conditional +// visibility/editability slots (as Studio Pro writes them when unset). +func makeEditabilityWidget(name string) bson.D { + return bson.D{ + {Key: "$Type", Value: "Forms$TextBox"}, + {Key: "Name", Value: name}, + {Key: "ConditionalVisibilitySettings", Value: nil}, + {Key: "ConditionalEditabilitySettings", Value: nil}, + } +} + +// TestSetWidgetConditionalSetting_AttributeIsEmptyString locks in the fix for +// issue #851: `alter page … set Editable = [expr]` produced a project Studio Pro +// refused to load with StorageLoadException "Conditional editability settings has +// an invalid value ” for property Attribute". +// +// Attribute is a BY_NAME AttributeIdentifier, so the absent value is the empty +// string, not null — the CREATE path already encodes it that way via the +// Forms$Conditional{Visibility,Editability}Settings TypeDefaults +// (EmptyStringFields: Attribute, see mdl/backend/modelsdk/widget_write.go). The +// ALTER path built the node by hand and wrote nil, so the same widget authored +// through ALTER instead of CREATE was unloadable. +func TestSetWidgetConditionalSetting_AttributeIsEmptyString(t *testing.T) { + cases := []struct { + prop string // MDL property as the visitor delivers the [expr] form + field string // BSON slot it lands in + }{ + {"EditableIf", "ConditionalEditabilitySettings"}, + {"VisibleIf", "ConditionalVisibilitySettings"}, + } + for _, tc := range cases { + t.Run(tc.prop, func(t *testing.T) { + rawData := makeRawPage(makeEditabilityWidget("b1")) + m := &Mutator{rawData: rawData, widgetFinder: findBsonWidget} + expr := "$currentObject/Slug != ''" + if err := m.SetWidgetProperty("b1", tc.prop, expr); err != nil { + t.Fatalf("SetWidgetProperty(%s) failed: %v", tc.prop, err) + } + node := bsonnav.DGetDoc(findBsonWidget(rawData, "b1").widget, tc.field) + if node == nil { + t.Fatalf("expected a %s node", tc.field) + } + if got := bsonnav.DGetString(node, "Expression"); got != expr { + t.Errorf("Expression = %q, want %q", got, expr) + } + attr, ok := lookupBsonKey(node, "Attribute") + if !ok { + t.Fatal("Attribute key missing — Studio Pro requires the slot to be present") + } + if attr != any("") { + t.Errorf("Attribute = %#v, want %#v — a null here is not a valid "+ + "AttributeIdentifier and Studio Pro refuses to load the page", attr, "") + } + // SourceVariable is a BY_ID reference and stays null when unset. + if sv, ok := lookupBsonKey(node, "SourceVariable"); !ok || sv != nil { + t.Errorf("SourceVariable = %#v (present=%v), want nil", sv, ok) + } + }) + } +} + func TestSetWidgetProperty_ButtonStyle(t *testing.T) { w1 := bson.D{ {Key: "$Type", Value: "Pages$ActionButton"}, diff --git a/mdl/executor/cmd_entities.go b/mdl/executor/cmd_entities.go index 480640858..33edaa03d 100644 --- a/mdl/executor/cmd_entities.go +++ b/mdl/executor/cmd_entities.go @@ -29,6 +29,9 @@ func buildEventHandlers(ctx *ExecContext, defs []ast.EventHandlerDef) ([]*domain if err != nil { return nil, mdlerrors.NewNotFound("microflow", mfQN) } + if err := checkBeforeCreateHandlerHasNoParameters(ctx, d, mfQN, mfID); err != nil { + return nil, err + } handlers = append(handlers, &domainmodel.EventHandler{ BaseElement: model.BaseElement{ ID: model.ID(types.GenerateID()), @@ -45,6 +48,42 @@ func buildEventHandlers(ctx *ExecContext, defs []ast.EventHandlerDef) ([]*domain return handlers, nil } +// checkBeforeCreateHandlerHasNoParameters refuses to wire a microflow that takes +// parameters to a BEFORE CREATE event handler. +// +// Mendix passes no object to a before-create handler — the object does not exist +// yet — so the build fails with +// +// [CE7247] "Microflow should not have parameters" at Event handler of entity … +// +// which `mxcli check` had no way to see. The handler a defaulting microflow +// actually wants is AFTER CREATE, which does receive the object. +// (mxcli-todo findings #14a) +func checkBeforeCreateHandlerHasNoParameters(ctx *ExecContext, d ast.EventHandlerDef, mfQN string, mfID model.ID) error { + if !strings.EqualFold(d.Moment, "Before") || !strings.EqualFold(d.Event, "Create") { + return nil + } + mf, err := ctx.Backend.GetMicroflow(mfID) + // A microflow created earlier in this same script is not readable back yet; + // skip rather than guess. mxbuild still catches it, and refusing on a failed + // read would break a legitimate script. + if err != nil || mf == nil { + return nil + } + if len(mf.Parameters) == 0 { + return nil + } + names := make([]string, 0, len(mf.Parameters)) + for _, p := range mf.Parameters { + names = append(names, "$"+p.Name) + } + return mdlerrors.NewValidationf( + "microflow %s takes %d parameter(s) (%s), but a BEFORE CREATE event handler is called with none — "+ + "Mendix has no object to pass yet, so this builds as CE7247 \"Microflow should not have parameters\"\n"+ + "hint: use ON AFTER CREATE, which does receive the object, or remove the parameters", + mfQN, len(mf.Parameters), strings.Join(names, ", ")) +} + func execCreateEntity(ctx *ExecContext, s *ast.CreateEntityStmt) error { if !ctx.Connected() { return mdlerrors.NewNotConnected() @@ -1077,9 +1116,18 @@ func execAlterEntity(ctx *ExecContext, s *ast.AlterEntityStmt) error { // Reject duplicate (same Moment + Event) for _, existing := range entity.EventHandlers { if existing.Moment == ehs[0].Moment && existing.Event == ehs[0].Event { + // An event handler has no other re-run route: a defensive + // drop-then-add fails on the drop when it is absent and on the + // add when it is present. IF NOT EXISTS makes the script + // idempotent, like ADD ATTRIBUTE. (mxcli-todo findings #18) + if s.IfNotExists { + fmt.Fprintf(ctx.Output, "Event handler %s %s already exists on %s, skipping\n", + s.EventHandler.Moment, s.EventHandler.Event, s.Name) + return nil + } return mdlerrors.NewAlreadyExistsMsg("event handler", fmt.Sprintf("%s %s", s.EventHandler.Moment, s.EventHandler.Event), - fmt.Sprintf("event handler already exists for %s %s on %s", s.EventHandler.Moment, s.EventHandler.Event, s.Name)) + fmt.Sprintf("event handler already exists for %s %s on %s (use ADD EVENT HANDLER IF NOT EXISTS to make the script re-runnable)", s.EventHandler.Moment, s.EventHandler.Event, s.Name)) } } entity.EventHandlers = append(entity.EventHandlers, ehs[0]) @@ -1104,9 +1152,14 @@ func execAlterEntity(ctx *ExecContext, s *ast.AlterEntityStmt) error { } } if idx < 0 { + if s.IfExists { + fmt.Fprintf(ctx.Output, "Event handler %s %s not present on %s, skipping\n", + s.EventHandler.Moment, s.EventHandler.Event, s.Name) + return nil + } return mdlerrors.NewNotFoundMsg("event handler", fmt.Sprintf("%s %s", s.EventHandler.Moment, s.EventHandler.Event), - fmt.Sprintf("event handler %s %s not found on %s", s.EventHandler.Moment, s.EventHandler.Event, s.Name)) + fmt.Sprintf("event handler %s %s not found on %s (use DROP EVENT HANDLER IF EXISTS to make the script re-runnable)", s.EventHandler.Moment, s.EventHandler.Event, s.Name)) } entity.EventHandlers = append(entity.EventHandlers[:idx], entity.EventHandlers[idx+1:]...) if err := ctx.Backend.UpdateEntity(dm.ID, entity); err != nil { diff --git a/mdl/executor/cmd_entities_before_create_test.go b/mdl/executor/cmd_entities_before_create_test.go new file mode 100644 index 000000000..f40054c21 --- /dev/null +++ b/mdl/executor/cmd_entities_before_create_test.go @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// mxcli-todo findings #14a: Mendix passes no object to a BEFORE CREATE handler — +// the object does not exist yet — so wiring a microflow that takes parameters +// there builds as CE7247 "Microflow should not have parameters". `mxcli check` +// could not see it, and the model was written before anyone found out. +func TestCheckBeforeCreateHandlerHasNoParameters(t *testing.T) { + withParams := µflows.Microflow{ + Name: "OCH_SetDefaults", + Parameters: []*microflows.MicroflowParameter{{Name: "Task"}}, + } + noParams := µflows.Microflow{Name: "OCH_Audit"} + + tests := []struct { + name string + moment string + event string + mf *microflows.Microflow + wantError bool + }{ + {"before create with parameters is refused", "Before", "Create", withParams, true}, + {"before create without parameters is fine", "Before", "Create", noParams, false}, + // Every other moment/event receives the object, so parameters are correct there. + {"after create with parameters is fine", "After", "Create", withParams, false}, + {"before commit with parameters is fine", "Before", "Commit", withParams, false}, + {"before delete with parameters is fine", "Before", "Delete", withParams, false}, + // MDL keywords are case-insensitive; the guard must not be. + {"lowercase spelling is still matched", "before", "create", withParams, true}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := &ExecContext{Backend: &mock.MockBackend{ + GetMicroflowFunc: func(model.ID) (*microflows.Microflow, error) { return tt.mf, nil }, + }} + def := ast.EventHandlerDef{Moment: tt.moment, Event: tt.event} + err := checkBeforeCreateHandlerHasNoParameters(ctx, def, "M.OCH_SetDefaults", model.ID("mf-1")) + + if tt.wantError { + if err == nil { + t.Fatal("expected the handler to be refused") + } + // The message has to carry the build error and the way out, or + // the user is no better off than with CE7247 alone. + for _, want := range []string{"CE7247", "AFTER CREATE", "$Task"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got: %v", want, err) + } + } + } else if err != nil { + t.Errorf("expected no error, got: %v", err) + } + }) + } +} + +// A microflow created earlier in the same script is not readable back yet. The +// guard must skip rather than refuse — mxbuild still catches the real case, and +// failing on an unreadable microflow would break legitimate scripts. +func TestCheckBeforeCreateHandler_UnreadableMicroflowIsSkipped(t *testing.T) { + ctx := &ExecContext{Backend: &mock.MockBackend{ + GetMicroflowFunc: func(model.ID) (*microflows.Microflow, error) { + return nil, errUnreadable + }, + }} + def := ast.EventHandlerDef{Moment: "Before", Event: "Create"} + if err := checkBeforeCreateHandlerHasNoParameters(ctx, def, "M.MF", model.ID("mf-1")); err != nil { + t.Errorf("expected the guard to skip an unreadable microflow, got: %v", err) + } +} + +var errUnreadable = &unreadableError{} + +type unreadableError struct{} + +func (e *unreadableError) Error() string { return "not readable yet" } diff --git a/mdl/executor/cmd_microflows_helpers.go b/mdl/executor/cmd_microflows_helpers.go index 8d3d1b5bc..5c8667ecd 100644 --- a/mdl/executor/cmd_microflows_helpers.go +++ b/mdl/executor/cmd_microflows_helpers.go @@ -333,7 +333,7 @@ func expressionToString(expr ast.Expression) string { return "if " + cond + " then " + thenStr + " else " + elseStr case *ast.SourceExpr: if e.Source != "" { - return e.Source + return normalizeMendixOperatorCase(e.Source) } return expressionToString(e.Expression) default: @@ -341,6 +341,85 @@ func expressionToString(expr ast.Expression) string { } } +// mendixLowercaseOperators are the word operators Mendix requires in lowercase. +// A rebuilt BinaryExpr/UnaryExpr already gets this via strings.ToLower on the +// operator; preserved source text does not, which is the whole bug below. +var mendixLowercaseOperators = map[string]bool{ + "and": true, "or": true, "not": true, "div": true, "mod": true, +} + +// normalizeMendixOperatorCase lowercases word operators in preserved expression +// source, leaving everything else — including string literals and member names — +// byte-identical. +// +// Some conditions are kept as a SourceExpr (original text plus the parsed tree) +// rather than rebuilt from the AST, and the raw branch skipped the lowercasing +// that a rebuilt expression gets. So `IF A != x AND B != empty` stored `AND` +// verbatim and the build failed with +// +// [CE0117] "Error(s) in expression." +// +// while the same condition written with `=` was rebuilt as a BinaryExpr and +// normalised — which is why it looked like `!=` inside a conjunction was +// unsupported. It is the casing, not the operator. (mxcli-todo findings #14b) +// +// A word preceded by `.`, `/` or `$` is a member or variable name, never an +// operator, so `Module.Enum.And` and `$Task/Mod` are left alone. +func normalizeMendixOperatorCase(src string) string { + var b strings.Builder + b.Grow(len(src)) + + inString := false + for i := 0; i < len(src); { + c := src[i] + if inString { + b.WriteByte(c) + if c == '\'' { + // '' is an escaped quote inside a Mendix string literal. + if i+1 < len(src) && src[i+1] == '\'' { + b.WriteByte(src[i+1]) + i += 2 + continue + } + inString = false + } + i++ + continue + } + if c == '\'' { + inString = true + b.WriteByte(c) + i++ + continue + } + if !isWordByte(c) { + b.WriteByte(c) + i++ + continue + } + j := i + for j < len(src) && isWordByte(src[j]) { + j++ + } + word := src[i:j] + prev := byte(0) + if i > 0 { + prev = src[i-1] + } + if prev != '.' && prev != '/' && prev != '$' && mendixLowercaseOperators[strings.ToLower(word)] { + b.WriteString(strings.ToLower(word)) + } else { + b.WriteString(word) + } + i = j + } + return b.String() +} + +func isWordByte(c byte) bool { + return c == '_' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') +} + // expressionToXPath converts an AST Expression to an XPath constraint string. // Unlike expressionToString (for Mendix expressions), XPath requires Mendix // tokens like [%CurrentDateTime%] to be quoted: '[%CurrentDateTime%]'. diff --git a/mdl/executor/cmd_microflows_operator_case_test.go b/mdl/executor/cmd_microflows_operator_case_test.go new file mode 100644 index 000000000..d8228825d --- /dev/null +++ b/mdl/executor/cmd_microflows_operator_case_test.go @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import "testing" + +// mxcli-todo findings #14b: a condition kept as a SourceExpr (original text plus +// the parsed tree) skipped the operator lowercasing a rebuilt BinaryExpr gets, +// so `IF A != x AND B != empty` stored `AND` verbatim and mxbuild answered +// CE0117. The same condition with `=` was rebuilt and normalised, which is what +// made it look like `!=` inside a conjunction was unsupported. +func TestNormalizeMendixOperatorCase(t *testing.T) { + tests := []struct { + name string + in string + want string + }{ + { + "the reported case", + "$Task/Status != M.Status.Done AND $Task/CompletedOn != empty", + "$Task/Status != M.Status.Done and $Task/CompletedOn != empty", + }, + {"already lowercase is unchanged", "$a = 1 and $b = 2", "$a = 1 and $b = 2"}, + {"mixed case", "$a = 1 And $b = 2 Or $c = 3", "$a = 1 and $b = 2 or $c = 3"}, + {"NOT", "NOT($a = 1)", "not($a = 1)"}, + {"DIV and MOD", "$a DIV $b MOD 2", "$a div $b mod 2"}, + + // Everything that is not an operator must survive byte-identical. + { + "a string literal is never touched", + "$a = 'AND' and $b = 'NOT OR'", + "$a = 'AND' and $b = 'NOT OR'", + }, + { + "an escaped quote does not end the literal early", + "$a = 'it''s AND then' and $b = 1", + "$a = 'it''s AND then' and $b = 1", + }, + { + "an enum value named And is a member, not an operator", + "$x/Kind = M.Enum.And and $y = 1", + "$x/Kind = M.Enum.And and $y = 1", + }, + { + "an attribute after / is a member", + "$Task/Mod = 1 AND $Task/Not = 2", + "$Task/Mod = 1 and $Task/Not = 2", + }, + { + "a variable named $And keeps its case", + "$And = 1 AND $b = 2", + "$And = 1 and $b = 2", + }, + { + "words merely containing an operator are left alone", + "$Android = 1 AND $Normal = 2", + "$Android = 1 and $Normal = 2", + }, + {"empty input", "", ""}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := normalizeMendixOperatorCase(tt.in); got != tt.want { + t.Errorf("normalizeMendixOperatorCase(%q)\n got %q\nwant %q", tt.in, got, tt.want) + } + }) + } +} diff --git a/mdl/executor/cmd_modules.go b/mdl/executor/cmd_modules.go index b87113041..e4e102339 100644 --- a/mdl/executor/cmd_modules.go +++ b/mdl/executor/cmd_modules.go @@ -5,6 +5,8 @@ package executor import ( "fmt" + "os" + "path/filepath" "sort" "strings" @@ -1058,9 +1060,56 @@ func execAlterModuleJarDep(ctx *ExecContext, s *ast.AlterModuleJarDepStmt) error } fmt.Fprintf(ctx.Output, "Updated jar dependencies for module '%s'\n", s.ModuleName) + warnUnvendoredJarDependencies(ctx, ms) return nil } +// warnUnvendoredJarDependencies says so when a declared dependency has no jar in +// the project's vendorlib/. +// +// Declaring and resolving are separate steps: this writes the coordinate to the +// model, and `list jar dependencies` will report it, but nothing downloads the +// jar — and MxBuild does not resolve it either, so the build stays green and the +// first symptom is a runtime "no driver found" exception from code that looks +// correctly configured. Studio Pro runs the resolution when you edit Module +// Settings; headless it has to be asked for. (mxcli-formula1 findings #12.) +func warnUnvendoredJarDependencies(ctx *ExecContext, ms *types.ModuleSettings) { + if ctx == nil || ctx.Backend == nil || ms == nil { + return + } + projectPath := ctx.Backend.Path() + if projectPath == "" { + return + } + vendorlib := filepath.Join(filepath.Dir(projectPath), "vendorlib") + + var missing []string + for _, d := range ms.JarDependencies { + if d == nil || d.ArtifactID == "" || d.Version == "" { + continue + } + jar := filepath.Join(vendorlib, fmt.Sprintf("%s-%s.jar", d.ArtifactID, d.Version)) + if _, err := os.Stat(jar); err != nil { + missing = append(missing, fmt.Sprintf("%s:%s:%s", d.GroupID, d.ArtifactID, d.Version)) + } + } + if len(missing) == 0 { + return + } + fmt.Fprintf(ctx.Output, + " Note: %s not in vendorlib/, so %s not on the classpath yet — the build will still succeed and fail at runtime.\n", + strings.Join(missing, ", "), pluralItIsTheyAre(len(missing))) + fmt.Fprintf(ctx.Output, " Run: mxcli sync-java-deps -p %s\n", projectPath) +} + +// pluralItIsTheyAre picks the pronoun+verb for a list of n coordinates. +func pluralItIsTheyAre(n int) string { + if n == 1 { + return "it is" + } + return "they are" +} + func applyJarDepAction(ms *types.ModuleSettings, action ast.JarDepAction, moduleName string) error { switch a := action.(type) { case *ast.AddJarDepAction: diff --git a/mdl/executor/cmd_notconnected_mock_test.go b/mdl/executor/cmd_notconnected_mock_test.go index b2bae29da..662d93b1c 100644 --- a/mdl/executor/cmd_notconnected_mock_test.go +++ b/mdl/executor/cmd_notconnected_mock_test.go @@ -93,7 +93,7 @@ func TestDescribeMermaid_Mock_NotConnected(t *testing.T) { func TestDescribeSettings_Mock_NotConnected(t *testing.T) { ctx, _ := newMockCtx(t, withBackend(disconnectedBackend())) - assertError(t, describeSettings(ctx)) + assertError(t, describeSettings(ctx, "")) } func TestDescribeBusinessEventService_Mock_NotConnected(t *testing.T) { diff --git a/mdl/executor/cmd_odata.go b/mdl/executor/cmd_odata.go index 3dbfd1934..28a0a1d89 100644 --- a/mdl/executor/cmd_odata.go +++ b/mdl/executor/cmd_odata.go @@ -392,26 +392,47 @@ func outputPublishedODataServiceMDL(ctx *ExecContext, svc *model.PublishedODataS es = entitySetByEntityName[et.Entity] } - // PUBLISH ENTITY line with modes - fmt.Fprintf(ctx.Output, " publish entity %s as '%s'", et.Entity, et.ExposedName) + // PUBLISH ENTITY line with modes. + // + // `AS ''` is the ENTITY SET's exposed name — that is what the + // served $metadata calls the set, and what re-executing this output + // must reproduce. Printing the entity TYPE's exposed name here + // silently renamed the set on a describe -> exec round trip + // (mxcli-formula1 findings #10.5). + exposedName := et.ExposedName + if es != nil && es.ExposedName != "" { + exposedName = es.ExposedName + } + fmt.Fprintf(ctx.Output, " publish entity %s as '%s'", et.Entity, exposedName) if es != nil { var modeProps []string if es.ReadMode != "" { - modeProps = append(modeProps, fmt.Sprintf("ReadMode: %s", es.ReadMode)) + modeProps = append(modeProps, fmt.Sprintf("ReadMode: %s", odataModeToMDL(es.ReadMode))) } if es.InsertMode != "" { - modeProps = append(modeProps, fmt.Sprintf("InsertMode: %s", es.InsertMode)) + modeProps = append(modeProps, fmt.Sprintf("InsertMode: %s", odataModeToMDL(es.InsertMode))) } if es.UpdateMode != "" { - modeProps = append(modeProps, fmt.Sprintf("UpdateMode: %s", es.UpdateMode)) + modeProps = append(modeProps, fmt.Sprintf("UpdateMode: %s", odataModeToMDL(es.UpdateMode))) } if es.DeleteMode != "" { - modeProps = append(modeProps, fmt.Sprintf("DeleteMode: %s", es.DeleteMode)) + modeProps = append(modeProps, fmt.Sprintf("DeleteMode: %s", odataModeToMDL(es.DeleteMode))) } if es.UsePaging { modeProps = append(modeProps, "UsePaging: Yes") modeProps = append(modeProps, fmt.Sprintf("PageSize: %d", es.PageSize)) } + // Only a turned-off query option is worth printing; true is the + // default and would be noise on every resource. + if es.Countable != nil && !*es.Countable { + modeProps = append(modeProps, "Countable: No") + } + if es.SkipSupported != nil && !*es.SkipSupported { + modeProps = append(modeProps, "SkipSupported: No") + } + if es.TopSupported != nil && !*es.TopSupported { + modeProps = append(modeProps, "TopSupported: No") + } if len(modeProps) > 0 { fmt.Fprintf(ctx.Output, " (\n %s\n )", strings.Join(modeProps, ",\n ")) } @@ -430,10 +451,17 @@ func outputPublishedODataServiceMDL(ctx *ExecContext, svc *model.PublishedODataS modifiers = append(modifiers, "Sortable") } if m.IsPartOfKey { - modifiers = append(modifiers, "IsPartOfKey") + // KEY is the spelling the syntax help documents; + // IsPartOfKey parses too, but only one belongs in + // output meant to be re-executed. + modifiers = append(modifiers, "KEY") } - line := fmt.Sprintf(" %s as '%s'", m.Name, m.ExposedName) + // The member is stored fully qualified + // (Module.Entity.Member), while `expose (...)` takes a bare + // member name — so emitting the stored form produced MDL + // that does not parse (mxcli-formula1 findings #10.5). + line := fmt.Sprintf(" %s as '%s'", bareMemberName(m.Name), m.ExposedName) if len(modifiers) > 0 { line += fmt.Sprintf(" (%s)", strings.Join(modifiers, ", ")) } @@ -1320,6 +1348,10 @@ func createODataService(ctx *ExecContext, stmt *ast.CreateODataServiceStmt) erro } if stmt.ServiceName != "" { svc.ServiceName = stmt.ServiceName + } else if svc.ServiceName == "" { + // Heal a service written before ServiceName was defaulted: + // re-running the script repairs a model that cannot build. + svc.ServiceName = svc.Name } if stmt.Summary != "" { svc.Summary = stmt.Summary @@ -1327,7 +1359,9 @@ func createODataService(ctx *ExecContext, stmt *ast.CreateODataServiceStmt) erro if stmt.Description != "" { svc.Description = stmt.Description } - svc.PublishAssociations = stmt.PublishAssociations + if stmt.PublishAssociationsSet { + svc.PublishAssociations = stmt.PublishAssociations + } if len(stmt.AuthenticationTypes) > 0 { svc.AuthenticationTypes = stmt.AuthenticationTypes } @@ -1353,6 +1387,17 @@ func createODataService(ctx *ExecContext, stmt *ast.CreateODataServiceStmt) erro containerID = folderID } + // Name (the document) and ServiceName (the name in the OData metadata + // document) are different properties, and Mendix requires the second to be + // non-empty — an empty one fails the build with CE0729 "The service name + // should not be empty", which `mxcli check` cannot see. Default it to the + // document name, exactly as the CONSUMED path does for CE0339 above. + // (mxcli-formula1 findings #10.1.) + serviceName := stmt.ServiceName + if serviceName == "" { + serviceName = stmt.Name.Name + } + newSvc := &model.PublishedODataService{ ContainerID: containerID, Name: stmt.Name.Name, @@ -1361,10 +1406,10 @@ func createODataService(ctx *ExecContext, stmt *ast.CreateODataServiceStmt) erro Version: stmt.Version, ODataVersion: stmt.ODataVersion, Namespace: stmt.Namespace, - ServiceName: stmt.ServiceName, + ServiceName: serviceName, Summary: stmt.Summary, Description: stmt.Description, - PublishAssociations: stmt.PublishAssociations, + PublishAssociations: publishAssociationsFor(stmt), AuthenticationTypes: stmt.AuthenticationTypes, } @@ -1378,6 +1423,18 @@ func createODataService(ctx *ExecContext, stmt *ast.CreateODataServiceStmt) erro newSvc.EntitySets = append(newSvc.EntitySets, entitySet) } + // An explicit false on a non-persistable entity is unbuildable whatever the + // key is: object-id mode needs a published ID, and Mendix forbids publishing + // the ID of a non-persistable entity. Say so rather than let CE7375 be the + // first anyone hears of it. + if !newSvc.PublishAssociations { + if nonPersistable := nonPersistablePublishedEntities(ctx, stmt.Entities); len(nonPersistable) > 0 { + fmt.Fprintf(ctx.Output, + " Warning: PublishAssociations is false, but %s %s non-persistable — associations-as-object-id requires a published ID, which Mendix forbids there. The build will fail with CE7375; remove the property to get the default (true).\n", + strings.Join(nonPersistable, ", "), pluralIsAre(len(nonPersistable))) + } + } + if err := ctx.Backend.CreatePublishedODataService(newSvc); err != nil { return mdlerrors.NewBackend("create OData service", err) } @@ -1722,6 +1779,9 @@ func astEntityDefToModel(ctx *ExecContext, def *ast.PublishedEntityDef) (*model. UpdateMode: def.UpdateMode, DeleteMode: def.DeleteMode, UsePaging: def.UsePaging, + Countable: def.Countable, + SkipSupported: def.SkipSupported, + TopSupported: def.TopSupported, PageSize: def.PageSize, } @@ -1782,3 +1842,87 @@ func fetchODataMetadata(metadataUrl string) (metadata string, hash string, err e } // Executor wrappers for unmigrated callers. +// nonPersistablePublishedEntities returns the qualified names of the published +// entities that are non-persistable, in statement order. Entities it cannot +// resolve are treated as persistable: this drives a silent default correction, +// so an unreadable entity must not change what gets written. +func nonPersistablePublishedEntities(ctx *ExecContext, defs []*ast.PublishedEntityDef) []string { + if ctx == nil || ctx.Backend == nil { + return nil + } + var out []string + seen := make(map[string]bool) + for _, def := range defs { + if def == nil || seen[def.Entity.String()] { + continue + } + seen[def.Entity.String()] = true + module, err := findModule(ctx, def.Entity.Module) + if err != nil { + continue + } + dm, err := ctx.Backend.GetDomainModel(module.ID) + if err != nil { + continue + } + for _, e := range dm.Entities { + if e.Name == def.Entity.Name && !e.Persistable { + out = append(out, def.Entity.String()) + break + } + } + } + return out +} + +// pluralIsAre picks the verb for a list of n names. +func pluralIsAre(n int) string { + if n == 1 { + return "is" + } + return "are" +} + +// publishAssociationsFor picks the PublishAssociations value to store. +// +// false means "expose associations as an associated object id", and Mendix then +// requires the system ID attribute to be published as the entity key — so a +// service whose key is an ordinary attribute (what MDL's `expose (Attr (KEY))` +// writes) fails the build with CE7375, and a non-persistable entity cannot +// satisfy it at all because publishing its ID is forbidden. Verified on 11.12.1: +// the identical service builds 0 errors with true and CE7375 with false, for a +// persistent entity with a unique key. +// +// Defaulting to true therefore does not pick a preference; it picks the value +// that can build from the MDL people actually write. An explicit +// `PublishAssociations: false` is still honoured — that author has published an +// ID key, or wants to know. (mxcli-formula1 findings #10.4.) +func publishAssociationsFor(stmt *ast.CreateODataServiceStmt) bool { + if !stmt.PublishAssociationsSet { + return true + } + return stmt.PublishAssociations +} + +// odataModeToMDL turns a stored Read/Change mode into the MDL spelling that +// parses back. The backend stores a microflow-backed mode as +// "CallMicroflow:Module.Name" (and accepts "MICROFLOW Module.Name" on the way +// in), but a bare `CallMicroflow:Qualified.Name` matches no MDL value — so +// DESCRIBE was emitting something it could not read (mxcli-formula1 #10.5). +func odataModeToMDL(mode string) string { + for _, prefix := range []string{"CallMicroflow:", "MICROFLOW ", "microflow "} { + if rest := strings.TrimPrefix(mode, prefix); rest != mode { + return "microflow " + strings.TrimSpace(rest) + } + } + return mode +} + +// bareMemberName strips a Module.Entity. prefix from a published member name, +// leaving the member name `expose (...)` accepts. +func bareMemberName(name string) string { + if i := strings.LastIndex(name, "."); i >= 0 { + return name[i+1:] + } + return name +} diff --git a/mdl/executor/cmd_odata_describe_roundtrip_test.go b/mdl/executor/cmd_odata_describe_roundtrip_test.go new file mode 100644 index 000000000..f536fe4f4 --- /dev/null +++ b/mdl/executor/cmd_odata_describe_roundtrip_test.go @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" +) + +// mxcli-formula1 findings #10.5: DESCRIBE emitted a form it could not parse. +// The project's review checklist asks DESCRIBE to produce re-executable MDL, so +// three separate slips each broke that: a stored mode spelling with no MDL +// equivalent, the entity TYPE's exposed name where the entity SET's belongs +// (silently renaming the set on a re-exec), and fully-qualified member names in +// an `expose (...)` clause that takes bare ones. +func TestDescribeODataService_EmitsReExecutableMDL(t *testing.T) { + mod := mkModule("F1") + svc := &model.PublishedODataService{ + BaseElement: model.BaseElement{ID: nextID("pos")}, + ContainerID: mod.ID, + Name: "LapApi", + ServiceName: "LapApi", + Path: "odata/f1/", + Version: "1.0.0", + ODataVersion: "OData4", + Namespace: "F1.Laps", + EntityTypes: []*model.PublishedEntityType{{ + Entity: "F1.Lap", + // The TYPE is exposed singular, the SET plural — Studio Pro's own + // convention, and the reason printing the wrong one is invisible + // until someone re-executes the output. + ExposedName: "Lap", + Members: []*model.PublishedMember{ + {Kind: "attribute", Name: "F1.Lap.LapKey", ExposedName: "lapKey", IsPartOfKey: true}, + {Kind: "attribute", Name: "F1.Lap.Driver", ExposedName: "driver", Filterable: true}, + }, + }}, + EntitySets: []*model.PublishedEntitySet{{ + ExposedName: "Laps", + EntityTypeName: "F1.Lap", + ReadMode: "CallMicroflow:F1.Read_Laps", + InsertMode: "NotSupported", + Countable: boolPtr(false), + }}, + } + h := mkHierarchy(mod) + withContainer(h, svc.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListPublishedODataServicesFunc: func() ([]*model.PublishedODataService, error) { + return []*model.PublishedODataService{svc}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, describeODataService(ctx, ast.QualifiedName{Module: "F1", Name: "LapApi"})) + out := buf.String() + + for _, want := range []string{ + "publish entity F1.Lap as 'Laps'", // the SET name, not the TYPE name + "ReadMode: microflow F1.Read_Laps", // not "CallMicroflow:F1.Read_Laps" + "LapKey as 'lapKey' (KEY)", // bare member, documented modifier + "driver", // the second member survives + "Countable: No", // a turned-off option is printed + } { + if !strings.Contains(out, want) { + t.Errorf("describe output should contain %q, got:\n%s", want, out) + } + } + + for _, unwanted := range []string{ + "CallMicroflow:", // parses as nothing + "F1.Lap.LapKey", // qualified member in an expose clause + "as 'Lap'", // the entity type name in the AS position + "IsPartOfKey", // parses, but KEY is the documented spelling + "SkipSupported", // unset options stay unprinted + "TopSupported", + } { + if strings.Contains(out, unwanted) { + t.Errorf("describe output should not contain %q, got:\n%s", unwanted, out) + } + } +} + +func TestODataModeToMDL(t *testing.T) { + tests := []struct{ in, want string }{ + {"CallMicroflow:M.Read", "microflow M.Read"}, + {"MICROFLOW M.Read", "microflow M.Read"}, + {"microflow M.Read", "microflow M.Read"}, + // Everything else is already an MDL spelling and must pass through. + {"source", "source"}, + {"NotSupported", "NotSupported"}, + {"", ""}, + } + for _, tt := range tests { + if got := odataModeToMDL(tt.in); got != tt.want { + t.Errorf("odataModeToMDL(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} + +func TestBareMemberName(t *testing.T) { + tests := []struct{ in, want string }{ + {"F1.Lap.LapKey", "LapKey"}, + {"Lap.LapKey", "LapKey"}, + {"LapKey", "LapKey"}, + {"", ""}, + } + for _, tt := range tests { + if got := bareMemberName(tt.in); got != tt.want { + t.Errorf("bareMemberName(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} diff --git a/mdl/executor/cmd_odata_mock_test.go b/mdl/executor/cmd_odata_mock_test.go index ebdad1cac..dd2214b15 100644 --- a/mdl/executor/cmd_odata_mock_test.go +++ b/mdl/executor/cmd_odata_mock_test.go @@ -462,7 +462,9 @@ func TestDescribeODataService_ExposeRoundtrip(t *testing.T) { assertNoError(t, describeODataService(ctx, ast.QualifiedName{Module: "MyModule", Name: "CatalogService"})) out := buf.String() - assertContainsStr(t, out, "IsPartOfKey") + // KEY, not IsPartOfKey: both parse, but DESCRIBE emits the spelling the + // syntax help documents (mxcli-formula1 #10.5). + assertContainsStr(t, out, "(KEY)") _, errs := visitor.Build(out) if len(errs) > 0 { diff --git a/mdl/executor/cmd_odata_publish_associations_test.go b/mdl/executor/cmd_odata_publish_associations_test.go new file mode 100644 index 000000000..e8a05da2b --- /dev/null +++ b/mdl/executor/cmd_odata_publish_associations_test.go @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// mxcli-formula1 findings #10.4, and wider than reported: PublishAssociations +// false means "expose associations as an associated object id", and Mendix then +// requires the system ID attribute to be published as the entity key. MDL's +// `expose (Attr (KEY))` publishes an ordinary attribute, so the default of false +// failed the build with CE7375 for EVERY published service — persistent +// entities included, not only the non-persistable case that surfaced it. +// +// Verified on 11.12.1: the identical service (persistent entity, unique key) +// builds 0 errors with true and CE7375 with false. +func TestCreateODataService_DefaultsPublishAssociationsToTrue(t *testing.T) { + for _, persistable := range []bool{true, false} { + ctx, created, _ := publishCtx(t, "Row", persistable) + if err := createODataService(ctx, publishStmt("Row")); err != nil { + t.Fatal(err) + } + if !(*created).PublishAssociations { + t.Errorf("persistable=%v: expected PublishAssociations to default to true", persistable) + } + } +} + +// An explicit false is the author's choice — they have published an ID key, or +// they want to see what Mendix says. It is written as given. +func TestCreateODataService_ExplicitPublishAssociationsFalseIsHonoured(t *testing.T) { + ctx, created, _ := publishCtx(t, "Row", true) + stmt := publishStmt("Row") + stmt.PublishAssociations = false + stmt.PublishAssociationsSet = true + if err := createODataService(ctx, stmt); err != nil { + t.Fatal(err) + } + if (*created).PublishAssociations { + t.Error("an explicit false must not be overridden by the default") + } +} + +// An explicit false over a non-persistable entity can never build, whatever the +// key is — publishing the ID of a non-persistable entity is forbidden. Warn +// rather than let CE7375 be the first anyone hears of it. +func TestCreateODataService_WarnsOnFalseWithNonPersistable(t *testing.T) { + ctx, _, buf := publishCtx(t, "Row", false) + stmt := publishStmt("Row") + stmt.PublishAssociations = false + stmt.PublishAssociationsSet = true + if err := createODataService(ctx, stmt); err != nil { + t.Fatal(err) + } + out := buf.String() + for _, want := range []string{"PublishAssociations", "Probe.Row", "non-persistable", "CE7375"} { + if !strings.Contains(out, want) { + t.Errorf("warning should mention %q, got: %s", want, out) + } + } +} + +// The warning is for the unbuildable combination only: a persistent entity with +// an explicit false is a legitimate choice and must stay quiet. +func TestCreateODataService_NoWarningForPersistentWithFalse(t *testing.T) { + ctx, _, buf := publishCtx(t, "Row", true) + stmt := publishStmt("Row") + stmt.PublishAssociations = false + stmt.PublishAssociationsSet = true + if err := createODataService(ctx, stmt); err != nil { + t.Fatal(err) + } + if strings.Contains(buf.String(), "Warning") { + t.Errorf("no warning expected for a persistent entity, got: %s", buf.String()) + } +} diff --git a/mdl/executor/cmd_odata_query_options_test.go b/mdl/executor/cmd_odata_query_options_test.go new file mode 100644 index 000000000..a7984e46d --- /dev/null +++ b/mdl/executor/cmd_odata_query_options_test.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// mxcli-formula1 findings #10.3: Countable, SkipSupported and TopSupported were +// hardcoded true in the BSON writer with no MDL to turn any of them off. That is +// not a cosmetic gap — Countable forces every read-microflow-backed resource to +// declare a System.ODataResponse parameter and compute a count, which over a +// 27533-row CSV scan is not free. +func TestPublishEntityQueryOptions_ParseAndCarry(t *testing.T) { + script := ` +create module Q; +create non-persistent entity Q.Row (K: string(20)); +create odata service Q.Api ( + path: 'odata/q/', + version: '1.0.0', + ODataVersion: OData4, + namespace: 'Q.Api' +) +authentication basic +{ + publish entity Q.Row as 'Rows' ( + ReadMode: microflow Q.Read, + Countable: No, + TopSupported: No + ) + expose ( K as 'k' (KEY) ); +}; +` + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("parsing the script: %v", errs) + } + + var def *ast.PublishedEntityDef + for _, stmt := range prog.Statements { + if s, ok := stmt.(*ast.CreateODataServiceStmt); ok && len(s.Entities) == 1 { + def = s.Entities[0] + } + } + if def == nil { + t.Fatal("expected one published entity") + } + + if def.Countable == nil || *def.Countable { + t.Errorf("Countable = %v, want an explicit false", def.Countable) + } + if def.TopSupported == nil || *def.TopSupported { + t.Errorf("TopSupported = %v, want an explicit false", def.TopSupported) + } + // Unmentioned stays nil, which the writer turns into Mendix's default of + // true — distinguishing "unset" from "false" is the whole point. + if def.SkipSupported != nil { + t.Errorf("SkipSupported = %v, want nil for an unmentioned property", *def.SkipSupported) + } + + // And the executor carries them onto the entity set it builds. + _, es := astEntityDefToModel(nil, def) + if es.Countable == nil || *es.Countable { + t.Errorf("entity set Countable = %v, want an explicit false", es.Countable) + } + if es.SkipSupported != nil { + t.Error("entity set SkipSupported should stay unset") + } +} diff --git a/mdl/executor/cmd_odata_service_name_test.go b/mdl/executor/cmd_odata_service_name_test.go new file mode 100644 index 000000000..1dfe52ae1 --- /dev/null +++ b/mdl/executor/cmd_odata_service_name_test.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// publishCtx wires a module with one entity of the given persistability, and +// returns a context plus a pointer to whatever CreatePublishedODataService is +// handed. +func publishCtx(t *testing.T, entityName string, persistable bool) (*ExecContext, **model.PublishedODataService, *bytes.Buffer) { + t.Helper() + mod := mkModule("Probe") + h := mkHierarchy(mod) + dm := &domainmodel.DomainModel{ + BaseElement: model.BaseElement{ID: nextID("dm")}, + ContainerID: mod.ID, + Entities: []*domainmodel.Entity{ + { + BaseElement: model.BaseElement{ID: nextID("ent")}, + Name: entityName, + Persistable: persistable, + Attributes: []*domainmodel.Attribute{ + {BaseElement: model.BaseElement{ID: nextID("attr")}, Name: "RowKey", Type: &domainmodel.StringAttributeType{}}, + }, + }, + }, + } + + var created *model.PublishedODataService + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + ListPublishedODataServicesFunc: func() ([]*model.PublishedODataService, error) { + return nil, nil + }, + CreatePublishedODataServiceFunc: func(svc *model.PublishedODataService) error { + created = svc + return nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + return ctx, &created, buf +} + +func publishStmt(entityName string) *ast.CreateODataServiceStmt { + return &ast.CreateODataServiceStmt{ + Name: ast.QualifiedName{Module: "Probe", Name: "ProbeApi"}, + Path: "odata/probe/", + Version: "1.0.0", + ODataVersion: "OData4", + Namespace: "Probe.Api", + Entities: []*ast.PublishedEntityDef{ + { + Entity: ast.QualifiedName{Module: "Probe", Name: entityName}, + ExposedName: "Rows", + ReadMode: "MICROFLOW Probe.Read_Rows", + }, + }, + } +} + +// mxcli-formula1 findings #10.1: Name (the document) and ServiceName (the name +// in the OData metadata document) are different properties, and CREATE only set +// the first — so every published service created purely from MDL failed the +// build with CE0729 "The service name should not be empty", which `mxcli check` +// cannot see. The CONSUMED path has defaulted this for CE0339 all along. +func TestCreateODataService_DefaultsServiceNameToDocumentName(t *testing.T) { + ctx, created, _ := publishCtx(t, "Row", true) + if err := createODataService(ctx, publishStmt("Row")); err != nil { + t.Fatal(err) + } + if *created == nil { + t.Fatal("expected the service to be created") + } + if got := (*created).ServiceName; got != "ProbeApi" { + t.Errorf("ServiceName = %q, want the document name %q", got, "ProbeApi") + } +} + +// An explicit ServiceName still wins — the default fills a gap, it does not +// override the author. +func TestCreateODataService_ExplicitServiceNameWins(t *testing.T) { + ctx, created, _ := publishCtx(t, "Row", true) + stmt := publishStmt("Row") + stmt.ServiceName = "PublicName" + if err := createODataService(ctx, stmt); err != nil { + t.Fatal(err) + } + if got := (*created).ServiceName; got != "PublicName" { + t.Errorf("ServiceName = %q, want %q", got, "PublicName") + } +} diff --git a/mdl/executor/cmd_pages_builder_inheritance_test.go b/mdl/executor/cmd_pages_builder_inheritance_test.go new file mode 100644 index 000000000..4f4c6c5bf --- /dev/null +++ b/mdl/executor/cmd_pages_builder_inheritance_test.go @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// mxcli-todo findings #12: Mendix stores a page's attribute reference against +// the entity that DECLARES the attribute. mxcli qualified it with the entity in +// context, so a binding to an inherited attribute produced a dangling reference +// and mxbuild failed with +// +// [CE1613] "The selected attribute 'TaskBoard.Person.FullName' no longer exists." +// +// Both `mxcli check` (including --references) and `mxcli lint` passed. +// +// The fixture mirrors the reporter's model: Person extends Administration.Account +// (which declares FullName/Email) and adds IsAvailable of its own; Task points at +// Person through an association. +func inheritancePB(entityContext string) *pageBuilder { + const ( + appID = model.ID("mod-app") + adminID = model.ID("mod-admin") + personID = model.ID("e-person") + taskID = model.ID("e-task") + accountID = model.ID("e-account") + ) + return &pageBuilder{ + entityContext: entityContext, + execCache: &executorCache{ + hierarchy: &ContainerHierarchy{moduleNames: map[model.ID]string{ + appID: "TaskBoard", + adminID: "Administration", + }}, + domainModels: []*domainmodel.DomainModel{ + { + ContainerID: appID, + Entities: []*domainmodel.Entity{ + { + BaseElement: model.BaseElement{ID: personID}, + Name: "Person", + GeneralizationRef: "Administration.Account", + Attributes: []*domainmodel.Attribute{ + {Name: "IsAvailable"}, + }, + }, + { + BaseElement: model.BaseElement{ID: taskID}, + Name: "Task", + Attributes: []*domainmodel.Attribute{{Name: "Title"}}, + }, + }, + CrossAssociations: []*domainmodel.CrossModuleAssociation{ + {Name: "Task_Assignee", ParentID: taskID, ChildRef: "TaskBoard.Person", Type: domainmodel.AssociationTypeReference}, + }, + Associations: []*domainmodel.Association{ + {Name: "Task_Assignee", ParentID: taskID, ChildID: personID, Type: domainmodel.AssociationTypeReference}, + }, + }, + { + ContainerID: adminID, + Entities: []*domainmodel.Entity{ + { + BaseElement: model.BaseElement{ID: accountID}, + Name: "Account", + Attributes: []*domainmodel.Attribute{ + {Name: "FullName"}, + {Name: "Email"}, + }, + }, + }, + }, + }, + }, + } +} + +func TestResolveAttributePath_InheritedAttribute(t *testing.T) { + pb := inheritancePB("TaskBoard.Person") + + // Inherited: must be qualified with the declaring entity, not the context. + if got := pb.resolveAttributePath("FullName"); got != "Administration.Account.FullName" { + t.Errorf("inherited attribute: got %q, want Administration.Account.FullName", got) + } + // Own: unchanged. + if got := pb.resolveAttributePath("IsAvailable"); got != "TaskBoard.Person.IsAvailable" { + t.Errorf("own attribute: got %q, want TaskBoard.Person.IsAvailable", got) + } + // Unknown name: no invented qualification, today's behaviour is kept. + if got := pb.resolveAttributePath("Nonexistent"); got != "TaskBoard.Person.Nonexistent" { + t.Errorf("unknown attribute: got %q, want the context qualification", got) + } + // Already qualified: untouched. + if got := pb.resolveAttributePath("Other.Entity.Attr"); got != "Other.Entity.Attr" { + t.Errorf("qualified attribute was rewritten: %q", got) + } +} + +// The same rule applies to the final attribute of an association path — the +// reporter's table had this failing too, and it goes through a different +// resolver. +func TestResolveAssociationAttributePath_InheritedFinalAttribute(t *testing.T) { + pb := inheritancePB("TaskBoard.Task") + + finalQN, steps, ok := pb.resolveAssociationAttributePath("Task_Assignee/FullName") + if !ok { + t.Fatal("expected the association path to resolve") + } + if finalQN != "Administration.Account.FullName" { + t.Errorf("inherited final attribute: got %q, want Administration.Account.FullName", finalQN) + } + if len(steps) != 1 || steps[0].DestinationEntity != "TaskBoard.Person" { + t.Errorf("steps = %+v, want one hop to TaskBoard.Person", steps) + } + + // An own attribute on the same destination keeps the destination entity. + finalQN, _, ok = pb.resolveAssociationAttributePath("Task_Assignee/IsAvailable") + if !ok || finalQN != "TaskBoard.Person.IsAvailable" { + t.Errorf("own final attribute: got %q (ok=%v), want TaskBoard.Person.IsAvailable", finalQN, ok) + } +} + +func TestDeclaringEntityFor(t *testing.T) { + pb := inheritancePB("TaskBoard.Person") + + tests := []struct { + entity, attr string + want string + wantOK bool + }{ + {"TaskBoard.Person", "IsAvailable", "TaskBoard.Person", true}, + {"TaskBoard.Person", "FullName", "Administration.Account", true}, + {"TaskBoard.Person", "Email", "Administration.Account", true}, + // Case-insensitive, since MDL identifiers are matched that way elsewhere. + {"TaskBoard.Person", "fullname", "Administration.Account", true}, + {"TaskBoard.Person", "Missing", "", false}, + {"Unknown.Entity", "Attr", "", false}, + {"", "Attr", "", false}, + {"TaskBoard.Person", "", "", false}, + } + for _, tt := range tests { + got, ok := pb.declaringEntityFor(tt.entity, tt.attr) + if got != tt.want || ok != tt.wantOK { + t.Errorf("declaringEntityFor(%q, %q) = (%q, %v), want (%q, %v)", + tt.entity, tt.attr, got, ok, tt.want, tt.wantOK) + } + } +} + +// A generalization cycle must not hang the walk. Mendix cannot express one, but +// a corrupt or partially-written model could. +func TestDeclaringEntityFor_CycleTerminates(t *testing.T) { + const modID = model.ID("mod") + pb := &pageBuilder{ + execCache: &executorCache{ + hierarchy: &ContainerHierarchy{moduleNames: map[model.ID]string{modID: "M"}}, + domainModels: []*domainmodel.DomainModel{{ + ContainerID: modID, + Entities: []*domainmodel.Entity{ + {Name: "A", GeneralizationRef: "M.B"}, + {Name: "B", GeneralizationRef: "M.A"}, + }, + }}, + }, + } + if _, ok := pb.declaringEntityFor("M.A", "Whatever"); ok { + t.Error("expected no declaring entity for a cyclic chain") + } +} diff --git a/mdl/executor/cmd_pages_builder_input.go b/mdl/executor/cmd_pages_builder_input.go index 65ecfaef1..73da8ba11 100644 --- a/mdl/executor/cmd_pages_builder_input.go +++ b/mdl/executor/cmd_pages_builder_input.go @@ -40,13 +40,89 @@ func (pb *pageBuilder) resolveAttributePath(attr string) string { if strings.Contains(attr, ".") { return attr } - // If we have an entity context, prefix the attribute with it + // If we have an entity context, prefix the attribute with it — but with the + // entity that actually DECLARES it, which for an inherited attribute is an + // ancestor rather than the context entity itself. if pb.entityContext != "" { + if declaring, ok := pb.declaringEntityFor(pb.entityContext, attr); ok { + return declaring + "." + attr + } return pb.entityContext + "." + attr } return attr } +// declaringEntityFor returns the entity in entityQN's generalization chain that +// declares attrName — entityQN itself when the attribute is its own. +// +// Mendix stores a page's attribute reference against the declaring entity. A +// reference qualified with a specialization that merely inherits the attribute +// is dangling, and the build fails with +// +// [CE1613] "The selected attribute 'Module.Sub.Attr' no longer exists." +// +// which reads as if the attribute had been deleted; it never existed there. +// Entity access rules resolve inherited members correctly, so this was the page +// layer alone. (mxcli-todo findings #12) +// +// ok is false when nothing in the chain declares the name — an unknown +// attribute, or a domain model we cannot read. The caller then keeps today's +// behaviour rather than inventing a qualification. +func (pb *pageBuilder) declaringEntityFor(entityQN, attrName string) (string, bool) { + if entityQN == "" || attrName == "" { + return "", false + } + // Resolution is best-effort: without a model to consult (no backend and + // nothing cached — e.g. a unit test building widgets in isolation) keep the + // caller's plain context qualification rather than panicking on the lookup. + if pb.backend == nil && (pb.execCache == nil || pb.execCache.domainModels == nil) { + return "", false + } + owners, parents, err := pb.entityAttributeOwners() + if err != nil { + return "", false + } + lower := strings.ToLower(attrName) + seen := map[string]bool{} + for cur := entityQN; cur != "" && !seen[cur]; cur = parents[cur] { + seen[cur] = true + if attrs, ok := owners[cur]; ok && attrs[lower] { + return cur, true + } + } + return "", false +} + +// entityAttributeOwners indexes, for every entity in the project, the set of +// attribute names it declares itself, plus each entity's direct parent. +func (pb *pageBuilder) entityAttributeOwners() (owners map[string]map[string]bool, parents map[string]string, err error) { + dms, err := pb.getDomainModels() + if err != nil { + return nil, nil, err + } + h, err := pb.getHierarchy() + if err != nil { + return nil, nil, err + } + owners = make(map[string]map[string]bool, len(dms)*8) + parents = make(map[string]string) + for _, dm := range dms { + mod := h.GetModuleName(dm.ContainerID) + for _, e := range dm.Entities { + qn := mod + "." + e.Name + attrs := make(map[string]bool, len(e.Attributes)) + for _, a := range e.Attributes { + attrs[strings.ToLower(a.Name)] = true + } + owners[qn] = attrs + if e.GeneralizationRef != "" { + parents[qn] = e.GeneralizationRef + } + } + } + return owners, parents, nil +} + // systemMemberBindingNames maps the name an audit member is DECLARED under to // the name Mendix actually stores it as. // diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index c09f0f4e5..e2409483e 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -1681,7 +1681,14 @@ func (pb *pageBuilder) resolveAssociationAttributePath(attrRef string) (finalQN current = dest } - return current + "." + storedSystemMemberName(attrName), steps, true + // The final attribute is qualified with the entity that DECLARES it, which + // for an inherited attribute is an ancestor of the association's destination + // — same rule (and same CE1613 when broken) as a direct binding. + stored := storedSystemMemberName(attrName) + if declaring, ok := pb.declaringEntityFor(current, stored); ok { + return declaring + "." + stored, steps, true + } + return current + "." + stored, steps, true } // associationDestination returns the entity reached by navigating assocQN from diff --git a/mdl/executor/cmd_security_demo_user_warning_test.go b/mdl/executor/cmd_security_demo_user_warning_test.go new file mode 100644 index 000000000..e39da330a --- /dev/null +++ b/mdl/executor/cmd_security_demo_user_warning_test.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/sdk/security" +) + +// mxcli-todo findings #15: with Security Level Off — what a blank mxcli template +// ships with — the runtime creates no accounts, so demo users sit in the model +// and never appear in the app. CREATE DEMO USER reported success and SHOW +// PROJECT SECURITY said "Demo Users Enabled: true", with nothing to explain the +// empty Administration.Account table. +func TestWarnDemoUsersInert(t *testing.T) { + tests := []struct { + level string + wantWarn bool + }{ + {security.SecurityLevelOff, true}, + {security.SecurityLevelPrototype, false}, + {security.SecurityLevelProduction, false}, + } + for _, tt := range tests { + var buf bytes.Buffer + warnDemoUsersInert(&ExecContext{Output: &buf}, tt.level) + got := buf.String() + if tt.wantWarn { + if !strings.Contains(got, "security level is Off") { + t.Errorf("level %q: expected a warning, got %q", tt.level, got) + } + if !strings.Contains(got, "alter project security level prototype") { + t.Errorf("level %q: the warning must name the fix, got %q", tt.level, got) + } + } else if got != "" { + t.Errorf("level %q: expected no warning, got %q", tt.level, got) + } + } +} diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 24409c306..3eb8fdbed 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -1132,9 +1132,27 @@ func execCreateDemoUser(ctx *ExecContext, s *ast.CreateDemoUserStmt) error { } fmt.Fprintf(ctx.Output, "Created demo user: %s (entity: %s)\n", s.UserName, entity) + warnDemoUsersInert(ctx, ps.SecurityLevel) return nil } +// warnDemoUsersInert says so when demo users cannot materialise. +// +// With Security Level Off — the level a blank mxcli template ships with — the +// runtime creates no accounts, serves no login page and enforces none of the +// row-level rules, so the demo users sit in the model and never appear. Nothing +// said this: `CREATE DEMO USER` reported success and `SHOW PROJECT SECURITY` +// reported "Demo Users Enabled: true", while the running app had zero accounts. +// (mxcli-todo findings #15) +func warnDemoUsersInert(ctx *ExecContext, level string) { + if level != security.SecurityLevelOff { + return + } + fmt.Fprintf(ctx.Output, " Note: project security level is Off, so the runtime creates no accounts "+ + "and this demo user will not appear in the app.\n"+ + " Raise it first: alter project security level prototype;\n") +} + // detectUserEntity finds the entity that generalizes System.User. // storedAuditMembers returns the audit members the entity actually stores, under // the names Mendix uses for them. They are entity FLAGS rather than entries in diff --git a/mdl/executor/cmd_settings.go b/mdl/executor/cmd_settings.go index 7dc728d71..f7a990a0a 100644 --- a/mdl/executor/cmd_settings.go +++ b/mdl/executor/cmd_settings.go @@ -44,9 +44,19 @@ func listSettings(ctx *ExecContext) error { for _, cfg := range ps.Configuration.Configurations { values := []string{} values = append(values, cfg.DatabaseType) - values = append(values, cfg.DatabaseUrl) + // An empty DatabaseUrl used to render as a bare ", ," — a gap where + // a value should be, which reads as a bug in the reader. + if cfg.DatabaseUrl != "" { + values = append(values, cfg.DatabaseUrl) + } values = append(values, "db="+cfg.DatabaseName) values = append(values, fmt.Sprintf("http=%d", cfg.HttpPortNumber)) + // The root URL decides the host the app answers on, so "did my root + // URL land?" is the obvious question to ask this command — and the + // summary used to be unable to answer it (mxcli-formula1 #8). + if cfg.ApplicationRootUrl != "" { + values = append(values, "url="+cfg.ApplicationRootUrl) + } if len(cfg.ConstantValues) > 0 { values = append(values, fmt.Sprintf("%d constants", len(cfg.ConstantValues))) } @@ -85,7 +95,11 @@ func listSettings(ctx *ExecContext) error { } // describeSettings outputs the full MDL description of all settings. -func describeSettings(ctx *ExecContext) error { +// describeSettings prints the project settings as re-executable `alter settings` +// statements. With configName set (DESCRIBE SETTINGS CONFIGURATION 'X') it +// prints only that configuration — the read form of `alter settings +// configuration 'X'`, which used to be a parse error. +func describeSettings(ctx *ExecContext, configName string) error { if !ctx.Connected() { return mdlerrors.NewNotConnected() } @@ -95,6 +109,10 @@ func describeSettings(ctx *ExecContext) error { return mdlerrors.NewBackend("read project settings", err) } + if configName != "" { + return describeSettingsConfiguration(ctx, ps, configName) + } + // Model settings if ps.Model != nil { ms := ps.Model @@ -122,34 +140,7 @@ func describeSettings(ctx *ExecContext) error { // Configuration settings if ps.Configuration != nil { for _, cfg := range ps.Configuration.Configurations { - var parts []string - parts = append(parts, fmt.Sprintf(" DatabaseType = '%s'", cfg.DatabaseType)) - parts = append(parts, fmt.Sprintf(" DatabaseUrl = '%s'", cfg.DatabaseUrl)) - parts = append(parts, fmt.Sprintf(" DatabaseName = '%s'", cfg.DatabaseName)) - parts = append(parts, fmt.Sprintf(" DatabaseUserName = '%s'", cfg.DatabaseUserName)) - parts = append(parts, fmt.Sprintf(" DatabasePassword = '%s'", cfg.DatabasePassword)) - parts = append(parts, fmt.Sprintf(" HttpPortNumber = %d", cfg.HttpPortNumber)) - parts = append(parts, fmt.Sprintf(" ServerPortNumber = %d", cfg.ServerPortNumber)) - if cfg.ApplicationRootUrl != "" { - parts = append(parts, fmt.Sprintf(" ApplicationRootUrl = '%s'", cfg.ApplicationRootUrl)) - } - fmt.Fprintf(ctx.Output, "alter settings configuration '%s'\n%s;\n\n", cfg.Name, strings.Join(parts, ",\n")) - - // Output constant overrides. A private override has no value in the - // model — emitting `value ''` would round-trip into a *shared* empty - // override, moving a value that is deliberately kept off the shared model - // into it. MDL does not author the shared/private choice, so describe - // reports it as a comment instead of a re-executable statement. - for _, cv := range cfg.ConstantValues { - if cv.IsPrivate { - fmt.Fprintf(ctx.Output, "-- constant '%s' has a private value in configuration '%s'\n"+ - "-- (stored on the developer's workstation; not part of the shared model)\n\n", - cv.ConstantId, cfg.Name) - continue - } - fmt.Fprintf(ctx.Output, "alter settings constant '%s' value '%s'\n in configuration '%s';\n\n", - cv.ConstantId, cv.Value, cfg.Name) - } + writeSettingsConfiguration(ctx, cfg) } } @@ -645,3 +636,52 @@ func settingsValueToString(val any) string { return fmt.Sprintf("%v", v) } } + +// writeSettingsConfiguration emits one configuration as re-executable MDL. +func writeSettingsConfiguration(ctx *ExecContext, cfg *model.ServerConfiguration) { + var parts []string + parts = append(parts, fmt.Sprintf(" DatabaseType = '%s'", cfg.DatabaseType)) + parts = append(parts, fmt.Sprintf(" DatabaseUrl = '%s'", cfg.DatabaseUrl)) + parts = append(parts, fmt.Sprintf(" DatabaseName = '%s'", cfg.DatabaseName)) + parts = append(parts, fmt.Sprintf(" DatabaseUserName = '%s'", cfg.DatabaseUserName)) + parts = append(parts, fmt.Sprintf(" DatabasePassword = '%s'", cfg.DatabasePassword)) + parts = append(parts, fmt.Sprintf(" HttpPortNumber = %d", cfg.HttpPortNumber)) + parts = append(parts, fmt.Sprintf(" ServerPortNumber = %d", cfg.ServerPortNumber)) + if cfg.ApplicationRootUrl != "" { + parts = append(parts, fmt.Sprintf(" ApplicationRootUrl = '%s'", cfg.ApplicationRootUrl)) + } + fmt.Fprintf(ctx.Output, "alter settings configuration '%s'\n%s;\n\n", cfg.Name, strings.Join(parts, ",\n")) + + // Output constant overrides. A private override has no value in the + // model — emitting `value ''` would round-trip into a *shared* empty + // override, moving a value that is deliberately kept off the shared model + // into it. MDL does not author the shared/private choice, so describe + // reports it as a comment instead of a re-executable statement. + for _, cv := range cfg.ConstantValues { + if cv.IsPrivate { + fmt.Fprintf(ctx.Output, "-- constant '%s' has a private value in configuration '%s'\n"+ + "-- (stored on the developer's workstation; not part of the shared model)\n\n", + cv.ConstantId, cfg.Name) + continue + } + fmt.Fprintf(ctx.Output, "alter settings constant '%s' value '%s'\n in configuration '%s';\n\n", + cv.ConstantId, cv.Value, cfg.Name) + } +} + +// describeSettingsConfiguration prints a single named configuration, or names +// the ones that exist when the requested name is not among them. +func describeSettingsConfiguration(ctx *ExecContext, ps *model.ProjectSettings, name string) error { + var available []string + if ps.Configuration != nil { + for _, cfg := range ps.Configuration.Configurations { + if strings.EqualFold(cfg.Name, name) { + writeSettingsConfiguration(ctx, cfg) + return nil + } + available = append(available, "'"+cfg.Name+"'") + } + } + return mdlerrors.NewNotFoundMsg("settings configuration", name, + fmt.Sprintf("settings configuration not found: '%s' (available: %s)", name, strings.Join(available, ", "))) +} diff --git a/mdl/executor/cmd_settings_configuration_test.go b/mdl/executor/cmd_settings_configuration_test.go new file mode 100644 index 000000000..bdf5bb77b --- /dev/null +++ b/mdl/executor/cmd_settings_configuration_test.go @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" +) + +// mxcli-formula1 findings #8: `alter settings configuration 'X'` is the write +// form, but the read form was a parse error, and the `show` summary omitted +// ApplicationRootUrl — so the obvious command for "did my root URL land?" could +// not answer it. +func settingsCtxWithConfigs(t *testing.T) *ExecContext { + t.Helper() + ps := &model.ProjectSettings{ + Configuration: &model.ConfigurationSettings{ + Configurations: []*model.ServerConfiguration{ + { + Name: "Default", DatabaseType: "Hsqldb", DatabaseName: "default", + HttpPortNumber: 8080, ApplicationRootUrl: "http://backend.local:8080/", + }, + {Name: "Test", DatabaseType: "PostgreSQL", DatabaseName: "test", HttpPortNumber: 8180}, + }, + }, + } + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetProjectSettingsFunc: func() (*model.ProjectSettings, error) { return ps, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + return ctx +} + +func TestDescribeSettingsConfiguration_ByName(t *testing.T) { + ctx := settingsCtxWithConfigs(t) + buf := ctx.Output.(*bytes.Buffer) + if err := describeSettings(ctx, "Default"); err != nil { + t.Fatal(err) + } + out := buf.String() + + if !strings.Contains(out, "alter settings configuration 'Default'") { + t.Errorf("expected the named configuration, got:\n%s", out) + } + if !strings.Contains(out, "ApplicationRootUrl = 'http://backend.local:8080/'") { + t.Errorf("expected the root URL, got:\n%s", out) + } + // Naming one configuration means one configuration, not all of them. + if strings.Contains(out, "'Test'") { + t.Errorf("expected only the named configuration, got:\n%s", out) + } +} + +// Case-insensitive, like the rest of MDL's name matching. +func TestDescribeSettingsConfiguration_CaseInsensitive(t *testing.T) { + ctx := settingsCtxWithConfigs(t) + if err := describeSettings(ctx, "default"); err != nil { + t.Fatalf("expected a case-insensitive match: %v", err) + } +} + +// A wrong name has to say which ones exist, or the user is guessing. +func TestDescribeSettingsConfiguration_UnknownNameListsAvailable(t *testing.T) { + ctx := settingsCtxWithConfigs(t) + err := describeSettings(ctx, "Nope") + if err == nil { + t.Fatal("expected an error for an unknown configuration") + } + for _, want := range []string{"Nope", "Default", "Test"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got: %v", want, err) + } + } +} + +// No name still prints everything, as before. +func TestDescribeSettings_AllConfigurations(t *testing.T) { + ctx := settingsCtxWithConfigs(t) + buf := ctx.Output.(*bytes.Buffer) + if err := describeSettings(ctx, ""); err != nil { + t.Fatal(err) + } + out := buf.String() + for _, want := range []string{"'Default'", "'Test'"} { + if !strings.Contains(out, want) { + t.Errorf("expected every configuration, %q missing from:\n%s", want, out) + } + } +} + +func TestShowSettings_SummaryCarriesRootURL(t *testing.T) { + ctx := settingsCtxWithConfigs(t) + buf := ctx.Output.(*bytes.Buffer) + if err := listSettings(ctx); err != nil { + t.Fatal(err) + } + out := buf.String() + if !strings.Contains(out, "url=http://backend.local:8080/") { + t.Errorf("summary should carry the root URL, got:\n%s", out) + } + // An empty DatabaseUrl leaves a gap where a value should be; it is omitted + // rather than rendered as a bare comma. + if strings.Contains(out, "Hsqldb, , ") { + t.Errorf("summary should skip an empty DatabaseUrl, got:\n%s", out) + } +} diff --git a/mdl/executor/cmd_settings_mock_test.go b/mdl/executor/cmd_settings_mock_test.go index d33aecaaa..be138f14f 100644 --- a/mdl/executor/cmd_settings_mock_test.go +++ b/mdl/executor/cmd_settings_mock_test.go @@ -44,7 +44,7 @@ func TestDescribeSettings_Mock(t *testing.T) { }, } ctx, buf := newMockCtx(t, withBackend(mb)) - assertNoError(t, describeSettings(ctx)) + assertNoError(t, describeSettings(ctx, "")) assertContainsStr(t, buf.String(), "alter settings") } @@ -61,7 +61,7 @@ func TestDescribeSettings_NotConnected(t *testing.T) { IsConnectedFunc: func() bool { return false }, } ctx, _ := newMockCtx(t, withBackend(mb)) - assertError(t, describeSettings(ctx)) + assertError(t, describeSettings(ctx, "")) } func TestShowSettings_BackendError(t *testing.T) { diff --git a/mdl/executor/cmd_settings_private_test.go b/mdl/executor/cmd_settings_private_test.go index facacf18a..3a1677775 100644 --- a/mdl/executor/cmd_settings_private_test.go +++ b/mdl/executor/cmd_settings_private_test.go @@ -110,7 +110,7 @@ func TestDescribeSettings_PrivateOverrideIsNotReExecutable(t *testing.T) { wrote := false ctx, out := newMockCtx(t, withBackend(privateConstantBackend(&wrote))) - if err := describeSettings(ctx); err != nil { + if err := describeSettings(ctx, ""); err != nil { t.Fatalf("describeSettings: %v", err) } got := out.String() diff --git a/mdl/executor/executor_query.go b/mdl/executor/executor_query.go index 86e6a45db..b7bcaab41 100644 --- a/mdl/executor/executor_query.go +++ b/mdl/executor/executor_query.go @@ -226,7 +226,7 @@ func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { case ast.DescribeDatabaseConnection: return describeDatabaseConnection(ctx, s.Name) case ast.DescribeSettings: - return describeSettings(ctx) + return describeSettings(ctx, s.Qualifier) case ast.DescribeFragment: return describeFragment(ctx, s.Name) case ast.DescribeImageCollection: diff --git a/mdl/executor/validate_database_type.go b/mdl/executor/validate_database_type.go new file mode 100644 index 000000000..4b63a37d9 --- /dev/null +++ b/mdl/executor/validate_database_type.go @@ -0,0 +1,78 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation of CREATE DATABASE CONNECTION's TYPE. +// +// mxcli writes the type string straight through to BSON, and mxbuild accepts +// anything: `type 'Redshift'` builds 0 errors on 11.12.1 and is simply not a +// database type Mendix has. The values are the ones Studio Pro's own connector +// editor offers, read out of the shipped bundle at +// modeler/ide-client/database-connector-editor/ (verified identical on 11.10.0, +// 11.12.1 and — per mxcli-formula1 findings #6 — 11.13.0). +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// databaseConnectionTypes is Studio Pro's picker, id -> label. +var databaseConnectionTypes = []struct{ ID, Label string }{ + {"MSSQL", "Microsoft SQL"}, + {"MySQL", "MySQL"}, + {"Oracle", "Oracle"}, + {"PostgreSQL", "PostgreSQL"}, + {"Snowflake", "Snowflake"}, + {"BYOD", "Other — bring your own JDBC driver"}, +} + +// ValidateDatabaseConnectionType warns (MDL-DB01) when a CREATE DATABASE +// CONNECTION names a type Studio Pro does not offer. +// +// A warning rather than an error: the set is version-specific, and mxbuild does +// not reject an unknown value, so mxcli cannot prove one wrong on a Mendix +// version it has not seen. Saying nothing is worse — the build is green and the +// connection simply does not work. +func ValidateDatabaseConnectionType(prog *ast.Program) []linter.Violation { + if prog == nil { + return nil + } + var out []linter.Violation + for _, stmt := range prog.Statements { + s, ok := stmt.(*ast.CreateDatabaseConnectionStmt) + if !ok || s.DatabaseType == "" { + continue + } + if knownDatabaseConnectionType(s.DatabaseType) { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL-DB01", + Severity: linter.SeverityWarning, + Message: fmt.Sprintf("database connection %s: type %q is not one Studio Pro offers — it is written to the model as-is and mxbuild does not check it, so the build stays green and the connection does not work", + s.Name.String(), s.DatabaseType), + Suggestion: fmt.Sprintf("Use one of: %s. For a JDBC driver Mendix has no entry for, use 'BYOD' — it skips the driver-presence check and takes the connection string as given.", + strings.Join(databaseConnectionTypeIDs(), ", ")), + }) + } + return out +} + +func knownDatabaseConnectionType(name string) bool { + for _, t := range databaseConnectionTypes { + if strings.EqualFold(t.ID, name) { + return true + } + } + return false +} + +func databaseConnectionTypeIDs() []string { + ids := make([]string, 0, len(databaseConnectionTypes)) + for _, t := range databaseConnectionTypes { + ids = append(ids, "'"+t.ID+"'") + } + return ids +} diff --git a/mdl/executor/validate_database_type_test.go b/mdl/executor/validate_database_type_test.go new file mode 100644 index 000000000..7d7510562 --- /dev/null +++ b/mdl/executor/validate_database_type_test.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// mxcli-formula1 findings #6: the skill documented `Redshift` (not in Studio +// Pro's picker on any version checked) and omitted `BYOD` (the only way to use +// a JDBC driver Mendix ships no entry for). mxcli writes the string straight +// through and mxbuild does not check it — `type 'Redshift'` builds 0 errors on +// 11.12.1 — so a wrong value hides behind a green build. +func TestValidateDatabaseConnectionType(t *testing.T) { + script := func(dbType string) string { + return ` +create module D; +create constant D.Cs type string default 'jdbc:x'; +create constant D.U type string default 'u'; +create constant D.P type string default 'p'; +create database connection D.Conn +type '` + dbType + `' +connection string @D.Cs +username @D.U +password @D.P +begin +end; +` + } + + tests := []struct { + dbType string + wantWarn bool + }{ + {"PostgreSQL", false}, + {"MSSQL", false}, + {"Oracle", false}, + {"MySQL", false}, + {"Snowflake", false}, + {"BYOD", false}, + // Matching is case-insensitive: the picker's id is the canonical + // spelling, but a different casing is not a different type. + {"postgresql", false}, + // Both of these were in the skill's table and neither is real. + {"Redshift", true}, + {"SQLServer", true}, + {"Postgres", true}, + } + + for _, tt := range tests { + t.Run(tt.dbType, func(t *testing.T) { + prog, errs := visitor.Build(script(tt.dbType)) + if len(errs) > 0 { + t.Fatalf("parsing the script: %v", errs) + } + got := ValidateDatabaseConnectionType(prog) + if tt.wantWarn { + if len(got) != 1 { + t.Fatalf("expected 1 warning for %q, got %d: %v", tt.dbType, len(got), got) + } + if got[0].RuleID != "MDL-DB01" { + t.Errorf("rule = %q, want MDL-DB01", got[0].RuleID) + } + // A warning, not an error: the set is version-specific and + // mxcli cannot prove a value wrong on a version it has not seen. + if got[0].Severity != linter.SeverityWarning { + t.Errorf("severity = %v, want warning", got[0].Severity) + } + // BYOD is the way out for a driver Mendix has no entry for, so + // the suggestion has to name it. + if !strings.Contains(got[0].Suggestion, "BYOD") { + t.Errorf("suggestion should point at BYOD, got: %s", got[0].Suggestion) + } + } else if len(got) != 0 { + t.Errorf("expected no warning for %q, got: %v", tt.dbType, got) + } + }) + } +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index a765a106e..c05dee69a 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -88,6 +88,11 @@ func (v *microflowValidator) validate(body []ast.MicroflowStatement) { // Duplicate loop iterator names — a Mendix loop variable is scoped to the whole // microflow, so reusing a name across loops is CE0111 at build time. v.checkDuplicateLoopVariables(body) + + // The other half of that rule: names are unique flow-wide, but a loop's + // variables are only VISIBLE inside its body, so using one after the loop + // is CE0108. + v.checkLoopScoping(body) } // checkDuplicateLoopVariables flags a loop iterator name used by more than one diff --git a/mdl/executor/validate_microflow_loop_scope.go b/mdl/executor/validate_microflow_loop_scope.go new file mode 100644 index 000000000..ef95ee682 --- /dev/null +++ b/mdl/executor/validate_microflow_loop_scope.go @@ -0,0 +1,356 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// checkLoopScoping flags a reference, from outside a loop, to a variable that +// only exists inside it — the loop iterator itself, or anything the loop body +// introduces (a retrieve, a create, a call output, …). +// +// Mendix scopes both to the loop body, so the reference builds as +// +// [CE0108] "Variable 'X' is defined but not in scope at this location." +// +// even though the flow is otherwise well-formed. MDL052 is the sibling rule for +// the other half of Mendix's loop-variable semantics: names must be unique +// across the WHOLE microflow (CE0111), while visibility stops at the loop body. +// +// Only names owned by a loop are considered, and any name that is also +// introduced outside a loop is dropped from the set first — so a re-declared +// name can never be reported. +func (v *microflowValidator) checkLoopScoping(body []ast.MicroflowStatement) { + owner := map[string]*ast.LoopStmt{} + ambiguous := map[string]bool{} + collectLoopScopedVars(body, owner, ambiguous) + if len(owner) == 0 { + return + } + + // A name claimed by two loops has no single owning body to judge references + // against — and it is already MDL052/CE0111 ("Duplicate variable name"). + for name := range ambiguous { + delete(owner, name) + } + + // A name introduced outside any loop is in scope after the loop regardless + // of what the loop does with it; never report those. + for name := range collectNonLoopDeclaredVars(body) { + delete(owner, name) + } + if len(owner) == 0 { + return + } + + v.walkLoopScope(body, map[*ast.LoopStmt]bool{}, owner, map[string]bool{}) +} + +// walkLoopScope walks the body with the set of loops whose bodies enclose the +// current position, reporting each out-of-scope name once. +func (v *microflowValidator) walkLoopScope( + body []ast.MicroflowStatement, + active map[*ast.LoopStmt]bool, + owner map[string]*ast.LoopStmt, + reported map[string]bool, +) { + for _, s := range body { + for _, name := range loopRefVars(s) { + if name == "" || reported[name] { + continue + } + loop, ok := owner[name] + if !ok || active[loop] { + continue + } + reported[name] = true + v.addViolation("MDL053", linter.SeverityError, + fmt.Sprintf("variable '$%s' is used outside the loop over '$%s' that defines it; "+ + "a Mendix loop variable — the iterator and anything the loop body creates — "+ + "is scoped to the loop body, so this builds as CE0108 "+ + "\"Variable '%s' is defined but not in scope at this location\"", + name, loop.ListVariable, name), + fmt.Sprintf("Move the statement into the loop body, or carry the value out "+ + "in a variable declared before the loop (e.g. 'declare $Result …' then "+ + "'set $Result = $%s' inside the loop)", name)) + } + + // Recurse into nested bodies, extending the active set for loops. + switch st := s.(type) { + case *ast.LoopStmt: + inner := make(map[*ast.LoopStmt]bool, len(active)+1) + for l := range active { + inner[l] = true + } + inner[st] = true + v.walkLoopScope(st.Body, inner, owner, reported) + continue + case *ast.WhileStmt: + v.walkLoopScope(st.Body, active, owner, reported) + case *ast.IfStmt: + v.walkLoopScope(st.ThenBody, active, owner, reported) + v.walkLoopScope(st.ElseBody, active, owner, reported) + case *ast.EnumSplitStmt: + for _, c := range st.Cases { + v.walkLoopScope(c.Body, active, owner, reported) + } + v.walkLoopScope(st.ElseBody, active, owner, reported) + case *ast.InheritanceSplitStmt: + for _, c := range st.Cases { + v.walkLoopScope(c.Body, active, owner, reported) + } + v.walkLoopScope(st.ElseBody, active, owner, reported) + } + + if eh := stmtErrorHandling(s); eh != nil && len(eh.Body) > 0 { + v.walkLoopScope(eh.Body, active, owner, reported) + } + } +} + +// collectLoopScopedVars maps every loop-scoped variable name to the loop whose +// body introduces it. Each loop claims only the names in its OWN body — a +// nested loop's names belong to the nested loop — so a name that still ends up +// claimed twice is a genuine duplicate (two sibling loops reusing an iterator); +// those go into ambiguous and are not reported here. +func collectLoopScopedVars(body []ast.MicroflowStatement, owner map[string]*ast.LoopStmt, ambiguous map[string]bool) { + claim := func(name string, loop *ast.LoopStmt) { + if name == "" { + return + } + if prev, seen := owner[name]; seen && prev != loop { + ambiguous[name] = true + } + owner[name] = loop + } + + for _, s := range body { + switch st := s.(type) { + case *ast.LoopStmt: + claim(st.LoopVariable, st) + for name := range declaredVarsOwnScope(st.Body) { + claim(name, st) + } + collectLoopScopedVars(st.Body, owner, ambiguous) + case *ast.WhileStmt: + collectLoopScopedVars(st.Body, owner, ambiguous) + case *ast.IfStmt: + collectLoopScopedVars(st.ThenBody, owner, ambiguous) + collectLoopScopedVars(st.ElseBody, owner, ambiguous) + case *ast.EnumSplitStmt: + for _, c := range st.Cases { + collectLoopScopedVars(c.Body, owner, ambiguous) + } + collectLoopScopedVars(st.ElseBody, owner, ambiguous) + case *ast.InheritanceSplitStmt: + for _, c := range st.Cases { + collectLoopScopedVars(c.Body, owner, ambiguous) + } + collectLoopScopedVars(st.ElseBody, owner, ambiguous) + } + if eh := stmtErrorHandling(s); eh != nil && len(eh.Body) > 0 { + collectLoopScopedVars(eh.Body, owner, ambiguous) + } + } +} + +// collectNonLoopDeclaredVars returns the names introduced anywhere OUTSIDE a +// loop body (branches and error handlers included — those are a different +// scoping question, covered by MDL005). +func collectNonLoopDeclaredVars(body []ast.MicroflowStatement) map[string]bool { + vars := map[string]bool{} + var walk func([]ast.MicroflowStatement) + walk = func(stmts []ast.MicroflowStatement) { + for _, s := range stmts { + for name := range collectDeclaredVars([]ast.MicroflowStatement{s}) { + vars[name] = true + } + switch st := s.(type) { + case *ast.LoopStmt: + // Deliberately not descended into: those names are loop-scoped. + case *ast.WhileStmt: + walk(st.Body) + case *ast.IfStmt: + walk(st.ThenBody) + walk(st.ElseBody) + case *ast.EnumSplitStmt: + for _, c := range st.Cases { + walk(c.Body) + } + walk(st.ElseBody) + case *ast.InheritanceSplitStmt: + for _, c := range st.Cases { + walk(c.Body) + } + walk(st.ElseBody) + } + if eh := stmtErrorHandling(s); eh != nil && len(eh.Body) > 0 { + walk(eh.Body) + } + } + } + walk(body) + return vars +} + +// declaredVarsOwnScope returns the variable names a loop body introduces +// itself — descending into branches, whiles, and error handlers, but NOT into a +// nested loop, whose names belong to that loop. +func declaredVarsOwnScope(body []ast.MicroflowStatement) map[string]bool { + vars := map[string]bool{} + var walk func([]ast.MicroflowStatement) + walk = func(stmts []ast.MicroflowStatement) { + for _, s := range stmts { + for name := range collectDeclaredVars([]ast.MicroflowStatement{s}) { + vars[name] = true + } + switch st := s.(type) { + case *ast.LoopStmt: + // Owned by the nested loop, not by this body. + case *ast.WhileStmt: + walk(st.Body) + case *ast.IfStmt: + walk(st.ThenBody) + walk(st.ElseBody) + case *ast.EnumSplitStmt: + for _, c := range st.Cases { + walk(c.Body) + } + walk(st.ElseBody) + case *ast.InheritanceSplitStmt: + for _, c := range st.Cases { + walk(c.Body) + } + walk(st.ElseBody) + } + if eh := stmtErrorHandling(s); eh != nil && len(eh.Body) > 0 { + walk(eh.Body) + } + } + } + walk(body) + return vars +} + +// loopRefVars returns the variable names a single statement reads, WITHOUT +// descending into nested bodies (walkLoopScope visits those itself, so that a +// reference is judged against the loops actually enclosing it). +// +// A statement kind missing here only costs a missed report, never a false one. +func loopRefVars(stmt ast.MicroflowStatement) []string { + var refs []string + add := func(names ...string) { + for _, n := range names { + if n != "" { + refs = append(refs, extractVarName(n)) + } + } + } + addExpr := func(exprs ...ast.Expression) { + for _, e := range exprs { + refs = append(refs, exprVarRefs(e)...) + } + } + addArgs := func(args []ast.CallArgument) { + for _, a := range args { + addExpr(a.Value) + } + } + addChanges := func(items []ast.ChangeItem) { + for _, c := range items { + addExpr(c.Value) + } + } + + switch s := stmt.(type) { + case *ast.MfSetStmt: + add(s.Target) + addExpr(s.Value) + case *ast.DeclareStmt: + addExpr(s.InitialValue) + case *ast.ReturnStmt: + addExpr(s.Value) + case *ast.CreateObjectStmt: + addChanges(s.Changes) + case *ast.ChangeObjectStmt: + add(s.Variable) + addChanges(s.Changes) + case *ast.MfCommitStmt: + add(s.Variable) + case *ast.DeleteObjectStmt: + add(s.Variable) + case *ast.RollbackStmt: + add(s.Variable) + case *ast.RetrieveStmt: + add(s.StartVariable) + addExpr(s.Where) + case *ast.IfStmt: + addExpr(s.Condition) + case *ast.WhileStmt: + addExpr(s.Condition) + case *ast.LoopStmt: + // The iterator is defined here, not referenced; the list is not. + add(s.ListVariable) + case *ast.EnumSplitStmt: + add(s.Variable) + case *ast.InheritanceSplitStmt: + add(s.Variable) + case *ast.CastObjectStmt: + add(s.ObjectVariable) + case *ast.LogStmt: + addExpr(s.Node, s.Message) + case *ast.CallMicroflowStmt: + addArgs(s.Arguments) + case *ast.CallNanoflowStmt: + addArgs(s.Arguments) + case *ast.CallJavaActionStmt: + addArgs(s.Arguments) + case *ast.CallJavaScriptActionStmt: + addArgs(s.Arguments) + case *ast.ExecuteDatabaseQueryStmt: + addArgs(s.Arguments) + addArgs(s.ConnectionArguments) + case *ast.ListOperationStmt: + add(s.InputVariable, s.SecondVariable) + addExpr(s.Condition, s.OffsetExpr, s.LimitExpr) + case *ast.AggregateListStmt: + add(s.InputVariable) + addExpr(s.Expression) + case *ast.AddToListStmt: + add(s.List) + if s.Value != nil { + addExpr(s.Value) + } else { + add(s.Item) + } + case *ast.RemoveFromListStmt: + add(s.Item, s.List) + case *ast.ShowPageStmt: + add(s.ForObject) + for _, a := range s.Arguments { + addExpr(a.Value) + } + case *ast.ShowMessageStmt: + addExpr(s.Message) + addExpr(s.TemplateArgs...) + case *ast.DownloadFileStmt: + add(s.FileDocument) + case *ast.ValidationFeedbackStmt: + if s.AttributePath != nil { + add(s.AttributePath.Variable) + } + addExpr(s.Message) + addExpr(s.TemplateArgs...) + } + + // Normalise: strip any leftover $ sigils from expression-derived names. + for i, r := range refs { + refs[i] = strings.TrimPrefix(r, "$") + } + return refs +} diff --git a/mdl/executor/validate_microflow_loop_scope_test.go b/mdl/executor/validate_microflow_loop_scope_test.go new file mode 100644 index 000000000..87201adac --- /dev/null +++ b/mdl/executor/validate_microflow_loop_scope_test.go @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// loopScopeViolations parses an MDL source containing exactly one microflow and +// returns the MDL053 messages the validator produces for it. +func loopScopeViolations(t *testing.T, src string) []string { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + var msgs []string + for _, stmt := range prog.Statements { + mf, ok := stmt.(*ast.CreateMicroflowStmt) + if !ok { + continue + } + for _, v := range ValidateMicroflow(mf) { + if v.RuleID == "MDL053" { + msgs = append(msgs, v.Message) + } + } + } + return msgs +} + +// TestLoopScope_IteratorUsedAfterLoop is the reported symptom: the loop +// iterator is referenced after `end loop`. Verified against mxbuild 11.12.1 as +// [CE0108] "Variable 'item' is defined but not in scope at this location." +func TestLoopScope_IteratorUsedAfterLoop(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($Items: list of Sample.Thing) +begin + loop $item in $Items + begin + change $item (Name = 'in'); + end loop; + change $item (Name = 'after'); +end; +`) + if len(msgs) != 1 { + t.Fatalf("expected 1 MDL053, got %d: %v", len(msgs), msgs) + } + if !strings.Contains(msgs[0], "$item") || !strings.Contains(msgs[0], "CE0108") { + t.Errorf("message should name the variable and CE0108, got: %s", msgs[0]) + } +} + +// A variable CREATED inside the loop body is loop-scoped too — same CE0108, +// confirmed against mxbuild 11.12.1. +func TestLoopScope_BodyVariableUsedAfterLoop(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($Items: list of Sample.Thing) +begin + loop $item in $Items + begin + $Inner = create Sample.Thing (Name = 'x'); + end loop; + change $Inner (Name = 'after'); +end; +`) + if len(msgs) != 1 { + t.Fatalf("expected 1 MDL053, got %d: %v", len(msgs), msgs) + } + if !strings.Contains(msgs[0], "$Inner") { + t.Errorf("message should name $Inner, got: %s", msgs[0]) + } +} + +// An inner loop's variable used in the outer loop body — after the inner +// `end loop` but still inside the outer one — is equally out of scope. +func TestLoopScope_InnerLoopVariableUsedInOuterBody(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($Outer: list of Sample.Thing, $Inner: list of Sample.Thing) +begin + loop $o in $Outer + begin + loop $i in $Inner + begin + change $i (Name = 'in'); + end loop; + change $i (Name = 'outer body'); + end loop; +end; +`) + if len(msgs) != 1 { + t.Fatalf("expected 1 MDL053, got %d: %v", len(msgs), msgs) + } + if !strings.Contains(msgs[0], "$i") { + t.Errorf("message should name $i, got: %s", msgs[0]) + } +} + +// A reference from a branch that follows the loop is just as out of scope as a +// reference at the top level. +func TestLoopScope_ReferenceInBranchAfterLoop(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($Items: list of Sample.Thing) +begin + loop $item in $Items + begin + log info node 'Sample' 'x'; + end loop; + if $Items != empty then + change $item (Name = 'after'); + end if; +end; +`) + if len(msgs) != 1 { + t.Fatalf("expected 1 MDL053, got %d: %v", len(msgs), msgs) + } +} + +// Everything that stays inside the loop body — including a nested branch and a +// nested loop reading the outer iterator — must not be reported. +func TestLoopScope_UsesInsideLoopAreClean(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($Items: list of Sample.Thing, $Others: list of Sample.Thing) +begin + declare $Total integer = 0; + loop $item in $Items + begin + if $item/Name != empty then + set $Total = $Total + 1; + end if; + loop $other in $Others + begin + change $other (Name = $item/Name); + end loop; + change $item (Name = 'still in scope'); + end loop; + return $Total; +end; +`) + if len(msgs) != 0 { + t.Fatalf("expected no MDL053, got: %v", msgs) + } +} + +// The carry-out idiom — declare before the loop, assign inside, read after — +// is the recommended fix and must stay clean. +func TestLoopScope_CarryOutVariableIsClean(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($Items: list of Sample.Thing) +begin + declare $LastName string = ''; + loop $item in $Items + begin + set $LastName = $item/Name; + end loop; + log info node 'Sample' $LastName; +end; +`) + if len(msgs) != 0 { + t.Fatalf("expected no MDL053, got: %v", msgs) + } +} + +// Two sibling loops REUSING an iterator name have no single owning body, so +// MDL053 must stay silent and leave the case to MDL052 (CE0111). Without this +// the first loop's own use of the name was reported as out of scope. +func TestLoopScope_DuplicateIteratorNameNotReported(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($A: list of Sample.Thing, $B: list of Sample.Thing) +begin + loop $R in $A + begin + change $R (Name = 'a'); + end loop; + loop $R in $B + begin + change $R (Name = 'b'); + end loop; +end; +`) + if len(msgs) != 0 { + t.Fatalf("expected no MDL053 for a duplicate iterator name (MDL052 owns it), got: %v", msgs) + } +} + +// Two sequential loops each using their own iterator: no cross-references, so +// nothing to report. +func TestLoopScope_SequentialLoopsAreClean(t *testing.T) { + msgs := loopScopeViolations(t, ` +create microflow Sample.MF ($A: list of Sample.Thing, $B: list of Sample.Thing) +begin + loop $a in $A + begin + change $a (Name = 'a'); + end loop; + loop $b in $B + begin + change $b (Name = 'b'); + end loop; +end; +`) + if len(msgs) != 0 { + t.Fatalf("expected no MDL053, got: %v", msgs) + } +} diff --git a/mdl/executor/validate_odata_properties.go b/mdl/executor/validate_odata_properties.go new file mode 100644 index 000000000..eb666627a --- /dev/null +++ b/mdl/executor/validate_odata_properties.go @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation of OData property names. +// +// The grammar accepts any `name: value` pair inside an OData property list, and +// the visitor's switch had no default — so `ReadMicroflow:` or `Pagesize:` was +// discarded in silence and the model was quietly missing what the author asked +// for. The ALTER path has always answered "unknown OData service property: %s"; +// this applies the same rule to CREATE and to the PUBLISH ENTITY block. +// (mxcli-formula1 findings, suggested issue 8.) +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// Known property names, in the spelling the syntax help uses. These are for the +// error message only — the visitor is the authority on what is accepted, and it +// matches case-insensitively. +var ( + knownODataServiceProps = []string{ + "Path", "Version", "ODataVersion", "Namespace", "ServiceName", + "Summary", "Description", "PublishAssociations", "Folder", + } + knownPublishEntityProps = []string{ + "ReadMode", "InsertMode", "UpdateMode", "DeleteMode", "UsePaging", "PageSize", + } + knownODataClientProps = []string{ + "Version", "ODataVersion", "MetadataUrl", "Timeout", "ProxyType", + "Description", "ServiceUrl", "UseAuthentication", "HttpUsername", + "HttpPassword", "ClientCertificate", "ConfigurationMicroflow", + "HeadersMicroflow", "ErrorHandlingMicroflow", "ProxyHost", "ProxyPort", + "ProxyUsername", "ProxyPassword", "Folder", + } + knownExternalEntityProps = []string{ + "EntitySet", "RemoteName", "Countable", "Creatable", "Deletable", + "Updatable", "AllowCreateChangeLocally", + } +) + +// ValidateODataProperties flags (MDL-ODATA01) property names in OData +// statements that no layer below will act on. +func ValidateODataProperties(prog *ast.Program) []linter.Violation { + if prog == nil { + return nil + } + var out []linter.Violation + for _, stmt := range prog.Statements { + switch s := stmt.(type) { + case *ast.CreateODataServiceStmt: + out = append(out, unknownODataProps( + "odata service "+s.Name.String(), s.UnknownProperties, knownODataServiceProps)...) + for _, e := range s.Entities { + if e == nil { + continue + } + out = append(out, unknownODataProps( + fmt.Sprintf("publish entity %s in %s", e.Entity.String(), s.Name.String()), + e.UnknownProperties, knownPublishEntityProps)...) + } + case *ast.CreateODataClientStmt: + out = append(out, unknownODataProps( + "odata client "+s.Name.String(), s.UnknownProperties, knownODataClientProps)...) + case *ast.CreateExternalEntityStmt: + out = append(out, unknownODataProps( + "external entity "+s.Name.String(), s.UnknownProperties, knownExternalEntityProps)...) + } + } + return out +} + +func unknownODataProps(location string, unknown, known []string) []linter.Violation { + var out []linter.Violation + for _, name := range unknown { + v := linter.Violation{ + RuleID: "MDL-ODATA01", + Severity: linter.SeverityError, + Message: fmt.Sprintf("%s: unknown property %q — it is accepted by the parser and then discarded, so the model will not have it", + location, name), + Suggestion: fmt.Sprintf("Known properties here: %s.", strings.Join(known, ", ")), + } + if near := closestProperty(name, known); near != "" { + v.Suggestion = fmt.Sprintf("Did you mean %q? Known properties here: %s.", near, strings.Join(known, ", ")) + } + out = append(out, v) + } + return out +} + +// closestProperty returns the known property a misspelling most likely meant, +// or "" when nothing is close enough to be worth guessing. Case-insensitive +// prefix/substring first, then a single edit. +func closestProperty(name string, known []string) string { + lower := strings.ToLower(name) + for _, k := range known { + lk := strings.ToLower(k) + if strings.HasPrefix(lk, lower) || strings.HasPrefix(lower, lk) || strings.Contains(lk, lower) { + return k + } + } + for _, k := range known { + if withinOneEdit(lower, strings.ToLower(k)) { + return k + } + } + return "" +} + +// withinOneEdit reports whether a and b differ by at most one insertion, +// deletion or substitution. +func withinOneEdit(a, b string) bool { + if a == b { + return true + } + if len(a) > len(b) { + a, b = b, a + } + if len(b)-len(a) > 1 { + return false + } + i, j, edits := 0, 0, 0 + for i < len(a) && j < len(b) { + if a[i] == b[j] { + i++ + j++ + continue + } + edits++ + if edits > 1 { + return false + } + if len(a) == len(b) { + i++ + } + j++ + } + return true +} diff --git a/mdl/executor/validate_odata_properties_test.go b/mdl/executor/validate_odata_properties_test.go new file mode 100644 index 000000000..f033f8dbb --- /dev/null +++ b/mdl/executor/validate_odata_properties_test.go @@ -0,0 +1,137 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// mxcli-formula1 findings, suggested issue 8: the OData property switches had +// no default, so a typo was accepted by the parser, dropped by the visitor, and +// the model was quietly missing what the author asked for. The ALTER path has +// always answered "unknown OData service property". +func TestValidateODataProperties(t *testing.T) { + const prologue = "create module T;\ncreate non-persistent entity T.Row (K: string(20));\n" + + service := func(props, entityProps string) string { + return prologue + ` +create odata service T.Api ( + path: 'odata/t/', + version: '1.0.0', + ODataVersion: OData4, + namespace: 'T.Api'` + props + ` +) +authentication basic +{ + publish entity T.Row as 'Rows' ( + ReadMode: source` + entityProps + ` + ) + expose ( K as 'k' (KEY) ); +}; +` + } + + tests := []struct { + name string + script string + wantN int + wantText []string + }{ + {"clean service", service("", ""), 0, nil}, + { + "misspelt service property", + service(",\n ServiceNam: 'Api'", ""), + 1, + // The guess is the whole point: the known-property list alone still + // leaves the reader diffing two spellings by eye. + []string{"ServiceNam", "ServiceName"}, + }, + { + "unknown publish-entity property", + service("", ",\n ReadMicroflow: microflow T.Read"), + 1, + []string{"ReadMicroflow", "ReadMode"}, + }, + { + "both at once", + service(",\n Pth: 'x'", ",\n PgSize: 20"), + 2, + nil, + }, + // Property matching is case-insensitive in the visitor, so a different + // casing is NOT a typo and must not be reported. (The finding listed + // `Pagesize:` as silently dropped; it is not — it is accepted.) + {"casing is not a typo", service("", ",\n Pagesize: 20"), 0, nil}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + prog, errs := visitor.Build(tt.script) + if len(errs) > 0 { + t.Fatalf("parsing the script: %v", errs) + } + got := ValidateODataProperties(prog) + if len(got) != tt.wantN { + t.Fatalf("got %d violations, want %d: %v", len(got), tt.wantN, got) + } + for _, want := range tt.wantText { + found := false + for _, v := range got { + if strings.Contains(v.Message+v.Suggestion, want) { + found = true + } + } + if !found { + t.Errorf("expected a violation mentioning %q, got: %v", want, got) + } + } + for _, v := range got { + if v.RuleID != "MDL-ODATA01" { + t.Errorf("rule = %q, want MDL-ODATA01", v.RuleID) + } + } + }) + } +} + +func TestValidateODataProperties_ClientAndExternalEntity(t *testing.T) { + script := ` +create module T; +create odata client T.Api ( + Version: '1.0', + ODataVersion: OData4, + MetadataUrl: 'https://example.com/$metadata', + Timout: 300 +); +` + prog, errs := visitor.Build(script) + if len(errs) > 0 { + t.Fatalf("parsing the script: %v", errs) + } + got := ValidateODataProperties(prog) + if len(got) != 1 { + t.Fatalf("got %d violations, want 1: %v", len(got), got) + } + if !strings.Contains(got[0].Suggestion, "Timeout") { + t.Errorf("expected a Timeout suggestion, got: %s", got[0].Suggestion) + } +} + +func TestClosestProperty(t *testing.T) { + known := []string{"ReadMode", "InsertMode", "PageSize"} + tests := []struct{ in, want string }{ + {"ReadMod", "ReadMode"}, // one deletion + {"Readmodes", "ReadMode"}, // one insertion, different case + {"PageSiz", "PageSize"}, // prefix + {"Countable", ""}, // nothing close — no guess is better than a wrong one + {"", "ReadMode"}, // empty is a prefix of everything; harmless + } + for _, tt := range tests { + if got := closestProperty(tt.in, known); got != tt.want { + t.Errorf("closestProperty(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} diff --git a/mdl/executor/validate_page_order.go b/mdl/executor/validate_page_order.go new file mode 100644 index 000000000..64b28fc48 --- /dev/null +++ b/mdl/executor/validate_page_order.go @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation of page-creation order within one script. +// The executor resolves a widget's page reference at the moment it writes the +// page, so a button pointing at a page created further down the same script +// fails partway through `exec` — after earlier statements have already been +// written. `--references` catches this (validateForwardPageRefs), but that needs +// a project; the ordering is a property of the script alone whenever the target +// is created by a plain CREATE, so plain `mxcli check` can catch it too. +// (mxcli-todo findings #9.) +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// ValidateScriptPageOrder flags (MDL-PAGE01) a page or snippet whose widgets +// reference a page that a LATER statement creates with a plain CREATE. +// +// The "plain CREATE" condition is what makes this sound without a project: a +// plain CREATE fails if the page already exists, so a script containing one +// asserts the page does not exist yet — the earlier reference therefore cannot +// resolve against the project either. `CREATE OR MODIFY` / `CREATE OR REPLACE` +// carry no such assertion, so those stay for `--references`, which can look. +func ValidateScriptPageOrder(prog *ast.Program) []linter.Violation { + if prog == nil { + return nil + } + + var out []linter.Violation + for i, stmt := range prog.Statements { + var widgets []*ast.WidgetV3 + var label string + switch s := stmt.(type) { + case *ast.CreatePageStmtV3: + widgets, label = s.Widgets, "page "+s.Name.String() + case *ast.CreateSnippetStmtV3: + widgets, label = s.Widgets, "snippet "+s.Name.String() + default: + continue + } + + refs := &widgetRefCollector{} + refs.collectFromWidgets(widgets) + refs.dedupe() + for _, ref := range refs.pages { + if createdEarlier(prog, ref, i) || !plainCreateAfter(prog, ref, i) { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL-PAGE01", + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s references page %s before it is created — the executor resolves page references in statement order, so this fails partway through `exec`", + label, ref), + Suggestion: fmt.Sprintf( + "Move the CREATE PAGE %s statement above this one. If the two pages link to each other, no ordering satisfies both: create one without the linking widget and add it afterwards with ALTER PAGE … INSERT.", + ref), + }) + } + } + return out +} + +// createdEarlier reports whether ref is created by any page statement at an +// index below fromIdx — in which case the reference resolves fine. +func createdEarlier(prog *ast.Program, ref string, fromIdx int) bool { + for j := 0; j < fromIdx; j++ { + if s, ok := prog.Statements[j].(*ast.CreatePageStmtV3); ok && s.Name.String() == ref { + return true + } + } + return false +} + +// plainCreateAfter reports whether ref is created after fromIdx by a CREATE that +// is neither OR MODIFY nor OR REPLACE. +func plainCreateAfter(prog *ast.Program, ref string, fromIdx int) bool { + for j := fromIdx + 1; j < len(prog.Statements); j++ { + s, ok := prog.Statements[j].(*ast.CreatePageStmtV3) + if !ok || s.Name.Module == "" || s.Name.String() != ref { + continue + } + if !s.IsModify && !s.IsReplace { + return true + } + } + return false +} diff --git a/mdl/executor/validate_page_order_test.go b/mdl/executor/validate_page_order_test.go new file mode 100644 index 000000000..8e3b95f4d --- /dev/null +++ b/mdl/executor/validate_page_order_test.go @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// mxcli-todo findings #9: a page whose button targets a page created further +// down the same script passed `mxcli check` and then failed partway through +// `exec` — after earlier statements had already been written to the .mpr. +// `--references` caught it, but only with a project; the ordering is knowable +// from the script alone whenever the target is created by a plain CREATE. +func TestValidateScriptPageOrder(t *testing.T) { + const board = ` +create page ORD.Board +( + title: 'Board', + layout: Atlas_Core.Atlas_Default +) +{ + container ctnMain { + linkbutton btnNew (caption: 'New', action: SHOW_PAGE ORD.TaskEdit) + } +} +` + const edit = ` +create page ORD.TaskEdit +( + title: 'Edit', + layout: Atlas_Core.Atlas_Default +) +{ + container ctnEdit { + dynamictext txtEdit (content: 'edit') + } +} +` + + tests := []struct { + name string + script string + wantFlag bool + }{ + {"forward reference to a plain CREATE", "create module ORD;\n" + board + edit, true}, + {"target created first", "create module ORD;\n" + edit + board, false}, + // CREATE OR MODIFY asserts nothing about whether the page already exists, + // so the reference may well resolve against the project. Only + // --references, which can look, may judge that one. + { + "forward reference to a CREATE OR MODIFY is left alone", + "create module ORD;\n" + board + strings.Replace(edit, "create page", "create or modify page", 1), + false, + }, + {"a page with no page references", "create module ORD;\n" + edit, false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + prog, errs := visitor.Build(tt.script) + if len(errs) > 0 { + t.Fatalf("parsing the script: %v", errs) + } + got := ValidateScriptPageOrder(prog) + if tt.wantFlag { + if len(got) != 1 { + t.Fatalf("expected exactly 1 violation, got %d: %v", len(got), got) + } + if got[0].RuleID != "MDL-PAGE01" { + t.Errorf("rule = %q, want MDL-PAGE01", got[0].RuleID) + } + // The message is only useful if it names both pages and the fix. + for _, want := range []string{"ORD.Board", "ORD.TaskEdit"} { + if !strings.Contains(got[0].Message, want) { + t.Errorf("message should mention %q, got: %s", want, got[0].Message) + } + } + if !strings.Contains(got[0].Suggestion, "ALTER PAGE") { + t.Errorf("suggestion should cover the cyclic case, got: %s", got[0].Suggestion) + } + } else if len(got) != 0 { + t.Errorf("expected no violations, got: %v", got) + } + }) + } +} + +func TestValidateScriptPageOrder_NilProgram(t *testing.T) { + if got := ValidateScriptPageOrder(nil); got != nil { + t.Errorf("expected nil for a nil program, got %v", got) + } +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 09dcf6e81..088236a23 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -702,6 +702,54 @@ func violation18(locationPrefix string, w *ast.WidgetV3, msg string) linter.Viol // validateStaticWidget checks value-level constraints on built-in (non-pluggable) // widgets that the grammar can't express and that otherwise fail silently or at // build time rather than at `mxcli check` time. +// validateConsumableConditional (MDL-WIDGET19) rejects a `Visible:` / `Editable:` +// value that no builder can consume, so an expression the visitor failed to turn +// into VisibleIf/EditableIf fails the command instead of vanishing from the page. +// +// The bracket form is routed to VisibleIf/EditableIf by the visitor; the plain +// slot then holds only a static form, which pages.StaticVisibleExpression reads +// from a bool or a string. Anything else is parse residue — the `[...]` matched +// the generic property-value alternative rather than an xpathConstraint — and the +// builder's `else if` simply doesn't fire. That is the silent-drop mechanism +// behind issue #852, where `trim(…)`/`length(…)` were unparseable as conditional +// expressions and the whole property disappeared; a missing Visible defaults to +// "always visible", so nothing downstream could notice. +// +// The grammar fix removes the known trigger. This rule is the general guard: the +// next function name promoted to a lexer token fails loudly here instead. +func validateConsumableConditional(w *ast.WidgetV3, locationPrefix string) []linter.Violation { + var out []linter.Violation + for _, p := range []struct{ plain, routed string }{ + {"Visible", "VisibleIf"}, + {"Editable", "EditableIf"}, + } { + if _, routed := w.Properties[p.routed]; routed { + continue + } + v, present := w.Properties[p.plain] + if !present || v == nil { + continue + } + switch v.(type) { + case bool, string: + continue // a static form StaticVisibleExpression consumes + } + out = append(out, linter.Violation{ + RuleID: "MDL-WIDGET19", + Severity: linter.SeverityError, + Message: fmt.Sprintf( + "%s: widget `%s` (%s) has a `%s` value that could not be parsed as a conditional expression "+ + "and would be dropped on write (leaving the widget unconditionally %s) — "+ + "check the expression inside `%s: [ ... ]`", + locationPrefix, w.Name, w.Type, strings.ToLower(p.plain), + map[string]string{"Visible": "visible", "Editable": "editable"}[p.plain], + strings.ToLower(p.plain), + ), + }) + } + return out +} + func validateStaticWidget(w *ast.WidgetV3, locationPrefix string) []linter.Violation { var out []linter.Violation @@ -742,6 +790,8 @@ func validateStaticWidget(w *ast.WidgetV3, locationPrefix string) []linter.Viola out = append(out, *v) } + out = append(out, validateConsumableConditional(w, locationPrefix)...) + // A DataView cannot use a database data source — a data view shows one object, // so Mendix offers only Context / Microflow / Nanoflow / Listen sources. // mxcli used to accept it: the modelsdk engine then errors "not yet supported — diff --git a/mdl/executor/validate_widgets_test.go b/mdl/executor/validate_widgets_test.go index de1e8c594..ce3f1e15d 100644 --- a/mdl/executor/validate_widgets_test.go +++ b/mdl/executor/validate_widgets_test.go @@ -154,6 +154,76 @@ func TestValidateStaticWidget_DataViewDatabaseSource(t *testing.T) { } } +// TestValidateStaticWidget_UnconsumableConditional — MDL-WIDGET19 is the safety +// net asked for in issue #852: the grammar fix stops `Visible: [trim(…)]` from +// falling through, but any FUTURE conditional expression the visitor cannot turn +// into VisibleIf/EditableIf would land in the plain Visible/Editable slot as a +// non-string, non-bool value, which the builder drops without a word. A dropped +// Visible reads as "always visible", so the failure is invisible until someone +// looks at the running app. +// +// Recognized static forms (bool, expression string) must NOT be flagged — they +// are consumed by pages.StaticVisibleExpression. +func TestValidateStaticWidget_UnconsumableConditional(t *testing.T) { + cases := []struct { + name string + widget *ast.WidgetV3 + want bool // expect an MDL-WIDGET19 violation + }{ + { + "unparsed bracket residue → flagged", + &ast.WidgetV3{Type: "dynamictext", Name: "a1", Properties: map[string]any{ + "Visible": []any{"trim($currentObject/Slug)", "!=", "''"}, + }}, + true, + }, + { + "unparsed Editable residue → flagged", + &ast.WidgetV3{Type: "textbox", Name: "b1", Properties: map[string]any{ + "Editable": []any{"length($currentObject/Slug)", ">", "0"}, + }}, + true, + }, + { + "routed to VisibleIf → not flagged", + &ast.WidgetV3{Type: "dynamictext", Name: "a2", Properties: map[string]any{ + "VisibleIf": "trim($currentObject/Slug) != ''", + }}, + false, + }, + { + "static bool → not flagged", + &ast.WidgetV3{Type: "dynamictext", Name: "a3", Properties: map[string]any{"Visible": false}}, + false, + }, + { + "expression string → not flagged", + &ast.WidgetV3{Type: "dynamictext", Name: "a4", Properties: map[string]any{ + "Visible": "$currentObject/Slug != ''", + }}, + false, + }, + { + "no visibility property → not flagged", + &ast.WidgetV3{Type: "dynamictext", Name: "a5", Properties: map[string]any{"Content": "x"}}, + false, + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got := false + for _, v := range validateStaticWidget(c.widget, "page X") { + if v.RuleID == "MDL-WIDGET19" { + got = true + } + } + if got != c.want { + t.Errorf("MDL-WIDGET19 present = %v, want %v", got, c.want) + } + }) + } +} + // TestValidateWidgetExpressionAssociations — MDL-WIDGET13 flags an association // step inside an expression-typed widget property (DynamicClasses/VisibleIf/ // EditableIf). Such expressions fail the build with CE0117; a data binding on the diff --git a/mdl/grammar/Makefile b/mdl/grammar/Makefile index 17c751492..74e46e155 100644 --- a/mdl/grammar/Makefile +++ b/mdl/grammar/Makefile @@ -1,14 +1,15 @@ # Makefile for MDL grammar generation # # Usage: +# make bootstrap - Install the pinned ANTLR4 toolchain (pip + a JVM) # make generate - Generate Go parser from MDLLexer.g4 and MDLParser.g4 # make clean - Remove generated files # -# Prerequisites: -# - ANTLR4 must be installed (https://www.antlr.org/) -# - macOS: brew install antlr4 -# - Linux: apt install antlr4 (or download JAR manually) -# - Or use: pip install antlr4-tools +# Prerequisites: an `antlr4` launcher on PATH and a JVM. The generator version +# is PINNED and load-bearing: a generator/runtime mismatch is a classic ANTLR +# failure mode, and go.mod requires the runtime below. CI pins the same pair. +# `antlr4-tools` downloads the ANTLR jar on first run, so the build needs +# network and a JVM, not only Go. LEXER = MDLLexer.g4 PARSER = MDLParser.g4 @@ -27,18 +28,40 @@ DOMAIN_FILES = \ domains/MDLCatalog.g4 \ domains/MDLSettings.g4 +# Pinned toolchain. Keep in step with the CI workflows and with the +# github.com/antlr4-go/antlr/v4 requirement in go.mod. +ANTLR_TOOLS_VERSION = 0.2.2 +ANTLR_VERSION = 4.13.2 +ANTLR_RUNTIME_VERSION = 4.13.1 + # Find antlr4 command (try common locations) ANTLR4 := $(shell which antlr4 2>/dev/null || which antlr 2>/dev/null) -.PHONY: generate clean check-antlr +.PHONY: generate clean check-antlr bootstrap check-antlr: ifndef ANTLR4 - $(error ANTLR4 not found. Install with: brew install antlr4 (macOS) or pip install antlr4-tools) + $(error ANTLR4 not found. Run `make -C mdl/grammar bootstrap`, or install it yourself: \ + pip install 'antlr4-tools==$(ANTLR_TOOLS_VERSION)' && export ANTLR4_TOOLS_ANTLR_VERSION=$(ANTLR_VERSION) \ + (macOS alternative: brew install antlr4). The generator version is pinned to $(ANTLR_VERSION) \ + against runtime $(ANTLR_RUNTIME_VERSION) in go.mod — a mismatch produces a parser that does not compile) endif +# bootstrap installs the pinned generator. It needs pip and a JVM; antlr4-tools +# downloads antlr4-$(ANTLR_VERSION)-complete.jar on first run. +bootstrap: + pip install 'antlr4-tools==$(ANTLR_TOOLS_VERSION)' + @echo "" + @echo "Installed antlr4-tools $(ANTLR_TOOLS_VERSION)." + @echo "Export the pinned generator version before building:" + @echo " export ANTLR4_TOOLS_ANTLR_VERSION=$(ANTLR_VERSION)" + generate: check-antlr $(LEXER) $(PARSER) $(DOMAIN_FILES) @mkdir -p $(OUTPUT_DIR) + @if [ -z "$$ANTLR4_TOOLS_ANTLR_VERSION" ]; then \ + echo "Note: ANTLR4_TOOLS_ANTLR_VERSION is unset; this build pins $(ANTLR_VERSION)."; \ + echo " export ANTLR4_TOOLS_ANTLR_VERSION=$(ANTLR_VERSION)"; \ + fi $(ANTLR4) -Dlanguage=Go -no-visitor -package $(PACKAGE) -lib domains -o $(OUTPUT_DIR) $(LEXER) $(PARSER) @echo "Generated Go parser in $(OUTPUT_DIR)/" diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index d367099a1..9cd613bbd 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -158,7 +158,7 @@ describeStatement | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name | DESCRIBE DATABASE CONNECTION qualifiedName // DESCRIBE DATABASE CONNECTION Module.Name - | DESCRIBE SETTINGS // DESCRIBE SETTINGS + | DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)? // DESCRIBE SETTINGS [CONFIGURATION 'Default'] | DESCRIBE FRAGMENT FROM PAGE qualifiedName WIDGET identifierOrKeyword // DESCRIBE FRAGMENT FROM PAGE Module.Page WIDGET name | DESCRIBE FRAGMENT FROM SNIPPET qualifiedName WIDGET identifierOrKeyword // DESCRIBE FRAGMENT FROM SNIPPET Module.Snippet WIDGET name | DESCRIBE IMAGE COLLECTION qualifiedName // DESCRIBE IMAGE COLLECTION Module.Name diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index 8a83b361e..b35a9f78a 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -211,14 +211,16 @@ alterEntityAction | SET ALLOW_CREATE_CHANGE_LOCALLY EQUALS (TRUE | FALSE) | ADD INDEX indexDefinition | DROP INDEX IDENTIFIER - | ADD EVENT HANDLER eventHandlerDefinition - | DROP EVENT HANDLER ON eventMoment eventType + | ADD EVENT HANDLER ifNotExists? eventHandlerDefinition + | DROP EVENT HANDLER ifExists? ON eventMoment eventType ; -// Idempotency guards for a re-runnable domain script: ADD ATTRIBUTE IF NOT -// EXISTS skips (with a notice) when the attribute is already present, and DROP -// ATTRIBUTE IF EXISTS skips when it is already gone — instead of erroring and -// halting the run. +// Idempotency guards for a re-runnable domain script: ADD ... IF NOT EXISTS +// skips (with a notice) when the member is already present, and DROP ... IF +// EXISTS skips when it is already gone — instead of erroring and halting the +// run. Accepted on ATTRIBUTE and on EVENT HANDLER, which has no other way to be +// re-run: a defensive drop-then-add fails on the drop when the handler is +// absent, and on the add when it is present. (mxcli-todo findings #18) ifNotExists : IF NOT EXISTS ; diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index 4bc34edd4..a4b8ae353 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -120,7 +120,6 @@ microflowStatement | annotation* caseStatement SEMICOLON | annotation* inheritanceSplitStatement SEMICOLON | annotation* castObjectStatement SEMICOLON - | annotation* setStatement SEMICOLON | annotation* createListStatement SEMICOLON // Must be before createObjectStatement to match "CREATE LIST OF" | annotation* createObjectStatement SEMICOLON | annotation* changeObjectStatement SEMICOLON @@ -170,6 +169,13 @@ microflowStatement | annotation* openWorkflowStatement SEMICOLON | annotation* lockWorkflowStatement SEMICOLON | annotation* unlockWorkflowStatement SEMICOLON + // LAST on purpose. Since SET became optional, `$X = ` overlaps every + // `VARIABLE EQUALS ` statement above — aggregates, list + // operations, RANGE. Those rules must keep winning: a lower-numbered + // setStatement swallowed `$Sum = sum($List.Price)` into a Change Variable + // whose fallback conversion drops the attribute, which mxbuild rejects + // (CE0015 / CE0109). Last means it only claims what nothing else parses. + | annotation* setStatement SEMICOLON ; declareStatement @@ -207,8 +213,13 @@ castObjectStatement | VARIABLE EQUALS CAST VARIABLE ; +// SET is optional: `$Total = 5;` is what everyone writes, and every other +// assignment form in MDL already works bare (`$X = HEAD($List)`, +// `$X = execute database query …`). Requiring the keyword only here made the +// rule unguessable — and the parse error named the token, not the missing +// keyword (mxcli-formula1 findings #13). setStatement - : SET (VARIABLE | attributePath) EQUALS expression + : SET? (VARIABLE | attributePath) EQUALS expression ; // $NewProduct = CREATE MfTest.Product (Name = $Name, Code = $Code); diff --git a/mdl/grammar/domains/MDLPage.g4 b/mdl/grammar/domains/MDLPage.g4 index cc931826f..a8522f2b5 100644 --- a/mdl/grammar/domains/MDLPage.g4 +++ b/mdl/grammar/domains/MDLPage.g4 @@ -152,13 +152,51 @@ xpathFunctionCall : xpathFunctionName LPAREN (xpathExpr (COMMA xpathExpr)*)? RPAREN ; +/** Function name inside a bracketed [ … ] constraint. + * + * Any single word may name a function, exactly as any single word may be a name + * part (see xpathQualifiedName). The enumerated form this replaces listed only + * IDENTIFIER, HYPHENATED_ID, NOT, TRUE, FALSE and CONTAINS, so a call to a + * function whose name is also a lexer keyword — `trim(…)`, `length(…)` — never + * matched xpathFunctionCall. The enclosing `Visible: [...]` / `Editable: [...]` + * then failed to parse as an xpathConstraint and fell through to the generic + * property-value alternative, so the widget's whole conditional property was + * dropped without a diagnostic (a dropped Visible reads as "always visible" at + * runtime). Issue #852. + * + * The grammar deliberately does NOT enumerate a valid function set, because + * xpathConstraint serves TWO contexts with DIFFERENT ones: + * + * - `Visible:` / `Editable:` — a Mendix *client expression*, where the string + * functions apply: trim(), length(), toUpperCase(), find(), contains(). + * - a datasource `where` clause — real *XPath*, where the function set is + * contains/starts-with/ends-with/string-length/not/true/false and the + * *-from-dateTime family, `length()` means list length rather than character + * count, and the aggregates (count/avg/min/max/sum) are Java-API-only. + * `empty` and `NULL` are keywords here (`[Name = empty]`), never calls. + * See docs.mendix.com/refguide/xpath-constraint-functions/ and + * .../xpath-keywords-and-system-variables/. + * + * One rule cannot encode both sets, and guessing wrong rejects valid MDL. So the + * grammar accepts any name and lets mxbuild adjudicate semantics — it reports an + * unknown or wrong-context function as CE0117 against the real version's rules, + * which no table here could track. Verified on 11.6.6: in a widget conditional + * trim/length/find pass while count/empty give CE0117; in a `where` clause + * `[Name = empty]`, `[Name = NULL]`, not(), contains(), starts-with() and + * string-length() all pass. + * + * xpathWord is a negated token set, so it self-maintains as the lexer grows new + * keywords — an enumerated list would silently reacquire this bug with the next + * function name that gets promoted to a token. NOT is spelled out because + * xpathWord excludes it (it is an operator elsewhere in the expression grammar) + * while `not(…)` is a legitimate call. + * + * This cannot swallow a path: xpathFunctionCall requires an LPAREN after the + * name, and no xpathStepValue may be followed by one, so `empty` alone still + * parses as a word via xpathPath — which is what keeps `[Name = empty]` working. */ xpathFunctionName - : IDENTIFIER - | HYPHENATED_ID + : xpathWord | NOT - | TRUE - | FALSE - | CONTAINS ; // ============================================================================= diff --git a/mdl/visitor/visitor_conditional_visibility_test.go b/mdl/visitor/visitor_conditional_visibility_test.go index 1549292c5..8d35c3762 100644 --- a/mdl/visitor/visitor_conditional_visibility_test.go +++ b/mdl/visitor/visitor_conditional_visibility_test.go @@ -88,3 +88,63 @@ func TestConditionalVisibility_EnumLiteralPreserved(t *testing.T) { t.Errorf("VisibleIf = %q, want %q", got, want) } } + +// Issue #852 — a widget conditional expression that calls a function whose name +// is also an MDL lexer keyword (trim, length, …) silently dropped the whole +// property: `xpathFunctionName` only admitted IDENTIFIER/HYPHENATED_ID plus a +// handful of keywords, so `trim(…)` never matched xpathFunctionCall and the +// expression built to nothing. The property then vanished from the page, and a +// dropped Visible defaults to "always visible" — a wrong-behaviour failure that +// passed both `mxcli check` and `mx check`. +// +// Non-keyword function names (toUpperCase, isMatch) were never affected; they +// are plain IDENTIFIERs. They are covered here so a future narrowing of the rule +// cannot regress them silently. +// +// Scope: this asserts the PARSER builds the call and the property survives, not +// that Mendix accepts the function. The two are deliberately separate — MDL does +// not adjudicate Mendix expression semantics at the grammar layer. `count` and +// `empty` are included because they are keyword tokens (the thing under test) +// even though mxbuild rejects them in a client expression with CE0117: `empty` +// is a literal (`$x != empty`), not a call. Verified against mxbuild 11.6.6, +// where trim/length/find pass and count/empty do not. The shipped example +// mdl-examples/bug-tests/852-conditional-keyword-functions.mdl uses only the +// valid set so it stays `mx check`-clean. +func TestConditionalVisibility_KeywordFunctionNames(t *testing.T) { + cases := []struct { + name string + expr string // what goes inside Visible: [ ... ] + want string + }{ + {"trim", "trim($currentObject/Slug) != ''", "trim($currentObject/Slug) != ''"}, + {"length", "length($currentObject/Slug) > 0", "length($currentObject/Slug) > 0"}, + {"empty", "empty($currentObject/Slug)", "empty($currentObject/Slug)"}, + {"count", "count($currentObject/Items) > 0", "count($currentObject/Items) > 0"}, + {"find", "find($currentObject/Slug, 'x') >= 0", "find($currentObject/Slug, 'x') >= 0"}, + // Already worked — guard against regressing them. + {"contains", "contains($currentObject/Slug, 'x')", "contains($currentObject/Slug, 'x')"}, + {"toUpperCase", "toUpperCase($currentObject/Slug) != ''", "toUpperCase($currentObject/Slug) != ''"}, + // A bare attribute inside a keyword-named call still gets rooted. + {"trim roots bare attr", "trim(Slug) != ''", "trim($currentObject/Slug) != ''"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + input := "CREATE PAGE M.P (Title: 'P') { CONTAINER ctn (Visible: [" + c.expr + "]) { DYNAMICTEXT t (Content: 'x') } };" + prog, errs := Build(input) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + ctn := findWidgetV3(prog.Statements[0].(*ast.CreatePageStmtV3).Widgets, "ctn") + if ctn == nil { + t.Fatal("container ctn not found") + } + got, ok := ctn.Properties["VisibleIf"].(string) + if !ok { + t.Fatalf("VisibleIf missing entirely — the property was dropped (Properties: %v)", ctn.Properties) + } + if got != c.want { + t.Errorf("VisibleIf = %q, want %q", got, c.want) + } + }) + } +} diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index 0dc368b5c..483f5b0d5 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -691,6 +691,7 @@ func (b *Builder) ExitAlterEntityAction(ctx *parser.AlterEntityActionContext) { Name: name, Operation: ast.AlterEntityAddEventHandler, EventHandler: eh, + IfNotExists: ctx.IfNotExists() != nil, }) } } @@ -720,6 +721,7 @@ func (b *Builder) ExitAlterEntityAction(ctx *parser.AlterEntityActionContext) { Name: name, Operation: ast.AlterEntityDropEventHandler, EventHandler: eh, + IfExists: ctx.IfExists() != nil, }) return } diff --git a/mdl/visitor/visitor_microflow_aggregate_test.go b/mdl/visitor/visitor_microflow_aggregate_test.go new file mode 100644 index 000000000..7787af2d6 --- /dev/null +++ b/mdl/visitor/visitor_microflow_aggregate_test.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// Making SET optional put `$X = ` in front of every +// `VARIABLE EQUALS ` statement in the grammar, so +// `$Sum = sum($List.Price)` stopped reaching aggregateListStatement and fell +// through to the SET conversion instead — which joined the list and the +// attribute into one name. mxbuild rejected the result: +// +// [CE0109] "Undefined variable 'ProductList.Price'." +// [CE0015] "Aggregate function must specify a valid attribute." +// +// Both spellings must produce the same aggregate, whichever rule claims them. +func TestAggregateSplitsListFromAttribute(t *testing.T) { + cases := []struct { + name, src string + wantOp ast.AggregateListOperationType + wantAttr string + }{ + {"bare sum", "$T = sum($ProductList.Price);", ast.AggregateSum, "Price"}, + {"bare average", "$T = average($ProductList.Price);", ast.AggregateAverage, "Price"}, + {"bare minimum", "$T = minimum($ProductList.Price);", ast.AggregateMinimum, "Price"}, + {"bare maximum", "$T = maximum($ProductList.Price);", ast.AggregateMaximum, "Price"}, + // The SET keyword routes through a different conversion; it was wrong + // there before the bare form ever reached it. + {"set sum", "set $T = sum($ProductList.Price);", ast.AggregateSum, "Price"}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := parseSingleAggregate(t, tc.src) + if got.Operation != tc.wantOp { + t.Errorf("operation = %v, want %v", got.Operation, tc.wantOp) + } + if got.InputVariable != "ProductList" { + t.Errorf("input variable = %q, want ProductList (the list, without the attribute)", got.InputVariable) + } + if got.Attribute != tc.wantAttr { + t.Errorf("attribute = %q, want %q", got.Attribute, tc.wantAttr) + } + }) + } +} + +// `sum($List, )` aggregates a value computed per item. Losing the +// expression leaves an aggregate with nothing to aggregate — CE0015. +func TestAggregateKeepsPerItemExpression(t *testing.T) { + for _, src := range []string{ + "$T = sum($ProductList, $currentObject/Price * 0.21);", + "set $T = sum($ProductList, $currentObject/Price * 0.21);", + } { + got := parseSingleAggregate(t, src) + if got.InputVariable != "ProductList" { + t.Errorf("%s: input variable = %q, want ProductList", src, got.InputVariable) + } + if !got.IsExpression || got.Expression == nil { + t.Errorf("%s: expression dropped (IsExpression=%v, Expression=%v)", src, got.IsExpression, got.Expression) + } + } +} + +// COUNT takes the list alone and must not acquire an attribute. +func TestAggregateCountTakesTheListAlone(t *testing.T) { + got := parseSingleAggregate(t, "$N = count($ProductList);") + if got.Operation != ast.AggregateCount || got.InputVariable != "ProductList" || got.Attribute != "" { + t.Errorf("got %+v, want COUNT over ProductList with no attribute", got) + } +} + +func parseSingleAggregate(t *testing.T, stmt string) *ast.AggregateListStmt { + t.Helper() + src := "create microflow M.A ($ProductList: list of M.Product)\nbegin\n " + stmt + "\nend;" + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors for %q: %v", stmt, errs) + } + for _, s := range prog.Statements { + cm, ok := s.(*ast.CreateMicroflowStmt) + if !ok { + continue + } + for _, st := range cm.Body { + if agg, ok := st.(*ast.AggregateListStmt); ok { + return agg + } + // A SET means the statement was swallowed as a plain value + // assignment — that is the regression, and it reads better as a + // failure here than as a nil dereference below. + if set, ok := st.(*ast.MfSetStmt); ok { + t.Fatalf("%q produced a Change Variable (target %q), not an aggregate", stmt, set.Target) + } + } + } + t.Fatalf("no AggregateListStmt produced by %q", stmt) + return nil +} diff --git a/mdl/visitor/visitor_microflow_bare_assign_test.go b/mdl/visitor/visitor_microflow_bare_assign_test.go new file mode 100644 index 000000000..40fc29ca1 --- /dev/null +++ b/mdl/visitor/visitor_microflow_bare_assign_test.go @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// mxcli-formula1 findings #13: `$N = 0;` did not parse — `no viable alternative +// at input '$N=0'`, an error naming the token rather than the missing keyword — +// while `DECLARE $N Integer = 0;` did, and every other assignment in MDL already +// worked bare (`$X = HEAD($List)`, `$X = execute database query …`). SET is now +// optional, so the bare form everyone reaches for means the same thing. +func TestBareAssignmentMatchesSetStatement(t *testing.T) { + cases := []struct { + name string + bare, full string + wantTarget string + }{ + {"integer literal", "$Total = 5;", "set $Total = 5;", "Total"}, + // A negative literal is the case that made this look like an + // expression-parsing bug rather than a missing statement form. + {"negative literal", "$Total = -1;", "set $Total = -1;", "Total"}, + {"expression over itself", "$Total = $Total + 1;", "set $Total = $Total + 1;", "Total"}, + {"string literal", "$Name = 'Hello';", "set $Name = 'Hello';", "Name"}, + // A plain variable target is stored without the sigil; an attribute + // path keeps it. Both spellings must agree on whichever it is. + {"attribute path", "$Order/Status = 'Pending';", "set $Order/Status = 'Pending';", "$Order/Status"}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + bare := parseSingleSet(t, tc.bare) + full := parseSingleSet(t, tc.full) + + if bare.Target != tc.wantTarget { + t.Errorf("bare target = %q, want %q", bare.Target, tc.wantTarget) + } + // Same statement, not merely both-parse: the two spellings must + // produce the same AST or they are two features, not one. + if bare.Target != full.Target { + t.Errorf("target differs: bare %q vs SET %q", bare.Target, full.Target) + } + }) + } +} + +// The keyword form must keep working — this is an addition, not a replacement, +// and existing scripts are full of it. +func TestSetKeywordStillParses(t *testing.T) { + if got := parseSingleSet(t, "set $Total = 7;"); got.Target != "Total" { + t.Errorf("target = %q, want Total", got.Target) + } +} + +// parseSingleSet parses one statement inside a microflow body and returns the +// MfSetStmt it produced. +func parseSingleSet(t *testing.T, stmt string) *ast.MfSetStmt { + t.Helper() + src := "create microflow M.ACT ($Order: M.O)\nbegin\n declare $Total integer = 0;\n declare $Name string;\n " + stmt + "\nend;" + prog, errs := Build(src) + if len(errs) > 0 { + t.Fatalf("parse errors for %q: %v", stmt, errs) + } + for _, s := range prog.Statements { + cm, ok := s.(*ast.CreateMicroflowStmt) + if !ok { + continue + } + for _, st := range cm.Body { + if set, ok := st.(*ast.MfSetStmt); ok { + return set + } + } + } + t.Fatalf("no MfSetStmt produced by %q", stmt) + return nil +} diff --git a/mdl/visitor/visitor_microflow_expression.go b/mdl/visitor/visitor_microflow_expression.go index b3e464cb9..b119922c2 100644 --- a/mdl/visitor/visitor_microflow_expression.go +++ b/mdl/visitor/visitor_microflow_expression.go @@ -526,6 +526,13 @@ func buildListAggregateAsFunction(ctx parser.IListAggregateOperationContext) ast } } + // The per-item expression of `sum($List, $currentObject/Price * 0.21)`. + // Without it the call reads as a one-argument aggregate, and whoever + // consumes it builds an aggregate with nothing to aggregate — CE0015. + if exprCtx := aggrCtx.Expression(); exprCtx != nil { + funcExpr.Arguments = append(funcExpr.Arguments, buildSourceExpression(exprCtx)) + } + return funcExpr } diff --git a/mdl/visitor/visitor_microflow_statements.go b/mdl/visitor/visitor_microflow_statements.go index 88bd14914..1ca597301 100644 --- a/mdl/visitor/visitor_microflow_statements.go +++ b/mdl/visitor/visitor_microflow_statements.go @@ -808,37 +808,13 @@ func buildSetStatement(ctx parser.ISetStatementContext) ast.MicroflowStatement { InputVariable: extractVariableName(funcCall.Arguments, 0), } case "SUM": - inputVar, attr := extractVariableAndAttribute(funcCall.Arguments, 0) - return &ast.AggregateListStmt{ - OutputVariable: targetVar, - Operation: ast.AggregateSum, - InputVariable: inputVar, - Attribute: attr, - } + return buildSetAggregate(targetVar, ast.AggregateSum, funcCall.Arguments) case "AVERAGE": - inputVar, attr := extractVariableAndAttribute(funcCall.Arguments, 0) - return &ast.AggregateListStmt{ - OutputVariable: targetVar, - Operation: ast.AggregateAverage, - InputVariable: inputVar, - Attribute: attr, - } + return buildSetAggregate(targetVar, ast.AggregateAverage, funcCall.Arguments) case "MINIMUM": - inputVar, attr := extractVariableAndAttribute(funcCall.Arguments, 0) - return &ast.AggregateListStmt{ - OutputVariable: targetVar, - Operation: ast.AggregateMinimum, - InputVariable: inputVar, - Attribute: attr, - } + return buildSetAggregate(targetVar, ast.AggregateMinimum, funcCall.Arguments) case "MAXIMUM": - inputVar, attr := extractVariableAndAttribute(funcCall.Arguments, 0) - return &ast.AggregateListStmt{ - OutputVariable: targetVar, - Operation: ast.AggregateMaximum, - InputVariable: inputVar, - Attribute: attr, - } + return buildSetAggregate(targetVar, ast.AggregateMaximum, funcCall.Arguments) } } @@ -903,31 +879,45 @@ func getArgumentExpression(args []ast.Expression, index int) ast.Expression { return args[index] } -// extractVariableAndAttribute extracts variable and attribute from $Var/Attr or $Var, Attr. -func extractVariableAndAttribute(args []ast.Expression, index int) (varName string, attrName string) { - if index >= len(args) { - return "", "" +// buildSetAggregate builds an aggregate activity from a SET whose value is a +// SUM/AVERAGE/MINIMUM/MAXIMUM call. +// +// It mirrors buildAggregateListStatement, which handles the same two spellings +// when they arrive through the aggregateListStatement rule: one argument is a +// list plus an attribute (`sum($List.Price)`), two arguments are a list plus an +// expression evaluated per item (`sum($List, $currentObject/Price * 0.21)`). +// Two conversions for one syntax is how the attribute went missing in the first +// place, so the two must agree. +func buildSetAggregate(targetVar string, op ast.AggregateListOperationType, args []ast.Expression) *ast.AggregateListStmt { + stmt := &ast.AggregateListStmt{OutputVariable: targetVar, Operation: op} + if len(args) == 0 { + return stmt } - // Check for attribute path like $Var/Attr - if pathExpr, ok := args[index].(*ast.AttributePathExpr); ok { - varName = pathExpr.Variable - if len(pathExpr.Path) > 0 { - attrName = pathExpr.Path[len(pathExpr.Path)-1] + + switch arg := args[0].(type) { + case *ast.AttributePathExpr: + stmt.InputVariable = arg.Variable + if len(arg.Path) > 0 { + stmt.Attribute = arg.Path[len(arg.Path)-1] } - return - } - // Check for simple variable - if varExpr, ok := args[index].(*ast.VariableExpr); ok { - varName = varExpr.Name - // Look for attribute in next argument - if index+1 < len(args) { - if identExpr, ok := args[index+1].(*ast.IdentifierExpr); ok { - attrName = identExpr.Name - } + case *ast.VariableExpr: + // `sum($List.Price)` reaches the expression parser as one variable whose + // name carries the dot, not as an attribute path. Left joined, mxbuild + // reports the whole thing as an undefined variable (CE0109). + stmt.InputVariable = arg.Name + if list, attr, ok := strings.Cut(arg.Name, "."); ok { + stmt.InputVariable, stmt.Attribute = list, attr } - return } - return "", "" + + // Any second argument is the per-item expression. Dropping it leaves an + // aggregate with nothing to aggregate, which mxbuild rejects with CE0015. + if len(args) > 1 { + stmt.IsExpression = true + stmt.Expression = args[1] + stmt.Attribute = "" + } + return stmt } // extractSortSpecs extracts sort specifications from function arguments. diff --git a/mdl/visitor/visitor_odata.go b/mdl/visitor/visitor_odata.go index 6abb7f27d..b45380676 100644 --- a/mdl/visitor/visitor_odata.go +++ b/mdl/visitor/visitor_odata.go @@ -70,6 +70,8 @@ func (b *Builder) ExitCreateODataClientStatement(ctx *parser.CreateODataClientSt stmt.ProxyPassword = value case "folder": stmt.Folder = value + default: + stmt.UnknownProperties = append(stmt.UnknownProperties, name) } } @@ -119,8 +121,11 @@ func (b *Builder) ExitCreateODataServiceStatement(ctx *parser.CreateODataService stmt.Description = value case "publishassociations": stmt.PublishAssociations = strings.EqualFold(value, "true") || strings.EqualFold(value, "yes") + stmt.PublishAssociationsSet = true case "folder": stmt.Folder = value + default: + stmt.UnknownProperties = append(stmt.UnknownProperties, name) } } @@ -188,6 +193,8 @@ func (b *Builder) ExitCreateExternalEntityStatement(ctx *parser.CreateExternalEn stmt.Updatable = &boolVal case "allowcreatechangelocally", "allowcreatingandchanginglocally", "createchangelocally": stmt.AllowCreateChangeLocally = &boolVal + default: + stmt.UnknownProperties = append(stmt.UnknownProperties, name) } } @@ -355,6 +362,14 @@ func parsePublishEntityBlock(ctx parser.IPublishEntityBlockContext) *ast.Publish if n, err := strconv.Atoi(value); err == nil { entity.PageSize = n } + case "countable": + entity.Countable = odataBoolPtr(value) + case "skipsupported": + entity.SkipSupported = odataBoolPtr(value) + case "topsupported": + entity.TopSupported = odataBoolPtr(value) + default: + entity.UnknownProperties = append(entity.UnknownProperties, name) } } @@ -431,3 +446,11 @@ func parseExposeMembers(ctx parser.IExposeClauseContext) []*ast.PublishedMemberD return members } + +// odataBoolPtr parses an OData property value as a bool, keeping "specified" +// distinct from "true": these properties default to true, so only an explicit +// value may turn one off. +func odataBoolPtr(value string) *bool { + b := strings.EqualFold(value, "true") || strings.EqualFold(value, "yes") + return &b +} diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 64d8727c5..8ae708a13 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -800,11 +800,18 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { return } - // Handle DESCRIBE SETTINGS + // Handle DESCRIBE SETTINGS [CONFIGURATION 'Name'] if ctx.SETTINGS() != nil { - b.statements = append(b.statements, &ast.DescribeStmt{ - ObjectType: ast.DescribeSettings, - }) + stmt := &ast.DescribeStmt{ObjectType: ast.DescribeSettings} + // `alter settings configuration 'X'` is the write form, so the read + // form has to accept the same shape — reaching for it and getting a + // parse error is the wrong lesson (mxcli-formula1 findings #8). + if ctx.CONFIGURATION() != nil { + if sl := ctx.STRING_LITERAL(); sl != nil { + stmt.Qualifier = unquoteString(sl.GetText()) + } + } + b.statements = append(b.statements, stmt) return } diff --git a/model/types.go b/model/types.go index 4b7a89fa7..7ee133361 100644 --- a/model/types.go +++ b/model/types.go @@ -473,6 +473,12 @@ type PublishedEntitySet struct { DeleteMode string `json:"deleteMode,omitempty"` UsePaging bool `json:"usePaging,omitempty"` PageSize int `json:"pageSize,omitempty"` + + // OData query options. nil means "not specified" and is written as Mendix's + // own default of true; only an explicit false turns one off. + Countable *bool `json:"countable,omitempty"` + SkipSupported *bool `json:"skipSupported,omitempty"` + TopSupported *bool `json:"topSupported,omitempty"` } // PublishedMember represents a member (attribute/association/id) published in an OData entity type.