feat: detect fake organizations created from email domains (CM-1411)

[skwowet]

Summary

When ingest creates an organization from an email domain, start a Temporal workflow that uses an LLM to decide whether it is a personal or vanity domain posing as a company. Affiliations stay on by default and are turned off only when the verdict is fake.

Changes

• findOrCreateOrganization now returns { id, created } so callers can tell create from find.
• After an email-domain org is created and linked to a member, data_sink_worker starts fakeOrganizationAnalysisWithLLM.
• Workflow loads a compact org+member payload, asks Claude Sonnet 4 for fake | genuine | unsure, and branches:
  • fake — set isAffiliationBlocked, apply affiliation cleanup, and refresh the org
  • genuine — no write
  • unsure (and unrecognized output) — insert into fakeOrganizationSuggestions
• Adds the suggestions table (org id PK + createdAt) and a single bulk-insert DAL function.
• LLM prompt/answer is already stored in llmPromptHistory; no extra results table.

[Copilot]

Pull request overview

Adds LLM-based detection of fake organizations created from email domains, including affiliation blocking and manual-review suggestions.

Changes:

• Tracks newly created organizations and triggers Temporal analysis.
• Adds LLM classification and verdict handling.
• Persists uncertain organizations for human review.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description / Review notes
services/libs/types/src/llm.ts	Configures the fake-organization LLM query.
services/libs/types/src/enums/temporal.ts	Adds the workflow ID.
services/libs/types/src/enums/llm.ts	Adds the query type.
services/libs/data-access-layer/src/organizations/types.ts	Defines creation-result metadata.
services/libs/data-access-layer/src/organizations/index.ts	Exports suggestion persistence.
services/libs/data-access-layer/src/organizations/fake.ts	Inserts review suggestions.
services/libs/data-access-layer/src/organizations/base.ts	Returns organization creation status.
services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts	Implements classification. Critical: unsafe workflow barrel import. Moderate: malformed output bypasses suggestions. Nit: redundant comment.
services/apps/profiles_worker/src/workflows.ts	Registers the workflow.
services/apps/profiles_worker/src/types/organization.ts	Defines workflow inputs and verdicts.
services/apps/profiles_worker/src/activities/organization/fakeAnalysis.ts	Loads context and applies verdicts. Critical: starts the wrong workflow type. Moderate: replica lag can permanently skip analysis.
services/apps/profiles_worker/src/activities.ts	Registers workflow activities.
services/apps/members_enrichment_worker/src/activities/enrichment.ts	Adapts to the new DAL return type.
services/apps/data_sink_worker/src/service/organization.service.ts	Adds workflow dispatch.
services/apps/data_sink_worker/src/service/member.service.ts	Triggers analysis for new domain organizations. Moderate: transient scheduling failures permanently skip analysis at both trigger sites.
services/apps/data_sink_worker/src/service/activity.service.ts	Updates the organization promise-cache type.
backend/src/database/migrations/V1788792681__fake-organization-suggestions.sql	Creates the suggestions table.
backend/src/api/public/v1/organizations/createOrganization.ts	Adapts the endpoint to the new return type.

Suppressed comments (1)

services/apps/data_sink_worker/src/service/member.service.ts:850

• The update path has the same one-shot scheduling gap: after a transient Temporal start failure, the retry observes created: false, so this organization can never reach the analysis workflow. Persist pending work (for example via an outbox) so retries can reschedule independently of the original created result.

              }
            }
          }

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Suppressed comments (6)

services/apps/data_sink_worker/src/service/member.service.ts:584

• The workflow launch is conditioned on transient created/orgsToAdd state after the organization and relationship have already committed. If the process crashes or the Temporal start call fails, a retry sees an existing organization/relationship and never enters this branch, permanently skipping analysis. Persist a durable pending-analysis/outbox marker in the creation transaction, or otherwise make retries launch from durable state rather than result.created.

              const addedIds = new Set(orgsToAdd.map((org) => org.id))
              for (const org of fromEmailDomain) {
                if (org.created && addedIds.has(org.id)) {
                  await orgService.startFakeOrganizationAnalysisWorkflow(org.id)

services/apps/data_sink_worker/src/service/member.service.ts:846

• The update path has the same at-most-once gap: organization creation and linking commit before the Temporal RPC, while a retry observes created === false or an existing member-organization row and skips this block. A transient dispatch failure therefore leaves the new organization unanalyzed. Drive the launch from durable pending-analysis state (or an outbox) so retries can recover it.

              const addedIds = new Set(orgsToAdd.map((org) => org.id))
              for (const org of fromEmailDomain) {
                if (org.created && addedIds.has(org.id)) {
                  await orgService.startFakeOrganizationAnalysisWorkflow(org.id)

services/apps/profiles_worker/src/activities/organization/fakeAnalysis.ts:60

• This workflow is started immediately after creating and linking the organization, but the initial activity reads from the replica. Replica lag can return no organization (or an incomplete member relation), causing the workflow to return permanently while its stable workflow ID prevents another analysis. Read this fresh payload from the writer to guarantee read-after-write consistency.

  const qx = pgpQx(svc.postgres.reader.connection())

services/apps/profiles_worker/src/activities/organization/fakeAnalysis.ts:165

• The first argument is the workflow type, but this enum resolves to organization-update, which is only the workflow-ID prefix. The registered/exported workflow type is organizationUpdate (as used by backend/src/services/organizationService.ts:1257), so Temporal creates an execution that the profiles worker cannot resolve and the required affiliation refresh/sync never runs.

    await svc.temporal.workflow.start(TemporalWorkflowId.ORGANIZATION_UPDATE, {

services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:58

• Malformed or non-object LLM output never reaches the default branch: parseLlmJson throws (and destructuring null also throws), so the workflow retries and ultimately fails instead of creating the promised human-review suggestion. Normalize parse failures to an undefined verdict so they follow the existing default path.

  const { verdict } = parseLlmJson<{ verdict?: FakeOrganizationVerdict }>(llm.answer)

services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:66

• This comment only restates the default branch. Repository guidance permits new comments only for non-obvious constraints, external quirks, legacy complexity, or ticketed TODOs (CLAUDE.md:72-84), so please remove it and let the control flow remain self-explanatory.

    // unsure, and any verdict the model made up, goes to human review

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:47

• This definition is wrong for multi-label public suffixes: in johnsmith.co.uk, the label immediately before the TLD is co, not the registrable label johnsmith. That can hide exactly the vanity domains this workflow is intended to detect. Define the label relative to the full public suffix (the repository's getDomainRootLabel helper already implements this distinction).

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

services/apps/profiles_worker/src/activities/organization/fakeAnalysis.ts:160

• These three state changes commit independently. If either affiliation update fails after isAffiliationBlocked is written, the organization remains marked blocked while existing affiliations can stay active or only be partially cleaned up. The established blocking path applies these changes in one transaction (backend/src/services/organizationService.ts:1089-1109); wrap them in qx.tx here as well so a failed activity leaves the previous state intact for retry.
  services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:62
• parseLlmJson can return the valid JSON value null; destructuring it throws before the default branch, even though the PR promises unrecognized output is sent to human review. Optional-chain the parsed object so syntactically valid responses without a verdict fall through, while malformed JSON can still retry as intended.

services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:43

• A registrable domain is not always the label before the TLD: for smith.co.uk, that label is co, while the registrable name is smith. This instruction can therefore suppress the intended name match for common multi-label public suffixes and produce a wrong verdict; describe it as the label immediately before the public suffix instead.

      Compare the domain's registrable name (label before the TLD, ignoring hyphens, dots, digits) to the member's displayName: given name, family name, given+family, initials. Any language or script.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (1)

services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:43

• The parenthetical definition of a registrable name is incorrect for multi-label public suffixes: for alice.co.uk, the label before the TLD is co, while the registrable label is alice. This can prevent the classifier from detecting vanity domains on common suffixes such as co.uk; describe it as the label immediately before the public suffix instead.

      Compare the domain's registrable name (label before the TLD, ignoring hyphens, dots, digits) to the member's displayName: given name, family name, given+family, initials. Any language or script.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

services/apps/data_sink_worker/src/service/organization.service.ts:99

• The batch-wide org promise cache returns the same { created: true } result to every member that shares this domain, so multiple callers can reach this starter. Without an explicit reuse policy, Temporal defaults to ALLOW_DUPLICATE: the workflow ID prevents overlap only while a run is open, but a later caller can start another completed workflow and repeat the LLM charge and writes. Set WorkflowIdReusePolicy.REJECT_DUPLICATE here and retain the already-started handling so each organization is analyzed once.

services/apps/data_sink_worker/src/service/member.service.ts:928

• result.created can come from a cached promise that was initiated by the earlier data.organizations path. When a payload contains an explicit organization and a matching verified email domain, that explicit organization is created first, this method reuses its promise, and the code later schedules fake-domain analysis even though the organization was not minted from the email domain. Preserve the creation provenance by setting created only when this method initiated the promise; this also leaves the initiating email-domain caller responsible for scheduling when other members reuse its promise.

          created: result.created,

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

services/apps/data_sink_worker/src/service/member.service.ts:909

• The shared batch cache now propagates the creator's created: true result to every concurrent waiter for the same domain. Each waiter consequently attempts to start fake-org analysis after linking its member; for a popular newly seen domain this generates repeated Temporal starts, and a later waiter can start another costly LLM run once the prior execution closes. Preserve created: true only for the call that populated the cache (or track analysis scheduling separately).
  services/apps/profiles_worker/src/activities/organization/fakeAnalysis.ts:134
• These writes establish a single affiliation-blocking invariant but currently commit independently. If the override update or affiliation deletion fails permanently, the organization remains blocked while existing member affiliations can remain active or only partially updated. Run the three database operations in one transaction so an activity failure leaves retryable state rather than a partial policy application.

services/apps/profiles_worker/src/workflows/organization/fakeOrganizationAnalysisWithLLM.ts:64

• parseLlmJson can return the valid JSON value null; destructuring it throws before the default branch, so this unrecognized output exhausts workflow retries instead of creating the promised human-review suggestion. Read verdict through optional chaining so a top-level null follows the default path (malformed non-JSON can still throw and retry).

  const { verdict } = parseLlmJson<{ verdict?: FakeOrganizationVerdict }>(llm.answer)