Skip to content

Terry Domain Projection Roadmap

Purpose

Этот документ фиксирует, чего не хватает, чтобы Terry мог вести pipeline от raw payload до быстрых structured notifications, raw-to-JSON transformation, schema-matched structured records и нормализованных domain tables для exchange products.

Целевой outcome:

  • Terry остается source/config/raw/replay runtime;
  • TransformActivity отвечает только за преобразование форматов вроде XML, HTML, файлов и PDF в JSON;
  • StoreStructuredDataActivity сопоставляет JSON поля с выбранной схемой и сохраняет schema-matched records;
  • domain projection пишет в typed normalized tables по bounded context, а не в отдельную таблицу на каждую биржу;
  • новые простые источники можно подключать конфигом, сложные источники получают coded processor, но используют общий canonical writer.

Codex Goal Format

Официальный Codex use case для long-running work рекомендует использовать /goal, когда задача имеет durable objective, verifiable stopping condition и validation loop: https://developers.openai.com/codex/use-cases/follow-goals.

Goal для реализации этой инициативы:

/goal Implement the Terry domain projection architecture described in docs/architecture/terry-domain-projection-roadmap.md without stopping until raw payloads can be transformed into JSON, JSON fields can be matched to structured schemas, schema-matched records can be projected into normalized reward/earn/loan domain tables, and the flow is verified by focused tests and parity checks.

Verifiable end state:

  • TransformActivity converts raw XML/HTML/file/PDF inputs into JSON outputs;
  • StoreStructuredDataActivity matches JSON fields to schema and writes schema-matched records with lineage and schema version;
  • projection activities idempotently write reward, earn and loan domain tables;
  • at least one pilot domain runs in parallel with the legacy per-exchange pipeline;
  • parity checks compare legacy and canonical outputs for the pilot domain;
  • implementation follows the phases and decision rules in docs/architecture/terry-domain-projection-roadmap.md;
  • docs describe operational replay, dry-run and rollback behavior.

Target Pipeline

Terry source config
  -> LoaderActivity.Execute [exists]
  -> Raw Store [exists inside LoaderActivity.Execute]
  -> CheckFreshnessActivity.Execute [exists]
  -> RecordRawDataActivity.Execute [exists]
  -> FinalizeRawDataActivity.Execute [exists]
  -> Fast source-level notification rules [stub: BuildPolicySubscriberNotificationsActivity.Execute currently returns skipped]
  -> TransformActivity.Execute [missing: converts raw XML/HTML/file/PDF/etc. into temporary JSON]
  -> StoreStructuredDataActivity.Execute [partial: maps JSON fields to schema and stores structured data; schema identity/key contract missing]
  -> Domain Projection [missing]
  -> Domain tables and continuous aggregates [partial: legacy per-exchange fct_* tables and aggregates exist; canonical domain tables missing]
  -> Domain-level notification rules [missing]

Current orchestration entrypoint: RunIngestionWorkflow.

Performance metrics note: UpdateSourcePerformanceActivity.Execute exists, but it is not part of the target business pipeline. It should remain an optional metrics side-effect that workflow stages or activities can call after success or failure.

Status legend:

  • exists: implemented in the current Terry workflow/activity layer;
  • partial: an implementation exists, but the target architecture needs a stronger contract or different semantics;
  • stub: callable activity exists, but target behavior is not implemented;
  • missing: not implemented yet.

Transform Activity Target Flow

TransformActivity.Execute should run after BuildPolicySubscriberNotificationsActivity.Execute and before StoreStructuredDataActivity.Execute.

It receives a TransformationRequest, validates raw input according to validator settings, converts source formats like XML, HTML, files or PDF into JSON according to processor settings, stores the transformed JSON file back into raw/object storage as temporary structured data, and returns a TransformationResponse through DataTransformerOutput.

flowchart LR
    %% BPMN-like node styles
    classDef start_msg fill:#fff,stroke:#333,stroke-width:2px,rx:50,ry:50;
    classDef end_msg fill:#fff,stroke:#000,stroke-width:4px,rx:50,ry:50;
    classDef gateway fill:#fff,stroke:#333,stroke-width:1px,rx:5,ry:5;
    classDef task fill:#fff,stroke:#333,stroke-width:1px,rx:10,ry:10;

    StartEvent(("Received task<br/>TransformationRequest"))

    ValidateTask["Validate raw data<br/>according to<br/>validator settings"]

    CheckValidation{"Success?"}

    TransformTask["Convert raw input<br/>to JSON according to<br/>processor settings"]

    CheckTransformation{"Success?"}

    SaveTask["Store transformed file<br/>in raw/object storage<br/>as temporary<br/>structured data"]

    UpdateSuccessTask["Update job with<br/>structured data info<br/>ResultCode = Success"]

    MergeFail{ }

    UpdateFailTask["Update job with<br/>error information<br/>ResultCode = Failed"]

    MergeFinal{ }

    EndEvent(("Send result<br/>TransformationResponse<br/>to DataTransformerOutput"))

    StartEvent --> ValidateTask
    ValidateTask --> CheckValidation

    CheckValidation -- Yes --> TransformTask
    CheckValidation -- No --> MergeFail

    TransformTask --> CheckTransformation

    CheckTransformation -- Yes --> SaveTask
    CheckTransformation -- No --> MergeFail

    SaveTask --> UpdateSuccessTask
    UpdateSuccessTask --> MergeFinal

    MergeFail --> UpdateFailTask
    UpdateFailTask --> MergeFinal

    MergeFinal --> EndEvent

    class StartEvent start_msg;
    class EndEvent end_msg;
    class CheckValidation,CheckTransformation,MergeFail,MergeFinal gateway;
    class ValidateTask,TransformTask,SaveTask,UpdateSuccessTask,UpdateFailTask task;

    linkStyle default stroke:#333,stroke-width:1px;

Target activity outcome:

  • validation failure returns ResultCode=Failed with machine-readable error information;
  • transformation failure returns ResultCode=Failed with raw-to-JSON processor error information;
  • success returns ResultCode=Success and a temporary structured data file reference that StoreStructuredDataActivity.Execute can match to a schema and persist;
  • the activity may update performance/job metadata as a side effect, but that metadata update is not the business output of the transformation.

Store Structured Data Activity Target Flow

StoreStructuredDataActivity.Execute should run after TransformActivity.Execute. It owns schema matching/correlation and persistence into Structured Store. There should not be a separate schema matching activity in the target pipeline.

Responsibilities:

  • receive StructuredDataStoreRequest;
  • correlate fields from the temporary structured JSON file with StructuredFieldName and dataSourceFields processor settings;
  • validate required schema fields and field types;
  • compute stable record_key values;
  • save schema-matched documents into Structured Store;
  • delete the temporary structured file from raw/object storage;
  • if active notification policies and subscribers exist, create notification send requests;
  • return StructuredDataStoreResponse with ResultCode=Success or ResultCode=Failed.
flowchart LR
    %% BPMN-like node styles
    classDef start_msg fill:#fff,stroke:#333,stroke-width:2px,rx:50,ry:50;
    classDef end_msg fill:#fff,stroke:#000,stroke-width:4px,rx:50,ry:50;
    classDef end_error fill:#fdd,stroke:#f00,stroke-width:2px,rx:50,ry:50;
    classDef gateway fill:#fff,stroke:#333,stroke-width:1px,rx:5,ry:5;
    classDef task fill:#fff,stroke:#333,stroke-width:1px,rx:10,ry:10;
    classDef error_boundary fill:#fff,stroke:#f00,stroke-width:1px,rx:50,ry:50,color:#f00;
    classDef note fill:#fff,stroke:#0288d1,stroke-width:1px,color:#0288d1,stroke-dasharray: 3 3;
    classDef sub_start fill:#dfd,stroke:#000,stroke-width:1px,rx:50,ry:50;

    StartEvent(("Received task<br/>StructuredDataStoreRequest"))

    StartNote["Diagram: Structured data<br/>storage algorithm"]
    StartEvent -.- StartNote

    T1["Correlate fields<br/>StructuredFieldName and<br/>temporary structured file<br/>using dataSourceFields"]
    Err1(("can"))

    T2["Save temporary structured<br/>file data into Structured Store<br/>using processor dataSourceFields<br/>settings"]
    Err2(("can"))

    T3["Delete temporary<br/>structured file from<br/>raw/object storage"]
    Err3(("can"))

    ErrorMerge{ }
    UpdateFailTask["Update job with<br/>error information<br/>ResultCode = Failed"]

    GW1{"Does the source have<br/>active notification policies<br/>except NewRawData?"}

    GW2{"Are there active<br/>subscribers for<br/>these policies?"}

    MergeSuccess{ }

    UpdateSuccessTask["Update job with<br/>stored document info,<br/>notification count,<br/>ResultCode = Success"]

    FinalMerge{ }
    EndEvent(("Send result<br/>StructuredDataStoreResponse<br/>to JobManager.Input"))

    subgraph NotificationSubprocess ["Build policy subscriber notifications"]
        direction LR
        SubStart(( ))
        SubT1["Build notification<br/>for each subscriber<br/>from template"]
        SubGW{"Success?"}
        SubT2["Create task for<br/>Notification Module<br/>to send notifications"]
        SubEnd(("Send task<br/>NotificationSendRequest<br/>to Notification.Input"))

        SubErrInternal(("A"))

        SubStart --> SubT1
        SubT1 --> SubGW
        SubGW -- Yes --> SubT2
        SubT2 --> SubEnd
        SubGW -- No --> SubErrInternal
    end

    SubBoundaryErr(("can"))
    SubErrLog["Save error<br/>to log"]
    SubEndErr(( ))

    StartEvent --> T1

    T1 --> T2
    T1 -.-> Err1 --> ErrorMerge

    T2 --> T3
    T2 -.-> Err2 --> ErrorMerge

    T3 --> GW1
    T3 -.-> Err3 --> ErrorMerge

    ErrorMerge --> UpdateFailTask
    UpdateFailTask --> FinalMerge

    GW1 -- No --> MergeSuccess
    GW1 -- Yes --> GW2

    GW2 -- No --> MergeSuccess
    GW2 -- "One or more policies" --> SubStart

    SubEnd --> MergeSuccess

    NotificationSubprocess -.-> SubBoundaryErr
    SubBoundaryErr --> SubErrLog
    SubErrLog --> SubEndErr

    MergeSuccess --> UpdateSuccessTask
    UpdateSuccessTask --> FinalMerge
    FinalMerge --> EndEvent

    class StartEvent,SubEnd,EndEvent start_msg;
    class EndEvent end_msg;
    class SubStart sub_start;
    class SubEndErr end_error;
    class GW1,GW2,SubGW,ErrorMerge,MergeSuccess,FinalMerge gateway;
    class T1,T2,T3,UpdateFailTask,UpdateSuccessTask,SubT1,SubT2,SubErrLog task;
    class Err1,Err2,Err3,SubBoundaryErr,SubErrInternal error_boundary;
    class StartNote note;

    linkStyle default stroke:#333,stroke-width:1px;

Bounded Contexts

Используем DDD как lightweight рамку для границ, а не как требование строить полную tactical DDD object model.

Context Responsibility Examples
Terry ingestion Source config, raw payload, freshness, replay, execution metadata terry.data_sources, terry.loader_configs, terry.raw_data_files
Structured store Schema-matched records for fast rules, preview, debugging and replay checkpoints schema_name, schema_version, record_key
Asset catalog Cross-exchange assets, networks and mapping quarantine global_assets, networks_catalog, exchange_assets
Reward campaigns Launchpool/xpool style promotional reward pools launchpool, xpool, campaign pools
Earn/staking Flexible earn, locked earn, simple earn and staking products APY/APR, duration, lock/redeem semantics
Loans Borrow/lend markets and offers borrow asset, collateral asset, LTV, term, liquidity
Notifications Source-level and domain-level policy evaluation and delivery fast alerts, canonical cross-exchange alerts

What Is Missing

1. Structured Store Schema Identity Contract

Current structured_data_documents behavior stores structured documents, but the architecture needs an explicit schema identity contract.

Missing:

  • schema_name and schema_version;
  • record_key for idempotent upsert;
  • source lineage: data_source_id, loader_config_id, raw_file_uid;
  • transformation lineage: transform_handler, transform_version;
  • schema matching lineage: mapping_template_id, mapping_config_version;
  • indexes for fast notification and projection queries by schema, key, source and time.

Acceptance criteria:

  • schema-matched records can be stored with stable schema identity;
  • repeated storage of the same transformed file remains idempotent;
  • a projection activity can select schema-matched records for one raw file or one job.

2. Raw-To-JSON Transform Contract

Current Terry config has processor_settings, but there is no implemented raw-to-JSON transformation runtime for non-JSON source formats.

Missing:

  • transform input contract: raw file UID and source format metadata;
  • transform output contract: temporary JSON file UID;
  • transform result summary: success/failure and output file metadata;
  • transform versioning and deterministic replay behavior;
  • error envelope for non-retryable data defects vs retryable infrastructure failures.

Acceptance criteria:

  • XML, HTML, file and PDF sources can be transformed into JSON;
  • transform can be rerun from raw payloads;
  • changing transform config creates a new version;
  • transform output can be compared across versions before schema matching.

3. Schema Matching DSL For Config-Only Sources

To avoid code changes for simple sources, Terry needs a constrained mapping DSL.

Missing:

  • record path selection;
  • field path mapping;
  • required/optional fields;
  • built-in transforms: trim, upper, lower, parse numeric, parse percent, parse timestamp, parse duration, normalize symbol;
  • enum normalization;
  • asset mapping policy references;
  • quarantine rules for missing or ambiguous fields;
  • dry-run preview and validation before activation.

Acceptance criteria:

  • a simple HTTP JSON launchpool/earn/loan source can be mapped from UI/config;
  • invalid mapping config fails before schedule activation;
  • mapping cannot execute arbitrary unsafe code.

4. Transformer And Schema Matcher Registry

Config-only mapping will not cover every exchange. Complex SDK and Playwright sources need coded transformers or store-time schema correlation handlers.

Missing:

  • registry keyed by processor_settings.handler;
  • stable handler names, for example:
  • transform.xml_to_json
  • transform.html_to_json
  • transform.pdf_to_json
  • match.reward_campaign.schema
  • match.earn.schema
  • match.loan.schema
  • custom.gate.launchpool
  • custom.bybit.easy_earn
  • custom.kucoin.loans
  • handler capability metadata for UI and validation;
  • common processor request/result structs;
  • processor version reporting.

Acceptance criteria:

  • adding a new coded processor does not require changing workflow control flow;
  • handler selection is data-driven through persisted config;
  • unknown or disabled handlers fail as non-retryable config errors.

5. Domain Canonical Schemas

The current model has many per-exchange fct_* tables. The target needs canonical tables per domain, not one table for everything and not one table per exchange.

Missing:

  • reward campaign tables for launchpool/xpool;
  • earn/staking tables;
  • loans tables;
  • shared response/quarantine conventions;
  • common idempotency keys;
  • migration plan from existing per-exchange tables;
  • compatibility views if existing readers depend on old table names.

Candidate domains:

exchange_reward_campaign_snapshots
exchange_reward_pool_snapshots
exchange_reward_quarantine

exchange_earn_product_snapshots
exchange_earn_pool_snapshots
exchange_earn_quarantine

exchange_loan_market_snapshots
exchange_loan_offer_snapshots
exchange_loan_quarantine

Acceptance criteria:

  • exchange_id is a dimension, not part of the table name;
  • launchpool/xpool, earn/staking and loans keep separate domain semantics;
  • raw source payload remains linked through raw_file_uid and payload hash;
  • Timescale hypertables and continuous aggregates remain available where needed.

6. Domain Projection Activities

Canonical structured records should be projected into domain tables after they are saved.

Missing:

  • ProjectRewardCampaignRecordsActivity;
  • ProjectEarnRecordsActivity;
  • ProjectLoanRecordsActivity;
  • idempotent writers for each domain;
  • projection result summary;
  • projection replay by raw_file_uid, structured batch or time range.

Acceptance criteria:

  • projection can be retried safely;
  • projection can be rerun without refetching the source;
  • projection failures do not corrupt raw or schema-matched structured data.

7. Notification Layer Split

Fast notifications and canonical/domain notifications have different contracts.

Missing:

  • source-level notification rules over raw or transformed JSON;
  • schema-level notification rules over schema-matched structured records;
  • domain-level rules over typed tables and aggregates;
  • policy ownership metadata;
  • dedupe keys for notification events.

Acceptance criteria:

  • quick source-specific alerts do not require domain projection;
  • cross-exchange alerts use canonical/domain data;
  • notification replay does not resend already delivered events unless requested.

8. Operations And Replay

The architecture needs operational controls before migration from existing exchange pipelines.

Missing:

  • replay command/API by raw_file_uid, data_source_id and time range;
  • dry-run mode for transform and projection;
  • parity reports comparing old per-exchange tables to new canonical tables;
  • dead-letter review flow;
  • metrics for transformed, schema-matched and projected counts;
  • rollout controls for active config versions.

Acceptance criteria:

  • one source can run old and new pipelines in parallel;
  • parity can be measured before switching readers;
  • failed projection can be repaired without losing raw lineage.

Implementation Phases

These phases are optimized for Codex execution. Each phase should be small enough for an agent to take as a standalone /goal, inspect the repo, make scoped changes, run focused verification, and stop with a clear result.

Phase design rules:

  • one phase owns one architectural boundary;
  • Temporal workflow semantic changes are isolated in their own phase;
  • domain work starts with one pilot domain, loans, before generalizing;
  • reader cutover happens only after parity evidence exists;
  • each phase has explicit prerequisites, allowed changes, likely files, acceptance criteria and stop/ask points.

Phase 0: Lock Architecture Decisions

Goal: make the remaining architecture choices explicit before migrations or workflow changes.

Codex goal:

/goal Complete Phase 0 from docs/architecture/terry-domain-projection-roadmap.md without stopping until the Terry domain projection decisions are documented: bounded contexts, pilot domain, structured schema identity, record key rules, transform/store/projection boundaries, and stop/ask rules for later implementation phases.

Codex plan command:

/plan Phase 0 from docs/architecture/terry-domain-projection-roadmap.md. Goal: lock architecture decisions before code. Context: read this roadmap, Terry docs, data-model docs, current migrations and RunIngestionWorkflow. Constraints: documentation only; do not change code, migrations or Temporal behavior. Done when: decisions and unresolved blockers are explicit enough for a later Codex task to implement Phase 1 without asking broad architecture questions.

Codex task packet:

  • Prerequisites: none.
  • Allowed changes: documentation only.
  • Likely files:
  • docs/architecture/terry-domain-projection-roadmap.md
  • docs/architecture/data-models.md if cross-links are needed
  • docs/terry/*.md if terminology needs alignment
  • Acceptance criteria:
  • loans is confirmed or rejected as the first pilot domain;
  • structured store identity fields are finalized;
  • record_key examples exist for loans, earn and reward records;
  • TransformActivity, StoreStructuredDataActivity and projection boundaries are unambiguous;
  • unresolved choices are listed as blockers.
  • Verification:
  • documentation review only.
  • Stop and ask:
  • if launchpool/xpool or earn/staking grouping remains ambiguous;
  • before changing code, migrations or workflow behavior.

Phase 1: Structured Store Identity And Read API

Goal: make schema-matched structured records identifiable, idempotent and queryable.

Codex goal:

/goal Complete Phase 1 from docs/architecture/terry-domain-projection-roadmap.md without stopping until terry.structured_data_documents can store schema_name, schema_version, record_key and lineage, repository read APIs can select records for later projection, and focused loader/database tests pass.

Codex plan command:

/plan Phase 1 from docs/architecture/terry-domain-projection-roadmap.md. Goal: add structured store identity and read APIs. Context: inspect StoreStructuredDataActivity, structured_data_documents migration, repository code and tests. Constraints: additive migration only; preserve existing idempotency and existing structured data behavior; no workflow or domain projection changes. Done when: plan lists exact schema/repository/activity/test changes and verification commands.

Codex task packet:

  • Prerequisites: Phase 0 schema identity and record key decisions are documented.
  • Allowed changes: additive migration, repository methods, loader record structs, tests and docs.
  • Likely files:
  • internal/db/migrations/*.sql
  • internal/db/migrations_test.go
  • internal/worker/loader/structured_data.go
  • internal/worker/loader/structured_data_test.go
  • internal/worker/repository/structured_data_document_repository.go
  • internal/worker/repository/repository_test.go
  • docs/activities/store-structured-data-activity.md
  • Acceptance criteria:
  • structured records store schema_name, schema_version, record_key, raw file lineage and transform/schema-correlation metadata;
  • idempotent upsert remains deterministic;
  • repository can read records by structured file, raw file, data source and schema identity;
  • no domain tables, projection activities or workflow order changes are added.
  • Verification:
go test ./internal/worker/loader ./internal/db ./internal/worker/repository
  • Stop and ask:
  • before replacing the table instead of adding compatible columns;
  • before destructive migrations or data backfills.

Phase 2: Store Structured Data Schema Correlation

Goal: make StoreStructuredDataActivity own JSON-field-to-schema correlation and record-key derivation for already-JSON temporary structured files.

Codex goal:

/goal Complete Phase 2 from docs/architecture/terry-domain-projection-roadmap.md without stopping until StoreStructuredDataActivity can correlate transformed JSON fields to configured schema fields, validate required fields/types, derive record_key values, persist schema-matched records, and pass focused loader tests.

Codex plan command:

/plan Phase 2 from docs/architecture/terry-domain-projection-roadmap.md. Goal: implement schema correlation inside StoreStructuredDataActivity. Context: inspect processor_settings, dataSourceFields, structured_data.go path helpers and tests. Constraints: deterministic config-only mapping; no arbitrary code execution; no separate schema-matching activity; no domain projection. Done when: plan defines config shape, validation, transforms, record key derivation and tests.

Codex task packet:

  • Prerequisites: Phase 1 structured store identity is implemented.
  • Allowed changes: schema correlation config parsing, safe transforms, StoreStructuredDataActivity changes, tests and docs.
  • Likely files:
  • internal/worker/loader/types.go
  • internal/worker/loader/structured_data.go
  • internal/worker/loader/structured_data_test.go
  • docs/terry/configuration.md
  • docs/activities/store-structured-data-activity.md
  • Acceptance criteria:
  • mapping config selects record paths and field paths;
  • required fields and type coercions fail with non-retryable config/data errors;
  • built-in transforms are deterministic and safe;
  • record_key can be derived from configured fields;
  • invalid config can be validated without persisting records.
  • Verification:
go test ./internal/worker/loader
  • Stop and ask:
  • before adding a general expression language;
  • if matching needs source-specific business logic better handled by a coded store-time correlation handler.

Phase 3: Transform Activity Raw-To-JSON

Goal: add TransformActivity as a raw-format-to-JSON activity, without relying on domain semantics.

Codex goal:

/goal Complete Phase 3 from docs/architecture/terry-domain-projection-roadmap.md without stopping until TransformActivity can convert raw XML/HTML/file/PDF-style inputs into temporary JSON output through a config-driven transformer registry, with request/response contracts, lineage and focused activity tests.

Codex plan command:

/plan Phase 3 from docs/architecture/terry-domain-projection-roadmap.md. Goal: implement TransformActivity and transformer registry. Context: inspect loader activity patterns, raw storage interfaces, processor_settings and current RunIngestionWorkflow tests. Constraints: keep transformation focused on raw-to-JSON; no schema matching in this activity; do not wire into RunIngestionWorkflow unless explicitly approved. Done when: plan defines TransformationRequest/Response, registry behavior, temporary JSON output and tests.

Codex task packet:

  • Prerequisites: Phase 0 transform boundary is documented.
  • Allowed changes: new activity, transformer registry, request/response structs, raw-to-JSON handlers, temporary JSON writing, tests and docs.
  • Likely files:
  • internal/worker/loader/transform*.go
  • internal/worker/loader/types.go
  • internal/worker/loader/*test.go
  • cmd/worker/main.go only for registration if approved
  • docs/activities/*.md
  • Acceptance criteria:
  • activity accepts TransformationRequest and returns TransformationResponse;
  • unknown transformer handlers fail as non-retryable config errors;
  • successful transform writes a temporary JSON file to raw/object storage;
  • validation/transform failures produce machine-readable result details;
  • tests cover success, config error, transform error and storage error.
  • Verification:
go test ./internal/worker/loader
  • Stop and ask:
  • before wiring into RunIngestionWorkflow;
  • before adding major parser dependencies.

Phase 4: Wire Terry Workflow Transform And Store Path

Goal: connect the Terry runtime sequence so raw loading can optionally flow through transformation and then structured storage.

Codex goal:

/goal Complete Phase 4 from docs/architecture/terry-domain-projection-roadmap.md without stopping until RunIngestionWorkflow can deterministically run BuildPolicySubscriberNotificationsActivity, optional TransformActivity, and StoreStructuredDataActivity in the target order, with Temporal versioning and focused workflow tests.

Codex plan command:

/plan Phase 4 from docs/architecture/terry-domain-projection-roadmap.md. Goal: wire TransformActivity into RunIngestionWorkflow safely. Context: inspect RunIngestionWorkflow, workflow.GetVersion markers, activity registration and workflow tests. Constraints: Temporal workflow semantics change requires explicit approval; preserve deterministic versioning; do not add domain projection in this phase. Done when: plan identifies version markers, activity order, fallback behavior and workflow tests.

Codex task packet:

  • Prerequisites: Phase 2 and Phase 3 are implemented; explicit approval for Temporal workflow semantic change.
  • Allowed changes: workflow ordering, version markers, activity registration, workflow tests and docs.
  • Likely files:
  • internal/worker/workflows/run_ingestion.go
  • internal/worker/workflows/run_ingestion_test.go
  • cmd/worker/main.go
  • docs/workflows/run-ingestion-workflow.md
  • docs/activities/*.md
  • Acceptance criteria:
  • target order is represented in workflow tests;
  • old workflow histories remain protected by workflow.GetVersion;
  • transform is optional when source config does not require it;
  • StoreStructuredData receives the transformed JSON temporary file;
  • performance metrics remain side effects, not business pipeline steps.
  • Verification:
go test ./internal/worker/workflows ./internal/worker/loader
  • Stop and ask:
  • before changing task queues, retry policy or schedule behavior;
  • if replay compatibility requires a broader Temporal migration strategy.

Phase 5: Loans Domain Tables And Writers

Goal: implement the first domain projection target for loans only.

Codex goal:

/goal Complete Phase 5 from docs/architecture/terry-domain-projection-roadmap.md without stopping until canonical loans domain tables and idempotent loans writers exist, preserve Terry lineage, keep existing per-exchange fct_* loans tables untouched, and pass database/repository tests.

Codex plan command:

/plan Phase 5 from docs/architecture/terry-domain-projection-roadmap.md. Goal: implement loans domain schema and writers as the first pilot domain. Context: inspect loans snapshot schema, existing loans activities, data-model docs and Timescale policies. Constraints: loans only; additive migrations; no reader cutover; no destructive changes to legacy fct_* tables. Done when: plan lists table shapes, writer APIs, lineage, indexes, aggregates and verification commands.

Codex task packet:

  • Prerequisites: Phase 0 confirms loans as pilot domain; Phase 1 lineage fields exist.
  • Allowed changes: additive loans domain migrations, writer/repository APIs, tests and data-model docs.
  • Likely files:
  • internal/db/migrations/*.sql
  • internal/db/migrations_test.go
  • internal/worker/models/*loans*.go
  • internal/worker/repository/*.go
  • internal/worker/repository/repository_test.go
  • docs/architecture/data-models.md
  • Acceptance criteria:
  • loans domain tables model market/offer semantics separately from earn/reward;
  • exchange_id is a column, not part of table names;
  • tables link to Terry raw/structured lineage;
  • writers are idempotent and preserve quarantine/error details;
  • legacy loans tables and ingestion functions are untouched.
  • Verification:
go test ./internal/db ./internal/worker/repository
  • Stop and ask:
  • before adding backfills or destructive migrations;
  • before adding Timescale policies that require compose verification.

Phase 6: Loans Projection Activity

Goal: project schema-matched structured loan records into the new loans domain tables.

Codex goal:

/goal Complete Phase 6 from docs/architecture/terry-domain-projection-roadmap.md without stopping until a loans projection activity can read schema-matched structured records, write idempotently to loans domain tables, support replay by raw/structured file, and pass focused activity/repository tests.

Codex plan command:

/plan Phase 6 from docs/architecture/terry-domain-projection-roadmap.md. Goal: implement loans projection activity. Context: inspect structured store read APIs, loans domain writers, Temporal activity patterns and loans fixtures/tests. Constraints: projection reads structured records, not raw payloads; do not wire into workflow unless explicitly approved; no reader cutover. Done when: plan defines activity input/output, replay selectors, error semantics and tests.

Codex task packet:

  • Prerequisites: Phase 1 read APIs and Phase 5 loans writers exist.
  • Allowed changes: loans projection activity, request/result contracts, replay helpers, tests and docs.
  • Likely files:
  • internal/worker/activities/*loans*projection*.go
  • internal/worker/activities/*loans*projection*_test.go
  • internal/worker/models/*.go
  • internal/worker/repository/*.go
  • cmd/worker/main.go only for registration if approved
  • docs/activities/*.md
  • Acceptance criteria:
  • projection reads schema-matched structured records by raw file, structured file or data source/time selector;
  • writes to loans domain tables are idempotent;
  • retryable DB/storage errors and non-retryable schema defects are separated;
  • activity tests cover success, duplicate replay, schema defect and writer failure.
  • Verification:
go test ./internal/worker/activities ./internal/worker/repository
  • Stop and ask:
  • before inserting projection into RunIngestionWorkflow;
  • if structured records do not contain enough fields for loans projection.

Phase 7: Loans Parallel Pilot And Parity

Goal: prove the new structured-store-to-domain-table path against the existing loans pipeline before any reader changes.

Codex goal:

/goal Complete Phase 7 from docs/architecture/terry-domain-projection-roadmap.md without stopping until the loans pilot can run old per-exchange ingestion and new structured projection in parallel, parity SQL compares counts/rates/liquidity/quarantine/aggregates, and mismatches are documented without reader cutover.

Codex plan command:

/plan Phase 7 from docs/architecture/terry-domain-projection-roadmap.md. Goal: run loans parity pilot. Context: inspect Gate/Bybit/KuCoin loans activities, legacy fct_* loans tables, new loans domain tables, fixtures and docs. Constraints: no reader cutover; no destructive data changes; keep old and new paths parallel. Done when: plan defines fixtures/data needs, parity SQL, acceptable deltas, mismatch report and verification commands.

Codex task packet:

  • Prerequisites: Phase 6 loans projection exists.
  • Allowed changes: fixture replay, parity SQL/report docs, tests and optional non-invasive pilot config.
  • Likely files:
  • docs/api/*loans*.md
  • docs/terry/operations.md
  • docs/architecture/data-models.md
  • internal/worker/activities/*loans*.go tests/fixtures if needed
  • SQL parity files under docs/ if no better location exists
  • Acceptance criteria:
  • old and new loans outputs can be compared from the same source snapshot;
  • parity covers counts, rates, liquidity, quarantine and aggregates where available;
  • mismatches include enough detail for triage;
  • no production schedule or reader is switched.
  • Verification:
  • focused tests for fixture replay, plus documented parity SQL.
  • Stop and ask:
  • if representative raw payloads, credentials, fixtures or DB snapshots are missing;
  • before enabling schedules or touching production-like data.

Phase 8: Expand Domain Families After Loans

Goal: generalize the proven loans pattern to reward campaigns and earn/staking.

Codex goal:

/goal Complete Phase 8 from docs/architecture/terry-domain-projection-roadmap.md without stopping until reward campaign and earn/staking domain schemas, writers and projection activities follow the proven loans pattern, with focused tests and no reader cutover.

Codex plan command:

/plan Phase 8 from docs/architecture/terry-domain-projection-roadmap.md. Goal: expand domain projection to reward and earn after loans pilot. Context: inspect launchpool/xpool/earn migrations, activities, docs and the implemented loans domain pattern. Constraints: do not collapse reward, earn and loans into one table; keep legacy fct_* tables; no reader cutover. Done when: plan lists domain-specific schema differences, writer/projection reuse, tests and verification commands.

Codex task packet:

  • Prerequisites: Phase 7 loans pilot has acceptable parity.
  • Allowed changes: additive reward/earn migrations, writers, projection activities, tests and docs.
  • Likely files:
  • internal/db/migrations/*.sql
  • internal/worker/models/*.go
  • internal/worker/repository/*.go
  • internal/worker/activities/*projection*.go
  • docs/architecture/data-models.md
  • docs/api/*launchpool*.md, docs/api/*earn*.md
  • Acceptance criteria:
  • reward campaign and earn/staking schemas remain separate;
  • writers and projectors reuse the loans pattern where appropriate;
  • legacy per-exchange tables are untouched;
  • tests cover at least one representative reward source and one earn source.
  • Verification:
go test ./internal/db ./internal/worker/repository ./internal/worker/activities
  • Stop and ask:
  • if product-family boundaries need to change;
  • before destructive migrations or reader changes.

Phase 9: Reader Compatibility And Cutover

Goal: move selected in-repo readers to canonical domain tables only after parity is proven.

Codex goal:

/goal Complete Phase 9 from docs/architecture/terry-domain-projection-roadmap.md without stopping until selected in-repo readers can use canonical domain data or compatibility views, old and new outputs are compared during an overlap window, and rollback is documented without dropping legacy tables.

Codex plan command:

/plan Phase 9 from docs/architecture/terry-domain-projection-roadmap.md. Goal: cut selected readers over safely. Context: search for fct_* readers, API/UI/reporting code and docs. Constraints: only in-repo readers unless external inventory is provided; do not remove legacy tables; rollback path must be explicit. Done when: plan identifies consumers, query/view changes, overlap validation, rollback and verification commands.

Codex task packet:

  • Prerequisites: acceptable parity for the selected domain and reader scope.
  • Allowed changes: in-repo reader queries, compatibility views, docs and tests.
  • Likely files:
  • files found by rg "fct_.*loans|fct_.*earn|fct_.*launchpool"
  • internal/db/migrations/*.sql for compatibility views
  • API/UI/reporting code if present in this repo
  • docs for cutover and rollback
  • Acceptance criteria:
  • selected readers use canonical tables or compatibility views;
  • old and new outputs are compared;
  • rollback does not require data loss;
  • legacy tables remain available.
  • Verification:
  • reader-specific tests plus parity query checks.
  • Stop and ask:
  • if readers live outside this repo;
  • before changing external API contracts or production dashboards.

Phase 10: Config-Only Onboarding And Operations

Goal: make simple source onboarding repeatable through config/admin workflows after backend contracts are stable.

Codex goal:

/goal Complete Phase 10 from docs/architecture/terry-domain-projection-roadmap.md without stopping until a simple new source can be configured without Go code, dry-run validated, activated, ingested, transformed to JSON if needed, schema-correlated, projected and inspected through documented operations.

Codex plan command:

/plan Phase 10 from docs/architecture/terry-domain-projection-roadmap.md. Goal: operationalize config-only onboarding. Context: inspect Terry config/admin flows, validation, transformer metadata, StoreStructuredData schema-correlation config and operations docs. Constraints: do not store secrets in config; UI work only if an existing admin surface is in scope; complex sources still use coded handlers. Done when: plan defines admin/CLI flow, dry-run preview, activation checks, handler discovery and end-to-end verification.

Codex task packet:

  • Prerequisites: Phase 2 config matching and Phase 3 transformer registry exist for the source type; projection path exists for the target domain.
  • Allowed changes: backend config/admin APIs or CLI flows, validation preview, handler capability metadata, docs and tests. UI changes are allowed only if an existing UI/admin surface is present and in scope.
  • Likely files:
  • Terry config/admin code if present in this repo
  • internal/worker/loader/types.go
  • internal/worker/loader/*config*
  • docs/terry/configuration.md
  • docs/terry/operations.md
  • tests for validation/dry-run/activation
  • Acceptance criteria:
  • simple source onboarding requires no Go code;
  • dry-run validates source, transform and schema correlation config;
  • activation refuses invalid config;
  • runbook shows transformed JSON, structured records and projected domain records.
  • Verification:
  • focused config/admin tests and documented dry-run command.
  • Stop and ask:
  • if a new UI/admin product surface must be designed;
  • before storing credentials or secrets in Terry config.

Codex Implementation Readiness

Codex can implement these phases as separate development goals.

Phase Codex readiness Why
Phase 0 Ready Docs-only architecture decision task.
Phase 1 Ready Local schema/repository/activity contract; no workflow semantic change.
Phase 2 Ready after Phase 1 Localized to StoreStructuredDataActivity and config validation.
Phase 3 Ready Activity and registry can be implemented independently; wiring is deferred.
Phase 4 Needs approval Temporal workflow semantic change.
Phase 5 Ready after Phase 0/1 Additive loans schema/writer work.
Phase 6 Ready after Phase 5 Projection activity can be tested behind interfaces.
Phase 7 Partially ready Needs representative payloads/fixtures or DB snapshots for real parity.
Phase 8 Ready after Phase 7 Reuses proven pilot pattern for more domains.
Phase 9 Partially ready External readers require user-provided inventory/access.
Phase 10 Partially ready Backend/CLI is feasible; new UI scope must be decided separately.

Recommended execution order:

Phase 0
  -> Phase 1
  -> Phase 2
  -> Phase 3
  -> Phase 4
  -> Phase 5
  -> Phase 6
  -> Phase 7
  -> Phase 8
  -> Phase 9
  -> Phase 10

Phase 2 and Phase 3 can be developed in parallel after Phase 1 if separate Codex sessions are used, but Phase 4 should wait for both.

Decision Rules For New Exchanges

Use config-only mapping when:

  • the source is JSON-like and stable;
  • one endpoint contains all required records;
  • field mapping is path-based;
  • built-in transforms are sufficient;
  • no complex pagination, signing or browser behavior is required.

Use coded processor when:

  • the source needs SDK-specific behavior;
  • multiple endpoints must be joined;
  • Playwright scraping is required;
  • asset mapping or rate normalization is source-specific;
  • quarantine logic needs custom diagnostics;
  • source instability requires custom fallback behavior.

In both cases, the writer should target the same schema-matched structured and domain contracts.

Risks

  • A too-generic schema can hide domain meaning and make analytics harder.
  • A too-code-heavy processor model can make every new source require a release.
  • Skipping schema-matched structured storage makes replay and debugging harder.
  • Skipping domain tables makes Timescale aggregates and cross-exchange queries weak.
  • Migrating all product families at once makes parity hard to prove.

Start with Phase 0 and Phase 1, then pilot loans through Phases 3-6. Loans are the best first domain because their current docs already describe a shared cross-exchange contract, while still exercising asset mapping, rates, liquidity, quarantine and aggregation.