Skip to content

System Overview

Актуальный scope

Проект содержит следующие рабочие направления:

  • синхронизация бирж из CoinGecko в exchanges;
  • синхронизация каталога сетей в networks_catalog и связанных таблицах;
  • синхронизация монет/сетей с 8 бирж в exchange_assets* и conservative global mapping;
  • snapshot Gate Launchpool в fct_gate_launchpool_* (Timescale hypertables + continuous aggregates);
  • snapshot Gate Simple Earn в fct_gate_simple_earn_* (Timescale hypertables + continuous aggregates);
  • snapshot Bybit Launchpool в fct_bybit_launchpool_* (Timescale hypertables + continuous aggregates);
  • snapshot Bybit Easy Earn в fct_bybit_easy_earn_* (Timescale hypertables + continuous aggregates);
  • snapshot KuCoin GemPool в fct_kucoin_launchpool_* (Timescale hypertables + continuous aggregates);
  • snapshot KuCoin Earn в fct_kucoin_earn_* (Timescale hypertables + continuous aggregates);
  • snapshot SDK-only loans catalogs для Gate / Bybit / KuCoin / WhiteBIT в fct_<exchange>_loans_*;
  • scraping BingX Launchpool через Playwright;
  • snapshot BingX Launchpool в fct_bingx_launchpool_* (Timescale hypertables + continuous aggregates).
  • scraping Bitget Launchpool через Playwright;
  • snapshot Bitget Launchpool в fct_bitget_launchpool_* (Timescale hypertables + continuous aggregates).
  • scraping BingX XPool через Playwright;
  • snapshot BingX XPool в fct_bingx_xpool_* (Timescale hypertables + continuous aggregates).
  • Terry universal ingestion runtime по persisted data_source_id с auto-upsert paused schedule run-ingestion.

Компоненты

  • cmd/worker — Temporal worker
  • internal/worker/workflows/sync_exchanges.go
  • internal/worker/workflows/sync_networks_catalog.go
  • internal/worker/workflows/sync_exchange_coins.go
  • internal/worker/workflows/sync_exchange_coins_child.go
  • internal/worker/workflows/sync_gate_launchpool.go
  • internal/worker/workflows/sync_bybit_launchpool.go
  • internal/worker/workflows/sync_bybit_easy_earn.go
  • internal/worker/workflows/sync_gate_simple_earn.go
  • internal/worker/workflows/sync_gate_launchpool_schedule.go
  • internal/worker/workflows/sync_gate_loans.go
  • internal/worker/workflows/sync_bybit_loans.go
  • internal/worker/workflows/sync_kucoin_launchpool.go
  • internal/worker/workflows/sync_kucoin_earn.go
  • internal/worker/workflows/sync_kucoin_loans.go
  • internal/worker/workflows/sync_whitebit_loans.go
  • internal/worker/workflows/scrape_bingx_launchpool.go
  • internal/worker/workflows/sync_bingx_launchpool.go
  • internal/worker/workflows/scrape_bitget_launchpool.go
  • internal/worker/workflows/sync_bitget_launchpool.go
  • internal/worker/workflows/scrape_bingx_xpool.go
  • internal/worker/workflows/sync_bingx_xpool.go
  • internal/worker/workflows/run_ingestion.go
  • internal/worker/workflows/run_ingestion_schedule.go
  • internal/worker/activities/sync_exchanges.go
  • internal/worker/activities/sync_networks_catalog.go
  • internal/worker/activities/sync_exchange_coins.go
  • internal/worker/activities/sync_gate_launchpool.go
  • internal/worker/activities/sync_bybit_launchpool.go
  • internal/worker/activities/sync_bybit_easy_earn.go
  • internal/worker/activities/sync_gate_simple_earn.go
  • internal/worker/activities/sync_gate_launchpool_schedule.go
  • internal/worker/activities/sync_gate_loans.go
  • internal/worker/activities/sync_bybit_loans.go
  • internal/worker/activities/sync_kucoin_launchpool.go
  • internal/worker/activities/sync_kucoin_earn.go
  • internal/worker/activities/sync_kucoin_loans.go
  • internal/worker/activities/sync_whitebit_loans.go
  • internal/worker/activities/scrape_bingx_launchpool.go
  • internal/worker/activities/sync_bingx_launchpool.go
  • internal/worker/activities/scrape_bitget_launchpool.go
  • internal/worker/activities/sync_bitget_launchpool.go
  • internal/worker/activities/scrape_bingx_xpool.go
  • internal/worker/activities/sync_bingx_xpool.go
  • internal/worker/activities/run_ingestion_schedule.go
  • internal/worker/loader/activity.go
  • internal/db/migrations/00001_baseline.sql
  • internal/db/migrations/00002_network_catalog.sql
  • internal/db/migrations/00003_exchange_coins_catalog.sql
  • internal/db/migrations/00005_gate_launchpool_snapshot_schema.sql
  • internal/db/migrations/00006_gate_launchpool_aggregates_policies.sql
  • internal/db/migrations/00020_gate_simple_earn_snapshot_schema.sql
  • internal/db/migrations/00021_gate_simple_earn_aggregates_policies.sql
  • internal/db/migrations/00007_bingx_launchpool_snapshot_schema.sql
  • internal/db/migrations/00008_bingx_launchpool_aggregates_policies.sql
  • internal/db/migrations/00009_bingx_xpool_snapshot_schema.sql
  • internal/db/migrations/00010_bingx_xpool_aggregates_policies.sql
  • internal/db/migrations/00013_bitget_launchpool_snapshot_schema.sql
  • internal/db/migrations/00014_bitget_launchpool_aggregates_policies.sql

Поток данных

flowchart LR
  CG[CoinGecko API] --> EX[SyncExchangesWorkflow]
  EX --> DB[(PostgreSQL exchanges)]
  CGP[CoinGecko /asset_platforms] --> NC[SyncNetworksCatalogWorkflow]
  CH[chainid.network] --> NC
  LL[DefiLlama /v2/chains] --> NC
  NC --> NDB[(PostgreSQL networks_catalog)]
  EXS[Bybit/Binance/Gate/Bitget/BingX/KuCoin/OKX/MEXC APIs] --> EC[SyncExchangeCoinsWorkflow]
  EC --> CDB[(PostgreSQL exchange_assets/global_assets)]
  GLP[Gate Launchpool API project-list] --> GLPW[SyncGateLaunchpoolWorkflow]
  GLPW --> TS[(Timescale fct_gate_launchpool_* + cagg)]
  BYLP[Bybit Launchpool API] --> BYLPW[SyncBybitLaunchpoolWorkflow]
  BYLPW --> BYLPTS[(Timescale fct_bybit_launchpool_* + cagg)]
  BYEE[Bybit Easy Earn API] --> BYEEW[SyncBybitEasyEarnWorkflow]
  BYEEW --> BYEETS[(Timescale fct_bybit_easy_earn_* + cagg)]
  GSE[Gate Simple Earn API market/list] --> GSEW[SyncGateSimpleEarnWorkflow]
  GSEW --> GSETS[(Timescale fct_gate_simple_earn_* + cagg)]
  KULP[KuCoin GemPool API] --> KULPW[SyncKuCoinLaunchpoolWorkflow]
  KULPW --> KULPTS[(Timescale fct_kucoin_launchpool_* + cagg)]
  KUE[KuCoin Earn API] --> KUEW[SyncKuCoinEarnWorkflow]
  KUEW --> KUETS[(Timescale fct_kucoin_earn_* + cagg)]
  GLO[Gate SDK loans] --> GLOW[SyncGateLoansWorkflow]
  GLOW --> GLTS[(Timescale fct_gate_loans_* + cagg)]
  BYLO[Bybit SDK loans] --> BYLOW[SyncBybitLoansWorkflow]
  BYLOW --> BYLTS[(Timescale fct_bybit_loans_* + cagg)]
  KLO[KuCoin SDK loans] --> KLOW[SyncKuCoinLoansWorkflow]
  KLOW --> KLTS[(Timescale fct_kucoin_loans_* + cagg)]
  WLO[WhiteBIT SDK loans] --> WLOW[SyncWhitebitLoansWorkflow]
  WLOW --> WLTS[(Timescale fct_whitebit_loans_* + cagg)]
  BX[BingX Launchpool UI/API] --> XP[ScrapeBingXLaunchpoolWorkflow]
  XP --> BLPW[SyncBingXLaunchpoolWorkflow]
  BLPW --> BTS[(Timescale fct_bingx_launchpool_* + cagg)]
  BTP[Bitget Launchpool UI/API] --> BTPW[ScrapeBitgetLaunchpoolWorkflow]
  BTPW --> BTLPW[SyncBitgetLaunchpoolWorkflow]
  BTLPW --> BTTS[(Timescale fct_bitget_launchpool_* + cagg)]
  BXX[BingX XPool UI/API] --> XPW[ScrapeBingXXPoolWorkflow]
  XPW --> BXPW[SyncBingXXPoolWorkflow]
  BXPW --> BXTS[(Timescale fct_bingx_xpool_* + cagg)]
  TDS[Terry data source id] --> RIW[RunIngestionWorkflow]
  RIW --> TR[(LoaderActivity.Execute)]
  TR --> TOUT[(LoaderOutput JSON + metadata)]

DB стратегия

После squashed baseline поддерживается только fresh-install через reset окружений.