Skip to content

KuCoin Earn Pipeline

SyncKuCoinEarnWorkflow собирает KuCoin Earn All Products через подтвержденный web API и пишет snapshots в TimescaleDB.

Endpoints

  • GET https://www.kucoin.com/_pxapi/pool-staking/v5/currencies-products?keyword=&layer=3&order_by=&exclude_advanced_group_apr=1&lang=en_US

Primary UI page:

  • https://www.kucoin.com/earn

Confirmed notes:

  • discovery показал, что https://www.kucoin.com/earn закрыт Cloudflare challenge для обычного curl, но сам ingestion endpoint работает как обычный JSON web API;
  • для confirmed request хватает browser-like headers (Accept, Accept-Language, Referer, User-Agent, x-site) без обязательной пользовательской cookie-сессии;
  • keyword=&layer=3 соответствует UI-фильтру All Products.

Mapping

  • project = currency row (USDT, ETH, SOL, ...)
  • subpool = конкретный продукт (id:category_group2|category|type)
  • project_coin <= top-level currency
  • reward_coin <= product income_currency
  • project_status агрегируется по всем subpool statuses
  • subpool_status:
  • PENDING => Upcoming
  • lock_end_time < now или ENDED => Ended
  • ONGOING / FULL / INTERESTING => Active
  • mixed catalog intentionally preserves Simple Earn, Staking, Advanced, KuCoin Wealth rows from the All Products view

Persistence

  • ingest_kucoin_earn_snapshot($1::jsonb)
  • request path marker: /_pxapi/pool-staking/v5/currencies-products?...
  • source marker: /_pxapi/pool-staking/v5/currencies-products

Tables:

  • fct_kucoin_earn_snapshot
  • fct_kucoin_earn_subpool_snapshot
  • fct_kucoin_earn_response_snapshot
  • fct_kucoin_earn_quarantine
  • fct_kucoin_earn_5min
  • fct_kucoin_earn_daily

Manual Run

docker compose exec -T temporal-admin-tools temporal workflow start \
  --address temporal:7233 --namespace default --task-queue default \
  --type SyncKuCoinEarnWorkflow \
  --workflow-id sync-kucoin-earn-manual-$(date +%s) \
  --input '{}'

Verification

  • Workflow result counters: projects_seen, projects_upserted, subpools_seen, subpools_upserted, quarantine_rows
  • SQL:
  • SELECT COUNT(*) FROM fct_kucoin_earn_snapshot;
  • SELECT COUNT(*) FROM fct_kucoin_earn_subpool_snapshot;
  • SELECT reason, COUNT(*) FROM fct_kucoin_earn_quarantine GROUP BY reason ORDER BY COUNT(*) DESC;