diff --git a/docs/DEPLOY_WORKERS_MCP.md b/docs/DEPLOY_WORKERS_MCP.md index bf67d3d..d17bd48 100644 --- a/docs/DEPLOY_WORKERS_MCP.md +++ b/docs/DEPLOY_WORKERS_MCP.md @@ -14,7 +14,7 @@ As tools deste Worker chamam fontes públicas oficiais: APIs JSON (BCB, IBGE, IPEA, SICONFI, Open Finance Directory) e os ZIPs/CSV da CVM em `dados.cvm.gov.br` para fundos abertos (`cvm_fund`). -Fora deste Worker (CDA/carteira, lâmina, perfil, B3 COTAHIST, ANBIMA XLS, +Fora deste Worker (lâmina, perfil, B3 COTAHIST, ANBIMA XLS, registry FTS5, code mode): `pip install openfindata` ou FastAPI interno. `cvm_fund` no Worker: @@ -22,10 +22,12 @@ registry FTS5, code mode): `pip install openfindata` ou FastAPI interno. - `dataset=catalog` + `cnpj` ou `q` — cadastro oficial RCVM 175 (`registro_fundo_classe.zip`: fundo + classe + subclasse). `cad_fi.csv` não lista fundos já adaptados à Resolução 175. -- `dataset=daily` + `cnpj` — série de cotas INF_DIARIO do mês - (`year`/`month`; default = mês UTC corrente). -- Não inclui CDA (composição da carteira): feed mensal atrasado e pesado, - separado do informe diário. Não usa Mais Retorno. +- `dataset=daily` + `cnpj` — série de cotas INF_DIARIO. Sem `year`/`month` + usa o mês mais recente no diretório CVM; `months` (1–3) olha para trás. +- `dataset=periods` — stamps YYYYMM publicados (`product=CDA` ou `INF_DIARIO`). +- `dataset=holdings` + `cnpj` — CDA (carteira). Sem `year`/`month` usa o + CDA mais recente. Scan em stream por CNPJ; linhas `CONFID` são sigilo, + não carteira aberta completa. Não usa Mais Retorno. ## Deploy @@ -57,9 +59,9 @@ curl -sS https://openfindata.com.br/health ``` Upstream calls no Worker têm timeout (15s) e teto de payload (2 MB; -8 MB no Directory Open Finance; 16 MB / 45s só em `cvm_fund`, porque o -cadastro e o INF_DIARIO mensal vêm em ZIP). O Worker faz scan em stream -do CSV deflate (não materializa os ~48 MB do INF_DIARIO). Séries BCB sem +8 MB no Directory Open Finance; 32 MB / 45s só em `cvm_fund`, porque +cadastro, INF_DIARIO e CDA vêm em ZIP). O Worker faz scan em stream do +CSV deflate (não materializa o CDA descompactado). Séries BCB sem intervalo caem em `last_n≤200`. Rate limits de `/mcp` não mudam. `/mcp` usa Workers Rate Limit bindings (não Cloudflare Queues): 60 req / diff --git a/docs/MCP_SURFACE.md b/docs/MCP_SURFACE.md index afa010a..785a421 100644 --- a/docs/MCP_SURFACE.md +++ b/docs/MCP_SURFACE.md @@ -6,8 +6,8 @@ > and [`docs/DEPLOY_WORKERS_MCP.md`](DEPLOY_WORKERS_MCP.md). Public `/mcp` is > 60 req/60s per IP with a 20/10s burst; overflow is 429 + Retry-After (no queue, > no code mode, no API key). Worker tools: the 9 JSON macro sources plus -> `cvm_fund` (RCVM 175 cadastro + INF_DIARIO). CDA/lâmina/perfil stay on -> the internal FastAPI catalog below. +> `cvm_fund` (RCVM 175 cadastro + INF_DIARIO + CDA holdings/periods). +> Lâmina/perfil stay on the internal FastAPI catalog below. ## Problem @@ -86,7 +86,7 @@ findata_run_code (code mode, opt-in) | `bcb_ptax` | `/ptax/usd`, `/ptax/usd/period`, `/ptax/{currency}` | `start`+`end` → period | | `bcb_focus` | `/focus/{indicators,annual,monthly,selic,top5}` | `horizon`, `panel`, `indicator` | | `cvm_company` | companies search/list, `fca/*`, `ipe` | `dataset=search\|list\|fca_*\|filings` | -| `cvm_fund` | `funds`, `funds/cadastro`, `funds/{daily,holdings,lamina,profile,periods}`, returns | `dataset`; `cnpj`/`q` → RCVM 175 | +| `cvm_fund` | `funds`, `funds/cadastro`, `funds/{daily,holdings,lamina,profile,periods}`, returns | `dataset`; `cnpj`/`q` → RCVM 175; omit year/month → latest CDA/INF_DIARIO | | `cvm_structured_fund` | `funds/{fii,fidc,fip}/*` | `kind` + `dataset` | | `b3_index` | index portfolio + monthly + list | `dataset`, omit `symbol` to list | | `tesouro_bonds` | bonds list/search/history | `dataset` | @@ -101,7 +101,7 @@ findata_run_code (code mode, opt-in) deliverable, not an afterthought. - **Consolidation can hide endpoint-specific params behind an enum.** Mitigated by documenting each `dataset`/`kind` value and validating bad combinations with - a `400` (e.g. `cvm_fund dataset=holdings` requires `cnpj`+`month`), matching the + a `400` (e.g. `cvm_fund dataset=holdings` requires `cnpj`; month defaults to latest), matching the REST API's `ValueError → 400` behaviour. - **Discoverability of rare endpoints.** A handful of niche REST routes are not individually surfaced as tools. They remain fully reachable over REST and via @@ -122,4 +122,6 @@ local/agent use. A production deployment should run it in a real sandbox - `bcb_ptax(start=2024-01-02, end=2024-01-05)` → daily PTAX USD series (the handoff's headline flow). - `cvm_fund(dataset=catalog, cnpj="38.729.027/0001-92")` → cadastro RCVM 175 (classe, condomínio, PL). - `cvm_fund(dataset=daily, cnpj="38729027000192", year=2026, month=8)` → INF_DIARIO (cota/PL/cotistas). +- `cvm_fund(dataset=periods, product="CDA")` → YYYYMM stamps + `latest`. +- `cvm_fund(dataset=holdings, cnpj="38729027000192")` → latest CDA carteira (CONFID = sigilo). - `findata_run_code("import findata; ...")` → runs in the sandbox, returns captured stdout. diff --git a/docs/agents/orientation.md b/docs/agents/orientation.md index af0935f..eaf62b0 100644 --- a/docs/agents/orientation.md +++ b/docs/agents/orientation.md @@ -24,7 +24,7 @@ mortos, fronteira atual, openfindata, findata. | Rede nos unit tests? | Proibido. `respx` nos unitários; live só `@pytest.mark.integration` | `AGENTS.md`, CI nightly | | Credenciais no repo? | Nunca. Fontes públicas preferidas; BdD usa billing project do operador via env | `AGENTS.md`, `docs/SOURCES_WITH_AUTH.md` | | MCP: 1:1 com REST ou curado? | Catálogo curado em `mcp_app` (~25 tools); REST intacto | `docs/MCP_SURFACE.md` | -| MCP público vs interno? | Worker `openfindata.com.br/mcp` (macro JSON + `cvm_fund`); FastAPI/Tailscale tem o catálogo completo | `docs/DEPLOY_WORKERS_MCP.md` | +| MCP público vs interno? | Worker `openfindata.com.br/mcp` (macro JSON + `cvm_fund` catalog/daily/holdings/periods); FastAPI/Tailscale tem o catálogo completo | `docs/DEPLOY_WORKERS_MCP.md` | | Code mode no MCP? | Opt-in via `FINDATA_MCP_CODE_MODE=1`; off por default | `docs/MCP_SURFACE.md`, `mcp_app.py` | | Charts: quais deps de plot? | Não adicionar matplotlib/pandas/plotly etc. só para gráfico | `AGENTS.md`, `docs/CHART_STANDARDS.md` | | Publicar no PyPI? | Só com aprovação humana explícita | `AGENTS.md` | diff --git a/src/findata/api/mcp_app.py b/src/findata/api/mcp_app.py index e42fe08..afbd66a 100644 --- a/src/findata/api/mcp_app.py +++ b/src/findata/api/mcp_app.py @@ -59,6 +59,7 @@ holdings, ipe, lamina, + latest_period, list_periods, profile, ) @@ -73,6 +74,36 @@ _MIN_YEAR_B3_COTAHIST = 1986 # B3 publishes COTAHIST since 1986 _RGF_MAX_PERIOD = 3 # RGF quadrimestre runs 1..3 +_DAILY_MONTHS_MAX = 12 +_YYYYMM_LEN = 6 + + +def _stamp_to_year_month(stamp: str) -> tuple[int, int]: + if len(stamp) != _YYYYMM_LEN or not stamp.isdigit(): + raise HTTPException(404, f"invalid CVM period stamp {stamp!r}") + return int(stamp[:4]), int(stamp[_YYYYMM_LEN - 2 :]) + + +def _add_months(year: int, month: int, delta: int) -> tuple[int, int]: + absolute = year * 12 + (month - 1) + delta + return absolute // 12, absolute % 12 + 1 + + +def _lookback_months(end_year: int, end_month: int, count: int) -> list[tuple[int, int]]: + return [_add_months(end_year, end_month, offset) for offset in range(-(count - 1), 1)] + + +async def _resolve_cvm_month(year: int | None, month: int | None, product: str) -> tuple[int, int]: + if (year is None) != (month is None): + raise HTTPException( + 400, "pass both `year` and `month`, or omit both for the latest published file" + ) + if year is not None and month is not None: + return year, month + latest = await latest_period("FI", f"DOC/{product}") + if not latest: + raise HTTPException(404, f"no published {product} period") + return _stamp_to_year_month(latest) # ── Registry: the entry point ───────────────────────────────────── @@ -316,8 +347,13 @@ async def cvm_fund( q: str | None = Query( None, min_length=2, description="catalog: name fragment when CNPJ is unknown" ), - year: int | None = Query(None, description="Reference year (required except catalog/periods)"), + year: int | None = Query( + None, description="Reference year; omit with month for latest CDA/INF_DIARIO" + ), month: int | None = Query(None, ge=1, le=12, description="Reference month (monthly datasets)"), + months: int = Query( + 1, ge=1, le=_DAILY_MONTHS_MAX, description="daily: lookback months including the end month" + ), horizon: Literal["monthly", "yearly"] = Query( "monthly", description="returns granularity (dataset=returns)" ), @@ -339,8 +375,10 @@ async def cvm_fund( """Open funds in one tool. ``catalog`` with ``cnpj`` or ``q`` reads the official RCVM 175 registro (fundo+classe+subclasse). Bare ``catalog`` still pages the legacy ``cad_fi.csv`` (non-adapted funds only). ``periods`` lists YYYYMM - stamps. ``daily`` is INF_DIARIO (cota/PL/cotistas). CDA ``holdings`` is a - separate monthly delayed feed and is not the cota series. + stamps. ``daily`` is INF_DIARIO (cota/PL/cotistas); omit ``year``/``month`` + for the latest published month, or pass ``months`` to look back. CDA + ``holdings`` is a separate monthly delayed feed — omit ``year``/``month`` + for the latest CDA. CONFID rows are sigilo, not a complete open book. """ if dataset == "catalog": if cnpj or q: @@ -348,17 +386,26 @@ async def cvm_fund( return (await funds.get_fund_catalog(True, None))[:limit] if dataset == "periods": return await list_periods("FI", f"DOC/{product}") - if year is None: - raise HTTPException(400, f"dataset={dataset} requires `year`") if dataset == "holdings": - if not cnpj or month is None: - raise HTTPException(400, "dataset=holdings requires `cnpj` and `month`") + if not cnpj: + raise HTTPException(400, "dataset=holdings requires `cnpj`") + year, month = await _resolve_cvm_month(year, month, "CDA") block_list = [b.strip() for b in blocks.split(",") if b.strip()] if blocks else None return await holdings.get_fund_holdings(cnpj, year, month, block_list) + if dataset == "daily": + year, month = await _resolve_cvm_month(year, month, "INF_DIARIO") + if months == 1: + return (await funds.get_fund_daily(year, month, cnpj))[:limit] + series: list[Any] = [] + for stamp_year, stamp_month in _lookback_months(year, month, months): + series.extend(await funds.get_fund_daily(stamp_year, stamp_month, cnpj)) + if len(series) >= limit: + break + return series[:limit] + if year is None: + raise HTTPException(400, f"dataset={dataset} requires `year`") if month is None: raise HTTPException(400, f"dataset={dataset} requires `month`") - if dataset == "daily": - return (await funds.get_fund_daily(year, month, cnpj))[:limit] if dataset == "lamina": return (await lamina.get_fund_lamina(year, month, cnpj))[:limit] if dataset == "profile": diff --git a/tests/test_mcp_surface.py b/tests/test_mcp_surface.py index a2de6e5..8f2c21f 100644 --- a/tests/test_mcp_surface.py +++ b/tests/test_mcp_surface.py @@ -12,12 +12,19 @@ from __future__ import annotations import importlib +import re +import httpx import pytest +import respx from fastapi.testclient import TestClient from findata.api.app import app from findata.api.mcp_app import mcp_app +from findata.http_client import clear_cache +from findata.sources.cvm._directory import _listing_cache +from tests.test_cvm_fund_cadastro import _daily_zip +from tests.test_cvm_funds import _LISTING_HTML, _make_cda_zip EXPECTED_TOOLS = 25 # curated tools with code mode OFF (the default) EXPECTED_REST_OPERATIONS = 97 # all REST routes (unconditional); bump when the surface changes @@ -103,12 +110,21 @@ def test_consolidated_tool_validates_missing_selector_args() -> None: assert "year" in r.json()["detail"] -def test_cvm_fund_holdings_requires_cnpj_and_month() -> None: - r = TestClient(mcp_app).get("/cvm/fund", params={"dataset": "holdings", "year": 2024}) +def test_cvm_fund_holdings_requires_cnpj() -> None: + r = TestClient(mcp_app).get("/cvm/fund", params={"dataset": "holdings"}) assert r.status_code == 400 assert "cnpj" in r.json()["detail"] +def test_cvm_fund_holdings_year_without_month_is_400() -> None: + r = TestClient(mcp_app).get( + "/cvm/fund", + params={"dataset": "holdings", "cnpj": "12.345.678/0001-99", "year": 2024}, + ) + assert r.status_code == 400 + assert "year" in r.json()["detail"] or "month" in r.json()["detail"] + + # ── code-mode gating ─────────────────────────────────────────────── @@ -154,3 +170,46 @@ def test_structured_fund_fip_rejects_dataset() -> None: "/cvm/structured-fund", params={"kind": "fip", "year": 2024, "dataset": "geral"} ) assert r.status_code == 400 + + +@respx.mock +def test_cvm_fund_holdings_defaults_to_latest_cda() -> None: + clear_cache() + _listing_cache.invalidate() + respx.get("https://dados.cvm.gov.br/dados/FI/DOC/CDA/DADOS/").mock( + return_value=httpx.Response(200, text=_LISTING_HTML) + ) + respx.get(re.compile(r"https://.*cda_fi_202603\.zip")).mock( + return_value=httpx.Response(200, content=_make_cda_zip()) + ) + r = TestClient(mcp_app).get( + "/cvm/fund", params={"dataset": "holdings", "cnpj": "12.345.678/0001-99"} + ) + assert r.status_code == 200 + body = r.json() + assert len(body) == 3 + assert {row["bloco"] for row in body} == {"BLC_4", "BLC_8"} + + +@respx.mock +def test_cvm_fund_daily_months_lookback() -> None: + clear_cache() + payload = _daily_zip() + respx.get(re.compile(r"https://.*inf_diario_fi_202607\.zip")).mock( + return_value=httpx.Response(200, content=payload) + ) + respx.get(re.compile(r"https://.*inf_diario_fi_202608\.zip")).mock( + return_value=httpx.Response(200, content=payload) + ) + r = TestClient(mcp_app).get( + "/cvm/fund", + params={ + "dataset": "daily", + "cnpj": "38729027000192", + "year": 2026, + "month": 8, + "months": 2, + }, + ) + assert r.status_code == 200 + assert len(r.json()) == 4 diff --git a/workers/mcp/package.json b/workers/mcp/package.json index 086c77a..c600eaf 100644 --- a/workers/mcp/package.json +++ b/workers/mcp/package.json @@ -1,7 +1,7 @@ { "name": "openfindata-mcp", "private": true, - "version": "0.3.1", + "version": "0.3.2", "type": "module", "scripts": { "dev": "wrangler dev", diff --git a/workers/mcp/public/index.html b/workers/mcp/public/index.html index 391520a..b1eefbb 100644 --- a/workers/mcp/public/index.html +++ b/workers/mcp/public/index.html @@ -40,10 +40,10 @@

MCP público no Cloudflare Workers

  • ipea_series ipea_search
  • tesouro_siconfi
  • openfinance_directory
    • -
  • cvm_fund — cadastro RCVM 175 + série de cotas INF_DIARIO (não CDA)
    • +
  • cvm_fund — cadastro RCVM 175, cotas INF_DIARIO e carteira CDA

    • - B3/ANBIMA/registry, CDA/lâmina/perfil e a API REST completa continuam no + B3/ANBIMA/registry, lâmina/perfil e a API REST completa continuam no pacote Python (pip install openfindata) e no FastAPI interno.

    Cliente MCP:

    diff --git a/workers/mcp/src/index.ts b/workers/mcp/src/index.ts index deb6072..b9f5915 100644 --- a/workers/mcp/src/index.ts +++ b/workers/mcp/src/index.ts @@ -11,7 +11,7 @@ export default { return Response.json({ status: "ok", surface: "mcp-worker", - version: "0.3.1", + version: "0.3.2", mcp: "/mcp", }); } diff --git a/workers/mcp/src/lib/http.ts b/workers/mcp/src/lib/http.ts index 376d96a..7726a68 100644 --- a/workers/mcp/src/lib/http.ts +++ b/workers/mcp/src/lib/http.ts @@ -1,4 +1,4 @@ -const USER_AGENT = "openfindata-mcp/0.3.1 (+https://github.com/robertoecf/OpenFinData)"; +const USER_AGENT = "openfindata-mcp/0.3.2 (+https://github.com/robertoecf/OpenFinData)"; export const FETCH_TIMEOUT_MS = 15_000; export const MAX_RESPONSE_BYTES = 2_000_000; diff --git a/workers/mcp/src/lib/zipCsv.ts b/workers/mcp/src/lib/zipCsv.ts index 07c9efa..f8c304e 100644 --- a/workers/mcp/src/lib/zipCsv.ts +++ b/workers/mcp/src/lib/zipCsv.ts @@ -53,6 +53,27 @@ function findZipEntry(zip: Uint8Array, name: string): ZipEntry { throw new Error(`zip entry not found: ${name}`); } +export function listZipEntryNames(zip: Uint8Array): string[] { + const names: string[] = []; + let offset = 0; + while (offset + 30 <= zip.length) { + const sig = u32(zip, offset); + if (sig === CENTRAL_SIG) { + break; + } + if (sig !== LOCAL_SIG) { + throw new Error("invalid zip local header"); + } + const compSize = u32(zip, offset + 18); + const nameLen = u16(zip, offset + 26); + const extraLen = u16(zip, offset + 28); + const nameStart = offset + 30; + names.push(LATIN1.decode(zip.subarray(nameStart, nameStart + nameLen))); + offset = nameStart + nameLen + extraLen + compSize; + } + return names; +} + function inflateRawStream(data: Uint8Array): ReadableStream { return new Blob([data]).stream().pipeThrough(new DecompressionStream("deflate-raw")); } diff --git a/workers/mcp/src/server.ts b/workers/mcp/src/server.ts index b009b50..6044421 100644 --- a/workers/mcp/src/server.ts +++ b/workers/mcp/src/server.ts @@ -24,7 +24,7 @@ function wrap(run: (args: T) => Promise) { export function createServer() { const server = new McpServer({ name: "openfindata", - version: "0.3.1", + version: "0.3.2", websiteUrl: "https://openfindata.com.br", }); @@ -141,13 +141,19 @@ export function createServer() { "cvm_fund", { description: - "CVM fund registry + open-fund cota series. catalog: RCVM 175 cadastro by CNPJ or name (any registered type; forma_condominio says Aberto/Fechado). daily: INF_DIARIO cota/PL/cotistas for one month (fundos abertos). Not CDA carteira. Not Mais Retorno.", + "CVM open-fund raw layer. catalog: RCVM 175 cadastro by CNPJ or name. daily: INF_DIARIO cota/PL/cotistas (omit year/month for latest published month; months=1..3 lookback). periods: available CDA or INF_DIARIO YYYYMM stamps. holdings: CDA carteira for one month (omit year/month for latest; CONFID is sigilo, not a complete open book). Not Mais Retorno.", inputSchema: { - dataset: z.enum(["catalog", "daily"]).default("catalog"), + dataset: z.enum(["catalog", "daily", "holdings", "periods"]).default("catalog"), cnpj: z.string().optional().describe("Fund CNPJ, punctuated or digits"), q: z.string().optional().describe("catalog: name fragment when CNPJ is unknown"), - year: z.number().int().min(2021).optional(), + year: z.number().int().min(2018).optional(), month: z.number().int().min(1).max(12).optional(), + months: z.number().int().min(1).max(3).optional().describe("daily: lookback months including the end month"), + product: z.enum(["CDA", "INF_DIARIO"]).optional().describe("periods: which directory to list"), + blocks: z + .string() + .optional() + .describe("holdings: comma list such as BLC_1,BLC_4 (BLC_1..8, CONFID, PL, FIE)"), limit: z.number().int().min(1).max(2000).optional(), }, }, diff --git a/workers/mcp/src/tools/cvm.test.ts b/workers/mcp/src/tools/cvm.test.ts index cd6f119..452aeb2 100644 --- a/workers/mcp/src/tools/cvm.test.ts +++ b/workers/mcp/src/tools/cvm.test.ts @@ -1,8 +1,8 @@ import assert from "node:assert/strict"; import { deflateRawSync } from "node:zlib"; import { afterEach, test } from "node:test"; -import { cvmFund } from "./cvm.ts"; -import { scanZipCsvForNeedles, zipFile } from "../lib/zipCsv.ts"; +import { cvmFund, listCvmZipMonths } from "./cvm.ts"; +import { listZipEntryNames, scanZipCsvForNeedles, zipFile } from "../lib/zipCsv.ts"; function crc32(data: Uint8Array): number { let crc = 0xffffffff; @@ -91,16 +91,39 @@ afterEach(() => { globalThis.fetch = originalFetch; }); -function mockZip(urlContains: string, zip: Uint8Array) { +function mockRoutes(routes: Array<{ match: string; body: Uint8Array | string; status?: number }>) { globalThis.fetch = (async (input: RequestInfo | URL) => { const url = String(input); - if (!url.includes(urlContains)) { + const route = routes + .filter((item) => url.includes(item.match)) + .sort((a, b) => b.match.length - a.match.length)[0]; + if (!route) { return new Response("missing", { status: 404 }); } - return new Response(zip, { status: 200 }); + return new Response(route.body, { status: route.status ?? 200 }); }) as typeof fetch; } +function mockZip(urlContains: string, zip: Uint8Array) { + mockRoutes([{ match: urlContains, body: zip }]); +} + +const CDA_HTML = + 'cda_fi_202606.zipcda_fi_202607.zip'; +const INF_HTML = + 'inf_diario_fi_202607.zipinf_diario_fi_202608.zip'; + +const CDA_BLC1 = + "CNPJ_FUNDO_CLASSE;DENOM_SOCIAL;DT_COMPTC;TP_APLIC;TP_ATIVO;EMISSOR;QT_POS_FINAL;VL_MERC_POS_FINAL;DS_ATIVO\n" + + "38.729.027/0001-92;AMW PREV;2026-07-31;Titulos Publicos;LFT;TESOURO;10;1000;LFT 2027\n" + + "21.494.444/0001-09;OUTRO;2026-07-31;Titulos Publicos;LFT;TESOURO;1;10;LFT 2027\n"; +const CDA_BLC2 = + "CNPJ_FUNDO_CLASSE;DENOM_SOCIAL;DT_COMPTC;TP_APLIC;TP_ATIVO;EMISSOR;QT_POS_FINAL;VL_MERC_POS_FINAL;DS_ATIVO\n" + + "38.729.027/0001-92;AMW PREV;2026-07-31;Cotas de Fundos;Cota;OUTRO FUNDO;5;500;Fundo X\n"; +const CDA_CONFID = + "CNPJ_FUNDO_CLASSE;DENOM_SOCIAL;DT_COMPTC;TP_APLIC\n" + + "38.729.027/0001-92;AMW PREV;2026-07-31;Sigilo\n"; + test("zipFile reads stored CSV", async () => { const zip = storeZip({ "registro_fundo.csv": FUNDO_CSV }); const bytes = await zipFile(zip, "registro_fundo.csv"); @@ -178,3 +201,142 @@ test("cvm_fund daily matches digit CNPJ to punctuated INF_DIARIO", async () => { assert.equal(body.series[0]?.cnpj, "38.729.027/0001-92"); assert.equal(body.series[0]?.vl_quota, 2.94); }); + +test("listCvmZipMonths reads only matching monthly zips", () => { + const months = listCvmZipMonths( + `${CDA_HTML}skipno`, + "cda_fi_", + ); + assert.deepEqual(months, ["202606", "202607"]); +}); + +test("listZipEntryNames walks every local header", () => { + const zip = storeZip({ "a.csv": "x\n", "b.csv": "y\n" }); + assert.deepEqual(listZipEntryNames(zip), ["a.csv", "b.csv"]); +}); + +test("cvm_fund periods lists CDA stamps and latest", async () => { + mockRoutes([{ match: "/CDA/DADOS/", body: CDA_HTML }]); + const result = await cvmFund({ dataset: "periods" }); + assert.equal(result.isError, undefined); + const body = JSON.parse(result.content[0].text) as { latest: string; periods: string[] }; + assert.equal(body.latest, "202607"); + assert.deepEqual(body.periods, ["202606", "202607"]); +}); + +test("cvm_fund holdings requires cnpj", async () => { + const result = await cvmFund({ dataset: "holdings", year: 2026, month: 7 }); + assert.equal(result.isError, true); +}); + +test("cvm_fund holdings scans every CDA block for one CNPJ", async () => { + mockRoutes([ + { + match: "cda_fi_202607.zip", + body: storeZip({ + "cda_fi_BLC_1_202607.csv": CDA_BLC1, + "cda_fi_BLC_2_202607.csv": CDA_BLC2, + "cda_fi_CONFID_202607.csv": CDA_CONFID, + }), + }, + ]); + const result = await cvmFund({ + dataset: "holdings", + cnpj: "38729027000192", + year: 2026, + month: 7, + }); + assert.equal(result.isError, undefined); + const body = JSON.parse(result.content[0].text) as { + holdings: Array<{ bloco: string; valor_mercado: number | null }>; + }; + assert.equal(body.holdings.length, 3); + assert.deepEqual( + body.holdings.map((row) => row.bloco), + ["BLC_1", "BLC_2", "CONFID"], + ); + assert.equal(body.holdings[0]?.valor_mercado, 1000); +}); + +test("cvm_fund holdings omits year/month and uses the latest CDA stamp", async () => { + mockRoutes([ + { match: "/CDA/DADOS/", body: CDA_HTML }, + { + match: "cda_fi_202607.zip", + body: storeZip({ "cda_fi_BLC_1_202607.csv": CDA_BLC1 }), + }, + ]); + const result = await cvmFund({ dataset: "holdings", cnpj: "38.729.027/0001-92" }); + assert.equal(result.isError, undefined); + const body = JSON.parse(result.content[0].text) as { year: number; month: number; holdings: unknown[] }; + assert.equal(body.year, 2026); + assert.equal(body.month, 7); + assert.equal(body.holdings.length, 1); +}); + +test("cvm_fund holdings honors a block whitelist", async () => { + mockRoutes([ + { + match: "cda_fi_202607.zip", + body: storeZip({ + "cda_fi_BLC_1_202607.csv": CDA_BLC1, + "cda_fi_BLC_2_202607.csv": CDA_BLC2, + }), + }, + ]); + const result = await cvmFund({ + dataset: "holdings", + cnpj: "38729027000192", + year: 2026, + month: 7, + blocks: "BLC_2", + }); + assert.equal(result.isError, undefined); + const body = JSON.parse(result.content[0].text) as { holdings: Array<{ bloco: string }> }; + assert.equal(body.holdings.length, 1); + assert.equal(body.holdings[0]?.bloco, "BLC_2"); +}); + +test("cvm_fund daily months=2 concatenates two INF_DIARIO months", async () => { + const july = + "TP_FUNDO_CLASSE;CNPJ_FUNDO_CLASSE;ID_SUBCLASSE;DT_COMPTC;VL_TOTAL;VL_QUOTA;VL_PATRIM_LIQ;CAPTC_DIA;RESG_DIA;NR_COTST\n" + + "CLASSES - FIF;38.729.027/0001-92;;2026-07-31;1;2.90;100;0;0;1\n"; + mockRoutes([ + { match: "inf_diario_fi_202607.zip", body: storeZip({ "inf_diario_fi_202607.csv": july }) }, + { match: "inf_diario_fi_202608.zip", body: storeZip({ "inf_diario_fi_202608.csv": DAILY_CSV }) }, + ]); + const result = await cvmFund({ + dataset: "daily", + cnpj: "38729027000192", + year: 2026, + month: 8, + months: 2, + }); + assert.equal(result.isError, undefined); + const body = JSON.parse(result.content[0].text) as { + from: string; + to: string; + series: Array<{ dt_comptc: string; vl_quota: number }>; + }; + assert.equal(body.from, "202607"); + assert.equal(body.to, "202608"); + assert.equal(body.series.length, 2); + assert.equal(body.series[0]?.vl_quota, 2.9); + assert.equal(body.series[1]?.vl_quota, 2.94); +}); + +test("cvm_fund daily without year/month uses the latest INF_DIARIO stamp", async () => { + mockRoutes([ + { match: "/INF_DIARIO/DADOS/", body: INF_HTML }, + { + match: "inf_diario_fi_202608.zip", + body: storeZip({ "inf_diario_fi_202608.csv": DAILY_CSV }), + }, + ]); + const result = await cvmFund({ dataset: "daily", cnpj: "38729027000192" }); + assert.equal(result.isError, undefined); + const body = JSON.parse(result.content[0].text) as { year: number; month: number; series: unknown[] }; + assert.equal(body.year, 2026); + assert.equal(body.month, 8); + assert.equal(body.series.length, 1); +}); diff --git a/workers/mcp/src/tools/cvm.ts b/workers/mcp/src/tools/cvm.ts index 02f43a6..645c1b2 100644 --- a/workers/mcp/src/tools/cvm.ts +++ b/workers/mcp/src/tools/cvm.ts @@ -1,13 +1,28 @@ -import { errorResult, getBytes, jsonResult } from "../lib/http.ts"; -import { cnpjDigits, optFloat, scanZipCsv, scanZipCsvForNeedles } from "../lib/zipCsv.ts"; +import { errorResult, getBytes, jsonResult, UpstreamError } from "../lib/http.ts"; +import { + cnpjDigits, + listZipEntryNames, + optFloat, + scanZipCsv, + scanZipCsvForNeedles, +} from "../lib/zipCsv.ts"; const REGISTRO_URL = "https://dados.cvm.gov.br/dados/FI/CAD/DADOS/registro_fundo_classe.zip"; const DAILY_URL = "https://dados.cvm.gov.br/dados/FI/DOC/INF_DIARIO/DADOS/inf_diario_fi_{ym}.zip"; +const CDA_URL = "https://dados.cvm.gov.br/dados/FI/DOC/CDA/DADOS/cda_fi_{ym}.zip"; +const CDA_LISTING_URL = "https://dados.cvm.gov.br/dados/FI/DOC/CDA/DADOS/"; +const DAILY_LISTING_URL = "https://dados.cvm.gov.br/dados/FI/DOC/INF_DIARIO/DADOS/"; const CVM_TIMEOUT_MS = 45_000; -const CVM_MAX_BYTES = 16_000_000; +const CVM_MAX_BYTES = 32_000_000; +const CVM_LISTING_MAX_BYTES = 2_000_000; const CATALOG_CLASS_CAP = 2_000; const DAILY_SCAN_CAP = 2_000; +const HOLDINGS_SCAN_CAP = 5_000; +const DAILY_MONTHS_MAX = 3; + +export type CvmDataset = "catalog" | "daily" | "holdings" | "periods"; +export type CvmPeriodProduct = "CDA" | "INF_DIARIO"; async function fetchCvmZip(url: string): Promise { return getBytes(url, { maxBytes: CVM_MAX_BYTES, timeoutMs: CVM_TIMEOUT_MS }); @@ -177,9 +192,66 @@ async function catalogByName(zip: Uint8Array, q: string, limit: number) { return fundos.map((row) => mapFundo(row, classesByFundo.get(row.ID_Registro_Fundo ?? "") ?? [])); } -function currentYearMonth(): { year: number; month: number } { - const now = new Date(); - return { year: now.getUTCFullYear(), month: now.getUTCMonth() + 1 }; +function formatYm(year: number, month: number): string { + return `${year}${String(month).padStart(2, "0")}`; +} + +function addMonths(year: number, month: number, delta: number): { year: number; month: number } { + const absolute = year * 12 + (month - 1) + delta; + return { year: Math.floor(absolute / 12), month: (absolute % 12) + 1 }; +} + +function lookbackMonths( + endYear: number, + endMonth: number, + count: number, +): { year: number; month: number }[] { + const out: { year: number; month: number }[] = []; + for (let i = count - 1; i >= 0; i -= 1) { + out.push(addMonths(endYear, endMonth, -i)); + } + return out; +} + +export function listCvmZipMonths(html: string, prefix: string): string[] { + const escaped = prefix.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + const re = new RegExp(`${escaped}(\\d{6})\\.zip`, "gi"); + const months = new Set(); + for (const match of html.matchAll(re)) { + months.add(match[1]!); + } + return [...months].sort(); +} + +async function listPublishedMonths(url: string, prefix: string): Promise { + const html = new TextDecoder("utf-8").decode( + await getBytes(url, { maxBytes: CVM_LISTING_MAX_BYTES, timeoutMs: 15_000 }), + ); + return listCvmZipMonths(html, prefix); +} + +async function resolveYearMonth( + args: { year?: number; month?: number }, + listingUrl: string, + prefix: string, +): Promise<{ year: number; month: number } | { error: string }> { + const hasYear = args.year !== undefined; + const hasMonth = args.month !== undefined; + if (hasYear !== hasMonth) { + return { error: "pass both `year` and `month`, or omit both for the latest published file" }; + } + if (hasYear && hasMonth) { + if (args.year! < 2018 || args.month! < 1 || args.month! > 12) { + return { error: "year must be >= 2018 and month 1-12" }; + } + return { year: args.year!, month: args.month! }; + } + const published = await listPublishedMonths(listingUrl, prefix); + const latest = published.at(-1); + if (!latest) { + return { error: `no published files at ${listingUrl}` }; + } + return { year: Number(latest.slice(0, 4)), month: Number(latest.slice(4, 6)) }; } function mapDaily(row: Record) { @@ -197,16 +269,115 @@ function mapDaily(row: Record) { }; } +function cdaBlockLabel(filename: string): string { + const base = filename.split("/").pop() ?? filename; + const upper = base.toUpperCase(); + const parts = base.replace(/\.csv$/i, "").split("_"); + if (parts.length >= 4 && parts[2] === "BLC") { + return `BLC_${parts[3]}`; + } + if (upper.includes("CONFID")) { + return base.toLowerCase().includes("fie") ? "FIE_CONFID" : "CONFID"; + } + if (base.toLowerCase().startsWith("cda_fie")) { + return "FIE"; + } + if (upper.includes("_PL_")) { + return "PL"; + } + return "OTHER"; +} + +function mapHolding(row: Record, bloco: string) { + return { + cnpj: row.CNPJ_FUNDO_CLASSE || row.CNPJ_FUNDO || "", + nome_fundo: row.DENOM_SOCIAL || row.DENOM_CLASSE || "", + dt_referencia: row.DT_COMPTC ?? "", + bloco, + tipo_aplicacao: row.TP_APLIC || null, + tipo_ativo: row.TP_ATIVO || null, + emissor: row.EMISSOR_LIGADO || row.EMISSOR || null, + cnpj_emissor: row.CNPJ_EMISSOR || row.CPF_CNPJ_EMISSOR || null, + tipo_negociacao: row.TP_NEGOC || null, + quantidade_final: optFloat(row.QT_POS_FINAL), + valor_mercado: optFloat(row.VL_MERC_POS_FINAL || row.VL_MERCADO || row.VL_MERC_POSICAO), + descricao: row.DS_ATIVO || row.CD_ATIVO || null, + }; +} + +function parseBlocks(raw: string | undefined): Set | null { + if (!raw?.trim()) { + return null; + } + return new Set( + raw + .split(",") + .map((item) => item.trim().toUpperCase()) + .filter(Boolean), + ); +} + +async function dailySeries( + digits: string, + year: number, + month: number, + limit: number, +): Promise[]> { + const ym = formatYm(year, month); + const zip = await fetchCvmZip(DAILY_URL.replace("{ym}", ym)); + const csvName = `inf_diario_fi_${ym}.csv`; + const rows = ( + await scanZipCsvForNeedles(zip, csvName, cnpjNeedles(digits), Math.min(limit, DAILY_SCAN_CAP)) + ).filter((row) => fieldCnpjEquals(row, ["CNPJ_FUNDO_CLASSE", "CNPJ_FUNDO"], digits)); + return rows.map(mapDaily); +} + +async function holdingsFromZip( + zip: Uint8Array, + digits: string, + blocks: Set | null, + limit: number, +) { + const needles = cnpjNeedles(digits); + const holdings: ReturnType[] = []; + for (const name of listZipEntryNames(zip)) { + if (!name.toLowerCase().endsWith(".csv")) { + continue; + } + const bloco = cdaBlockLabel(name); + if (blocks && !blocks.has(bloco)) { + continue; + } + const rows = ( + await scanZipCsvForNeedles(zip, name, needles, Math.min(limit - holdings.length, HOLDINGS_SCAN_CAP)) + ).filter((row) => fieldCnpjEquals(row, ["CNPJ_FUNDO_CLASSE", "CNPJ_FUNDO"], digits)); + for (const row of rows) { + holdings.push(mapHolding(row, bloco)); + if (holdings.length >= limit) { + return holdings; + } + } + } + return holdings; +} + export async function cvmFund(args: { - dataset?: "catalog" | "daily"; + dataset?: CvmDataset; cnpj?: string; q?: string; year?: number; month?: number; + months?: number; + product?: CvmPeriodProduct; + blocks?: string; limit?: number; }) { const dataset = args.dataset ?? "catalog"; - const limit = Math.min(args.limit ?? (dataset === "daily" ? 500 : 20), dataset === "daily" ? 2000 : 100); + const limit = Math.min( + args.limit ?? (dataset === "holdings" ? 2000 : dataset === "daily" ? 500 : 20), + dataset === "daily" || dataset === "holdings" ? 2000 : 100, + ); + if (dataset === "catalog") { const digits = cnpjDigits(args.cnpj); const q = args.q?.trim() ?? ""; @@ -219,28 +390,79 @@ export async function cvmFund(args: { : await catalogByName(zip, q, limit); return jsonResult(rows); } + + if (dataset === "periods") { + const product = args.product ?? "CDA"; + const listingUrl = product === "CDA" ? CDA_LISTING_URL : DAILY_LISTING_URL; + const prefix = product === "CDA" ? "cda_fi_" : "inf_diario_fi_"; + const periods = await listPublishedMonths(listingUrl, prefix); + return jsonResult({ + source: product === "CDA" ? "cvm_cda" : "cvm_inf_diario", + product, + latest: periods.at(-1) ?? null, + periods, + }); + } + const digits = cnpjDigits(args.cnpj); if (digits.length < 8) { - return errorResult("dataset=daily requires `cnpj`"); + return errorResult(`dataset=${dataset} requires \`cnpj\``); } - const fallback = currentYearMonth(); - const year = args.year ?? fallback.year; - const month = args.month ?? fallback.month; - if (year < 2021 || month < 1 || month > 12) { - return errorResult("dataset=daily requires year>=2021 and month 1-12"); + + if (dataset === "holdings") { + const resolved = await resolveYearMonth(args, CDA_LISTING_URL, "cda_fi_"); + if ("error" in resolved) { + return errorResult(resolved.error); + } + const ym = formatYm(resolved.year, resolved.month); + const zip = await fetchCvmZip(CDA_URL.replace("{ym}", ym)); + const holdings = await holdingsFromZip(zip, digits, parseBlocks(args.blocks), limit); + return jsonResult({ + source: "cvm_cda", + year: resolved.year, + month: resolved.month, + cnpj: digits, + truncated: holdings.length >= limit, + note: "CDA is a delayed monthly feed. CONFID rows are confidential (sigilo), not a complete open book.", + holdings, + }); + } + + const months = Math.min(Math.max(args.months ?? 1, 1), DAILY_MONTHS_MAX); + const resolved = await resolveYearMonth(args, DAILY_LISTING_URL, "inf_diario_fi_"); + if ("error" in resolved) { + return errorResult(resolved.error); + } + const window = lookbackMonths(resolved.year, resolved.month, months); + const series: ReturnType[] = []; + const missing: string[] = []; + let remaining = limit; + for (const stamp of window) { + const ym = formatYm(stamp.year, stamp.month); + try { + const chunk = await dailySeries(digits, stamp.year, stamp.month, remaining); + series.push(...chunk); + remaining = Math.max(limit - series.length, 0); + if (remaining === 0) { + break; + } + } catch (error) { + if (error instanceof UpstreamError && error.status === 404) { + missing.push(ym); + continue; + } + throw error; + } } - const ym = `${year}${String(month).padStart(2, "0")}`; - const zip = await fetchCvmZip(DAILY_URL.replace("{ym}", ym)); - const csvName = `inf_diario_fi_${ym}.csv`; - const rows = ( - await scanZipCsvForNeedles(zip, csvName, cnpjNeedles(digits), Math.min(limit, DAILY_SCAN_CAP)) - ).filter((row) => fieldCnpjEquals(row, ["CNPJ_FUNDO_CLASSE", "CNPJ_FUNDO"], digits)); return jsonResult({ source: "cvm_inf_diario", - year, - month, + year: resolved.year, + month: resolved.month, + months, + from: formatYm(window[0]!.year, window[0]!.month), + to: formatYm(resolved.year, resolved.month), cnpj: digits, - note: "CDA carteira is a separate delayed monthly feed and is not included.", - series: rows.map(mapDaily), + missing, + series, }); }