macula_content_transfer (macula v10.1.1)
View SourceAddressable content-store put/get, with a real, peer-visible abort, real pause/resume, and (for chunked content) parallel multi-stream chunk transfer — the foundation macula_feeder/ macula_download (and, later, macula_pusher/macula_upload) build on. See PLAN_PUSH_UPLOAD.md, Phases 1-3.
macula:put_content/2/get_content/2 are ONE opaque blocking call each: pick a link, open a dedicated content stream, run the transfer, close the stream — no handle exists mid-transfer, so cancelling meant killing whatever process happened to be blocked in the call. That killed the caller's wait, but never touched the underlying stream: macula_station_link — not the killed process — owns the stream's content_stream_bufs/content_pending state, so a terminate/2 that never runs (because the blocking call's own process was exit(_, kill)d mid-gen_server:call) leaked that state on the link forever, cleaned up only by the eventual content_call_timeout firing against an already-dead caller.
This module owns the picked link and the open stream itself, in its own gen_server state, updated as soon as a worker process resolves them — so cancel/1,3 can always reach in and tear the stream down explicitly, from any point in the transfer's lifecycle, with nothing left to time out.
Cancel is a real abort, not a dropped connection
cancel/3 resets every currently-open content stream's send side via macula_quic:reset_stream/2 (through macula_station_link: abort_content_stream/4) — a QUIC RESET_STREAM frame the PEER's own read genuinely observes as {quic, stream_closed, PeerStream, {reset, Code}}, not merely a connection that went away. This is NOT macula_stream:abort/3 (streaming RPC's abort) — that targets a macula_stream gen_server's own STREAM_ERROR application framing, and a content-transfer stream is not one of those; it is a raw QUIC dedicated stream owned directly by macula_station_link (see that module's open_content_stream/1). The two "stream" concepts share a name and nothing else — do not reuse macula_stream:abort/3 here.
Lifecycle
start_put/2,3, start_put_station/4,5, start_get/2,3, start_get_station/4,5 return {ok, Pid} immediately; the resolve/dial sequence runs in a linked worker. await/1,2 blocks for the outcome ({ok, Mcid} / {ok, Bytes} / {error, Reason}) — repeatable and from any process; the result is cached once known. The process does NOT self-terminate on completion (a second await/1 after success must still answer) — call cancel/1 when done with the handle, whether the transfer succeeded, failed, or is still in flight; on an already-resolved transfer this is a pure reap (nothing left to abort).
Pause/resume (chunked content only)
Single-block content is one wire round trip — there is no "between chunks" for it to pause at, so pause/1 on a single-block transfer is a harmless no-op (the transfer just runs to completion). For chunked content, each chunk's own put/get is still ONE uninterrupted blocking call underneath (pausing mid-chunk would leave a half-sent block the station can't verify) — what pause/1 actually controls is whether the NEXT chunk starts, on EVERY stream, once whichever chunk is currently in flight on it finishes. resume/1 re-arms every stream from exactly its own next un-sent/un-fetched chunk, never from the start. cancel/1,3 still works at any point, paused or not — every stream with a chunk step in flight has it killed and reset exactly as described above; a stream idle between chunks (nothing in flight) just gets its stream reset directly.
Multi-stream parallel chunk transfer (chunked content only)
Chunks are distributed round-robin (Index rem StreamCount) across up to stream_count dedicated content streams on the SAME link (Opts's stream_count key, default 4, capped at the actual chunk count so a 2-chunk transfer never opens more than 2 streams) — each stream runs its own independent chunk-by-chunk loop concurrently, all driven by this ONE gen_server via handle_continue/2 (never by the streams' own worker processes, which each do exactly one network call and report back). The manifest is put (or, for a get, its chunks reassembled and verified) only once every stream has drained its own share. A get doesn't know the chunk count — and therefore how many streams are worth opening — until its manifest is fetched, so it starts on the ONE stream the connect step already opened and expands to more once the count is known; a put knows upfront and opens every extra stream immediately. If opening an extra stream fails, this degrades gracefully to fewer streams rather than failing the transfer — a single-stream transfer is still a correct, if slower, one. A single stream's own chunk failing (a genuine {error, _}, not a crash) fails the WHOLE transfer, same as a sequential transfer would — every other stream's in-flight work is killed and every stream reset before the caller's await/1,2 sees the error.
Correlation-id registry
Each transfer mints a share_id (crypto:strong_rand_bytes(16), overridable via Opts's share_id key so a wrapper that already publishes it in a sharing.*_started_v1 mesh fact — see macula_feeder/macula_download — can keep the same id) and registers it in macula_content_transfer_registry, so a caller that only saw the id in a published fact, not the pid, can still resolve it to cancel/1,3.
Summary
Functions
Block for the transfer's outcome: {ok, Mcid} (put), {ok, Bytes} (get), or {error, Reason}. Safe to call more than once, from more than one process, before or after the result is known.
As await/1 with an explicit timeout on THIS call only — a timeout here does not cancel the transfer itself.
As cancel/3 with a default code/message.
Cancel Pid's transfer and reap the process. Resets every currently-open content stream with Code — genuinely peer-visible, see the moduledoc. Message is local-only (logged at the link; QUIC RESET_STREAM carries only the numeric code on the wire). If the transfer already resolved (success or failure), this is a pure reap — nothing left to abort. Either way await/1,2 answers {error, cancelled} to anyone still waiting.
Pause a chunked transfer between chunks, on every stream — each stream's in-flight chunk (if any) still completes, no stream closes, but none of them starts its next chunk until resume/1. A no-op on a single-block transfer or one that has already resolved (nothing to pause either way).
Resume a transfer paused via pause/1 — every stream continues from exactly its own next un-sent/un-fetched chunk, never from the start. A no-op if not actually paused, not chunked, or already resolved.
This transfer's share_id, for publishing in a mesh fact or looking itself up later via macula_content_transfer_registry.
Start an addressable get through the pool's own connected link. See macula:get_content/2.
As start_get/2. Opts may carry share_id and stream_count (see start_put/3).
As start_get/2, dialing Station directly — the addressable counterpart to macula:get_content_station/4.
As start_get_station/4. Opts as start_put_station/5.
Start an addressable put through the pool's own connected link (whichever macula_client:pick_connected_link/1 picks).
As start_put/2. Opts may carry share_id (binary, overrides the minted default) and stream_count (positive integer, default 4 — see the moduledoc's "Multi-stream" section; irrelevant for a single-block transfer).
As start_put/2, dialing Station directly (reusing a live link or dialing + waiting up to TimeoutMs for one) instead of picking from the pool's existing links — the addressable counterpart to macula:put_content_station/4.
As start_put_station/4. Opts may carry share_id, stream_count (see start_put/3) plus a per-call TLS trust override for this dial — verify, expected_node_id, pin_tls_cert (see macula:put_content_station/5).
Types
-type dial() :: {pooled, macula:pool()} | {station, macula:pool(), macula_client:seed(), pos_integer(), map()}.
-type kind() :: put | get.
Functions
Block for the transfer's outcome: {ok, Mcid} (put), {ok, Bytes} (get), or {error, Reason}. Safe to call more than once, from more than one process, before or after the result is known.
As await/1 with an explicit timeout on THIS call only — a timeout here does not cancel the transfer itself.
-spec cancel(pid()) -> ok.
As cancel/3 with a default code/message.
-spec cancel(pid(), non_neg_integer(), binary()) -> ok.
Cancel Pid's transfer and reap the process. Resets every currently-open content stream with Code — genuinely peer-visible, see the moduledoc. Message is local-only (logged at the link; QUIC RESET_STREAM carries only the numeric code on the wire). If the transfer already resolved (success or failure), this is a pure reap — nothing left to abort. Either way await/1,2 answers {error, cancelled} to anyone still waiting.
-spec pause(pid()) -> ok.
Pause a chunked transfer between chunks, on every stream — each stream's in-flight chunk (if any) still completes, no stream closes, but none of them starts its next chunk until resume/1. A no-op on a single-block transfer or one that has already resolved (nothing to pause either way).
-spec resume(pid()) -> ok.
Resume a transfer paused via pause/1 — every stream continues from exactly its own next un-sent/un-fetched chunk, never from the start. A no-op if not actually paused, not chunked, or already resolved.
-spec start_get(macula:pool(), macula:mcid()) -> {ok, pid()}.
Start an addressable get through the pool's own connected link. See macula:get_content/2.
-spec start_get(macula:pool(), macula:mcid(), map()) -> {ok, pid()}.
As start_get/2. Opts may carry share_id and stream_count (see start_put/3).
-spec start_get_station(macula:pool(), macula_client:seed(), macula:mcid(), pos_integer()) -> {ok, pid()}.
As start_get/2, dialing Station directly — the addressable counterpart to macula:get_content_station/4.
-spec start_get_station(macula:pool(), macula_client:seed(), macula:mcid(), pos_integer(), map()) -> {ok, pid()}.
As start_get_station/4. Opts as start_put_station/5.
-spec start_put(macula:pool(), binary()) -> {ok, pid()}.
Start an addressable put through the pool's own connected link (whichever macula_client:pick_connected_link/1 picks).
-spec start_put(macula:pool(), binary(), map()) -> {ok, pid()}.
As start_put/2. Opts may carry share_id (binary, overrides the minted default) and stream_count (positive integer, default 4 — see the moduledoc's "Multi-stream" section; irrelevant for a single-block transfer).
-spec start_put_station(macula:pool(), macula_client:seed(), binary(), pos_integer()) -> {ok, pid()}.
As start_put/2, dialing Station directly (reusing a live link or dialing + waiting up to TimeoutMs for one) instead of picking from the pool's existing links — the addressable counterpart to macula:put_content_station/4.
-spec start_put_station(macula:pool(), macula_client:seed(), binary(), pos_integer(), map()) -> {ok, pid()}.
As start_put_station/4. Opts may carry share_id, stream_count (see start_put/3) plus a per-call TLS trust override for this dial — verify, expected_node_id, pin_tls_cert (see macula:put_content_station/5).