macula_content_transfer (macula v10.1.1)

View Source

Addressable content-store put/get, with a real, peer-visible abort, real pause/resume, and (for chunked content) parallel multi-stream chunk transfer — the foundation macula_feeder/ macula_download (and, later, macula_pusher/macula_upload) build on. See PLAN_PUSH_UPLOAD.md, Phases 1-3.

macula:put_content/2/get_content/2 are ONE opaque blocking call each: pick a link, open a dedicated content stream, run the transfer, close the stream — no handle exists mid-transfer, so cancelling meant killing whatever process happened to be blocked in the call. That killed the caller's wait, but never touched the underlying stream: macula_station_link — not the killed process — owns the stream's content_stream_bufs/content_pending state, so a terminate/2 that never runs (because the blocking call's own process was exit(_, kill)d mid-gen_server:call) leaked that state on the link forever, cleaned up only by the eventual content_call_timeout firing against an already-dead caller.

This module owns the picked link and the open stream itself, in its own gen_server state, updated as soon as a worker process resolves them — so cancel/1,3 can always reach in and tear the stream down explicitly, from any point in the transfer's lifecycle, with nothing left to time out.

Cancel is a real abort, not a dropped connection

cancel/3 resets every currently-open content stream's send side via macula_quic:reset_stream/2 (through macula_station_link: abort_content_stream/4) — a QUIC RESET_STREAM frame the PEER's own read genuinely observes as {quic, stream_closed, PeerStream, {reset, Code}}, not merely a connection that went away. This is NOT macula_stream:abort/3 (streaming RPC's abort) — that targets a macula_stream gen_server's own STREAM_ERROR application framing, and a content-transfer stream is not one of those; it is a raw QUIC dedicated stream owned directly by macula_station_link (see that module's open_content_stream/1). The two "stream" concepts share a name and nothing else — do not reuse macula_stream:abort/3 here.

Lifecycle

start_put/2,3, start_put_station/4,5, start_get/2,3, start_get_station/4,5 return {ok, Pid} immediately; the resolve/dial sequence runs in a linked worker. await/1,2 blocks for the outcome ({ok, Mcid} / {ok, Bytes} / {error, Reason}) — repeatable and from any process; the result is cached once known. The process does NOT self-terminate on completion (a second await/1 after success must still answer) — call cancel/1 when done with the handle, whether the transfer succeeded, failed, or is still in flight; on an already-resolved transfer this is a pure reap (nothing left to abort).

Pause/resume (chunked content only)

Single-block content is one wire round trip — there is no "between chunks" for it to pause at, so pause/1 on a single-block transfer is a harmless no-op (the transfer just runs to completion). For chunked content, each chunk's own put/get is still ONE uninterrupted blocking call underneath (pausing mid-chunk would leave a half-sent block the station can't verify) — what pause/1 actually controls is whether the NEXT chunk starts, on EVERY stream, once whichever chunk is currently in flight on it finishes. resume/1 re-arms every stream from exactly its own next un-sent/un-fetched chunk, never from the start. cancel/1,3 still works at any point, paused or not — every stream with a chunk step in flight has it killed and reset exactly as described above; a stream idle between chunks (nothing in flight) just gets its stream reset directly.

Multi-stream parallel chunk transfer (chunked content only)

Chunks are distributed round-robin (Index rem StreamCount) across up to stream_count dedicated content streams on the SAME link (Opts's stream_count key, default 4, capped at the actual chunk count so a 2-chunk transfer never opens more than 2 streams) — each stream runs its own independent chunk-by-chunk loop concurrently, all driven by this ONE gen_server via handle_continue/2 (never by the streams' own worker processes, which each do exactly one network call and report back). The manifest is put (or, for a get, its chunks reassembled and verified) only once every stream has drained its own share. A get doesn't know the chunk count — and therefore how many streams are worth opening — until its manifest is fetched, so it starts on the ONE stream the connect step already opened and expands to more once the count is known; a put knows upfront and opens every extra stream immediately. If opening an extra stream fails, this degrades gracefully to fewer streams rather than failing the transfer — a single-stream transfer is still a correct, if slower, one. A single stream's own chunk failing (a genuine {error, _}, not a crash) fails the WHOLE transfer, same as a sequential transfer would — every other stream's in-flight work is killed and every stream reset before the caller's await/1,2 sees the error.

Correlation-id registry

Each transfer mints a share_id (crypto:strong_rand_bytes(16), overridable via Opts's share_id key so a wrapper that already publishes it in a sharing.*_started_v1 mesh fact — see macula_feeder/macula_download — can keep the same id) and registers it in macula_content_transfer_registry, so a caller that only saw the id in a published fact, not the pid, can still resolve it to cancel/1,3.

Summary

Functions

Block for the transfer's outcome: {ok, Mcid} (put), {ok, Bytes} (get), or {error, Reason}. Safe to call more than once, from more than one process, before or after the result is known.

As await/1 with an explicit timeout on THIS call only — a timeout here does not cancel the transfer itself.

As cancel/3 with a default code/message.

Cancel Pid's transfer and reap the process. Resets every currently-open content stream with Code — genuinely peer-visible, see the moduledoc. Message is local-only (logged at the link; QUIC RESET_STREAM carries only the numeric code on the wire). If the transfer already resolved (success or failure), this is a pure reap — nothing left to abort. Either way await/1,2 answers {error, cancelled} to anyone still waiting.

Pause a chunked transfer between chunks, on every stream — each stream's in-flight chunk (if any) still completes, no stream closes, but none of them starts its next chunk until resume/1. A no-op on a single-block transfer or one that has already resolved (nothing to pause either way).

Resume a transfer paused via pause/1 — every stream continues from exactly its own next un-sent/un-fetched chunk, never from the start. A no-op if not actually paused, not chunked, or already resolved.

This transfer's share_id, for publishing in a mesh fact or looking itself up later via macula_content_transfer_registry.

Start an addressable get through the pool's own connected link. See macula:get_content/2.

As start_get/2. Opts may carry share_id and stream_count (see start_put/3).

As start_get/2, dialing Station directly — the addressable counterpart to macula:get_content_station/4.

Start an addressable put through the pool's own connected link (whichever macula_client:pick_connected_link/1 picks).

As start_put/2. Opts may carry share_id (binary, overrides the minted default) and stream_count (positive integer, default 4 — see the moduledoc's "Multi-stream" section; irrelevant for a single-block transfer).

As start_put/2, dialing Station directly (reusing a live link or dialing + waiting up to TimeoutMs for one) instead of picking from the pool's existing links — the addressable counterpart to macula:put_content_station/4.

As start_put_station/4. Opts may carry share_id, stream_count (see start_put/3) plus a per-call TLS trust override for this dial — verify, expected_node_id, pin_tls_cert (see macula:put_content_station/5).

Types

dial/0

-type dial() ::
          {pooled, macula:pool()} | {station, macula:pool(), macula_client:seed(), pos_integer(), map()}.

kind/0

-type kind() :: put | get.

Functions

await(Pid)

-spec await(pid()) -> {ok, term()} | {error, term()}.

Block for the transfer's outcome: {ok, Mcid} (put), {ok, Bytes} (get), or {error, Reason}. Safe to call more than once, from more than one process, before or after the result is known.

await(Pid, Timeout)

-spec await(pid(), timeout()) -> {ok, term()} | {error, term()}.

As await/1 with an explicit timeout on THIS call only — a timeout here does not cancel the transfer itself.

cancel(Pid)

-spec cancel(pid()) -> ok.

As cancel/3 with a default code/message.

cancel(Pid, Code, Message)

-spec cancel(pid(), non_neg_integer(), binary()) -> ok.

Cancel Pid's transfer and reap the process. Resets every currently-open content stream with Code — genuinely peer-visible, see the moduledoc. Message is local-only (logged at the link; QUIC RESET_STREAM carries only the numeric code on the wire). If the transfer already resolved (success or failure), this is a pure reap — nothing left to abort. Either way await/1,2 answers {error, cancelled} to anyone still waiting.

pause(Pid)

-spec pause(pid()) -> ok.

Pause a chunked transfer between chunks, on every stream — each stream's in-flight chunk (if any) still completes, no stream closes, but none of them starts its next chunk until resume/1. A no-op on a single-block transfer or one that has already resolved (nothing to pause either way).

resume(Pid)

-spec resume(pid()) -> ok.

Resume a transfer paused via pause/1 — every stream continues from exactly its own next un-sent/un-fetched chunk, never from the start. A no-op if not actually paused, not chunked, or already resolved.

share_id(Pid)

-spec share_id(pid()) -> binary().

This transfer's share_id, for publishing in a mesh fact or looking itself up later via macula_content_transfer_registry.

start_get(Pool, Mcid)

-spec start_get(macula:pool(), macula:mcid()) -> {ok, pid()}.

Start an addressable get through the pool's own connected link. See macula:get_content/2.

start_get(Pool, Mcid, Opts)

-spec start_get(macula:pool(), macula:mcid(), map()) -> {ok, pid()}.

As start_get/2. Opts may carry share_id and stream_count (see start_put/3).

start_get_station(Pool, Station, Mcid, TimeoutMs)

-spec start_get_station(macula:pool(), macula_client:seed(), macula:mcid(), pos_integer()) ->
                           {ok, pid()}.

As start_get/2, dialing Station directly — the addressable counterpart to macula:get_content_station/4.

start_get_station(Pool, Station, Mcid, TimeoutMs, Opts)

-spec start_get_station(macula:pool(), macula_client:seed(), macula:mcid(), pos_integer(), map()) ->
                           {ok, pid()}.

As start_get_station/4. Opts as start_put_station/5.

start_put(Pool, Bytes)

-spec start_put(macula:pool(), binary()) -> {ok, pid()}.

Start an addressable put through the pool's own connected link (whichever macula_client:pick_connected_link/1 picks).

start_put(Pool, Bytes, Opts)

-spec start_put(macula:pool(), binary(), map()) -> {ok, pid()}.

As start_put/2. Opts may carry share_id (binary, overrides the minted default) and stream_count (positive integer, default 4 — see the moduledoc's "Multi-stream" section; irrelevant for a single-block transfer).

start_put_station(Pool, Station, Bytes, TimeoutMs)

-spec start_put_station(macula:pool(), macula_client:seed(), binary(), pos_integer()) -> {ok, pid()}.

As start_put/2, dialing Station directly (reusing a live link or dialing + waiting up to TimeoutMs for one) instead of picking from the pool's existing links — the addressable counterpart to macula:put_content_station/4.

start_put_station(Pool, Station, Bytes, TimeoutMs, Opts)

-spec start_put_station(macula:pool(), macula_client:seed(), binary(), pos_integer(), map()) ->
                           {ok, pid()}.

As start_put_station/4. Opts may carry share_id, stream_count (see start_put/3) plus a per-call TLS trust override for this dial — verify, expected_node_id, pin_tls_cert (see macula:put_content_station/5).