macula_upload behaviour (macula v10.1.1)

View Source

Behaviour for supervised content uploads (the receiver side of a push-initiated upload — PLAN_PUSH_UPLOAD.md Phase 6, the recipient macula_pusher pushes at).

advertise/5,6 registers an upload procedure. Each inbound push starts one supervised, ephemeral receiver: it reads the manifest macula_pusher passed as the stream's open-time Args (decoded via macula_manifest:from_wire/1, the same decode content sharing's own _content.get_manifest path already relies on), accumulates pushed chunks, and once the sender half-closes, reassembles and verifies them against the manifest with macula_manifest:verify/2 — receiver-side verification, never sender-trusted, matching content-sharing's existing "content is self-verifying by hash" model exactly; the sender's claimed manifest proves nothing on its own. Delivers {ok, Mcid, Bytes} | {error, _} to Module:handle_uploaded/2, and publishes sharing.upload_started_v1 / sharing.upload_completed_v1 mesh facts around each push's lifetime.

A correction from the plan's literal wording

The plan that named this module described it as "mirroring macula_download's shape" while, in the same sentence, also saying its public API is advertise/advertise_directmacula_download doesn't have those; it has start_link/4,5 (a one-shot, caller-initiated fetch). What this module actually is — a long-lived, ADVERTISED provider spawning one ephemeral child per inbound push — mirrors macula_streamer's shape, not macula_download's. Built directly on top of it: this module IS a macula_streamer callback module internally (?MODULE is passed as macula_streamer:advertise/6's own Module argument, closing over the caller's Module/Args in its own state), reusing Phase 5's supervision, client_stream receive loop (handle_chunk/2), and abort-wired cancel for free — "inherits... cancel from day one, nothing to retrofit later," per the plan's own intro to this phase.

== The terminal reply, and why handle_uploaded/2 return values are ignored ==

Once the sender half-closes, macula_streamer calls this module's handle_eof/1 (Phase 6's new optional callback on that module) — the one place with access to the raw stream needed to set a terminal reply. Verification happens right there: {reply, {ok, Mcid}, NewState} on success or {reply, {error, Reason}, NewState} on failure, which macula_streamer turns into macula_stream:set_reply/2 / set_error/2 — exactly the channel macula_pusher's own macula:await_reply/1 blocks on. The Mcid itself needs no separate round trip: it's a field already present in the manifest both sides hold, deterministic from the same bytes the sender is pushing (macula_manifest mirrors macula-station's own algorithm BYTE-FOR-BYTE) — trustworthy to echo back specifically BECAUSE verification against the actually-received bytes already passed, not despite it.

Module:handle_uploaded/2 — the LOCAL delivery, to whatever application registered this upload handler — fires separately, from this module's own terminate/2 (after the wire-level reply is already set, so a slow local callback can never block it). Because terminate/2 can no longer act on a {stop, _, _} return the way a live handle_info clause could, handle_uploaded/2's return value is ignored — any(), not {noreply,_} | {stop,_,_} — the same shape macula_stream_sink's own handle_close/2 already uses for the identical reason (it too only ever fires from terminate/2). This is a deliberate divergence from macula_feeder/macula_download's handle_fed/handle_downloaded callbacks, which DO get a meaningful {stop,_,_} because they fire from a live handle_info clause, not terminate/2 — "mirrors macula_feeder's shape exactly" does not hold for this specific callback's return contract, traced to WHERE each one is actually invoked from.

A malicious or buggy sender that never half-closes, or that pushes more chunks than its own manifest declared, is bounded: exceeding the declared chunk_count stops the receiver (a genuine {error, too_many_chunks}, abort-wired same as any other non-normal stop) rather than accumulating chunks without limit — the manifest is a system-boundary input from an untrusted remote peer, worth guarding explicitly, unlike macula_manifest:verify/2's own eventual size_mismatch, which would only catch this after the fact.

Direct-dial

advertise_direct/6,7 is macula_streamer:advertise_direct/6,7 verbatim, mode => client_stream folded into Opts before forwarding — see that module's own "Direct-dial" section. (Building this surfaced a real, separate bug in macula_streamer:advertise_direct/7 itself: it called the arity-5 advertise/5, silently discarding whatever modeOpts carried, so ANY direct-dial-advertised client_stream procedure — not just this module's — would have been served as server_stream instead, with no error anywhere to say so. Fixed at the source, in macula_streamer.erl itself, per this project's "fix bugs in owned libraries immediately" rule — not specific to Phase 6, but found while building it.)

Example

   -module(doc_upload).
   -behaviour(macula_upload).
   -export([init/1, handle_uploaded/2]).
  
   init(Parent) -> {ok, Parent}.
  
   handle_uploaded(Result, Parent) ->
       Parent ! {uploaded, Result},
       ok.
   {ok, _Sup} = macula_upload:advertise(Pool, Realm,
       <<"bulk.ingest">>, doc_upload, self()).

Summary

Functions

Advertise Procedure on Pool/Realm as an upload target. Each push sent at it starts one supervised, ephemeral receiver, threading state through Module:init/1 and delivering the final outcome to Module:handle_uploaded/2.

As advertise/5. Opts may include announce (default true) for this module's OWN sharing.upload_* facts.

As advertise/5, and additionally publishes a signed procedure_advertisement DHT record naming this pool's connected station as the server, so macula_pusher:start_link_direct/5,6 can resolve and dial here directly. See macula_streamer:advertise_direct/6,7.

As advertise_direct/6, with Opts forwarded to macula_direct_dial:publish_advertisement/5 — e.g. cert_chain => ChainPem (Slice 7c Direction B, managed realms only).

Stop advertising Procedure.

Callbacks

handle_uploaded/2

-callback handle_uploaded(Result :: {ok, macula:mcid(), binary()} | {error, term()}, State :: term()) ->
                             any().

init/1

-callback init(Args :: term()) -> {ok, State :: term()} | {stop, Reason :: term()}.

Functions

advertise(Pool, Realm, Procedure, Module, Args)

-spec advertise(macula:pool(), macula:realm(), macula:procedure(), module(), term()) ->
                   {ok, pid()} | {error, term()}.

Advertise Procedure on Pool/Realm as an upload target. Each push sent at it starts one supervised, ephemeral receiver, threading state through Module:init/1 and delivering the final outcome to Module:handle_uploaded/2.

advertise(Pool, Realm, Procedure, Module, Args, Opts)

-spec advertise(macula:pool(), macula:realm(), macula:procedure(), module(), term(), map()) ->
                   {ok, pid()} | {error, term()}.

As advertise/5. Opts may include announce (default true) for this module's OWN sharing.upload_* facts.

unadvertise(Pool, Realm, Procedure)

-spec unadvertise(macula:pool(), macula:realm(), macula:procedure()) -> ok.

Stop advertising Procedure.