macula_streamer behaviour (macula v10.1.1)

View Source

Behaviour for supervised streaming RPC providers.

advertise_stream/5 on the raw SDK takes a bare handler fun invoked as Handler(StreamPid, Args) in a transient process spawned per inbound STREAM_OPEN (see the internal macula_station_link advertise_stream/5 — "this link spawns a server-side macula_stream and dispatches Handler(StreamPid, Args) in a transient process"). This is the provider-side counterpart to macula_stream_sink: each inbound stream starts one supervised macula_streamer child (under a simple_one_for_one factory this module owns), threading state through Module:init/1 and Module:handle_open/2, and publishing streaming.started_v1 / streaming.completed_v1 mesh facts around the stream's lifetime.

Sending is push-based and driven from outside the callback: once Module:handle_open/2 has done whatever registration it needs (e.g. stashing self() in a registry keyed by some connection id), any process holding this streamer's pid can call send/2,3 / close/1 on it. This module does not prescribe the discovery mechanism.

For client_stream mode — a consumer pushing chunks INTO the provider, e.g. a batch upload — export the optional handle_chunk/2 callback (mirroring macula_stream_sink's consumer-side callback exactly) and this module drives the same linked-reader recv/2 loop for you, on the provider side. A server_stream-mode module that doesn't export it is unaffected.

A client_stream provider that also needs to hand the consumer a terminal result (not just accept chunks) exports the optional handle_eof/1 callback: called once, when the consumer's own close_send/1 surfaces here as end-of-stream, in place of the default unconditional {stop, normal, State}. Returning {reply, Result, NewState} sets the stream's terminal reply (macula_stream:set_reply/2 for {ok, Value}, set_error/2 for {error, Reason}) so the consumer's own macula:await_reply/1,2 unblocks with it, before stopping. A module that doesn't export handle_eof/1 keeps the exact prior behavior — no reply is ever set, eof just stops the stream.

This is the general-purpose RPC streaming feature (call_stream/5, advertise_stream/5, e.g. a logs.tail_v1-style procedure) — unrelated to content sharing's own chunked-transfer protocol; see macula_feeder / macula_download for that.

Cancel

Stopping this gen_server for any non-normal reason (a crash, the underlying stream dying, Module:handle_open/2/handle_chunk/2 returning a non-normal stop) sends the peer an explicit macula_stream:abort/3 STREAM_ERROR, not just a graceful close — the peer learns the transfer was cancelled/failed instead of mistaking it for an ordinary end-of-stream. A normal stop closes both sides cleanly instead.

Direct-dial

advertise/5,6 registers the handler with the pool's advertise- gossip mechanism only — nothing published lets a caller on another station find this procedure without a route having propagated between the two stations first. advertise_direct/6,7 does that AND publishes a signed procedure_advertisement DHT record naming this pool's currently-connected station as the server — the exact same record type and publish function macula_response:advertise_direct/6,7 uses for plain RPC (a procedure_advertisement does not distinguish RPC from streaming), so a caller using macula_stream_sink:start_link_direct/5,6 can resolve and dial here directly, in one hop, regardless of whether the two stations have a routing edge between them.

Example

   -module(log_tailer_provider).
   -behaviour(macula_streamer).
   -export([init/1, handle_open/2]).
  
   init(Registry) -> {ok, Registry}.
  
   handle_open(#{topic := Topic}, Registry) ->
       Registry ! {tailer_ready, Topic, self()},
       {ok, Registry}.
   {ok, _Sup} = macula_streamer:advertise(Pool, Realm,
       <<"logs.tail_v1">>, log_tailer_provider, self()).
  
   %% elsewhere, once the provider has announced its pid:
   ok = macula_streamer:send(TailerPid, <<"a log line\n">>).

A client_stream-mode provider exports handle_chunk/2 instead, and never calls send/2,3 itself — the consumer is the one pushing:

   -module(batch_upload_provider).
   -behaviour(macula_streamer).
   -export([init/1, handle_open/2, handle_chunk/2]).
  
   init(Parent) -> {ok, {Parent, []}}.
  
   handle_open(_StreamArgs, State) -> {ok, State}.
  
   handle_chunk(Chunk, {Parent, Acc}) ->
       {noreply, {Parent, [Chunk | Acc]}}.
   {ok, _Sup} = macula_streamer:advertise(Pool, Realm,
       <<"bulk.ingest">>, batch_upload_provider, self(),
       #{mode => client_stream}).

Summary

Functions

Advertise Procedure on Pool/Realm. Starts a private factory supervisor for per-stream provider children and registers a dispatch handler with macula:advertise_stream/5. Returns the supervisor pid so the caller can supervise it (or ignore it).

As advertise/5. Opts may include announce (default true), mode (default server_stream), and reuse_sup — an existing supervisor pid (as returned by a prior advertise/5,6 call) to re-send the wire ADVERTISE frame on without starting a new factory supervisor. Use this for periodic re-advertise (a station's registration for a procedure is tied to the connection that sent it, and does not survive that connection being replaced — see advertise_direct/6,7's own doc) — calling plain advertise/5,6 on a timer would leak one orphaned supervisor per tick, since each call otherwise starts a fresh one.

As advertise/5, and additionally publishes a signed procedure_advertisement DHT record naming this pool's connected station as the server, so macula_stream_sink:start_link_direct/5,6 can resolve and dial here directly. Identity signs it — reuse the same one across re-advertises so each one doesn't mint a fresh advertiser identity.

As advertise_direct/6, with Opts forwarded BOTH to advertise/6 (so mode/announce/reuse_sup apply here too, e.g. mode => client_stream) and to macula_direct_dial:publish_advertisement/5 (e.g. cert_chain => ChainPem, Slice 7c Direction B, managed realms only) — each side reads only the keys it recognizes, so one Opts map serves both. reuse_sup matters here specifically: a station's wire-level registration for a procedure is tied to whichever connection sent the ADVERTISE frame, and does not survive that connection being replaced (reconnect, station-side eviction, etc.) — a periodic re-advertise with reuse_sup => Sup (the pid this function returned the first time) re-sends both the wire frame and the DHT record without leaking a new supervisor per tick.

Close the send side of the stream.

Send a chunk out on the stream this streamer owns.

As send/2, with an explicit encoding.

Stop advertising. Does not stop the factory supervisor returned by advertise/5,6 — callers that want to tear it down should exit(Sup, shutdown) themselves.

Callbacks

handle_chunk/2

(optional)
-callback handle_chunk(Chunk :: term(), State :: term()) ->
                          {noreply, NewState :: term()} | {stop, Reason :: term(), NewState :: term()}.

handle_eof/1

(optional)
-callback handle_eof(State :: term()) ->
                        {noreply, NewState :: term()} |
                        {reply, {ok, term()} | {error, term()}, NewState :: term()} |
                        {stop, Reason :: term(), NewState :: term()}.

handle_open/2

-callback handle_open(StreamArgs :: term(), State :: term()) ->
                         {ok, NewState :: term()} | {stop, Reason :: term(), NewState :: term()}.

init/1

-callback init(Args :: term()) -> {ok, State :: term()} | {stop, Reason :: term()}.

terminate/2

(optional)
-callback terminate(Reason :: term(), State :: term()) -> any().

Functions

advertise(Pool, Realm, Procedure, Module, Args)

-spec advertise(macula:pool(), macula:realm(), macula:procedure(), module(), term()) ->
                   {ok, pid()} | {error, term()}.

Advertise Procedure on Pool/Realm. Starts a private factory supervisor for per-stream provider children and registers a dispatch handler with macula:advertise_stream/5. Returns the supervisor pid so the caller can supervise it (or ignore it).

advertise(Pool, Realm, Procedure, Module, Args, Opts)

-spec advertise(macula:pool(), macula:realm(), macula:procedure(), module(), term(), map()) ->
                   {ok, pid()} | {error, term()}.

As advertise/5. Opts may include announce (default true), mode (default server_stream), and reuse_sup — an existing supervisor pid (as returned by a prior advertise/5,6 call) to re-send the wire ADVERTISE frame on without starting a new factory supervisor. Use this for periodic re-advertise (a station's registration for a procedure is tied to the connection that sent it, and does not survive that connection being replaced — see advertise_direct/6,7's own doc) — calling plain advertise/5,6 on a timer would leak one orphaned supervisor per tick, since each call otherwise starts a fresh one.

close(Pid)

-spec close(pid()) -> ok.

Close the send side of the stream.

send(Pid, Chunk)

-spec send(pid(), binary()) -> ok | {error, term()}.

Send a chunk out on the stream this streamer owns.

send(Pid, Chunk, Encoding)

-spec send(pid(), binary() | term(), macula_stream:encoding()) -> ok | {error, term()}.

As send/2, with an explicit encoding.

unadvertise(Pool, Realm, Procedure)

-spec unadvertise(macula:pool(), macula:realm(), macula:procedure()) -> ok.

Stop advertising. Does not stop the factory supervisor returned by advertise/5,6 — callers that want to tear it down should exit(Sup, shutdown) themselves.