Macula Clustering Guide

This guide covers Macula's LAN clustering capabilities, including gossip-based discovery, the Cluster API, distribution management, and cookie management. The modules live in src/macula_cluster_system/, separate from src/macula_dist_system/ (distribution over the mesh). The two do not depend on each other.

Note: Clustering via gossip is designed for LAN environments (same subnet). For WAN connectivity across networks, use the relay mesh (see the Dist Over Mesh guide).

Overview

The Macula Cluster API (macula_cluster.erl) provides a standardized interface for:

• Cluster Formation - Starting clusters with various discovery strategies
• Distribution Management - Starting and verifying Erlang distribution
• Cookie Management - Resolving, setting, and persisting cluster cookies
• Node Monitoring - Subscribing to node join/leave events

Cluster Strategies

Any other strategy value returns {error, {unknown_strategy, Strategy}} and does not start distribution. For zero-config discovery on a LAN use gossip; otherwise use static with a node list.

Quick Start

Via macula_cluster API (Recommended)

%% Start with gossip strategy (default). Gossip needs a shared secret of at
%% least 32 bytes, here or in MACULA_GOSSIP_SECRET.
ok = macula_cluster:start_cluster(#{
    strategy => gossip,
    secret => <<"at least 32 bytes of shared secret">>
}).

%% With static node list
ok = macula_cluster:start_cluster(#{
    strategy => static,
    nodes => ['node1@host1', 'node2@host2']
}).

Erlang (Direct Gossip)

%% Start with defaults, the secret taken from MACULA_GOSSIP_SECRET
{ok, _Pid} = macula_cluster_gossip:start_link(#{}).

%% With custom configuration
{ok, _Pid} = macula_cluster_gossip:start_link(#{
    multicast_addr => {230, 1, 1, 251},
    port => 45892,
    broadcast_interval => 1500,
    secret => <<"at least 32 bytes of shared secret">>
}).

Elixir (Phoenix Application)

def start(_type, _args) do
  :macula_cluster.start_cluster(%{
    strategy: :gossip,
    secret: System.fetch_env!("MACULA_GOSSIP_SECRET")
  })

  children = [
    # ... your supervision tree
  ]

  Supervisor.start_link(children, strategy: :one_for_one)
end

Gossip Clustering

The macula_cluster_gossip module provides automatic cluster discovery using UDP multicast, implemented natively in Erlang.

How It Works

1. Join Multicast Group: Each node joins the UDP multicast group (default: 230.1.1.251:45892)
2. Broadcast Heartbeats: Nodes periodically broadcast their Erlang node name
3. Discover Peers: When a heartbeat is received from an unknown node, it's added to discovered set
4. Connect via Distribution: Newly discovered nodes are connected using net_kernel:connect_node/1
5. Full Mesh: All nodes eventually connect to form a full mesh cluster

Configuration Options

Shared Secret

Gossip does not start without a shared secret of at least 32 bytes, given as the secret option or in MACULA_GOSSIP_SECRET. start_link/1 then returns {error, secret_required} or {error, {secret_too_short, #{bytes => N, required => 32}}}, and macula_cluster:start_cluster/1 returns either as {error, {gossip_strategy_failed, Reason}}.

Every gossip packet carries an HMAC-SHA256 tag over the node name:

MACULA_GOSSIP:node_name|<HMAC-SHA256 in hex>

A node takes a packet only when its tag verifies against the secret and the announced name is a node name whose host is a hostname, an IPv4 address or an IPv6 literal.

Environment Variables

MACULA_GOSSIP_ADDR=230.1.1.251
MACULA_GOSSIP_PORT=45892
MACULA_GOSSIP_SECRET=at-least-32-bytes-of-shared-secret
CLUSTER_STRATEGY=gossip

Gossip Query API

%% Get all discovered nodes (may not be connected yet)
Discovered = macula_cluster_gossip:get_discovered().

%% Get connected nodes
Connected = macula_cluster_gossip:get_connected().

%% Force immediate broadcast (useful for testing)
ok = macula_cluster_gossip:broadcast_now().

Distribution Management

Ensure Distributed Mode

ok = macula:ensure_distributed().

If the node is already distributed, returns ok immediately. Otherwise, starts distribution with a generated node name in the format macula_host@hostname.

Cookie Management

macula sets no distribution cookie, and reads or writes no cookie file. A node's cookie is its release's own configuration, which OTP applies when distribution starts:

• The node reads .erlang.cookie in the HOME it started with. OTP refuses a cookie file its group or others can access, and creates an owner-only one (mode 0400) with a random cookie when there is none.
• Nodes that form a cluster need the same cookie. Give each node the same cookie file, owned by the user the node runs as, with mode 0400 or 0600. In a container, mount it read-only at that user's $HOME/.erlang.cookie.
• A release that passes -setcookie itself takes the cookie from wherever that flag gets it. Elixir's mix release reads its releases/COOKIE file, or RELEASE_COOKIE when that is set; prefer the file.

Keep the cookie out of environment variables and command lines: /proc/<pid>/environ, ps and docker inspect show them.

macula:get_cookie/0

Returns the running node's cookie, as erlang:get_cookie/0 does, and raises not_distributed on a node that is not distributed. Deprecated, and removed in 11.0.0: call erlang:get_cookie/0.

Cookie = macula:get_cookie().

macula:set_cookie/1

Sets the cookie of the running node, and raises not_distributed on a node that is not distributed. It writes no file, so a restarted node has its cookie file's cookie again. Deprecated, and removed in 11.0.0: call erlang:set_cookie/1.

ok = macula:set_cookie(my_secret_cookie).
ok = macula:set_cookie(<<"my_secret_cookie">>).

Node Monitoring

Subscribe to Events

ok = macula:monitor_nodes().

receive
    {nodeup, Node} -> io:format("Node joined: ~p~n", [Node]);
    {nodedown, Node} -> io:format("Node left: ~p~n", [Node])
end.

Unsubscribe

ok = macula:unmonitor_nodes().

Gossip-Specific Callbacks

%% Using a PID
{ok, _} = macula_cluster_gossip:start_link(#{callback => self(), secret => Secret}).

receive
    {macula_cluster, nodeup, Node} -> handle_join(Node);
    {macula_cluster, nodedown, Node} -> handle_leave(Node)
end.

%% Using module/function callback
{ok, _} = macula_cluster_gossip:start_link(#{
    callback => {my_module, handle_cluster_event},
    secret => Secret
}).

Docker Compose Example

services:
  node1:
    image: my-app:latest
    network_mode: host  # Required for UDP multicast
    environment:
      - RELEASE_NODE=node1@localhost
      - CLUSTER_STRATEGY=gossip

      - CLUSTER_SECRET=demo_secret
    volumes:
      # The shared cookie file, owned by the container's user, mode 0400.
      - ./erlang.cookie:/home/app/.erlang.cookie:ro
      - MACULA_GOSSIP_SECRET=${MACULA_GOSSIP_SECRET}  # at least 32 bytes, the same on every node

  node2:
    image: my-app:latest
    network_mode: host
    environment:
      - RELEASE_NODE=node2@localhost
      - CLUSTER_STRATEGY=gossip

      - CLUSTER_SECRET=demo_secret
    volumes:
      - ./erlang.cookie:/home/app/.erlang.cookie:ro
      - MACULA_GOSSIP_SECRET=${MACULA_GOSSIP_SECRET}  # at least 32 bytes, the same on every node

Network Requirements

Firewall Rules

# Allow UDP multicast traffic
iptables -A INPUT -p udp --dport 45892 -j ACCEPT
iptables -A OUTPUT -p udp --dport 45892 -j ACCEPT

# For Erlang distribution (EPMD and distribution ports)
iptables -A INPUT -p tcp --dport 4369 -j ACCEPT
iptables -A INPUT -p tcp --dport 9100:9200 -j ACCEPT

Docker Networking

Host Networking (Recommended for development):

network_mode: host

Macvlan (Production):

networks:
  macvlan_net:
    driver: macvlan
    driver_opts:
      parent: eth0
    ipam:
      config:
        - subnet: 192.168.1.0/24

Troubleshooting

Nodes Not Discovering Each Other

1. Check multicast support: ping -c 3 230.1.1.251

2. Verify UDP port is open: ss -ulnp | grep 45892

3. Check Docker networking: Must use host networking for multicast
4. Verify cookie matches: erlang:get_cookie().

Authentication Failures

Nodes with different secrets drop each other's packets and never discover each other; the dropped packets are logged at debug level. Ensure all nodes use the same MACULA_GOSSIP_SECRET, or the same secret option.

Gossip Does Not Start

{error, secret_required} or {error, {secret_too_short, ...}} means the node has no shared secret of at least 32 bytes. Set MACULA_GOSSIP_SECRET, or pass secret.

High CPU Usage

If broadcast_interval is too low, increase it:

{ok, _} = macula_cluster_gossip:start_link(#{broadcast_interval => 5000, secret => Secret}).

Security Best Practices

1. Use a random shared secret - Gossip requires one of at least 32 bytes; generate it with, for example, openssl rand -hex 32
2. Rotate secrets periodically - Coordinate rotation across all nodes
3. Use network segmentation - Limit multicast scope with VLANs
4. Set TTL appropriately - multicast_ttl => 1 limits to same subnet

Testing

# Run gossip clustering tests
rebar3 eunit --module=macula_cluster_gossip_tests

# Run cluster API tests
rebar3 eunit --module=macula_cluster_tests

# Run full cluster test suite
rebar3 eunit --dir=test/macula_cluster_system

Related Documentation

• Dist Over Mesh Guide - WAN connectivity via relay mesh
• Erlang Distribution Protocol