Replication Coordinator

Many other IDM systems configure replication on each node of the topology. This means that the administrator is responsible for ensuring all nodes are connected properly, and that agreements are bidirectional. As well this requires manual work for administrators to configure each node individually, as well as monitoring individually. This adds a significant barrier to "stateless" configurations.

In Kanidm we want to avoid this - we want replication to be coordinated to make deployment of replicas as easy as possible for new sites.

Kanidm Replication Coordinator

The intent of the replication coordinator (KRC) is to allow nodes to subscribe to the KRC which configures the state of replication across the topology.

1. Out of band -                ┌────────────────┐
 issue KRC ca + ────────────────┤                │
   Client JWT.                  │                │
        │       ┌──────────────▶│                │──────────────────────┐
        │       │2. HTTPS       │     Kanidm     │                      │
        │     JWT in Bearer     │  Replication   │            5. Issue repl config
        │  Request repl config  │  Coordinator   │             with partner public
        │  Send self signed ID  │                │                     key
        │       │  cert         │                │                      │
        │       │     ┌─────────│                │◀────────┐            │
        │       │     │         │                │       4. HTTPS       │
        │       │     │         └────────────────┘    JWT in Bearer     │
        │       │   3. Issue                       Request repl config  │
        │       │  repl config                     Send self signed ID  │
        │       │     │                                    cert         │
        │       │     │                                    │            │
        │       │     │                                    │            │
        │       │     │                                    │            │
        │       │     │                                    │            │
        ▼       │     ▼                                    │            ▼
       ┌────────────────┐                                ┌─┴──────────────┐
       │                │                                │                │
       │                │                                │                │
       │                │       5. mTLS with self        │                │
       │                │──────────signed cert──────────▶│                │
       │ Kanidm Server  │      Perform replication       │ Kanidm Server  │
       │     (node)     │                                │     (node)     │
       │                │                                │                │
       │                │                                │                │
       │                │                                │                │
       │                │                                │                │
       └────────────────┘                                └────────────────┘

Kanidm Node Configuration

There are some limited cases where an administrator may wish to manually define replication configuration for their deployments. In these cases the admin can manually configure replication parameters in the Kanidm configuration.

A kanidm node for replication requires either:

  • The URL to the KRC
  • the KRC CA cert
  • KRC issued configuration JWT

OR

  • A replication configuration map

A replication configuration map contains a set of agreements and their direction of operation.

All replicas require:

  • The direct url that other nodes can reach them on (this is NOT the origin of the server!)

Pull mode

This is the standard mode. The map contains for each node to pull replication data from. This logically maps to the implementation of the underlying replication mechanism.

  • the url of the node's replication endpoint.
  • The self-signed node certificate to be pinned for the connection.
  • If a refresh required message is received, if automatic refresh should be carried out.

Push mode

This mode is unlikely to be developed as it does not match the way that replication works.

Worked examples

Manual configuration

There are two nodes, A and B.

The administrator configures both kanidm servers with replication urls.

# Server A
[replication]
origin = "repl://kanidmd_a:8444"
bindaddress = "[::]:8444"
# Server B
[replication]
origin = "repl://kanidmd_b:8444"
bindaddress = "[::]:8444"

The administrator extracts their replication certificates with the kanidmd binary admin features. This will reflect the node_url in the certificate.

kanidmd replication get-certificate

For each node, a replication configuration is created in json.

For A pulling from B.

[replication."repl://kanidmd_b:8444"]
type = "mutual-pull"
partner_cert = "M..."
automatic_refresh = false

For B pulling from A.

[replication."repl://kanidmd_a:8444"]
type = "mutual-pull"
partner_cert = "M..."
automatic_refresh = true

Notice that automatic refresh only goes from A -> B and not the other way around. This allows one server to be "authoritative".

KRC Configuration

The KRC is enabled as a replication parameter. This informs the node that it must not contact other nodes for its replication topology, and it prepares the node for serving that replication metadata. This is analgous to a single node operation configuration.

[replication]
origin = "repl://kanidmd_a:8444"
bindaddress = "[::]:8444"

krc_enable = true

# krc_url -- unset
# krc_ca_dir -- unset

All other nodes will have a configuration of:

[replication]
origin = "repl://kanidmd_b:8444"
bindaddress = "[::]:8444"

# krc_enable -- unset / false

# krc_url = https://private.name.of.krc.node
krc_url = https://kanidmd_a
# must contain ca that signs kanidmd_a's tls_chain.
krc_ca_dir = /path/to/ca_dir

The domain will automatically add a Default Site. The KRC implies its own membership to "Default Site" and it will internally add itself to the Default Site.

The KRC can then issue Tokens that define which Site a new replica should join. Initially we will only allow Default Site (and will disallow creation of other sites).

The new replica will load its KRC token from the environment variable KANIDMD_KRC_TOKEN_PATH. This value will contain a file path where the JWT is stored. This is compatible with systemd credentials and docker secrets. By default the value if unset will be defined by a profile default (/etc/kanidm/krc.token or /data/krc.token).

A new replica can then contact the krc_url validating the presented TLS chain with the roots from krc_ca_dir to assert the legitimacy of the KRC. Only once these are asserted, then the KRC token can be sent to the instance as a Bearer token. The new replica will also provide its mTLS certificate and its server UUID.

Once validated, the KRC will create or update the server's replica entry. The replica entry in the database will contain the active mTLS cert of the replica and a reference to the replication site that the token referenced.

This will additionally add the "time first seen" to the server entry.

From this, for each server in the replication site associated to the token, the KRC will provide a replication config map to the new replica providing all URL's and mTLS certs.

Anytime the replica checks in, if the KRC replication map has changed a new one will be provided, or the response will be None for no changes.

To determine no changes we use a "generation". This is where any change to a replication site or server entries will increment the generation counter. This allows us to detect when a client requires a new configuration or not.

If a server's entry in the database is marked to be Revoked then it will remain in the database, but be inelligible for replication participation. This is to allow for forced removal of a potentially compromised node.

The KRC will periodically examine its RUV. For any server entry whose UUID is not contained in the RUV, and whose "time first seen + trime window" is less than now, then the server entry will be REMOVED for inactivity since it has now been trimmed from the RUV.

Moving the Replication Coordinator Role

Since the coordinator is part of a kanidmd server, there must be a process to move the KRC to another node.

Imagine the following example. Here, Node A is acting as the KRC.

┌─────────────────┐                ┌─────────────────┐
│                 │                │                 │
│                 │                │                 │
│     Node A      │◀───────────────│     Node B      │
│                 │                │                 │
│                 │                │                 │
└─────────────────┘                └─────────────────┘
         ▲     ▲
         │     │
         │     │
         │     └────────────────────────────┐
         │                                  │
         │                                  │
         │                                  │
┌─────────────────┐                ┌─────────────────┐
│                 │                │                 │
│                 │                │                 │
│     Node C      │                │     Node D      │
│                 │                │                 │
│                 │                │                 │
└─────────────────┘                └─────────────────┘

This would allow Node A to be aware of B, C, D and then create a full mesh.

We wish to decommision Node A and promote Node B to become the new KRC. Imagine at this point we cut over Node D to point its KRC at Node B.

┌─────────────────┐                ┌─────────────────┐
│                 │                │                 │
│                 │                │                 │
│     Node A      │                │     Node B      │
│                 │                │                 │
│                 │                │                 │
└─────────────────┘                └─────────────────┘
         ▲                                  ▲
         │                                  │
         │                                  │
         │                                  │
         │                                  │
         │                                  │
         │                                  │
┌─────────────────┐                ┌─────────────────┐
│                 │                │                 │
│                 │                │                 │
│     Node C      │                │     Node D      │
│                 │                │                 │
│                 │                │                 │
└─────────────────┘                └─────────────────┘

Since we still have the Server Entry records in the Default Site on both Node A and Node B, then all nodes will continue to participate in full mesh, and will update certificates as required.

Since all servers would still be updating their RUV's and by proxy, updating RUV's to their partners then no nodes will be trimmed from the topology.

This allows a time window where servers can be moved from Node A to Node B.

Gruesome Details

Server Start Up Process

Token is read from a file defined in the env.
	works with systemd + docker secrets

Token is JWT with HS256. (OR JWE + AES-GCM)

Read the token
- if token domain_uuid != our domain_uuid -> set status to "waiting"
    - empty replication config map
- if token domain_uuid == domain_uuid -> status to "ok"
    - use cached replication config map

No TOKEN -> Implies KRC role.
- Set status to "ok", we are the domain_uuid source.

Client Process

connect to KRC
- provide token for site binding
- submit my server_uuid
- submit my public cert with the request
- submit current domain_uuid + generation if possible

- reply from KRC -> repl config map.
    - config_map contains issuing KRC server uuid.
    - if config_map generation > current config_map
        - reload config.
    - if config_map == None
      - current map remains valid.

KRC Process

- Validate Token
- is server_uuid present as a server entry?
    - if no: add it with site association
    - if yes: verify site associated to token
- is server_uuid certificate the same as before?
    - if no: replace it.
- compare domain_uuid + generation
    - if different supply config
    - else None (no change)

FUTURE: Possible Read Only nodes

For R/O nodes, we need to define how R/W will pass through. I can see a possibility like

                                No direct line
       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ of sight─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐

       │                                                               ▼
┌────────────┐                 ┌─────────────┐────OOB Write────▶┌─────────────┐
│            │                 │ Remote Kani │                  │             │
│   Client   │─────Write──────▶│   Server    │                  │    Main     │
│            │                 │             │                  │             │
└────────────┘                 └─────────────┘◀───Replication───└─────────────┘

This could potentially even have some filtering rules about what's allowed to proxy writes through. Generally though I think that RO will need a lot more thought, for now I want to focus on just simple cases like a pair / group of four replicas. 😅

Requirements

  • Cryptographic (key) only authentication
  • Node to Node specific authentication
  • Scheduling of replication
  • Multiple nodes
  • Direction of traffic?
  • Use of self-signed issued certs for nodes.
  • Nodes must reject if incoming clients have the same certs.