Why I built my own MongoDB image to run a replica set in production

There comes a moment in an application's life when a single database is no longer enough. Not because it's slow. Because it's alone.

A single MongoDB instance is a single point of failure. The day that container goes down — full disk, kernel panic, a VM reboot gone wrong — your app doesn't slow down: it stops. No failover, no replica ready to take over. Just an error screen and a ringing phone.

That's the starting problem. MongoDB's classic answer is the replica set: several nodes holding the same data, electing a primary, and failing over automatically when that primary disappears. On paper, it's simple. In a reproducible, multi-machine, multi-tenant Docker environment, it's a lot less so. This article tells where it actually breaks, and how I ended up solving the problem once and for all.

The full lab (Dockerfile, init.sh, init.js, mongod.conf) is available on github.com/denisakp/dev-life-labs, in the mongo-replica folder.

The context: a multi-tenant platform, one database per client

Before getting into it: every client on the platform runs on its own isolated VM, with its own Docker Compose stack. That means the database isn't deployed once, but as many times as there are clients. Anything fragile or manual in deploying MongoDB gets multiplied by the number of VMs.

That constraint changes everything. A procedure that "works if you're careful" is not acceptable when you have to replay it on ten different machines, on macOS in dev and AlmaLinux in prod. I needed something that works every time, with no intervention.

Why a replica set, concretely

A MongoDB replica set is three nodes in my case — mongo1, mongo2, mongo3. Only one is primary at any moment: it receives all writes. The other two are secondaries that replicate the data continuously.

The point isn't volume or performance. It's availability. If the primary goes down, the surviving nodes hold an election and pick a new one within seconds. The application reconnects automatically, provided it uses the right connection string (more on that). Three nodes is the minimum for an election to have a clear majority — two survivors out of three is enough to decide; a single survivor out of two is not.

In my config, mongo1 gets priority: 2, the others stay at 1:

members: [
  { _id: 0, host: 'mongo1:27017', priority: 2 },
  { _id: 1, host: 'mongo2:27017' },
  { _id: 2, host: 'mongo3:27017' }
]

That higher priority simply tells MongoDB: "if mongo1 is healthy, it's the primary." It makes behavior predictable, which simplifies debugging and monitoring.

The real trap: node-to-node authentication

Here's where the "3 containers and you're done" tutorial falls apart.

As soon as a replica set is protected by authentication — and in production it must be — nodes don't just authenticate clients. They must also authenticate each other. MongoDB uses a keyfile for this: a shared secret every node holds.

You generate it once:

openssl rand -base64 756 > rs_keyfile

And MongoDB is uncompromising about this file. It requires:

• permissions exactly 400 (read-only for the owner, nothing for others),
• ownership by the mongodb user.

One bit too many, and the node refuses to start. Not a warning: a flat refusal.

The natural reflex is to mount this keyfile as a volume from the host:

volumes:
  - ./mongodb/rs_keyfile:/etc/mongodb/keyfile:ro   # ❌ the trap

Except the permissions of a host-mounted file depend on the host. On my macOS dev machine they don't look like those on an AlmaLinux VM. The file that starts perfectly locally crashes the container in production. You patch it with a chmod, it works today, it breaks on the next git clone on another machine. The behavior is never reproducible.

Now multiply the problem: three nodes, four config files each (keyfile, mongod.conf, init scripts) — that's twelve volume mounts per deployment. Twelve chances to get a path or a permission wrong. On every VM. Exactly the kind of friction that turns a deployment into a lottery.

The solution: bake everything into the image

The breakthrough was to stop mounting configuration at runtime and freeze it at build time. Instead of asking the host to provide the right files with the right permissions, I build an image that already contains them, correctly.

FROM mongo:8.2-noble

# The keyfile gets its 400 permissions here, once and for all
COPY --chown=mongodb:mongodb --chmod=400 rs_keyfile /etc/mongodb/keyfile

COPY --chown=mongodb:mongodb --chmod=644 mongod.conf /etc/mongod.conf
COPY --chown=mongodb:mongodb --chmod=755 init.sh /init.sh
COPY --chown=mongodb:mongodb --chmod=644 init.js /init.js

# The directory holding the keyfile is locked down too
RUN chown -R mongodb:mongodb /etc/mongodb && \
    chmod 700 /etc/mongodb

CMD ["--config", "/etc/mongod.conf"]

The --chmod=400 on COPY isn't a suggestion: it's etched into the image layer. Whatever the host, whatever the OS, the keyfile always has the right permissions and owner. The reproducibility problem disappears, because the host no longer has a say.

The embedded mongod.conf is minimal — it points at the now-guaranteed keyfile:

replication:
  replSetName: okydookReplSet
security:
  keyFile: /etc/mongodb/keyfile
net:
  bindIp: 0.0.0.0
  port: 27017

This shift in perspective has four direct consequences:

• Consistency — same behavior on macOS in dev and on the prod VMs. The host no longer touches the config.
• Simplicity — no more twelve mounts. Compose only mounts the data volumes (/data/db), which is their real job.
• Versioning — a config change becomes a new image tag. You know exactly which configuration runs on which node, and you can roll back.
• Security — the configuration is immutable. It can't be accidentally modified at runtime, and you shrink the attack surface by removing host↔container bindings.

Build the image

To run the lab on a single machine, you don't need a registry at all — build the image locally from the Dockerfile:

docker build -t mongo-replica:latest .

That's exactly what the lab's docker-compose.yml does on its own: each node declares build: . with image: mongo-replica:latest, so a plain docker compose up builds the image once and reuses it for the three nodes. No registry, no docker login.

Going multi-VM: push to a registry

In production I don't run the three nodes on one machine — they live on separate VMs (linux/amd64, AlmaLinux on Proxmox), while my dev machine is arm64. Building locally on each VM no longer cuts it: I build once for the target platform and push to a registry I control, then every VM pulls the same image:

docker buildx build \
  --platform linux/amd64 \
  -t <your-registry>/mongo-replica:latest \
  --push \
  .

Then, in the compose file, swap build: . for image: <your-registry>/mongo-replica:latest. Replace <your-registry> with the registry you push to. Since this image bakes in your keyfile and config, it has no business on a public registry: push it to a private one — GitHub Container Registry (ghcr.io/<your-user>), a private Docker Hub repo, GitLab, or a self-hosted registry. A public registry is only fine if the image holds no secrets (e.g. you generate the keyfile at runtime instead of baking it in).

Each VM then just pulls the image (after a docker login to the private registry). Deploying a database becomes: pull, run. No more files to copy by hand.

The compose file, before and after

The benefit shows up best in docker-compose.yml. Before, each node drowned in mounts:

mongo1:
  image: mongo:8.2-noble
  volumes:
    - mongo1_data:/data/db
    - mongo1_config:/data/configdb
    - $PWD/scripts/mongo/rs_keyfile:/etc/mongodb/keyfile:ro   # ❌ host permissions
    - $PWD/scripts/mongo/mongod.conf:/etc/mongod.conf:ro      # ❌ × 3 nodes = 12 mounts

After, the custom image carries the config, compose keeps only the data:

services:
  mongo1:
    container_name: mongo1
    build: .                       # local build → in prod: image: <your-registry>/mongo-replica:latest
    image: mongo-replica:latest
    hostname: mongo1
    networks: [ mongo-cluster ]
    volumes:
      - mongo1_data:/data/db
      - mongo1_config:/data/configdb
    environment:
      MONGO_INITDB_ROOT_USERNAME: '${MONGO_ADMIN_USER:-admin}'
      MONGO_INITDB_ROOT_PASSWORD: '${MONGO_ADMIN_PASSWD:-veryStringPassword}'

  # mongo2 and mongo3: strictly identical, only the suffix changes
  mongo-init:
    container_name: mongo-init
    image: mongo-replica:latest
    restart: "no"
    entrypoint: ["/bin/bash", "/init.sh"]
    networks: [ mongo-cluster ]
    depends_on: [ mongo1, mongo2, mongo3 ]
    environment:
      MONGO_ROOT_USERNAME: '${MONGO_ADMIN_USER:-admin}'
      MONGO_ROOT_PASSWORD: '${MONGO_ADMIN_PASSWD:-veryStringPassword}'
      MONGO_DB_USER: '${DB_USERNAME:-okydook}'
      MONGO_DB_PASSWORD: '${DB_PASSWORD:-mypassword}'

No keyfile, no mongod.conf, no mounted scripts: they live in the image. That's the whole point. The full docker-compose.yml — all three nodes, networks, volumes — is in the lab repo.

Orchestrating the bootstrap: the real hidden headache

Building the image isn't enough. A replica set doesn't initialize itself: you have to run, once, the rs.initiate() command that tells the three nodes they form a cluster. And you have to do it at the right moment — neither before the nodes are ready, nor twice.

I handed that job to a dedicated ephemeral container, mongo-init, which does only that then shuts down. Its script patiently waits for MongoDB to respond before acting:

until mongosh --host mongo1:27017 -u "$MONGO_ROOT_USERNAME" -p "$MONGO_ROOT_PASSWORD" \
  --authenticationDatabase admin --eval "db.adminCommand('ping')" > /dev/null 2>&1; do
  echo "Waiting for MongoDB connection..."
  sleep 5
done

Two details matter enormously here, because they make the difference between an automated deployment and one you have to babysit:

Initialization is idempotent. The script first checks whether the replica set already exists before trying to create it:

try {
  var status = rs.status();
  if (status.ok === 1) {
    isInitialized = true;
    print("ReplicaSet already initiated");
  }
} catch(e) {
  print("ReplicaSet not yet initiated, proceeding...");
}

Without that guard, restarting the stack would try to re-initialize a live cluster — and crash it. With it, you can run docker compose up as many times as you want: if the cluster exists, the script notices and moves on. That's what makes the deployment safely replayable.

The application user is created right after, with the strict rights it needs and nothing more — readWrite on the single okydook database, not admin access:

db.createUser({
  user: process.env.MONGO_DB_USER,
  pwd: process.env.MONGO_DB_PASSWORD,
  roles: [{ role: 'readWrite', db: 'okydook' }]
});

Finally, Compose chains dependencies cleanly: the admin UI doesn't just wait for mongo-init to start, but to finish successfully:

depends_on:
  mongo-init:
    condition: service_completed_successfully

That condition is subtle but essential: it guarantees you never expose an interface on a half-initialized cluster.

The final piece: connect to the cluster, not a node

All this work would be wasted if the application pointed at a single node. Because if it only knows mongo1 and mongo1 goes down, it's blind to the freshly elected new primary — high availability becomes useless.

So the connection string lists all three nodes and explicitly names the replica set:

mongodb://USER:PASS@mongo1:27017,mongo2:27017,mongo3:27017/okydook?replicaSet=okydookReplSet&authSource=okydook

The three nodes answer on port 27017 inside the Docker network — these are the mongo-cluster network hostnames, not host-mapped ports. Listing mongo2:27018 or mongo3:27019 here is a common mistake: those ports only exist outside the network.

The replicaSet=okydookReplSet parameter is what changes everything. With it, the MongoDB driver doesn't treat the three addresses as three independent databases: it understands it's a cluster, discovers its topology, tracks who's primary at any instant, and automatically routes writes to the right node — including after a failover. The application never even knows an election happened. That's exactly the intended effect.

Verifying it all runs

Once the stack is up (docker compose up -d), two checks are enough. First the keyfile — the thing that crashed the volume-mounted setup — which must be 400 inside the container:

docker exec -it mongo1 ls -la /etc/mongodb/keyfile
# -r-------- 1 mongodb mongodb ...

Then cluster state: rs.status() should show mongo1 as PRIMARY, mongo2/mongo3 as SECONDARY, all with health: 1.

docker exec -it mongo1 mongosh -u admin -p PASSWORD \
  --authenticationDatabase admin --eval "rs.status()"

The other diagnostic commands (app-user auth, ping) are listed in the README of the lab repo.

What this journey taught me

The starting problem looked like "MongoDB must be highly available." The real problem, the one that cost me time, was elsewhere: making a distributed setup reproducible across environments that don't look alike.

The lesson I keep: when a configuration is fragile because of the environment — permissions, paths, host state — the right answer isn't to document it better or add defensive chmods. It's to take the environment out of the equation. Bake the config into the image, make init idempotent, declare dependencies explicitly: each of those decisions turns a "be careful" step into an "it just works" step.

And maybe that's the real meaning of infrastructure as code: not just writing infra into files, but methodically eliminating every place where a human has to be careful.

In a next article, I'll take on the logical sequel: how to back up and restore this MongoDB cluster in a containerized environment. Because a highly available cluster with no backup strategy is just a fancier way to lose your data.

References

• Full lab: dev-life-labs/mongo-replica
• MongoDB replica set keyfile authentication
• MongoDB Replica Set documentation
• Docker multi-platform builds
• GitHub Container Registry
• Docker Compose documentation