Hub deployment

Security is a critical part of our deployment. This is a guide to the security practices Hazy applies to it's development and release process and best practices for deployment of the Hub within the enterprise.

Development and Release Security Procedures

The Hub application is continuously analysed during the development process and as part of the release procedure in order to detect and prevent security vulnerabilities.

  • Static Analysis. Code is continuously scanned by a static analysis tool to detect potential vulnerabilities before release.

  • Dependencies. Hazy leverages the automatic vulnerability scanning of dendendencies provided by Github. Any dependencies with security issues are fixed.

  • Minimal Images. Hub container images are based off a distroless image. These images contain only the application and the minimal dependencies required to run that application.

Restricting what's in your runtime container to precisely what's necessary for your app is a best practice employed by Google and other tech giants that have used containers in production for many years. It improves the signal to noise of scanners (e.g. CVE) and reduces the burden of establishing provenance to just what you need.

  • Vulnerability Scans. As an automatic part of the container build and release process, all container images are analysed using the Trivy vulnerability scanner.

    Any issues are checked for applicability and severity before the image is released to customers.

Authentication Token Storage

Hazy follows best practices for authentication tokens and token storage.

  • All user passwords are hashed using the Argon2id algorithm before being written to the database.

  • API keys, used for API access, are secure random strings that are easily revoked from the admin dashboard.

Audit Logging

The Hub can be configured to send all events to an external log -- either an external file or via UDP.

This includes notification of:

  • All HTTP requests, including HTTP method, URL, response and response duration
  • All modifications to the Hub state (when set to trace level)
  • Significant modifications to the user database, such as user creation, password changes, role assignments etc
  • Notification of successful logins
  • Notification of unsuccessful logins
  • Notification of authorization failures

All events include the originating user (if one exists) and originating IP address.

See the audit log documentation for more information.

Container Security

The Hub, as a networked application which runs multiple internal databases, requires more capabilities than the typical synthesiser installation.

Despite this, it has been designed to operate with as many security mitigations as possible:

  • Requires no external network access

    Hazy applications will never attempt to connect to an external service (unless required by the Customer's installation topology)

  • Read only root

    To limit the risk of security breaches, the Hub container can be run with a read-only root filesystem

  • Runs services as an un-privileged user

    Because of the need to run multiple services, the Hub container needs to be run as the root user (although the meaning of "root" is namespace dependent) but the services themselves, both internal and externally facing, run as unprivileged users, specifically the nobody user.

    This limits the damage that can be done were the publicly visible services be compromised.

Docker

Throughout this document it is assumed that you need to be root to run root-full docker containers (that is when the Docker daemon is configured to run as root and the default owner of any container process is therefore also root - this is the default for new Docker installations). It is possible (and often preferable) to configure Docker to enable non-root access to the daemon, in which case you can remove any use of sudo from the various docker commands below.

To run the Hub container with all applicable security options:

sudo docker run \
  --name hub \
  -p 4000:4000 \
  -v hub_data:/mnt/data \
  -v hub_db:/var/db \
  --read-only \
  --tmpfs /tmp:rw,noexec \
  --tmpfs /var/run:rw,exec \
  --security-opt=no-new-privileges \
  --cap-drop ALL \
  --cap-add CHOWN \
  --cap-add DAC_OVERRIDE \
  --cap-add FOWNER \
  --cap-add FSETID \
  --cap-add KILL \
  --cap-add SETGID \
  --cap-add SETUID \
  hazy-hub:release

Rootless Docker

To further enhance the security of your installation, we recommend configuring Docker to run container without root privileges.

There are two options available to do this:

Isolate containers with a user namespace

By configuring the Docker daemon with to remap containers to a non-privileged user, we allow the Hub container to run as root within a user namespace.

The best way to prevent privilege-escalation attacks from within a container is to configure your container’s applications to run as unprivileged users. For containers whose processes must run as the root user within the container, you can re-map this user to a less-privileged user on the Docker host. The mapped user is assigned a range of UIDs which function within the namespace as normal UIDs from 0 to 65536, but have no privileges on the host machine itself.

https://docs.docker.com/engine/security/userns-remap/

This gives all the advantages of running the container as a non-root user (e.g. by using the --user flag to docker run) whilst keeping the application's ability to run as root within the container.

Run the Docker daemon as a non-root user

Rootless mode allows running the Docker daemon and containers as a non-root user to mitigate potential vulnerabilities in the daemon and the container runtime.

https://docs.docker.com/engine/security/rootless/

In this configuration, no Docker process, not even the Docker daemon, runs as root.

Due to the power of Linux namespaces the above docker run command works identically with either of these more secure modes enabled.

Due to file permission issues, switching between the three runtime configurations (rootfull, rootless and user namespaced) will cause problems unless the files within the two hub volumes have their permissions manually migrated after re-configuring the Docker daemon.

Also note that when using Docker volumes rather than bind mounts, switching between these modes will also require either manually moving the data (as well as altering ownership) or re-starting the Hub configuration from scratch.

For these reasons we recommend experimenting with the Docker runtime modes before committing to a final Hub configuration as the likelihood is that your data will be lost between transitions.

Kubernetes

The example Kubernetes configuration for the Hub includes a securityContext entry that limits the Hub container's capabilities.

Podman

Podman runs in either root-full or root-less mode depending on the user that runs the podman command.

The Hub supports both root-full and root-less modes in Podman (although see above regarding the dangers of data loss when switching runtime modes).

Because podman run aims to be compatible with docker run the Docker command above works equally well for Podman:

podman run \
  --name hub \
  -p 4000:4000 \
  -v hub_data:/mnt/data \
  -v hub_db:/var/db \
  --health-cmd "/usr/local/bin/hub-healthcheck" \
  --health-interval "30s" \
  --health-start-period "60s" \
  --read-only \
  --tmpfs /tmp:rw,noexec \
  --tmpfs /var/run:rw,exec \
  --security-opt=no-new-privileges \
  --cap-drop ALL \
  --cap-add CHOWN \
  --cap-add DAC_OVERRIDE \
  --cap-add FOWNER \
  --cap-add FSETID \
  --cap-add KILL \
  --cap-add SETGID \
  --cap-add SETUID \
  hazy-hub:release

The only difference being that Podman does not support the HEALTHCHECK configuration defined within the container itself, so it must be passed explicitly to podman run.

TLS

The Hazy Hub only provides unsecured HTTP access. Integration into the organisation's public key infrastructure should be handled by a proxy service, as detailed in the installation guide.