Networking

• All access is defined via standard Ingress specification files. They’re tested to work with the NGINX ingress controller, which is pretty standard. To make them work, one way is to setup this controller as shown in the /ingress-nginx directory.

• The bulk of all traffic is via https, hence you need a certificate as well. Included are examples for Let’s Encrypt in the /letsencrypt directory, but you can also use your own certificate. However you set this up, this is a general Kubernetes question and goes beyond the scope of this guide. (see Ingress TLS)

• As a minimum, you need a domain/subdomain and a LoadBalancer for your cluster, which routes the traffic to the NGINX controller. It’s beyond the scope of this guide, but such a LoadBalancer is either provided by your public cloud provider, or you can use a MetalLB setup.

Beyond the HTTPS traffic, there is also (optionally) an SSH Gateway. It makes it possible to access projects via SSH. That traffic is pure TCP on port 22, which needs a special configuration for the NGINX controller.

If you’re not using the NGINX Ingress Controller, you might have to adjust some details. Please check both ingress.yaml HELM Chart templates in hub and static for up-to-date details. The relevant settings are in the annotations, prefixed with nginx.ingress.kubernetes.io – see NGINX Annotations for more details. Basically:

• Session Affinity: reconnecting the websockets is more stable and faster, if they’re sticky with specific hubs.

• Body Size: this is relevant for uploading files. The uploader uses chunking, so, it’s just important to allow more than the size of a chunk.

• /metrics endpoint: you don’t want to expose that endpoint to the public. That’s why this snippet is added:

  nginx.ingress.kubernetes.io/server-snippet: |
    location = "/metrics" {
      deny all;
      return 404;
    }
  

  Warning

  Newer NGINX controllers block adding custom server-snippet entries, because of this security issue.

  If this happens, your HELM deployment will fail with that error:

  admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
  nginx.ingress.kubernetes.io/server-snippet annotation cannot be used.
  Snippet directives are disabled by the Ingress administrator
  

  Ref.: Ingress Nginx → User Guide → ConfigMap: allow-snippet-annotations

  Inclusion of this snippet is controlled by the global.networkingConfiguration.hideMetrics setting in your my-values.yaml. By default it is true. Set hideMetrics: false in your configuration to not include that snippet. This will unblock this roadblock immediately and you can deploy CoCalc OnPrem.

  However, if you trust what’s being deployed in your cluster, you can tell the NGINX controller to accept such annotations.

  • Via it’s HELM chart: it’s the allowSnippetAnnotations: true setting.

  • Modify the controller’s ConfigMap directly: i.e. use kubectl get configmap to find it and then edit it, it has a name like nginx-ingress-ingress-nginx-controller

    The effective content should end up like:

    data:
      allow-snippet-annotations: "true"
    

    … and after saving the edited ConfigMap, the controller should notice that and restart automatically.