.. index::  Networking setup


.. _setup-networking:

Networking
----------

-  All access is defined via standard :term:`Ingress` specification files.
   They're tested to work with the :term:`NGINX ingress controller`, which is pretty standard.
   To make them work, one way is to setup this controller as shown in the ``/ingress-nginx`` directory.
-  The bulk of all traffic is via https, hence you need a certificate as well.
   Included are examples for :term:`Let's Encrypt` in the ``/letsencrypt`` directory,
   but you can also use your own certificate.
   However you set this up, this is a general Kubernetes question and goes beyond the scope of this guide.
   (see :term:`Ingress TLS`)
-  As a minimum, you need a domain/subdomain and
   a :term:`LoadBalancer` for your cluster, which routes the traffic to the NGINX controller.
   It's beyond the scope of this guide, but such a LoadBalancer is either provided
   by your public cloud provider, or you can use a :term:`MetalLB` setup.

Beyond the HTTPS traffic, there is also (optionally) an :ref:`SSH Gateway <ssh-gateway>`.
It makes it possible to access projects via SSH.
That traffic is pure TCP on port 22,
which needs a special configuration for the NGINX controller.

If you're **not** using the :term:`NGINX Ingress Controller`, you might have to adjust some details.
Please check both ``ingress.yaml`` :term:`HELM Chart <HELM Charts>` templates in ``hub`` and ``static`` for up-to-date details.
The relevant settings are in the annotations, prefixed with ``nginx.ingress.kubernetes.io`` – see `NGINX Annotations <https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/>`_ for more details. Basically:

- **Session Affinity**: reconnecting the websockets is more stable and faster, if they're sticky with specific hubs.
- **Body Size**: this is relevant for `uploading files <https://doc.cocalc.com/howto/upload.html>`_. The uploader uses chunking, so, it's just important to allow more than the size of a chunk.
- **/metrics endpoint**: you don't want to expose that endpoint to the public. That's why this snippet is added:

  .. code:: yaml

      nginx.ingress.kubernetes.io/server-snippet: |
        location = "/metrics" {
          deny all;
          return 404;
        }

  .. warning::

    Newer NGINX controllers block adding custom ``server-snippet`` entries,
    because of `this security issue <https://github.com/kubernetes/ingress-nginx/issues/7837>`_.

    If this happens, your HELM deployment will fail with that error::

        admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
        nginx.ingress.kubernetes.io/server-snippet annotation cannot be used.
        Snippet directives are disabled by the Ingress administrator

    Ref.: `Ingress Nginx → User Guide → ConfigMap: allow-snippet-annotations <https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#allow-snippet-annotations>`_

  Inclusion of this snippet is controlled by the ``global.networkingConfiguration.hideMetrics`` setting in your :ref:`my-values.yaml <my-values.yaml>`. By default it is true.
  Set ``hideMetrics: false`` in your configuration to not include that snippet.
  This will unblock this roadblock immediately and you can deploy CoCalc OnPrem.

  However, if you trust what's being deployed in your cluster,
  you can tell the NGINX controller to accept such annotations.

  * Via it's HELM chart: it's the ``allowSnippetAnnotations: true`` setting.
  * Modify the controller's :term:`ConfigMap` directly: i.e. use ``kubectl get configmap`` to find it and then edit it, it has a name like ``nginx-ingress-ingress-nginx-controller``

    The effective content should end up like::

        data:
          allow-snippet-annotations: "true"

    ... and after saving the edited ConfigMap, the controller should notice that and restart automatically.