Project Pods

These settings deal with the projects running as Pods in your cluster. All default values should be fine, though.

• manage.timeout_pending_projects_min: All Manage services are responsible for starting and stopping project pods. If for some reason such a project pod is in the state Pending for too long, it will be killed. This is the timeout in minutes.

• manage.project.tag is the default image tag, on which a project will run. It’s possible to customize the software environment in various ways, including changing the default project image. This is where the upstream project image tag will be set.

• The Prepull service can be enabled or disabled via manage.prepull.enabled. Associated with it are the taints and labels for project nodes, which are defined in manage.project.dedicatedProjectNodesTaint and *.dedicatedProjectNodesLabel.

• manage.mem_request_adjustment: basically a safeguard to avoid too large memory requests by project pods. Applied on the computed memory request of a project’s pod quotas – see Scaling/Projects for more context. The lim is a hard upper bound in MiB, pct is dynamic in percent of memory limit.

• manage.timeout_pending_projects_min: projects stuck in “Pending” are deleted after this timeout.

• manage.watchdog_stream_s: “manage” listens to a stream of changes coming from your Kubernetes API. If it is behind a LoadBalancer, this stream might be interrupted, without even knowing that it is. So, if there is no new data coming in for that time, the stream is restarted.

• manage.project.serviceAccountName customize the ServiceAccount used by the project (if this isn’t set, it will not set it in the template and hence use the default).

• Settings related to hub’s resources and multiple_replicas are explained in scaling frontend service.

• Projects are prohibited from accessing internal services. That’s accomplished using a set of NetworkPolicy rules. If you want to allow projects to access internal services, you can set networkPolicy.allowProjectEgress to a list of rules. Check out the template file cocalc/templates/network-policies.yaml for more details about this.

• manage.project.init: this is a setup script, which is sourced right before the local server of a project is started. It allows you to tune parameters, change environment variables, or call scripts from your compute environment. Use your powers wisely! The /cocalc/values.yaml file shows two configurable aspects. Tune them to your needs based on your experience:

  • Blobstore: this is a local store of data, which has been generated in Jupyter Notebooks – in particular, images. They are served from a small web-server for efficiency and e.g. TimeTravel uses them to show old plots, no longer part of the notebook. There are two implementations: an Sqlite3 based one (old, has been the only implementation) and a file-based store (new, now the default, especially made for NFS backed file-systems, which are known to cause problems with sqlite databases). You can selected the implementation by setting COCALC_JUPYTER_BLOBSTORE_IMPL=sqlite or disk. Disk is recommended and the default. The other tuning parameters for the disk based store are a size limit and number of files:

    • JUPYTER_BLOBSTORE_DISK_PRUNE_SIZE_MB=100: prune disk usage of jupyter blobstore

    • JUPYTER_BLOBSTORE_DISK_PRUNE_ENTRIES=1000: max number of files in that blobstore cache

  • Jupyter Kernel Pool: This is a mechanism to spin up one or more kernels without a notebook, to make them ready for use. E.g. when you restart a kernel, it will pick one from the pool and you see the notebook running without a delay. The tradeoff is memory usage. By default, one kernel will be in the pool. You can tune this as well:

    • COCALC_JUPYTER_POOL_SIZE=1: size of pool, set to 0 to disable it

    • COCALC_JUPYTER_POOL_TIMEOUT_S=900: after that time, clean up old kernels in the pool

    • COCALC_JUPYTER_POOL_LAUNCH_DELAY_MS=5000: delay before spawning an additional kernel