Who will allocate GPU resources among users of the ML platform if there are 4 servers with video cards and 5 users request resources?

The distribution of GPU resources among users of an ML platform is usually implemented through orchestration and resource management systems, such as Kubernetes with GPU support or specialized resource managers (e.g., NVIDIA GPU Cloud, Slurm, or custom solutions).

If there are 4 servers with graphics cards and 5 users, the distribution occurs based on requests and prioritization policies. For example:

• The orchestrator monitors available GPUs on all servers.
• Users submit requests for a certain number of GPUs.
• The system allocates GPUs, aiming to distribute the load optimally, considering occupancy and priorities.
• If GPUs are insufficient, requests are queued or users are returned an error with a suggestion to try again later.

It is also important to have monitoring and usage restrictions for GPUs to avoid conflicts and overloads.

Example: In Kubernetes, using the Device Plugin for NVIDIA GPUs, resource limits and requests can be set in pod manifests, and kube-scheduler will distribute pods across nodes with available GPUs.