Distribute and Conquer

Summary:

Borg is the cluster management system that assign well-defined jobs and tasks (latency-sensitive or batch) to different clusters with sophisticated assigning and scheduling algorithms. It offers container to each task and a nice debug environment as well. The scalability of Borg is achieved with the Borgmaster and scheduler .

Strong points:

1. Jobs with constraints and preference (dependency, environment, etc.) and other task properties are good for general-purpose task assignment. Each submitted task can be configured and handed to suitable clusters for better locale or performance.
2. Built-in HTTP server for almost every Borg tasks to publish information about the health of the task and performance metrics. A serrvice called Sigma provides web-based UI for processing logs and debug information. Also the task/job control is easy and straightforward, which is quite similar to linux processes or SLURM resource management. To be even more detailed on the issue of debugging, jobs and tasks submission and usages are recorded via Dremel.
3. All the scheduling optimizations seem amazing. Score caching is used to store the evaluation of machine performance. The performance will only be updated upon any new tasks/jobs execution. Equivalence classes become useful when assigning jobs with identical constraints. Relaxed randomization is a time saver. The scheduler will not try to find the best machine/cluster to assign the task, it will find a “good enough” machine randomly for the current task.

Weak points:

1. Borg chooses containers over VMs for the sake of performance. While containers can pack more jobs and lighter comparing to full virtual environment, this option might suffer from security and container management issues. However, it’s just a trade-off and I think Borg is in the right direction. Actually, Google continues to use Docker in Kubernetes, which is another cluster management system.
2. I agree with the restriction of jobs in the lesson learned chapter. Borg focus too much on individual tasks and jobs are the only way to group them together. This is inflexible when it comes to any group operations to the tasks since there’s no other ways to describe the relation between two tasks. A label or attribute-grouping would be better.
3. This paper is loaded with performance analysis and it has probably the most graphs and charts among all the Google papers. However, there’s no performance comparison between similar product like Mesos or even their own Kubernetes. Aside from the performance, they didn’t talk much about the structural difference between those similar resource management systems either.