This blog describes the concept of cgroups and static service pools in Cloudera Manager. It assumes that you already have a cloudera cluster running. If not, then you can download a Cloudera Quick-Start VM from Cloudera.

Defining cgroups

Linux Control Groups (cgroups – abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
https://en.wikipedia.org/wiki/Cgroups

Cloudera Manager and cgroups

  • Cloudera Manager supports the Linux control groups (cgroups) kernel feature.
  • Administrators can impose per-resource restrictions and limits on services and roles.
  • This provides the ability to allocate resources using cgroups to enable isolation of compute frameworks from one another.
  • Configurable Resource Parameters
  1. CPU Shares – Processes with higher CPU shares will be given more CPU time
  2. I/O Weight – Greater the I/O weight, the higher priority will be given to I/O requests made by the role
  3. Memory Limit – Sets Hard / Soft limits to the memory for the roles. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process.

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_mc_cgroups.html

Static Service Pools

  • Static service pools isolate the services in your cluster from one another, so that load on one service has a bounded impact on other services.
  • Services are allocated a static percentage of total resources—CPU, memory, and I/O weight—which are not shared with other services.
  • Cloudera Manager computes recommended memory, CPU, and I/O configurations for the worker roles of the services that correspond to the percentage assigned to each service.
  • Implemented per role group within a cluster, using Linux control groups (cgroups) and cooperative memory limits
  • Can be used to control access to resources by HBase, HDFS, Impala, MapReduce, Solr, Spark, YARN, and add-on services.

http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_mc_service_pools.html

Managing Static Service Pools through Cloudera Manager

  • Login to the Cloudera Manager application
  • Browse to Clusters > Cluster_name > Resource Management > Static Service Pools

Cloudera Manager - static service pools

Viewing Static Service Pool Status

The dashboard for Static Service Pools contains the summary of cluster’s overall and per-service CPU & Memory usage.

Cloudera Manager - Static Service Pool dashboardTo read about configuring Static service pools, please refer to the blog Configuring Static Service Pools.

Share this:

Leave a Reply

Your email address will not be published. Required fields are marked *