Kubernetes Node Configuration

This section describes the Linux kernel configuration requirements for each Kubernetes node that is used in a Greenplum for Kubernetes cluster.

Memory Configuration

Pivotal Greenplum requires that you set the vm.overcommit_memory setting to 2 (no overcommit), which returns an out-of-memory error if an application fails to allocate memory. This is the only safe setting for the database, and is required for any production deployment. See Virtual memory settings in Linux - The Problem with Overcommit for additional details.

On Ubuntu systems, you can configure this setting by adding the following line to the /etc/sysctl.conf file:

vm.overcommit_memory=2

Note: By default Kubernetes uses vm.overcommit_memory=1 with the assumption that applications always overcommit memory. This is frequently cited as a best practice in Kubernetes distributions to avoid memory allocation failures. However, it conflicts with the requirements for a database server environment. For Greenplum deployments, always set vm.overcommit_memory=2 and ensure that Greenplum is the only application deployed in a given Kubernetes environment.

Network Configuration

If your Kubernetes deployment uses large nodes (for example, nodes with 40+ cores), and you want to create many segments on each node (more than 16 segments), then you must configure the ARP cache setting on each node to allow the numerous connections that are created by the interconnect process between Greenplum segments.

On Ubuntu systems, update the /etc/sysctl.conf to add the settings:

net.ipv4.neigh.default.gc_thresh1=30000
net.ipv4.neigh.default.gc_thresh2=32000
net.ipv4.neigh.default.gc_thresh3=32768
net.ipv6.neigh.default.gc_thresh1=30000
net.ipv6.neigh.default.gc_thresh2=32000
net.ipv6.neigh.default.gc_thresh3=32768

If you fail to increase these thresholds but deploy a large number of segments on a node, queries that access numerous segments will generate interconnect errors such as:

ERROR:  Interconnect error writing an outgoing packet: Invalid argument ...

File Configuration

Greenplum parallel query processing against multiple segments requires a high open file limit for each Kubernetes node.

On Ubuntu systes, update the /etc/systemd/systemd.conf file to include the setting:

DefaultLimitNOFILE=65536