Kubernetes Node Configuration
This section describes the Linux kernel configuration requirements for each Kubernetes node that is used in a Greenplum for Kubernetes cluster.
Memory Configuration
Pivotal Greenplum requires that you set the vm.overcommit_memory
setting to 2
(no overcommit), which returns an out-of-memory error
if an application fails to allocate memory. This is the only safe setting for the database, and is required for any production deployment. See
Virtual memory settings in Linux - The Problem with Overcommit for additional details.
On Ubuntu systems, you can configure this setting by adding the following line to the /etc/sysctl.conf
file:
vm.overcommit_memory=2
Note: By default Kubernetes uses vm.overcommit_memory=1
with the assumption that applications always overcommit memory. This is frequently cited as a best practice in Kubernetes distributions to avoid memory allocation failures. However, it conflicts with the requirements for a database server environment. For Greenplum deployments, always set vm.overcommit_memory=2
and ensure that Greenplum is the only application deployed in a given Kubernetes environment.
Network Configuration
If your Kubernetes deployment uses large nodes (for example, nodes with 40+ cores), and you want to create many segments on each node (more than 16 segments), then you must configure the ARP cache setting on each node to allow the numerous connections that are created by the interconnect process between Greenplum segments.
On Ubuntu systems, update the /etc/sysctl.conf
to add the settings:
net.ipv4.neigh.default.gc_thresh1=30000
net.ipv4.neigh.default.gc_thresh2=32000
net.ipv4.neigh.default.gc_thresh3=32768
net.ipv6.neigh.default.gc_thresh1=30000
net.ipv6.neigh.default.gc_thresh2=32000
net.ipv6.neigh.default.gc_thresh3=32768
If you fail to increase these thresholds but deploy a large number of segments on a node, queries that access numerous segments will generate interconnect errors such as:
ERROR: Interconnect error writing an outgoing packet: Invalid argument ...
File Configuration
Greenplum parallel query processing against multiple segments requires a high open file limit for each Kubernetes node.
On Ubuntu systes, update the /etc/systemd/systemd.conf
file to include the setting:
DefaultLimitNOFILE=65536