Kubernetes Node Configuration
This section describes the Linux kernel configuration requirements for each Kubernetes node that is used in a Greenplum for Kubernetes cluster.
Pivotal Greenplum requires that you set the
vm.overcommit_memory setting to
2 (no overcommit), which returns an out-of-memory error
if an application fails to allocate memory. This is the only safe setting for the database, and is required for any production deployment. See
Virtual memory settings in Linux - The Problem with Overcommit for additional details.
On Ubuntu systems, you can configure this setting by adding the following line to the
Note: By default Kubernetes uses
vm.overcommit_memory=1 with the assumption that applications always overcommit memory. This is frequently cited as a best practice in Kubernetes distributions to avoid memory allocation failures. However, it conflicts with the requirements for a database server environment. For Greenplum deployments, always set
vm.overcommit_memory=2 and ensure that Greenplum is the only application deployed in a given Kubernetes environment.
If your Kubernetes deployment uses large nodes (for example, nodes with 40+ cores), and you want to create many segments on each node (more than 16 segments), then you must configure the ARP cache setting on each node to allow the numerous connections that are created by the interconnect process between Greenplum segments.
On Ubuntu systems, update the
/etc/sysctl.conf to add the settings:
net.ipv4.neigh.default.gc_thresh1=30000 net.ipv4.neigh.default.gc_thresh2=32000 net.ipv4.neigh.default.gc_thresh3=32768 net.ipv6.neigh.default.gc_thresh1=30000 net.ipv6.neigh.default.gc_thresh2=32000 net.ipv6.neigh.default.gc_thresh3=32768
If you fail to increase these thresholds but deploy a large number of segments on a node, queries that access numerous segments will generate interconnect errors such as:
ERROR: Interconnect error writing an outgoing packet: Invalid argument ...
Greenplum parallel query processing against multiple segments requires a high open file limit for each Kubernetes node.
On Ubuntu systes, update the
/etc/systemd/systemd.conf file to include the setting: