Installing PKS for Greenplum for Kubernetes, Using Google Cloud Platform (GCP)

Understanding Communication Channel and Load Balancer Requirements

The following image illustrates 3 kinds of communication channels required for the Greenplum Service in PKS.

Each cli tool, pks, kubectl and psql needs its own channel of communication. pks has one global channel that is set up just once during the installation of the PKS system. Each kubernetes cluster requires its own communication channel. In particular, when there are multiple Kubernetes clusters, the kubectl command must be configured to target a one kubernetes cluster at a time, such as “Cluster 1” or “Cluster 2” in the diagram above, via the pks get-credentials command. Similarly, psql needs to target one Greenplum Service at a time, as controlled by its connection string.

Preparing to Install Greenplum for Kubernetes

This release of Pivotal Greenplum for Kubernetes can be deployed with Pivotal Container Services (PKS) running on Google Cloud Platform (GCP). Management scripts are provided to help you configure required GCP and Kubernetes resources, as well as to deploy a Greenplum for Kubernetes cluster.

Follow each of the steps in this section to install required software and configure resources before you attempt to deploy Greenplum for Kubernetes.

Step 1: Install PKS

Follow the instructions in Installing PKS on GCP in the Pivotal Container Service documentation to install PKS on Google Cloud Platform. Note that this step involves completing instructions in each of these sections of the PKS documentation:

• GCP Prerequisites and Resource Requirements
• Preparing to Deploy PKS on GCP
• Deploying Ops Manager to GCP
• Configuring Ops Manager on GCP
• Installing and Configuring PKS on GCP

Completing these instructions creates a new PKS installation including an Operations Manager URL, which will be referenced in later steps as .

Step 2: Install Client Tools

Pivotal Greenplum for Kubernetes provides a number of scripts to automate configuration tasks in Google Cloud Platform, PKS, and Kubernetes. These scripts require your client system to have a number of tools installed. Follow these steps:

1. On MacOS systems, use homebrew to install required client system tools:

   $ brew install jq yq kubernetes-helm docker kubectl
   

   The above command installs several tools that are required for for the Greenplum for Kubernetes scripts:

   • jq - for parsing json on the command line
   • yq - for parsing yaml on the command line
   • helm - the CLI for the Helm package manager for Kubernetes
   • docker - software container platform and related tools
   • kubectl - the CLI for deploying and managing applciations on Kubernetes

2. Download and install the Google cloud tools. Ensure that the gcloud utility is available before you continue.

3. Install the PKS CLI tool from Pivotal Network at Pivotal Container Service (PKS).

Step 3: Configure the GCP Service Account

1. Login to GCP with the gcloud tool and set a new project:

   $ gcloud auth login
   $ gcloud projects list
   $ gcloud config set project <project_name>
   

2. As a best practice, create a separate service account to use for deploying Greenplum for Kubernetes:

   Note: Update the location of the key.json file in Values-common.yaml key dockerRegistryKeyJson.

   The Greenplum for Kubernetes service account must have the following permissions, which are automatically set by the script:

   • Compute Viewer
   • Compute Network Admin
   • Storage Object Creator

   You can optionally grant these permissions to a different service account using IAM & admin in the GCP console.

3. Activate the service account using the key file key.json:

   $ gcloud auth activate-service-account --key-file=./key.json
   

   Note: The gpcloud team uses a service account key “gpcloud-pks (workstation)” stored in LastPass.

4. Set a preferred availability zone with a command similar to:

   $ gcloud config set compute/zone us-west1-a
   

   Update the zone name as necessary for your configuration.

5. Set up Google Container Registry to host Docker images

   Enable the Container Registry API

   Configure docker to use the gcloud command-line tool as a credential helper for the activated service-account:

   $ gcloud auth configure-docker
   

For additional details see instructions.

Step 4: Configure the GCP Firewall

In order to provide communication between the nodes and containers running Greenplum, you must create a firewall rule and add it to the default deployment tag. This creates routes between nodes and their containers. Follow these steps:

1. Access the Operations Manager configuration page at https://<OPSMAN_IP_OR_FQDN>/infrastructure/iaas_configuration/edit. Look for the field named Default Deployment Tag and record its value. If it is empty, record the value, greenplum-tag.

2. Export a variable for the Default Deployment Tag value, as in:

   $ export DEFAULT_DEPLOYMENT_TAG="greenplum-tag"
   

   The Default Deployment Tag controls the default firewall rules for every VM that is created.

3. Determine the Google Parent Network name. To do so, access the Operations Manager page at https://<OPSMAN_IP_OR_FQDN>/infrastructure/networks/edit and disclose one of the networks that you created for PKS. The Google Network Name value uses the format <parent-network-name>/<subnet-name>/<region-name>. Because all of the networks are based on the underlying Google Network, the first part of the full network name, <parent-network-name> corresponds to the Google Parent Network.

   For example, if the value of the Google Network Name reads gpcloud-pks-net/foo/us-west1 then gpcloud-pks-net corresponds to the Google Parent network.

4. Export an environment for the Google parent network:

   $ export GOOGLE_PARENT_NETWORK="gpcloud-pks-net"
   

5. Finally, create a firewall rule to allow inter-container traffic using DEFAULT_DEPLOYMENT_TAG and GOOGLE_PARENT_NETWORK variables that you exported:

   $ gcloud compute firewall-rules create greenplum-rule \
   --network=${GOOGLE_PARENT_NETWORK} \
   --action=ALLOW \
   --rules=tcp:1024-65535,tcp:22,icmp,udp \
   --source-ranges=0.0.0.0/0 \
   --target-tags ${DEFAULT_DEPLOYMENT_TAG}
   

Step 5: Configure the Kubernetes Load Balancer

Note: This step should not be necessary after the release of PKS 1.1.

Create or re-use a Load Balancer for a new Kubernetes cluster. If a TCP load balancer from a previous kubernetes cluster exists, reuse it by specifying the front-end IP address below in the Step 6: Deploy a Kubernetes Cluster.

If no load balancer exists, use these commands to create one:

$ export LB_NAME=greenplum-cluster1-lb
$ gcloud compute target-pools create ${LB_NAME}
$ REGION=$(gcloud config get-value compute/region)
$ gcloud compute forwarding-rules create ${LB_NAME}-forwarding-rule \
    --target-pool ${LB_NAME} --ports 8443 --ip-protocol TCP --region ${REGION}
$ echo " Front-end IP Address of load balancer for ${LB_NAME}:"
$ gcloud compute forwarding-rules describe ${LB_NAME}-forwarding-rule --region ${REGION} --format="value(IP_ADDRESS)"

You will reset the backend instances to attach to the new Kubernetes cluster in the next section.

Step 6: Deploy a Kubernetes Cluster

Follow these steps to deploy a new PKS cluster using the firewall rules and load balancer you configured for Kubernetes:

1. Follow the instructions to Log in to PKS CLI.

2. Execute the create_pks_cluster_on_gcp.bash script, specifying the IP address of the Load Balancer’s front end. The command uses the syntax create_cluster.bash <my_cluster-name> <IP_address_of_Load_Balancer>. For example:

   $ cd workspace
   $ samples/scripts/create_pks_cluster_on_gcp.bash gpdb 1.1.1.1
   

   This creates a Kubernetes cluster and assigns it to be accessed via the specified load balancer. You can use either an existing Load Balancer or a newly-created one, as described in Step 5: Set Up the Kubernetes Load Balancer.

3. Use the kubectl command to show the system containers and newly-created nodes available for deploying Greenplum for Kubernetes:

   $ kubectl get pods --namespace kube-system
   $ kubectl get nodes 
   

Deploying Greenplum for Kubernetes on the PKS Cluster

After you complete the above steps, follow the instructions in Deploying Pivotal Greenplum for Kubernetes to deploy the Greenplum service on the new PKS cluster.

Troubleshooting Your PKS Installation

A Greenplum query fails to write an outgoing packet

The Greenplum cluster is initialized and running, but a query returns an error similar to:

ERROR: Interconnect error writing an outgoing packet: Invalid argument (seg0 slice1 <ip>:<port> pid=1193)

This error occurs when ports are not garbage collected quickly enough. The problem is commonplace in systems that have many containers on a single kubernetes node, and the containers heavily use different ports to communicate with one another (as is the case with Greenplum segments).

To work around this problem, set the following sysctl attribute on the worker nodes:

net.ipv4.neigh.default.gc_thresh1 = 30000

A container fails to pull its image with “ErrImagePull”

The container image is pulled from the private Google Container Registry (GCR), which requires a secret to be pushed to the cluster.

The deploy.sh script should do this, and it relies on the key file provided to it on the command line.

That key file can be created with a new service account if does not exist with the correct permissions.

Get UAA login after certificate change (after any URL change for UAA domain name)

If you see an error about 401 from BOSH like

bosh -e pks vms
Using environment '192.168.101.10' as anonymous user

Finding deployments:
  Director responded with non-successful status code '401' response 'Not authorized: '/deployments'
'

Exit code 1

Go to the credentials web page like https://<ops manager>/infrastructure/director/credentials (with different IP address)

and look for Bosh command line credentials, which looks something like

{"credential":"BOSH_CLIENT=<Some User> BOSH_CLIENT_SECRET=<Some Secret> BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=192.168.101.10 bosh "}

use: * BOSH_CLIENT as the “Client ID” * BOSH_CLIENT_SECRET as “Client secret” * “Admin” for User name * associated password from Uaa Admin User Credentials

in the command for uaac token owner get, for example:

uaac token owner get
Client ID:  ops_manager
Client secret:  ********************************
User name:  Admin
Password:  ********************************

Can access Ops Manager, but cannot Access UAA

If you have a problem like

pks login -a https://pks-0.gpcloud.gpdb.pivotal.io:9021 -u dummy -p <some password> -k

Error: Post https://pks-0.gpcloud.gpdb.pivotal.io:8443/oauth/token: dial tcp 35.197.67.138:8443: getsockopt: connection refused

That can be a symptom of having recycled the VM running PKS API, such that the external IP address defined in the domain name is old. So use gcloud or Google Cloud Console to determine the current VM for PKS API. You can distinguish it because it has two labels “job” and “instance-group” which both have values “pivotal container service”. Get the external IP address for this and change the DNS definition to that external IP address. Use a command like

watch dig '<my domain name>'

to see when that DNS entry is updated for your local workstation. When it updates, try the pks login command again.

FAQ

What permissions are required for the Google Service Account?

The service account used by PKS must have the compute.admin role to create a new service. As part of creating a service, it must create firewall rules and create an external ip for the load balancer.

How can I use bosh to obtain a shell?

In order to get a Bash shell on a particular node, go through Operations Manager to use it as a jump box:

1. In the Google Cloud web interface, look for “compute instances” and identify Ops Manager.

2. Click the “ssh” button to launch an ssh shell on that instance.

3. After launching the ssh shell, use the BOSH CLI command from the Ops Manager Interface Credentials Tab. Look for a Bosh Commandline Credentials line that looks like:

   BOSH_CLIENT=<SOME_CLIENT> BOSH_CLIENT_SECRET=<SOME_SECRET> BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<SOME_IP> bosh
   

4. Create an alias named pks and use it in commands below:

   BOSH_CLIENT=<SOME_CLIENT> BOSH_CLIENT_SECRET=<SOME_SECRET> BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate bosh alias-env pks -e <SOME_IP>
   

5. For convenience, export the above two variables in the current shell:

   export BOSH_CLIENT=<SOME_CLIENT>
   export BOSH_CLIENT_SECRET=<SOME_SECRET>
   export BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
   

   After exporting those variables, the following command should work:

   bosh -e pks deployments
   

6. Determine the IP address of the node you want by listing the nodes for each pod:

   kubectl get pods -o wide
   

   bosh -e pks vms
   bosh -e pks ssh <VM_NAME_FROM_COMMAND_ABOVE>
   

How do I set permissions for MacOS HomeBrew?

If necessary, execute the command:

sudo chown -R $USER:admin /usr/local