Upgrading Greenplum for Kubernetes

This topic describes how to upgrade VMware Tanzu Greenplum for Kubernetes from version 2.x to a subsequent version 2.y release. The upgrade process involves first deleting any existing Greenplum cluster deployments, and then upgrading the Greenplum Operator to the latest version. You then use the new Greenplum Operator to re-create earlier cluster deployments, using the same manifest files. During this process, you re-use any existing persistent volumes so that Greenplum cluster data is preserved.

Prerequisites

  • This procedure assumes that you have installed a previous minor version Greenplum for Kubernetes (version 2.x). Greenplum for Kubernetes supports upgrades only from the prior minor version (for example, from version 2.0 to version 2.1). If your installed version of the product is more than one version prior, you will need to upgrade incrementally to reach your target version, using the upgrade instructions associated with each new version.

  • Before you can install the newer Greenplum for Kubernetes version, ensure that you have installed all required software and prepared your Kubernetes environment as described in Prerequisites.

Procedure

Follow these steps to upgrade the Greenplum Operator resource:

  1. Navigate to the workspace directory of the Greenplum for Kubernetes installation (or to the location of the Kubernetes manifest that you used to deploy the cluster). For example:

    $ cd ./greenplum-for-kubernetes-*/workspace
    
  2. Execute the kubectl delete command, specifying the manifest that you used to deploy the cluster. For example:

    $ kubectl delete -f ./my-gp-instance.yaml
    

    kubectl stops the Greenplum for Kubernetes instance and deletes the kubernetes resources for the Greenplum deployment.

  3. Use kubectl to monitor the progress of terminating Greenplum resources in your cluster. For example, if your cluster deployment was named my-greenplum:

    $ kubectl get all -l greenplum-cluster=my-greenplum
    
    NAME                                     READY     STATUS        RESTARTS   AGE
    pod/greenplum-operator-7b5ddcb79-vnwvc   1/1       Running       0          9m
    pod/master-0                             0/1       Terminating   0          5m
    pod/segment-a-0                          0/1       Terminating   0          5m
    pod/segment-a-1                          0/1       Terminating   0          5m
    pod/segment-b-0                          0/1       Terminating   0          5m
    
    NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   26m
    
    NAME                                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/greenplum-operator   1         1         1            1           9m
    
    NAME                                           DESIRED   CURRENT   READY     AGE
    replicaset.apps/greenplum-operator-7b5ddcb79   1         1         1         9m
    
  4. The deletion process is complete when the segment pods are no longer available (and no resources remain):

    $ kubectl get all -l greenplum-cluster=my-greenplum
    
    No resources found in default namespace.
    

    The Greenplum Operator should remain:

    $ kubectl get all
    
    NAME                                     READY     STATUS    RESTARTS   AGE
    pod/greenplum-operator-7b5ddcb79-vnwvc   1/1       Running   0          34m
    
    NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   50m
    
    NAME                                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/greenplum-operator   1         1         1            1           34m
    
    NAME                                           DESIRED   CURRENT   READY     AGE
    replicaset.apps/greenplum-operator-7b5ddcb79   1         1         1         34m
    
  5. Repeat the previous steps for each version cluster that you deployed.

  6. Download the new Greenplum for Kubernetes software from VMware Tanzu Network. The download file has the name: greenplum-for-kubernetes-<version>.tar.gz.

  7. Go to the directory where you downloaded the new version of Greenplum for Kubernetes, and unpack the downloaded software. For example:

    $ cd ~/Downloads
    $ tar xzf greenplum-for-kubernetes-*.tar.gz
    

    The above command unpacks the distribution into a new directory named greenplum-for-kubernetes-<version>.

  8. Go into the new greenplum-for-kubernetes-<version> directory:

    $ cd ./greenplum-for-kubernetes-*
    
  9. For Minikube deployments only, ensure that the local docker daemon interacts with the Minikube docker container registry:

    $ eval $(minikube docker-env)
    

    Note: To undo this docker setting in the current shell, run eval "$(docker-machine env -u)".

  10. Load the new Greenplum Operator Docker image to the Docker registry. For example:

    $ docker load -i ./images/greenplum-operator
    
    96eda0f553ba: Loading layer [==================================================>]  127.6MB/127.6MB
    bd95983a8d99: Loading layer [==================================================>]  11.78kB/11.78kB
    293b479c17a5: Loading layer [==================================================>]  15.87kB/15.87kB
    fa1693d66d0b: Loading layer [==================================================>]  3.072kB/3.072kB
    5d5f706821ef: Loading layer [==================================================>]  92.66MB/92.66MB
    3c59653fa97d: Loading layer [==================================================>]   40.5MB/40.5MB
    4457bbd6a302: Loading layer [==================================================>]  40.85MB/40.85MB
    Loaded image: greenplum-operator:v1.12.0
    
  11. Load the Greenplum for Kubernetes Docker image to the local Docker registry as well. For example:

    $ docker load -i ./images/greenplum-for-kubernetes
    
    b04caf3e714d: Loading layer [==================================================>]  114.7MB/114.7MB
    5d990f175f75: Loading layer [==================================================>]  12.31MB/12.31MB
    029e8dc80732: Loading layer [==================================================>]  7.226MB/7.226MB
    e7a72a352b50: Loading layer [==================================================>]  744.3MB/744.3MB
    ...
    Loaded image: greenplum-for-kubernetes:v1.12.0
    
  12. Verify that the new Docker images are now available:

    $ docker images "greenplum-*"
    
    REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
    greenplum-operator         v1.12.0             e82305e017a4        45 hours ago        295MB
    greenplum-for-kubernetes   v1.12.0             647cef39bbd8        45 hours ago        3.15GB
    greenplum-operator         v1.11.0             852cafa7ac90        7 weeks ago         269MB
    greenplum-for-kubernetes   v1.11.0             3819d17a577a        7 weeks ago         3.05GB
    
  13. (Skip this step if you are using local docker images such as on Minikube.) If you push the Greenplum for Kubernetes docker images to a different container registry:

    1. If you want to push the Greenplum for Kubernetes docker images to a different container registry, set the project name and image repo name and then use Docker to push the images. For example, to push the images to Google Cloud Registry using the current Google Cloud project name:

      $ gcloud auth configure-docker
      
      $ PROJECT=$(gcloud config list core/project --format='value(core.project)')
      $ IMAGE_REPO="gcr.io/${PROJECT}"
      
      $ GREENPLUM_IMAGE_NAME="${IMAGE_REPO}/greenplum-for-kubernetes:$(cat ./images/greenplum-for-kubernetes-tag)"
      $ docker tag $(cat ./images/greenplum-for-kubernetes-id) ${GREENPLUM_IMAGE_NAME}
      $ docker push ${GREENPLUM_IMAGE_NAME}
      
      $ OPERATOR_IMAGE_NAME="${IMAGE_REPO}/greenplum-operator:$(cat ./images/greenplum-operator-tag)"
      $ docker tag $(cat ./images/greenplum-operator-id) ${OPERATOR_IMAGE_NAME}
      $ docker push ${OPERATOR_IMAGE_NAME}
      
    2. Copy the values yaml file from your existing deployment, or create a new YAML file in the workspace subdirectory as shown below:

      $ touch workspace/operator-values-overrides.yaml
      
    3. If you pushed the Greenplum Operator and Greenplum Database Docker images to a container registry, add two additional lines to the configuration file to indicate the registry where you pushed the images. For example, if you are using Google Cloud Registry with a project named “gp-kubernetes”, you would add the properties:

      operatorImageRepository: gcr.io/gp-kubernetes/greenplum-operator
      greenplumImageRepository: gcr.io/gp-kubernetes/greenplum-for-kubernetes
      

      Note: If you did not tag the images with a container registry prefix or project name (for example, if you are using your own local Minikube deployment), then you can skip this step.

  14. Use helm to upgrade the Greenplum Operator to the latest release, specifying a customized YAML configuration file if you created one. For example:

    $ helm upgrade greenplum-operator -f workspace/operator-values-overrides.yaml operator/
    

    If you did not create a YAML configuration file (as in the case with Minikube) omit the -f option:

    $ helm upgrade greenplum-operator operator/
    

    Helm upgrades to the new release in the Kubernetes namespace specified in the current Kubernetes context. If you want to upgrade a Greenplum Operator from a different namespace, include the --namespace option in the helm command.


    The command displays the following message and concludes with a link to this documentation:

    Release "greenplum-operator" has been upgraded. Happy Helming!
    NAME: greenplum-operator
    LAST DEPLOYED: Wed Feb 12 11:41:47 2020
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None
    NOTES:
    greenplum-operator has been installed.
    
    Please see documentation at:
    http://greenplum-kubernetes.docs.pivotal.io/
    
  15. Use watch kubectl get all -l app=greenplum-operator to ensure that the Greenplum Operator pod is in the Running state. For example:

    $ watch kubectl get all -l app=greenplum-operator
    
    NAME                                    READY   STATUS    RESTARTS   AGE
    pod/greenplum-operator-ddbc89dc-v67ng   1/1     Running   0          116s
    
    NAME                                                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    service/greenplum-validating-webhook-service-ddbc89dc-v67ng   ClusterIP   10.100.181.61   <none>        443/TCP   115s
    
    NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/greenplum-operator   1/1     1            1           64m
    
    NAME                                            DESIRED   CURRENT   READY   AGE
    replicaset.apps/greenplum-operator-667ccc59fd   0         0         0       64m
    replicaset.apps/greenplum-operator-ddbc89dc     1         1         1       116s
    
  16. Return to the workspace directory that contains the manifest files you used to deploy your clusters (or copy those manifest files to the new Greenplum for Kubernetes version workspace directory.

  17. Use kubectl apply and specify your manifest file to send the deployment request to the Greenplum Operator. For example, to use the sample my-gp-instance.yaml file:

    $ kubectl apply -f ./my-gp-instance.yaml 
    
    greenplumcluster.greenplum.pivotal.io/my-greenplum created
    

    The Greenplum Operator deploys the necessary Greenplum resources according to your specification, using the existing Persistent Volume Claims as-is with their available data.

  18. Use kubectl get all -l greenplum-cluster=<your Greenplum cluster instance name> and wait until all Greenplum cluster pods have the status Running:

    $ watch kubectl get all -l greenplum-cluster=my-greenplum
    
    NAME              READY   STATUS    RESTARTS   AGE
    pod/master-0      1/1     Running   0          34s
    pod/master-1      1/1     Running   0          34s
    pod/segment-a-0   1/1     Running   0          34s
    pod/segment-b-0   1/1     Running   0          34s
    
    NAME                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
    service/agent       ClusterIP      None            <none>        22/TCP           34s
    service/greenplum   LoadBalancer   10.107.25.227   <pending>     5432:32149/TCP   34s
    
    NAME                         READY   AGE
    statefulset.apps/master      2/2     34s
    statefulset.apps/segment-a   1/1     34s
    statefulset.apps/segment-b   1/1     34s
    

    At this point, the upgraded cluster is available. If you are using a persistent storageClass, the updated cluster is created with the same Persistent Volume Claims (PVCs) and data.

  19. If your cluster is configured to use a standby master, connect to the master-0 pod and execute the gpstart command manually. For example:

    kubectl exec -it master-0 -- bash -c "source /usr/local/greenplum-db/greenplum_path.sh; gpstart"
    
    20200212:19:45:55:000517 gpstart:master-0:gpadmin-[INFO]:-Starting gpstart with args: 
    20200212:19:45:55:000517 gpstart:master-0:gpadmin-[INFO]:-Gathering information and validating the environment...
    20200212:19:45:55:000517 gpstart:master-0:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.24.2 build dev'
    20200212:19:45:55:000517 gpstart:master-0:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
    20200212:19:45:55:000517 gpstart:master-0:gpadmin-[INFO]:-Starting Master instance in admin mode
    20200212:19:45:56:000517 gpstart:master-0:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
    20200212:19:45:56:000517 gpstart:master-0:gpadmin-[INFO]:-Obtaining Segment details from master...
    20200212:19:45:56:000517 gpstart:master-0:gpadmin-[INFO]:-Setting new master era
    20200212:19:45:56:000517 gpstart:master-0:gpadmin-[INFO]:-Master Started...
    20200212:19:45:56:000517 gpstart:master-0:gpadmin-[INFO]:-Shutting down master
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:---------------------------
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Master instance parameters
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:---------------------------
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Database                 = template1
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Master Port              = 5432
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Master directory         = /greenplum/data-1
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Timeout                  = 600 seconds
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Master standby start     = On
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:---------------------------------------
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-Segment instances that will be started
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:---------------------------------------
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-   Host          Datadir                  Port    Role
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-   segment-a-0   /greenplum/data          40000   Primary
    20200212:19:45:58:000517 gpstart:master-0:gpadmin-[INFO]:-   segment-b-0   /greenplum/mirror/data   50000   Mirror
    
    Continue with Greenplum instance startup Yy|Nn (default=N):
    

    Press Y to continue the startup.

  20. Describe your Greenplum cluster to observe that the update succeeded.

    $ kubectl describe greenplumClusters/my-greenplum
    
    Name:         my-greenplum
    Namespace:    default
    Labels:       <none>
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"greenplum.pivotal.io/v1","kind":"GreenplumCluster","metadata":{"annotations":{},"name":"my-greenplum",   "namespace":"default"...
    API Version:  greenplum.pivotal.io/v1
    Kind:         GreenplumCluster
    Metadata:
      Creation Timestamp:  2020-02-12T19:44:19Z
      Finalizers:
        stopcluster.greenplumcluster.pivotal.io
      Generation:        3
      Resource Version:  6167
      Self Link:         /apis/greenplum.pivotal.io/v1/namespaces/default/greenplumclusters/my-greenplum
      UID:               204c1dc7-f7b8-4fae-a784-3cc06579211a
    Spec:
      Master And Standby:
        Anti Affinity:              no
        Cpu:                        0.5
        Host Based Authentication:  # host   all   gpadmin   1.2.3.4/32   trust
    # host   all   gpuser    0.0.0.0/0   md5
    
        Memory:              800Mi
        Storage:             1G
        Storage Class Name:  standard
        Worker Selector:
      Segments:
        Anti Affinity:          no
        Cpu:                    0.5
        Memory:                 800Mi
        Primary Segment Count:  1
        Storage:                2G
        Storage Class Name:     standard
        Worker Selector:
    Status:
      Instance Image:    greenplum-for-kubernetes:v1.12.0
      Operator Version:  greenplum-operator:v1.12.0
      Phase:             Running
    Events:              <none>
    

    The Phase should be Running and the images should show the latest version.