Failing Over to a Standby Master

Follow these steps to failover to a standby master in a Greenplum for Kubernetes cluster, and then initialize the previous master instance to operate as the standby.

  1. Use the Greenplum gpstate utility to determine which of the master hosts is acting as the passive standby master. For example:

    $ kubectl exec -it master-0 bash -- -c "source /opt/gpdb/; gpstate"
    20181017:20:51:06:002457 gpstate:master-0:gpadmin-[INFO]:-Starting gpstate with args:
    20181017:20:51:07:002457 gpstate:master-0:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.11.3 build dev'
    20181017:20:51:08:002457 gpstate:master-0:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.11.3 build dev) on x86_64-pc-linux-gnu, compiled by    GCC gcc (Ubuntu 6.4.0-17ubuntu1~16.04) 6.4.0 20180424, 64-bit compiled on Oct 10 2018 22:25:23'
    20181017:20:51:09:002457 gpstate:master-0:gpadmin-[INFO]:-Obtaining Segment details from master...
    20181017:20:51:09:002457 gpstate:master-0:gpadmin-[INFO]:-Gathering data from segments...
    20181017:20:51:21:002457 gpstate:master-0:gpadmin-[INFO]:-Greenplum instance status summary
    20181017:20:51:21:002457 gpstate:master-0:gpadmin-[INFO]:-----------------------------------------------------
    20181017:20:51:21:002457 gpstate:master-0:gpadmin-[INFO]:-   Master instance                                           = Active
    20181017:20:51:21:002457 gpstate:master-0:gpadmin-[INFO]:-   Master standby                                            = master-1.agent.default.svc.cluster.local
    20181017:20:51:21:002457 gpstate:master-0:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive

    This output confirms that the master-1 host is currently serving as the standby master. The remaining steps will use master-1 to indicate the instance that is being promoted to operate as the active master instance.

  2. First, use gpstop -y to stop the cluster, leaving the current standby master instance running:

    $ kubectl exec -it master-0 bash -- -c "source /opt/gpdb/; gpstop -y"

    Enter Y when prompted to shut down the Greenplum instance.

  3. Login to the standby master host and execute gpactivatestandby to activate the host as the standby master:

    $ kubectl exec -it master-1 bash -- -c "source /opt/gpdb/; bagpactivatestandby -d /greenplum/data-1  -f"
    20181017:21:39:02:000721 gpactivatestandby:master-1:gpadmin-[INFO]:------------------------------------------------------
    20181017:21:39:02:000721 gpactivatestandby:master-1:gpadmin-[INFO]:-Standby data directory    = /greenplum/data-1
    20181017:21:39:02:000721 gpactivatestandby:master-1:gpadmin-[INFO]:-Standby port              = 5432
    20181017:21:39:02:000721 gpactivatestandby:master-1:gpadmin-[INFO]:-Standby running           = yes
    20181017:21:39:02:000721 gpactivatestandby:master-1:gpadmin-[INFO]:-Force standby activation  = no
    20181017:21:39:02:000721 gpactivatestandby:master-1:gpadmin-[INFO]:------------------------------------------------------
    Do you want to continue with standby master activation? Yy|Nn (default=N):

    Enter Y when prompted to activate the standby master.

  4. After the container named “master-1” becomes master, you will need to use a new external IP address to access the Greenplum service. Execute the command:

    $ kubectl patch service greenplum -p '{"spec":{"selector":{"": "master-1"}}}'
    service "greenplum" patched
  5. With the new master instance operational, you can continue starting the cluster using the gpstart utility:

    $ kubectl exec -it master-1 bash -- -c "source /opt/gpdb/; gpstart"

    Enter Y when prompted to activate the start the cluster.

  6. At this point, executing gpstate shows that no standby master instance is currently configured:

    $ kubectl exec -it master-1 bash -- -c "source /opt/gpdb/; gpstate"
    20181017:21:51:31:001142 gpstate:master-1:gpadmin-[INFO]:-Starting gpstate with args:
    20181017:21:51:32:001142 gpstate:master-1:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.11.3 build dev'
    20181017:21:51:33:001142 gpstate:master-1:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.11.3 build dev) on x86_64-pc-linux-gnu, compiled by    GCC gcc (Ubuntu 6.4.0-17ubuntu1~16.04) 6.4.0 20180424, 64-bit compiled on Oct 10 2018 22:25:23'
    20181017:21:51:33:001142 gpstate:master-1:gpadmin-[INFO]:-Obtaining Segment details from master...
    20181017:21:51:33:001142 gpstate:master-1:gpadmin-[INFO]:-Gathering data from segments...
    20181017:21:51:42:001142 gpstate:master-1:gpadmin-[INFO]:-Greenplum instance status summary
    20181017:21:51:43:001142 gpstate:master-1:gpadmin-[INFO]:-----------------------------------------------------
    20181017:21:51:43:001142 gpstate:master-1:gpadmin-[INFO]:-   Master instance                                           = Active
    20181017:21:51:43:001142 gpstate:master-1:gpadmin-[INFO]:-   Master standby                                            = No master standby configured

    Continue following the remaining steps to initialize the previous master instance as the standby master.

  7. Remove the master data directory from the inactive master instance:

    $ kubectl exec master-0  -- /bin/bash -c 'source /opt/gpdb/; rm -rf ${MASTER_DATA_DIRECTORY}'
  8. On the active master host, prepare and initialize the new standby master host:

    $ kubectl exec master-1  -- /bin/bash -c "source /opt/gpdb/; /home/gpadmin/tools/sshKeyScan"
    $ kubectl exec master-1  -- /bin/bash -c "source /opt/gpdb/; gpinitstandby -a -s master-0"
  9. At this point the active master runs on the pod named “master-1” and the standby master runs on the pod named “master-0.”