5.13 Replication

Server replication makes it easier to configure and maintain multiple Administrative Servers, provide high availability, and have more flexibility when spanning geographically distant server installations.

Using replicated servers, you can synchronize multiple administrative servers by propagating configuration and session changes made on one server (the Master) to all of the servers in a replication group (the Slaves).

NOTE:To configure Metering for high availability, you must configure a master and at least two slaves.

Before you begin, review these topics:

5.13.1 Replication Database

Beginning in version 12.5, Management and Security Server uses a database to store information on replicated servers. This implementation introduces some terms to the Replication UI.

Database. The database contains a cluster of MSS servers.

A node represents one server in the cluster.
Database Status. The current status displays on the Configure Settings-Replication panel.

After you add or edit settings and click Apply, the Database Status changes to Down, progresses to Initializing, and then to Up.

Wait until the status is Up before proceeding with other settings.
Details. Information about each node in the cluster. Replication is functioning properly when each node reports Status: UP and Owns: 100%. (That is, the server contains 100% of the replicated data.)

Related topics

5.13.2 How to Configure Replication

Perform the configuration tasks in this order to set the trust relationships.

Identify which MSS server will be the Master, and which ones will be Slave servers.

As you decide, consider the Server role: Master and Slave servers.
Import the certificates for the Master and all of the Slave servers to the Trusted Certificates store.
Configure the Master server, and add the Slave servers, one at a time. See Master Server Role for detailed steps.
Configure each Slave server, and add the Master server. See Slave Server Role for detailed steps.

Server role: Master and Slave servers

To implement server replication, you will configure one Master server and one or more Slave server

Before you begin, consider the roles of the Master and Slave servers and how to secure the connections

Slave servers are configured according to the settings on this Master server. Any subsequent changes to the Master server will be automatically replicated to the Slave servers.
Only one Master server can be configured, but multiple Slave servers can be associated with that Master server.
You can modify the configuration on the Master server and push those updates to the Slaves.
You can also modify the configuration on a Slave server and submit the changes, but they will not be applied until the requested change is implemented on the Master server.
All configuration elements of the Master Administrative Server are replicated to a Slave server -- except:
- - log files
- - Credential Store settings
- - some Certificates settings
- - the Web Agent name, when SiteMinder is used for authentication. (The Web Agent name must be set separately for each replicated machine.)

5.13.3 Securing Replication Connections

You have the option to use HTTP or HTTPS as your server-to-server communication transport. HTTPS is recommended.

Certificate Requirements. The notes below provide an overview of certificate requirements for replication.

When using HTTPS

When configuring Replication to use HTTPS, you must establish trust between the Master and the Slaves by entering all of the servers’ certificates into the Trusted Certificate store.

The certificate of the current Administrative Server must be included in the list along with the certificates of the remote Administrative Servers. This step is required for all servers, Master or Slave, in the management cluster. In the case of self-signed or other certificates that are not well known, the certificates must be imported to all Administrative Servers in the cluster.

Perform these steps before adding the Master or Slave servers.

Open Administrative Console > Configure Settings - Trusted Certificates.
Under Certificate Store, click Management and Security Server.
View the list of certificates trusted by the Management and Security Server.

Confirm that a trusted certificate is already present, or continue to import a certificate.
Click + Import to add a certificate.

NOTE:If you installed MSS on z Linux with no JRE (using the UNIX no-jre installer), see the Caution in the Configure master server section. You may need to edit a file.

When using a Security Proxy

If you are using a Security Proxy server, you must also import the certificates for all the remote Administrative Servers to each Security Proxy server. Use the Security Proxy Wizard to import these certificates.

When you create a secure session that connects to a Security Proxy server, the session is thereafter linked to this specific Security Proxy. When this session is replicated to other servers in the cluster, the session is then initiated from a different Administrative Server, but the session itself will still connect to the original Security Proxy for which it was configured.

If client authorization is enabled on the Security Proxy Server, then the Security Proxy Server will only accept connections from sessions initiated from Administrative Servers it trusts (that is, their certificates are in the Security Proxy Trusted Certificate list).

In order for connections from replicated servers to succeed in this environment, the certificates from every Administrative Server in the cluster need to be imported to the Security Proxy server. If there are multiple Security Proxy Servers in the cluster, then this operation needs to be done on each of these Security Proxy Servers.

NOTE:The Concurrency Lock Timeout, included in the former Administrative WebStation interface, applies to managing sessions. See the Note in Edit a session.

Related topics

5.13.4 Standalone Server Role

This is the default role, and applies to Administrative Servers that are not configured for replication. The Standalone server role is used to reset Replication, such as when upgrading.

To set up Replication, configure a Master server, and add Slave servers.

5.13.5 Master Server Role

Use the Master server role to set the configuration of this Administrative Server as the one that will be replicated to the specified Slave Administrative Servers.

Configure master server

On the MASTER server that you want to replicate:

Click Master.
Check the Database Status. If Up, continue.

If Down or Initializing, wait until the changes take effect and the status is Up.
Database Network Adapter. Choose a network adapter associated with an IP address that the other MSS servers in the cluster can access.
Global protocol. Choose whether to Use HTTPS for server to server communication (between the Master and Slave servers). HTTPS is the default.
CAUTION:When HTTPS is checked:
- Verify that a certificate for the Slave server is in the Master server’s trusted store before proceeding with replication setup. See Securing Replication Connections for more information.
- You cannot change the transport and retain previously configured Slave servers. You must delete existing Slave servers before changing the protocol from HTTP to HTTPS (or vice versa).
- If you installed MSS on z Linux with no JRE (using the UNIX no-jre installer), you must edit this file in the MSS installation directory:
  
  mss/server/microservices/cassandra/conf/cassandra.yaml
  
  Change # algorithm: SunX509
  
  to algorithm: IBMX509 (uncommented)
  
  You do not need to restart the MSS server.
  
  Without this change, the cassandra database fails to restart after replication is configured, and this exception is added to the mss/server/logs/cassandra.log:
  
  "Caused by: java.security.NoSuchAlgorithmException: SunX509 TrustManagerFactory not available"
  
  To resolve the exception, set the Replication server back to Standalone. Edit cassandra.yaml, as above, and proceed with Replication.
Configure Master Server. Change the default Passphrase, if desired.

The Master server and all associated Slave servers must share the same passphrase. Use these fields to change the default passphrase.

If you make no changes to the passphrase on any of the Administrative Servers in the cluster, the passphrase is automatically the same on all servers. A changed passphrase is in effect for all replication implementations, whether or not HTTPS is used.
Next step: Add a slave server.

Slave server list

The Administrative Servers that have been replicated with this Master server’s configuration are in the list of Slave servers.

Add a slave server

Before you add a replication server, remember:

When you set up a Master server and then specify Slave servers, any existing configuration for the Slave server will be overwritten, and existing sessions on the Slave server will be deleted.
Any security certificates that are required for an HTTPS transport or for a Security Proxy must be in place before the Slave server is added here. For more information, see Securing Replication Connections.

To add a Slave Server:

Click + Add, which expands the dialog.
Enter the Slave server host name.

Add the fully qualified name of a Slave server that is to communicate with the Master server. This name is used to identify where changes are sent and also validates the requests coming in from the Slave server. Enter the name of each Slave server here.
Enter the Slave port.

The default host port is 443 for HTTPS and 80 for HTTP. Use the ports that you specified for HTTP and HTTPS during Management and Security Server setup.
Enter the Servlet context.

This entry specifies the web application context for the Slave server. The servlet context is used in the URL that accesses the Slave server, and is specified when the Administrative Server is installed.

The default, mss, is the correct value if you used the automated installer and did not change the default context as part of setup.
Click Save. The server is added to the list of Slave servers.
Click Apply to save the Master configuration.

Test Connection

To confirm that your Master server can locate a Slave server, select the Slave server in the list, and click Test Connection. This is a one-way test; it does not confirm that the Slave server is correctly configured to reach the Master server. A message displays the result.

NOTE:Test Connection fails when:

you are using HTTPS and the required certificates are missing. For more information, see Securing Replication Connections.
the Server role is switched to Standalone to configure Replication. After the server role is changed back to Master or Slave, Test Connection becomes available

5.13.6 Slave Server Role

Use the Slave server role to replicate the configuration of the Master Administrative Server.

Configure slave server

Before you add a replication server, remember:

When you set up a Master server and then specify Slave servers, any existing configuration for the Slave server will be overwritten, and existing sessions on the Slave server will be deleted.
Any security certificates required for an HTTPS transport or for a Security Proxy must be in place before the Slave server is added. For more information, see Securing Replication Connections.

On the SLAVE server that will replicate the Master:

Click Slave.
Check the Database Status. If Up, continue.

If Down or Initializing, wait until the changes take effect and the status is Up.
Database Network Adapter. Choose a network adapter associated with an IP address that the other MSS servers in the cluster can access.
Global protocol. Choose whether to Use HTTPS for server to server communication (between the Master and Slave servers). HTTPS is the default.

When you use HTTPS:
- Verify that a certificate for the Slave server is in the Master server’s trusted store before proceeding with replication setup. See Securing Replication Connections for more information.
- You cannot change the transport and retain previously configured Slave servers. You must delete existing Slave servers before changing the protocol from HTTP to HTTPS (or vice versa).
- If you installed MSS on z Linux with no JRE, be aware of the Caution stated in the Configure master server section.
Change the default Passphrase, if desired.

The Master server and all associated Slave servers must share the same passphrase. Use these fields to change the default passphrase. A changed passphrase is in effect for all replication implementations, whether or not HTTPS is used.
Next step: Add the master server.

Add the master server

A Slave serve can be associated with only one Master Administrative Server. When configured, the Master is listed on this panel. To change the Master, delete the one in the list and click + Add.

To add a Master Server:

Click + Add, which expands the dialog.
Enter the Master server host name.

Add the fully qualified name of a Slave server that is to communicate with the Master server. This name is used to identify where changes are sent and also validates the requests coming in from the Slave server. Enter the name of each Slave server here.
Enter the Master port.

The default host port is 443 for HTTPS and 80 for HTTP. Use the ports that you specified for HTTP and HTTPS during Management and Security Server setup.
Enter the Servlet context.

This entry specifies the web application context for the Master server. The servlet context is used in the URL that accesses the Master server, and is specified when the Administrative Server is installed.

The default, mss, is the correct value if you used the automated installer and did not change the default context as part of setup.
Click Save. to add the Master server to the list .
Click Apply to save the Slave configuration.
Then, return to the Master server to be sure each Slave is added to the list.

Test Connection

To confirm that your Slave server can locate the Master server, click Test Connection. This is a one-way test; it does not confirm that the Master server is correctly configured to reach the Slave server.

A message displays the result. Note: The test fails if you are using HTTPS and the required certificates are missing. For more information, see Securing Replication Connections.

5.13.7 Removing Replicated Servers

When a node is Down, it must be removed from the cluster.

To remove a server from the database cluster:

Open the Database Status Details dialog.
Scroll to the node (server) that is Down, and click Remove.
Close the dialog.

Related topics

5.13.8 Copying Package Data

If you are replicating a server that contains packages for Windows-based sessions, the assignments and settings are replicated automatically. However, the package data must be manually copied to each Slave server.

Package data needs to be manually copied from the Master server to each Slave server when:

new packages are uploaded to the Master server.
existing packages are updated or deleted from the Master server.

To copy the package data:

Upload, update, or delete packages on the Master server.
Delete all .zip files from the /MSSData/deploy/packages/ directory from each Slave server.
Manually copy all of the .zip files from the /MSSData/deploy/packages/ directory on the Master server to the analogous location on each Slave server.
To confirm success: Log in to Management and Security Server on a Slave server as a user who is authorized to receive the package. Verify that the package downloaded and installed successfully.

NOTE:If your client already has the package, first uninstall it from the client, and delete it from C:\Users\<username>\AppData\Local\Temp\MicroFocusPkgs before performing this verification.

5.13.9 Upgrading Replication Servers

If server replication is enabled, you must disable it on every server with replication before you upgrade. Follow these steps:

In the Administrative Console, click Configure Settings - Replication. Select the Standalone option. Click Apply.
Repeat this step for the Master server and all the Slave servers.
Upgrade all the servers.
Configure the Master server from Standalone back to the Master server role, and add the Slave servers.
Configure the Slave servers from Standalone back to the Slave server role, and add the Master server.

5.13.10 Troubleshooting Replication

If you encounter any of these problems while configuring Replication, apply the appropriate workaround. For further assistance, contact Support.

Problem: When setting an MSS server to a Slave role, the Database Status remains at Initializing, and in the Details view, the Owns value for all instances is less than 100%.

Workaround:
1. Set the server back to Standalone role.
2. Change to the Master server role, and open Details.
3. Remove the newly added Slave.
4. Add the server as a Slave again, and Apply the settings.
Problem: When removing a database instance from a cluster using the Details view, the instance remains in the Leaving state.

(NOTE: The Leaving process for a database instance can take several minutes to complete.)

Workaround:
1. If the Leaving state does not change after a few minutes, open a command prompt in the <install-dir>/server/microservices/cassandra/bin directory.
2. Enter the command: nodetool removenode force
Problem: When setting a Slave to Standalone role, the Database Status remains at Initializing, and the Details view shows more than just the Local Instance.

(NOTE: The process for transitioning a server to Standalone role can take several minutes to complete.)

Workaround:
1. Open a command prompt in the <install-dir>/server/microservices/cassandra/bin directory.
2. Enter the command: nodetool removenode force
3. Open the Details view.
  - If only the Local Instance is showing, the status should change to Up.
  - If more than just the Local Instance displays, go back to the command prompt above, and enter the command: nodetool removenode <host-id>
    
    where <host-id> is the Host Id shown in the Details view for the instance(s) to be removed.
Problem: When setting an MSS system to Slave role, and the Database Status remains Down.

Workaround:
1. Change to the Master server role, and open Details.
2. Remove the newly added slave, if present.
3. Restart the Slave server.
Problem: When a server is on DHCP, either MSS fails to start, or the Database Status remains Down.

Workaround:
1. As a administrator, open a command prompt in the <install-dir>/server/microservices/cassandra/conf directory.
2. Open cassandra.yaml in a text editor.
3. Note the value of "listen_interface", which should be similar to "eth0".
4. Use a network utility such as ifconfig or ipconfig to determine the IP address of your network interface.
5. Set the value of "seeds:" to your IP address that corresponds to the listen_interface value (step 3).
6. Restart MSS.

Related topics