5.13 Replication

Server replication makes it easier to configure and maintain multiple Administrative Servers, provide high availability, and have more flexibility when spanning geographically distant server installations.

Using replicated servers, you can synchronize multiple administrative servers by propagating configuration and session changes made on one server (the Master) to all of the servers in a replication group (the Slaves).

NOTE:To configure Metering for high availability, you must configure a master and at least two slaves.

Before you begin, review these topics:

5.13.1 Replication Database

Management and Security Server (MSS) uses a database to store information on replicated servers. This implementation introduces some terms to the Replication UI.

Database. The database contains a cluster of MSS servers.

A node represents one server in the cluster.
Database Status. The current status displays on the Configure Settings-Replication panel.

After you add or edit settings and click Apply, the MSS server initializes and implements the changed settings.The Database Status indicates the phase of the initialization process.

When the Database Status reaches the Up phase (indicating that the initialization is complete), you can make additional changes if needed.
Details. Information about each node in the cluster. Replication is functioning properly when each node reports Status: UP and Owns: 100%. (That is, the server contains 100% of the replicated data.)

Related topics

5.13.2 How to Configure Replication

Perform the configuration tasks in this order to set the trust relationships.

Identify which MSS server will be the Master, and which ones will be Slave servers.

As you decide, consider the Server role: Master and Slave servers.
For each server to be clustered, import the certificates for the Master and all of the Slave servers to the Trusted Certificates store. See Importing Server Certificates for detailed steps.
Configure the Master server. See Master Server Role for detailed steps.
Configure each Slave server. See Slave Server Role for detailed steps.

Server role: Master and Slave servers

To implement server replication, you will configure one Master server and one or more Slave server

Before you begin, consider the roles of the Master and Slave servers and how to secure the connections

Slave servers are configured according to the settings on this Master server. Any subsequent changes to the Master server will be automatically replicated to the Slave servers.
Only one Master server can be configured, but multiple Slave servers can be associated with that Master server.
You can modify the configuration on the Master server and push those updates to the Slaves.
You can also modify the configuration on a Slave server and submit the changes, but they will not be applied until the requested change is implemented on the Master server.
All configuration elements of the Master Administrative Server are replicated to a Slave server -- except:
- - log files
- - Credential Store settings
- - some Certificates settings
- - the Web Agent name, when SiteMinder is used for authentication. (The Web Agent name must be set separately for each replicated machine.)

5.13.3 Importing Server Certificates

Before you can configure replication, you must establish trust between the Master and the Slaves by entering all of the servers’ certificates into the Trusted Certificate store. This is required regardless of your security settings.

The certificate of the current Administrative Server, along with the certificates of the remote Administrative Servers must be included in the list.This step is required for all servers, Master or Slave, in the management cluster. Self-signed, or other certificates that are not well known, must be imported to all Administrative Servers in the cluster.

Before you add the Master or Slave servers:

Open Administrative Console > Configure Settings > Trusted Certificates.
Under Certificate Store, click Management and Security Server.
In the list of certificates that are trusted by MSS, confirm that a trusted certificate is present. If not, continue importing the certificate.
Copy the remote server certificate, found here, <install-dir>/MSS/server/etc/servletcontainer.cer, to this location on the local server, <mss-data-dir>/certificates. You must provide a unique file name when you copy the file. Best practice is to name the file <server-name>.cer where server-name is the name of the remote server.
Click + Import to add a certificate.
Enter the file name.
For the friendly name use the server name you provided in Step 4.
Click Import.

NOTE:If you installed MSS on z/Linux without a JRE (using the UNIX no-jre installer, see the Caution in the Configure master server section. You may need to edit a file.

5.13.4 Securing Replication Connections

You have the option to use HTTP or HTTPS as your server-to-server communication transport. HTTPS is recommended.

When using a Security Proxy

If you are using a Security Proxy server, you must also import the certificates for all the remote Administrative Servers to each Security Proxy server. Use the Security Proxy Wizard to import these certificates.

When you create a secure session that connects to a Security Proxy server, the session is thereafter linked to this specific Security Proxy. When this session is replicated to other servers in the cluster, the session is then initiated from a different Administrative Server, but the session itself will still connect to the original Security Proxy for which it was configured.

If client authorization is enabled on the Security Proxy Server, then the Security Proxy Server will only accept connections from sessions initiated from Administrative Servers it trusts (that is, their certificates are in the Security Proxy Trusted Certificate list).

In order for connections from replicated servers to succeed in this environment, the certificates from every Administrative Server in the cluster need to be imported to the Security Proxy server. If there are multiple Security Proxy Servers in the cluster, then this operation needs to be done on each of these Security Proxy Servers.

NOTE:The Concurrency Lock Timeout, included in the former Administrative WebStation interface, applies to managing sessions. See the Note in Edit a session.

Related topics

5.13.5 Standalone Server Role

This is the default role, and applies to Administrative Servers that are not configured for replication. The Standalone server role is used to reset Replication, such as when upgrading.

To set up Replication, configure a Master server, and add Slave servers.

5.13.6 Master Server Role

Use the Master server role to set the configuration of this Administrative Server as the one that will be replicated to the specified Slave Administrative Servers.

Configure master server

Configure master server

On the MASTER server that you want to replicate:

Click Master.
Check the Database Status. If Up, continue.

If Down or Initializing, wait until the changes take effect and the status is Up.
Database Network Adapter. Choose a network adapter associated with an IP address that the other MSS servers in the cluster can access.
Global protocol. Choose whether to Use HTTPS for server to server communication (between the Master and Slave servers). HTTPS is the default. If you choose to clear the HTTPS requirement, the Importing Server Certificates requirements still apply.
CAUTION:When HTTPS is checked:
- You cannot change the transport and retain previously configured Slave servers. You must delete existing Slave servers before changing the protocol from HTTP to HTTPS (or vice versa).
- If you installed MSS on z Linux with no JRE (using the UNIX no-jre installer), you must edit this file in the MSS installation directory:
  
  mss/server/microservices/cassandra/conf/cassandra.yaml
  
  Change # algorithm: SunX509
  
  to algorithm: IBMX509 (uncommented)
  
  You do not need to restart the MSS server.
  
  Without this change, the cassandra database fails to restart after replication is configured, and this exception is added to the mss/server/logs/cassandra.log:
  
  "Caused by: java.security.NoSuchAlgorithmException: SunX509 TrustManagerFactory not available"
  
  To resolve the exception, set the Replication server back to Standalone. Edit cassandra.yaml, as above, and proceed with Replication.
Configure Master Server. Change the default Passphrase, if desired.

The Master server and all associated Slave servers must share the same passphrase. Use these fields to change the default passphrase.

If you make no changes to the passphrase on any of the Administrative Servers in the cluster, the passphrase is automatically the same on all servers. A changed passphrase is in effect for all replication implementations, whether or not HTTPS is used.
Click Apply. To proceed, choose Yes on the Settings update confirmation. This confirmation causes the database to restart to allow for remote communication.
After the database has transitioned back to an UP status, click Details and confirm that the Local Instance Address is Not 127.0.0.1.

The next step is to proceed to the Slave Server Role section for information on adding slaves. You should complete all the steps for each slave before adding additional ones.

5.13.7 Slave Server Role

Use the Slave server role to replicate the configuration of the Master Administrative Server.

Configure slave server

Before you add a replication server, remember:

When you set up a Master server and then specify Slave servers, any existing configuration for the Slave server will be overwritten, and existing sessions on the Slave server will be deleted.
It is important that you have followed the steps for importing trust certificates before you continue. This is a requirement regardless of your security settings.

On the SLAVE server that will replicate the Master:

Click Slave.
Check the Database Status. If Up, continue.

If Down or Initializing, wait until the changes take effect and the status is Up.
Database Network Adapter. Choose a network adapter associated with an IP address that the other MSS servers in the cluster can access.
Global protocol. Choose whether to Use HTTPS for server to server communication (between the Master and Slave servers). HTTPS is the default.

When you use HTTPS:
- You cannot change the transport and retain previously configured Slave servers. You must delete existing Slave servers before changing the protocol from HTTP to HTTPS (or vice versa).
- If you installed MSS on z Linux with no JRE, be aware of the Caution stated in the Configure master server section.
Change the default Passphrase, if desired.

The Master server and all associated Slave servers must share the same passphrase. Use these fields to change the default passphrase. A changed passphrase is in effect for all replication implementations, whether or not HTTPS is used.
Next step: Add the master server.

Add the master server

A Slave server can be associated with only one Master Administrative Server. When configured, the Master is listed on this panel. To change the Master, delete the one in the list and click + Add.

To add a Master Server:

Click + Add, which expands the dialog.
Enter the Master server host name.

Add the fully qualified name of the Master server.
Enter the Master port.

The default host port is 443 for HTTPS and 80 for HTTP. Use the ports that you specified for HTTP and HTTPS during Management and Security Server setup.
Enter the Servlet context.

This entry specifies the web application context for the Master server. The servlet context is used in the URL that accesses the Master server, and is specified when the Administrative Server is installed.

The default, mss, is the correct value if you used the automated installer and did not change the default context as part of setup.
Click Save. to add the Master server to the list.
Select the server checkbox and click Test Connection to ensure the slave servers can communicate with the master.
Click Apply to save the configuration. To proceed, choose Yes on the Settings update confirmation. This confirmation causes the database to re-initialize to start the initial replication process with the master server.

This process can take some time. When the Database status transitions through, completing with an Up status, the initial replication is complete.
Click Details and ensure that all servers in the table show an 100% value in the Owns section.
The next step: Adding a slave to the master server.

Add a slave to the master server

From the Configure Settings > Replication panel on the master server, click +Add to add a slave server to the Slave Server table.
Enter the host name of the slave server you are adding.
Enter the port of the salve server. This will vary depending on what you entered in the User HTTPS for server to server communication setting. Click Save.
Select the newly added slave server and click Test Connection to ensure the master can now communicate with the slave.
Click Apply to save the newly added Slave. To proceed, choose Yes on the Settings update confirmation box. The Master will then contact all the slaves to let them know about the newly added slave. The database status will read Status unavailable due to registry restart while contact is being made, but will eventually transition back to the Up status when the process is complete.

Remove a running slave from a cluster

You can remove a running slave from a cluster and the slave will maintain all of its current data. However, after being removed from the cluster, the server will no longer receive replicated data.

To remove a running slave from a cluster:

From the Configure Settings > Replication panel on the slave server you want to remove.
Select the server from the Master server list and click Delete.
Set the Server Role to Standalone and click Apply.
To proceed, click Yes on the Settings update confirmation panel. The database status will cycle through Down, Initializing, and finally Up when the process is complete.
Click Details and verify that the address is 127.0.0.1.
On the Configure Settings > Replication panel on the Master server, click Details and then Remove on the slave server you removed.
To proceed, click Yes on the Settings update confirmation panel. The database status will cycle through Down, Initializing, and finally Up when the process is complete.
Select the removed slave from the Slave Server table and click Delete.
Click Apply to save the removed Slave settings. To proceed, click Yes on the Settings update confirmation panel. The master contacts all the slaves to inform them about the removed slave. While this occurs, the Database status will be Status unavailable due to registry restart. The process completes with an Up status.

Remove a down slave from a cluster

When a slave server goes down and cannot be restarted, it must be removed from the cluster.

WARNING:If a down server is ever restarted, then the original cluster will be stuck in the Initializing status. To bring the server back into the cluster follow the first workaround in the Troubleshooting Replication. To remove the server from the cluster follow the steps in Remove a running slave from a cluster.

To remove a server from the database cluster:

From the Configure Settings > Replication panel on the Master server, open the Database Status Details dialog box and click Remove for the down server.
To proceed, click Yes on the Settings update confirmation panel. The database status cycles through Initializing and completes with an Up status once the server has been removed.
Select the down server from the Slave Server table and click Delete.
Click Apply to save the removed Slave settings. To proceed, click Yes on the Settings update confirmation panel. The master contacts all the slaves to inform them about the removed slave. While this occurs, the Database status will be Status unavailable due to registry restart. The process completes with an Up status.

Since the down server will not be able to be contacted you will receive an error message stating the it could not be updated. This is expected and you can safely dismiss it.

Related topics

5.13.8 Copying Package Data

If you are replicating a server that contains packages for Windows-based sessions, the assignments and settings are replicated automatically. However, the package data must be manually copied to each Slave server.

Package data needs to be manually copied from the Master server to each Slave server when:

new packages are uploaded to the Master server.
existing packages are updated or deleted from the Master server.

To copy the package data:

Upload, update, or delete packages on the Master server.
Delete all .zip files from the /MSSData/deploy/packages/ directory from each Slave server.
Manually copy all of the .zip files from the /MSSData/deploy/packages/ directory on the Master server to the analogous location on each Slave server.
To confirm success: Log in to Management and Security Server on a Slave server as a user who is authorized to receive the package. Verify that the package downloaded and installed successfully.

NOTE:If your client already has the package, first uninstall it from the client, and delete it from C:\Users\<username>\AppData\Local\Temp\MicroFocusPkgs before performing this verification.

5.13.9 Upgrading Replication Servers

If server replication is enabled, you must disable it on every server with replication before you upgrade. Follow these steps:

Set each Master server to Standalone and wait for the Database status to transition to Up.
Set each Slave server to Standalone and wait for the Database status to transition to Up.
Upgrade all the servers.
Follow the steps to:
- Configure master server
- Add a slave to the master server

5.13.10 Troubleshooting Replication

If you encounter any of these problems while configuring Replication, apply the appropriate workaround. For further assistance, contact Support.

Problem: When setting an MSS server to a Slave role, the Database Status remains at Initializing, and in the Details view, the Owns value for all instances is less than 100%.

Workaround:
1. Set the server back to Standalone role and apply the change.
2. From the Configure Settings > Replication panel on the Master server, open the Database Status Details dialog box and click Remove for the down server.
3. To proceed, click Yes on the Settings update confirmation panel. The database status cycles through Initializing and completes with an Up status once the server has been removed.
4. Add the server as a Slave again, and Apply the settings.
Problem: When removing a database instance from a cluster using the Details view, the instance remains in the Leaving state.

(NOTE: The Leaving process for a database instance can take several minutes to complete.)

Workaround:
1. If the Leaving state does not change after a few minutes, open a command prompt in the <install-dir>/server/microservices/cassandra/bin directory.
2. Enter the command: nodetool removenode force
Problem: When setting a Slave to Standalone role, the Database Status remains at Initializing, and the Details view shows more than just the Local Instance.

(NOTE: The process for transitioning a server to Standalone role can take several minutes to complete.)

Workaround:
1. Open a command prompt in the <install-dir>/server/microservices/cassandra/bin directory.
2. Enter the command: nodetool removenode force
3. Open the Details view.
  - If only the Local Instance is showing, the status should change to Up.
  - If more than just the Local Instance displays, go back to the command prompt above, and enter the command: nodetool removenode <host-id>
    
    where <host-id> is the Host Id shown in the Details view for the instance(s) to be removed.
Problem: When setting an MSS system to Slave role, and the Database Status remains Down.

Workaround:
1. Change to the Master server role, and open Details.
2. Remove the newly added slave, if present.
3. Restart the Slave server.
Problem: When a server is on DHCP, either MSS fails to start, or the Database Status remains Down.

Workaround:
1. As a administrator, open a command prompt in the <install-dir>/server/microservices/cassandra/conf directory.
2. Open cassandra.yaml in a text editor.
3. Note the value of "listen_interface", which should be similar to "eth0".
4. Use a network utility such as ifconfig or ipconfig to determine the IP address of your network interface.
5. Set the value of "seeds:" to your IP address that corresponds to the listen_interface value (step 3).
6. Restart MSS.

Related topics