Administration guide > Operate the deployment environment
Ovveride catalog service quorum
When the data center enters a failure scenario, consider overriding quorum so that container server events are not ignored. Use the xsadmin tool to query about and perform quorum tasks, such as the quorum status and overriding quorum.
Before you begin
The following message indicates that quorum has been lost. Look for this message in the catalog service logs.
CWOBJ1254W: The catalog service is waiting for quorum.
WebSphere eXtreme Scale expects to lose quorum for the following reasons:
- Catalog service JVM member fails
- Network brown out
- Data center loss
You should only override quorum in a data center failure scenario. When you override quorum, any surviving catalog server instance can be used. All survivors are notified when one is told to override quorum.
- Query quorum status with the xsadmin tool.
xsadmin -ch cathost -p 1099 -quorumstatus
Use this option to display the quorum status of a catalog service instance. One of the following outcomes is displayed:
- Quorum is disabled: The catalog servers are running in a quorum-disabled mode. This is a development or single only data center mode. It is not recommended for multi-data-center configurations.
- Quorum is enabled and the catalog server has quorum: Quorum is enabled and the system is working normally.
- Quorum is enabled but the catalog server is waiting for quorum: Quorum is enabled and quorum has been lost.
- Quorum is enabled and the quorum is overridden: Quorum is enabled and quorum has been overridden.
- Quorum status is outlawed: When a brown out occurs, splitting the catalog service into two partitions, A and B. The catalog server A has overridden quorum. The network partition resolves and the server in the B partition is outlawed, requiring a JVM restart. It also occurs if the catalog JVM in B restarts during the brown out and then the brown out clears.
- Override quorum with the xsadmin tool.
xsadmin -ch cathost -p 1099 -overridequorum
- Diagnose quorum with the xsadmin tool.
- Display a list of the core groups:
Use the -coregroups option to display a list of all the core groups for the catalog server.
xsadmin –ch cathost –p 1099 –coregroups
- Teardown servers:
Use the -teardown option to remove a server manually from the data grid. This is not needed normally since servers are automatically removed when they are detected as failed, but the command is provided for use under IBM support help. See Stop servers gracefully with the xsadmin tool for more information about using this command.
xsadmin –ch cathost –p 1099 –g Grid –teardown server1,server2,server3
- Display the route table:
Use the -routetable option to display the current route table by simulating a new client connection to the data grid. It also validates the route table by confirming that all container servers are recognizing their role in the route table, such as which type of shard for which partition.
xsadmin –ch cathost –p 1099 –g myGrid -routetable
- Check the map sizes:
Use the -mapsizes option to verify that key distribution is uniform over the shards in the key. If some container servers have significantly more keys than others, then it is likely the hash function on the key objects has a poor distribution.
xsadmin -ch cathost -p 1099 -g myGrid -m myMapSet -mapsizes myMap
- Set trace strings:
Use the -settracespec option to set the trace settings for all JVMs that match the filter specified for the xsadmin command. This setting only changes the trace settings until another command is used or the JVMs modified fail or stop.
xsadmin –ch cathost –p 1099 –g myGrid –fh host1 –settracespec ObjectGrid*=event=enabled
This string enables trace for all JVMs on the server with the specified host name, in this case host1.
- Display unassigned shards:
Use the -unassigned option to display the list of shards that cannot be placed on the data grid. Shards cannot be placed when the placement service has a constraint that is preventing placement. For example, if you start JVMs on a single physical server while in production mode, then only primary shards can be placed. Replicas are not assigned until JVMs start on a second physical server. The placement service only places replicas on JVMs with different IP addresses than the JVMs that are hosting the primary shards. Having no JVMs in a zone can also cause shards to be unassigned.
xsadmin –ch cathost –p 1099 –g myGrid –unassigned
Parent topic:Operate the deployment environment
Catalog server quorums
Start and stop stand-alone servers
Start and stop servers in a WAS environment
Use the embedded server API to start and stop servers
Manage ObjectGrid availability
Monitor with the xsAdmin sample utility
Administer programmatically with Managed Beans (MBeans)