Security components troubleshooting tips: WAS v5.1

Security components troubleshooting tips

This document explains basic resources and steps for diagnosing security related issues in the WAS, including...

What log files to look at and what to look for in them.
A general approach to isolating and resolving security problems.
When and how to enable security-related trace.
An overview and table of security-related CORBA minor codes.

The following security-related problems are addressed elsewhere in the information center...

Errors and access problems after enabling security
Errors after enabling SSL, or SSL-related error messages
Errors trying to configure and enable security

If none of these steps solves the problem, check to see if the problem has been identified and documented using the links in Diagnosing and fixing problems: Resources for learning. If you do not see a problem that resembles yours, or if the information provided does not solve your problem, contact IBM support for further assistance.
Note that for an overview of WAS security components such as SAS and how they work, see Getting started with security.

Log filesSDSF output logs

When troubleshooting the security component, browse the JVM logs for the server that hosts the resource you are trying to access. The following is a sample of messages you would expect to see from a server in which the security service has started successfully
SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: SWAM SASRas A JSAS0004I: Principal name: MYHOSTNAME/aServerID SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered. NameServerImp I NMSV0720I: Do Security service listener registration. SecurityCompo A SECJ0242A: Security service is starting UserRegistryI A SECJ0136I: Custom Registry:com.ibm.ws.security.registry.nt. NTLocalDomainRegistryImpl has been initialized SecurityCompo A SECJ0202A: Admin application initialized successfully SecurityCompo A SECJ0203A: Naming application initialized successfully SecurityCompo A SECJ0204A: Rolebased authorizer initialized successfully SecurityCompo A SECJ0205A: Security Admin mBean registered successfully SecurityCompo A SECJ0243A: Security service started successfully SecurityCompo A SECJ0210A: Security enabled true

The following is an example of messages from a server which cannot start the security service, in this case because the administrative user ID and password given to communicate with the user registry is wrong, or the user registry itself is down or misconfigured
SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: SWAM SASRas A JSAS0004I: Principal name: MYHOSTNAME/aServerID SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered. NameServerImp I NMSV0720I: Do Security service listener registration. SecurityCompo A SECJ0242A: Security service is starting UserRegistryI A SECJ0136I: Custom Registry:com.ibm.ws.security. registry.nt.NTLocalDomainRegistryImpl has been initialized Authenticatio E SECJ4001E: Login failed for badID/<null> javax.security.auth.login.LoginException: authentication failed: bad user/password

The following is an example of messages from a server for which LDAP has been specified as the security mechanism, but the LDAP keys have not been properly configured
SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: LTPA SASRas A JSAS0004I: Principal name: MYHOSTNAME/anID SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered. NameServerImp I NMSV0720I: Do Security service listener registration. SecurityCompo A SECJ0242A: Security service is starting UserRegistryI A SECJ0136I: Custom Registry:com.ibm.ws.security.registry.nt. NTLocalDomainRegistryImpl has been initialized SecurityServe E SECJ0237E: One or more vital LTPAServerObject configuration attributes are null or not available. The attributes and values are password : LTPA password does exist, expiration time 30, private key <null>, public key <null>, and shared key <null>.

A problem with the SSL configuration might lead to the following message. You should ensure that the keystore location and keystore passwords are valid. Also, ensure the keystore has a valid personal certificate and that the personal certificate public key or CA root has been extracted on put into the truststore
SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: SWAM SASRas A JSAS0004I: Principal name: MYHOSTNAME/aServerId SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered. SASRas E JSAS0026E: [SecurityTaggedComponentAssistorImpl.register] Exception connecting object to the ORB. Check the SSL configuration to ensure that the SSL keyStore and trustStore properties are set properly. If the problem persists, contact support for assistance. org.omg.CORBA.OBJ_ADAPTER: ORB_CONNECT_ERROR (5) - couldn't get Server Subcontract minor code: 4942FB8F completed: No

General approach for troubleshooting security-related issues

When troubleshooting security-related problems, the following questions are very helpful and should be considered:

Does the problem occur when security is disabled? This is a good litmus test to determine that a problem is security related. However, just because a problem only occurs when security is enabled does not always make it a security problem. More troubleshooting is necessary to ensure the problem is really security-related.

Did security appear to initialize properly? A lot of security code is visited during initialization. So you will likely see problems there first if the problem is configuration related. The following sequence of messages generated in the SystemOut.log indicate normal code initialization of an appserver. This sequence will vary based on the configuration, but the messages are similar
SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: SWAM SASRas A JSAS0004I: Principal name: BIRKT20/pbirk SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered. NameServerImp I NMSV0720I: Do Security service listener registration. SecurityCompo A SECJ0242A: Security service is starting UserRegistryI A SECJ0136I: Custom Registry:com.ibm.ws.security.registry.nt. NTLocalDomainRegistryImpl has been initialized SecurityCompo A SECJ0202A: Admin application initialized successfully SecurityCompo A SECJ0203A: Naming application initialized successfully SecurityCompo A SECJ0204A: Rolebased authorizer initialized successfully SecurityCompo A SECJ0205A: Security Admin mBean registered successfully SecurityCompo A SECJ0243A: Security service started successfully SecurityCompo A SECJ0210A: Security enabled true

Is there a stack trace or exception printed in the SystemOut.log? A single stack trace tells a lot about the problem. What code initiated the code that failed? What is the failing component? Which class did the failure actually come from? Sometimes the stack trace is all that is needed to solve the problem and it can pinpoint the root cause. Other times, it can only give us a clue, and could actually be misleading. When support analyzes a stack trace, they may request additional trace if it is not clear what the problem is. If it appears to be security related and the solution cannot be determined from the stack trace or problem description, you will be asked to gather the following trace specification: SASRas=all=enabled:com.ibm.ws.security.*=all=enabled from all processes involved.

Is this a distributed security problem or a local security problem?

If the problem is local, that is the code involved does not make a remote method invocation, then troubleshooting is isolated to a single process. It is important to know when a problem is local versus distributed since the behavior of the ORB, among other components, is different between the two. Once a remote method invocation takes place, an entirely different security code path is entered.
When you know that the problem involves two or more servers, the techniques of troubleshooting change. You will need to trace all servers involved simultaneously so that the trace shows the client and server sides of the problem. Try to make sure the timestamps on all machines match as closely as possible so that you can find the request and reply pair from two different processes. Enable both SAS and Security trace using the trace specification: SASRas=all=enabled:com.ibm.ws.security.*=all=enabled.

Is the problem related to authentication or authorization? Most security problems fall under one of these two categories. Authentication is the process of determing who the caller is. Authorization is the process of validating that the caller has the proper authority to invoke the requested method. When authentication fails, typically this is related to either the authentication protocol, authentication mechanism or user registry. When authorization fails, this is usually related to the application bindings from assembly and/or deployment and to the caller's identity who is accessing the method and the roles required by the method.

Is this a Web or EJB request?
Web requests have a completely different code path than EJB requests. Also, there are different security features for Web requests than for EJB requests, requiring a completely different body of knowledge to resolve. For example, when using the LTPA authentication mechanism, the Single SignOn feature is available for Web requests but not for EJB requests. Web requests involve HTTP header information not required by EJB requests due to the protocol differences. Also, the Web container (or servlet engine) is involved in the entire process. Any of these components could be involved in the problem and all should be considered during troubleshooting, based on the type of request and where the failure occurs.
Secure EJB requests heavily involve the ORB and Naming components since they flow over the RMI/IIOP protocol. In addition, when work flow management (WLM) is enabled, other behavior changes in the code can be observed. All of these components interact closely for security to work properly in this environment. At times, trace in any or all of these components might be necessary to troubleshoot problems in this area. The trace specification to begin with is SASRas=all=enabled:com.ibm.ws.security.*=all=enabled. ORB trace is also very beneficial when the SAS/Security trace does not seem to pinpoint the problem.

Does the problem seem to be related to the SSL?
The Secure Socket Layer (SSL) is a totally distinct separate layer of security. Troubleshooting SSL problems are usually separate from troubleshooting authentication and/or authorization problems. There are many things to consider. Usually, SSL problems are first time setup problems because the configuration can be difficult. Each client must contain the server's signer certificate. During mutual authentication, each server must contain the client's signer certificate. Also, there can be protocol differences (SSLv3 vs. TLS), and listener port problems related to stale IORs (i.e., IORs from a server reflecting the port prior to the server cycling).
For SSL problems, we sometimes request an SSL trace to determine what is happening with the SSL handshake. The SSL handshake is the process which occurs when a client opens a socket to a server. If anything goes wrong with the key exchange, cipher exchange, etc. the handshake will fail and thus the socket is invalid. Tracing JSSE (the SSL implementation used in WAS) involves the following steps:

Ensure that the client and server processes contain an ibmjsse-debug.jar file in the java/jre/lib/ext directory. The ibmjsse-debug.jar is shipped with the product. You can locate the file under installation_directory\web\docs\jsse. Make sure you remove the existing ibmjsse.jar file from this directory after putting in the ibmjsse-debug.jar. If both exist in the /ext directory, the JSSE trace will not be complete.
Set the following system property on the client and server processes... -Djavax.net.debug=true. For the server, add this to the Generic JVM Arguments property of the Java virtual machine settings page.
Turn on ORB trace as well.
Recreate the problem. The SystemOut.log of both processes should contain the JSSE trace. You will find trace similar to the following
SSLConnection: install <com.ibm.sslite.e@3ae78375> >> handleHandshakeV2 <com.ibm.sslite.e@3ae78375> >> handshakeV2 type = 1 >> clientHello: SSLv2. SSL client version: 3.0 ... ... ... JSSEContext: handleSession[Socket[addr=null,port=0,localport=0]] << sendServerHello. SSL version: 3.0 SSL_RSA_WITH_RC4_128_MD5 HelloRandom ... ... ... << sendCertificate. << sendServerHelloDone. >> handleData <com.ibm.sslite.e@3ae78375> >> handleHandshake <com.ibm.sslite.e@3ae78375> >> handshakeV3 type = 16 >> clientKeyExchange. >> handleData <com.ibm.sslite.e@3ae78375> >> handleChangeCipherSpec <com.ibm.sslite.e@3ae78375> >> handleData <com.ibm.sslite.e@3ae78375> >> handleHandshake <com.ibm.sslite.e@3ae78375> >> handshakeV3 type = 20 >> finished. << sendChangeCipherSpec. << sendFinished.

Tracing security

The classes which implement WAS security are:

com.ibm.ws.security.*
com.ibm.websphere.security.*
com.ibm.WebSphereSecurityImpl.*
SASRas
To view detailed information on the runtime behavior of security, enable trace on the following components and review the output...

com.ibm.ws.security.*=all=enabled:com.ibm:WebSphereSecurityImpl.*= all=enabled:com.ibm.websphere.security.*=all=enabled. This trace statement collects the trace for the security runtime.
com.ibm.ws.console.security.*=all=enabled. This trace statement collects the trace for the security center GUI.
SASRas=all=enabled. This trace statement collects the trace for SAS (low-level authentication logic).

Fine tuning SAS traces... If a subset of classes need to be traced for the SAS/CSIv2 component, a system property can be specified with the class names comma separated: com.ibm.CORBA.securityTraceFilter=SecurityConnectionInterceptorImpl, VaultImpl, ...

Fine tuning Security traces... If a subset of packages need to be traced, specify a trace specification more detailed than com.ibm.ws.security.*=all=enabled. For example, to trace just dynamic policy code, you can specify com.ibm.ws.security.policy.*=all=enabled. To disable dynamic policy trace, you can specify com.ibm.ws.security.policy.*=all=disabled.

Configuring CSIv2 or SAS Trace Settings Situations arise where reviewing trace for the CSIv2 or SAS authentication protocols can assist in troubleshooting difficult problems. This section decribes how to enable to CSIv2/SAS trace.

Enabling Client-Side CSIv2/SAS Trace To enable CSIv2 and SAS trace on a pure client, the following steps need to be taken:

Edit the file TraceSettings.properties in the /WebSphere/AppServer/properties directory.
In this file, change traceFileName= to point to the path in which you want the ouput file created. Make sure you put a double backslash (\\) between each subdirectory. For example, traceFileName=c:\\WebSphere\\AppServer\\logs\\sas_client.log
In this file, add the trace specification string: SASRas=all=enabled. Any additional trace strings can be added on separate lines.
Point to this file from within your client application. On the Java command line where you launch the client, add the following system property: -DtraceSettingsFile=TraceSettings.properties.
Note that Do not give the fully qualified path to the TraceSettings.properties file. Verify that the TraceSettings.properties file is in your classpath.

Enabling Server-Side CSIv2/SAS Trace To enable SAS trace in an appserver, complete the following:

Add the trace specification, SASRas=all=enabled, to the server.xml file or add it to the Trace settings within the WebConsole GUI.
Typically it is best to also trace the authorization security runtime in addition to the authentication protocol runtime. To do this, use the following two trace specifications in combination: SASRas=all=enabled:com.ibm.ws.security.*=all=enabled.
When troubleshooting a connection type problem, it is beneficial to trace both SAS/CSIv2 and the ORB. To do this, use the following three trace specifications: SASRas=all=enabled:com.ibm.ws.security.*=all=enabled:ORBRas=all=enabled.
In addition to adding these trace specifications, for ORB trace there are a couple of system properties that also need to be set. Go to the ORB settings in the GUI and add the following two properties: com.ibm.CORBA.Debug=true and com.ibm.CORBA.CommTrace=true.

CSIv2 CORBA Minor Codes

Whatever exceptions might occur within the security code on either the client or server, the eventual exception will become a CORBA exception. So any exception that occurs gets "wrapped" by a CORBA exception, because the CORBA architecture is used by the security service for its own inter-process communication. CORBA exceptions are generic, and indicate a problem in communication between two components. CORBA minor codes are more specific, and indicate the underlying reason that a component could not complete a request.
The following shows the CORBA Minor codes which a client can expect to receive after executing a security-related request such as authentication. It also includes the CORBA exception type that the minor code would appear in.
The following exception shows an example of a CORBA exception where the minor code is 49424300. From the table below, this minor code indicates Authentication Failure. Typically, a descriptive message is also included in the exception to assist in troubleshooting the problem. Here, the detailed message is "Exception caught invoking authenticateBasicAuthData from SecurityServer for user jdoe. Reason: com.ibm.WebSphereSecurity..uthenticationFailedException" which indicates that the authentication failed for user "jdoe".
The completed field in the exception indicates whether the method was completed or not. In the case of a NO_PERMISSION, the method should never get invoked, so it will always be "completed:No". Other exceptions which are caught on the server side could have a completed status of "Maybe" or "Yes"
org.omg.CORBA.NO_PERMISSION: Caught WSSecurityContextException in WSSecurityContext.acceptSecContext(), reason: Major Code[0] Minor Code[0] Message[Exception caught invoking authenticateBasicAuthData from SecurityServer for user jdoe. Reason: com.ibm.WebSphereSecurity.AuthenticationFailedException] minor code: 49424300 completed: No at com.ibm.ISecurityLocalObjectBaseL13Impl.PrincipalAuthFailReason. map_auth_fail_to_minor_code(PrincipalAuthFailReason.java:83) at com.ibm.ISecurityLocalObjectBaseL13Impl.CSIServerRI.receive_request (CSIServerRI.java:1569) at com.ibm.rmi.pi.InterceptorManager.iterateReceiveRequest (InterceptorManager.java:739) at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:398) at com.ibm.rmi.iiop.ORB.process(ORB.java:313) at com.ibm.CORBA.iiop.ORB.process(ORB.java:1581) at com.ibm.rmi.iiop.GIOPConnection.doWork(GIOPConnection.java:1827) at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:81) at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:91) at com.ibm.ws.util.CachedThread.run(ThreadPool.java:149)

The following table shows the CORBA Minor codes which a client can expect to receive after executing a security-related request such as authentication. It also includes the CORBA exception type that the minor code would appear in.

Minor code name Minor code value (in hex) Exception type (all in the package of org.omg.CORBA .*) Minor code description Retry performed (when authenticationRe tryEnabled=true)
AuthenticationFailed 49424300 NO_PERMISSION This is a generic authentication failed error. It does not give any details about whether the userid or password is invalid. Some registries can choose to use this type of error code, others might choose to use the next three types which are more specific. Yes
InvalidUserid 49424301 NO_PERMISSION This occurs when the registry returns bad userid. Yes
InvalidPassword 49424302 NO_PERMISSION This occurs when the registry returns bad password. Yes
InvalidSecurityCredentials 49424303 NO_PERMISSION This is a generic error indicating that the credentials are bad for whatever reason. It could be that they don't have the right attributes set. Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
InvalidRealm 49424304 NO_PERMISSION This occurs when the REALM in the token received from the client does not match the server's current realm. No
ValidationFailed 49424305 NO_PERMISSION A validation failure occurs when a token is sent from the client or server to a target server but the token format or the expiration is invalid. Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
CredentialTokenExpired 49424306 NO_PERMISSION This is more specific about why the validation failed. In this case, the token has a absolute lifetime, and this lifetime has expired. Therefore, it is no longer a valid token and cannot be used. Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
InvalidCredentialToken 49424307 NO_PERMISSION This is more specific about why the validation failed. In this case, the token cannot be decrypted or the data within it is not readable. Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
SessionDoesNotExist 49424308 NO_PERMISSION This indicates that the CSIv2 session does not exist on the server. Typically, a retry occurs automatically and will successfully create a new session. Yes
SessionConflictingEvidence 49424309 NO_PERMISSION This indicates that a session already exists on the server which matches the context_id sent over by the client, however, the information provided by the client for this EstablishContext message is different from the information originally provided to establish the session. Yes
SessionRejected 4942430A NO_PERMISSION This indicates that the session referenced by the client has been previously rejected by the server. Yes
SecurityServerNotAvailable 4942430B NO_PERMISSION This error occurs when the server cannot contact the security server (whether local or remote) in order to authenticate or validate. No
InvalidIdentityToken 4942430C NO_PERMISSION This error indicates that identity cannot be obtained from the identity token when Identity Assertion is enabled. No
IdentityServerNotTrusted 4942430D NO_PERMISSION This indicates that the server id of the sending server is not on the target server's trusted principal list. No
InvalidMessage 4942430E NO_PERMISSION This indicates that the CSIv2 message format is invalid for the receiving server. No
AuthenticationNotSupported 49421090 NO_PERMISSION This error occurs when a mechanism does not support authentication (very rare). No
InvalidSecurityMechanism 49421091 NO_PERMISSION This is used to indicate that the specified security mechanism is not known. No
CredentialNotAvailable 49421092 NO_PERMISSION This indicates a credential is not available when it is required. No
SecurityMechanismNotSupported 49421093 NO_PERMISSION This error occurs when a security mechanism specified in the CSIv2 token is not implemented on the server. No
ValidationNotSupported 49421094 NO_PERMISSION This error occurs when a mechanism does not support validation (such as LocalOS). This error should not occur since the LocalOS credential is not a forwardable credential, therefore, validation should never need to be called on it. No
CredentialTokenNotSet 49421095 NO_PERMISSION This is used to indicate the token inside the credential is null. No
ServerConnectionFailed 494210A0 COMM_FAILURE This error is used when a connection attempt fails. Yes (via ORB retry)
CorbaSystemException 494210B0 INTERNAL This is a generic CORBA specific exception in system code. No
JavaException 494210B1 INTERNAL This is a generic error that indicated an unexpected Java exception occurred. No
ValueIsNull 494210B2 INTERNAL This is used to indicate that a value or parameter passed in was null. No
EffectivePolicyNotPresent 494210B3 INTERNAL This indicates that an effective policy object for CSIv2 is not present. This object is used to determine what security configuration features have been specified. No
NullPointerException 494210B4 INTERNAL This is used to indicate that a NullPointerException was caught in the runtime. No
ErrorGettingClassInstance 494210B5 INTERNAL This indicates a problem loading a class dynamically. No
MalFormedParameters 494210B6 INTERNAL This indicates parameters are not valid. No
DuplicateSecurityAttributeType 494210B7 INTERNAL A duplicate credential attribute has been specified during the set_attributes operation. No
MethodNotImplemented 494210C0 NO_IMPLEMENT A method invoked has not been implemented. No
GSSFormatError 494210C5 BAD_PARAM This indicates that a GSS encoding or decoding routine has thrown an exception. No
TagComponentFormatError 494210C6 BAD_PARAM This indicates that a tag component cannot be read properly. No
InvalidSecurityAttributeType 494210C7 BAD_PARAM This indicates an attribute type specified during the set_attributes operation is an invalid type. No
SecurityConfigError 494210CA INITIALIZE A problem exists between the client and server configuration. No

For current information available from IBM Support on known problems and their resolution, see IBM Support page.
IBM Support has documents that can save you time gathering information needed to resolve this problem. Before opening a PMR, see IBM Support page.

See Also
Troubleshooting installation problems

Does the problem occur when security is disabled?	This is a good litmus test to determine that a problem is security related. However, just because a problem only occurs when security is enabled does not always make it a security problem. More troubleshooting is necessary to ensure the problem is really security-related.
Did security appear to initialize properly?	A lot of security code is visited during initialization. So you will likely see problems there first if the problem is configuration related. The following sequence of messages generated in the SystemOut.log indicate normal code initialization of an appserver. This sequence will vary based on the configuration, but the messages are similar SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: SWAM SASRas A JSAS0004I: Principal name: BIRKT20/pbirk SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered. NameServerImp I NMSV0720I: Do Security service listener registration. SecurityCompo A SECJ0242A: Security service is starting UserRegistryI A SECJ0136I: Custom Registry:com.ibm.ws.security.registry.nt. NTLocalDomainRegistryImpl has been initialized SecurityCompo A SECJ0202A: Admin application initialized successfully SecurityCompo A SECJ0203A: Naming application initialized successfully SecurityCompo A SECJ0204A: Rolebased authorizer initialized successfully SecurityCompo A SECJ0205A: Security Admin mBean registered successfully SecurityCompo A SECJ0243A: Security service started successfully SecurityCompo A SECJ0210A: Security enabled true
Is there a stack trace or exception printed in the SystemOut.log?	A single stack trace tells a lot about the problem. What code initiated the code that failed? What is the failing component? Which class did the failure actually come from? Sometimes the stack trace is all that is needed to solve the problem and it can pinpoint the root cause. Other times, it can only give us a clue, and could actually be misleading. When support analyzes a stack trace, they may request additional trace if it is not clear what the problem is. If it appears to be security related and the solution cannot be determined from the stack trace or problem description, you will be asked to gather the following trace specification: SASRas=all=enabled:com.ibm.ws.security.*=all=enabled from all processes involved.
Is this a distributed security problem or a local security problem?	If the problem is local, that is the code involved does not make a remote method invocation, then troubleshooting is isolated to a single process. It is important to know when a problem is local versus distributed since the behavior of the ORB, among other components, is different between the two. Once a remote method invocation takes place, an entirely different security code path is entered. When you know that the problem involves two or more servers, the techniques of troubleshooting change. You will need to trace all servers involved simultaneously so that the trace shows the client and server sides of the problem. Try to make sure the timestamps on all machines match as closely as possible so that you can find the request and reply pair from two different processes. Enable both SAS and Security trace using the trace specification: SASRas=all=enabled:com.ibm.ws.security.*=all=enabled.
Is the problem related to authentication or authorization?	Most security problems fall under one of these two categories. Authentication is the process of determing who the caller is. Authorization is the process of validating that the caller has the proper authority to invoke the requested method. When authentication fails, typically this is related to either the authentication protocol, authentication mechanism or user registry. When authorization fails, this is usually related to the application bindings from assembly and/or deployment and to the caller's identity who is accessing the method and the roles required by the method.
Is this a Web or EJB request?	Web requests have a completely different code path than EJB requests. Also, there are different security features for Web requests than for EJB requests, requiring a completely different body of knowledge to resolve. For example, when using the LTPA authentication mechanism, the Single SignOn feature is available for Web requests but not for EJB requests. Web requests involve HTTP header information not required by EJB requests due to the protocol differences. Also, the Web container (or servlet engine) is involved in the entire process. Any of these components could be involved in the problem and all should be considered during troubleshooting, based on the type of request and where the failure occurs. Secure EJB requests heavily involve the ORB and Naming components since they flow over the RMI/IIOP protocol. In addition, when work flow management (WLM) is enabled, other behavior changes in the code can be observed. All of these components interact closely for security to work properly in this environment. At times, trace in any or all of these components might be necessary to troubleshoot problems in this area. The trace specification to begin with is SASRas=all=enabled:com.ibm.ws.security.*=all=enabled. ORB trace is also very beneficial when the SAS/Security trace does not seem to pinpoint the problem.
Does the problem seem to be related to the SSL?	The Secure Socket Layer (SSL) is a totally distinct separate layer of security. Troubleshooting SSL problems are usually separate from troubleshooting authentication and/or authorization problems. There are many things to consider. Usually, SSL problems are first time setup problems because the configuration can be difficult. Each client must contain the server's signer certificate. During mutual authentication, each server must contain the client's signer certificate. Also, there can be protocol differences (SSLv3 vs. TLS), and listener port problems related to stale IORs (i.e., IORs from a server reflecting the port prior to the server cycling). For SSL problems, we sometimes request an SSL trace to determine what is happening with the SSL handshake. The SSL handshake is the process which occurs when a client opens a socket to a server. If anything goes wrong with the key exchange, cipher exchange, etc. the handshake will fail and thus the socket is invalid. Tracing JSSE (the SSL implementation used in WAS) involves the following steps: Ensure that the client and server processes contain an ibmjsse-debug.jar file in the java/jre/lib/ext directory. The ibmjsse-debug.jar is shipped with the product. You can locate the file under installation_directory\web\docs\jsse. Make sure you remove the existing ibmjsse.jar file from this directory after putting in the ibmjsse-debug.jar. If both exist in the /ext directory, the JSSE trace will not be complete. Set the following system property on the client and server processes... -Djavax.net.debug=true. For the server, add this to the Generic JVM Arguments property of the Java virtual machine settings page. Turn on ORB trace as well. Recreate the problem. The SystemOut.log of both processes should contain the JSSE trace. You will find trace similar to the following SSLConnection: install <com.ibm.sslite.e@3ae78375> >> handleHandshakeV2 <com.ibm.sslite.e@3ae78375> >> handshakeV2 type = 1 >> clientHello: SSLv2. SSL client version: 3.0 ... ... ... JSSEContext: handleSession[Socket[addr=null,port=0,localport=0]] << sendServerHello. SSL version: 3.0 SSL_RSA_WITH_RC4_128_MD5 HelloRandom ... ... ... << sendCertificate. << sendServerHelloDone. >> handleData <com.ibm.sslite.e@3ae78375> >> handleHandshake <com.ibm.sslite.e@3ae78375> >> handshakeV3 type = 16 >> clientKeyExchange. >> handleData <com.ibm.sslite.e@3ae78375> >> handleChangeCipherSpec <com.ibm.sslite.e@3ae78375> >> handleData <com.ibm.sslite.e@3ae78375> >> handleHandshake <com.ibm.sslite.e@3ae78375> >> handshakeV3 type = 20 >> finished. << sendChangeCipherSpec. << sendFinished.

Minor code name	Minor code value (in hex)	Exception type (all in the package of org.omg.CORBA .*)	Minor code description	Retry performed (when authenticationRe tryEnabled=true)
AuthenticationFailed	49424300	NO_PERMISSION	This is a generic authentication failed error. It does not give any details about whether the userid or password is invalid. Some registries can choose to use this type of error code, others might choose to use the next three types which are more specific.	Yes
InvalidUserid	49424301	NO_PERMISSION	This occurs when the registry returns bad userid.	Yes
InvalidPassword	49424302	NO_PERMISSION	This occurs when the registry returns bad password.	Yes
InvalidSecurityCredentials	49424303	NO_PERMISSION	This is a generic error indicating that the credentials are bad for whatever reason. It could be that they don't have the right attributes set.	Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
InvalidRealm	49424304	NO_PERMISSION	This occurs when the REALM in the token received from the client does not match the server's current realm.	No
ValidationFailed	49424305	NO_PERMISSION	A validation failure occurs when a token is sent from the client or server to a target server but the token format or the expiration is invalid.	Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
CredentialTokenExpired	49424306	NO_PERMISSION	This is more specific about why the validation failed. In this case, the token has a absolute lifetime, and this lifetime has expired. Therefore, it is no longer a valid token and cannot be used.	Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
InvalidCredentialToken	49424307	NO_PERMISSION	This is more specific about why the validation failed. In this case, the token cannot be decrypted or the data within it is not readable.	Yes, if client has BasicAuth credential (token based credential was rejected in the first place).
SessionDoesNotExist	49424308	NO_PERMISSION	This indicates that the CSIv2 session does not exist on the server. Typically, a retry occurs automatically and will successfully create a new session.	Yes
SessionConflictingEvidence	49424309	NO_PERMISSION	This indicates that a session already exists on the server which matches the context_id sent over by the client, however, the information provided by the client for this EstablishContext message is different from the information originally provided to establish the session.	Yes
SessionRejected	4942430A	NO_PERMISSION	This indicates that the session referenced by the client has been previously rejected by the server.	Yes
SecurityServerNotAvailable	4942430B	NO_PERMISSION	This error occurs when the server cannot contact the security server (whether local or remote) in order to authenticate or validate.	No
InvalidIdentityToken	4942430C	NO_PERMISSION	This error indicates that identity cannot be obtained from the identity token when Identity Assertion is enabled.	No
IdentityServerNotTrusted	4942430D	NO_PERMISSION	This indicates that the server id of the sending server is not on the target server's trusted principal list.	No
InvalidMessage	4942430E	NO_PERMISSION	This indicates that the CSIv2 message format is invalid for the receiving server.	No
AuthenticationNotSupported	49421090	NO_PERMISSION	This error occurs when a mechanism does not support authentication (very rare).	No
InvalidSecurityMechanism	49421091	NO_PERMISSION	This is used to indicate that the specified security mechanism is not known.	No
CredentialNotAvailable	49421092	NO_PERMISSION	This indicates a credential is not available when it is required.	No
SecurityMechanismNotSupported	49421093	NO_PERMISSION	This error occurs when a security mechanism specified in the CSIv2 token is not implemented on the server.	No
ValidationNotSupported	49421094	NO_PERMISSION	This error occurs when a mechanism does not support validation (such as LocalOS). This error should not occur since the LocalOS credential is not a forwardable credential, therefore, validation should never need to be called on it.	No
CredentialTokenNotSet	49421095	NO_PERMISSION	This is used to indicate the token inside the credential is null.	No
ServerConnectionFailed	494210A0	COMM_FAILURE	This error is used when a connection attempt fails.	Yes (via ORB retry)
CorbaSystemException	494210B0	INTERNAL	This is a generic CORBA specific exception in system code.	No
JavaException	494210B1	INTERNAL	This is a generic error that indicated an unexpected Java exception occurred.	No
ValueIsNull	494210B2	INTERNAL	This is used to indicate that a value or parameter passed in was null.	No
EffectivePolicyNotPresent	494210B3	INTERNAL	This indicates that an effective policy object for CSIv2 is not present. This object is used to determine what security configuration features have been specified.	No
NullPointerException	494210B4	INTERNAL	This is used to indicate that a NullPointerException was caught in the runtime.	No
ErrorGettingClassInstance	494210B5	INTERNAL	This indicates a problem loading a class dynamically.	No
MalFormedParameters	494210B6	INTERNAL	This indicates parameters are not valid.	No
DuplicateSecurityAttributeType	494210B7	INTERNAL	A duplicate credential attribute has been specified during the set_attributes operation.	No
MethodNotImplemented	494210C0	NO_IMPLEMENT	A method invoked has not been implemented.	No
GSSFormatError	494210C5	BAD_PARAM	This indicates that a GSS encoding or decoding routine has thrown an exception.	No
TagComponentFormatError	494210C6	BAD_PARAM	This indicates that a tag component cannot be read properly.	No
InvalidSecurityAttributeType	494210C7	BAD_PARAM	This indicates an attribute type specified during the set_attributes operation is an invalid type.	No
SecurityConfigError	494210CA	INITIALIZE	A problem exists between the client and server configuration.	No