SYMPTOMS

Data Protector GUI shows error message “Unable to get Authentication token. Check if AppServer is running.”

Data Protector GUI shows error message “User Authentication failed.”

Data Protector user migration failed during installation (Data Protector GUI doesn’t show users and
schedules).

SCOPE
Data Protector 10.03 CM platforms.

CAUSES
1. Hostname resolution is inconsistent.
2. Certificates generated are not in sync state or were not created for appropriate host names.
3. The Data Protector GUI/CLI is unable to correctly authenticate.
4. User migration was not successfully completed for all users.

DATA COLLECTION
For analyzing of problems next data should be collected:
a. AppServer logs
Win: %DP_DATA_DIR%\log\AppServer
UNIX: /var/opt/omni/log/AppServer
b. Standalone.xml, dp-webservice-roles.properties, jce-webservice-roles.properties
Win: %DP_SDATA_DIR%\Config\server\AppServer
UNIX: /etc/opt/omni/server/AppServer
c. All webservices.properties files found under location
Win: %DP_DATA_DIR%\Config\client\components
UNIX: /etc/opt/omni/client/components
d. Output of script created according chapter “Application Server users issues” ->
“General procedure to list Application Server users in Data Protector IDB”
e. Certificate files
Win: %DP_SDATA_DIR%\Config\server\certificates
UNIX: /etc/opt/omni/server/certificates
f. JRE cacerts file
Win: "%DP_HOME_DIR%\jre\lib\security\cacerts"
UNIX: /opt/omni/jre/lib/security/cacerts
IDENTIFICATION
Application Server Certificates issues
1. Check if server.log contains only service registration errors (eventually service registration is not
successful, e.g.: it is expected that initial registration will fail):

WARN [com.hp.im.jce.serviceregistry.ServiceRegistrationHandler] (Timer-6) Failed to register service

ServiceDescription [id=null, name=dp-loginprovider, version=1.1.0, properties=null,
url=https://hostname.domain.net:7116/dp-loginprovider/restws, expiresAt=null]:
java.lang.NullPointerException

WARN [com.hp.im.jce.serviceregistry.ServiceRegistrationHandler] (Timer-6) Failed to register service

ServiceDescription [id=null, name=dp-loginprovider, version=1.1.0, properties=null,
url=https://hostname.domain.net:7116/dp-loginprovider/restws, expiresAt=null]: IOException

2. Check if server.log contains any error like:

Caused by: javax.net.ssl.SSLException: Certificate for <hostname> doesn't match any of the subject
alternative names: [hostname.domain.com].

07:34:37,417 ERROR [org.keycloak.adapters.BearerTokenRequestAuthenticator] (default task-25) Failed to

verify token: org.keycloak.common.VerificationException: Invalid token issuer. Expected
'https://hostname:7116/auth/realms/DataProtector', but was
'https://hostname.domain.com:7116/auth/realms/DataProtector'

3. If errors listed under point 1. or 2. were found, then you can also verify that all applicable host
names are listed among DNSName entries within server certificate:
a. Check for keystore password in webservice.properties file
UNIX:
cat /etc/opt/omni/client/components/webservice.properties
WIN:
type "%DP_DATA_DIR%\Config\client\components\webservice.properties"

b. List DNSName entries for server certificate as:

UNIX:
/opt/omni/jre/bin/keytool -v -list -keystore /etc/opt/omni/server/certificates/server/server.keystore -
storepass store_password | grep -i DNS
WIN:
"%DP_HOME_DIR%\jre\bin\keytool.exe" -v -list -keystore
"%DP_DATA_DIR%\Config\Server\certificates\server\server.keystore" -storepass store_password | findstr -i
DNS

4. If errors under 1. are found verify also that “cacerts” file is in sync with ca.truststore:
a. Get “store_password” as described in 3.a

b. List content of ca.truststore (list trusted CA certificates – normally only one should be imported into
ca.truststore)
WIN:
"%DP_HOME_DIR%\jre\bin\keytool.exe" -v -list -keystore
"%DP_DATA_DIR%\Config\Server\certificates\server\ca.truststore" -storepass store_password
UNIX:
/opt/omni/jre/bin/keytool -v -list -keystore /etc/opt/omni/server/certificates/server/ca.truststore -
storepass store_password

c. List content of cacerts file:

WIN:
"%DP_HOME_DIR%\jre\bin\keytool.exe" -v -list -keystore "%DP_HOME_DIR%\jre\lib\security\cacerts" -storepass
changeit
UNIX:
/opt/omni/jre/bin/keytool -v -list -keystore /opt/omni/jre/lib/security/cacerts" -storepass changeit

d. Verify that ca certificate MD5 finger print (search for MD5 checksum content in output of command run
with b.) exist in java “cacerts” file (i.e.: was also imported to “cacerts” file)
Example of finger print to search in b. output:
MD5: 47:02:83:7F:68:7C:5E:7E:89:57:F3:84:00:AF:39:40
If MD5 fingerprint cannot be found in “cacerts” file, this would require certificates regeneration (or more
specifically only the steps to import ca certificate to “cacerts” file can be executed).

5. If errors under 1. can be seen and certificates were regenerated with no_ca omnigencert.pl option, then
verify client.keystore Owner name matches the value in in dp-webservice-roles.properties and jce-
webservice-roles.properties files
a. Get “store_password” as described in 3.a
b. List content of client.keystore
WIN:
"%DP_HOME_DIR%\jre\bin\keytool.exe" -v -list -keystore "%DP_DATA_DIR%\Config\Server\certificates\server\
client.keystore " -storepass store_password
UNIX:
/opt/omni/jre/bin/keytool -v -list -keystore /etc/opt/omni/server/certificates/client/client.keystore -
storepass store_password

c. Verify that client certificate Owner value matches the value written in dp-webservice-roles.properties,
jce-webservice-roles.properties files, e.g.:

client.keystore output
Alias name: cn=dpinst-17 administrator, o=micro focus, st=md, c=us
Creation date: May 22, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=dpinst-17 Administrator, O=MICRO FOCUS, ST=MD, C=US
Issuer: CN=CA dpinst-17, O=MICRO FOCUS, ST=MD, C=US

dp-webservice-roles.properties content
CN\=dpinst-17\ Administrator,\ O\=MICRO\ FOCUS,\ ST\=MD,\ C\=US=ServiceUser

Value must match on case-sensitive level, if names do not match, then update .properties files, restart
Application server and verify if this resolves the issue.

Execute steeps described under Resolution chapter “Hostname inconsistently resolved” when:
1. Any error under 2. is found (check also conditions for certificate regeneration)

If any of the following is true then execute “Certificate Regeneration” found under Resolution chapter:
1. If any error listed under 1. and not all host names were added to DNSName list verified by 3.
2. If any error listed under 2. and not all host names were added to DNSName list verified by 3.
3. If JRE “cacerts” file is out of synch with ca.truststore
Hostname is not resolving uniformly
See previous chapter under point 2. how to possibly identify these issues.

To resolve the issues with host names resolution, follow the steeps described under Resolution chapter
“Hostname inconsistently resolved”. You can also use steps described there, to verify that Data Protector
uses same host name value (either short name or FQDN name; or same FQDN of the same short hostname) across
all Data Protector configuration files/entries listed there.

Application Server (Keycloak) users issues

General procedure to list Application Server users in Data Protector IDB
1. Create SQL script file with the following content and save it in a file, for example scriptname.sql.
select * from dp_config_host;
select * from dp_keycloak_tokens;
select * from dp_config_host_credentials;

2. Execute the created SQL script:

“omnidbutil -run_script scriptname.sql –detail”

Application Server master users verification

Execute the general procedure described previously and verify:
a. dp_config_host_credentials table must contain entries for “username” fields: “master_admin” and
“DpKeycloakUser”
When these users cannot be found follow the steeps described under Resolution chapter “Resetting
Application Server users credentials”.

Application Server master_admin user credentials does not match credentials in Data Protector IDB
Execute the general procedure described previously:
1. Note down the password value for master_admin user, e.g:
uuid | seq_id | host_uuid | host_seq_id |
username | password
--------------------------------------+--------+--------------------------------------+-------------+------
-----------------------------------------+--------------------------------------------------------------
2aa0ee31-625f-4158-bbe2-f886b27c15a4 | 1105 | 2aa0ee31-625f-4158-bbe2-f886b27c15a4 | 1104 |
master_admin | QFBBCGBBFEBBKGBBLGBBDEBBNHBBIIBBJFBBLFBBLHBBQHBBPFBB
2. Execute util_cmd –decode EncodePassword, e.g:
util_cmd –decode QFBBCGBBFEBBKGBBLGBBDEBBNHBBIIBBJFBBLFBBLHBBQHBBPFBB

3. Open web browser and navigate to https://CellManagerHostName:7116/auth/admin and enter “master_admin”

and “DecodedPassword” from step 2

If login was unsuccessful follow the steps described under Resolution chapter “Resetting Application Server
users credentials”.

Application Server dp_config_host verification

Execute the general procedure described previously and verify:
a. “name” field in dp_config_host table must be set and must contain valid host name of the Cell Manager
system (as used across Data Protector configuration).
Important: matching system host name against the one in dp_config_host is case sensitive, which means if
not matching exactly could result in users query issues, which could reflect as:
- GUI/CLI authentication errors
- General example would be “omniuser –list” does not list any users

If there is hostname mismatching, then follow the steps described under Resolution chapter “Hostname
inconsistently resolved” (in some cases it could be enough to update the host name in the dp_config_host to
correct value).

Data Protector Users were not migrated to Application Server database

Proceed with steps down when:
a. During installation user migration was reported as failed
b. “General procedure to list Application Server users in Data Protector IDB” shows no Data Protector user
or some are missing
c. “omniuser –list” does not list any users

Scenario under c. could indicate also that DpKeyCloak user credentials are incorrect, to verify this
additionally check that no other known errors are found, execute “omniuser –list” with debug option and
check DBSM log created for error like:
[ -2] getKeycloaktoken: Invalid input parameter
In case such error is reported you can skip steps a. and b. in the procedure down.
First check the chapter “Identification” and execute corresponding resolution steps if any applicable:
a. Certificate regeneration
b. Hostname inconsistently resolved
c. Resetting Application Server users credentials

After all general issues were resolved follow the steps described under Resolution chapter “Re-Migrate Data
Protector Users and re-migrate schedules”.
RESOLUTION

Hostname inconsistently resolved

Short system hostname is consistent, however in configuration sometimes FQDN equivalent is being used or
vice versa

Example:
‘myhostname’ vs. ‘myhostname.mydoimain.com

This indicates that that Cell Manager host name resolving is not uniformly configured (e.g.: full Netbios
host name does not corresponds to the resolved FQDN of the host).

1. Resolve the system configuration inconsistencies in the host name resolution first and proceed with
updating Data Protector configuration with using the same host name value (based on updated system
configuration) in:

UNIX:
/etc/opt/omni/client/components – make change in all webservice.properties including in subfolders
/etc/opt/omni/client/cell_server
/etc/opt/omni/server/cell/cell_info
/etc/opt/omni/server/AppServer/standalone.xml

Win:
C:\ProgramData\OmniBack\Config\client\components – make change in all webservice.properties including in
subfolders
C:\ProgramData\OmniBack\Config\Server\AppServer\standalone.xml
C:\ProgramData\OmniBack\Config\Server\cell\cell_info
HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett-Packard\OpenView\OmniBackII\Site - CellServer

2. Execute:
omnidbutil –config_cell_name CellManagerName

3. Restart Data Protector Application Server and proceed with any further steps needed.
All other inconsistencies in hostname resolution (multihoming environment, different FQDN resolution for
same host)

Examples:
‘myhostname1’ vs. ‘myhostname2’
‘myhostname.mydomain1.com’ vs. ‘myhostname.mydomain2.com’

To resolve such inconsistencies in host name resolving, steps for changing Cell Manager name should be
followed as described in:
https://docs.microfocus.com/itom/Data_Protector:10.03/Install/install_landing/Appendix_B:_System_preparatio
n_and_maintenance_tasks#ChangetheCellManagername.

After these changes restart Data Protector Application Server and proceed with any further steps.

Certificate regeneration
To resolve certificates issue follow procedure described in:
https://docs.microfocus.com/itom/Data_Protector:10.03/Admin/Installation/Description/DES_Regenerate_Certifi
cates.

After certificates regeneration, verify that Server Application logs contain no new known certificates
errors.

Resetting Application Server users credentials

Reset password for user “master_admin” and “DpKeycloakUser” on Keycloak server and change password for this
user in IDB. Could also require re-migrating all Data Protector users to the AppServer database.

1. Create keycloak master user in order to access Keycloak server and change “master_admin” and
“DpKeycloakUser” user password.

2. Execute
Win:
“C:\Program Files\OmniBack\AppServer\bin\add-user-keycloak.bat -r master -u username -p password”
The result should be:
“Added 'master_dpuser' to 'C:\ProgramData\OmniBack\Config\Server\AppServer\keycloak-add-user.json', restart
server to load user”

UNIX:
/opt/omni/AppServer/bin/add-user-keycloak.sh --sc /etc/opt/omni/server/AppServer -r master -u username -
p password
The result should be:
“Added 'master_dpuser' to '/etc/opt/omni/server/AppServer/keycloak-add-user.json', restart server to load
user”

username and password can be selected, for example master_dpuser / PassMaster

3. Restart only Data Protector Application Server service (hpdp-as)

3. Open web browser and navigate to https://CellManagerHostName:7116/auth/admin and enter “username” and
“password” from step 1.

4. In the upward left corner expand DataProtector and select Master realm. Then change the password for the
user:
a. In left pane select “Users” and then press “View all users”.
b. In the list, click on the “ID” of user “master_admin”.
c. Open “Credentials” tab, where you can enter new password and set “Temporary” switch to OFF.
Press “Reset Password” and confirm new password by pressing “Change password” button to confirm password
change.

5. Now in left-up corner where “Master” sign is, select “DataProtector” realm.
a. Repeat the steps 4.a-c to change password for user “dpkeycloakuser” in “DataProtector” realm.
Note: The password must meet the following requirements:
- Must be 8-20 characters long
- Includes at least one upper case letter
- Includes at least one of these special character: an asterisk ( * ), a dot ( . ), an hyphen ( - ), or an
underscore ( _ )
- Includes at least one numeral
- Does not include spaces
b. Delete all DP users except “dpkeycloakuser”

Note: If Data Protector GUI was previously working and at some time started to show error message “Unable
to get Authentication token.”, then you can skip this step (5.b) and steps 6.a
6. Create SQL script file with the following content and save it in a file, for example scriptname.sql.
a. If step 5.b was executed next SQL script content must be used:
delete from dp_keycloak_tokens;
delete from dp_config_host_credentials;
delete from dp_config_host;

Note: This will delete any existing Data Protector users already populated in Application Server database.
They will be re-migrated from userlist file with step 11, however any custom password configured will be
lost. In case steps 5.b was skipped use script content under 6.b

b. If step 5.b was skipped then use next SQL script:

delete from dp_keycloak_tokens;
delete from dp_config_host_credentials where username = ‘DpKeycloakUser’ OR username = ‘master_admin’;
delete from dp_config_host;

Note: Executing last SQL statement can report errors like down, these error can be ignored.
psql:deletekc2.sql:6: ERROR: update or delete on table "dp_config_host" violates foreign key constraint
"dp_host_credentails_fk" on table "dp_config_hos
t_credentials"
DETAIL: Key (uuid, seq_id)=(2aa0ee31-625f-4158-bbe2-f886b27c15a4, 1104) is still referenced from table
"dp_config_host_credentials".

7. Execute the created SQL script:

“omnidbutil -run_script scriptname.sql –detail”

8. Go back to CLI and run following command

“omnidbutil -add_restuser_cred -server CellManagerHostName -user master_admin -pass password”
“password” is the password as set in step 4.c

“omnidbutil -add_restuser_cred -server CellManagerHostName -user DpKeycloakUser -pass password”

“password” is the password as set in step 5.c

If command goes through, you will get “DONE”

9. Verify user credentials were successfully added according steps described in chapter “Identification”
under “Application Server users entries verification”
10. Restart Data Protector Application Server

Re-migrate Data Protector users and re-migrate schedules

Edit userlist file
Win:
C:\ProgramData\OmniBack\Config\Server\users\userlist
UNIX:
/etc/opt/omni/server/users/UserList
Change end flag “Yes” to “No” for users where this flag exist and is set to “Yes”.

Execute command
Win:
"C:\Program Files\OmniBack\bin\perl.exe" "C:\Program Files\OmniBack\bin\userMigrate.pl"
UNIX:
/opt/omni/bin/perl /opt/omni/sbin/userMigrate.pl

If needed (failed during installation) execute migration of legacy schedules executing:

“omnidbutil –migrate_schedules”
“omnidbutil –optimize_schedules”