EtherAssure

Training

System
Administration
Training
Session 4 System Administration Training

Servers and application architecture

Servers and Probes troubleshooting
CLI commands
Log files
Issue/bug escalation process

(Backup and system restore)

(North Bound Interface maintenance)

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 2

EtherAssure / NetComplete

EtherAssure is a versatile Ethernet test solution comprised of several combined Hardware

and software applications

NetComplete system
Element Management System (EMS)
Ethernet Test and Troubleshooting Solution - NetAnalyst NGT
Ethernet Performance Monitoring (PM)
ESAM (JMEP mediator software)
QT600/vQT probes
JMEP smart SFP, MA100
Packet Portal system
System Manager
PRE (IV mediator software)
PDG
LivePM

MTS3800 / HST3000

eNodeB reflector/initiator (not Viavi)

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 3

EtherAssure Deployment scenario (with Packet Portal
and JMEP)

See Viavi Low Level Description for Hardware deployment

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 4

 NetComplete Deployment Scenario

Hardware & Software Single or multiple Server Includes

Application Functions (AS)
Database Functions (DS)
Mediation Functions (MS)

See Viavi Low Level Description for Hardware scaling

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 5

NetComplete Third-party software options

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 6

NetComplete System Requirements Application TCP/IP
Port Requirements

Required TCP/IP Sockets (default):

Main default for NetComplete:
162 SNMP
80, 8080 HTTP (80 for Solaris, 8080 for Linux)
443 HTTPS (optional)
22 SSH
123 NTP
83, 8083 Intermediate Results Collection (NetMon tests)
25 SMTP forwarding

See Viavi Low Level Document for IP port to open

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 7

 NetComplete System Requirements Client Side Hardware
Requirements
Minimum Configuration

PC CPU - Pentium IV 2 GHz or higher

Memory - 1 GB or higher
Disk Storage - At least 40 MB free
Graphic Card - Support 1280 x 1024 resolution with 256 colors or more
Monitor
Keyboard and mouse required

Recommended Configuration
PC CPU Core 2 Duo 2.8 GHz or higher
Memory - 2 GB or higher
Disk Storage - At least 40 MB free
Graphic Card - Support 1280 x 1024 resolution with 256 colors or more
Monitor
Keyboard and mouse required

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 8

Java 1.8

Administrator needs to add the http://NetCOmpleteAS-IP:8080 in java Exception

from the Java configuration panel

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 9

Licensing
NetComplete Application Licensing

Mechanics of NetComplete application licensing

Ethernet T&T
(NGT)
Application Server

When an Active or
NetAnalyst NGT Passive test is
invoked, a single
Session License Session License
seed is used.

NetAnalyst NGT license location: /ada/netanalyst/Licenses

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 11

Walk-thru of upgrading existing NetAnalyst NGT License

NetAnalyst NGT License Upgrade Process

Step 1: Request all testers to terminate their client sessions

Step 2: Log in as [RUN_AS_USER] to the ETH T&T AS server
Step 3: At the command prompt, type: /ada/netanalyst/src/scripts/enterLicense
Step 4: Follow the prompts for supplying the session license number, product ID, customer ID, and date

Enter the full NGT license: xxxxxxnn|NANGT1000|ntmst|today

Step 5: Restart application processes: /ada/startup/npiu_init restart

Verify total number of available NetAnalyst NGT Session Licenses:

Step 1: Type: /ada/netanalyst/src/scripts/checkLicenseFile -v

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 12

JMEP licensing
JMEP licensing

Each JMEP has is own License

License is required when JMEP is added to the system

License is downloaded from Viavi Central site, requiring JMEP serial number and
initial Commercial order (to get access to the site)

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 14

Discover and license JMEP
if License is not found in the initial Input license File,
Select the JMEPs which you want to license and link to this ESAM
Click the Export Redemption File button to create the Redemption Input file to
send the license request to the Viavi Solutions Software Center

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 15

Discover and license JMEP
Log in to the Viavi Solutions Software Center at the following URL:
https://jdsu.subscribenet.com/control/jdsu/purchases
Select Order History and select your order.
In the Activation menu, select Redeem Discovery Keys.
Select Browse and choose the redemption input file that you created in an earlier
step.
Click Submit Redemption Input file

The Viavi Solutions Software Center will use the Redemption Input file to create
an Activation File for the specific JMEPs. The Activation File is also known as
the Redemption Output File.
You will receive an email with the Activation File attached. Save this Activation
File and save it on your local computer
2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 16
Redemption file example

<?xml version="1.0" encoding="UTF-8" ?>

<RedemptionInputFile>
<ProbeIds>
<ProbeId>A74AXF68413Z</ProbeId>
<ProbeId>B74A2F68413N</ProbeId>
<ProbeId>C74A2FV8413A</ProbeId>
</ProbeIds>
<SerialNumbers>
<SerialNumber>3132313059303744462020202020</SerialNumber>
<SerialNumber>31323130593037444E2020202020</SerialNumber>
<SerialNumber>3132313059303745452020202020</SerialNumber>
</SerialNumbers>
</RedemptionInputFile>

<?xml version="1.0" encoding="UTF-8" ?>

<RedemptionOutputFile>
<CustomerKey ProbeId="AE3D9B68E411" SerialNumber="3132313059303744462020202020"
DiscoveryKey= "ED8454F141AF8D0FA8188715C8E279D2" ManufacturerText="1000" />
Licenses Key: <CustomerKey ProbeId="B61ECE0B580C" SerialNumber="31323130593037444E2020202020"
DiscoveryKey= "BE7BB4868E36204493F2067562926CD2" ManufacturerText="1001" />
<CustomerKey ProbeId="8E2BCE6EDB0E" SerialNumber="3132313059303745452020202020"
DiscoveryKey= "3EFF64CA4E230D448362B2FDD331A975" ManufacturerText="1002" />
</RedemptionOutputFile>

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 17

NetComplete installation licenses

The available feature and measurement depends on the installation License keys.

The licenses are put in /ada/license/licenses.txt

Refer to Viavi Support for upgrading your software and your licences

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 18

Performing Basic Server
Operations
NetComplete Application Processes
Commands and Definitions
 NetComplete Application Startup/Shutdown Commands

use the user adatdce

On any NetComplete server :

/ada/startup npiu_init [restart] [stop] [start] [pstat] [swstat]
start/stop/restart : This will start/stop/restart the server software
pstat : list current process
swstat: list software versioning on this server

With multiple servers (AS/MS-DS or AS-MS-DS)

Stop in this order : MS - AS - DS
Start in this order : DS - AS - MS

(Optional) Execute this command to wait for the entire system to start :
/ada/startup/npiu_27_cws wait-started

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 20

NetComplete Server Processes

Application process scripts are executed from the adatdce/ada1tdce account.

Processes Script on each AS/MS/DS :
Command: /ada/startup/npiu_init pstat
Definition: Displays NetComplete processes

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 21

NetComplete Server Processes

Application scripts are executed from the adatdce account.

Software Package Version Script :
Command: /ada/startup/npiu_init swstat
Definition: Displays NetComplete software versions

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 22

NetComplete Server Processes

Application process checks are executed from the adatdce account.

Oracle processes
Command: ps aef | grep oracle
Definition: Displays Oracle processes on Server

Sample Output:

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 23

NetComplete Server Processes (Commands and Definitions)

Command: /ada/NTCdbsetup/bin/verify_sqlplus_connectivity a RDBMS_UID

Definition: Verify ability to connect to the DB

Sample Output:

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 24

Others restart

These commands would run on AS, If you encountering error on:

Login or Jboss : /ada/startup/npiu_27_cws restart

ESAM and JMEP configuration : /ada/startup/npiu_19_jgw restart

NGT : /ada/startup/npiu_35_nangt restart

Cognos (Reports) : /ada/startup/npiu_41_cognos restart

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 25

NetComplete Log Files
NetComplete Application Suite Log Files

Application Log Files Directory Specific Log Files of Interest

NGT /ada/log/netanalyst nangt_start_log.latest

nangt_stop_log.latest
NGTArchive_day*.tar.gz
NGT Database /ada/log/IS/webNMS hbn.txt, stderr.txt, nmserr.txt

CWS /ada/log/ACTRsvweb server.log, error_log

NTC-Portal /ada/log/ACTRsvweb access_log

Access log
NTC-Portal /ada/log/ACTRsvweb EMS.log

Flow config /ada/log/ACTRsvweb EVCConfig.log

Flows collection /ada/log/ master.YYYYMMDD

Note : some ERROR could be seen, but not affecting your system. Check with your Viavi
support

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 27

QT600
QT600 Resetting Lost Passwords

If you have lost the root password, use the steps in the QT_600 Installation
Guide around page 28 Resetting the root password. This has to be done
from the local console port on the QT.

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 29

QT600 Resetting Lost Passwords

If you have lost the admin or http passwords, use the steps in the QT_600
Installation Guide around page 29 Configuring the QT-600-10 security
settings. This must be done from the local console port on the QT.

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 30

Trouble Shooting Steps

From the QT600 Installation Guide

Note: I highly recommend taking the time

read the user manuals.

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 31

QT600 log files

Login as root to the QT ssh root@IP (pw = celogic)

Run : saveInfo.bash
Run tar zcvf saveInfoQTName.tar.gz /tmp/cap/debug

Download and send to Viavi the saveInfoQTName.tar.gz

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 32

Active QT Processes Page 1

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 33

Active QT Processes Page 2

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 34

Active QT Processes Page 3

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 35

QT600 Upgrade Trouble Shooting Steps

Kick off upgrade from Portal GUI

To check status of upgrade:
Login as root to the QT ssh root@IP (pw = celogic)
# tail f /opt/upgrade/Upgrade.log

upgrade.b 22387 root 255r REG 0,13 7775 73734 /tmp/upgrade.bash

1970.01.07-23:36:55-UTC Upgrading: downloading
Downloading http://24.30.213.98:80/distributions/qt600/QT600-BL0421-01/upgrade.tgz
Fail to download initial job file from http://24.30.213.98:80/distributions/qt600/QT600-
BL0421-01/upgrade
"Upgrade.log" line 1 of 40 --2%--
NOTE: - If you notice the date is incorrect, indicates that the NTP was not set
correctly on the QT.

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 36

Check QT600 Twamp synchronization
When using NTP or native linux PTP

chronyc sourcestats
root@QT600:~> chronyc sourcestats
210 Number of sources = 2
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
PTP0 0 0 0 +0.000 2000.000 +0ns 4000ms
10.129.181.124 9 5 137m -0.010 0.047 -323ns 74us
chronyc sources
root@QT600:~> chronyc sources
210 Number of sources = 2
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
#? PTP0 0 2 0 10y +0ns[ +0ns] +/- 0ns
^* 10.129.181.124 3 10 377 58 -52us[ -135us] +/- 8880us
chronyc tracking
root@QT600:~> chronyc tracking
Reference ID : 10.129.181.124 (10.129.181.124)
Stratum :4
Ref time (UTC) : Fri Nov 4 11:05:49 2016
System time : 0.000065516 seconds slow of NTP time
Last offset : -0.000082856 seconds
RMS offset : 0.000068960 seconds
Frequency : 47.219 ppm slow
Residual freq : -0.010 ppm
Skew : 0.025 ppm
Root delay : 0.006316 seconds
Root dispersion : 0.001578 seconds
Update interval : 1029.7 seconds
Leap status : Normal
2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 37
Check QT600 Twamp synchronization

When using external PTP Meinberg N2X

echo 1 > /proc/driver/qt600-gprmc
to turn debug on then
cat /proc/driver/qt600-gprmc
a couple of times to see if the count increments

cat /proc/qt600/qteth | grep -i sync

Should report slice1(or 2)TimeSyncStatus <> NA

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 38

NGT Backup Directory
NetAnalyst NGT Backup Directory

NetAnalyst NGT user adatdce uses a CRON job which runs at 1am which creates a backup tar file of the
following directories and stores the gunzip tarball file at /ada/NA_backups :

0 1 * * * /ada/netanalyst/src/scripts/NANGT_databackup -q -c 2>&1

This will backup :

-/ada/netanalyst
-/data
-Services.xml
-qt600_service_parameters.xml
-test_products
-User stored test scripts (baselines)
-Group stored test scripts (baselines)
-test_setup
-User stored test xml files

File system backup is the responsibility of each customer.

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 40

NetAnalyst NGT Backup setting

To change the Backup setting, performs:

1. Connect to the NetAnalyst NGT server as $RUN_AS_USER.

2. Type bash to open a bash shell.
3. Initialize the shell with the NetAnalyst NGT environment variables by typing
. /ada/netanalyst/src/scripts/setTM_env.sh
4. Change directories by typing cd $NANGT_BASE_DIR/src/scripts/customer
5. Create a file called 10backupsettings_env.sh.
6. Open the file you just created in a text editor
7. Add location of Backup : NANGT_BACKUP_TARGET=<valid path name>;export NANGT_BACKUP
TARGET
8. Add compress: NANGT_BACKUP_COMPRESS=FALSE;export NANGT_BACKUP_COMPRESS
9. Define number of saved backup:
NANGT_BACKUP_MAX_HELD=<integer 0-6>;export NANGT_BACKUP_MAX_HELD
10. Save the file
11. Restart the server : /ada/startup/npiu_init restart

0919 NetComplete 3.8 Administrator Training

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 41
 NetAnalyst NGT Restore Backup

To restore a Backup, perform:

1. Connect to the NetAnalyst NGT server as $RUN_AS_USER.
2. Initialize the shell with the NetAnalyst NGT environment variables:
. $BASEDIR/netanalyst/src/scripts/setTM_env.sh
3. Change directories : $NANGT_BASEDIR/src/scripts
4. Run the restore script by doing one of the following:
To restore a specific backup file: ./NANGT_datarestore </directory/><file_name>
where <file_name> and </directory/> is the file and directory location where the backup file
exists.
To restore backup files in a specific directory: ./NANGT_datarestore </directory/>
To restore a backup file from tape: ./NANGT_datarestore -t or ./NANGT_datarestore --tape
5. The restore is complete when Done appears on the console.

0919 NetComplete 3.8 Administrator Training

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 42
Oracle Flash Recovery Area
 Managing the Oracle Flash Recovery Area

To manage the Oracle flash recovery area for your NetComplete system, first review
the following sections and items in the Oracle Database Backup and Recovery User's
Guide, 11g Release 2 (11.2), Part Number E10642-01 provided by the Oracle
Corporation:
The Fast Recovery Area in Chapter 3
Overview of the Fast Recovery Area in Chapter 5, up through table 5-2 and also
How Oracle Manages Disk Space in the Fast Recovery Area

Database parameters for the flash recovery area

db_recovery_file_dest Specifies default location for the fast recovery area.

db_recovery_file_dest_size Specifies the hard limit on total space to be used by

database recovery files created in the fast recovery
area
To update parameters
ADA_HOME=/ada . /ada/NTCdb/bin/setdbenv
ORACLE_SID=dbsid sqlplus /nolog
connect / as sysdba
show parameter db_recovery_file_dest_size
alter system set db_recovery_file_dest_size=54g; # depend
exit
0919 NetComplete 3.8 Administrator Training
2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 44
 Report on the status of the flash recovery area on DS

/ada/NTCdb/bin/backup_db -log_dir /ada/log/backup_db -log_retention 14 -log_file_prefix

tdce_flash_usage_ -oracle_home /ada/NTCdb/oracle/oraserver_121010_64 -oracle_sid tdce /
flash_usage

Results logged to '/ada/log/backup_db/tdce_flash_usage_20141103_132814.log'.

0919 NetComplete 3.8 Administrator Training 2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 45
 Adjusting the size of the flash recovery area - on DS

The size of the flash recovery area can be adjusted by the database administrator by setting the value of
database parameter db_recovery_file_dest_size
(syntax: db_recovery_file_dest_size = integer [ K | M | G ]).
For example, the following commands will list the current value of db_recovery_file_dest_size and then
immediately change its value:
ORACLE_SID=dbsid sqlplus /nolog
connect / as sysdba
show parameter db_recovery_file_dest_size
alter system set db_recovery_file_dest_size=54g;
Exit

After changing the value in the database, change the value of the corresponding NetComplete
system parameter in $BASE_DIR/parms/NTCdbserver_parms.
On the EDS, using an editor, update system parameter NTC_DB_FLASH_SIZE. In this example, the value
would be set to 54

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 46

Database backup
Main Oracle file are located in Database Server, in these folder

Oracle Software Files /ada/oracle

Oracle Database /ada/db1/oradata
Oracle Archive Logs /ada/db1/archivelog
Oracle Backup Files /ada/db1/flash_recovery_area

Backup script could be ordered as Service to save and export to other server :

/ada/NTCdb/oracle : Done in maintenance mode

/ada/db1/flash_recovery_area : Daily
/ada/NTCdb/diag /ada/NTCdb/admin : Daily

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 47

Backup Oracle for Maintenance
To create a backup
Stop MS/AS/DS, using /ada/startup/npiu_ini stop
Log as adatdce on DS Server
If Linux OS :
tar zcvf BackupNTCdb_YYYYMMDD.tar.gz /ada/NTCdb/oracle
If SunOS :
gtar -zcvf BackupNTCdb_YYYYMMDD.tar.gz /ada/NTCdb/oracle

YYYYMMDD is the current date

Restart DS/AS/MS, using /ada/startup/npiu_ini start

To restore a backup:
Stop MS/AS/DS, using /ada/startup/npiu_ini stop
Log as adatdce on DS Server
If Linux OS :
tar zxvf BackupNTCdb_YYYYMMDD.tar.gz
If SunOS :
gtar -zxvf BackupNTCdb_YYYYMMDD.tar.gz
Restart DS/AS/MS, using /ada/startup/npiu_ini start
2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 48
Daily Backup of Oracle
To create a backup
Log as adatdce on DS Server
If Linux OS :
tar zcvf Backup_DiagAdminOracle_YYYYMMDD.tar.gz /ada/NTCdb/diag
/ada/NTCdb/admin
tar zcvf Backup_fraOracle_YYYYMMDD.tar.gz /ada/db1/flash_recovery_area

YYYYMMDD is the current day date

To restore a backup:
Stop MS/AS/DS, using /ada/startup/npiu_ini stop
Log as adatdce on DS Server
tar zxvf Backup_DiagAdminOracle_YYYYMMDD.tar.gz
tar zxvf Backup_fraOracle_YYYYMMDD.tar.gz
Restart DS/AS/MS, using /ada/startup/npiu_ini start

2015 Viavi Solutions, Inc. | Viavi Confidential and Proprietary Information 49