Tandberg - Basic - Troubleshooting Video Quality

1. Verify Endpoint model and firmware
2. Verify Call Diagnostics/System Information:

Video:
Check:
- Video Protocol(codec)
- Resolution
- Frame Rate
- Call rate

Audio
Check:
- Audio protocol (codec)
- Audio rate
Check TX/RX Packet loss

Gathering call details:

1. Verify QoS settings and marking from Tandberg Menu

2. Verify IPLR is enabled:
Transmit: Enables/disables Intelligent Packetloss Recovery
usage: iplr

3. Verify call details from Telnet:
xstatus Call 1

*s Call 1 (status=Synced, type=Vtlph, protocol=H323, direction=Incoming, logTag=
9):
     CallRate: 1920
     RemoteNumber: "3006131"
     IncomingNumber: "13302220001"
     IncomingSubAddress: ""
     Mute: Off
     Microphone: On
     Duration: 303
     MuteOutgoing: Off
     CallOnHold: False
     RemoteSiteOnHold: False
     MultiwayProgress: Off
     Channels 1 (type=Incoming):
       Rate: 1920
       Restrict: Off
       IPLR: Off
       Encryption (status=Off): /
       Audio (status=Active):
         Protocol: G722
         Rate: 64
       Video 1 (status=Active):
         Protocol: H264
         Resolution: w720p
         Rate: 1856
       Video 2 (status=Inactive): /
       Data (status=Inactive): /
     Channels 2 (type=Outgoing):
       Rate: 1920
       Restrict: Off
       IPLR: On
       Encryption (status=Off): /
       Audio (status=Active):
         Protocol: G722
         Rate: 64
       Video 1 (status=Active):
         Protocol: H264
         Resolution: w720p
         Rate: 1856
       Video 2 (status=Inactive): /
       Data (status=Inactive): /
*s/end

OK

4. Verify real time statistics:

Enable syslog
syslog on
syslog 3

H323MCS-1 pkts  23703, loss      0, jitter   0, maxjitter   1, drop    0, rate  640 (Audio input)
H323MCS-1 pkts 108081, loss      0, jitter   1, maxjitter   4, drop    0, rate 17140 (Main Video input)
H323MCS-1 pkts      2, loss      0, jitter   0, maxjitter   0, drop    0, rate 0 (Data input)
H323MCS-1 pkts  23703, loss      0, jitter   0, maxjitter   0, drop    0, rate 640 (Audio output)
H323MCS-1 pkts  73652, loss      0, jitter   7, maxjitter   7, drop    0, rate 12690 (Main Video output)
H323MCS-1 pkts     19, loss      0, jitter   0, maxjitter   0, drop    0, rate 0 (Data output)

5.  Enable PacketLossDownSpeed:
xConfiguration PacketlossDownSpeed Mode:

6. Collect sniffer captures



* some commands may vary depending Endpoint model

General - Replay video from packet capture

1) Download videosnarf
Download the tar file:
videosnarf-0.63.tar.gz
http://sourceforge.net/projects/ucsniff/files/

2) Build and Compile VideoSnarf
Extract file above
cd videosnarf-(version)
./configure
make
make install

3) Download FFmpeg
Download the tar file
ffmpeg-0.6.tar.gz
http://ffmpeg.org/download.html

4) Build and Compile FFmpeg
Extract file above
cd ffmpeg-(version)
./configure
make
make install

5) Obtain packet capture using Wireshark and then filter it by source IP address and UDP port

Once you have filtered packet capture, Save As this filtered packet capture as new file containing only the stream you want to replay (Use Save As and then Selected Displayed packets)

6) Use Videosnarf to detect RTP streams in the packet capture
videosnarf -i filename.pcap -c
Will generate H264-media-1.264 file

7) Use FFMPEG to convert H264 file to AVI video
ffmpeg -i H264-media-1.264 myvideo.avi

8) Play your video.avi in your favorite video player

CTS System - Device names and numbers

TelePresence systems contains different components in the system board to process Video signals, when you go through the logs you will find reference to them, here is the device name and number used:

Video Encoder
Device 0 - Main Camera
Device 2 - PC Video Input (VGA)
Device 4 - Document Camera

At any time only one either Device 2 or Device 4 are active, never both.

Video Decoder
Device 1 - Plasma
Device 3 - Projector Display

Scope ID 15: Primary Video Encoder
Scope ID 16: Primary Video Decoder

Thanks to Shankar  for sharing this 

CTS System - SNMP Decode dateandTime

We have a value in our MIB for notifying when an specific call started:

Object ctpcStartDateAndTime

"This object specifies the value of local date and time when a call is started.
 
http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en&translate=Translate&typeName=DateAndTime

This Object has variable type as "DateAndTime". DateAndTime is a standard Textual Convention which is defined in the SNMPv2-TC. DateAndTime is resolved to base data type OCTET STRING. The DISPLAY-HINT format for DataAndTime is given as follows.

DISPLAY-HINT "2d-1d-1d,1d:1d:1d.1d,1a1d:1d"

The date-time specification is as follows.


"A date-time specification.

field octets contents range
----- ------ -------- -----
1 1-2 year 0..65536
2 3 month 1..12
3 4 day 1..31
4 5 hour 0..23
5 6 minutes 0..59
6 7 seconds 0..60
(use 60 for leap-second)
7 8 deci-seconds 0..9
8 9 direction from UTC '+' / '-'
9 10 hours from UTC 0..11
10 11 minutes from UTC 0..59

For example, Tuesday May 26, 1992 at 1:30:15 PM EDT would be
displayed as:

1992-5-26,13:30:15.0,-4:0

Note that if only local time is known, then timezone information (fields 8-10) is not present."

Valid Usage

SnmpVar snmpvar = syntax.CreateVariable("'07:D2:09:03:0C:14:20:03:2B:07:00'");//hex format, length 11 bytes SnmpVar snmpvar = syntax.CreateVariable("'07:D2:09:03:0C:14:20:03'");//hex format, length 8 bytes SnmpVar snmpvar = syntax.CreateVariable("2002-9-21,13:53:32.3,-7:0");//string format, length 11 bytes SnmpVar snmpvar = syntax.CreateVariable("2002-9-21,13:53:32.3");//string format, length 8 byte

Example:

07 DA 08 09 0E 13 31 00 

07DA = 2010

08   = August

09   = 9th

0E   = 14

13   = 13 minutes

31   = 31 seconds

00   = deciseconds

The value of year is in network-byte order.

CTS system - Basic Video quality troubleshooting

High level description

---

1. Obtain an accurate problem description:
  • Display flickering, snowy
  • Pinkish,greenish,blueish
  • Black Display during loopback as well as during the call
  • Half image, zoom in
  • Other....

Obtain pictures or video when problem is happening

2. Problem details:
• New install, upgrade?
• Reproducible, random
• Start of the call, end of the call, after display is idle?
• Point to point calls?, Multi-point calls?
• Do we have a video or a picture when problem occurs.
• Problem only with one type of unit? Interop ?
• Problem appears after firmware upgrade? Any network changes? 
• Any message during the failure (i.e Network congestion etc.)
• Verify type of Display and generation (CTS 500 GEN2 37", CTS 1000 GEN2 65", etc)
• Obtain app code and boot code:
  - App code, boot code and firmware must be the same on all three displays

Isolating the problem

---

• We can try first a loopback with encoder enabled, this will confirm codec itself is not dropping packets

From SSH CLI:
diag display loopback full enable

• If ICMP is enabled across the path we can use mtr to see which device may be introducing delay, packet loss, etc. 

 utils network mtr count 100 tos  160

Replace 160 for your current QoS marking:

http://mytelepresence.blogspot.com/2010/04/telepresence-system-mtr-utility.html

• Verify Packets sent/receive show proper QoS value and proper policy maps are matched

• Network devices:

Verify SRND guide to make sure we meet network requirements:

http://www.cisco.com/en/US/docs/solutions/Enterprise/Video/TP-Book.html

1) ‘show run’ output

In addition, I would suggest collecting the following information to further isolate network traffic:

1)  Interf type x/y <- WAN interface
2) ‘load-interval 30’
3) Reproduce problem and keep call up for perhaps 5 mins and then collect
the following information while the call is still up and while packet loss is being seen:

a) ‘show interf intftype x/y’
b) ‘show policy-map interf intftype x/y ’
c) ‘show queue intftype x/y’

The above should allow us to (a) determine what our guaranteed bandwidth really is (b) determine where we are seeing drops (ex: interface drops vs software queuing drops).