StorNext - cvadmin tricks
Cvadmin, a powerful command line tool to query the running file systems on the MDC.
Common cvadmin usage: cvadmin (no arguments)
If run on a client, and it shows the file systems available, it means that the metadata connection is working correctly. On an HA system, you can determine which MDC is the primary by which entries have an asterisk (*) next to it.
StorNext Administrator Enter command(s) For command help, enter "help" or "?". List FSS File System Services (* indicates service is in control of FS): 1>*data_vol1[0] located on mdc1:32892 (pid 17650) 2>*data_vol2[0] located on mdc1:32900 (pid 17649) 3> data_vol1[1] located on mdc2:32825 (pid 9623) 4> data_vol2[1] located on mdc2:32826 (pid 9624) Select FSM "none"
Inside cvadmin use select to view info about a file system
cvadmin> select data_vol1
This shows an overview of the file system. Type who to see which clients are connected to that file system, how long they have been connected, and information on licensing
snadmin (data_vol1) > who Who (File System "data_vol1") # SNFS I.D. Type Location Up Time License Expires -- ---------- ---- --------------- --------- --------------- 0> FSM mdc1 12d 2h 10m N/A 1> CLI client_01-san 12d 2h 10m N/A 3> 151762DD74 CLI mdc-1 1d 1h 56m Permanent 4> ADM mdc1 0d 0h 0m N/A 10> CLI 192.168.10.110 12d 2h 13m Permanent 12> 151762DD74 CLI 192.168.10.121 6d 0h 42m Permanent
Type who long to get a more detailed output on the file system status and it’s stripegroup statistics.
Index : 10 Type : Client Location : 192.168.10.110 Up Time : 12d 2h 13m License : I.D. 15177A3B98 permanent Latency : 37037us measured on 02-17-2014 21:39:15 Revision : 4.3.2 build 30118 for Linux 2.6.32-71.el6.x86_64 x86_64 Build Date: Fri Jan 4 02:04:48 MST 2013 Build Dir : /home/mlund/nightly/VM-0-RedHat60AS-26x86-64-SP0/sn/buildinfo Index : 12 Type : Client Location : 192.168.10.121 Up Time : 6d 0h 45m License : I.D. 15177A3B98 permanent Latency : 77us measured on 02-17-2014 21:37:08 Revision : 4.3.2 build 30118 for Windows v7.0 AMD64 Build Date: Fri Jan 4 02:08:57 MST 2013 Build Dir : C:/sbx/sn/buildinfo Reconnects: 6 times in the last 8 days
This is a quite helpful option that can help to quick check the right IP assignments and versions (not provided for Xsan).
cvadmin -e 'select data_vol1;who long' | grep -e Location -e Revision Location : mdc-1 (192.168.10.2) Location : 192.168.20.110 (192.168.10.110 -> 192.168.1.2) Revision : 4.3.2 build 30118 for Linux 2.6.32-71.el6.x86_64 x86_64 Location : 192.168.20.121 (192.168.10.121 -> 192.168.1.2) Location : mdc-1 (192.168.10.2) (192.168.10.2 -> 192.168.10.2) Revision : 4.3.2 build 30118 for Linux 2.6.32-71.el6.x86_64 x86_64
Another useful option that can help troubleshooting performance issues is the latency-test. While there are many Linux tools as netcap, iperf, ttcp and so on, cvadmin has a build in tool to measure the latency to a client.
Inside the cvadmin tool run a latency test against a single client or all attached clients. It is based on the client index number that you can see via who, not the IP address of that client.
snadmin (data_vol1) > latency-test 10 Test started on client 1o (192.168.20.110)...latency 1582us
snadmin (data_vol1) > latency-test all Test started on client 10 (192.168.20.110)...latency 142857us Test started on client 12 (192.168.20.121)...latency 216us
As you can see, the repeated test result for client 10 is quite different. This may happen as the client is busy with some other activities. I have seen clients spiking with those latency numbers and I would recommend to test client by client rather than do a broadcast like latency test.
If you still need more information why this latency number is high, you can gather more details about the client running stats IPADDR (not stat) inside cvadmin. Note, this time the IP is required and it won’t accept the client ID like in the example before.
snadmin (data_vol1) > stats 192.168.10.110 File paths shown are relative to the mount point for data_vol1
Top files in bytes read/written: read bytes write bytes read count write count Relative File Path 1529444352 0 14834 0 scanity/05.03.2014/picture/project/reel04/2048x1488/0370453.dpx 410200064 0 1684 0 scanity/05.03.2014/picture/project/reel04/2048x1488/0370454.dpx
Cvadmin is the tool used to stop, start, and activate file systems. Inside cvadmin use the command “start data_vol1” to start that file system on the local machine. If that file system is already running and active on another MDC (fail over /HA mode), then the local MDC will start it in secondary (standby) mode.
If you use “stop data_vol1” on an HA system, it will stop both the primary and secondary instances of it. You can be more specific about which to stop by specifying a file system with a number, such as “stop 1“, where 1 is a file system in a list.
The internal commands of cvadmin can be used on the regular linux command line by using the -e and -F argument. This becomes useful for scripting or other non-interactive activities. For example, to see who is connected to file system data_vol1 without entering the cvadmin prompt, use the following.
cvadmin -e "select; who -F data_vol1"
This will output the list of connections the same as if you typed who inside cvadmin.
To see a list of the current file systems like you would upon first entering the cvadmin prompt, use the select command without specifying a file system.
cvadmin -e select