Profiling with frametest or another (but deeper) how-to

Topic

For those who don’t know frametest for testing frame-based performance, I created this guide to run a test against the storage via the file system. Frametest hasn’t seen any update in 15+ years but is still a powerful tool that is free. Yes, we have AJA and BlackMagic Disktest for our UI users. But if you, like me, like to automate and graph the output from this kind of applications, this how-to may be helpful.

Situation

Let’s assume you have a client attached to your storage (NAS or SAN or even DAS) and get merely mediocre performance when using Adobe Premiere or Resolve. To look a little deeper, a tool that can write or read some sequence-based DPX /EXR sized files came in handy.

Ready, set …

Download frametest from below. This zip file contains versions for Windows, Linux and macOS (Intel) and a small how-to text. 

Unfortunately, this tool won’t run on all Linux distributions. Debian buster and newer, for instance, won’t comply which might also be true for Ubuntu 18+. There are usually the 32bit libs missing. Running yum install glibc.i686 will solve the problem on RHEL based distros in most cases.

To get the help file, run frametest without any argument and all the options will be shown to you.

… GO!

The goal is to simulate a set-up using an application like Resolve or similar.

So let’s have some fun and create 3000 4K files, each 50MB (Note: the largest frame size supported is ~66MB) and let them run as slow as we can. Meaning, with a single thread and no buffer. Frametest will write into the folder you defined at the end of the command line.

frametest –w 4k –n 3000 –t 1  /mnt/server-xyz/TEST_1

Of course, in real life, the application will use more than 1 thread, but the purpose of this initial test is to give you an idea of how clean the connection to the server is. If you see results above 1300MB/s, you will be able to play a 4K sequence from the storage. If the throughput comes back below that number, you will have to increase your threads and add some buffer. We can also add a frame drop option which will alert you when the buffer is exhausted and your application will drop a frame.

Try this (or something along those lines):

frametest –w 4k –n 3000 –q 10 –b 5 –f 24 –t 4  /mnt/server-xyz/TEST_2

This will write 3000 4K files again, but using 4 threads this time. In addition, we added a buffer (-q) of 10 frames which we prefilled with 5 frames (-b) and told frametest to report if our framerate (-f) goes below 24 fps. In addition, we are writing into a clean folder, as a rewrite is more time consuming than a generic write.

Trying to push just as much over the wire as possible, remove the –f option from the example above, and also fiddle with the number of threads until you find the perfect setting for your needs.

If 4K, 50MB per frame is not what you want to test and none of the predefined options (sd”, “hd”, “2k”, “4k”) suit you, use the –z to tell frametest in KB what you need.

frametest –w 1 -z 40000 –n 3000 –t 4  /mnt/server-xyz/TEST_3

Another option which is often overlooked, especially in SAN environments, is the –e option. This will create empty frames and test for the open/close time which basically represents a metadata or IOPS test to see how many frames one could write or read.

frametest -e –n 3000 -t 4  /mnt/server-xyz/TEST_4

Reading the content back is done with the option –r. In general, the same options can be applied to run frametest as a single threaded test, multithreaded and in a frame drop simulation.

frametest –r 4k –n 3000 –q 10 –b 5 –f 24 –t 4  /mnt/server-xyz/TEST_1

Frametest creates files with the name “frame[counter].tst”.
So how can you use frametest to read back your existing DPX or EXR sequence? Use the –name option. Assuming your sequence is named “shot-172.[counter].exr”, use the name of the first frame as start frame.

frametest –r 4k –n 3000 –t 4 --name=shot-172.000001.exr /mnt/server-xyz/DPX-12

Want to have more information because you want to automate or graph the test? Use the –x option to write out a CSV compatible file which results in:

frametest –r 4k –n 3000 –t 4 -x read-out.csv TEST_2

Bonus

Can you use frametest for streaming media test? Yes you can do that. For example we read a file back and simulate a read of 24MB/s

frametest -r -s -z 1024 -t 1 -n 1440 -f 24 -q 8 /mnt/server-xyz/big_testfile

Options

-w size  Perfrom write test with 'size' KB frames or "sd", "hd", "2k", "4k"
    -e       Perform write test with empty frames (open/close test)
-r       Perform read test using existing frames (default)
-s       Perform streaming test to/from a single file
-n num   Number of frames to read or write (default = 1800)
-a num   Use async I/O with 'num' overlapped operations (default is sync)
-t num   Use multithreading I/O, with 'num' threads at one time
-o       Handle completed I/Os out-of-order (not used in read and sync IO)
-f rate  Limit speed to 'rate' frames per second
-q num   Maximum 'num' of queued buffers before a frame is dropped
-b num   Pre-buffer 'num' frames before starting "playback" (default = 5)
-v       Create/access files in reverse order
-m       Create/access files in a random pattern
-l num   Number of frames to loop (default = 0: don't loop)
-i num   Number of I/Os per frame (default = 1, ignored in async mode)
-z size  Specifies a custom 'size' for read or write test in KB
-d num   Use directories, with 'num' files per directory
-p secs  Pause 'secs' before starting test
-c perc  Crop mode, read middle 'perc'% of frame (ignored in write test)
-u secs  Display update interval in seconds (default = 1s)
-h secs  Histogram displays last 'secs' worth of info (default = all)
-x file  Export detailed timing to a file (comma separated), append '.csv'
  
--header=size   Header 'size' in KB for non-stream mode (default = 64)
--name=name     Use 'name' as the filename pattern specifying the first file
--noinfo        Don't display version and copyright information
--prealloc=num  In streaming mode (-s), preallocate 'num' frames of space. In non-streaming mode, the 'num' is ignored
--verbose       Verbose/debugging mode 

Another handy tool to profile is vidio where you can find a how-to here.