Question Sequential write testing setup

[GVM2014]

Hi, I want to run some sequential write tests on some of my SSDs to find out the size of their cache and how much their speed drops when the cache is full. I have several questions on this.

1. From what I've understood, Iometer is very reliable, but it is a bit confusing to set up. Can someone give a link to a tutorial or post some instructions how to set up Iometer to do a sequential write (both for the full drive and for a part of the drive - namely, in the last case, how to simply hammer a SSD with several hundred GBs of data, without writing the full disk; for instance if you want to test write 500 GB only on a 2 TB drive)?

Yes, I am aware that a full write test is data-destructive, so I'll make sure to run it on empty drives. Also, I am aware that, technically, such tests will shorten the lifespan of the drive by a bit, but the TBW of modern SSDs is so high, that this is not a concern (the typical 400-600 TBW of a 1 TB SSD is so high that I am unlikely to use it all up during its lifespan).

2. I would also like to clarify something about CrystalDiskMark, which is one of the most popular benchmarking tools. It can use test files of maximum 64 GB and run 9 passes. Does that mean that, when someone selects the size of the test file to be 64 GB and to run for 9 passes, the sequential read/write phase of the CrystalDiskMark test will continuously read/write a total of 576 GB of data on a drive?

3. In the past, I have used AIDA64 Extreme Disk Benchmark Linear Write and HD Tune Pro File Benchmark for this purpose, but I came to doubt their reliability. Is that correct? Also, I want to confirm something: in HD Tune Pro File Benchmark you can select a size for a test file: does this mean, if you select, say, 200 GB (an example), it will sequentially write and read 200 GB of data on the drive?

Thanks in advance.

 

[Maxxify]

I answered you very briefly on AnandTech and will do so again here. I wouldn't rely on AIDA64 or HD Tune Pro. CrystalDiskMark uses Microsoft's DiskSpd which you can download and use the cmdline to run tests manually.

diskspd /?

For options/switches.

diskspd -b1M -w100 -c1G -o16 -Sh write_test.dat > write_test.txt

Would write with a 1MB block size, 100% write workload, to a 1GB file with 16 threads and direct I/O, parsing the output to a text file. So you can produce any type of workload you want without relying on CDM. This is NOT destructive (except to the output file).

Many sites use FIO instead of IOMeter, this includes AnandTech, LTT, PCPartPicker, etc. This will work on Windows with ioengine=windowsaio. Job writetest.fio:

ioengine=windowsaio //use Windows I/O engine
rw=write //100% writes
direct=1 //Bypass caching
bs=128K //128KB block size
size=100G //100GB file size
iodepth=32 //QD of 32
write_bw_log_seqwrite //log or output name
log_avg_msec=500 //interval to measure
filename=slc.test //working file name

fio writetest.fio

You can import from the log into excel as CSV and then you can manipulate it. I tested with my 2xSN750 RAID and this had high speeds through the static SLC cache (~12GB per drive, so 24GB) then direct-to-TLC of over 3GB/s (which is accurate). You can graph from this data, etc. Check FIO's examples folder to learn.

IOMeter: Sean Webster (was the SSD reviewer here) could probably help you, he's on here, my discord server (moderator), Reddit. I have used his scripts but the setup is similar to FIO.

 

[GVM2014]

I answered you very briefly on AnandTech and will do so again here. I wouldn't rely on AIDA64 or HD Tune Pro. CrystalDiskMark uses Microsoft's DiskSpd which you can download and use the cmdline to run tests manually.

diskspd /?

For options/switches.

diskspd -b1M -w100 -c1G -o16 -Sh write_test.dat > write_test.txt

Would write with a 1MB block size, 100% write workload, to a 1GB file with 16 threads and direct I/O, parsing the output to a text file. So you can produce any type of workload you want without relying on CDM. This is NOT destructive (except to the output file).

Many sites use FIO instead of IOMeter, this includes AnandTech, LTT, PCPartPicker, etc. This will work on Windows with ioengine=windowsaio. Job writetest.fio:

ioengine=windowsaio //use Windows I/O engine
rw=write //100% writes
direct=1 //Bypass caching
bs=128K //128KB block size
size=100G //100GB file size
iodepth=32 //QD of 32
write_bw_log_seqwrite //log or output name
log_avg_msec=500 //interval to measure
filename=slc.test //working file name

fio writetest.fio

You can import from the log into excel as CSV and then you can manipulate it. I tested with my 2xSN750 RAID and this had high speeds through the static SLC cache (~12GB per drive, so 24GB) then direct-to-TLC of over 3GB/s (which is accurate). You can graph from this data, etc. Check FIO's examples folder to learn.

IOMeter: Sean Webster (was the SSD reviewer here) could probably help you, he's on here, my discord server (moderator), Reddit. I have used his scripts but the setup is similar to FIO.

Well, I posted the question here because I know tomshardware uses Iometer.

As for the rest of your post, I appreciate you are trying to help, but what I was hoping for was a software like CrystalDiskMark, where you select an option and click a button. Microsoft's DiskSpd and FIO can be useful tools, but they seem waaaaay to complicated for an average user like me: I honestly have no clue what those lines of commands which you posted actually mean or even how to install this stuff. There is no point in DiskSpd and FIO being reliable if they literally require programming knowledge to use them.
For instance, searching on Google for Microsoft's DiskSpd took me here: https://learn.microsoft.com/en-us/a...-overview#quick-start-install-and-run-diskspd. However, these "instructions" could have just as well been written in Chinese (and, btw, it also says DISKSPD has a difficult time emulating complex workloads....)
Iometer, on the other hand, seems more intuitive, albeit not entirely.

I am quite surprised, though, that no one else answered this question. Does nobody besides Sean Webster know how to use Iometer?
The lack of other suggestions is equally surprising. Does this mean that no software for these kind of tests, more user-friendly than the likes of FIO, Diskspd or even Iometer, actually exist?

PS: how about question 2 from the thread opener?

 

[Maxxify]

Well, I posted the question here because I know tomshardware uses Iometer.

As for the rest of your post, I appreciate you are trying to help, but what I was hoping for was a software like CrystalDiskMark, where you select an option and click a button. Microsoft's DiskSpd and FIO can be useful tools, but they seem waaaaay to complicated for an average user like me: I honestly have no clue what those lines of commands which you posted actually mean or even how to install this stuff. There is no point in DiskSpd and FIO being reliable if they literally require programming knowledge to use them.
For instance, searching on Google for Microsoft's DiskSpd took me here: https://learn.microsoft.com/en-us/a...-overview#quick-start-install-and-run-diskspd. However, these "instructions" could have just as well been written in Chinese (and, btw, it also says DISKSPD has a difficult time emulating complex workloads....)
Iometer, on the other hand, seems more intuitive, albeit not entirely.

I am quite surprised, though, that no one else answered this question. Does nobody besides Sean Webster know how to use Iometer?
The lack of other suggestions is equally surprising. Does this mean that no software for these kind of tests, more user-friendly than the likes of FIO, Diskspd or even Iometer, actually exist?

PS: how about question 2 from the thread opener?

CDM is literally a GUI front-end for DiskSpd. It's true IOMeter has a basic GUI, but you would probably struggle with that, too. IOMeter itself isn't crazy complex but if you can't handle basic FIO/diskspd lines then the icm's won't make sense to you. There's a wrapper for FIO called ezFIO but that would probably be too much, too. HD Tune can be semi-accurate if you set it up right. It needs to be set for maximum Accurate (rather than Fast) with a 128 KB block size. Don't use short stroke (do full). However, this models at QD1 AFAIK.

 

[MWink64]

If you have no data on the drive and want to go real quick and dirty, you could use the write test on HDDScan or Victoria. Both can produce line charts with the transfer rate. If using Victoria, make sure to maximize the window before starting the test or else the chart will be smaller. The results may not be as precise as something like IOMeter but it's way simpler to do. BTW, you have to first remove all partitions from the drive or else you'll get errors (this only applies to the write test).

Don't quote me on this but I'm pretty sure the answer to your CrystalDiskMark question is no. It will write 64GB once, before starting the tests. The subsequent reads/writes seem to depend on the speed of the drive. Imagine my surprise when testing my first NVMe drive with the default settings (1GB, 5 pass) resulted in over 200GB of both reads and writes. I'm pretty sure the inverse is also true because it can take longer to write the test file than it does to run the tests on really slow drives.

By the way, I'm starting to sour on CrystalDiskMark. I'm finding it often gets results that are much rosier than real world performance, especially when testing forms of flash-based media. I don't know how it works but I wonder if it's displaying peak performance, rather than averages. A lot of cheap drives have "sawtooth" writes, constantly bouncing between high and very low speeds. If the transfer rate constantly and evenly bounces between 10 and 100MB/s, it shouldn't get a score of 100MB/s, it should get a score of 55MB/s.

 

[GVM2014]

If you have no data on the drive and want to go real quick and dirty, you could use the write test on HDDScan or Victoria. Both can produce line charts with the transfer rate. If using Victoria, make sure to maximize the window before starting the test or else the chart will be smaller. The results may not be as precise as something like IOMeter but it's way simpler to do. BTW, you have to first remove all partitions from the drive or else you'll get errors (this only applies to the write test).

Don't quote me on this but I'm pretty sure the answer to your CrystalDiskMark question is no. It will write 64GB once, before starting the tests. The subsequent reads/writes seem to depend on the speed of the drive. Imagine my surprise when testing my first NVMe drive with the default settings (1GB, 5 pass) resulted in over 200GB of both reads and writes. I'm pretty sure the inverse is also true because it can take longer to write the test file than it does to run the tests on really slow drives.

By the way, I'm starting to sour on CrystalDiskMark. I'm finding it often gets results that are much rosier than real world performance, especially when testing forms of flash-based media. I don't know how it works but I wonder if it's displaying peak performance, rather than averages. A lot of cheap drives have "sawtooth" writes, constantly bouncing between high and very low speeds. If the transfer rate constantly and evenly bounces between 10 and 100MB/s, it shouldn't get a score of 100MB/s, it should get a score of 55MB/s.

I thought it was well-known that CrystalDiskMark's results are the maximum speeds of the drive, not the averages.

I would like to give it a try to Iometer, because the interface seems pretty easy to use. I got the basic principles and I figured out, roughly, how to run a test, but, for better accuracy, I wish to find some tutorial/manual who would explain what some of the commands in the software represent them, because there are some commands that are not clear to me. I mean things like: "Use Fixed Seed", "Test Connection Rate" (from the "disk targets" section), "Transfer Request Size". "Reply Size", "Align I/Os on" (from Edit Access Specification"), "Cycling Options", "Number of Workers to Spawn Automatically" (in "Test Setup". Some options are clear what they mean, but others are not. Does no such manual actually exist?

 

[Maxxify]

Some options are clear what they mean, but others are not. Does no such manual actually exist?

There are many testing and preconditioning guides out there, and it's possible I don't list them in my resources (I have shared them with reviewers, though). I may have forgotten. In any case, they are largely for more serious testing, but in regards to your needs they might help you get started here.

Start at page 6 here to see what I mean. Those settings - 128KB block, 100% sequential, 100% write, 4KB align - is a typical sustained write workload. QD32 (outstanding IO) is as well. For SNIA they do 2xCapacity based on steady state average write speed. Nowadays, a larger block size like 1MB might work better.

 

[GVM2014]

There are many testing and preconditioning guides out there, and it's possible I don't list them in my resources (I have shared them with reviewers, though). I may have forgotten. In any case, they are largely for more serious testing, but in regards to your needs they might help you get started here.

Start at page 6 here to see what I mean. Those settings - 128KB block, 100% sequential, 100% write, 4KB align - is a typical sustained write workload. QD32 (outstanding IO) is as well. For SNIA they do 2xCapacity based on steady state average write speed. Nowadays, a larger block size like 1MB might work better.

Thanks for reply, this is what I was looking for. Overall, I figured out how to use Iometer and what the majority of the commands are for.

But, since this guide does not include any info on the "disk targets", perhaps you could provide me with some input about some of its features: "Maximum Disk Size" and "Starting Disk Sector" is clear what they are (determining the size of the test area and its starting point), an so is "Write IO Data Pattern", but I have some questions about the other 3 features.

1. What should be the value for "# of Outstanding I/O" per target? I've read in a guide it should be set to 16 or 32, but I've not identified when that guide was posted, so the info could be outdated. Is the best value for this feature still 16 or 32? Should it vary according to CPU performance, perhaps (depending on number of cores/threads)?

2. What is "Used Fixed Seed" and "Test Connection Rate"?

Also, I have some questions regarding Tomshardware's ssd reviews:

3. The reviews, in the section dedicated to sequential write results, specify how fast will the cache recover once the drive goes to idle. For instance (just an example, from the SN570 1 TB): "It absorbed roughly 12-13GB of data at 3.2 GBps before performance degraded. Once the cache was saturated, write speeds averaged 615 MBps for the remainder of the test. Thanks to the SN570’s small cache capacity, it recovers nearly immediately during each idle round." How do the reviewers know exactly how fast does the cache recover? (Iometer shows the speed, but I haven't noticed any feature which would show cache recovery time). It is just an estimation or do the reviews measure it somehow?

4. When a review says "the cache recovers during idle round", does this mean the drive had to stop writing or does it also have to stop reading for the cache to recover?

5. How do they produce all those graphs displaying the results? In the client I've downloaded, the results are shown as numbers only (either in the "Results display" tab or in the txt file. Is there something I miss?

 

[MWink64]

5. How do they produce all those graphs displaying the results? In the client I've downloaded, the results are shown as numbers only (either in the "Results display" tab or in the txt file. Is there something I miss?

I'm actually trying to do the same thing as you. I too have been playing with IOMeter but am not confident enough to try and answer your other questions. I do think I've figured out the graphing part. I don't think IOMeter itself is creating the graphs. I made some by importing the .CSV file produced by IOMeter into LibreOffice Calc (Excel should also work) and using it to create the graphs. I was able to make ones very similar to the ones you see in the reviews.

BTW, I've only done some preliminary tests but was shocked to see how small the pSLC cache appears to be on some drives. A couple of 1TB TLC drives only absorbed about 7GB and 20GB, before dropping to ~60MB/s and ~90MB/s respectively. This was from a virtually empty state. Interestingly, an even cheaper, smaller QLC drive was able to absorb far more. It will be interesting to see the results, when I'm able to do more extensive testing.

 

[Maxxify]

Thanks for reply, this is what I was looking for. Overall, I figured out how to use Iometer and what the majority of the commands are for.

When you see numbers reported on data sheets and in marketing it's usually sequential at QD32 (32 outstanding I/O). Seed has to do with the RNG (random number generator), you may have seen this term with other things like Minecraft maps, using the time stamp is common. Connection Rate has to do with the workers opening/closing disks. This stuff is handled in the IOMeter manual. Tom's Hardware (Sean) determines recovery with idle periods in-between sequential runs. This is somewhat predictable depending on the cache type, size, and flash type. Tom's uses Excel to graph the output data. (I've sent Sean this thread but he is not so interested)

 

[Maxxify]

I'm actually trying to do the same thing as you. I too have been playing with IOMeter but am not confident enough to try and answer your other questions. I do think I've figured out the graphing part. I don't think IOMeter itself is creating the graphs. I made some by importing the .CSV file produced by IOMeter into LibreOffice Calc (Excel should also work) and using it to create the graphs. I was able to make ones very similar to the ones you see in the reviews.

BTW, I've only done some preliminary tests but was shocked to see how small the pSLC cache appears to be on some drives. A couple of 1TB TLC drives only absorbed about 7GB and 20GB, before dropping to ~60MB/s and ~90MB/s respectively. This was from a virtually empty state. Interestingly, an even cheaper, smaller QLC drive was able to absorb far more. It will be interesting to see the results, when I'm able to do more extensive testing.

Yep, CSV is a common output for data, Excel (or equivalent) is typically used for graphing. Sometimes you have to parse or convert the data, for example when using PowerShell Objects (PSO). Static SLC will always be small because it has to come from flash outside of the user area (as it is dedicated) so a 1TB drive will usually have about 12GB (plenty of examples: SN570, SN750, Intel's QLC series). This equates to 36GB or 48GB (TLC or QLC) of the OP. Drives more often have hybrid with some dynamic SLC nowadays, also applies to the Intel QLC series and P41 Plus although these have unique algorithms. Cheaper QLC drives usually have full-drive SLC caching or caches that are 1/4 the user available size, but this means far worse steady state performance.

There's a lot more I can say about all this technically because static SLC can be used in various ways but ultimately taking flash away from OP is a trade-off. Even the size of static can vary over a drive's lifespan as you require more ECC with more wear. But you get the idea.