Question Samsung 990 PRO - 4TByte Periodic Performance drop every 3Gbyte when doing larger continues write

[eyalgutman67]

Hi all
i am trying to build a recorder system base on FPGA board that connect via PCI Express Gen4 x 4 . I need to do around 400Gbyte write on around 4GByte/sec. I selceted the Samasung 990 Pro since it have cache of 420Gbyte. Its almost work perfect but each 3GByte we have large stall that can not work in our system. I don't get answer from Samsung. Any idea from the experts?
Thanks
Eyal

 

[COLGeek]

What are the specs for the system this SSD is being used with (make and model of all installed components)?

 

[eyalgutman67]

What are the specs for the system this SSD is being used with (make and model of all installed components)?

Hi , this is an internal system that get external stream of samples and using NVME IP that is design over PCI Exp (and all off that is on FPGA (agelix7) that is connect directly to the Samsung NVME. Since we are monitoring the low level interface we can see exactly when the NVME is doing the push back . i wonder is there some" garbage calculation" or turbowrite area that need to be extend?

 

[USAFRet]

Hi , this is an internal system that get external stream of samples and using NVME IP that is design over PCI Exp (and all off that is on FPGA (agelix7) that is connect directly to the Samsung NVME. Since we are monitoring the low level interface we can see exactly when the NVME is doing the push back . i wonder is there some" garbage calculation" or turbowrite area that need to be extend?

What is the composition of the files here?
A lot of tiny files may well exhibit this behavior.

 

[eyalgutman67]

What is the composition of the files here?
A lot of tiny files may well exhibit this behavior.

This is not a file, its one large stream of data that is inject (write) directly to the NVME. so you can look on that as one big file. There is no PC involve her since its "hardware system"

 

[COLGeek]

So the SSD is always reading and writing as the stream "flows". Right?

How hot is the SSD getting (if monitoring)? The repetition of the lag is surely tied to cache size and write speed.

As an experiment, can you slow the stream to see how it impacts the performance?

 

[USAFRet]

This is not a file, its one large stream of data that is inject (write) directly to the NVME. so you can look on that as one big file. There is no PC involve her since its "hardware system"

The DRAM cache is 1GB.

The SLC write cache is "approx. 442 GB (432 GB Dynamic + 10 GB Static)"
https://www.techpowerup.com/ssd-specs/samsung-990-pro-4-tb.d863

 

[eyalgutman67]

So the SSD is always reading and writing as the stream "flows". Right?

How hot is the SSD getting (if monitoring)? The repetition of the lag is surely tied to cache size and write speed.

As an experiment, can you slow the stream to see how it impacts the performance?

We are doing long write and after a while (lets assume 30 minutes) we are doing the read.. so we dont mix. Also this issue is only on the write (read work perfectly), We did slow down by half the write and it still "stall" for some period on the 3GByte write boundary. we try to keep the system cold using external fan, so i guess it some internal configuration but i am new to this martial ..thanks!!!

 

[eyalgutman67]

The DRAM cache is 1GB.

The SLC write cache is "approx. 442 GB (432 GB Dynamic + 10 GB Static)"
https://www.techpowerup.com/ssd-specs/samsung-990-pro-4-tb.d863

Exactly !!! so what can happen on 3GByte write boundary. I will share the measurements now

 

[USAFRet]

Exactly !!! so what can happen on 3GByte write boundary. I will share the measurements now

What application or drive is writing to the 990?

 

[eyalgutman67]

Hi all
i am trying to build a recorder system base on FPGA board that connect via PCI Express Gen4 x 4 . I need to do around 400Gbyte write on around 4GByte/sec. I selceted the Samasung 990 Pro since it have cache of 420Gbyte. Its almost work perfect but each 3GByte we have large stall that can not work in our system. I don't get answer from Samsung. Any idea from the experts?
Thanks
Eyal

What application or drive is writing to the 990?

This is HW apllication that get sampels from the "antena" and try to record them. So no PC is on the loop. the "driver" is also coded by HW and it come from some IP vendor. The idea is that we can not push back for long time the stream that come so any push back that come from the nvme that is bigger then some buffer size will mess the application

 

[eyalgutman67]

The idea was that till 400MByte and since we have sustained data rate of the 4GByte/sec we will not hit any wall ..

 

[USAFRet]

This is HW apllication that get sampels from the "antena" and try to record them. So no PC is on the loop. the "driver" is also coded by HW and it come from some IP vendor. The idea is that we can not push back for long time the stream that come so any push back that come from the nvme that is bigger then some buffer size will mess the application

I'm just thinking there is some limitation on the Source, leading to this 3GB blip.

Have you tried this with the 990 in a different M.2 port, or with a completely different drive?

 

[COLGeek]

I'm just thinking there is some limitation on the Source, leading to this 3GB blip.

Have you tried this with the 990 in a different M.2 port, or with a completely different drive?

Agreed. I, too, wonder if this issue is being caused on the "source" side.

 

[eyalgutman67]

Agreed. I, too, wonder if this issue is being caused on the "source" side.

The "HW Drive" is encrypted HW IP that i bought. It may happen in the source but the vendor that supply the driver said he don't have this number on his code ... i got new SSD from
WD BLACK SN850X that also have big cache. I will check and update the forum. Thank you and i will keep all updated.