Question RS3621xs+ write performance issue after restart

[AndersNyman]

Hi!

After a fresh install of our Synology RS3621xs+ with 12xHAT5300 16TB the performence is good.
Read: ~760MB/s
Write: ~1GB/s
Looks like this:

But after restarting the server write performance drops and vary much more. Not very satisfying.

• We also have a Synology RS3617xs+ on the network that works good.
• Cables have been switched
• We make use of the two 10GbE ports with Linked Aggregation.
• The volume is almost empty.
• Minimum SMB2, max SMB3.
• Raid10
• DSM 6.2.4-25556

It's the second time we do a factory reset and get good performance after the installation is complete, but after a restart performance drops.

Anyone has a clue why write performance drops? And suggestions on how to do further troubleshooting are more than welcome!

Best,
Anders

 

[kanewolf]

Hi!

After a fresh install of our Synology RS3621xs+ with 12xHAT5300 16TB the performence is good.
Read: ~760MB/s
Write: ~1GB/s
Looks like this:

But after restarting the server write performance drops and vary much more. Not very satisfying.

• We also have a Synology RS3617xs+ on the network that works good.
• Cables have been switched
• We make use of the two 10GbE ports with Linked Aggregation.
• The volume is almost empty.
• Minimum SMB2, max SMB3.
• Raid10
• DSM 6.2.4-25556

It's the second time we do a factory reset and get good performance after the installation is complete, but after a restart performance drops.

Anyone has a clue why write performance drops? And suggestions on how to do further troubleshooting are more than welcome!

Best,
Anders

How long after a reboot of the Synology? It may still be doing background tasks after reboot. What if you reboot and wait 20 min ?
It could be a link aggregation problem. If you remove one 10GE do you get consistent results ?

 

[AndersNyman]

How long after a reboot of the Synology? It may still be doing background tasks after reboot. What if you reboot and wait 20 min ?
It could be a link aggregation problem. If you remove one 10GE do you get consistent results ?

Yes, have been waiting for hours between benchmarks.
Yes, have done tests with only one 10GbE connection.

 

[kanewolf]

Yes, have been waiting for hours between benchmarks.
Yes, have done tests with only one 10GbE connection.

"Yes" isn't an easy to interpret answer.
Do you mean that there is no change from the behavior you showed in your original post in both cases ??

 

[AndersNyman]

"Yes" isn't an easy to interpret answer.
Do you mean that there is no change from the behavior you showed in your original post in both cases ??

We have found that some changes have temporarily improved performance, but after a restart it's back to slow speeds.
So i dont think the changes themselves have anything to do with the problem. For example, after changing the default gateway we get back to normal speed. Turning off LinkedAggrigation have improved the speed. But after a restart of the server we are back to slow speeds again. Very random...
Hard to troubleshoot.

Another speed test i did right now. 😬

Will try to add a external network card.

 

[kanewolf]

We have found that some changes have temporarily improved performance, but after a restart it's back to slow speeds.
So i dont think the changes themselves have anything to do with the problem. For example, after changing the default gateway we get back to normal speed. Turning off LinkedAggrigation have improved the speed. But after a restart of the server we are back to slow speeds again. Very random...
Hard to troubleshoot.

Another speed test i did right now. 😬

Will try to add a external network card.

A network card in what? The Synology? That is a bad idea, IMO.
What tool are you using for this performance test ?

 

[AndersNyman]

I'm using ATTO Disk Benchmark 4.

I gave it a try and added another 10GbE network card (Synology E10G21-F2 Network Adapter 2-Port SFP+).

• Started up and did a bound of the two 10GbE ports.
• Ran benchmark that im happy with...

• Restarted the RackStation
• New benchmark:

• Restarted again and did wait 10 minutes
• New benchmark:

Conclusion:
Nothing wrong with the built in network card. Same behavior as before. The results are very random and not satisfying.

 

[kanewolf]

Have you tried an alternate benchmark? Intel NAS test for example -- https://www.intel.com/content/www/us/en/products/docs/storage/nas-performance-toolkit.html

What layout are you using for the drives? RAID6 with all 12? RAID6 with 8+2 and 2 hot spares ?

 

[AndersNyman]

Have you tried an alternate benchmark? Intel NAS test for example -- https://www.intel.com/content/www/us/en/products/docs/storage/nas-performance-toolkit.html

What layout are you using for the drives? RAID6 with all 12? RAID6 with 8+2 and 2 hot spares ?

About alternative benchmark, We have been copying 5GB files from an to a windows workstation and get same results. But i will try black magics benchmark software as well.

We are using RAID10. Have also tried to switch to RAID5 but it gave the same result so we are back on RAID10.

I'm in contact with the synology support at the moment.

 

[roawar2]

Please let us know if you have any success with Synology support. We are also seeing strange performance issues on the 10GB ports with RS3621xs+ and 12x HAT5300 8TB drives.

Write performance on 10GB port is sometime slower than the 1GB port which is defintely not correct. I have also tried changing cables etc. In our case we are using a single 10GB port and no link aggregation.

After reading your post with similar sounding issues I am now suspecting a firmware software bug of some sort.

 

[AndersNyman]

Please let us know if you have any success with Synology support. We are also seeing strange performance issues on the 10GB ports with RS3621xs+ and 12x HAT5300 8TB drives.

Write performance on 10GB port is sometime slower than the 1GB port which is defintely not correct. I have also tried changing cables etc. In our case we are using a single 10GB port and no link aggregation.

After reading your post with similar sounding issues I am no suspecting a firmware software bug of some sort.

Interesting, thanks for letting me know you have similar issues.

The Synology support are very helpful, they are investigating. No conclusion yet.

We found a way of reproducing this behaviour.

1. Press the reset button on the back of NAS (hold for 4 seconds until beep).
2. Write speed is good. About 1GB/s
3. Restart NAS
4. Write speed back to ~100MB/s

I have done this procedure 4 times now. Every time i restart the nas write speed drops.
To me it feels like its software related to NIC firmware / drivers.

 

[AndersNyman]

We had some further progress during the weekend. And we have normal writing speed now, even after a restart of the NAS.

I believe the problem had to do with the switch the NAS was connected to. And it altered the problem down to the NAS even if we connected it straight to a computer.
After restarting the switch and a reset of the NAS everything is working good.