[SOLVED] I need more memory for work

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
Hi !

Sometimes each frame of image/video file takes 300-400 megabites , to see the results it needs caching.
One second of such video takes 10gigs of ram in cache , which makes impossible to cache 30second video and watch it
to make work comfortable in such environment through RAM, it needs almost 512 gigs of RAM, which is complete overkill.

what if buy fastest nvme, let's say rocket 4gen sarbent with 5000mb/s sequential read speed (hope read frame by frame from drive pretty much means "sequential read" right ?
and set paging file of any size on 1TB drive like that. I know it's still not close to be DDR4 RAM , but at least way better than just render out and watch every time I need to see result.
what do you think ? will caching be fast with above solution ?
 
Last edited:
Solution
right, that exact situation of putting cache into pagefile.
And you can't control it like that.
They are two different things, controlled by different entities.

I have a SATA III drive dedicated to cache files, scratch space, libraries for various applications.
Lightroom, Hitfilm Express, Paintshop Pro, Rhino3D, etc.

Use of the pagefile is controlled by the OS.

They can live on the same drive.
But they don't work together.

Ralston18

Titan
Moderator
First: update your post to include full system hardware specs and OS.

Second: what application software are you using? Most software provides some listing of hardware requirements in the form of minimum, recommended, and best.

You do not want minimum and you do want as much best as you can afford.

Third: whatever hardware is needed, the motherboard must support that hardware configuration (including RAM) and the PSU must be capable of providing the necessary power (wattage) to all the component during times of peak demand.

More information needed.
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
What are you talking about ha ? I was just asking about possible solution not particular one.
Question was about substituting DDR4 RAM with PCI gen 4 nvme.
If we are talking about fastest nvme 4gen it automatically means that system is new and bandwidths are high
and that means win should be new and 64bit , it also means CPU is not 10 years old. what PSU has to do with pagefile .. )) ok good PSU , good one, which will be enough to supply average new system 800-1000W.
 

USAFRet

Titan
Moderator
What are you talking about ha ? I was just asking about possible solution not particular one.
Question was about substituting DDR4 RAM with PCI gen 4 nvme.
If we are talking about fastest nvme 4gen it automatically means that system is new and bandwidths are high
and that means win should be new and 64bit , it also means CPU is not 10 years old. what PSU has to do with pagefile .. )) ok good PSU , good one, which will be enough to supply average new system 800-1000W.
An NVMe is much much much slower than actual RAM.

A 1TB pagefile? No.
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
An NVMe is much much much slower than actual RAM.

A 1TB pagefile? No.

no , not 1TB pagefile), it would be overkill also, just 128-256gigs let's say.
I just dont see any other way , it would be at least better than nothing , because 256-512 fast ram will cost as 2 new average workstation pcs, of course it will have performance penalty but again think it will be way better than just render them out every time.
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
I don't have that possible worstation, I'm just asking if anyone have experience with setting pagefile on fast nvme while caching some video data into it. sata interface will do nothing on regular ssds , I need something fast
and someone who can share experience of working with such configuration.

PS
512gigs of fast ram costs almost 5k$ compared to 300-500$ of possible PCI4 or PCI5 nvme.
PCI5 is around , which will double the bandwidth afaik, maybe that will do the trick.
 

kanewolf

Titan
Moderator
Hi !

Sometimes each frame of image/video file takes 300-400 megabites , to see the results it needs caching.
One second of such video takes 10gigs of ram in cache , which makes impossible to cache 30second video and watch it
to make work comfortable in such environment through RAM, it needs almost 512 gigs of RAM, which is complete overkill.

what if buy fastest nvme, let's say rocket 4gen sarbent with 5000mb/s sequential read speed (hope read frame by frame from drive pretty much means "sequential read" right ?
and set paging file of any size on 1TB drive like that. I know it's still not close to be DDR4 RAM , but at least way better than just render out and watch every time I need to see result.
what do you think ? will caching be fast with above solution ?
I would like to know what is generating this amount of data. 8K Red Monstro cameras generate less than 350MB/s
Is this a scientific sensor of some kind?
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
I wonder if your render is not configured optimally. I just don't hear anybody else having issues like this.
But, I don't do graphic design for a living.

" wonder if your render is not configured optimally." no comments )

by the way there is no such thing like render configured optimally when related to filesize , that might be related to noise, yes, but not for filesize, size directly depends on bit depth and resolution at first place.
You don't hear because as you said you don't do graphic designs.

In other words you can't configure renderer so it becomes less in filesize if your bit depth and resolution needs to be fixed, files will differ in size little bit, but predictably anyway , so I'm taking maximum filesizes I've experinced.

don't compare renderoutput to camera output , it's different.
exr have it's own compression, without compression it goes above 1gig per frame.

some frames of deep exr goes even above 500mb compressed. but if you will make 100% raw exr it will go above sky. it's severe format but only one that gives great control over grading and other kinds of post work.

so ,question is opened , anyone with video editing experience and pagefile on nvme ?
 

kanewolf

Titan
Moderator
You don't hear because as you said you don't do graphic designs.
And I admitted as much. BUT, have have been an engineer in high performance computing for 35 year. And I know that having a preconceived answer to a problem can be narrow focused. Examining the PROBLEM rather than the proposed solution, is how you get smart people to help you.
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
Afraid, best way to help with such question is to share experience of working with that exact situation
when one edits video and sends cache into m.2 interface. All other will be just speculations on how it might work.
But anyway THANK YOU !! ))
 

kanewolf

Titan
Moderator
Afraid, best way to help with such question is to share experience of working with that exact situation
when one edits video and sends cache into m.2 interface. All other will be just speculations on how it might work.
But anyway THANK YOU !! ))
Have you posted this question on the support forum for your render software or for nuke? That should be where the experts are.
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
Have you posted this question on the support forum for your render software or for nuke? That should be where the experts are.
I did that on discord channels but unfortunately as for today they're working solely either with less sized files or compressed videofiles. Also I don't have access to write directly big studios corporate emails with request of sharing such info. Plus , It's something that is not related to nuke in particular.
Moreover blazing fast nvmes I've mentioned above more likely will be accessible to communities here rather than there, as for today.
 

USAFRet

Titan
Moderator
32GB physical RAM and 256GB "pagefile" is inferior to 64GB RAM and 1GB pagefile, IF the system and application is using more than 32GB RAM but less than 64GB RAM.

The pagefile is only used in extremis, when the system runs out of physical RAM.


Cache file? Yes, put that on whatever the fastest drive in the system is.
In my personal use with Adobe Lightroom, I see zero perf difference if the LR cache file is on a SATA III SSD vs on the Intel 660p NVMe.
Yes, I've tested.
Obviously, that is not the same as your vid editing and rendering. But...
 

kanewolf

Titan
Moderator
I did that on discord channels but unfortunately as for today they're working solely either with less sized files or compressed videofiles. Also I don't have access to write directly big studios corporate emails with request of sharing such info. Plus , It's something that is not related to nuke in particular.
Moreover blazing fast nvmes I've mentioned above more likely will be accessible to communities here rather than there, as for today.
You are still focusing on a solution that you are looking for validation. You haven't provided any description of the rest of the system the NVMe would be part of. You could be asking about a Sarbent NVMe and have a motherboard with an X2 NVMe interface.
But if the "expert" forums haven't helped and you believe that similar specific experience is required, you may not find it here.
 

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
USAFRet
" that is not the same as your vid editing and rendering. But... "
it's not same, yes.

kanewolf , "You haven't provided any description of the rest of the system the NVMe would be part of. "

it's not necessary, you want me to build virtual PC in my post , in same time you know very well that you will focus only on its actually involved parts anyway , which is nvme and ddr4 ram (no matter how much it will be).

asus motherboard with PCI4 gen , did that helped you ?
PSU that will feed enough power to system , and 8-12 core modern CPU.

"But if the "expert" forums haven't helped and you believe that similar specific experience is required, you may not find it here. "

not similar specific experience , just experience with putting video cache on nvme, it's broader.

I posted question couple of hours ago, and you're sending me somewhere to other forums, I never said I need answers now or today, sometimes I'm getting most valuable information after weeks after I post them here.
 
Last edited:

peaceduke

Honorable
Dec 18, 2014
230
0
10,690
That is exactly what the initial post read like.
" I know it's still not close to be DDR4 RAM , but at least way better than just render out and watch every time I need to see result. "
still exactly like that ?
Yes I am. Because that is where YOU drove the conversation. "Afraid, best way to help with such question is to share experience of working with that exact situation "
right, that exact situation of putting cache into pagefile.
 

kanewolf

Titan
Moderator
" I know it's still not close to be DDR4 RAM , but at least way better than just render out and watch every time I need to see result. "
still exactly like that ?

right, that exact situation of putting cache into pagefile.
Cache is managed by an application.
Pagefile is manged by the operating system. What is in the pagefile is not under application control.
You can put cache on an NVMe
You can put pagefile on an NVMe.
But you can't control what is in a pagefile at the application layer.