- Status
Thanks for this article. Just wanted to post an FYI that there will be an informal birds-of-a-feather session on renderfarming at this year's SIGGRAPH conference in New Orleans. The details about this and the other BoFs are here: http://www.siggraph.org/s2009/community/birds/index.php
[citation][nom]Anonymous[/nom]How about network booting the cluster - we have found it easier to manage upgrades/patches as you just need to reboot the nodes that need upgrading. Also makes each node a little cheaper and saves a fair bit on power consumption.Another idea we have been playing with is using cheap USB keyfobs either as system drives or to persist config data etc. - much faster boot times, very low power consumption and great MTTF.[/citation]
Mistoffeles
Distinguished
- Aug 9, 2006
- 6
- 0
- 18,510
We did this stuff back before the Pentium was released, much less GPU-based rendering. Never mind Maya farms and GPU rendering, we did it all on 486DXs running 3DS (pre-MAX) and made it more cost effective than purpose-build render farms by using the company's design and engineering workstations evenings and weekends. All just to produce cheap in-house walkthroughs of client sites before ground was even broken on the building (net cost of the work was 15-20% of the cost of farming it out to a contractor).
After I left that job, I went into the business myself for a while, but I stupidly went for an SGI server rather than a farm of low-end machines like we had back at the company. Expensive mistake, I barely broke even on it. Cheap consumer hardware is the way to go, and it really doesn't matter whether GPU rendering becomes an issue because you can just stick some video cards in your rendering machines and reconfigure.
Oh, and regarding the efficiency, putting 100 men to work for 1 hour each is only inefficient if you haven't got several more jobs lined up when that one is complete. I'm sure you can extend the analogy to a rendering farm. The company I worked for got great efficiency from using about 100 486s as compared to the 16 brand new Pentiums one potential contractor had bought to do nothing but build a render farm.
After I left that job, I went into the business myself for a while, but I stupidly went for an SGI server rather than a farm of low-end machines like we had back at the company. Expensive mistake, I barely broke even on it. Cheap consumer hardware is the way to go, and it really doesn't matter whether GPU rendering becomes an issue because you can just stick some video cards in your rendering machines and reconfigure.
Oh, and regarding the efficiency, putting 100 men to work for 1 hour each is only inefficient if you haven't got several more jobs lined up when that one is complete. I'm sure you can extend the analogy to a rendering farm. The company I worked for got great efficiency from using about 100 486s as compared to the 16 brand new Pentiums one potential contractor had bought to do nothing but build a render farm.
"For example, render times totaled 40 million hours for Monsters vs. Aliens, 30 million hours for Madagascar: Escape 2 Africa, and 6.6 million hours for Revenge of the Sith. "
It is not really quite true, and it is specially true with Madagascar :Madagascar is a cheap product, lousy fur, low poly everywhere and so on. Dreamworks did say it just for disincentive potential competitors.
Years ago, Disney Pixar's render farm was a 2000 cpu, or you could say, to render the movie will take 2.3 years (and a big electric bill) and it is not true.
It is not really quite true, and it is specially true with Madagascar :Madagascar is a cheap product, lousy fur, low poly everywhere and so on. Dreamworks did say it just for disincentive potential competitors.
Years ago, Disney Pixar's render farm was a 2000 cpu, or you could say, to render the movie will take 2.3 years (and a big electric bill) and it is not true.
As has been said earlier, when you are counting hours of render time you count the total, then divide them by the number of processors. Bringing up these total render times is the point, in an article on building a renderfarm/ render nodes. 2000 CPUs years ago was likely 2000 *nodes*,and these days 2000 *nodes* means 8000-16000 cpu cores. Note that ILM's render farm consists of 5700 cores, right *now*- which would mean a little over seven months rendering- not unreasonable for an all-CG feature film.
Revenge of the Sith was not all cg, and so took less time to render. Is that so hard to believe?
Revenge of the Sith was not all cg, and so took less time to render. Is that so hard to believe?
This guy is talking about old stuff these days. such a shame.
He should be chatting about CL + Cpu not give a waste of web space on just CPU render systems.
Sure he spoke a little about cpu+CL.. but.. 90% of this is a total waste.
The render industry is underoing a Huge re-think due to CL.
And my other Gripe..
What clueless person will talk about using MS on relyable/functional render farms... ohh yea this guy.. (clueless.) sorry I am not impressed.. catch up with what is realy going on out there.
He should be chatting about CL + Cpu not give a waste of web space on just CPU render systems.
Sure he spoke a little about cpu+CL.. but.. 90% of this is a total waste.
The render industry is underoing a Huge re-think due to CL.
And my other Gripe..
What clueless person will talk about using MS on relyable/functional render farms... ohh yea this guy.. (clueless.) sorry I am not impressed.. catch up with what is realy going on out there.
Most small VFX shops still use Windows render nodes, especially if the application calls for it. Some larger ones (Digital Domain) have their render nodes dual-booting Windows and Linux so that they can use either as the application calls for it.
The VFX shops are looking at what OpenCL might do but the majority of effects and animation shops to NOT develop their own renderers. Thus, they are paying attention to what is going on- but, until the commercial renderer of their choice actually supports using OpenCL to speed rendering and is stable and usable in a production environment, they cannot and will not plan around GPU-based rendering. They can plan for having the capability in the future, but they aren't going to bid on their next job with the expectation that the next version of mental ray is going to be 10x faster due to having GPU-based acceleration. Until they know from experience that it does, they are going to hope that it might but not plan and bid jobs based on it. Planning and bidding jobs based on expected future capabilities of your 3d software is a good way to end up out of business.
The VFX shops are looking at what OpenCL might do but the majority of effects and animation shops to NOT develop their own renderers. Thus, they are paying attention to what is going on- but, until the commercial renderer of their choice actually supports using OpenCL to speed rendering and is stable and usable in a production environment, they cannot and will not plan around GPU-based rendering. They can plan for having the capability in the future, but they aren't going to bid on their next job with the expectation that the next version of mental ray is going to be 10x faster due to having GPU-based acceleration. Until they know from experience that it does, they are going to hope that it might but not plan and bid jobs based on it. Planning and bidding jobs based on expected future capabilities of your 3d software is a good way to end up out of business.
I must say that to put forward a story is ok.. but tom's Hardware is to be detailing cutting edge stuff. Not what most small VFX shops have. (2pc's and a 15 year old pimpled kid).
how about Pixar? .. light and magic.. the real stuff/company's.
How about an article on there views on this new technology.. be a good read/story I think. And forbid.. me .. Mac's doing CL.. they can ao it now.. but you knew that..(wink)
As for buisness models on what to do.. I think you have to ask the big guys that realy do it before you pass comments. I think there commments and considerations will be realisitc and a worthwhile read.
nuff said.
how about Pixar? .. light and magic.. the real stuff/company's.
How about an article on there views on this new technology.. be a good read/story I think. And forbid.. me .. Mac's doing CL.. they can ao it now.. but you knew that..(wink)
As for buisness models on what to do.. I think you have to ask the big guys that realy do it before you pass comments. I think there commments and considerations will be realisitc and a worthwhile read.
nuff said.
When I talk about 'small' VFX shops, I mean shops with less than a hundred employees. I did check into how the 'big shops' do it, (including talking to people who work at said shops, and people who designed the network architecture for said shops) but this article wasn't about building a new renderfarm for ILM, it was about building one for a freelance animator or a small shop.
Pixar has a mixed renderfarm, it used to be Itaniums but there's no publicity data on exactly what they are running now. They haven't been running Suns since 2003.
ILM's renderfarm is a lot of linux boxes.
Both places have over a thousand employees, including people dedicated to administering their farm. Both places have more people dedicated to administering their systems than most smaller shops have in total employees. Both places also have in-house development teams for applications and their renderers that smaller shops do not have- often these dev teams consist of dozens of people in and of themselves. They can afford to be Linux-based because they have people to develop applications for their use when there are no commercial applications to do a particular function for Linux. Smaller shops don't burn a few dozen employees just for a Linux development team when they can just use windows and have it work fine.
By the way, I have worked at a few VFX shops, both as an artist and in systems, and as a freelance animator, for the past several years. The content of this article is not conjecture, it is based off of what boutique shops are actually doing for their renderfarms.
Pixar has a mixed renderfarm, it used to be Itaniums but there's no publicity data on exactly what they are running now. They haven't been running Suns since 2003.
ILM's renderfarm is a lot of linux boxes.
Both places have over a thousand employees, including people dedicated to administering their farm. Both places have more people dedicated to administering their systems than most smaller shops have in total employees. Both places also have in-house development teams for applications and their renderers that smaller shops do not have- often these dev teams consist of dozens of people in and of themselves. They can afford to be Linux-based because they have people to develop applications for their use when there are no commercial applications to do a particular function for Linux. Smaller shops don't burn a few dozen employees just for a Linux development team when they can just use windows and have it work fine.
By the way, I have worked at a few VFX shops, both as an artist and in systems, and as a freelance animator, for the past several years. The content of this article is not conjecture, it is based off of what boutique shops are actually doing for their renderfarms.
well
I think if you wish to talk about small old render shops and new ones that pop up and vanish now and then.. feel free.. this in reality does not show the real market and where the trends are moving. Your story is nice.. but spoken the wrong way around. People look at big fish, as they do make standards for the rest of the smaller fish. Talking about new technology with a bunch of small shops with the 2pc's and a bunch of 15 year olds is not a reality of any standard. Thoughts on writiing and seeing stuff like this must be provided better and from the right sources.. (( like you said.. Pixar.
And my last statement.. Get MS out of your head. it is a tool that is used in appropriate areas. . I will also correct your other problem.. Linux is not what is the right word you have to use.. try this new word.. 'UNIX'..
Roll it around your mouth a bit.. and try that.. put it in every sentince you have ever typed... (( that is it.. problem solved).
I look forward to a most enjoyable new story with your now new corrected concepts.
I think if you wish to talk about small old render shops and new ones that pop up and vanish now and then.. feel free.. this in reality does not show the real market and where the trends are moving. Your story is nice.. but spoken the wrong way around. People look at big fish, as they do make standards for the rest of the smaller fish. Talking about new technology with a bunch of small shops with the 2pc's and a bunch of 15 year olds is not a reality of any standard. Thoughts on writiing and seeing stuff like this must be provided better and from the right sources.. (( like you said.. Pixar.
And my last statement.. Get MS out of your head. it is a tool that is used in appropriate areas. . I will also correct your other problem.. Linux is not what is the right word you have to use.. try this new word.. 'UNIX'..
Roll it around your mouth a bit.. and try that.. put it in every sentince you have ever typed... (( that is it.. problem solved).
I look forward to a most enjoyable new story with your now new corrected concepts.
UNIX? You mean like SGI, which everyone has dumped?
Linux is what they are using (except for Pixar's OSX machines). Sorry to hear you think everyone is still using UNIX, because that hasn't been the case since.. oh, 1999 or so.
I'm sure the owners at EdenFX, Zoic, et al will love to hear you call them "small old render shops". They have Emmy Awards for VFX. Do you?
Linux is what they are using (except for Pixar's OSX machines). Sorry to hear you think everyone is still using UNIX, because that hasn't been the case since.. oh, 1999 or so.
I'm sure the owners at EdenFX, Zoic, et al will love to hear you call them "small old render shops". They have Emmy Awards for VFX. Do you?
Ohh dear me.. gosh.. You are a MS noob.
ok.. education 101.. I do not want to see more silly statements like above.
Try to understand this..
UNIX = Irix(SGI), Solaris, SunOs, MAC-OSX, HP-UX, Linux, Ubuntu, RedHAT, OSX, ....etc..
you get the idea.. UNIX is not an operating system specific word..
And Get off your hate SGI cart. SGI make Great computers. These computers are
definately worth while in there niche areas, (Just like MS)..
Do not jump about throwing rocks atanything because you have an MS hat on. Open your view and see both disadvantages and advantages of each system & service.. Looking unbiased at all things will help reviews and yourself..
ok.. education 101.. I do not want to see more silly statements like above.
Try to understand this..
UNIX = Irix(SGI), Solaris, SunOs, MAC-OSX, HP-UX, Linux, Ubuntu, RedHAT, OSX, ....etc..
you get the idea.. UNIX is not an operating system specific word..
And Get off your hate SGI cart. SGI make Great computers. These computers are
definately worth while in there niche areas, (Just like MS)..
Do not jump about throwing rocks atanything because you have an MS hat on. Open your view and see both disadvantages and advantages of each system & service.. Looking unbiased at all things will help reviews and yourself..
Try to understand this. Prior to 1998 or so, most VFX studios were Unix based. (SGI IRIX, in particular.) then the "Windows NT Revolution" happened and many switched to NT workstations with certain things still being done on their old SGIs. Some of the studios had a large investment in proprietary tools that were written for IRIX and it was easiest to port them to Linux when their old SGI systems became to slow to be practical to use. SGI even supplied extensions for Linux that made this easier, and actually built Intel-based Linux boxes (and Intel-based Windows boxes, which bombed because they still used proprietary rather than commodity components...) So some of the largest studios moved some of their work to Linux, specifically the ones with a large investment in proprietary tools- read this as, ILM.
Pixar was Sun-based for a long long time. They bought an Intel Itanium renderfarm in 2003, and moved their workstations to OSX. (Why? Ask Steve Jobs, he was CEO of both companies at the time...) Their renderfarm, however, is Linux-based. Pixar is the only major animation studio that is using OSX for their primary workstation OS.
Weta Digital came into the game a little late, and thus almost missed the entire "SGI-based" phase, and is primarily Linux-based.
Digital Domain weaned themselves off SGI, and moved most of their artists to Windows, while their render farm dual-boots Windows and Linux. (Many of their workstations do as well.)
Sony Pictures Imageworks uses Windows and Linux both for their workstations.
Rhythm and Hues uses a lot of Linux, and some Windows.
By this point, you should be getting the idea... the smaller the studio the more likely they are using Windows. And there are more seats- more working animators- in the smaller studios using Windows day after day than their are using Linux at the larger studios. And Windows works *just fine* for animation... I don't have massive stability problems with my workstation and the only time I've had stability problems it has been hardware-related... i.e. hardware failures.
*made* great computers. Past tense. I have an Indy and an O2, and for their day, they were nice. Nowadays, any PCIe-based Windows box is faster, even for graphics, and that has been the case for *years*. The last workstation SGI made, the Fuel, was $14,000 in its minimum configuration... I could buy a workstation-class machine and a farm of ten nodes for that.
Which hasn't been content creation, for quite a few years. The final nail in the coffin of SGI graphics machines was Flame becoming available as a Linux box.
I'm not saying you shouldn't use Linux. I'm not saying people don't use Linux. But if you aren't familiar with Linux, and are working as a freelance animator, or setting up a boutique studio or a virtual studio, then perhaps your time isn't best spent learning Linux, when Windows or OSX will work just fine. Most of the visual effects you see on TV are created on... wait for it... Windows. Quite a bit of the VFX you see in feature films is created in Windows as well. Most content creation for games is done in windows- as most of it is done in 3D Studio Max, which only runs under Windows.
Like I said.. showing a silly bias to 1 flavour of software (MS) is wrong. As a writer you have to have and display an unbiased and open mind, (you certainly show some knowledge of computers).
All you write was correct.. (even the use of the nice new word UNIx.. very good.
Lots of ommissions.. but as you wrote what was convienent to push your points.
For example, I myself have no favour to any operating system. I do not protect the grail of Linux ' unixes. I see what is used and where it is most usefull. Something you will slowly learn grasp as you get older and wiser. So no point attacking me with your MS rocks. You are only displaying what you do not know.
I will have 1 last word at pointing you at wisdom. If your view on SGI are 1 out of 10. and one day somebody comes to you with a use that SGI can do very very well. ( for example something that can use SGI's Huge buss bandwith ).
Now we all know that no other company in the world has nowhere near that performance. But at the same time we know that SGI CPU's are on the best side. very very poor performance. But you know that right..
OK so that is a quick and dirty example for you.. because you Hate SGI soo much. (Mind you Sun CPU performance is not that much faster than SGI. but Sun have different uses. (I bet you never tested massive systems or knew the constraints that bind them.. but.. I also bet you have never asked ..
I ask you to go try philosophy as a side enhancement.
All you write was correct.. (even the use of the nice new word UNIx.. very good.
Lots of ommissions.. but as you wrote what was convienent to push your points.
For example, I myself have no favour to any operating system. I do not protect the grail of Linux ' unixes. I see what is used and where it is most usefull. Something you will slowly learn grasp as you get older and wiser. So no point attacking me with your MS rocks. You are only displaying what you do not know.
I will have 1 last word at pointing you at wisdom. If your view on SGI are 1 out of 10. and one day somebody comes to you with a use that SGI can do very very well. ( for example something that can use SGI's Huge buss bandwith ).
Now we all know that no other company in the world has nowhere near that performance. But at the same time we know that SGI CPU's are on the best side. very very poor performance. But you know that right..
OK so that is a quick and dirty example for you.. because you Hate SGI soo much. (Mind you Sun CPU performance is not that much faster than SGI. but Sun have different uses. (I bet you never tested massive systems or knew the constraints that bind them.. but.. I also bet you have never asked ..
I ask you to go try philosophy as a side enhancement.
SGI Fuel- 3.2 GB/s memory bandwidth, 1.6 GB/s graphics interconnect bandwidth
Generic core 2 duo PC- 6.4 GB/s memory bandwidth, 8 GB/s graphics interconnect bandwidth
Like i said, even Flame, which used to be SGI's killed app for graphics, has moved off of SGIs.
Ewww p5b-vm se. I tested a Dell vostro desktop that had faster onboard graphics than that one. Plus they are somewhat prone to being DOA like many other Asus product these days.
DDR3 offers little benefit? Well reading tom's hardware own reviews that is true on a core2 system but not necessarily completely true on an i7.
DDR3 offers little benefit? Well reading tom's hardware own reviews that is true on a core2 system but not necessarily completely true on an i7.
globolstaff02
Distinguished
- Aug 19, 2009
- 1
- 0
- 18,510
Welcome to the site, thank you for supporting the site. It's people like you who read and try things yourself that make our lives a bit easier. Thank you.
website update
website update
The Molecule, a visual effect and motion graphic studio, based out of mid-town Manhattan just blogged about their render farm. Pretty interesting blog http://themoleculevfx.blogspot.com/
- Status
TRENDING THREADS
-
Question Is LGA1150 still viable in 2025(26,27,28,29,30)?- Started by MaIwarebytes
- Replies: 3
-
Question No POST on new AM5 build - - - and the CPU & DRAM lights are on ?
- Started by Uknownflowet
- Replies: 13
-
-
-
Question Gigabyte Eagle B650 AX CPU led is on, but PC seems to work fine except longer booting
- Started by TopekaMarek
- Replies: 5
-
Facebook
Twitter
Instagram