Showing posts with label blender. Show all posts
Showing posts with label blender. Show all posts

Monday, 9 March 2015

Blender Render Farm Tutorial

This tutorial will talk you through how to use the Render Farm in S721. You will be able to submit a job, and collect the files when finished.

  1. Load up Blender 2.73a found in Applications > Blender 2.73a > Blender
  2. You will need to enable the render farm plugin
    1. Go to File > User Preferences
    2. Go to "Render" on the left hand list
    3. Enable "Network Renderer"
    4. Close this window
    5. At the top go to where it says "Blender Render" and select "Network Render":
    6. On the lower right hand panel you will see the controls for network rendering:
    7. Once a job is submitted you will have to wait until it is complete before collecting the files.
On the Master Machine (the Mac Pro teaching machine) you can log in and access the queue viewer.
  1. Open a web browser on this machine
  2. in the URL type
    1. http://localhost:8000
 Done

Thursday, 12 February 2015

Blender render farm - built in Network renderer

Well, I tested this last year, but I can't believe it, I can't find any mention of me doing it. It either means I forgot to write it up, or didn't write it up in this blog! Oh well, here's a new writeup:

The only files I have to go from are two files named:
master.blend
slave.blend

Basically, these files have presets in them, so whichever machine opens them either becomes the master or the slave.

Here's a document I found which has 'slave.blend' mentioned in it, looks very promising.

  1. First thing is to load blender, and load the Network Renderer plugin. Go to:
    1. File > User Preferences > Render > Network render > tick:
  2. Up the top, next to File, Edit menus, there is a dropdown usually on "Scene". Select "Network Render":
  3. On the far right you should see the Network Render settings:
  4. Click Master on the machine you want to control jobs. Make sure it's a machine which is always on, perhaps with storage space which can be shared to other machines (not worked out where rendered files are stored)
    1. Click Start Service
  5. You can now view the jobs and clients connected to the master by browsing to the webpage:
    1. http://ipAddressOfMaster:8000:
      1. I had trouble with this, but found it was because I was on the master machine and so had to type:
        1. http://localhost:8000/
      2. There is also a newer jquery interface: /html/newui#interface:
  6.  Now start another machine as a slave
  7. On the final machine, load up a nice file and click "Send Job":
  8. The systems so good, the end files end up where-ever you said for the output (at the bottom of the client list). You can even download renders from previous jobs!!

Now that I got it all working, the plan was to save a master.blend file and a slave.blend file and have them load up when the machines are started, from command line. Here it goes:

 To run the master:
/Applications/Blender\ 2.73a/blender.app/Contents/MacOS/blender -b /Users/Shared/master.blend --addons netrender -a -noaudio -nojoystick

The slave:
/Applications/Blender\ 2.73a/blender.app/Contents/MacOS/blender -b /Users/Shared/slave.blend --addons netrender -a -noaudio -nojoystick 

Seems to work very nicely. One thing to note is that the IP address of the master has to be saved in the slave.blend file. I don't know how to add this through the command line.

The next thing is to get the slave.blend command to launch at system startup, and run as root.

Found this question and answer in stackoverflow 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>uk.ac.chesterfield.dci.blender.client</string> <!-- org.mongodb.mongodb perhaps? -->

    <key>KeepAlive</key>
    <true/>
  
    <key>RunAtLoad</key>
    <true/>
  
    <key>ProcessType</key>
    <string>Background</string>

    <key>UserName</key>
    <string>root</string>

    <key>ProgramArguments</key>
    <array>
            <string>/Applications/Blender 2.73a/blender.app/Contents/MacOS/blender</string>
            <string>-b</string>
            <string>/Users/Shared/slave.blend</string>
            <string>--addons</string>
            <string>netrender</string>
            <string>-a</string>
            <string>-noaudio</string>
            <string>-nojoystick</string>
    </array>
</dict>
</plist>

Wednesday, 21 May 2014

Blender render farm for OS X in background

Hi, I want to set up a render farm at our College to run on any machine in the background with minimal installation and setup.

It looks like the built in render farm to blender is only acceptable on smaller scales where all machines are logged in, and have blender open and running. This isn't the right solution for me.

A while back I set up a render farm for Maya using OS X Qmaster. This runs in the background without anyone logged in, receives jobs and executes command line arguments. A network share is mounted on all machines running it. Looks like Qmaster isn't set up correctly on Mavericks, also might be some compatibility issues for it too. Looks like you can set up with command line:

sudo SystemStarter start Qmaster
qmasterprefs -list
Current settings:
 -sharingType servicesAndQuickCluster
 -launchContentAgent off
Bus error: 10
qmasterprefs -cluster on servername "test1"
Bus error: 10

I'm getting "Bus Error: 10" every time I run any commands

I also setup Blender to render over Xgrid, but Apple has pulled this out of the new servers so I'll give this a miss for now.

Farmerjoe render farm, looks alright. Need to install Perl I think.

DrQueue looks interesting, but you do need to compile it first.

Backburner is more promising, as we can also use it for Maya and any autodesk app. Found an article here which almost proves it. So here's my attempt a few years ago to get it working. I tried once more but couldn't get the web monitor working, might be because I've got the server app installed? Or it's mavericks? I shall continue with this.

Monday, 15 February 2010

Xgrid and Blender part 3

Well, hopefully the final installment of the Xgrid and Blender saga. So, now I started using gridstuffer, a very useful app. By setting up a text file, and giving it some lines of code you want to submit to the xgrid, it'll render the files to where you specify in the blender file. Unfortunately Blender can only really render to the tmp folder because for some reason it has trouble with permissions in ALL other folders (doh!).

So I finally tried a couple of other commands in blenders command line (which I put in the text file) to see if I could get that to tell blender where to render to. Success! It actually renders to the "Output" folder which you specify in gridstuffer (or generally in Xgrid!).

So the final command looks like this:



/Applications/Blender/blender.app/Contents/MacOS/blender -b KitchenTopFreeTree2.blend -o //kitchen -F TGA -x 1 -f 1
/Applications/Blender/blender.app/Contents/MacOS/blender -b KitchenTopFreeTree2.blend -o //kitchen -F TGA -x 1 -f 2
/Applications/Blender/blender.app/Contents/MacOS/blender -b KitchenTopFreeTree2.blend -o //kitchen -F TGA -x 1 -f 3

the -o specifies the folder and file to save. The -F is the file type, -x specifies whether to add numbers to the end. The -f specifies the frame number!

So the blender file is in the input directory, and the // defines it is in the root of the output directory.

Looking at how the whole thing works, I think the machine you submit it all from is the machine which distributes and collects all the files needed. The server just works out which machines to send it to. I don't think that the files have to be on a share? I shall test this theory out.

So the only pre-requsits are that the machines are on the xgrid network, and they have blender installed in the right directory with the right name (case-sensitive)

References:

http://wiki.blender.org/index.php/Doc:Manual/Render/Command_Line_Options

Wednesday, 10 February 2010

X-Grid and Blender part 2

So, after looking at the said articles I found that they are actually slightly out of date (2004!) and OS X 10.5 no longer supports the GUI for submitting jobs in that particular way. And plus, to my dismay, I cannot find it to download ANYWHERE! I tried many many things alas and I've ended up spending about 3 days slowly getting there, and I have got somewhere. I think I'll go through all the troubleshooting I went through, so I can say what I have and have not tried.

So to start with I had my xgrid all setup (previous post), running xgrid admin from my machine. For a test setup I thought I'd try and just get one machine to render an animation, and then get two and so on. So I started with my machine (although I should have started with a third machine). So I made a blender file "box.blend" and tried some command line rendering. Finally I got this command to render the animation:

/Applications/Blender/blender.app/Contents/MacOS/blender -b /Volumes/Shared/xgridprojects/in/box.blend -a

note: I saved it on a network share on the xserve mounted in /Volumes/Shared so that other machines in the future could access this file.

note 2: this is case sensitive so watch out for Blender or blender in the applications folder. I think I renamed it 'Blender' and it prefers to be 'blender' (this held me up for 15 mins!)

the -b renders in the background (put before the file) and the -a renders the animation you have setup in the blend file (default to /tmp/). The /tmp/ directory turned out maybe to be the best directory to render to as blender, when run by this xgrid user (no-one) can't seem to write to any other folders.

The -a attribute renders as many frames as are in the animation. At first this seemed the best but when using with xgrid, the alternative option is -f [frame number] which renders a specific frame. Also, -a can be used with -s [start frame] and -e [end frame] but I think these need to be plaeed carefully either before or after the '-a' attribute.

So, now I knew the commands which xgrid needed to send to each machine I could begin. A big 'downer' was the fact I didn't have the GUI in the tutorial so it was command line stuff. The first thing was to know how to send a job:

xgrid -job submit [-gid grid-identifier] [-si stdin] [-in indir]
           [-dids jobid [, jobid]*] [-email email-address]
           [-art art-path -artid art-identifier] [-artequal art-value]
           [-artmin art-value] [-artmax art-value] cmd [arg1 [...]]

so;

xgrid -job submit -gid xserve.chesterfield.ac.uk /Applications/Blender/blender.app/Contents/MacOS/blender -b /Volumes/Shared/xgridprojects/in/box.blend -a

I think this might have worked, but I saw that actually you can have input directories and output directories! So this is where the network mount might have come in:

xgrid -job submit -gid xserve.chesterfield.ac.uk -in /Volumes/Shared/xgridprojects/in/ -out /Volumes/Shared/xgridprojects/out/ /Applications/Blender/blender.app/Contents/MacOS/blender -b box.blend -a

notice how I didn't have to specify where my box.blend file was, as it automatically looked in the "in" folder. Now this folder was mounted on the computer I was rendering on through xgrid, but I now think that xgrid manages these files all by itself, so I don't think that it has to be mounted (I shall try tomorrow). I think the main point is that it is on the machine which submitted the job!

Now that this worked I tried it on two machines! Alas, it only used one machine to render the entire animation! This was because I didn't specify the variable or 'range' (which was in the tutorial) which changes. The only solution I found was to specify lots of jobs, each rendering a specific part of the animaton. Luckily I found an application for submitting 'Batch' jobs  which is called 'GridStuffer'. You can give it a text file with the cmd for the specific jobs, a different line for each job you want, and it will send it to the xgrid as lots of different jobs. Not only that but if a job fails, it'll keep sending it until it is complete (fully optional though).

So with this handy tool I made a text file like this:

/Applications/Blender/blender.app/Contents/MacOS/blender -b /Volumes/Shared/xgridprojects/in/box.blend -f 1
/Applications/Blender/blender.app/Contents/MacOS/blender -b /Volumes/Shared/xgridprojects/in/box.blend -f 2
/Applications/Blender/blender.app/Contents/MacOS/blender -b /Volumes/Shared/xgridprojects/in/box.blend -f 3
/Applications/Blender/blender.app/Contents/MacOS/blender -b /Volumes/Shared/xgridprojects/in/box.blend -f 4
etc....

and it will send them all to the xgrid under different jobs, and keep sending them until they are done. That way the xgrid application will send each job to a different machine and utilise all the processers it needs, YAY!

I think you don't even need the blend file on the server, I think it sends it from that machine (but this theory needs testing). I think what you do need is the blender application on each machine although I seem to remember it saying you didn't!! But anyway, not the hardest thing to do.

One of the main stumbling points at the moment is trying to get all the rendered files from all of the machines '/tmp/' folders. A script could be written and sent out over ARD, but I would prefer it if I could get xgrid to move them, or blender to save to a networked space, which would mean mounting on each machine before jobs are sent out. Preferably all the machines would automount a volume on bootup which could be useful for a lot of other things. Apparenly this might be done by NFS shares but needs looking into.

Well, till another day. Hope this helps sometime again and it wasn't all in vain.

Monday, 8 February 2010

X-Grid setup

So, I've managed to get all the Apple Macs in Art and Design Directorate at Chesterfield College! I should hopefully reach 400GHz when all the machines are finally switched on. Here's the general gist of how I did it:


  1. Set up Xserve with XGrid. I did have a problem switching on XGrid but that was fixed on my previous post. Go to "Server Admin" and set it as the Controller. I didn't make it an agent as I didn't want to slow it down (it's only a G4). It's as simple as that
  2. Check XGrid is working by going to your machine and loading "Xgrid Admin". Now connect to your Xserver and all should be there (no agents though)
  3. Set the clients up. This can be done in two ways, Workgroup Manager or Apple Remote Desktop.
    1. I used a combination. I set up my machine by going to the Sharing Pane in System Preferences and clicking "XGrid" and specifying the servers address (DNS). Not using authentication at the moment.
    2. Now go to Workgroup Manager and manage computer groups and preferences, go to Details and add: "/Library/Preferences/com.apple.xgrid.agent" and I always make the changes "Always". This should put these preferences on all the machines in this group.
    3. Start the service on each client machine. I did this through ARD's command line. I've also put how to change preferences on the client machines if Workgroup Manager doesn't work (run as Root):
      1. sudo xgridctl a stop
      1. sudo defaults write /Library/Preferences/com.apple.xgrid.agent ControllerName xserve.chesterfield.ac.uk
      2. sudo defaults write /Library/Preferences/com.apple.xgrid.agent ControllerAuthentication None
      3. sudo defaults write /Library/Preferences/com.apple.xgrid.agent BindToFirstAvailable 0
      4. sudo defaults write /Library/Preferences/com.apple.xgrid.agent ResolveNameAsNetService 0
      5. sudo defaults read /Library/Preferences/com.apple.xgrid.agent
      6. sudo xgridctl a start
      7. sudo xgridctl a status
Now everything should be working and through XGrid Admin you should see all the clients connecting. Next thing to do is to submit some jobs!

Saturday, 6 February 2010

Xgrid Startup troubles

Well, after a very long time of X-Grid being untested, I decided to open it up and see if I could utilise it for Blender (see other post), but alas, on startup of Xgrid on the server it crashes! Doh!

So, after a little looking through the logs I knew I wouldn't find anything so I googled it and after a few pages the answer was found:

http://lists.apple.com/archives/xgrid-users/2007/Sep/msg00012.html

So, basically do this in terminal on the server (or ssh)

sudo xgridctl c stop sudo xgridctl c off sudo rm /var/xgrid/controller/datastore.db sudo rm /var/xgrid/controller/status sudo xgridctl c start sudo xgridctl c on

He's added more, but this fixed it. I'll put the rest on though just incase it helps:


optionally, you could do:
sudo rm /var/xgrid/controller/blobs/*

(this can take a while if your database is old and has grown a lot of crap; another option is thus to rename the blobs to blobs-old and create a new one with the same permissions; then while the controller restarts happily, delete blobs-old).

Carefully check the permissions and ownership before messing up with / var/xgrid/controller.
The owner of most of the files is xgridcontroller (the name can be clipped of in a simple ls -al).



Just need to work out how to send a job to it (which is surely quite important!)

Thursday, 4 February 2010

Blender Render Farm with X-Grid

Well, with my new project only begun I thought I'd optimistically look into a render farm using X-Grid on our Mac OS X 10.4 server (a little outdated though).

So, after not much time I found this article:
http://www.atpm.com/10.06/blender.shtml

and it looks almost complete. Watch this space, although you might have to wait until I have a large enough project to render!

Also, here are the Blender command line arguments:
http://download.blender.org/documentation/html/a19823.html

I'd just like to note that if all goes well, I could easily adapt this for Maya!!

Thursday, 21 January 2010

Camera Tracking with Voodoo and Blender

Well, it's a little bit of a minefield camera tracking using free software as you'd expect. Heres what I've been able to find out in a few hours:

To start with you will need a video file which you want to camera track. I used a simple camera and recorded video on it.

Next you need to convert this to an image sequence (not the most straight forward things to do suprisingly). After 20 mins of looking I found VirtualDub does this and many more video conversions (worth looking at). Also, you can get blender to do it for you by putting a camera with the video in the background and rendering out the sequence (handy).

Now you can get round to using Voodoo. Open the sequence and click on ¨Track¨ down in the bottom right (took me a while to find this the first time! Should be a menu item for it). This takes a while but eventually you'll have a nice bunch of points.

You need to export this as a python script for Blender.

To get this into Blender, you need to open up the text editor, open the .py script that was saved, then click File> Run Python Script (easy!!)

Now you have the camera animated and 3D points corrisponding to points in the real world.

I managed to find the Blender side of things from this web address: http://fictionality.co.uk/doku.php?id=tutorials:matchmoving

I might add to this when I find out more. But I'm still in the testing stages.

Sunday, 10 January 2010

3d tree generation

I want to create realistic trees and bushes for a project to visualise Chesterfield covered in trees. I want to use free programs and came across Gen3 - http://wiki.blender.org/index.php/Extensions:Py/Scripts/Manual/Wizards/Gen3 - http://blendernewbies.blogspot.com/2009/07/gen3-tree-script-url-updated.html for blender and Arbaro http://arbaro.sourceforge.net/

Looks like it is easy to create shrubs and young trees but a little harder to create mature trees. There is some mature trees which look impressive though. Both these programs are based on the findings in the document Jason Weber & Joseph Penn: "Creation and Rendering of Realistic Trees" -PDF - Creation and Rendering of Realistic Trees which gives information on tree generation and some presets for trees. Might be helpful.