PhotoShop Interop

Recently, there was a thread on CGTalk about whether it was possible to save out multiple render passes into a layered photoshop file from 3dsMax itself. This question had been sitting in the back of my mind ever since and I decided to look into whether it was possible. My first thought was that there was an external image library out there that could do the job. However, finding a good solution meant I didn’t want to have to purchase software for something that was purely research. I already own a great library – AIL, which is based on the freeimage library. Alas, while this can open Photoshop documents, it can not create, structure and save them. However, after a bit of research on the Adobe site, I found that it is indeed possible to do all these things using DotNet.

With Photoshop, Adobe has included an object library assembly. This is a clever assembly that allows you to write custom scripts and automate the Photoshop UI. “Hang on Shirley”, I hear you cry, “We don’t need that – I own a Mac and we’ve got actions for that, I can do everything I need from there! It’s fun, too”.

Well, yes and no. This programmed method still could be the ‘droids you are looking for’.

Actions mean that you obviously need to have it recorded at some point, and be able to trigger the start of the action from within the photoshop application. However, when using maps in 3dsMax there are many occasions when you need to perform a tweak here and there to a map, and the workflow is broken as you have to open it in another program and perform the tasks.

Some other benefits (from the Adobe scripting guide) are -

  • You can add conditional logic, so that the script automatically makes “decisions” based on the currentsituation. For example, you could write a script that decides which color border to add depending onthe size of the selected area in an image: “If the selected area is smaller than 2 x 4 inches, add a greenborder; otherwise add a red border.”
  • A single script can perform actions that involve multiple applications. For example, depending on thescripting language you are using, you could target both Photoshop CS3 and another Adobe CreativeSuite 3 Application, such as Illustrator® CS3, in the same script.
  • You can open, save, and rename files using scripts.
  • You can copy scripts from one computer to another. If you were using an Action and then switchedcomputers, you’d have to recreate the Action.
  • Scripts provide more versatility for automatically opening files. When opening a file in an action, youmust hard code the file location. In a script, you can use variables for file paths.

With this in mind, you are presented with the opportunity that you can automate Photoshop from within the 3dsMax interface. For example, you could have a maxfile that renders multiple passes and calls a post-render script when it finishes, triggering the creation of a psd composite of all the images. You could use it to call a function to overlay an Ambient Occlusion pass on notification of a completed render. You could select an object in the 3dsMax viewport and have Photoshop apply a Gaussian Blur or an artistic effect to the Diffusemap without leaving the 3dsMax Interface. The applications can be as specialised as you want but completely customisable in the same breath.

Creating a Photoshop Assembly for 3dsMax

Because the access to Photoshop is via COM, you need to use an interop assembly in order to use it within a DotNet context. With my fuzzy logic, The interop assembly basically handles the calls between the DotNet assembly and the COM object. I have demonstrated this before, using the COM assembly DSOFile in order to extract the 3DSMax file thumbnail, and there seems to be a small snag that I haven’t been able to resolve as of yet.

Trying to directly use COM interop via the DotNet bridge in 3dsMax can only get you so far. At some point you will (via interop) be accessing a COM object itself. This part I don’t explain fully myself, but as Max uses reflection it reaches a point at which certain objects are created and viewed explicitly as the system.__COMobject type. This is a pain, – one because you can’t cast it to the correct object type in 3dsMax as this is a core function, and two because as a system.__COMobject, you can only access the properties and methods of that type, not the underlying class that the interop assembly is giving you access to. Here’s an example of my failed attempt using just the interop assembly and attempting to set this up via MaxScript – (Listener output is in blue)

dotnet.loadassembly ((getdir#scripts)+"\LoneRobot\ClassLib\Interop.Photoshop")
dotNetObject:System.Reflection.Assembly
ps = dotnetobject "Photoshop.ApplicationClass"
dotNetObject:Photoshop.ApplicationClass
showproperties ps
.ActiveDocument : <Photoshop.Document>
.Application : <Photoshop.Application>, read-only
.BackgroundColor : <Photoshop.SolidColor>
.ColorSettings : <System.String>, read-only
.DisplayDialogs : <Photoshop.PsDialogModes>
.Documents : <Photoshop.Documents>, read-only
.Fonts : <Photoshop.TextFonts>, read-only
.ForegroundColor : <Photoshop.SolidColor>
.FreeMemory : <System.Double>, read-only
.Locale : <System.String>, read-only
.MacintoshFileTypes : <System.Object>, read-only
.MeasurementLog : <Photoshop.MeasurementLog>, read-only
.Name : <System.String>, read-only
.Notifiers : <Photoshop.Notifiers>, read-only
.NotifiersEnabled : <System.Boolean>
.Path : <System.String>, read-only
.Preferences : <Photoshop.Preferences>, read-only
.PreferencesFolder : <System.String>, read-only
.RecentFiles : <System.Object>, read-only
.ScriptingBuildDate : <System.String>, read-only
.ScriptingVersion : <System.String>, read-only
.Version : <System.String>, read-only
.Visible : <System.Boolean>
.WinColorSettings : <System.String>, read-only
.WindowsFileTypes : <System.Object>, read-only
ps.documents
dotNetObject:System.__ComObject
-- this should be a Photoshop.Documents class??
show ps.documents
false
showmethods ps.documents
.<System.Runtime.Remoting.ObjRef>CreateObjRef <System.Type>requestedType
.<System.Boolean>Equals <System.Object>obj
.[static]<System.Boolean>Equals <System.Object>objA <System.Object>objB
.<System.Int32>GetHashCode()
.<System.Object>GetLifetimeService()
.<System.Type>GetType()
.<System.Object>InitializeLifetimeService()
.[static]<System.Boolean>ReferenceEquals <System.Object>objA <System.Object>objB
.<System.String>ToString()
-- ie showing the methods of the COM object, not the documents class.

I cant say exactly what is going on, except that Max is returning the COM object’s methods rather than the methods the interop assembly is allowing from the Adobe Object library. Maybe this is something to do with Reflection or the way 3dsMax sees DotNet, I really don’t know. If anyone reading this has any ideas, feel free to correct me. Just using the application class methods, like load() will work, so you can open a photoshop file via max, it’s just that you can’t do an awful lot after that. :-(

It’s not all bad, as there is no such limitation in the Visual Studio IDE. Which means you can write a wrapper to expose the functions you want to call within max. This is actually a nice way of testing these things, as the VS IDE is a far superior environment to test and debug your classes, and you also get help from intellisense and prompting you when you make all sorts of errors.

Coding a COM Wrapper

The next bit assumes the following criteria, and was done in this environment -

  • A Visual Studio 2008 VB project on Vista x86
  • PhotoShop CS3 (this is the newest and only version I have)
  • 3dsMax 2009

For different versions of Photoshop, there may be slight differences but the principles should essentially be the same. Also note that the code examples are in Visual Basic, but would be almost identical if done in C#. Once the photoshop object is created, the methods and property access would be the same. A wrapper is a class library that provides you with convenient access to the photoshop application, so that you can make general calls from max like photoshop.blur() or photoshop.strokeselection red. It means you can write an assembly with whatever functions you want and deploy them wherever you like.

Creating the Interop Assembly

Open the Visual Studio IDE. Choose File>New Project>Class Library (VB or C#)

Name it something suitable. Not PatSharpesHairPiece or MrCommotasticsPhotoShopofHorrors, something meaningful. Actually that last one isn’t that bad.

Click the show all files button on the Solution Explorer -

Right Click the References branch and choose Add Reference (Or from the menu Project>Add Reference)

When the dialog loads, instead of .NET, pick the COM tab. Choose Adobe Photoshop Object Library from the list.

Bingo, you’ve created a reference to photoshop -

Save, and Build your project. Depending on the compile parameters, you will have created an interop assembly in one of the folders. Usually this is the projectfolder>bin>debug folder. If you check in this folder in the solution explorer you will see listed Interop.Photoshop.dll

That’s the first part, Visual Studio automatically creates the COM interop assembly for you. Thanks Visual Studio, and much obliged I am too.

Bring on the Shop

Now a drop of the good stuff. Take a look at how the photoshop object is constructed. I have highlighted the three main aspects to show what we will be using in this tutorial -

Most important is the Application class. This creates an instance of the photoshop application. If photoshop is not open, Instantiating this object will open Photoshop for you. Don’t worry, you can make the UI a background process after this if you don’t want to (or need) see what is going on. All operations on individual documents are through the Documents class, strangely, and individual layers are accessed via the ArtLayers class. Collections index members are from 1 rather than 0 like in a VB array. The first document
created has an index of 1.

If you look up in the Document object in the
Visual Basic Object Browser, you will see that there is no Add() method for the object. However, the Add()
method is available for the Documents object. Similarly, the ArtLayer object does not have an Add()
method; the ArtLayers object does. You have to add these via these collections than the individual object class. Not only that, you need to refer to them via the Application class. So to adjust the brightness and contrast of layer 4 you would have to do it with the following code -

<ApplicationObject>.ActiveDocument.ArtLayers(4).AdjustBrightnessContrast()

In your class library, place an import (or using in C#) statement at the top of the class so you don’t have to keep writing the full address. I have called my class PhotoShopBot, surprisingly.

Imports Photoshop
Public Class PhotoshopBot
End Class

In order for the class to perform the automation you are going to program, It will need an instance of the Photoshop application object. It makes sense to declare this as a private member variable at the start, and instantiate it in the new sub. That way, you only create one photoshop application object, which can then be used if you extend the class library in the future and add more methods. You don’t want to be creating it each time you call a function, that would be wasteful.

Imports Photoshop
Private PSDapp As Application
Public Class PhotoshopBot
Public Sub New()
PSDapp = New Application()
End Sub
End Class

Currently, all this class is doing is instantiating the application object, so in theory creating a dotnetobject in max from the resulting assembly would actually open up the photoshop interface!

Now, define a sub for the actual work you want the class to do. Since the post on CGTalk was discussing whether it was possible to create a PSD file with layers from multiple render passes, so this is what I will setup in this tutorial.

Imports Photoshop
Private PSDapp As Application
Public Class PhotoshopBot
Public Sub New()
PSDapp = New Application()
End Sub
Public Sub BasicImageCompositor(ByVal Images() As String, _
ByVal OutputFileName As String, _
ByVal HideApplication As Boolean, _
ByVal OpenOnCompletion As Boolean)
End Sub
End Class

BasicImageCompositor is going to work by providing an array of filenames. This could be passed from a maxscript using the GetFiles function to create the array, so could be generated dynamically from a folder’s contents. For the purposes of this basic subroutine, the layer order will be done bottom up by the order of the array elements, so make sure the background layer is first and each subsequent layer next. You could give your passes names so that this would automatically arrange them into the correct order after calling sort on the array. The reason it is an array of strings is that the Application object calls load on a string filename, don’t forget windows only opens a small selection of filetypes natively.

The thing to remember is that you need to keep track of the active document before you do anything. This is a property that is set via the Documents collection of the Application object. Let’s look at the entire class -

Imports Photoshop

Public Class PhotoshopBot

Private PSDapp As Application

Public Sub New()
PSDapp = New Application()
End Sub

Public Sub BasicImageCompositor(ByVal Images() As String, _
ByVal OutputFileName As String, _
ByVal HideApplication As Boolean, _
ByVal OpenOnCompletion As Boolean)

' get a fileinfo object to check the folder validity
Dim OutFile As New IO.FileInfo(OutputFileName)

If OutFile.Directory.Exists Then
' hide the photoshop UI
If HideApplication Then PSDapp.Visible = False
' Suppress the Photoshop dialogs (if any)
PSDapp.DisplayDialogs = 3
'   enum 3 =
'   PsDialogModes.psDisplayNoDialogs()
'load the first image and duplicate it into a new image document

If Images.Length > 0 Then
Dim BackGroundLayer As String = Images(0)
PSDapp.Load(BackGroundLayer)
PSDapp.ActiveDocument = PSDapp.Documents.Item(1)
PSDapp.ActiveDocument.Duplicate(OutFile.Name)
'close the original
'close(1) bypasses the save dialog
PSDapp.Documents.Item(1).Close(1)
'set the active document to the duplicate
PSDapp.ActiveDocument = PSDapp.Documents.Item(1)
'loop through the rest of the array, copying and pasting it into the
'duplicated document
For imageindex = LBound(Images) + 1 To UBound(Images)
PSDapp.Load(Images(imageindex))
PSDapp.ActiveDocument = PSDapp.Documents.Item(2)
PSDapp.ActiveDocument.Selection.SelectAll()
PSDapp.ActiveDocument.ArtLayers(1).Copy()
PSDapp.Documents.Item(2).Close(1)
PSDapp.ActiveDocument = PSDapp.Documents.Item(1)
PSDapp.ActiveDocument.Paste()
Next

'save it
PSDapp.ActiveDocument.SaveAs(OutputFileName)
PSDapp.ActiveDocument.Close(1)
' show the ui and load the image, if desired
If HideApplication Then PSDapp.Visible = True
If OpenOnCompletion Then PSDapp.Load(OutputFileName)
End If
Else
' if the folder doesn't exist it will throw this exception
Throw New IO.DirectoryNotFoundException("Directory does not exist")
End If

End Sub

End Class

That’s it, compile the project and copy the two dlls it makes to your max directory or wherever you load your dotnet stuff from. Depending on your project and namespaces used this will be different but mine is photoshop.dll and photoshop.interop.dll

Using this assembly in 3DSMax

Now there is the simple issue of creating an instance of this class and calling the only method it has. I am going to composite the following images -

Background – (Click to save if you want to try this example)

Other layers

As for 3dsMax, here is the script to automate the creation of the PSD file. Don’t forget your paths will be different to mine!

dotnet.loadassembly ((getdir#scripts)+"\LoneRobot\ClassLib\Interop.Photoshop")

dotnet.loadassembly ((getdir#scripts)+"\LoneRobot\ClassLib\Photoshop")

oPhotoShop = dotnetobject "LoneRobot.Imaging.PhotoshopBot"

showmethods PshopApp

imagepath = @"C:\LoneRobot Script Development\PhotoShopInterop\testimages"

imagearray = #((imagepath + "\bg.jpg"),(imagepath + "\layer1.png"),(imagepath + "\layer2.png"),(imagepath + "\layer3.png"),(imagepath + "\layer4.png"))

oPhotoShop.BasicImageCompositor imagearray (imagepath +@"\testoutput.psd") false true

All things being fair and just, you should have this in Photoshop – Ah crap!

Problem with this is that the copy/paste automation has placed the layers bang in the middle of the composition. This is how photoshop does this, regardless of image, if there is a transparent area around the image it takes the bounding box of the layer and pastes it smack in the middle of the comp. What it needs is a method for placing the layers when they are put in the composition. Looks like we will have to return to our class library and implement this.

Adding layer placement to the class

In a somewhat hacky way, photoshop does allow you to paste a layer into a certain position – If you select a marquee somewhere on the composition, it will use that as a location point to place the pasted layer. It takes the center of the layer and aligns it to the center of the selection marquee. Because of this, we can try to mimic this behavior in our loop.

So firstly, we need to work out how to automate the selection procedure in Photoshop. The Photoshop application object wants a mildly ambiguous Region object to be passed to the select function. This is listed in the Adobe help as

Array(Array(0, 0), Array(0, 100), Array(100,100), Array(100,0))

Now this is VBScript so my best guess is that in VB it is an array of Integer Arrays, with each array being the coordinate of the drawing surface. The other thing to note is if you traced the draw order, it means that it is drawing the selection marquee point-by-point in an anti-clockwise direction. This is a little confusing
to say the least, but since drawing selection marquees are pretty important,
It would be worth finding a way of calling this method with a simpler structure. To create a Photoshop selection region via VB the syntax is slightly different -

 Dim psdArray1() as Object = {x, y}

Dim psdArray2() as Object = {w, y}

Dim psdArray3() as Object = {x, h}

Dim psdArray4() as Object = {x, h}

Dim psdArray() as Object = {psdArray1, psdArray2, psdArray3, psdArray4}

PSDapp.Selection.Select(psdArray)

It seems that casting to anything else than a generic object causes an exception with the COM assembly. The best thing, in order to keep this DotNetty, would be to write a function to return the selection region by providing a Drawing.Rectangle object as an argument. That way it’s much easier to visualise the area that you want to select. Here’s what i came up with -

Public Function SelectionMarquee(ByVal Area As Drawing.Rectangle) As Object

Dim psArray1() As Object = {Area.X, Area.Y}

Dim psArray2() As Object = {Area.X, Area.Y + Area.Width}

Dim psArray3() As Object = {Area.X + Area.Height, Area.Y + Area.Width}

Dim psArray4() As Object = {Area.X + Area.Height, Area.Y}

Dim psArray() As Object = {psArray1, psArray2, psArray3, psArray4}

Return psArray

End Function

My idea is to pass a drawing.point array from max, since you are simply saying “paste this layer into position X,Y”. The function will then construct a rectangle with a height and width of 1 pixel. Then, when pasting the layer, it will be at the XY coordinates specified. Let’s see how that looks -

Public Sub BasicImageCompositor(ByVal Images() As String, _

ByVal Placement() As Drawing.Point, _

ByVal OutputFileName As String, _

ByVal HideApplication As Boolean, _

ByVal OpenOnCompletion As Boolean)

' get a fileinfo object to check the folder validity

Dim OutFile As New IO.FileInfo(OutputFileName)

If OutFile.Directory.Exists Then

' hide the photoshop UI

If HideApplication Then PSDapp.Visible = False

' Suppress the Photoshop dialogs (if any)

PSDapp.DisplayDialogs = 3

'   enum 3 =

'   PsDialogModes.psDisplayNoDialogs()

'load the first image and duplicate it into a new image document

If Images.Length > 0 Then

Dim BackGroundLayer As String = Images(0)

PSDapp.Load(BackGroundLayer)

PSDapp.ActiveDocument = PSDapp.Documents.Item(1)

PSDapp.ActiveDocument.Duplicate(OutFile.Name)

'close the original

'close(1) bypasses the save dialog

PSDapp.Documents.Item(1).Close(1)

'set the active document to the duplicate

PSDapp.ActiveDocument = PSDapp.Documents.Item(1)

'loop through the rest of the array, copying and pasting it into the

'duplicated document

For imageindex = LBound(Images) + 1 To UBound(Images)

PSDapp.Load(Images(imageindex))

PSDapp.ActiveDocument = PSDapp.Documents.Item(2)

PSDapp.ActiveDocument.Selection.SelectAll()

PSDapp.ActiveDocument.ArtLayers(1).Copy()

PSDapp.Documents.Item(2).Close(1)

PSDapp.ActiveDocument = PSDapp.Documents.Item(1)

'Select the point at which to paste the layer

Try

Dim Location As Drawing.Point = CType(Placement(imageindex), Drawing.Point)

Dim Marquee As Object = SelectionMarquee(New Drawing.Rectangle(Location.X, Location.Y, 1, 1))

PSDapp.ActiveDocument.Selection.Select(Marquee)

Catch ex As Exception

End Try

PSDapp.ActiveDocument.Paste()

Next

'save it

PSDapp.ActiveDocument.SaveAs(OutputFileName)

PSDapp.ActiveDocument.Close(1)

' show the ui and load the image, if desired

If HideApplication Then PSDapp.Visible = True

If OpenOnCompletion Then PSDapp.Load(OutputFileName)

End If

Else

' if the folder doesn't exist it will throw this exception

Throw New IO.DirectoryNotFoundException("Directory does not exist")

End If

End Sub

So now you need to change the code you call from 3dsMax to include a placement argument. This is just an array of dotnet point objects.

oPhotoShop = dotnetobject "LoneRobot.Imaging.PhotoshopBot"

imagepath = @"C:\LoneRobot Script Development\PhotoShopInterop\testimages"

imagearray = #((imagepath + "\bg.jpg"),(imagepath + "\layer1.png"),(imagepath + "\layer2.png"),(imagepath + "\layer3.png"),(imagepath + "\layer4.png"))

LocationArray = #((DotNetObject "System.Drawing.Point" 0 0),(DotNetObject "System.Drawing.Point" 411 513),(DotNetObject "System.Drawing.Point" 109 531),(DotNetObject "System.Drawing.Point" 193 444),(DotNetObject "System.Drawing.Point" 314 421))

oPhotoShop.BasicImageCompositor imagearray LocationArray (imagepath +@"\testoutput.psd") false true

The first point in the array is (0,0) – This is because it represents the Background layer position, so is disregarded by the loop in the class.

What do we get now?

An automated composition with layer placement, all from within 3dsMax!

Huzzah!

I hope this tutorial has been helpful and you can see the potential of being able to use this within 3DSmax. You really can automate just about everything, without even going near the photoshop interface. I’m quite excited about the possibilities of this, so I will perhaps write more in the future.

Further reading -

http://www.adobe.com/devnet/photoshop/scripting/

The download package has the assemblies and the VB version of the class itself. If you want the test images, just leech them from here. Cheers!

Update!

I have updated the wrapper with some more methods listed below. There are a few things you can now change with this class. A simple showmethods from 3ds max will give you all the info you need to use these functions. The enums are used within a few of the functions, like setting the blend modes. It’s just an easier way to specify these.

I would advise always using the function setactivelayer before any code, as it is a boolean function that returns true on setting the correct layer, and false if it doesn’t exist.

If you’ve got this far then you know what you are doing anyway! Enjoy!

Lastest Update!

I’ve added a whole load of classes exposing the save options for various other photoshop image formats. You can now save out to the following image types with this assembly -

  • .bmp
  • .jpg
  • .tga
  • .png
  • .eps
  • .psd
  • .gif
  • .tiff

Download at the bottom as usual!

New Update!

Just managed to get a handle on programming the Action Descriptor using VB. This is reasonably straight forward, you can use the Adobe Scripting Listener to record the actions. It’s amost the same as VB, as it is VBscript. This has basically allowed me to now integrate pretty much any action or command into the assembly. You could make a highly complex series of commands and wrap it into a fuction and call it straight from 3dsMax. Take a look at the new methods – there are many not exposed by the Adobe Object Library. If you think of a useful action or operation you’d like to see in this assembly, let me know below and i’ll try to integrate it for you.

download script

WIDGETS NEEDED!

Go ahead and add some widgets here! Admin > Appearance > Widgets