Sunday, August 24, 2014

3D Touch Up Painter for Unity

Why a 3D painting add-on for Unity?

Sometimes an artist may work on a model for hours only to discover that there's a minor error in the texture. Other times an artist may create a shader that requires weights to be inputted via vertex color to work properly. Maybe after seeing their final in game scenery they may decide that the model needs look a little more rigid or flatter.

ZBrush Polypaint
All these require that you reopen the model, texture, or both in an outside program. Do your fix, put it back in Unity to look at in the engine to see if you got it right. In particular this can happen a lot when you're using vertex color weights. In comes Unity Painter.

Unity Painter

Unity Painter will be able to easily modify the following.
  1. Vertex Color
  2. Texture Color
  3. Vertex Position
Instead of listing and explaining how painting will work I'll just provide a screen shot of the options I'm working on currently. For now I'm only worried about spherical brushes.

Good Code Follows Good Data

So far I've been doing A LOT of testing. One of my first tests was getting my brush to work just like Photoshop. This was important to me since I the artist to feel as comfortable as possible. This was surprisingly annoying. At first glance you'd think the hardness falloff would be linear.
Brush with hardness zero

However, after finding my brush working nothing like Photoshop. I dove deeper and wrote a small program to help me find out what the falloff is. The graph below shows a single color channel go from zero to one as you increase your distance from the center of the brush. As you can see this falloff is not linear, or cubic splined, or even cos interpolated. Instead it follows a natural log falloff that has uses a Perlin smooth step function to smooth it out.

Perlin Smoothed Natural Log Interpolation
Other preliminary tests have shown some other Photoshop weirdness. Such as,
  1. The Photoshop brush GUI gizmo will decrease in size by as much as 80% at zero hardness.
  2. Photoshop will increase its draw radius by as much as 55% at zero hardness.

Start At The End - UV Space to World Space

It's not hard to check whether or not a point is inside a sphere. Therefore its trivial to check whether or not a model's vertex lies inside a spherical brush. However, what about a UV space pixel on a texture? How do I find whether or not I should draw to a pixel on my texture when I'm painting in world space? My basic method is the following with adjustments needing to be made if the model is not in the same pose as it was unwrapped in.
  1. For each triangle find the pixels that it covers in UV space.
  2. For each triangle create a texture to world space matrix and find each pixel's world space coordinates
  3. Create a new data structure of world space positions for each texture pixel.
  4. For each brush stroke use the new data structure to find out which pixel is inside the sphere.

Step 1

We can use Unity's Mesh class to find the mesh's triangles, the triangles vertices, and the vertices corresponding UV. From here you can use barycentric coordinates to test out if your pixel in UV space lies inside the triangle. Click to find out more.

Step 2

We can also use the UV points for each triangle vertex to construct the following UV space triangle equation \[P_{t} = P_{0} + su + tv \,\, for \, s \geq 0,  \, t \geq 0, \,  s + t \leq 1\] \[and\] \[u = P_{1} - P_{0}\] \[v = P_{2} - P_{0}\]

For my method we need to find the weights s and t. Note that \(P_{t} = (x,y) \) and therefore we can say \[x = su_{x} + tv_{x}\] \[y = su_{y} + tv_{y}\] Using the \(x\) and \(y\) components of \(u\) and \(v\) this allows us to construct the following matrix that when reduced will yield us \(s\) and \(t\).
Reduce the matrix to echelon form.

Now in this form it is clear that a new matrix can be constructed from our \(s\) and \(t\). This new matrix allows us to calculate the weights of any pixel that lies in our triangle given its UV coordinates.
The weights \(s\) and \(t\) are important since the weights for the pixel will not change for the triangle even when the triangle in in world space! Assuming that the model is still in its original pose. That means that to calculate the pixel's world space coordinates all we have to do is recalculate the triangle equation using the vertice's world space points, apply the weights for that pixel, and done, you know have its world space point.

Step 3 and 4

Since a texture is just an array we can store each world space position in an an array of equal size and dimension.

The quick and dirty way to do step 4 is to just check every pixel if its inside our spherical brush but this can be painstakingly slow for large textures. Instead we can examine which triangles intersect or lie inside our sphere and then test out only the pixels that belong to those triangles. 


I've already spoke about how the model essentially can't be in the middle of an animation. It needs to be in its original pose since I don't take into account it's bone blend weights.

My second concern has to do with bad unwraps. A bad unwrap can cause textures to be pulled, stretched or squeezed. This happens when the UV triangle doesn't match the ratio's of the model triangle. I haven't fully tested this algorithm out yet but I believe this problem to be purely an unwrap one. If it does end up being a major problem, back to the drawing board.

Ninja edit: I realized on my way to work this morning that it may not be too hard to construct a texel to world space matrix that accounts for triangle ratio changes when you assume the texture lies on the same plane as the triangle.

Roadkill showing texture deformation due to unwrapping

Till next time...

I hope to have this done or mostly done by the next time I post. I'll be sure to post anything interesting I find along the way. In particular I want to write a little more about how to use Unity's Editor.OnPreviewGUI() since there is literally nothing useful out there on it. It's own example throws an error. I've had to use the C# Reflector to figure it out so I'll be writing about that too.