Unreal Engine 4 Paint Filter Tutorial

As time passes, video games continue to look better and better. And in an era of video games with amazing visuals, it can be hard to make your game stand out. A way to make your game’s aesthetic more unique is to use non-photorealistic rendering.

Non-photorealistic rendering encompasses a wide range of rendering techniques. These include but are not limited to cel shading, toon outlines and cross hatching. You can even make your game look more like a painting! One of the techniques to accomplish this is Kuwahara filtering.

To implement Kuwahara filtering, you will learn how to:

• Calculate mean and variance for multiple kernels
• Output the mean of the kernel with lowest variance
• Use Sobel to find a pixel’s local orientation
• Rotate the sampling kernels based on the pixel’s local orientation

Since this tutorial uses HLSL, you should be familiar with it or a similar language such as C#.

• Part 1: Cel Shading
• Part 2: Toon Outline
• Part 3: Custom Shaders Using HLSL
• Part 4: Paint Filter (you are here!)

Getting Started

Start by downloading the materials for this tutorial (you can find a link at the top or bottom of this tutorial). Unzip it and navigate to PaintFilterStarter and open PaintFilter.uproject. You will see the following scene:

To save time, the scene already contains a Post Process Volume with PP_Kuwahara. This is the material (and its shader files) you will be editing.

To start, let’s go over what the Kuwahara filter is and how it works.

Kuwahara Filter

When taking photos, you may notice a grainy texture over the image. This is noise and just like the noise coming from your loud neighbors, you probably don’t want to see or hear it.

A common way to remove noise is to use a low-pass filter such as a blur. Below is the noise image after box blurring with a radius of 5.

Most of the noise is now gone but all the edges have lost their hardness. If only there was a filter that could smooth the image and preserve the edges!

As you might have guessed, the Kuwahara filter meets these requirements. Let’s look at how it works.

How Kuwahara Filtering Works

Like convolution, Kuwahara filtering uses kernels but instead of using one kernel, it uses four. The kernels are arranged so that they overlap by one pixel (the current pixel). Below is an example of the kernels for a 5×5 Kuwahara filter.

First, you calculate the mean (average color) for each kernel. This essentially blurs the kernel which has the effect of smoothing out noise.

For each kernel, you also calculate the variance. This is basically a measure of how much a kernel varies in color. For example, a kernel with similar colors will have low variance. If the colors are dissimilar, the kernel will have high variance.

Finally, you find the kernel with the lowest variance and output its mean. This selection based on variance is how the Kuwahara filter preserves edges. Let’s look at a few examples.

Kuwahara Filtering Examples

Below is a 10×10 grayscale image. You can see that there is an edge going from the bottom-left to the top-right. You can also see that some areas of the image have noise.

First, select a pixel and determine which kernel has the lowest variance. Here is a pixel near the edge and its associated kernels:

As you can see, kernels lying on the edge have varying colors. This indicates high variance and means the filter will not select them. By not selecting kernels lying on an edge, the filter avoids the problem of blurred edges.

For this pixel, the filter will select the green kernel since it is the most homogeneous. The output will then be the mean of the green kernel which is a color close to black.

Here’s another edge pixel and its kernels:

This time the yellow kernel has the least variance since it’s the only one not on the edge. So the output will be the mean of the yellow kernel which is a color close to white.

Below is a comparison between box blurring and Kuwahara filtering — each with a radius of 5.

As you can see, Kuwahara filtering does a great job at smoothing and edge preserving. In this case, the filter actually hardened the edge!

Incidentally, this edge-preserving smoothing feature can give an image a painterly look. Since brush strokes generally have hard edges and low noise, the Kuwahara filter is a great choice for converting realistic images to a painterly style.

Here is the result of running a photo through Kuwahara filters of varying size:

It looks pretty good, doesn’t it? Let’s go ahead and start creating the Kuwahara filter.

Creating the Kuwahara Filter

For this tutorial, the filter is split into two shader files: Global.usf and Kuwahara.usf. The first file will store a function to calculate a kernel’s mean and variance. The second file is the filter’s entry point and will call the aforementioned function for each kernel.

First, you will create the function to calculate mean and variance. Open the project folder in your OS and then go to the Shaders folder. Afterwards, open Global.usf. Inside, you will see the GetKernelMeanAndVariance() function.

Before you start building the function, you will need an extra parameter. Change the function’s signature to:

float4 GetKernelMeanAndVariance(float2 UV, float4 Range)

To sample in a grid, you need two for loops: one for horizontal offsets and another for vertical offsets. The first two channels of Range will contain the bounds for the horizontal loop. The second two will contain the bounds for the vertical loop. For example, if you are sampling the top-left kernel and the filter has a radius of 2, Range would be:

Range = float4(-2, 0, -2, 0);

Now it’s time to start sampling.