Unreal Engine 4 Custom Shaders Tutorial

In this Unreal Engine 4 tutorial, you will learn how to create custom shaders using HLSL By Tommy Tran.

4.7 (23) · 2 Reviews

Download materials
Save for later
Share

The material editor is a great tool for artists to create shaders thanks to its node-based system. However, it does have its limitations. For example, you cannot create things such as loops and switch statements.

Luckily, you can get around these limitations by writing your own code. To do this, you can create a Custom node which will allow you to write HLSL code.

In this tutorial, you will learn how to:

  • Create a Custom node and set up its inputs
  • Convert material nodes to HLSL
  • Edit shader files using an external text editor
  • Create HLSL functions

To demonstrate all of this, you will use HLSL to desaturate the scene image, output different scene textures and create a Gaussian blur.

The tutorial also assumes you are familiar with a C-type language such as C++ or C#. If you know a syntactically similar language such as Java, you should still be able to follow along.

Note: This tutorial assumes you already know the basics of using Unreal Engine. If you are new to Unreal Engine, check out our 10-part Unreal Engine for Beginners tutorial series.

The tutorial also assumes you are familiar with a C-type language such as C++ or C#. If you know a syntactically similar language such as Java, you should still be able to follow along.

Note: This tutorial is part of a 4-part tutorial series on shaders in Unreal Engine:

Getting Started

Start by downloading the materials for this tutorial (you can find a link at the top or bottom of this tutorial). Unzip it and navigate to CustomShadersStarter and open CustomShaders.uproject. You will see the following scene:

unreal engine shaders

First, you will use HLSL to desaturate the scene image. To do this, you need to create and use a Custom node in a post process material.

Creating a Custom Node

Navigate to the Materials folder and open PP_Desaturate. This is the material you will edit to create the desaturation effect.

unreal engine shaders

First, create a Custom node. Just like other nodes, it can have multiple inputs but is limited to one output.

unreal engine shaders

Next, make sure you have the Custom node selected and then go to the Details panel. You will see the following:

unreal engine shaders

Here is what each property does:

  • Code: This is where you will put your HLSL code
  • Output Type: The output can range from a single value (CMOT Float 1) up to a four channel vector (CMOT Float 4).
  • Description: The text that will display on the node itself. This is a good way to name your Custom nodes. Set this to Desaturate.
  • Inputs: This is where you can add and name input pins. You can then reference the inputs in code using their names. Set the name for input 0 to SceneTexture.

unreal engine shaders

To desaturate the image, replace the text inside Code with the following:

return dot(SceneTexture, float3(0.3,0.59,0.11));
Note: dot() is an intrinsic function. These are functions built into HLSL. If you need a function such as atan() or lerp(), check if there is already a function for it.

Finally, connect everything like so:

unreal engine shaders

Summary:

  1. SceneTexture:PostProcessInput0 will output the color of the current pixel
  2. Desaturate will take the color and desaturate it. It will then output the result to Emissive Color

Click Apply and then close PP_Desaturate. The scene image is now desaturated.

unreal engine shaders

You might be wondering where the desaturation code came from. When you use a material node, it gets converted into HLSL. If you look through the generated code, you can find the appropriate section and copy-paste it. This is how I converted the Desaturation node into HLSL.

In the next section, you will learn how to convert a material node into HLSL.

Converting Material Nodes to HLSL

For this tutorial, you will convert the SceneTexture node into HLSL. This will be useful later on when you create a Gaussian blur.

First, navigate to the Maps folder and open GaussianBlur. Afterwards, go back to Materials and open PP_GaussianBlur.

unreal engine shaders

Unreal will generate HLSL for any nodes that contribute to the final output. In this case, Unreal will generate HLSL for the SceneTexture node.

To view the HLSL code for the entire material, select Window\HLSL Code. This will open a separate window with the generated code.

unreal engine shaders

Note: If the HLSL Code window is blank, you need to enable Live Preview in the Toolbar.
unreal engine shaders

Since the generated code is a few thousand lines long, it’s quite difficult to navigate. To make searching easier, click the Copy button and paste it into a text editor (I use Notepad++). Afterwards, close the HLSL Code window.

Now, you need to find where the SceneTexture code is. The easiest way to do this is to find the definition for CalcPixelMaterialInputs(). This function is where the engine calculates all the material outputs. If you look at the bottom of the function, you will see the final values for each output:

PixelMaterialInputs.EmissiveColor = Local1;
PixelMaterialInputs.Opacity = 1.00000000;
PixelMaterialInputs.OpacityMask = 1.00000000;
PixelMaterialInputs.BaseColor = MaterialFloat3(0.00000000,0.00000000,0.00000000);
PixelMaterialInputs.Metallic = 0.00000000;
PixelMaterialInputs.Specular = 0.50000000;
PixelMaterialInputs.Roughness = 0.50000000;
PixelMaterialInputs.Subsurface = 0;
PixelMaterialInputs.AmbientOcclusion = 1.00000000;
PixelMaterialInputs.Refraction = 0;
PixelMaterialInputs.PixelDepthOffset = 0.00000000;

Since this is a post process material, you only need to worry about EmissiveColor. As you can see, its value is the value of Local1. The LocalX variables are local variables the function uses to store intermediate values. If you look right above the outputs, you will see how the engine calculates each local variable.

MaterialFloat4 Local0 = SceneTextureLookup(GetDefaultSceneTextureUV(Parameters, 14), 14, false);
MaterialFloat3 Local1 = (Local0.rgba.rgb + Material.VectorExpressions[1].rgb);

The final local variable (Local1 in this case) is usually a "dummy" calculation so you can ignore it. This means SceneTextureLookup() is the function for the SceneTexture node.

Now that you have the correct function, let’s test it out.

Using the SceneTextureLookup Function

First, what do the parameters do? This is the signature for SceneTextureLookup():

float4 SceneTextureLookup(float2 UV, int SceneTextureIndex, bool Filtered)

Here is what each parameter does:

  • UV: The UV location to sample from. For example, a UV of (0.5, 0.5) will sample the middle pixel.
  • SceneTextureIndex: This will determine which scene texture to sample from. You can find a table of each scene texture and their index below. For example, to sample Post Process Input 0, you would use 14 as the index.
  • Filtered: Whether the scene texture should use bilinear filtering. Usually set to false.

unreal engine shaders

To test, you will output the World Normal. Go to the material editor and create a Custom node named Gaussian Blur. Afterwards, put the following in the Code field:

return SceneTextureLookup(GetDefaultSceneTextureUV(Parameters, 8), 8, false);

This will output the World Normal for the current pixel. GetDefaultSceneTextureUV() will get the UV for the current pixel.

This is an example of how custom HLSL can break between versions of Unreal.

Note: Before 4.19, you were able to get UVs by supplying a TextureCoordinate node as an input. In 4.19, the correct way is to use GetDefaultSceneTextureUV() and supply your desired index.

This is an example of how custom HLSL can break between versions of Unreal.

Next, disconnect the SceneTexture node. Afterwards, connect Gaussian Blur to Emissive Color and click Apply.

unreal engine shaders

At this point, you will get the following error:

[SM5] /Engine/Generated/Material.ush(1410,8-76):  error X3004: undeclared identifier 'SceneTextureLookup'

This is telling you that SceneTextureLookup() does not exist in your material. So why does it work when using a SceneTexture node but not in a Custom node? When you use a SceneTexture, the compiler will include the definition for SceneTextureLookup(). Since you are not using one, you cannot use the function.

Luckily, the fix for this is easy. Set the SceneTexture node to the same texture as the one you are sampling. In this case, set it to WorldNormal.

Afterwards, connect it to the Gaussian Blur. Finally, you need to set the input pin’s name to anything besides None. For this tutorial, set it to SceneTexture.

unreal engine shaders

Note: As of writing, there is an engine bug where the editor will crash if the scene textures are not the same. However, once it works, you can freely change the scene texture in the Custom node.

Now the compiler will include the definition for SceneTextureLookup().

Click Apply and then go back to the main editor. You will now see the world normal for each pixel.

unreal engine shaders

Right now, editing code in the Custom node isn’t too bad since you are working with little snippets. However, once your code starts getting longer, it becomes difficult to maintain.

To improve the workflow, Unreal allows you to include external shader files. With this, you can write code in your own text editor and then switch back to Unreal to compile.

Tommy Tran

Contributors

Tommy Tran

Author

Over 300 content creators. Join our team.