Press Button, Drink Coffee : An Overview of UE4 build pipeline and maintenance

「Press Button, Drink Coffee」
An overview of
UE4 build pipeline and maintenance

#UE4 | @UNREALENGINE
Introduction
Axel Riffard
Epic Games Japan
Support Engineer
● French 🇫🇷
● Twitter : @AxRiff
● Hobbies : Whisky, Cats and Disneyland
● Currently Playing: Dragon Quest XI, COD Mobile, The Witcher III (Switch)

FIRST OF ALL …

HELL YEAH POLAND !

Ruiner !
Frostpunk !
Dead Island !
Dying Light !
The Vanishing of Ethan Carter !
And …

Geralt of Rivia !
<3

Ok, back on topic

Disclaimer
There is no such thing as THE perfect pipeline.
Rely on experience ! In this session, I will only talk to you about mine and
give you a few best practices. But donʼt trust me, trust yourself !

Agenda
●CI : What is it ?
●UnrealAutomationTool
●Building the Engine and your Project
●The other stuff
●The Unreal Way

Dev
2
Dev
1
Dev
3
Versioning
System
Why ?

● Local build breaks after updating
● Finding the bad commit wastes time
● Communication issues, fights in the team
● Non technical members are scared to commit things
● Building locally every time is a waste
Consequences

CI Software
CI = Continuous Integration
A combination of software and hardware allowing to merge the team source
code and assets several times every day.

Why ?
Dev
2
Dev
1
Dev
3
Versioning
System

CI Software
In 2019, there is no bad choice.
I have worked with both TeamCity and Jenkins, and was satisfied with
both of them

CI Software
Jenkins TeamCity

Jenkins
Merits︓
● Free Open Source
● Plenty of documentation and plugins
● Huge Community
Drawbacks︓
● Commercial Support exists but is complicated
● Some plugins are unreliable

TeamCity
Merits :
● Building pipelines is crazy fast
● Technical support is great
Drawbacks :
● Limitations on the Free version

CI Software
Every advice can be applied to TeamCity, or any CI solution.
Also, totally unrelated, but this cat looks like Jenkins mascot, so Iʼm going
to use him instead.

Build Machine
Master → Manages the tasks.
Slave → Does the tasks.

Build Machine
VersioningDEV PC
CI Server
Master
Slack
Commit
Slave 1 Slave 2 Slave 3
Company
Server
Poll
Distribute
Versioning

Build Machine – How many
Prototyping Stage :
● Just the Master
Production Stage :
● One Master
● One slave for every release platform

Build Machine - OS
Master︓Win64 or Linux
Slave︓Win64（for platforms SDK), MacOS for Apple Platforms

Build Machine - Windows
Storage︓SSD
Memory ︓32GB or more
CPU︓i7 / i9

Build Machine - MacOS
No C++ and Shader compiling distribution on Mac,
So get as many cores as you can for your CPU.
If itʼs just for a build machine, no need for GPU.
High spec Mac Mini is good enough.

Agenda
●Continuous Integration : What is it ?
●UnrealAutomationTool
●Building the Engine and your Project
●The other stuff
●The Unreal Way

Building in UE4
● Engine and Editor
● Game sources
● Game Assets (lights, shaders, etc.)
● Tools

Build Commands
Find all the command lines here
https://github.com/AxRiff/UnrealEngineJenkins

UnrealAutomationTool
UnrealAutomationTool is a complete system to build and test your games
inside of Unreal Engine
UE4 is mainly written in C++, but UnrealAutomationTool is made in C#

Depending on your company size, type of project, workflow, etc., you can
use different parts of UnrealAutomationTool
● BuildGraph Custom Script
● BuildGraph InstalledBuild Script
● BuildCookRun Command Line Tool

BuildGraph

BuildGraph
BuildGraph is a script based build automation system inside of
● XML
● By Epic for Epic
● Original approach

BuildGraph
Youʼve got plenty of sample codes in
Engine/Build/Graph/Examples/

BuildGraph
Merits :
● Reduce dependency on CI framework.
● Work on it locally -> No more Jenkins master
● XML Based : easy to read and write
● Platform Agnostic
● Can be shared between projects
Drawbacks :
● Steep learning curve at first

BuildGraph
● Agents: Group of nodes being executed on the same build machine
● Nodes: Sequence of Tasks
● Tasks: Build Process Step (Updating, Compiling, etc)

BuildGraph
<Agent Name="Default Agent" Type="CompileWin64">

<Node Name="Update Version Files">
<SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/>
</Node>

<Node Name="Compile Tools Win64" Requires="Update Version Files" Produces="#ToolBinaries">
<Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/>
</Node>

<Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" Produces="#EditorBinaries">
<Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/>
</Node>

<Property Name="GameBinaries" Value=""/>
<ForEach Name="GameTarget" Values="$(GameTargets)">
<ForEach Name="TargetPlatform" Values="$(TargetPlatforms)">
<Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)">
<Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/>
</Node>
<Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/>
</ForEach>
</ForEach>
</Agent>

BuildGraph
</Node>
</Node>
</Node>
<Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64"
Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)">
<Compile Target="$(GameTarget)" Platform="$(TargetPlatform)"
Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/>
</Node>
</ForEach>
</ForEach>
</Agent>

BuildGraph
</Node>
</Node>
</Node>
</Node>
</ForEach>
</ForEach>
</Agent>
<SetVersion Change="$(Change)" Branch="$(EscapedBranch)"
Licensee="$(Licensee)" If="$(Versioned)"/>
</Node>

BuildGraph
</Node>
</Node>
</Node>
</Node>
</ForEach>
</ForEach>
</Agent>
<Node Name="Compile Tools Win64" Requires="Update Version Files"
Produces="#ToolBinaries">
<Compile Target="UnrealHeaderTool" Platform="Win64"
Configuration="Development" Tag="#ToolBinaries"/>
</Node>

BuildGraph
</Node>
</Node>
</Node>
</Node>
</ForEach>
</ForEach>
</Agent>
<Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64"
Produces="#EditorBinaries">
<Compile Target="$(EditorTarget)" Platform="Win64"
Configuration="Development" Tag="#EditorBinaries"/>
</Node>

BuildGraph
</Node>
</Node>
</Node>
</Node>
</ForEach>
</ForEach>
</Agent>
<Node Name="Compile $(GameTarget) $(TargetPlatform)"
Requires="Compile Tools Win64"
Configuration="Development"
Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/>
</Node>
<Property Name="GameBinaries"
Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/>
</ForEach>
</ForEach>

InstalledBuild

InstalledBuild
● Think of it as your own Unreal Launcher
● Created by Epic to deploy a non editable engine
Engine/Build/InstalledEngineBuild.xml
● Easy to use for Designers and Artists
● Engine build only. Canʼt use it to build a project.

InstalledBuild
Installed Build mains steps
1. Update Version File
2. Compile UnrealHeaderTool Win64
3. Compile UE4Editor Win64
4. Compile UE4Game Win64
5. Compile UE4Game (Platform)
6. Build Tools Win64
7. Build Tools CS
8. Make Feature Packs
9. Build DDC Win64

InstalledBuild
Engine/Build/BatchFiles/RunUAT.bat BuildGraph
-target”Make Installed Build Win64”
-script=/Engine/Build/InstalledEngineBuild.xml
-clean

BuildCookRun

BuildCookRun
● Script located at /Engine/Build/BatchFiles/
● From build to package, do anything
● Highly customizable
● Command line based : nostalgic feeling ! <3

BuildCookRun, BuildGraph, InstalledBuild
● Depending on the situation, the best choice will differ
● I think adaptability is the most important thing in game dev, so I
personally use BuildCookRun most of the time.

Command Lines

Engine DLL Build
If you donʼt edit the Engine a lot, trigger it manually
If you extend the Engine a lot, do it once a day. Morning or Lunch Time
is preferred

Build Machine
Jenkins Build
Commit to
Source Control
Update
Notify

Build Command
%ENGINE_ROOT%/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -
project=Samples/StarterContent/StarterContent.uproject -
platform=%ENGINE_PLATFORM_TARGET% -
clientconfig=%ENGINE_COMPILATION_TARGET% -build
ENGINE_ROOT=EngineRoot
ENGINE_COMPILATION_TARGET=Development
ENGINE_PLATFORM_TARGET=Win64
Yup, thatʼs a little hard to read

Build Command
Engine/Build/BatchFiles/RunUAT.bat BuildCookRun
-project=Samples/StarterContent/StarterContent.uproject
-platform=Win64
-clientconfig=Development
-build

Build Command
-project=Samples/StarterContent/StarterContent.uproject
-platform=Win64
-build
You need a uproject to use RunUAT. You
can use a sample project or blank one !

Project DLL build
Poll every 10 minutes to build
Jenkins Build
Commit to
Source Control
Update
Notify

Build Command
%ENGINE_ROOT%/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun
-project= project=%GAME_SUB_PATH%/%GAME_NAME%.uproject -
clientconfig=%ENGINE_COMPILATION_TARGET% -build
ENGINE_ROOT=EngineRoot
ENGINE_COMPILATION_TARGET=Development
ENGINE_PLATFORM_TARGET=Win64
GAME_SUB_PATH=C:/GamePath
GAME_NAME=GameName

Build Command
-project=C:/PathToTheGame/GameName.uproject
-platform=Win64
-build

Packaging

Nightly Build
● Well, itʼs build every night.
● Development, Test（for profiling), Shipping for every release platform.
Time doesnʼt matter, so clean build

Nightly Build Flow
ZipJenkins Build
Copy to
Shared Folder
Update
Notify

Build Command
%ENGINE_ROOT%/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -
project=%GAME_SUB_PATH%/%GAME_NAME%.uproject -noP4 -
clientconfig=%ENGINE_COMPILATION_TARGET% -cook -allmaps -build -stage
-pak -archive -archivedirectory=%TEMP_DIR%
ENGINE_ROOT=PathToYourEngine
GAME_SUB_PATH=PathToYourProject
GAME_NAME=UprojectFileWithoutTheExtension
ENGINE_COMPILATION_TARGET=Development|Shipping|etc
ENGINE_PLATFORM_TARGET=Win64|etc

Build Command
-project=GamePath/GameName.uproject
-noP4
-platform=Win64
-cook
-allmaps
-build
-stage
-pak
-archive
-archivedirectory=C:/TempDirectory/

Build Command
-noP4
-platform=Win64
-cook
-allmaps
-build
-stage
-pak
-archive
Argument to zip all the assets in one “pak” file

Build Command
robocopy %ARCHIVE_DIR%/latest %ARCHIVE_DIR%/%ENGINE_PLATF
ORM_TARGET%/archive /E /IS /MOV
set PATH=%PATH%;C:/Program Files/7-Zip/
7z.exe a -tzip -mmt -
mx1 %ARCHIVE_DIR%/latest/%BUILD_NUMBER%.zip %TEMP_DIR%/
exit 0

Build Command
Wait a sec !!

Build Command
https://github.com/electronicarts/EACopy
Twitter : @honk_dice
EA DICE Studios has open sourced EACopy, a faster alternative to
robocopy !
Give it a try !!!

Incremental Build
● Manual Trigger
● For those “Hey mate, I need to show the build in 40 minutes to our
investor / boss / publisher, do smth plz lolwut”
● Contrary to the nightly build, faster is better so …
● Use -partialgc
● Donʼt make it clean

Incremental Build Flow
Jenkins Build
Send to
DevKit
Update
Notify
For DevKit deployment, check your platform
But you might get an idea
/Engine/Source/Programs/AutomationTool/PlatformName/PlatformNamePlatfo
rm.Automation.cs

Build Command
-noP4
-platform=Win64
-cook
-allmaps
-build
-stage
-pak
-archive
-partialgc
Instead of cleaning the memory when you run out of it, it will free it
up at the end of the packaging process

Jenkins Plugins

Plugins
Build Pipeline Plugin
https://wiki.jenkins.io/display/JENKINS/Build+Pipeline+Plugin

Plugins
Jenkins Slack Plugin
https://github.com/jenkinsci/slack-plugin

Plugins
Retaliation
https://github.com/codedance/Retaliation

Other CI Jobs

Commandlet
Command Line Applet is a command program to be run inside of Unreal
Engine environment.

Shared DDC
Derived Data Cache (DDC) is a feature to keep assets that were
prepared for every platform.
To make it simple, itʼs an asset caching feature.

Shared DDC
Shared
FolderDev PC 1
Copy Built Asset
Build Assets on
1st launch

Shared DDC
Shared
Folder
Dev PC 5Dev PC 4
Dev PC 3Dev PC 2
Copy Built Asset

Shared DDC
Shared
FolderDev PC 4
Modify / Add
Asset
Copy Built Asset

Shared DDC
https://docs.unrealengine.com/en-
US/Engine/Basics/DerivedDataCache/index.html
BaseEngine.ini
[InstalledDerivedDataBackendGraph]
MinimumDaysToKeepFile=7
Root=(Type=KeyLength, Length=120, Inner=AsyncPut)
AsyncPut=(Type=AsyncPut, Inner=Hierarchy)
Hierarchy=(Type=Hierarchical, Inner=Boot, Inner=Pak, Inner=EnginePak, Inner=Local, Inner=Shared)
Boot=(Type=Boot, Filename=%GAMEDIR%DerivedDataCache/Boot.ddc, MaxCacheSize=256)
Local=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, PurgeTransient=true, DeleteUnused=true,
UnusedFileAge=17, FoldersToClean=-1, Path=../../../Engine/DerivedDataCache)
Shared=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=19,
FoldersToClean=-1, Path=//mystudio.net/DDC, EnvPathOverride=UE-SharedDataCachePath)
AltShared=(Type=FileSystem, ReadOnly=true, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=19,
FoldersToClean=-1, Path=//mystudio.net/DDC2, EnvPathOverride=UE-SharedDataCachePath2)
Pak=(Type=ReadPak, Filename=%GAMEDIR%DerivedDataCache/DDC.ddp)
EnginePak=(Type=ReadPak, Filename=../../../Engine/DerivedDataCache/DDC.ddp)

Argument Type Comment
ReadOnly Bool Make it false when Shared
Clean Bool Delete files that went over UnusedFileAge limit
Flush Bool Delete all DDC
DeleteUnused Bool Il false, never deletes old assets (overrides UnusedFileAge and Clean)
UnusedFileAge Int Limit of days after which an asset will be deleted, Fortnite sets it to 10.
FoldersToClean Int Max limit of folders to be flushed each time. ０ or -1 means no limit.
Shared DDC

Shared DDC
// Flush and Clean argument DerivedDataBackends.cpp:553
if( bFlush )
{
IFileManager::Get().DeleteDirectory( *(Path / TEXT(“”)), false, true );
}
else if( bClean )
{
DeleteOldFiles( *Path );
}
/** Delete the old files in a directory **/
void DeleteOldFiles(const TCHAR* Directory)
{
float MinimumDaysToKeepFile = 7;
GConfig->GetFloat( *GraphName, TEXT("MinimumDaysToKeepFile"), MinimumDaysToKeepFile, GEngineIni );
check(MinimumDaysToKeepFile > 0.0f); // sanity
//@todo
}

Shared DDC
//@todo

Shared DDC
UE4Editor-cmd.exe ProjectName –run=DerivedDataCache -fill
del /s /q c:/ServerShareDDCPath/*

Lighting Build
● All lights except the ones calculated in real time are to be built.
● Of course, Epic is providing the tools for it.
But …

Lighting Build

Itʼs SUPER heavy and slow
Let your CI pipe do it !
Lighting Build

Lighting Build
UE4Editor-cmd.exe ProjectName -run=resavepackages
-buildlighting -allowcommandletrendering

Lighting Build
There are times you want to check the lights in the editor / game
In this case, build locally, and Swarm is your new BFF

Lighting Build - Swarm
With Swarm, you can distribute the task of
building lights between the computers of your
team.
You will need a Coordinator. Your Jenkins master
should do the trick.
https://wiki.unrealengine.com/Swarm_Agent_Trou
bleshooting

Engine/Binaries/DotNET/SwarmCoordinator.exe
Engine/Binaries/DotNET/SwarmAgent.exe
To add it at startup,
Create a shortcut the the Agent and copy it here.
%APPDATA%/Microsoft/Windows/Start Menu/Programs/Startup

Redirectors
● When you move or rename an asset in the editor, a Redirector is created to
know the ”new address”
● Super convenient feature but it has a tendency to bug, so you will need to
clean them up frequently
● Once a month is fine, but do it once a week in the critical phases of the
project (alpha, beta, gold, patch release)
https://docs.unrealengine.com/en-US/Engine/Basics/Redirectors/index.html

Redirectors
UE4Editor-cmd.exe ProjectName -run=fixupredirects

The Unreal Way :
Fortnite

Fortnite
● Only engineers have Visual Studio
● We are building with BuildGraph managed by ElectricCommander
● We are heavily using UnrealGameSync

Iteration
Iteration can seem complicated without Visual Studio.
But actually, itʼs pretty manageable.
● Heavily rely on PIE for testing
● Leverage your CI software properly

Testing a Level on Device
So letʼs say your artist has just changed a characterʼs hair color and
wants to see how it looks on the release platform screen before
committing…
First, create an incremental build task on your CI software, and commit
the binaries to Project/Binaries/PlatformName

Custom Launch Profile
Then in the editor, click on the triangle on
the right side of of “Launch”, scroll down
and create a new Custom Launch Profile
Stored in
/Engine/Programs/UnrealFrontend/Profiles
In the profile, untick the Build step, and
then you can deploy the currently edited
level on the machine connected to the
computer, even without Visual Studio.
Magic !!

Build Machine Jobs
What I think a UE4 build pipeline should have :
● Engine Build
● Project Build
● Nightly Package
● Incremental Package
● Incremental Package for the Device Launcher
● Lightmaps Build
● Fixup Redirectors Command
● SharedDDC Cleanup Command
● SharedDDC Fillup Command
Any suggestion on what to add ? Tell me in Q&A !!!

Versioning
● Epic uses Perforce. If you want to be as close as possible to our
workstyle, you know what to do
● SVN might be a little slow, but it is very reliable and still heavily used
● GIT is free, modern design choices and is very used for mobile and
social games (in Japan !)
SVN and Perforce have Editor support !
GIT too, but itʼs still experimental !

UnrealGameSync
Made by Epic for Epic. Heavily used on Fortnite
Itʼs Perforce only, and it allows non technical staff not to have to deal with
P4V

UnrealGameSync

UnrealGameSync
P4 Server
Sync
Get
Engine
Local Builds
(Binary Engine)
Build Machine

UnrealGameSync
Get the latest version anytime on GitHub and Perforce.
UGS dev is unrelated to UE4, so always get the latest version even if
youʼve locked your engine version !
https://docs.unrealengine.com/latest/en/Programming/Deployment/Unre
alGameSync/Reference/

https://github.com/gpageot/JenkinsUE4
Twitter : @GregoryPAGEOT
https://github.com/AxRiff/UnrealEngineJenkins
Twitter : @AxRiff
https://github.com/electronicarts/EACopy
Twitter : @honk_dice
Manga by @shiba_zushi, Technical Artist at Epic Games Japan.
Links

Press Button, Drink Coffee : An Overview of UE4 build pipeline and maintenance

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Press Button, Drink Coffee : An Overview of UE4 build pipeline and maintenance

Similar to Press Button, Drink Coffee : An Overview of UE4 build pipeline and maintenance (20)

More from エピック・ゲームズ・ジャパン Epic Games Japan

More from エピック・ゲームズ・ジャパン Epic Games Japan (20)

Recently uploaded

Recently uploaded (20)

Press Button, Drink Coffee : An Overview of UE4 build pipeline and maintenance