Speaker: Alex Bennée Speaker Bio: Long time systems and embedded developer with a side of Dynamic Binary Translation. Alex started learning to program in the 80s in an era of classic home computers that allowed you to get down and dirty at the system level. After graduating with a degree in Chemistry he's worked on a variety of projects including Fruit Machines, Line Cards, CCTV recorders and point-to-multipoint wireless microwave systems. Since the turn of the century his primary focus has been working with FLOSS platforms, especially Linux. An alumni of Transitive he has a broad experience of cross-platform virtualization as well as a strong background in telecommunications and networking. A keen Emacs user he will happily answer questions and proselytise for the One True Editor (tm). Talk Title: Setting up an SVE developer environment Talk Abstract: ARM's SVE instructions provide a whole new set of features for HPC computing. Engineers are understandably keen to start making sure their software is ready to take advantage as soon as hardware arrives. Up until now that has meant building and running in slow system emulation models. Fortunately the latest QEMU now supports SVE instruction in it's Linux user-emulation mode. Alex will talk through how to set this up so building and testing your SVE enabled code is as easy as running make.

1. Alex Bennée
HPC Workshop, 26th July 2018
Setting up for SVE Development

2. Agenda
● Introductions
● Building SVE programs
● Running & Debugging SVE programs with QEMU
● Setting up a SVE development environment with QEMU

3. Introduction
● Alex Bennée
● Linaro Virtualization Engineer
● QEMU TCG, KVM
● alex.bennee@linaro.org
● IRC: ajb-linaro/stsquad

4. ARM’s Scaleable Vector Extensions
●
●
○
●
○
○
○
●
○
○
○

5. Compilers
● GCC-8
○ Free and Open Source
○ -march=armv8.2-a+sve
● ARM HPC Compiler
○ Commercially licensed compiler
○ LLVM based
○ -march=armv8-a+sve

6. Building with SVE enabled
● Environment Variables
○ CFLAGS=”-march=armv8.2-a+sve” ./configure
● Configure Parameters
○ ./configure
--extra-cflags="-march=armv8.2-a+sve"

7. Libraries
● Currently Upstreaming
○ glibc
○ musl
○ bionic
● cortex-strings
○ staging ground for mem/string functions
○ can be LD_PRELOADED
● If important to your workload
○ Build yourself!

8. Running SVE programs
● Real Hardware
● Fast Models
○ Foundation Model
○ Fixed Virtual Platform
● ARMIE
● QEMU 3.0 linux-user

9. What is QEMU?
●
○
○
○
○
●
○
○
○

10. What is QEMU linux-user mode?
●
○
○
○
●
○
○

11. The Tiny Code Generator (TCG)

12. Building QEMU
$ ./configure --target-list=aarch64-linux-user --static
$ make [-jN]

13. Running with QEMU
$ ./tests/Test_simd
Illegal instruction (core dumped)
$ qemu-aarch64 ./tests/Test_simd
...
Current Grid git commit hash=ec9939c1ba8d61dbb2c583bfc2024c8f4baf608c:
(HEAD -> develop, origin/develop, origin/HEAD)
Grid : Message : 809944 usec : Requesting 1073741824 byte stencil comms
buffers

14. QEMU options
● CPU selection
○ -cpu [cortex-a53|cortex-a57|max]
● Debug output
○ -d [in_asm,...]
○ - dfilter 0xstart+0xrange
○ -D filename
● GDB Stub
○ -g 1234
● Help
○ -help
○ -flag help

15. QEMU gdbserver
Run program:
qemu-aarch64 -g 1234 <program>
Attach debugger:
gdb <program> -ex "target remote localhost:1234"
● Currently SVE registers (z/p) not viewable

16. Cross Architecture
Considerations

17. Program Start-up (ELF)
● Loader (INTERP)
● ld-linux.so
● finds and links libraries
● Link Libraries
● libc.so.6
● and many more….
● specific versions (HWCAP)
● Jump to main()

18. Avoiding Library Clashes
● chroot
○ set new /
● containers
○ uses namespaces/control groups
● multi-arch
○ file-system layout

19. Multi-Arch Support
● Builds on multi-lib
● supporting 32/64 bit user-space
● Side-by-side
● Shared Objects (.so libs)
● Headers
● Static libraries
● Versions must align

20. binfmt_misc
● See /proc/sys/fs/binfmt_misc/
● jar
● python
● qemu-FOO
● When kernel can’t run binary
● check magic numbers
● call helper
● System-wide setting
○ Shared with containers

21. Setup binfmt_misc
● Distro specific
● Install qemu-user/qemu-user-binfmt
● use the static packages
● $(QEMU_SRC)/scripts/qemu-binfmt-conf.sh
○ update-binfmts (Debian)
○ systemd
○ direct /proc/sys/fs/binfmt_misc access

22. Docker Builds in QEMU
● Builds on different Distros
○ CentOS, Fedora, Ubuntu, Debian, Travis
○ make docker-test-quick@centos7 J=n
● Cross-builds for different hosts
○ Using Debian Stretch + Multi-Arch Cross Compilers
○ Good for shaking out 64 bit assumptions!
○ make docker-test-build@debian-arm64-cross TARGET_LIST=aarch64-softmmu J=n
● Building TCG test-cases
○ Debian Cross Compilers (Stretch & Sid)
○ make -jN check-tcg
● Building linux-user containers
○ make docker-test-unit@debian-powerpc-user-cross J=n

23. Docker Make Targets in QEMU
● Run test on distro
○ make docker-TEST-SCRIPT@DISTRO J=n
● Run test on all enabled distros
○ make docker-test-unit J=n
○ make docker-test-quick J=n
● Run all tests on all images
○ make docker-all-tests
● Help
○ make docker

24. Docker binfmt targets
● Pre-requisites
○ binfmt_misc setup for /usr/bin/qemu-FOO
○ ./configure --disable-system --static
○ debootstrap/fakeroot installed
● Debootstrap
○ make docker-binfmt-image-debian-FOO DEB_ARCH=ARCH DEB_TYPE=DISTRO
EXECUTABLE=QEMU

25. Building Ubuntu ARM64 image
make docker-binfmt-image-debian-ubuntu-bionic-arm64
DEB_ARCH=arm64 DEB_TYPE=bionic DEB_URL=http://ports.ubuntu.com
EXECUTABLE=./aarch64-linux-user/qemu-aarch64 V=1

26. Running your Ubuntu ARM64 image
$ docker run --rm -it qemu:debian-ubuntu-bionic-arm64 /bin/bash
root@e68be4cb7b0f:/# uname -a
Linux e68be4cb7b0f 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
root@e68be4cb7b0f:/# exit

27. Setup Developer Environment
$ docker run -it qemu:debian-ubuntu-bionic-arm64 /bin/bash
root@c4dc9b5426ad:/# sed -i 's/main/main universe/' /etc/apt/sources.list
root@c4dc9b5426ad:/# apt update
root@c4dc9b5426ad:/# apt install -y gcc-8 g++-8 wget libtool autoconf libtool gdb less
root@c4dc9b5426ad:/# update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 1000
root@c4dc9b5426ad:/# update-alternatives --install /usr/bin/++ g++ /usr/bin/g++-8 1000
root@c4dc9b5426ad:/# gcc --version
gcc (Ubuntu 8-20180414-1ubuntu2) 8.0.1 20180414 (experimental) [trunk revision 259383]
...
root@c4dc9b5426ad:/# exit

28. Save Developer Environment
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
c4dc9b5426ad qemu:debian-ubuntu-bionic-arm64 "/bin/bash" 2 hours ago Exited (0) adoring_goodall
e174632927ba 238519b386bc "/bin/sh" 5 hours ago Exited (0) friendly_mayer
ad33c7bc7558 0da2cdd3455f "/bin/sh" 8 hours ago Exited (0) silly_noether
…
$ docker commit -m "setup arm64 env" adoring_goodall development:bionic-arm64-sve
sha256:25b770e5ce8a5b55ebbccad3b90b58a3474c4acdc5a70ca8ad42fdaf9f273f53

29. Running Development Environment
$ git clone https://git.linaro.org/toolchain/cortex-strings.git cortex-strings.git
$ cd cortex-strings.git
$ docker run --rm -it -u $(id -u) -v $(pwd):$(pwd) -w $(pwd) development:bionic-arm64-sve /bin/bash
user@container:~/cortex-strings.git $ ./autogen.sh
user@container:~/cortex-strings.git $ ./configure --with-sve --enable-static --disable-shared
user@container:~/cortex-strings.git $ make -j

30. Did it work?
user@container:~/cortex-strings.git $ make check -j
PASS: tests/test-memset
…
PASS: tests/test-strcmp
======================================================================
Testsuite summary for cortex-strings 1.1-2012.06~dev
======================================================================
# TOTAL: 12
# PASS: 12

31. Advantages
● “Native” build
○ configure runs as if on real HW
○ no need of “cross” compiles
○ tests work out of the box
● Container
○ Is stable and fixed, works every time
○ Running environment is ephemeral
● Source Code
○ Stays on host file-system
○ Use your own tools/editors independent of container

32. Useful Links
● https://www.linaro.org/blog/sve-in-qemu-linux-user/

33. Thank You!