jarmitage/.bashrc (bela)

## .bashrc (bela)
#!/bin/bash
alias watchdistcc="watch -n 0.4 rm -rf \"~/.distcc/lock/*\""

## .bashrc (host)
#!/bin/bash
alias distccdbela="distccd --verbose --no-detach --daemon --allow 192.168.7.2 --log-level debug --log-file ~/.distcc/distccd.log --enable-tcp-insecure & tail -f ~/.distcc/distccd.log"
alias killdistccd="killall distccd"

## 2018-12-04_cross-compiling.md

      
    Raw
  

              2018-12-04_cross-compiling.md
            
          
  layout
  title
  date
  categories
  description
  image
  author
  
  
  post
  Using the `distcc` distributed compiler with Bela
  2018-12-04
  
  
  Tutorials
  Software
  
  
  Hints on getting faster compile times without a cross-compiler
  bela-and-supercollider-live-coding-sensors/alo-allik.jpg
  giulio
  
  
Compiling programs on the BeagleBone Black, depending on what you are developing and how frequently, can be slow.
Distributed compilation allows you to use a more powerful processor (a host in this article, likely your personal laptop/desktop) to do the grunt work for you.
This article explains how to use the distributed compiler distcc to compile programs on Bela, including Bela programs themselves (although the latter workflow is currently "imperfect").
This article is broken down into these sections:

Requirements
Installing distcc
Setting up distcc
Using distcc


Requirements


A Bela, preferably with image version ≥v0.3.1. The version is displayed when you ssh root@192.168.7.2 into Bela.
A host machine (e.g. your laptop) connected to Bela by Ethernet over USB
A recent release of distcc: https://github.com/distcc/distcc/releases
Other dependencies (macOS):

autoconf (e.g. brew install autoconf)
Clang 3.9. This can be installed via MacPorts (try sudo port install clang @3.9 or sudo port install clang-3.9) (apparently this fails on newer versions of macOS), or download the pre-built binaries from LLVM.org, (Direct download), and add them to your PATH environment variable.


Optional: if you may want to cross-compile without distcc later, then install sshfs ("FUSE for OS X" on macOS: https://osxfuse.github.io/)

For Ubuntu 18.04:
sudo apt update
sudo apt install autoconf clang-3.9

Installing distcc

Installing on Bela

Since Bela image version v0.3.1, distcc is installed by default.
If you are running an older version of the Bela image, you may want to consider upgrading rather than installing distcc manually.
See this article for more on flashing a Bela image: https://github.com/BelaPlatform/Bela/wiki/Manage-your-SD-card
Installing on the host

To install distcc on your host machine, first download the latest distcc release from here: https://github.com/distcc/distcc/releases. Documentation for distcc can be found here: https://distcc.github.io/.
Steps to install (tested on macOS Mojave):
cd /path/to/distcc

autoconf configure.ac # if this fails, run the line below instead
sudo ./autogen.sh # if `aclocal` not found: `brew install automake`

./configure --without-libiberty --disable-Werror
sudo make && sudo make install

For information about the flags for ./configure, see: marksatt/DistCode#13
Setting up distcc

Setting up on Bela


NOTE 2022-01-05: As of Bela Image 0.3.8f (and possibly earlier, haven't tested), the following setup on Bela is already done by default.

There are two steps to setting up:

Creating and making executable scripts that will run clang-3.9 on Bela with custom options as needed
Creating additional executable scripts that will run the above script using distcc

The goal of setting up distcc on Bela is to have a simple syntax for running distcc, without depending on edits to a profile such as ~/.bashrc.
To achieve this, the environment variables for distcc are going to be set in the scripts we create in the second step.
The clang-3.9 scripts

On Bela, create the following script in /usr/local/bin/clang-3.9-arm:
#!/bin/bash
clang-3.9 $@

Then, create the following script in /usr/local/bin/clang++-3.9-arm:
#!/bin/bash
clang++-3.9 $@ -stdlib=libstdc++

Make the scripts executable:
chmod +x /usr/local/bin/clang-3.9-arm
chmod +x /usr/local/bin/clang++-3.9-arm

... Or do the above in a single step if you are feeling fly:
printf '#!/bin/bash\nclang-3.9 $@' > /usr/local/bin/clang-3.9-arm
printf '#!/bin/bash\nclang++-3.9 $@' > /usr/local/bin/clang++-3.9-arm
chmod +x /usr/local/bin/clang-3.9-arm
chmod +x /usr/local/bin/clang++-3.9-arm

The distcc scripts

On Bela, create the following script in  /usr/local/bin/distcc-clang:
#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc

distcc clang-3.9-arm $@

Then, create the following script in  /usr/local/bin/distcc-clang++:
#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc

distcc clang++-3.9-arm $@

Make these scripts executable:
chmod +x /usr/local/bin/distcc-clang
chmod +x /usr/local/bin/distcc-clang++

Setting up on the host

This section will differ depending on if you installed Clang 3.9 via MacPorts or from LLVM.org.
Namely, the MacPorts executables will be called clang-mp-3.9, clang++-mp-3.9, etc, whereas the LLVM.org executables will be named clang-3.9, clang++, etc.
Alter the following instructions as necessary.
Only one set of scripts are needed on the host, which essentially create a shortcut to running clang-3.9 with some specific command line options to make it generate code for ARM, our target platform.
Create the folder ~/arm or similar (if you choose something else, then modify the scripts that follow accordingly):
mkdir -p ~/arm

On the host, create the following script in /usr/local/bin/clang-3.9-arm:
#!/bin/bash
/path/to/your/clang-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@

Create the following script in /usr/local/bin/clang++-3.9-arm:
#!/bin/bash
/path/to/your/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@

Make them executable:
sudo chmod +x /usr/local/bin/clang-3.9-arm
sudo chmod +x /usr/local/bin/clang++-3.9-arm


Note on setting up 1: We created clang-3.9-arm on the host so that we can easily compile to ARM.
distcc requires us to have on Bela an executable file called clang-3.9-arm, so that the local and remote compiler are "the same".
While we could have on Bela used a symlink to /usr/bin/clang, the method we have used allows us more flexibility to inject custom options as needed.


Note on setting up 2: We added --sysroot ~/arm $@ to the host scripts.
These should not be necessary with distcc, since all the includes are resolved on the client (Bela), and so the host does not really need any paths.
However if you wanted to cross-compile without distcc you would need this path.
To do this you will also need to have installed sshfs and run sshfs root@192.168.7.2: ~/arm.

Using distcc

Using distcc requires two processes:

Running the distccd daemon on the host
Compile programs from Bela

Start distccd (the distcc daemon) on the host:
distccd --verbose --no-detach --daemon --allow 192.168.7.2 --log-level debug --log-file ~/distccd.log


macOS note: --make-me-a-botnet may also be required, otherwise distccd will search in the wrong paths and when monitoring the log ~/distccd.log you will something like: distccd[53245] (dcc_warn_masquerade_whitelist) CRITICAL! /usr/local/lib/distcc not found.
It seems that distcc thinks that user installed software on a Mac should be in /usr instead of /usr/local, but this changed a few macOS' ago.

On Bela, you should now be able to compile a test program:
distcc-clang ~/Bela/resources/network/udp-client.c

Which will produce the build output a.out.
This will not try to compile remotely, you need to have -c for something to be sent out to the server. The reason for this is that without -c then there is a linking stage included, and the linking is always done locally:
distcc-clang -c ~/Bela/resources/network/udp-client.c

Using distcc by default on Bela

Make the following edit to the Bela Makefile at /root/Bela/Makefile:
https://github.com/BelaPlatform/Bela/blob/master/Makefile#L25
DISTCC ?= 1 # set this to 1 to use distcc by default

Using distcc to compile Bela programs

While it is possible to compile Bela programs using distcc, there is currently an issue with re-compiling dependencies:

"if files are compiled with distcc, then dependency files point to the wrong target" - BelaPlatform/Bela#354

What this means is that when a .h file is modified, make may fail to realise that all the .cpp files that include it need to be recompiled.
To workaround this, if you are editing a .h file, make sure that the .cpp file is also touched (or the old .o file is deleted), so that make will know that it has to recompile it.
Monitoring distcc

To see the logs, run on the host:
tail -f ~/distccd.log

To make sure the process is actually called on Bela, run on Bela:
watch -n 0.4 bash -c "ps aux | grep clang-3.9-arm"

watch -n 0.4 bash -c "ps aux | grep clang++-3.9-arm"

From Bela, you should also be able to see it in top or htop and the macOS app "Activity Monitor" on the host.
On Bela, you can also inspect the stderr returned by the host
cat /tmp/distcc/*txt

More usage notes

You may experience an issue where the DISTCC_BACKOFF_PERIOD flag is not obeyed, possibly due to the version of distcc being used on Bela.
You can compensate for this by running a program such as:
watch -n 0.4 rm -rf "~/.distcc/lock/*"

This will compensate for DISTCC_BACKOFF_PERIOD=0 not working by removing periodically any "lock" files that distcc uses internally to timeout the backoff.

  
## clang++-3.9-arm (bela)
#!/bin/bash
clang++-3.9 $@ -stdlib=libstdc++

## clang++-3.9-arm (host)
#!/bin/bash
/path/to/your/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@

## clang-3.9-arm (bela)
#!/bin/bash
clang-3.9 $@

## clang-3.9-arm (host)
#!/bin/bash
/path/to/your/clang-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@

## distcc-clang (bela)
#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc

distcc clang-3.9-arm $@

## distcc-clang++ (bela)
#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc

distcc clang++-3.9-arm $@
	#!/bin/bash
	alias watchdistcc="watch -n 0.4 rm -rf \"~/.distcc/lock/*\""
	#!/bin/bash
	alias distccdbela="distccd --verbose --no-detach --daemon --allow 192.168.7.2 --log-level debug --log-file ~/.distcc/distccd.log --enable-tcp-insecure & tail -f ~/.distcc/distccd.log"
	alias killdistccd="killall distccd"
	#!/bin/bash
	/path/to/your/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
	#!/bin/bash
	/path/to/your/clang-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
	#!/bin/bash
	export DISTCC_HOSTS=192.168.7.1
	export DISTCC_VERBOSE=0
	export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
	export DISTCC_BACKOFF_PERIOD=0
	export TMPDIR=/tmp/distcc
	mkdir -p /tmp/distcc

	distcc clang-3.9-arm $@