Workload Specification¶
Attention
FireMarshal is still in alpha. You are encouraged to try it out and use it for new workloads. The old-style workload generation is still supported (see Defining Custom Workloads for details).
Workloads are defined by a configuration file and corresponding workload source
directory, both typically in the firesim/sw/firesim-software/workloads/
directory. Most paths in the configuration file are assumed to be relative to
the workload source directory.
Example Configuration File¶
FireMarshal supports many configuration options (detailed below), many of which
are not commonly used. We will now walk through an example that uses most of
the common options: workloads/example-fed.json
. In this example, we produce
a 2-node workload that runs two benchmarks: quicksort and spam-filtering. This
will require installing a number of packages on Fedora, as well as
cross-compiling some code. The configuration is as follows:
{
"name" : "example-fed",
"base" : "fedora-base.json",
"overlay" : "overlay",
"guest-init" : "guest-init.sh",
"host-init" : "host-init.sh",
"jobs" : [
{
"name" : "qsort",
"run" : "runQsort.sh"
},
{
"name" : "spamBench",
"run" : "runSpam.sh"
}
]
}
The name
field is required and (by convention) should match the name of the
configuration file. Next is the base
(fedora-base.json). This option
specifies an existing workload to base off of. FireMarshal will first build
fedora-base.json
, and use a copy of its rootfs for example-fed before
applying the remaining options. Additionally, if fedora-base.json specifies any
configuration options that we do not include, we will inherit those (e.g. we
will use the linux-config
option specified by fedora-base). Notice that we
do not specify a workload source directory. FireMarshal will look in
workloads/example-fed/
for any sources specified in the remaining options.
Next come a few options that specify common setup options used by all jobs in
this workload. The overlay
option specifies a filesystem overlay to copy
into our rootfs. In this case, it includes the source code for our benchmarks
(see workloads/example-fed/overlay
). Next is a host-init
option, this
is a script that should be run on the host before building. In our case, it
cross-compiles the quicksort benchmark (cross-compilation is much faster than
natively compiling).
#!/bin/bash
echo "Building qsort benchmark"
cd overlay/root/qsort
make
Next is guest-init
, this script should run exactly once natively within our
workload. For example-fed, this script installs a number of packages that are
required by our benchmarks. Note that guest-init scripts are run during the
build process; this can take a long time, especially with fedora. You will see
linux boot messages and may even see a login prompt. There is no need to login
or interact at all, the guest-init script will run in the background. Note that
guest-init.sh ends with a poweroff
command, all guest-init scripts should
include this (leave it off to debug the build process).
#!/bin/bash
echo "Installing the real time tool (not the shell builtin)"
dnf install -y time
echo "Installing the spambayes python module for the spam benchmark"
pip install spambayes
poweroff
Finally, we specify the two jobs that will run on each simulated node. Job
descriptions have the same format and options as normal workloads. However,
notice that the job descriptions are much shorter than the basic descriptions.
Jobs implicitly inherit from the root configuration. In this case, both qsort
and spamBench will have the overlay and host/guest-init scripts already set up
for them. If needed, you could override these options with a different base
option in the job description. In our case, we need only provide a custom
run
option to each workload. The run option specifies a script that should
run natively in each job every time the job is launched. In our case, we run
each benchmark, collecting some statistics along the way, and then shutdown.
Finishing a run script with poweroff
is a common pattern that allows
workloads to run automatically (no need to log-in or interact at all).
#!/bin/bash
set -x
cd root/qsort
/usr/bin/time -f "%S,%M,%F" ./qsort 10000 2> ../run_result.csv
poweroff
We can now build and launch this workload:
./marshal build workloads/example-fed.json
./marshal launch -j qsort workloads/example-fed.json
./marshal launch -j spamBench workloads/example-fed.json
For more examples, see the test/
directory that contains many workloads
used for testing FireMarshal.
Bare-Metal Workloads¶
FireMarshal was primarily designed to support linux-based workloads. However,
it provides basic support for bare-metal workloads. Take test/bare.json
as
an example:
{
"name" : "bare",
"base" : "bare",
"host-init" : "build.sh",
"bin" : "hello",
"testing" : {
"refDir" : "refOutput"
}
}
This workload creates a simple “Hello World” bare-metal workload. This workload
simply inherits from the “bare” distro in its base
option. This tells
FireMarshal to not attempt to build any linux binaries or rootfs’s for this
workload. It then includes a simple host-init script that simply calls the
makefile to build the bare-metal boot-binary. Finally, it hard-codes a path to
the generated boot-binary. Note that we can still use all the standard
FireMarshal commands with bare-metal workloads. In this case, we provide a
testing specification that simply compares the serial port output against the
known good output of “Hello World!”.
A complete discussion of generating bare-metal boot-binaries is out of scope for this documentation.
Configuration File Options¶
Below is a complete list of configuration options available to FireMarshal.
name¶
Name to use for the workload. Derived objects (rootfs/bootbin) will be named according to this option.
Non-heritable
base¶
Configuration file to inherit from. FireMarshal will look in the same directory
as the workload config file for the base configuration (or the workdir if
--workdir
was passed to the marshal command). A copy of the rootfs from base
will be used when building this workload. Additionally, most configuration
options will be inherited if not explicitly provided (options that cannot be
inherited will be marked as ‘non-heritable’ in this documentation).
In addition to normal configuration files, you may inherit from several hard-coded “distros” including: fedora, br (buildroot), and bare. This is not recommended for the linux-based distros because the fedora-base.json and br-base.json configurations include useful additions to get things like serial ports or the network to work. However, basing on the ‘bare’ distro is the recommended way to generate bare-metal workloads.
Non-heritable
spike¶
Path to binary for spike (riscv-isa-sim) to use when running this workload in spike. Useful for custom forks of spike to support custom instructions or hardware models. Defaults to the version of spike on your PATH (typically the one include with riscv-tools).
linux-src¶
Path to riscv-linux source directory to use when building the boot-binary for
this workload. Defaults to the riscv-linux source submoduled at
firesim/sw/firesim-software/riscv-linux
.
linux-config¶
Linux configuration file to use when building linux. Take care when using a custom configuration, FireSim may require certain boot arguments and device drivers to work properly.
host-init¶
A script to run natively on your host (i.e., your manager instance where you invoked FireMarshal) from the workload source directory each time you explicitly build this workload.
Non-heritable
guest-init
^^^^^^^^^^^^^^^ A script to run natively on the guest (i.e., your workload
running in qemu) exactly once while building. The guest init script will be run
from the root directory with root privileges. This script should end with a
call to poweroff
to make the build process fully automated. Otherwise, the
user will need to log in and shut down manually on each build.
post_run_hook¶
A script or command to run on the output of your run. At least the serial port output of
each run is captured, along with any file outputs specified in the outputs
option. The script will be called like so:
cd workload-dir
post_run_hook /path/to/output
The output directory will follow roughly the following format:
runOutput/name-DATETIME-RAND/
name-job/
uartlog
OUTPUT_FILE1
...
OUTPUT_FILEN
When running as part of the test
command, there will be a folder for each
job in the workload.
overlay¶
Filesystem overlay to apply to the workload rootfs. An overlay should match the rootfs directory structure, with the overlay directory corresponding to the root directory. This is especially useful for overriding system configuration files (e.g. /etc/fstab). The owner of all copied files will be changed to root in the workload rootfs after copying.
files¶
A list of files to copy into the rootfs. The file list has the following format:
[ ["src1", "dst1"], ["src2", "dst2"], ... ]
The source paths are relative to the workload source directory, the destination paths are absolute with respect to the workload rootfs (e.g. [“file1”, “/root/”]). The ownership of each file will be changed to ‘root’ after copying.
outputs¶
A list of files to copy out of the workload rootfs after running. Each path should be absolute with respect to the workload rootfs. Files will be placed together in the output directory. You cannot specify the directory structure of the output.
run¶
A script to run automatically every time this workload runs. The script will
run after all other initialization finishes, but does not require the user to
log in (run scripts run concurrently with any user interaction). Run scripts
typically end with a call to poweroff
to make the workload fully automated,
but this can be omitted if you would like to interact with the workload after
its run script has finished.
Note
Unlike FireSim workloads, the FireMarshal launch command uses
the same rootfs for each run (not a copy), so you should avoid using poweroff
-f
to prevent filesystem corruption.
Non-heritable
command¶
A command to run every time this workload runs. The command will be run from
the root directory and will automatically call poweroff
when complete (the
user does not need to include this).
Non-heritable
workdir¶
Directory to use as the workload source directory. Defaults to a directory with the same name as the configuration file.
Non-heritable
launch¶
Enable/Disable launching of a job when running the ‘test’ command. This is occasionally needed for special ‘dummy’ workloads or other special-purpose jobs that only make sense when running on FireSim. Defaults to ‘yes’.
jobs¶
A list of configurations describing individual jobs that make up this workload.
This list is ordered (FireSim places these jobs in-order in simulation slots).
Job descriptions have the same syntax and options as normal workloads. The one
exception is that jobs implicitly inherit from the parent workload unless a
base
option is explicitly provided. The job name will be appended to the
workload name when creating boot-binaries and rootfs’s. For example, a workload
called “foo” with two jobs named ‘bar’ and ‘baz’ would create 3 rootfs’s:
foo.img, foo-bar.img, and foo-baz.img.
Non-heritable: You cannot use jobs as a base
, only base workloads.
bin¶
Explicit path to the boot-binary to use. This will override any generated binaries created during the build process. This is particularly useful for bare-metal workloads that generate their own raw boot code.
Non-heritable
img¶
Explicit path to the rootfs to use. This will override any generated rootfs created during the build process. This is mostly used for debugging.
Non-heritable
testing¶
Provide details of how to test this workload. The test
command will ignore
any workload that does not have a testing
field. This option is a map with
the following options (only refDir
is required):
Non-heritable
refDir¶
Path to a directory containing reference outputs for this workload. Directory structures are compared directly (same folders, same file names). Regular files are compared exactly. Serial outputs (uartlog) need only match a subset of outputs; the entire reference uartlog contents must exist somewhere (contiguously) in the test uartlog.
buildTimeout¶
Maximum time (in seconds) that the workload should take to build. The test will fail if building takes longer than this. Defaults to infinite.
Note
workloads with many jobs and guest-init scripts, could take a very long time to build.
runTimeout¶
Maximum time (in seconds) that any particular job should take to run and exit. The test will fail if a job runs for longer than this before exiting. Defaults to infinite.
strip¶
Attempt to clean up the uartlog output before comparing against the reference. This will remove all lines not generated by a run script or command, as well as stripping out any extra characters that might be added by the run-system (e.g. the systemd timestamps on Fedora). This option is highly recommended on Fedora due to it’s non-deterministic output.