Targets

FireSim generates SoC models by transforming RTL emitted by a Chisel generator, such as the Rocket SoC generator. Subject to conditions outlined in Restrictions on Target RTL, if it can be generated by Chisel, it can be simulated in FireSim.

Restrictions on Target RTL

Current limitations in Golden Gate place the following restrictions on the (FIR)RTL that can be transformed and thus used in FireSim:

  1. The top-level module must have no inputs or outputs. Input stimulus and output capture must be implemented using target RTL or target-to-host Bridges.
  2. All target clocks must be generated by a single RationalClockBridge.
  3. Black boxes must be “clock-gateable” by replacing its input clock with a gated equivalent which will be used to stall simulation time in that module.
    1. As a consequence, target clock-gating cannot be implemented using black-box primitives, and must instead be modeled by adding clock-enables to all state elements of the gated clock domain (i.e., by adding an enable or feedback mux on registers to conditionally block updates, and by gating write-enables on memories).
  4. Asynchronous reset must only be implemented using Rocket Chip’s black-box async reset. These are replaced with synchronously reset registers using a FIRRTL transformation.

Including Verilog IP

FireSim now supports target designs that incorporate Verilog IP using the standard BlackBox interface from Chisel. For an example of how to add Verilog IP to a target system based on Rocket Chip, see the Incorporating Verilog Blocks section of the Chipyard documentation.

  1. For the transform to work, the Chisel Blackbox that wraps the Verilog IP must have input clocks that can safely be clock-gated.
  2. The compiler that produces the decoupled simulator (“FAME Transform”) automatically recognizes such blackboxes inside the target design.
  3. The compiler automatically gates each clock of the Verilog IP to ensure that it deterministically advances in lockstep with the rest of the simulator.
  4. This allows any Verilog module, subject to the constraint above, to be instantiated anywhere in the target design using the standard Chisel Blackbox interface.

Multiple Clock Domains

FireSim can support simulating targets that have multiple clock domains. As stated above, all clocks must be generated using a single RationalClockBridge. For most users the default FireSim test harness in Chipyard will suffice, if you need to define a custom test harness instantiate the RationalClockBridge like so:

  // Here we request three target clocks (the base clock is implicit). All
  // clocks beyond the base clock are specified using the RationalClock case
  // class which gives the clock domain's name, and its clock multiplier and
  // divisor relative to the base clock.
  val clockBridge = Module(new RationalClockBridge(RationalClock("HalfRate", 1, 2), 
                                                   RationalClock("ThirdRate", 1, 3)))

  // The clock bridge has a single output: a Vec[Clock] of the requested clocks
  // in the order they were specified, which we are now free to use through our
  // Chisel design.  While not necessary, here we unassign the Vec to give them
  // more informative references in our Chisel.
  val Seq(fullRate, halfRate, thirdRate) = clockBridge.io.clocks.toSeq

Further documentation can be found in the source (sim/midas/src/main/scala/midas/widgets/ClockBridge.scala).

The Base Clock

By convention, target time is specified in cycles of the base clock, which is defined to be the zeroth output clock of the RationalClockBridge. While we suggest making the base clock the fastest clock in your system, as in any microprocessor-based system it will likely correspond to your core clock frequency, this is not a constraint.

Limitations:

  • The number of target clocks FireSim can simulate is bounded by the number of BUFGCE resources available on the host FPGA, as these are used to independently clock-gate each target clock.

  • As its name suggests, the RationalClockBridge can only generate target clocks that are rationally related. Specifically, all requested frequencies must be expressable in the form:

    \[f_{i} = \frac{f_{lcm}}{k_{i}}\]
    Where,
    • \(f_{i}\) is the desired frequency of the \(i^{th}\) clock
    • \(f_{lcm}\), is the least-common multiple of all requested frequencies
    • \(k_{i}\) is a 16-bit unsigned integer

    An arbitrary frequency can be modeled using a sufficiently precise rational multiple. Golden Gate will raise a compile-time error if it cannot support a desired frequency.

  • Each bridge module must reside entirely within a single clock domain. The Bridge’s target interface must contain a single input clock, and all inputs and outputs of the bridge module must be latched and launched, respectively, by registers in the same clock domain.

Provided Target Designs

Target Generator Organization

FireSim provides multiple projects, each for a different type of target. Each project has its own chisel generator that invokes Golden Gate, its own driver sources, and a makefrag that plugs into the Make-based build system that resides in sim/. These projects are:

  1. firesim (Default): rocket chip-based targets. These include targets with either BOOM or rocket pipelines, and should be your starting point if you’re building an SoC with the Rocket Chip generator.
  2. midasexamples: the Golden Gate example designs, a set of simple chisel circuits like GCD, that demonstrate how to use Golden Gate. These are useful test cases for bringing up new Golden Gate features.
  3. fasedtests: designs to do integration testing of FASED memory-system timing models.

Projects have the following directory structure:

sim/
├-Makefile # Top-level makefile for projects where FireSim is the top-level repo
├-Makefrag # Target-agnostic makefrag, with recipes to generate drivers and RTL simulators
├-src/main/scala/{target-project}/
│                └─Makefrag # Defines target-specific make variables and recipes.
├-src/main/cc/{target-project}/
│             ├─{driver-csrcs}.cc # The target's simulation driver, and sofware-model sources
│             └─{driver-headers}.h
└-src/main/makefrag/{target-project}/
                     ├─Generator.scala # Contains the main class that generates target RTL and calls Golden Gate
                     └─{other-scala-sources}.scala

Specifying A Target Instance

To generate a specific instance of a target, the build system leverages four Make variables:

  1. TARGET_PROJECT: this points the Makefile (sim/Makefile) at the right target-specific Makefrag, which defines the generation and MIDAS-level software-simulation recipes. The makefrag for the default target project is defined at sim/src/main/makefrag/firesim.
  2. DESIGN: the name of the top-level Chisel module to generate (a Scala class name). These are defined in FireChip Chipyard generator.
  3. TARGET_CONFIG: specifies a Config instance that is consumed by the target design’s generator. For the default firesim target project, predefined configs are described in in the FireChip Chipyard generator.
  4. PLATFORM_CONFIG: specifies a Config instance that is consumed by Golden Gate and specifies compiler-level and host-land parameters, such as whether to enable assertion synthesis, or multi-ported RAM optimizations. Common platform configs are described in firesim-lib/sim/src/main/scala/configs/CompilerConfigs.scala).

TARGET_CONFIG and PLATFORM_CONFIG are strings that are used to construct a Config instance (derives from RocketChip’s parameterization system, Config, see freechips.rocketchip.config). These strings are of the form “{…_}{<Class Name>_}<Class Name>”. Only the final, base class name is compulsory: class names that are prepended with “_” are used to create a compound Config instance.

// Specify by setting TARGET_CONFIG=Base
class Base extends Config((site, here, up) => {...})
class Override1 extends Config((site, here, up) => {...})
class Override2 extends Config((site, here, up) => {...})
// Specify by setting TARGET_CONFIG=Compound
class Compound extends Config(new Override2 ++ new Override1 ++ new Base)
// OR by setting TARGET_CONFIG=Override2_Override1_Base
// Can specify undefined classes this way. ex: TARGET_CONFIG=Override2_Base

With this scheme, you don’t need to define a Config class for every instance you wish to generate. We use this scheme to specify FPGA frequencies (eg. “BaseF1Config_F90MHz”) in manager build recipes, but it’s also very useful for doing sweeping over a parameterization space.

Note that the precedence of Configs decreases from left to right in a string. Appending a config to an existing one will only have an effect if it sets a field not already set in higher precendence Configs. For example, “BaseF1Config_F90MHz” is equivalent to “BaseF1Config_F90MHz_F80MHz” as DesiredHostFrequency resolves to 90 MHz, but “F90MHz_BaseF1Config” is distinct from “F80MHz_F90MHz_BaseF1Config” in that DesiredHostFrequency resolves to 90 and 80 MHz respectively.

How a particular Config resolves it’s Field s can be unintuitive for complex compound Config s. One precise way to check a config is doing what you expect is to open the scala REPL, instantiate an instance of the desired Config, and inspect its fields.

$ make sbt # Launch into SBT's shell with extra FireSim arguments

sbt:firechip> console # Launch the REPL

scala> val inst = (new firesim.firesim.FireSimRocketChipConfig).toInstance # Make an instance

inst: freechips.rocketchip.config.Config = FireSimRocketChipConfig

scala> import freechips.rocketchip.subsystem._ # Get some important Fields

import freechips.rocketchip.subsystem.RocketTilesKey

scala> inst(RocketTilesKey).size # Query number of cores

res2: Int = 1

scala> inst(RocketTilesKey).head.dcache.get.nWays # Query L1 D$ associativity

res3: Int = 4

Rocket Chip Generator-based SoCs (firesim project)

Using the Make variables listed above, we give examples of generating different targets using the default Rocket Chip-based target project.

Rocket-based SoCs

Three design classes use Rocket scalar in-order pipelines.

Single core, Rocket pipeline (default)

make TARGET_CONFIG=FireSimRocketConfig

Single-core, Rocket pipeline, with network interface

make TARGET_CONFIG=WithNIC_FireSimRocketChipConfig

Quad-core, Rocket pipeline

make TARGET_CONFIG=FireSimQuadRocketConfig

BOOM-based SoCs

The BOOM (Berkeley Out-of-Order Machine) superscalar out-of-order pipelines can also be used with the same design classes that the Rocket pipelines use. Only the TARGET_CONFIG needs to be changed, as shown below:

Single-core BOOM

make TARGET_CONFIG=FireSimLargeBoomConfig

Single-core BOOM, with network interface

make TARGET_CONFIG=WithNIC_FireSimBoomConfig

Generating A Different FASED Memory-Timing Model Instance

Golden Gate’s memory-timing model generator, FASED, can elaborate a space of different DRAM model instances: we give some typical ones here. These targets use the Makefile-defined defaults of DESIGN=FireSim PLATFORM_CONFIG=BaseF1Config.

Quad-rank DDR3 first-ready, first-come first-served memory access scheduler

::
make TARGET_CONFIG=DDR3FRFCFS_FireSimRocketConfig

As above, but with a 4 MiB (maximum simulatable capacity) last-level-cache model

make TARGET_CONFIG=DDR3FRFCFSLLC4MB_FireSimRocketConfig

FASED timing-model configurations are passed to the FASED Bridges in your Target’s FIRRTL, and so must be prepended to TARGET_CONFIG.

Midas Examples (midasexamples project)

This project can generate a handful of toy target-designs (set with the make variable DESIGN). Each of these designs has their own chisel source file and serves to demostrate the features of Golden Gate.

Some notable examples are:

  1. GCD: the “Hello World!” of hardware.
  2. WireInterconnect: demonstrates how combinational paths can be modeled with Golden Gate.
  3. PrintfModule: demonstrates synthesizable printfs
  4. AssertModule: demonstrates synthesizable assertions

To generate a target, set the make variable TARGET_PROJECT=midasexamples. so that the right project makefrag is sourced.

Examples

To generate the GCD midasexample:

make DESIGN=GCD TARGET_PROJECT=midasexamples

FASED Tests (fasedtests project)

This project generates target designs capable of driving considerably more bandwidth to an AXI4-memory slave than current FireSim targets. These are used to do integration and stress testing of FASED instances.

Examples

Generate a synthesizable AXI4Fuzzer (based off of Rocket Chip’s TL fuzzer), driving a DDR3 FR-FCFS-based FASED instance.

make TARGET_PROJECT=fasedtests DESIGN=AXI4Fuzzer TARGET_CONFIG=FRFCFSConfig

As above, now configured to drive 10 million transactions through the instance.

make TARGET_PROJECT=fasedtests DESIGN=AXI4Fuzzer TARGET_CONFIG=NT10e7_FRFCFSConfig