Bridge Deep Dive

In this section, we’ll walkthrough a simple Target-to-Host bridge associated with Chipyard called the UARTBridge. It serves as an example of how to use FireSim as a library to create your own bridges. The UARTBridge uses host-MMIO to model a UART device.

Reading the Target-to-Host Bridges section is a prerequisite to reading these sections.

UART Bridge (Host-MMIO)

Source code for the UART Bridge lives in Chipyard in the ${CY_DIR}/generators/firechip area. Specifically, the following directories:

chipyard/generators/firechip/
    bridgeinterfaces/
        src/main/scala/
            UART.scala # Chisel IOs and Scala case classes shared between FireSim/Chipyard
    goldengateimplementations/
        src/main/scala/
            UARTBridge.scala # BridgeModule definition
    bridgestubs/
        src/main/
            scala/uart/UARTBridge.scala # Target-Side Bridge
            cc/bridges/uart.cc          # Bridge Driver source
            cc/bridges/uart.h           # Bridge Driver header
    chip/
        src/main/
            cc/firesim/firesim_top.cc # Driver instantiation in the main simulation driver
            makefrag/firesim/ # Target-specific build rules
                build.mk      # Definition of the Chisel elaboration step
                config.mk     # Target-specific configuration and path setup
                driver.mk     # Build rules for the driver
                metasim.mk    # Custom run commands for meta-simulation

Target Side

The first order of business when designing a new bridge is to implement its target side. In the case of UART we’ve defined a Chisel BlackBox [1] extending Bridge. We’ll instantiate this BlackBox and connect it to UART IO in the top-level of our chip. We first define a class that captures the target-side interface of the Bridge (in ${CY_DIR}/generators/firechip/bridgeinterfaces/src/main/scala/UART.scala):

class UARTPortIO extends Bundle {
   val txd = Output(Bool())
   val rxd = Input(Bool())
 }

 class UARTBridgeTargetIO extends Bundle {
   val clock = Input(Clock())
   val uart = Flipped(new UARTPortIO)
   // Note this reset is optional and used only to reset target-state modeled
   // in the bridge. This reset is just like any other Bool included in your target
   // interface, simply appears as another Bool in the input token.
   val reset = Input(Bool())
 }

Here, we define a case class (in ${CY_DIR}/generators/firechip/bridgeinterfaces/src/main/scala/UART.scala) that carries additional metadata to the host-side BridgeModule. For UART, this is simply the clock-division required to produce the baudrate:

// Out bridge module constructor argument. This captures all of the extra
// metadata we'd like to pass to the host-side BridgeModule. Note, we need to
// use a single case class to do so, even if it is simply to wrap a primitive
// type, as is the case for the div Int.
case class UARTKey(div: Int)

Both these IOs and the case class needs be fully isolated from the target. This means that they should compile with FireSim’s Chisel version (Chisel 3) and not include any target-specific classes, IOs, etc. Both the IOs and the case class is compiled in the target and is copied to FireSim to be compiled there as well.

Finally, we define the actual target-side module (specifically, a BlackBox)(in ${CY_DIR}/generators/firechip/bridgestubs/src/main/scala/uart/UARTBridge.scala):

class UARTBridge(initBaudRate: BigInt, freqMHz: Int)(implicit p: Parameters) extends BlackBox
    with Bridge[HostPortIO[UARTBridgeTargetIO]] {
  // Module portion corresponding to this bridge
  val moduleName = "firechip.firesimonly.bridges.UARTBridgeModule"
  // Since we're extending BlackBox this is the port will connect to in our target's RTL
  val io = IO(new UARTBridgeTargetIO)
  // Implement the bridgeIO member of Bridge using HostPort. This indicates that
  // we want to divide io, into a bidirectional token stream with the input
  // token corresponding to all of the inputs of this BlackBox, and the output token consisting of
  // all of the outputs from the BlackBox
  val bridgeIO = HostPort(io)

  // Do some intermediate work to compute our host-side BridgeModule's constructor argument
  val div = (BigInt(freqMHz) * 1000000 / initBaudRate).toInt

  // And then implement the constructorArg member
  val constructorArg = Some(UARTKey(div))

  // Finally, and this is critical, emit the Bridge Annotations -- without
  // this, this BlackBox would appear like any other BlackBox to Golden Gate
  generateAnnotations()
}

To make it easier to instantiate our target-side module, we’ve also defined an optional companion object (in ${CY_DIR}/generators/firechip/bridgestubs/src/main/scala/uart/UARTBridge.scala):

object UARTBridge {
  def apply(clock: Clock, uart: sifive.blocks.devices.uart.UARTPortIO, reset: Bool, freqMHz: Int)(implicit p: Parameters): UARTBridge = {
    val ep = Module(new UARTBridge(uart.c.initBaudRate, freqMHz))
    ep.io.uart.txd := uart.txd
    uart.rxd := ep.io.uart.rxd
    ep.io.clock := clock
    ep.io.reset := reset
    ep
  }
}

This target-side module doesn’t compile with FireSim at all (except for the APIs given by the firesim-lib Scala project).

That completes the target-side definition.

Host-Side BridgeModule

The remainder of the file is dedicated to the host-side BridgeModule definition. Here we have to process tokens generated by the target, and expose a memory-mapped interface to the bridge driver.

Inspecting the top of the class (in ${CY_DIR}/generators/firechip/goldengateimplementations/src/main/scala/UARTBridge.scala):

// Our UARTBridgeModule definition, note:
// 1) it takes one parameter, key, of type UARTKey --> the same case class we captured from the target-side
// 2) It accepts one implicit parameter of type Parameters
// 3) It extends BridgeModule passing the type of the HostInterface
//
// While the Scala type system will check if you parameterized BridgeModule
// correctly, the types of the constructor arugument (in this case UARTKey),
// don't match, you'll only find out later when Golden Gate attempts to generate your module.
class UARTBridgeModule(key: UARTKey)(implicit p: Parameters) extends BridgeModule[HostPortIO[UARTBridgeTargetIO]]()(p) {
  lazy val module = new BridgeModuleImp(this) {
    val div = key.div
    // This creates the interfaces for all of the host-side transport
    // AXI4-lite for the simulation control bus, =
    // AXI4 for DMA
    val io = IO(new WidgetIO())

    // This creates the host-side interface of your TargetIO
    val hPort = IO(HostPort(new UARTBridgeTargetIO))

    // Generate some FIFOs to capture tokens...
    val txfifo = Module(new Queue(UInt(8.W), 128))
    val rxfifo = Module(new Queue(UInt(8.W), 128))

    val target = hPort.hBits.uart
    // In general, your BridgeModule will not need to do work every host-cycle. In simple Bridges,
    // we can do everything in a single host-cycle -- fire captures all of the
    // conditions under which we can consume and input token and produce a new
    // output token
    val fire = hPort.toHost.hValid && // We have a valid input token: toHost ~= leaving the transformed RTL
               hPort.fromHost.hReady && // We have space to enqueue a new output token
               txfifo.io.enq.ready      // We have space to capture new TX data
    val targetReset = fire & hPort.hBits.reset
    rxfifo.reset := reset.asBool || targetReset
    txfifo.reset := reset.asBool || targetReset

    hPort.toHost.hReady := fire
    hPort.fromHost.hValid := fire

Most of what follows is responsible for modeling the timing of the UART. As a bridge designer, you’re free to take as many host-cycles as you need to process tokens. In simpler models, like this one, it’s often easiest to write logic that operates in a single cycle but gate state-updates using a “fire” signal that is asserted when the required tokens are available.

Now, we’ll skip to the end to see how to add registers to the simulator’s memory map that can be accessed using MMIO from bridge driver.

// Exposed the head of the queue and the valid bit as a read-only registers
// with name "out_bits" and out_valid respectively
genROReg(txfifo.io.deq.bits, "out_bits")
genROReg(txfifo.io.deq.valid, "out_valid")

// Generate a writeable register, "out_ready", that when written to dequeues
// a single element in the tx_fifo. Pulsify derives the register back to false
// after pulseLength cycles to prevent multiple dequeues
Pulsify(genWORegInit(txfifo.io.deq.ready, "out_ready", false.B), pulseLength = 1)

// Generate regisers for the rx-side of the UART; this is eseentially the reverse of the above
genWOReg(rxfifo.io.enq.bits, "in_bits")
Pulsify(genWORegInit(rxfifo.io.enq.valid, "in_valid", false.B), pulseLength = 1)
genROReg(rxfifo.io.enq.ready, "in_ready")

// This method invocation is required to wire up all of the MMIO registers to
// the simulation control bus (AXI4-lite)
genCRFile()

This module is injected into the FireSim compiler and is never compiled by the target. Thus it has access to all APIs given by MIDAS.

Host-Side Driver

To complete our host-side definition, we need to define a CPU-hosted bridge driver. Bridge Drivers extend the bridge_driver_t interface, which declares 5 virtual methods a concrete bridge driver must implement:

/**
 * @brief Base class for Bridge Drivers
 *
 * Bridge Drivers are the CPU-hosted component of a Target-to-Host Bridge. A
 * Bridge Driver interacts with their accompanying FPGA-hosted BridgeModule
 * using MMIO (via read() and write() methods).
 */
class bridge_driver_t : public widget_t {
public:
  using widget_t::widget_t;

  ~bridge_driver_t() override = default;

  /**
   * Initialize the bridge state.
   *
   * This method can be overriden to initialise the hardware through MMIO.
   */
  virtual void init() {}

  /**
   * Final post-run cleanup.
   *
   * The analog of init(), this provides a final opportunity to interact with
   * the FPGA before destructors are called at the end of simulation. Useful
   * for doing end-of-simulation clean up that requires communication with
   * the hardware through MMIO or streams.
   */
  virtual void finish() {}

  /**
   * Does work that allows the bridge to advance in simulation time.
   *
   * Progresses for one or more cycles. The standard FireSim driver calls the
   * tick methods of all registered bridge drivers. Bridges whose BridgeModule
   * is free-running need not implement this method
   */
  virtual void tick() {}

  /**
   * Returns true if the bridge requests explicit termination.
   */
  virtual bool terminate() { return false; }

  /**
   * If the bridge driver calls for termination, encode a cause here.
   *
   * 0 = PASS. All other codes are bridge-implementation defined
   */
  virtual int exit_code() { return 0; }

protected:
  void write(size_t addr, uint32_t data);

  uint32_t read(size_t addr);
};

The declaration of the UART bridge is inlined below from ${CY_DIR}/generators/firechip/bridgestubs/src/main/cc/bridges/uart.h:

#ifndef __UART_H
#define __UART_H

#include "bridges/serial_data.h"
#include "core/bridge_driver.h"

#include <cstdint>
#include <memory>
#include <optional>
#include <signal.h>
#include <string>
#include <vector>

/**
 * Structure carrying the addresses of all fixed MMIO ports.
 *
 * This structure is instantiated when all bridges are populated based on
 * the target configuration.
 */
struct UARTBRIDGEMODULE_struct {
  uint64_t out_bits;
  uint64_t out_valid;
  uint64_t out_ready;
  uint64_t in_bits;
  uint64_t in_valid;
  uint64_t in_ready;
};

/**
 * Base class for callbacks handling data coming in and out a UART stream.
 */
class uart_handler {
public:
  virtual ~uart_handler() = default;

  virtual std::optional<char> get() = 0;
  virtual void put(char data) = 0;
};

class uart_t final : public bridge_driver_t {
public:
  /// The identifier for the bridge type used for casts.
  static char KIND;

  /// Creates a bridge which interacts with standard streams or PTY.
  uart_t(simif_t &simif,
         const UARTBRIDGEMODULE_struct &mmio_addrs,
         int uartno,
         const std::vector<std::string> &args);

  ~uart_t() override;

  void tick() override;

private:
  const UARTBRIDGEMODULE_struct mmio_addrs;
  std::unique_ptr<uart_handler> handler;

  serial_data_t<char> data;

  void send();
  void recv();
};

#endif // __UART_H

The bulk of the driver’s work is done in its tick() method. Here, the driver polls the BridgeModule and then does some work. Note: the name, tick is vestigial: one invocation of tick() may do work corresponding to an arbitrary number of target cycles. It’s critical that tick be non-blocking, as waiting for work from the BridgeModule may deadlock the simulator.

Build-System Modifications

The final consideration in adding your bridge concerns the build system. You should be able to host the Scala sources for your bridge with rest of your target RTL: SBT will make sure those classes are available on the runtime classpath. If you’re hosting your bridge driver sources outside of the existing directories, you’ll need to modify your target-project make fragments to include them. The default Chipyard/Rocket Chip-based one lives in ${CY_DIR}/generators/firechip/chip/src/main/makefrag/firesim.

Here the main order of business is to add header and source files to DRIVER_H and DRIVER_CC respectively in driver.mk, by modifying the lines below:

##########################
# Driver Sources & Flags #
##########################

ifeq (,$(wildcard $(RISCV)/lib/libriscv.so))
$(warning libriscv not found)
LRISCV=
else
LRISCV=-lriscv
endif

firechip_lib_dir = $(chipyard_dir)/generators/firechip/chip/src/main/cc
firechip_bridgestubs_lib_dir = $(chipyard_dir)/generators/firechip/bridgestubs/src/main/cc
testchipip_csrc_dir = $(chipyard_dir)/generators/testchipip/src/main/resources/testchipip/csrc

# DRIVER_H only used to update recipe pre-reqs (ok to track more files)

# fesvr and related srcs
DRIVER_H += \
                $(shell find $(testchipip_csrc_dir) -name "*.h") \
                $(shell find $(firechip_bridgestubs_lib_dir)/fesvr -name "*.h")
DRIVER_CC += \
                $(testchipip_csrc_dir)/cospike_impl.cc \
                $(testchipip_csrc_dir)/testchip_tsi.cc \
                $(testchipip_csrc_dir)/testchip_dtm.cc \
                $(testchipip_csrc_dir)/testchip_htif.cc \
                $(firechip_bridgestubs_lib_dir)/fesvr/firesim_tsi.cc \
                $(firechip_bridgestubs_lib_dir)/fesvr/firesim_dtm.cc \
                $(RISCV)/lib/libfesvr.a
# Disable missing override warning for testchipip.
TARGET_CXX_FLAGS += \
                -isystem $(testchipip_csrc_dir) \
                -isystem $(RISCV)/include \
                -Wno-inconsistent-missing-override
TARGET_LD_FLAGS += \
                -L$(RISCV)/lib \
                -Wl,-rpath,$(RISCV)/lib \
                $(LRISCV)

# top-level sources
DRIVER_CC += $(addprefix $(firechip_lib_dir)/firesim/, $(addsuffix .cc, firesim_top))
TARGET_CXX_FLAGS += -I$(firechip_bridgestubs_lib_dir)/bridge/test

# bridge sources
DRIVER_H += $(shell find $(firechip_bridgestubs_lib_dir) -name "*.h")
DRIVER_CC += \
                $(wildcard \
                        $(addprefix \
                                $(firechip_bridgestubs_lib_dir)/, \
                                $(addsuffix .cc,bridges/* bridges/tracerv/* bridges/cospike/*) \
                        ) \
                )
TARGET_CXX_FLAGS += \
                -I$(firechip_bridgestubs_lib_dir) \
                -I$(firechip_bridgestubs_lib_dir)/bridge \
                -I$(firechip_bridgestubs_lib_dir)/bridge/tracerv \
                -I$(firechip_bridgestubs_lib_dir)/bridge/cospike
TARGET_LD_FLAGS += \
        -l:libdwarf.so -l:libelf.so \
        -lz \

# other
TARGET_CXX_FLAGS += \
                -I$(GENERATED_DIR) \
                -g

Then the other .mk files in the directory handle copying sources from the target to FireSim, building RTL, and more. That’s it! At this point you should be able to both test your bridge in software simulation using metasimulation, or deploy it to an FPGA.