Bridge Walkthrough

In this section, we’ll walkthrough a simple Target-to-Host bridge, the UARTBridge, provided with FireSim to demonstrate how to integrate your own. The UARTBridge uses host-MMIO to model a UART device.

Reading the Bridges section is a prerequisite to reading these sections.

UART Bridge (Host-MMIO)

Source code for the UART Bridge lives in the following directories:

sim/
├-firesim-lib/src/main/
│                 ├-scala/bridges/UARTBridge.scala # Target-Side Bridge and BridgeModule Definitions
│                 ├-cc/brides/uart.cc # Bridge Driver source
│                 └-cc/brides/uart.h  # Bridge Driver header
├-src/main/cc/firesim/firesim_top.cc  # Driver instantiation in the main simulation driver
└-src/main/makefrag/firesim/Makefrag  # Build system modifications to compile Bridge Driver code

Target Side

The first order of business when designing a new bridge is to implement its target side. In the case of UART we’ve defined a Chisel BlackBox [1] extending Bridge. We’ll instantiate this BlackBox and connect it to UART IO in the top-level of our chip. We first define a class that captures the target-side interface of the Bridge:

class UARTBridgeTargetIO extends Bundle {
  val uart = Flipped(new UARTPortIO)
  // Note this reset is optional and used only to reset target-state modelled
  // in the bridge This reset just like any other Bool included in your target
  // interface, simply appears as another Bool in the input token.
  val reset = Input(Bool())
}
[1]You can also extend a non-BlackBox Chisel Module, but any Chisel source contained within will be removed by Golden Gate. You may wish to do this to enclose a synthesizable model of the Bridge for other simulation backends, or simply to wrap a larger chunk RTL you wish to model in the host-side of the Bridge.

Here, we define a case class that carries additional metadata to the host-side BridgeModule. For UART, this is simply the clock-division required to produce the baudrate:

// Out bridge module constructor argument. This captures all of the extra
// metadata we'd like to pass to the host-side BridgeModule. Note, we need to
// use a single case-class to do so, even if it is simply to wrap a primitive
// type, as is the case for UART (int)
case class UARTKey(div: Int)

Finally, we define the actual target-side module (specifically, a BlackBox):

class UARTBridge(implicit p: Parameters) extends BlackBox
    with Bridge[HostPortIO[UARTBridgeTargetIO], UARTBridgeModule] {
  // Since we're extending BlackBox this is the port will connect to in our target's RTL
  val io = IO(new UARTBridgeTargetIO)
  // Implement the bridgeIO member of Bridge using HostPort. This indicates that
  // we want to divide io, into a bidirectional token stream with the input
  // token corresponding to all of the inputs of this BlackBox, and the output token consisting of 
  // all of the outputs from the BlackBox
  val bridgeIO = HostPort(io)

  // Do some intermediate work to compute our host-side BridgeModule's constructor argument
  val frequency = p(PeripheryBusKey).frequency
  val baudrate = 3686400L
  val div = (p(PeripheryBusKey).frequency / baudrate).toInt

  // And then implement the constructorArg member
  val constructorArg = Some(UARTKey(div))

  // Finally, and this is critical, emit the Bridge Annotations -- without
  // this, this BlackBox would appear like any other BlackBox to golden-gate
  generateAnnotations()
}

To make it easier to instantiate our target-side module, we’ve also defined an optional companion object:

object UARTBridge {
  def apply(uart: UARTPortIO)(implicit p: Parameters): UARTBridge = {
    val ep = Module(new UARTBridge)
    ep.io.uart <> uart
    ep
  }
}

That completes the target-side definition.

Host-Side BridgeModule

The remainder of the file is dedicated to the host-side BridgeModule definition. Here we have to process tokens generated by the target, and expose a memory-mapped interface to the bridge driver.

Inspecting the top of the class:

// Our UARTBridgeModule definition, note:
// 1) it takes one parameter, key, of type UARTKey --> the same case class we captured from the target-side
// 2) It accepts one implicit parameter of type Parameters
// 3) It extends BridgeModule passing the type of the HostInterface
//
// While the scala-type system will check if you parameterized BridgeModule
// correctly, the types of the constructor arugument (in this case UARTKey),
// don't match, you'll only find out later when Golden Gate attempts to generate your module.
class UARTBridgeModule(key: UARTKey)(implicit p: Parameters) extends BridgeModule[HostPortIO[UARTBridgeTargetIO]]()(p) {
  val div = key.div
  // This creates the interfaces for all of the host-side transport
  // AXI4-lite for the simulation control bus, =
  // AXI4 for DMA
  val io = IO(new WidgetIO())

  // This creates the host-side interface of your TargetIO
  val hPort = IO(HostPort(new UARTBridgeTargetIO))

  // Generate some FIFOs to capture tokens...
  val txfifo = Module(new Queue(UInt(8.W), 128))
  val rxfifo = Module(new Queue(UInt(8.W), 128))

  val target = hPort.hBits.uart
  // In general, your BridgeModule will not need to do work every host-cycle. In simple Bridges,
  // we can do everything in a single host-cycle -- fire captures all of the
  // conditions under which we can consume and input token and produce a new
  // output token
  val fire = hPort.toHost.hValid && // We have a valid input token: toHost ~= leaving the transformed RTL
             hPort.fromHost.hReady && // We have space to enqueue a new output token
             txfifo.io.enq.ready      // We have space to capture new TX data
  val targetReset = fire & hPort.hBits.reset
  rxfifo.reset := reset.toBool || targetReset
  txfifo.reset := reset.toBool || targetReset

  hPort.toHost.hReady := fire
  hPort.fromHost.hValid := fire

Most of what follows is responsible for modeling the timing of the UART. As a bridge designer, you’re free to take as many host-cycles as you need to process tokens. In simpler models, like this one, it’s often easiest to write logic that operates in a single cycle but gate state-updates using a “fire” signal that is asserted when the required tokens are available.

Now, we’ll skip to the end to see how to add registers to the simulator’s memory map that can be accessed using MMIO from bridge driver.

  // Exposed the head of the queue and the valid bit as a read-only registers
  // with name "out_bits" and out_valid respectively
  genROReg(txfifo.io.deq.bits, "out_bits")
  genROReg(txfifo.io.deq.valid, "out_valid")

  // Generate a writeable register, "out_ready", that when written to dequeues
  // a single element in the tx_fifo. Pulsify derives the register back to false
  // after pulseLength cycles to prevent multiple dequeues
  Pulsify(genWORegInit(txfifo.io.deq.ready, "out_ready", false.B), pulseLength = 1)

  // Generate regisers for the rx-side of the UART; this is eseentially the reverse of the above
  genWOReg(rxfifo.io.enq.bits, "in_bits")
  Pulsify(genWORegInit(rxfifo.io.enq.valid, "in_valid", false.B), pulseLength = 1)
  genROReg(rxfifo.io.enq.ready, "in_ready")

  // This method invocation is required to wire up all of the MMIO registers to
  // the simulation control bus (AXI4-lite)
  genCRFile()

Host-Side Driver

To complete our host-side definition, we need to define a CPU-hosted bridge driver. Bridge Drivers extend the bridge_driver_t interface, which declares 5 virtual methods a concrete bridge driver must implement:

// Bridge Drivers are the CPU-hosted component of a Target-to-Host Bridge. A
// Bridge Driver interacts with their accompanying FPGA-hosted BridgeModule
// using MMIO (via read() and write() methods) or CPU-mastered DMA (via pull()
// and push()).

class bridge_driver_t
{
public:
  bridge_driver_t(simif_t* s): sim(s) { }
  virtual ~bridge_driver_t() {};
  // Initialize BridgeModule state -- this can't be done in the constructor currently
  virtual void init() = 0;
  // Does work that allows the Bridge to advance in simulation time (one or more cycles)
  // The standard FireSim driver calls the tick methods of all registered bridge drivers.
  // Bridges whose BridgeModule is free-running need not implement this method
  virtual void tick() = 0;
  // Indicates the simulation should terminate.
  // Tie off to false if the brige will never call for the simulation to teriminate.
  virtual bool terminate() = 0;
  // If the bridge driver calls for termination, encode a cause here. 0 = PASS All other
  // codes are bridge-implementation defined
  virtual int exit_code() = 0;
  // The analog of init(), this provides a final opportunity to interact with
  // the FPGA before destructors are called at the end of simulation. Useful
  // for doing end-of-simulation clean up that requires calling {read,write,push,pull}.
  virtual void finish() = 0;

The declaration of the Uart bridge driver lives at sim/firesim-lib/src/main/cc/bridges/uart.h. It is inlined below:

//See LICENSE for license details
#ifndef __UART_H
#define __UART_H

#include "serial.h"
#include <signal.h>

// The definition of the primary constructor argument for a bridge is generated
// by Golden Gate at compile time _iff_ the bridge is instantiated in the
// target. As a result, all bridge driver definitions conditionally remove
// their sources if the constructor class has been defined (the
// <cname>_struct_guard macros are generated along side the class definition.)
//
// The name of this class and its guards are always BridgeModule class name, in
// all-caps, suffixed with "_struct" and "_struct_guard" respectively.

#ifdef UARTBRIDGEMODULE_struct_guard
class uart_t: public bridge_driver_t
{
    public:
        uart_t(simif_t* sim, UARTBRIDGEMODULE_struct * mmio_addrs, int uartno);
        ~uart_t();
        virtual void tick();
        // Our UART bridge's initialzation and teardown procedures don't
        // require interaction with the FPGA (i.e., MMIO), and so we don't need
        // to define init and finish methods (we can do everything in the
        // ctor/dtor)
        virtual void init() {};
        virtual void finish() {};
        // Our UART bridge never calls for the simulation to terminate
        virtual bool terminate() { return false; }
        // ... and thus, never returns a non-zero exit code
        virtual int exit_code() { return 0; }

    private:
        UARTBRIDGEMODULE_struct * mmio_addrs;
        serial_data_t<char> data;
        int inputfd;
        int outputfd;
        int loggingfd;
        void send();
        void recv();
};
#endif // UARTBRIDGEMODULE_struct_guard

#endif // __UART_H

The bulk of the driver’s work is done in its tick() method. Here, the driver polls the BridgeModule and then does some work. Note: the name, tick is vestigial: one invocation of tick() may do work corresponding to an arbitrary number of target cycles. It’s critical that tick be non-blocking, as waiting for work from the BridgeModule may deadlock the simulator.

Registering the Driver

With the Bridge Driver implemented, we now have to register it in the main simulator simulator class defined in sim/src/main/cc/firesim/firesim_top.cc. Here, we rely on the C preprocessor macros to instantiate the bridge driver only when the corresponding BridgeModule is present:

    // Here we instantiate our driver once for each bridge in the target
    // Golden Gate emits a <BridgeModuleClassName>_<id>_PRESENT macro for each instance
    // which you may use to conditionally instantiate your driver
    #ifdef UARTBRIDGEMODULE_0_PRESENT
    // Create an instance of the constructor argument (this has all of
    // addresses of the BridgeModule's memory mapped registers)
    UARTBRIDGEMODULE_0_substruct_create;
    // Instantiate the driver; register it in the main simulation class
    add_bridge_driver(new uart_t(this, UARTBRIDGEMODULE_0_substruct, 0));
    #endif

    // Repeat the code above with modified indices as many times as necessary
    // to support the maximum expected number of bridge instances
    #ifdef UARTBRIDGEMODULE_1_PRESENT
    UARTBRIDGEMODULE_1_substruct_create;
    add_bridge_driver(new uart_t(this, UARTBRIDGEMODULE_1_substruct, 1));
    #endif

Build-System Modifications

The final consideration in adding your bridge concerns the build system. You should be able to host the Scala sources for your bridge with rest of your target RTL: SBT will make sure those classes are available on the runtime classpath. If you’re hosting your bridge driver sources outside of the existing directories, you’ll need to modify your target-project Makefrag to include them. The default Chipyard/Rocket Chip-based one lives here: sim/src/main/makefrag/firesim/Makefrag

Here the main order of business is to add header and source files to DRIVER_H and DRIVER_CC respectively, by modifying the lines below:

##########################
# Driver Sources & Flags #
##########################

firesim_lib_dir = $(firesim_base_dir)/firesim-lib/src/main/cc
driver_dir = $(firesim_base_dir)/src/main/cc
DRIVER_H = $(shell find $(driver_dir) -name "*.h") \
		   $(shell find $(firesim_lib_dir) -name "*.h")

DRIVER_CC = $(wildcard $(addprefix $(driver_dir)/, $(addsuffix .cc, firesim/*))) \
            $(wildcard $(addprefix $(firesim_lib_dir)/, $(addsuffix .cc, bridges/* fesvr/*)))  \
			$(RISCV)/lib/libfesvr.a

TARGET_CXX_FLAGS := -g -I$(firesim_lib_dir) -I$(driver_dir)/firesim -I$(RISCV)/include
TARGET_LD_FLAGS :=

That’s it! At this point you should be able to both test your bridge in software simulation using MIDAS-level simulation, or deploy it to an FPGA.