# FAT - FPGA Acceleration of Transformers

FAT is a research prototype for splitting transformer-style ONNX models and
offloading one or more encoder layers to an FPGA accelerator. The Python CLI
handles model preparation, optional ONNX quantization, model slicing, FPGA
kernel build orchestration, and inference/evaluation against an ONNX Runtime
reference.

The repository intentionally keeps the model/data payloads separate from the
source code. All model/data payloads must be located at `data/<name>/`.



## Repository Layout

- `main.py` is the CLI entry point.
- `build_fpga.py` stages repo-local Xilinx build assets into a generated
  workspace, renders model settings into the staged HLS source, and launches
  Vitis HLS/Vivado when requested.
- `extraction.py` loads `splice.py` files and extracts encoder parameters from
  ONNX initializers.
- `inference.py` runs ONNX-only, split-model, FPGA, and comparison inference.
- `kernel.py` wraps the PYNQ overlay, DMA buffers, and accelerator registers.
- `quantization.py` creates an ONNX Runtime static-quantized model.
- `utils.py` loads `model.yaml` and provides small C++ header export helpers.

## Data Directory Contract

Each experiment should live in its own directory:

```text
data/<experiment>/
  model.onnx
  model.yaml
  input.pkl
  splice.py
```

Required files:

- `model.onnx`: the ONNX model to inspect, split, quantize, or run.
- `model.yaml`: model/kernel metadata plus parameter extraction rules. The
  `model` section describes dimensions such as sequence length, hidden size,
  head count, encoder count, and feed-forward size. The `kernel` section
  describes FPGA storage/accumulation precision and how many encoder layers are
  offloaded. The `parameters` section maps the expected accelerator parameter
  names to ONNX initializer names or node-input lookup rules.
- `input.pkl`: a pickled collection of input samples for inference. Samples must
  match the ONNX model input shape. If the model uses padding, padded rows are
  expected to use the same sentinel convention as the model export.
- `splice.py`: defines `splice(data_dir, count)`. This function chooses the
  ONNX split points and usually calls `extraction.split_into_3(...)` to produce
  `model_a.onnx`, `model_b.onnx`, and `model_c.onnx`.

Optional files:

- `encoder.bit` and `encoder.hwh`: FPGA overlay artifacts used by `kernel.py`.
  These can be generated with `--synthesis` or supplied with an example data
  directory.
  
Generated files:

- `model_a.onnx`, `model_b.onnx`, `model_c.onnx`: split model stages created by
  `--split`.
- `model_quant.onnx`: quantized model created by `--quantize`.
- `output.pkl`: inference output written by `--onnx` or `--inference`.
- `encoder.bit`, `encoder.hwh`: copied into the data directory after a Vivado
  synthesis run.

## Setup

Create a Python environment and install the runtime dependencies:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

ONNX-only workflows can run on a normal workstation. FPGA inference requires a
PYNQ-capable target with the expected overlay IP names and DMA layout. Synthesis
requires external Vivado/Vitis installations, while the project assets now live
under `xilinx/` in this repository.

## Common Commands

Inspect an ONNX model and its initializers:

```bash
python main.py data/<experiment> --view
```

Run the original ONNX model only:

```bash
python main.py data/<experiment> --onnx
```

Split the model into pre-encoder, FPGA-offloaded encoder, and post-encoder
stages:

```bash
python main.py data/<experiment> --split
```

Run FPGA inference after split models and overlay artifacts are present:

```bash
python main.py data/<experiment> --inference
```

Compare FPGA output with the ONNX split-stage reference:

```bash
python main.py data/<experiment> --evaluate
```

Use deterministic synthetic inputs instead of `input.pkl` for a quick smoke
test:

```bash
python main.py data/<experiment> --onnx --random-inputs --random-count 4 --random-seed 1
```

Quantize `model.onnx` into `model_quant.onnx`:

```bash
python main.py data/<experiment> --quantize
```

Preview the FPGA build workflow without launching vendor tools:

```bash
python main.py data/<experiment> --synthesis --build-dry-run
```

Run only one synthesis stage:

```bash
python main.py data/<experiment> --synthesis vitis
python main.py data/<experiment> --synthesis vivado
```

## FPGA Build Configuration

`build_fpga.py` resolves tool and Xilinx asset paths from environment
variables. By default it uses the repo-local `xilinx/` tree and stages each
build into `.cache/<experiment>/`:

- `FAT_XILINX_ROOT`
- `FAT_BUILD_ROOT`

- `FAT_HLS_DIR`
- `FAT_VIVADO_DIR`
- `FAT_HLS_SCRIPT`
- `FAT_VIVADO_TCL`
- `FAT_VITIS_SETTINGS`
- `FAT_VIVADO_SETTINGS`
- `FAT_HLS_SOURCE`
- `FAT_VIVADO_OUT_DIR`
- `FAT_VIVADO_JOBS`

If Vivado or Vitis are not installed in a standard location, set
`FAT_VITIS_SETTINGS` and `FAT_VIVADO_SETTINGS` to the corresponding
`settings64.sh` files. FAT also checks common local layouts such as `tools/`
near the release directory and its parent directories.

The HLS source must contain marker blocks named `DATA_TYPES`, `MODEL_CONFIG`,
and `NUM_LAYERS`. During synthesis, FAT replaces those blocks with values from
`model.yaml` in the staged workspace before running the vendor tools, so the
versioned source under `xilinx/` is never edited in place.

Vivado projects are recreated from Tcl instead of checked-in `.xpr` state. FAT
stages `xilinx/vivado/run.tcl` and `xilinx/vivado/create_bd.tcl`, points Vivado
at the staged HLS IP repository from `hls/export/export`, creates a fresh
ZCU102 project under `.cache/<experiment>/vivado/project/`, and copies the
resulting `encoder.bit` and `encoder.hwh` back into the data directory.

## Naming Note

Some FPGA/HLS files still use the name `siso`. This is short for
single-in, single-out, from one of the earliest accelerator implementations.
The name stuck because changing it later would have touched many build scripts,
generated interfaces, and integration points.