HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level

Synthesis

Andy He * 1 Darren Key * 1 Mason Bulling * 1 Andrew Chang * 1 Skyler Shapiro * 1 Everett Lee 1

Abstract 1. Introduction
arXiv:2405.00738v1 [cs.AR] 29 Apr 2024

Hardware accelerators have long appeared in comput-

ing (Merritt, 2021) to improve performance compared to
Graphics Processing Units (GPUs) have become general-purpose CPUs through specialized operations, high
the leading hardware accelerator for deep learn- parallelism, and efficient memory systems (Dally et al.,
ing applications and are used widely in training 2020). The use of accelerators for deep learning have
and inference of transformers; transformers have been especially significant to accommodate models that
achieved state-of-the-art performance in many ar- are rapidly scaling up in size and complexity, such as
eas of machine learning and are especially used transformer-based Large Language Models (LLMs) which
in most modern Large Language Models (LLMs). have become increasingly complex with a massive influx of
However, GPUs require large amounts of energy, research following the advent of OpenAI’s ChatGPT. Meta’s
which poses environmental concerns, demands popular Llama 2 model, for instance, is trained on 2 tril-
high operational costs, and causes GPUs to be lion tokens and ranges up to 70 billion parameters (Touvron
unsuitable for edge computing. We develop an et al., 2023a). GPUs are currently the dominant accelera-
accelerator for transformers, namely, Llama 2, tors for general deep learning tasks as they can be easily
an open-source state-of-the-art LLM, using high leveraged to develop extremely efficient implementations of
level synthesis (HLS) on Field Programmable parallel basic linear algebra subroutines (BLAS), which are
Gate Arrays (FPGAs). HLS allows us to rapidly commonly used in deep learning algorithms. (Xiong & Xu,
prototype FPGA designs without writing code at 2020).
the register-transfer level (RTL). We name our
method HLSTransform, and the FPGA designs However, the most glaring tradeoff to using GPUs is their
we synthesize with HLS achieve up to a 12.75x massive demand for power, resulting in high carbon emis-
reduction and 8.25x reduction in energy used per sions and energy costs. The carbon footprint of training
token on the Xilinx Virtex UltraScale+ VU9P Llama 2 is officially estimated at 539 tons carbon dioxide
FPGA compared to an Intel Xeon Broadwell E5- equivalent (Touvron et al., 2023b), which is almost 72x the
2686 v4 CPU and NVIDIA RTX 3090 GPU re- amount the average US household produces per year at 7.5
spectively, while increasing inference speeds by tons (CCFPD). However, while model training takes large
up to 2.46x compared to CPU and maintaining amounts of energy, energy spent running inference on the
0.53x the speed of an RTX 3090 GPU despite the model is typically larger; NVIDIA and Amazon estimate
GPU’s 4 times higher base clock rate. With the that over 80% of their energy usage for AI models is spent in
lack of existing open-source FPGA accelerators inference, and for Google, 60% of their energy usage for AI
for transformers, we open-source our code and models is for inference (McDonald et al., 2022) (Patterson,
document our steps for synthesis. We hope this 2022). Inference dominates emissions in ChatGPT-like ser-
work will serve as a step in democratizing the use vices from the querying of millions of users, producing 25x
of FPGAs in transformer inference and inspire the carbon emissions of GPT-3 (Samsi et al., 2023) (Chien
research into energy-efficient inference methods et al., 2023).
as a whole. The code can be found on GitHub. High energy consumption also poses a problem for oper-
ational costs and for edge computing applications. High
energy consumption forces the inference of LLMs and deep
*
learning models to be mostly allocated to GPU clusters. An
Equal contribution 1 Cornell University, Ithaca, NY. Corre- article from Sequoia Capital estimates that for data cen-
spondence to: Darren Key <[email protected]>.
ters, the price from energy alone is roughly equal to the
amount spent on buying GPUs (Cahn, 2023). For applica-

1
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

tions requiring real-time inference on the edge, in addition In summary, through our methods which we name
to monetary reasons, a dedicated GPU is often impractical HLSTransform, we demonstrate the following:
as it cannot draw sufficient and sustained power.
While GPU acceleration will likely remain dominant in the 1. Low power and energy consumption
near future despite the power disadvantage, there is value Energy savings up to a 12.75x reduction of total energy
in exploring different avenues of hardware acceleration as consumption compared to CPU and an 8.25x reduction
deep learning tasks continue to diverge into highly specific of total energy consumption compared to GPU.
applications. Further, as transformers become more and
more ubiquitous, there is a case to be made for designing 2. Fast inference speeds and low latency
model-specific hardware accelerators solely to optimize in- Acceleration up to 2.46x in inference speed in compar-
ference. To that end, Field Programmable Gate Arrays (FP- ison to CPU, and maintaining up to 0.53x in inference
GAs) are another desirable choice for accelerators as they speed in comparison to GPU, despite the GPU having
offer a hardware reconfigurable for specific tasks enabled by 4x higher base clock rate.
a large number of programmable logic gates, making them
inexpensive to iterate hardware designs on. Furthermore, 3. Verification of HLS tools for faster deployment
FPGAs are distinguished for their reduced power consump- Ensuring HLS tools run properly to synthesize appro-
tion, which on average is only 28% of GPUs (Cong et al., priate FPGA designs for this study. We also test the
2018). learning curve of the tools for the use of any developer
without extensive hardware backgrounds.
What limits the adoption of FPGAs currently is the high
barrier of entry and relative lack of research compared to
GPUs. FPGAs are commonly used to prototype hardware We open-source our code and document our FPGA syn-
designs for system-on-chip (SoC) and Application Specific thesis to the public, available in our GitHub repo here:
Integrated Circuit (ASIC), which is typically done on the github.com/HLSTransform/submission. To the
register-transfer level (RTL) using hardware description lan- best of our knowledge, our model is one of the first open-
guages like Verilog. However, the design and verification of source HLS-based implementations for transformers. In
RTL modules are known to be extremely complex and time- our research process, the lack of documentation for many
consuming. High Level Synthesis (HLS) is a methodology steps of the process combined with the absence of existing
that seeks to address that complexity by allowing developers open-source FPGA accelerators for transformers served as
to write hardware descriptions in more accessible, high-level a high barrier to entry, and we hope our work serves a step
languages like C or C++. HLS tools convert high-level code forward in democratizing the usage and research of FPGAs
input into RTL code that optimizes for performance, area, for transformer inference.
and energy consumption, leading to faster prototyping and
iteration for FPGAs. Furthermore, the nature of HLS tools 2. Related Work
and availability of Vitis C / RTL co-simulation make it sim-
ple to verify the correctness of the synthesized hardware We delineate a few studies that relate to FPGA accelerators
designs; these factors allow HLS to significantly shorten the for transformers and the application of high level synthesis.
traditional hardware development cycle.
2.1. Existing Hardware Accelerators for Transformers
In this literature, we employ HLS tools to design FPGAs for on FPGA
accelerating Llama 2 inference. In addition to the large GPU
power footprint of LLMs that may be addressed with FP- Existing hardware accelerators for transformers on FPGA
GAs, the complex data flow of transformer models (Li et al., incorporate specialized techniques to optimize performance
2020) often comprises of nonlinearities or token encoding on FPGAs. Column Balanced Block Pruning (Peng et al.,
subroutines (such as RoPE) that are difficult to accelerate 2021) and FTrans (Li et al., 2020) are two novel frameworks
on GPUs but could be better suited for FPGAs. Llama 2 is for transformer models suitable for FPGA acceleration. By
chosen in particular due to its open-source implementations incorporating weight pruning to employ sparse matrix mul-
and superb performance (Touvron et al., 2023b), making tiplication, these papers are able to achieve multiple folds of
it a popular and well researched choice. We use Andrej improvements in transformer inference compared to CPUs
Karpathy’s llama2.c repository (Karpathy, 2023) to develop and GPUs in terms of performance and energy efficiency.
our methods on a relatively small (110M parameters) model We instead strive to maintain dense matrix multiplication
to allow for our financial and compute constraints. We focus in our methods to allow for general application to exist-
on inference over training due to its higher energy usage ing transformer models. Similarly, NPE (Khan et al., 2021)
and greater suitability for FPGAs. introduces a framework for FPGA acceleration on transform-
ers, utilizing piecewise linear approximations for nonlinear

2
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

functions (e.g. softmax and GELU) to achieve speedups.

In contrast, we compute exact values for nonlinear func-
tions. Our methodology allows us to avoid needing to train Kernel Code

Host Code (C++)

FPGA-specific models and avoid potential accuracy trade- (C++/Verilog)

offs associated with these novel pruning or approximation

techniques. The only potential accuracy tradeoffs are from User Code g++ Vitis HLS

our usage of quantization, where we follow the well-tested Vitis Platform

quantization algorithm “Q8 0”, explored further in Section Host Executable Kernel Binary

3.2.
Xilinx Runtime

2.2. hls4ml
Hardware Platform

We aim to inspire the democratization of FPGA acceler-

ators for deep learning using HLS. Fast Machine Learn-
ing Lab’s hls4ml (Fahim et al., 2021) is an open-source
workflow that enables fast prototyping of machine learning Figure 1. Vitis HLS development workflow.
algorithms via Python for FPGAs (FastML Team, 2023)
(Duarte et al., 2018). hls4ml has been successful in being
one of the first open source HLS tools for deep learning espe- The host sends the input parameters, such as the token and
cially in Python, but a major limitation is its lack of support position to the FPGA via direct memory access (DMA).
for attention layers used in transformer models. The tool The FPGA is responsible for writing the output to a shared-
mainly supports convolutional neural networks (CNNs) and buffer that can be accessed by both the host and the kernel.
feed-forward deep neural networks (DNNs), but transformer The host reads the output and performs sampling to extract
models like Llama 2 are unique in requiring attention layers the next token.
and novel techniques such as Rotary Position Embeddings, We focus on three HLS optimizations: pipelining, unrolling,
which are not yet supported by this framework. and array partitioning. We also implement software-level op-
timizations; in addition to memory limitations, FPGAs also
3. Methods have constraints regarding Digital Signal Processor (DSP)
blocks, which are specialized hardware modules within an
We follow the same architecture outlined in the original FPGA that are optimized for efficient floating point arith-
Llama 2 paper (Touvron et al., 2023a): the standard Trans- metic calculations. However, the number of available DSP
former architecture (Vaswani et al., 2023), rotary posi- blocks is limited and varies depending on the FPGA model;
tion embeddings (Su et al., 2021), grouped query atten- to address DSP and on-chip memory bottlenecks, we first
tion (Ainslie et al., 2023), RMS norm for pre-normalization quantized the weights from 32-bit (single-precision) IEEE
(Zhang & Sennrich, 2019), and the SwiGlu activation func- floating points to 8-bit signed integers.
tion (Shazeer, 2020). Since FPGAs are constrained in per-
formance by the amount of on-chip memory, we selected 3.2. Int-8 Quantization
a small 110M parameter model trained on the TinyStories
dataset to test our designs (Eldan & Li, 2023). We discuss Included in Karpathy’s work, we employ an 8-bit integer
the limitations of the small model size further in the Lim- quantized forward pass to run our inference on FPGAs
itations and Future Works section. More details on model (Karpathy, 2023). The quantization process is post-training;
architecture are included in the Appendix. i.e. it is independent from model training.
We perform symmetric quantization, scaling each weight
3.1. Implementation between [-127, 127]. Each weight is divided into sections
Our implementation of Llama 2 is built on Andrej Karpa- of equal size, each of which is quantized by the following
thy’s llama2.c repository. For our HLS toolchain, we chose formula, where w here represents a vector of weights in
Vitis, as it is both widely used and directly supported by that section and the square brackets denote the rounding
the FPGAs available to us on AWS. The code is split into function.
two portions, the host and the kernel. The kernel code
contains the hardware description for one iteration of the w
w = ⌈127 ∗ ⌋
computationally-intensive forward inference pass and is ∥w∥∞
synthesized for the FPGA, while the host is responsible for
driving the kernel code. The host interfaces with the FPGA This quantization has been noted to perform well empiri-
accelerator through the Xilinx Runtime Library (XRT). cally, used in Georgi Gerganov’s popular GGML library

3
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

for efficient CPU transformer inference and referred to as to increased throughput. The pipeline pragma is applied
“Q8 0” quantization in the library (Gerganov). We quantize to the main loops responsible for computing matrix-vector
the embedding, attention, and the feedforward weights. The multiplication and rotary position embeddings.
RMSNorm params, which are sensitive to error, are kept in
float32 precision. 3.3.2. L OOP U NROLLING
Although quantization leads to decreased model accuracy, Loop unrolling is an optimization technique that increases
the accuracy dropoff is minimal, and we explore the ef- the efficiency of hardware implementations derived from
fects of quantization in Section 4.1. Quantization allows high-level code. This process involves expanding the loop
for smaller weights, which permits us to better utilize body multiple times in order to reduce the number of itera-
the limited memory bandwidth on the FPGA and per- tions. By doing this, loop unrolling enables the simultaneous
form integer-only calculations, which provides inference execution of multiple consecutive loop iterations, as long as
speedups through lower precision arithmetic calculations there are no intra-loop data dependencies.
(Kim et al., 2021).
In other words, if a loop is executed N times and we un-
roll it M times, the loop body will be replicated M times
3.3. Optimization of Llama 2 Accelerator Using HLS
within each iteration, thereby reducing the total number
Pragmas
of iterations to N/M . This technique is especially useful
Pragmas in High-Level Synthesis (HLS) are directives used in hardware design because it can lead to more parallel
to guide the HLS compiler in the process of converting the operations, allowing the hardware to perform more tasks
high-level code into a hardware description, typically used simultaneously at the cost of chip space.
when indicating to the compiler that a specific optimization
should be performed on some section of the code. 3.3.3. M EMORY PARTITIONING
The application of HLS partitioning pragmas is a critical
step in the design of the Llama 2 deep learning accelera-
for(int i = 0; i < 2; i++) {

tor. Typically, FPGA BRAM is implemented as a dual-port

op_read;
RD memory, which greatly restricts the degree to which code
op_execute;
EX can be parallelized on chip. By dividing arrays and memory
op_write;
WR
} structures into smaller, independent blocks, different data
segments can be processed in parallel. Memory partitioning
ensures more efficient utilization of the available compu-
RD RD
EX WR RD EX WR RD RD
EX WR tational resources, thereby enhancing the throughput for
RD EX WR
matrix multiplication operations, a common bottleneck in
No pipelining
Pipelining with II = 1

(6 cycle latency) (4 cycle latency)

neural network computations.

3.3.4. B URST R EADS / W RITES OVER AXI4

Figure 2. Pipelining two iterations of instructions with read, exe- In general, a dual-port memory bank can support two reads
cute, and write stages. per cycle. Since global memory cannot be partitioned com-
pletely due to the limitation on the number of memory chan-
3.3.1. P IPELINING nels available to the FPGA, we instead utilize burst reads
and writes into local on-chip buffers. By using a technique
Pipelining HLS is a technique used to enhance the perfor- called widening, global memory can be accessed through
mance of hardware circuits generated from high-level code. dual-port 256-bit wide lines, allowing the simultaneous read
This method involves dividing a process into several stages, of 64 8-bit integers per cycle. Efficient data transfer between
each separated by registers. Similar to an assembly line, the FPGA and external memory is essential, given the large
pipelining allows different stages of a computation to oc- amount of parameters that need to be read from memory
cur in parallel but on different sets of data. In HLS, this before any computations can begin.
means translating high-level programming constructs into
pipelined hardware structures.
4. Results and Discussion
For example, in a computation involving multiple arith-
metic operations, HLS can break down these operations We evaluate the perplexity, latency, power, and energy con-
into stages, where each stage performs a part of the com- sumption of the 110M parameter Llama 2 model across
putation. By doing so, while one stage is processing one CPU, GPU, and FPGA. We provide more details of the eval-
set of data, the next stage can work on another, leading uation setup in the Appendix. We run our benchmarks for

4
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

256 tokens and the max context length of 1024 tokens to sults from the system simulations (Khan et al., 2021), and
test both the short and long text generation domains. we provide a report of our full timings in the Appendix.
Our FPGA designs were synthesized targeting the Ultra-
scale+ VU9P platform available on AWS, and the synthe- Table 2. I NFERENCE SPEED ( TOKENS PER SECOND )
sized designs were then exported to an Amazon Machine
Image (AMI) using a custom toolchain provided by Ama- H ARDWARE 256 TOKENS ↑ 1024 TOKENS ↑
zon (AWS). We use the f1.2xlarge instance from AWS to CPU 23.21 TOKS / S 19.63 TOKS / S
host the FPGA, and we use the t2.2xlarge instance for our GPU 107.00 TOKS / S 107.24 TOKS / S
CPU benchmarks (8 vCPUs, 2.3 GHz Intel Xeon Broadwell FPGA 57.11 TOKS / S 57.11 TOKS / S
E5-2686 v4), the same CPUs used in the FPGA instance,
and an NVIDIA RTX 3090 GPU for our GPU benchmarks.
We use the original Llama 2 implementation provided by
Meta for our GPU experiments. We run all samples with Table 3. I NFERENCE L ATENCY ( MILLISECONDS )
non-batched inference (batch size 1).
H ARDWARE 256 TOKENS ↓ 1024 TOKENS ↓
While we run benchmarks of FPGA performance against
CPUs and GPUs, we are unable to provide equitable quan- CPU 43.08 MS 50.94 MS
GPU 9.34 MS 9.32 MS
tized benchmarks for GPUs, as the different scaling factors FPGA 17.51 MS 17.51 MS
per section in the quantization algorithm used would require
specialized kernels to make this efficient. To provide equi-
table comparisons, we also provide perplexity benchmarks, According to Table 2, the FPGA is 2.46x the inference speed
a common metric for model quality, along with inference of CPU and 0.53x the inference speed of GPU.
latency and energy consumption benchmarks to demonstrate
minimal tradeoffs to accuracy while fully utilizing the opti- Although the GPU performs inference faster than the FPGA,
mized integer-arithmetic abilities of FPGAs. one of the primary bottlenecks of deep learning inference is
memory bandwidth and the availability of on-chip memory
4.1. Perplexity (Balasubramanian et al., 2021). A RTX 3090 has 24GB
VRAM running at 1219 MHz with a base core clock of 1395
We measure perplexity on the validation dataset for TinyS- MHz (TechPowerUp, 2024). In comparison, a VU9P FPGA
tories, for both the quantized and unquantized models of the has 345.9 MB of combined on-chip BRAM and URAM,
110M parameter model; perplexity is a common metric for running at a much slower clock speed of around 200-300
model quality that measures a model’s uncertainty about its MHz depending on the module; however, with much lower
predictions. Our experimental setup is detailed further in clock speeds, the FPGA is able to achieve better efficiency
the Appendix. on power and energy consumption, as shown below.

4.3. Energy and Power Consumption

Table 1. P ERPLEXITY ( LOWER IS BETTER )
We utilize the CodeCarbon library, also used by Hugging-
M ODEL AVERAGE PERPLEXITY ( PPL ) ↓ Face to provide carbon estimations for the BLOOM LLM, to
Q UANTIZED 110M 2.9679 provide energy consumption metrics for CPU and GPU per-
U NQUANTIZED 110M 2.9667 formance (Heikkiläarchive, 2022) (Workshop et al., 2022)
U NQUANTIZED 42M 3.1810 (Courty et al., 2023). For GPU benchmarks, CodeCar-
bon sources energy consumption directly from NVIDIA’s
The quantized model is able to retain nearly identical levels NVML library. For the AWS CPU benchmarks, energy
of performance (0.04% increase in perplexity) as the un- consumption cannot be directly sourced since AWS uses hy-
quantized model while utilizing integer only computations. pervisors, and CodeCarbon uses an estimation derived from
We include the perplexity benchmark for a 42 million pa- empirical energy consumption data (Courty et al., 2023).
rameter model as reference, which is 7.22% higher than the As CodeCarbon does not handle FPGA energy consumption
unquantized 110 million parameter model. measurement, energy consumption metrics for FPGA is
provided by Vivado and AWS provided tools (AWS).
4.2. Latency and Speed
The average power consumption of the FPGA is consider-
We measure inference latency in milliseconds and inference ably lower than the average power consumption for both
speed in tokens per second. Similar to NPE, an existing CPU and GPU. For 256 tokens, the average FPGA power
hardware accelerator for FPGAs, we obtain our timing re- consumption achieves a 4.72x reduction in the average

5
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

VU19P, has an on-chip memory capacity of 224 MB (AMD).

Table 4. P OWER CONSUMPTION ON FPGA (WATTS )
In contrast, most LLMs are much larger than the maximum
size FPGAs can load on chip; for instance, Llama 2 has three
FPGA 256 TOKENS ↓ 1024 TOKENS ↓
pretrained LLMs of size 7, 13, and 70 billion, while GPT-3
AVERAGE 9W 9W uses 175 billion parameters (Touvron et al., 2023a) (Brown
M AX 12 W 11 W
et al., 2020). Since the parameters cannot be pre-initialized
on on-chip memory banks due to memory constraints, the
weights are instead on off-chip global memory interfaced
Table 5. AVERAGE POWER CONSUMPTION (WATTS ) via the AXI4 protocol, making it possible to run inference on
larger models. However, external memory accesses quickly
H ARDWARE 256 TOKENS ↓ 1024 TOKENS ↓ become a major bottleneck in inference latency as only 64
CPU 42.5 W 42.5 W 8-bit integers can be read per cycle.
GPU 126.9 W 130.6 W
FPGA 9W 9W As a result, we limit our model size to 110M parameters.
Despite the model size, there are many practical applications
of similar model sizes. For instance, BERT has a base
model size of 110M parameters, while ALBERT xlarge
power consumption of the CPU, and a 14.10x reduction has a model size of 68M parameters; these models achieve
in the average power consumption of the GPU. For 1024 state-of-the-art or near state-of-the-art performances on a
tokens, the FPGA achieves a 14.51x reduction of the power multitude of NLP tasks and are in widespread use (Rogers
consumption of the GPU, reaching a maximum of only 12 et al., 2020). Several Llama variants, such as LiteLlama
watts. and TinyLlama, also have considerably smaller parameter
To calculate the total energy consumption, we need the sizes of 460M parameters and 1.1B parameters respectively,
duration of inference; therefore we introduce a new metric, while achieving considerable generation capabilities for the
the total energy consumption per token, calculated by using size (Han) (Zhang et al., 2024).
the inference latency and average power consumption. We Several future directions to be explored for fitting larger
measure the energy consumption per token in milliwatt models on FPGA include using greater levels of quanti-
hours per token. zation (i.e. 4-bit precision) or using multiple FPGAs in
unison. “Q4 0” quantization utilizes the same quantization
Table 6. T OTAL ENERGY CONSUMPTION ( MILLIWATT HOUR PER technique applied to 4-bit integers, and has seen success in
TOKEN , M W H / TOK ) implementations in Gerganov’s GGML library, and ongo-
ing research exists for other quantization schemes, such as
H ARDWARE 256 TOKENS ↓ 1024 TOKENS ↓ 2-bit LLMs (Chee et al., 2023). Fully-integer quantization
CPU 0.51 M W H / TOK 0.60 M W H / TOK methods also serve as a potential research path, which both
GPU 0.33 M W H / TOK 0.34 M W H / TOK reduces parameter size and inference latency by making all
FPGA 0.04 M W H / TOK 0.04 M W H / TOK weights and all calculations involve only integers, such as
the ones explored in I-BERT (Kim et al., 2021). Model par-
allelism schema utilizing multiple FPGAs may also help run
For 256 tokens, the FPGA reaches a 12.75x reduction in larger models by sharding a model across multiple FPGAs.
energy consumption over the CPU and 8.25x reduction in
energy consumption over the GPU, while for 1024 tokens, 5.2. Batch Size
the FPGA achieves a 15x reduction over the CPU and a 8.5x
reduction over the GPU. Through HLSTransform, we are Another limitation of our work is our focus on the non-
able to achieve high savings in energy per token. batched inference domain; i.e. inference with batch size 1.
The large VRAM capacity and parallel computation nature
of GPUs make the GPUs suitable for tasks requiring high
5. Limitations and Future Work
throughput, which may make the GPU overall more power
We note several limitations regarding our work, and we efficient in the high batch regime. An interesting future
provide potential research directions: research direction is the optimization of batched inference
on FPGAs.
5.1. Model Size
A key limitation of our work is the on-chip memory bot-
tlenecks that accompany FPGAs; for example, one of Xil-
inx’s high-end commercial FPGAs, the Virtex UltraScale+

6
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

6. Conclusion CCFPD. C02 facts chart. URL https:

//www.ccfpd.org/Portals/0/Assets/PDF/
We propose a new hardware accelerator for transformers Facts Chart.pdf.
on FPGA, HLSTranform, which achieves up to a 12.75x
reduction and 8.25x reduction in total energy consumption Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit
per token, compared to a 2.3 GHz Intel Xeon Broadwell E5- quantization of large language models with guarantees.
2686 v4 CPU and a NVIDIA RTX 3090 GPU, respectively. 2023.
Our FPGA accelerator maintains 0.53x the inference speed
of an RTX 3090 GPU and is 2.46x as fast as the inference Chien, A. A., Lin, L., Nguyen, H., Rao, V., Sharma, T., and
speed of the Intel Xeon Broadwell E5-2686 v4 CPU; these Wijayawardana, R. Reducing the carbon impact of gener-
results are achieved via synthesis combined with pipelining, ative ai inference (today and in 2035). In Proceedings of
memory unrolling, and memory partitioning and transfer the 2nd Workshop on Sustainable Computer Systems, pp.
optimizations, with the addition of 8-bit integer quantization. 1–7, 2023.
Through our study, we provide a proof-of-concept for the
Cong, J., Fang, Z., Lo, M., Wang, H., Xu, J., and Zhang,
usage of High Level Synthesis (HLS) as a much quicker
S. Understanding performance differences of fpgas and
way of prototyping FPGA designs.
gpus. In 2018 IEEE 26th Annual International Sympo-
As transformers become more widely used and as model sium on Field-Programmable Custom Computing Ma-
sizes continue to increase, energy consumption from AI- chines (FCCM), pp. 93–96. IEEE, 2018.
related applications will increase correspondingly. In-
creased energy consumption comes with vast environmental Courty, B., Schmidt, V., Goyal-Kamal, M., Feldand, B.,
concerns and monetary costs, as well as limiting applica- Lecourt, J., et al. Codecarbon: Estimate and track
tions that restrict power consumption such as edge comput- carbon emissions from machine learning computing.
ing; as a result, energy-efficient methods for inference that 2023. doi: 10.5281/zenodo.4658424. URL https://
provide more sustainable solutions may become a much zenodo.org/doi/10.5281/zenodo.4658424.
more pressing issue. We hope that our work serves as a step Dally, W. J., Turakhia, Y., and Han, S. Domain-specific
forward in energy-efficient methods for AI. hardware accelerators. Communications of the ACM,
63(7), 48-57, 2020.
References
Duarte, J. et al. Fast inference of deep neural networks in
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., fpgas for particle physics. JINST, 13(07):P07027, 2018.
Lebrón, F., and Sanghai, S. Gqa: Training generalized doi: 10.1088/1748-0221/13/07/P07027.
multi-query transformer models from multi-head check-
points. 2023. Eldan, R. and Li, Y. Tinystories: How small can language
models be and still speak coherent english? 2023.
AMD. Amd virtex ultrascale+. URL https:
//www.xilinx.com/products/silicon- Fahim, F., Hawks, B., Herwig, C., Hirschauer, J., Jindariani,
devices/fpga/virtex-ultrascale-plus- S., Tran, N., Carloni, L. P., Di Guglielmo, G., Harris, P.,
vu19p.html#ProductTable. Krupa, J., et al. hls4ml: An open-source codesign work-
flow to empower scientific low-power machine learning
AWS. aws-fpga. URL https://github.com/aws/ devices. arXiv preprint arXiv:2103.05579, 2021.
aws-fpga.
FastML Team. fastmachinelearning/hls4ml. 2023.
Balasubramanian, A., Kumar, A., Liu, Y., Cao, H., doi: 10.5281/zenodo.1201549. URL https://
Venkataraman, S., and Akella, A. Accelerating deep github.com/fastmachinelearning/hls4ml.
learning inference via learned caches. 2021. Gerganov, G. ggml. URL https://github.com/
ggerganov/ggml.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Han, X. Litellama-460m-1t. URL https://
Askell, A., et al. Language models are few-shot learners. huggingface.co/ahxt/LiteLlama-460M-1T.
Advances in neural information processing systems, 33:
1877–1901, 2020. Heikkiläarchive, M. We’re getting a better idea
of ai’s true carbon footprint. 2022. URL
Cahn, D. AI $200b question. 2023. URL https://www.technologyreview.com/2022/
https://www.sequoiacap.com/article/ 11/14/1063192/were-getting-a-better-
follow-the-gpus-perspective/. idea-of-ais-true-carbon-footprint/.

7
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

Karpathy, A. llama2.c. 2023. URL https:// Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
github.com/karpathy/llama2.c. A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
Bhosale, S., et al. Llama 2: Open foundation and fine-
Khan, H., Khan, A., Khan, Z., Huang, L. B., Wang, tuned chat models. arXiv preprint arXiv:2307.09288,
K., and He, L. Npe: An fpga-based overlay pro- 2023a.
cessor for natural language processing. 2021. doi:
10.1145/3431920.3439477. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Bhosale, S., et al. Llama 2: Open foundation and fine-
Keutzer, K. I-bert: Integer-only bert quantization. 2021. tuned chat models. arXiv preprint arXiv:2307.09288,
2023b.
Li, B., Pandey, S., Fang, H., Lyv, Y., Li, J., Chen, J., Xie, M.,
Wan, L., Liu, H., and Ding, C. Ftrans: energy-efficient Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
acceleration of transformers using fpga. In Proceedings of L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention
the ACM/IEEE International Symposium on Low Power is all you need. 2023.
Electronics and Design, pp. 175–180, 2020. Workshop, B., Scao, T. L., Fan, A., Akiki, C., Pavlick, E.,
Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon,
McDonald, J., Li, B., Frey, N., Tiwari, D., Gadepally, V.,
F., et al. Bloom: A 176b-parameter open-access multilin-
and Samsi, S. Great power, great responsibility: Rec-
gual language model. arXiv preprint arXiv:2211.05100,
ommendations for reducing energy for training language
2022.
models. arXiv preprint arXiv:2205.09646, 2022.
Xiong, C. and Xu, N. Performance comparison of blas on
Merritt, R. What is accelerated computing? NVIDIA Blog?, cpu, gpu and fpga. 2020 IEEE 9th Joint International
2021. Information Technology and Artificial Intelligence Con-
ference (ITAIC), 2020.
Patterson, D. Good news about the carbon footprint
of machine learning training. 2022. URL https: Zhang, B. and Sennrich, R. Root mean square layer normal-
//blog.research.google/2022/02/good- ization. 2019.
news-about-carbon-footprint-of.html.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An
Peng, H., Huang, S., Geng, T., Li, A., Jiang, W., Liu, H., open-source small language model. 2024.
Wang, S., and Ding, C. Accelerating transformer-based
deep learning models on fpgas using column balanced
block pruning. In 2021 22nd International Symposium on
Quality Electronic Design (ISQED), pp. 142–148, 2021.
doi: 10.1109/ISQED51717.2021.9424344.

Rogers, A., Kovaleva, O., and Rumshisky, A. A primer in

bertology: What we know about how bert works. 2020.

Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A.,
Jones, M., Bergeron, W., Kepner, J., Tiwari, D., and
Gadepally, V. From words to watts: Benchmarking the
energy costs of large language model inference. In 2023
IEEE High Performance Extreme Computing Conference
(HPEC), pp. 1–9. IEEE, 2023.

Shazeer, N. Glu variants improve transformer. 2020.

Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu,
Y. Roformer: Enhanced transformer with rotary position
embedding. 2021.

TechPowerUp. Nvidia geforce rtx 3090. 2024. URL

https://www.techpowerup.com/gpu-specs/
geforce-rtx-3090.c3622.

8
 HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis

A. Appendix
A.1. Experimental Setup
For all our experiments, we use a sampling temperature of 1, an empty prompt (prompt is “”), and top-p sampling at 1. We
run all our experiments 100 times and take the average for our results.
We use Karpathy’s provided 110M model, which has an embedding dim of 768, 12 layers. 12 heads, 12 KV heads, and a
max context length of 1024.

A.2. Timing Results

Table 7. We obtain our timing results from the synthesis as shown below.
Module Name Start Interval Best (cycles) Avg (cycles) Worst (cycles) Best (absolute) Avg (absolute) Worst (absolute)
forward Pipeline 1 771 771 771 771 3.084 us 3.084 us 3.084 us
rmsnorm 768 Pipeline 1 770 770 770 770 3.080 us 3.080 us 3.080 us
rmsnorm 768 Pipeline 2 771 771 771 771 3.084 us 3.084 us 3.084 us
rmsnorm 768 Pipeline sum of squares 5413 5413 5413 5413 21.652 us 21.652 us 21.652 us
rmsnorm 768 Pipeline norm and scale 23 23 23 23 92.000 ns 92.000 ns 92.000 ns
rmsnorm 768 Pipeline 5 770 770 770 770 3.080 us 3.080 us 3.080 us
rmsnorm 768 s 7822 7822 7822 7822 31.288 us 31.288 us 31.288 us
round 1 1 1 1 4.000 ns 4.000 ns 4.000 ns
p hls fptosi float i8 1 1 1 1 4.000 ns 4.000 ns 4.000 ns
quantize 768 Pipeline main loop 198 198 198 198 0.792 us 0.792 us 0.792 us
quantize 768 Pipeline 2 770 770 770 770 3.080 us 3.080 us 3.080 us
quantize 768 Pipeline 3 14 14 14 14 56.000 ns 56.000 ns 56.000 ns
quantize 768 s 971 971 971 971 3.884 us 3.884 us 3.884 us
matmul 768 768 Pipeline x buff 50 50 50 50 0.200 us 0.200 us 0.200 us
matmul 768 768 Pipeline xs buff 5 5 5 5 20.000 ns 20.000 ns 20.000 ns
matmul 768 768 Pipeline VITIS LOOP 225 1 20900 20900 20900 20900 83.600 us 83.600 us 83.600 us
matmul 768 768 s 20977 20977 20977 20977 83.908 us 83.908 us 83.908 us
pow generic float s 1 15 15 15 60.000 ns 60.000 ns 60.000 ns
sin or cos float s 1 18 18 18 72.000 ns 72.000 ns 72.000 ns
forward Pipeline rotation1 119 119 119 119 0.476 us 0.476 us 0.476 us
forward Pipeline 3 839 839 839 839 3.356 us 3.356 us 3.356 us
forward Pipeline 4 839 839 839 839 3.356 us 3.356 us 3.356 us
forward Pipeline iterate 530 1̃554 530 1042 1554 2.120 us 4.168 us 6.216 us
forward Pipeline max 2 2̃61 2 133 261 8.000 ns 0.532 us 1.044 us
forward Pipeline exp 24 5̃6 24 40 56 96.000 ns 0.160 us 0.224 us
forward Pipeline sum 10 1̃546 10 778 1546 40.000 ns 3.112 us 6.184 us
forward Pipeline norm 9 2̃5 9 17 25 36.000 ns 68.000 ns 0.100 us
forward Pipeline 10 66 66 66 66 0.264 us 0.264 us 0.264 us
forward Pipeline acc 89 1̃625 89 857 1625 0.356 us 3.428 us 6.500 us
forward Pipeline residual 61 61 61 61 0.244 us 0.244 us 0.244 us
matmul 768 2048 Pipeline x buff 50 50 50 50 0.200 us 0.200 us 0.200 us
matmul 768 2048 Pipeline xs buff 5 5 5 5 20.000 ns 20.000 ns 20.000 ns
matmul 768 2048 Pipeline VITIS LOOP 225 1 55460 55460 55460 55460 0.222 ms 0.222 ms 0.222 ms
matmul 768 2048 s 55537 55537 55537 55537 0.222 ms 0.222 ms 0.222 ms
forward Pipeline swi glu 552 552 552 552 2.208 us 2.208 us 2.208 us
forward Pipeline 14 2050 2050 2050 2050 8.200 us 8.200 us 8.200 us
quantize 2048 Pipeline main loop 221 221 221 221 0.884 us 0.884 us 0.884 us
quantize 2048 Pipeline 2 2050 2050 2050 2050 8.200 us 8.200 us 8.200 us
quantize 2048 Pipeline 3 34 34 34 34 0.136 us 0.136 us 0.136 us
quantize 2048 s 2274 2274 2274 2274 9.096 us 9.096 us 9.096 us
matmul 2048 768 Pipeline x buff 130 130 130 130 0.520 us 0.520 us 0.520 us
matmul 2048 768 Pipeline xs buff 10 10 10 10 40.000 ns 40.000 ns 40.000 ns
matmul 2048 768 Pipeline VITIS LOOP 225 1 52526 52526 52526 52526 0.210 ms 0.210 ms 0.210 ms
matmul 2048 768 s 52659 52659 52659 52659 0.211 ms 0.211 ms 0.211 ms
forward Pipeline residual2 58 58 58 58 0.232 us 0.232 us 0.232 us
matmul 768 32000 Pipeline x buff 50 50 50 50 0.200 us 0.200 us 0.200 us
matmul 768 32000 Pipeline xs buff 5 5 5 5 20.000 ns 20.000 ns 20.000 ns
matmul 768 32000 Pipeline VITIS LOOP 225 1 864190 864190 864190 864190 3.457 ms 3.457 ms 3.457 ms
matmul 768 32000 s 864311 864311 864311 864311 3.457 ms 3.457 ms 3.457 ms
forward 4160108 4̃892636 4160107 4377403 4892635 16.640 ms 17.510 ms 19.571 ms