This library provides a set of Keras primitives (tf.keras.Layer and tf.keras.Model) that can be assembled into transformer-based models. They are flexible, validated, interoperable, and both TF1 and TF2 compatible.

• layers are the fundamental building blocks for NLP models. They can be used to assemble new tf.keras layers or models.

• networks are combinations of tf.keras layers (and possibly other networks). They are tf.keras models that would not be trained alone. It encapsulates common network structures like a transformer encoder into an easily handled object with a standardized configuration.

• models are combinations of tf.keras layers and models that can be trained. Several pre-built canned models are provided to train encoder networks. These models are intended as both convenience functions and canonical examples.

• losses contains common loss computation used in NLP tasks.

Please see the colab NLP modeling library intro.ipynb for how to build transformer-based NLP models using above primitives.

Besides the pre-defined primitives, it also provides scaffold classes to allow easy experimentation with noval achitectures, e.g., you don’t need to fork a whole Transformer object to try a different kind of attention primitive, for instance.

• TransformerScaffold implements the Transformer from ["Attention Is All You Need"] (https://arxiv.org/abs/1706.03762), with a customizable attention layer option. Users can pass a class to attention_cls and associated config to attention_cfg, in which case the scaffold will instantiate the class with the config, or pass a class instance to attention_cls.

• EncoderScaffold implements the transformer encoder from "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", with customizable embedding subnetwork (which will replace the standard embedding logic) and/or a custom hidden layer (which will replace the Transformer instantiation in the encoder).

Please see the colab customize_encoder.ipynb for how to use scaffold classes to build noval achitectures.

BERT and ALBERT models in this repo are implemented using this library. Code examples can be found in the corresponding model folder.