Huawei PanGu

Huawei PanGu

Developer(s)	Huawei
Initial release	3.0, July 7, 2023 (2023-07-07)
Stable release	5.0 / June 21, 2024 (2024-06-21)
Available in	Chinese, English, Russian
Type	Large language model
License	Proprietary

Huawei PanGu, PanGu, PanGu-Σ or PanGu-π (Chinese: 盘古大模型; pinyin: pángǔ dà móxíng) is a multimodal large language model developed by Huawei. It was announced on July 7, 2023, positioned as a contender to other multimodal large language models.

The name of the large learning language model, PanGu, was derived from the Chinese mythology and folklore of Pangu, a primordial character related to the creation of the world.

History

Early development

In April 2023, Huawei released a paper detailing the development of PanGu-Σ, a colossal language model featuring 1.085 trillion parameters. Developed within Huawei's MindSpore 5 framework, PanGu-Σ underwent training for over 100 days on a cluster system equipped with 512 Ascend 910 AI accelerator chips, processing 329 billion tokens in more than 40 natural and programming languages.

PanGu-Σ incorporates Random Routed Experts (RRE) and the Transformer decoder architecture, allowing easy extraction of sub-models for various applications like conversation, translation, code production, and natural language interpretation. The model achieves 6.3 times faster training throughput compared to MoE models with the same hyper-parameters. In the Chinese domain, it outperforms previous state-of-the-art models across 16 tasks in a zero-shot setting. Trained on datasets from 40 domains, including Chinese, English, Bilingual, and code, PanGu-Σ excels in few-shot natural-language understanding, open-domain discussion, question answering, machine translation, and code creation.

Launch

During the Huawei Developer Conference on July 7, 2023, Huawei introduced PanGu 3.0, a large language model (LLM), tailored for sectors like government, finance, manufacturing, mining, and meteorology utilizing Huawei Cloud solutions. In the subsequent month, Huawei launched the Celia Virtual Assistant with advanced AI features, capable of generating long text replies based on user voice commands and set to release with HarmonyOS 4.0 for eligible devices.

The LLM was designed for enterprises seeking advantages in the AI industry, focusing on task execution over creative work, unlike traditional models used for general purposes like chatbots, poetry, and visual content creation.

Using the same technology as ChatGPT, Huawei's LLM features a hierarchical architecture, allowing customers to adapt the model to various tasks and train it on their own datasets, making it versatile across various industries.

Updates

On August 5, 2023, Huawei partnered with European Centre for Medium-Range Weather Forecasts (ECMWF) to launch a global weather forecasting AI model. This model used Huawei Cloud solutions and the PanGu-Weather Model with MindSpore. It is accessible on the ECMWF website and aims to provide accurate weather data.

On December 19, 2023, Huawei announced its financial services on the PanGu-powered AI Finance platform for the global market. The tech giant introduced this product at the 2023 Huawei Cloud Fintech Summit, aiming to reshape the digital finance industry with efficient features to boost Fintech firms worldwide. The platform incorporated a variety of advanced technologies, including AI, big data analytics, and blockchain.

On June 21, 2024, at HDC 2024, Huawei announced upgraded PanGu 5.0 alongside HarmonyOS NEXT. This version integrated with Harmony Intelligence, which features a smarter Celia (Xiaoyi) and focuses on generative AI updates to its LLM platform for creating new content, such as text, code, or images. Aiming to make PanGu accessible to a wide range of developers and businesses, it offered scalable options: smaller models requiring less computational power for those with limited resources, and larger models with increased capacities for complex tasks requiring more processing power.

Technical specifications

PanGu Large Model 3.0, designed for industry use, was structured with a 5+N+X three-tier architecture.

• First Layer (L0): Comprises PanGu's five basic large models to provide a variety of capabilities for different industry scenarios. These include Natural Language Processing (NLP) models, Visual models, Multimodal models, Prediction models, and Scientific Computing models.
• Second Layer (L1): Consists of N large industry-specific models. These models are trained using public data from various industries, such as government, finance, manufacturing, mining, and weather. Additionally, it uses customers' own data from L0 and L1 to train proprietary models tailored for each customer.
• Third Layer (L2): Provides customers with detailed scenario-specific models. This layer focuses on specific applications or business needs, offering ready-to-use model services.

The updated Huawei PanGu Model 5.0 by Huawei Cloud business division offered three key features: adaptability for different business scenarios, multi-style modeling, and advanced intelligence. Huawei divided the AI model platform into four series, each with different parameter scales:

• PanGu E Series: The Embedded version supports smart apps on phones, tablets, PCs, and other devices, with a parameter scale of 1 billion.
• PanGu P Series: The Professional version features a 10-billion parameter scale, ideal for low-latency and low-cost reasoning conditions.
• PanGu U Series: The Ultra version comes in two variants, with 135 billion and 230 billion parameters, capable of handling complex tasks and serving as a base for large models.
• PanGu S Series: The Super PanGu is the top-tier edition, featuring trillion-level parameters, designed to manage advanced AI technology scenarios such as cross-domain or multi-tasking applications.

See also

• Large Language Model
• Gemini
• GPT-4