What's New?

Hugging Face welcomes Falcon 180B, the largest open-source language model ever with 180 billion parameters, developed by Technology Innovation Institute from UAE.

Trained on 3.5 trillion tokens from RefinedWeb dataset and using more about 7 million GPU hours (about 4096 GPUs), the model represents the longest single-epoch pretraining for an open model. 

Why Does It Matter?

Falcon 180B not only tops the leaderboard for open-access models but also rivals proprietary ones like PaLM-2 Large. Its performance metrics span a range of NLP tasks, placing it between GPT-3.5 and GPT-4 depending on the evaluation benchmark.

Some of Falcon 180B's main takeways are

• Advanced multiquery attention architecture, scaled up from Falcon 40B

• The dataset includes web data, technical papers, and a small fraction of code.

• Commercial use is possible but restricted, excluding "hosting use."