Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://huggingface.co/tiiuae/falcon-rw-1b below:

Website Navigation

• 😻

  Sharathhebbar24/One-stop-for-Open-source-models

• 🕵️

  tararad/Liketropy-LLM-Detector

• 🚀

  PhoenixDecim/slm_financial_rag

• 🏃

  saifa8237/Beauty_Reels

• 📚

  tsissam/AI_Powered_University_of_CapeTown_Course_Advisor_Chatbot

• 🏢

  swapnasg/ChatFriend

• 📊

  Occvltvs/whisperbox-ai

• 🐠

  angel1987/LiteraLingo_Dev

• 🦀

  Deepakraj2006/RAG_GRAD

• 📖

  UmaGeeth/Story_Poem

• 🌍

  sabry011/code-ai-chat

• 👁

  isnel55/DevMasterGPTplus

• TOS
• Privacy
• About
• Jobs
• Models
• Datasets
• Spaces
• Pricing
• Docs

tiiuae/falcon-rw-1b · Hugging Face

Falcon-RW-1B is a 1B parameters causal decoder-only model built by TII and trained on 350B tokens of RefinedWeb. It is made available under the Apache 2.0 license.

See the 📓 paper on arXiv for more details.

RefinedWeb is a high-quality web dataset built by leveraging stringent filtering and large-scale deduplication. Falcon-RW-1B, trained on RefinedWeb only, matches or outperforms comparable models trained on curated data.

⚠️ Falcon is now available as a core model in the transformers library! To use the in-library version, please install the latest version of transformers with pip install git+https://github.com/huggingface/transformers.git, then simply remove the trust_remote_code=True argument from from_pretrained().

⚠️ This model is intended for use as a research artifact, to study the influence of training on web data alone. If you are interested in state-of-the-art models, we recommend using Falcon-7B/40B, both trained on >1,000 billion tokens.

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-rw-1b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

💥 Falcon LLMs require PyTorch 2.0 for use with transformers!

• Developed by: https://www.tii.ae;
• Model type: Causal decoder-only;
• Language(s) (NLP): English;
• License: Apache 2.0.

• Paper: https://arxiv.org/abs/2306.01116.

Research on large language models, specifically the influence of adequately filtered and deduplicated web data on the properties of large language models (fairness, safety, limitations, capabilities, etc.).

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful.

Broadly speaking, we would recommend Falcon-7B/40B for any use not directly related to research on web data pipelines.

Falcon-RW-1B is trained on English data only, and will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.

We recommend users of Falcon-RW-1B to consider finetuning it for the specific set of tasks of interest, and for guardrails and appropriate precautions to be taken for any production use.

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-rw-1b"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

Falcon-RW-1B was trained on 350B tokens of RefinedWeb, a high-quality filtered and deduplicated web dataset. The data was tokenized with the GPT-2 tokenizer.

Falcon-RW-1B was trained on 32 A100 40GB GPUs, using only data parallelism with ZeRO.

Hyperparameters were adapted from the GPT-3 paper (Brown et al., 2020).

Training happened in early December 2022 and took about six days.

See the 📓 paper on arXiv for in-depth evaluation.

Falcon-RW-1B is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).

The architecture is adapted from the GPT-3 paper (Brown et al., 2020), but uses ALiBi (Ofir et al., 2021) and FlashAttention (Dao et al., 2022).

Falcon-RW-1B was trained on AWS SageMaker, on 32 A100 40GB GPUs in P4d instances.

Falcon-RW-1B was trained a custom distributed training codebase, Gigatron. It uses a 3D parallelism approach combined with ZeRO and high-performance Triton kernels (FlashAttention, etc.)

@article{refinedweb,
  title={The {R}efined{W}eb dataset for {F}alcon {LLM}: outperforming curated corpora with web data, and web data only},
  author={Guilherme Penedo and Quentin Malartic and Daniel Hesslow and Ruxandra Cojocaru and Alessandro Cappelli and Hamza Alobeidli and Baptiste Pannier and Ebtesam Almazrouei and Julien Launay},
  journal={arXiv preprint arXiv:2306.01116},
  eprint={2306.01116},
  eprinttype = {arXiv},
  url={https://arxiv.org/abs/2306.01116},
  year={2023}
}

falconllm@tii.ae

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4