Anyscale is a fully-managed Ray platform, on which you can build, deploy, and manage scalable AI and Python applications
This example goes over how to use LangChain to interact with Anyscale Endpoint.
%pip install -qU langchain-community
ANYSCALE_API_BASE = "..."
ANYSCALE_API_KEY = "..."
ANYSCALE_MODEL_NAME = "..."
import os
os.environ["ANYSCALE_API_BASE"] = ANYSCALE_API_BASE
os.environ["ANYSCALE_API_KEY"] = ANYSCALE_API_KEY
from langchain.chains import LLMChain
from langchain_community.llms import Anyscale
from langchain_core.prompts import PromptTemplate
template = """Question: {question}
Answer: Let's think step by step."""
prompt = PromptTemplate.from_template(template)
llm = Anyscale(model_name=ANYSCALE_MODEL_NAME)
question = "When was George Washington president?"
llm_chain.invoke({"question": question})
With Ray, we can distribute the queries without asynchronized implementation. This not only applies to Anyscale LLM model, but to any other Langchain LLM models which do not have _acall or _agenerate implemented
prompt_list = [
"When was George Washington president?",
"Explain to me the difference between nuclear fission and fusion.",
"Give me a list of 5 science fiction books I should read next.",
"Explain the difference between Spark and Ray.",
"Suggest some fun holiday ideas.",
"Tell a joke.",
"What is 2+2?",
"Explain what is machine learning like I am five years old.",
"Explain what is artifical intelligence.",
]
import ray
@ray.remote(num_cpus=0.1)
def send_query(llm, prompt):
resp = llm.invoke(prompt)
return resp
futures = [send_query.remote(llm, prompt) for prompt in prompt_list]
results = ray.get(futures)
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4