lakeFS provides scalable version control over the data lake, and uses Git-like semantics to create and access those versions.
This notebooks covers how to load document objects from a lakeFS path (whether it's an object or a prefix).
Replace ENDPOINT, LAKEFS_ACCESS_KEY, and LAKEFS_SECRET_KEY values with your own.
from langchain_community.document_loaders import LakeFSLoader
ENDPOINT = ""
LAKEFS_ACCESS_KEY = ""
LAKEFS_SECRET_KEY = ""
lakefs_loader = LakeFSLoader(
lakefs_access_key=LAKEFS_ACCESS_KEY,
lakefs_secret_key=LAKEFS_SECRET_KEY,
lakefs_endpoint=ENDPOINT,
)
You can specify a prefix or a complete object path to control which files to load.
Specify the repository, reference (branch, commit id, or tag), and path in the corresponding REPO, REF, and PATH to load the documents from:
REPO = ""
REF = ""
PATH = ""
lakefs_loader.set_repo(REPO)
lakefs_loader.set_ref(REF)
lakefs_loader.set_path(PATH)
docs = lakefs_loader.load()
docs
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4