Core OME Concepts
This section of the documentation helps you learn about the components, APIs and abstractions that OME uses to represent model serving, model training, models, and dedicated AI clusters.
APIs Base ModelThe BaseModel CRD manages the lifecycle of foundational Hugging Face compatible and TensorRT LLM/AI models such as GPT, BERT, and other architectures， model type, model format, capabilities, model sizes and model configurations. These base models can be used for both training and serving. This resource has both namespace-scoped and cluster-scoped which can be used to define base models for different models.
Fine-Tuned WeightThe FineTunedWeight CRD manages the weights of models fine-tuned from a base model, allowing for task-specific optimization.
Serving RuntimeThe ServingRuntime CRD manages the runtime environment for model serving, allowing for dynamic scaling and configuration of model-serving containers. This resource has both namespace-scoped and cluster-scoped which can be used to define serving runtimes for different models.
Inference ServiceThe InferenceService CRD manages the entire lifecycle of model-serving workloads, including model versioning, scaling, and traffic routing. It supports real-time inference for both single-node and multi-node deployments, ensuring seamless model updates and efficient scaling.
IngressOME supports a range of ingress controllers for external access to model serving workloads. This section provides an overview of the available ingress controller options, including their capabilities, configuration, and features.
BenchmarkOME integrates with the latest GenAI-Bench to provide real-time benchmarking capabilities for AI models. This section provides an overview of the GenAI-Bench benchmarks, how they work, and how to use them with OME.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4