A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awss3receiver below:

awss3receiver package - github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awss3receiver - Go Packages

AWS S3 Receiver Overview

Receiver for retrieving trace previously stored in S3 by the AWS S3 Exporter.

Configuration

The following exporter configuration parameters are supported.

Name Description Default Required starttime The time at which to start retrieving data. Required if fetching by time endtime The time at which to stop retrieving data. Required if fetching by time s3downloader: region AWS region. "us-east-1" Optional s3_bucket S3 bucket Required s3_prefix prefix for the S3 key (root directory inside bucket). Required s3_partition time granularity of S3 key: hour or minute "minute" Optional file_prefix file prefix defined by user Optional endpoint overrides the endpoint used by the exporter instead of constructing it from region and s3_bucket Optional endpoint_partition_id partition id to use if endpoint is specified. "aws" Optional s3_force_path_style set this to true to force the request to use path-style addressing false Optional sqs: queue_url The URL of the SQS queue that receives S3 bucket notifications Required if fetching by SQS notification region AWS region of the SQS queue Required if fetching by SQS notification endpoint Custom endpoint for the SQS service Optional max_messages Maximum number of messages to retrieve in a single SQS request 10 Optional wait_time Wait time in seconds for long polling SQS requests 20 Optional encodings: An array of entries with the following properties: Optional extension Extension to use for decoding a key with a matching suffix. Required suffix Key suffix to match against. Required notifications: opampextension Name of the OpAMP Extension to use to send ingest progress notifications.

There are two modes of operation:

  1. Time Range Mode - Specify starttime and endtime to fetch data from a specific time range.
  2. SQS Message Mode - Subscribe to SQS messages to process new objects as they arrive.
SQS Message Configuration

The receiver can subscribe to an SQS queue that receives S3 event notifications:

sqs:
  # Required: The ARN of the SQS queue that receives S3 bucket notifications
  queue_url: "https:https://sqs.us-east-1.amazonaws.com/123456789012/test-queue" 
  # Required: The AWS region of the SQS queue
  region: "us-east-1"

Note: You must configure your S3 bucket to send event notifications to the SQS queue. Time-based configuration (starttime/endtime) and SQS configuration cannot be used together.

Time format for starttime and endtime

The starttime and endtime fields are used to specify the time range for which to retrieve data. The time format is either RFC3339,YYYY-MM-DD HH:MM or simply YYYY-MM-DD, in which case the time is assumed to be 00:00.

Encodings

By default, the receiver understands the following encodings:

The encodings options allows you to specify Encoding Extensions to use to decode keys with matching suffixes.

Example Configuration 
extension:
  # example of text encoding extension
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"
    
receivers:
  awss3:
    starttime: "2024-01-01 01:00"
    endtime: "2024-01-02"
    s3downloader:
        region: "us-west-1"
        s3_bucket: "mybucket"
        s3_prefix: "trace"
        s3_partition: "minute"
    encodings:
      - extension: text_encoding
        suffix: ".txt"

receivers:
  awss3/sqs_traces:
    s3downloader:
      region: us-east-1
      s3_bucket: mybucket
      s3_prefix: mytrace
    sqs:
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue"
      region: "us-east-1"

exporters:
  otlp:
    endpoint: otelcol:4317

service:
  pipelines:
    traces:
      receivers: [awss3/traces]
      exporters: [otlp]
      
    traces/sqs:
      receivers: [awss3/sqs_traces]
      exporters: [otlp]
Notifications

The receiver can send notifications of ingest progress to an OpAmp server using the custom message capability of "org.opentelemetry.collector.receiver.awss3" and message type "TimeBasedIngestStatus". The format of the notifications is a ProtoBuf formatted OLTP logs message with a single Log Record. The body of the record is set to status and the timestamp of the record is used to hold the ingest time. The record also has the following attributes:

Attribute Description telemetry_type The type of telemetry being ingested. One of "traces", "metrics", or "logs". ingest_status The status of the data ingestion. One of "ingesting", "failed", or "completed". start_time The time to start retrieving data as an Int64, nanoseconds since Unix epoch. end_time The time to stop retrieving data as an Int64, nanoseconds since Unix epoch. failure_message Error message if ingest_status is "failed".

The "ingesting" status is sent at the beginning of the ingest process before data has been retrieved for the specified time. If during the processing of the data an error occurs a status message with ingest_status set to "failed" status with the time of the data being ingested when the failure occurred. If the ingest process completes successfully a status message with ingest_status set to "completed" is sent.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4