OpenSearch

🔎 What is Amazon OpenSearch Service?

Amazon OpenSearch Service is a fully managed search and analytics engine based on open-source OpenSearch and Elasticsearch.

✅ It's used for real-time log analytics, search, monitoring, and observability over large datasets.

---

🧠 Core Concepts

Term	Description
Index	Like a database in SQL. Holds documents and mappings.
Document	A JSON object (record) to be indexed and queried.
Mapping	Defines structure and types of fields in an index.
Shard	A portion of an index. OpenSearch splits indexes into shards.
Replica	A copy of a shard used for HA and load balancing.
Node	A single EC2 instance in the OpenSearch cluster.
Cluster	A group of nodes working together to store and search data.
Domain	AWS-specific term for a managed OpenSearch cluster.

---

🧰 Use Cases

Use Case	Why OpenSearch?
📜 Log Analytics (e.g., VPC, ALB)	Ingest, index, and query logs in near real-time
🔍 Full-Text Search	For web apps, product search, enterprise search
📊 Dashboarding + Visualization	Kibana/OpenSearch Dashboards for insights
🛡️ Security Data Lake	SIEM-like threat detection and event analysis
🧪 Observability Stack	Pair with Prometheus, Grafana, FluentBit/Fluentd

---

🏗️ Architecture (Managed by AWS)

+--------------+     +----------------------+     +-----------------+
|   Log Source | --> |   Ingestion Pipeline | --> | OpenSearch Index|
+--------------+     +----------------------+     +-----------------+
                           |    ↑
                           v    |
                    OpenSearch Dashboards (GUI)

• Can use Kinesis, Firehose, Fluentd, Fluent Bit, or Logstash to ingest data.

• Query using OpenSearch SQL, DSL, or REST APIs.

• Visualize with OpenSearch Dashboards.

---

💡 Key Features

Feature	Description
🔎 Full-Text Search	Text relevance, keyword matching, stemming, etc.
📊 Aggregations	For metrics, analytics, faceting
📦 Index Templates	Define default settings/mappings for indices
🧠 Anomaly Detection	Built-in ML to detect log anomalies
📉 Index Lifecycle Policy	Auto delete, rollover, shrink, freeze cold data
🔐 Fine-Grained Access	Control access at index/field-level with Cognito/IAM/SAML
🔄 Snapshot & Restore	For backups to S3

---

🔐 Security Features

Layer	Options
AuthN	IAM, Cognito, SAML, Basic Auth
AuthZ	Fine-grained roles, index/field-level permissions
In-Transit Encryption	TLS (HTTPS)
At-Rest Encryption	AWS KMS or built-in encryption
Network Control	VPC, IP-based access policies

---

🛠️ Terraform Example: Basic OpenSearch Domain

resource "aws_opensearch_domain" "example" {
  domain_name           = "example-domain"
  engine_version        = "OpenSearch_2.11"

  cluster_config {
    instance_type = "t3.small.search"
    instance_count = 2
  }

  ebs_options {
    ebs_enabled = true
    volume_size = 10
  }

  node_to_node_encryption {
    enabled = true
  }

  encrypt_at_rest {
    enabled = true
  }

  domain_endpoint_options {
    enforce_https = true
  }

  access_policies = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Effect = "Allow",
      Principal = "*",
      Action = "es:*",
      Resource = "arn:aws:es:${var.region}:${var.account_id}:domain/example-domain/*"
    }]
  })
}

---

🧪 Indexing Example

Here’s a simple document and index using OpenSearch API:

1. Index a Document

curl -XPOST -u 'admin:admin' https://your-endpoint/_doc/1 -H 'Content-Type: application/json' -d '
{
  "user": "yuvaraj",
  "message": "Hello OpenSearch",
  "timestamp": "2025-06-15T12:00:00"
}'

2. Search the Document

curl -XGET -u 'admin:admin' "https://your-endpoint/_search?q=message:Hello"

---

📊 OpenSearch Dashboards

• Built-in GUI (like Kibana)

• Visualize:

  • Pie charts, bar graphs

  • Real-time dashboards

  • Anomaly detection

  • Trace analytics (Jaeger/Zipkin integration)

---

🔁 Integrations

Tool	Use With OpenSearch For...
Amazon Kinesis	Real-time log streaming
Amazon S3 + Firehose	Batched log delivery
Fluentd / Fluent Bit	Lightweight log forwarding from EC2 or ECS
CloudWatch Logs	Forward logs to OpenSearch for analysis
Lambda	Custom processors or filters

---

💰 Pricing Overview (2024)

Item	Cost Example
t3.small.search node	~$0.036/hour
Storage (EBS)	~$0.10/GB/month
Snapshots to S3	Free
Data Transfer	Standard AWS data transfer fees

🧠 Tip: Use index lifecycle policies and compression to reduce storage costs.

---

✅ TL;DR Summary

Feature	Amazon OpenSearch Service
Engine	OpenSearch / Elasticsearch (1.5–7.x)
Fully Managed	✅ Yes
Data Ingest	Kinesis, Firehose, Fluentd, etc.
Visualization	OpenSearch Dashboards (Kibana fork)
Query Language	DSL, SQL, Lucene
Security	IAM + Fine-grained + KMS + TLS
Ideal For	Search, log analytics, observability
Serverless Mode	❌ Not available yet (compute-based)

---