Docker Compose Configuration for ElasticSearch v8.15.3 on Ubuntu 24.04.1 with Static IP and Custom Settings

docker-compose.yml

version: '3.7'

services:
  elasticsearch:
    image: ${ES_IMAGE}  # The Elasticsearch Docker image, provided via the environment variable.
    container_name: ${ES_CONTAINER_NAME}  # The name of the Elasticsearch container, provided via the environment variable.
    environment:
      - discovery.type=${ES_DISCOVERY_TYPE}  # The discovery type for Elasticsearch cluster. Options include "single-node" for a single-node setup or "zen-disco" for multi-node cluster discovery.
      - bootstrap.memory_lock=${ES_MEMORY_LOCK}  # Memory locking setting. This ensures Elasticsearch has full control over memory usage.
      - xpack.security.enabled=${ES_SECURITY_ENABLED}  # Whether to enable the X-Pack security module, which enables security features.
    ports:
      - "${ES_HTTP_PORT}:${ES_HTTP_PORT}"  # Exposes the Elasticsearch HTTP port to the outside. The port is provided via the ${ES_HTTP_PORT} environment variable.
      - "${ES_TRANSPORT_PORT}:${ES_TRANSPORT_PORT}"  # Exposes the Elasticsearch transport (internal communication) port. The port is provided via the ${ES_TRANSPORT_PORT} environment variable.
    ulimits:
      memlock:
        soft: -1  # Soft memory lock limit. -1 means no limit on memory locking.
        hard: -1  # Hard memory lock limit. -1 means no limit on memory locking.
    volumes:
      - ${ES_DATA_VOLUME_PATH}:/usr/share/elasticsearch/data  # The path to store Elasticsearch data on the host machine. The path is provided via the ${ES_DATA_VOLUME_PATH} environment variable.
    deploy:
      resources:
        limits:
          memory: ${ES_MEMORY_LIMIT}  # The maximum amount of memory the Elasticsearch container can use. The limit is provided via the ${ES_MEMORY_LIMIT} environment variable.
          cpus: '${ES_CPU_LIMIT}'  # The CPU limit for the Elasticsearch container. The limit is provided via the ${ES_CPU_LIMIT} environment variable.
        reservations:
          memory: ${ES_MEMORY_RESERVATION}  # The reserved memory for the Elasticsearch container. The value is provided via the ${ES_MEMORY_RESERVATION} environment variable.
    networks:
      estack_network:
        ipv4_address: 172.20.0.2  # Assigns a static IP address from the desired range.

networks:
  estack_network:
    driver: bridge  # Use the bridge network driver for the custom network.
    ipam:
      driver: default
      config:
        - subnet: "172.20.0.0/16"  # Set the subnet to 172.20.0.0/16 for your custom network.

Dockerized-ElasticSearch-8-Ubuntu-24.04.1.md

Dockerized Elasticsearch 8 on Ubuntu 24.04.1

This document provides a step-by-step guide to setting up Elasticsearch 8 with Docker on an Ubuntu 24.04.1 system. It also explains the configuration of Elasticsearch, including memory limits, security settings, and the use of environment variables from a .env file for dynamic configuration.

Table of Contents

• System Requirements
• Project Structure
• Step 1: Create the .env File
• Step 2: Docker Compose Configuration
• Step 3: Running Elasticsearch with Docker Compose
  • Run Elasticsearch in Detached Mode
  • Check the Status of the Elasticsearch Container
  • Access Elasticsearch
• Step 4: Shutting Down Elasticsearch
• Troubleshooting
  • Elasticsearch Not Starting
  • Causes of Elasticsearch Startup Issues
  • Solutions for Elasticsearch Startup Issues
  • Docker Memory Allocation
• Conclusion

---

System Requirements

Before starting, ensure the following:

• Ubuntu 24.04.1 system is ready.

• Docker and Docker Compose are installed. If not, you can install them using the instructions below:

  sudo apt update
  sudo apt install docker.io docker-compose

• Docker Daemon is running.

Docker Installation:

sudo systemctl start docker
sudo systemctl enable docker

• Docker Compose Installation:

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Verify the installation:

docker --version
docker-compose --version

---

Project Structure

Below is the recommended project structure for the Elasticsearch Docker setup:

elasticsearch-docker/
│
├── docker-compose.yml
├── .env
└── README.md

---

Step 1: Create the .env File

To manage configuration variables dynamically, we will use a .env file to store environment variables. These variables will be referenced in the docker-compose.yml file.

Create a .env file with the following content:

# .env file

# Elasticsearch Configuration
ES_IMAGE=docker.elastic.co/elasticsearch/elasticsearch:8.15.3
ES_CONTAINER_NAME=es01
ES_DISCOVERY_TYPE=single-node
ES_MEMORY_LOCK=true
ES_SECURITY_ENABLED=false
ES_HTTP_PORT=9200
ES_TRANSPORT_PORT=9300
ES_DATA_VOLUME_PATH=/home/ubuntu/Elastic31012025
ES_MEMORY_LIMIT=8G
ES_CPU_LIMIT=1
ES_MEMORY_RESERVATION=4G

• ES_IMAGE: The Docker image for Elasticsearch.
• ES_CONTAINER_NAME: The name of the Elasticsearch container.
• ES_DISCOVERY_TYPE: The type of node discovery; set to single-node for single node Elasticsearch.
• ES_MEMORY_LOCK: Set to true to lock memory.
• ES_SECURITY_ENABLED: Disable X-Pack security.
• ES_HTTP_PORT and ES_TRANSPORT_PORT: Set the HTTP and transport ports.
• ES_DATA_VOLUME_PATH: Path on the host system for Elasticsearch data storage.
• ES_MEMORY_LIMIT, ES_CPU_LIMIT, ES_MEMORY_RESERVATION: Define the resource limits and reservations for Elasticsearch.

---

Step 2: Docker Compose Configuration

The docker-compose.yml file will use the variables defined in the .env file. Below is the configuration for Elasticsearch 8 with Docker Compose:

version: '3.7'

services:
  elasticsearch:
    image: ${ES_IMAGE}  # Pulls from .env file
    container_name: ${ES_CONTAINER_NAME}  # Container name from .env
    environment:
      - discovery.type=${ES_DISCOVERY_TYPE}  # Single node setup from .env
      - bootstrap.memory_lock=${ES_MEMORY_LOCK}  # Memory lock from .env
      - xpack.security.enabled=${ES_SECURITY_ENABLED}  # Security settings from .env
    ports:
      - "${ES_HTTP_PORT}:${ES_HTTP_PORT}"  # HTTP port from .env
      - "${ES_TRANSPORT_PORT}:${ES_TRANSPORT_PORT}"  # Transport port from .env
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ${ES_DATA_VOLUME_PATH}:/usr/share/elasticsearch/data  # Data volume from .env
    deploy:
      resources:
        limits:
          memory: ${ES_MEMORY_LIMIT}  # Memory limit from .env
          cpus: '${ES_CPU_LIMIT}'  # CPU limit from .env
        reservations:
          memory: ${ES_MEMORY_RESERVATION}  # Memory reservation from .env

volumes:
  esdata:
    driver: local

Explanation of Configuration:

• Image: The Elasticsearch Docker image is pulled from the Docker registry, defined in the .env file (docker.elastic.co/elasticsearch/elasticsearch:8.15.3).
• Container Name: The container is named es01, which is dynamically set from the .env file.
• Environment Variables: The configuration of Elasticsearch (discovery type, memory lock, security) is controlled via environment variables from the .env file.
• Ports: The ports 9200 and 9300 are exposed for HTTP and transport communication, respectively.
• Memory and CPU Limits: The deploy section sets memory and CPU limits for the Elasticsearch container. These values come from the .env file.
• Data Volume: The path /home/ubuntu/Elastic31012025 is mounted as a volume to persist Elasticsearch data. This allows data to persist even after the container is stopped or removed.
• Memory Lock: The memlock ulimit is set to ensure that Elasticsearch locks memory, preventing the operating system from swapping memory to disk.

---

Step 3: Running Elasticsearch with Docker Compose

With the .env and docker-compose.yml files configured, you can now launch the Elasticsearch container with Docker Compose.

Run Elasticsearch in Detached Mode

Navigate to your project directory (where docker-compose.yml is located) and run:

sudo docker-compose up -d

This command will pull the necessary Docker image (if not already pulled), create the container, and start Elasticsearch in the background.

Check the Status of the Elasticsearch Container

To verify that the container is running, use the following command:

sudo docker-compose ps

You should see the es01 container running, and it will be accessible via the ports defined in the .env file (9200 for HTTP and 9300 for transport).

Access Elasticsearch

Once the container is running, you can access Elasticsearch by navigating to:

http://localhost:9200

You should see a JSON response with details about the Elasticsearch node.

---

Step 4: Shutting Down Elasticsearch

To stop the Elasticsearch container, use the following command:

sudo docker-compose down

This will stop and remove the container but will retain the volume where the data is stored.

---

Troubleshooting

Elasticsearch Not Starting

If Elasticsearch is not starting, check the logs for any issues:

sudo docker-compose logs elasticsearch

This will display the logs for the elasticsearch service, helping you identify any configuration issues or errors.

Causes of Elasticsearch Startup Issues

If you're encountering an error related to Elasticsearch not starting, here are some common causes:

1. Permissions Issue: Elasticsearch might not be able to access or write to the /usr/share/elasticsearch/data directory. This could be due to incorrect directory permissions.

2. Multiple Nodes Using the Same Data Path: If multiple nodes are trying to use the same data path, it can cause a conflict.

3. Existing Lock File: If the node.lock file already exists and Elasticsearch cannot obtain a lock on it, this may prevent it from starting.

Solutions for Elasticsearch Startup Issues

To resolve the above issues, follow these solutions:

1. Check Permissions: Ensure Elasticsearch has the proper permissions to access the data directory. Run the following commands to fix permissions:

   sudo chown -R 1000:1000 /home/ubuntu/elasticsearch
   sudo chmod -R 755 /home/ubuntu/elasticsearch

   The 1000:1000 user and group are the default UID/GID for Elasticsearch when running in Docker.

2. Remove Existing Lock Files: If a node.lock file is preventing Elasticsearch from starting, delete it:

   sudo rm /home/ubuntu/elasticsearch/node.lock

3. Ensure Single Node Configuration: Confirm that the discovery.type is set to single-node in your .env file, and check that no other Elasticsearch instances are running that may be using the same

directory.

4. Check Docker Volumes: If you're using Docker volumes, ensure they are not already in use by another container. List existing volumes:

   docker volume ls

   If necessary, remove the existing volume:

   docker volume rm <volume_name>

5. Elasticsearch Logs: Review the Elasticsearch logs located at /usr/share/elasticsearch/logs/docker-cluster.log for more details. Ensure the directory exists and is writable.

Docker Memory Allocation

If Elasticsearch is having memory issues, increase the memory allocation for Docker on your system. You can adjust the memory allocation in the Docker settings or docker-compose.yml.

---

Conclusion

This guide has walked you through the process of setting up Elasticsearch 8 with Docker on Ubuntu 24.04.1. By following these steps, you should now have a fully functional Elasticsearch instance running in a Docker container.

env

# ElasticSearch Environment Variables
ES_IMAGE=docker.elastic.co/elasticsearch/elasticsearch:8.15.3
ES_CONTAINER_NAME=es01
ES_DISCOVERY_TYPE=single-node
ES_MEMORY_LOCK=true
ES_SECURITY_ENABLED=false
ES_HTTP_PORT=9200
ES_TRANSPORT_PORT=9300
ES_DATA_VOLUME_PATH=/home/ubuntu/elasticsearch/data
ES_MEMORY_LIMIT=8G
ES_CPU_LIMIT=1
ES_MEMORY_RESERVATION=4G

🚀 Elasticsearch Docker Setup

Great job setting up Elasticsearch 8.15.3 with Docker! Here’s a quick breakdown of what’s happening in your setup:

1. Docker & Docker Compose Version Check
   ✅ You're running Docker 28.0.2 and Docker Compose 2.34.0, both are up-to-date. No compatibility issues here.

2. Elasticsearch Container Deployment
   🔹 docker-compose up -d --build correctly pulls the necessary images and starts the container in detached mode.
   🔹 The network (estack_estack_network) and container (es01) are successfully created & started.

3. Container Running Status
   📌 docker ps confirms that Elasticsearch is running, exposing:

   • 9200/tcp → REST API (used for search queries & cluster management)
   • 9300/tcp → Cluster communication

4. Next Steps & Verification
   ✅ Ensure that Elasticsearch is fully operational by running:

   curl -X GET "http://localhost:9200"

   or check logs for any startup issues:

   docker logs es01

🔍 Potential Issues & Fixes:

• Elasticsearch Not Starting? Check if another node is trying to use the same data directory.
• Permissions Issue? Ensure that your data directory is writable:

  sudo chown -R 1000:1000 /home/ubuntu/elasticsearch
  sudo chmod -R 755 /home/ubuntu/elasticsearch

• Stale Lock File? If Elasticsearch fails due to a lock, remove the node.lock file:

  sudo rm /home/ubuntu/elasticsearch/node.lock

Overall, everything looks good! 🚀 If you hit any roadblocks, check the logs and verify the volume bindings.
Happy coding! 🛠️

🚀 Elasticsearch Cluster Health Check

Great job setting up Elasticsearch and verifying the cluster status via localhost:9200. Based on the visualized JSON response, here’s my breakdown:

✅ Key Observations

1. Elasticsearch is Running

   • The response confirms that the cluster is up and responding.
   • The cluster UUID, name, and tagline fields are correctly retrieved.

2. Version & Build Details

   • The response provides comprehensive metadata, including:
     • Version Number → Ensures compatibility with clients/plugins.
     • Build Flavor & Type → Confirms whether this is an official build or custom.
     • Lucene Version → Important for index compatibility.
     • Build Hash & Date → Useful for debugging and comparing with release versions.

3. Index & Wire Compatibility

   • Shows the minimum supported index and wire compatibility versions, which is crucial when upgrading.

🔍 Recommended Next Steps

• Check Cluster Health: Run

  curl -X GET "http://localhost:9200/_cluster/health?pretty"

  This ensures that the cluster is in a green or yellow state, meaning all nodes and indices are functioning correctly.

• Verify Indexes: Run

  curl -X GET "http://localhost:9200/_cat/indices?v"

  This helps confirm if your indices are properly created and active.

• Security Considerations:

  • If running in production, disable open access on localhost:9200 to prevent unauthorized access.
  • Use TLS and authentication mechanisms if exposing Elasticsearch externally.

🚀 Overall, this confirms a successful Elasticsearch deployment. Just verify the cluster health and indices, and you're good to go!
Happy indexing! 🛠️