How to Persist data from a Docker Container Running on an AWS EC2 instance

by | Nov 9, 2020 | AWS, BlogPosts, Tech Tips | 0 comments

Sometimes when running an application; you want to persist data. A common use-case for this is as a redundancy measure so that if the application goes down, no data currently being processed by the application is lost. 

In this example I will persist data from a container running on an EC2 onto an EBS volume. I will make the following assumptions of the reader:  

  • You want to automatically configure and start the application using a user-data script or equivalent; and therefore you will run the commands as the root user. 
  • You are using an EC2 AMI which has docker pre-installed; if you are not, you can find instructions to install docker on an EC2 here.
  • You have an EBS volume attached to your EC2 instance as /dev/sda1. 

I will use the official Logstash container available here. 

Note: I will not prefix any of the commands here with “sudo”; this is because commands run in user-data scripts are run as root and therefore “sudo” is not required. 

  1. Let’s use bash for our user-data script.
#!/bin/bash
source ~/.bash_profile
  1. Then create a file system on the external drive.
mkfs --type ext4 /dev/sda1
  1. Mount the external drive onto a mount-point in your Linux directory tree. This specific mount point will later be mounted onto the docker-container.
mount /dev/sda1 /Logstash 
  1. This is a crucial and easily missed step. Change the owner of the directory that you want to mount onto the docker-container to a user other than the root user; on EC2’s the ec2-user is convenient.
chown ec2-user /apps/Logstash 
  1. Enable and start Docker.
# Enable docker.
systemctl enable docker
systemctl start docker 
  1. Run the docker image and mount the mount-point of the external drive onto the directory within the container, that contains the data to be persisted.
docker run –-mount type=bind,source=/apps/logstash/data,target=/usr/share/logstash docker.elastic.co/logstash/logstash:7.9.2 
If a mounted drive is owned by root; the application running on the docker container will be unable to write data to the mounted drive which can cause the following:
  • Data is not persisted on external drive.
  • Application failure to start successfully.
When the Logstash application is started, it stores any persisted data in the /apps/logstash/data directory on the container. Storing this information on an external drive, such as an EBS volume, provides an added layer of redundancy. If the EC2 or container goes down or is restarted, then the data which was persisted by the Logstash application will still available then the Logstash application is restarted via the mounted EBS volume. Good luck with your docker journey.

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *