Ansible playbook to deploy containers within VPC - 29/09/2018 06:23 EDT

Each of the deployed containers must have a auto-scaling group; all ports should be open within the VPC.

The stack being deployed is the ELK Stack. Elasticsearch should be able to get data from the Logstash container. Logstash should be able to connect to the internet to get access to data that may be stored in different locations. Kibana will be published to the internet and be available for access from anywhere.

Ansible requirements: Deployment for Elasticsearch, Logstash, and Kibana.

We need ansible playbooks to create a cluster of Elasticsearch, Logstash, and Kibana – each having its own auto-scaling cluster and load-balancers behind the same VPC. The images should be taken from our AWS ECS Repository. We should be able to edit the path to the ECR repositories through ansible and only images with the tag “latest” should be spun in the cluster. If a new image is pushed to ECR, then ansible should spin the new instance and destroy the earlier outdated image.

There are certain requirements for Elasticsearch such as mmap count (needs to be 262144) must be editable through ansible. SSD storage for each container must be customisable through Ansible.

All the clusters should be behind the same VPC with all ports and communications open between the servers within the cluster. Logstash should be able to communicate to the internet and must have a permanent Elastic IP. Kibana should be publicly accessible through an Elastic IP.

We need clear instructions for deployment of ansible (prefer deployment on S3 but we are open to a better cost-efficient method.) Need instructions on adding a new docker container within the same VPC (in the future, we also want to implement the Machine Learning stack of ELK – this will require a new container within the VPC). All AWS credentials should be editable from a single file.

Analytics and monitoring:
a. We would prefer Prometheus to monitor all the instances within the cluster. This should also be deployed through ansible
b. Simple dashboards such as CPU Utilization, number of servers/nodes, RAM usage stats should be displayed in Grafana (also deployed through ansible)
c. We want email alerts that a specific deployment has been undertaken successfully.

Cost minimization:
a. We want ansible to monitor whether specified clusters can be run on spot-instances. If the spot-request is not fulfilled within a few minutes (say 3 minutes) --- then an on-demand instance to be spun.
b. We should be able to schedule the instances such that we can shutdown instances during specific hours of the day or even during specific days
c. The type and number of instances for each image will be different. These will also change from time-to-time. Currently, we only need one instance running for each container but going forward, we will be scaling Elasticsearch and Logstash to two instances.
d. Readme to contain instructions on adding another image to the existing cluster

Please see other ELK-specific requirements here:
https://medium.com/@devfire/deploying-the-elk-stack-on-amazon-ecs-part-2-34c841e3b774

Amazon Web Services Docker Elasticsearch Szoftverarchitektúra

Projektazonosító: #17863359

A projektről