Blog / Devops

How to download the entire contents of a folder in an S3 bucket with Ansible

June 21, 2018

Just ran into this issue today. I needed to be able to download all the files inside a folder stored in an S3 bucket with Ansible.

The aws_s3 module docs on the Ansible site didn't provide an example on how to do this, but I did find a couple of articles that pointed me in the right direction.

- name: Get list of files from S3
aws_s3:
mode: list
aws_access_key: "{{ aws_access_key_id }}"
aws_secret_key: "{{ …

How to delete all queues in RabbitMQ

January 21, 2018

I recently had to do this for one of my older Django projects that uses Celery as I made the mistake of using the broker as the result backend.

Using "amqp" as the result backend is not recommended in production as this will create a queue for each task. I noticed I was getting a warning for the file descriptors in the RabbitMQ web management dashboard.

From the web management (available by installing the management …

Using SSH Agent Forwarding with Ansible

December 24, 2017

I've been using SSH agent forwarding with Ansible for the last few projects I've been working on and I thought I'd just share my setup here.

The neat thing with SSH agent forwarding is not having to store your SSH keys on your servers when pulling down your Git repo during deployment.

For example, I can deploy new code to my servers through Ansible from my local machine using my local SSH keys. The servers …

Generating a signed URL for an Amazon S3 file using boto

April 9, 2017

I was refactoring some code for EZ Exporter, a data exporter app for Shopify, last week as our customer base has been growing pretty steadily these last few months. I figured it's time to do some optimization to make sure the app is ready for future growth.

One of the functionalities that we needed to optimize is how we handle manual downloads. Initially, I just made it very simple by returning the data as …

Ansible Playbook for provisioning a Graylog central logging server

February 24, 2017

Graylog Message Stream

I first used Graylog a few years back while working for a tech startup in New York City. I was really impressed with it and it was probably the best log management software I've used at the time.

I've been meaning to set one up for Highview Apps for some time now, as we now have 5 servers for the different Django apps we've built and we will soon be adding another 3. Without centralized …

Older