How to Upload Data from a Local Rails Application to the Cloud using S3 Compatible Object Storage

Background

In a recent client consultation, we were tasked with migrating a Firebase application to Rails, including transferring both the database and associated images to the cloud. After reviewing several resources, I found that guidance on handling image migrations was sparse, so I decided to write this article to help other developers facing similar challenges.

Note: This guide will not cover the Firebase-to-Rails migration process, as it’s simply a matter of using a Rake task to export data from Firebase as JSON and then importing it into the Rails database.

Let’s Get Started

In this guide, we will deploy the Rails app on a simple virtual machine using Kamal, and walk through the process of migrating both the database and image files to a cloud environment. The steps include using SCP for the database transfer, Docker for the database import, and Magalu’s mgc command-line tool for image uploads. The same methods can likely be adapted for other cloud providers like S3 or DigitalOcean Spaces.

Steps Overview

• Database Migration (PostgreSQL)
• Uploading Image Files to Object Storage (Magalu or S3-Compatible Service)
• Data Processing and Cleanup

Database Migration

To migrate your production database to the cloud, follow these steps:

1. Transfer the Database Using SCP

   First, copy your production database file from your local machine to the cloud server using scp:

   scp /path/to/your/project_db.sql user@your-server:/destination-path/
   

2. Import the Database

   After creating an empty database in the cloud, import the database file using Docker:

   docker exec -i project-db psql -U postgres -d project_production < ./path_for/project_db.sql
   

Uploading Image Files to Object Storage

For this step, we’ll use Magalu’s Object Storage, which is compatible with S3. If you’re using S3 or DigitalOcean Spaces, you can follow the same steps.

One of the main challenges here is that Rails’ ActiveStorage treats file storage differently between the local Disk service and the cloud-based S3 service. ActiveStorage stores images in different folder structures for disk and s3 storage service:

• Disk Service: Files are stored in a folder structure like storage/folder/key (where the folder is derived from the first four digits of the file’s key).
• S3 Service: Files are stored as key without the folder structure.

Here’s the relevant code snippet from Rails’ DiskService:

# https://github.com/rails/rails/blob/dd8f7185faeca6ee968a6e9367f6d8601a83b8db/activestorage/lib/active_storage/service/disk_service.rb#L101
class ActiveStorage::Service::DiskService < ActiveStorage::Service
  def path_for(key)
    File.join root, folder_for(key), key
  end
end

To ensure smooth migration, we need to flatten the folder structure and move all images into a single directory for upload to S3 or Magalu.

1. Prepare Image Files for Upload

From your Rails storage folder, locate all stored images and copy them to a temporary directory:

This command is basically removing the folder structure and copying all files to a new folder.

find storage -type f -exec bash -c 'cp "$1" "tempstorage/$(basename "$1")"'  {} \;

2. Upload Images to Cloud Storage Using Magalu CLI

Once your images are in the tempstorage folder, you can use Magalu’s mgc command-line tool to upload the files to your cloud storage bucket:

./mgc object-storage objects upload-dir --dst="project-bucket" --src="/home/pedro/IdeaProjects/project/api/tempstorage"

3. Update Rails ActiveStorage to Use Cloud Storage

After uploading the images, you need to tell Rails to use the cloud storage service (Magalu, S3, or another compatible service) for all files. In the Rails console, run the following command:

# In this example, I set the service name as 'magalu_cloud'
ActiveStorage::Blob.update_all(service_name: :magalu_cloud)

Data Processing and Cleanup

After migrating the images and database, you may need to process or clean up some of your data. For example, you may need to correct naming inconsistencies or perform other data adjustments.

1. Run a Background Job for Data Cleanup

To perform bulk updates or corrections, use a Rake task. Here’s an example of how to run a task that updates some records in the database:

./bin/rake task:update_something_task

Conclusion

By following these steps, you can successfully migrate your Rails application’s data to the cloud. If you’re using S3 or DigitalOcean Spaces instead of Magalu, simply adjust the relevant commands for your chosen provider. Happy migrating!