pnispel

Storage Profiles

Summary

Storage profiles are stored at the company level and used to route a companies files to the correct storage location. There are three storage profiles right now: default govcloud and apse2 with the main regions being us-east-1, us-gov-west-1, and ap-southeast-2 respectively.

Moving a company from one storage profile to another

1. Switch the company storage profile in the rails_console: Company.find(<company_id>).update(storage_profile_key: <new_profile_name>)
2. Use the storage profile migrator to migrate old files to the new profile

pnispel

batch_size = 1000
start_id = LoginInformation.minimum(LoginInformation.primary_key) || 0
max_id = LoginInformation.maximum(LoginInformation.primary_key) || 0
end_id = [start_id + batch_size, max_id].min

while start_id <= max_id
  LoginInformation.where("login_informations.id BETWEEN #{start_id} and #{end_id} AND temp_id_uuid_mappings.id = login_informations.id").update_all("uuid = temp_id_uuid_mappings.uuid FROM temp_id_uuid_mappings")
  start_id += batch_size
  end_id = [start_id + batch_size, max_id].min
end

pnispel

Regional Storage

Regional storage invovles the storage, retrieval, and processing of files in an arbitrary AWS availability region.

Concepts

AWS regions

AWS has regions all over the world. The regions that we are interested in are those in us-east, us-west, us-gov-west, and ap-southeast.

Storage Services

Storage services are all the services that interact with our S3 Buckets. The gist linked above has an overview of the major services.

pnispel

File Storage

File storage encompasses all the services that interact with AWS S3.

Concepts

Storage Profiles

Storage profiles are a way for us to determine where to read and store files. For now, the profiles are:

• default which has purposes that point to buckets in us-east and us-west
• govcloud which has buckets in us-gov-west
• apse2 which has buckets in ap-southeast-2

pnispel

S3 Backup

The Problem

Storage backups are an import part of a fault-tolerant application. In Procore's current state, an S3 Bucket deletion would be catastrophic.

Goal

The goal is to have a storage backup solution that is robust, autonomous, scalable, and easily deployed in an emergency.

Options

1. Cloudfiles backup - This was our previous solution. At some point this solution stopped working, but its possible to bring it back, possibly through a lambda function.

pnispel

Backpressure Backfills

When backfilling millions of records, it becomes dangerous to try and update them all at once by hand. This technique tries to mitigate some of the dangers by creating a testible script that's sensitive to the current queue depth. It's most useful for things like data backfills, but the technique can be used for any type of sidekiq work that will require millions of jobs.

The goal of this technique is to allow running backfills at any time including peak hours without stressing the procore infrastructure.

Goals

• Maximize dispersion of work - we want to disperse the work over time in a way that correlates with the amount of strain currently put on Procore
• Minimize heavy database use - an obvious issue because if the database is getting hammered it can cause issues with locks and queuing
• Decrease variability - we want the results of the backfill to be as predictable as possible

pnispel

module Storage
  class CompanyStorageProfileMigrator
    attr_reader :company_id
    attr_reader :new_profile
    attr_reader :max_number_of_records_to_update
    attr_accessor :manifest

    def initialize(company_id:, new_profile:, max_number_of_records_to_update: Float::INFINITY)
      @company_id = company_id
      @new_profile = new_profile

pnispel

class Child extends Component {
  onChange() {
    this.props.onUpdate(this.inputEl.val)
  }
  
  render() {
    return (
      <div>
        <input onChange={this.onChange} value={this.props.someFoo} ref={(el) => this.inputEl = el} /> 
      </div>

pnispel

class Filter {
  render() {
    const { onChange, options } = this.props
    
    return (
      <select onChange={onChange}>
        {options.map((option) => <option value={option.value}>{option.label}</option>)}
      </select>
    )
  }

pnispel

module Storage
  module Migrations
    # Implements the action of moving a document from its current location to
    # the location its internal data suggests.
    class Move
      attr_reader :document
      attr_reader :from
      attr_reader :to

      def initialize(document, to_usd)