Production Engineering

Overview

The goal of this document is to outline strict processes those that have access to PRODUCTION systems MUST follow.

It is intended to mitigate most classes of known threats while still allowing for high productivity via compartmentalization.

Production access where one has access to the personal data or assets of others is to be taken very seriously, and assigned only to those who have tasks that can not be performed without it.

These are the rules we wish to meet and model at #! and futher recommend for those who are managing targeted production systems.

Assumptions

All of our screens are visible to an adversary
All of our keyboards are logging to an adversary
Any firmware/bootloaders not verified on every boot are compromised
Any host OS with network access is compromised
Any guest OS used for any purpose other than prod access is compromised
At least one member of the PRODUCTION team is always compromised
At least one maintainer of third party code we depend on is compromised

Requirements

A PRODUCTION ENGINEER SHOULD have the following:
- A clear set of tasks that can’t be completed without access
- Experience in Red/Blue team, CTF, or documented CVE discoveries
- A demonstrated working knowledge of:
  - Low level unix understanding. E.g. /proc, filesystems, kernel modules
  - Low level debugging techniques. E.g. strace, inotify, /proc, LD_PRELOAD
  - Linux kernel security features. E.g. seccomp, pam, cgroups, permissions
  - Common attack classes. E.g: XSS, Buffer Overflows, Social Engineering.
- A majority passed interview panel with current PRODUCTION access team
- An extensive background check clean of any evidence of dishonesty
- Training on secret coercion and duress protocols shared with team
A PRODUCTION ENGINEER MUST NOT expose a CRITICAL SECRET to:
- a screen
- a keyboard
- an unsupervised peer
- an internet connected OS other than destination that requires it
A recommended ENTROPY SOURCE MUST be used to generate a CRITICAL SECRET
An OS that accesses PRIVILEGED SYSTEMS MUST NOT be used for anything else
Any OS without verified boot from firmware up MUST NOT be trusted
Manual PRIVILEGED SYSTEM mutations MUST be approved, witnessed, and recorded
PRIVILEGED SYSTEM mutatations MUST be automated and repeatable via code
Any code without SECURITY REVIEW MUST NOT be trusted
Any code we can not review ourselves as desired MUST NOT be trusted
PRIVILEGED SYSTEM access requires physical interaction with an approved HSM

Implementation

Tools

HARDENED WORKSTATION
PERSONAL HSM
ENTROPY SOURCE

Setup

Install QubesOS

Enable full disk encryption
Set up Verified Boot
- Choose “factory reset” after QubesOS install in boot options
- Sign with PERSONAL HSM
- Verify every boot by inserting PERSONAL HSM and observing green LED
Set up FDE password in hardware device via PERSONAL HSM
Set up Challenge Response authentication with PERSONAL HSM
- PERSONAL HSM + password to unlock screen
- Automatically lock screen when PERSONAL HSM removed

Configure Qubes

Vault

MUST NOT have internet access
Example use cases:
- Personal GPG keychain management
- Bulk encrypting/decrypting documents
- Provisioning OTP devices

Production

MUST have internet access limited to only bastion servers
MUST manage all credentials via individually approved hardware decryption
- password-store via PERSONAL HSM or mooltipass
SHOULD use whonix as the network gateway
Used only to reach PRODUCTION bastion servers and systems beyond them

Work

MUST have internet access limited to organization needs and partners
MUST manage all credentials via individually approved hardware decryption
- password-store via PERSONAL HSM or mooltipass
SHOULD use whonix as the network gateway
Example use cases:
- Read only access to AWS panel
- Observe Kibana logs
- Check organization email and chat

Personal

No internet access limits
SHOULD only be used for personal services
SHOULD use whonix as the network gateway
Example use cases:
- Check personal email / chat
- Personal finance

Development

No internet access limits
SHOULD only be used for development
SHOULD use whonix as the network gateway
SHOULD manage all credentials via individually approved hardware decryption
- password-store via PERSONAL HSM
Example use cases:
- Read online documentation
- Authoring code
- Submitting PRs
- Doing code review

Disposable

SHOULD only be used for development
SHOULD use whonix as the network gateway
Example use cases:
- Pentesting
- Testing untrusted code or applications
- Competitive research
- Explore dark web portals for data dumps

Setup Keychain

Follow “Advanced” GPG setup guide in Vault Qube
- Daily driver PERSONAL HSM holds subkeys
- Separate locked-away PERSONAL HSM holds master key
- Separate locked-away flash drive holds pubkeys and encrypted private keys

Workflow

The following describes the workflow taken to get signed code/artifacts that have become stablized in the DEVELOPMENT environment and are ready to enter the pipeline towards production.

Local

PRODUCTION ENGINEER or Software Engineer
- Authors/tests changes to INFRASTRUCTURE REPOSOTORY in LOCAL environment
- makes feature branch
- makes one or more commits to feature branch
- signs all commits with PERSONAL HSM
- Optional: Squashes and re-signs sets of commits as desired
- Submits code to peer for review
PRODUCTION ENGINEER
- Verifies changes work as intended in “local” environment
- Verifies changes have solid health checks and recovery practices
- Merges reviewed branch into master branch with signed merge commit

Staging

PRODUCTION ENGINEER #1
- Copies desired changes from LOCAL templates to STAGING templates
- makes feature branch
- makes one or more commits to feature branch
- signs all commits with PERSONAL HSM
- Optional: Squashes and re-signs sets of commits as desired
- Submits code to peer for review
PRODUCTION ENGINEER #2
- Optional: significant changes/migrations are tested in “sandbox” env.
- Merges reviewed branch into master branch with signed merge commit
PRODUCTION ENGINEER #1
- Logs into STAGING toolbox
- Deploys changes from STAGING “toolbox”

Production

PRODUCTION ENGINEER #1
- Copies desired changes from STAGING templates to PRODUCTION templates
- makes feature branch
- makes one or more commits to feature branch
- signs all commits with PERSONAL HSM
- Optional: Squashes and re-signs sets of commits as desired
- Submits code to peer for review
PRODUCTION ENGINEER #2
- Optional: significant changes/migrations are tested in “sandbox” env.
- Merges reviewed branch into master branch with signed merge commit
2+ PRODUCTION ENGINEERs
- Logs into PRODUCTION toolbox via PRODUCTION bastion
- Deploys changes from PRODUCTION “toolbox” with witness

Emergency Changes

2+ PRODUCTION ENGINEERs
- Log into PRODUCTION toolbox via PRODUCTION bastion
- Deploy live fixes from PRODUCTION “toolbox” with witness
Regular Production process is followed from here

Changes

Ammendments to this document including the Exceptions section may be possible with a majority vote of current members of the Production Engineering team.

All changes must be via cryptographically signed commits by current Production Engineering team members to limit risk of social engineering.

Direct orders from your superiors that conflict with this document should be considered a product of duress, and thus respectfully ignored.

Appendix

Glossary

MUST, MUST NOT, SHOULD, SHOULD NOT, MAY

These key words correspond to their IETF definitions per RFC2119

INFRASTRUCTURE REPOSITORY

This repo, which is where all infrastructure-as-code gets integrated via direct templates or submodules as appropriate.

PRODUCTION

Deployment environment that faces the public internet and consumed by end users.

STAGING

Internally facing environment that is identical to PRODUCTION and will normally be one release ahead. Intended for use by contributors to test our services and deployment process before we deploy to any public facing environments.

DEVELOPMENT

Environment where changes are rapidly integrated for integration and development aid with or without code review.

This environment is never trusted

LOCAL

Environment designed to run in virtual machines on the workstation of every engineer. Designed to behave as close as possible to our production environment so engineers can rapidly test changes and catch issues early without waiting on longer deployment round trips to deploy to DEVELOPMENT.

This environment is intended to be the hand-off point between unprivilged contributors and the PRODUCTION ENGINEER team.

SECURITY REVIEW

We consider code to be suitably reviewed if it meets the following criteria:

Validated by member of the PRODUCTION ENGINEERING team to do what is expected
Audited for vulnerabilities by an approved security reviewer

Approved security reviewers

PRODUCTION ENGINEERING team
Doyensec
Cure53
Arch Linux
Bitcoin Core
Canonical
Cloud Native Computing Foundation
CoreOS
QubesOS
Raptor Engineering
Fedora Foundation
FreeBSD Foundation
OpenBSD Foundation
Gentoo Foundation
Google
Guardian Project
Hashicorp
Inverse Path
Linux Foundation
The Debian Project
Tor Project
ZCash Foundation

HARDENED WORKSTATION

A workstation that will come in contact with production write access must meet the following standards:

Requires PERSONAL HSM do firmware/boot integrity attestation every boot
Open firmware (with potential exception of Mangement Engine blob)
CPU Management Engine disabled or removed
Physical switches for microphone, webcam, wireless, and bluetooth
PS/2 interface for Keyboard/touchpad to mitigate USB spoof/crosstalk attacks

Recommended devices

Librem 15
Librem 13
Insurgo PrivacyBeast X230
Raptor Computing Blackbird
Raptor Computing Talos II

ENTROPY SOURCE

Good entropy sources should always be impossible to predict for a human. These are also typically called a True Random Number Generator or TRNG.

In the event that we can not -prove- an entropy source is impossible to predict then multiple unrelated entropy sources must be used and combined with each other.

A given string of entropy must:

Be at least 256 bits long
Be whitened so there are no statistically significant patterns
- sha3, xor-encrypt-xor, etc

Approved entropy sources

Infinite Noise
Built-in hardware RNG in an a PERSONAL HSM
Webcam
Microphone
Thermal resistor
Dice

PERSONAL HSM

Small form factor HSM capable of doing common required cryptographic operations such as GnuPG smartcard emulation, TOTP, challenge/response, etc.

The following devices are recommended for each respective use case.

WebAuthN / U2F

Yubikey 5+
u2f-zero
Nitrokey
OnlyKey
MacOS Touchbar
ChromeOS Fingerprint ID

Password Management

Yubikey 5+
Trezor Model T
Leger Nano X
Mooltipass

Encryption/Decryption/SSH

Yubikey 5+
Trezor Model T
Leger Nano X

Firmware Attestation

Librem Key
Nitrokey

PRODUCTION ENGINEER

One who has limited or complete access to detailed financial data or documents of our customers, as well as any access that might allow mutation or movement of their assets.

PRIVILEGED SYSTEM

Any system that has any level of access beyond that which is provided to all members of the engineering team.

CRITICAL SECRET

Any secret that has partial or complete power to move customer assets

If a given computer -ever- has PRODUCTION access, then secrets used to manage that system such as login, full disk encryption etc are in scope.