sbt release process (#75)

* bin/release.sh script to check for required env variables, run tests, then run `sbt release`
* MAINTAINERS.md that describes steps needed to release
* implemented sbt release to run our release to docker and sonatype

MAINTAINERS.md

@@ -1,11 +1,18 @@
 # Maintainers
 
 ## Table of Contents
-- [Pushing images to Docker Hub](#pushing-images-to-docker-hub)
+* [Docker Content Trust](#docker-content-trust)
+    * [Docker Hub Account](#docker-hub-account)
+    * [Delegating Image Signing](#delegating-image-signing)
+        * [Setting up Notary](#setting-up-notary)
+        * [Generating a Personal Delegation Key](#generating-a-personal-delegation-key)
+        * [Adding a Delegation Key To a Repository](#adding-a-delegation-key-to-a-repository)
+        * [Pushing a Signed Image with your Delegation Key](#pushing-a-signed-image-with-your-delegation-key)
+* [Sonatype Credentials](#sonatype-credentials)
+* [Release Process](#release-process)
 
-## Pushing images to Docker Hub
+## Docker Content Trust
 
-### Docker content trust
 Official VinylDNS Docker images are signed when being pushed to Docker Hub. Docs for Docker Content Trust can be found 
 at https://docs.docker.com/engine/security/trust/content_trust/.
 
@@ -33,29 +40,23 @@ do not change the names of the keys.
 Docker expects these keys to be saved in `~/.docker/trust/private`. Each key is encrypted with a passphrase, that you 
 must have available when pushing an image.
 
-### Pushing a signed image
+### Docker Hub Account
-First make sure you have been given the correct permissions in the vinyldns org on Docker Hub. Then, publish the image 
-you will be pushing locally first. For the API, run `sbt ;project:api;docker:publishLocal`, for the portal, 
-run `sbt ;project:portal;docker:publishLocal`. The image tag will be whatever the project version is set to in 
-`build.sbt` 
 
-Then make sure `DOCKER_CONTENT_TRUST=1` is in your environment, and run `docker push vinyldns/<repo>:<tag>`. e.g. 
+If you don't have one already, make an account on Docker Hub. Get added as a Collaborator to vinyldns/api, vinyldns/portal,
-`docker push vinyldns/api:0.1.0`. When prompted, enter the passphrase for the root key, then the passphrase for the 
+and vinyldns/bind9
-Docker repo you are pushing to. 
 
-### Delegating image signing
+### Delegating Image Signing
-The above method will work as long as a pusher has the required keys and passphrases. Optionally, the following steps can be taken
+Someone with our keys can sign images when pushing, but instead of sharing those keys we can utilize 
-for core maintainers to push signed images via notary, without having to store the keys on their machine.
+notary to delegate image signing permissions in a safer way. Notary will have you make a public-private key pair and 
+upload your public key. This way you only need your private key, and a developer's permissions can easily be revoked. 
 
-The documentation reference for this is https://docs.docker.com/engine/security/trust/trust_delegation/#generating-delegation-keys
+#### Setting up Notary
-
-#### Setting up notary
 If you do not already have notary: 
 
 1. Download the latest release for your machine at https://github.com/theupdateframework/notary/releases, 
 for example, on a mac download the precompiled binary `notary-Darwin-amd64`
 1. Rename the binary to notary, and choose a location where it will live,
-e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/notary`
+e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/notary; cd ~/Documents`
 1. Make it executable, e.g. `chmod +x notary`
 1. Add notary to your path, e.g. `vim ~/.bashrc`, add `export PATH="$PATH":<path to notary>`
 1. Create a `~/.notary/config.json` with
@@ -69,27 +70,90 @@ e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/nota
  }
 ```
 
-You can test notary with `notary -s https://notary.docker.io -d ~/.docker/trust" list docker.io/vinyldns/api`, in which
+You can test notary with `notary -s https://notary.docker.io -d ~/.docker/trust list docker.io/vinyldns/api`, in which
-you should see tagged images for the API
+you should see tagged images for the VinylDNS API
 
-#### Generating a personal delegation key
+> Note: you'll pretty much always use the `-s https://notary.docker.io -d ~/.docker/trust` args when running notary,
-1. cd to a directory where you will save your delegation keys
+it will be easier for you to alias a command like `notarydefault` to `notary -s https://notary.docker.io -d ~/.docker/trust`
+in your `.bashrc` 
+
+#### Generating a Personal Delegation Key
+1. `cd` to a directory where you will save your delegation keys and certs
 1. Generate your private key: `openssl genrsa -out delegation.key 2048`
-1. Generate your public key: `openssl req -new -sha256 -key delegation.key -out delegation.csr`
+1. Generate your public key: `openssl req -new -sha256 -key delegation.key -out delegation.csr`, all fields are optional,
+but when it gets to your email it makes sense to add that
 1. Self-sign your public key (valid for one year): 
 `openssl x509 -req -sha256 -days 365 -in delegation.csr -signkey delegation.key -out delegation.crt`
 1. Change the `delegation.crt` to some unique name, like `my-name-vinyldns-delegation.crt`
 1. Give your `my-name-vinyldns-delegation.crt` to someone that has the root keys and passphrases so 
-they can add your delegation key to the repository
+they can upload your delegation key to the repository
 
-#### Adding a delegation key to a repository
+#### Adding a Delegation Key to a Repository
-This expects you to have the keys and passhphrases for the project that you are adding the delegation to
+This expects you to have the root keys and passphrases for the Docker repositories
 
-1. `notary delegation add docker.io/vinyldns/api targets/releases <team members delegation crt path> --all-paths`
+1. List current keys: `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`
-1. `notary publish docker.io/vinyldns/api`
+1. Add team member's public key: `notary delegation add docker.io/vinyldns/api targets/releases <team members delegation crt path> --all-paths`
-1. Repeat above steps for `docker.io/vinyldns/portal`, `docker.io/vinyldns/bind9`
+1. Push key: `notary publish docker.io/vinyldns/api`
+1. Repeat above steps for `docker.io/vinyldns/portal`
+
+Add their key ID to the table below, it can be viewed with `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`.
+It will be the one that didn't show up when you ran step one of this section
+
+| Key ID | Name |
+|------------------------------------------------------------------|----------------|
+| 66027c822d68133da859f6639983d6d3d9643226b3f7259fc6420964993b499a | Nima Eskandary |
+| | |
+
+#### Pushing a Signed Image with your Delegation Key
+1. Run `notary key import <path to private delegation key> --role user`
+1. You will have to create a passphrase for this key that encrypts it at rest. Use a password generator to make a 
+strong password and save it somewhere safe, like apple keychain or some other password manager
+1. From now on `docker push` will be try to sign images with the delegation key if it was configured for that Docker 
+repository
+
+## Sonatype Credentials
+
+The core module is pushed to oss.sonatype.org under io.vinyldns
+
+To be able to push to sonatype you will need the pgp key used to sign the module. We use a [blackbox](https://github.com/StackExchange/blackbox/)
+repo to share this key and its corresponding passphrase. Follow these steps to set it up properly on your local
+
+1. Ensure you have a gpg key setup on your machine by running `gpg -K`, if you do not then run `gpg --gen-key` to create one,
+note you will have to generate a strong passphrase and save it in some password manager
+1. Make sure you have blackbox, on mac this would be `brew install blackbox`
+1. Clone our blackbox repo, get the git url from another maintainer
+1. Run `blackbox_addadmin <the email associated with your gpg key>`
+1. Commit your changes to the blackbox repo and push to master
+1. Have an existing admin pull the repo and run `gpg --keyring keyrings/live/pubring.kbx --export | gpg --import`, and `blackbox_update_all_files`
+1. Have the existing admin commit and push those changes to master
+1. Back to you - pull the changes, and now you should be able to read those files
+1. Run `blackbox_edit_start vinyldns-sonatype-key.asc.gpg` to temporarily decrypt the sonatype signing key
+1. Run `gpg --import vinyldns-sonatype-key.asc` to import the sonatype signing key to your keyring
+1. Run `blackbox_edit_end vinyldns-sonatype-key.asc.gpg` to re-encrypt the sonatype signing key
+1. Run `blackbox_cat vinyldns-sonatype.txt.gpg` to view the passphrase for that key - you will need this passphrase handy when releasing
+1. Create a file `~/.sbt/1.0/vinyldns-gpg-credentials` with the content
+
+``` 
+realm=GnuPG Key ID
+host=gpg
+user=vinyldns@gmail.com
+password=ignored-must-use-pinentry
+```
+
+## Release Process
+
+We are using sbt-release to run our release steps and auto-bump the version in `version.sbt`. The `bin/release.sh`
+script will first run functional tests, then kick off `sbt release`, which also runs unit and integration tests before
+running the release
+
+1. Follow [Docker Content Trust](#docker-content-trust) to setup a notary delegation for yourself
+1. Follow [Sonatype Credentials](#sonatype-credentials) to setup the sonatype pgp signing key on your local
+1. Make sure you're logged in to Docker with `docker login`
+1. Export `DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE` in your env with your notary key passphrase 
+1. Run `bin/release.sh` _Note: the arg "skip-tests" will skip unit, integration and functional testing before a release_
+1. You will be asked to confirm the version which originally comes from `version.sbt`. _NOTE: if the version ends with 
+`SNAPSHOT`, then the docker latest tag won't be applied and the core module will only be published to the sonatype
+staging repo._
+1. When it comes to the sonatype stage, you will need the passphrase handy for the signing key, [Sonatype Credentials](#sonatype-credentials)
+1. Assuming things were successful, make a pr since sbt release auto-bumped `version.sbt` and made a commit for you
 
-#### Pushing trusted data as a collaborator
-Run `notary key import <path to private delegation key> --role user`, after this `docker push` will sign 
-images with the delegation key if your public key has been added to the repository, and you do not have the 
-root keys and passphrases

bin/release.sh

@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+#
+# Will run all tests, and if they pass, will push new Docker images for vinyldns/api and vinyldns/portal, and push
+# the core module to Maven Central
+#
+# Command line args:
+#   skip-tests: skips functional, unit, and integration tests
+#
+# Necessary environment variables:
+#   DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: passphrase for notary delegation key
+#
+# sbt release will auto-bump version.sbt and make a commit on your local
+#
+
+printf "\nnote: follow the guides in MAINTAINERS.md to setup notary delegation (Docker) and get sonatype key (Maven) \n"
+
+DIR=$( cd $(dirname $0) ; pwd -P )
+
+# gpg sbt plugin fails if this is not set
+export GPG_TTY=$(tty)
+
+# force image signing
+export DOCKER_CONTENT_TRUST=1
+
+##
+# Checking for uncommitted changes
+##
+
+printf "\nchecking for uncommitted changes... \n"
+if ! (cd "$DIR" && git add . && git diff-index --quiet HEAD --)
+then
+    printf "\nerror: attempting to release with uncommitted changes\n"
+    exit 1
+fi
+
+##
+# Checking for environment variables
+##
+
+printf "\nchecking for notary key passphrase in env... \n"
+if [[ -z "${DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE}" ]]; then
+    printf "\nerror: DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE must be set in environment\n"
+    exit 1
+fi
+
+##
+# running tests
+##
+
+if [ "$1" != "skip-tests" ]; then
+    printf "\nrunning api func tests... \n"
+    "$DIR"/remove-vinyl-containers.sh
+    if ! "$DIR"/func-test-api.sh
+    then
+        printf "\nerror: bin/func-test-api.sh failed \n"
+        exit 1
+    fi
+    "$DIR"/remove-vinyl-containers.sh
+
+    printf "\nrunning portal func tests... \n"
+    if ! "$DIR"/func-test-portal.sh
+    then
+        printf "\nerror: bin/func-test-portal.sh failed \n"
+        exit 1
+    fi
+
+    printf "\nrunning verify... \n"
+    if ! "$DIR"/verify.sh
+    then
+        printf "\nerror: bin/verify.sh failed \n"
+        exit 1
+    fi
+else
+    printf "\nskipping tests... \n"
+fi
+
+##
+# run release
+##
+
+printf "\nrunning sbt release... \n"
+cd "$DIR"/../ && sbt release
+
+printf "\nrelease finished \n"

build.sbt

@@ -5,6 +5,7 @@ import com.typesafe.sbt.packager.docker._
 import scoverage.ScoverageKeys.{coverageFailOnMinimum, coverageMinimum}
 import org.scalafmt.sbt.ScalafmtPlugin._
 import microsites._
+import ReleaseTransformations._
 
 resolvers ++= additionalResolvers
 
@@ -45,7 +46,6 @@ def scalaStyleSettings: Seq[Def.Setting[_]] = scalaStyleCompile ++ scalaStyleTes
 // settings that should be inherited by all projects
 lazy val sharedSettings = Seq(
   organization := "vinyldns",
-  version := "0.8.0-SNAPSHOT",
   scalaVersion := "2.12.6",
   organizationName := "Comcast Cable Communications Management, LLC",
   startYear := Some(2018),
@@ -108,7 +108,6 @@ lazy val apiDockerSettings = Seq(
   dockerBaseImage := "openjdk:8u171-jdk",
   dockerUsername := Some("vinyldns"),
   packageName in Docker := "api",
-  dockerUpdateLatest := true,
   dockerExposedPorts := Seq(9000),
   dockerEntrypoint := Seq("/opt/docker/bin/api"),
   dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath
@@ -121,7 +120,7 @@ lazy val apiDockerSettings = Seq(
   bashScriptExtraDefines += """addJava "-Dconfig.file=${app_home}/../conf/application.conf"""",
   bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml"""", // adds logback
   bashScriptExtraDefines += "(cd ${app_home} && ./wait-for-dependencies.sh && cd -)",
-  credentials in Docker := Seq(Credentials(Path.userHome / ".iv2" / ".dockerCredentials")),
+  credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials")),
   dockerCommands ++= Seq(
     Cmd("USER", "root"), // switch to root so we can install netcat
     ExecCmd("RUN", "apt-get", "update"),
@@ -135,7 +134,6 @@ lazy val portalDockerSettings = Seq(
   dockerBaseImage := "openjdk:8u171-jdk",
   dockerUsername := Some("vinyldns"),
   packageName in Docker := "portal",
-  dockerUpdateLatest := true,
   dockerExposedPorts := Seq(9001),
   dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath
   dockerExposedVolumes := Seq("/opt/docker/conf"), // mount extra config to the classpath
@@ -146,7 +144,7 @@ lazy val portalDockerSettings = Seq(
   // adds config file to mount
   bashScriptExtraDefines += """addJava "-Dconfig.file=/opt/docker/conf/application.conf"""",
   bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=/opt/docker/conf/logback.xml"""",
-  credentials in Docker := Seq(Credentials(Path.userHome / ".iv2" / ".dockerCredentials"))
+  credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials"))
 )
 
 lazy val noPublishSettings = Seq(
@@ -210,14 +208,30 @@ lazy val coreBuildSettings = Seq(
   scalacOptions ++= scalacOptionsByV(scalaVersion.value).filterNot(_ == "-Ywarn-unused:params")
 ) ++ pbSettings
 
+import xerial.sbt.Sonatype._
 lazy val corePublishSettings = Seq(
   publishMavenStyle := true,
   publishArtifact in Test := false,
   pomIncludeRepository := { _ => false },
   autoAPIMappings := true,
-  credentials += Credentials(Path.userHome / ".ivy2" / ".credentials"),
   publish in Docker := {},
-  mainClass := None
+  mainClass := None,
+  homepage := Some(url("https://vinyldns.io")),
+  scmInfo := Some(
+    ScmInfo(
+      url("https://github.com/vinyldns/vinyldns"),
+      "scm:git@github.com:vinyldns/vinyldns.git"
+    )
+  ),
+  developers := List(
+    Developer(id="pauljamescleary", name="Paul James Cleary", email="pauljamescleary@gmail.com", url=url("https://github.com/pauljamescleary")),
+    Developer(id="rebstar6", name="Rebecca Star", email="rebstar6@gmail.com", url=url("https://github.com/rebstar6")),
+    Developer(id="nimaeskandary", name="Nima Eskandary", email="nimaesk1@gmail.com", url=url("https://github.com/nimaeskandary")),
+    Developer(id="mitruly", name="Michael Ly", email="michaeltrulyng@gmail.com", url=url("https://github.com/mitruly")),
+    Developer(id="britneywright", name="Britney Wright", email="blw06g@gmail.com", url=url("https://github.com/britneywright")),
+  ),
+  sonatypeProfileName := "io.vinyldns",
+  credentials += Credentials(Path.userHome / ".sbt" / "1.0" / "vinyldns-gpg-credentials")
 )
 
 lazy val core = (project in file("modules/core")).enablePlugins(AutomateHeaderPlugin)
@@ -228,6 +242,7 @@ lazy val core = (project in file("modules/core")).enablePlugins(AutomateHeaderPl
   .settings(libraryDependencies ++= coreDependencies ++ coreTestDependencies.map(_ % "test"))
   .settings(scalaStyleCompile ++ scalaStyleTest)
   .settings(
+    organization := "io.vinyldns",
     coverageMinimum := 85,
     coverageFailOnMinimum := true,
     coverageHighlighting := true
@@ -305,9 +320,87 @@ lazy val docSettings = Seq(
   fork in tut := true
 )
 
-lazy val docs = (project in file("modules/docs")).enablePlugins(MicrositesPlugin)
+lazy val docs = (project in file("modules/docs"))
+  .enablePlugins(MicrositesPlugin)
   .settings(docSettings)
 
+// release stages
+
+lazy val setSonatypeReleaseSettings = ReleaseStep(action = oldState => {
+  // sonatype publish target, and sonatype release steps, are different if version is SNAPSHOT
+  val extracted = Project.extract(oldState)
+  val v = extracted.get(Keys.version)
+  val snap = v.endsWith("SNAPSHOT")
+  if (!snap) {
+    val publishToSettings = Some("releases" at "https://oss.sonatype.org/" + "service/local/staging/deploy/maven2")
+    val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
+
+    // create sonatypeReleaseCommand with releaseSonatype step
+    val sonatypeCommand = Command.command("sonatypeReleaseCommand") {
+      "project core" ::
+      "publish" ::
+      "releaseSonatype" ::
+      _
+    }
+
+    newState.copy(definedCommands = newState.definedCommands :+ sonatypeCommand)
+  } else {
+    val publishToSettings = Some("snapshots" at "https://oss.sonatype.org/" + "content/repositories/snapshots")
+    val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
+
+    // create sonatypeReleaseCommand without releaseSonatype step
+    val sonatypeCommand = Command.command("sonatypeReleaseCommand") {
+      "project core" ::
+        "publish" ::
+        _
+    }
+
+    newState.copy(definedCommands = newState.definedCommands :+ sonatypeCommand)
+  }
+})
+
+lazy val setDockerReleaseSettings = ReleaseStep(action = oldState => {
+  // dockerUpdateLatest is set to true if the version is not a SNAPSHOT
+  val extracted = Project.extract(oldState)
+  val v = extracted.get(Keys.version)
+  val snap = v.endsWith("SNAPSHOT")
+  if (!snap) {
+    extracted
+      .appendWithSession(Seq(dockerUpdateLatest in api := true, dockerUpdateLatest in portal := true), oldState)
+  } else oldState
+})
+
+lazy val initReleaseStage = Seq[ReleaseStep](
+  releaseStepCommand(";project root"), // use version.sbt file from root
+  inquireVersions, // have a developer confirm versions
+  setReleaseVersion,
+  setDockerReleaseSettings,
+  setSonatypeReleaseSettings
+)
+
+lazy val dockerPublishStage = Seq[ReleaseStep](
+  releaseStepCommandAndRemaining(";project api;docker:publish"),
+  releaseStepCommandAndRemaining(";project portal;docker:publish")
+)
+
+lazy val sonatypePublishStage = Seq[ReleaseStep](
+  releaseStepCommandAndRemaining(";sonatypeReleaseCommand")
+)
+
+lazy val finalReleaseStage = Seq[ReleaseStep] (
+  releaseStepCommand("project root"), // use version.sbt file from root
+  commitReleaseVersion,
+  tagRelease,
+  setNextVersion,
+  commitNextVersion
+)
+
+releaseProcess :=
+  initReleaseStage ++
+  dockerPublishStage ++
+  sonatypePublishStage ++
+  finalReleaseStage
+
 // Validate runs static checks and compile to make sure we can go
 addCommandAlias("validate-api",
   ";project api; clean; headerCheck; test:headerCheck; it:headerCheck; scalastyle; test:scalastyle; " +

project/plugins.sbt

@@ -33,3 +33,7 @@ addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0")
 addSbtPlugin("com.typesafe.sbt" % "sbt-license-report" % "1.2.0")
 
 addSbtPlugin("com.47deg"  % "sbt-microsites" % "0.7.22")
+
+addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.3")
+
+addSbtPlugin("io.crashbox" % "sbt-gpg" % "0.2.0")

version.sbt

@@ -1 +1 @@
-version in ThisBuild := "0.1"
+version in ThisBuild := "0.8.0-SNAPSHOT"