diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 71d3183d8..54e0f3f1c 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -1,11 +1,18 @@ # Maintainers ## Table of Contents -- [Pushing images to Docker Hub](#pushing-images-to-docker-hub) +* [Docker Content Trust](#docker-content-trust) + * [Docker Hub Account](#docker-hub-account) + * [Delegating Image Signing](#delegating-image-signing) + * [Setting up Notary](#setting-up-notary) + * [Generating a Personal Delegation Key](#generating-a-personal-delegation-key) + * [Adding a Delegation Key To a Repository](#adding-a-delegation-key-to-a-repository) + * [Pushing a Signed Image with your Delegation Key](#pushing-a-signed-image-with-your-delegation-key) +* [Sonatype Credentials](#sonatype-credentials) +* [Release Process](#release-process) -## Pushing images to Docker Hub +## Docker Content Trust -### Docker content trust Official VinylDNS Docker images are signed when being pushed to Docker Hub. Docs for Docker Content Trust can be found at https://docs.docker.com/engine/security/trust/content_trust/. @@ -33,29 +40,23 @@ do not change the names of the keys. Docker expects these keys to be saved in `~/.docker/trust/private`. Each key is encrypted with a passphrase, that you must have available when pushing an image. -### Pushing a signed image -First make sure you have been given the correct permissions in the vinyldns org on Docker Hub. Then, publish the image -you will be pushing locally first. For the API, run `sbt ;project:api;docker:publishLocal`, for the portal, -run `sbt ;project:portal;docker:publishLocal`. The image tag will be whatever the project version is set to in -`build.sbt` +### Docker Hub Account -Then make sure `DOCKER_CONTENT_TRUST=1` is in your environment, and run `docker push vinyldns/:`. e.g. -`docker push vinyldns/api:0.1.0`. When prompted, enter the passphrase for the root key, then the passphrase for the -Docker repo you are pushing to. +If you don't have one already, make an account on Docker Hub. Get added as a Collaborator to vinyldns/api, vinyldns/portal, +and vinyldns/bind9 -### Delegating image signing -The above method will work as long as a pusher has the required keys and passphrases. Optionally, the following steps can be taken -for core maintainers to push signed images via notary, without having to store the keys on their machine. +### Delegating Image Signing +Someone with our keys can sign images when pushing, but instead of sharing those keys we can utilize +notary to delegate image signing permissions in a safer way. Notary will have you make a public-private key pair and +upload your public key. This way you only need your private key, and a developer's permissions can easily be revoked. -The documentation reference for this is https://docs.docker.com/engine/security/trust/trust_delegation/#generating-delegation-keys - -#### Setting up notary +#### Setting up Notary If you do not already have notary: 1. Download the latest release for your machine at https://github.com/theupdateframework/notary/releases, for example, on a mac download the precompiled binary `notary-Darwin-amd64` 1. Rename the binary to notary, and choose a location where it will live, -e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/notary` +e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/notary; cd ~/Documents` 1. Make it executable, e.g. `chmod +x notary` 1. Add notary to your path, e.g. `vim ~/.bashrc`, add `export PATH="$PATH":` 1. Create a `~/.notary/config.json` with @@ -69,27 +70,90 @@ e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/nota } ``` -You can test notary with `notary -s https://notary.docker.io -d ~/.docker/trust" list docker.io/vinyldns/api`, in which -you should see tagged images for the API +You can test notary with `notary -s https://notary.docker.io -d ~/.docker/trust list docker.io/vinyldns/api`, in which +you should see tagged images for the VinylDNS API -#### Generating a personal delegation key -1. cd to a directory where you will save your delegation keys +> Note: you'll pretty much always use the `-s https://notary.docker.io -d ~/.docker/trust` args when running notary, +it will be easier for you to alias a command like `notarydefault` to `notary -s https://notary.docker.io -d ~/.docker/trust` +in your `.bashrc` + +#### Generating a Personal Delegation Key +1. `cd` to a directory where you will save your delegation keys and certs 1. Generate your private key: `openssl genrsa -out delegation.key 2048` -1. Generate your public key: `openssl req -new -sha256 -key delegation.key -out delegation.csr` +1. Generate your public key: `openssl req -new -sha256 -key delegation.key -out delegation.csr`, all fields are optional, +but when it gets to your email it makes sense to add that 1. Self-sign your public key (valid for one year): `openssl x509 -req -sha256 -days 365 -in delegation.csr -signkey delegation.key -out delegation.crt` 1. Change the `delegation.crt` to some unique name, like `my-name-vinyldns-delegation.crt` 1. Give your `my-name-vinyldns-delegation.crt` to someone that has the root keys and passphrases so -they can add your delegation key to the repository +they can upload your delegation key to the repository -#### Adding a delegation key to a repository -This expects you to have the keys and passhphrases for the project that you are adding the delegation to +#### Adding a Delegation Key to a Repository +This expects you to have the root keys and passphrases for the Docker repositories -1. `notary delegation add docker.io/vinyldns/api targets/releases --all-paths` -1. `notary publish docker.io/vinyldns/api` -1. Repeat above steps for `docker.io/vinyldns/portal`, `docker.io/vinyldns/bind9` +1. List current keys: `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api` +1. Add team member's public key: `notary delegation add docker.io/vinyldns/api targets/releases --all-paths` +1. Push key: `notary publish docker.io/vinyldns/api` +1. Repeat above steps for `docker.io/vinyldns/portal` + +Add their key ID to the table below, it can be viewed with `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`. +It will be the one that didn't show up when you ran step one of this section + +| Key ID | Name | +|------------------------------------------------------------------|----------------| +| 66027c822d68133da859f6639983d6d3d9643226b3f7259fc6420964993b499a | Nima Eskandary | +| | | + +#### Pushing a Signed Image with your Delegation Key +1. Run `notary key import --role user` +1. You will have to create a passphrase for this key that encrypts it at rest. Use a password generator to make a +strong password and save it somewhere safe, like apple keychain or some other password manager +1. From now on `docker push` will be try to sign images with the delegation key if it was configured for that Docker +repository + +## Sonatype Credentials + +The core module is pushed to oss.sonatype.org under io.vinyldns + +To be able to push to sonatype you will need the pgp key used to sign the module. We use a [blackbox](https://github.com/StackExchange/blackbox/) +repo to share this key and its corresponding passphrase. Follow these steps to set it up properly on your local + +1. Ensure you have a gpg key setup on your machine by running `gpg -K`, if you do not then run `gpg --gen-key` to create one, +note you will have to generate a strong passphrase and save it in some password manager +1. Make sure you have blackbox, on mac this would be `brew install blackbox` +1. Clone our blackbox repo, get the git url from another maintainer +1. Run `blackbox_addadmin ` +1. Commit your changes to the blackbox repo and push to master +1. Have an existing admin pull the repo and run `gpg --keyring keyrings/live/pubring.kbx --export | gpg --import`, and `blackbox_update_all_files` +1. Have the existing admin commit and push those changes to master +1. Back to you - pull the changes, and now you should be able to read those files +1. Run `blackbox_edit_start vinyldns-sonatype-key.asc.gpg` to temporarily decrypt the sonatype signing key +1. Run `gpg --import vinyldns-sonatype-key.asc` to import the sonatype signing key to your keyring +1. Run `blackbox_edit_end vinyldns-sonatype-key.asc.gpg` to re-encrypt the sonatype signing key +1. Run `blackbox_cat vinyldns-sonatype.txt.gpg` to view the passphrase for that key - you will need this passphrase handy when releasing +1. Create a file `~/.sbt/1.0/vinyldns-gpg-credentials` with the content + +``` +realm=GnuPG Key ID +host=gpg +user=vinyldns@gmail.com +password=ignored-must-use-pinentry +``` + +## Release Process + +We are using sbt-release to run our release steps and auto-bump the version in `version.sbt`. The `bin/release.sh` +script will first run functional tests, then kick off `sbt release`, which also runs unit and integration tests before +running the release + +1. Follow [Docker Content Trust](#docker-content-trust) to setup a notary delegation for yourself +1. Follow [Sonatype Credentials](#sonatype-credentials) to setup the sonatype pgp signing key on your local +1. Make sure you're logged in to Docker with `docker login` +1. Export `DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE` in your env with your notary key passphrase +1. Run `bin/release.sh` _Note: the arg "skip-tests" will skip unit, integration and functional testing before a release_ +1. You will be asked to confirm the version which originally comes from `version.sbt`. _NOTE: if the version ends with +`SNAPSHOT`, then the docker latest tag won't be applied and the core module will only be published to the sonatype +staging repo._ +1. When it comes to the sonatype stage, you will need the passphrase handy for the signing key, [Sonatype Credentials](#sonatype-credentials) +1. Assuming things were successful, make a pr since sbt release auto-bumped `version.sbt` and made a commit for you -#### Pushing trusted data as a collaborator -Run `notary key import --role user`, after this `docker push` will sign -images with the delegation key if your public key has been added to the repository, and you do not have the -root keys and passphrases diff --git a/bin/release.sh b/bin/release.sh new file mode 100755 index 000000000..416c4f38f --- /dev/null +++ b/bin/release.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# +# Will run all tests, and if they pass, will push new Docker images for vinyldns/api and vinyldns/portal, and push +# the core module to Maven Central +# +# Command line args: +# skip-tests: skips functional, unit, and integration tests +# +# Necessary environment variables: +# DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: passphrase for notary delegation key +# +# sbt release will auto-bump version.sbt and make a commit on your local +# + +printf "\nnote: follow the guides in MAINTAINERS.md to setup notary delegation (Docker) and get sonatype key (Maven) \n" + +DIR=$( cd $(dirname $0) ; pwd -P ) + +# gpg sbt plugin fails if this is not set +export GPG_TTY=$(tty) + +# force image signing +export DOCKER_CONTENT_TRUST=1 + +## +# Checking for uncommitted changes +## + +printf "\nchecking for uncommitted changes... \n" +if ! (cd "$DIR" && git add . && git diff-index --quiet HEAD --) +then + printf "\nerror: attempting to release with uncommitted changes\n" + exit 1 +fi + +## +# Checking for environment variables +## + +printf "\nchecking for notary key passphrase in env... \n" +if [[ -z "${DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE}" ]]; then + printf "\nerror: DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE must be set in environment\n" + exit 1 +fi + +## +# running tests +## + +if [ "$1" != "skip-tests" ]; then + printf "\nrunning api func tests... \n" + "$DIR"/remove-vinyl-containers.sh + if ! "$DIR"/func-test-api.sh + then + printf "\nerror: bin/func-test-api.sh failed \n" + exit 1 + fi + "$DIR"/remove-vinyl-containers.sh + + printf "\nrunning portal func tests... \n" + if ! "$DIR"/func-test-portal.sh + then + printf "\nerror: bin/func-test-portal.sh failed \n" + exit 1 + fi + + printf "\nrunning verify... \n" + if ! "$DIR"/verify.sh + then + printf "\nerror: bin/verify.sh failed \n" + exit 1 + fi +else + printf "\nskipping tests... \n" +fi + +## +# run release +## + +printf "\nrunning sbt release... \n" +cd "$DIR"/../ && sbt release + +printf "\nrelease finished \n" diff --git a/build.sbt b/build.sbt index c8e9ecaa5..f71a03292 100644 --- a/build.sbt +++ b/build.sbt @@ -5,6 +5,7 @@ import com.typesafe.sbt.packager.docker._ import scoverage.ScoverageKeys.{coverageFailOnMinimum, coverageMinimum} import org.scalafmt.sbt.ScalafmtPlugin._ import microsites._ +import ReleaseTransformations._ resolvers ++= additionalResolvers @@ -45,7 +46,6 @@ def scalaStyleSettings: Seq[Def.Setting[_]] = scalaStyleCompile ++ scalaStyleTes // settings that should be inherited by all projects lazy val sharedSettings = Seq( organization := "vinyldns", - version := "0.8.0-SNAPSHOT", scalaVersion := "2.12.6", organizationName := "Comcast Cable Communications Management, LLC", startYear := Some(2018), @@ -108,7 +108,6 @@ lazy val apiDockerSettings = Seq( dockerBaseImage := "openjdk:8u171-jdk", dockerUsername := Some("vinyldns"), packageName in Docker := "api", - dockerUpdateLatest := true, dockerExposedPorts := Seq(9000), dockerEntrypoint := Seq("/opt/docker/bin/api"), dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath @@ -121,7 +120,7 @@ lazy val apiDockerSettings = Seq( bashScriptExtraDefines += """addJava "-Dconfig.file=${app_home}/../conf/application.conf"""", bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml"""", // adds logback bashScriptExtraDefines += "(cd ${app_home} && ./wait-for-dependencies.sh && cd -)", - credentials in Docker := Seq(Credentials(Path.userHome / ".iv2" / ".dockerCredentials")), + credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials")), dockerCommands ++= Seq( Cmd("USER", "root"), // switch to root so we can install netcat ExecCmd("RUN", "apt-get", "update"), @@ -135,7 +134,6 @@ lazy val portalDockerSettings = Seq( dockerBaseImage := "openjdk:8u171-jdk", dockerUsername := Some("vinyldns"), packageName in Docker := "portal", - dockerUpdateLatest := true, dockerExposedPorts := Seq(9001), dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath dockerExposedVolumes := Seq("/opt/docker/conf"), // mount extra config to the classpath @@ -146,7 +144,7 @@ lazy val portalDockerSettings = Seq( // adds config file to mount bashScriptExtraDefines += """addJava "-Dconfig.file=/opt/docker/conf/application.conf"""", bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=/opt/docker/conf/logback.xml"""", - credentials in Docker := Seq(Credentials(Path.userHome / ".iv2" / ".dockerCredentials")) + credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials")) ) lazy val noPublishSettings = Seq( @@ -210,14 +208,30 @@ lazy val coreBuildSettings = Seq( scalacOptions ++= scalacOptionsByV(scalaVersion.value).filterNot(_ == "-Ywarn-unused:params") ) ++ pbSettings +import xerial.sbt.Sonatype._ lazy val corePublishSettings = Seq( publishMavenStyle := true, publishArtifact in Test := false, pomIncludeRepository := { _ => false }, autoAPIMappings := true, - credentials += Credentials(Path.userHome / ".ivy2" / ".credentials"), publish in Docker := {}, - mainClass := None + mainClass := None, + homepage := Some(url("https://vinyldns.io")), + scmInfo := Some( + ScmInfo( + url("https://github.com/vinyldns/vinyldns"), + "scm:git@github.com:vinyldns/vinyldns.git" + ) + ), + developers := List( + Developer(id="pauljamescleary", name="Paul James Cleary", email="pauljamescleary@gmail.com", url=url("https://github.com/pauljamescleary")), + Developer(id="rebstar6", name="Rebecca Star", email="rebstar6@gmail.com", url=url("https://github.com/rebstar6")), + Developer(id="nimaeskandary", name="Nima Eskandary", email="nimaesk1@gmail.com", url=url("https://github.com/nimaeskandary")), + Developer(id="mitruly", name="Michael Ly", email="michaeltrulyng@gmail.com", url=url("https://github.com/mitruly")), + Developer(id="britneywright", name="Britney Wright", email="blw06g@gmail.com", url=url("https://github.com/britneywright")), + ), + sonatypeProfileName := "io.vinyldns", + credentials += Credentials(Path.userHome / ".sbt" / "1.0" / "vinyldns-gpg-credentials") ) lazy val core = (project in file("modules/core")).enablePlugins(AutomateHeaderPlugin) @@ -228,6 +242,7 @@ lazy val core = (project in file("modules/core")).enablePlugins(AutomateHeaderPl .settings(libraryDependencies ++= coreDependencies ++ coreTestDependencies.map(_ % "test")) .settings(scalaStyleCompile ++ scalaStyleTest) .settings( + organization := "io.vinyldns", coverageMinimum := 85, coverageFailOnMinimum := true, coverageHighlighting := true @@ -305,9 +320,87 @@ lazy val docSettings = Seq( fork in tut := true ) -lazy val docs = (project in file("modules/docs")).enablePlugins(MicrositesPlugin) +lazy val docs = (project in file("modules/docs")) + .enablePlugins(MicrositesPlugin) .settings(docSettings) +// release stages + +lazy val setSonatypeReleaseSettings = ReleaseStep(action = oldState => { + // sonatype publish target, and sonatype release steps, are different if version is SNAPSHOT + val extracted = Project.extract(oldState) + val v = extracted.get(Keys.version) + val snap = v.endsWith("SNAPSHOT") + if (!snap) { + val publishToSettings = Some("releases" at "https://oss.sonatype.org/" + "service/local/staging/deploy/maven2") + val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState) + + // create sonatypeReleaseCommand with releaseSonatype step + val sonatypeCommand = Command.command("sonatypeReleaseCommand") { + "project core" :: + "publish" :: + "releaseSonatype" :: + _ + } + + newState.copy(definedCommands = newState.definedCommands :+ sonatypeCommand) + } else { + val publishToSettings = Some("snapshots" at "https://oss.sonatype.org/" + "content/repositories/snapshots") + val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState) + + // create sonatypeReleaseCommand without releaseSonatype step + val sonatypeCommand = Command.command("sonatypeReleaseCommand") { + "project core" :: + "publish" :: + _ + } + + newState.copy(definedCommands = newState.definedCommands :+ sonatypeCommand) + } +}) + +lazy val setDockerReleaseSettings = ReleaseStep(action = oldState => { + // dockerUpdateLatest is set to true if the version is not a SNAPSHOT + val extracted = Project.extract(oldState) + val v = extracted.get(Keys.version) + val snap = v.endsWith("SNAPSHOT") + if (!snap) { + extracted + .appendWithSession(Seq(dockerUpdateLatest in api := true, dockerUpdateLatest in portal := true), oldState) + } else oldState +}) + +lazy val initReleaseStage = Seq[ReleaseStep]( + releaseStepCommand(";project root"), // use version.sbt file from root + inquireVersions, // have a developer confirm versions + setReleaseVersion, + setDockerReleaseSettings, + setSonatypeReleaseSettings +) + +lazy val dockerPublishStage = Seq[ReleaseStep]( + releaseStepCommandAndRemaining(";project api;docker:publish"), + releaseStepCommandAndRemaining(";project portal;docker:publish") +) + +lazy val sonatypePublishStage = Seq[ReleaseStep]( + releaseStepCommandAndRemaining(";sonatypeReleaseCommand") +) + +lazy val finalReleaseStage = Seq[ReleaseStep] ( + releaseStepCommand("project root"), // use version.sbt file from root + commitReleaseVersion, + tagRelease, + setNextVersion, + commitNextVersion +) + +releaseProcess := + initReleaseStage ++ + dockerPublishStage ++ + sonatypePublishStage ++ + finalReleaseStage + // Validate runs static checks and compile to make sure we can go addCommandAlias("validate-api", ";project api; clean; headerCheck; test:headerCheck; it:headerCheck; scalastyle; test:scalastyle; " + diff --git a/project/plugins.sbt b/project/plugins.sbt index 7138c5f07..7742442a4 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -33,3 +33,7 @@ addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0") addSbtPlugin("com.typesafe.sbt" % "sbt-license-report" % "1.2.0") addSbtPlugin("com.47deg" % "sbt-microsites" % "0.7.22") + +addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.3") + +addSbtPlugin("io.crashbox" % "sbt-gpg" % "0.2.0") diff --git a/version.sbt b/version.sbt index 06f50fc03..d8a83a96c 100644 --- a/version.sbt +++ b/version.sbt @@ -1 +1 @@ -version in ThisBuild := "0.1" +version in ThisBuild := "0.8.0-SNAPSHOT"