Support building docker images locally and without SBT (#753)

Fixes #744 

Overhauls our docker release process to run releases of docker outside of SBT.  This was tested through the signing process deploying SNAPSHOT images to docker hub.

All images do a git clone to pull vinyldns, and perform the build of the container based on that git clone.

- A new `build` directory contains the things to be built, for this PR only the docker images are here.  Anticipate possibly pulling in other artifacts in the future.
- `build/docker` contains all of the docker setup
- `build/docker/test` contains a func test docker image that can be used to test any VinylDNS API instance
- `build/docker/test-bind9` contains our bind9 setup
- `build/docker/api` is a MySQL only API distribution
- `build/docker/portal` is a MySQL only Portal distribution

In addition, a few files are added to make the build easier:

- `build/start.sh` starts a given vinyldns version
- `build/stop.sh` stops the current running vinyldns setup
- `build/release.sh` this is the **MAIN** guy that performs the build.  Lots of arguments that are useful for testing.  Can provide a repository (for publishing to a docker register that is not docker hub), for building clean vs. assuming docker images.  Besides building everything, also runs func tests (smoke tests) against the docker image before completing.

Added a `README.md` for others to follow along with

To test..

1. Make sure you increase your docker machine settings to 4GB (the default of 2GB failed)
2. Go to the `build` directory
3. From there, run `./release.sh -c -t 111` to only build the docker images, this should build `0.9.4-b111`
4. Start it up, run `./start.sh -v 0.9.4-b111`
5. Login to the portal at http://localhost:9001
6. When finished, run `./stop.sh` from the build directory
*If you see an error code 137 error, you need to increase your memory*

bin/docker-up-vinyldns.sh

@@ -75,7 +75,7 @@ function wait_for_portal {
 # initial var setup
 DIR=$( cd $(dirname $0) ; pwd -P )
 TIMEOUT=60
-DOCKER_COMPOSE_CONFIG="${DIR}/../docker/docker-compose-build.yml"
+DOCKER_COMPOSE_CONFIG="${DIR}/../docker/docker-compose-quick-start.yml"
 # empty service starts up all docker services in compose file
 SERVICE=""
 # when CLEAN is set to 1, existing docker images are deleted so they are re-pulled

build/README.md

@@ -0,0 +1,103 @@
+## Building VinylDNS
+
+This folder contains scripts and everything you need to build and test VinylDNS from your own machine.
+
+## Pre-requisites
+
+- `docker` - you will need docker and docker-compose installed locally
+
+## Local Build and Test
+
+1. `./release.sh --clean`
+1. Open up `version.sbt` in the root to know the directory (or capture in the script output)
+1. Once complete, run a test `./start.sh --version 0.9.4-SNAPSHOT` (replace 0.9.4 with the value in version.sbt).
+1. Login to the portal at http://localhost:9001 to verify everything looks good
+1. Run `./stop.sh` to bring everything down
+
+### Release Process
+
+1. If you are using image signing / docker notary, be sure you set the environment variable `export DOCKER_CONTENT_TRUST=1`.  
+Whether you sign or not is up to your organization.  You need to have notary setup to be able to sign properly.
+1. Be sure to login to your docker registry, typically done by `docker login` in the terminal you will release from.
+1. The actual version number is pulled from the local `version.sbt` based on the branch specified (defaults to master)
+1. Run `./release.sh --push --clean --tag [your tag here] --branch [your branch here]`
+    1. typically the `tag` is a build number that you maintain, for example a build number in Jenkins.  Using this field is recommended.  This value will be appended to the generated version as `-b[TAG]`; for example `0.9.4-b123` if using `123` for the tag.
+    1. the `branch` defaults to `master` if not specified, you can choose any branch or tag from https://github.com/vinyldns/vinyldns
+1. The version generated will be whatever the version is in the `version.sbt` on the `branch` specified (defaults to master)
+1. Each of the images are built using the branch specified and the correct version
+1. The func tests are run with only smoke tests against the API image to verify it is working
+1. If everything passes, and the user specifies `--push`, the images are tagged and released to the docker repository (defaults to docker hub)
+
+### Release Script
+Does a clean build off of remote master and tags it with 
+`./release.sh --clean --push --tag 123`
+
+The release script is used for doing a release.  It takes the following parameters:
+
+- `-b | --branch [BRANCH]` - what branch to pull from, can be any PR branch or a tag like `v0.9.3`, defaults to `master`
+- `-c | --clean` - a flag that indicates to perform a build.  If omitted, the release script will look for a 
+pre-built image locally
+- `-p | --push` - a flag that indicates to push to the remote docker registry.  The default docker registry 
+is `docker.io`
+- `-r | --repository [REPOSITORY]` - a URL to your docker registry, defaults to `docker.io`
+- `-t | --tag [TAG]` - a build qualifer for this build.  For example, pass in the build number for your 
+continuous integration tool
+
+## Docker Images
+
+The build will generate several VinylDNS docker images that are used to deploy into any environment VinylDNS
+
+- `vinyldns/api` - this is the heart of the VinylDNS system, the backend API
+- `vinyldns/portal` - the VinylDNS web UI
+- `vinyldns/test-bind9` - a DNS server that is configured to support running the functional tests
+- `vinyldns/test` - a container that will execute functional tests, and exit success or failure when the tests are complete
+
+### vinyldns/api
+
+The default build for vinyldns api assumes an **ALL MYSQL** installation. 
+
+**Environment Variables**
+- `VINYLDNS_VERSION` - this is the version of VinylDNS the API is running, typically you will not set this as 
+it is set as part of the container build
+
+**Volumes**
+- `/opt/docker/conf/` - if you need to have your own application config file.  This is **MANDATORY** for
+any production environments.  Typically, you will add your own `application.conf` file in here with your settings.
+- `/opt/docker/lib_extra/` - if you need to have additional jar files available to your VinylDNS instance.
+Rarely used, but if you want to bring your own message queue or database you can put the `jar` files there
+
+### vinyldns/portal
+
+The default build for vinyldns portal assumes an **ALL MYSQL** installation.
+
+**Environment Variables**
+- `VINYLDNS_VERSION` - this is the version of VinylDNS the API is running, typically you will not set this as 
+it is set as part of the container build
+
+**Volumes**
+- `/opt/docker/conf/` - if you need to have your own application config file.  This is **MANDATORY** for
+any production environments.  Typically, you will add your own `application.conf` file in here with your settings.
+- `/opt/docker/lib_extra/` - if you need to have additional jar files available to your VinylDNS instance.
+Rarely used, but if you want to bring your own message queue or database you can put the `jar` files there
+
+### vinyldns/test-bind9
+
+This pulls correct DNS configuration to run func tests.  You can largely disregard what is in here
+
+### vinyldns/test
+
+This is used to run functional tests against a vinyldns instance.  **This is very useful for verifying 
+your environment as part of doing an upgrade.**  By default, it will run against a local docker-compose setup.
+
+**Environment Variables**
+- `VINYLDNS_URL` - the url to the vinyldns you will test against
+- `DNS_IP` - the IP address to the `vinyldns/test-bind9` container that you will use for test purposes
+- `TEST_PATTERN` - the actual functional test you want to run.  *Important, set to empty string to run 
+ALL test; otherwise, omit the environment variable when you run to just run smoke tests*.
+
+**Example**
+
+This example will run all functional tests on the given VinylDNS url and DNS IP address
+`docker run -e VINYLDNS_URL="https://my.vinyldns.example.com" -e DNS_IP="1.2.3.4" -e TEST_PATTERN=""`
+
+

build/docker/api/.jvmopts

@@ -0,0 +1,6 @@
+-Xms512M
+-Xmx1024M
+-Xss2M
+-XX:MaxMetaspaceSize=512M
+-XX:ReservedCodeCacheSize=512M
+-Djava.net.preferIPv4Stack=true

build/docker/api/Dockerfile

@@ -0,0 +1,31 @@
+FROM hseeberger/scala-sbt:8u212_1.2.8_2.12.9 as builder
+
+ARG BRANCH=master
+ARG VINYLDNS_VERSION
+
+RUN git clone -b ${BRANCH} --single-branch --depth 1 https://github.com/vinyldns/vinyldns.git /vinyldns
+
+# The default jvmopts are huge, meant for running everything, use a paired down version
+COPY .jvmopts /vinyldns
+
+RUN cd /vinyldns ; sbt "set version in ThisBuild := \"${VINYLDNS_VERSION}\"" api/stage
+
+FROM openjdk:8u201-jdk-alpine3.9
+
+RUN apk add --update --no-cache netcat-openbsd bash
+
+COPY --from=builder /vinyldns/modules/api/target/universal/stage /opt/docker
+
+# This will set the vinyldns version, make sure to have this in config... version = ${?VINYLDNS_VERSION}
+ARG VINYLDNS_VERSION
+ENV VINYLDNS_VERSION=$VINYLDNS_VERSION
+
+RUN mkdir -p /opt/docker/lib_extra
+
+# Mount the volume for config file and lib extras
+# Note: These volume names are used in the build.sbt
+VOLUME ["/opt/docker/lib_extra/", "/opt/docker/conf"]
+
+EXPOSE 9000
+
+ENTRYPOINT ["/opt/docker/bin/api"]

build/docker/api/application.conf

@@ -0,0 +1,170 @@
+vinyldns {
+  version = "unknown"
+  version = ${?VINYLDNS_VERSION}
+
+  queue {
+    class-name = "vinyldns.mysql.queue.MySqlMessageQueueProvider"
+    polling-interval = 250.millis
+    messages-per-poll = 10
+
+    settings = {
+      name = "vinyldns"
+      driver = "org.mariadb.jdbc.Driver"
+      migration-url = "jdbc:mariadb://vinyldns-mysql:3306/?user=root&password=pass"
+      url = "jdbc:mariadb://vinyldns-mysql:3306/vinyldns?user=root&password=pass"
+      user = "root"
+      password = "pass"
+
+      # see https://github.com/brettwooldridge/HikariCP
+      connection-timeout-millis = 1000
+      idle-timeout = 10000
+      max-lifetime = 30000
+      maximum-pool-size = 5
+      minimum-idle = 0
+
+      my-sql-properties = {
+        cachePrepStmts=true
+        prepStmtCacheSize=250
+        prepStmtCacheSqlLimit=2048
+        rewriteBatchedStatements=true
+      }
+    }
+  }
+
+  rest {
+    host = "0.0.0.0"
+    port = 9000
+  }
+
+  sync-delay = 10000
+
+  crypto {
+    type = "vinyldns.core.crypto.NoOpCrypto"
+  }
+
+  data-stores = ["mysql"]
+
+  mysql {
+    settings {
+      # JDBC Settings, these are all values in scalikejdbc-config, not our own
+      # these must be overridden to use MYSQL for production use
+      # assumes a docker or mysql instance running locally
+      name = "vinyldns"
+      driver = "org.mariadb.jdbc.Driver"
+      migration-url = "jdbc:mariadb://vinyldns-mysql:3306/?user=root&password=pass"
+      url = "jdbc:mariadb://vinyldns-mysql:3306/vinyldns?user=root&password=pass"
+      user = "root"
+      password = "pass"
+      # see https://github.com/brettwooldridge/HikariCP
+      connection-timeout-millis = 1000
+      idle-timeout = 10000
+      max-lifetime = 600000
+      maximum-pool-size = 20
+      minimum-idle = 20
+      register-mbeans = true
+    }
+    # Repositories that use this data store are listed here
+    repositories {
+      zone {
+      }
+      batch-change {
+      }
+      user {
+      }
+      record-set {
+      }
+      zone-change {
+      }
+      record-change {
+      }
+      group {
+      }
+      group-change {
+      }
+      membership {
+      }
+    }
+  }
+
+  defaultZoneConnection {
+    name = "vinyldns."
+    keyName = "vinyldns."
+    key = "nzisn+4G2ldMn0q1CV3vsg=="
+    primaryServer = "vinyldns-bind9"
+  }
+
+  defaultTransferConnection {
+    name = "vinyldns."
+    keyName = "vinyldns."
+    key = "nzisn+4G2ldMn0q1CV3vsg=="
+    primaryServer = "vinyldns-bind9"
+  }
+
+  backends = [
+    {
+      id = "func-test-backend"
+      zone-connection {
+        name = "vinyldns."
+        key-name = "vinyldns."
+        key = "nzisn+4G2ldMn0q1CV3vsg=="
+        primary-server = "vinyldns-bind9"
+      }
+      transfer-connection {
+        name = "vinyldns."
+        key-name = "vinyldns."
+        key = "nzisn+4G2ldMn0q1CV3vsg=="
+        primary-server = "vinyldns-bind9"
+      }
+    }
+  ]
+
+  batch-change-limit = 1000
+
+  # FQDNs / IPs that cannot be modified via VinylDNS
+  # regex-list used for all record types except PTR
+  # ip-list used exclusively for PTR records
+  high-value-domains = {
+    regex-list = [
+      "high-value-domain.*" # for testing
+    ]
+    ip-list = [
+      # using reverse zones in the vinyldns/bind9 docker image for testing
+      "192.0.2.252",
+      "192.0.2.253",
+      "fd69:27cc:fe91:0:0:0:0:ffff",
+      "fd69:27cc:fe91:0:0:0:ffff:0"
+    ]
+  }
+
+  # types of unowned records that users can access in shared zones
+  shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"]
+
+  manual-batch-review-enabled = true
+}
+
+akka {
+  loglevel = "INFO"
+  loggers = ["akka.event.slf4j.Slf4jLogger"]
+  logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
+  logger-startup-timeout = 30s
+
+  actor {
+    provider = "akka.actor.LocalActorRefProvider"
+  }
+}
+
+akka.http {
+  server {
+    # The time period within which the TCP binding process must be completed.
+    # Set to `infinite` to disable.
+    bind-timeout = 5s
+
+    # Show verbose error messages back to the client
+    verbose-error-messages = on
+  }
+
+  parsing {
+    # Spray doesn't like the AWS4 headers
+    illegal-header-warnings = on
+  }
+}

build/docker/api/logback.xml

@@ -0,0 +1,32 @@
+<configuration>
+    <!-- Test configuration, log to console so we can get the docker logs -->
+    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
+        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
+            <pattern>%d [test] %-5p | \(%logger{4}:%line\) | %msg %n</pattern>
+        </encoder>
+    </appender>
+
+    <logger name="org.flywaydb" level="DEBUG">
+        <appender-ref ref="CONSOLE"/>
+    </logger>
+
+    <logger name="org.flywaydb.core.internal.dbsupport.SqlScript" level="DEBUG">
+        <appender-ref ref="CONSOLE"/>
+    </logger>
+
+    <logger name="org.flywaydb.core.internal.command.DbMigrate" level="DEBUG">
+        <appender-ref ref="CONSOLE"/>
+    </logger>
+
+    <logger name="vinyldns.core.route.Monitor" level="OFF">
+        <appender-ref ref="CONSOLE"/>
+    </logger>
+
+    <logger name="scalikejdbc.StatementExecutor$$anon$1" level="OFF">
+        <appender-ref ref="CONSOLE"/>
+    </logger>
+
+    <root level="INFO">
+        <appender-ref ref="CONSOLE"/>
+    </root>
+</configuration>

build/docker/docker-compose.yml

@@ -0,0 +1,77 @@
+version: "3.0"
+services:
+  mysql:
+    image: "mysql:5.7"
+    container_name: "vinyldns-mysql"
+    environment:
+      MYSQL_ROOT_PASSWORD: 'pass'
+      MYSQL_ROOT_HOST: '%'
+    logging:
+      driver: none
+    ports:
+      - "19002:3306"
+
+  bind9:
+    build:
+      context: ./test-bind9
+      args:
+          BRANCH: master
+    image: "vinyldns/test-bind9:${VINYLDNS_VERSION}"
+    container_name: "vinyldns-bind9"
+    logging:
+      driver: none
+    ports:
+      - "19001:53/udp"
+      - "19001:53"
+
+  api:
+    build:
+      context: ./api
+    image: "vinyldns/api:${VINYLDNS_VERSION}"
+    container_name: "vinyldns-api"
+    environment:
+      MYSQL_ROOT_PASSWORD: 'pass'
+      MYSQL_ROOT_HOST: '%'
+    logging:
+      driver: none
+    ports:
+      - "9000:9000"
+    volumes:
+      - ./api/application.conf:/opt/docker/conf/application.conf
+      - ./api/logback.xml:/opt/docker/conf/logback.xml
+    depends_on:
+      - mysql
+
+  ldap:
+    image: rroemhild/test-openldap
+    container_name: "vinyldns-ldap"
+    ports:
+      - "19008:389"
+
+  portal:
+    build:
+      context: ./portal
+    image: "vinyldns/portal:${VINYLDNS_VERSION}"
+    container_name: "vinyldns-portal"
+    environment:
+      MYSQL_ROOT_PASSWORD: 'pass'
+      MYSQL_ROOT_HOST: '%'
+    logging:
+      driver: none
+    ports:
+      - "9001:9000"
+    volumes:
+      - ./portal/application.conf:/opt/docker/conf/application.conf
+    depends_on:
+      - api
+      - ldap
+
+  functest:
+    build:
+      context: ./test
+    image: "vinyldns/test:${VINYLDNS_VERSION}"
+    environment:
+      TEST_PATTERN: "test_verify_production"
+    container_name: "vinyldns-functest"
+    depends_on:
+      - api

build/docker/portal/.jvmopts

@@ -0,0 +1,6 @@
+-Xms512M
+-Xmx1024M
+-Xss2M
+-XX:MaxMetaspaceSize=512M
+-XX:ReservedCodeCacheSize=512M
+-Djava.net.preferIPv4Stack=true

build/docker/portal/Dockerfile

@@ -0,0 +1,44 @@
+FROM hseeberger/scala-sbt:8u212_1.2.8_2.12.9 as builder
+
+ARG BRANCH=master
+ARG VINYLDNS_VERSION
+
+RUN git clone -b ${BRANCH} --single-branch --depth 1 https://github.com/vinyldns/vinyldns.git /vinyldns
+
+# The default jvmopts are huge, meant for running everything, use a paired down version
+COPY .jvmopts /vinyldns
+
+# Needed for preparePortal
+RUN apt-get update \
+    && apt-get install -y \
+    apt-transport-https \
+    curl \
+    gnupg \
+    && curl -sL https://deb.nodesource.com/setup_6.x | bash - \
+    && apt-get install -y nodejs \
+    && npm install -g grunt-cli
+
+RUN cd /vinyldns ; sbt "set version in ThisBuild := \"${VINYLDNS_VERSION}\"" portal/preparePortal universal:packageZipTarball
+
+FROM openjdk:8u201-jdk-alpine3.9
+
+RUN apk add --update --no-cache netcat-openbsd bash
+
+COPY --from=builder /vinyldns/modules/portal/target/universal/portal.tgz /
+
+RUN mkdir -p /opt && \
+    tar -xzvf /portal.tgz && \
+    mv /portal /opt/docker && \
+    mkdir -p /opt/docker/lib_extra
+
+# This will set the vinyldns version, make sure to have this in config... version = ${?VINYLDNS_VERSION}
+ARG VINYLDNS_VERSION
+ENV VINYLDNS_VERSION=$VINYLDNS_VERSION
+
+# Mount the volume for config file and lib extras
+# Note: These volume names are used in the build.sbt
+VOLUME ["/opt/docker/lib_extra/", "/opt/docker/conf"]
+
+EXPOSE 9000
+
+ENTRYPOINT ["/opt/docker/bin/portal"]

build/docker/portal/application.conf

@@ -0,0 +1,83 @@
+LDAP {
+  # For OpenLDAP, this would be a full DN to the admin for LDAP / user that can see all users
+  user = "cn=admin,dc=planetexpress,dc=com"
+
+  # Password for the admin account
+  password = "GoodNewsEveryone"
+
+  # Keep this as an empty string for OpenLDAP
+  domain = ""
+
+  # This will be the name of the LDAP field that carries the user's login id (what they enter in the username in login form)
+  userNameAttribute = "uid"
+
+  # For ogranization, leave empty for this demo, the domainName is what matters, and that is the LDAP structure
+  # to search for users that require login
+  searchBase = [
+    {organization = "", domainName = "ou=people,dc=planetexpress,dc=com"},
+  ]
+  context {
+    initialContextFactory = "com.sun.jndi.ldap.LdapCtxFactory"
+    securityAuthentication = "simple"
+
+    # Note: The following assumes a purely docker setup, using container_name = vinyldns-ldap
+    providerUrl = "ldap://vinyldns-ldap:389"
+  }
+
+  # This is only needed if keeping vinyldns user store in sync with ldap (to auto lock out users who left your
+  # company for example)
+  user-sync {
+    enabled = false
+    hours-polling-interval = 1
+  }
+}
+
+# Note: This MUST match the API or strange errors will ensure, NoCrypto should not be used for production
+crypto {
+  type = "vinyldns.core.crypto.NoOpCrypto"
+}
+
+http.port = 9000
+
+data-stores = ["mysql"]
+
+portal.vinyldns.backend.url = "http://vinyldns-api:9000"
+
+# Note: The default mysql settings assume a local docker compose setup with mysql named vinyldns-mysql
+# follow the configuration guide to point to your mysql
+# Only 3 repositories are needed for portal: user, task, user-change
+mysql {
+  settings {
+    # JDBC Settings, these are all values in scalikejdbc-config, not our own
+    # these must be overridden to use MYSQL for production use
+    # assumes a docker or mysql instance running locally
+    name = "vinyldns"
+    driver = "org.mariadb.jdbc.Driver"
+    migration-url = "jdbc:mariadb://vinyldns-mysql:3306/?user=root&password=pass"
+    url = "jdbc:mariadb://vinyldns-mysql:3306/vinyldns?user=root&password=pass"
+    user = "root"
+    password = "pass"
+    # see https://github.com/brettwooldridge/HikariCP
+    connection-timeout-millis = 1000
+    idle-timeout = 10000
+    max-lifetime = 600000
+    maximum-pool-size = 20
+    minimum-idle = 20
+    register-mbeans = true
+  }
+
+  repositories {
+    user {
+    }
+    task {
+    }
+    user-change {
+    }
+  }
+}
+
+# You generate this yourself following https://www.playframework.com/documentation/2.7.x/ApplicationSecret
+play.http.secret.key = "rpkTGtoJvLIdIV?WU=0@yW^x:pcEGyAt`^p/P3G0fpbj9:uDnD@caSjCDqA0@tB="
+
+vinyldns.version = "unknown"
+vinyldns.version = ${?VINYLDNS_VERSION}

build/docker/test-bind9/Dockerfile

@@ -0,0 +1,11 @@
+FROM alpine/git:1.0.7 as gitcheckout
+
+ARG BRANCH=master
+
+RUN git clone -b ${BRANCH} --single-branch --depth 1 https://github.com/vinyldns/vinyldns.git /vinyldns
+
+FROM vinyldns/bind9:0.0.5
+
+COPY --from=gitcheckout /vinyldns/docker/bind9/zones/* /var/cache/bind/zones/
+
+COPY --from=gitcheckout /vinyldns/docker/bind9/etc/named.conf.local /var/cache/bind/config

build/docker/test/Dockerfile

@@ -0,0 +1,28 @@
+FROM alpine/git:1.0.7 as gitcheckout
+
+ARG BRANCH=master
+
+RUN git clone -b ${BRANCH} --single-branch --depth 1 https://github.com/vinyldns/vinyldns.git /vinyldns
+
+FROM python:2.7.16-alpine3.9
+
+RUN apk add --update --no-cache bind-tools netcat-openbsd bash curl
+
+# The run script is what actually runs our func tests
+COPY run.sh /app/run.sh
+COPY run-tests.py /app/run-tests.py
+
+RUN chmod a+x /app/run.sh && chmod a+x /app/run-tests.py
+
+# Copy over the functional test directory, this must have been copied into the build context previous to this building!
+COPY --from=gitcheckout /vinyldns/modules/api/functional_test/ /app/
+
+# Install our func test requirements
+RUN pip install --index-url https://pypi.python.org/simple/ -r /app/requirements.txt
+
+ENV VINYLDNS_URL=""
+ENV DNS_IP=""
+ENV TEST_PATTERN="test_verify_production"
+
+# set the entry point for the container to start vinyl, specify the config resource
+ENTRYPOINT ["/app/run.sh"]

build/docker/test/run-tests.py

@@ -0,0 +1,16 @@
+#!/usr/bin/env python
+import os
+import sys
+
+basedir = os.path.dirname(os.path.realpath(__file__))
+
+report_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), '../target/pytest_reports')
+if not os.path.exists(report_dir):
+    os.system('mkdir -p ' + report_dir)
+
+import pytest
+
+result = 1
+result = pytest.main(list(sys.argv[1:]))
+
+sys.exit(result)

build/docker/test/run.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+
+# Assume defaults of local docker-compose if not set
+if [ -z "${VINYLDNS_URL}" ]; then
+  VINYLDNS_URL="http://vinyldns-api:9000"
+fi
+if [ -z "${DNS_IP}" ]; then
+  DNS_IP=$(dig +short vinyldns-bind9)
+fi
+
+# Assume all tests if not specified
+if [ -z "${TEST_PATTERN}" ]; then
+  TEST_PATTERN=
+else
+  TEST_PATTERN="-k ${TEST_PATTERN}"
+fi
+
+echo "Waiting for API to be ready at ${VINYLDNS_URL} ..."
+DATA=""
+RETRY=60
+SLEEP_DURATION=1
+while [ "$RETRY" -gt 0 ]
+do
+    DATA=$(curl -I -s "${VINYLDNS_URL}/ping" -o /dev/null -w "%{http_code}")
+    if [ $? -eq 0 ]
+    then
+        break
+    else
+        echo "Retrying" >&2
+
+        let RETRY-=1
+        sleep "$SLEEP_DURATION"
+
+        if [ "$RETRY" -eq 0 ]
+        then
+          echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
+          exit 1
+        fi
+    fi
+done
+
+echo "Running live tests against ${VINYLDNS_URL} and DNS server ${DNS_IP}"
+
+cd /app
+
+# Cleanup any errant cached file copies
+find . -name "*.pyc" -delete
+find . -name "__pycache__" -delete
+
+ls -al
+
+# -m plays havoc with -k, using variables is a headache, so doing this by hand
+# run parallel tests first (not serial)
+set -x
+./run-tests.py live_tests -n2 -v -m "not skip_production and not serial" ${TEST_PATTERN} --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} --teardown=False
+ret1=$?
+
+# IMPORTANT! pytest exists status code 5 if no tests are run, force that to 0
+if [ "$ret1" = 5 ]; then
+  echo "No tests collected."
+  ret1=0
+fi
+
+./run-tests.py live_tests -n0 -v -m "not skip_production and serial" ${TEST_PATTERN} --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} --teardown=True
+ret2=$?
+if [ "$ret2" = 5 ]; then
+  echo "No tests collected."
+  ret2=0
+fi
+
+if [ $ret1 -ne 0 ] || [ $ret2 -ne 0 ]; then
+  exit 1
+else
+  exit 0
+fi
+

build/release.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+
+CURDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+function usage() {
+  printf "usage: release.sh [OPTIONS]\n\n"
+  printf "builds and releases vinyldns artifacts\n\n"
+  printf "options:\n"
+  printf "\t-b, --branch: the branch of tag to use for the build; default is master\n"
+  printf "\t-c, --clean: indicates a fresh build or attempt to work with existing images; default is off\n"
+  printf "\t-p, --push: indicates docker will push to the repository; default is off\n"
+  printf "\t-r, --repository [REPOSITORY]: the docker repository where this image will be pushed; default is docker.io\n"
+  printf "\t-t, --tag [TAG]: sets the qualifier for the semver version; default is -SNAPSHOT\n"
+}
+
+# Default the build to -SNAPSHOT if not set
+BUILD_TAG="-SNAPSHOT"
+REPOSITORY="docker.io"
+DOCKER_PUSH=0
+DO_BUILD=0
+BRANCH="master"
+
+while [ "$1" != "" ]; do
+  case "$1" in
+  -b | --branch)
+    BRANCH="$2"
+    shift 2
+    ;;
+  -c | --clean)
+    DO_BUILD=1
+    shift
+    ;;
+  -p | --push)
+    DOCKER_PUSH=1
+    shift
+    ;;
+  -r | --repository)
+    REPOSITORY="$2"
+    shift 2
+    ;;
+  -t | --tag)
+    BUILD_TAG="-b$2"
+    shift 2
+    ;;
+  *)
+    usage
+    exit
+    ;;
+  esac
+done
+
+BASEDIR=$CURDIR/../
+
+# Clear out our target
+rm -rf $CURDIR/target && mkdir -p $CURDIR/target
+
+# Download just the version.sbt file from the branch specified, we use this to calculate the version
+wget "https://raw.githubusercontent.com/vinyldns/vinyldns/${BRANCH}/version.sbt" -P "${CURDIR}/target"
+
+# Calculate the version by using version.sbt, this will pull out something like 0.9.4
+V=$(find $CURDIR/target -name "version.sbt" | head -n1 | xargs grep "[ \\t]*version in ThisBuild :=" | head -n1 | sed 's/.*"\(.*\)".*/\1/')
+echo "VERSION ON BRANCH ${BRANCH} IS ${V}"
+if [[ "$V" == *-SNAPSHOT ]]; then
+  export VINYLDNS_VERSION="${V%?????????}${BUILD_TAG}"
+else
+  export VINYLDNS_VERSION="$V${BUILD_TAG}"
+fi
+
+if [ $DO_BUILD -eq 1 ]; then
+  docker-compose -f $CURDIR/docker/docker-compose.yml build \
+    --no-cache \
+    --parallel \
+    --build-arg VINYLDNS_VERSION="${VINYLDNS_VERSION}" \
+    --build-arg BRANCH="${BRANCH}"
+
+  if [ $? -eq 0 ]; then
+    # Runs smoke tests to make sure the new images are sound
+    docker-compose -f $CURDIR/docker/docker-compose.yml --log-level ERROR up --exit-code-from functest
+  fi
+
+  if [ $? -eq 0 ]; then
+    docker tag vinyldns/test-bind9:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test-bind9:$VINYLDNS_VERSION
+    docker tag vinyldns/test:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test:$VINYLDNS_VERSION
+    docker tag vinyldns/api:$VINYLDNS_VERSION $REPOSITORY/vinyldns/api:$VINYLDNS_VERSION
+    docker tag vinyldns/portal:$VINYLDNS_VERSION $REPOSITORY/vinyldns/portal:$VINYLDNS_VERSION
+  fi
+fi
+
+if [ $DOCKER_PUSH -eq 1 ]; then
+  docker push $REPOSITORY/vinyldns/test-bind9:$VINYLDNS_VERSION
+  docker push $REPOSITORY/vinyldns/test:$VINYLDNS_VERSION
+  docker push $REPOSITORY/vinyldns/api:$VINYLDNS_VERSION
+  docker push $REPOSITORY/vinyldns/portal:$VINYLDNS_VERSION
+fi

build/start.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+CURDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+function usage() {
+  printf "usage: start.sh [OPTIONS]\n\n"
+  printf "starts a specific version of vinyldns\n\n"
+  printf "options:\n"
+  printf "\t-v, --version: the version to start up; required\n"
+}
+
+function wait_for_url() {
+  URL=$1
+  DATA=""
+  RETRY="60"
+  echo "pinging $URL ..."
+  while [ "$RETRY" -gt 0 ]; do
+    DATA=$(curl -I -s "${URL}" -o /dev/null -w "%{http_code}")
+    if [ $? -eq 0 ]; then
+      echo "Succeeded in connecting to ${URL}!"
+      break
+    else
+      echo "Retrying" >&2
+
+      let RETRY-=1
+      sleep 1
+
+      if [ "$RETRY" -eq 0 ]; then
+        echo "Exceeded retries waiting for ${URL} to be ready, failing"
+        exit 1
+      fi
+    fi
+  done
+}
+
+# Default the build to -SNAPSHOT if not set
+VINYLDNS_VERSION=
+
+while [ "$1" != "" ]; do
+  case "$1" in
+  -v | --version)
+    VINYLDNS_VERSION="$2"
+    shift 2
+    ;;
+  *)
+    usage
+    exit
+    ;;
+  esac
+done
+
+if [ -z "$VINYLDNS_VERSION" ]; then
+  echo "VINYLDNS_VERSION not set"
+  usage
+  exit
+else
+  export VINYLDNS_VERSION=$VINYLDNS_VERSION
+fi
+
+# Actually starts up our docker images
+docker-compose -f $CURDIR/docker/docker-compose.yml up --no-build -d api portal
+
+# Waits for the URL to be available
+wait_for_url "http://localhost:9001"
+
+if [ $? -eq 0 ]; then
+  echo "VinylDNS started and available at http://localhost:9001"
+  exit 0
+else
+  echo "VinylDNS startup failed!"
+  $CURDIR/stop.sh
+  exit 1
+fi

build/stop.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+CURDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+docker-compose -f $CURDIR/docker/docker-compose.yml down

docker/docker-compose-quick-start.yml

@@ -3,7 +3,7 @@ services:
   mysql:
     image: "mysql:5.7"
     env_file:
-      .env.build
+      .env.quickstart
     container_name: "vinyldns-mysql"
     ports:
       - "19002:3306"
@@ -11,7 +11,7 @@ services:
   bind9:
     image: "vinyldns/bind9:0.0.4"
     env_file:
-      .env.build
+      .env.quickstart
     container_name: "vinyldns-bind9"
     ports:
       - "19001:53/udp"
@@ -41,7 +41,7 @@ services:
   api:
     image: "vinyldns/api:${VINYLDNS_VERSION}"
     env_file:
-      .env.build
+      .env.quickstart
     container_name: "vinyldns-api"
     ports:
       - "${REST_PORT}:${REST_PORT}"
@@ -55,7 +55,7 @@ services:
   portal:
     image: "vinyldns/portal:${VINYLDNS_VERSION}"
     env_file:
-      .env.build
+      .env.quickstart
     ports:
       - "${PORTAL_PORT}:${PORTAL_PORT}"
     container_name: "vinyldns-portal"