Updates

- Remove old, unused scripts in `bin/`
- Remove old images from release
  - `test` and `test-bind` are no longer necessary. Test images are in a different repo now
- Remove Docker image creation from sbt build config - actual `Dockerfile` files are easier to deal with

- Update scripts in `bin/` to utilize new Docker images
- Update documentation for changes
- Update all Docker Compose and configuration to use exposed ports on the `integration` image (19001, 19002, etc) both inside the container and outside to make testing more consistent irrespective of method
- Update FlywayDB dependency to v8 to fix a weird logging bug that showed up during integration testing. See: https://github.com/flyway/flyway/issues/2270

- Add `test/api/integration` Docker container definition to be used for any integration testing

- Move `module/api/functional_test` to `test/api/functional` to centralize the "integration-type" external tests and testing utilities

- Move functional testing and integration image to the `test/` folder off of the root to reduce confusion with `bin/` and `docker/`

.gitignore

@@ -32,3 +32,7 @@ tmp.out
 project/metals.sbt
 .bsp
 docker/data
+**/.virtualenv
+**/.venv*
+**/*cache*
+

AUTHORS.md

@@ -22,6 +22,7 @@ Thank you! If you have contributed in any way, but do not see your name here, pl
 - Luke Cori
 - Jearvon Dharrie
 - Andrew Dunn
+- Ryan Emerle
 - David Grizzanti
 - Alejandro Guirao
 - Daniel Jin

DEVELOPER_GUIDE.md

@@ -1,13 +1,14 @@
 # Developer Guide
 
 ## Table of Contents
+
 - [Developer Requirements](#developer-requirements)
 - [Project Layout](#project-layout)
 - [Running VinylDNS Locally](#running-vinyldns-locally)
 - [Testing](#testing)
-- [Validating VinylDNS](#validating-vinyldns)
 
 ## Developer Requirements
+
 - Scala 2.12
 - sbt 1+
 - Java 8 (at least u162)
@@ -21,49 +22,54 @@
 Make sure that you have the requirements installed before proceeding.
 
 ## Project Layout
-[SYSTEM_DESIGN.md](SYSTEM_DESIGN.md) provides a high-level architectural overview of VinylDNS and interoperability of its components.
 
-The main codebase is a multi-module Scala project with multiple sub-modules.  To start working with the project,
+[SYSTEM_DESIGN.md](SYSTEM_DESIGN.md) provides a high-level architectural overview of VinylDNS and interoperability of
-from the root directory run `sbt`.  Most of the code can be found in the `modules` directory.
+its components.
-The following modules are present:
+
+The main codebase is a multi-module Scala project with multiple sub-modules. To start working with the project, from the
+root directory run `sbt`. Most of the code can be found in the `modules` directory. The following modules are present:
 
 * `root` - this is the parent project, if you run tasks here, it will run against all sub-modules
 * [`core`](#core): core modules that are used by both the API and portal, such as cryptography implementations.
-* [`api`](#api): the API is the main engine for all of VinylDNS.  This is the most active area of the codebase, as everything else typically just funnels through
+* [`api`](#api): the API is the main engine for all of VinylDNS. This is the most active area of the codebase, as
-the API.
+  everything else typically just funnels through the API.
-* [`portal`](#portal): The portal is a user interface wrapper around the API.  Most of the business rules, logic, and processing can be found in the API.  The
+* [`portal`](#portal): The portal is a user interface wrapper around the API. Most of the business rules, logic, and
-_only_ features in the portal not found in the API are creation of users and user authentication.
+  processing can be found in the API. The
+  _only_ features in the portal not found in the API are creation of users and user authentication.
 * [`docs`](#documentation): documentation for VinylDNS.
 
 ### Core
+
 Code that is used across multiple modules in the VinylDNS ecosystem live in `core`.
 
 #### Code Layout
+
 * `src/main` - the main source code
 * `src/test` - unit tests
 
 ### API
-The API is the RESTful API for interacting with VinylDNS.  The following technologies are used:
+
+The API is the RESTful API for interacting with VinylDNS. The following technologies are used:
 
 * [Akka HTTP](https://doc.akka.io/docs/akka-http/current/) - Used primarily for REST and HTTP calls.
 * [FS2](https://functional-streams-for-scala.github.io/fs2/) - Used for backend change processing off of message queues.
-FS2 has back-pressure built in, and gives us tools like throttling and concurrency.
+  FS2 has back-pressure built in, and gives us tools like throttling and concurrency.
 * [Cats Effect](https://typelevel.org/cats-effect/) - We are currently migrating away from `Future` as our primary type
-and towards cats effect IO.  Hopefully, one day, all the things will be using IO.
+  and towards cats effect IO. Hopefully, one day, all the things will be using IO.
 * [Cats](https://typelevel.org/cats) - Used for functional programming.
-* [PureConfig](https://pureconfig.github.io/) - For loading configuration values.  We are currently migrating to
+* [PureConfig](https://pureconfig.github.io/) - For loading configuration values. We are currently migrating to use
-use PureConfig everywhere.  Not all the places use it yet.
+  PureConfig everywhere. Not all the places use it yet.
 
 The API has the following dependencies:
-* MySQL - the SQL database that houses zone data
+
-* DynamoDB - where all of the other data is stored
+* MySQL - the SQL database that houses the data
 * SQS - for managing concurrent updates and enabling high-availability
 * Bind9 - for testing integration with a real DNS system
 
 #### Code Layout
+
 The API code can be found in `modules/api`.
 
-* `functional_test` - contains the python black box / regression tests
 * `src/it` - integration tests
 * `src/main` - the main source code
 * `src/test` - unit tests
@@ -71,27 +77,31 @@ The API code can be found in `modules/api`.
 
 The package structure for the source code follows:
 
-* `vinyldns.api.domain` - contains the core front-end logic.  This includes things like the application services,
+* `vinyldns.api.domain` - contains the core front-end logic. This includes things like the application services,
-repository interfaces, domain model, validations, and business rules.
+  repository interfaces, domain model, validations, and business rules.
-* `vinyldns.api.engine` - the back-end processing engine.  This is where we process commands including record changes,
+* `vinyldns.api.engine` - the back-end processing engine. This is where we process commands including record changes,
-zone changes, and zone syncs.
+  zone changes, and zone syncs.
 * `vinyldns.api.protobuf` - marshalling and unmarshalling to and from protobuf to types in our system
 * `vinyldns.api.repository` - repository implementations live here
 * `vinyldns.api.route` - HTTP endpoints
 
 ### Portal
+
 The project is built using:
+
 * [Play Framework](https://www.playframework.com/documentation/2.6.x/Home)
 * [AngularJS](https://angularjs.org/)
 
-The portal is _mostly_ a shim around the API.  Most actions in the user interface are translated into API calls.
+The portal is _mostly_ a shim around the API. Most actions in the user interface are translated into API calls.
 
 The features that the Portal provides that are not in the API include:
+
 * Authentication against LDAP
-* Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials for them in the
+* Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials
-database with their LDAP information.
+  for them in the database with their LDAP information.
 
 #### Code Layout
+
 The portal code can be found in `modules/portal`.
 
 * `app` - source code for portal back-end
@@ -108,38 +118,54 @@ The portal code can be found in `modules/portal`.
 * `test` - unit tests for portal back-end
 
 ### Documentation
-Code used to build the microsite content for the API, operator and portal guides at https://www.vinyldns.io/. Some settings for the microsite
+
-are also configured in `build.sbt` of the project root.
+Code used to build the microsite content for the API, operator and portal guides at https://www.vinyldns.io/. Some
+settings for the microsite are also configured in `build.sbt` of the project root.
 
 #### Code Layout
+
 * `src/main/resources` - Microsite resources and configurations
 * `src/main/tut` - Content for microsite web pages
 
 ## Running VinylDNS Locally
-VinylDNS can be started in the background by running the [quickstart instructions](README.md#quickstart) located in the README. However, VinylDNS
+
-can also be run in the foreground.
+VinylDNS can be started in the background by running the [quickstart instructions](README.md#quickstart) located in the
+README. However, VinylDNS can also be run in the foreground.
 
 ### Starting the API Server
-To start the API for integration, functional, or portal testing.  Start up sbt by running `sbt` from the root directory.
+
-* `dockerComposeUp` to spin up the dependencies on your machine from the root project.
+Before starting the API service, you can start the dependencies for local development:
+```
+cd test/api/integration
+make build && make run-bg
+```
+This will start a container running in the background with necessary prerequisites.
+
+Once the prerequisites are running, you can start up sbt by running `sbt` from the root directory.
+
 * `project api` to change the sbt project to the API
 * `reStart` to start up the API server
 * Wait until you see the message `VINYLDNS SERVER STARTED SUCCESSFULLY` before working with the server
 * To stop the VinylDNS server, run `reStop` from the api project
-* To stop the dependent Docker containers, change to the root project `project root`, then run `dockerComposeStop` from the API project
+* To stop the dependent Docker containers, change to the root project `project root`, then run `dockerComposeStop` from
+  the API project
 
-See the [API Configuration Guide](https://www.vinyldns.io/operator/config-api) for information regarding API configuration.
+See the [API Configuration Guide](https://www.vinyldns.io/operator/config-api) for information regarding API
+configuration.
 
 ### Starting the Portal
-To run the portal locally, you _first_ have to start up the VinylDNS API Server (see instructions above).  Once
-that is done, in the same `sbt` session or a different one, go to `project portal` and then execute `;preparePortal; run`.
 
-See the [Portal Configuration Guide](https://www.vinyldns.io/operator/config-portal) for information regarding portal configuration.
+To run the portal locally, you _first_ have to start up the VinylDNS API Server (see instructions above). Once that is
+done, in the same `sbt` session or a different one, go to `project portal` and then execute `;preparePortal; run`.
+
+See the [Portal Configuration Guide](https://www.vinyldns.io/operator/config-portal) for information regarding portal
+configuration.
 
 ### Loading test data
+
 Normally the portal can be used for all VinylDNS requests. Test users are locked down to only have access to test zones,
-which the portal connection modal has not been updated to incorporate. To connect to a zone with testuser, you will need to use an alternative
+which the portal connection modal has not been updated to incorporate. To connect to a zone with testuser, you will need
-client and set `isTest=true` on the zone being connected to.
+to use an alternative client and set `isTest=true` on the zone being connected to.
 
 Use the vinyldns-js client (Note, you need Node installed):
 
@@ -159,78 +185,95 @@ You should now be able to see the zone in the portal at localhost:9001 when logg
 ```
 
 ## Testing
+
 ### Unit Tests
-1. First, start up your Scala build tool: `sbt`.  Running *clean* immediately after starting is recommended.
+
-1. (Optionally) Go to the project you want to work on, for example `project api` for the API; `project portal` for the portal.
+1. First, start up your Scala build tool: `sbt`. Running *clean* immediately after starting is recommended.
+1. (Optionally) Go to the project you want to work on, for example `project api` for the API; `project portal` for the
+   portal.
 1. Run _all_ unit tests by just running `test`.
 1. Run an individual unit test by running `testOnly *MySpec`.
-1. If you are working on a unit test and production code at the same time, use `~` (eg. `~testOnly *MySpec`) to automatically background compile for you!
+1. If you are working on a unit test and production code at the same time, use `~` (e.g., `~testOnly *MySpec`) to
+   automatically background compile for you!
 
 ### Integration Tests
-Integration tests are used to test integration with _real_ dependent services.  We use Docker to spin up those
+
-backend services for integration test development.
+Integration tests are used to test integration with _real_ dependent services. We use Docker to spin up those backend
+services for integration test development.
 
 1. Type `dockerComposeUp` to start up dependent background services
 1. Go to the target module in sbt, example: `project api`
 1. Run all integration tests by typing `it:test`.
 1. Run an individual integration test by typing `it:testOnly *MyIntegrationSpec`
 1. You can background compile as well if working on a single spec by using `~it:testOnly *MyIntegrationSpec`
-1. You must stop (`dockerComposeStop`) and start (`dockerComposeUp`) the dependent services from the root project (`project root`) before you rerun the tests.
+1. You must stop (`dockerComposeStop`) and start (`dockerComposeUp`) the dependent services from the root
-1. For the mysql module, you may need to wait up to 30 seconds after starting the services before running the tests for setup to complete.
+   project (`project root`) before you rerun the tests.
+1. For the mysql module, you may need to wait up to 30 seconds after starting the services before running the tests for
+   setup to complete.
 
 #### Running both
 
 You can run all unit and integration tests for the api and portal by running `sbt verify`
 
 ### Functional Tests
-When adding new features, you will often need to write new functional tests that black box / regression test the
-API.  We have over 350 (and growing) automated regression tests.  The API functional tests are written in Python
-and live under `modules/api/functional_test`.
 
-#### Running functional tests
+When adding new features, you will often need to write new functional tests that black box / regression test the API.
-To run functional tests, make sure that you have started the API server (directions above).
-Then in another terminal session:
 
-1. `cd modules/api/functional_test`
+- The API functional tests are written in Python and live under `test/api/functional`.
-1. `./run.py live_tests -v`
+The Portal functional tests are written in JavaScript and live under `test/portal/functional`.
 
-You can run a specific test by name by running `./run.py live_tests -v -k <name of test function>`
+#### Running Functional Tests
 
-You run specific tests for a portion of the project, say recordsets, by running `./run.py live_tests/recordsets -v`
+To run functional tests you can simply execute the following command:
 
-#### Our Setup
+```
-We use [pytest](https://docs.pytest.org/en/latest/) for python tests.  It is helpful that you browse the documentation
+make build && make run
-so that you are familiar with pytest and how our functional tests operate.
+```
+During iterative test development, you can use `make run-local` which will mount the current functional tests in the
+container, allowing for easier test development.
 
-We also use [PyHamcrest](https://pyhamcrest.readthedocs.io/en/release-1.8/) for matchers in order to write easy
+Additionally, you can pass `--interactive` to `make run` or `make run-local` to drop to a shell inside the container.
-to read tests.  Please browse that documentation as well so that you are familiar with the different matchers
+From there you can run tests with the `/functional_test/run.sh` command. This allows for finer-grained control over the
-for PyHamcrest.  There aren't a lot, so it should be quick.
+test execution process as well as easier inspection of logs.
 
 
-In the `modules/api/functional_test` directory are a few important files for you to be familiar with:
+##### API Functional Tests
+You can run a specific test by name by running `make run -- -k <name of test function>`. Any arguments after
+`make run --` will be passed to the test runner [`test/api/functional/run.sh`](test/api/functional/run.sh).
 
-* vinyl_client.py - this provides the interface to the VinylDNS API.  It handles signing the request for you, as well
-as building and executing the requests, and giving you back valid responses.  For all new API endpoints, there should
-be a corresponding function in the vinyl_client
-* utils.py - provides general use functions that can be used anywhere in your tests.  Feel free to contribute new
-functions here when you see repetition in the code
 
-Functional tests run on every build, and are designed to work _in every environment_.  That means locally, in Docker,
-and in production environments.
 
-In the `modules/api/functional_test/live_tests` directory, we have directories / modules for different areas of the application.
+#### Setup
 
-* membership - for managing groups and users
+We use [pytest](https://docs.pytest.org/en/latest/) for python tests. It is helpful that you browse the documentation so
-* recordsets - for managing record sets
+that you are familiar with pytest and how our functional tests operate.
-* zones - for managing zones
+
-* internal - for internal endpoints (not intended for public consumption)
+We also use [PyHamcrest](https://pyhamcrest.readthedocs.io/en/release-1.8/) for matchers in order to write easy to read
-* batch - for managing batch updates
+tests. Please browse that documentation as well so that you are familiar with the different matchers for PyHamcrest.
+There aren't a lot, so it should be quick.
+
+In the `test/api/functional` directory are a few important files for you to be familiar with:
+
+* `vinyl_client.py` - this provides the interface to the VinylDNS API. It handles signing the request for you, as well
+  as building and executing the requests, and giving you back valid responses. For all new API endpoints, there should
+  be a corresponding function in the vinyl_client
+* `utils.py` - provides general use functions that can be used anywhere in your tests. Feel free to contribute new
+  functions here when you see repetition in the code
+
+In the `test/api/functional/tests` directory, we have directories / modules for different areas of the application.
+
+* `batch` - for managing batch updates
+* `internal` - for internal endpoints (not intended for public consumption)
+* `membership` - for managing groups and users
+* `recordsets` - for managing record sets
+* `zones` - for managing zones
 
 ##### Functional Test Context
-Our func tests use pytest contexts.  There is a main test context that lives in `shared_zone_test_context.py`
+
-that creates and tears down a shared test context used by many functional tests.  The
+Our functional tests use `pytest` contexts. There is a main test context that lives in `shared_zone_test_context.py`
-beauty of pytest is that it will ensure that the test context is stood up exactly once, then all individual tests
+that creates and tears down a shared test context used by many functional tests. The beauty of pytest is that it will
-that use the context are called using that same context.
+ensure that the test context is stood up exactly once, then all individual tests that use the context are called using
+that same context.
 
 The shared test context sets up several things that can be reused:
 
@@ -243,30 +286,33 @@ The shared test context sets up several things that can be reused:
 1. A classless IPv4 reverse zone
 1. A parent zone that has child zones - used for testing NS record management and zone delegations
 
+##### Partitioning
+
+Each of the test zones are configured in a `partition`. By default, there are four partitions. These partitions are
+effectively copies of the zones so that parallel tests can run without interfering with one another.
+
+For instance, there are four zones for the `ok` zone: `ok1`, `ok2`, `ok3`, and `ok4`. The functional tests will handle
+distributing which zone is being used by which of the parallel test runners.
+
+As such, you should **never** hardcode the name of the zone. Always get the zone from the `shared_zone_test_context`.
+For instance, to get the `ok` zone, you would write:
+
+```python
+zone = shared_zone_test_context.ok_zone
+zone_name = shared_zone_test_context.ok_zone["name"]
+zone_id = shared_zone_test_context.ok_zone["id"]
+```
+
 ##### Really Important Test Context Rules!
 
-1. Try to use the `shared_zone_test_context` whenever possible!  This reduces the time
+1. Try to use the `shared_zone_test_context` whenever possible!  This reduces the time it takes to run functional
-it takes to run functional tests (which is in minutes).
+   tests (which is in minutes).
-1. Limit changes to users, groups, and zones in the shared test context, as doing so could impact downstream tests
+1. Be mindful of changes to users, groups, and zones in the shared test context, as doing so could impact downstream
+   tests
 1. If you do modify any entities in the shared zone context, roll those back when your function completes!
 
 ##### Managing Test Zone Files
-When functional tests are run, we spin up several Docker containers.  One of the Docker containers is a Bind9 DNS
+
-server.  If you need to add or modify the test DNS zone files, you can find them in
+When functional tests are run, we spin up several Docker containers. One of the Docker containers is a Bind9 DNS server.
+If you need to add or modify the test DNS zone files, you can find them in
 `docker/bind9/zones`
-
-## Validating VinylDNS
-VinylDNS comes with a build script `./build.sh` that validates VinylDNS compiles, verifies that unit tests pass, and then runs functional tests.
-Note: This takes a while to run, and typically is only necessary if you want to simulate the same process that runs on the build servers.
-
-When functional tests run, you will see a lot of output intermingled together across the various containers.  You can view only the output
-of the functional tests at `target/vinyldns-functest.log`.  If you want to see the Docker log output from any one container, you can view
-them after the tests complete at:
-
-* `target/vinyldns-api.log` - the API server logs
-* `target/vinyldns-bind9.log` - the Bind9 DNS server logs
-* `target/vinyldns-elasticmq.log` - the ElasticMQ (SQS) server logs
-* `target/vinyldns-functest.log` - the output of running the functional tests
-* `target/vinyldns-mysql.log` - the MySQL server logs
-
-When the func tests complete, the entire Docker setup will be automatically torn down.

MAINTAINERS.md

@@ -2,12 +2,6 @@
 
 ## Table of Contents
 * [Docker Content Trust](#docker-content-trust)
-    * [Docker Hub Account](#docker-hub-account)
-    * [Delegating Image Signing](#delegating-image-signing)
-        * [Setting up Notary](#setting-up-notary)
-        * [Generating a Personal Delegation Key](#generating-a-personal-delegation-key)
-        * [Adding a Delegation Key To a Repository](#adding-a-delegation-key-to-a-repository)
-        * [Pushing a Signed Image with your Delegation Key](#pushing-a-signed-image-with-your-delegation-key)
 * [Sonatype Credentials](#sonatype-credentials)
 * [Release Process](#release-process)
 
@@ -26,8 +20,6 @@ the [vinyldns organization](https://hub.docker.com/u/vinyldns/dashboard/). Namel
 * vinyldns/api: images for vinyldns core api engine 
 * vinyldns/portal: images for vinyldns web client
 * vinyldns/bind9: images for local DNS server used for testing 
-* vinyldns/test-bind9: contains the setup to run functional tests 
-* vinyldns/test: has the actual functional tests pinned to a version of VinylDNS 
 
 The offline root key and repository keys are managed by the core maintainer team. The keys managed are:
 
@@ -35,8 +27,6 @@ The offline root key and repository keys are managed by the core maintainer team
 * api key: used to sign tagged images in vinyldns/api
 * portal key: used to sign tagged images in vinyldns/portal
 * bind9 key: used to sign tagged images in the vinyldns/bind9
-* test-bind9 key: used to sign tagged images in the vinyldns/test-bind9
-* test key: used to sign tagged images in the vinyldns/test
 
 These keys are named in a <hash>.key format, e.g. 5526ecd15bd413e08718e66c440d17a28968d5cd2922b59a17510da802ca6572.key,
 do not change the names of the keys. 
@@ -44,77 +34,6 @@ do not change the names of the keys.
 Docker expects these keys to be saved in `~/.docker/trust/private`. Each key is encrypted with a passphrase, that you 
 must have available when pushing an image.
 
-### Docker Hub Account
-
-If you don't have one already, make an account on Docker Hub. Get added as a Collaborator to vinyldns/api, vinyldns/portal,
-and vinyldns/bind9
-
-### Delegating Image Signing
-Someone with our keys can sign images when pushing, but instead of sharing those keys we can utilize 
-notary to delegate image signing permissions in a safer way. Notary will have you make a public-private key pair and 
-upload your public key. This way you only need your private key, and a developer's permissions can easily be revoked. 
-
-#### Setting up Notary
-If you do not already have notary: 
-
-1. Download the latest release for your machine at https://github.com/theupdateframework/notary/releases, 
-for example, on a mac download the precompiled binary `notary-Darwin-amd64`
-1. Rename the binary to notary, and choose a location where it will live,
-e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/notary; cd ~/Documents`
-1. Make it executable, e.g. `chmod +x notary`
-1. Add notary to your path, e.g. `vim ~/.bashrc`, add `export PATH="$PATH":<path to notary>`
-1. Create a `~/.notary/config.json` with
- 
-``` 
-{
-   "trust_dir" : "~/.docker/trust",
-   "remote_server": {
-     "url": "https://notary.docker.io"
-   }
- }
-```
-
-You can test notary with `notary -s https://notary.docker.io -d ~/.docker/trust list docker.io/vinyldns/api`, in which
-you should see tagged images for the VinylDNS API
-
-> Note: you'll pretty much always use the `-s https://notary.docker.io -d ~/.docker/trust` args when running notary,
-it will be easier for you to alias a command like `notarydefault` to `notary -s https://notary.docker.io -d ~/.docker/trust`
-in your `.bashrc` 
-
-#### Generating a Personal Delegation Key
-1. `cd` to a directory where you will save your delegation keys and certs
-1. Generate your private key: `openssl genrsa -out delegation.key 2048`
-1. Generate your public key: `openssl req -new -sha256 -key delegation.key -out delegation.csr`, all fields are optional,
-but when it gets to your email it makes sense to add that
-1. Self-sign your public key (valid for one year): 
-`openssl x509 -req -sha256 -days 365 -in delegation.csr -signkey delegation.key -out delegation.crt`
-1. Change the `delegation.crt` to some unique name, like `my-name-vinyldns-delegation.crt`
-1. Give your `my-name-vinyldns-delegation.crt` to someone that has the root keys and passphrases so 
-they can upload your delegation key to the repository
-
-#### Adding a Delegation Key to a Repository
-This expects you to have the root keys and passphrases for the Docker repositories
-
-1. List current keys: `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`
-1. Add team member's public key: `notary delegation add docker.io/vinyldns/api targets/releases <team members delegation crt path> --all-paths`
-1. Push key: `notary publish docker.io/vinyldns/api`
-1. Repeat above steps for `docker.io/vinyldns/portal`
-
-Add their key ID to the table below, it can be viewed with `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`.
-It will be the one that didn't show up when you ran step one of this section
-
-| Name | Key ID |
-|----------------|------------------------------------------------------------------
-| Nima Eskandary | 66027c822d68133da859f6639983d6d3d9643226b3f7259fc6420964993b499a, cdca33de91c54f801d89240d18b5037e274461ba1c88c10451070c97e9f665b4 |
-| Rebecca Star | 04285e24d3b9a8b614b34da229669de1f75c9faa471057e8b4a7d60aac0d5bf5 |
-| Michael Ly |dd3a5938fc927de087ad4b59d6ac8f62b6502d05b2cc9b0623276cbac7dbf05b |
-
-#### Pushing a Signed Image with your Delegation Key
-1. Run `notary key import <path to private delegation key> --role user`
-1. You will have to create a passphrase for this key that encrypts it at rest. Use a password generator to make a 
-strong password and save it somewhere safe, like apple keychain or some other password manager
-1. From now on `docker push` will be try to sign images with the delegation key if it was configured for that Docker 
-repository
 
 ## Sonatype Credentials
 
@@ -163,7 +82,7 @@ running the release
 1. Run `bin/release.sh` _Note: the arg "skip-tests" will skip unit, integration and functional testing before a release_
 1. You will be asked to confirm the version which originally comes from `version.sbt`. _NOTE: if the version ends with 
 `SNAPSHOT`, then the docker latest tag won't be applied and the core module will only be published to the sonatype
-staging repo.
+staging repo._
 1. When it comes to the sonatype stage, you will need the passphrase handy for the signing key, [Sonatype Credentials](#sonatype-credentials)
 1. Assuming things were successful, make a pr since sbt release auto-bumped `version.sbt` and made a commit for you
 1. Run `./build/docker-release.sh --branch [TAG CREATED FROM PREVIOUS STEP, e.g. v0.9.3] --clean --push`

README.md

@@ -1,7 +1,5 @@
-[![Join the chat at https://gitter.im/vinyldns](https://badges.gitter.im/vinyldns/vinyldns.svg)](https://gitter.im/vinyldns)
 ![Build](https://github.com/vinyldns/vinyldns/workflows/Continuous%20Integration/badge.svg)
 [![CodeCov ](https://codecov.io/gh/vinyldns/vinyldns/branch/master/graph/badge.svg)](https://codecov.io/gh/vinyldns/vinyldns)
-[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/2682/badge)](https://bestpractices.coreinfrastructure.org/projects/2682)
 [![License](https://img.shields.io/github/license/vinyldns/vinyldns)](https://github.com/vinyldns/vinyldns/blob/master/LICENSE)
 [![conduct](https://img.shields.io/badge/%E2%9D%A4-code%20of%20conduct-blue.svg)](https://github.com/vinyldns/vinyldns/blob/master/CODE_OF_CONDUCT.md)
 
@@ -23,17 +21,16 @@ secure RESTful API, and integration with infrastructure automation tools like An
 It is designed to integrate with your existing DNS infrastructure, and provides extensibility to fit your installation.
 
 VinylDNS helps secure DNS management via:
-* AWS Sig4 signing of all messages to ensure that the message that was sent was not altered in transit
+- AWS Sig4 signing of all messages to ensure that the message that was sent was not altered in transit
-* Throttling of DNS updates to rate limit concurrent updates against your DNS systems
+- Throttling of DNS updates to rate limit concurrent updates against your DNS systems
-* Encrypting user secrets and TSIG keys at rest and in-transit
+- Encrypting user secrets and TSIG keys at rest and in-transit
-* Recording every change made to DNS records and zones
+- Recording every change made to DNS records and zones
 
 Integration is simple with first-class language support including:
-* java
+- Java
-* ruby
+- Python
-* python
+- Go
-* go-lang
+- JavaScript
-* javascript
 
 ## Table of Contents
 - [Quickstart](#quickstart)
@@ -59,7 +56,7 @@ There exist several clients at <https://github.com/vinyldns> that can be used to
 
 ## Things to try in the portal
 1. View the portal at <http://localhost:9001> in a web browser
-1. Login with the credentials ***professor*** and ***professor***
+1. Login with the credentials `testuser` and `testpassword`
 1. Navigate to the `groups` tab: <http://localhost:9001/groups>
 1. Click on the **New Group** button and create a new group, the group id is the uuid in the url after you view the group
 1. View zones you connected to in the `zones` tab: <http://localhost:9001/zones>.  For a quick test, create a new zone named "ok" with an email of "test@test.com" and choose a group you created from the previous step. (Note, see [Developer Guide](DEVELOPER_GUIDE.md#loading-test-data) for creating a zone)
@@ -79,7 +76,7 @@ TTL = 300, IP Addressess = 1.1.1.1`
 1. A similar `docker/.env.quickstart` can be modified to change the default ports for the Portal and API. You must also modify their config files with the new port: https://www.vinyldns.io/operator/config-portal & https://www.vinyldns.io/operator/config-api
 
 ## Code of Conduct
-This project and everyone participating in it are governed by the [VinylDNS Code Of Conduct](CODE_OF_CONDUCT.md).  By
+This project, and everyone participating in it, are governed by the [VinylDNS Code Of Conduct](CODE_OF_CONDUCT.md).  By
 participating, you agree to this Code.  Please report any violations to the code of conduct to vinyldns-core@googlegroups.com.
 
 ## Developer Guide
@@ -89,15 +86,14 @@ See [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md) for instructions on setting up Viny
 See the [Contributing Guide](CONTRIBUTING.md).
 
 ## Contact
-- [Gitter](https://gitter.im/vinyldns)
 - If you have any security concerns please contact the maintainers directly vinyldns-core@googlegroups.com
 
 ## Maintainers and Contributors
 The current maintainers (people who can merge pull requests) are:
-- Paul Cleary
+
-- Ryan Emerle
+- Ryan Emerle ([@remerle](https://github.com/remerle))
-- Sriram Ramakrishnan
+- Sriram Ramakrishnan ([@sramakr](https://github.com/sramakr))
-- Jim Wakemen
+- Jim Wakemen ([@jwakemen](https://github.com/jwakemen))
 
 See [AUTHORS.md](AUTHORS.md) for the full list of contributors to VinylDNS.
 

bin/build.sh

@@ -1,34 +0,0 @@
-#!/usr/bin/env bash
-DIR=$( cd $(dirname $0) ; pwd -P )
-
-echo "Verifying code..."
-#${DIR}/verify.sh
-
-#step_result=$?
-step_result=0
-if  [ ${step_result} != 0 ]
-then
-    echo "Failed to verify build!!!"
-    exit ${step_result}
-fi
-
-echo "Func testing the api..."
-${DIR}/func-test-api.sh
-
-step_result=$?
-if  [ ${step_result} != 0 ]
-then
-    echo "Failed API func tests!!!"
-    exit ${step_result}
-fi
-
-echo "Func testing the portal..."
-${DIR}/func-test-portal.sh
-step_result=$?
-if  [ ${step_result} != 0 ]
-then
-    echo "Failed Portal func tests!!!"
-    exit ${step_result}
-fi
-
-exit 0

bin/docker-publish-api.sh

@@ -1,10 +0,0 @@
-#!/usr/bin/env bash
-DIR=$( cd $(dirname $0) ; pwd -P )
-
-cd $DIR/../
-
-echo "Publishing docker image..."
-sbt clean docker:publish
-publish_result=$?
-cd $DIR
-exit ${publish_result}

bin/docker-up-api-server.sh

@@ -1,58 +0,0 @@
-#!/bin/bash
-######################################################################
-# Copies the contents of `docker` into target/scala-2.12
-# to start up dependent services via docker compose.  Once
-# dependent services are started up, the fat jar built by sbt assembly
-# is loaded into a docker container.  The api will be available
-# by default on port 9000
-######################################################################
-
-
-DIR=$( cd $(dirname $0) ; pwd -P )
-
-set -a # Required in order to source docker/.env
-# Source customizable env files
-source "$DIR"/.env
-source "$DIR"/../docker/.env
-
-WORK_DIR="$DIR"/../target/scala-2.12
-mkdir -p "$WORK_DIR"
-
-echo "Copy all Docker to the target directory so we can start up properly and the Docker context is small..."
-cp -af "$DIR"/../docker "$WORK_DIR"/
-
-echo "Copy the vinyldns.jar to the API Docker folder so it is in context..."
-if [[ ! -f "$DIR"/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
-    echo "vinyldns.jar not found, building..."
-    cd "$DIR"/../
-    sbt api/clean api/assembly
-    cd "$DIR"
-fi
-cp -f "$DIR"/../modules/api/target/scala-2.12/vinyldns.jar "$WORK_DIR"/docker/api
-
-echo "Starting API server and all dependencies in the background..."
-docker-compose -f "$WORK_DIR"/docker/docker-compose-func-test.yml --project-directory "$WORK_DIR"/docker up --build -d api
-
-echo "Waiting for API to be ready at ${VINYLDNS_API_URL} ..."
-DATA=""
-RETRY=40
-while [ "$RETRY" -gt 0 ]
-do
-    DATA=$(curl -I -s "${VINYLDNS_API_URL}/ping" -o /dev/null -w "%{http_code}")
-    if [ $? -eq 0 ]
-    then
-        echo "Succeeded in connecting to VinylDNS API!"
-        break
-    else
-        echo "Retrying" >&2
-
-        let RETRY-=1
-        sleep 1
-
-        if [ "$RETRY" -eq 0 ]
-        then
-          echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
-          exit 1
-        fi
-    fi
-done

bin/docker-up-dns-server.sh

@@ -1,5 +0,0 @@
-#!/bin/bash
-DIR=$( cd $(dirname $0) ; pwd -P )
-
-echo "Starting ONLY the bind9 server.  To start an api server use the api server script"
-docker-compose -f $DIR/../docker/docker-compose-func-test.yml --project-directory $DIR/../docker up -d bind9

bin/func-test-api-testbind9.sh

@@ -1,57 +0,0 @@
-#!/bin/bash
-######################################################################
-# Copies the contents of `docker` into target/scala-2.12
-# to start up dependent services via docker compose.  Once
-# dependent services are started up, the fat jar built by sbt assembly
-# is loaded into a docker container.  Finally, the func tests run inside
-# another docker container
-# At the end, we grab all the logs and place them in the target
-# directory
-######################################################################
-
-DIR=$( cd $(dirname $0) ; pwd -P )
-WORK_DIR=$DIR/../target/scala-2.12
-mkdir -p $WORK_DIR
-
-echo "Cleaning up unused networks..."
-docker network prune -f
-
-echo "Copy all docker to the target directory so we can start up properly and the docker context is small..."
-cp -af $DIR/../docker $WORK_DIR/
-
-echo "Copy over the functional tests as well as those that are run in a container..."
-mkdir -p $WORK_DIR/functest
-rsync -av --exclude='.virtualenv' $DIR/../modules/api/functional_test $WORK_DIR/docker/functest
-
-echo "Copy the vinyldns.jar to the api docker folder so it is in context..."
-if [[ ! -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
-    echo "vinyldns jar not found, building..."
-    cd $DIR/../
-    sbt api/clean api/assembly
-    cd $DIR
-fi
-cp -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar $WORK_DIR/docker/api
-
-echo "Starting docker environment and running func tests..."
-
-# If PAR_CPU is unset; default to auto
-if [ -z "${PAR_CPU}" ]; then
-  export PAR_CPU=auto
-fi
-
-docker-compose -f $WORK_DIR/docker/docker-compose-func-test-testbind9.yml --project-directory $WORK_DIR/docker --log-level ERROR up --build --exit-code-from functest
-test_result=$?
-
-echo "Grabbing the logs..."
-
-docker logs vinyldns-api > $DIR/../target/vinyldns-api.log 2>/dev/null
-docker logs vinyldns-bind9 > $DIR/../target/vinyldns-bind9.log 2>/dev/null
-docker logs vinyldns-mysql > $DIR/../target/vinyldns-mysql.log 2>/dev/null
-docker logs vinyldns-elasticmq > $DIR/../target/vinyldns-elasticmq.log 2>/dev/null
-docker logs vinyldns-functest > $DIR/../target/vinyldns-functest.log 2>/dev/null
-
-echo "Cleaning up docker containers..."
-$DIR/./remove-vinyl-containers.sh
-
-echo "Func tests returned result: ${test_result}"
-exit ${test_result}

bin/func-test-api-travis.sh

@@ -1,57 +0,0 @@
-#!/bin/bash
-######################################################################
-# Copies the contents of `docker` into target/scala-2.12
-# to start up dependent services via docker compose.  Once
-# dependent services are started up, the fat jar built by sbt assembly
-# is loaded into a docker container.  Finally, the func tests run inside
-# another docker container
-# At the end, we grab all the logs and place them in the target
-# directory
-######################################################################
-
-DIR=$( cd $(dirname $0) ; pwd -P )
-WORK_DIR=$DIR/../target/scala-2.12
-mkdir -p $WORK_DIR
-
-echo "Cleaning up unused networks..."
-docker network prune -f
-
-echo "Copy all docker to the target directory so we can start up properly and the docker context is small..."
-cp -af $DIR/../docker $WORK_DIR/
-
-echo "Copy over the functional tests as well as those that are run in a container..."
-mkdir -p $WORK_DIR/functest
-rsync -av --exclude='.virtualenv' $DIR/../modules/api/functional_test $WORK_DIR/docker/functest
-
-echo "Copy the vinyldns.jar to the api docker folder so it is in context..."
-if [[ ! -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
-    echo "vinyldns jar not found, building..."
-    cd $DIR/../
-    sbt build-api
-    cd $DIR
-fi
-cp -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar $WORK_DIR/docker/api
-
-echo "Starting docker environment and running func tests..."
-
-if [ -z "${PAR_CPU}" ]; then
-  export PAR_CPU=2
-fi
-
-docker-compose -f $WORK_DIR/docker/docker-compose-func-test.yml --project-directory $WORK_DIR/docker up --build --exit-code-from functest
-test_result=$?
-
-echo "Grabbing the logs..."
-docker logs vinyldns-functest
-docker logs vinyldns-api > $DIR/../target/vinyldns-api.log 2>/dev/null
-docker logs vinyldns-bind9 > $DIR/../target/vinyldns-bind9.log 2>/dev/null
-docker logs vinyldns-mysql > $DIR/../target/vinyldns-mysql.log 2>/dev/null
-docker logs vinyldns-elasticmq > $DIR/../target/vinyldns-elasticmq.log 2>/dev/null
-docker logs vinyldns-dynamodb > $DIR/../target/vinyldns-dynamodb.log 2>/dev/null
-docker logs vinyldns-functest > $DIR/../target/vinyldns-functest.log 2>/dev/null
-
-echo "Cleaning up docker containers..."
-$DIR/./remove-vinyl-containers.sh
-
-echo "Func tests returned result: ${test_result}"
-exit ${test_result}

bin/func-test-api.sh

@@ -1,57 +1,7 @@
-#!/bin/bash
+#!/usr/bin/env bash
-######################################################################
+set -euo pipefail
-# Copies the contents of `docker` into target/scala-2.12
-# to start up dependent services via docker compose.  Once
-# dependent services are started up, the fat jar built by sbt assembly
-# is loaded into a docker container.  Finally, the func tests run inside
-# another docker container
-# At the end, we grab all the logs and place them in the target
-# directory
-######################################################################
 
-DIR=$( cd $(dirname $0) ; pwd -P )
+DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
-WORK_DIR=$DIR/../target/scala-2.12
-mkdir -p $WORK_DIR
 
-echo "Cleaning up unused networks..."
+cd "$DIR/../test/api/functional"
-docker network prune -f
+make
-
-echo "Copy all docker to the target directory so we can start up properly and the docker context is small..."
-cp -af $DIR/../docker $WORK_DIR/
-
-echo "Copy over the functional tests as well as those that are run in a container..."
-mkdir -p $WORK_DIR/functest
-rsync -av --exclude='.virtualenv' $DIR/../modules/api/functional_test $WORK_DIR/docker/functest
-
-echo "Copy the vinyldns.jar to the api docker folder so it is in context..."
-if [[ ! -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
-    echo "vinyldns jar not found, building..."
-    cd $DIR/../
-    sbt api/clean api/assembly
-    cd $DIR
-fi
-cp -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar $WORK_DIR/docker/api
-
-echo "Starting docker environment and running func tests..."
-
-# If PAR_CPU is unset; default to auto
-if [ -z "${PAR_CPU}" ]; then
-  export PAR_CPU=auto
-fi
-
-docker-compose -f $WORK_DIR/docker/docker-compose-func-test.yml --project-directory $WORK_DIR/docker --log-level ERROR up --build --exit-code-from functest
-test_result=$?
-
-echo "Grabbing the logs..."
-
-docker logs vinyldns-api > $DIR/../target/vinyldns-api.log 2>/dev/null
-docker logs vinyldns-bind9 > $DIR/../target/vinyldns-bind9.log 2>/dev/null
-docker logs vinyldns-mysql > $DIR/../target/vinyldns-mysql.log 2>/dev/null
-docker logs vinyldns-elasticmq > $DIR/../target/vinyldns-elasticmq.log 2>/dev/null
-docker logs vinyldns-functest > $DIR/../target/vinyldns-functest.log 2>/dev/null
-
-echo "Cleaning up docker containers..."
-$DIR/./remove-vinyl-containers.sh
-
-echo "Func tests returned result: ${test_result}"
-exit ${test_result}

bin/func-test-portal.sh

@@ -1,46 +1,7 @@
-#!/bin/bash
+#!/usr/bin/env bash
-######################################################################
+set -euo pipefail
-# Runs e2e tests against the portal
-######################################################################
 
-DIR=$( cd $(dirname $0) ; pwd -P )
+DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
-WORK_DIR=$DIR/../modules/portal
 
-function check_for() {
+cd "$DIR/../test/portal/functional"
-    which $1 >/dev/null 2>&1
+make
-    EXIT_CODE=$?
-    if [ ${EXIT_CODE} != 0 ]
-    then
-        echo "$1 is not installed"
-        exit ${EXIT_CODE}
-    fi
-}
-
-cd $WORK_DIR
-check_for python
-check_for npm
-
-# if the program exits before this has been captured then there must have been an error
-EXIT_CODE=1
-
-# javascript code generate
-npm install
-grunt default
-
-TEST_SUITES=('grunt unit')
-
-for TEST in "${TEST_SUITES[@]}"
-do
-    echo "##### Running test: [$TEST]"
-    $TEST
-    EXIT_CODE=$?
-    echo "##### Test [$TEST] ended with status [$EXIT_CODE]"
-    if  [ ${EXIT_CODE} != 0 ]
-    then
-        cd -
-        exit ${EXIT_CODE}
-    fi
-done
-
-cd -
-exit 0

bin/generate-aes-256-hex-key.sh

@@ -1,18 +0,0 @@
-#!/bin/bash
-
-# Generate 256-bit AES key.
-#
-# Usage:
-#  $ ./generate-aes-256-hex-key.sh [passphrase]
-# * passphrase: Optional passphrase used to generate secret key. A pseudo-random passphrase will be used if
-#               one is not provided.
-
-if [[ ! -z "$1" ]]
-then
-    echo "Using user-provided passphrase."
-fi
-
-PASSPHRASE=${1:-$(openssl rand 32)}
-
-KEY=$(openssl enc -aes-256-cbc -k "$PASSPHRASE" -P -md sha1 | awk -F'=' 'NR == 2 {print $2}')
-echo "Your 256-bit AES hex key: $KEY"

bin/release.sh

@@ -34,13 +34,11 @@ if [ "$1" != "skip-tests" ]; then
   fi
 
   printf "\nrunning api func tests... \n"
-  "$DIR"/remove-vinyl-containers.sh
   if ! "$DIR"/func-test-api.sh
   then
       printf "\nerror: bin/func-test-api.sh failed \n"
       exit 1
   fi
-  "$DIR"/remove-vinyl-containers.sh
 
   printf "\nrunning portal func tests... \n"
   if ! "$DIR"/func-test-portal.sh

bin/verify.sh

@@ -1,18 +1,14 @@
-#!/bin/bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
 echo 'Running tests...'
 
-echo 'Stopping any docker containers...'
+cd "$DIR/../test/api/integration"
-./bin/remove-vinyl-containers.sh
+make build && make run -- sbt ";validate;verify"
-
-echo 'Starting up docker for integration testing and running unit and integration tests on all modules...'
-sbt ";validate;verify"
 verify_result=$?
 
-echo 'Stopping any docker containers...'
+if [ ${verify_result} -eq 0 ]; then
-./bin/remove-vinyl-containers.sh
-
-if [ ${verify_result} -eq 0 ]
-then
     echo 'Verify successful!'
     exit 0
 else

build.sbt

@@ -1,7 +1,6 @@
 import CompilerOptions._
 import Dependencies._
 import Resolvers._
-import com.typesafe.sbt.packager.docker._
 import microsites._
 import org.scalafmt.sbt.ScalafmtPlugin._
 import sbtrelease.ReleasePlugin.autoImport.ReleaseTransformations._
@@ -11,7 +10,7 @@ import scala.util.Try
 
 resolvers ++= additionalResolvers
 
-lazy val IntegrationTest = config("it") extend Test
+lazy val IntegrationTest = config("it").extend(Test)
 
 // settings that should be inherited by all projects
 lazy val sharedSettings = Seq(
@@ -21,11 +20,12 @@ lazy val sharedSettings = Seq(
   startYear := Some(2018),
   licenses += ("Apache-2.0", new URL("https://www.apache.org/licenses/LICENSE-2.0.txt")),
   scalacOptions ++= scalacOptionsByV(scalaVersion.value),
-  scalacOptions in(Compile, doc) += "-no-link-warnings",
+  scalacOptions in (Compile, doc) += "-no-link-warnings",
   // Use wart remover to eliminate code badness
   wartremoverErrors := (
     if (getPropertyFlagOrDefault("build.lintOnCompile", true))
-      Seq(Wart.EitherProjectionPartial,
+      Seq(
+        Wart.EitherProjectionPartial,
         Wart.IsInstanceOf,
         Wart.JavaConversions,
         Wart.Return,
@@ -33,22 +33,21 @@ lazy val sharedSettings = Seq(
         Wart.ExplicitImplicitTypes
       )
     else Seq.empty
-    ),
+  ),
 
   // scala format
-  scalafmtOnCompile := getPropertyFlagOrDefault("build.scalafmtOnCompile", true),
+  scalafmtOnCompile := getPropertyFlagOrDefault("build.scalafmtOnCompile", false),
-  scalafmtOnCompile in IntegrationTest := getPropertyFlagOrDefault("build.scalafmtOnCompile", true),
 
   // coverage options
   coverageMinimum := 85,
   coverageFailOnMinimum := true,
-  coverageHighlighting := true,
+  coverageHighlighting := true
 )
 
 lazy val testSettings = Seq(
   parallelExecution in Test := true,
   parallelExecution in IntegrationTest := false,
-  fork in IntegrationTest := true,
+  fork in IntegrationTest := false,
   testOptions in Test += Tests.Argument("-oDNCXEPQRMIK", "-l", "SkipCI"),
   logBuffered in Test := false,
   // Hide stack traces in tests
@@ -75,74 +74,16 @@ lazy val apiAssemblySettings = Seq(
   // there are some odd things from dnsjava including update.java and dig.java that we don't use
   assemblyMergeStrategy in assembly := {
     case "update.class" | "dig.class" => MergeStrategy.discard
-    case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "index.js") => MergeStrategy.discard
+    case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "index.js") =>
-    case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "template.js") => MergeStrategy.discard
+      MergeStrategy.discard
+    case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "template.js") =>
+      MergeStrategy.discard
     case x =>
       val oldStrategy = (assemblyMergeStrategy in assembly).value
       oldStrategy(x)
   }
 )
 
-lazy val apiDockerSettings = Seq(
-  dockerBaseImage := "adoptopenjdk/openjdk11:jdk-11.0.7_10-alpine",
-  dockerUsername := Some("vinyldns"),
-  packageName in Docker := "api",
-  dockerExposedPorts := Seq(9000),
-  dockerEntrypoint := Seq("/opt/docker/bin/api"),
-  dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath
-  dockerExposedVolumes := Seq("/opt/docker/conf"), // mount extra config to the classpath
-
-  // add extra libs to class path via mount
-  scriptClasspath in bashScriptDefines ~= (cp => cp :+ "${app_home}/../lib_extra/*"),
-
-  // adds config file to mount
-  bashScriptExtraDefines += """addJava "-Dconfig.file=${app_home}/../conf/application.conf"""",
-  bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml"""", // adds logback
-
-  // this is the default version, can be overridden
-  bashScriptExtraDefines += s"""addJava "-Dvinyldns.base-version=${(version in ThisBuild).value}"""",
-  bashScriptExtraDefines += "(cd ${app_home} && ./wait-for-dependencies.sh && cd -)",
-  credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials")),
-  dockerCommands ++= Seq(
-    Cmd("USER", "root"), // switch to root so we can install netcat
-    ExecCmd("RUN", "apk", "add", "--update", "--no-cache", "netcat-openbsd", "bash"),
-    Cmd("USER", "1001:0") // switch back to the daemon user
-  ),
-)
-
-lazy val portalDockerSettings = Seq(
-  dockerBaseImage := "adoptopenjdk/openjdk11:jdk-11.0.7_10-alpine",
-  dockerUsername := Some("vinyldns"),
-  packageName in Docker := "portal",
-  dockerExposedPorts := Seq(9001),
-  dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath
-  dockerExposedVolumes := Seq("/opt/docker/conf"), // mount extra config to the classpath
-
-  // add extra libs to class path via mount
-  scriptClasspath in bashScriptDefines ~= (cp => cp :+ "${app_home}/../lib_extra/*"),
-
-  // adds config file to mount
-  bashScriptExtraDefines += """addJava "-Dconfig.file=${app_home}/../conf/application.conf"""",
-  bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml"""",
-
-  // this is the default version, can be overridden
-  bashScriptExtraDefines += s"""addJava "-Dvinyldns.base-version=${(version in ThisBuild).value}"""",
-
-  // needed to avoid access issue in play for the RUNNING_PID
-  // https://github.com/lightbend/sbt-reactive-app/issues/177
-  bashScriptExtraDefines += s"""addJava "-Dplay.server.pidfile.path=/dev/null"""",
-
-  // wait for mysql
-  bashScriptExtraDefines += "(cd ${app_home}/../ && ls && ./wait-for-dependencies.sh && cd -)",
-  dockerCommands ++= Seq(
-    Cmd("USER", "root"), // switch to root so we can install netcat
-    ExecCmd("RUN", "apk", "add", "--update", "--no-cache", "netcat-openbsd", "bash"),
-    Cmd("USER", "1001:0") // switch back to the user that runs the process
-  ),
-
-  credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials"))
-)
-
 lazy val noPublishSettings = Seq(
   publish := {},
   publishLocal := {},
@@ -164,7 +105,7 @@ lazy val portalPublishSettings = Seq(
     case (file, _) => file.getName.equals("local.conf")
   }),
   // for local.conf to be excluded in jars
-  mappings in(Compile, packageBin) ~= (_.filterNot {
+  mappings in (Compile, packageBin) ~= (_.filterNot {
     case (file, _) => file.getName.equals("local.conf")
   })
 )
@@ -181,8 +122,7 @@ lazy val allApiSettings = Revolver.settings ++ Defaults.itSettings ++
   sharedSettings ++
   apiAssemblySettings ++
   testSettings ++
-  apiPublishSettings ++
+  apiPublishSettings
-  apiDockerSettings
 
 lazy val api = (project in file("modules/api"))
   .enablePlugins(JavaAppPackaging, AutomateHeaderPlugin)
@@ -197,23 +137,18 @@ lazy val api = (project in file("modules/api"))
     r53 % "compile->compile;it->it"
   )
 
-val killDocker = TaskKey[Unit]("killDocker", "Kills all vinyldns docker containers")
+lazy val root = (project in file("."))
-lazy val root = (project in file(".")).enablePlugins(DockerComposePlugin, AutomateHeaderPlugin)
+  .enablePlugins(AutomateHeaderPlugin)
   .configs(IntegrationTest)
   .settings(headerSettings(IntegrationTest))
   .settings(sharedSettings)
   .settings(
-    inConfig(IntegrationTest)(scalafmtConfigSettings),
+    inConfig(IntegrationTest)(scalafmtConfigSettings)
-    killDocker := {
-      import scala.sys.process._
-      "./bin/remove-vinyl-containers.sh" !
-    },
   )
   .aggregate(core, api, portal, mysql, sqs, r53)
 
 lazy val coreBuildSettings = Seq(
   name := "core",
-
   // do not use unused params as NoOpCrypto ignores its constructor, we should provide a way
   // to write a crypto plugin so that we fall back to a noarg constructor
   scalacOptions ++= scalacOptionsByV(scalaVersion.value).filterNot(_ == "-Ywarn-unused:params")
@@ -221,7 +156,9 @@ lazy val coreBuildSettings = Seq(
 lazy val corePublishSettings = Seq(
   publishMavenStyle := true,
   publishArtifact in Test := false,
-  pomIncludeRepository := { _ => false },
+  pomIncludeRepository := { _ =>
+    false
+  },
   autoAPIMappings := true,
   publish in Docker := {},
   mainClass := None,
@@ -235,7 +172,8 @@ lazy val corePublishSettings = Seq(
   sonatypeProfileName := "io.vinyldns"
 )
 
-lazy val core = (project in file("modules/core")).enablePlugins(AutomateHeaderPlugin)
+lazy val core = (project in file("modules/core"))
+  .enablePlugins(AutomateHeaderPlugin)
   .settings(sharedSettings)
   .settings(coreBuildSettings)
   .settings(corePublishSettings)
@@ -257,7 +195,8 @@ lazy val mysql = (project in file("modules/mysql"))
   .settings(libraryDependencies ++= mysqlDependencies ++ commonTestDependencies.map(_ % "test, it"))
   .settings(
     organization := "io.vinyldns"
-  ).dependsOn(core % "compile->compile;test->test")
+  )
+  .dependsOn(core % "compile->compile;test->test")
   .settings(name := "mysql")
 
 lazy val sqs = (project in file("modules/sqs"))
@@ -271,8 +210,9 @@ lazy val sqs = (project in file("modules/sqs"))
   .settings(Defaults.itSettings)
   .settings(libraryDependencies ++= sqsDependencies ++ commonTestDependencies.map(_ % "test, it"))
   .settings(
-    organization := "io.vinyldns",
+    organization := "io.vinyldns"
-  ).dependsOn(core % "compile->compile;test->test")
+  )
+  .dependsOn(core % "compile->compile;test->test")
   .settings(name := "sqs")
 
 lazy val r53 = (project in file("modules/r53"))
@@ -287,53 +227,51 @@ lazy val r53 = (project in file("modules/r53"))
   .settings(libraryDependencies ++= r53Dependencies ++ commonTestDependencies.map(_ % "test, it"))
   .settings(
     organization := "io.vinyldns",
-    coverageMinimum := 65,
+    coverageMinimum := 65
-  ).dependsOn(core % "compile->compile;test->test")
+  )
+  .dependsOn(core % "compile->compile;test->test")
   .settings(name := "r53")
 
 val preparePortal = TaskKey[Unit]("preparePortal", "Runs NPM to prepare portal for start")
-val checkJsHeaders = TaskKey[Unit]("checkJsHeaders", "Runs script to check for APL 2.0 license headers")
+val checkJsHeaders =
-val createJsHeaders = TaskKey[Unit]("createJsHeaders", "Runs script to prepend APL 2.0 license headers to files")
+  TaskKey[Unit]("checkJsHeaders", "Runs script to check for APL 2.0 license headers")
+val createJsHeaders =
+  TaskKey[Unit]("createJsHeaders", "Runs script to prepend APL 2.0 license headers to files")
 
-lazy val portal = (project in file("modules/portal")).enablePlugins(PlayScala, AutomateHeaderPlugin)
+lazy val portal = (project in file("modules/portal"))
+  .enablePlugins(PlayScala, AutomateHeaderPlugin)
   .settings(sharedSettings)
   .settings(testSettings)
   .settings(portalPublishSettings)
-  .settings(portalDockerSettings)
   .settings(
     name := "portal",
     libraryDependencies ++= portalDependencies,
     routesGenerator := InjectedRoutesGenerator,
     coverageExcludedPackages := "<empty>;views.html.*;router.*;controllers\\.javascript.*;.*Reverse.*",
     javaOptions in Test += "-Dconfig.file=conf/application-test.conf",
-
     // ads the version when working locally with sbt run
     PlayKeys.devSettings += "vinyldns.base-version" -> (version in ThisBuild).value,
-
     // adds an extra classpath to the portal loading so we can externalize jars, make sure to create the lib_extra
     // directory and lay down any dependencies that are required when deploying
     scriptClasspath in bashScriptDefines ~= (cp => cp :+ "lib_extra/*"),
     mainClass in reStart := None,
-
     // we need to filter out unused for the portal as the play framework needs a lot of unused things
-    scalacOptions ~= { opts => opts.filterNot(p => p.contains("unused")) },
+    scalacOptions ~= { opts =>
-
+      opts.filterNot(p => p.contains("unused"))
+    },
     // runs our prepare portal process
     preparePortal := {
       import scala.sys.process._
       "./modules/portal/prepare-portal.sh" !
     },
-
     checkJsHeaders := {
       import scala.sys.process._
       "./bin/add-license-headers.sh -d=modules/portal/public/lib -f=js -c" !
     },
-
     createJsHeaders := {
       import scala.sys.process._
       "./bin/add-license-headers.sh -d=modules/portal/public/lib -f=js" !
     },
-
     // change the name of the output to portal.zip
     packageName in Universal := "portal"
   )
@@ -365,8 +303,16 @@ lazy val docSettings = Seq(
   mdocIn := (sourceDirectory in Compile).value / "mdoc",
   micrositeCssDirectory := (resourceDirectory in Compile).value / "microsite" / "css",
   micrositeCompilingDocsTool := WithMdoc,
-  micrositeFavicons := Seq(MicrositeFavicon("favicon16x16.png", "16x16"), MicrositeFavicon("favicon32x32.png", "32x32")),
+  micrositeFavicons := Seq(
-  micrositeEditButton := Some(MicrositeEditButton("Improve this page", "/edit/master/modules/docs/src/main/mdoc/{{ page.path }}")),
+    MicrositeFavicon("favicon16x16.png", "16x16"),
+    MicrositeFavicon("favicon32x32.png", "32x32")
+  ),
+  micrositeEditButton := Some(
+    MicrositeEditButton(
+      "Improve this page",
+      "/edit/master/modules/docs/src/main/mdoc/{{ page.path }}"
+    )
+  ),
   micrositeFooterText := None,
   micrositeHighlightTheme := "atom-one-light",
   includeFilter in makeSite := "*.html" | "*.css" | "*.png" | "*.jpg" | "*.jpeg" | "*.gif" | "*.js" | "*.swf" | "*.md" | "*.webm" | "*.ico" | "CNAME" | "*.yml" | "*.svg" | "*.json" | "*.csv"
@@ -384,8 +330,10 @@ lazy val setSonatypeReleaseSettings = ReleaseStep(action = oldState => {
   val v = extracted.get(Keys.version)
   val snap = v.endsWith("SNAPSHOT")
   if (!snap) {
-    val publishToSettings = Some("releases" at "https://oss.sonatype.org/" + "service/local/staging/deploy/maven2")
+    val publishToSettings =
-    val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
+      Some("releases".at("https://oss.sonatype.org/" + "service/local/staging/deploy/maven2"))
+    val newState =
+      extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
 
     // create sonatypeReleaseCommand with releaseSonatype step
     val sonatypeCommand = Command.command("sonatypeReleaseCommand") {
@@ -397,8 +345,10 @@ lazy val setSonatypeReleaseSettings = ReleaseStep(action = oldState => {
 
     newState.copy(definedCommands = newState.definedCommands :+ sonatypeCommand)
   } else {
-    val publishToSettings = Some("snapshots" at "https://oss.sonatype.org/" + "content/repositories/snapshots")
+    val publishToSettings =
-    val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
+      Some("snapshots".at("https://oss.sonatype.org/" + "content/repositories/snapshots"))
+    val newState =
+      extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
 
     // create sonatypeReleaseCommand without releaseSonatype step
     val sonatypeCommand = Command.command("sonatypeReleaseCommand") {
@@ -437,21 +387,24 @@ releaseProcess :=
     finalReleaseStage
 
 // Let's do things in parallel!
-addCommandAlias("validate", "; root/clean; " +
+addCommandAlias(
-  "all core/headerCheck core/test:headerCheck " +
+  "validate",
-  "api/headerCheck api/test:headerCheck api/it:headerCheck " +
+  "; root/clean; " +
-  "mysql/headerCheck mysql/test:headerCheck mysql/it:headerCheck " +
+    "all core/headerCheck core/test:headerCheck " +
-  "r53/headerCheck r53/test:headerCheck r53/it:headerCheck " +
+    "api/headerCheck api/test:headerCheck api/it:headerCheck " +
-  "sqs/headerCheck sqs/test:headerCheck sqs/it:headerCheck " +
+    "mysql/headerCheck mysql/test:headerCheck mysql/it:headerCheck " +
-  "portal/headerCheck portal/test:headerCheck; " +
+    "r53/headerCheck r53/test:headerCheck r53/it:headerCheck " +
-  "portal/createJsHeaders;portal/checkJsHeaders;" +
+    "sqs/headerCheck sqs/test:headerCheck sqs/it:headerCheck " +
-  "root/compile;root/test:compile;root/it:compile"
+    "portal/headerCheck portal/test:headerCheck; " +
+    "portal/createJsHeaders;portal/checkJsHeaders;" +
+    "root/compile;root/test:compile;root/it:compile"
 )
 
-addCommandAlias("verify", "; project root; killDocker; dockerComposeUp; " +
+addCommandAlias(
-  "project root; coverage; " +
+  "verify",
-  "all test it:test; " +
+  "; project root; coverage; " +
-  "project root; coverageReport; coverageAggregate; killDocker"
+    "all test it:test; " +
+    "project root; coverageReport; coverageAggregate"
 )
 
 // Build the artifacts for release

build/README.md

@@ -50,8 +50,6 @@ The build will generate several VinylDNS docker images that are used to deploy i
 
 - `vinyldns/api` - this is the heart of the VinylDNS system, the backend API
 - `vinyldns/portal` - the VinylDNS web UI
-- `vinyldns/test-bind9` - a DNS server that is configured to support running the functional tests
-- `vinyldns/test` - a container that will execute functional tests, and exit success or failure when the tests are complete
 
 ### vinyldns/api
 
@@ -80,25 +78,3 @@ it is set as part of the container build
 any production environments.  Typically, you will add your own `application.conf` file in here with your settings.
 - `/opt/docker/lib_extra/` - if you need to have additional jar files available to your VinylDNS instance.
 Rarely used, but if you want to bring your own message queue or database you can put the `jar` files there
-
-### vinyldns/test-bind9
-
-This pulls correct DNS configuration to run func tests.  You can largely disregard what is in here
-
-### vinyldns/test
-
-This is used to run functional tests against a vinyldns instance.  **This is very useful for verifying 
-your environment as part of doing an upgrade.**  By default, it will run against a local docker-compose setup.
-
-**Environment Variables**
-- `VINYLDNS_URL` - the url to the vinyldns you will test against
-- `DNS_IP` - the IP address to the `vinyldns/test-bind9` container that you will use for test purposes
-- `TEST_PATTERN` - the actual functional test you want to run.  *Important, set to empty string to run 
-ALL test; otherwise, omit the environment variable when you run to just run smoke tests*.
-
-**Example**
-
-This example will run all functional tests on the given VinylDNS url and DNS IP address
-`docker run -e VINYLDNS_URL="https://my.vinyldns.example.com" -e DNS_IP="1.2.3.4" -e TEST_PATTERN=""`
-
-

build/docker-release.sh

@@ -107,15 +107,11 @@ if [ $DO_BUILD -eq 1 ]; then
   fi
 
   if [ $? -eq 0 ]; then
-    docker tag vinyldns/test-bind9:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test-bind9:$VINYLDNS_VERSION
-    docker tag vinyldns/test:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test:$VINYLDNS_VERSION
     docker tag vinyldns/api:$VINYLDNS_VERSION $REPOSITORY/vinyldns/api:$VINYLDNS_VERSION
     docker tag vinyldns/portal:$VINYLDNS_VERSION $REPOSITORY/vinyldns/portal:$VINYLDNS_VERSION
 
     if [ $TAG_LATEST -eq 1 ]; then
         echo "Tagging latest..."
-        docker tag vinyldns/test-bind9:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test-bind9:latest
-        docker tag vinyldns/test:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test:latest
         docker tag vinyldns/api:$VINYLDNS_VERSION $REPOSITORY/vinyldns/api:latest
         docker tag vinyldns/portal:$VINYLDNS_VERSION $REPOSITORY/vinyldns/portal:latest
     fi
@@ -123,15 +119,11 @@ if [ $DO_BUILD -eq 1 ]; then
 fi
 
 if [ $DOCKER_PUSH -eq 1 ]; then
-  docker push $REPOSITORY/vinyldns/test-bind9:$VINYLDNS_VERSION
-  docker push $REPOSITORY/vinyldns/test:$VINYLDNS_VERSION
   docker push $REPOSITORY/vinyldns/api:$VINYLDNS_VERSION
   docker push $REPOSITORY/vinyldns/portal:$VINYLDNS_VERSION
 
   if [ $TAG_LATEST -eq 1 ]; then
     echo "Pushing latest..."
-    docker push $REPOSITORY/vinyldns/test-bind9:latest
-    docker push $REPOSITORY/vinyldns/test:latest
     docker push $REPOSITORY/vinyldns/api:latest
     docker push $REPOSITORY/vinyldns/portal:latest
   fi

build/docker/test/Dockerfile

@@ -1,28 +0,0 @@
-FROM alpine/git:1.0.7 as gitcheckout
-
-ARG BRANCH=master
-
-RUN git clone -b ${BRANCH} --single-branch --depth 1 https://github.com/vinyldns/vinyldns.git /vinyldns
-
-FROM python:2.7.16-alpine3.9
-
-RUN apk add --update --no-cache bind-tools netcat-openbsd bash curl
-
-# The run script is what actually runs our func tests
-COPY run.sh /app/run.sh
-COPY run-tests.py /app/run-tests.py
-
-RUN chmod a+x /app/run.sh && chmod a+x /app/run-tests.py
-
-# Copy over the functional test directory, this must have been copied into the build context previous to this building!
-COPY --from=gitcheckout /vinyldns/modules/api/functional_test/ /app/
-
-# Install our func test requirements
-RUN pip install --index-url https://pypi.python.org/simple/ -r /app/requirements.txt
-
-ENV VINYLDNS_URL=""
-ENV DNS_IP=""
-ENV TEST_PATTERN="test_verify_production"
-
-# set the entry point for the container to start vinyl, specify the config resource
-ENTRYPOINT ["/app/run.sh"]

build/docker/test/run-tests.py

@@ -1,16 +0,0 @@
-#!/usr/bin/env python
-import os
-import sys
-
-basedir = os.path.dirname(os.path.realpath(__file__))
-
-report_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), '../target/pytest_reports')
-if not os.path.exists(report_dir):
-    os.system('mkdir -p ' + report_dir)
-
-import pytest
-
-result = 1
-result = pytest.main(list(sys.argv[1:]))
-
-sys.exit(result)

build/docker/test/run.sh

@@ -1,76 +0,0 @@
-#!/usr/bin/env bash
-
-# Assume defaults of local docker-compose if not set
-if [ -z "${VINYLDNS_URL}" ]; then
-  VINYLDNS_URL="http://vinyldns-api:9000"
-fi
-if [ -z "${DNS_IP}" ]; then
-  DNS_IP=$(dig +short vinyldns-bind9)
-fi
-
-# Assume all tests if not specified
-if [ -z "${TEST_PATTERN}" ]; then
-  TEST_PATTERN=
-else
-  TEST_PATTERN="-k ${TEST_PATTERN}"
-fi
-
-echo "Waiting for API to be ready at ${VINYLDNS_URL} ..."
-DATA=""
-RETRY=60
-SLEEP_DURATION=1
-while [ "$RETRY" -gt 0 ]
-do
-    DATA=$(curl -I -s "${VINYLDNS_URL}/ping" -o /dev/null -w "%{http_code}")
-    if [ $? -eq 0 ]
-    then
-        break
-    else
-        echo "Retrying" >&2
-
-        let RETRY-=1
-        sleep "$SLEEP_DURATION"
-
-        if [ "$RETRY" -eq 0 ]
-        then
-          echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
-          exit 1
-        fi
-    fi
-done
-
-echo "Running live tests against ${VINYLDNS_URL} and DNS server ${DNS_IP}"
-
-cd /app
-
-# Cleanup any errant cached file copies
-find . -name "*.pyc" -delete
-find . -name "__pycache__" -delete
-
-ls -al
-
-# -m plays havoc with -k, using variables is a headache, so doing this by hand
-# run parallel tests first (not serial)
-set -x
-./run-tests.py live_tests -n2 -v -m "not skip_production and not serial" ${TEST_PATTERN} --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} --teardown=False
-ret1=$?
-
-# IMPORTANT! pytest exists status code 5 if no tests are run, force that to 0
-if [ "$ret1" = 5 ]; then
-  echo "No tests collected."
-  ret1=0
-fi
-
-./run-tests.py live_tests -n0 -v -m "not skip_production and serial" ${TEST_PATTERN} --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} --teardown=True
-ret2=$?
-if [ "$ret2" = 5 ]; then
-  echo "No tests collected."
-  ret2=0
-fi
-
-if [ $ret1 -ne 0 ] || [ $ret2 -ne 0 ]; then
-  exit 1
-else
-  exit 0
-fi
-

docker/api/Dockerfile

@@ -10,7 +10,6 @@ RUN chmod a+x /app/run.sh
 COPY docker.conf /app/docker.conf
 
 EXPOSE 9000
-EXPOSE 2551
 
 # set the entry point for the container to start vinyl, specify the config resource
 ENTRYPOINT ["/app/run.sh"]

docker/api/docker.conf

@@ -106,17 +106,17 @@ vinyldns {
 
     settings {
       # AWS access key and secret.
-      access-key = "x"
+      access-key = "test"
       access-key = ${?AWS_ACCESS_KEY}
-      secret-key = "x"
+      secret-key = "test"
       secret-key = ${?AWS_SECRET_ACCESS_KEY}
 
       # Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed.
-      signing-region = "x"
+      signing-region = "us-east-1"
       signing-region = ${?SQS_REGION}
 
       # Endpoint to access queue
-      service-endpoint = "http://vinyldns-elasticmq:9324/"
+      service-endpoint = "http://vinyldns-localstack:19007/"
       service-endpoint = ${?SQS_ENDPOINT}
 
       # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change.

docker/api/run.sh

@@ -3,20 +3,10 @@
 # gets the docker-ized ip address, sets it to an environment variable
 export APP_HOST=`ip addr show eth0 | grep 'inet ' | awk '{print $2}' | cut -f1 -d'/'`
 
-export DYNAMO_ADDRESS="vinyldns-dynamodb"
-export DYNAMO_PORT=8000
-export JOURNAL_HOST="vinyldns-dynamodb"
-export JOURNAL_PORT=8000
 export MYSQL_ADDRESS="vinyldns-mysql"
 export MYSQL_PORT=3306
 export JDBC_USER=root
 export JDBC_PASSWORD=pass
-export DNS_ADDRESS="vinyldns-bind9"
-export DYNAMO_KEY="local"
-export DYNAMO_SECRET="local"
-export DYNAMO_TABLE_PREFIX=""
-export ELASTICMQ_ADDRESS="vinyldns-elasticmq"
-export DYNAMO_ENDPOINT="http://${DYNAMO_ADDRESS}:${DYNAMO_PORT}"
 export JDBC_URL="jdbc:mariadb://${MYSQL_ADDRESS}:${MYSQL_PORT}/vinyldns?user=${JDBC_USER}&password=${JDBC_PASSWORD}"
 export JDBC_MIGRATION_URL="jdbc:mariadb://${MYSQL_ADDRESS}:${MYSQL_PORT}/?user=${JDBC_USER}&password=${JDBC_PASSWORD}"
 
@@ -27,7 +17,7 @@ RETRY=40
 SLEEP_DURATION=1
 while [ "$RETRY" -gt 0 ]
 do
-    DATA=$(nc -vzw1 vinyldns-mysql 3306)
+    DATA=$(nc -vzw1 ${MYSQL_ADDRESS} ${MYSQL_PORT})
     if [ $? -eq 0 ]
     then
         break

docker/bind9/README.md

@@ -18,6 +18,5 @@ When used in a container, or to run `named`, the files in this directory should
 
 | Directory | Target |
 |:---|:---|
-| `etc/named.conf.local` | `/etc/bind/` |
+| `etc/named.conf.*` | `/etc/bind/` |
-| `etc/named.partition*.conf` | `/var/bind/config/` |
 | `zones/` | `/var/bind/` |

docker/bind9/etc/named.conf.local

@@ -29,7 +29,7 @@ key "vinyldns-sha512." {
   secret "xfKA0DYb88tiUGND+cWddwUg3/SugYSsdvCfBOJ1jr8MEdgbVRyrlVDEXLsfTUGorQ3ShENdymw2yw+rTr+lwA==";
 };
 
-include "/var/bind/config/named.partition1.conf";
+include "/etc/bind/named.conf.partition1";
-include "/var/bind/config/named.partition2.conf";
+include "/etc/bind/named.conf.partition2";
-include "/var/bind/config/named.partition3.conf";
+include "/etc/bind/named.conf.partition3";
-include "/var/bind/config/named.partition4.conf";
+include "/etc/bind/named.conf.partition4";

docker/docker-compose-func-test-testbind9.yml

@@ -1,60 +0,0 @@
-version: "3.0"
-services:
-  mysql:
-    image: "mysql:5.7"
-    env_file:
-      .env
-    container_name: "vinyldns-mysql"
-    ports:
-    - "19002:3306"
-    logging:
-      driver: none
-
-  bind9:
-    image: "vinyldns/test-bind9:0.9.4"
-    container_name: "vinyldns-bind9"
-    ports:
-    - "19001:53/tcp"
-    - "19001:53/udp"
-    logging:
-      driver: none
-
-  localstack:
-    image: localstack/localstack:0.10.4
-    container_name: "vinyldns-localstack"
-    ports:
-      - "19000:19000"
-      - "19006:19006"
-      - "19007:19007"
-      - "19009:19009"
-    environment:
-      - SERVICES=sns:19006,sqs:19007,route53:19009
-      - START_WEB=0
-      - HOSTNAME_EXTERNAL=vinyldns-localstack
-
-  # this file is copied into the target directory to get the jar!  won't run in place as is!
-  api:
-    build:
-      context: api
-    env_file:
-      .env
-    container_name: "vinyldns-api"
-    ports:
-    - "9000:9000"
-    depends_on:
-    - mysql
-    - bind9
-    - localstack
-    logging:
-      driver: none
-
-  functest:
-    build:
-      context: functest
-    env_file:
-      .env
-    environment:
-    - PAR_CPU=${PAR_CPU}
-    container_name: "vinyldns-functest"
-    depends_on:
-    - api

docker/docker-compose-func-test.yml

@@ -1,87 +0,0 @@
-version: "3.5"
-
-services:
-
-  # this file is copied into the target directory to get the jar!  won't run in place as is!
-  api:
-    build:
-      context: api
-    env_file:
-      .env
-    container_name: "vinyldns-api"
-    ports:
-      - "9000:9000"
-    depends_on:
-      - mysql
-      - bind9
-      - localstack
-    networks:
-      vinyldns:
-        ipv4_address: 172.10.10.2
-
-  mysql:
-    image: "mysql:5.7"
-    env_file:
-      .env
-    container_name: "vinyldns-mysql"
-    ports:
-      - "19002:3306"
-    logging:
-      driver: none
-    networks:
-      vinyldns:
-        ipv4_address: 172.10.10.3
-
-  localstack:
-    image: localstack/localstack:0.10.4
-    container_name: "vinyldns-localstack"
-    ports:
-      - "19006:19006"
-      - "19007:19007"
-      - "19009:19009"
-    environment:
-      - SERVICES=sns:19006,sqs:19007,route53:19009
-      - START_WEB=0
-      - HOSTNAME_EXTERNAL=vinyldns-localstack
-    networks:
-      vinyldns:
-        ipv4_address: 172.10.10.4
-
-  bind9:
-    image: "vinyldns/bind9:0.0.4"
-    env_file:
-      .env
-    container_name: "vinyldns-bind9"
-    volumes:
-      - ./bind9/etc:/var/cache/bind/config
-      - ./bind9/zones:/var/cache/bind/zones
-    ports:
-      - "19001:53/tcp"
-      - "19001:53/udp"
-    logging:
-      driver: none
-    networks:
-      vinyldns:
-        ipv4_address: 172.10.10.10
-
-  functest:
-    build:
-      context: functest
-    env_file:
-      .env
-    environment:
-      - PAR_CPU=${PAR_CPU}
-    container_name: "vinyldns-functest"
-    depends_on:
-      - api
-    networks:
-      - vinyldns
-
-networks:
-  # Custom network so that we have some control over IP space and deterministic container IPs
-  vinyldns:
-    name: vinyldns
-    driver: bridge
-    ipam:
-      config:
-        - subnet: 172.10.10.0/24

docker/docker-compose-quick-start.yml

@@ -9,7 +9,7 @@ services:
       - "19002:3306"
 
   bind9:
-    image: "vinyldns/bind9:0.0.4"
+    image: "vinyldns/bind9:0.0.5"
     env_file:
       .env.quickstart
     container_name: "vinyldns-bind9"

docker/docker-compose.yml

@@ -8,7 +8,7 @@ services:
       - "19002:3306"
 
   bind9:
-    image: vinyldns/bind9:0.0.4<skipPull>
+    image: vinyldns/bind9:0.0.5<skipPull>
     env_file:
       .env
     ports:

docker/elasticmq/Dockerfile

@@ -1,10 +0,0 @@
-FROM alpine:3.2
-FROM anapsix/alpine-java:8_server-jre
-
-EXPOSE 9324
-
-COPY run.sh /elasticmq/run.sh
-COPY custom.conf /elasticmq/custom.conf
-COPY elasticmq-server-0.13.2.jar /elasticmq/server.jar
-
-ENTRYPOINT ["/elasticmq/run.sh"]

docker/elasticmq/custom.conf

@@ -1,22 +0,0 @@
-node-address {
-  protocol = http
-  host = "localhost"
-  host = ${?QUEUE_HOST}
-  port = 9324
-  context-path = ""
-}
-
-rest-sqs {
-  enabled = true
-  bind-port = 9324
-  bind-hostname = "0.0.0.0"
-  // Possible values: relaxed, strict
-  sqs-limits = relaxed
-}
-
-queues {
-  vinyldns {
-    defaultVisibilityTimeout = 10 seconds
-    receiveMessageWait = 0 seconds
-  }
-}

docker/elasticmq/run.sh

@@ -1,8 +0,0 @@
-#!/usr/bin/env bash
-
-# gets the docker-ized ip address, sets it to an environment variable
-export APP_HOST=`ip addr show eth0 | grep 'inet ' | awk '{print $2}' | cut -f1 -d'/'`
-
-echo "APP HOST = ${APP_HOST}"
-
-java -Djava.net.preferIPv4Stack=true -Dconfig.file=/elasticmq/custom.conf -jar /elasticmq/server.jar

docker/email/.gitignore

@@ -1,2 +0,0 @@
-*
-!.gitignore

docker/functest/Dockerfile

@@ -1,23 +0,0 @@
-FROM python:2.7.15-stretch
-
-# Install dns utils so we can run dig
-RUN apt-get update && apt-get install dnsutils -y
-
-# The run script is what actually runs our func tests
-COPY run.sh /app/run.sh
-RUN chmod a+x /app/run.sh
-
-COPY run-tests.py /app/run-tests.py
-RUN chmod a+x /app/run-tests.py
-
-# Copy over the functional test directory, this must have been copied into the build context previous to this building!
-ADD functional_test /app
-
-# Install our func test requirements
-RUN pip install --index-url https://pypi.python.org/simple/ -r /app/requirements.txt
-
-# Specifies how many CPUs to use for func tests; the more the better or specifiy "auto" for optimal results
-ENV PAR_CPU=2
-
-# set the entry point for the container to start vinyl, specify the config resource
-ENTRYPOINT ["/app/run.sh"]

docker/functest/run-tests.py

@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-import os
-import sys
-
-basedir = os.path.dirname(os.path.realpath(__file__))
-
-report_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), '../target/pytest_reports')
-if not os.path.exists(report_dir):
-    os.system('mkdir -p ' + report_dir)
-
-import pytest
-
-result = 1
-result = pytest.main(list(sys.argv[1:]))
-
-sys.exit(result)
-
-

docker/functest/run.sh

@@ -1,81 +0,0 @@
-#!/usr/bin/env bash
-
-# Assume defaults of local docker-compose if not set
-if [ -z "${VINYLDNS_URL}" ]; then
-  VINYLDNS_URL="http://vinyldns-api:9000"
-fi
-if [ -z "${DNS_IP}" ]; then
-  DNS_IP=$(dig +short vinyldns-bind9)
-fi
-
-# Assume all tests if not specified
-if [ -z "${TEST_PATTERN}" ]; then
-  TEST_PATTERN=
-else
-  TEST_PATTERN="-k ${TEST_PATTERN}"
-fi
-
-if [ -z "${PAR_CPU}" ]; then
-  export PAR_CPU=2
-fi
-
-echo "Waiting for API to be ready at ${VINYLDNS_URL} ..."
-DATA=""
-RETRY=60
-SLEEP_DURATION=1
-while [ "$RETRY" -gt 0 ]
-do
-    DATA=$(curl -I -s "${VINYLDNS_URL}/ping" -o /dev/null -w "%{http_code}")
-    if [ $? -eq 0 ]
-    then
-        break
-    else
-        echo "Retrying" >&2
-
-        let RETRY-=1
-        sleep "$SLEEP_DURATION"
-
-        if [ "$RETRY" -eq 0 ]
-        then
-          echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
-          exit 1
-        fi
-    fi
-done
-
-echo "Running live tests against ${VINYLDNS_URL} and DNS server ${DNS_IP}"
-
-cd /app
-
-# Cleanup any errant cached file copies
-find . -name "*.pyc" -delete
-find . -name "__pycache__" -delete
-
-result=0
-# If PROD_ENV is not true, we are in a local docker environment so do not skip anything
-if [ "${PROD_ENV}" = "true" ]; then
-    # -m plays havoc with -k, using variables is a headache, so doing this by hand
-    # run parallel tests first (not serial)
-    echo "./run-tests.py live_tests -n${PAR_CPU} -v -m \"not skip_production and not serial\" -v --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False"
-    ./run-tests.py live_tests -n${PAR_CPU} -v -m "not skip_production and not serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False
-    result=$?
-    if [ $result -eq 0 ]; then
-      # run serial tests second (serial marker)
-      echo "./run-tests.py live_tests -n0 -v -m \"not skip_production and serial\" -v --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True"
-      ./run-tests.py live_tests -n0 -v -m "not skip_production and serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True
-      result=$?
-    fi
-else
-    # run parallel tests first (not serial)
-    echo "./run-tests.py live_tests -n${PAR_CPU} -v -m \"not serial\" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False"
-    ./run-tests.py live_tests -n${PAR_CPU} -v -m "not serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False
-    result=$?
-    if [ $result -eq 0 ]; then
-      # run serial tests second (serial marker)
-      echo "./run-tests.py live_tests -n0 -v -m \"serial\" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True"
-      ./run-tests.py live_tests -n0 -v -m "serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True
-      result=$?
-    fi
-fi
-
-exit $result

modules/api/functional_test/Dockerfile

@@ -1,33 +0,0 @@
-# Build VinylDNS API if the JAR doesn't already exist
-FROM vinyldns/build:base-api as vinyldns-api
-COPY modules/api/functional_test/docker.conf modules/api/functional_test/vinyldns*.jar /opt/vinyldns/
-COPY . /build/
-WORKDIR /build
-
-##  Run the build if we don't already have a vinyldns.jar
-RUN if [ ! -f /opt/vinyldns/vinyldns.jar ]; then \
-      env SBT_OPTS="-XX:+UseConcMarkSweepGC -Xmx4G -Xms1G" \
-      sbt -Dbuild.scalafmtOnCompile=false -Dbuild.lintOnCompile=fase ";project api;coverageOff;assembly" \
-      && cp modules/api/target/scala-2.12/vinyldns.jar /opt/vinyldns/; \
-    fi
-
-# Build the testing image, copying data from `vinyldns-api`
-FROM vinyldns/build:base-test
-SHELL ["/bin/bash","-c"]
-COPY --from=vinyldns-api /opt/vinyldns /opt/vinyldns
-
-# Local bind server files
-COPY docker/bind9/etc/named.conf.local /etc/bind/
-COPY docker/bind9/etc/*.conf /var/bind/config/
-COPY docker/bind9/zones/ /var/bind/
-RUN named-checkconf
-
-# Copy over the functional tests
-COPY modules/api/functional_test /functional_test
-
-ENTRYPOINT ["/bin/bash", "-c", "/initialize.sh && \
-                                (java -Dconfig.file=/opt/vinyldns/docker.conf -jar /opt/vinyldns/vinyldns.jar &> /opt/vinyldns/vinyldns.log &) && \
-                                echo -n 'Starting VinylDNS API..' && \
-                                timeout 30s grep -q 'STARTED SUCCESSFULLY' <(timeout 30s tail -f /opt/vinyldns/vinyldns.log) && \
-                                echo 'done.' && \
-                                /bin/bash"]

modules/api/functional_test/Makefile

@@ -1,25 +0,0 @@
-SHELL=bash
-ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
-
-# Check that the required version of make is being used
-REQ_MAKE_VER:=3.82
-ifneq ($(REQ_MAKE_VER),$(firstword $(sort $(MAKE_VERSION) $(REQ_MAKE_VER))))
-   $(error The version of MAKE $(REQ_MAKE_VER) or higher is required; you are running $(MAKE_VERSION))
-endif
-
-.ONESHELL:
-
-.PHONY: all build run
-
-all: build run
-
-build:
-	@set -euo pipefail
-	trap 'if [ -f modules/api/functional_test/vinyldns.jar ]; then rm modules/api/functional_test/vinyldns.jar;	fi' EXIT
-	cd ../../..
-	if [ -f modules/api/target/scala-2.12/vinyldns.jar ]; then cp modules/api/target/scala-2.12/vinyldns.jar modules/api/functional_test/vinyldns.jar; fi
-	docker build -t vinyldns-test -f modules/api/functional_test/Dockerfile .
-
-run:
-	@set -euo pipefail
-	docker run -it --rm -p 9000:9000 -p 19001:53/tcp -p 19001:53/udp vinyldns-test

modules/api/src/it/scala/vinyldns/api/backend/dns/DnsBackendIntegrationSpec.scala

@@ -59,7 +59,7 @@ class DnsBackendIntegrationSpec extends AnyWordSpec with Matchers {
         config.tsigUsage
       )
       val testZone = Zone(
-        "open.",
+        "open1.",
         "test@test.com",
         connection = Some(testConnection)
       )

modules/api/src/it/scala/vinyldns/api/domain/zone/ZoneViewLoaderIntegrationSpec.scala

@@ -1,3 +1,4 @@
+
 /*
  * Copyright 2018 Comcast Cable Communications Management, LLC
  *
@@ -35,7 +36,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
 
   "ZoneViewLoader" should {
     "return a ZoneView upon success" in {
-      val zone = Zone("vinyldns.", "test@test.com")
+      val zone = Zone("vinyldns1.", "test@test.com")
       DnsZoneViewLoader(zone, backendResolver.resolve(zone), 10000)
         .load()
         .unsafeRunSync() shouldBe a[ZoneView]
@@ -44,7 +45,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
     "return a failure if the transfer connection is bad" in {
       assertThrows[IllegalArgumentException] {
         val zone = Zone(
-          "vinyldns.",
+          "vinyldns1.",
           "bad@transfer.connection",
           connection = Some(
             ZoneConnection(
@@ -77,7 +78,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
     "return a failure if the zone is larger than the max zone size" in {
       assertThrows[ZoneTooLargeError] {
         val zone = Zone(
-          "vinyldns.",
+          "vinyldns1.",
           "test@test.com",
           connection = Some(
             ZoneConnection(
@@ -86,6 +87,14 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
               "nzisn+4G2ldMn0q1CV3vsg==",
               "127.0.0.1:19001"
             )
+          ),
+          transferConnection = Some(
+            ZoneConnection(
+              "vinyldns.",
+              "vinyldns.",
+              "nzisn+4G2ldMn0q1CV3vsg==",
+              "127.0.0.1:19001"
+            )
           )
         )
         DnsZoneViewLoader(zone, backendResolver.resolve(zone), 1)

modules/api/src/it/scala/vinyldns/api/notifier/sns/SnsNotifierIntegrationSpec.scala

@@ -16,25 +16,23 @@
 
 package vinyldns.api.notifier.sns
 
+import cats.effect.IO
+import com.amazonaws.auth.{AWSStaticCredentialsProvider, BasicAWSCredentials}
+import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration
+import com.amazonaws.services.sns.AmazonSNSClientBuilder
+import com.amazonaws.services.sqs.AmazonSQSClientBuilder
 import com.typesafe.config.{Config, ConfigFactory}
-import vinyldns.core.notifier._
+import org.joda.time.DateTime
-import vinyldns.api.MySqlApiIntegrationSpec
+import org.json4s.DefaultFormats
-import vinyldns.mysql.MySqlIntegrationSpec
+import org.json4s.jackson.JsonMethods._
 import org.scalatest.matchers.should.Matchers
 import org.scalatest.wordspec.AnyWordSpecLike
-import vinyldns.core.domain.batch._
+import vinyldns.api.MySqlApiIntegrationSpec
-import vinyldns.core.domain.record.RecordType
-import vinyldns.core.domain.record.AData
-import org.joda.time.DateTime
 import vinyldns.core.TestMembershipData._
-import cats.effect.IO
+import vinyldns.core.domain.batch._
-import com.amazonaws.services.sns.AmazonSNSClientBuilder
+import vinyldns.core.domain.record.{AData, RecordType}
-import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration
+import vinyldns.core.notifier._
-import com.amazonaws.services.sqs.AmazonSQSClientBuilder
+import vinyldns.mysql.MySqlIntegrationSpec
-import org.json4s.jackson.JsonMethods._
-import org.json4s.DefaultFormats
-import com.amazonaws.auth.BasicAWSCredentials
-import com.amazonaws.auth.AWSStaticCredentialsProvider
 
 class SnsNotifierIntegrationSpec
     extends MySqlApiIntegrationSpec
@@ -93,7 +91,7 @@ class SnsNotifierIntegrationSpec
       val sqs = AmazonSQSClientBuilder
         .standard()
         .withEndpointConfiguration(
-          new EndpointConfiguration("http://127.0.0.1:19007", "us-east-1")
+          new EndpointConfiguration("http://127.0.0.1:19003", "us-east-1")
         )
         .withCredentials(credentialsProvider)
         .build()
@@ -105,6 +103,7 @@ class SnsNotifierIntegrationSpec
         notifier <- new SnsNotifierProvider()
           .load(NotifierConfig("", snsConfig), userRepository)
         _ <- notifier.notify(Notification(batchChange))
+        _ <- IO { Thread.sleep(100) }
         messages <- IO { sqs.receiveMessage(queueUrl).getMessages }
         _ <- IO {
           sns.deleteTopic(topic)

modules/api/src/it/scala/vinyldns/api/route53/Route53ApiIntegrationSpec.scala

@@ -56,7 +56,7 @@ class Route53ApiIntegrationSpec
           "test",
           Some("access"),
           Some("secret"),
-          "http://127.0.0.1:19009",
+          "http://127.0.0.1:19003",
           "us-east-1"
         )
       )

modules/api/src/main/resources/reference.conf

@@ -16,43 +16,41 @@ vinyldns {
       {
         class-name = "vinyldns.api.backend.dns.DnsBackendProviderLoader"
         settings = {
-          legacy = true # set this to true to attempt to load legacy config YAML
+          legacy = false
-          backends = []
+          backends = [
-
+            {
-          # if not legacy then this...
+              id = "default"
-          # legacy = false
+              zone-connection = {
-          # backends = [
+                name = "vinyldns."
-          #  {
+                key-name = "vinyldns."
-          #    id = "default"
+                key = "nzisn+4G2ldMn0q1CV3vsg=="
-          #    zone-connection = {
+                primary-server = "127.0.0.1:19001"
-          #      name = "vinyldns."
+              }
-          #      keyName = "vinyldns."
+              transfer-connection = {
-          #      key = "nzisn+4G2ldMn0q1CV3vsg=="
+                name = "vinyldns."
-          #      primaryServer = "127.0.0.1:19001"
+                key-name = "vinyldns."
-          #    }
+                key = "nzisn+4G2ldMn0q1CV3vsg=="
-          #    transfer-connection = {
+                primary-server = "127.0.0.1:19001"
-          #      name = "vinyldns."
+              },
-          #      keyName = "vinyldns."
+              tsig-usage = "always"
-          #      key = "nzisn+4G2ldMn0q1CV3vsg=="
+            },
-          #      primaryServer = "127.0.0.1:19001"
+            {
-          #    }
+              id = "func-test-backend"
-          #  },
+              zone-connection = {
-          #  {
+                name = "vinyldns."
-          #    id = "func-test-backend"
+                key-name = "vinyldns."
-          #    zone-connection = {
+                key = "nzisn+4G2ldMn0q1CV3vsg=="
-          #      name = "vinyldns."
+                primary-server = "127.0.0.1:19001"
-          #      keyName = "vinyldns."
+              }
-          #      key = "nzisn+4G2ldMn0q1CV3vsg=="
+              transfer-connection = {
-          #      primaryServer = "127.0.0.1:19001"
+                name = "vinyldns."
-          #    }
+                key-name = "vinyldns."
-          #    transfer-connection = {
+                key = "nzisn+4G2ldMn0q1CV3vsg=="
-          #      name = "vinyldns."
+                primary-server = "127.0.0.1:19001"
-          #      keyName = "vinyldns."
+              },
-          #      key = "nzisn+4G2ldMn0q1CV3vsg=="
+              tsig-usage = "always"
-          #      primaryServer = "127.0.0.1:19001"
+            }
-          #    }
+          ]
-          #  }
-          #]
         }
       }
     ]
@@ -76,10 +74,10 @@ vinyldns {
       # secret-key = "x"
 
       # Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed.
-      signing-region = "x"
+      signing-region = "us-east-1"
 
       # Endpoint to access queue
-      service-endpoint = "http://localhost:19007/"
+      service-endpoint = "http://localhost:19003/"
 
       # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change.
       queue-name = "vinyldns"
@@ -141,9 +139,9 @@ vinyldns {
     class-name = "vinyldns.api.notifier.sns.SnsNotifierProvider"
     settings {
       topic-arn = "arn:aws:sns:us-east-1:000000000000:batchChanges"
-      access-key = "vinyldnsTest"
+      access-key = "test"
-      secret-key = "notNeededForSnsLocal"
+      secret-key = "test"
-      service-endpoint = "http://127.0.0.1:19006"
+      service-endpoint = "http://127.0.0.1:19003"
       signing-region = "us-east-1"
     }
   }

modules/api/src/test/scala/vinyldns/api/backend/dns/DnsBackendSpec.scala

@@ -101,7 +101,7 @@ class DnsBackendSpec
 
   override def beforeEach(): Unit = {
     doReturn(mockMessage).when(mockMessage).clone()
-    doReturn(new Array[DNS.Record](0)).when(mockMessage).getSectionArray(DNS.Section.ADDITIONAL)
+    doReturn(new java.util.ArrayList[DNS.Record](0)).when(mockMessage).getSection(DNS.Section.ADDITIONAL)
     doReturn(DNS.Rcode.NOERROR).when(mockMessage).getRcode
     doReturn(mockMessage).when(mockResolver).send(messageCaptor.capture())
     doReturn(DNS.Lookup.SUCCESSFUL).when(mockDnsQuery).result
@@ -160,7 +160,7 @@ class DnsBackendSpec
       val conn = zoneConnection.copy(primaryServer = "dns.comcast.net:19001")
 
       val dnsConn = DnsBackend("test", conn, None, new NoOpCrypto())
-      val simpleResolver = dnsConn.resolver.asInstanceOf[DNS.SimpleResolver]
+      val simpleResolver = dnsConn.resolver
 
       val address = simpleResolver.getAddress
 
@@ -172,7 +172,7 @@ class DnsBackendSpec
       val conn = zoneConnection.copy(primaryServer = "dns.comcast.net")
 
       val dnsConn = DnsBackend("test", conn, None, new NoOpCrypto())
-      val simpleResolver = dnsConn.resolver.asInstanceOf[DNS.SimpleResolver]
+      val simpleResolver = dnsConn.resolver
 
       val address = simpleResolver.getAddress
 
@@ -267,14 +267,14 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       dnsRecord.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord.getTTL shouldBe testA.ttl
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord shouldBe a[DNS.ARecord]
       dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -286,7 +286,7 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val rrset = sentMessage.getSectionRRsets(DNS.Section.UPDATE)(0)
+      val rrset = sentMessage.getSectionRRsets(DNS.Section.UPDATE).asScala.head
       rrset.getName.toString shouldBe "a-record.vinyldns."
       rrset.getTTL shouldBe testA.ttl
       rrset.getType shouldBe DNS.Type.A
@@ -298,7 +298,7 @@ class DnsBackendSpec
 
       records should contain theSameElementsAs expected
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -327,20 +327,20 @@ class DnsBackendSpec
       val sentMessage = messageCaptor.getValue
 
       // Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       emptyRecord.getName.toString shouldBe "updated-a-record.vinyldns."
       emptyRecord.getType shouldBe DNS.Type.A
       emptyRecord.getDClass shouldBe DNS.DClass.ANY
 
       // The second section in the replace is the data that is being passed in, this is different than an add
-      val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
+      val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
       dnsRecord.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord.getTTL shouldBe testA.ttl
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord shouldBe a[DNS.ARecord]
       dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -353,20 +353,20 @@ class DnsBackendSpec
       val sentMessage = messageCaptor.getValue
 
       // Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       emptyRecord.getName.toString shouldBe "a-record.vinyldns."
       emptyRecord.getType shouldBe DNS.Type.A
       emptyRecord.getDClass shouldBe DNS.DClass.ANY
 
       // The second section in the replace is the data that is being passed in, this is different than an add
-      val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
+      val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
       dnsRecord.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord.getTTL shouldBe 300
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord shouldBe a[DNS.ARecord]
       dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -378,7 +378,7 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE)
 
       emptyRecord shouldBe empty
       result shouldBe a[NoError]
@@ -393,13 +393,13 @@ class DnsBackendSpec
       val sentMessage = messageCaptor.getValue
 
       // Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       emptyRecord.getName.toString shouldBe "updated-a-record.vinyldns."
       emptyRecord.getType shouldBe DNS.Type.A
       emptyRecord.getDClass shouldBe DNS.DClass.ANY
 
       // The second section in the replace is the data that is being passed in, this is different than an add
-      val dnsRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
+      val dnsRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
       dnsRecord1.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord1.getTTL shouldBe testA.ttl
       dnsRecord1.getType shouldBe DNS.Type.A
@@ -407,7 +407,7 @@ class DnsBackendSpec
       val dnsRecord1Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
       List("1.1.1.1", "2.2.2.2") should contain(dnsRecord1Data)
 
-      val dnsRecord2 = sentMessage.getSectionArray(DNS.Section.UPDATE)(2)
+      val dnsRecord2 = sentMessage.getSection(DNS.Section.UPDATE).asScala(2)
       dnsRecord2.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord2.getTTL shouldBe testA.ttl
       dnsRecord2.getType shouldBe DNS.Type.A
@@ -415,7 +415,7 @@ class DnsBackendSpec
       val dnsRecord2Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
       List("1.1.1.1", "2.2.2.2") should contain(dnsRecord2Data)
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -443,7 +443,7 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE)
 
       emptyRecord shouldBe empty
       result shouldBe a[NoError]
@@ -457,20 +457,20 @@ class DnsBackendSpec
       val sentMessage = messageCaptor.getValue
 
       // A NONE update is sent for each DNS record that is getting deleted
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       emptyRecord.getName.toString shouldBe "a-record.vinyldns."
       emptyRecord.getType shouldBe DNS.Type.A
       emptyRecord.getDClass shouldBe DNS.DClass.NONE
 
       // The second section in the replace is the data that is being passed in, this is different than an add
-      val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
+      val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
       dnsRecord.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord.getTTL shouldBe testA.ttl
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord shouldBe a[DNS.ARecord]
       dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -482,7 +482,7 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)
+      val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala
 
       emptyRecord shouldBe empty
       result shouldBe a[NoError]
@@ -497,14 +497,14 @@ class DnsBackendSpec
       val sentMessage = messageCaptor.getValue
 
       // A NONE update is sent for each DNS record that is getting deleted
-      val deleteRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val deleteRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       deleteRecord1.getName.toString shouldBe "a-record.vinyldns."
       deleteRecord1.getType shouldBe DNS.Type.A
       deleteRecord1.getDClass shouldBe DNS.DClass.NONE
       val deleteRecord1Data = deleteRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
       List("4.4.4.4", "3.3.3.3") should contain(deleteRecord1Data)
 
-      val deleteRecord2 = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
+      val deleteRecord2 = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
       deleteRecord2.getName.toString shouldBe "a-record.vinyldns."
       deleteRecord2.getType shouldBe DNS.Type.A
       deleteRecord2.getDClass shouldBe DNS.DClass.NONE
@@ -512,7 +512,7 @@ class DnsBackendSpec
       List("4.4.4.4", "3.3.3.3") should contain(deleteRecord2Data)
 
       // The second section in the replace is the data that is being passed in, this is different than an add
-      val dnsRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(2)
+      val dnsRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala(2)
       dnsRecord1.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord1.getTTL shouldBe testA.ttl
       dnsRecord1.getType shouldBe DNS.Type.A
@@ -520,7 +520,7 @@ class DnsBackendSpec
       val dnsRecord1Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
       List("1.1.1.1", "2.2.2.2") should contain(dnsRecord1Data)
 
-      val dnsRecord2 = sentMessage.getSectionArray(DNS.Section.UPDATE)(3)
+      val dnsRecord2 = sentMessage.getSection(DNS.Section.UPDATE).asScala(3)
       dnsRecord2.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord2.getTTL shouldBe testA.ttl
       dnsRecord2.getType shouldBe DNS.Type.A
@@ -528,7 +528,7 @@ class DnsBackendSpec
       val dnsRecord2Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
       List("1.1.1.1", "2.2.2.2") should contain(dnsRecord2Data)
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -556,14 +556,14 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       dnsRecord.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord.getTTL shouldBe 0
       dnsRecord.getDClass shouldBe DNS.DClass.ANY
       dnsRecord should not be a[DNS.ARecord]
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]
@@ -575,14 +575,14 @@ class DnsBackendSpec
 
       val sentMessage = messageCaptor.getValue
 
-      val dnsRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val dnsRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
       dnsRecord1.getName.toString shouldBe "a-record.vinyldns."
       dnsRecord1.getType shouldBe DNS.Type.A
       dnsRecord1.getTTL shouldBe 0
       dnsRecord1.getDClass shouldBe DNS.DClass.ANY
       dnsRecord1 should not be a[DNS.ARecord]
 
-      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
 
       result shouldBe a[NoError]

modules/api/src/test/scala/vinyldns/api/backend/dns/DnsConversionsSpec.scala

@@ -288,7 +288,7 @@ class DnsConversionsSpec
 
   override protected def beforeEach(): Unit = {
     doReturn(mockMessage).when(mockMessage).clone()
-    doReturn(new Array[DNS.Record](0)).when(mockMessage).getSectionArray(DNS.Section.ADDITIONAL)
+    doReturn(new java.util.ArrayList[DNS.Record]()).when(mockMessage).getSection(DNS.Section.ADDITIONAL)
   }
 
   "Collapsing multiple records to record sets" should {
@@ -572,47 +572,47 @@ class DnsConversionsSpec
   "Converting to an update message" should {
     "work for an Add message" in {
       val dnsMessage = toAddRecordMessage(rrset(testDnsA), testZoneName).right.value
-      val dnsRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val dnsRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala.head
       dnsRecord.getName.toString shouldBe "a-record."
       dnsRecord.getTTL shouldBe testA.ttl
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord shouldBe a[DNS.ARecord]
       dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
 
-      val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
     }
     "work for an Update message" in {
       val dnsMessage =
         toUpdateRecordMessage(rrset(testDnsA), rrset(testDnsAReplace), testZoneName).right.value
       // Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
-      val emptyRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val emptyRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala.head
       emptyRecord.getName.toString shouldBe "a-record-2."
       emptyRecord.getType shouldBe DNS.Type.A
       emptyRecord.getDClass shouldBe DNS.DClass.ANY
 
       // The second section in the replace is the data that is being passed in, this is different than an add
-      val dnsRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(1)
+      val dnsRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala(1)
       dnsRecord.getName.toString shouldBe "a-record."
       dnsRecord.getTTL shouldBe testA.ttl
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord shouldBe a[DNS.ARecord]
       dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
 
-      val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
     }
     "work for a Delete message" in {
       val dnsMessage = toDeleteRecordMessage(rrset(testDnsA), testZoneName).right.value
 
-      val dnsRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(0)
+      val dnsRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala.head
       dnsRecord.getName.toString shouldBe "a-record."
       dnsRecord.getType shouldBe DNS.Type.A
       dnsRecord.getTTL shouldBe 0
       dnsRecord.getDClass shouldBe DNS.DClass.ANY
       dnsRecord should not be a[DNS.ARecord]
 
-      val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE)(0)
+      val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
       zoneRRset.getName.toString shouldBe "vinyldns."
     }
   }

modules/api/src/universal/bin/wait-for-dependencies.sh

@@ -1,28 +0,0 @@
-#!/usr/bin/env bash
-
-# the mysql address, default to a local docker setup
-MYSQL_ADDRESS=${MYSQL_ADDRESS:-vinyldns-mysql}
-MYSQL_PORT=${MYSQL_PORT:-3306}
-echo "Waiting for MYSQL to be ready on $MYSQL_ADDRESS:$MYSQL_PORT"
-DATA=""
-RETRY=30
-while [ $RETRY -gt 0 ]
-do
-    DATA=$(nc -vzw1 $MYSQL_ADDRESS $MYSQL_PORT)
-    if [ $? -eq 0 ]
-    then
-        break
-    else
-        echo "Retrying" >&2
-
-        let RETRY-=1
-        sleep .5
-
-        if [ $RETRY -eq 0 ]
-        then
-          echo "Exceeded retries waiting for MYSQL to be ready on $MYSQL_ADDRESS:$MYSQL_PORT, failing"
-          return 1
-        fi
-    fi
-done
-

modules/mysql/src/main/scala/vinyldns/mysql/MySqlConnector.scala

@@ -27,7 +27,7 @@ object MySqlConnector {
 
   private val logger = LoggerFactory.getLogger("MySqlConnector")
 
-  def runDBMigrations(config: MySqlConnectionConfig): IO[Unit] = {
+  def runDBMigrations(config: MySqlConnectionConfig): IO[Unit] =
     // We can skip migrations for h2, we'll use the test/ddl.sql for initializing
     // that for testing
     if (config.driver.contains("h2")) IO.unit
@@ -61,7 +61,6 @@ object MySqlConnector {
         logger.info("migrations complete")
       }
     }
-  }
 
   def getDataSource(settings: MySqlDataSourceSettings): IO[HikariDataSource] = IO {
 

modules/portal/karma.conf.js

@@ -39,7 +39,7 @@ module.exports = function(config) {
         // level of logging
         // possible values: LOG_DISABLE || LOG_ERROR || LOG_WARN || LOG_INFO || LOG_DEBUG
         logLevel: config.LOG_INFO,
-        
+
         plugins: [
             'karma-jasmine',
             'karma-chrome-launcher',
@@ -66,7 +66,13 @@ module.exports = function(config) {
         // - Safari (only Mac)
         // - PhantomJS
         // - IE (only Windows)
-        browsers: ['ChromeHeadless'],
+        browsers: ['ChromeHeadlessNoSandbox'],
+        customLaunchers: {
+            ChromeHeadlessNoSandbox: {
+                base: 'ChromeHeadless',
+                flags: ['--no-sandbox']
+            }
+        },
 
         // Continuous Integration mode
         // if true, it capture browsers, run tests and exit

modules/r53/src/it/scala/vinyldns/route53/backend/Route53IntegrationSpec.scala

@@ -52,7 +52,7 @@ class Route53IntegrationSpec
           "test",
           Option("access"),
           Option("secret"),
-          "http://127.0.0.1:19009",
+          "http://127.0.0.1:19003",
           "us-east-1"
         )
       )

modules/sqs/src/it/resources/application.conf

@@ -4,10 +4,10 @@ sqs {
   messages-per-poll = 10
   max-retries = 100
   settings = {
-    access-key = "x"
+    access-key = "test"
-    secret-key = "x"
+    secret-key = "test"
-    signing-region = "x"
+    signing-region = "us-east-1"
-    service-endpoint = "http://localhost:19007/"
+    service-endpoint = "http://localhost:19003/"
     queue-name = "sqs-override-name"
   }
 }

modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueIntegrationSpec.scala

@@ -147,13 +147,6 @@ class SqsMessageQueueIntegrationSpec
       result shouldBe empty
     }
 
-    "succeed when attempting to remove item from empty queue" in {
-      queue
-        .remove(SqsMessage(MessageId("does-not-exist"), rsAddChange))
-        .attempt
-        .unsafeRunSync() should beRight(())
-    }
-
     "succeed when attempting to remove item from queue" in {
       queue.send(rsAddChange).unsafeRunSync()
       val result = queue.receive(MessageCount(1).right.value).unsafeRunSync()

modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueProviderIntegrationSpec.scala

@@ -36,7 +36,7 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
           |    max-retries = 100
           |
           |    settings {
-          |      service-endpoint = "http://localhost:19007/"
+          |      service-endpoint = "http://localhost:19003/"
           |      queue-name = "queue-name"
           |    }
           |    """.stripMargin)
@@ -59,8 +59,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
           |    settings {
           |      access-key = "x"
           |      secret-key = "x"
-          |      signing-region = "x"
+          |      signing-region = "us-east-1"
-          |      service-endpoint = "http://localhost:19007/"
+          |      service-endpoint = "http://localhost:19003/"
           |      queue-name = "new-queue"
           |    }
           |    """.stripMargin)
@@ -86,8 +86,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
           |    settings {
           |      access-key = "x"
           |      secret-key = "x"
-          |      signing-region = "x"
+          |      signing-region = "us-east-1"
-          |      service-endpoint = "http://localhost:19007/"
+          |      service-endpoint = "http://localhost:19003/"
           |      queue-name = "bad*queue*name"
           |    }
           |    """.stripMargin)
@@ -108,8 +108,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
           |    settings {
           |      access-key = "x"
           |      secret-key = "x"
-          |      signing-region = "x"
+          |      signing-region = "us-east-1"
-          |      service-endpoint = "http://localhost:19007/"
+          |      service-endpoint = "http://localhost:19003/"
           |      queue-name = "queue.fifo"
           |    }
           |    """.stripMargin)
@@ -131,8 +131,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
           |    settings {
           |      access-key = "x"
           |      secret-key = "x"
-          |      signing-region = "x"
+          |      signing-region = "us-east-1"
-          |      service-endpoint = "http://localhost:19007/"
+          |      service-endpoint = "http://localhost:19003/"
           |      queue-name = "new-queue"
           |    }
           |    """.stripMargin)

project/Dependencies.scala

@@ -33,7 +33,7 @@ object Dependencies {
     "dnsjava"                   %  "dnsjava"                        % "3.4.2",
     "org.apache.commons"        %  "commons-lang3"                  % "3.4",
     "org.apache.commons"        %  "commons-text"                   % "1.4",
-    "org.flywaydb"              %  "flyway-core"                    % "5.1.4",
+    "org.flywaydb"              %  "flyway-core"                    % "8.0.0",
     "org.json4s"                %% "json4s-ext"                     % "3.5.3",
     "org.json4s"                %% "json4s-jackson"                 % "3.5.3",
     "org.scalikejdbc"           %% "scalikejdbc"                    % scalikejdbcV,
@@ -77,7 +77,7 @@ object Dependencies {
   )
 
   lazy val mysqlDependencies = Seq(
-    "org.flywaydb"              %  "flyway-core"                    % "5.1.4",
+    "org.flywaydb"              %  "flyway-core"                    % "8.0.0",
     "org.mariadb.jdbc"          %  "mariadb-java-client"            % "2.3.0",
     "org.scalikejdbc"           %% "scalikejdbc"                    % scalikejdbcV,
     "org.scalikejdbc"           %% "scalikejdbc-config"             % scalikejdbcV,

project/plugins.sbt

@@ -8,8 +8,6 @@ addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
 
 addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.0")
 
-addSbtPlugin("io.github.davidmweber" % "flyway-sbt" % "5.0.0")
-
 addSbtPlugin("org.wartremover" % "sbt-wartremover" % "2.4.10")
 
 addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.25")

test/api/functional/Dockerfile

@@ -0,0 +1,29 @@
+# Build VinylDNS API if the JAR doesn't already exist
+FROM vinyldns/build:base-build as base-build
+ARG DOCKERFILE_PATH="test/api/functional"
+COPY "${DOCKERFILE_PATH}/vinyldns.*" /opt/vinyldns/
+COPY . /build/
+WORKDIR /build
+
+##  Run the build if we don't already have a vinyldns.jar
+RUN if [ ! -f /opt/vinyldns/vinyldns.jar ]; then \
+      env SBT_OPTS="-XX:+UseConcMarkSweepGC -Xmx4G -Xms1G" \
+      sbt -Dbuild.scalafmtOnCompile=false -Dbuild.lintOnCompile=fase ";project api;coverageOff;assembly" \
+      && cp modules/api/target/scala-2.12/vinyldns.jar /opt/vinyldns/; \
+    fi
+
+# Build the testing image, copying data from `vinyldns-api`
+FROM vinyldns/build:base-test
+SHELL ["/bin/bash","-c"]
+ARG DOCKERFILE_PATH
+COPY --from=base-build /opt/vinyldns /opt/vinyldns
+
+# Local bind server files
+COPY docker/bind9/etc/named.conf.* /etc/bind/
+COPY docker/bind9/zones/ /var/bind/
+RUN named-checkconf
+
+# Copy over the functional tests
+COPY  ${DOCKERFILE_PATH}/test /functional_test
+
+ENTRYPOINT ["/bin/bash", "-c", "/initialize.sh bind localstack vinyldns-api && /functional_test/run.sh \"$@\""]

test/api/functional/Dockerfile.dockerignore

@@ -1,6 +1,5 @@
-**/.venv_win
+**/.venv*
 **/.virtualenv
-**/.venv
 **/target
 **/docs
 **/out

test/api/functional/Makefile

@@ -0,0 +1,45 @@
+SHELL=bash
+IMAGE_NAME=vinyldns-api-test
+ROOT_DIR:=$(shell dirname $(realpath  $(lastword $(MAKEFILE_LIST))))
+RELATIVE_ROOT_DIR:=$(shell realpath --relative-to=../../.. $(ROOT_DIR))
+VINYLDNS_JAR_PATH?=modules/api/target/scala-2.12/vinyldns.jar
+
+# Check that the required version of make is being used
+REQ_MAKE_VER:=3.82
+ifneq ($(REQ_MAKE_VER),$(firstword $(sort $(MAKE_VERSION) $(REQ_MAKE_VER))))
+   $(error The version of MAKE $(REQ_MAKE_VER) or higher is required; you are running $(MAKE_VERSION))
+endif
+
+# Extract arguments for `make run`
+EXTRACT_ARGS=true
+ifeq (run,$(firstword $(MAKECMDGOALS)))
+	EXTRACT_ARGS=true
+endif
+ifeq ($(EXTRACT_ARGS),true)
+  # use the rest as arguments for "run"
+  RUN_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
+  # ...and turn them into do-nothing targets
+  $(eval $(RUN_ARGS):;@:)
+endif
+
+
+.ONESHELL:
+
+.PHONY: all build run run-local
+
+all: build run
+
+build:
+	@set -euo pipefail
+	trap 'if [ -f "$(ROOT_DIR)/vinyldns.jar" ]; then rm $(ROOT_DIR)/vinyldns.jar;	fi' EXIT
+	cd ../../..
+	if [ -f modules/api/target/scala-2.12/vinyldns.jar ]; then cp modules/api/target/scala-2.12/vinyldns.jar $(ROOT_DIR)/vinyldns.jar; fi
+	docker build -t $(IMAGE_NAME) $(DOCKER_PARAMS)--build-arg DOCKERFILE_PATH="$(RELATIVE_ROOT_DIR)" -f "$(ROOT_DIR)/Dockerfile" .
+
+run:
+	@set -euo pipefail
+	docker run -it --rm $(DOCKER_PARAMS) -p 9000:9000 -p 19003:19003 -p 19002:19002 -p 19001:19001/tcp -p 19001:19001/udp $(IMAGE_NAME) -- $(RUN_ARGS)
+
+run-local:
+	@set -euo pipefail
+	docker run -it --rm $(DOCKER_PARAMS) -p 9000:9000 -p 19003:19003 -p 19002:19002 -p 19001:19001/tcp -p 19001:19001/udp -v "$$(pwd)/test:/functional_test" $(IMAGE_NAME) -- $(RUN_ARGS)

test/api/functional/test/conftest.py

@@ -29,7 +29,7 @@ def pytest_addoption(parser: _pytest.config.argparsing.Parser) -> None:
     Adds additional options that we can parse when we run the tests, stores them in the parser / py.test context
     """
     parser.addoption("--url", dest="url", action="store", default="http://localhost:9000", help="URL for application to root")
-    parser.addoption("--dns-ip", dest="dns_ip", action="store", default="127.0.0.1", help="The ip address for the dns name server to update")
+    parser.addoption("--dns-ip", dest="dns_ip", action="store", default="127.0.0.1:19001", help="The ip address for the dns name server to update")
     parser.addoption("--resolver-ip", dest="resolver_ip", action="store", help="The ip address for the dns server to use for the tests during resolution. This is usually the same as `--dns-ip`")
     parser.addoption("--dns-zone", dest="dns_zone", action="store", default="vinyldns.", help="The zone name that will be used for testing")
     parser.addoption("--dns-key-name", dest="dns_key_name", action="store", default="vinyldns.", help="The name of the key used to sign updates for the zone")

test/api/functional/test/run.sh

@@ -10,4 +10,9 @@ if [ "$1" == "--update" ]; then
 fi
 
 cd "${ROOT_DIR}"
-"./pytest.sh" "${UPDATE_DEPS}" -n4 --suppress-no-test-exit-code -v live_tests "$@"
+if [ "$1" == "--interactive" ]; then
+  shift
+  bash
+else
+  "./pytest.sh" "${UPDATE_DEPS}" -n4 --suppress-no-test-exit-code -v tests "$@"
+fi

test/api/functional/test/tests/recordsets/create_recordset_test.py

@@ -1,6 +1,6 @@
 import pytest
 
-from live_tests.test_data import TestData
+from tests.test_data import TestData
 from utils import *
 
 

test/api/functional/test/tests/recordsets/delete_recordset_test.py

@@ -1,6 +1,6 @@
 import pytest
 
-from live_tests.test_data import TestData
+from tests.test_data import TestData
 from utils import *