mir/vinyldns
2
0
Fork 0
mirror of https://github.com/VinylDNS/vinyldns synced 2025-08-22 10:10:12 +00:00
Code Issues Releases Wiki Activity

Updates

Browse Source 
- Remove old, unused scripts in `bin/`
- Remove old images from release
  - `test` and `test-bind` are no longer necessary. Test images are in a different repo now
- Remove Docker image creation from sbt build config - actual `Dockerfile` files are easier to deal with

- Update scripts in `bin/` to utilize new Docker images
- Update documentation for changes
- Update all Docker Compose and configuration to use exposed ports on the `integration` image (19001, 19002, etc) both inside the container and outside to make testing more consistent irrespective of method
- Update FlywayDB dependency to v8 to fix a weird logging bug that showed up during integration testing. See: https://github.com/flyway/flyway/issues/2270

- Add `test/api/integration` Docker container definition to be used for any integration testing

- Move `module/api/functional_test` to `test/api/functional` to centralize the "integration-type" external tests and testing utilities

- Move functional testing and integration image to the `test/` folder off of the root to reduce confusion with `bin/` and `docker/`
This commit is contained in:
Emerle, Ryan 2021-10-15 15:06:04 -04:00
parent 0a1b533192
commit 07b683cbd0
123 changed files with 978 additions and 1393 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.gitignore vendored
View File

@ -32,3 +32,7 @@ tmp.out
project/metals.sbt
.bsp
docker/data
**/.virtualenv
**/.venv*
**/*cache*

1
AUTHORS.md
View File

@ -22,6 +22,7 @@ Thank you! If you have contributed in any way, but do not see your name here, pl
- Luke Cori
- Jearvon Dharrie
- Andrew Dunn
- Ryan Emerle
- David Grizzanti
- Alejandro Guirao
- Daniel Jin

234
DEVELOPER_GUIDE.md
View File

@ -1,13 +1,14 @@
# Developer Guide
## Table of Contents
- [Developer Requirements](#developer-requirements)
- [Project Layout](#project-layout)
- [Running VinylDNS Locally](#running-vinyldns-locally)
- [Testing](#testing)
- [Validating VinylDNS](#validating-vinyldns)
## Developer Requirements
- Scala 2.12
- sbt 1+
- Java 8 (at least u162)
@ -21,49 +22,54 @@
Make sure that you have the requirements installed before proceeding.
## Project Layout
[SYSTEM_DESIGN.md](SYSTEM_DESIGN.md) provides a high-level architectural overview of VinylDNS and interoperability of its components.
The main codebase is a multi-module Scala project with multiple sub-modules. To start working with the project,
from the root directory run `sbt`. Most of the code can be found in the `modules` directory.
The following modules are present:
[SYSTEM_DESIGN.md](SYSTEM_DESIGN.md) provides a high-level architectural overview of VinylDNS and interoperability of
its components.
The main codebase is a multi-module Scala project with multiple sub-modules. To start working with the project, from the
root directory run `sbt`. Most of the code can be found in the `modules` directory. The following modules are present:
* `root` - this is the parent project, if you run tasks here, it will run against all sub-modules
* [`core`](#core): core modules that are used by both the API and portal, such as cryptography implementations.
* [`api`](#api): the API is the main engine for all of VinylDNS. This is the most active area of the codebase, as everything else typically just funnels through
the API.
* [`portal`](#portal): The portal is a user interface wrapper around the API. Most of the business rules, logic, and processing can be found in the API. The
_only_ features in the portal not found in the API are creation of users and user authentication.
* [`api`](#api): the API is the main engine for all of VinylDNS. This is the most active area of the codebase, as
everything else typically just funnels through the API.
* [`portal`](#portal): The portal is a user interface wrapper around the API. Most of the business rules, logic, and
processing can be found in the API. The
_only_ features in the portal not found in the API are creation of users and user authentication.
* [`docs`](#documentation): documentation for VinylDNS.
### Core
Code that is used across multiple modules in the VinylDNS ecosystem live in `core`.
#### Code Layout
* `src/main` - the main source code
* `src/test` - unit tests
### API
The API is the RESTful API for interacting with VinylDNS. The following technologies are used:
* [Akka HTTP](https://doc.akka.io/docs/akka-http/current/) - Used primarily for REST and HTTP calls.
* [FS2](https://functional-streams-for-scala.github.io/fs2/) - Used for backend change processing off of message queues.
FS2 has back-pressure built in, and gives us tools like throttling and concurrency.
FS2 has back-pressure built in, and gives us tools like throttling and concurrency.
* [Cats Effect](https://typelevel.org/cats-effect/) - We are currently migrating away from `Future` as our primary type
and towards cats effect IO. Hopefully, one day, all the things will be using IO.
and towards cats effect IO. Hopefully, one day, all the things will be using IO.
* [Cats](https://typelevel.org/cats) - Used for functional programming.
* [PureConfig](https://pureconfig.github.io/) - For loading configuration values. We are currently migrating to
use PureConfig everywhere. Not all the places use it yet.
* [PureConfig](https://pureconfig.github.io/) - For loading configuration values. We are currently migrating to use
PureConfig everywhere. Not all the places use it yet.
The API has the following dependencies:
* MySQL - the SQL database that houses zone data
* DynamoDB - where all of the other data is stored
* MySQL - the SQL database that houses the data
* SQS - for managing concurrent updates and enabling high-availability
* Bind9 - for testing integration with a real DNS system
#### Code Layout
The API code can be found in `modules/api`.
* `functional_test` - contains the python black box / regression tests
* `src/it` - integration tests
* `src/main` - the main source code
* `src/test` - unit tests
@ -72,26 +78,30 @@ The API code can be found in `modules/api`.
The package structure for the source code follows:
* `vinyldns.api.domain` - contains the core front-end logic. This includes things like the application services,
repository interfaces, domain model, validations, and business rules.
repository interfaces, domain model, validations, and business rules.
* `vinyldns.api.engine` - the back-end processing engine. This is where we process commands including record changes,
zone changes, and zone syncs.
zone changes, and zone syncs.
* `vinyldns.api.protobuf` - marshalling and unmarshalling to and from protobuf to types in our system
* `vinyldns.api.repository` - repository implementations live here
* `vinyldns.api.route` - HTTP endpoints
### Portal
The project is built using:
* [Play Framework](https://www.playframework.com/documentation/2.6.x/Home)
* [AngularJS](https://angularjs.org/)
The portal is _mostly_ a shim around the API. Most actions in the user interface are translated into API calls.
The features that the Portal provides that are not in the API include:
* Authentication against LDAP
* Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials for them in the
database with their LDAP information.
* Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials
for them in the database with their LDAP information.
#### Code Layout
The portal code can be found in `modules/portal`.
* `app` - source code for portal back-end
@ -108,38 +118,54 @@ The portal code can be found in `modules/portal`.
* `test` - unit tests for portal back-end
### Documentation
Code used to build the microsite content for the API, operator and portal guides at https://www.vinyldns.io/. Some settings for the microsite
are also configured in `build.sbt` of the project root.
Code used to build the microsite content for the API, operator and portal guides at https://www.vinyldns.io/. Some
settings for the microsite are also configured in `build.sbt` of the project root.
#### Code Layout
* `src/main/resources` - Microsite resources and configurations
* `src/main/tut` - Content for microsite web pages
## Running VinylDNS Locally
VinylDNS can be started in the background by running the [quickstart instructions](README.md#quickstart) located in the README. However, VinylDNS
can also be run in the foreground.
VinylDNS can be started in the background by running the [quickstart instructions](README.md#quickstart) located in the
README. However, VinylDNS can also be run in the foreground.
### Starting the API Server
To start the API for integration, functional, or portal testing. Start up sbt by running `sbt` from the root directory.
* `dockerComposeUp` to spin up the dependencies on your machine from the root project.
Before starting the API service, you can start the dependencies for local development:
```
cd test/api/integration
make build && make run-bg
```
This will start a container running in the background with necessary prerequisites.
Once the prerequisites are running, you can start up sbt by running `sbt` from the root directory.
* `project api` to change the sbt project to the API
* `reStart` to start up the API server
* Wait until you see the message `VINYLDNS SERVER STARTED SUCCESSFULLY` before working with the server
* To stop the VinylDNS server, run `reStop` from the api project
* To stop the dependent Docker containers, change to the root project `project root`, then run `dockerComposeStop` from the API project
* To stop the dependent Docker containers, change to the root project `project root`, then run `dockerComposeStop` from
the API project
See the [API Configuration Guide](https://www.vinyldns.io/operator/config-api) for information regarding API configuration.
See the [API Configuration Guide](https://www.vinyldns.io/operator/config-api) for information regarding API
configuration.
### Starting the Portal
To run the portal locally, you _first_ have to start up the VinylDNS API Server (see instructions above). Once
that is done, in the same `sbt` session or a different one, go to `project portal` and then execute `;preparePortal; run`.
See the [Portal Configuration Guide](https://www.vinyldns.io/operator/config-portal) for information regarding portal configuration.
To run the portal locally, you _first_ have to start up the VinylDNS API Server (see instructions above). Once that is
done, in the same `sbt` session or a different one, go to `project portal` and then execute `;preparePortal; run`.
See the [Portal Configuration Guide](https://www.vinyldns.io/operator/config-portal) for information regarding portal
configuration.
### Loading test data
Normally the portal can be used for all VinylDNS requests. Test users are locked down to only have access to test zones,
which the portal connection modal has not been updated to incorporate. To connect to a zone with testuser, you will need to use an alternative
client and set `isTest=true` on the zone being connected to.
which the portal connection modal has not been updated to incorporate. To connect to a zone with testuser, you will need
to use an alternative client and set `isTest=true` on the zone being connected to.
Use the vinyldns-js client (Note, you need Node installed):
@ -159,78 +185,95 @@ You should now be able to see the zone in the portal at localhost:9001 when logg
```
## Testing
### Unit Tests
1. First, start up your Scala build tool: `sbt`. Running *clean* immediately after starting is recommended.
1. (Optionally) Go to the project you want to work on, for example `project api` for the API; `project portal` for the portal.
1. (Optionally) Go to the project you want to work on, for example `project api` for the API; `project portal` for the
portal.
1. Run _all_ unit tests by just running `test`.
1. Run an individual unit test by running `testOnly *MySpec`.
1. If you are working on a unit test and production code at the same time, use `~` (eg. `~testOnly *MySpec`) to automatically background compile for you!
1. If you are working on a unit test and production code at the same time, use `~` (e.g., `~testOnly *MySpec`) to
automatically background compile for you!
### Integration Tests
Integration tests are used to test integration with _real_ dependent services. We use Docker to spin up those
backend services for integration test development.
Integration tests are used to test integration with _real_ dependent services. We use Docker to spin up those backend
services for integration test development.
1. Type `dockerComposeUp` to start up dependent background services
1. Go to the target module in sbt, example: `project api`
1. Run all integration tests by typing `it:test`.
1. Run an individual integration test by typing `it:testOnly *MyIntegrationSpec`
1. You can background compile as well if working on a single spec by using `~it:testOnly *MyIntegrationSpec`
1. You must stop (`dockerComposeStop`) and start (`dockerComposeUp`) the dependent services from the root project (`project root`) before you rerun the tests.
1. For the mysql module, you may need to wait up to 30 seconds after starting the services before running the tests for setup to complete.
1. You must stop (`dockerComposeStop`) and start (`dockerComposeUp`) the dependent services from the root
project (`project root`) before you rerun the tests.
1. For the mysql module, you may need to wait up to 30 seconds after starting the services before running the tests for
setup to complete.
#### Running both
You can run all unit and integration tests for the api and portal by running `sbt verify`
### Functional Tests
When adding new features, you will often need to write new functional tests that black box / regression test the
API. We have over 350 (and growing) automated regression tests. The API functional tests are written in Python
and live under `modules/api/functional_test`.
#### Running functional tests
To run functional tests, make sure that you have started the API server (directions above).
Then in another terminal session:
When adding new features, you will often need to write new functional tests that black box / regression test the API.
1. `cd modules/api/functional_test`
1. `./run.py live_tests -v`
- The API functional tests are written in Python and live under `test/api/functional`.
The Portal functional tests are written in JavaScript and live under `test/portal/functional`.
You can run a specific test by name by running `./run.py live_tests -v -k <name of test function>`
#### Running Functional Tests
You run specific tests for a portion of the project, say recordsets, by running `./run.py live_tests/recordsets -v`
To run functional tests you can simply execute the following command:
#### Our Setup
We use [pytest](https://docs.pytest.org/en/latest/) for python tests. It is helpful that you browse the documentation
so that you are familiar with pytest and how our functional tests operate.
```
make build && make run
```
During iterative test development, you can use `make run-local` which will mount the current functional tests in the
container, allowing for easier test development.
We also use [PyHamcrest](https://pyhamcrest.readthedocs.io/en/release-1.8/) for matchers in order to write easy
to read tests. Please browse that documentation as well so that you are familiar with the different matchers
for PyHamcrest. There aren't a lot, so it should be quick.
Additionally, you can pass `--interactive` to `make run` or `make run-local` to drop to a shell inside the container.
From there you can run tests with the `/functional_test/run.sh` command. This allows for finer-grained control over the
test execution process as well as easier inspection of logs.
In the `modules/api/functional_test` directory are a few important files for you to be familiar with:
##### API Functional Tests
You can run a specific test by name by running `make run -- -k <name of test function>`. Any arguments after
`make run --` will be passed to the test runner [`test/api/functional/run.sh`](test/api/functional/run.sh).
* vinyl_client.py - this provides the interface to the VinylDNS API. It handles signing the request for you, as well
as building and executing the requests, and giving you back valid responses. For all new API endpoints, there should
be a corresponding function in the vinyl_client
* utils.py - provides general use functions that can be used anywhere in your tests. Feel free to contribute new
functions here when you see repetition in the code
Functional tests run on every build, and are designed to work _in every environment_. That means locally, in Docker,
and in production environments.
In the `modules/api/functional_test/live_tests` directory, we have directories / modules for different areas of the application.
#### Setup
* membership - for managing groups and users
* recordsets - for managing record sets
* zones - for managing zones
* internal - for internal endpoints (not intended for public consumption)
* batch - for managing batch updates
We use [pytest](https://docs.pytest.org/en/latest/) for python tests. It is helpful that you browse the documentation so
that you are familiar with pytest and how our functional tests operate.
We also use [PyHamcrest](https://pyhamcrest.readthedocs.io/en/release-1.8/) for matchers in order to write easy to read
tests. Please browse that documentation as well so that you are familiar with the different matchers for PyHamcrest.
There aren't a lot, so it should be quick.
In the `test/api/functional` directory are a few important files for you to be familiar with:
* `vinyl_client.py` - this provides the interface to the VinylDNS API. It handles signing the request for you, as well
as building and executing the requests, and giving you back valid responses. For all new API endpoints, there should
be a corresponding function in the vinyl_client
* `utils.py` - provides general use functions that can be used anywhere in your tests. Feel free to contribute new
functions here when you see repetition in the code
In the `test/api/functional/tests` directory, we have directories / modules for different areas of the application.
* `batch` - for managing batch updates
* `internal` - for internal endpoints (not intended for public consumption)
* `membership` - for managing groups and users
* `recordsets` - for managing record sets
* `zones` - for managing zones
##### Functional Test Context
Our func tests use pytest contexts. There is a main test context that lives in `shared_zone_test_context.py`
that creates and tears down a shared test context used by many functional tests. The
beauty of pytest is that it will ensure that the test context is stood up exactly once, then all individual tests
that use the context are called using that same context.
Our functional tests use `pytest` contexts. There is a main test context that lives in `shared_zone_test_context.py`
that creates and tears down a shared test context used by many functional tests. The beauty of pytest is that it will
ensure that the test context is stood up exactly once, then all individual tests that use the context are called using
that same context.
The shared test context sets up several things that can be reused:
@ -243,30 +286,33 @@ The shared test context sets up several things that can be reused:
1. A classless IPv4 reverse zone
1. A parent zone that has child zones - used for testing NS record management and zone delegations
##### Partitioning
Each of the test zones are configured in a `partition`. By default, there are four partitions. These partitions are
effectively copies of the zones so that parallel tests can run without interfering with one another.
For instance, there are four zones for the `ok` zone: `ok1`, `ok2`, `ok3`, and `ok4`. The functional tests will handle
distributing which zone is being used by which of the parallel test runners.
As such, you should **never** hardcode the name of the zone. Always get the zone from the `shared_zone_test_context`.
For instance, to get the `ok` zone, you would write:
```python
zone = shared_zone_test_context.ok_zone
zone_name = shared_zone_test_context.ok_zone["name"]
zone_id = shared_zone_test_context.ok_zone["id"]
```
##### Really Important Test Context Rules!
1. Try to use the `shared_zone_test_context` whenever possible! This reduces the time
it takes to run functional tests (which is in minutes).
1. Limit changes to users, groups, and zones in the shared test context, as doing so could impact downstream tests
1. Try to use the `shared_zone_test_context` whenever possible! This reduces the time it takes to run functional
tests (which is in minutes).
1. Be mindful of changes to users, groups, and zones in the shared test context, as doing so could impact downstream
tests
1. If you do modify any entities in the shared zone context, roll those back when your function completes!
##### Managing Test Zone Files
When functional tests are run, we spin up several Docker containers. One of the Docker containers is a Bind9 DNS
server. If you need to add or modify the test DNS zone files, you can find them in
When functional tests are run, we spin up several Docker containers. One of the Docker containers is a Bind9 DNS server.
If you need to add or modify the test DNS zone files, you can find them in
`docker/bind9/zones`
## Validating VinylDNS
VinylDNS comes with a build script `./build.sh` that validates VinylDNS compiles, verifies that unit tests pass, and then runs functional tests.
Note: This takes a while to run, and typically is only necessary if you want to simulate the same process that runs on the build servers.
When functional tests run, you will see a lot of output intermingled together across the various containers. You can view only the output
of the functional tests at `target/vinyldns-functest.log`. If you want to see the Docker log output from any one container, you can view
them after the tests complete at:
* `target/vinyldns-api.log` - the API server logs
* `target/vinyldns-bind9.log` - the Bind9 DNS server logs
* `target/vinyldns-elasticmq.log` - the ElasticMQ (SQS) server logs
* `target/vinyldns-functest.log` - the output of running the functional tests
* `target/vinyldns-mysql.log` - the MySQL server logs
When the func tests complete, the entire Docker setup will be automatically torn down.

83
MAINTAINERS.md
View File

@ -2,12 +2,6 @@
## Table of Contents
* [Docker Content Trust](#docker-content-trust)
* [Docker Hub Account](#docker-hub-account)
* [Delegating Image Signing](#delegating-image-signing)
* [Setting up Notary](#setting-up-notary)
* [Generating a Personal Delegation Key](#generating-a-personal-delegation-key)
* [Adding a Delegation Key To a Repository](#adding-a-delegation-key-to-a-repository)
* [Pushing a Signed Image with your Delegation Key](#pushing-a-signed-image-with-your-delegation-key)
* [Sonatype Credentials](#sonatype-credentials)
* [Release Process](#release-process)
@ -26,8 +20,6 @@ the [vinyldns organization](https://hub.docker.com/u/vinyldns/dashboard/). Namel
* vinyldns/api: images for vinyldns core api engine
* vinyldns/portal: images for vinyldns web client
* vinyldns/bind9: images for local DNS server used for testing
* vinyldns/test-bind9: contains the setup to run functional tests
* vinyldns/test: has the actual functional tests pinned to a version of VinylDNS
The offline root key and repository keys are managed by the core maintainer team. The keys managed are:
@ -35,8 +27,6 @@ The offline root key and repository keys are managed by the core maintainer team
* api key: used to sign tagged images in vinyldns/api
* portal key: used to sign tagged images in vinyldns/portal
* bind9 key: used to sign tagged images in the vinyldns/bind9
* test-bind9 key: used to sign tagged images in the vinyldns/test-bind9
* test key: used to sign tagged images in the vinyldns/test
These keys are named in a <hash>.key format, e.g. 5526ecd15bd413e08718e66c440d17a28968d5cd2922b59a17510da802ca6572.key,
do not change the names of the keys.
@ -44,77 +34,6 @@ do not change the names of the keys.
Docker expects these keys to be saved in `~/.docker/trust/private`. Each key is encrypted with a passphrase, that you
must have available when pushing an image.
### Docker Hub Account
If you don't have one already, make an account on Docker Hub. Get added as a Collaborator to vinyldns/api, vinyldns/portal,
and vinyldns/bind9
### Delegating Image Signing
Someone with our keys can sign images when pushing, but instead of sharing those keys we can utilize
notary to delegate image signing permissions in a safer way. Notary will have you make a public-private key pair and
upload your public key. This way you only need your private key, and a developer's permissions can easily be revoked.
#### Setting up Notary
If you do not already have notary:
1. Download the latest release for your machine at https://github.com/theupdateframework/notary/releases,
for example, on a mac download the precompiled binary `notary-Darwin-amd64`
1. Rename the binary to notary, and choose a location where it will live,
e.g. `cd ~/Downloads/; mv notary-Darwin-amd64 notary; mv notary ~/Documents/notary; cd ~/Documents`
1. Make it executable, e.g. `chmod +x notary`
1. Add notary to your path, e.g. `vim ~/.bashrc`, add `export PATH="$PATH":<path to notary>`
1. Create a `~/.notary/config.json` with
```
{
"trust_dir" : "~/.docker/trust",
"remote_server": {
"url": "https://notary.docker.io"
}
}
```
You can test notary with `notary -s https://notary.docker.io -d ~/.docker/trust list docker.io/vinyldns/api`, in which
you should see tagged images for the VinylDNS API
> Note: you'll pretty much always use the `-s https://notary.docker.io -d ~/.docker/trust` args when running notary,
it will be easier for you to alias a command like `notarydefault` to `notary -s https://notary.docker.io -d ~/.docker/trust`
in your `.bashrc`
#### Generating a Personal Delegation Key
1. `cd` to a directory where you will save your delegation keys and certs
1. Generate your private key: `openssl genrsa -out delegation.key 2048`
1. Generate your public key: `openssl req -new -sha256 -key delegation.key -out delegation.csr`, all fields are optional,
but when it gets to your email it makes sense to add that
1. Self-sign your public key (valid for one year):
`openssl x509 -req -sha256 -days 365 -in delegation.csr -signkey delegation.key -out delegation.crt`
1. Change the `delegation.crt` to some unique name, like `my-name-vinyldns-delegation.crt`
1. Give your `my-name-vinyldns-delegation.crt` to someone that has the root keys and passphrases so
they can upload your delegation key to the repository
#### Adding a Delegation Key to a Repository
This expects you to have the root keys and passphrases for the Docker repositories
1. List current keys: `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`
1. Add team member's public key: `notary delegation add docker.io/vinyldns/api targets/releases <team members delegation crt path> --all-paths`
1. Push key: `notary publish docker.io/vinyldns/api`
1. Repeat above steps for `docker.io/vinyldns/portal`
Add their key ID to the table below, it can be viewed with `notary -s https://notary.docker.io -d ~/.docker/trust delegation list docker.io/vinyldns/api`.
It will be the one that didn't show up when you ran step one of this section
| Name | Key ID |
|----------------|------------------------------------------------------------------
| Nima Eskandary | 66027c822d68133da859f6639983d6d3d9643226b3f7259fc6420964993b499a, cdca33de91c54f801d89240d18b5037e274461ba1c88c10451070c97e9f665b4 |
| Rebecca Star | 04285e24d3b9a8b614b34da229669de1f75c9faa471057e8b4a7d60aac0d5bf5 |
| Michael Ly |dd3a5938fc927de087ad4b59d6ac8f62b6502d05b2cc9b0623276cbac7dbf05b |
#### Pushing a Signed Image with your Delegation Key
1. Run `notary key import <path to private delegation key> --role user`
1. You will have to create a passphrase for this key that encrypts it at rest. Use a password generator to make a
strong password and save it somewhere safe, like apple keychain or some other password manager
1. From now on `docker push` will be try to sign images with the delegation key if it was configured for that Docker
repository
## Sonatype Credentials
@ -163,7 +82,7 @@ running the release
1. Run `bin/release.sh` _Note: the arg "skip-tests" will skip unit, integration and functional testing before a release_
1. You will be asked to confirm the version which originally comes from `version.sbt`. _NOTE: if the version ends with
`SNAPSHOT`, then the docker latest tag won't be applied and the core module will only be published to the sonatype
staging repo.
staging repo._
1. When it comes to the sonatype stage, you will need the passphrase handy for the signing key, [Sonatype Credentials](#sonatype-credentials)
1. Assuming things were successful, make a pr since sbt release auto-bumped `version.sbt` and made a commit for you
1. Run `./build/docker-release.sh --branch [TAG CREATED FROM PREVIOUS STEP, e.g. v0.9.3] --clean --push`

32
README.md
View File

@ -1,7 +1,5 @@
[![Join the chat at https://gitter.im/vinyldns](https://badges.gitter.im/vinyldns/vinyldns.svg)](https://gitter.im/vinyldns)
![Build](https://github.com/vinyldns/vinyldns/workflows/Continuous%20Integration/badge.svg)
[![CodeCov ](https://codecov.io/gh/vinyldns/vinyldns/branch/master/graph/badge.svg)](https://codecov.io/gh/vinyldns/vinyldns)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/2682/badge)](https://bestpractices.coreinfrastructure.org/projects/2682)
[![License](https://img.shields.io/github/license/vinyldns/vinyldns)](https://github.com/vinyldns/vinyldns/blob/master/LICENSE)
[![conduct](https://img.shields.io/badge/%E2%9D%A4-code%20of%20conduct-blue.svg)](https://github.com/vinyldns/vinyldns/blob/master/CODE_OF_CONDUCT.md)
@ -23,17 +21,16 @@ secure RESTful API, and integration with infrastructure automation tools like An
It is designed to integrate with your existing DNS infrastructure, and provides extensibility to fit your installation.
VinylDNS helps secure DNS management via:
* AWS Sig4 signing of all messages to ensure that the message that was sent was not altered in transit
* Throttling of DNS updates to rate limit concurrent updates against your DNS systems
* Encrypting user secrets and TSIG keys at rest and in-transit
* Recording every change made to DNS records and zones
- AWS Sig4 signing of all messages to ensure that the message that was sent was not altered in transit
- Throttling of DNS updates to rate limit concurrent updates against your DNS systems
- Encrypting user secrets and TSIG keys at rest and in-transit
- Recording every change made to DNS records and zones
Integration is simple with first-class language support including:
* java
* ruby
* python
* go-lang
* javascript
- Java
- Python
- Go
- JavaScript
## Table of Contents
- [Quickstart](#quickstart)
@ -59,7 +56,7 @@ There exist several clients at <https://github.com/vinyldns> that can be used to
## Things to try in the portal
1. View the portal at <http://localhost:9001> in a web browser
1. Login with the credentials ***professor*** and ***professor***
1. Login with the credentials `testuser` and `testpassword`
1. Navigate to the `groups` tab: <http://localhost:9001/groups>
1. Click on the **New Group** button and create a new group, the group id is the uuid in the url after you view the group
1. View zones you connected to in the `zones` tab: <http://localhost:9001/zones>. For a quick test, create a new zone named "ok" with an email of "test@test.com" and choose a group you created from the previous step. (Note, see [Developer Guide](DEVELOPER_GUIDE.md#loading-test-data) for creating a zone)
@ -79,7 +76,7 @@ TTL = 300, IP Addressess = 1.1.1.1`
1. A similar `docker/.env.quickstart` can be modified to change the default ports for the Portal and API. You must also modify their config files with the new port: https://www.vinyldns.io/operator/config-portal & https://www.vinyldns.io/operator/config-api
## Code of Conduct
This project and everyone participating in it are governed by the [VinylDNS Code Of Conduct](CODE_OF_CONDUCT.md). By
This project, and everyone participating in it, are governed by the [VinylDNS Code Of Conduct](CODE_OF_CONDUCT.md). By
participating, you agree to this Code. Please report any violations to the code of conduct to vinyldns-core@googlegroups.com.
## Developer Guide
@ -89,15 +86,14 @@ See [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md) for instructions on setting up Viny
See the [Contributing Guide](CONTRIBUTING.md).
## Contact
- [Gitter](https://gitter.im/vinyldns)
- If you have any security concerns please contact the maintainers directly vinyldns-core@googlegroups.com
## Maintainers and Contributors
The current maintainers (people who can merge pull requests) are:
- Paul Cleary
- Ryan Emerle
- Sriram Ramakrishnan
- Jim Wakemen
- Ryan Emerle ([@remerle](https://github.com/remerle))
- Sriram Ramakrishnan ([@sramakr](https://github.com/sramakr))
- Jim Wakemen ([@jwakemen](https://github.com/jwakemen))
See [AUTHORS.md](AUTHORS.md) for the full list of contributors to VinylDNS.

34
bin/build.sh
View File

@ -1,34 +0,0 @@
#!/usr/bin/env bash
DIR=$( cd $(dirname $0) ; pwd -P )
echo "Verifying code..."
#${DIR}/verify.sh
#step_result=$?
step_result=0
if [ ${step_result} != 0 ]
then
echo "Failed to verify build!!!"
exit ${step_result}
fi
echo "Func testing the api..."
${DIR}/func-test-api.sh
step_result=$?
if [ ${step_result} != 0 ]
then
echo "Failed API func tests!!!"
exit ${step_result}
fi
echo "Func testing the portal..."
${DIR}/func-test-portal.sh
step_result=$?
if [ ${step_result} != 0 ]
then
echo "Failed Portal func tests!!!"
exit ${step_result}
fi
exit 0

10
bin/docker-publish-api.sh
View File

@ -1,10 +0,0 @@
#!/usr/bin/env bash
DIR=$( cd $(dirname $0) ; pwd -P )
cd $DIR/../
echo "Publishing docker image..."
sbt clean docker:publish
publish_result=$?
cd $DIR
exit ${publish_result}

58
bin/docker-up-api-server.sh
View File

@ -1,58 +0,0 @@
#!/bin/bash
######################################################################
# Copies the contents of `docker` into target/scala-2.12
# to start up dependent services via docker compose. Once
# dependent services are started up, the fat jar built by sbt assembly
# is loaded into a docker container. The api will be available
# by default on port 9000
######################################################################
DIR=$( cd $(dirname $0) ; pwd -P )
set -a # Required in order to source docker/.env
# Source customizable env files
source "$DIR"/.env
source "$DIR"/../docker/.env
WORK_DIR="$DIR"/../target/scala-2.12
mkdir -p "$WORK_DIR"
echo "Copy all Docker to the target directory so we can start up properly and the Docker context is small..."
cp -af "$DIR"/../docker "$WORK_DIR"/
echo "Copy the vinyldns.jar to the API Docker folder so it is in context..."
if [[ ! -f "$DIR"/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
echo "vinyldns.jar not found, building..."
cd "$DIR"/../
sbt api/clean api/assembly
cd "$DIR"
fi
cp -f "$DIR"/../modules/api/target/scala-2.12/vinyldns.jar "$WORK_DIR"/docker/api
echo "Starting API server and all dependencies in the background..."
docker-compose -f "$WORK_DIR"/docker/docker-compose-func-test.yml --project-directory "$WORK_DIR"/docker up --build -d api
echo "Waiting for API to be ready at ${VINYLDNS_API_URL} ..."
DATA=""
RETRY=40
while [ "$RETRY" -gt 0 ]
do
DATA=$(curl -I -s "${VINYLDNS_API_URL}/ping" -o /dev/null -w "%{http_code}")
if [ $? -eq 0 ]
then
echo "Succeeded in connecting to VinylDNS API!"
break
else
echo "Retrying" >&2
let RETRY-=1
sleep 1
if [ "$RETRY" -eq 0 ]
then
echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
exit 1
fi
fi
done

5
bin/docker-up-dns-server.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
DIR=$( cd $(dirname $0) ; pwd -P )
echo "Starting ONLY the bind9 server. To start an api server use the api server script"
docker-compose -f $DIR/../docker/docker-compose-func-test.yml --project-directory $DIR/../docker up -d bind9

57
bin/func-test-api-testbind9.sh
View File

@ -1,57 +0,0 @@
#!/bin/bash
######################################################################
# Copies the contents of `docker` into target/scala-2.12
# to start up dependent services via docker compose. Once
# dependent services are started up, the fat jar built by sbt assembly
# is loaded into a docker container. Finally, the func tests run inside
# another docker container
# At the end, we grab all the logs and place them in the target
# directory
######################################################################
DIR=$( cd $(dirname $0) ; pwd -P )
WORK_DIR=$DIR/../target/scala-2.12
mkdir -p $WORK_DIR
echo "Cleaning up unused networks..."
docker network prune -f
echo "Copy all docker to the target directory so we can start up properly and the docker context is small..."
cp -af $DIR/../docker $WORK_DIR/
echo "Copy over the functional tests as well as those that are run in a container..."
mkdir -p $WORK_DIR/functest
rsync -av --exclude='.virtualenv' $DIR/../modules/api/functional_test $WORK_DIR/docker/functest
echo "Copy the vinyldns.jar to the api docker folder so it is in context..."
if [[ ! -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
echo "vinyldns jar not found, building..."
cd $DIR/../
sbt api/clean api/assembly
cd $DIR
fi
cp -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar $WORK_DIR/docker/api
echo "Starting docker environment and running func tests..."
# If PAR_CPU is unset; default to auto
if [ -z "${PAR_CPU}" ]; then
export PAR_CPU=auto
fi
docker-compose -f $WORK_DIR/docker/docker-compose-func-test-testbind9.yml --project-directory $WORK_DIR/docker --log-level ERROR up --build --exit-code-from functest
test_result=$?
echo "Grabbing the logs..."
docker logs vinyldns-api > $DIR/../target/vinyldns-api.log 2>/dev/null
docker logs vinyldns-bind9 > $DIR/../target/vinyldns-bind9.log 2>/dev/null
docker logs vinyldns-mysql > $DIR/../target/vinyldns-mysql.log 2>/dev/null
docker logs vinyldns-elasticmq > $DIR/../target/vinyldns-elasticmq.log 2>/dev/null
docker logs vinyldns-functest > $DIR/../target/vinyldns-functest.log 2>/dev/null
echo "Cleaning up docker containers..."
$DIR/./remove-vinyl-containers.sh
echo "Func tests returned result: ${test_result}"
exit ${test_result}

57
bin/func-test-api-travis.sh
View File

@ -1,57 +0,0 @@
#!/bin/bash
######################################################################
# Copies the contents of `docker` into target/scala-2.12
# to start up dependent services via docker compose. Once
# dependent services are started up, the fat jar built by sbt assembly
# is loaded into a docker container. Finally, the func tests run inside
# another docker container
# At the end, we grab all the logs and place them in the target
# directory
######################################################################
DIR=$( cd $(dirname $0) ; pwd -P )
WORK_DIR=$DIR/../target/scala-2.12
mkdir -p $WORK_DIR
echo "Cleaning up unused networks..."
docker network prune -f
echo "Copy all docker to the target directory so we can start up properly and the docker context is small..."
cp -af $DIR/../docker $WORK_DIR/
echo "Copy over the functional tests as well as those that are run in a container..."
mkdir -p $WORK_DIR/functest
rsync -av --exclude='.virtualenv' $DIR/../modules/api/functional_test $WORK_DIR/docker/functest
echo "Copy the vinyldns.jar to the api docker folder so it is in context..."
if [[ ! -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
echo "vinyldns jar not found, building..."
cd $DIR/../
sbt build-api
cd $DIR
fi
cp -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar $WORK_DIR/docker/api
echo "Starting docker environment and running func tests..."
if [ -z "${PAR_CPU}" ]; then
export PAR_CPU=2
fi
docker-compose -f $WORK_DIR/docker/docker-compose-func-test.yml --project-directory $WORK_DIR/docker up --build --exit-code-from functest
test_result=$?
echo "Grabbing the logs..."
docker logs vinyldns-functest
docker logs vinyldns-api > $DIR/../target/vinyldns-api.log 2>/dev/null
docker logs vinyldns-bind9 > $DIR/../target/vinyldns-bind9.log 2>/dev/null
docker logs vinyldns-mysql > $DIR/../target/vinyldns-mysql.log 2>/dev/null
docker logs vinyldns-elasticmq > $DIR/../target/vinyldns-elasticmq.log 2>/dev/null
docker logs vinyldns-dynamodb > $DIR/../target/vinyldns-dynamodb.log 2>/dev/null
docker logs vinyldns-functest > $DIR/../target/vinyldns-functest.log 2>/dev/null
echo "Cleaning up docker containers..."
$DIR/./remove-vinyl-containers.sh
echo "Func tests returned result: ${test_result}"
exit ${test_result}

60
bin/func-test-api.sh
View File

@ -1,57 +1,7 @@
#!/bin/bash
######################################################################
# Copies the contents of `docker` into target/scala-2.12
# to start up dependent services via docker compose. Once
# dependent services are started up, the fat jar built by sbt assembly
# is loaded into a docker container. Finally, the func tests run inside
# another docker container
# At the end, we grab all the logs and place them in the target
# directory
######################################################################
#!/usr/bin/env bash
set -euo pipefail
DIR=$( cd $(dirname $0) ; pwd -P )
WORK_DIR=$DIR/../target/scala-2.12
mkdir -p $WORK_DIR
DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
echo "Cleaning up unused networks..."
docker network prune -f
echo "Copy all docker to the target directory so we can start up properly and the docker context is small..."
cp -af $DIR/../docker $WORK_DIR/
echo "Copy over the functional tests as well as those that are run in a container..."
mkdir -p $WORK_DIR/functest
rsync -av --exclude='.virtualenv' $DIR/../modules/api/functional_test $WORK_DIR/docker/functest
echo "Copy the vinyldns.jar to the api docker folder so it is in context..."
if [[ ! -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar ]]; then
echo "vinyldns jar not found, building..."
cd $DIR/../
sbt api/clean api/assembly
cd $DIR
fi
cp -f $DIR/../modules/api/target/scala-2.12/vinyldns.jar $WORK_DIR/docker/api
echo "Starting docker environment and running func tests..."
# If PAR_CPU is unset; default to auto
if [ -z "${PAR_CPU}" ]; then
export PAR_CPU=auto
fi
docker-compose -f $WORK_DIR/docker/docker-compose-func-test.yml --project-directory $WORK_DIR/docker --log-level ERROR up --build --exit-code-from functest
test_result=$?
echo "Grabbing the logs..."
docker logs vinyldns-api > $DIR/../target/vinyldns-api.log 2>/dev/null
docker logs vinyldns-bind9 > $DIR/../target/vinyldns-bind9.log 2>/dev/null
docker logs vinyldns-mysql > $DIR/../target/vinyldns-mysql.log 2>/dev/null
docker logs vinyldns-elasticmq > $DIR/../target/vinyldns-elasticmq.log 2>/dev/null
docker logs vinyldns-functest > $DIR/../target/vinyldns-functest.log 2>/dev/null
echo "Cleaning up docker containers..."
$DIR/./remove-vinyl-containers.sh
echo "Func tests returned result: ${test_result}"
exit ${test_result}
cd "$DIR/../test/api/functional"
make

49
bin/func-test-portal.sh
View File

@ -1,46 +1,7 @@
#!/bin/bash
######################################################################
# Runs e2e tests against the portal
######################################################################
#!/usr/bin/env bash
set -euo pipefail
DIR=$( cd $(dirname $0) ; pwd -P )
WORK_DIR=$DIR/../modules/portal
DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
function check_for() {
which $1 >/dev/null 2>&1
EXIT_CODE=$?
if [ ${EXIT_CODE} != 0 ]
then
echo "$1 is not installed"
exit ${EXIT_CODE}
fi
}
cd $WORK_DIR
check_for python
check_for npm
# if the program exits before this has been captured then there must have been an error
EXIT_CODE=1
# javascript code generate
npm install
grunt default
TEST_SUITES=('grunt unit')
for TEST in "${TEST_SUITES[@]}"
do
echo "##### Running test: [$TEST]"
$TEST
EXIT_CODE=$?
echo "##### Test [$TEST] ended with status [$EXIT_CODE]"
if [ ${EXIT_CODE} != 0 ]
then
cd -
exit ${EXIT_CODE}
fi
done
cd -
exit 0
cd "$DIR/../test/portal/functional"
make

18
bin/generate-aes-256-hex-key.sh
View File

@ -1,18 +0,0 @@
#!/bin/bash
# Generate 256-bit AES key.
#
# Usage:
# $ ./generate-aes-256-hex-key.sh [passphrase]
# * passphrase: Optional passphrase used to generate secret key. A pseudo-random passphrase will be used if
# one is not provided.
if [[ ! -z "$1" ]]
then
echo "Using user-provided passphrase."
fi
PASSPHRASE=${1:-$(openssl rand 32)}
KEY=$(openssl enc -aes-256-cbc -k "$PASSPHRASE" -P -md sha1 | awk -F'=' 'NR == 2 {print $2}')
echo "Your 256-bit AES hex key: $KEY"

2
bin/release.sh
View File

@ -34,13 +34,11 @@ if [ "$1" != "skip-tests" ]; then
fi
printf "\nrunning api func tests... \n"
"$DIR"/remove-vinyl-containers.sh
if ! "$DIR"/func-test-api.sh
then
printf "\nerror: bin/func-test-api.sh failed \n"
exit 1
fi
"$DIR"/remove-vinyl-containers.sh
printf "\nrunning portal func tests... \n"
if ! "$DIR"/func-test-portal.sh

18
bin/verify.sh
View File

@ -1,18 +1,14 @@
#!/bin/bash
#!/usr/bin/env bash
set -euo pipefail
DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
echo 'Running tests...'
echo 'Stopping any docker containers...'
./bin/remove-vinyl-containers.sh
echo 'Starting up docker for integration testing and running unit and integration tests on all modules...'
sbt ";validate;verify"
cd "$DIR/../test/api/integration"
make build && make run -- sbt ";validate;verify"
verify_result=$?
echo 'Stopping any docker containers...'
./bin/remove-vinyl-containers.sh
if [ ${verify_result} -eq 0 ]
then
if [ ${verify_result} -eq 0 ]; then
echo 'Verify successful!'
exit 0
else

173
build.sbt
View File

@ -1,7 +1,6 @@
import CompilerOptions._
import Dependencies._
import Resolvers._
import com.typesafe.sbt.packager.docker._
import microsites._
import org.scalafmt.sbt.ScalafmtPlugin._
import sbtrelease.ReleasePlugin.autoImport.ReleaseTransformations._
@ -11,7 +10,7 @@ import scala.util.Try
resolvers ++= additionalResolvers
lazy val IntegrationTest = config("it") extend Test
lazy val IntegrationTest = config("it").extend(Test)
// settings that should be inherited by all projects
lazy val sharedSettings = Seq(
@ -21,11 +20,12 @@ lazy val sharedSettings = Seq(
startYear := Some(2018),
licenses += ("Apache-2.0", new URL("https://www.apache.org/licenses/LICENSE-2.0.txt")),
scalacOptions ++= scalacOptionsByV(scalaVersion.value),
scalacOptions in(Compile, doc) += "-no-link-warnings",
scalacOptions in (Compile, doc) += "-no-link-warnings",
// Use wart remover to eliminate code badness
wartremoverErrors := (
if (getPropertyFlagOrDefault("build.lintOnCompile", true))
Seq(Wart.EitherProjectionPartial,
Seq(
Wart.EitherProjectionPartial,
Wart.IsInstanceOf,
Wart.JavaConversions,
Wart.Return,
@ -36,19 +36,18 @@ lazy val sharedSettings = Seq(
),
// scala format
scalafmtOnCompile := getPropertyFlagOrDefault("build.scalafmtOnCompile", true),
scalafmtOnCompile in IntegrationTest := getPropertyFlagOrDefault("build.scalafmtOnCompile", true),
scalafmtOnCompile := getPropertyFlagOrDefault("build.scalafmtOnCompile", false),
// coverage options
coverageMinimum := 85,
coverageFailOnMinimum := true,
coverageHighlighting := true,
coverageHighlighting := true
)
lazy val testSettings = Seq(
parallelExecution in Test := true,
parallelExecution in IntegrationTest := false,
fork in IntegrationTest := true,
fork in IntegrationTest := false,
testOptions in Test += Tests.Argument("-oDNCXEPQRMIK", "-l", "SkipCI"),
logBuffered in Test := false,
// Hide stack traces in tests
@ -75,74 +74,16 @@ lazy val apiAssemblySettings = Seq(
// there are some odd things from dnsjava including update.java and dig.java that we don't use
assemblyMergeStrategy in assembly := {
case "update.class" | "dig.class" => MergeStrategy.discard
case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "index.js") => MergeStrategy.discard
case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "template.js") => MergeStrategy.discard
case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "index.js") =>
MergeStrategy.discard
case PathList("scala", "tools", "nsc", "doc", "html", "resource", "lib", "template.js") =>
MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
)
lazy val apiDockerSettings = Seq(
dockerBaseImage := "adoptopenjdk/openjdk11:jdk-11.0.7_10-alpine",
dockerUsername := Some("vinyldns"),
packageName in Docker := "api",
dockerExposedPorts := Seq(9000),
dockerEntrypoint := Seq("/opt/docker/bin/api"),
dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath
dockerExposedVolumes := Seq("/opt/docker/conf"), // mount extra config to the classpath
// add extra libs to class path via mount
scriptClasspath in bashScriptDefines ~= (cp => cp :+ "${app_home}/../lib_extra/*"),
// adds config file to mount
bashScriptExtraDefines += """addJava "-Dconfig.file=${app_home}/../conf/application.conf"""",
bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml"""", // adds logback
// this is the default version, can be overridden
bashScriptExtraDefines += s"""addJava "-Dvinyldns.base-version=${(version in ThisBuild).value}"""",
bashScriptExtraDefines += "(cd ${app_home} && ./wait-for-dependencies.sh && cd -)",
credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials")),
dockerCommands ++= Seq(
Cmd("USER", "root"), // switch to root so we can install netcat
ExecCmd("RUN", "apk", "add", "--update", "--no-cache", "netcat-openbsd", "bash"),
Cmd("USER", "1001:0") // switch back to the daemon user
),
)
lazy val portalDockerSettings = Seq(
dockerBaseImage := "adoptopenjdk/openjdk11:jdk-11.0.7_10-alpine",
dockerUsername := Some("vinyldns"),
packageName in Docker := "portal",
dockerExposedPorts := Seq(9001),
dockerExposedVolumes := Seq("/opt/docker/lib_extra"), // mount extra libs to the classpath
dockerExposedVolumes := Seq("/opt/docker/conf"), // mount extra config to the classpath
// add extra libs to class path via mount
scriptClasspath in bashScriptDefines ~= (cp => cp :+ "${app_home}/../lib_extra/*"),
// adds config file to mount
bashScriptExtraDefines += """addJava "-Dconfig.file=${app_home}/../conf/application.conf"""",
bashScriptExtraDefines += """addJava "-Dlogback.configurationFile=${app_home}/../conf/logback.xml"""",
// this is the default version, can be overridden
bashScriptExtraDefines += s"""addJava "-Dvinyldns.base-version=${(version in ThisBuild).value}"""",
// needed to avoid access issue in play for the RUNNING_PID
// https://github.com/lightbend/sbt-reactive-app/issues/177
bashScriptExtraDefines += s"""addJava "-Dplay.server.pidfile.path=/dev/null"""",
// wait for mysql
bashScriptExtraDefines += "(cd ${app_home}/../ && ls && ./wait-for-dependencies.sh && cd -)",
dockerCommands ++= Seq(
Cmd("USER", "root"), // switch to root so we can install netcat
ExecCmd("RUN", "apk", "add", "--update", "--no-cache", "netcat-openbsd", "bash"),
Cmd("USER", "1001:0") // switch back to the user that runs the process
),
credentials in Docker := Seq(Credentials(Path.userHome / ".ivy2" / ".dockerCredentials"))
)
lazy val noPublishSettings = Seq(
publish := {},
publishLocal := {},
@ -164,7 +105,7 @@ lazy val portalPublishSettings = Seq(
case (file, _) => file.getName.equals("local.conf")
}),
// for local.conf to be excluded in jars
mappings in(Compile, packageBin) ~= (_.filterNot {
mappings in (Compile, packageBin) ~= (_.filterNot {
case (file, _) => file.getName.equals("local.conf")
})
)
@ -181,8 +122,7 @@ lazy val allApiSettings = Revolver.settings ++ Defaults.itSettings ++
sharedSettings ++
apiAssemblySettings ++
testSettings ++
apiPublishSettings ++
apiDockerSettings
apiPublishSettings
lazy val api = (project in file("modules/api"))
.enablePlugins(JavaAppPackaging, AutomateHeaderPlugin)
@ -197,23 +137,18 @@ lazy val api = (project in file("modules/api"))
r53 % "compile->compile;it->it"
)
val killDocker = TaskKey[Unit]("killDocker", "Kills all vinyldns docker containers")
lazy val root = (project in file(".")).enablePlugins(DockerComposePlugin, AutomateHeaderPlugin)
lazy val root = (project in file("."))
.enablePlugins(AutomateHeaderPlugin)
.configs(IntegrationTest)
.settings(headerSettings(IntegrationTest))
.settings(sharedSettings)
.settings(
inConfig(IntegrationTest)(scalafmtConfigSettings),
killDocker := {
import scala.sys.process._
"./bin/remove-vinyl-containers.sh" !
},
inConfig(IntegrationTest)(scalafmtConfigSettings)
)
.aggregate(core, api, portal, mysql, sqs, r53)
lazy val coreBuildSettings = Seq(
name := "core",
// do not use unused params as NoOpCrypto ignores its constructor, we should provide a way
// to write a crypto plugin so that we fall back to a noarg constructor
scalacOptions ++= scalacOptionsByV(scalaVersion.value).filterNot(_ == "-Ywarn-unused:params")
@ -221,7 +156,9 @@ lazy val coreBuildSettings = Seq(
lazy val corePublishSettings = Seq(
publishMavenStyle := true,
publishArtifact in Test := false,
pomIncludeRepository := { _ => false },
pomIncludeRepository := { _ =>
false
},
autoAPIMappings := true,
publish in Docker := {},
mainClass := None,
@ -235,7 +172,8 @@ lazy val corePublishSettings = Seq(
sonatypeProfileName := "io.vinyldns"
)
lazy val core = (project in file("modules/core")).enablePlugins(AutomateHeaderPlugin)
lazy val core = (project in file("modules/core"))
.enablePlugins(AutomateHeaderPlugin)
.settings(sharedSettings)
.settings(coreBuildSettings)
.settings(corePublishSettings)
@ -257,7 +195,8 @@ lazy val mysql = (project in file("modules/mysql"))
.settings(libraryDependencies ++= mysqlDependencies ++ commonTestDependencies.map(_ % "test, it"))
.settings(
organization := "io.vinyldns"
).dependsOn(core % "compile->compile;test->test")
)
.dependsOn(core % "compile->compile;test->test")
.settings(name := "mysql")
lazy val sqs = (project in file("modules/sqs"))
@ -271,8 +210,9 @@ lazy val sqs = (project in file("modules/sqs"))
.settings(Defaults.itSettings)
.settings(libraryDependencies ++= sqsDependencies ++ commonTestDependencies.map(_ % "test, it"))
.settings(
organization := "io.vinyldns",
).dependsOn(core % "compile->compile;test->test")
organization := "io.vinyldns"
)
.dependsOn(core % "compile->compile;test->test")
.settings(name := "sqs")
lazy val r53 = (project in file("modules/r53"))
@ -287,53 +227,51 @@ lazy val r53 = (project in file("modules/r53"))
.settings(libraryDependencies ++= r53Dependencies ++ commonTestDependencies.map(_ % "test, it"))
.settings(
organization := "io.vinyldns",
coverageMinimum := 65,
).dependsOn(core % "compile->compile;test->test")
coverageMinimum := 65
)
.dependsOn(core % "compile->compile;test->test")
.settings(name := "r53")
val preparePortal = TaskKey[Unit]("preparePortal", "Runs NPM to prepare portal for start")
val checkJsHeaders = TaskKey[Unit]("checkJsHeaders", "Runs script to check for APL 2.0 license headers")
val createJsHeaders = TaskKey[Unit]("createJsHeaders", "Runs script to prepend APL 2.0 license headers to files")
val checkJsHeaders =
TaskKey[Unit]("checkJsHeaders", "Runs script to check for APL 2.0 license headers")
val createJsHeaders =
TaskKey[Unit]("createJsHeaders", "Runs script to prepend APL 2.0 license headers to files")
lazy val portal = (project in file("modules/portal")).enablePlugins(PlayScala, AutomateHeaderPlugin)
lazy val portal = (project in file("modules/portal"))
.enablePlugins(PlayScala, AutomateHeaderPlugin)
.settings(sharedSettings)
.settings(testSettings)
.settings(portalPublishSettings)
.settings(portalDockerSettings)
.settings(
name := "portal",
libraryDependencies ++= portalDependencies,
routesGenerator := InjectedRoutesGenerator,
coverageExcludedPackages := "<empty>;views.html.*;router.*;controllers\\.javascript.*;.*Reverse.*",
javaOptions in Test += "-Dconfig.file=conf/application-test.conf",
// ads the version when working locally with sbt run
PlayKeys.devSettings += "vinyldns.base-version" -> (version in ThisBuild).value,
// adds an extra classpath to the portal loading so we can externalize jars, make sure to create the lib_extra
// directory and lay down any dependencies that are required when deploying
scriptClasspath in bashScriptDefines ~= (cp => cp :+ "lib_extra/*"),
mainClass in reStart := None,
// we need to filter out unused for the portal as the play framework needs a lot of unused things
scalacOptions ~= { opts => opts.filterNot(p => p.contains("unused")) },
scalacOptions ~= { opts =>
opts.filterNot(p => p.contains("unused"))
},
// runs our prepare portal process
preparePortal := {
import scala.sys.process._
"./modules/portal/prepare-portal.sh" !
},
checkJsHeaders := {
import scala.sys.process._
"./bin/add-license-headers.sh -d=modules/portal/public/lib -f=js -c" !
},
createJsHeaders := {
import scala.sys.process._
"./bin/add-license-headers.sh -d=modules/portal/public/lib -f=js" !
},
// change the name of the output to portal.zip
packageName in Universal := "portal"
)
@ -365,8 +303,16 @@ lazy val docSettings = Seq(
mdocIn := (sourceDirectory in Compile).value / "mdoc",
micrositeCssDirectory := (resourceDirectory in Compile).value / "microsite" / "css",
micrositeCompilingDocsTool := WithMdoc,
micrositeFavicons := Seq(MicrositeFavicon("favicon16x16.png", "16x16"), MicrositeFavicon("favicon32x32.png", "32x32")),
micrositeEditButton := Some(MicrositeEditButton("Improve this page", "/edit/master/modules/docs/src/main/mdoc/{{ page.path }}")),
micrositeFavicons := Seq(
MicrositeFavicon("favicon16x16.png", "16x16"),
MicrositeFavicon("favicon32x32.png", "32x32")
),
micrositeEditButton := Some(
MicrositeEditButton(
"Improve this page",
"/edit/master/modules/docs/src/main/mdoc/{{ page.path }}"
)
),
micrositeFooterText := None,
micrositeHighlightTheme := "atom-one-light",
includeFilter in makeSite := "*.html" | "*.css" | "*.png" | "*.jpg" | "*.jpeg" | "*.gif" | "*.js" | "*.swf" | "*.md" | "*.webm" | "*.ico" | "CNAME" | "*.yml" | "*.svg" | "*.json" | "*.csv"
@ -384,8 +330,10 @@ lazy val setSonatypeReleaseSettings = ReleaseStep(action = oldState => {
val v = extracted.get(Keys.version)
val snap = v.endsWith("SNAPSHOT")
if (!snap) {
val publishToSettings = Some("releases" at "https://oss.sonatype.org/" + "service/local/staging/deploy/maven2")
val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
val publishToSettings =
Some("releases".at("https://oss.sonatype.org/" + "service/local/staging/deploy/maven2"))
val newState =
extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
// create sonatypeReleaseCommand with releaseSonatype step
val sonatypeCommand = Command.command("sonatypeReleaseCommand") {
@ -397,8 +345,10 @@ lazy val setSonatypeReleaseSettings = ReleaseStep(action = oldState => {
newState.copy(definedCommands = newState.definedCommands :+ sonatypeCommand)
} else {
val publishToSettings = Some("snapshots" at "https://oss.sonatype.org/" + "content/repositories/snapshots")
val newState = extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
val publishToSettings =
Some("snapshots".at("https://oss.sonatype.org/" + "content/repositories/snapshots"))
val newState =
extracted.appendWithSession(Seq(publishTo in core := publishToSettings), oldState)
// create sonatypeReleaseCommand without releaseSonatype step
val sonatypeCommand = Command.command("sonatypeReleaseCommand") {
@ -437,7 +387,9 @@ releaseProcess :=
finalReleaseStage
// Let's do things in parallel!
addCommandAlias("validate", "; root/clean; " +
addCommandAlias(
"validate",
"; root/clean; " +
"all core/headerCheck core/test:headerCheck " +
"api/headerCheck api/test:headerCheck api/it:headerCheck " +
"mysql/headerCheck mysql/test:headerCheck mysql/it:headerCheck " +
@ -448,10 +400,11 @@ addCommandAlias("validate", "; root/clean; " +
"root/compile;root/test:compile;root/it:compile"
)
addCommandAlias("verify", "; project root; killDocker; dockerComposeUp; " +
"project root; coverage; " +
addCommandAlias(
"verify",
"; project root; coverage; " +
"all test it:test; " +
"project root; coverageReport; coverageAggregate; killDocker"
"project root; coverageReport; coverageAggregate"
)
// Build the artifacts for release

24
build/README.md
View File

@ -50,8 +50,6 @@ The build will generate several VinylDNS docker images that are used to deploy i
- `vinyldns/api` - this is the heart of the VinylDNS system, the backend API
- `vinyldns/portal` - the VinylDNS web UI
- `vinyldns/test-bind9` - a DNS server that is configured to support running the functional tests
- `vinyldns/test` - a container that will execute functional tests, and exit success or failure when the tests are complete
### vinyldns/api
@ -80,25 +78,3 @@ it is set as part of the container build
any production environments. Typically, you will add your own `application.conf` file in here with your settings.
- `/opt/docker/lib_extra/` - if you need to have additional jar files available to your VinylDNS instance.
Rarely used, but if you want to bring your own message queue or database you can put the `jar` files there
### vinyldns/test-bind9
This pulls correct DNS configuration to run func tests. You can largely disregard what is in here
### vinyldns/test
This is used to run functional tests against a vinyldns instance. **This is very useful for verifying
your environment as part of doing an upgrade.** By default, it will run against a local docker-compose setup.
**Environment Variables**
- `VINYLDNS_URL` - the url to the vinyldns you will test against
- `DNS_IP` - the IP address to the `vinyldns/test-bind9` container that you will use for test purposes
- `TEST_PATTERN` - the actual functional test you want to run. *Important, set to empty string to run
ALL test; otherwise, omit the environment variable when you run to just run smoke tests*.
**Example**
This example will run all functional tests on the given VinylDNS url and DNS IP address
`docker run -e VINYLDNS_URL="https://my.vinyldns.example.com" -e DNS_IP="1.2.3.4" -e TEST_PATTERN=""`

8
build/docker-release.sh
View File

@ -107,15 +107,11 @@ if [ $DO_BUILD -eq 1 ]; then
fi
if [ $? -eq 0 ]; then
docker tag vinyldns/test-bind9:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test-bind9:$VINYLDNS_VERSION
docker tag vinyldns/test:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test:$VINYLDNS_VERSION
docker tag vinyldns/api:$VINYLDNS_VERSION $REPOSITORY/vinyldns/api:$VINYLDNS_VERSION
docker tag vinyldns/portal:$VINYLDNS_VERSION $REPOSITORY/vinyldns/portal:$VINYLDNS_VERSION
if [ $TAG_LATEST -eq 1 ]; then
echo "Tagging latest..."
docker tag vinyldns/test-bind9:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test-bind9:latest
docker tag vinyldns/test:$VINYLDNS_VERSION $REPOSITORY/vinyldns/test:latest
docker tag vinyldns/api:$VINYLDNS_VERSION $REPOSITORY/vinyldns/api:latest
docker tag vinyldns/portal:$VINYLDNS_VERSION $REPOSITORY/vinyldns/portal:latest
fi
@ -123,15 +119,11 @@ if [ $DO_BUILD -eq 1 ]; then
fi
if [ $DOCKER_PUSH -eq 1 ]; then
docker push $REPOSITORY/vinyldns/test-bind9:$VINYLDNS_VERSION
docker push $REPOSITORY/vinyldns/test:$VINYLDNS_VERSION
docker push $REPOSITORY/vinyldns/api:$VINYLDNS_VERSION
docker push $REPOSITORY/vinyldns/portal:$VINYLDNS_VERSION
if [ $TAG_LATEST -eq 1 ]; then
echo "Pushing latest..."
docker push $REPOSITORY/vinyldns/test-bind9:latest
docker push $REPOSITORY/vinyldns/test:latest
docker push $REPOSITORY/vinyldns/api:latest
docker push $REPOSITORY/vinyldns/portal:latest
fi

28
build/docker/test/Dockerfile
View File

@ -1,28 +0,0 @@
FROM alpine/git:1.0.7 as gitcheckout
ARG BRANCH=master
RUN git clone -b ${BRANCH} --single-branch --depth 1 https://github.com/vinyldns/vinyldns.git /vinyldns
FROM python:2.7.16-alpine3.9
RUN apk add --update --no-cache bind-tools netcat-openbsd bash curl
# The run script is what actually runs our func tests
COPY run.sh /app/run.sh
COPY run-tests.py /app/run-tests.py
RUN chmod a+x /app/run.sh && chmod a+x /app/run-tests.py
# Copy over the functional test directory, this must have been copied into the build context previous to this building!
COPY --from=gitcheckout /vinyldns/modules/api/functional_test/ /app/
# Install our func test requirements
RUN pip install --index-url https://pypi.python.org/simple/ -r /app/requirements.txt
ENV VINYLDNS_URL=""
ENV DNS_IP=""
ENV TEST_PATTERN="test_verify_production"
# set the entry point for the container to start vinyl, specify the config resource
ENTRYPOINT ["/app/run.sh"]

16
build/docker/test/run-tests.py
View File

@ -1,16 +0,0 @@
#!/usr/bin/env python
import os
import sys
basedir = os.path.dirname(os.path.realpath(__file__))
report_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), '../target/pytest_reports')
if not os.path.exists(report_dir):
os.system('mkdir -p ' + report_dir)
import pytest
result = 1
result = pytest.main(list(sys.argv[1:]))
sys.exit(result)

76
build/docker/test/run.sh
View File

@ -1,76 +0,0 @@
#!/usr/bin/env bash
# Assume defaults of local docker-compose if not set
if [ -z "${VINYLDNS_URL}" ]; then
VINYLDNS_URL="http://vinyldns-api:9000"
fi
if [ -z "${DNS_IP}" ]; then
DNS_IP=$(dig +short vinyldns-bind9)
fi
# Assume all tests if not specified
if [ -z "${TEST_PATTERN}" ]; then
TEST_PATTERN=
else
TEST_PATTERN="-k ${TEST_PATTERN}"
fi
echo "Waiting for API to be ready at ${VINYLDNS_URL} ..."
DATA=""
RETRY=60
SLEEP_DURATION=1
while [ "$RETRY" -gt 0 ]
do
DATA=$(curl -I -s "${VINYLDNS_URL}/ping" -o /dev/null -w "%{http_code}")
if [ $? -eq 0 ]
then
break
else
echo "Retrying" >&2
let RETRY-=1
sleep "$SLEEP_DURATION"
if [ "$RETRY" -eq 0 ]
then
echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
exit 1
fi
fi
done
echo "Running live tests against ${VINYLDNS_URL} and DNS server ${DNS_IP}"
cd /app
# Cleanup any errant cached file copies
find . -name "*.pyc" -delete
find . -name "__pycache__" -delete
ls -al
# -m plays havoc with -k, using variables is a headache, so doing this by hand
# run parallel tests first (not serial)
set -x
./run-tests.py live_tests -n2 -v -m "not skip_production and not serial" ${TEST_PATTERN} --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} --teardown=False
ret1=$?
# IMPORTANT! pytest exists status code 5 if no tests are run, force that to 0
if [ "$ret1" = 5 ]; then
echo "No tests collected."
ret1=0
fi
./run-tests.py live_tests -n0 -v -m "not skip_production and serial" ${TEST_PATTERN} --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} --teardown=True
ret2=$?
if [ "$ret2" = 5 ]; then
echo "No tests collected."
ret2=0
fi
if [ $ret1 -ne 0 ] || [ $ret2 -ne 0 ]; then
exit 1
else
exit 0
fi

1
docker/api/Dockerfile
View File

@ -10,7 +10,6 @@ RUN chmod a+x /app/run.sh
COPY docker.conf /app/docker.conf
EXPOSE 9000
EXPOSE 2551
# set the entry point for the container to start vinyl, specify the config resource
ENTRYPOINT ["/app/run.sh"]

8
docker/api/docker.conf
View File

@ -106,17 +106,17 @@ vinyldns {
settings {
# AWS access key and secret.
access-key = "x"
access-key = "test"
access-key = ${?AWS_ACCESS_KEY}
secret-key = "x"
secret-key = "test"
secret-key = ${?AWS_SECRET_ACCESS_KEY}
# Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed.
signing-region = "x"
signing-region = "us-east-1"
signing-region = ${?SQS_REGION}
# Endpoint to access queue
service-endpoint = "http://vinyldns-elasticmq:9324/"
service-endpoint = "http://vinyldns-localstack:19007/"
service-endpoint = ${?SQS_ENDPOINT}
# Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change.

12
docker/api/run.sh
View File

@ -3,20 +3,10 @@
# gets the docker-ized ip address, sets it to an environment variable
export APP_HOST=`ip addr show eth0 | grep 'inet ' | awk '{print $2}' | cut -f1 -d'/'`
export DYNAMO_ADDRESS="vinyldns-dynamodb"
export DYNAMO_PORT=8000
export JOURNAL_HOST="vinyldns-dynamodb"
export JOURNAL_PORT=8000
export MYSQL_ADDRESS="vinyldns-mysql"
export MYSQL_PORT=3306
export JDBC_USER=root
export JDBC_PASSWORD=pass
export DNS_ADDRESS="vinyldns-bind9"
export DYNAMO_KEY="local"
export DYNAMO_SECRET="local"
export DYNAMO_TABLE_PREFIX=""
export ELASTICMQ_ADDRESS="vinyldns-elasticmq"
export DYNAMO_ENDPOINT="http://${DYNAMO_ADDRESS}:${DYNAMO_PORT}"
export JDBC_URL="jdbc:mariadb://${MYSQL_ADDRESS}:${MYSQL_PORT}/vinyldns?user=${JDBC_USER}&password=${JDBC_PASSWORD}"
export JDBC_MIGRATION_URL="jdbc:mariadb://${MYSQL_ADDRESS}:${MYSQL_PORT}/?user=${JDBC_USER}&password=${JDBC_PASSWORD}"
@ -27,7 +17,7 @@ RETRY=40
SLEEP_DURATION=1
while [ "$RETRY" -gt 0 ]
do
DATA=$(nc -vzw1 vinyldns-mysql 3306)
DATA=$(nc -vzw1 ${MYSQL_ADDRESS} ${MYSQL_PORT})
if [ $? -eq 0 ]
then
break

3
docker/bind9/README.md
View File

@ -18,6 +18,5 @@ When used in a container, or to run `named`, the files in this directory should
| Directory | Target |
|:---|:---|
| `etc/named.conf.local` | `/etc/bind/` |
| `etc/named.partition*.conf` | `/var/bind/config/` |
| `etc/named.conf.*` | `/etc/bind/` |
| `zones/` | `/var/bind/` |

8
docker/bind9/etc/named.conf.local
View File

@ -29,7 +29,7 @@ key "vinyldns-sha512." {
secret "xfKA0DYb88tiUGND+cWddwUg3/SugYSsdvCfBOJ1jr8MEdgbVRyrlVDEXLsfTUGorQ3ShENdymw2yw+rTr+lwA==";
};
include "/var/bind/config/named.partition1.conf";
include "/var/bind/config/named.partition2.conf";
include "/var/bind/config/named.partition3.conf";
include "/var/bind/config/named.partition4.conf";
include "/etc/bind/named.conf.partition1";
include "/etc/bind/named.conf.partition2";
include "/etc/bind/named.conf.partition3";
include "/etc/bind/named.conf.partition4";

0
docker/bind9/etc/named.partition1.conf → docker/bind9/etc/named.conf.partition1
View File

0
docker/bind9/etc/named.partition2.conf → docker/bind9/etc/named.conf.partition2
View File

0
docker/bind9/etc/named.partition3.conf → docker/bind9/etc/named.conf.partition3
View File

0
docker/bind9/etc/named.partition4.conf → docker/bind9/etc/named.conf.partition4
View File

60
docker/docker-compose-func-test-testbind9.yml
View File

@ -1,60 +0,0 @@
version: "3.0"
services:
mysql:
image: "mysql:5.7"
env_file:
.env
container_name: "vinyldns-mysql"
ports:
- "19002:3306"
logging:
driver: none
bind9:
image: "vinyldns/test-bind9:0.9.4"
container_name: "vinyldns-bind9"
ports:
- "19001:53/tcp"
- "19001:53/udp"
logging:
driver: none
localstack:
image: localstack/localstack:0.10.4
container_name: "vinyldns-localstack"
ports:
- "19000:19000"
- "19006:19006"
- "19007:19007"
- "19009:19009"
environment:
- SERVICES=sns:19006,sqs:19007,route53:19009
- START_WEB=0
- HOSTNAME_EXTERNAL=vinyldns-localstack
# this file is copied into the target directory to get the jar! won't run in place as is!
api:
build:
context: api
env_file:
.env
container_name: "vinyldns-api"
ports:
- "9000:9000"
depends_on:
- mysql
- bind9
- localstack
logging:
driver: none
functest:
build:
context: functest
env_file:
.env
environment:
- PAR_CPU=${PAR_CPU}
container_name: "vinyldns-functest"
depends_on:
- api

87
docker/docker-compose-func-test.yml
View File

@ -1,87 +0,0 @@
version: "3.5"
services:
# this file is copied into the target directory to get the jar! won't run in place as is!
api:
build:
context: api
env_file:
.env
container_name: "vinyldns-api"
ports:
- "9000:9000"
depends_on:
- mysql
- bind9
- localstack
networks:
vinyldns:
ipv4_address: 172.10.10.2
mysql:
image: "mysql:5.7"
env_file:
.env
container_name: "vinyldns-mysql"
ports:
- "19002:3306"
logging:
driver: none
networks:
vinyldns:
ipv4_address: 172.10.10.3
localstack:
image: localstack/localstack:0.10.4
container_name: "vinyldns-localstack"
ports:
- "19006:19006"
- "19007:19007"
- "19009:19009"
environment:
- SERVICES=sns:19006,sqs:19007,route53:19009
- START_WEB=0
- HOSTNAME_EXTERNAL=vinyldns-localstack
networks:
vinyldns:
ipv4_address: 172.10.10.4
bind9:
image: "vinyldns/bind9:0.0.4"
env_file:
.env
container_name: "vinyldns-bind9"
volumes:
- ./bind9/etc:/var/cache/bind/config
- ./bind9/zones:/var/cache/bind/zones
ports:
- "19001:53/tcp"
- "19001:53/udp"
logging:
driver: none
networks:
vinyldns:
ipv4_address: 172.10.10.10
functest:
build:
context: functest
env_file:
.env
environment:
- PAR_CPU=${PAR_CPU}
container_name: "vinyldns-functest"
depends_on:
- api
networks:
- vinyldns
networks:
# Custom network so that we have some control over IP space and deterministic container IPs
vinyldns:
name: vinyldns
driver: bridge
ipam:
config:
- subnet: 172.10.10.0/24

2
docker/docker-compose-quick-start.yml
View File

@ -9,7 +9,7 @@ services:
- "19002:3306"
bind9:
image: "vinyldns/bind9:0.0.4"
image: "vinyldns/bind9:0.0.5"
env_file:
.env.quickstart
container_name: "vinyldns-bind9"

2
docker/docker-compose.yml
View File

@ -8,7 +8,7 @@ services:
- "19002:3306"
bind9:
image: vinyldns/bind9:0.0.4<skipPull>
image: vinyldns/bind9:0.0.5<skipPull>
env_file:
.env
ports:

10
docker/elasticmq/Dockerfile
View File

@ -1,10 +0,0 @@
FROM alpine:3.2
FROM anapsix/alpine-java:8_server-jre
EXPOSE 9324
COPY run.sh /elasticmq/run.sh
COPY custom.conf /elasticmq/custom.conf
COPY elasticmq-server-0.13.2.jar /elasticmq/server.jar
ENTRYPOINT ["/elasticmq/run.sh"]

22
docker/elasticmq/custom.conf
View File

@ -1,22 +0,0 @@
node-address {
protocol = http
host = "localhost"
host = ${?QUEUE_HOST}
port = 9324
context-path = ""
}
rest-sqs {
enabled = true
bind-port = 9324
bind-hostname = "0.0.0.0"
// Possible values: relaxed, strict
sqs-limits = relaxed
}
queues {
vinyldns {
defaultVisibilityTimeout = 10 seconds
receiveMessageWait = 0 seconds
}
}

8
docker/elasticmq/run.sh
View File

@ -1,8 +0,0 @@
#!/usr/bin/env bash
# gets the docker-ized ip address, sets it to an environment variable
export APP_HOST=`ip addr show eth0 | grep 'inet ' | awk '{print $2}' | cut -f1 -d'/'`
echo "APP HOST = ${APP_HOST}"
java -Djava.net.preferIPv4Stack=true -Dconfig.file=/elasticmq/custom.conf -jar /elasticmq/server.jar

2
docker/email/.gitignore vendored
View File

@ -1,2 +0,0 @@
*
!.gitignore

23
docker/functest/Dockerfile
View File

@ -1,23 +0,0 @@
FROM python:2.7.15-stretch
# Install dns utils so we can run dig
RUN apt-get update && apt-get install dnsutils -y
# The run script is what actually runs our func tests
COPY run.sh /app/run.sh
RUN chmod a+x /app/run.sh
COPY run-tests.py /app/run-tests.py
RUN chmod a+x /app/run-tests.py
# Copy over the functional test directory, this must have been copied into the build context previous to this building!
ADD functional_test /app
# Install our func test requirements
RUN pip install --index-url https://pypi.python.org/simple/ -r /app/requirements.txt
# Specifies how many CPUs to use for func tests; the more the better or specifiy "auto" for optimal results
ENV PAR_CPU=2
# set the entry point for the container to start vinyl, specify the config resource
ENTRYPOINT ["/app/run.sh"]

18
docker/functest/run-tests.py
View File

@ -1,18 +0,0 @@
#!/usr/bin/env python
import os
import sys
basedir = os.path.dirname(os.path.realpath(__file__))
report_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), '../target/pytest_reports')
if not os.path.exists(report_dir):
os.system('mkdir -p ' + report_dir)
import pytest
result = 1
result = pytest.main(list(sys.argv[1:]))
sys.exit(result)

81
docker/functest/run.sh
View File

@ -1,81 +0,0 @@
#!/usr/bin/env bash
# Assume defaults of local docker-compose if not set
if [ -z "${VINYLDNS_URL}" ]; then
VINYLDNS_URL="http://vinyldns-api:9000"
fi
if [ -z "${DNS_IP}" ]; then
DNS_IP=$(dig +short vinyldns-bind9)
fi
# Assume all tests if not specified
if [ -z "${TEST_PATTERN}" ]; then
TEST_PATTERN=
else
TEST_PATTERN="-k ${TEST_PATTERN}"
fi
if [ -z "${PAR_CPU}" ]; then
export PAR_CPU=2
fi
echo "Waiting for API to be ready at ${VINYLDNS_URL} ..."
DATA=""
RETRY=60
SLEEP_DURATION=1
while [ "$RETRY" -gt 0 ]
do
DATA=$(curl -I -s "${VINYLDNS_URL}/ping" -o /dev/null -w "%{http_code}")
if [ $? -eq 0 ]
then
break
else
echo "Retrying" >&2
let RETRY-=1
sleep "$SLEEP_DURATION"
if [ "$RETRY" -eq 0 ]
then
echo "Exceeded retries waiting for VinylDNS API to be ready, failing"
exit 1
fi
fi
done
echo "Running live tests against ${VINYLDNS_URL} and DNS server ${DNS_IP}"
cd /app
# Cleanup any errant cached file copies
find . -name "*.pyc" -delete
find . -name "__pycache__" -delete
result=0
# If PROD_ENV is not true, we are in a local docker environment so do not skip anything
if [ "${PROD_ENV}" = "true" ]; then
# -m plays havoc with -k, using variables is a headache, so doing this by hand
# run parallel tests first (not serial)
echo "./run-tests.py live_tests -n${PAR_CPU} -v -m \"not skip_production and not serial\" -v --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False"
./run-tests.py live_tests -n${PAR_CPU} -v -m "not skip_production and not serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False
result=$?
if [ $result -eq 0 ]; then
# run serial tests second (serial marker)
echo "./run-tests.py live_tests -n0 -v -m \"not skip_production and serial\" -v --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True"
./run-tests.py live_tests -n0 -v -m "not skip_production and serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True
result=$?
fi
else
# run parallel tests first (not serial)
echo "./run-tests.py live_tests -n${PAR_CPU} -v -m \"not serial\" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False"
./run-tests.py live_tests -n${PAR_CPU} -v -m "not serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=False
result=$?
if [ $result -eq 0 ]; then
# run serial tests second (serial marker)
echo "./run-tests.py live_tests -n0 -v -m \"serial\" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True"
./run-tests.py live_tests -n0 -v -m "serial" --url=${VINYLDNS_URL} --dns-ip=${DNS_IP} ${TEST_PATTERN} --teardown=True
result=$?
fi
fi
exit $result

33
modules/api/functional_test/Dockerfile
View File

@ -1,33 +0,0 @@
# Build VinylDNS API if the JAR doesn't already exist
FROM vinyldns/build:base-api as vinyldns-api
COPY modules/api/functional_test/docker.conf modules/api/functional_test/vinyldns*.jar /opt/vinyldns/
COPY . /build/
WORKDIR /build
## Run the build if we don't already have a vinyldns.jar
RUN if [ ! -f /opt/vinyldns/vinyldns.jar ]; then \
env SBT_OPTS="-XX:+UseConcMarkSweepGC -Xmx4G -Xms1G" \
sbt -Dbuild.scalafmtOnCompile=false -Dbuild.lintOnCompile=fase ";project api;coverageOff;assembly" \
&& cp modules/api/target/scala-2.12/vinyldns.jar /opt/vinyldns/; \
fi
# Build the testing image, copying data from `vinyldns-api`
FROM vinyldns/build:base-test
SHELL ["/bin/bash","-c"]
COPY --from=vinyldns-api /opt/vinyldns /opt/vinyldns
# Local bind server files
COPY docker/bind9/etc/named.conf.local /etc/bind/
COPY docker/bind9/etc/*.conf /var/bind/config/
COPY docker/bind9/zones/ /var/bind/
RUN named-checkconf
# Copy over the functional tests
COPY modules/api/functional_test /functional_test
ENTRYPOINT ["/bin/bash", "-c", "/initialize.sh && \
(java -Dconfig.file=/opt/vinyldns/docker.conf -jar /opt/vinyldns/vinyldns.jar &> /opt/vinyldns/vinyldns.log &) && \
echo -n 'Starting VinylDNS API..' && \
timeout 30s grep -q 'STARTED SUCCESSFULLY' <(timeout 30s tail -f /opt/vinyldns/vinyldns.log) && \
echo 'done.' && \
/bin/bash"]

25
modules/api/functional_test/Makefile
View File

@ -1,25 +0,0 @@
SHELL=bash
ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
# Check that the required version of make is being used
REQ_MAKE_VER:=3.82
ifneq ($(REQ_MAKE_VER),$(firstword $(sort $(MAKE_VERSION) $(REQ_MAKE_VER))))
$(error The version of MAKE $(REQ_MAKE_VER) or higher is required; you are running $(MAKE_VERSION))
endif
.ONESHELL:
.PHONY: all build run
all: build run
build:
@set -euo pipefail
trap 'if [ -f modules/api/functional_test/vinyldns.jar ]; then rm modules/api/functional_test/vinyldns.jar; fi' EXIT
cd ../../..
if [ -f modules/api/target/scala-2.12/vinyldns.jar ]; then cp modules/api/target/scala-2.12/vinyldns.jar modules/api/functional_test/vinyldns.jar; fi
docker build -t vinyldns-test -f modules/api/functional_test/Dockerfile .
run:
@set -euo pipefail
docker run -it --rm -p 9000:9000 -p 19001:53/tcp -p 19001:53/udp vinyldns-test

2
modules/api/src/it/scala/vinyldns/api/backend/dns/DnsBackendIntegrationSpec.scala
View File

@ -59,7 +59,7 @@ class DnsBackendIntegrationSpec extends AnyWordSpec with Matchers {
config.tsigUsage
)
val testZone = Zone(
"open.",
"open1.",
"test@test.com",
connection = Some(testConnection)
)

15
modules/api/src/it/scala/vinyldns/api/domain/zone/ZoneViewLoaderIntegrationSpec.scala
View File

@ -1,3 +1,4 @@
/*
* Copyright 2018 Comcast Cable Communications Management, LLC
*
@ -35,7 +36,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
"ZoneViewLoader" should {
"return a ZoneView upon success" in {
val zone = Zone("vinyldns.", "test@test.com")
val zone = Zone("vinyldns1.", "test@test.com")
DnsZoneViewLoader(zone, backendResolver.resolve(zone), 10000)
.load()
.unsafeRunSync() shouldBe a[ZoneView]
@ -44,7 +45,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
"return a failure if the transfer connection is bad" in {
assertThrows[IllegalArgumentException] {
val zone = Zone(
"vinyldns.",
"vinyldns1.",
"bad@transfer.connection",
connection = Some(
ZoneConnection(
@ -77,7 +78,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
"return a failure if the zone is larger than the max zone size" in {
assertThrows[ZoneTooLargeError] {
val zone = Zone(
"vinyldns.",
"vinyldns1.",
"test@test.com",
connection = Some(
ZoneConnection(
@ -86,6 +87,14 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers {
"nzisn+4G2ldMn0q1CV3vsg==",
"127.0.0.1:19001"
)
),
transferConnection = Some(
ZoneConnection(
"vinyldns.",
"vinyldns.",
"nzisn+4G2ldMn0q1CV3vsg==",
"127.0.0.1:19001"
)
)
)
DnsZoneViewLoader(zone, backendResolver.resolve(zone), 1)

31
modules/api/src/it/scala/vinyldns/api/notifier/sns/SnsNotifierIntegrationSpec.scala
View File

@ -16,25 +16,23 @@
package vinyldns.api.notifier.sns
import cats.effect.IO
import com.amazonaws.auth.{AWSStaticCredentialsProvider, BasicAWSCredentials}
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration
import com.amazonaws.services.sns.AmazonSNSClientBuilder
import com.amazonaws.services.sqs.AmazonSQSClientBuilder
import com.typesafe.config.{Config, ConfigFactory}
import vinyldns.core.notifier._
import vinyldns.api.MySqlApiIntegrationSpec
import vinyldns.mysql.MySqlIntegrationSpec
import org.joda.time.DateTime
import org.json4s.DefaultFormats
import org.json4s.jackson.JsonMethods._
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpecLike
import vinyldns.core.domain.batch._
import vinyldns.core.domain.record.RecordType
import vinyldns.core.domain.record.AData
import org.joda.time.DateTime
import vinyldns.api.MySqlApiIntegrationSpec
import vinyldns.core.TestMembershipData._
import cats.effect.IO
import com.amazonaws.services.sns.AmazonSNSClientBuilder
import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration
import com.amazonaws.services.sqs.AmazonSQSClientBuilder
import org.json4s.jackson.JsonMethods._
import org.json4s.DefaultFormats
import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.auth.AWSStaticCredentialsProvider
import vinyldns.core.domain.batch._
import vinyldns.core.domain.record.{AData, RecordType}
import vinyldns.core.notifier._
import vinyldns.mysql.MySqlIntegrationSpec
class SnsNotifierIntegrationSpec
extends MySqlApiIntegrationSpec
@ -93,7 +91,7 @@ class SnsNotifierIntegrationSpec
val sqs = AmazonSQSClientBuilder
.standard()
.withEndpointConfiguration(
new EndpointConfiguration("http://127.0.0.1:19007", "us-east-1")
new EndpointConfiguration("http://127.0.0.1:19003", "us-east-1")
)
.withCredentials(credentialsProvider)
.build()
@ -105,6 +103,7 @@ class SnsNotifierIntegrationSpec
notifier <- new SnsNotifierProvider()
.load(NotifierConfig("", snsConfig), userRepository)
_ <- notifier.notify(Notification(batchChange))
_ <- IO { Thread.sleep(100) }
messages <- IO { sqs.receiveMessage(queueUrl).getMessages }
_ <- IO {
sns.deleteTopic(topic)

2
modules/api/src/it/scala/vinyldns/api/route53/Route53ApiIntegrationSpec.scala
View File

@ -56,7 +56,7 @@ class Route53ApiIntegrationSpec
"test",
Some("access"),
Some("secret"),
"http://127.0.0.1:19009",
"http://127.0.0.1:19003",
"us-east-1"
)
)

82
modules/api/src/main/resources/reference.conf
View File

@ -16,43 +16,41 @@ vinyldns {
{
class-name = "vinyldns.api.backend.dns.DnsBackendProviderLoader"
settings = {
legacy = true # set this to true to attempt to load legacy config YAML
backends = []
# if not legacy then this...
# legacy = false
# backends = [
# {
# id = "default"
# zone-connection = {
# name = "vinyldns."
# keyName = "vinyldns."
# key = "nzisn+4G2ldMn0q1CV3vsg=="
# primaryServer = "127.0.0.1:19001"
# }
# transfer-connection = {
# name = "vinyldns."
# keyName = "vinyldns."
# key = "nzisn+4G2ldMn0q1CV3vsg=="
# primaryServer = "127.0.0.1:19001"
# }
# },
# {
# id = "func-test-backend"
# zone-connection = {
# name = "vinyldns."
# keyName = "vinyldns."
# key = "nzisn+4G2ldMn0q1CV3vsg=="
# primaryServer = "127.0.0.1:19001"
# }
# transfer-connection = {
# name = "vinyldns."
# keyName = "vinyldns."
# key = "nzisn+4G2ldMn0q1CV3vsg=="
# primaryServer = "127.0.0.1:19001"
# }
# }
#]
legacy = false
backends = [
{
id = "default"
zone-connection = {
name = "vinyldns."
key-name = "vinyldns."
key = "nzisn+4G2ldMn0q1CV3vsg=="
primary-server = "127.0.0.1:19001"
}
transfer-connection = {
name = "vinyldns."
key-name = "vinyldns."
key = "nzisn+4G2ldMn0q1CV3vsg=="
primary-server = "127.0.0.1:19001"
},
tsig-usage = "always"
},
{
id = "func-test-backend"
zone-connection = {
name = "vinyldns."
key-name = "vinyldns."
key = "nzisn+4G2ldMn0q1CV3vsg=="
primary-server = "127.0.0.1:19001"
}
transfer-connection = {
name = "vinyldns."
key-name = "vinyldns."
key = "nzisn+4G2ldMn0q1CV3vsg=="
primary-server = "127.0.0.1:19001"
},
tsig-usage = "always"
}
]
}
}
]
@ -76,10 +74,10 @@ vinyldns {
# secret-key = "x"
# Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed.
signing-region = "x"
signing-region = "us-east-1"
# Endpoint to access queue
service-endpoint = "http://localhost:19007/"
service-endpoint = "http://localhost:19003/"
# Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change.
queue-name = "vinyldns"
@ -141,9 +139,9 @@ vinyldns {
class-name = "vinyldns.api.notifier.sns.SnsNotifierProvider"
settings {
topic-arn = "arn:aws:sns:us-east-1:000000000000:batchChanges"
access-key = "vinyldnsTest"
secret-key = "notNeededForSnsLocal"
service-endpoint = "http://127.0.0.1:19006"
access-key = "test"
secret-key = "test"
service-endpoint = "http://127.0.0.1:19003"
signing-region = "us-east-1"
}
}

64
modules/api/src/test/scala/vinyldns/api/backend/dns/DnsBackendSpec.scala
View File

@ -101,7 +101,7 @@ class DnsBackendSpec
override def beforeEach(): Unit = {
doReturn(mockMessage).when(mockMessage).clone()
doReturn(new Array[DNS.Record](0)).when(mockMessage).getSectionArray(DNS.Section.ADDITIONAL)
doReturn(new java.util.ArrayList[DNS.Record](0)).when(mockMessage).getSection(DNS.Section.ADDITIONAL)
doReturn(DNS.Rcode.NOERROR).when(mockMessage).getRcode
doReturn(mockMessage).when(mockResolver).send(messageCaptor.capture())
doReturn(DNS.Lookup.SUCCESSFUL).when(mockDnsQuery).result
@ -160,7 +160,7 @@ class DnsBackendSpec
val conn = zoneConnection.copy(primaryServer = "dns.comcast.net:19001")
val dnsConn = DnsBackend("test", conn, None, new NoOpCrypto())
val simpleResolver = dnsConn.resolver.asInstanceOf[DNS.SimpleResolver]
val simpleResolver = dnsConn.resolver
val address = simpleResolver.getAddress
@ -172,7 +172,7 @@ class DnsBackendSpec
val conn = zoneConnection.copy(primaryServer = "dns.comcast.net")
val dnsConn = DnsBackend("test", conn, None, new NoOpCrypto())
val simpleResolver = dnsConn.resolver.asInstanceOf[DNS.SimpleResolver]
val simpleResolver = dnsConn.resolver
val address = simpleResolver.getAddress
@ -267,14 +267,14 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
dnsRecord.getName.toString shouldBe "a-record.vinyldns."
dnsRecord.getTTL shouldBe testA.ttl
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord shouldBe a[DNS.ARecord]
dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -286,7 +286,7 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val rrset = sentMessage.getSectionRRsets(DNS.Section.UPDATE)(0)
val rrset = sentMessage.getSectionRRsets(DNS.Section.UPDATE).asScala.head
rrset.getName.toString shouldBe "a-record.vinyldns."
rrset.getTTL shouldBe testA.ttl
rrset.getType shouldBe DNS.Type.A
@ -298,7 +298,7 @@ class DnsBackendSpec
records should contain theSameElementsAs expected
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -327,20 +327,20 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
// Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
emptyRecord.getName.toString shouldBe "updated-a-record.vinyldns."
emptyRecord.getType shouldBe DNS.Type.A
emptyRecord.getDClass shouldBe DNS.DClass.ANY
// The second section in the replace is the data that is being passed in, this is different than an add
val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
dnsRecord.getName.toString shouldBe "a-record.vinyldns."
dnsRecord.getTTL shouldBe testA.ttl
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord shouldBe a[DNS.ARecord]
dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -353,20 +353,20 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
// Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
emptyRecord.getName.toString shouldBe "a-record.vinyldns."
emptyRecord.getType shouldBe DNS.Type.A
emptyRecord.getDClass shouldBe DNS.DClass.ANY
// The second section in the replace is the data that is being passed in, this is different than an add
val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
dnsRecord.getName.toString shouldBe "a-record.vinyldns."
dnsRecord.getTTL shouldBe 300
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord shouldBe a[DNS.ARecord]
dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -378,7 +378,7 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE)
emptyRecord shouldBe empty
result shouldBe a[NoError]
@ -393,13 +393,13 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
// Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
emptyRecord.getName.toString shouldBe "updated-a-record.vinyldns."
emptyRecord.getType shouldBe DNS.Type.A
emptyRecord.getDClass shouldBe DNS.DClass.ANY
// The second section in the replace is the data that is being passed in, this is different than an add
val dnsRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
val dnsRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
dnsRecord1.getName.toString shouldBe "a-record.vinyldns."
dnsRecord1.getTTL shouldBe testA.ttl
dnsRecord1.getType shouldBe DNS.Type.A
@ -407,7 +407,7 @@ class DnsBackendSpec
val dnsRecord1Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
List("1.1.1.1", "2.2.2.2") should contain(dnsRecord1Data)
val dnsRecord2 = sentMessage.getSectionArray(DNS.Section.UPDATE)(2)
val dnsRecord2 = sentMessage.getSection(DNS.Section.UPDATE).asScala(2)
dnsRecord2.getName.toString shouldBe "a-record.vinyldns."
dnsRecord2.getTTL shouldBe testA.ttl
dnsRecord2.getType shouldBe DNS.Type.A
@ -415,7 +415,7 @@ class DnsBackendSpec
val dnsRecord2Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
List("1.1.1.1", "2.2.2.2") should contain(dnsRecord2Data)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -443,7 +443,7 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE)
emptyRecord shouldBe empty
result shouldBe a[NoError]
@ -457,20 +457,20 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
// A NONE update is sent for each DNS record that is getting deleted
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
emptyRecord.getName.toString shouldBe "a-record.vinyldns."
emptyRecord.getType shouldBe DNS.Type.A
emptyRecord.getDClass shouldBe DNS.DClass.NONE
// The second section in the replace is the data that is being passed in, this is different than an add
val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
dnsRecord.getName.toString shouldBe "a-record.vinyldns."
dnsRecord.getTTL shouldBe testA.ttl
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord shouldBe a[DNS.ARecord]
dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -482,7 +482,7 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val emptyRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)
val emptyRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala
emptyRecord shouldBe empty
result shouldBe a[NoError]
@ -497,14 +497,14 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
// A NONE update is sent for each DNS record that is getting deleted
val deleteRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val deleteRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
deleteRecord1.getName.toString shouldBe "a-record.vinyldns."
deleteRecord1.getType shouldBe DNS.Type.A
deleteRecord1.getDClass shouldBe DNS.DClass.NONE
val deleteRecord1Data = deleteRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
List("4.4.4.4", "3.3.3.3") should contain(deleteRecord1Data)
val deleteRecord2 = sentMessage.getSectionArray(DNS.Section.UPDATE)(1)
val deleteRecord2 = sentMessage.getSection(DNS.Section.UPDATE).asScala(1)
deleteRecord2.getName.toString shouldBe "a-record.vinyldns."
deleteRecord2.getType shouldBe DNS.Type.A
deleteRecord2.getDClass shouldBe DNS.DClass.NONE
@ -512,7 +512,7 @@ class DnsBackendSpec
List("4.4.4.4", "3.3.3.3") should contain(deleteRecord2Data)
// The second section in the replace is the data that is being passed in, this is different than an add
val dnsRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(2)
val dnsRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala(2)
dnsRecord1.getName.toString shouldBe "a-record.vinyldns."
dnsRecord1.getTTL shouldBe testA.ttl
dnsRecord1.getType shouldBe DNS.Type.A
@ -520,7 +520,7 @@ class DnsBackendSpec
val dnsRecord1Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
List("1.1.1.1", "2.2.2.2") should contain(dnsRecord1Data)
val dnsRecord2 = sentMessage.getSectionArray(DNS.Section.UPDATE)(3)
val dnsRecord2 = sentMessage.getSection(DNS.Section.UPDATE).asScala(3)
dnsRecord2.getName.toString shouldBe "a-record.vinyldns."
dnsRecord2.getTTL shouldBe testA.ttl
dnsRecord2.getType shouldBe DNS.Type.A
@ -528,7 +528,7 @@ class DnsBackendSpec
val dnsRecord2Data = dnsRecord1.asInstanceOf[DNS.ARecord].getAddress.getHostAddress
List("1.1.1.1", "2.2.2.2") should contain(dnsRecord2Data)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -556,14 +556,14 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val dnsRecord = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val dnsRecord = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
dnsRecord.getName.toString shouldBe "a-record.vinyldns."
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord.getTTL shouldBe 0
dnsRecord.getDClass shouldBe DNS.DClass.ANY
dnsRecord should not be a[DNS.ARecord]
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]
@ -575,14 +575,14 @@ class DnsBackendSpec
val sentMessage = messageCaptor.getValue
val dnsRecord1 = sentMessage.getSectionArray(DNS.Section.UPDATE)(0)
val dnsRecord1 = sentMessage.getSection(DNS.Section.UPDATE).asScala.head
dnsRecord1.getName.toString shouldBe "a-record.vinyldns."
dnsRecord1.getType shouldBe DNS.Type.A
dnsRecord1.getTTL shouldBe 0
dnsRecord1.getDClass shouldBe DNS.DClass.ANY
dnsRecord1 should not be a[DNS.ARecord]
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = sentMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
result shouldBe a[NoError]

16
modules/api/src/test/scala/vinyldns/api/backend/dns/DnsConversionsSpec.scala
View File

@ -288,7 +288,7 @@ class DnsConversionsSpec
override protected def beforeEach(): Unit = {
doReturn(mockMessage).when(mockMessage).clone()
doReturn(new Array[DNS.Record](0)).when(mockMessage).getSectionArray(DNS.Section.ADDITIONAL)
doReturn(new java.util.ArrayList[DNS.Record]()).when(mockMessage).getSection(DNS.Section.ADDITIONAL)
}
"Collapsing multiple records to record sets" should {
@ -572,47 +572,47 @@ class DnsConversionsSpec
"Converting to an update message" should {
"work for an Add message" in {
val dnsMessage = toAddRecordMessage(rrset(testDnsA), testZoneName).right.value
val dnsRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(0)
val dnsRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala.head
dnsRecord.getName.toString shouldBe "a-record."
dnsRecord.getTTL shouldBe testA.ttl
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord shouldBe a[DNS.ARecord]
dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
}
"work for an Update message" in {
val dnsMessage =
toUpdateRecordMessage(rrset(testDnsA), rrset(testDnsAReplace), testZoneName).right.value
// Update record issues a replace, the first section is an EmptyRecord containing the name and type to replace
val emptyRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(0)
val emptyRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala.head
emptyRecord.getName.toString shouldBe "a-record-2."
emptyRecord.getType shouldBe DNS.Type.A
emptyRecord.getDClass shouldBe DNS.DClass.ANY
// The second section in the replace is the data that is being passed in, this is different than an add
val dnsRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(1)
val dnsRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala(1)
dnsRecord.getName.toString shouldBe "a-record."
dnsRecord.getTTL shouldBe testA.ttl
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord shouldBe a[DNS.ARecord]
dnsRecord.asInstanceOf[DNS.ARecord].getAddress.getHostAddress shouldBe "10.1.1.1"
val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
}
"work for a Delete message" in {
val dnsMessage = toDeleteRecordMessage(rrset(testDnsA), testZoneName).right.value
val dnsRecord = dnsMessage.getSectionArray(DNS.Section.UPDATE)(0)
val dnsRecord = dnsMessage.getSection(DNS.Section.UPDATE).asScala.head
dnsRecord.getName.toString shouldBe "a-record."
dnsRecord.getType shouldBe DNS.Type.A
dnsRecord.getTTL shouldBe 0
dnsRecord.getDClass shouldBe DNS.DClass.ANY
dnsRecord should not be a[DNS.ARecord]
val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE)(0)
val zoneRRset = dnsMessage.getSectionRRsets(DNS.Section.ZONE).asScala.head
zoneRRset.getName.toString shouldBe "vinyldns."
}
}

28
modules/api/src/universal/bin/wait-for-dependencies.sh
View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
# the mysql address, default to a local docker setup
MYSQL_ADDRESS=${MYSQL_ADDRESS:-vinyldns-mysql}
MYSQL_PORT=${MYSQL_PORT:-3306}
echo "Waiting for MYSQL to be ready on $MYSQL_ADDRESS:$MYSQL_PORT"
DATA=""
RETRY=30
while [ $RETRY -gt 0 ]
do
DATA=$(nc -vzw1 $MYSQL_ADDRESS $MYSQL_PORT)
if [ $? -eq 0 ]
then
break
else
echo "Retrying" >&2
let RETRY-=1
sleep .5
if [ $RETRY -eq 0 ]
then
echo "Exceeded retries waiting for MYSQL to be ready on $MYSQL_ADDRESS:$MYSQL_PORT, failing"
return 1
fi
fi
done

3
modules/mysql/src/main/scala/vinyldns/mysql/MySqlConnector.scala
View File

@ -27,7 +27,7 @@ object MySqlConnector {
private val logger = LoggerFactory.getLogger("MySqlConnector")
def runDBMigrations(config: MySqlConnectionConfig): IO[Unit] = {
def runDBMigrations(config: MySqlConnectionConfig): IO[Unit] =
// We can skip migrations for h2, we'll use the test/ddl.sql for initializing
// that for testing
if (config.driver.contains("h2")) IO.unit
@ -61,7 +61,6 @@ object MySqlConnector {
logger.info("migrations complete")
}
}
}
def getDataSource(settings: MySqlDataSourceSettings): IO[HikariDataSource] = IO {

8
modules/portal/karma.conf.js
View File

@ -66,7 +66,13 @@ module.exports = function(config) {
// - Safari (only Mac)
// - PhantomJS
// - IE (only Windows)
browsers: ['ChromeHeadless'],
browsers: ['ChromeHeadlessNoSandbox'],
customLaunchers: {
ChromeHeadlessNoSandbox: {
base: 'ChromeHeadless',
flags: ['--no-sandbox']
}
},
// Continuous Integration mode
// if true, it capture browsers, run tests and exit

2
modules/r53/src/it/scala/vinyldns/route53/backend/Route53IntegrationSpec.scala
View File

@ -52,7 +52,7 @@ class Route53IntegrationSpec
"test",
Option("access"),
Option("secret"),
"http://127.0.0.1:19009",
"http://127.0.0.1:19003",
"us-east-1"
)
)

8
modules/sqs/src/it/resources/application.conf
View File

@ -4,10 +4,10 @@ sqs {
messages-per-poll = 10
max-retries = 100
settings = {
access-key = "x"
secret-key = "x"
signing-region = "x"
service-endpoint = "http://localhost:19007/"
access-key = "test"
secret-key = "test"
signing-region = "us-east-1"
service-endpoint = "http://localhost:19003/"
queue-name = "sqs-override-name"
}
}

7
modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueIntegrationSpec.scala
View File

@ -147,13 +147,6 @@ class SqsMessageQueueIntegrationSpec
result shouldBe empty
}
"succeed when attempting to remove item from empty queue" in {
queue
.remove(SqsMessage(MessageId("does-not-exist"), rsAddChange))
.attempt
.unsafeRunSync() should beRight(())
}
"succeed when attempting to remove item from queue" in {
queue.send(rsAddChange).unsafeRunSync()
val result = queue.receive(MessageCount(1).right.value).unsafeRunSync()

18
modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueProviderIntegrationSpec.scala
View File

@ -36,7 +36,7 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
| max-retries = 100
|
| settings {
| service-endpoint = "http://localhost:19007/"
| service-endpoint = "http://localhost:19003/"
| queue-name = "queue-name"
| }
| """.stripMargin)
@ -59,8 +59,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
| settings {
| access-key = "x"
| secret-key = "x"
| signing-region = "x"
| service-endpoint = "http://localhost:19007/"
| signing-region = "us-east-1"
| service-endpoint = "http://localhost:19003/"
| queue-name = "new-queue"
| }
| """.stripMargin)
@ -86,8 +86,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
| settings {
| access-key = "x"
| secret-key = "x"
| signing-region = "x"
| service-endpoint = "http://localhost:19007/"
| signing-region = "us-east-1"
| service-endpoint = "http://localhost:19003/"
| queue-name = "bad*queue*name"
| }
| """.stripMargin)
@ -108,8 +108,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
| settings {
| access-key = "x"
| secret-key = "x"
| signing-region = "x"
| service-endpoint = "http://localhost:19007/"
| signing-region = "us-east-1"
| service-endpoint = "http://localhost:19003/"
| queue-name = "queue.fifo"
| }
| """.stripMargin)
@ -131,8 +131,8 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers {
| settings {
| access-key = "x"
| secret-key = "x"
| signing-region = "x"
| service-endpoint = "http://localhost:19007/"
| signing-region = "us-east-1"
| service-endpoint = "http://localhost:19003/"
| queue-name = "new-queue"
| }
| """.stripMargin)

4
project/Dependencies.scala
View File

@ -33,7 +33,7 @@ object Dependencies {
"dnsjava" % "dnsjava" % "3.4.2",
"org.apache.commons" % "commons-lang3" % "3.4",
"org.apache.commons" % "commons-text" % "1.4",
"org.flywaydb" % "flyway-core" % "5.1.4",
"org.flywaydb" % "flyway-core" % "8.0.0",
"org.json4s" %% "json4s-ext" % "3.5.3",
"org.json4s" %% "json4s-jackson" % "3.5.3",
"org.scalikejdbc" %% "scalikejdbc" % scalikejdbcV,
@ -77,7 +77,7 @@ object Dependencies {
)
lazy val mysqlDependencies = Seq(
"org.flywaydb" % "flyway-core" % "5.1.4",
"org.flywaydb" % "flyway-core" % "8.0.0",
"org.mariadb.jdbc" % "mariadb-java-client" % "2.3.0",
"org.scalikejdbc" %% "scalikejdbc" % scalikejdbcV,
"org.scalikejdbc" %% "scalikejdbc-config" % scalikejdbcV,

2
project/plugins.sbt
View File

@ -8,8 +8,6 @@ addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.0")
addSbtPlugin("io.github.davidmweber" % "flyway-sbt" % "5.0.0")
addSbtPlugin("org.wartremover" % "sbt-wartremover" % "2.4.10")
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.25")

29
test/api/functional/Dockerfile Normal file
View File

@ -0,0 +1,29 @@
# Build VinylDNS API if the JAR doesn't already exist
FROM vinyldns/build:base-build as base-build
ARG DOCKERFILE_PATH="test/api/functional"
COPY "${DOCKERFILE_PATH}/vinyldns.*" /opt/vinyldns/
COPY . /build/
WORKDIR /build
## Run the build if we don't already have a vinyldns.jar
RUN if [ ! -f /opt/vinyldns/vinyldns.jar ]; then \
env SBT_OPTS="-XX:+UseConcMarkSweepGC -Xmx4G -Xms1G" \
sbt -Dbuild.scalafmtOnCompile=false -Dbuild.lintOnCompile=fase ";project api;coverageOff;assembly" \
&& cp modules/api/target/scala-2.12/vinyldns.jar /opt/vinyldns/; \
fi
# Build the testing image, copying data from `vinyldns-api`
FROM vinyldns/build:base-test
SHELL ["/bin/bash","-c"]
ARG DOCKERFILE_PATH
COPY --from=base-build /opt/vinyldns /opt/vinyldns
# Local bind server files
COPY docker/bind9/etc/named.conf.* /etc/bind/
COPY docker/bind9/zones/ /var/bind/
RUN named-checkconf
# Copy over the functional tests
COPY ${DOCKERFILE_PATH}/test /functional_test
ENTRYPOINT ["/bin/bash", "-c", "/initialize.sh bind localstack vinyldns-api && /functional_test/run.sh \"$@\""]

3
modules/api/functional_test/Dockerfile.dockerignore → test/api/functional/Dockerfile.dockerignore
View File

@ -1,6 +1,5 @@
**/.venv_win
**/.venv*
**/.virtualenv
**/.venv
**/target
**/docs
**/out

45
test/api/functional/Makefile Normal file
View File

@ -0,0 +1,45 @@
SHELL=bash
IMAGE_NAME=vinyldns-api-test
ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
RELATIVE_ROOT_DIR:=$(shell realpath --relative-to=../../.. $(ROOT_DIR))
VINYLDNS_JAR_PATH?=modules/api/target/scala-2.12/vinyldns.jar
# Check that the required version of make is being used
REQ_MAKE_VER:=3.82
ifneq ($(REQ_MAKE_VER),$(firstword $(sort $(MAKE_VERSION) $(REQ_MAKE_VER))))
$(error The version of MAKE $(REQ_MAKE_VER) or higher is required; you are running $(MAKE_VERSION))
endif
# Extract arguments for `make run`
EXTRACT_ARGS=true
ifeq (run,$(firstword $(MAKECMDGOALS)))
EXTRACT_ARGS=true
endif
ifeq ($(EXTRACT_ARGS),true)
# use the rest as arguments for "run"
RUN_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
# ...and turn them into do-nothing targets
$(eval $(RUN_ARGS):;@:)
endif
.ONESHELL:
.PHONY: all build run run-local
all: build run
build:
@set -euo pipefail
trap 'if [ -f "$(ROOT_DIR)/vinyldns.jar" ]; then rm $(ROOT_DIR)/vinyldns.jar; fi' EXIT
cd ../../..
if [ -f modules/api/target/scala-2.12/vinyldns.jar ]; then cp modules/api/target/scala-2.12/vinyldns.jar $(ROOT_DIR)/vinyldns.jar; fi
docker build -t $(IMAGE_NAME) $(DOCKER_PARAMS)--build-arg DOCKERFILE_PATH="$(RELATIVE_ROOT_DIR)" -f "$(ROOT_DIR)/Dockerfile" .
run:
@set -euo pipefail
docker run -it --rm $(DOCKER_PARAMS) -p 9000:9000 -p 19003:19003 -p 19002:19002 -p 19001:19001/tcp -p 19001:19001/udp $(IMAGE_NAME) -- $(RUN_ARGS)
run-local:
@set -euo pipefail
docker run -it --rm $(DOCKER_PARAMS) -p 9000:9000 -p 19003:19003 -p 19002:19002 -p 19001:19001/tcp -p 19001:19001/udp -v "$$(pwd)/test:/functional_test" $(IMAGE_NAME) -- $(RUN_ARGS)

0
modules/api/functional_test/.gitignore → test/api/functional/test/.gitignore vendored Executable file → Normal file
View File

0
modules/api/functional_test/__init__.py → test/api/functional/test/__init__.py
View File

0
modules/api/functional_test/aws_request_signer.py → test/api/functional/test/aws_request_signer.py
View File

2
modules/api/functional_test/conftest.py → test/api/functional/test/conftest.py
View File

@ -29,7 +29,7 @@ def pytest_addoption(parser: _pytest.config.argparsing.Parser) -> None:
Adds additional options that we can parse when we run the tests, stores them in the parser / py.test context
"""
parser.addoption("--url", dest="url", action="store", default="http://localhost:9000", help="URL for application to root")
parser.addoption("--dns-ip", dest="dns_ip", action="store", default="127.0.0.1", help="The ip address for the dns name server to update")
parser.addoption("--dns-ip", dest="dns_ip", action="store", default="127.0.0.1:19001", help="The ip address for the dns name server to update")
parser.addoption("--resolver-ip", dest="resolver_ip", action="store", help="The ip address for the dns server to use for the tests during resolution. This is usually the same as `--dns-ip`")
parser.addoption("--dns-zone", dest="dns_zone", action="store", default="vinyldns.", help="The zone name that will be used for testing")
parser.addoption("--dns-key-name", dest="dns_key_name", action="store", default="vinyldns.", help="The name of the key used to sign updates for the zone")

0
modules/api/functional_test/pytest.ini → test/api/functional/test/pytest.ini
View File

0
modules/api/functional_test/pytest.sh → test/api/functional/test/pytest.sh Executable file → Normal file
View File

0
modules/api/functional_test/requirements.txt → test/api/functional/test/requirements.txt
View File

7
modules/api/functional_test/run.sh → test/api/functional/test/run.sh Executable file → Normal file
View File

@ -10,4 +10,9 @@ if [ "$1" == "--update" ]; then
fi
cd "${ROOT_DIR}"
"./pytest.sh" "${UPDATE_DEPS}" -n4 --suppress-no-test-exit-code -v live_tests "$@"
if [ "$1" == "--interactive" ]; then
shift
bash
else
"./pytest.sh" "${UPDATE_DEPS}" -n4 --suppress-no-test-exit-code -v tests "$@"
fi

0
modules/api/functional_test/live_tests/authentication_test.py → test/api/functional/test/tests/authentication_test.py
View File

0
modules/api/functional_test/live_tests/batch/approve_batch_change_test.py → test/api/functional/test/tests/batch/approve_batch_change_test.py
View File

0
modules/api/functional_test/live_tests/batch/cancel_batch_change_test.py → test/api/functional/test/tests/batch/cancel_batch_change_test.py
View File

0
modules/api/functional_test/live_tests/batch/create_batch_change_test.py → test/api/functional/test/tests/batch/create_batch_change_test.py
View File

0
modules/api/functional_test/live_tests/batch/get_batch_change_test.py → test/api/functional/test/tests/batch/get_batch_change_test.py
View File

0
modules/api/functional_test/live_tests/batch/list_batch_change_summaries_test.py → test/api/functional/test/tests/batch/list_batch_change_summaries_test.py
View File

0
modules/api/functional_test/live_tests/batch/reject_batch_change_test.py → test/api/functional/test/tests/batch/reject_batch_change_test.py
View File

0
modules/api/functional_test/live_tests/conftest.py → test/api/functional/test/tests/conftest.py
View File

0
modules/api/functional_test/live_tests/internal/color_test.py → test/api/functional/test/tests/internal/color_test.py
View File

0
modules/api/functional_test/live_tests/internal/health_test.py → test/api/functional/test/tests/internal/health_test.py
View File

0
modules/api/functional_test/live_tests/internal/ping_test.py → test/api/functional/test/tests/internal/ping_test.py
View File

0
modules/api/functional_test/live_tests/internal/status_test.py → test/api/functional/test/tests/internal/status_test.py
View File

0
modules/api/functional_test/live_tests/list_batch_summaries_test_context.py → test/api/functional/test/tests/list_batch_summaries_test_context.py
View File

0
modules/api/functional_test/live_tests/list_groups_test_context.py → test/api/functional/test/tests/list_groups_test_context.py
View File

0
modules/api/functional_test/live_tests/list_recordsets_test_context.py → test/api/functional/test/tests/list_recordsets_test_context.py
View File

0
modules/api/functional_test/live_tests/list_zones_test_context.py → test/api/functional/test/tests/list_zones_test_context.py
View File

0
modules/api/functional_test/live_tests/membership/create_group_test.py → test/api/functional/test/tests/membership/create_group_test.py
View File

0
modules/api/functional_test/live_tests/membership/delete_group_test.py → test/api/functional/test/tests/membership/delete_group_test.py
View File

0
modules/api/functional_test/live_tests/membership/get_group_changes_test.py → test/api/functional/test/tests/membership/get_group_changes_test.py
View File

0
modules/api/functional_test/live_tests/membership/get_group_test.py → test/api/functional/test/tests/membership/get_group_test.py
View File

0
modules/api/functional_test/live_tests/membership/list_group_admins_test.py → test/api/functional/test/tests/membership/list_group_admins_test.py
View File

0
modules/api/functional_test/live_tests/membership/list_group_members_test.py → test/api/functional/test/tests/membership/list_group_members_test.py
View File

0
modules/api/functional_test/live_tests/membership/list_my_groups_test.py → test/api/functional/test/tests/membership/list_my_groups_test.py
View File

0
modules/api/functional_test/live_tests/membership/update_group_test.py → test/api/functional/test/tests/membership/update_group_test.py
View File

0
modules/api/functional_test/live_tests/production_verify_test.py → test/api/functional/test/tests/production_verify_test.py
View File

2
modules/api/functional_test/live_tests/recordsets/create_recordset_test.py → test/api/functional/test/tests/recordsets/create_recordset_test.py
View File

@ -1,6 +1,6 @@
import pytest
from live_tests.test_data import TestData
from tests.test_data import TestData
from utils import *

2
modules/api/functional_test/live_tests/recordsets/delete_recordset_test.py → test/api/functional/test/tests/recordsets/delete_recordset_test.py
View File

@ -1,6 +1,6 @@
import pytest
from live_tests.test_data import TestData
from tests.test_data import TestData
from utils import *

0
modules/api/functional_test/live_tests/recordsets/get_recordset_test.py → test/api/functional/test/tests/recordsets/get_recordset_test.py
View File

0
modules/api/functional_test/live_tests/recordsets/list_recordset_changes_test.py → test/api/functional/test/tests/recordsets/list_recordset_changes_test.py
View File

Some files were not shown because too many files have changed in this diff Show More