diff --git a/AUTHORS.md b/AUTHORS.md index bd5098d89..6e42640fd 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -3,14 +3,9 @@ This project would not be possible without the generous contributions of many people. Thank you! If you have contributed in any way, but do not see your name here, please open a PR to add yourself (in alphabetical order by last name)! -## DNS SMEs - -- Joe Crowe -- David Back -- Hong Ye - ## Contributors +- David Back - Mike Ball - Tommy Barker - Robert Barrimond @@ -23,6 +18,7 @@ in any way, but do not see your name here, please open a PR to add yourself (in - Peter Cline - Kemar Cockburn - Luke Cori +- Joe Crowe - Jearvon Dharrie - Andrew Dunn - Ryan Emerle @@ -54,3 +50,4 @@ in any way, but do not see your name here, please open a PR to add yourself (in - Andrew Wang - Peter Willis - Britney Wright +- Hong Ye diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 5bfce78b9..4d66eab0e 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -2,10 +2,28 @@ ## Table of Contents -- [Developer Requirements](#developer-requirements) +- [Developer Requirements (Local)](#developer-requirements-local) +- [Developer Requirements (Docker)](#developer-requirements-docker) - [Project Layout](#project-layout) + * [Core](#core) + * [API](#api) + * [Portal](#portal) + * [Documentation](#documentation) - [Running VinylDNS Locally](#running-vinyldns-locally) + * [Starting the API Server](#starting-the-api-server) + * [Starting the Portal](#starting-the-portal) - [Testing](#testing) + * [Unit Tests](#unit-tests) + * [Integration Tests](#integration-tests) + + [Running both](#running-both) + * [Functional Tests](#functional-tests) + + [Running Functional Tests](#running-functional-tests) + - [API Functional Tests](#api-functional-tests) + + [Setup](#setup) + - [Functional Test Context](#functional-test-context) + - [Partitioning](#partitioning) + - [Really Important Test Context Rules!](#really-important-test-context-rules) + - [Managing Test Zone Files](#managing-test-zone-files) ## Developer Requirements (Local) @@ -41,14 +59,14 @@ its components. The main codebase is a multi-module Scala project with multiple sub-modules. To start working with the project, from the root directory run `sbt`. Most of the code can be found in the `modules` directory. The following modules are present: -* `root` - this is the parent project, if you run tasks here, it will run against all sub-modules -* [`core`](#core): core modules that are used by both the API and portal, such as cryptography implementations. -* [`api`](#api): the API is the main engine for all of VinylDNS. This is the most active area of the codebase, as +- `root` - this is the parent project, if you run tasks here, it will run against all sub-modules +- [`core`](#core): core modules that are used by both the API and portal, such as cryptography implementations. +- [`api`](#api): the API is the main engine for all of VinylDNS. This is the most active area of the codebase, as everything else typically just funnels through the API. -* [`portal`](#portal): The portal is a user interface wrapper around the API. Most of the business rules, logic, and +- [`portal`](#portal): The portal is a user interface wrapper around the API. Most of the business rules, logic, and processing can be found in the API. The _only_ features in the portal not found in the API are creation of users and user authentication. -* [`docs`](#documentation): documentation for VinylDNS. +- [`docs`](#documentation): documentation for VinylDNS. ### Core @@ -56,76 +74,76 @@ Code that is used across multiple modules in the VinylDNS ecosystem live in `cor #### Code Layout -* `src/main` - the main source code -* `src/test` - unit tests +- `src/main` - the main source code +- `src/test` - unit tests ### API The API is the RESTful API for interacting with VinylDNS. The following technologies are used: -* [Akka HTTP](https://doc.akka.io/docs/akka-http/current/) - Used primarily for REST and HTTP calls. -* [FS2](https://functional-streams-for-scala.github.io/fs2/) - Used for backend change processing off of message queues. +- [Akka HTTP](https://doc.akka.io/docs/akka-http/current/) - Used primarily for REST and HTTP calls. +- [FS2](https://functional-streams-for-scala.github.io/fs2/) - Used for backend change processing off of message queues. FS2 has back-pressure built in, and gives us tools like throttling and concurrency. -* [Cats Effect](https://typelevel.org/cats-effect/) - A replacement of `Future` with the `IO` monad -* [Cats](https://typelevel.org/cats) - Used for functional programming. -* [PureConfig](https://pureconfig.github.io/) - For loading configuration values. +- [Cats Effect](https://typelevel.org/cats-effect/) - A replacement of `Future` with the `IO` monad +- [Cats](https://typelevel.org/cats) - Used for functional programming. +- [PureConfig](https://pureconfig.github.io/) - For loading configuration values. The API has the following dependencies: -* MySQL - the SQL database that houses the data -* SQS - for managing concurrent updates and enabling high-availability -* Bind9 - for testing integration with a real DNS system +- MySQL - the SQL database that houses the data +- SQS - for managing concurrent updates and enabling high-availability +- Bind9 - for testing integration with a real DNS system #### Code Layout The API code can be found in `modules/api`. -* `src/it` - integration tests -* `src/main` - the main source code -* `src/test` - unit tests -* `src/universal` - items that are packaged in the Docker image for the VinylDNS API +- `src/it` - integration tests +- `src/main` - the main source code +- `src/test` - unit tests +- `src/universal` - items that are packaged in the Docker image for the VinylDNS API The package structure for the source code follows: -* `vinyldns.api.domain` - contains the core front-end logic. This includes things like the application services, +- `vinyldns.api.domain` - contains the core front-end logic. This includes things like the application services, repository interfaces, domain model, validations, and business rules. -* `vinyldns.api.engine` - the back-end processing engine. This is where we process commands including record changes, +- `vinyldns.api.engine` - the back-end processing engine. This is where we process commands including record changes, zone changes, and zone syncs. -* `vinyldns.api.protobuf` - marshalling and unmarshalling to and from protobuf to types in our system -* `vinyldns.api.repository` - repository implementations live here -* `vinyldns.api.route` - HTTP endpoints +- `vinyldns.api.protobuf` - marshalling and unmarshalling to and from protobuf to types in our system +- `vinyldns.api.repository` - repository implementations live here +- `vinyldns.api.route` - HTTP endpoints ### Portal The project is built using: -* [Play Framework](https://www.playframework.com/documentation/2.6.x/Home) -* [AngularJS](https://angularjs.org/) +- [Play Framework](https://www.playframework.com/documentation/2.6.x/Home) +- [AngularJS](https://angularjs.org/) The portal is _mostly_ a shim around the API. Most actions in the user interface are translated into API calls. The features that the Portal provides that are not in the API include: -* Authentication against LDAP -* Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials +- Authentication against LDAP +- Creation of users - when a user logs in for the first time, VinylDNS automatically creates a user and new credentials for them in the database with their LDAP information. #### Code Layout The portal code can be found in `modules/portal`. -* `app` - source code for portal back-end - * `models` - data structures that are used by the portal - * `views` - HTML templates for each web page - * `controllers` - logic for updating data -* `conf` - configurations and endpoint routes -* `public` - source code for portal front-end - * `css` - stylesheets - * `images` - images, including icons, used in the portal - * `js` - scripts - * `mocks` - mock JSON used in Grunt tests - * `templates` - modal templates -* `test` - unit tests for portal back-end +- `app` - source code for portal back-end + - `models` - data structures that are used by the portal + - `views` - HTML templates for each web page + - `controllers` - logic for updating data +- `conf` - configurations and endpoint routes +- `public` - source code for portal front-end + - `css` - stylesheets + - `images` - images, including icons, used in the portal + - `js` - scripts + - `mocks` - mock JSON used in Grunt tests + - `templates` - modal templates +- `test` - unit tests for portal back-end ### Documentation @@ -134,8 +152,8 @@ settings for the microsite are also configured in `build.sbt` of the project roo #### Code Layout -* `src/main/resources` - Microsite resources and configurations -* `src/main/mdoc` - Content for microsite web pages +- `src/main/resources` - Microsite resources and configurations +- `src/main/mdoc` - Content for microsite web pages ## Running VinylDNS Locally @@ -146,20 +164,19 @@ README. However, VinylDNS can also be run in the foreground. Before starting the API service, you can start the dependencies for local development: -``` -cd test/api/integration -make build && make run-bg +```shell +quickstart/quickstart-vinyldns.sh --deps-only ``` This will start a container running in the background with necessary prerequisites. Once the prerequisites are running, you can start up sbt by running `sbt` from the root directory. -* `project api` to change the sbt project to the API -* `reStart` to start up the API server -* Wait until you see the message `VINYLDNS SERVER STARTED SUCCESSFULLY` before working with the server -* To stop the VinylDNS server, run `reStop` from the api project -* To stop the dependent Docker containers: `utils/clean-vinyldns-containers.sh` +- `project api` to change the sbt project to the API +- `reStart` to start up the API server +- Wait until you see the message `VINYLDNS SERVER STARTED SUCCESSFULLY` before working with the server +- To stop the VinylDNS server, run `reStop` from the api project +- To stop the dependent Docker containers: `utils/clean-vinyldns-containers.sh` See the [API Configuration Guide](https://www.vinyldns.io/operator/config-api) for information regarding API configuration. @@ -167,9 +184,9 @@ configuration. ### Starting the Portal To run the portal locally, you _first_ have to start up the VinylDNS API Server. This can be done by following the -instructions for [Staring the API Server](#Starting the API Server) or by using the QuickStart: +instructions for [Staring the API Server](#starting-the-api-server) or by using the QuickStart: -``` +```shell quickstart/quickstart-vinyldns.sh --api-only ``` @@ -179,29 +196,6 @@ execute `;preparePortal; run`. See the [Portal Configuration Guide](https://www.vinyldns.io/operator/config-portal) for information regarding portal configuration. -### Loading test data - -Normally the portal can be used for all VinylDNS requests. Test users are locked down to only have access to test zones, -which the portal connection modal has not been updated to incorporate. To connect to a zone with testuser, you will need -to use an alternative client and set `isTest=true` on the zone being connected to. - -Use the vinyldns-js client (Note, you need Node installed): - -``` -git clone https://github.com/vinyldns/vinyldns-js.git -cd vinyldns-js -npm install -export VINYLDNS_API_SERVER=http://localhost:9000 -export VINYLDNS_ACCESS_KEY_ID=testUserAccessKey -export VINYLDNS_SECRET_ACCESS_KEY=testUserSecretKey -npm run repl -> var groupId; -> vinyl.createGroup({"name": "test-group", "email":"test@test.com", members: [{id: "testuser"}], admins: [{id: "testuser"}]}).then(res => {groupId = res.id}).catch(err => {console.log(err)}); -> vinyl.createZone ({name: "ok.", isTest: true, adminGroupId: groupId, email: "test@test.com"}).then(res => { console.log(res) }).catch(err => { console.log(err) }) - -You should now be able to see the zone in the portal at localhost:9001 when logged in as username=testuser password=testpassword -``` - ## Testing ### Unit Tests @@ -247,7 +241,7 @@ When adding new features, you will often need to write new functional tests that To run functional tests you can simply execute the following commands: -``` +```shell build/func-test-api.sh build/func-test-portal.sh ``` @@ -259,9 +253,9 @@ These command will run the API functional tests and portal functional tests resp To run functional tests you can simply execute `build/func-test-api.sh`, but if you'd like finer-grained control, you can work with the `Makefile` in `test/api/functional`: -``` -cd test/api/functional -make build && make run +```shell +# Build and then run the function test container +make -C test/api/functional build run ``` During iterative test development, you can use `make run-local` which will bind-mount the current functional tests in @@ -289,20 +283,20 @@ There aren't a lot, so it should be quick. In the `modules/api/src/test/functional` directory are a few important files for you to be familiar with: -* `vinyl_python.py` - this provides the interface to the VinylDNS API. It handles signing the request for you, as well +- `vinyl_python.py` - this provides the interface to the VinylDNS API. It handles signing the request for you, as well as building and executing the requests, and giving you back valid responses. For all new API endpoints, there should be a corresponding function in the vinyl_client -* `utils.py` - provides general use functions that can be used anywhere in your tests. Feel free to contribute new +- `utils.py` - provides general use functions that can be used anywhere in your tests. Feel free to contribute new functions here when you see repetition in the code In the `modules/api/src/test/functional/tests` directory, we have directories / modules for different areas of the application. -* `batch` - for managing batch updates -* `internal` - for internal endpoints (not intended for public consumption) -* `membership` - for managing groups and users -* `recordsets` - for managing record sets -* `zones` - for managing zones +- `batch` - for managing batch updates +- `internal` - for internal endpoints (not intended for public consumption) +- `membership` - for managing groups and users +- `recordsets` - for managing record sets +- `zones` - for managing zones ##### Functional Test Context diff --git a/MAINTAINERS.md b/MAINTAINERS.md index a048cc87f..6a0e5ebc6 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -27,12 +27,6 @@ The offline root key and repository keys are managed by the core maintainer team * api key: used to sign tagged images in vinyldns/api * portal key: used to sign tagged images in vinyldns/portal -These keys are named in a .key format, e.g. 5526ecd15bd413e08718e66c440d17a28968d5cd2922b59a17510da802ca6572.key, -do not change the names of the keys. - -Docker expects these keys to be saved in `~/.docker/trust/private`. Each key is encrypted with a passphrase, that you -must have available when pushing an image. - ## Release Process The release process is automated by GitHub Actions. diff --git a/README.md b/README.md index 4068fd50f..6e262c095 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,9 @@ ![Build](https://github.com/vinyldns/vinyldns/workflows/Continuous%20Integration/badge.svg) -[![CodeCov ](https://codecov.io/gh/vinyldns/vinyldns/branch/master/graph/badge.svg)](https://codecov.io/gh/vinyldns/vinyldns) [![License](https://img.shields.io/github/license/vinyldns/vinyldns)](https://github.com/vinyldns/vinyldns/blob/master/LICENSE) [![conduct](https://img.shields.io/badge/%E2%9D%A4-code%20of%20conduct-blue.svg)](https://github.com/vinyldns/vinyldns/blob/master/CODE_OF_CONDUCT.md)

- + VinylDNS that can be used to make API requests, using the endpoint `http://localhost:9000` -## Things to try in the portal +## Things to Try in the Portal 1. View the portal at in a web browser 2. Login with the credentials `professor` and `professor` 3. Navigate to the `groups` tab: 4. Click on the **New Group** button and create a new group, the group id is the uuid in the url after you view the group -5. View zones you connected to in the `zones` tab: . For a quick test, create a new zone - named `ok` with an email of `test@test.com` and choose a group you created from the previous step. (Note, - see [Developer Guide](DEVELOPER_GUIDE.md#loading-test-data) for creating a zone) -6. You will see that some records are preloaded in the zoned already, this is because these records are preloaded in the +5. Connect a zone by going to the `zones` tab: . + 1. Click the `-> Connect` button + 2. For `Zone Name` enter `ok` with an email of `test@test.com` + 3. For `Admin Group`, choose a group you created from the previous step + 4. Leave everything else as-is and click the `Connect` button at the bottom of the form +6. A new zone `ok` should appear in your `My Zones` tab _(you may need to refresh your browser)_ +7. You will see that some records are preloaded in the zone already, this is because these records are preloaded in the local docker DNS server and VinylDNS automatically syncs records with the backend DNS server upon zone connection -7. From here, you can create DNS record sets in the **Manage Records** tab, and manage zone settings and ***ACL rules*** +8. From here, you can create DNS record sets in the **Manage Records** tab, and manage zone settings and ***ACL rules*** in the **Manage Zone** tab -8. To try creating a DNS record, click on the **Create Record Set** button under +9. To try creating a DNS record, click on the **Create Record Set** button under Records, `Record Type = A, Record Name = my-test-a, TTL = 300, IP Addressess = 1.1.1.1` -9. Click on the **Refresh** button under Records, you should see your new record created +10. Click on the **Refresh** button under Records, you should see your new record created ### Verifying Your Changes @@ -96,7 +101,7 @@ This tells `dig` to use `127.0.0.1` as the resolver on port `19001`. The `+short verbose. Finally, the record we're looking up is `my-test-a.ok`. You can see the returned output of `1.1.1.1` matches the record data we entered. -## Other things to note +### Other things to note 1. Upon connecting to a zone for the first time, a zone sync is executed to provide VinylDNS a copy of the records in the zone diff --git a/SYSTEM_DESIGN.md b/SYSTEM_DESIGN.md index 8654c7526..1eedc2bdd 100644 --- a/SYSTEM_DESIGN.md +++ b/SYSTEM_DESIGN.md @@ -1,56 +1,57 @@ # System Design ## Table of Contents + - [Components](#components) - [Process Flow](#process-flow) - [Integration](#integration) ## Components -The following diagram illustrates the major components in the VinylDNS ecosystem and the external systems they interact with. +The following diagram illustrates the major components in the VinylDNS ecosystem and the external systems they interact +with. -![VinylDNS Architecture Diagram](img/VinylDNS_overview.png) +![VinylDNS Architecture Diagram](img/vinyldns_overview.png) -* API - RESTful endpoints to allow interaction with VinylDNS - -* Database - stores information that the VinylDNS application needs - -* DNS servers - communicates DNS changes and resolves DNS records - -* Message queue - temporarily stores DNS requests for processing - -* LDAP Service - application protocol used to authenticate user access to the VinylDNS portal - -* Portal - graphical user interface to interact with the VinylDNS API - -* Tooling - external libraries and utilities used to interact with the VinylDNS API +| Component | Description | +|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Portal | Web user interface to interact with the VinylDNS API | +| API | RESTful endpoints to allow interaction with VinylDNS | +| API Worker Nodes | These are API components with `processing-disabled` set to `false` (see [documentation](https://www.vinyldns.io/operator/config-api.html#processing-disabled)) | +| Message queue | Queue for DNS commands to enable flow control to the DNS backends (see [documentation](https://www.vinyldns.io/operator/pre.html#message-queues)) | +| Database | Stores information about users, membership, and DNS records | +| DNS Backend(s) | The DNS backend servers which VinylDNS will query and update. | +| LDAP Service | The optional LDAP service that VinylDNS can be configured to communicate with (see [documentation](https://www.vinyldns.io/operator/setup-ldap.html#setup-ldap)) | ## Process Flow 1. LDAP service authenticates user credentials and grants access to the portal. 1. If the user is accessing the portal for the first time, VinylDNS credentials are generated and stored. 1. User navigates portal or uses integration tooling to generate a signed API request. -1. When the API receives a request, it loads the credentials for the calling user from the database and validates the request signature to ensure that the request was not modified in transit. +1. When the API receives a request, it loads the credentials for the calling user from the database and validates the + request signature to ensure that the request was not modified in transit. 1. The request is then validated to ensure that: - - the request data is correct + - the request data is correct - the request passes all validation checks - the user has access to make the change 1. Assuming the request is in good order, the request is put on a message queue for handling. -1. One of the VinylDNS API server instances pulls the message from the queue for processing. For record changes, a DDNS message is issued to the DNS backend server. -1. When the message completes processing, it is removed from the message queue. The changes are applied to the VinylDNS database along with an audit record for the request. +1. One of the VinylDNS API server instances pulls the message from the queue for processing. For record changes, a DDNS + message is issued to the DNS backend server. +1. When the message completes processing, it is removed from the message queue. The changes are applied to the VinylDNS + database along with an audit record for the request. ## Integration -Integrating with VinylDNS is simple since each API endpoint is effectively a distinct DNS operation (eg. create record, update record, delete record, etc.). The only requirement for sending a request is generating the correct AWS SIG4 signature without content length and providing the corresponding HTTP headers so that VinylDNS can verify it. See [API Authentication](https://www.vinyldns.io/api/auth-mechanism.html) for more details. +Integrating with VinylDNS is simple since each API endpoint is effectively a distinct DNS operation (eg. create record, +update record, delete record, etc.). The only requirement for sending a request is generating the correct AWS SIG4 +signature without content length and providing the corresponding HTTP headers so that VinylDNS can verify it. +See [API Authentication](https://www.vinyldns.io/api/auth-mechanism.html) for more details. The current tooling available to perform VinylDNS API requests include: * [go-vinyldns](https://github.com/vinyldns/go-vinyldns) - Golang client package - -* [vinyldns-cli](https://github.com/vinyldns/vinyldns-cli) - command line utility written in Golang - +* [terraform-provider-vinyldns](https://github.com/vinyldns/terraform-provider-vinyldns) - A [Terraform](https://terraform.io/) provider for VinylDNS +* [vinyldns-cli](https://github.com/vinyldns/vinyldns-cli) - Command line utility written in Golang * [vinyldns-java](https://github.com/vinyldns/vinyldns-java) - Java client - +* [vinyldns-js](https://github.com/vinyldns/vinyldns-js) - JavaScript client * [vinyldns-python](https://github.com/vinyldns/vinyldns-python) - Python client library - -* [vinyldns-ruby](https://github.com/vinyldns/vinyldns-ruby) - Ruby gem diff --git a/build.sbt b/build.sbt index a1af2654b..006c919a6 100644 --- a/build.sbt +++ b/build.sbt @@ -5,6 +5,7 @@ import org.scalafmt.sbt.ScalafmtPlugin._ import scoverage.ScoverageKeys.{coverageFailOnMinimum, coverageMinimum} import scala.language.postfixOps +import scala.sys.env import scala.util.Try lazy val IntegrationTest = config("it").extend(Test) @@ -16,9 +17,9 @@ lazy val sharedSettings = Seq( organizationName := "Comcast Cable Communications Management, LLC", startYear := Some(2018), licenses += ("Apache-2.0", new URL("https://www.apache.org/licenses/LICENSE-2.0.txt")), - maintainer := "VinylDNS Maintainers ", + maintainer := "VinylDNS Maintainers ", scalacOptions ++= scalacOptionsByV(scalaVersion.value), - scalacOptions in (Compile, doc) += "-no-link-warnings", + scalacOptions in(Compile, doc) += "-no-link-warnings", // Use wart remover to eliminate code badness wartremoverErrors := ( if (getPropertyFlagOrDefault("build.lintOnCompile", true)) @@ -31,13 +32,13 @@ lazy val sharedSettings = Seq( Wart.ExplicitImplicitTypes ) else Seq.empty - ), + ), // scala format scalafmtOnCompile := getPropertyFlagOrDefault("build.scalafmtOnCompile", false), // coverage options coverageMinimum := 85, coverageFailOnMinimum := true, - coverageHighlighting := true + coverageHighlighting := true, ) lazy val testSettings = Seq( @@ -287,7 +288,6 @@ lazy val docs = (project in file("modules/docs")) .settings(docSettings) - def getPropertyFlagOrDefault(name: String, value: Boolean): Boolean = sys.props.get(name).flatMap(propValue => Try(propValue.toBoolean).toOption).getOrElse(value) diff --git a/build/docker/.env b/build/docker/.env index 4bb852ad4..f5bcadd7d 100644 --- a/build/docker/.env +++ b/build/docker/.env @@ -7,7 +7,7 @@ TEST_LOGIN=false # API Settings REST_PORT=9000 -SQS_ENDPOINT=http://vinyldns-integration:19003 +SQS_SERVICE_ENDPOINT=http://vinyldns-integration:19003 SNS_SERVICE_ENDPOINT=http://vinyldns-integration:19003 MYSQL_ENDPOINT=vinyldns-integration:19002 DEFAULT_DNS_ADDRESS=vinyldns-integration:19001 diff --git a/build/docker/api/application.conf b/build/docker/api/application.conf index b1f9eb837..5857c4599 100644 --- a/build/docker/api/application.conf +++ b/build/docker/api/application.conf @@ -49,7 +49,7 @@ vinyldns { key-name = ${?DEFAULT_DNS_KEY_NAME} key = "nzisn+4G2ldMn0q1CV3vsg==" key = ${?DEFAULT_DNS_KEY_SECRET} - primary-server = "127.0.0.1" + primary-server = "127.0.0.1:19001" primary-server = ${?DEFAULT_DNS_ADDRESS} } transfer-connection = { @@ -58,7 +58,7 @@ vinyldns { key-name = ${?DEFAULT_DNS_KEY_NAME} key = "nzisn+4G2ldMn0q1CV3vsg==" key = ${?DEFAULT_DNS_KEY_SECRET} - primary-server = "127.0.0.1" + primary-server = "127.0.0.1:19001" primary-server = ${?DEFAULT_DNS_ADDRESS} }, tsig-usage = "always" @@ -71,7 +71,7 @@ vinyldns { key-name = ${?DEFAULT_DNS_KEY_NAME} key = "nzisn+4G2ldMn0q1CV3vsg==" key = ${?DEFAULT_DNS_KEY_SECRET} - primary-server = "127.0.0.1" + primary-server = "127.0.0.1:19001" primary-server = ${?DEFAULT_DNS_ADDRESS} } transfer-connection = { @@ -80,7 +80,7 @@ vinyldns { key-name = ${?DEFAULT_DNS_KEY_NAME} key = "nzisn+4G2ldMn0q1CV3vsg==" key = ${?DEFAULT_DNS_KEY_SECRET} - primary-server = "127.0.0.1" + primary-server = "127.0.0.1:19001" primary-server = ${?DEFAULT_DNS_ADDRESS} }, tsig-usage = "always" diff --git a/build/sbt.sh b/build/sbt.sh index 730b36b61..e02750d64 100755 --- a/build/sbt.sh +++ b/build/sbt.sh @@ -4,4 +4,4 @@ set -euo pipefail DIR=$(cd -P -- "$(dirname -- "$0")" && pwd -P) cd "$DIR/../test/api/integration" -make build DOCKER_PARAMS="--build-arg SKIP_API_BUILD=true" && make run-local WITH_ARGS="sbt" DOCKER_PARAMS="-e RUN_SERVICES=none" +make build DOCKER_PARAMS="--build-arg SKIP_API_BUILD=true" && make run-local WITH_ARGS="sbt" DOCKER_PARAMS="-e RUN_SERVICES=none --env-file \"$DIR/../test/api/integration/.env.integration\"" diff --git a/img/VinylDNS_overview.png b/img/VinylDNS_overview.png deleted file mode 100644 index b673b0778..000000000 Binary files a/img/VinylDNS_overview.png and /dev/null differ diff --git a/img/vinyldns_overview.png b/img/vinyldns_overview.png new file mode 100755 index 000000000..cc33b79d8 Binary files /dev/null and b/img/vinyldns_overview.png differ diff --git a/modules/api/src/it/resources/application.conf b/modules/api/src/it/resources/application.conf index df6975a45..5857c4599 100644 --- a/modules/api/src/it/resources/application.conf +++ b/modules/api/src/it/resources/application.conf @@ -1,63 +1,268 @@ -akka { - loglevel = "OFF" - log-dead-letters-during-shutdown = off - log-dead-letters = 0 - logger-startup-timeout = 60s -} - vinyldns { + base-version = "0.0.0-local-dev" + version = ${vinyldns.base-version} # default to the base version if not overridden + version = ${?VINYLDNS_VERSION} # override the base version via env var + + # How often to any particular zone can be synchronized in milliseconds + sync-delay = 10000 + sync-delay = ${?SYNC_DELAY} + + # If we should start up polling for change requests, set this to false for the inactive cluster + processing-disabled = false + processing-disabled = ${?PROCESSING_DISABLED} + + # Number of records that can be in a zone + max-zone-size = 60000 + max-zone-size = ${?MAX_ZONE_SIZE} + + # Types of unowned records that users can access in shared zones + shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] + + # Batch change settings + batch-change-limit = 1000 + batch-change-limit = ${?BATCH_CHANGE_LIMIT} + manual-batch-review-enabled = true + manual-batch-review-enabled = ${?MANUAL_BATCH_REVIEW_ENABLED} + scheduled-changes-enabled = true + scheduled-changes-enabled = ${?SCHEDULED_CHANGES_ENABLED} + multi-record-batch-change-enabled = true + multi-record-batch-change-enabled = ${?MULTI_RECORD_BATCH_CHANGE_ENABLED} + + # configured backend providers + backend { + # Use "default" when dns backend legacy = true + # otherwise, use the id of one of the connections in any of your backends + default-backend-id = "default" + + # this is where we can save additional backends + backend-providers = [ + { + class-name = "vinyldns.api.backend.dns.DnsBackendProviderLoader" + settings = { + legacy = false + backends = [ + { + id = "default" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + }, + { + id = "func-test-backend" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + } + ] + } + } + ] + } + + + queue { + class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + + messages-per-poll = 10 + polling-interval = 250.millis + + settings { + # AWS access key and secret. + access-key = "test" + access-key = ${?AWS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?AWS_SECRET_ACCESS_KEY} + + # Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed. + signing-region = "us-east-1" + signing-region = ${?SQS_REGION} + + # Endpoint to access queue + service-endpoint = "http://vinyldns-integration:19003/" + service-endpoint = ${?SQS_SERVICE_ENDPOINT} + + # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change. + queue-name = "vinyldns" + queue-name = ${?SQS_QUEUE_NAME} + } + } + + email { + class-name = "vinyldns.api.notifier.email.EmailNotifierProvider" + class-name = ${?EMAIL_CLASS_NAME} + settings = { + from = "VinylDNS " + from = ${?EMAIL_FROM} + } + } + + sns { + class-name = "vinyldns.apadi.notifier.sns.SnsNotifierProvider" + class-name = ${?SNS_CLASS_NAME} + settings { + topic-arn = "arn:aws:sns:us-east-1:000000000000:batchChanges" + topic-arn = ${?SNS_TOPIC_ARN} + access-key = "test" + access-key = ${?SNS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?SNS_SECRET_KEY} + service-endpoint = "http://vinyldns-integration:19003" + service-endpoint = ${?SNS_SERVICE_ENDPOINT} + signing-region = "us-east-1" + signing-region = ${?SNS_REGION} + } + } + + rest { + host = "0.0.0.0" + port = 9000 + port=${?API_SERVICE_PORT} + } + + approved-name-servers = [ - "172.17.42.1." + "172.17.42.1.", + "ns1.parent.com." + "ns1.parent.com1." + "ns1.parent.com2." + "ns1.parent.com3." + "ns1.parent.com4." ] + # Note: This MUST match the Portal or strange errors will ensue, NoOpCrypto should not be used for production + crypto { + type = "vinyldns.core.crypto.NoOpCrypto" + type = ${?CRYPTO_TYPE} + secret = ${?CRYPTO_SECRET} + } + + data-stores = ["mysql"] mysql { settings { - # see https://github.com/brettwooldridge/HikariCP - connection-timeout-millis = 1000 - idle-timeout = 10000 - max-lifetime = 600000 - maximum-pool-size = 5 - minimum-idle = 1 - my-sql-properties = { - cachePrepStmts=true - prepStmtCacheSize=250 - prepStmtCacheSqlLimit=2048 - rewriteBatchedStatements=true - } - + # JDBC Settings, these are all values in scalikejdbc-config, not our own + # these must be overridden to use MYSQL for production use + # assumes a docker or mysql instance running locally + name = "vinyldns" + name = ${?DATABASE_NAME} + driver = "org.h2.Driver" + driver = ${?JDBC_DRIVER} + migration-url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + migration-url = ${?JDBC_MIGRATION_URL} + url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + url = ${?JDBC_URL} + user = "sa" + user = ${?JDBC_USER} + password = "" + password = ${?JDBC_PASSWORD} } + # TODO: Remove the need for these useless configuration blocks repositories { zone { - # no additional settings for now } batch-change { - # no additional settings for now } user { - } record-set { - - } - group { - - } - membership { - - } - group-change { - } zone-change { - } record-change { - + } + group { + } + group-change { + } + membership { } } } + backends = [] + + # FQDNs / IPs that cannot be modified via VinylDNS + # regex-list used for all record types except PTR + # ip-list used exclusively for PTR records + high-value-domains = { + regex-list = [ + "high-value-domain.*" # for testing + ] + ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.2.252", + "192.0.2.253", + "fd69:27cc:fe91:0:0:0:0:ffff", + "fd69:27cc:fe91:0:0:0:ffff:0" + ] + } + + # FQDNs / IPs / zone names that require manual review upon submission in batch change interface + # domain-list used for all record types except PTR + # ip-list used exclusively for PTR records + manual-review-domains = { + domain-list = [ + "needs-review.*" + ] + ip-list = [ + "192.0.1.254", + "192.0.1.255", + "192.0.2.254", + "192.0.2.255", + "192.0.3.254", + "192.0.3.255", + "192.0.4.254", + "192.0.4.255", + "fd69:27cc:fe91:0:0:0:ffff:1", + "fd69:27cc:fe91:0:0:0:ffff:2", + "fd69:27cc:fe92:0:0:0:ffff:1", + "fd69:27cc:fe92:0:0:0:ffff:2", + "fd69:27cc:fe93:0:0:0:ffff:1", + "fd69:27cc:fe93:0:0:0:ffff:2", + "fd69:27cc:fe94:0:0:0:ffff:1", + "fd69:27cc:fe94:0:0:0:ffff:2" + ] + zone-name-list = [ + "zone.requires.review." + "zone.requires.review1." + "zone.requires.review2." + "zone.requires.review3." + "zone.requires.review4." + ] + } + # FQDNs / IPs that cannot be modified via VinylDNS # regex-list used for all record types except PTR # ip-list used exclusively for PTR records @@ -66,31 +271,65 @@ vinyldns { "high-value-domain.*" # for testing ] ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.1.252", + "192.0.1.253", "192.0.2.252", "192.0.2.253", + "192.0.3.252", + "192.0.3.253", + "192.0.4.252", + "192.0.4.253", "fd69:27cc:fe91:0:0:0:0:ffff", - "fd69:27cc:fe91:0:0:0:ffff:0" + "fd69:27cc:fe91:0:0:0:ffff:0", + "fd69:27cc:fe92:0:0:0:0:ffff", + "fd69:27cc:fe92:0:0:0:ffff:0", + "fd69:27cc:fe93:0:0:0:0:ffff", + "fd69:27cc:fe93:0:0:0:ffff:0", + "fd69:27cc:fe94:0:0:0:0:ffff", + "fd69:27cc:fe94:0:0:0:ffff:0" ] } - # types of unowned records that users can access in shared zones - shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] + global-acl-rules = [ + { + group-ids: ["global-acl-group-id"], + fqdn-regex-list: [".*shared[0-9]{1}."] + }, + { + group-ids: ["another-global-acl-group"], + fqdn-regex-list: [".*ok[0-9]{1}."] + } + ] +} - crypto { - type = "vinyldns.core.crypto.NoOpCrypto" - } +akka { + loglevel = "INFO" + loggers = ["akka.event.slf4j.Slf4jLogger"] + logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" + logger-startup-timeout = 30s - email.settings.smtp { - port = 19025 + actor { + provider = "akka.actor.LocalActorRefProvider" } } -# Global settings -scalikejdbc.global.loggingSQLAndTime.enabled=true -scalikejdbc.global.loggingSQLAndTime.logLevel=error -scalikejdbc.global.loggingSQLAndTime.warningEnabled=true -scalikejdbc.global.loggingSQLAndTime.warningThresholdMillis=1000 -scalikejdbc.global.loggingSQLAndTime.warningLogLevel=warn -scalikejdbc.global.loggingSQLAndTime.singleLineMode=false -scalikejdbc.global.loggingSQLAndTime.printUnprocessedStackTrace=false -scalikejdbc.global.loggingSQLAndTime.stackTraceDepth=10 +akka.http { + server { + # The time period within which the TCP binding process must be completed. + # Set to `infinite` to disable. + bind-timeout = 5s + + # Show verbose error messages back to the client + verbose-error-messages = on + } + + parsing { + # Spray doesn't like the AWS4 headers + illegal-header-warnings = on + } +} + +# You can provide configuration overrides via local.conf if you don't want to replace everything in +# this configuration file +include "local.conf" diff --git a/modules/api/src/it/scala/vinyldns/api/backend/dns/DnsBackendIntegrationSpec.scala b/modules/api/src/it/scala/vinyldns/api/backend/dns/DnsBackendIntegrationSpec.scala index 22e5fa22a..0fd274b79 100644 --- a/modules/api/src/it/scala/vinyldns/api/backend/dns/DnsBackendIntegrationSpec.scala +++ b/modules/api/src/it/scala/vinyldns/api/backend/dns/DnsBackendIntegrationSpec.scala @@ -34,15 +34,16 @@ import vinyldns.core.domain.zone.{Algorithm, Zone, ZoneConnection} class DnsBackendIntegrationSpec extends AnyWordSpec with Matchers { private val testConnection = ZoneConnection( - "test", - "test", + "vinyldns.", + "vinyldns.", "nzisn+4G2ldMn0q1CV3vsg==", - "127.0.0.1:19001", + sys.env.getOrElse("DEFAULT_DNS_ADDRESS", "127.0.0.1:19001"), Algorithm.HMAC_MD5 ) "DNSBackend" should { "connect to a zone without a tsig key for transfer or update" in { + val config = DnsBackendConfig( "test", diff --git a/modules/api/src/it/scala/vinyldns/api/domain/zone/ZoneViewLoaderIntegrationSpec.scala b/modules/api/src/it/scala/vinyldns/api/domain/zone/ZoneViewLoaderIntegrationSpec.scala index c04795003..b757d5518 100644 --- a/modules/api/src/it/scala/vinyldns/api/domain/zone/ZoneViewLoaderIntegrationSpec.scala +++ b/modules/api/src/it/scala/vinyldns/api/domain/zone/ZoneViewLoaderIntegrationSpec.scala @@ -51,7 +51,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers { "vinyldns.", "vinyldns.", "nzisn+4G2ldMn0q1CV3vsg==", - "127.0.0.1:19001" + sys.env.getOrElse("DEFAULT_DNS_ADDRESS", "127.0.0.1:19001") ) ), transferConnection = @@ -84,7 +84,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers { "vinyldns.", "vinyldns.", "nzisn+4G2ldMn0q1CV3vsg==", - "127.0.0.1:19001" + sys.env.getOrElse("DEFAULT_DNS_ADDRESS", "127.0.0.1:19001") ) ), transferConnection = Some( @@ -92,7 +92,7 @@ class ZoneViewLoaderIntegrationSpec extends AnyWordSpec with Matchers { "vinyldns.", "vinyldns.", "nzisn+4G2ldMn0q1CV3vsg==", - "127.0.0.1:19001" + sys.env.getOrElse("DEFAULT_DNS_ADDRESS", "127.0.0.1:19001") ) ) ) diff --git a/modules/api/src/it/scala/vinyldns/api/notifier/sns/SnsNotifierIntegrationSpec.scala b/modules/api/src/it/scala/vinyldns/api/notifier/sns/SnsNotifierIntegrationSpec.scala index 5677647e8..86fe39540 100644 --- a/modules/api/src/it/scala/vinyldns/api/notifier/sns/SnsNotifierIntegrationSpec.scala +++ b/modules/api/src/it/scala/vinyldns/api/notifier/sns/SnsNotifierIntegrationSpec.scala @@ -16,7 +16,7 @@ package vinyldns.api.notifier.sns -import cats.effect.IO +import cats.effect.{IO, Timer} import com.amazonaws.auth.{AWSStaticCredentialsProvider, BasicAWSCredentials} import com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration import com.amazonaws.services.sns.AmazonSNSClientBuilder @@ -26,6 +26,7 @@ import org.joda.time.DateTime import org.json4s.DefaultFormats import org.json4s.jackson.JsonMethods._ import org.scalatest.matchers.should.Matchers +import org.scalatest.time.SpanSugar.convertIntToGrainOfTime import org.scalatest.wordspec.AnyWordSpecLike import vinyldns.api.MySqlApiIntegrationSpec import vinyldns.core.TestMembershipData._ @@ -34,6 +35,8 @@ import vinyldns.core.domain.record.{AData, RecordType} import vinyldns.core.notifier._ import vinyldns.mysql.MySqlIntegrationSpec +import scala.concurrent.ExecutionContext + class SnsNotifierIntegrationSpec extends MySqlApiIntegrationSpec with MySqlIntegrationSpec @@ -43,7 +46,7 @@ class SnsNotifierIntegrationSpec import vinyldns.api.domain.DomainValidations._ implicit val formats = DefaultFormats - + implicit val timer: Timer[IO] = IO.timer(ExecutionContext.global) val snsConfig: Config = ConfigFactory.load().getConfig("vinyldns.sns.settings") "Sns Notifier" should { @@ -91,7 +94,7 @@ class SnsNotifierIntegrationSpec val sqs = AmazonSQSClientBuilder .standard() .withEndpointConfiguration( - new EndpointConfiguration("http://127.0.0.1:19003", "us-east-1") + new EndpointConfiguration(sys.env.getOrElse("SNS_SERVICE_ENDPOINT","http://vinyldns-integration:19003"), "us-east-1") ) .withCredentials(credentialsProvider) .build() @@ -103,7 +106,7 @@ class SnsNotifierIntegrationSpec notifier <- new SnsNotifierProvider() .load(NotifierConfig("", snsConfig), userRepository) _ <- notifier.notify(Notification(batchChange)) - _ <- IO { Thread.sleep(100) } + _ <- IO.sleep(1.seconds) messages <- IO { sqs.receiveMessage(queueUrl).getMessages } _ <- IO { sns.deleteTopic(topic) diff --git a/modules/api/src/it/scala/vinyldns/api/route53/Route53ApiIntegrationSpec.scala b/modules/api/src/it/scala/vinyldns/api/route53/Route53ApiIntegrationSpec.scala index 11d2d166f..d5ca3daf3 100644 --- a/modules/api/src/it/scala/vinyldns/api/route53/Route53ApiIntegrationSpec.scala +++ b/modules/api/src/it/scala/vinyldns/api/route53/Route53ApiIntegrationSpec.scala @@ -39,7 +39,7 @@ import scala.collection.JavaConverters._ import scala.util.matching.Regex class Route53ApiIntegrationSpec - extends AnyWordSpec + extends AnyWordSpec with ScalaFutures with Matchers with MockitoSugar @@ -49,6 +49,7 @@ class Route53ApiIntegrationSpec with MySqlApiIntegrationSpec { private val testZone = Zone("example.com.", "test@test.com", backendId = Some("test")) + private def testConnection: Route53Backend = Route53Backend .load( @@ -56,7 +57,7 @@ class Route53ApiIntegrationSpec "test", Some("access"), Some("secret"), - "http://127.0.0.1:19003", + sys.env.getOrElse("R53_SERVICE_ENDPOINT", "http://localhost:19003"), "us-east-1" ) ) diff --git a/modules/api/src/main/resources/application.conf b/modules/api/src/main/resources/application.conf index 433095214..5857c4599 100644 --- a/modules/api/src/main/resources/application.conf +++ b/modules/api/src/main/resources/application.conf @@ -1,15 +1,317 @@ -################################################################################################################ -# This configuration is used primarily when running re-start or starting Vinyll locally. The configuration -# presumes a stand-alone Vinyll server with no backend services. -################################################################################################################ -akka { - loglevel = "ERROR" +vinyldns { + base-version = "0.0.0-local-dev" + version = ${vinyldns.base-version} # default to the base version if not overridden + version = ${?VINYLDNS_VERSION} # override the base version via env var - # The following settings are required to have Akka logging output to SLF4J and logback; without - # these, akka will output to STDOUT + # How often to any particular zone can be synchronized in milliseconds + sync-delay = 10000 + sync-delay = ${?SYNC_DELAY} + + # If we should start up polling for change requests, set this to false for the inactive cluster + processing-disabled = false + processing-disabled = ${?PROCESSING_DISABLED} + + # Number of records that can be in a zone + max-zone-size = 60000 + max-zone-size = ${?MAX_ZONE_SIZE} + + # Types of unowned records that users can access in shared zones + shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] + + # Batch change settings + batch-change-limit = 1000 + batch-change-limit = ${?BATCH_CHANGE_LIMIT} + manual-batch-review-enabled = true + manual-batch-review-enabled = ${?MANUAL_BATCH_REVIEW_ENABLED} + scheduled-changes-enabled = true + scheduled-changes-enabled = ${?SCHEDULED_CHANGES_ENABLED} + multi-record-batch-change-enabled = true + multi-record-batch-change-enabled = ${?MULTI_RECORD_BATCH_CHANGE_ENABLED} + + # configured backend providers + backend { + # Use "default" when dns backend legacy = true + # otherwise, use the id of one of the connections in any of your backends + default-backend-id = "default" + + # this is where we can save additional backends + backend-providers = [ + { + class-name = "vinyldns.api.backend.dns.DnsBackendProviderLoader" + settings = { + legacy = false + backends = [ + { + id = "default" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + }, + { + id = "func-test-backend" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + } + ] + } + } + ] + } + + + queue { + class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + + messages-per-poll = 10 + polling-interval = 250.millis + + settings { + # AWS access key and secret. + access-key = "test" + access-key = ${?AWS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?AWS_SECRET_ACCESS_KEY} + + # Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed. + signing-region = "us-east-1" + signing-region = ${?SQS_REGION} + + # Endpoint to access queue + service-endpoint = "http://vinyldns-integration:19003/" + service-endpoint = ${?SQS_SERVICE_ENDPOINT} + + # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change. + queue-name = "vinyldns" + queue-name = ${?SQS_QUEUE_NAME} + } + } + + email { + class-name = "vinyldns.api.notifier.email.EmailNotifierProvider" + class-name = ${?EMAIL_CLASS_NAME} + settings = { + from = "VinylDNS " + from = ${?EMAIL_FROM} + } + } + + sns { + class-name = "vinyldns.apadi.notifier.sns.SnsNotifierProvider" + class-name = ${?SNS_CLASS_NAME} + settings { + topic-arn = "arn:aws:sns:us-east-1:000000000000:batchChanges" + topic-arn = ${?SNS_TOPIC_ARN} + access-key = "test" + access-key = ${?SNS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?SNS_SECRET_KEY} + service-endpoint = "http://vinyldns-integration:19003" + service-endpoint = ${?SNS_SERVICE_ENDPOINT} + signing-region = "us-east-1" + signing-region = ${?SNS_REGION} + } + } + + rest { + host = "0.0.0.0" + port = 9000 + port=${?API_SERVICE_PORT} + } + + + approved-name-servers = [ + "172.17.42.1.", + "ns1.parent.com." + "ns1.parent.com1." + "ns1.parent.com2." + "ns1.parent.com3." + "ns1.parent.com4." + ] + + # Note: This MUST match the Portal or strange errors will ensue, NoOpCrypto should not be used for production + crypto { + type = "vinyldns.core.crypto.NoOpCrypto" + type = ${?CRYPTO_TYPE} + secret = ${?CRYPTO_SECRET} + } + + data-stores = ["mysql"] + mysql { + settings { + # JDBC Settings, these are all values in scalikejdbc-config, not our own + # these must be overridden to use MYSQL for production use + # assumes a docker or mysql instance running locally + name = "vinyldns" + name = ${?DATABASE_NAME} + driver = "org.h2.Driver" + driver = ${?JDBC_DRIVER} + migration-url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + migration-url = ${?JDBC_MIGRATION_URL} + url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + url = ${?JDBC_URL} + user = "sa" + user = ${?JDBC_USER} + password = "" + password = ${?JDBC_PASSWORD} + } + + # TODO: Remove the need for these useless configuration blocks + repositories { + zone { + } + batch-change { + } + user { + } + record-set { + } + zone-change { + } + record-change { + } + group { + } + group-change { + } + membership { + } + } + } + + backends = [] + + # FQDNs / IPs that cannot be modified via VinylDNS + # regex-list used for all record types except PTR + # ip-list used exclusively for PTR records + high-value-domains = { + regex-list = [ + "high-value-domain.*" # for testing + ] + ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.2.252", + "192.0.2.253", + "fd69:27cc:fe91:0:0:0:0:ffff", + "fd69:27cc:fe91:0:0:0:ffff:0" + ] + } + + # FQDNs / IPs / zone names that require manual review upon submission in batch change interface + # domain-list used for all record types except PTR + # ip-list used exclusively for PTR records + manual-review-domains = { + domain-list = [ + "needs-review.*" + ] + ip-list = [ + "192.0.1.254", + "192.0.1.255", + "192.0.2.254", + "192.0.2.255", + "192.0.3.254", + "192.0.3.255", + "192.0.4.254", + "192.0.4.255", + "fd69:27cc:fe91:0:0:0:ffff:1", + "fd69:27cc:fe91:0:0:0:ffff:2", + "fd69:27cc:fe92:0:0:0:ffff:1", + "fd69:27cc:fe92:0:0:0:ffff:2", + "fd69:27cc:fe93:0:0:0:ffff:1", + "fd69:27cc:fe93:0:0:0:ffff:2", + "fd69:27cc:fe94:0:0:0:ffff:1", + "fd69:27cc:fe94:0:0:0:ffff:2" + ] + zone-name-list = [ + "zone.requires.review." + "zone.requires.review1." + "zone.requires.review2." + "zone.requires.review3." + "zone.requires.review4." + ] + } + + # FQDNs / IPs that cannot be modified via VinylDNS + # regex-list used for all record types except PTR + # ip-list used exclusively for PTR records + high-value-domains = { + regex-list = [ + "high-value-domain.*" # for testing + ] + ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.1.252", + "192.0.1.253", + "192.0.2.252", + "192.0.2.253", + "192.0.3.252", + "192.0.3.253", + "192.0.4.252", + "192.0.4.253", + "fd69:27cc:fe91:0:0:0:0:ffff", + "fd69:27cc:fe91:0:0:0:ffff:0", + "fd69:27cc:fe92:0:0:0:0:ffff", + "fd69:27cc:fe92:0:0:0:ffff:0", + "fd69:27cc:fe93:0:0:0:0:ffff", + "fd69:27cc:fe93:0:0:0:ffff:0", + "fd69:27cc:fe94:0:0:0:0:ffff", + "fd69:27cc:fe94:0:0:0:ffff:0" + ] + } + + global-acl-rules = [ + { + group-ids: ["global-acl-group-id"], + fqdn-regex-list: [".*shared[0-9]{1}."] + }, + { + group-ids: ["another-global-acl-group"], + fqdn-regex-list: [".*ok[0-9]{1}."] + } + ] +} + +akka { + loglevel = "INFO" loggers = ["akka.event.slf4j.Slf4jLogger"] logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" logger-startup-timeout = 30s + + actor { + provider = "akka.actor.LocalActorRefProvider" + } } akka.http { @@ -28,143 +330,6 @@ akka.http { } } -vinyldns { - queue { - class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" - - messages-per-poll = 10 - polling-interval = 250.millis - max-retries = 100 # Max retries for message on queue; currently only applies to MySqlMessageQueue - - settings { - # AWS access key and secret. - access-key = "x" - secret-key = "x" - } - } - data-stores = ["mysql"] - - mysql { - settings { - # see https://github.com/brettwooldridge/HikariCP - connection-timeout-millis = 1000 - idle-timeout = 10000 - max-lifetime = 600000 - maximum-pool-size = 5 - minimum-idle = 1 - register-mbeans = true - } - - repositories { - zone { - # no additional settings for now - } - batch-change { - # no additional settings for now - } - user { - - } - record-set { - - } - group { - - } - membership { - - } - group-change { - - } - zone-change { - - } - record-change { - - } - } - } - - sync-delay = 10000 # 10 second delay for resyncing zone - - batch-change-limit = 1000 # Max change limit per batch request - - # this key is used in order to encrypt/decrypt DNS TSIG keys. We use this dummy one for test purposes, this - # should be overridden with a real value that is hidden for production deployment - crypto { - type = "vinyldns.core.crypto.JavaCrypto" - secret = "8B06A7F3BC8A2497736F1916A123AA40E88217BE9264D8872597EF7A6E5DCE61" - } - - # FQDNs / IPs that cannot be modified via VinylDNS - # regex-list used for all record types except PTR - # ip-list used exclusively for PTR records - high-value-domains = { - regex-list = [ - "high-value-domain.*" # for testing - ] - ip-list = [ - "192.0.2.252", - "192.0.2.253", - "fd69:27cc:fe91:0:0:0:0:ffff", - "fd69:27cc:fe91:0:0:0:ffff:0" - ] - } - - # FQDNs / IPs / zone names that require manual review upon submission in batch change interface - # domain-list used for all record types except PTR - # ip-list used exclusively for PTR records - manual-review-domains = { - domain-list = [ - "needs-review.*" - ] - ip-list = [ - "192.0.2.254", - "192.0.2.255", - "fd69:27cc:fe91:0:0:0:ffff:1", - "fd69:27cc:fe91:0:0:0:ffff:2" - ] - zone-name-list = [ - "zone.requires.review." - ] - } - - # types of unowned records that users can access in shared zones - shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] - - backends = [ - { - id = "func-test-backend" - zone-connection { - name = "vinyldns." - key-name = "vinyldns." - key = "nzisn+4G2ldMn0q1CV3vsg==" - primary-server = "127.0.0.1:19001" - } - transfer-connection { - name = "vinyldns." - key-name = "vinyldns." - key = "nzisn+4G2ldMn0q1CV3vsg==" - primary-server = "127.0.0.1:19001" - } - } - ] - - multi-record-batch-change-enabled = true - - # feature flag for manual batch review - manual-batch-review-enabled = true - scheduled-changes-enabled = true - - global-acl-rules = [ - { - group-ids: ["global-acl-group-id"], - fqdn-regex-list: [".*shared."] - }, - { - group-ids: ["another-global-acl-group"], - fqdn-regex-list: [".*ok."] - } - ] -} +# You can provide configuration overrides via local.conf if you don't want to replace everything in +# this configuration file +include "local.conf" diff --git a/modules/api/src/main/resources/reference.conf b/modules/api/src/main/resources/reference.conf index e7ced14b2..fdbb0fb00 100644 --- a/modules/api/src/main/resources/reference.conf +++ b/modules/api/src/main/resources/reference.conf @@ -148,16 +148,24 @@ vinyldns { defaultZoneConnection { name = "vinyldns." + name= ${?DEFAULT_DNS_KEY_NAME} keyName = "vinyldns." + keyName= ${?DEFAULT_DNS_KEY_NAME} key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} primaryServer = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} } defaultTransferConnection { name = "vinyldns." + name= ${?DEFAULT_DNS_KEY_NAME} keyName = "vinyldns." + keyName= ${?DEFAULT_DNS_KEY_NAME} key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} primaryServer = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} } batch-change-limit = 1000 diff --git a/modules/api/src/main/resources/vinyldns-ascii.txt b/modules/api/src/main/resources/vinyldns-ascii.txt index bb10a843a..b84f40a76 100644 --- a/modules/api/src/main/resources/vinyldns-ascii.txt +++ b/modules/api/src/main/resources/vinyldns-ascii.txt @@ -1,6 +1,6 @@ - .__ .__ .___ -___ _|__| ____ ___.__.| | __| _/____ ______ -\ \/ / |/ < | || | / __ |/ \ / ___/ - \ /| | | \___ || |__/ /_/ | | \\___ \ - \_/ |__|___| / ____||____/\____ |___| /____ > - \/\/ \/ \/ \/ + _ ___ ______ _ _______ +| | / (_)___ __ __/ / __ \/ | / / ___/ +| | / / / __ \/ / / / / / / / |/ /\__ \ +| |/ / / / / / /_/ / / /_/ / /| /___/ / +|___/_/_/ /_/\__, /_/_____/_/ |_//____/ + /____/ diff --git a/modules/api/src/test/resources/application.conf b/modules/api/src/test/resources/application.conf index b1bd68c8d..18faa24ec 100644 --- a/modules/api/src/test/resources/application.conf +++ b/modules/api/src/test/resources/application.conf @@ -1,30 +1,227 @@ -akka { - loglevel = "OFF" - loggers = ["akka.testkit.TestEventListener"] - log-dead-letters-during-shutdown = off - log-dead-letters = 0 - test.timefactor = 5 - logger-startup-timeout = 30s -} - vinyldns { - active-node-count = 3 - sync-delay = 10000 # 10 second delay for resyncing zone - color = "blue" + base-version = "0.0.0-local-dev" + version = ${vinyldns.base-version} # default to the base version if not overridden + version = ${?VINYLDNS_VERSION} # override the base version via env var + + # How often to any particular zone can be synchronized in milliseconds + sync-delay = 10000 + sync-delay = ${?SYNC_DELAY} + + # If we should start up polling for change requests, set this to false for the inactive cluster + processing-disabled = false + processing-disabled = ${?PROCESSING_DISABLED} + + # Number of records that can be in a zone + max-zone-size = 60000 + max-zone-size = ${?MAX_ZONE_SIZE} + + # Types of unowned records that users can access in shared zones + shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] + + # Batch change settings + batch-change-limit = 1000 + batch-change-limit = ${?BATCH_CHANGE_LIMIT} + manual-batch-review-enabled = true + manual-batch-review-enabled = ${?MANUAL_BATCH_REVIEW_ENABLED} + scheduled-changes-enabled = true + scheduled-changes-enabled = ${?SCHEDULED_CHANGES_ENABLED} + multi-record-batch-change-enabled = true + multi-record-batch-change-enabled = ${?MULTI_RECORD_BATCH_CHANGE_ENABLED} + + # configured backend providers + backend { + # Use "default" when dns backend legacy = true + # otherwise, use the id of one of the connections in any of your backends + default-backend-id = "default" + + # this is where we can save additional backends + backend-providers = [ + { + class-name = "vinyldns.api.backend.dns.DnsBackendProviderLoader" + settings = { + legacy = false + backends = [ + { + id = "default" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + }, + { + id = "func-test-backend" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + } + ] + } + } + ] + } + + + queue { + class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + + messages-per-poll = 10 + polling-interval = 250.millis + + settings { + # AWS access key and secret. + access-key = "test" + access-key = ${?AWS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?AWS_SECRET_ACCESS_KEY} + + # Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed. + signing-region = "us-east-1" + signing-region = ${?SQS_REGION} + + # Endpoint to access queue + service-endpoint = "http://vinyldns-integration:19003/" + service-endpoint = ${?SQS_SERVICE_ENDPOINT} + + # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change. + queue-name = "vinyldns" + queue-name = ${?SQS_QUEUE_NAME} + } + } + + email { + class-name = "vinyldns.api.notifier.email.EmailNotifierProvider" + class-name = ${?EMAIL_CLASS_NAME} + settings = { + from = "VinylDNS " + from = ${?EMAIL_FROM} + } + } + + sns { + class-name = "vinyldns.apadi.notifier.sns.SnsNotifierProvider" + class-name = ${?SNS_CLASS_NAME} + settings { + topic-arn = "arn:aws:sns:us-east-1:000000000000:batchChanges" + topic-arn = ${?SNS_TOPIC_ARN} + access-key = "test" + access-key = ${?SNS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?SNS_SECRET_KEY} + service-endpoint = "http://vinyldns-integration:19003" + service-endpoint = ${?SNS_SERVICE_ENDPOINT} + signing-region = "us-east-1" + signing-region = ${?SNS_REGION} + } + } + + notifiers = ["test-notifier"] + + test-notifier { + class-name = "someclass" + settings { + value = "test" + } + } + rest { - host = "127.0.0.1" + host = "0.0.0.0" port = 9000 + port=${?API_SERVICE_PORT} } - # this key is used in order to encrypt/decrypt DNS TSIG keys. We use this dummy one for test purposes, this - # should be overridden with a real value that is hidden for production deployment - crypto { - type = "vinyldns.core.crypto.JavaCrypto" - secret = "8B06A7F3BC8A2497736F1916A123AA40E88217BE9264D8872597EF7A6E5DCE61" - } + + approved-name-servers = [ - "some.test.ns." + "172.17.42.1.", + "ns1.parent.com." + "ns1.parent.com1." + "ns1.parent.com2." + "ns1.parent.com3." + "ns1.parent.com4." ] + # Note: This MUST match the Portal or strange errors will ensue, NoOpCrypto should not be used for production + crypto { + type = "vinyldns.core.crypto.NoOpCrypto" + type = ${?CRYPTO_TYPE} + secret = ${?CRYPTO_SECRET} + } + + data-stores = ["mysql"] + mysql { + settings { + # JDBC Settings, these are all values in scalikejdbc-config, not our own + # these must be overridden to use MYSQL for production use + # assumes a docker or mysql instance running locally + name = "vinyldns" + name = ${?DATABASE_NAME} + driver = "org.h2.Driver" + driver = ${?JDBC_DRIVER} + migration-url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + migration-url = ${?JDBC_MIGRATION_URL} + url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + url = ${?JDBC_URL} + user = "sa" + user = ${?JDBC_USER} + password = "" + password = ${?JDBC_PASSWORD} + } + + # TODO: Remove the need for these useless configuration blocks + repositories { + zone { + } + batch-change { + } + user { + } + record-set { + } + zone-change { + } + record-change { + } + group { + } + group-change { + } + membership { + } + } + } + + backends = [] + # FQDNs / IPs that cannot be modified via VinylDNS # regex-list used for all record types except PTR # ip-list used exclusively for PTR records @@ -33,6 +230,7 @@ vinyldns { "high-value-domain.*" # for testing ] ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing "192.0.2.252", "192.0.2.253", "fd69:27cc:fe91:0:0:0:0:ffff", @@ -48,80 +246,99 @@ vinyldns { "needs-review.*" ] ip-list = [ + "192.0.1.254", + "192.0.1.255", "192.0.2.254", "192.0.2.255", + "192.0.3.254", + "192.0.3.255", + "192.0.4.254", + "192.0.4.255", "fd69:27cc:fe91:0:0:0:ffff:1", - "fd69:27cc:fe91:0:0:0:ffff:2" + "fd69:27cc:fe91:0:0:0:ffff:2", + "fd69:27cc:fe92:0:0:0:ffff:1", + "fd69:27cc:fe92:0:0:0:ffff:2", + "fd69:27cc:fe93:0:0:0:ffff:1", + "fd69:27cc:fe93:0:0:0:ffff:2", + "fd69:27cc:fe94:0:0:0:ffff:1", + "fd69:27cc:fe94:0:0:0:ffff:2" ] zone-name-list = [ - "zone.needs.review." + "zone.requires.review." + "zone.requires.review1." + "zone.requires.review2." + "zone.requires.review3." + "zone.requires.review4." ] } - # types of unowned records that users can access in shared zones - shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] - - # used for testing only - string-list-test = ["test"] - - mysql.repositories { - zone { - # no additional settings for now - } - batch-change { - # no additional settings for now - } - user { - - } - record-set { - - } - group { - - } - membership { - - } - group-change { - - } - zone-change { - - } - record-change { - - } + # FQDNs / IPs that cannot be modified via VinylDNS + # regex-list used for all record types except PTR + # ip-list used exclusively for PTR records + high-value-domains = { + regex-list = [ + "high-value-domain.*" # for testing + ] + ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.1.252", + "192.0.1.253", + "192.0.2.252", + "192.0.2.253", + "192.0.3.252", + "192.0.3.253", + "192.0.4.252", + "192.0.4.253", + "fd69:27cc:fe91:0:0:0:0:ffff", + "fd69:27cc:fe91:0:0:0:ffff:0", + "fd69:27cc:fe92:0:0:0:0:ffff", + "fd69:27cc:fe92:0:0:0:ffff:0", + "fd69:27cc:fe93:0:0:0:0:ffff", + "fd69:27cc:fe93:0:0:0:ffff:0", + "fd69:27cc:fe94:0:0:0:0:ffff", + "fd69:27cc:fe94:0:0:0:ffff:0" + ] } - notifiers = ["test-notifier"] - - test-notifier { - class-name = "someclass" - settings { - value = "test" - } - } - - backends = [ + global-acl-rules = [ { - id = "test" - zone-connection { - name = "zoneconn." - key-name = "vinyldns." - key = "test-key" - primary-server = "127.0.0.1:19001" - } - transfer-connection { - name = "transferconn." - key-name = "vinyldns." - key = "test-key" - primary-server = "127.0.0.1:19001" - } + group-ids: ["global-acl-group-id"], + fqdn-regex-list: [".*shared[0-9]{1}."] + }, + { + group-ids: ["another-global-acl-group"], + fqdn-regex-list: [".*ok[0-9]{1}."] } ] - - manual-batch-review-enabled = true - - scheduled-changes-enabled = true } + +akka { + loglevel = "INFO" + loggers = ["akka.event.slf4j.Slf4jLogger"] + logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" + logger-startup-timeout = 30s + + actor { + provider = "akka.actor.LocalActorRefProvider" + } +} + +akka.http { + server { + # The time period within which the TCP binding process must be completed. + # Set to `infinite` to disable. + bind-timeout = 5s + + # Show verbose error messages back to the client + verbose-error-messages = on + } + + parsing { + # Spray doesn't like the AWS4 headers + illegal-header-warnings = on + } +} + +# You can provide configuration overrides via local.conf if you don't want to replace everything in +# this configuration file +include "local.conf" diff --git a/modules/api/src/test/scala/vinyldns/api/config/VinylDNSConfigSpec.scala b/modules/api/src/test/scala/vinyldns/api/config/VinylDNSConfigSpec.scala index b278b637d..f61b43701 100644 --- a/modules/api/src/test/scala/vinyldns/api/config/VinylDNSConfigSpec.scala +++ b/modules/api/src/test/scala/vinyldns/api/config/VinylDNSConfigSpec.scala @@ -16,15 +16,18 @@ package vinyldns.api.config +import cats.effect.{ContextShift, IO} import org.scalatest.BeforeAndAfterAll import org.scalatest.matchers.should.Matchers import org.scalatest.wordspec.AnyWordSpec +import vinyldns.api.backend.dns.DnsBackendProviderConfig import vinyldns.core.domain.zone.ZoneConnection import vinyldns.core.repository.RepositoryName._ class VinylDNSConfigSpec extends AnyWordSpec with Matchers with BeforeAndAfterAll { private val underTest: VinylDNSConfig = VinylDNSConfig.load().unsafeRunSync() + private implicit val cs: ContextShift[IO] = IO.contextShift(scala.concurrent.ExecutionContext.global) "VinylDNSConfig" should { "load the rest config" in { @@ -63,27 +66,19 @@ class VinylDNSConfigSpec extends AnyWordSpec with Matchers with BeforeAndAfterAl notifierConfigs.head.settings.getString("value").shouldBe("test") } - "load default keys" in { - val defaultConn = - ZoneConnection("vinyldns.", "vinyldns.", "nzisn+4G2ldMn0q1CV3vsg==", "127.0.0.1:19001") - - underTest.configuredDnsConnections.defaultZoneConnection - .decrypted(underTest.crypto) shouldBe - defaultConn - underTest.configuredDnsConnections.defaultTransferConnection - .decrypted(underTest.crypto) shouldBe - defaultConn - } "load specified backends" in { - val zc = ZoneConnection("zoneconn.", "vinyldns.", "test-key", "127.0.0.1:19001") - val tc = zc.copy(name = "transferconn.") + val zc = ZoneConnection("vinyldns.", "vinyldns.", "nzisn+4G2ldMn0q1CV3vsg==", sys.env.getOrElse("DEFAULT_DNS_ADDRESS", "127.0.0.1:19001")) + val tc = zc.copy() - val backends = underTest.configuredDnsConnections.dnsBackends + val backends = underTest.backendConfigs.backendProviders backends.length shouldBe 1 - backends.head.id shouldBe "test" - backends.head.zoneConnection.decrypted(underTest.crypto) shouldBe zc - backends.head.transferConnection.decrypted(underTest.crypto) shouldBe tc + val config = DnsBackendProviderConfig.load(backends.head.settings).unsafeRunSync() + config.backends.length shouldBe 2 + + config.backends.head.id shouldBe "default" + config.backends.head.zoneConnection.decrypted(underTest.crypto) shouldBe zc + config.backends.head.transferConnection.get.decrypted(underTest.crypto) shouldBe tc } } } diff --git a/modules/api/src/universal/conf/application.conf b/modules/api/src/universal/conf/application.conf index 6b90dbe92..5857c4599 100644 --- a/modules/api/src/universal/conf/application.conf +++ b/modules/api/src/universal/conf/application.conf @@ -1,124 +1,317 @@ -# The default application.conf is not intended to be used in production. It assumes a docker-compose -# setup for all of the services. Provide your own application.conf on the docker mount with your -# own settings vinyldns { - - # Should be provided in the start up script via an environment - base-version = "unset" + base-version = "0.0.0-local-dev" version = ${vinyldns.base-version} # default to the base version if not overridden version = ${?VINYLDNS_VERSION} # override the base version via env var - queue.settings.service-endpoint = "http://vinyldns-elasticmq:9324/" + # How often to any particular zone can be synchronized in milliseconds + sync-delay = 10000 + sync-delay = ${?SYNC_DELAY} + + # If we should start up polling for change requests, set this to false for the inactive cluster + processing-disabled = false + processing-disabled = ${?PROCESSING_DISABLED} + + # Number of records that can be in a zone + max-zone-size = 60000 + max-zone-size = ${?MAX_ZONE_SIZE} + + # Types of unowned records that users can access in shared zones + shared-approved-types = ["A", "AAAA", "CNAME", "PTR", "TXT"] + + # Batch change settings + batch-change-limit = 1000 + batch-change-limit = ${?BATCH_CHANGE_LIMIT} + manual-batch-review-enabled = true + manual-batch-review-enabled = ${?MANUAL_BATCH_REVIEW_ENABLED} + scheduled-changes-enabled = true + scheduled-changes-enabled = ${?SCHEDULED_CHANGES_ENABLED} + multi-record-batch-change-enabled = true + multi-record-batch-change-enabled = ${?MULTI_RECORD_BATCH_CHANGE_ENABLED} + + # configured backend providers + backend { + # Use "default" when dns backend legacy = true + # otherwise, use the id of one of the connections in any of your backends + default-backend-id = "default" + + # this is where we can save additional backends + backend-providers = [ + { + class-name = "vinyldns.api.backend.dns.DnsBackendProviderLoader" + settings = { + legacy = false + backends = [ + { + id = "default" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + }, + { + id = "func-test-backend" + zone-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + } + transfer-connection = { + name = "vinyldns." + key-name = "vinyldns." + key-name = ${?DEFAULT_DNS_KEY_NAME} + key = "nzisn+4G2ldMn0q1CV3vsg==" + key = ${?DEFAULT_DNS_KEY_SECRET} + primary-server = "127.0.0.1:19001" + primary-server = ${?DEFAULT_DNS_ADDRESS} + }, + tsig-usage = "always" + } + ] + } + } + ] + } + + + queue { + class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + + messages-per-poll = 10 + polling-interval = 250.millis + + settings { + # AWS access key and secret. + access-key = "test" + access-key = ${?AWS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?AWS_SECRET_ACCESS_KEY} + + # Regional endpoint to make your requests (eg. 'us-west-2', 'us-east-1', etc.). This is the region where your queue is housed. + signing-region = "us-east-1" + signing-region = ${?SQS_REGION} + + # Endpoint to access queue + service-endpoint = "http://vinyldns-integration:19003/" + service-endpoint = ${?SQS_SERVICE_ENDPOINT} + + # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change. + queue-name = "vinyldns" + queue-name = ${?SQS_QUEUE_NAME} + } + } + + email { + class-name = "vinyldns.api.notifier.email.EmailNotifierProvider" + class-name = ${?EMAIL_CLASS_NAME} + settings = { + from = "VinylDNS " + from = ${?EMAIL_FROM} + } + } + + sns { + class-name = "vinyldns.apadi.notifier.sns.SnsNotifierProvider" + class-name = ${?SNS_CLASS_NAME} + settings { + topic-arn = "arn:aws:sns:us-east-1:000000000000:batchChanges" + topic-arn = ${?SNS_TOPIC_ARN} + access-key = "test" + access-key = ${?SNS_ACCESS_KEY} + secret-key = "test" + secret-key = ${?SNS_SECRET_KEY} + service-endpoint = "http://vinyldns-integration:19003" + service-endpoint = ${?SNS_SERVICE_ENDPOINT} + signing-region = "us-east-1" + signing-region = ${?SNS_REGION} + } + } - # host and port the server binds to. This should not be changed rest { host = "0.0.0.0" port = 9000 + port=${?API_SERVICE_PORT} } - # the delay between zone syncs so we are not syncing too often - sync-delay = 10000 - # crypto settings for symmetric cryptography of secrets in the system - # Note: for production systems secrets should not live in plain text in a file + approved-name-servers = [ + "172.17.42.1.", + "ns1.parent.com." + "ns1.parent.com1." + "ns1.parent.com2." + "ns1.parent.com3." + "ns1.parent.com4." + ] + + # Note: This MUST match the Portal or strange errors will ensue, NoOpCrypto should not be used for production crypto { type = "vinyldns.core.crypto.NoOpCrypto" + type = ${?CRYPTO_TYPE} + secret = ${?CRYPTO_SECRET} } data-stores = ["mysql"] - - # default settings point to the setup from docker compose mysql { settings { + # JDBC Settings, these are all values in scalikejdbc-config, not our own + # these must be overridden to use MYSQL for production use + # assumes a docker or mysql instance running locally name = "vinyldns" - driver = "org.mariadb.jdbc.Driver" - migration-url = "jdbc:mariadb://vinyldns-mysql:3306/?user=root&password=pass" - url = "jdbc:mariadb://vinyldns-mysql:3306/vinyldns?user=root&password=pass" - user = "root" - password = "pass" - - # see https://github.com/brettwooldridge/HikariCP - connection-timeout-millis = 1000 - max-lifetime = 600000 - maximum-pool-size = 20 - register-mbeans = true + name = ${?DATABASE_NAME} + driver = "org.h2.Driver" + driver = ${?JDBC_DRIVER} + migration-url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + migration-url = ${?JDBC_MIGRATION_URL} + url = "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;IGNORECASE=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" + url = ${?JDBC_URL} + user = "sa" + user = ${?JDBC_USER} + password = "" + password = ${?JDBC_PASSWORD} } + + # TODO: Remove the need for these useless configuration blocks repositories { zone { - # no additional settings for now } batch-change { - # no additional settings for now } user { - # no additional settings for now } record-set { - # no additional settings for now - } - group { - # no additional settings for now - } - membership { - # no additional settings for now - } - group-change { - # no additional settings for now } zone-change { - # no additional settings for now } record-change { - # no additional settings for now + } + group { + } + group-change { + } + membership { } } } - # the DDNS connection information for the default dns backend - defaultZoneConnection { - name = "vinyldns." - keyName = "vinyldns." - key = "nzisn+4G2ldMn0q1CV3vsg==" - primaryServer = "localhost:19001" + backends = [] + + # FQDNs / IPs that cannot be modified via VinylDNS + # regex-list used for all record types except PTR + # ip-list used exclusively for PTR records + high-value-domains = { + regex-list = [ + "high-value-domain.*" # for testing + ] + ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.2.252", + "192.0.2.253", + "fd69:27cc:fe91:0:0:0:0:ffff", + "fd69:27cc:fe91:0:0:0:ffff:0" + ] } - # the AXFR connection information for the default dns backend - defaultTransferConnection { - name = "vinyldns." - keyName = "vinyldns." - key = "nzisn+4G2ldMn0q1CV3vsg==" - primaryServer = "localhost:19001" + # FQDNs / IPs / zone names that require manual review upon submission in batch change interface + # domain-list used for all record types except PTR + # ip-list used exclusively for PTR records + manual-review-domains = { + domain-list = [ + "needs-review.*" + ] + ip-list = [ + "192.0.1.254", + "192.0.1.255", + "192.0.2.254", + "192.0.2.255", + "192.0.3.254", + "192.0.3.255", + "192.0.4.254", + "192.0.4.255", + "fd69:27cc:fe91:0:0:0:ffff:1", + "fd69:27cc:fe91:0:0:0:ffff:2", + "fd69:27cc:fe92:0:0:0:ffff:1", + "fd69:27cc:fe92:0:0:0:ffff:2", + "fd69:27cc:fe93:0:0:0:ffff:1", + "fd69:27cc:fe93:0:0:0:ffff:2", + "fd69:27cc:fe94:0:0:0:ffff:1", + "fd69:27cc:fe94:0:0:0:ffff:2" + ] + zone-name-list = [ + "zone.requires.review." + "zone.requires.review1." + "zone.requires.review2." + "zone.requires.review3." + "zone.requires.review4." + ] } - backends = [ + # FQDNs / IPs that cannot be modified via VinylDNS + # regex-list used for all record types except PTR + # ip-list used exclusively for PTR records + high-value-domains = { + regex-list = [ + "high-value-domain.*" # for testing + ] + ip-list = [ + # using reverse zones in the vinyldns/bind9 docker image for testing + "192.0.1.252", + "192.0.1.253", + "192.0.2.252", + "192.0.2.253", + "192.0.3.252", + "192.0.3.253", + "192.0.4.252", + "192.0.4.253", + "fd69:27cc:fe91:0:0:0:0:ffff", + "fd69:27cc:fe91:0:0:0:ffff:0", + "fd69:27cc:fe92:0:0:0:0:ffff", + "fd69:27cc:fe92:0:0:0:ffff:0", + "fd69:27cc:fe93:0:0:0:0:ffff", + "fd69:27cc:fe93:0:0:0:ffff:0", + "fd69:27cc:fe94:0:0:0:0:ffff", + "fd69:27cc:fe94:0:0:0:ffff:0" + ] + } + + global-acl-rules = [ { - id = "func-test-backend" - zone-connection { - name = "vinyldns." - key-name = "vinyldns." - key = "nzisn+4G2ldMn0q1CV3vsg==" - primary-server = "localhost:19001" - } - transfer-connection { - name = "vinyldns." - key-name = "vinyldns." - key = "nzisn+4G2ldMn0q1CV3vsg==" - primary-server = "localhost:19001" - } + group-ids: ["global-acl-group-id"], + fqdn-regex-list: [".*shared[0-9]{1}."] + }, + { + group-ids: ["another-global-acl-group"], + fqdn-regex-list: [".*ok[0-9]{1}."] } ] - - # the max number of changes in a single batch change. Change carefully as this has performance - # implications - batch-change-limit = 1000 } -# Akka settings, these should not need to be modified unless you know akka http really well. akka { loglevel = "INFO" loggers = ["akka.event.slf4j.Slf4jLogger"] logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" logger-startup-timeout = 30s + + actor { + provider = "akka.actor.LocalActorRefProvider" + } } akka.http { @@ -132,7 +325,11 @@ akka.http { } parsing { - # akka-http doesn't like the AWS4 headers + # Spray doesn't like the AWS4 headers illegal-header-warnings = on } } + +# You can provide configuration overrides via local.conf if you don't want to replace everything in +# this configuration file +include "local.conf" diff --git a/modules/docs/src/main/mdoc/operator/config-api.md b/modules/docs/src/main/mdoc/operator/config-api.md index d07a6821d..6e6c25349 100644 --- a/modules/docs/src/main/mdoc/operator/config-api.md +++ b/modules/docs/src/main/mdoc/operator/config-api.md @@ -52,7 +52,7 @@ snippet... access-key = ${AWS_ACCESS_KEY} secret-key = ${AWS_SECRET_ACCESS_KEY} signing-region = ${SQS_REGION} - service-endpoint = ${SQS_ENDPOINT} + service-endpoint = ${SQS_SERVICE_ENDPOINT} queue-name = ${SQS_QUEUE_NAME} } ``` diff --git a/modules/docs/src/main/mdoc/operator/config-portal.md b/modules/docs/src/main/mdoc/operator/config-portal.md index c61dc99d9..0d31f9a1e 100644 --- a/modules/docs/src/main/mdoc/operator/config-portal.md +++ b/modules/docs/src/main/mdoc/operator/config-portal.md @@ -251,7 +251,7 @@ portal.vinyldns.backend.url = "http://vinyldns-api:9000" portal.test_login = false # configuration for the users and groups store -data-stores = ["dynamodb", "mysql"] +data-stores = ["mysql"] mysql { class-name = "vinyldns.mysql.repository.MySqlDataStoreProvider" @@ -284,37 +284,6 @@ mysql { } } -dynamodb { - class-name = "vinyldns.dynamodb.repository.DynamoDBDataStoreProvider" - - settings { - key = "x" - secret = "x" - endpoint = "http://vinyldns-dynamodb:8000" - region = "us-east-1" - } - - repositories { - user-change { - table-name = "userChangeTest" - provisioned-reads = 30 - provisioned-writes = 20 - } - } - -LDAP { - user="test" - password="test" - domain="test" - - searchBase = [{organization = "someDomain", domainName = "DC=test,DC=test,DC=com"}, {organization = "anotherDomain", domainName = "DC=test,DC=com"}] - - context { - initialContextFactory = "com.sun.jndi.ldap.LdapCtxFactory" - securityAuthentication = "simple" - providerUrl = "ldaps://somedomain.com:9999" - } - } play.filters.enabled += "play.filters.csrf.CSRFFilter" diff --git a/modules/mysql/src/it/resources/application.conf b/modules/mysql/src/it/resources/application.conf index d134a7265..d17e78063 100644 --- a/modules/mysql/src/it/resources/application.conf +++ b/modules/mysql/src/it/resources/application.conf @@ -1,5 +1,7 @@ mysql { class-name = "vinyldns.mysql.repository.MySqlDataStoreProvider" + endpoint = "localhost:19002" + endpoint = ${?MYSQL_ENDPOINT} settings { # JDBC Settings, these are all values in scalikejdbc-config, not our own @@ -7,8 +9,8 @@ mysql { # assumes a docker or mysql instance running locally name = "vinyldns2" driver = "org.mariadb.jdbc.Driver" - migration-url = "jdbc:mariadb://localhost:19002/" - url = "jdbc:mariadb://localhost:19002/vinyldns2" + migration-url = "jdbc:mariadb://"${mysql.endpoint}"/" + url = "jdbc:mariadb://"${mysql.endpoint}"/vinyldns2" user = "root" password = "pass" @@ -43,8 +45,8 @@ queue { settings = { name = "vinyldns2" driver = "org.mariadb.jdbc.Driver" - migration-url = "jdbc:mariadb://localhost:19002/?user=root&password=pass" - url = "jdbc:mariadb://localhost:19002/vinyldns2?user=root&password=pass" + migration-url = "jdbc:mariadb://"${mysql.endpoint}"/?user=root&password=pass" + url = "jdbc:mariadb://"${mysql.endpoint}"/vinyldns2?user=root&password=pass" user = "root" password = "pass" diff --git a/modules/mysql/src/main/resources/test/ddl.sql b/modules/mysql/src/main/resources/test/ddl.sql index c54ad1a54..851ac6672 100644 --- a/modules/mysql/src/main/resources/test/ddl.sql +++ b/modules/mysql/src/main/resources/test/ddl.sql @@ -6,7 +6,7 @@ -- Ex: "jdbc:h2:mem:vinyldns;MODE=MYSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;INIT=RUNSCRIPT FROM 'classpath:test/ddl.sql'" -- -CREATE TABLE batch_change +CREATE TABLE IF NOT EXISTS batch_change ( id char(36) not null primary key, user_id char(36) not null, @@ -22,16 +22,16 @@ CREATE TABLE batch_change cancelled_timestamp datetime null ); -create index batch_change_approval_status_index +CREATE INDEX IF NOT EXISTS batch_change_approval_status_index on batch_change (approval_status); -create index batch_change_user_id_created_time_index +CREATE INDEX IF NOT EXISTS batch_change_user_id_created_time_index on batch_change (user_id, created_time); -create index batch_change_user_id_index +CREATE INDEX IF NOT EXISTS batch_change_user_id_index on batch_change (user_id); -create table group_change +CREATE TABLE IF NOT EXISTS group_change ( id char(36) not null primary key, group_id char(36) not null, @@ -39,10 +39,10 @@ create table group_change data blob not null ); -create index group_change_group_id_index +CREATE INDEX IF NOT EXISTS group_change_group_id_index on group_change (group_id); -create table `groups` +CREATE TABLE IF NOT EXISTS `groups` ( id char(36) not null primary key, name varchar(256) not null, @@ -52,10 +52,10 @@ create table `groups` email varchar(256) not null ); -create index groups_name_index +CREATE INDEX IF NOT EXISTS groups_name_index on `groups` (name); -create table membership +CREATE TABLE IF NOT EXISTS membership ( user_id char(36) not null, group_id char(36) not null, @@ -63,7 +63,7 @@ create table membership primary key (user_id, group_id) ); -create table message_queue +CREATE TABLE IF NOT EXISTS message_queue ( id char(36) not null primary key, message_type tinyint null, @@ -75,16 +75,16 @@ create table message_queue attempts int default 0 not null ); -create index message_queue_inflight_index +CREATE INDEX IF NOT EXISTS message_queue_inflight_index on message_queue (in_flight); -create index message_queue_timeout_index +CREATE INDEX IF NOT EXISTS message_queue_timeout_index on message_queue (timeout_seconds); -create index message_queue_updated_index +CREATE INDEX IF NOT EXISTS message_queue_updated_index on message_queue (updated); -create table record_change +CREATE TABLE IF NOT EXISTS record_change ( id char(36) not null primary key, zone_id char(36) not null, @@ -93,13 +93,13 @@ create table record_change data blob not null ); -create index record_change_created_index +CREATE INDEX IF NOT EXISTS record_change_created_index on record_change (created); -create index record_change_zone_id_index +CREATE INDEX IF NOT EXISTS record_change_zone_id_index on record_change (zone_id); -create table recordset +CREATE TABLE IF NOT EXISTS recordset ( id char(36) not null primary key, zone_id char(36) not null, @@ -112,16 +112,16 @@ create table recordset unique (zone_id, name, type) ); -create index recordset_fqdn_index +CREATE INDEX IF NOT EXISTS recordset_fqdn_index on recordset (fqdn); -create index recordset_owner_group_id_index +CREATE INDEX IF NOT EXISTS recordset_owner_group_id_index on recordset (owner_group_id); -create index recordset_type_index +CREATE INDEX IF NOT EXISTS recordset_type_index on recordset (type); -create table single_change +CREATE TABLE IF NOT EXISTS single_change ( id char(36) not null primary key, seq_num smallint not null, @@ -138,13 +138,13 @@ create table single_change on delete cascade ); -create index single_change_batch_change_id_index +CREATE INDEX IF NOT EXISTS single_change_batch_change_id_index on single_change (batch_change_id); -create index single_change_record_set_change_id_index +CREATE INDEX IF NOT EXISTS single_change_record_set_change_id_index on single_change (record_set_change_id); -create table stats +CREATE TABLE IF NOT EXISTS stats ( id bigint auto_increment primary key, name varchar(255) not null, @@ -152,13 +152,13 @@ create table stats created datetime not null ); -create index stats_name_created_index +CREATE INDEX IF NOT EXISTS stats_name_created_index on stats (name, created); -create index stats_name_index +CREATE INDEX IF NOT EXISTS stats_name_index on stats (name); -create table task +CREATE TABLE IF NOT EXISTS task ( name varchar(255) not null primary key, in_flight bit not null, @@ -166,7 +166,7 @@ create table task updated datetime null ); -create table user +CREATE TABLE IF NOT EXISTS user ( id char(36) not null primary key, user_name varchar(256) not null, @@ -174,13 +174,13 @@ create table user data blob not null ); -create index user_access_key_index +CREATE INDEX IF NOT EXISTS user_access_key_index on user (access_key); -create index user_user_name_index +CREATE INDEX IF NOT EXISTS user_user_name_index on user (user_name); -create table user_change +CREATE TABLE IF NOT EXISTS user_change ( change_id char(36) not null primary key, user_id char(36) not null, @@ -188,7 +188,7 @@ create table user_change created_timestamp bigint(13) not null ); -create table zone +CREATE TABLE IF NOT EXISTS zone ( id char(36) not null primary key, name varchar(256) not null, @@ -198,13 +198,13 @@ create table zone unique (name) ); -create index zone_admin_group_id_index +CREATE INDEX IF NOT EXISTS zone_admin_group_id_index on zone (admin_group_id); -create index zone_name_index +CREATE INDEX IF NOT EXISTS zone_name_index on zone (name); -create table zone_access +CREATE TABLE IF NOT EXISTS zone_access ( accessor_id char(36) not null, zone_id char(36) not null, @@ -214,13 +214,13 @@ create table zone_access on delete cascade ); -create index zone_access_accessor_id_index +CREATE INDEX IF NOT EXISTS zone_access_accessor_id_index on zone_access (accessor_id); -create index zone_access_zone_id_index +CREATE INDEX IF NOT EXISTS zone_access_zone_id_index on zone_access (zone_id); -create table zone_change +CREATE TABLE IF NOT EXISTS zone_change ( change_id char(36) not null primary key, zone_id char(36) not null, @@ -228,11 +228,13 @@ create table zone_change created_timestamp bigint(13) not null ); -create index zone_change_created_timestamp_index +CREATE INDEX IF NOT EXISTS zone_change_created_timestamp_index on zone_change (created_timestamp); -create index zone_change_zone_id_index +CREATE INDEX IF NOT EXISTS zone_change_zone_id_index on zone_change (zone_id); +DELETE FROM task WHERE name = 'user_sync'; + INSERT IGNORE INTO task(name, in_flight, created, updated) VALUES ('user_sync', 0, NOW(), NULL); diff --git a/modules/mysql/src/main/scala/vinyldns/mysql/MySqlConnector.scala b/modules/mysql/src/main/scala/vinyldns/mysql/MySqlConnector.scala index 25c25b56b..ee4bc6e79 100644 --- a/modules/mysql/src/main/scala/vinyldns/mysql/MySqlConnector.scala +++ b/modules/mysql/src/main/scala/vinyldns/mysql/MySqlConnector.scala @@ -29,8 +29,8 @@ object MySqlConnector { private val logger = LoggerFactory.getLogger("MySqlConnector") def runDBMigrations(config: MySqlConnectionConfig): IO[Unit] = - // We can skip migrations for h2, we'll use the test/ddl.sql for initializing - // that for testing + // We can skip migrations for h2, we'll use the test/ddl.sql for initializing + // that for testing if (config.driver.contains("h2")) IO.unit else { val migrationConnectionSettings = MySqlDataSourceSettings( @@ -90,14 +90,21 @@ object MySqlConnector { def retry[T](times: Int, delayMs: Int)(op: => T) = Iterator .range(0, times) - .map(_ => Try(op)) - .flatMap { - case Success(t) => Some(t) - case Failure(_) => - logger.warn("failed to startup database connection, retrying..") - Thread.sleep(delayMs) - None - } + .map(index => (index, Try(op))) + .flatMap(x => { + val (currentIndex, result) = x + result match { + case Success(t) => Some(t) + case Failure(e) => + logger.warn("failed to startup database connection, retrying..") + // Hard abort if we exhaust retries + if (currentIndex >= times - 1) { + throw e + } + Thread.sleep(delayMs) + None + } + }) .toSeq .head diff --git a/modules/portal/conf/application.conf b/modules/portal/conf/application.conf index f6d6c77b6..35c7809e1 100644 --- a/modules/portal/conf/application.conf +++ b/modules/portal/conf/application.conf @@ -1,23 +1,62 @@ -http.port = 9001 +LDAP { + # For OpenLDAP, this would be a full DN to the admin for LDAP / user that can see all users + user = "cn=admin,dc=planetexpress,dc=com" -crypto { - type = "vinyldns.core.crypto.JavaCrypto" - secret = "8B06A7F3BC8A2497736F1916A123AA40E88217BE9264D8872597EF7A6E5DCE61" -} + # Password for the admin account + password = "GoodNewsEveryone" -data-stores = ["mysql"] -data-stores = ${?DATA_STORES} + # Keep this as an empty string for OpenLDAP + domain = "" -mysql { - repositories { - user { - } - task { - } - user-change { - } + # This will be the name of the LDAP field that carries the user's login id (what they enter in the username in login form) + userNameAttribute = "uid" + + # For organization, leave empty for this demo, the domainName is what matters, and that is the LDAP structure + # to search for users that require login + searchBase = [ + {organization = "", domainName = "ou=people,dc=planetexpress,dc=com"}, + ] + context { + initialContextFactory = "com.sun.jndi.ldap.LdapCtxFactory" + initialContextFactory = ${?LDAP_INITIAL_CONTEXT_CLASS} + securityAuthentication = "simple" + securityAuthentication = ${?LDAP_SECURITY_AUTH} + + # Note: The following assumes a purely docker setup, using container_name = vinyldns-ldap + providerUrl = "ldap://vinyldns-ldap:19004" + providerUrl = ${?LDAP_PROVIDER_URL} + } + + # This is only needed if keeping vinyldns user store in sync with ldap (to auto lock out users who left your + # company for example) + user-sync { + enabled = false + enabled = ${?USER_SYNC_ENABLED} + hours-polling-interval = 1 + hours-polling-interval = ${?USER_SYNC_POLL_INTERVAL} } } -// Local.conf has files specific to your environment, for example your own LDAP settings +# Note: This MUST match the API or strange errors will ensue, NoOpCrypto should not be used for production +crypto { + type = "vinyldns.core.crypto.NoOpCrypto" + type = ${?CRYPTO_TYPE} + secret = ${?CRYPTO_SECRET} +} + +http.port = 9001 +http.port = ${?PORTAL_PORT} + +data-stores = ["mysql"] + +# Must be true to manage shared zones through the portal +shared-display-enabled = true +shared-display-enabled = ${?SHARED_ZONES_ENABLED} + +# You generate this yourself following https://www.playframework.com/documentation/2.7.x/ApplicationSecret +play.http.secret.key = "changeme" +play.http.secret.key = ${?PLAY_HTTP_SECRET_KEY} + +# You can provide configuration overrides via local.conf if you don't want to replace everything in +# this configuration file include "local.conf" diff --git a/modules/r53/src/it/scala/vinyldns/route53/backend/Route53IntegrationSpec.scala b/modules/r53/src/it/scala/vinyldns/route53/backend/Route53IntegrationSpec.scala index dbd5a17e7..cd799f182 100644 --- a/modules/r53/src/it/scala/vinyldns/route53/backend/Route53IntegrationSpec.scala +++ b/modules/r53/src/it/scala/vinyldns/route53/backend/Route53IntegrationSpec.scala @@ -30,7 +30,7 @@ import vinyldns.core.domain.{Fqdn, record} import vinyldns.core.domain.record.{RecordSet, RecordType} class Route53IntegrationSpec - extends AnyWordSpec + extends AnyWordSpec with BeforeAndAfterAll with BeforeAndAfterEach with Matchers { @@ -52,7 +52,7 @@ class Route53IntegrationSpec "test", Option("access"), Option("secret"), - "http://127.0.0.1:19003", + sys.env.getOrElse("R53_SERVICE_ENDPOINT", "http://localhost:19003"), "us-east-1" ) ) diff --git a/modules/sqs/src/it/resources/application.conf b/modules/sqs/src/it/resources/application.conf index 49705a5b0..b235d0f8c 100644 --- a/modules/sqs/src/it/resources/application.conf +++ b/modules/sqs/src/it/resources/application.conf @@ -8,6 +8,7 @@ sqs { secret-key = "test" signing-region = "us-east-1" service-endpoint = "http://localhost:19003/" + service-endpoint = ${?SQS_SERVICE_ENDPOINT} queue-name = "sqs-override-name" } } diff --git a/modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueProviderIntegrationSpec.scala b/modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueProviderIntegrationSpec.scala index 496212e8a..561d7936b 100644 --- a/modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueProviderIntegrationSpec.scala +++ b/modules/sqs/src/it/scala/vinyldns/sqs/queue/SqsMessageQueueProviderIntegrationSpec.scala @@ -15,6 +15,7 @@ */ package vinyldns.sqs.queue + import com.typesafe.config.ConfigFactory import org.scalatest.matchers.should.Matchers import org.scalatest.wordspec.AnyWordSpec @@ -24,50 +25,53 @@ import pureconfig._ import pureconfig.generic.auto._ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers { - val undertest = new SqsMessageQueueProvider() + val underTest = new SqsMessageQueueProvider() + private val sqsEndpoint = sys.env.getOrElse("SQS_SERVICE_ENDPOINT", "http://localhost:19003") "load" should { "fail if a required setting is not provided" in { val badConfig = - ConfigFactory.parseString(""" - | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" - | polling-interval = 250.millis - | messages-per-poll = 10 - | max-retries = 100 - | - | settings { - | service-endpoint = "http://localhost:19003/" - | queue-name = "queue-name" - | } - | """.stripMargin) + ConfigFactory.parseString( + s""" + | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + | polling-interval = 250.millis + | messages-per-poll = 10 + | max-retries = 100 + | + | settings { + | service-endpoint = "$sqsEndpoint" + | queue-name = "queue-name" + | } + | """.stripMargin) val badSettings = ConfigSource.fromConfig(badConfig).loadOrThrow[MessageQueueConfig] - a[pureconfig.error.ConfigReaderException[MessageQueueConfig]] should be thrownBy undertest + a[pureconfig.error.ConfigReaderException[MessageQueueConfig]] should be thrownBy underTest .load(badSettings) .unsafeRunSync() } "create the queue if the queue is non-existent" in { val nonExistentQueueConfig = - ConfigFactory.parseString(""" - | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" - | polling-interval = 250.millis - | messages-per-poll = 10 - | max-retries = 100 - | - | settings { - | access-key = "x" - | secret-key = "x" - | signing-region = "us-east-1" - | service-endpoint = "http://localhost:19003/" - | queue-name = "new-queue" - | } - | """.stripMargin) + ConfigFactory.parseString( + s""" + | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + | polling-interval = 250.millis + | messages-per-poll = 10 + | max-retries = 100 + | + | settings { + | access-key = "x" + | secret-key = "x" + | signing-region = "us-east-1" + | service-endpoint = "$sqsEndpoint" + | queue-name = "new-queue" + | } + | """.stripMargin) val messageConfig = ConfigSource.fromConfig(nonExistentQueueConfig).loadOrThrow[MessageQueueConfig] - val messageQueue = undertest.load(messageConfig).unsafeRunSync() + val messageQueue = underTest.load(messageConfig).unsafeRunSync() noException should be thrownBy messageQueue .asInstanceOf[SqsMessageQueue] @@ -77,65 +81,68 @@ class SqsMessageQueueProviderIntegrationSpec extends AnyWordSpec with Matchers { "fail with InvalidQueueName if an invalid queue name is given" in { val invalidQueueNameConfig = - ConfigFactory.parseString(""" - | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" - | polling-interval = 250.millis - | messages-per-poll = 10 - | max-retries = 100 - | - | settings { - | access-key = "x" - | secret-key = "x" - | signing-region = "us-east-1" - | service-endpoint = "http://localhost:19003/" - | queue-name = "bad*queue*name" - | } - | """.stripMargin) + ConfigFactory.parseString( + s""" + | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + | polling-interval = 250.millis + | messages-per-poll = 10 + | max-retries = 100 + | + | settings { + | access-key = "x" + | secret-key = "x" + | signing-region = "us-east-1" + | service-endpoint = "$sqsEndpoint" + | queue-name = "bad*queue*name" + | } + | """.stripMargin) val messageConfig = ConfigSource.fromConfig(invalidQueueNameConfig).loadOrThrow[MessageQueueConfig] - assertThrows[InvalidQueueName](undertest.load(messageConfig).unsafeRunSync()) + assertThrows[InvalidQueueName](underTest.load(messageConfig).unsafeRunSync()) } "fail with InvalidQueueName if a FIFO queue is specified" in { val fifoQueueName = - ConfigFactory.parseString(""" - | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" - | polling-interval = 250.millis - | messages-per-poll = 10 - | max-retries = 100 - | - | settings { - | access-key = "x" - | secret-key = "x" - | signing-region = "us-east-1" - | service-endpoint = "http://localhost:19003/" - | queue-name = "queue.fifo" - | } - | """.stripMargin) + ConfigFactory.parseString( + s""" + | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + | polling-interval = 250.millis + | messages-per-poll = 10 + | max-retries = 100 + | + | settings { + | access-key = "x" + | secret-key = "x" + | signing-region = "us-east-1" + | service-endpoint = "$sqsEndpoint" + | queue-name = "queue.fifo" + | } + | """.stripMargin) val messageConfig = ConfigSource.fromConfig(fifoQueueName).loadOrThrow[MessageQueueConfig] - assertThrows[InvalidQueueName](undertest.load(messageConfig).unsafeRunSync()) + assertThrows[InvalidQueueName](underTest.load(messageConfig).unsafeRunSync()) } } "MessageQueueLoader" should { "invoke SQS provider properly" in { val nonExistentQueueConfig = - ConfigFactory.parseString(""" - | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" - | polling-interval = 250.millis - | messages-per-poll = 10 - | max-retries = 100 - | - | settings { - | access-key = "x" - | secret-key = "x" - | signing-region = "us-east-1" - | service-endpoint = "http://localhost:19003/" - | queue-name = "new-queue" - | } - | """.stripMargin) + ConfigFactory.parseString( + s""" + | class-name = "vinyldns.sqs.queue.SqsMessageQueueProvider" + | polling-interval = 250.millis + | messages-per-poll = 10 + | max-retries = 100 + | + | settings { + | access-key = "x" + | secret-key = "x" + | signing-region = "us-east-1" + | service-endpoint = "$sqsEndpoint" + | queue-name = "new-queue" + | } + | """.stripMargin) val messageConfig = ConfigSource.fromConfig(nonExistentQueueConfig).loadOrThrow[MessageQueueConfig] diff --git a/quickstart/.env b/quickstart/.env index 4a0f5894f..fb0b428e2 100644 --- a/quickstart/.env +++ b/quickstart/.env @@ -11,8 +11,9 @@ TEST_LOGIN=false # API Settings REST_PORT=9000 -SQS_ENDPOINT=http://vinyldns-integration:19003 +SQS_SERVICE_ENDPOINT=http://vinyldns-integration:19003 SNS_SERVICE_ENDPOINT=http://vinyldns-integration:19003 +R53_SERVICE_ENDPOINT=http://vinyldns-integration:19003 MYSQL_ENDPOINT=vinyldns-integration:19002 DEFAULT_DNS_ADDRESS=vinyldns-integration:19001 diff --git a/test/api/functional/application.conf b/test/api/functional/application.conf index 5b02a3601..a79c5cf1b 100644 --- a/test/api/functional/application.conf +++ b/test/api/functional/application.conf @@ -85,7 +85,7 @@ vinyldns { # Endpoint to access queue service-endpoint = "http://localhost:19003/" - service-endpoint = ${?SQS_ENDPOINT} + service-endpoint = ${?SQS_SERVICE_ENDPOINT} # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change. queue-name = "vinyldns" diff --git a/test/api/integration/.env.integration b/test/api/integration/.env.integration new file mode 100755 index 000000000..12521a5c3 --- /dev/null +++ b/test/api/integration/.env.integration @@ -0,0 +1,27 @@ +# Overrides that can be used when dependent services are running in the vinyldns-integration host +# and 'sbt' or tests are running in another container. More importantly, when these services are +# not accessible via "localhost" +# +# Any docker container that wishes to to use these, can simply specify this file using the `--env-file` +# argument. + +# General settings +VINYLDNS_API_URL=http://vinyldns-integration:9000 +VINYLDNS_PORTAL_URL=http://vinyldns-integration:9001 + +# Portal settings +VINYLDNS_BACKEND_URL=http://vinyldns-integration:9000 +LDAP_PROVIDER_URL=ldap://vinyldns-integration:19004 + +# API Settings +SQS_SERVICE_ENDPOINT=http://vinyldns-integration:19003 +SNS_SERVICE_ENDPOINT=http://vinyldns-integration:19003 +R53_SERVICE_ENDPOINT=http://vinyldns-integration:19003 +MYSQL_ENDPOINT=vinyldns-integration:19002 +DEFAULT_DNS_ADDRESS=vinyldns-integration:19001 + +JDBC_DRIVER=org.mariadb.jdbc.Driver +JDBC_USER=root +JDBC_PASSWORD=pass +JDBC_URL=jdbc:mariadb://vinyldns-integration:19002/vinyldns?user=${JDBC_USER}&password=${JDBC_PASSWORD} +JDBC_MIGRATION_URL=jdbc:mariadb://vinyldns-integration:19002/?user=${JDBC_USER}&password=${JDBC_PASSWORD} diff --git a/test/api/integration/Makefile b/test/api/integration/Makefile index 1bbb79ff8..4abd336cb 100644 --- a/test/api/integration/Makefile +++ b/test/api/integration/Makefile @@ -38,14 +38,16 @@ build: run: @set -euo pipefail + docker network create --driver bridge vinyldns_net &> /dev/null || true USE_TTY="" && test -t 1 && USE_TTY="-t" - docker run -i $${USE_TTY} --rm $(DOCKER_PARAMS) $(IMAGE_NAME) $(ARG_SEPARATOR) $(WITH_ARGS) + docker run -i $${USE_TTY} --rm --network vinyldns_net $(DOCKER_PARAMS) $(IMAGE_NAME) $(ARG_SEPARATOR) $(WITH_ARGS) run-bg: @set -euo pipefail docker stop $(IMAGE_NAME) &> /dev/null || true + docker network create --driver bridge vinyldns_net &> /dev/null || true USE_TTY="" && test -t 1 && USE_TTY="-t" - docker run -d $${USE_TTY} --name $(IMAGE_NAME) --rm $(DOCKER_PARAMS) -e RUN_SERVICES="deps-only tail-logs" -p 19001-19003:19001-19003 -p 19001:19001/udp $(IMAGE_NAME) + docker run -d $${USE_TTY} --name $(IMAGE_NAME) --rm --network vinyldns_net $(DOCKER_PARAMS) -e RUN_SERVICES="deps-only tail-logs" -p 19001-19003:19001-19003 -p 19001:19001/udp $(IMAGE_NAME) stop-bg: @set -euo pipefail @@ -53,8 +55,9 @@ stop-bg: run-local: @set -euo pipefail + docker network create --driver bridge vinyldns_net &> /dev/null || true USE_TTY="" && test -t 1 && USE_TTY="-t" - docker run -i $${USE_TTY} --rm $(DOCKER_PARAMS) -v "$(ROOT_DIR)/../../..:/build" $(IMAGE_NAME) -- $(WITH_ARGS) + docker run -i $${USE_TTY} --rm --network vinyldns_net $(DOCKER_PARAMS) -v "$(ROOT_DIR)/../../..:/build" $(IMAGE_NAME) -- $(WITH_ARGS) clean-containers: @set -euo pipefail diff --git a/test/api/integration/application.conf b/test/api/integration/application.conf index a13a8b256..d6a7500d5 100644 --- a/test/api/integration/application.conf +++ b/test/api/integration/application.conf @@ -114,7 +114,7 @@ vinyldns { # Endpoint to access queue service-endpoint = "http://localhost:19003/" - service-endpoint = ${?SQS_ENDPOINT} + service-endpoint = ${?SQS_SERVICE_ENDPOINT} # Queue name. Should be used in conjunction with service endpoint, rather than using a queue url which is subject to change. queue-name = "vinyldns"