2
0
mirror of https://github.com/VinylDNS/vinyldns synced 2025-08-21 17:37:15 +00:00

71 Commits

Author SHA1 Message Date
Nicholas Spadaccino
7eed595bd4
Merge branch 'master' into dependency_upgrade 2025-07-24 17:50:56 -04:00
Jay07GIT
95f6e72c4a
update
Signed-off-by: Jay07GIT <jeyraj931@gmail.com>
2025-03-04 15:46:29 +05:30
Jay07GIT
ee2581cc3a
Dependencies upgraded for both mac M1 and Intel chip 2022-11-21 11:03:53 +05:30
Jay07GIT
e805423e3b
Update in tests 2022-08-18 15:01:14 +05:30
Emerle, Ryan
a40c231853
Add twitter handle to documentation [ci skip] 2022-05-23 09:20:29 -04:00
Emerle, Ryan
02d702f461
Performance tuning
- Add `getGroupsAbridged` which returns a subset of group data for dropdowns and other places where all groups are listed
- Remove unnecessary checks for `canSeeGroup` in `groups.scala.html` since all users can see all groups
- Move `ZoneController` initialization in `manageZone.scala.html` to higher level to avoid waiting for groups to load when expanding the select box
- Add `PreparePortalHook` to automatically run `prepare-portal.sh` when `project porta; run` is executed
2022-05-19 14:41:11 -04:00
Emerle, Ryan
b24cf8db54
Updates
- Rename RecordSetData as RecordSetCache
- Refactoring
- Fix ip address storage to be binary instead of string
- Align recordset_data table contents with migration tooling
- Add feature flag for recordset cache `use-recordset-cache`
- Add feature flag for loading test data `load-test-data`
2022-05-13 13:31:37 -04:00
Emerle, Ryan
f34e6928b2
Update CoC and contact details 2021-12-14 17:16:30 -05:00
Ryan Emerle
85577a27f0
Consistency Updates
- Clean up documentation
- Update architecure diagram
- Fix discrepencies between local and docker test execution
- Fix inconsistencies in various configuration files used for tests and
execution
2021-12-08 14:36:00 -05:00
Emerle, Ryan
e1743e5342
Updates
- Simplify build config
- Add TTY check to Makefiles for running Docker containers
- Update `fs2` to latest patch
- Update `sbt-assembly` plugin
- Update portal to remove chatty console
- Update portal scripts to add license header
- Update prepare-portal/Gruntfile to combine js and css where applicable
- Remove unused gentelella files from final portal artifact
- Add support for shared zones to quickstart/docker images
- Consolidate built artifacts in `artifacts/` to make eventual release easier
2021-11-02 17:06:24 -04:00
Emerle, Ryan
f2db11e89a
Update Release
- Remove sonatype release from sbt build
- Remove instructions about release - it will be automated by GHA
- Remove `sbt-release` plugin
2021-10-29 12:23:08 -04:00
Emerle, Ryan
5fe33eee22
Update docs
- Fix broken links
- Fix formatting
- Add Makefile for running via docker
- Move README.md from `modules/docs/src/main/mdoc` to `modules/docs` to be consistent with `modules/portal`
2021-10-21 14:21:37 -04:00
Emerle, Ryan
9ce466aa0c
Fix microsite broken dependencies and update docs 2021-10-21 11:44:53 -04:00
Emerle, Ryan
a075c3c35e Updates
- Move away from using multiple images for "quickstart" and instead use a single "integration" image which provides all of the dependencies

- Update `docker-up-vinyldns.sh` to support the new `integration` image
- Update `remove-vinyl-containers.sh` to more cleanly.. clean up
- Update `verify.sh` to more reliably run `sbt` targets
- Update `build/docker/api/application.conf` to allow for overrides and default to the `vinyldns-integration` image
- Update `build/docker/portal/application.conf` to allow overrides and use `vinyldns-integration` image
  - Update `build/docker/portal/Dockerfile` to use `vinyldns/build:base-build-portal` to reduce need to download dependencies over and over
- Update `api/assembly` sbt target to output to `assembly` rather than some deeply nested folder in `**/target`
- Update documentation to reflect changes

- Move `docker/` directory to `quickstart/` to reduce confusion with the `build/docker` directory
- Move `bin/` to `utils/` since the files are binaries

- Add `.dockerignore` to root
2021-10-20 09:07:19 -04:00
Emerle, Ryan
07b683cbd0 Updates
- Remove old, unused scripts in `bin/`
- Remove old images from release
  - `test` and `test-bind` are no longer necessary. Test images are in a different repo now
- Remove Docker image creation from sbt build config - actual `Dockerfile` files are easier to deal with

- Update scripts in `bin/` to utilize new Docker images
- Update documentation for changes
- Update all Docker Compose and configuration to use exposed ports on the `integration` image (19001, 19002, etc) both inside the container and outside to make testing more consistent irrespective of method
- Update FlywayDB dependency to v8 to fix a weird logging bug that showed up during integration testing. See: https://github.com/flyway/flyway/issues/2270

- Add `test/api/integration` Docker container definition to be used for any integration testing

- Move `module/api/functional_test` to `test/api/functional` to centralize the "integration-type" external tests and testing utilities

- Move functional testing and integration image to the `test/` folder off of the root to reduce confusion with `bin/` and `docker/`
2021-10-15 15:06:04 -04:00
Emerle, Ryan
0a1b533192 WIP - Functional Test Updates
- Update `dnsjava` library
- Add support for H2 database
- Update functional tests to support parallel runs
- Remove the ability to specify number of processes for functional tests - always 4 now
- Add `Makefile` and `Dockerfile` in `functional_test` to make it easier to run tests without spinning up multiple containers
2021-10-08 15:52:09 -04:00
Paul Cleary
25acdb13c7
Burn dynamo burn (#1025)
Fixes #971 

Changes in this pull request:
- Remove the module
- Update `build.sbt`
- Update docs
- Update docker things
- Update configs everywhere
2020-10-23 17:26:52 -04:00
Paul Cleary
b9a56bc123
[ci skip] Fixing menu for docs (#1024)
Had some side bar issues, the `menu.yml` was inadvertantly not merge because of .gitignore
2020-10-23 14:12:38 -04:00
Paul Cleary
97e80e54e8
Update microsite, move to mdoc (#1023)
We used to rely on `tut` for docs, however it is deprecated to be replaced with mdoc.

Moved to an `mdoc` folder structure and updated all of the links (what a pain).
2020-10-23 08:22:26 -04:00
Paul Cleary
97ef44907b
Include r53 in docker build (#1005)
* Include r53 in docker build

* Removing unnecessary config file

* Decrease coverage min for route 53
2020-10-10 10:53:34 -04:00
Paul Cleary
c56a9cad38
Update build.sbt (#996) 2020-10-04 15:28:33 -04:00
Paul Cleary
a988bcd9a8
Add backend provider (#980)
Introduces the concept of a `Backend` into VinylDNS.  This will allow support for any DNS backend in the future, including AwS Route53 for example.  This is consistent with other "provider" things for dynamic loading of classes (Notifier, Repository, Queue, etc.)

The initial implementation builds on what we have already, that is when creating a zone one can choose a `backendId` that is configured in the `application.conf`.  If no `backendId` is specified, we attempt to map like we do today, so the exact same functionality.

We expand that by allowing one to map a `backendId` to a different provider (like aws). 

After this PR:
1. If someone specifies a zone connection on a zone, it will work exactly like it does today, namely go through the `DnsBackend` to connect.
2. If someone specifies a `backendId` when setting up a zone, the naive mapping will take place to map that zone to the `Backend` implementation that is configured with that `backendId`.   For example, if you have configured a backend id `aws` that connects to Route53, and you specify `aws` when connecting the zone, it will connect to it in Route 53 **Note: we still do not support zone create, but that is much closer to reality with this PR, much much**
3. If someone specifies NEITHER, the `defaultBackendId` will be used, which could be on any one of the backend providers configured.

To start, there is a new `vinyldns.core.domain.backend` package that contains the main classes for the system.  In there you will find the following:

- `BackendProvider` - this is to be implemented by each provider.  Adds a means of pre-loading zones, and providing connections to zones. 
- `Backend` - provides connectivity to a particular backend instance.  For example, a particular DNS Authoritative server.  This is where the real work happens of interacting with whatever backend.  For example, `DnsConnection` implements this to send DDNS messages to the DNS system.  Consider this the "main" thing to implement, where the rubber meets the road, the meat and potatoes
- `BackendProviderLoader` - to be implemented by each provider, knows how to load it's single instance `BackendProvider`, as well as possibly pre-loading configured `Backends` or anything else it needs to do to get ready.  It provides a dynamic hook via the `def load` method that is called by the `BackendLoader` to load a specific `Backend`
- `BackendResolver` - the main, default, BackendResolver.  It holds all `BackendProvider` instances loaded via the `BackendLoader` and provides right now a naive lookup mechanism to find `Backend`s.  Really, this is more of a `Router` or `Resolver`, as in the future it could use more advanced techniques to finding connections than right now
- `BackendConfigs` - used by the `BackendRegistry` as the entrypoint into configuration for all backends
- `BackendProviderConfig` - a single backend provider configuration, specifies a `className` that should be the `BackendProviderLoader` implementation to be loaded, and a `settings` that is passed into the `BackendProvider` to load itself.  This is consistent with other providers.
- `BackendResponse` - uniform responses across all providers to the rest of the VinylDNS System

**Workflow**
During initialization of the system:

1. The `BackendResolver` loads the `BackendConfigs` from the application configuration.  This contains configuration for ALL backends
2. The `BackendResolver` utilizes the `BackendLoader` to dynamically load each backend individually.  If any backend cannot be loaded, it will fail.
3. The `BackendLoader` creates a new instance of each `className` for each `BackendConfig`, this points to the `BackendProviderLoader` implementation which takes care of loading the specific `BackendProvider` provided the configuration
4. The `BackendProviderLoader` does any initialization necessary to ensure it is ready.  In the case of `Route53`, it will pre-load and cache all hosted zones that are available for the AWS account that is configured.  For Route53, a single `Route53Backend` is setup right now.  For `DnsBackend`, a connection (server, port, tsig key) is setup for each DNS Authoritative system to integrate with.

During runtime of the system:

1. When anything is needed, the `BackendResolver` is consulted that will determine how to lookup the `Backend` that is needed.  This is done right now by naively scanning all `BackendProvider` instances it has to say "can anyone connect to this zone".  More intelligent discovery rules can be added in the future
2. Once a `Backend` is obtained, any operation can be performed:
    1. `ZoneConnectionValidator` uses `zoneExists` and `loadZone` to validate a zone is usable by VinylDNS
    2. `RecordSetChangeHandler` uses `resolve` and `applyChange` to apply changes to the DNS backend
    3. `ZoneSyncHandler` and `DnsZoneViewLoader` use `loadZone` in order to load records into VinylDNS

**What else is here**

- Provided an implementation of a backend provider for DNS via `Backend`
- Updated all of VinylDNS to use `Backends` instead of hard coded to DNS
- Provided an implementation of a backend provider for AWS Route 53 as an example to follow for other providers


**Example configuration**

```
vinyldns {
  backend {
    default-backend-id = "r53"

    backend-providers = [
      {
        class-name = "vinyldns.route53.backend.Route53BackendProviderLoader"
        settings = {
          backends = [
            {
              id = "test"
              access-key = "vinyldnsTest"
              secret-key = "notNeededForSnsLocal"
              service-endpoint = "http://127.0.0.1:19009"
              signing-region = "us-east-1"
            }
          ]
        }
      }
    ]
  }
}
```
2020-09-30 09:17:32 -04:00
Paul Cleary
0725093636
Skip CI for certain tests (#983)
The `EmailNotifierIntegrationSpec` does not work in Github actions for some reason, likely due to sending an email.

1. Added a `SkipCI` tag that will enable us to by default skip certain tests in CI.
2. Updated the `build.sbt` to by default exclude `SkipCI` tests
3. Added `taggedAs(SkipCI)` to the email integration test
2020-09-29 12:58:10 -04:00
Ryan Emerle
4310adb667
Add support for simple request tracing (#977)
- Add request tracing header for calls from `portal->API`
- Update request logging to include `trace.id`
- Tame down the monitor logging to prevent log pollution

Closes #976
2020-09-15 13:40:31 -04:00
Paul Cleary
1de710b991
Update dependencies (#939)
* Updating dependencies

Updated almost all dependencies to current.  There were some issues with
akka-http 10.1.11 so I stayed with 10.1.10 for the time being.

Func tests passed locally and manual review of the UI looks to be good

Significant changes are:

- `pureconfig` - this update had breaking syntax, so I had to update everywhere
  we use pureconfig.  Functionally it is the same, just different syntax
- `scalatest` - this was a big change, as scalatest has refactored out things
  like Mockito and scalacheck.  Many imports changed.
- `Java11` - formally moved everything to java 11.  This required some new
  dependencies like `javax.activation` and `java.xml.bind`

* Updating travis to JDK 11

* Finishing JDK 11 update

In order to update to JDK 11, needed to modify several docker things.

Removed timeout test that was causing issues as timeout tests here are not good
for running in travis.
2020-04-28 09:08:03 -05:00
Paul Cleary
7f1ac230be
Update Release (#890)
Updated release process:

- `bin/release.sh` - added checks so we can only release from master, and can only release from upstream
- `build.sbt` - removed sbt publishing of docker images, we will now use `build/docker-release.sh` for that release
- `build/release.sh` -- renamed --> `build/docker-release.sh`
- `build/docker-release.sh` - added a version override to make it simple to force a version
2019-10-30 07:04:11 -04:00
Paul Cleary
a24cdddc9f
Build optimizations (#883)
A few specific build optimizations:

1. Consolidated `dockerComposeUp` to only use a single `root/docker/docker-compose.yml` instead of each module having its own docker-compose files.  This eliminates additional waits for docker containers to startup and stop, as well as reduces memory consumption during the build
2. Cleaned up `VinylDNSSpec` - I noticed that this spec was taking 3 minutes to run!  I discovered that the way we were mocking the `WSClient` was largely to blame.  Was able to get tests to run in **16 SECONDS** using a library called `mock-ws`.  This is where we see most savings.
3. Added back `dynamodb-local` instead of running it in `localstack`.  Integration tests for dynamodb were very slow in localstack.  This added an additional 20-30 second improvement.

After doing several tests locally running the following command...

```
> SECONDS=0; sbt verify; echo "DURATION = $SECONDS SECONDS"
```

Current master took 535 seconds to run; with these optimizations it took **211 SECONDS** - that is a 60% improvement.

The initial Travis builds reported a run time of 13 minutes as opposed to 19 minutes; this would save some 6 minutes off of Travis build times (or 30% improvement).
2019-10-21 12:02:27 -04:00
Paul Cleary
730c6f8897
Par func tests (#838)
Major overhaul of func tests to allow them to run in parallel.  Major changes include:

1. Consolidate all separate test fixtures into a single test fixture in the `shared_zone_test_context`
1. Add `xdist` to allow running tests in parallel
1. Add hooks in main `conftest.py` to setup the test fixture before workers run, and tear it down when workers are finished
1. After fixture is setup, save state in a local `tmp.out` so the workers will use that state instead of trying to recreate the fixture.
1. Add a `utils.generate_record_name` which generates a unique record name in order to avoid conflicts when running tests in parallel
1. Add a `pytest.mark.serial` for func tests that just cannot be run in serial
1. Tests are now run in two phases, first we run in parallel, and if that is successful, we run the serial tests
1. Add a `--teardown` flag, this allows us to reuse the test fixture between the two phases parallel and serial
2019-09-18 15:02:25 -04:00
Paul Cleary
d7516e0b85
Fix docker releases (#787)
* Fix docker releases

There was an issue starting the docker containers due to how native packager
works where we were seeing issues with the container being able to start.

The issue was that we were assuming a "daemon" user to run the containers under.
At some point this changed to "1001:0".  As a result, there were not sufficient
privileges to start the containers because the "daemon" user was invalid or did
not have access to the scripts created by sbt native packager.

* `build.sbt` - update the user to "1001:0" for our custom install.  Cleaned up the hardcoded references in the script extras to `/opt/docker` to use the variable `app_home` instead.
* `plugins.sbt` - updated to the latest sbt native packager
2019-08-07 14:24:29 -04:00
Paul Cleary
e0456491fb
Updating play to 2.7 (#783)
* Upgrade to Play 2.7
* Upgrade to akka-http 10.1.9
* Upgrade to akk 2.5.23
2019-08-07 09:13:05 -04:00
Nima Eskandary
619d9c9e46
update latest tag on release (#712)
* chage how we set dockerUpdateLatest
2019-06-26 16:11:50 -04:00
Juan Valencia
382b9a54cf Include *.csv in the built microsite (#625)
* Include `.csv` files as part of the ones that be included when building the site
2019-05-10 08:12:31 -04:00
Nima Eskandary
4fa0b82664
change openjdk version to openjdk:8u201-jdk-alpine3.9 (#597) 2019-04-24 12:17:17 -04:00
Paul Cleary
002c5f0e51
Changing base image to alpine (#596) 2019-04-24 08:55:18 -04:00
Nima Eskandary
3dadc35a00 upgrade to scala 2.12.18 (#569) 2019-04-08 15:50:27 -04:00
Nima Eskandary
1a2d12a7e9
have portal wait for mysql (#527)
* have portal docker image wait for mysql before starting up
2019-03-18 12:00:41 -04:00
Rebecca Star
cc430ce90c
test data cleanup (#418)
* test data cleanup
2019-01-07 16:43:39 -05:00
Marco
1de36ca6b5 remove sonatype credential setting and update docs (#306) 2018-12-17 12:24:27 -05:00
Paul Cleary
469e49e6f6
Adding version to login (#372)
* Adding version to login

* Allow override of version

* Have default version based on build

* Adding version to local dev
2018-12-05 15:29:22 -05:00
Rebecca Star
3216aa5bbe
fix sonotype command (#378) 2018-12-03 11:35:18 -05:00
Rebecca Star
ad99525cdf
portal rely on mysql (#361) 2018-11-19 17:41:58 -05:00
Rebecca Star
9d58713f92
fix portal configuration for local run (#358) 2018-11-19 09:35:47 -05:00
Paul Cleary
5832fcbb73
Add MySqlRecordSetRepository (#309)
* Add MySqlRecordSetRepository
* Updated docker for mysql to use general_log for fun sql debug times
* Made sure to use rewriteBatchStatements to acheive new hights for bulk inserts
* `MySqlDataStoreProvider` support for the record set repo
2018-10-30 15:29:41 -04:00
Rebecca Star
bfb8aa7117
Mysql queue loader (#303)
* mysql queue loader

* use namedDB

* cleanup, make logging tweeaks
2018-10-29 11:27:43 -04:00
Varsha Chnadrashekar
b5f059a6fd Adding shutdown to DataStoreProvider (#293)
* Adding shutdown to DataStore

* Calling shutdown from Boot
2018-10-23 09:28:58 -04:00
Rebecca Star
8adc4e6073
move logging dep to core (#294) 2018-10-19 10:58:39 -04:00
Michael Ly
db6412b0d1
Implement message queue in AWS SQS (#255)
* Move SQS to module.

* Implement SqsMessageQueue
2018-10-12 10:21:44 -04:00
Britney Wright
99214f9e35
add edit button to doc site (#275) 2018-10-10 14:31:43 -04:00
Nima Eskandary
3c4c5f2e99
exclude filter for local.conf (#267)
* exclude `modules/portal/conf/local.conf` from packaging
2018-10-09 13:05:21 -04:00
David Margolin
0161676a56 update scala (#253)
Update scala to 2.12.7
2018-10-03 17:17:36 -04:00