0

BIND 9.19 Planning: Testing

Matthijs Mekking edited this page 2021-11-04 10:35:56 +00:00

Table of Contents

• The problems with the system tests
• Thoughts
• Goals
• Other test topics
• Journal files issues
• XFR Resilience
• inline-signing review, test uplift.
• A Golden Oldie: moving GitLab CI to AWS

Back to agenda: https://gitlab.isc.org/isc-projects/bind9/-/wikis/BIND-9.19-Plan

Attendees: Matthijs, Petr, Michał K., Michal N., Aram, Artem, Evan, Mark, Ondrej.

The problems with the system tests

Thoughts

Petr: System Tests are not treated as first class code. We can do better.

Petr: There is a lot of boiler plate & repetition in the shell script system tests. We should try to refactor repeated bits of test code into helper functions as this makes reviews and reading easier.

Michał: Boiler plate work started, but if we are going to move Python it doesn't make sense to complete that work. At least follow current conventions.

Ondrej: Suggests hard deadline of no new bash tests.

Matthijs: If you can justify why writing it in bash is better it should be allowed.

Michał: We don't have Python building blocks yet. We should focus on this, this makes writing tests in the future easier.

Ondrej: Suggests that from 9.18.0 we should default to writing tests in Python.

Artem: Agrees in principle. But suggests not rewriting existing tests.

Ondrej: Current system tests often will have errors that we don't know about.

Evan: We might make new mistakes in Python.

Evan: The tools (dig, rndc, etc.) are a part of BIND 9. We need to test those as well. There is room for both using dnspython and using tools for sending queries.

Michał: Start building basic Python blocks for testing

1. Send query (log and check expected response)
2. RNDC calls
3. Grepping through log files and waiting for log lines to appear

Ondrej: Pytest advantage: if split into small test functions, they can be run in parralel.

Michał: But we can't spin up 200 named servers.

Clarification: Running 200 named servers at the same time is problematic, spinning them up for short tests a few after another is not so bad.

Can we use Deckard instead of "ns1 root server" (or ldns-testns or something similar). This is similar to our "Perl ans" servers, except we don't need to write Perl code. Deckard is using a file that basically describes with pseudo language how to respond to certain classes of answers.

See https://github.com/CZ-NIC/deckard

Goals

• Python written system tests (with few exceptions if justifiable).
• Implement a "library" of reusable components that could replace "well-known" shell blocks.
• Most repetition:
  • Sending queries (and log responses, check against expected output).
  • rndc commands
  • Waiting for log messages to appear.
• Property based testing (Hypothesis) to generally improve test coverage & write less spaghetti code.
• Don't use our own software to test the server (replace dig with dnspython for example, replace ns1 servers with a mock server like Deckard for example).

The idea being that this should facilitate writing Python tests more quickly/conveniently (it's pretty much the Wild West right now).

Other test topics

Journal files issues

Test first and see if/what is the problem (some sort of "framework" for "cross-version" tests?)

Ondrej: Make interop testing framework with other vendors

Evan: Compatibility testing with previous versions of our files.

QA: On the roadmap but not high priority.

Having them only in Gitlab might be okay - use prebuilt Docker images and swap files between them.

XFR Resilience

Work on xfr/axfr resilience and (lower) impact on other processing as more users are doing transfers (large transfers) constantly.

This is tightly coupled with RBTDB refactoring.

Priority proportional to number of support tickets.

How is the response rate & latency affected by zone transfers.

How long it takes from starting to transfer to serving the zone.

Memory consumption. NSD has limit of 4 times zone size (we should aim for 2, soft limit 4 with AFXR).

Also consider file size, disk I/O.

For all these things we should calculate the base line. So we can make sure that refactoring/changes do not make matters worse.

inline-signing review, test uplift.

Related to zone.c issues, make changes when server is offline.

A Golden Oldie: moving GitLab CI to AWS

Discussed in 2018. There is hope. We know there is a bottleneck, there is a possible solution, but it comes at the cost of complexity. This year we had 20k credits from AWS. Not next year.

Mark suggests to have more non Linux platforms and some high core count machines (the Redhat case).