|
|
## The Basics
|
|
|
## General recommendation
|
|
|
|
|
|
The basic BIND 9 memory management object is a memory context: the application can have as many as is practical. There are two reasons for a separate memory context: a) logical separation - this includes both separate accounting, and different configuration, and b) contention and speed - access to a memory context pinned on a specific thread will not be blocked by different threads.
|
|
|
|
|
|
## Limiting memory use
|
|
|
All deployments should use [jemalloc](http://jemalloc.net/) library if possible. To use jemalloc with BIND 9.16, force the linkage via extra `LDFLAGS` variable:
|
|
|
```
|
|
|
./configure LDFLAGS="-ljemalloc"
|
|
|
```
|
|
|
|
|
|
The configuration option `max-cache-size` only affects the memory context in the cache and ADB (address database). All other memory contexts are unconstrained. This means setting the `max-cache-size` to 100% would lead to the OOM Reaper finding your BIND 9 process and killing it.
|
|
|
## Differences between BIND versions
|
|
|
|
|
|
### BIND 9.16 uses more memory than BIND 9.11
|
|
|
|
... | ... | @@ -17,26 +18,34 @@ There are two reasons for this: |
|
|
|
|
|
BIND 9.18 uses less memory than 9.16, similar to the memory usage in 9.11. The part that sends and processes outgoing DNS messages (server side) was refactored to use the new networking code and therefore uses half as many threads as BIND 9.16 used.
|
|
|
|
|
|
The other major change implemented in BIND 9.18 was the replacement of the internal memory allocator with the jemalloc memory allocator. The internal memory allocator kept pools of memory for later reuse and would never free up the reserved memory. The jemalloc memory allocator is much better suited to the memory usage patterns that BIND 9 exhibits and is able to be both fast and memory efficient.
|
|
|
The other major change implemented in BIND 9.18 was the replacement of the old internal memory allocator from 2000 with the [jemalloc](http://jemalloc.net/) memory allocator.
|
|
|
|
|
|
The internal memory allocator kept pools of memory for later reuse and would never free up the reserved memory. The jemalloc memory allocator is much better suited to the memory usage patterns that BIND 9 exhibits and is able to be both fast and memory efficient.
|
|
|
|
|
|
Our general recommendation for all deployments is to use jemalloc if possible. You can use jemalloc with BIND 9.16 by forcing the linkage via extra LDFLAGS (./configure LDFLAGS="-ljemalloc" should do the trick).
|
|
|
## Limiting memory use
|
|
|
|
|
|
The configuration statement [`max-cache-size`](https://bind9.readthedocs.io/en/latest/reference.html#namedconf-statement-max-cache-size) only affects the DNS cache and ADB (address database). All other memory uses remain unconstrained. This means setting the `max-cache-size` to 100% will eventually allow cache+ADB to consume all system memory and nothing will remain for other uses. This will subsequently trigger [Out-Of-Memory Reaper](https://www.baeldung.com/linux/memory-overcommitment-oom-killer) finding your BIND 9 process and killing it without mercy.
|
|
|
|
|
|
## Terminology
|
|
|
|
|
|
The basic BIND 9 memory management object is a "memory context": a running BIND instance has many of them. There are two reasons for separate memory contexts:
|
|
|
1. logical separation - this includes both separate accounting, and different configuration, and
|
|
|
2. contention and speed - access to a memory context pinned on a specific thread will not be blocked by different threads.
|
|
|
|
|
|
## Measuring Memory
|
|
|
|
|
|
Measuring real memory usage can be tricky, but fortunately, there are some tools to help with that.
|
|
|
|
|
|
### Measuring Memory Internally
|
|
|
### Measuring Memory Internally - BIND's view of memory usage
|
|
|
|
|
|
The [statistics channel](https://bind9.readthedocs.io/en/latest/reference.html#namedconf-statement-statistics-channels) exposes counters for the memory contexts. The important values are 'InUse' and 'Malloced'. The 'InUse' counter shows the memory used "externally" and 'Malloced' includes the management overhead (the more memory contexts the more overhead there is).
|
|
|
The BIND [statistics channel](https://bind9.readthedocs.io/en/latest/reference.html#namedconf-statement-statistics-channels) exposes counters for individual memory contexts. The important values are 'InUse' and `Malloced`. The `InUse` counter shows the memory actively used by BIND and `Malloced` includes the memory management overhead. Values reported by the operating system should be close to `Malloced` values.
|
|
|
|
|
|
You can use attached [memory-json.py](uploads/d907c0f06b7d36469443019920bf4b05/memory-json.py) script to parse the statistics channel output from URL `/json/v1`.
|
|
|
|
|
|
In case BIND was compiled only with XML output support, take statistics from URL `/xml/v3` and convert it to JSON using [xq command](https://kislyuk.github.io/yq/#xml-support):
|
|
|
To summarize BIND's view of memory use, use script [memory-json.py](uploads/d907c0f06b7d36469443019920bf4b05/memory-json.py). The script parses the statistics channel output from URL `<bind>/json/v1` and produces summaries. In case BIND was compiled only with XML output support, take statistics from URL `<bind>/xml/v3` and convert it to JSON using [xq command](https://kislyuk.github.io/yq/#xml-support):
|
|
|
```
|
|
|
xq < stats.xml > stats.json
|
|
|
```
|
|
|
|
|
|
Finally, the script above outputs summary data (this is from `main` branch):
|
|
|
Finally, the script above outputs summary data (this is from `main` BIND branch):
|
|
|
```
|
|
|
./memory-json.py stats.json
|
|
|
```
|
... | ... | @@ -59,11 +68,14 @@ MALLOCED: 13.3MiB == 13.3MiB |
|
|
|
|
|
### Measuring Memory Externally
|
|
|
|
|
|
The rule of thumb is "Don't use the 'top' command" - there are better tools that are less misleading. There are two tools the are easily available on modern Linux systems - **pmap and smem**.
|
|
|
The rule of thumb is
|
|
|
> Don't use the 'top' command
|
|
|
|
|
|
There are better tools that are less misleading. There are two tools the are easily available on modern Linux systems - **pmap** and **smem**.
|
|
|
|
|
|
#### pmap
|
|
|
|
|
|
`pmap` provides detailed statistics, but can be too chatty - the basic usage is `pmap -x -p <pid>`. It prints information about all pages used by the program which includes shared libraries, the program itself and the heap. The important number is the last one "Dirty" - it shows the memory "used" by the BIND 9.
|
|
|
[pmap](https://linux.die.net/man/1/pmap) provides detailed statistics, but can be too chatty - the basic usage is `pmap -x -p <pid>`. It prints information about all pages used by the program which includes shared libraries, the program itself and the heap. The important number is the last one "Dirty" - it shows the memory "used" by the BIND 9.
|
|
|
|
|
|
Example `pmap` output could look like this:
|
|
|
|
... | ... | @@ -82,7 +94,7 @@ total kB 760180 74324 60708 |
|
|
|
|
|
#### smem
|
|
|
|
|
|
`smem` provides fewer details, so if you want only a single number, run `smem -P named` and look for the USS column - this provides the information about memory used by the program sans the shared library. The PSS column adds shared libraries divided by the number of programs using those libraries, and RSS is the normal Resident Size.
|
|
|
[smem](https://linux.die.net/man/8/smem) provides fewer details, so if you want only a single number, run `smem -P named` and look for the USS column - this provides the information about memory used by the program sans the shared library. The PSS column adds shared libraries divided by the number of programs using those libraries, and RSS is the normal Resident Size.
|
|
|
|
|
|
```
|
|
|
$ smem -P named -a
|
... | ... | @@ -100,9 +112,9 @@ BIND 9 uses several external libraries - OpenSSL, libuv, libxml2, json-c and pos |
|
|
#### Memory fragmentation
|
|
|
There's quite a lot of churn in the memory allocations and deallocations on a busy server, and memory gets fragmented - the default Linux allocator isn't particularly good with the BIND 9 memory usage patterns. Using jemalloc is strongly recommended as it handles memory fragmentation much better and is also faster.
|
|
|
|
|
|
## Memory Profiling
|
|
|
## Memory Profiling with jemalloc
|
|
|
|
|
|
When compiled (or even linked using `LD_PRELOAD`), `jemalloc` can produce **heap** snapshots based on triggers (time, size, ...). This can be later analysed using the `jeprof` tool to see where the memory went.
|
|
|
When compiled (or even linked using `LD_PRELOAD`), `jemalloc` [can produce](https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling) **heap** snapshots based on triggers (time, size, ...). This can be later analyzed using the [`jeprof` tool](https://manpages.debian.org/unstable/libjemalloc-dev/jeprof.1.en.html) to see where the memory went.
|
|
|
|
|
|
The basics are:
|
|
|
|
... | ... | |