Make Cassandra connection parameters configurable [ISC-support #13376]
The request from the customer:
I have been spending some time setting Kea up with Cassandra across multiple datacenters, and I believe there is a configuration issue with the current CQL connection manager. It forces consistency to CASS_CONSISTENCY_QUORUM (and I believe it does so for both reads and writes).
Cassandra considers a "cluster" to include all servers across all datacenters. In a deployment with 3 servers in Site1, 3 servers in Site2, and a replication factor of 3 at each site (each node holding a copy of all data), a quorum is 4 servers. This means that any read or write to Site1 must cross the country from Site1 to at least one node in Site2. This ensures strong consistency but doesn't work well with multi-datacenter latency. Worse yet, it means that a datacenter-wide failure of Site2 creates a failure of Site1 because a quorum can't be achieved, as only 3 servers would be available.
I think CASS_CONSISTENCY_LOCAL_QUORUM is probably a better default- it still requires a quorum but is aware of the Cassandra network topology when calculating the servers required for a quorum. In this case, a write to Site1 would require 2 of the 3 servers at that site. This also works with Cassandra's 'SimpleStrategy' if multiple data centers aren't being used- and it would be equivalent to CASS_CONSISTENCY_QUORUM in SimpleStrategy. Replication across datacenters would be eventually consistent, but there would be strong consistency within a datacenter.
Ideally, the consistency for reads and writes would be independently configurable via the backend configuration. This would let the user assess the risk of conflicts and pick the consistency that makes the most sense for their deployment. https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlConfigConsistency.html