IGNITE-28952 [ducktests] MDC tests for majority based topology validation - #13576
Open
maksaska wants to merge 5 commits into
Open
IGNITE-28952 [ducktests] MDC tests for majority based topology validation#13576maksaska wants to merge 5 commits into
maksaska wants to merge 5 commits into
Conversation
The MBean name of a metric registry was spelled out at every call site, and
every one of them read an attribute as next(mbean.Attr).strip(). Both are now
in one place:
* metric_registry_pattern(group, name) builds the name pattern, honouring the
two rules the JMX exporter follows - the registry name is split at its FIRST
dot, and the tail is quoted only when it is not purely alphanumeric. The
pattern the rebalance metrics used, '.*group=cacheGroups.*name="%s"',
required the quotes unconditionally and so could not find the registry of an
alphanumeric cache group name at all.
* The pattern ends at the end of the line. 'name' sorts last of an MBean
name's properties, so without the anchor a registry whose name is a prefix
of another one ("myCache" against "myCacheV2") matched the wrong bean.
* JmxMBean.value()/bool_value() read a single valued attribute.
* cache_mbean()/cache_group_mbean() expose the two registries a test asks for.
Also documents why IgniteAwareService.await_rebalance() builds a JmxClient of
its own instead of the node's memoized one: the memoized client holds the pid of
the incarnation it was built for, and await_rebalance() runs right after a
restart.
check_jmx_utils.py covers the pattern, including both cases that used to be
wrong.
…data centers
MdcTopologyValidator has two modes and the number of data centers picks one:
with an EVEN DC count a segment stays writable while it sees the main DC, with
an ODD one while it sees a majority of the DC set. The fixture only ever built a
two DC cluster, so only the first mode could be tested.
MdcCluster now takes the DC set it spans (dcs=, two by default) and compiles the
cache parameters from it:
* mdc_topology_params() emits mainDc or datacenters, never both -
MdcTopologyValidator.checkConfiguration() rejects the pair;
* min_backups() is the smallest backup count giving every DC one copy of every
partition, and is what generate_data() uses by default;
* _with_cache_params() is the single point an application that creates the
cache is handed all of it, so no call site can configure a cache that
disagrees with the DC set. That is what lets the transactional test drop its
hand written mainDc.
verify_half_ring_healthy()/verify_split_brain() generalize into
verify_segment_healthy()/verify_segments(): a segment is now a DC or a group of
DCs that still see each other, so a three DC cluster with one DC cut off is
expressed as verify_segments((DC_1, DC_2), DC_3).
NetworkGroupManager gains enable/disable_network_partitions(*pairs). A cluster of
three or more groups is cut apart along several links at once, and every chain a
node takes part in has to be installed by the same single SSH round-trip: rolling
the links out one after the other would present the cluster with intermediate
segmentations it would legitimately react to.
The cache level MdcTopologyValidator becomes optional, through the cache
parameter topologyValidator and the mdc_cache_topology_validator global, for a
fork whose validator is configured elsewhere. The affinity backup filter moves
into a protected MdcCacheAwareApplication.backupFilter() for the same reason: a
fork that spreads the copies by something finer than the data center overrides
one method rather than repeating the cache configuration.
check_mdc_cluster.py and check_partition.py cover all of the above without a
cluster.
…ed validation
Three scenarios over a three DC cluster, each asserting something two DC mode
cannot show:
* test_minority_dc_isolation - one DC is cut off. The two DCs left keep
writing, the isolated one goes read-only while still serving every read.
Parametrized over the isolated DC, because no DC is privileged here: cutting
off DC1 - the main DC of the equivalent two DC cluster, and the DC the others
discovered through - is just as survivable as cutting off DC3.
* test_three_way_split_blocks_all_writes - every cross-DC link drops at once.
No segment holds a majority, so unlike the two DC case, where one half always
survives as writable, the whole cluster goes read-only.
* test_writes_survive_single_dc_loss - the same guarantee without any network
impairment: losing one DC leaves a majority, losing a second one does not.
Every partition owns exactly one copy per DC (backups = 2), so every segment
down to a single isolated DC still serves every read - which is what separates
the read assertions from the write ones throughout.
test_minority_dc_isolation also reads the cluster's own verdict on the MDC
guarantees off a node over JMX. IsCacheAffinityConfigurationMdcSafe is about the
cache CONFIGURATION and holds in every segment; IsCachePartitionDistributionSafe
is about the current assignment, and is only reported while the cluster is split
until the expected value is confirmed by a run.
maksaska
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
The Contribution Checklist
The description explains WHAT and WHY was made instead of HOW.
The following pattern must be used:
IGNITE-XXXX Change summarywhereXXXX- number of JIRA issue.(see the Maintainers list)
the
green visaattached to the JIRA ticket (see tabPR Checkat TC.Bot - Instance 1 or TC.Bot - Instance 2)Notes
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.