framework: do not double-execute an async job after returning its queue item - #14039
framework: do not double-execute an async job after returning its queue item#14039nagaboinaramgopal wants to merge 1 commit into
Conversation
c397396 to
9259d8b
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.20 #14039 +/- ##
============================================
+ Coverage 16.34% 19.78% +3.44%
- Complexity 13574 19993 +6419
============================================
Files 5669 6371 +702
Lines 501368 575900 +74532
Branches 60903 70495 +9592
============================================
+ Hits 81964 113970 +32006
- Misses 410219 449500 +39281
- Partials 9185 12430 +3245
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ue item In executeQueueItem, when persisting the executing MS id fails (the DB-deadlock case the catch block exists for), the queue item is returned to the queue so it can be retried. Execution then fell through to scheduleExecution(job), so the job was dispatched now AND re-dequeued and dispatched again by the heartbeat, running the same job (VM start/deploy, volume create, snapshot, etc.) twice concurrently and defeating the sync queue's serialization. Return after returning the item.
9259d8b to
8c06db5
Compare
|
@blueorangutan package |
|
@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19093 |
|
@nagaboinaramgopal the method you are touching has two consecutive try blocks, both of which have nested try block. If you have time, do you feel like cleaning up the code? (does not need to be in the scope of this PR) |
|
@blueorangutan test |
|
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
3 participants
Description
In executeQueueItem, when persisting the executing management-server id fails
(the DB-deadlock case the catch block exists for), the queue item is returned to
the queue so it can be retried later. Execution then fell through to
scheduleExecution(job), so the job was dispatched now AND re-dequeued and
dispatched again by the heartbeat, running the same job (VM start/deploy, volume
create, snapshot, etc.) twice concurrently and defeating the sync queue's
serialization.
Return after returning the item so the job runs once, on the retry.
Types of changes
Feature/Enhancement Scale or Bug Severity
Bug Severity
How Has This Been Tested?
Added a unit test that makes the executing-msid update throw and asserts the
queue item is returned and the job is not also scheduled. Also built the standard
packages and deployed on a KVM advanced zone.