Shutdown expunged resources cleanup executor properly, and allow other components to configure/start/stop on error by sureshanaparti · Pull Request #9723 · apache/cloudstack

sureshanaparti · 2024-09-23T14:58:06Z

Description

This PR shutdowns expunged resources cleanup executor when obj is available (when config expunged.resources.purge.enabled is true), allows other components to configure/start/stop on error, and adds some logs in component lifecycle classes.

Noticed this exception with custom logs, the remaining components fails to stop after this exception.

WARN  [o.a.c.s.l.CloudStackExtendedLifeCycleStart] (SpringContextShutdownHook:null) (logid:) Error on stopping beans - null
java.lang.NullPointerException
        at org.apache.cloudstack.resource.ResourceCleanupServiceImpl.stop(ResourceCleanupServiceImpl.java:584)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$2.with(CloudStackExtendedLifeCycle.java:105)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:159)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.stopBeans(CloudStackExtendedLifeCycle.java:101)
        at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.stop(CloudStackExtendedLifeCycleStart.java:32)
        at org.apache.cloudstack.spring.lifecycle.AbstractSmartLifeCycle.stop(AbstractSmartLifeCycle.java:49)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:234)
        at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:54)
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:373)
        at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:206)
        at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:129)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1069)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:993)

Fixes #9722

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

Mgmt2 service stopped =>

How Has This Been Tested?

Manually tested management server start & stop.

2024-09-23 18:20:51,111 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterServiceServletAdapter
2024-09-23 18:20:51,111 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (SpringContextShutdownHook:[]) (logid:) stopping bean ClusterManagerImpl

[root@ol8 ~]# cat stopping-beans-check.txt | wc -l
665

stopping-beans-check.txt

How did you try to break this feature and the system with this change?

…n config expunged.resources.purge.enabled is true), and added some logs in component lifecycle classes

sureshanaparti · 2024-09-23T14:59:09Z

@blueorangutan package

blueorangutan · 2024-09-23T15:00:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-09-23T15:06:12Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 4.48%. Comparing base (b1f683d) to head (a6eb0b0).
Report is 44 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (b1f683d) and HEAD (a6eb0b0). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (b1f683d) HEAD (a6eb0b0)

unittests 1 0

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #9723       +/-   ##
============================================
- Coverage     15.77%   4.48%   -11.30%     
============================================
  Files          5621     392     -5229     
  Lines        491564   32154   -459410     
  Branches      61174    5672    -55502     
============================================
- Hits          77562    1441    -76121     
+ Misses       405545   30707   -374838     
+ Partials       8457       6     -8451

Flag	Coverage Δ
uitests	`4.48% <ø> (+0.43%)`	⬆️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-09-23T16:06:57Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11181

sureshanaparti · 2024-09-23T16:12:30Z

@blueorangutan test

blueorangutan · 2024-09-23T16:14:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti · 2024-09-23T19:05:40Z

@blueorangutan package

sureshanaparti · 2024-09-23T19:10:21Z

@blueorangutan package

blueorangutan · 2024-09-23T19:12:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-23T20:13:02Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11182

DaanHoogland · 2024-09-24T06:41:48Z

@sureshanaparti , this looks like a good cleanup. I wonder what it fixes other than just the looks of the code, though. We have two shutdown issues:

prolonged time of shutdown
no status update for shutdown MSses
does this address voth, @sureshanaparti ? (I can see you showed some evidence for the second)

sureshanaparti · 2024-09-24T09:07:05Z

@sureshanaparti , this looks like a good cleanup. I wonder what it fixes other than just the looks of the code, though. We have two shutdown issues:

prolonged time of shutdown

no status update for shutdown MSses
does this address voth, @sureshanaparti ? (I can see you showed some evidence for the second)

@DaanHoogland this updates MS status to Down when service is stopped/shutdown. (it doesn't address prolonged time of shutdown)

DaanHoogland

clgtm

blueorangutan · 2024-09-24T10:05:31Z

[SF] Trillian test result (tid-11539)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 62071 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9723-t11539-kvm-ol8.zip
Smoke tests completed. 134 look OK, 2 have errors, 5 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_03_secured_to_nonsecured_vm_migration	`Error`	375.65	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	282.02	test_vm_life_cycle.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_webhook_delivery	`Skipped`	---	test_webhook_delivery.py
all_test_webhook_lifecycle	`Skipped`	---	test_webhook_lifecycle.py

JoaoJandre

CLGTM, did not test it

DaanHoogland · 2024-09-25T15:02:04Z

@blueorangutan test keepEnv

blueorangutan · 2024-09-25T15:04:05Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-09-27T02:36:52Z

[SF] Trillian test result (tid-11551)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 125751 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9723-t11551-kvm-ol8.zip
Smoke tests completed. 133 look OK, 5 have errors, 3 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_01_secure_vm_migration	`Error`	135.10	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	135.11	test_vm_life_cycle.py
test_12_start_vm_multiple_volumes_allocated	`Error`	1109.52	test_vm_life_cycle.py
test_12_start_vm_multiple_volumes_allocated	`Error`	1109.53	test_vm_life_cycle.py
test_13_destroy_and_expunge_vm	`Error`	32.81	test_vm_life_cycle.py
test_14_destroy_vm_delete_protection	`Error`	38.62	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:teardown	`Error`	81.55	test_vm_life_cycle.py
ContextSuite context=TestCreateVolume>:setup	`Error`	0.00	test_volumes.py
ContextSuite context=TestVolumeEncryption>:setup	`Error`	0.00	test_volumes.py
ContextSuite context=TestVolumes>:setup	`Error`	0.00	test_volumes.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	41204.43	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Error`	50.88	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Error`	172.41	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	67.11	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	218.98	test_vpc_redundant.py
ContextSuite context=TestVPCRedundancy>:teardown	`Error`	326.28	test_vpc_redundant.py
all_test_vm_strict_host_tags	`Skipped`	---	test_vm_strict_host_tags.py
all_test_vnf_templates	`Skipped`	---	test_vnf_templates.py
all_test_vpc_ipv6	`Skipped`	---	test_vpc_ipv6.py

blueorangutan · 2024-09-27T06:05:49Z

[SF] Trillian test result (tid-11554)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 59928 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9723-t11554-kvm-ol8.zip
Smoke tests completed. 122 look OK, 1 have errors, 18 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
ContextSuite context=TestNatRuleUsage>:setup	`Error`	973.44	test_usage.py
ContextSuite context=TestPublicIPUsage>:setup	`Error`	1990.90	test_usage.py
ContextSuite context=TestSnapshotUsage>:setup	`Error`	2414.42	test_usage.py
ContextSuite context=TestTemplateUsage>:setup	`Error`	2540.51	test_usage.py
ContextSuite context=TestVmUsage>:setup	`Error`	2627.83	test_usage.py
ContextSuite context=TestVolumeUsage>:setup	`Error`	2792.50	test_usage.py
ContextSuite context=TestVpnUsage>:setup	`Error`	2859.77	test_usage.py
all_test_vm_autoscaling	`Skipped`	---	test_vm_autoscaling.py
all_test_vm_deployment_planner	`Skipped`	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	`Skipped`	---	test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import	`Skipped`	---	test_vm_lifecycle_unmanage_import.py
all_test_vm_schedule	`Skipped`	---	test_vm_schedule.py
all_test_vm_snapshot_kvm	`Skipped`	---	test_vm_snapshot_kvm.py
all_test_vm_snapshots	`Skipped`	---	test_vm_snapshots.py
all_test_vm_strict_host_tags	`Skipped`	---	test_vm_strict_host_tags.py
all_test_vnf_templates	`Skipped`	---	test_vnf_templates.py
all_test_volumes	`Skipped`	---	test_volumes.py
all_test_vpc_ipv6	`Skipped`	---	test_vpc_ipv6.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_webhook_delivery	`Skipped`	---	test_webhook_delivery.py
all_test_webhook_lifecycle	`Skipped`	---	test_webhook_lifecycle.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

sureshanaparti · 2024-09-27T07:16:30Z

@blueorangutan package

blueorangutan · 2024-09-27T07:18:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-27T08:19:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11214

sureshanaparti · 2024-09-27T08:32:04Z

@blueorangutan test

blueorangutan · 2024-09-27T08:34:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

DaanHoogland · 2024-09-27T09:30:17Z

Tested. Both the status update and the prolonged shutdown time have been fixed by this.

blueorangutan · 2024-09-28T03:44:29Z

[SF] Trillian test result (tid-11558)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 66748 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9723-t11558-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_01_migrate_VM_and_root_volume	`Error`	136.04	test_vm_life_cycle.py

yadvr · 2024-09-28T07:48:47Z

cc @JoaoJandre - this has been tested, should we merge this?

…r components to configure/start/stop on error (apache#9723)

sureshanaparti added 2 commits September 23, 2024 18:37

Shutdown expunged resources cleanup executor when obj is available (o…

cccf010

…n config expunged.resources.purge.enabled is true), and added some logs in component lifecycle classes

continue with other components on error

583d0dd

sureshanaparti requested review from JoaoJandre, shwstppr and vishesh92 September 23, 2024 14:58

sureshanaparti added this to the 4.20.0.0 milestone Sep 23, 2024

sureshanaparti requested a review from DaanHoogland September 23, 2024 15:01

JoaoJandre reviewed Sep 23, 2024

View reviewed changes

review comments

a6eb0b0

sureshanaparti requested a review from JoaoJandre September 23, 2024 19:06

DaanHoogland approved these changes Sep 24, 2024

View reviewed changes

DaanHoogland added the Severity:Critical Critical bug label Sep 24, 2024

JoaoJandre approved these changes Sep 24, 2024

View reviewed changes

yadvr approved these changes Sep 24, 2024

View reviewed changes

DaanHoogland merged commit c159471 into apache:main Sep 30, 2024

DaanHoogland deleted the ms-state-update-issue-during-service-stop branch September 30, 2024 09:09

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Oct 14, 2024

Shutdown expunged resources cleanup executor properly, and allow othe…

9747aa1

…r components to configure/start/stop on error (apache#9723)

Uh oh!

Conversation

sureshanaparti commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Sep 23, 2024

Uh oh!

blueorangutan commented Sep 23, 2024

Uh oh!

codecov Bot commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Sep 23, 2024

Uh oh!

sureshanaparti commented Sep 23, 2024

Uh oh!

blueorangutan commented Sep 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Sep 23, 2024

Uh oh!

sureshanaparti commented Sep 23, 2024

Uh oh!

blueorangutan commented Sep 23, 2024

Uh oh!

blueorangutan commented Sep 23, 2024

Uh oh!

DaanHoogland commented Sep 24, 2024

Uh oh!

sureshanaparti commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Sep 24, 2024

Uh oh!

JoaoJandre left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Sep 25, 2024

Uh oh!

blueorangutan commented Sep 25, 2024

Uh oh!

blueorangutan commented Sep 27, 2024

Uh oh!

blueorangutan commented Sep 27, 2024

Uh oh!

sureshanaparti commented Sep 27, 2024

Uh oh!

blueorangutan commented Sep 27, 2024

Uh oh!

blueorangutan commented Sep 27, 2024

Uh oh!

sureshanaparti commented Sep 27, 2024

Uh oh!

blueorangutan commented Sep 27, 2024

Uh oh!

DaanHoogland commented Sep 27, 2024

Uh oh!

blueorangutan commented Sep 28, 2024

Uh oh!

yadvr commented Sep 28, 2024

Uh oh!

Reviewers

sureshanaparti commented Sep 23, 2024 •

edited

Loading

codecov Bot commented Sep 23, 2024 •

edited

Loading

sureshanaparti commented Sep 24, 2024 •

edited

Loading