Investigate cluster CPU spikes

[ejsmith]

Happening every 8 hours. Orphan data cleanup is the only thing that happens every 8 hours. Doesn't look like the job itself is taking a lot of CPU, but ES CPU spikes during that time. So my guess would be whatever we are doing in orphan data cleanup is very expensive in ES.

[ejsmith]

The job is searching for orphaned documents across all time. Since we are running this on a regular basis, we can probably restrict it to only check events within the last 3 days or so (forgot how far back we allow events to be post dated).

[ejsmith]

I guess what would be better is to find recently deleted stacks, projects, orgs and see if there is any matching events. Ideally we wouldn't have to worry about this happening. We need to bullet proof the process of deleting projects, stacks and orgs and make sure it's impossible for for them to be deleted if there are any matching events. We might have a concurrency issue where a stack is deleted but a new event comes in at the same time and gets added to that stack even though it was just deleted.

[niemyjski]

@ejsmith We only allow event submission for the past 3 days. Maybe we only run the full check once a month or once a week?

[ejsmith]

We're closing this as part of an open issue cleanup. This looks like a historical production/cluster investigation from 2020 rather than a reproducible current product bug.

If the same 8-hour CPU spike is visible in current telemetry, please open a new issue with recent metrics, traces, and the affected deployment details so we can investigate from current evidence.