This is the Insights Advisor Backend repository. It hosts the Advisor API, the
Advisor Service, and the inventory event consumer. This is the top-level README
for those processes. To see more detailed information about the API or Service,
look at their respective READMEs in the api or service directories.
Advisor grew out of the original Insights project, which provided a system for hosts to upload a 'snapshot' of their state as it related to Red Hat components, and for reports of Insights rules to be recorded and shown for that system.
Advisor has three main processes. The 'service' receives Kafka messages with
the results of processing Insights rules on a system and records them in the
database. The inventory event consumer stores HBI host lifecycle events in
Advisor's database. The 'API' displays this data to the user. The service is
in service/; the API and inventory event consumer are in api/.
Advisor Backend is designed to run in containers, with the other parts of the Red Hat Lightspeed (Insights) system, in an OpenShift cluster. This is managed inside Red Hat using https://github.com/RedHatInsights/clowder.
It's possible to run just the basic components Advisor needs using Podman, which is how we suggest you do this in development.
The projects under https://github.com/RedHatInsights/ are related to the Insights project.
python 3.12+
pipenv
podman and podman-compose
Setup the Python environment:
pipenv shell
pipenv install
pipenv install --dev
To start the database and Kafka, run required migrations and load required fixtures you will need to run the following commands:
export ADVISOR_DB_HOST=localhost
podman-compose up -d advisor-db init-kafka kafka
./container_init_localdev.sh
Feature flags are supported by Unleash. Connection to an Unleash server is done by a Clowder provided config.
Local development can be done by supplying bootstrapped feature flags via
UNLEASH_BOOTSTRAP_FILE environment variable. UNLEASH_BOOTSTRAP_FILE should
point to a JSON file conforming to Unleash's API
/api/client/features.
The Insights client collects data from a system, and the Insights core is used to process this data and check it against the Insights production ruleset. This generates reports which are sent through Kafka to the Advisor service.
The Advisor service organises these into an upload, which contains zero or more reports. Each upload relates to a host by its Inventory UUID; each report therefore links to the host (via the upload) and the rule, and stores the details that the rule generated (which is basically a JSON object).
The API provides access to the content for the rules, and the reports of
them occurring on systems. Rules have categories, impact levels, and tags,
as well as content fields that use the DoT JavaScript templating language
and MarkDown to render the data actually captured by the rule into meaningful
information to display to the user.
Systems normally upload each day, and there are several types of upload -
subscription-manager and insights-client are the two most common. If a
system hasn't uploaded in fourteen days, it is considered stale, and after
twenty-one days it is hidden from display. After twenty-eight days HBI
deletes the system - this prevents 'ephemeral' systems
which are brought up, run for a limited time, and then shut down (without
sending a 'delete' notification) from cluttering up the database.
Advisor consumes host creation, update, and deletion events from Inventory and
stores the required host data in its local AdvisorInventoryHost table.
The advisor_inventory_service management command
(api/advisor/api/management/commands/advisor_inventory_service.py) runs a
Kafka consumer that stores host data from HBI in Advisor's own
AdvisorInventoryHost and Host tables. It consumes messages from the
platform.inventory.events topic in configurable batches controlled by the
INVENTORY_BATCH_SIZE setting.
Messages are consumed in batches and dispatched to handle_inventory_event(), which:
- Classifies each message by its
typefield (created,updated, ordelete). - Parses valid messages into typed value objects (
ParsedInventoryHostorParsedDeleteEvent). - Deduplicates upserts by
(org_id, host_id)within the batch — when multiple events target the same host, only the last one is applied. - Executes bulk DB operations:
bulk_create(update_conflicts=True)for upserts, and a combinedQfilter for deletes.
The consumer handles three event types. Not all fields in the HBI message are used by Advisor; required fields are marked below.
{
"type": "created", // required: "created" or "updated"
"timestamp": "<timestamp>",
"platform_metadata": "<metadata_json_doc>",
"metadata": { // required
"request_id": "<request_id>" // required
},
"host": { // required
"id": "<uuid>", // required
"account": "<account_number>", // optional
"org_id": "<org_id>", // required
"display_name": "<display_name>", // required
"ansible_host": "<ansible_host>",
"fqdn": "<fqdn>",
"insights_id": "<insights_id>", // required (must be non-empty)
"subscription_manager_id": "<sub_mgr_id>",
"satellite_id": "<satellite_id>", // optional
"bios_uuid": "<bios_uuid>",
"ip_addresses": ["<ip_addresses>"],
"mac_addresses": ["<mac_addresses>"],
"facts": ["<facts>"],
"provider_id": "<provider_id>",
"provider_type": "<provider_type>",
"created": "<created_date>", // required
"updated": "<updated_date>", // required
"last_check_in": "<last_check_in>", // required
"stale_timestamp": "<stale_timestamp>", // required
"stale_warning_timestamp": "<stale_warning_timestamp>",
"culled_timestamp": "<culled_timestamp>",
"reporter": "<reporter>", // required
"tags": [<tags>], // required
"system_profile": { // required
"host_type": "<host_type>",
"operating_system": {
"name": "<os_name>",
"major": <os_major>,
"minor": <os_minor>
},
"bootc_status": {
"booted": {
"image": "<image>",
"image_digest": "<digest>"
}
},
"owner_id": "<owner_uuid>",
"rhc_client_id": "<rhc_uuid>",
"workloads": {<workloads>},
"system_update_method": "<method>"
},
"per_reporter_staleness": { // required
"<reporter>": {
"stale_timestamp": "<timestamp>",
"stale_warning_timestamp": "<timestamp>",
"culled_timestamp": "<timestamp>"
}
},
"groups": [{ // required
"id": "<group_uuid>",
"name": "<group_name>"
}],
"openshift_cluster_id": "<openshift_cluster_id>"
}
}Fields within system_profile are all optional — the consumer handles missing or
empty values gracefully. Only the first entry in groups is used to populate
workspace_id and workspace_name.
{
"type": "delete", // required
"id": "<uuid>", // required — the inventory host ID
"timestamp": "<timestamp>",
"account": "<account>", // optional
"org_id": "<org_id>", // required
"insights_id": "<insights_id>",
"request_id": "<request_id>" // required
}Delete events remove the matching AdvisorInventoryHost and Host records,
along with dependent Upload, CurrentReport, and HostAck records, in one
database transaction.
There are four main types of data stored in Advisor.
Some data is more or less fixed, and this is loaded from fixtures; this covers
the Ruleset, RuleCategory, SystemType and UploadSource models. In
production and stage environments the Pathway model data is also loaded
from a fixture.
The content in the Rule model and its associated Tag, RuleImpact,
Resolution, ResolutionRisk and Playbook models is loaded from data
written by the rule content team. This data presents information about the
rules and how they affect a specific system - data from the report is
interpolated into some of these fields.
This data is loaded by the import_content Django command. In the stage and
production environments this command is run in the container_init.sh
script during the container environment initialisation process.
The command takes a -c option that is given a directory path that contains
the rule and playbook content, either in its direct form of the actual
insights-content and insights-playbooks Git repositories, or as the
dumped YAML form of those repositories (see below).
Red Hat's content is its own intellectual property and the insights-content
and insights-playbooks repositories are not available externally. For an
example of what these can look like, the api/test_content/ directory has
the content and playbooks directories which match these repository
structures and the data contained within the basic_test_data fixture.
The import process is designed to quickly load this data into the data, using bulk insert and update operations where possible.
During the container build process, the import_content command is invoked
(in the Dockerfile) using the --dump and --compress options. This reads
the Git content and playbook repositories and then writes these out to two
YAML files (compressed using zlib to be gzip compatible). This, plus the
content repository's config.yaml file, get written into the container image.
System data is primarily stored in the AdvisorInventoryHost model by the
inventory event consumer described above. We also use a Host model to keep
track of Advisor-specific data, such as Satellite IDs, and as the foreign key
target for uploads and reports.
Each time a system runs insights-client we store each individual result in
the CurrentReport model. Zero or more reports are grouped together into an
Upload object; this allows us to track that a report on one day does not
appear in a following upload (which means the rule has been resolved on that
system).
Data in the Ack and HostAck models tracks if a user does not want to see
particular recommendations, either for all hosts (Ack) or only for specific
hosts (HostAck).
Users can leave ratings for rules in the RuleRating model - positive,
negative, or neutral.
The WeeklyReportSubscription model tracks users subscribing to receive
weekly reports.
The RuleTopic model allows us to group rules together - for example, rules
related to managing Postgres on a Satellite. There are only a limited number
of these. They were created online using the API, and there is no fixture
At one stage product management decided that for all accounts, any new
person that was added to the account would automatically be subscribed to the
weekly report. However, it was decided that aproximately 310 accounts would
not have this enabled, so the SubscriptionExcludedAccount model tracks
those accounts. The API endpoints that allow the UI to automatically create
a subscription will return a 405 Method Not Allowed when the UI attempts
to create a user in one of these accounts.
The following environment variables can be used to configure the Advisor API. Most have sensible defaults for local development.
DJANGO_SESSION_KEY- for security reasons this is not exposed in the codebase. It is generated by Clowder and stored in theadvisor-backend/.envfile.ENV_NAME- Default:localADVISOR_ENV- Default:dev- These should probably be combined.
ALLOWED_HOSTS- Django's list of which hosts we will serve.- Default:
testserver,insights-advisor-api,advisor-api,localhost,127.0.0.1,.insights.openshiftapps.com,.svc.cluster.local
- Default:
These control how what URL paths we accept.
PATH_PREFIX- Default:api/PLATFORM_BASE_PATH- Default:r/insights/platform/ADVISOR_PATH_NAME- Default:insightsTASKS_PATH_NAME- Default:tasks
ADVISOR_DB_HOST- Default: `` (empty, allows local socket connection)ADVISOR_DB_PORT_NUM- Default: `` (empty)ADVISOR_DB_NAME- Default:insightsapiADVISOR_DB_USER- Default:insightsapiADVISOR_DB_PASSWORD- Default:InsightsDataADVISOR_DB_ENGINE- Default:django_prometheus.db.backends.postgresqlADVISOR_DB_READONLY_HOST- If set, this allows use of a read-only replica of the database.ADVISOR_DB_SSL_MODE- added optionally if presentADVISOR_DB_SSL_CA- added optionally if presentADVISOR_INVENTORY_HOST_NUM_PARTITIONS- Default:1. Sets the number of hash partitions created foradvisor_inventory_hostby its initial migration and used by the partitioned-index helper. Set it before applying that migration.
Role-Based Access Control is how we manage authentication and permissions in Advisor. Most requests need to be checked against an external RBAC server. For testing RBAC is normally not enabled - this bypasses RBAC checks and allows access.
RBAC_ENABLED- Default:falseRBAC_URL- The URL of the RBAC server.RBAC_PSK- The pre-shared key used to authenticate with the RBAC server.RBAC_CLIENT_ID- Default:advisor
Kessel is the next generaton of this RBAC system, providing more flexibility in how organisations manage their own users and what systems and groups those users can see. For testing this is normally disabled as well.
KESSEL_ENABLED- Default:falseKESSEL_URL- Default:device under testKESSEL_AUTH_ENABLED- Default:falseKESSEL_AUTH_CLIENT_ID- Default: `` (empty)KESSEL_AUTH_CLIENT_SECRET- Default: `` (empty)KESSEL_AUTH_OIDC_ISSUER- Default: `` (empty)KESSEL_INSECURE- Default:trueKESSEL_TIMEOUT- Default:10- deadline in seconds for each Kessel gRPC call.
CLOWDER_ENABLED- Default:false- this mainly controls how logs are generated and formatted.INVENTORY_SERVER_URL- The URL of the Inventory service.PLAYBOOK_DISPATCHER_URL- The URL of the Playbook Dispatcher service.SOURCES_API_URL- The URL of the Sources API service.REMEDIATIONS_URL- The URL of the Remediations service.MIDDLEWARE_HOST_URL- The URL of the Back-office proxyMIDDLEWARE_CERT_FILE- Default:./api/certs/backoffice-proxy.ca.crtMIDDLEWARE_API_TOKEN- Default:PROXY_TOKENMIDDLEWARE_CLIENT_ID- Default:insights-advisor-mailerPLATFORM_HOSTNAME_URL- Default:cert.console.stage.redhat.comCLOUD_CONNECTOR_HOST- The host name of the Cloud Connector service.CLOUD_CONNECTOR_PORT- The port of the Cloud Connector service.
PDAPI_PSK- the pre-shared key used by the Tasks API to talk to the Playbook Dispatcher API.GROUP_ID- Default:tasks_callbackTASKS_BATCH_SIZE- Default:50TASKS_REWRITE_INTERNAL_URLS- Default:falseTASKS_REWRITE_INTERNAL_URLS_FOR- Default:internal.console.
UNLEASH_TOKEN- Default:*:*.advisorUNLEASH_URL- Default:http://localhost:4242/apiUNLEASH_APP_NAME- Default:insights-advisor-apiUNLEASH_CACHE_DIR- Default:/tmp/unleashcacheUNLEASH_REFRESH_INTERVAL- Default:5UNLEASH_FAKE_INITIALIZE- Default:trueUNLEASH_BOOTSTRAP_FILE- A JSON file with feature flags set, in the same format as the Unleash API returns. Defaults to no file.
PROMETHEUS_PATH- Default:metrics(or from Clowder config)PROMETHEUS_PORT- Default:8000(or from Clowder config)SPLUNK_HEC_TOKEN- Default: `` (empty)ENABLE_SPLUNK_HEC- Default:false
OPENSHIFT_BUILD_NAME- Default:devOPENSHIFT_BUILD_COMMIT- Default:devOPENSHIFT_BUILD_REFERENCE- Default: `` (empty)
MAIL_HOST- Default:mail.corp.redhat.comDEFAULT_FROM_EMAIL- Default:Red Hat Hybrid Cloud Console <noreply@redhat.com>
BOOTSTRAP_SERVERS- Needed for connecting to KafkaENABLE_KAFKA_SSL- Default:falseKAFKA_SSL_CERT- Default:/opt/certs/kafka-cacertKAFKA_SECURITY_PROTOCOL- Default:SASL_SSLKAFKA_SASL_MECHANISMS- Default:SCRAM-SHA-512KAFKA_SASL_USERNAME-KAFKA_SASL_PASSWORD-
These are the actual topics read and written to when we use Kafka
ENGINE_RESULTS_TOPIC- Default:platform.engine.resultsINVENTORY_EVENTS_TOPIC- Default:platform.inventory.eventsRULE_HITS_TOPIC- Default:platform.insights.rule-hitsPAYLOAD_TRACKER_TOPIC- probably should default toplatform.payload-statusREMEDIATIONS_HOOK_TOPIC- Default:platform.remediation-updates.advisorWEBHOOKS_TOPIC- probably should default toplatform.notifications.ingressTASKS_UPDATES_TOPIC- Default:platform.playbook-dispatcher.runsTASKS_SOURCES_TOPIC- Default:platform.sources.event-streamTASKS_UPLOAD_TOPIC- Default:platform.upload.announce
LOG_LEVEL- Default:INFODB_RETRY_CONSTANT- Default:3THREAD_POOL_SIZE- Default:30INVENTORY_BATCH_SIZE- Default:50. Number of HBI events handled in one batch by the inventory event consumer.DISABLE_PROMETHEUS- Default:falseFILTER_OUT_NON_RHEL- Default:trueFILTER_OUT_NON_RHEL_RULE_ID- Default:other_linux_system|OTHER_LINUX_SYSTEM,other_linux_system|OTHER_LINUX_SYSTEM_V2,other_linux_system|CONVERT2RHEL_SUPPORTEDFILTER_OUT_RHEL6- Default:trueFILTER_OUT_RHEL6_RULE_IDS- Default:rhel6_upgrade|RHEL6_HAS_TO_UPGRADE_WARN,rhel6_upgrade|RHEL6_HAS_TO_UPGRADE_WARN_V1,rhel6_upgrade|RHEL6_HAS_TO_UPGRADE_ERRORLOG_DB_QUERIES- Default:false
AWS_ACCESS_KEY_ID-AWS_SECRET_ACCESS_KEY-AWS_REGION_NAME-CW_LOG_GROUP- Default:platform-dev(non-Clowder)CW_LOG_STREAM-CW_CREATE_LOG_GROUP- Default:false
STATIC_URL- Default:/apps/insights/STATIC_ROOT- Default:/tmp/staticENABLE_AUTOSUB- Default:false
The stale timestamps of the hosts in the DB will need to be updated so they are in the futures. The timestamps in the fixtures are well into the past now and need to be updated in the DB so the hosts will show up in queries:
python api/advisor/manage.py freshen_hostsStart the service
export ADVISOR_DB_HOST=localhost
podman-compose up advisor-service
Sending in fake engine results
pipenv shell
python service/manual_test/send_fake_engine_results.py
python api/advisor/manage.py freshen_hosts
Start Service dependencies. We still use podman-compose here but only for the dependencies. This method is meant for more rapid development.
export ADVISOR_DB_HOST=localhost
podman-compose up -d advisor-db init-kafka kafka advisor-api
Start Service manually
BOOTSTRAP_SERVERS=localhost:9092 PROMETHEUS_PORT=8001 python service/service.py
Sending in engine results for processing.
python service/manual_test/send_fake_engine_results.py
python api/advisor/manage.py freshen_hosts
With the database and Kafka running, start the consumer in a separate shell:
BOOTSTRAP_SERVERS=localhost:9092 PROMETHEUS_PORT=8002 \
python api/advisor/manage.py advisor_inventory_serviceStart the API
export ADVISOR_DB_HOST=localhost
podman-compose up advisor-api
Verify the API is running
curl http://localhost:8000/api/insights/v1/status/ready
curl http://localhost:8000/api/insights/v1/status/live
You should see output corresponding to the request from podman as well as your curl command.
Start API dependencies. We still use podman-compose here but only for the dependencies. This method is meant for more rapid development.
Note: all the following commands assume you have activated the pipenv shell and run export ADVISOR_DB_HOST=localhost
Start the DB:
podman-compose up -d advisor-dbPopulate the DB (if this is the first time running).
./container_init_localdev.sh... or manually run the following commands (which are mostly in the container_init_localdev.sh script):
python api/advisor/manage.py migrate
python api/advisor/manage.py loaddata rulesets rule_categories system_types \
upload_sources basic_test_data basic_task_test_dataStart the API manually:
LOG_LEVEL=DEBUG api/advisor/manage.py runserver --insecure 0.0.0.0:8000The runserver --insecure flag is for serving static files even if DEBUG is False.
It should only be used for local development.
LOG_LEVEL=DEBUG api/advisor/manage.py runserver --noreload --insecure 0.0.0.0:8000NOTE: If you want to enable the Auto-Subscribe endpoint, define the
ENABLE_AUTOSUB environment variable to true before running the server.
ENABLE_AUTOSUB=true LOG_LEVEL=DEBUG api/advisor/manage.py runserver --insecure 0.0.0.0:8000Follow the Tasks README.md file under tasks folder.
The local APIs can be accessed via OpenAPI Swagger UIs:
- http://localhost:8000/api/insights/v1/openapi/swagger/
- http://localhost:8000/api/tasks/v1/schema/swagger-ui/
Because the OpenAPI schema is generated from the permissions of the user, you
will need to insert the x-rh-identity header in your browser to provide the
header that the 3Scales would normally provide. To do this:
- Install a header modifying plugin in your browser, e.g. "ModHeader" for FireFox.
- Set it up to add the
x-rh-identityheader with this value:eyJpZGVudGl0eSI6IHsiYWNjb3VudF9udW1iZXIiOiAiMTIzNDU2NyIsICJvcmdfaWQiOiAiOTg3NjU0MyIsICJ0eXBlIjogIlVzZXIiLCAiYXV0aF90eXBlIjogImp3dCIsICJ1c2VyIjogeyJ1c2VyX2lkIjogIjE2Nzc3MjE2IiwgInVzZXJuYW1lIjogInRlc3RpbmciLCAiaXNfaW50ZXJuYWwiOiB0cnVlfX19Cg==
You can generate a similar header using this command:
$ echo '{"identity": {"account_number": "1234567", "org_id": "9876543", "type": "User", "auth_type": "jwt", "user": {"user_id": "16777216", "username": "testing", "is_internal": true}}}' | base64 -w 0 ; echo
Note, you can decode the header using:
$ echo <x-rh-identity> | base64 -d
If you need to vary the data in the header structure, this is the way to do it.
Likewise, if using curl then the header can be provided in this way:
$ curl -H 'x-rh-identity: eyJpZGVudGl0eSI6IHsiYWNjb3VudF9udW1iZXIiOiAiMTIzNDU2NyIsICJvcmdfaWQiOiAiOTg3NjU0MyIsICJ0eXBlIjogIlVzZXIiLCAiYXV0aF90eXBlIjogImp3dCIsICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJ0ZXN0aW5nIiwgImlzX2ludGVybmFsIjogdHJ1ZX19fQo=' http://localhost:8000/api/insights/v1/rule/
To run lint tests run the following pipenv script. This will lint both the Service and the API
pipenv run linter
Before running the tests, make sure the database is running first:
export ADVISOR_DB_HOST=localhost
podman-compose up -d advisor-db init-kafka kafka
To run Service Tests
pipenv run testservice
To run API Tests
pipenv run testapi
To run Tasks API Tests
pipenv run testtasks
To monitor the API locally with Prometheus and Grafana, start both services:
podman-compose up -d prometheus grafana
Prometheus scrapes the API metrics endpoint and is available at http://localhost:9090.
Grafana is available at http://localhost:3000 (no login required). The Advisor Backend API dashboard is automatically provisioned and will appear under Dashboards.
The Prometheus datasource is pre-configured. You can also run Grafana without the local Prometheus container by port-forwarding from an OpenShift cluster, which is useful for testing dashboards against real production or stage data:
oc port-forward --address 0.0.0.0 <prometheus-pod> 9090:9090
podman-compose up -d grafana
The --address 0.0.0.0 flag is required so the Grafana container can reach the
forwarded port via host.containers.internal. If your forwarded port differs from
9090, edit grafana/provisioning/datasources/prometheus.yml and update the url
field, then restart the grafana container.
The following items would improve the README documentation:
- Add installation instructions for prerequisites (podman, podman-compose, specific versions)
- Document location and contents of podman-compose.yml file
- Create a complete first-time setup checklist (start to finish)
- Explain what Clowder is and how it's used
- Clarify the relationship between API and Service (when to run each, how they interact)
- Document external dependencies:
- Location of insights-content and insights-playbooks repositories
- Add production authentication/authorization flow explanation
- This is mainly done in the API readme.
- Explain send_fake_engine_results.py and other manual test scripts
- This is explained in the Service readme.
- Document typical development workflow (running API only, Service only, or both)
- Mostly covered; would like feedback on other people's development workflow
- Add complete port reference (database, services, etc.)
- Not sure this is necessary; ports are usually encoded in environment variables (described above).
- Add repository structure overview (what's in api/ vs service/ directories)
- Document where to find specific functionality
- Add troubleshooting section with common issues and solutions
- Not sure where to start on this; I don't think we have a 'common issues' topic that's specific enough to write on.
- Document CI/CD process (PR checks, deployment workflow)
- Beyond the scope of this document - internal processes are documented internally.
All outstanding issues or feature requests should be filed as Issues on this GitHub page. MRs should be submitted against the main branch for any new features or changes, and pass ALL testing above.
The Advisor Architecture Document gives some more information for Red Hat associates who need to maintain Advisor.
