Monitoring Connectors and Ingestion

The pages under the Monitoring link display information about connectors and ingestion. In these pages, you can manually trigger ingestion, monitor the status of ingestion, or view diagnostics about ingestion.

Ingestion Trigger

Lucidum ingests data from all connectors every 24 hours. To ingest all data from all connectors, on demand:

1. Navigate to the Lucidum UI.

2. In the left pane, click Monitoring.

3. Select Ingestion Trigger

4. In the Ingestion Trigger page, click the Ingest All Data button.

Notice that the Ingestion Trigger page also displays:

• Task_ID. The Lucidum task. Usually run_<connector name>.

• Status. Status of ingestion from the connector. Sucess or failure.

• Start Date. Date ingestion started.

• End Date. Date ingested ended.

• Duration. Number of seconds for ingestion

Ingestion Status

After you have configured connectors, Lucidum can start ingesting data and displaying information about your environment. By default, Lucidum ingests data every 24 hours. After 24 hours, information appears on the Ingestion Status page.

If you want to start ingesting data immediately, you can go to Monitoring > Ingestion Trigger and click Ingest All Data to immediately start ingesting data.

The Ingestion Status page is a convenient way to see what data Lucidum has ingested.

To view the Ingestion Status page:

1. Navigate to the Lucidum UI.

2. In the left pane, click Monitoring.

3. Select Ingestion Status.

The Data Ingestion Flow chart shows all the sources of data and if the last ingestion by Lucidum was successful or timed out.

Mousing over the solid circle next to a data source displays information about the last time Lucidum ingested data from that data source.

Ingestion Metrics

The Ingestion Metrics page displays technical details on connectors and ingestion.

To view the Ingestion Status page:

1. Navigate to the Lucidum UI.

2. In the left pane, click Monitoring.

3. Select Ingestion Metrics.

The Ingestion Metrics page displays:

• Platform. Name of the platform vendor from which Lucidum ingests data. For example, “aws” or “api”.

• Data Source. Name of the connector and if applicable, the specific service within a connector. For example, “aws_iam” or “oci_storage”.

• Ingestion Duration (minutes). Total time ingestion took, in minutes.

• Output Records. Number of records generated or updated from this instance of ingestion.

• Run time. Date the ingestion occurred.

• Details. Detailed output of ingestion, in JSON format.