Show Run Details

Show Run Details is the displays logging data for each time the Flow was run.

View Log Data

To view the Flow log data:

  1. From the Flow overview page, click the gear icon in the Settings column next to the a currently running Flow.
    Click Show run details.

  2. Hero Platform_ brings you to a new log page for the selected Flow.

The default retention for Flow execution history is 30 days.

The retention period can be modified by opening the hero_flow_default.conf file and configuring the value for history-storage.retention.


The following data is available for each Flow run:

  • Run ID
  • Number of input records
  • Number of output records
  • Number of rejected fields
  • Start timestamp
  • End timestamp
  • Flow runtime
  • User
  • Flow version

View or Download Flow Logs

View or download a .zip file that contains a .log file of the Flow run details.

To view a Flow log:

  1. Navigate to the Show run details page.
  2. Click the kabab menu icon on the right side of the Flow run row. 
  3. Click View logs.

To download a Flow log:

  1. Navigate to the Show run details page.
  2. Click the kabab menu icon on the right side of the Flow run row. 
  3. Click Download log file.

The log file is downloaded as a .zip file to your local file system.

After unzipping the downloaded file, the following information is available:

Job Information

  • Build details:
    • Timestamp
    • Build revision ID
    • Version number
    • Build number
  • Flow name
  • Flow ID
  • Schedule ID
  • Execution ID

Example

Build details: BuildVersionUtils.HeroFlowVersion(_builtTimeStamp=2022-07-11T15:26:32.419+0200, _buildRevision=18d1bf4d1035c510ec4d496ba4a9be97b5541799, _version=6, _buildNumber=2022.8.48, _lastAppliedUpgradeNumber=1)

Flow name:     'Flow Demo'
Flow ID:       87d41b9a-02a6-4186-b896-e177c6be523d
Schedule ID:   48xk72nNkQ9CQTYPN18yWz
Execution ID:  6hz919pxUQ40Qks3BooXYG

Execution Plan

  • Input names/IDs
    • Operations
  • Node names/IDs
    • Operations
  • Output names/IDs
    • Operations


Example

------------------------------
Execution plan with 1 segment(s)
	Segment 0: dependsOn=[], intermediateInputs=0, intermediateOutputs=0
		Node: 'User Data' (c6bec2ce-234a-42f4-a3b1-16b8282d4730)
			Operation: ImmutableExternalTupleInput (id=51hDwQi1LTLeiMEFuxDvtG)
			Operation: ImmutableTupleMetrics (id=7UDulWShUdWPscDqa7so5u)
		Node: 'Upper' (43abcd22-3877-4b6b-864d-cadd41a72646)
			Operation: MapTupleStream(identity) (id=6ml3uld93KYEd9EK0oykYf)
			Operation: ImmutableTupleMetrics (id=4aC9PNaRwmNKoSlwBhdcDS)
			Operation: MapTupleStream(FunctionTypeMapOperator(ImmutableSimpleFunctionDefinition)) (id=6nj3u8XEc2YALkjGQOCLT)
			Operation: ImmutableTupleMetrics (id=5eFl2r2bykIP5CHs1mEuB4)
		Node: 'Auto Output 2022-07-12 08:57' (674b1d6d-0fb9-4515-9c18-70275a749daf)
			Operation: MapTupleStream(identity) (id=12eJn6yLJH9bnp2EKvYkL8)
			Operation: ImmutableTupleMetrics (id=1nBMazpZYjg0tA01NivnMb)
			Operation: FlatMapTupleStream(TupleStore) (id=7BWzDPuW4I5CqtBJ9S7nNy)
			Operation: ImmutableTupleMetrics (id=1THwhJxVuJcYLWE7CyG0nN)
			Operation: ImmutableTupleSink (id=4xPLBe0qdoWxaxCOyUQPl2)
------------------------------

Execution Metrics (JSON)

  • Timings:
    • Started at
    • Stopped at
    • Runtime
    • Runtime in ms
  • Status
  • Operators: (All nodes in the Flow including Inputs/functions/Outputs) 
    • Name
    • Node ID
    • Input count
    • External Input count
    • Output count
    • External Output count
    • Rejected count
    • Timings:
      • Runtime
      • Runtime in ms
      • Average runtime
      • Average runtime in ms


Example

{
  "timings": {
    "startedAt": "2022-07-12T09:06:35.237Z",
    "stoppedAt": "2022-07-12T09:06:37.292Z",
    "runtime": "00:02m",
    "runtimeMs": 2055
  },
  "status": "SUCCEEDED",
  "operators": [
    {
      "name": "Auto Output 2022-07-12 08:57",
      "nodeId": "674b1d6d-0fb9-4515-9c18-70275a749daf",
      "inputs": 14,
	  "externalInputs": 0,
      "outputs": 14,
	  "externalOutputs": 14,
      "rejected": 0,
      "timings": {
        "runtime": "\u003c 1 sec",
        "runtimeMs": 237,
        "averageRuntime": "\u003c 1 sec",
        "averageRuntimeMs": 15
      }
    },
    {
      "name": "User Data",
      "nodeId": "c6bec2ce-234a-42f4-a3b1-16b8282d4730",
      "inputs": 0,
      "externalInputs": 14,
      "outputs": 14,
      "externalOutputs": 0,
      "rejected": 0,
      "timings": {
        "runtime": "\u003c 1 sec",
        "runtimeMs": 11,
        "averageRuntime": "\u003c 1 sec",
        "averageRuntimeMs": 0
      }
    },
    {
      "name": "Upper",
      "nodeId": "43abcd22-3877-4b6b-864d-cadd41a72646",
      "inputs": 14,
      "externalInputs": 0,
      "outputs": 14,
      "externalOutputs": 0,
      "rejected": 0,
      "timings": {
        "runtime": "\u003c 1 sec",
        "runtimeMs": 0,
        "averageRuntime": "\u003c 1 sec",
        "averageRuntimeMs": 0
      }
    }
  ]
}

Execution Log File

Example

2022-07-12T09:06:35.253Z [ClusterSystem-akka.actor.default-dispatcher-18] INFO  a.h.p.c.f.FlowExecutionActor:213 - flowExecution-5: Starting FlowExecutionActor. flowId=87d41b9a-02a6-4186-b896-e177c6be523d, executionId=6hz919pxUQ40Qks3BooXYG, settings=ExecutionSettings{bleedCorruptedData=false, executionMode=STRICT, maxAttempts=3, maxParallelism=1}, recovering=NO, actorRef=Actor[akka://ClusterSystem/system/singletonManagerflowManagementActor/flowManagementActor/flowDeploymentActorV2-5/flowExecutionActorV2-5/flowExecution-5#1240496143]
2022-07-12T09:06:35.259Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.b.a.SequenceNumberPersistentActor:55 - flowExecution-5: Recovery is turned off, we delete all snapshots
2022-07-12T09:06:35.260Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.b.a.SequenceNumberPersistentActor:35 - flowExecution-5: Deleting all snapshots for persistenceId=flowExecution-87d41b9a-02a6-4186-b896-e177c6be523d
2022-07-12T09:06:35.260Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.p.c.f.FlowExecutionActor:301 - flowExecution-5: Flow in status RUNNING. Starting execution.
2022-07-12T09:06:35.263Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.backend.akka.SnapshotPersister:133 - flowExecution-5: Saving snapshot FlowExecutionState{runIndex=1, status=RUNNING, queuedStages=2, inProgressStages=1, completedStages=0}
2022-07-12T09:06:35.270Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.p.c.f.FlowExecutionActor:194 - flowExecution-5: Added Actor[akka://ClusterSystem/system/singletonManagerflowManagementActor/flowManagementActor/flowDeploymentActorV2-5/flowExecutionActorV2-5/$$a-adapter#-665643777] as subscriber.
2022-07-12T09:06:35.276Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.backend.akka.SnapshotPersister:145 - flowExecution-5: Deleting snapshots succeeded.
2022-07-12T09:06:35.276Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.backend.akka.SnapshotPersister:162 - flowExecution-5: Saved snapshot 1 with success.
2022-07-12T09:06:35.548Z [ClusterSystem-akka.actor.default-dispatcher-17] INFO  a.h.p.c.f.FlowExecutionActor:377 - flowExecution-5: Stage 0 completed. Running stage will be marked as completed.
2022-07-12T09:06:35.549Z [ClusterSystem-akka.actor.default-dispatcher-17] INFO  a.h.backend.akka.SnapshotPersister:133 - flowExecution-5: Saving snapshot FlowExecutionState{runIndex=1, status=RUNNING, queuedStages=1, inProgressStages=1, completedStages=1}
2022-07-12T09:06:35.556Z [ClusterSystem-akka.actor.default-dispatcher-17] INFO  a.h.backend.akka.SnapshotPersister:162 - flowExecution-5: Saved snapshot 2 with success.
2022-07-12T09:06:36.313Z [ClusterSystem-akka.actor.default-dispatcher-17] INFO  a.h.backend.akka.SnapshotPersister:133 - flowExecution-5: Saving snapshot FlowExecutionState{runIndex=1, status=RUNNING, queuedStages=1, inProgressStages=1, completedStages=1}
2022-07-12T09:06:36.314Z [ClusterSystem-akka.actor.default-dispatcher-18] INFO  a.h.backend.akka.SnapshotPersister:162 - flowExecution-5: Saved snapshot 3 with success.
2022-07-12T09:06:36.316Z [ClusterSystem-akka.actor.default-dispatcher-18] INFO  a.h.p.c.f.FlowExecutionActor:377 - flowExecution-5: Stage 1 completed. Running stage will be marked as completed.
2022-07-12T09:06:36.316Z [ClusterSystem-akka.actor.default-dispatcher-18] INFO  a.h.backend.akka.SnapshotPersister:133 - flowExecution-5: Saving snapshot FlowExecutionState{runIndex=1, status=RUNNING, queuedStages=0, inProgressStages=1, completedStages=2}
2022-07-12T09:06:36.324Z [ClusterSystem-akka.actor.default-dispatcher-6] INFO  a.h.backend.akka.SnapshotPersister:162 - flowExecution-5: Saved snapshot 4 with success.
2022-07-12T09:06:36.563Z [ClusterSystem-akka.actor.default-dispatcher-31] INFO  a.h.p.c.f.FlowExecutionActor:377 - flowExecution-5: Stage 2 completed. Running stage will be marked as completed.
2022-07-12T09:06:36.564Z [ClusterSystem-akka.actor.default-dispatcher-31] INFO  a.h.backend.akka.SnapshotPersister:133 - flowExecution-5: Saving snapshot FlowExecutionState{runIndex=1, status=SUCCEEDED, queuedStages=0, inProgressStages=0, completedStages=3}
2022-07-12T09:06:36.568Z [ClusterSystem-akka.actor.default-dispatcher-16] INFO  a.h.backend.akka.SnapshotPersister:162 - flowExecution-5: Saved snapshot 5 with success.
2022-07-12T09:06:37.292Z [ClusterSystem-akka.actor.default-dispatcher-31] INFO  a.h.w.e.d.FlowExecutionActorV2:134 - flowExecutionActorV2-5: Shutting down gracefully.
2022-07-12T09:06:37.292Z [ClusterSystem-akka.actor.default-dispatcher-14] INFO  a.h.p.c.f.FlowExecutionActor:322 - flowExecution-5: Shutting down gracefully.
2022-07-12T09:06:37.292Z [ClusterSystem-akka.actor.default-dispatcher-14] INFO  a.h.b.a.SequenceNumberPersistentActor:35 - flowExecution-5: Deleting all snapshots for persistenceId=flowExecution-87d41b9a-02a6-4186-b896-e177c6be523d
2022-07-12T09:06:37.293Z [ClusterSystem-akka.actor.default-dispatcher-14] INFO  a.h.p.c.f.FlowExecutionActor:154 - flowExecution-5: Stopped.
2022-07-12T09:06:37.293Z [ClusterSystem-akka.actor.default-dispatcher-14] INFO  a.h.w.e.d.FlowExecutionActorV2:238 - Execution actor delegate has terminated gracefully. Shutting down.
2022-07-12T09:06:37.293Z [ClusterSystem-akka.actor.default-dispatcher-14] INFO  a.h.w.e.d.FlowExecutionActorV2:167 - flowExecutionActorV2-5: Stopped

Stage Logs

  • Stage ID
  • Description
  • Type
  • Status
  • Timings:
    • Started at
    • Stopped at
    • Runtime
    • Runtime in ms
  • External Inputs
  • External Outputs
  • Rejected
  • Queued tasks
  • Running tasks
  • Operators:
    • Name
    • Node ID
    • Runtime metrics:
      • Inputs
      • External Inputs
      • Outputs
      • External Outputs
      • Timings:
        • Runtime
        • Runtime in ms
        • Average runtime
        • Average runtime in ms


Example

{
  "stageId": "1",
  "description": "Processing stage (Segment 0, grouped inputs\u003d0, flat inputs\u003d0)",
  "type": "BATCH_PROCESSING",
  "status": "COMPLETED",
  "timings": {
    "startedAt": "2022-07-12T09:06:35.549Z",
    "stoppedAt": "2022-07-12T09:06:36.313Z",
    "runtime": "\u003c 1 sec",
    "runtimeMs": 764
  },
  "externalInputs": 0,
  "externalOutputs": 0,
  "rejected": 0,
  "queuedTasks": 0,
  "runningTasks": 0,
  "completedTasks": 1,
  "operators": [
    {
      "name": "Auto Output 2022-07-12 08:57",
      "nodeId": "674b1d6d-0fb9-4515-9c18-70275a749daf",
      "inputs": 14,
      "externalInputs": 0,
      "outputs": 14,
	  "externalOutputs": 14,
      "rejected": 0,
      "timings": {
        "runtime": "\u003c 1 sec",
        "runtimeMs": 237,
        "averageRuntime": "\u003c 1 sec",
        "averageRuntimeMs": 15
      }
    },
    {
      "name": "User Data",
      "nodeId": "c6bec2ce-234a-42f4-a3b1-16b8282d4730",
      "inputs": 0,
	  "externalInputs": 14,
      "outputs": 14,
      "externalOutputs": 0,
      "rejected": 0,
      "timings": {
        "runtime": "\u003c 1 sec",
        "runtimeMs": 11,
        "averageRuntime": "\u003c 1 sec",
        "averageRuntimeMs": 0
      }
    },
    {
      "name": "Upper",
      "nodeId": "43abcd22-3877-4b6b-864d-cadd41a72646",
      "inputs": 14,
      "externalInputs": 0
      "outputs": 14,
      "externalOutputs": 0,
      "rejected": 0,
      "timings": {
        "runtime": "\u003c 1 sec",
        "runtimeMs": 0,
        "averageRuntime": "\u003c 1 sec",
        "averageRuntimeMs": 0
      }
    }
  ]
}

Containerized Function (Docker)

  • Name
  • Container consumes:
    • CPU time
    • Memory usage
  • Consumed resources totals
    • Uptime
    • Consumed CPU time
    • Used CPUs average
    • Configured CPUs
    • Max memory used

Example

--------- Consumed resources ----------
Uptime:              171s
Consumed CPU time:   217.5s (185.2s user, 32.2s system)
Used CPUs (average): 1.27 (1.08 user, 0.19 system)
Configured CPUs:     1.9 (utilized: 66.8% total, 56.9% user, 9.9% system)
Max memory used:     1.7 GB/2.0 GB

Execution Statistics

To view execution statistics on a node by node basis:

  1. Navigate to the Show run details page.
  2. Click the kabab menu icon on the right side of the Flow run row. 
  3. Click Execution Statistics.


    A read-only version of the Flow is displayed.
    Under the node name, click the number to view statistics for individual nodes in your Flow. 
    View:
    • The number of input records the node processed coming from the previous node.
    • The number of external input records the Input processed coming from an external data source.
    • The number of output records the node processed coming from the previous node.
    • The number of external output records the Output processed that are written to an external data source.
    • The number of errors the node recorded.
    • The total runtime of the node during execution.
    • The average runtime of the node during execution.

Filter Logs Displayed

  • Field data can be sorted alphanumerically by clicking the field name at the top of the column.
  • Hide all the Flows that have run successfully and had no inputs or outputs by marking the box Hide successful, 0 inputs and 0 outputs.
    • This feature is useful when a Flow has been set to run on a schedule but no new data needed to be processed.
    • This feature box is marked by default.