ores.compute.telemetry

Table of Contents

The grid telemetry surface: one request and reply for the dashboard summary, and one fire-and-forget publish from a wrapper node reporting what it has done since the last sample.

These are operations rather than entity verbs, so they are declared here instead of being derived from an entity. The grid summary is computed, not stored, and the node sample arrives from outside the service, so neither is a row a caller addresses by key.

The handler that serves them stays hand-written beside this model, because it reads the newest stored samples and calls the grid-stats function.

1. Physical space

The shell commands are switched off. The grid summary reaches a person through the hand-written compute grid-stats command, which sends get_grid_stats_request, composes it with the host listing, and can watch the grid while a batch runs. A generated unit would send the same request and print the reply, so it would be a second command for one read.

2. Includes

#include <cstdint>
#include <string>
#include <string_view>
#include <vector>

3. Messages

3.1. get_grid_stats_request

/**
 * @brief Asks for the current grid summary.
 *
 * It carries no fields, because the caller's session decides what the
 * summary covers.
 */

3.2. node_stats_summary

/**
 * @brief One node's contribution to the grid summary.
 */

3.2.1. host_id

/**
 * @brief The node the summary describes.
 */

3.2.2. machine

/**
 * @brief The computer the node runs on, empty when the node did not say.
 */

3.2.3. tasks_completed

/**
 * @brief Tasks the node finished since its wrapper started.
 */

3.2.4. tasks_failed

/**
 * @brief Failed tasks the grid summary attributes to the node.
 */

3.2.5. tasks_since_last

/**
 * @brief Tasks the node finished since its previous sample.
 */

3.2.6. avg_task_duration_ms

/**
 * @brief Mean task duration over the tasks since the last sample.
 */

3.2.7. max_task_duration_ms

/**
 * @brief The node's slowest task in the interval the summary covers.
 */

3.2.8. input_bytes_fetched

/**
 * @brief Bytes the node fetched for those tasks.
 */

3.2.9. output_bytes_uploaded

/**
 * @brief Bytes the node uploaded for those tasks.
 */

3.2.10. seconds_since_hb

/**
 * @brief Seconds since the node last heartbeated.
 */

3.3. get_grid_stats_response

/**
 * @brief The grid summary, as the dashboard renders it.
 *
 * The counters mirror one stored grid sample, and the node summaries come
 * from the most recent sample of every node.
 */

3.3.1. success

/**
 * @brief Whether the summary was produced.
 */

3.3.2. message

/**
 * @brief Why it was not, when it was not.
 */

3.3.3. total_hosts

/**
 * @brief Hosts registered for the tenant.
 */

3.3.4. online_hosts

/**
 * @brief Hosts that heartbeated inside the online window.
 */

3.3.5. idle_hosts

/**
 * @brief Online hosts with nothing in flight.
 */

3.3.6. results_inactive

/**
 * @brief Results in server state 1.
 */

3.3.7. results_unsent

/**
 * @brief Results in server state 2.
 */

3.3.8. results_in_progress

/**
 * @brief Results in server state 4.
 */

3.3.9. results_done

/**
 * @brief Results in server state 5.
 */

3.3.10. total_workunits

/**
 * @brief Workunits the tenant holds.
 */

3.3.11. total_batches

/**
 * @brief Batches the tenant holds.
 */

3.3.12. active_batches

/**
 * @brief Batches that are not closed.
 */

3.3.13. outcomes_success

/**
 * @brief Results that finished with outcome 1 in the last day.
 */

3.3.14. outcomes_client_error

/**
 * @brief Results that finished with outcome 3 in the last day.
 */

3.3.15. outcomes_no_reply

/**
 * @brief Results that finished with outcome 4 in the last day.
 */

3.3.16. sampled_at

/**
 * @brief ISO-8601 timestamp of when the sample was taken.
 */

3.3.17. node_summaries

/**
 * @brief Per-node summaries from the most recent node samples.
 */

3.4. get_grid_series_request

/**
 * @brief Asks for the stored grid samples over a time range, as columns.
 *
 * The read serves the whole installation and needs the permission that
 * get_grid_stats needs. The range widens to whole buckets, counted from the
 * Unix epoch. A range that needs more than 5000 buckets is refused, so the
 * caller widens the bucket and never pulls raw rows.
 */

3.4.1. start_time

/**
 * @brief ISO-8601 start of the range, inclusive.
 */

3.4.2. end_time

/**
 * @brief ISO-8601 end of the range, exclusive.
 */

3.4.3. bucket_seconds

/**
 * @brief Width of a bucket in seconds.
 */

3.5. get_grid_series_response

/**
 * @brief The grid samples of the range as columns on a regular time axis.
 *
 * Bucket i starts at start_time plus i times bucket_seconds, so the axis is
 * implicit and every column has bucket_count entries. A column holds the value
 * of the newest sample in the bucket. A bucket with no sample has samples[i]
 * equal to 0 and zeros in every column, so a caller must read samples before
 * it draws a value: a gap is not a measurement of zero. The outcome counts and
 * results_done are totals that only grow, so a caller draws throughput as the
 * difference between two buckets.
 */

3.5.1. success

/**
 * @brief Whether the series was read.
 */

3.5.2. message

/**
 * @brief Why it was not, when it was not.
 */

3.5.3. start_time

/**
 * @brief ISO-8601 start of the first bucket.
 */

3.5.4. bucket_seconds

/**
 * @brief Width of a bucket in seconds.
 */

3.5.5. bucket_count

/**
 * @brief Number of buckets, and the length of every column.
 */

3.5.6. samples

/**
 * @brief Samples in each bucket. 0 marks a gap.
 */

3.5.7. total_hosts

/**
 * @brief Hosts registered, per bucket.
 */

3.5.8. online_hosts

/**
 * @brief Hosts online, per bucket.
 */

3.5.9. idle_hosts

/**
 * @brief Hosts online with no work, per bucket.
 */

3.5.10. results_inactive

/**
 * @brief Results not yet sendable, per bucket.
 */

3.5.11. results_unsent

/**
 * @brief Results waiting for a node, per bucket.
 */

3.5.12. results_in_progress

/**
 * @brief Results a node is running, per bucket.
 */

3.5.13. results_done

/**
 * @brief Results finished, a total that only grows, per bucket.
 */

3.5.14. total_workunits

/**
 * @brief Work units, per bucket.
 */

3.5.15. total_batches

/**
 * @brief Batches, per bucket.
 */

3.5.16. active_batches

/**
 * @brief Batches with work left, per bucket.
 */

3.5.17. outcomes_success

/**
 * @brief Results that ended in success, a total that only grows, per bucket.
 */

3.5.18. outcomes_client_error

/**
 * @brief Results that ended in a client error, a total that only grows, per bucket.
 */

3.5.19. outcomes_no_reply

/**
 * @brief Results that ended with no reply, a total that only grows, per bucket.
 */

3.6. get_node_series_request

/**
 * @brief Asks for the stored node samples over a time range, as columns.
 *
 * The read serves the whole installation and needs the permission that
 * get_grid_stats needs. The range widens to whole buckets, counted from the
 * Unix epoch. A range that needs more than 5000 buckets, or more than 200000
 * cells of nodes times buckets, is refused, so the caller widens the bucket and
 * never pulls raw rows.
 */

3.6.1. start_time

/**
 * @brief ISO-8601 start of the range, inclusive.
 */

3.6.2. end_time

/**
 * @brief ISO-8601 end of the range, exclusive.
 */

3.6.3. bucket_seconds

/**
 * @brief Width of a bucket in seconds.
 */

3.7. node_series_node

/**
 * @brief One node that reported in the range.
 */

3.7.1. host_id

/**
 * @brief The node's host id.
 */

3.7.2. machine

/**
 * @brief The computer the node runs on, from its newest sample that names one. Empty when none did.
 */

3.8. get_node_series_response

/**
 * @brief The node samples of the range as columns on a regular time axis.
 *
 * Bucket j starts at start_time plus j times bucket_seconds, as in the grid
 * series. The nodes are listed once, ordered by machine and then host id, so the
 * nodes of one computer sit together. Every column below has
 * nodes.size() times bucket_count entries, row-major: the cell of node i in
 * bucket j is at i times bucket_count plus j. A cell with no sample has
 * samples 0 and zeros in every column, so a caller must read samples before it
 * draws a value: a gap is not a measurement of zero. Only nodes with a sample in
 * the range are listed.
 */

3.8.1. success

/**
 * @brief Whether the series was read.
 */

3.8.2. message

/**
 * @brief Why it was not, when it was not.
 */

3.8.3. start_time

/**
 * @brief ISO-8601 start of the first bucket.
 */

3.8.4. bucket_seconds

/**
 * @brief Width of a bucket in seconds.
 */

3.8.5. bucket_count

/**
 * @brief Number of buckets.
 */

3.8.6. nodes

/**
 * @brief The nodes that reported in the range.
 */

3.8.7. samples

/**
 * @brief Samples in each cell. 0 marks a gap.
 */

3.8.8. tasks_completed

/**
 * @brief Tasks finished since the wrapper started, at the cell's newest sample.
 */

3.8.9. tasks_failed

/**
 * @brief Tasks that failed since the wrapper started, at the cell's newest sample.
 */

3.8.10. tasks_since_last

/**
 * @brief Tasks finished in the cell, summed over its samples.
 */

3.8.11. avg_task_duration_ms

/**
 * @brief Mean task duration in the cell, weighted by the tasks of each sample. 0 when the cell finished none.
 */

3.8.12. max_task_duration_ms

/**
 * @brief Longest task in the cell.
 */

3.8.13. input_bytes_fetched

/**
 * @brief Bytes fetched in the cell, summed over its samples.
 */

3.8.14. output_bytes_uploaded

/**
 * @brief Bytes uploaded in the cell, summed over its samples.
 */

3.9. node_sample_message

/**
 * @brief One node's report, published fire-and-forget from a wrapper.
 *
 * It carries no response, so the wrapper does not wait for one. The
 * publishing node has no session and the handler reads it without one, so
 * the message is public rather than claiming a token it never sends.
 */

3.9.1. tenant_id

/**
 * @brief The tenant the node works for.
 */

3.9.2. host_id

/**
 * @brief The reporting node.
 */

3.9.3. machine

/**
 * @brief The computer the node runs on, empty when the node did not say.
 */

3.9.4. sampled_at

/**
 * @brief ISO-8601 timestamp of the sample.
 */

3.9.5. tasks_completed

/**
 * @brief Tasks finished since the wrapper started.
 */

3.9.6. tasks_failed

/**
 * @brief Tasks that failed since the wrapper started.
 */

3.9.7. tasks_since_last

/**
 * @brief Tasks finished since the previous sample.
 */

3.9.8. avg_task_duration_ms

/**
 * @brief Mean task duration over the tasks since the last sample.
 */

3.9.9. max_task_duration_ms

/**
 * @brief Longest task duration over the tasks since the last sample.
 */

3.9.10. input_bytes_fetched

/**
 * @brief Bytes fetched for those tasks.
 */

3.9.11. output_bytes_uploaded

/**
 * @brief Bytes uploaded for those tasks.
 */

3.9.12. seconds_since_hb

/**
 * @brief Seconds since this node last heartbeated.
 */

3.10. get_grid_usage_request

/**
 * @brief Asks who uses the grid in the session's tenant, with the work counted by party.
 *
 * It carries no fields, because the caller's session decides the tenant and the
 * parties. A tenant's administrator reads the tenant. A session that acts for
 * parties reads those parties. A session of the system tenant reads the system
 * tenant's work only, and another tenant is read by entering it.
 */

3.11. get_grid_usage_response

/**
 * @brief The work of the session's tenant counted by party, as columns.
 *
 * Group i is described by keys[i] and by entry i of every other column. The
 * group is a party, and group_kind says "party". A key that is empty stands for
 * work with no party, which predates the party scope. The share of a group is its
 * entry in a column divided by the sum of that column, which the caller
 * computes.
 */

3.11.1. success

/**
 * @brief Whether the usage was read.
 */

3.11.2. message

/**
 * @brief Why it was not, when it was not.
 */

3.11.3. group_kind

/**
 * @brief What a group is: always "party".
 */

3.11.4. keys

/**
 * @brief The id of each group, empty for work with no party.
 */

3.11.5. batches

/**
 * @brief Batches in each group.
 */

3.11.6. workunits

/**
 * @brief Workunits in each group.
 */

3.11.7. results

/**
 * @brief Results in each group.
 */

3.11.8. results_unsent

/**
 * @brief Results waiting to run in each group.
 */

3.11.9. results_in_progress

/**
 * @brief Results running in each group.
 */

3.11.10. results_done

/**
 * @brief Results finished in each group.
 */

3.11.11. outcomes_success

/**
 * @brief Results that finished with success in the last 24 hours.
 */

3.11.12. outcomes_failed

/**
 * @brief Results that finished with an error or no reply in the last 24 hours.
 */

3.11.13. oldest_unsent_seconds

/**
 * @brief How long the oldest waiting result of each group has waited, in seconds.
 *
 * Zero when the group has no result waiting. The wait is counted from the start of the
 * current version of the oldest waiting result, which is the time it was queued while
 * nothing rewrites a result that is still unsent.
 */

4. See also

Emacs 29.3 (Org mode 9.6.15)