Sprint Backlog 08

Updates to sprint and product backlog.

We need to scan all of our finance notebooks so we can use them with AI. Each sprint will have a story similar to this until we scan and process them all.

At the end of the sprint, generate the sprint summary using the prompt.

Time spent doing the demo. This is the first one so it will be very basic.

We need to validate the scripts and fix all issues we find.

This pull request significantly refines the integration of TimescaleDB within the ORES SQL schema, making it more adaptable to different TimescaleDB license tiers and ensuring robust functionality. It introduces runtime license detection to intelligently enable or disable advanced TimescaleDB features, alongside implementing per-database extension installation for PostgreSQL. Furthermore, this PR substantially expands the project's documentation by adding detailed architectural overviews for the ores.assets and ores.comms.analyser components, and a comprehensive guide to the overall SQL schema, enhancing clarity and maintainability for developers and users alike.

Highlights:

• Enhanced TimescaleDB Integration: The system now correctly integrates with both Apache and Timescale license editions of TimescaleDB, dynamically adjusting features based on the detected license.
• Per-Database Extension Installation: PostgreSQL extensions are now installed on a per-database basis, ensuring proper functionality within each ORES database instance, including the template.
• Runtime License Detection: SQL scripts now include runtime checks for the TimescaleDB license, gracefully skipping advanced features like compression, retention policies, and continuous aggregates if only the Apache license is present.
• New Component Documentation: Detailed Org-mode documentation has been added for the ores.assets and ores.comms.analyser components, outlining their architecture, domain types, and usage.
• Comprehensive SQL Schema Documentation: The ores.sql.org file has been added, providing extensive documentation on the ORES SQL schema, including architecture, design patterns, PostgreSQL extensions (especially TimescaleDB), and database creation processes. Schema Prefix for TimescaleDB Functions: TimescaleDB function calls in sessions_create.sql and session_stats_create.sql now explicitly use the public. schema prefix to avoid issues when search_path is set to 'ores'.

Rationale: implemented in previous sprint.

Refactor client manager to use it.

Since we have to seed from SQL scripts, having code seeding as well is just confusing.

This pull request fundamentally shifts the database seeding mechanism from C++ application code to SQL population scripts. This change centralizes the initial setup of permissions, roles, and system flags directly within the database template, simplifying application startup logic and ensuring consistent data across new instances. Consequently, the test suites have been refactored to accommodate this pre-seeded state, improving test reliability and reflecting the new database initialization flow.

Highlights:

• Migration to SQL for Database Seeding: The C++ rbac_seeder and system_flags_seeder have been replaced with dedicated SQL population scripts for permissions, roles, and system flags, centralizing initial data setup within the database template.
• Directory Renaming for Clarity: The sql/data/ directory, containing various population scripts and assets, has been renamed to sql/populate/ to better reflect its purpose.
• Test Suite Adaptation: Numerous test cases across ores.iam, ores.refdata, and ores.variability have been updated to correctly handle pre-seeded database data, moving away from assumptions of empty databases and instead accounting for existing entries.
• Streamlined Instance Initialization: The init_instance.sql script has been simplified, removing direct seeding calls as the population is now handled during the database template creation process.
• Documentation Refinements: A modeling link in projects/modeling/system_model.org was corrected, and doc/recipes/sql_recipes.org was updated with new SQL commands and output examples.

We added quite a few new features during the holidays but due to the laptop constraints we could not validate them properly. Check they all work correctly and fix what needs fixing. Features:

• session monitoring. Looks good for tracking, cannot view in app.
• flags display. Looks good.
• Roles and permissions: cannot edit, display should use a table.

We have a lot of boost program options parsers and options. We need to do an inventory of all of the implementations and figure out if we can have common code under utility.

This pull request significantly refactors the command-line interface (CLI) configuration across various applications by introducing a modular and composable architecture. The primary goal is to ensure consistency in how command-line options are defined and parsed, making it easier to manage and extend CLI functionality. This change centralizes common options and provides dedicated classes for specific configurations like server, client, and compression settings, while also enhancing logging capabilities with a new PID inclusion option.

Highlights:

• Composable CLI Configuration: Introduced new composable configuration classes (common_configuration, server_configuration, client_configuration, compression_configuration) to standardize command-line options across applications.
• Standardized CLI Flags: Standardized common CLI flags such as -h/–help, -v/–version, and –verbose across all relevant applications.
• Log PID Inclusion: Added a new –log-include-pid option to include the process ID in log filenames, enhancing log management and identification.
• Application Migration: Migrated ores.comms.service, ores.cli, ores.qt, and ores.comms.analyser applications to utilize the new configuration architecture and standardized flags.

We created the application but did not validate it.

Notes:

• need support for dark and light themes.
• currencies are hard-coded.
• password written to log file.
• can't see details in accounts dialog.
• edit currency dialog is too tall.
• use images for site logo, etc.

The shell is specific to the binary protocol.

This pull request implements a significant renaming effort, transitioning the ores.shell component to ores.comms.shell. This change clarifies the component's specific function as an interactive shell for the binary communications protocol. The modifications are comprehensive, touching all aspects of the project from file paths and C++ namespaces to build configurations and user-facing documentation, ensuring consistency and improved clarity regarding the component's purpose.

Highlights:

• Project Renaming: The ores.shell project has been renamed to ores.comms.shell to accurately reflect its role as a shell specifically designed for the binary communications protocol.
• Namespace Updates: All C++ namespaces within the project have been updated from ores::shell to ores::comms::shell.
• Build System & Paths: CMake targets, include paths, and header guards across the project have been modified to align with the new naming convention.
• Documentation Alignment: System UML diagrams, documentation references, and shell recipe examples have been updated to reflect the ores.comms.shell name.

We need to make sure the swagger works, create recipes etc.

At present the following is not working or is a snag:

• can we make the details windows unmaximisable.
• We need to either make the columns configurable, or remove some less useful columns such as symbol, fraction etc. These are still visible in details.
• can we refactor reload logic into common entity code?
• wrong icon in revert for currency history. no option to reload history
• when creating a new currency we cannot attach a flag to it.

Rationale: this seems to work now, with the rename to SKILLS.md.

We don't seem to be able to use skills reliably.

Links:

• Claude: Agent Skills
• reddit: Claude Code skills activate 20% of the time. Here's how I got to 84%.
• How to Make Claude Code Skills Activate Reliably

Rationale: implemented with telemetry work.

It would be very useful to be able to browse both client and server logs from the client (either command line or qt). This will be even more useful with AI agents. Add a simple component that periodically loads up the logs to the database.

Possible names: log shipping, log forwarding, etc.

Rationale: you can't maximise currencies any longer.

At present we end up with two tool bars, one below the other.

Rationale: implemented.

When we add support for HTTP/REST, we need to ensure it uses JWT.

Links:

• Building Secure JWT Authentication in Go with PostgreSQL
• GH jwt-cpp

This pull request focuses on a significant upgrade to the sqlgen library, moving to version 0.6.0. This update allows for the removal of several previous workarounds, including custom vcpkg overlay ports and a single_connection hack for aggregation queries. The primary impact is the adoption of sqlgen's native offset() and limit() operators, which streamlines pagination logic across various repositories, improving the robustness and maintainability of database interactions throughout the codebase.

Highlights:

• SQLGen Update: The sqlgen library has been updated to version 0.6.0 by updating the vcpkg submodule to the latest release.
• Overlay Port Removal: Custom vcpkg overlay ports for sqlgen have been removed as the necessary fixes and features are now merged upstream into the main sqlgen repository.
• Native Pagination Support: Manual offset and limit workarounds in database queries have been replaced with sqlgen's new native offset() and limit() operators, simplifying pagination logic.
• Single Connection Removal: The single_connection hack, previously used for aggregation queries due to a sqlgen limitation (issue Fix cli tests #99), has been removed as sqlgen 0.6.0 resolves this issue. Consequently, the single_connection member has been removed from the database context entirely.
• Expanded Pagination: Pagination support (offset/limit + total count) has been added to feature_flags_repository, role_repository, permission_repository, image_repository, tag_repository, and login_info_repository.

Now available in vcpkg with all required features.

At present we disabled pagination in UI because sqlgen does not support offset. We implemented this feature in sqlgen. When it is released to vcpkg we should update our code to use it.

Remember also this:

// HACK: Using single connection instead of session because sqlgen sessions
// doesn't seem to support SELECT FROM with aggregations. Plain connections
// work fine. This is a temporary workaround until the sqlgen library is
// fixed. See: https://github.com/getml/sqlgen/issues/99
struct count_result {
    long long count;
};

Links:

• GH #100: Feature request: offset support
• https://vcpkg.io/en/packages?query=sqlgen

Rationale: already implemented.

vcpkg will support faker soon:

• #38583: [faker-cxx] add new port

When that is available, we should try to add support for it.

This pull request significantly enhances the user experience for managing feature flags within the Qt application. It introduces real-time feedback through event subscriptions and visual cues like pulsing highlights for recent changes and colored badges for status. Additionally, it improves data presentation by adding new columns and refining the detail dialog, while also updating the backend protocol to support these new data points. The changes aim to make feature flag management more intuitive and informative for users.

Highlights:

• Real-time Updates: The UI now subscribes to feature flag change events, enabling real-time updates and notifications when flags are modified externally.
• Recency Highlighting: Recently modified feature flags in the list view will now display a pulsing yellow highlight to draw attention to their recent changes.
• Enhanced 'Enabled' Column: The 'Enabled' column in the feature flags list now uses colored badges (green for 'Yes', gray for 'No') for improved visual clarity.
• New Data Columns: The feature flags list has been augmented with 'Version' and 'Recorded At' columns, providing more detailed historical context for each flag.
• Improved Detail Dialog UX: The feature flag detail dialog has been updated to use a combo box for the 'Enabled' status and now displays the 'Recorded At' timestamp.
• Optimized Reload Behavior: Saving a feature flag or receiving an external change notification now marks the list as 'stale' instead of triggering an immediate auto-reload, improving responsiveness.
• Protocol Updates: The underlying communication protocol has been updated to version 20.0 to include the new 'recorded_at' and 'version' fields for feature flags.
• Logging Cleanup: All MXN-specific debug logging has been removed from the ImageCache component, streamlining log output.

We need to create fake data so we can explore the problem domain. This is something to work on in the future. We can use LLMs to help with the fake data, where it makes sense.

Example:

This pull request significantly expands the application's data management capabilities by adding full support for ISO 3166-1 countries. It establishes new data structures, persistence mechanisms, and a service layer for country-related operations, alongside user interface components for interactive management. The changes ensure that country data is stored bitemporally, allowing for historical tracking and auditing, and provides a framework for associating countries with visual assets like flags.

Highlights:

• ISO 3166-1 Country Support: Introduced a comprehensive system for managing ISO 3166-1 countries, including domain models, bitemporal repository, and a dedicated service layer within the ores.refdata module.
• Database Schema and Population: Added new SQL scripts to create a bitemporal countries table and populate it with data for all 249 ISO 3166-1 countries. Also included drop scripts for cleanup.
• Country Image Linkage: Implemented a new country_images table and associated domain, repository, and mapper components in ores.assets to link countries with their flag images.
• Web UI Components: Developed new Wt UI components, country_list_widget and country_dialog, enabling full CRUD (Create, Read, Update, Delete) operations for country data through the web interface.
• Service Integration: Integrated the new country_service into the application_context to make country management functionalities available across the application.

Add standard ISO country codes.

This pull request significantly enhances the database's change tracking capabilities by introducing a comprehensive amendment control schema. The new tables and associated logic provide a structured way to record and categorize reasons for data modifications and deletions, which is crucial for maintaining data integrity and meeting stringent regulatory requirements. This foundational change enables a more auditable and compliant data management framework.

Highlights:

• New Temporal Tables for Amendment Control: Introduced three new temporal tables: reason_categories, amendment_reasons, and entity_reason_categories. These tables are designed to track changes and provide historical context for data amendments and deletions.
• Regulatory-Compliant Taxonomy: Implemented a taxonomy for amendment reasons that aligns with key financial regulatory standards, including BCBS 239, FRTB, FINRA, and MiFID II, ensuring robust auditability and compliance.
• Idempotent Data Population: Added idempotent population scripts that seed the new tables with initial data, including 3 reason categories ('system', 'common', 'trade') and 17 specific amendment reasons. This ensures consistent setup across environments.
• Temporal Versioning and Soft Deletes: Each new table incorporates temporal versioning via valid_from and valid_to columns, managed by BEFORE INSERT triggers. Deletion is handled as a soft delete using INSTEAD OF DELETE rules, preserving historical data.

This pull request significantly enhances the system's data integrity and auditability by introducing a robust change tracking mechanism. It establishes a structured framework for recording and categorizing reasons behind all entity modifications, crucial for regulatory compliance and transparent data management. This foundational change impacts both the database schema and the user interface, ensuring that every alteration is justified and documented.

Highlights:

• Change Tracking Infrastructure: Introduced 'change_reasons' and 'change_reason_categories' database tables for comprehensive entity modification tracking.
• Entity Integration: Integrated 'change_reason_code' and 'change_commentary' fields into all temporal entities across the system for audit trails.
• Qt User Interface for Management: Implemented a Qt UI for administrators to manage change reason categories and individual reasons.
• Mandatory Reason Selection: Added a 'ChangeReasonDialog' popup to enforce mandatory reason selection when saving existing entities in the UI.
• Client-Side Caching: Developed a 'ChangeReasonCache' for efficient client-side caching and event-driven refresh of change reasons.
• History Visibility: Enhanced entity history dialogs to display the associated change reason and commentary.
• Protocol Version Update: Updated the messaging protocol version to 21.1, standardizing naming conventions (e.g., 'list' to 'get', 'create'/'update' to 'save') and incorporating change tracking fields.

Review:

• both: missing reason for change in change reasons. Done.
• categories: does not ask for reason on save, no option to add comment. Done.
• categories: remove description from main list, add header configuration so users can add them if they want. Done
• change reasons are not showing categories in combo box. would be nice if there was a box to show the category details near the combo box. Done.
• both: recorded at uses human readable time but no other dialog does. Done.
• change reason: make description a bigger box. Done.
• change reason cannot save. Done.
• history: don't need change reason and commentary in full details, it's already at the top. Done.
• save dialog: rename "save changes" to just "save". Done.
• details dialogs: should allow save even when no fields have changed, but then it should be hard coded to touch. will be checked on the server side. Done.
• save dialog: cannot be touch if any of the fields changed. will be checked on the server side. Done.
• details dialogs: meta-data missing change management fields. Done.
• code: CHECK_DIFF_STRING: why not use a regular c++ function? avoid macros. Done.
• both change management dialogs: should be possible to add, delete and edit. Done.
• change reason: code column should be wider.
• change reason: commentary column is too narrow.
• both change management dialogs: remove description, will show in details dialog.
• change reason: is grid following CSS with spacing, etc? Done.
• both change management dialogs: recorded at should use human readable time. Done.
• both change management dialogs: icons not following skill icon-guidelines. Done.
• both change management dialogs: no history. Done.
• both change management dialogs: details is not a standard details window like all other details. Done.
• change reason: use emblems like we do in accounts for yes and no, with yes in green. Done.
• both change management: menu icons for change are not very informative. have a look at the icons for change reasons and change reason categories: /home/marco/Development/fluentui-system-icons-mayin/assets. Done.
• both change management dialogs: not using standard icons. see skill icon-guidelines. Done.
• refactor: have a common save dialog.

• updating flag does not seem to update after reload. Done.
• can't see flag changes in history. Done.
• cannot revert to version in history. Get an error saying incorrect version. Done.

We should be able to see the log from qt.

Notes:

• it should be possible to change the log level from the UI for both the Qt app and the service.

Links:

• GH: Qt-LogViewer

• we need skills for http end points, shell, cli, qt and wt.

We need to have a quick go at modeling the key entities necessary to support books. Use FPML as the basis and iterate with Gemini and Qwen. It does not have to be super complex but it must at least conform to known use cases, or ignore them explicitly as too complex for now.

At present we are lacking a lot of features in shell for the new entities.

This pull request refactors the eventing system to support dynamic discovery of event channels, eliminating the need for hardcoded channel lists in client applications. It introduces a server-side registry for event channels, new protocol messages for querying this registry, and updates the shell client to leverage this new discovery mechanism. This change centralizes event channel management, improves system flexibility, and simplifies client-side event subscription.

Highlights:

• Dynamic Event Channel Discovery: Clients now query the server for available event channels instead of relying on hardcoded lists, enhancing flexibility and maintainability.
• New Protocol Messages: Introduced list_event_channels_request (0x0015) and list_event_channels_response (0x0016) to facilitate server-side channel discovery.
• Event Channel Registry Service: A new event_channel_registry service component was added to manage and provide information about all registered event channels, including human-readable descriptions.
• Expanded Event Coverage: Added registration for country_changed, change_reason_changed, and change_reason_category_changed events, making them discoverable and subscribable.
• Shell Client Update: The events channels command now dynamically fetches channel information from the server, and events listen * subscribes to all dynamically discovered channels.

• on my account, email saved successfully is not very visible. Instead, make save button enabled after changes, once saved disable it. My account should be a regular dialog, allowing you to keep it open and interact with rest of the app.
• session history should not be maximisable.
• sessions still have 0 bytes sent and received.
• add any details such as instance id to session.
• last login field does not fit the screen.
• cannot reload account in account details. Need reload button.
• remove "grant administrative privileges" from details screen.
• old users remain online. Server should check sessions on start and update table.
• cannot tell who unlocked an account and when.
• do not use "old" for accounts who never logged in. Maybe "unused"?
• at present you can't tell if you asked for a password reset. We probably should use Status to mean: locked, unlocked, reset. And maybe "Age" or "Last Login" to mean: online, never, recent, old.
• add right-click menu with the context options from the toolbar.
• after reload, lock etc remember which line we were on and return to it.
• email updated message is truncated.
• user cannot lock it's own account.

Need to check this works and add UI for it.

It seems we did not add database support for telemetry.

Ores.QT:

• logging about telemetry generates telemetry events. Done.
• unknown in telemetry. Done.

2026-01-07 12:03:24.195664 [INFO] [ores.telemetry.messaging.telemetry_message_handler] Persisted 29 telemetry log entries from unknown

• event viewer is always on top
• can't see timestamps properly in event viewer. can't see event type.
• make accounts dialog bigger. do not allow resizing.
• update flag does not update cache in currency dialog.

• event details is not a proper dialog.
• new events raise the dialog.
• cannot see account created events but can see currencies created.
• when a user logs in or out it does not generate an event, should it?
• saving new flag does not save it in server side.
• non admin users can see roles, update feature flags.
• account is locked due to many failed attempts: just say account is locked.
• remove check box in main feature flags table.
• replace the blurb in feature flags with a simple combo for enabled/disabled.
• enabling feature flag does not show events in event viewer on any ores.qt, reload does not go yellow.
• can't add, delete etc for roles. Make dialog bigger, do not allow maximising. Probably has no eventing.
• cannot add or remove permissions for a role, cannot add new permissions (which probably makes sense?)
• cannot distinguish accounts created by user or by sign-up.
• event viewer should have a ring buffer with the last N events. always captures even when dialog is not switched on. We don't know when we want to see events up front.
• error events should be marked with an error so we can see them quickly.