Sprint Backlog 01

Updates to sprint and product backlog.

• add a new organisation and repo.
• add a simple project with vcpkg support.

Result:

• Git repository: https://github.com/OreStudio/OreStudio/

Created VisualOre and setup emacs to connect to it.

Result:

• Discord channel: https://discord.gg/gcrYsjW3pd

Commands:

account add discord <email> <password>

Links:

• bitlbee-discord
• emacs wiki BitlBee
• Discord on Emacs

Setup a basic build with vcpkg.

Result:

• vcpkg in git repository: https://github.com/OreStudio/OreStudio/

Get a basic green build in github, showing up in CDash.

Result:

• Workflow definitions: https://github.com/OreStudio/OreStudio/tree/main/.github/workflows
• Pipelines: https://github.com/OreStudio/OreStudio/actions

We should create a basic package, and upload it to github.

Result:

• Packages available in action: https://github.com/OreStudio/OreStudio/actions/runs/9830998231

We need to decide on the basic architecture for the project. As this is a side-project, mainly for learning about ORE, it is going to be very simple: one or more fat clients connecting to a postgres database. We don't mind having business logic in the database; the most important thing is for it to be really quick to add new features with the least amount of layers and duplication.

Result:

• Structure in github: https://github.com/OreStudio/OreStudio/

Create a trivial application in Qt that exercises the pipelines.

Result:

• Qt project: https://github.com/OreStudio/OreStudio/tree/main/projects/ores.qt

It would be nice to have a simple splash screen with the project logo.

Example:

#include "mainwindow.h"
#include <QApplication>
#include <QSplashScreen>
#include <QTimer>

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);

    QSplashScreen *splash = new QSplashScreen;
    splash->setPixmap(QPixmap("C:/Users/9589693153/Desktop/Vids/pk.png"));
    splash->show();

    MainWindow w;

    QTimer::singleShot(5000, splash, SLOT(close()));
    QTimer::singleShot(5000, &w, SLOT(show()));

    // w.show();

    return a.exec();
}

Source: QT C++ GUI Tutorial - How to display a Splash Screen in Qt

Result:

• Code: QT main.cpp

We need a basic website for the project.

Result:

• site: https://orestudio.github.io/OreStudio/

We need to be able to draw some simple PlantUML diagrams to describe the code as we go along. They should be linked to the documentation.

Result:

• Modeling folder

At present coverage is very low because its picking up some files from vcpkg install. We need to ignore those.

Result:

• CTestCustom.cmake

For some reason we are not getting syntax highlighting in the website.

Result:

• The issue was with the markers on source code blocks.

Its a bit annoying to have both a readme and an index file, since they are both just landing pages. Look into merging them.

Result:

• readme.org

Explain the objective of the project and the approach.

Result:

• readme.org: section "Project Overview".

Copy across from dogen basic logging.

Result:

• log folder

We need to check for memory issues with nightlies.

Notes:

• error with valgrind appears to be related to DWARF version: #56550: Clang Generates DWARF Info That Valgrind Can't Read. Change it to v4.

Result:

• Nightly workflow.
• Nightly in CDash.

We are using CMake in a legacy way when defining warnings, etc. We need to move towards the modern approach, by defining toolchain files and using presets correctly.

Result:

• presets file, with workflow.

We need to create the project documentation in Doxygen.

Result:

• Doxygen documentation.

Move assorted notes we have in the domain over to the documentation folder.

Notes:

• Fix issues with publishing by ignoring emacs package directories and vcpkg.
• add basic support for bibliography as per progen setup (manual bib file).

Result:

• Domain folder in github.
• Domain page on website.

At present we get the following failure:

### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x25
### unhandled dwarf2 abbrev form code 0x23
==6610== Valgrind: debuginfo reader: ensure_valid failed:
==6610== Valgrind:   during call to ML_(img_get)
==6610== Valgrind:   request for range [1940368405, +4) exceeds
==6610== Valgrind:   valid image size of 7099560 for image:
==6610== Valgrind:   "/home/runner/work/OreStudio/OreStudio/build/output/linux-clang-debug/projects/ores.ore.test/ores.ore.test"
==6610==
==6610== Valgrind: debuginfo reader: Possibly corrupted debuginfo file.
==6610== Valgrind: I can't recover.  Giving up.  Sorry.
==6610==

This normally means we are generating DWARF5:

• #56550: Clang Generates DWARF Info That Valgrind Can't Read

We still get the error with DWARF2 v4. Try v3. The problem is with valgrind:

• Bug 452758: Valgrind does not read properly DWARF5 as generated by Clang14

We need to upgrade to latest valgrind. Try moving to latest ubuntu LTS which has valgrind 3.22.

Updating to Ubunbtu 22.04 and valgrind 3.22 resolved the issue.

Result:

• Nightly build is now green.
• commit.

We need to do a simple check of the packages just to make sure they install and run.

Result

Linux package looks correct:

# dpkg -i orestudio_0.0.1_amd64.deb
Selecting previously unselected package orestudio.
(Reading database ... 872299 files and directories currently installed.)
Preparing to unpack orestudio_0.0.1_amd64.deb ...
Unpacking orestudio (0.0.1) ...
Setting up orestudio (0.0.1) ...
# ores.console --help
ORE Studio is a User Interface for Open Source Risk Engine (ORE).Console provides a CLI based version of the interface.
ORE Studio is created by the ORE Studio project.
ores.console uses a command-based interface: <command> <options>.
See below for a list of valid commands.

Global options:

General:
  -h [ --help ]           Display usage and exit.
  -v [ --version ]        Output version information and exit.

Logging:
  -e [ --log-enabled ]    Generate a log file.
  -l [ --log-level ] arg  What level to use for logging. Valid values: trace,
                          debug, info, warn, error. Defaults to info.
  --log-to-console        Output logging to the console, as well as to file.
  --log-directory arg     Where to place the log files.

Commands:

   data           Operates directly on data.

For command specific options, type <command> --help.
# ores.console --version
OreStudio v0.0.1
Copyright (C) 2024 Marco Craveiro.
License GPLv3: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Build: Provider = 'github' Number = '73' Commit = 'a6ca706350b5063fe65b39484350f25bc6bfb806' Timestamp = '2024/07/05 23:59:06'
IMPORTANT: build details are NOT for security purposes.
# apt-get remove orestudio

GUI starts as expected.

Windows package looks very small:

$ ls -lh *msi
-rw-r--r-- 1 marco marco 448K Jul  6 00:01 OreStudio-0.0.1-win64.msi

Its also not installing under wine at present:

$ wine OreStudio-0.0.1-win64.msi
0024:err:module:import_dll Library zlib1.dll (which is needed by L"C:\\windows\\syswow64\\user32.dll") not found
0024:err:module:import_dll Library user32.dll (which is needed by L"C:\\windows\\syswow64\\gdi32.dll") not found
0024:err:module:import_dll Library gdi32.dll (which is needed by L"C:\\windows\\syswow64\\shell32.dll") not found
0024:err:module:import_dll Library zlib1.dll (which is needed by L"C:\\windows\\syswow64\\user32.dll") not found
0024:err:module:import_dll Library user32.dll (which is needed by L"C:\\windows\\syswow64\\gdi32.dll") not found
0024:err:module:import_dll Library gdi32.dll (which is needed by L"C:\\windows\\syswow64\\shlwapi.dll") not found
0024:err:module:import_dll Library zlib1.dll (which is needed by L"C:\\windows\\syswow64\\user32.dll") not found
0024:err:module:import_dll Library user32.dll (which is needed by L"C:\\windows\\syswow64\\shlwapi.dll") not found
0024:err:module:import_dll Library shlwapi.dll (which is needed by L"C:\\windows\\syswow64\\shell32.dll") not found
0024:err:module:import_dll Library zlib1.dll (which is needed by L"C:\\windows\\syswow64\\user32.dll") not found
0024:err:module:import_dll Library user32.dll (which is needed by L"C:\\windows\\syswow64\\shell32.dll") not found
0024:err:module:import_dll Library shell32.dll (which is needed by L"C:\\windows\\syswow64\\start.exe") not found
0024:err:module:import_dll Library zlib1.dll (which is needed by L"C:\\windows\\syswow64\\user32.dll") not found
0024:err:module:import_dll Library user32.dll (which is needed by L"C:\\windows\\syswow64\\start.exe") not found
0024:err:module:loader_init Importing dlls for L"C:\\windows\\syswow64\\start.exe" failed, status c0000135

This needs to be tested on a real windows box.

OSX package is not being uploaded:

Warning: No files were found with the provided path: ./build/output/macos-clang-release/packages/ORES-0.0.1-Darwin-x86_64.dmg. No artifacts will be uploaded.

This is because the filename is incorrect:

CPack: - package: /Users/runner/work/OreStudio/OreStudio/build/output/macos-clang-release/packages/OreStudio-0.0.1-Darwin.dmg generated.

Add a simple command to command line options that deals with data, for example:

ores.console data --import currencies.xml

Notes:

• missing version support. Need template, etc. Try to use existing CMake variables to setup the version.

Result:

$ ./ores.console --help
ORE Studio is a User Interface for Open Source Risk Engine (ORE).
Console provides a CLI based version of the interface.
ORE Studio is created by the ORE Studio project.
ores.console uses a command-based interface: <command> <options>.
See below for a list of valid commands.

Global options:

General:
  -h [ --help ]           Display usage and exit.
  -v [ --version ]        Output version information and exit.

Logging:
  -e [ --log-enabled ]    Generate a log file.
  -l [ --log-level ] arg  What level to use for logging. Valid values: trace,
                          debug, info, warn, error. Defaults to info.
  --log-to-console        Output logging to the console, as well as to file.
  --log-directory arg     Where to place the log files.

Commands:

   import         Imports data into the system.

For command specific options, type <command> --help.

For some reason we see the dummy test coming out twice in the nightlies. We checked CTest submission and it seems there is only one entry. This could be some kind of timing bug in CDash.

This is now happening for GCC as well. This is as expected:

• duplication of test execution with ctest_memcheck() and sanitizers

We need to run the tests just once to avoid this issue.

Result:

• Nightly build has tests only once.

Using org-babel, create a simple recipe file that exercises all options for the command line tool.

Result:

• recipes in git.
• console recipes in site.

We started to struggle with how we use pie charts in the sprint backlog. The main issue is described here:

• SO: R prevent overlap in pie charts

But it seems more generally that we should avoid them altogether:

• Save the Pies for Dessert

Links:

• Horizontal barplot with R and ggplot2

Result:

• Bar chart added to sprint backlog.

We should tag the stories by their "type" or "kind" and then generate a table with how the resourcing was split. Sample tags:

• code: real coding task.
• infra: infrastructure work such as build, tooling, etc.
• agile: other types of overhead such as agile bookkeeping, release notes, etc.
• doc: documentation related tasks.

Seems like this is not easy to do within org itself:

• Org clocktable, how to sum time by tags
• org-mode clock table by tag

But we could perhaps do it within R.

Result:

• Bar chart added to sprint backlog.

We are using C++ 17 at present. Change it to 20 or even 23 and see what breaks.

Seems like 20 is the highest we can go for now.

Result:

• presets updated to C++ 20.

Update Debian to latest, install required packages and build ORE Studio in laptop.

Tasks:

• install R.
• get latest cunene with R support.

• seems like the current libsystemd does not build. This is a bit weird because we should be pinned to the same vcpkg commit. However, for some reason it fails to build in the laptop. It seems there already is a fix in main, so we can probably solve the issue by doing an update to vcpkg. We just need to understand why it works everywhere else. The issue appears to be with a filesystem:

  • #39365: [libsystemd] Build error on x64-linux

  We just need to make sure we are running latest vcpkg as the fix has made it in via this MR:

  • #39741: [libsystemd] Update to 256.2

Create a table and write currencies to it.

We need to have the ability to read and write currencies from XML. Copy all currency input data from examples.

• remove manual JSON serialisation code, add real serialisers.

We need some kind of tree view and list view.

Links:

• Model/View Tutorial

We are still getting a lot of vcpkg warnings, so our custom setup is not working.

CMake Warning at ports/icu/portfile.cmake:2 (message):
CMake Warning at ports/qtbase/portfile.cmake:39 (message):
CMake Warning at scripts/cmake/vcpkg_buildpath_length_warning.cmake:4 (message):
CMake Warning at scripts/cmake/vcpkg_copy_pdbs.cmake:44 (message):

Links:

• CTest performance tip: Use CTestCustom.cmake, not .ctest

This project seems to have quite a few additional badges that are of interest:

• https://github.com/jkriege2/JKQtPlotter/blob/master/README.md

It may also be interesting in the future:

About

an extensive Qt5 & Qt6 Plotter framework (including a feature-richt plotter widget, a speed-optimized, but limited variant and a LaTeX equation renderer!), written fully in C/C++ and without external dependencies

We need to see how people setup the UI and copy the approach.

Results:

• follow the approach of creating widgets in Qt Creator and then inheriting from the generated code to extend it with signals and slots, etc.
• copy code from fast track with themes, about, links to website, discord, etc.
• use remix icons. Follow the fast track approach to embed assets into the application.
• follow the fast track approach for the applications settings, including remembering window sizes, positions, etc.

Links:

• QLog: use of flags, complex UI with widgets similar to what we are trying to do. Use of colours in grid view.
• Search GitHub for Qt6 apps
• Guitar: GUI for diffs, graph for the git trees. Interesting icons for the workspace / explorer. Shell / log for git actions.
• FastTrack: complex UI with themes, tabs, docking widgets.
• RemixIcon: icons

We need to do some minor tidy-ups to the Qt UI:

• rename the tree widget after workspaces, worksets or some such. Left for later on its own story.
• add a prodigy launcher for the UI. done
• add logging to UI. done
• columns do not have names. done
• rows are too big. done
• when you select currencies from the main menu, it should open up the currencies tab. done
• make the currencies table widget follow the main window size. done

Notes:

• spent some time analysing FastTrack, a Qt application. Need to do a write up of the features we can copy across. Created its own story for this analysis.

Links:

• Qt6: SQL Examples
• Qt6 - Qt SQL
• Draw in an empty QTabWidget
• New Thread to add new tab when menu item is clicked
• Model/View Tutorial

We need to be able to represent the data in the filesystem for the ORE samples in a way that allows users to load and save the samples. The data in the filesystem has the following structure:

• Examples
  • Example 1:
    • Input
    • ExpectedOutput, e.g. Output
  • Example 2
  • …
  • Example n
  • Input

The overall context under which valuation is taking place needs a name. Options:

• environment
• workspace
• sandbox: seems to imply the data is not "real" but that is not always the case. This seems more like a label we could apply to the top-level container rather than its overall name.

Each entry within the top-level container is then itself another kind of container. Options:

• namespace
• package

There are two types of these containers:

• libraries: components whose content is made to be shared with other components; and
• executables: components whose content is sufficient to execute computations.
• component

Elements within a component is of two kinds: input and output. However, this may not necessarily be a good classification because some outputs can be used as inputs to output other elements. Perhaps this is more of a tag.

Notes:

• move this analysis to documentation.

It may be easier to integrate diagrams with roam if they are org-mode documents. Experiment with babel for this.

We added a test password to the repo on purpose to see if it was going to be detected by the github actions:

std::string connection_string("postgresql://ores:ores@localhost:5433/oresdb");

It wasn't. We need to figure out which actions need to be setup for this. Add any other actions we may be missing.

The build seems to be failing:

-- SCCache NOT found.
 CMake Error at /usr/local/share/cmake-3.30/Modules/CMakeDetermineSystem.cmake:152 (message):
   Could not find toolchain file:
   /home/runner/work/OreStudio/OreStudio/vcpkg/scripts/buildsystems/vcpkg.cmake
 Call Stack (most recent call first):
 CMakeLists.txt:61 (project)


 CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
 CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
 -- Configuring incomplete, errors occurred!
 ~/work/OreStudio/OreStudio ~/work/OreStudio/OreStudio
 ~/work/OreStudio/OreStudio
 cpp/autobuilder: No supported build command succeeded.
 cpp/autobuilder: autobuild summary.
 Error: We were unable to automatically build your code. Please replace the call to the autobuild action with your custom build steps. Encountered a fatal error while running "/opt/hostedtoolcache/CodeQL/2.18.0/x64/codeql/cpp/tools/autobuild.sh". Exit code was 1 and last log line was: cpp/autobuilder: autobuild summary. See the logs for more details.

This may be due to a missing sub-module for vcpkg.

Try binding to relational database using Qt SQL infrastructure.

Notes:

• create a model inheriting from QSqlRelationalTableModel.
• to connect:

QSqlDatabase db = QSqlDatabase::addDatabase("QPSQL");
        db.setHostName("localhost");
        db.setDatabaseName("pg_nhx");
        db.setUserName("postgres");
        db.setPassword("postgres");
        db.open();

Links:

• Qt/C++ - Lesson 005. QSqlRelationalTableModel - The work with relational tables
• qt-qsqlrelationaltablemodel-example
• PostgreSQL and Trying to connect using Qt
• index : qt/qtbase.git
• SQL Examples