LLM Integration

This page is a capture in the deferred bucket of the product backlog — a pre-sprint idea, not yet pulled into a sprint as a story.

LLMs can be useful when learning a new subject as they can provide additional context to the information displayed in the screen. For example, a user can ask the LLM to explain a graph or a table. It would probably be fairly straight forward to allow dumping some of the information in a format that is friendly to LLMs (.e.g./ PNG, Markdown, plain text) and then make an API call to a local or remote LLM. We could probably create a set of useful canned prompts (explain this report, explain this chart).

On a more blue skies approach, one could conceive asking the LLM for suggestions on how to act, on the basis of the analysis. This could result in suggestions for action the user could implement, or even on actions directly taken based on the LLM's suggestions. This is conceptually straightforward: the LLM could for example generate a well defined JSON with the proposed action, and the system would look for some predefined markers in the LLM output:

----- ACTION START
<JSON>
----- ACTION END

The JSON payload would describe the action:

{
    "action": "some_action_type",
    "key1": "value1",
    ....

A trivial lookup table could de-serialise the JSON and execute the action. All that is required is for the LLM to "learn" how to generate JSON compliant with the desired format, which should be quite straightforward (perhaps with the help of fine-tuning). Agents probably provide most of this infrastructure already. The key thing is to ensure all functionality in the core becomes UI agnostic such that one could bolt an NLP UI around it.

Links:

• How-to guides: Prompting
• Practical Techniques to constraint LLM output in JSON format