"For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.
> simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability
N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?
A simpler spec can be used by a simpler agent. So, maybe that's the use-case here... use by smaller/cheaper agents that run in parallel as opposed to large models running one visualization at a time.
Or at least, maybe that's the idea?
IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.
A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration.
When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while.
In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).
Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.
> Json as a common language for human in visualization has been around for a while
Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.
Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.
Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.
So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.
I'm sorry, but as someone who creates data visualisation as a big part of my job, I wouldn't say the charts on the website look good. Most aren't awful either, but by no means are they an improvement over what I'd get by telling any coding agent to make a chart with Vega-Lite or Observable Plot, and probably worse than if I had some decent instructions/skills.
I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity.
This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.
Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.
This is cool to see from a research team. A few weeks ago I was exploring a similar idea with ntcharts, where a user or LLM can specify a chart in a Golang or JSON object...
and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.
But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...
This is fun! We started thinking it would just be an engineering task in the beginning, but doing a solid intermediate language turned out to be a research project (the paper will be out soon).
Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45
The charts are very nice, and I think the visualisation layer for LLMs is a very interesting problem.
I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.
I'm terrible at diagrams, so I gave GPT very generic descriptions of one of our project, to convert in to that mermaid style, then for Lucid I pasted it in there, and had a visualization of what I needed. Worked out nicely.
interesting how you don't discuss literally anything about the project actually posted and spam your thing. Not pointing you out, seen many other comments like this on HN but always felt a bit weird about them
It's more like a simple high-level spec to make it easier. The idea is that you don't have to fill position / axes details just to make the chart work. The compiler has a bit of magic of using semantic types to optimize what parameters will be set in ECharts.
In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!
N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?
Or at least, maybe that's the idea?
IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.
A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.
I feel we will be seeing this more and more in the near future.
Isnt graphviz there for the same reason?
Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!
Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.
Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.
My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.
Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.
So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.
I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity.
Interesting to see a Chinese University collab for once.
MCP setup: https://microsoft.github.io/flint-chart/#/mcp
Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.
I find that besides training better models, designing new language for agents is also a super viable paths to improve their performance!
https://www.chartjs.org/
https://echarts.apache.org/en/index.html
https://vega.github.io/vega/
.....
.....
https://github.com/microsoft/flint-chart/blob/main/packages/...
and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.
But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...
https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...
Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45
I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.
Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc
Source: https://github.com/espressoplease/smalldocs
I'm terrible at diagrams, so I gave GPT very generic descriptions of one of our project, to convert in to that mermaid style, then for Lucid I pasted it in there, and had a visualization of what I needed. Worked out nicely.
They can do a lot of cool things! Mermaid gallery here: https://smalldocs.org/s/xZrc-lNW1kbXpoIuU3l_ky#k=-0ehGe2B-hR...
but enterprise
In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!