Have you benchmarked it in real condition to test if it truly reduces costs ? I feel like many of those type of tools often end up making the llm more lost about the logs and make it more spend than just feeding the raw logs or letting it code to discover the content itself
A few days ago I made something to use in addition to head/tail: Venetianblinds.js shows equally spaced chunks of a file https://github.com/firasd/venetianblinds
So I can kinda see these as being part of the same workflow like VBlinds for 'what does this file even look like' before calling the decompose
Finally, a tool that tackles this problem without forcing you to adopt an entire ecosystem. I appreciate how this leans into the Unix philosophy by just focusing on the decomposition step and doing it well. Really appreciate the modular architecture here.
Thank you and we do really appreciate it. The Unix philosophy was certainly one of the influences upon us, we wanted CtrlB Decompose to carry out one task well, converting noisy logs into structured and compact representations which are easier for both humans and LLMs to handle.
Thanks a lot, I truly do appreciate the thoughtful feedback.
We agree that there are a great many interesting downstream possibilities concerning OpenTelemetry, correlation, derived metrics, and more sophisticated telemetry pipelines once you are able to reliably extract structure from noisy logs.
To us, CtrlB Decompose represents the initial step. Although we launched it as a standalone tool for producing logs that are friendly to LLMs, it also feeds into part of CtrlB's special log compression pipeline. The identical structured representation aids in reducing both storage space and token usage without losing the information required for search, investigation, and AI-assisted debugging.
The idea you've advanced regarding enriching the logs with trace context and generating higher-level signals before sending them on downstream is indeed a very interesting one, and we are currently considering it as the project develops. Thank you for bringing it to our attention!
A few days ago I made something to use in addition to head/tail: Venetianblinds.js shows equally spaced chunks of a file https://github.com/firasd/venetianblinds
So I can kinda see these as being part of the same workflow like VBlinds for 'what does this file even look like' before calling the decompose
We agree that there are a great many interesting downstream possibilities concerning OpenTelemetry, correlation, derived metrics, and more sophisticated telemetry pipelines once you are able to reliably extract structure from noisy logs.
To us, CtrlB Decompose represents the initial step. Although we launched it as a standalone tool for producing logs that are friendly to LLMs, it also feeds into part of CtrlB's special log compression pipeline. The identical structured representation aids in reducing both storage space and token usage without losing the information required for search, investigation, and AI-assisted debugging.
The idea you've advanced regarding enriching the logs with trace context and generating higher-level signals before sending them on downstream is indeed a very interesting one, and we are currently considering it as the project develops. Thank you for bringing it to our attention!