Rendered at 04:19:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Syntaf 12 hours ago [-]
I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience.
We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents.
We paired this with skills initially, but found that the way folks built skills was often too prescriptive and limited to the authors own specific function in the company. A 2k line long skill suffers from the same gaps as we do, if an agent is just following a laundry list it’s less likely to reason about the request it’s doing.
So we instead asked ourselves: what if we just _let_ the agent reason about the work to be done and only provided the tools + guardrails to gather context and perform accounting work?
Turns out frontier models are GOOD at what they do, they outperformed our highly prescriptive skills and were able to work across a larger set of tasks even without instruction on how to do those tasks.
It’s a breath of fresh air from the decade of CRUD I’ve worked on, harness engineering is very neat.
rush86999 6 hours ago [-]
I think you've really hit the mark on how the harness should be structured:
1. Guardrails - deterministic, social intelligence, team alignment & accountability
2. Learn by doing
3. make it stupid easy for the agent to research and access data
4. DRY
Hi. This is very interesting, could you link to the research? There is a dearth of proper research studies that A/B test what approach is best in terms of harness structure based on repeatable benchmark data with relevant sample uses-cases.
Multi-agent / org (core/agent_radio/, core/fleet_orchestration/):
- Stanford Virtual Biotech — bioRxiv 2026.02.23.707551, Zou Lab — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — 37k agents, CSO->scientists->reviewer->re-delegation, Merck external validation of B7-H3 design. Basis for VFS + hierarchy.
- Debate or Vote (Choi & Li) — NeurIPS 2025 — https://arxiv.org/abs/2508.17536 — MAD gains = majority vote, not debate (why we didn't build debate)
Sandbox / eval:
- DABstep — arXiv:2506.23719 — https://arxiv.org/abs/2506.23719 — 450 real Adyen tasks, justifies code-interpreter + sandbox isolation
- Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 — provenance delimiters cut injection ASR 50% -> <2%
Governance:
- OWASP Top 10 for Agentic Applications 2026 — globally peer-reviewed by 100+ experts, Dec 2025 — https://genai.owasp.org/resource/owasp-top-10-for-agentic-ap... — HIGH. Atom maps 1:1 (Goal Hijack -> match-confidence + oracle, Tool Misuse -> sandbox whitelist/caps, Privilege Abuse -> capability bindings, Memory Poisoning -> verified-episode graduation, etc.) docs/marketing/RESEARCH_NOTES.md:130
- NIST AI Agent Standards Initiative — Feb 17 2026, NIST CAISI — https://www.nist.gov/artificial-intelligence/ai-agent-standa... + RFI summary May 2026 https://www.nist.gov/publications/summary-analysis-responses... — HIGH (US gov standard). Defines the 4 enterprise minimums Atom implements: identification, authorization, access delegation, logging.
- Stanford Virtual Biotech — bioRxiv 2026.02.23.707551 — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — CSO -> 4 divisions -> 8 scientists -> reviewer -> re-delegation, no debate, no SFT — HIGH (Stanford Zou lab + Merck external validation). Basis for Atom's fleet hierarchy core/agent_radio/ and why maturity is routing not security.
- Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 — HIGH — provenance delimiters <provenance type="tool_output"> cut indirect injection ASR 50% -> <2%, used in core/provenance.py:10
- IntentGuard — https://arxiv.org/abs/2512.00966 + OpenReview — HIGH — intent tracing ASR 100% -> 8.5% on AgentDojo/Mind2Web, basis for sandbox egress allowlist + core/sandbox_tripwire.py
YZF 10 hours ago [-]
This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask.
As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the model, it fills the context with conflicting instructions, it reduces the ability of the agent to respond to novel situations (and really in real life most situations are going to be novel). If you want to follow a process or a checklist you probably shouldn't use an LLM, or you should use it for some sub-tasks in the checklist/process but something more deterministic to work through the list.
msdz 6 hours ago [-]
> When I use LLMs I leave things as open as possible. I just give them the information they need and my ask.
How do you handle security?
Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?
YZF 5 hours ago [-]
Sandboxing and reviewing the output. I don't have any incredible insight to add here- that's the same process I think most of us are doing.
wonnage 10 hours ago [-]
That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly
0x457 9 hours ago [-]
> . “why app slow” obviously doesn’t work because the task is underspecified.
Not always. In my case LLM goes to grafana mcp, pulls metrics/traces/cpu profiles. Figures out what is slow and proposes a solution.
hombre_fatal 9 hours ago [-]
I wouldn't agree. Sota models can do self-directed sampling, profiling, benchmarking, read call trees, etc. to give you a report of the app's bottlenecks and then recommend solutions that can be vetted.
I do this constantly.
As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgrading the quality of their work.
wonnage 6 hours ago [-]
In my experience if you’re at the point where you have something to sample then the hard part is already done.
In a perfect world everything is covered by distributed tracing and the problems are only in your application code and the agent just needs to find the data
In reality the data is often missing or misleading. “Your observability sucks”? Yeah, but that’s life
cortesoft 6 hours ago [-]
> “Your observability sucks”? Yeah, but that’s life
You could start by asking your AI "help me add better observability to our stack"
YZF 9 hours ago [-]
I use skills. The skills are not typically "how to perform a task in detail" they are more about what relevant tools and knowledge are required to work in a domain. That is I give the LLM the information it needs about the system but not a sequence of how to accomplish a task. I treat it more like a human and less like a computer.
pests 5 hours ago [-]
> “why app slow” obviously doesn’t work because the task is underspecified
Definitely not true and like everyone else is saying, shows how people still underestimate these models.
I have been working on a simple vite + react app lately and commonly ask Gemini/Antigravity to just "improve speeds", "x is running slow, check it out" and have no complaints.
gmadsen 9 hours ago [-]
It really doesn’t need to be that much more specified, give it context to the tools and level of analysis you expect then “why app slow” is a reasonable prompt
jimjimjim 3 hours ago [-]
disturbingly, when I was using antigravity with gemini pro it was actually quite good at working out 'why app slow' types of problems. Maybe I've been lucky but it seems really good at determining why something might be wrong. It may ask for more logging or diagnostics and run for a long time but it was really digging in and making changes or suggestions to solve the problems.
avadodin 6 hours ago [-]
This vibe people sentiment is not wrong per se.
If you want outlier performance from these models it is best to just ask in the most high level prompt of the most minimal harness and let them loose.
Any extra information reduces their performance.
However, as often as these models output masterpieces, they also produce utter garbage so our current choice is for them to have a process to follow that can be reviewed by humans and LLMs.
trzy 4 hours ago [-]
This doesn’t seem to work when the harness feeds images and asks the agent to do things in the real world. It fails to devise ways to keep track of its progress and fails to utilize its tools effectively.
ljm 11 hours ago [-]
I've been building a harness (on top of Pi for that matter) and have had similar experiences. Pi itself helps a lot with it being extensible by design but it's definitely been a challenge to make certain things work in an expected way.
The native app I'm building on top, which I hope people who are less technical (or not technical at all) will use, is even more interesting because it's not just supposed to shell out to the CLI for everything and needs its own state.
newsomix9xl 12 hours ago [-]
Can you post a generic version of code for this somewhere (e.g. codeberg or whatever)?
I find your description intriguing but I'd like to see it to make sure I understand it.
Syntaf 8 hours ago [-]
Sorry I can't share what we're doing here directly!
I will however say that this page alone does a pretty good job of illustrating what an agent harness might look like: https://docs.agno.com/tools/overview
* System prompt
* Tool calls
* Model definition
Everything else (guards / etc) can just exist as code abstractions between the agent layer and the tool layer.
rpastuszak 11 hours ago [-]
Just came here to say the same :)
soulofmischief 3 hours ago [-]
This couldn't have been said a year ago. It's amazing to watch. I have been building harnesses and applying networked agents to various domains since the GPT-3 API came out, and even two years ago, frontier models were just not at acceptable quality to make these harnesses useful. Everything changed overnight near the end of 2025. What will next year hold?
pdhborges 12 hours ago [-]
So you still have CLIs but they have I presume an help command that describes the capabilities right.
Could you give an example of an accounting guardrail you created?
dpritchett 12 hours ago [-]
I’ve also found that Claude and friends are eerily good at using classic Unix CLI tools so I build mine in the same style, not unlike the `gh` CLI from GitHub, though with an agent-first design shape.
Usually I’m returning TSV as a default format and I add a `help-all` subcommand to list every available command at once when needed. Another thing that helps is adding just-in-time context-sensitive hints, such as: user has just run a list query with at least one result. Add a one-liner to the response explaining the command shape for getting the detail view of the first response.
In terms of skill files, I like to have my CLI generate them dynamically at runtime by walking their own current command tree and then feeding that through a text template.
Yeah the CLI can provide schema for commands via the usual ‘—help’ syntax, so agents are able to discover + explore commands on their own.
As for an example: if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed, if it attempts to do so without the requisite information we deny the tool call and ask the agent to escalate back to the client for proof of receipt.
Often times this results in the agent not doing the work and instead sending a message back to the client asking for proof of the transaction.
For humans on our platform there may be valid situations where we’d want to allow this, but for our agent this is a hard guardrail thus why it’s not just standard validation for any JE posting on our platform.
pdhborges 11 hours ago [-]
if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed
And that rule is encoded in the CLI?
Syntaf 11 hours ago [-]
It’s actually encoded in an abstraction that we call “gates” which run before any tool call an agent makes, this allows us to prevent the tool call from happening and return a cited code + explanation on why their tool call was not executed
Does anyone have a suggestion for a harness that is good at handoff?
When I say handoff, I mean:
* handoff from a terminal CLI to webui (on a phone)?
* handoff from one team member, to another?
* handoff from one communication modality, like writing a prompt in a TUI, to email?
* handoff from one model to another, or one provider (openrouter)( to another (llama.cpp)
Does such a thing exist?
I used to think that a PR would be a good place to centralize all this. Who cares what IDE, or developer, or location. But, now I feel like an agent harness might contain that better.
Why do I want handoff? I keep losing context of where my harness is running. Sometimes I am inside an isolated VM. Sometimes I'm on my laptop, sometimes I'm on my home machine with the big GPU for local models. If I could spin up a harness that could identify itself inside my tailscale network, then I could probably have a single web UI which allows me to keep all that context straight.
I'm tempted to experiment with Pi to configure such a thing. But, perhaps there are patterns out there already with a harness I have not considered.
We're designing it like a meta-harness actually. The goal is to let users put any agent(s) (Claude, Codex, Pi, OpenCode) on a sandbox on any computer (local, K8S pod, cloud VPS, sandbox provider, etc.)
And then be able to interact with the agent(s) from anywhere: terminal, phone, web, or load them into the Codex, Claude, or Cursor apps
Under the hood, I think of part of our tech like mashing together Tailscale and Firecracker.
The edges are rough right now, but if you kick the tires, I appreciate any feedback! (my email is dylan@amika.dev)
And besides what we're building, there are cool projects like Amp Orbs, YC's QM harness, etc.
itishappy 10 hours ago [-]
The session is "just" the raw chat history in it's entirety (human and agent) and can be disseminated as such. This is what enables swapping between models, you simply send the whole context.
Not sure how others do it, but opencode stores sessions in a sqlite db and you can extract them and share them as needed.
Pro-tip: Building your own extremely minimal harness takes about 15m and is both fun and enlightening. Agents are unsurprisingly quite good at it, but ask them to walk you through it step by step.
YZF 10 hours ago [-]
I do this all the time in my workflow. Use any harness. Ask it to create a markdown file with the information required for the handoff. Use that downstream. Keep a "repo" of those markdown files. Are you trying to orchestrate or manage this sort of process?
ds_opseeker 11 hours ago [-]
Sounds like you want an orchestration.
Let's assume handoff happens when one "agent" finishes its work on one task, i.e. "submit a PR".
At that point you want to exit the agent/clear context etc (any context the next actor needs should be in the handoff artifact).
And the orchestrator calls the next agent with the artifact.
And then do a search for these terms on HN and get some idea of their shortcomings vs a 'real' orchestration tool like Airflow or Dagster
speed_spread 5 hours ago [-]
Hermes has /handoff to go from cli or desktop to IM, possibly other ways.
izgorodin 5 hours ago [-]
[flagged]
ni10c 10 hours ago [-]
Author here. It’s ironic because this post was clearly geared towards non-hackers. But now that we’re here.. the other analogy I considered presenting was:
harness = chassis,
model = engine,
fuel = tokens,
agent = car
I’m curious what y’all might think and whether that analogy carries more explanatory power
asQuirreL 9 hours ago [-]
The first analogy that comes to mind, growing out of "harness", is more like harness = harness, model = horse (rather than harness as in climbing harness).
I guess you could say that tokens = hay, and agent = horse and cart, from there? Not sure how useful the hay part is but you could observe from the second that there are many different things you could harness a horse to (also a plough, or a coach, or just a saddle) based on your goal.
johnmw 3 hours ago [-]
I wonder if I'm the only person whose first analogy that came to mind was: model = toddler, harness = baby bouncing jumper harness
troyvit 9 hours ago [-]
I'm a climber so I'm biased but I really liked your climbing harness example because of the configuration you're able to easily make to the harness.
Saying the harness is like a car's chassis doesn't work as well for me because the chassis isn't as configurable as a climbing harness for as little work.
Getting deeper into the climbing analogy you can even swap out the harnesses themselves for wildly different climbs. Like using Claude Code with a bunch of agents for medical software (climbing K2 where that extra padding comes in super handy) and pi.dev with a local model for a respectable web project (sport route where you'll be back in a few hours and it's safe to be a little more exposed).
I'm glad your article made HN, and thank you for pi!
mempko 5 hours ago [-]
I actually use the computer as a metaphor. The LLM is the CPU. The harness is the motherboard which controls communication between the CPU and other components like memory, hardrive, and inputs. In other words, how the LLM interacts with the outside world, and outside world with the LLM.
I've been building coding harnesses since 2021 and believed in their value for a long time. Harnesses matter a lot, look at what claude did for Anthropic.
theturtletalks 13 hours ago [-]
Harnesses are the next frontier. If LLMs are electricity, harnesses are the “electronics.” Right now, it’s like an AC vs DC between Claude and ChatGPT, but once that settles, the harnesses will be the actual value providers.
And Pi is the best harness because of the amazing extension system. You can build extensions that turn Pi into a stock trader, software factory, anything. I tried switching to another harness but none have extension functionality as good as Pi.
Even if there is a new harness or agent project, I tell Pi to dig into the codebase and then make me an extension that brings that functionality into Pi. I did it with Prime Intellect’s and Deepseek’s harnesses and those are built on Pi.
amelius 13 hours ago [-]
> If LLMs are electricity, harnesses are the “electronics.” (...) the harnesses will be the actual value providers.
Don't get ahead of yourself. Harnesses are not exactly rocket science and will be a commodity.
The real value providers here are the hardware, then the LLM as a distant second, and at a much larger distance the harness.
Labs are now post-training models with Harness so that Harness now gets absorbed into the weights.
layer8 12 hours ago [-]
I’d say that harnesses almost by definition are the parts that you want to keep customizable. That won’t get absorbed into the weights.
datsci_est_2015 10 hours ago [-]
Depends on your product strategy. If you only care about how your model will be used in the context of a harness (perhaps, specifically the harness that you designed), then the incentive is plainly there to optimize the weights within the context of the harness.
goosejuice 12 hours ago [-]
My naive intuition is that as harnesses converge on shape and models improve the first party advantage will mostly disappear.
wyre 3 hours ago [-]
What is being absorbed into weights is tool usage. It is incredibly counterproductive to train models on a specific harness, when instead it can be trained to reason about the tools that it has available to itself and how to best use those tools to accomplish it's goal.
Would you rather hire an engineer that can adopt to your org's prefered tooling, or hire an engineer that can only perform well with their own favorite tools? It's the same thing.
gritzko 12 hours ago [-]
Either part can be branded a "commodity" or a "sovereign privilege" depending on supply and demand.
Solar goes all the way up => power is commodity.
Some hyperscaler goes bankrupt => hardware is commodity.
Models get real good => output is a commodity, no profitable problems to solve anymore.
Open source models get good => models are commodity.
theturtletalks 13 hours ago [-]
I was saying more the custom skills and extensions that make the harness not a commodity. Yes people will use Claude Code, Codex, or Pi but their customizations will make their harness unique and more powerful.
romanhounds 13 hours ago [-]
[dead]
mpawelski 13 hours ago [-]
> Harnesses are the next frontier. If LLMs are electricity, harnesses are the “electronics.”
I really though this comment was a satire ...
DarmokTanagra 10 hours ago [-]
Its literally the same people who were making hyperbolic crypto claims a few years ago.
This entire forum is infested with shameless hype chasers and biological linkedin bots.
wwalexander 12 hours ago [-]
E = mc^2 + AI
tomrod 8 hours ago [-]
If we take this at face value, this means AI = 0 !
_superposition_ 12 hours ago [-]
In a sense they are the last frontier imo. At some point a harness will be built that can modify itself to fit the needs of the majority of people's workflows and evolve with them.
layer8 11 hours ago [-]
Then people will want to share and exchange their evolved harnesses. Ways will be found to modularize certain aspects to enable mixing and matching.
I’m thinking of how in cyberpunk, people are replacing their cybernetic enhancements all the time. You could alternatively bioengineer your own body towards the desired outcomes, but that’s more constrained by the trajectory your body has already taken, whereas the promise of cybernetic parts is that they are more independently replaceable. (Probably an illusion in practice, but I’m talking about the fictional ideal.)
As another analogy, monolithic software tends to quickly become hard to change significantly, whereas a plugin architecture tends to be more flexible and modular, and people can share and combine their various plugins.
grey-area 13 hours ago [-]
Sadly, many people have bought into the cult that LLMs will lead to AGI. I guess if that is your worldview then all this babbling about new frontiers makes more sense.
They probably used an LLM to come up with this bizarre metaphor.
jbstack 11 hours ago [-]
I find it difficult to understand people who are wildly skeptical about LLMs leading to AGI (assuming we can even agree on what that means). Consider:
- They can already reason better than many humans and are still improving all the time
- Harnesses are improving all the time
- We're already exploring things like long term memory, long term goals, and other things that humans have which LLMs traditionally lack
- An AI agent can read and reason about every piece of AI research ever published, including looking for insights that humans may have missed. A team of humans could never do this even if they dedicated their whole lives to it.
- They can design and execute experiments on a mass scale to determine what does and doesn't work
- Large AI labs have more than sufficient resources and motivation to throw at the problem, and are in fact doing this.
Topfi 10 hours ago [-]
So you believe LLMs (despite their inherent deficiencies vs EBMs [0], etc.) can lead to what you'd consider AGI, but you also admit that there is no agreement what AGI actually would be and you further don't provide your own definition? But you are surprised that some (like e.g. Yann LeCun (I am very convinced by his published works beyond his authority in the field, but am willing to admit his could be seen as a biased position)) are skeptical?
If you provide what you'd consider AGI, we may not agree on that definition, but I and other skeptics could at least discuss with you whether A.) that seems reasonably achievable given LLMs inherent limitations and B.) whether any of what you'd listed is actually likely to get us there.
As it stands, neither is possible without knowing what you believe AGI to be, but for what it's worth, coming from someone who both does see LLMs as valuable tools but whose definition for AGI also contains, among other things, reliable self-assessment of factual uncertainty [1] and basic counting and grade school maths [0][2] without tools or eternally scaling training data, I have yet to read any evidence that LLMs can achieve my, rather strict, metric for AGI.
These models are amazing tools, their ability to leverage massive amounts of high quality training data to further sciences truly awe inspiring, but that does not mean intelligence, at least in my definition that requires some internals these models have never been proven to possess. It's nuts that solving Erdos problems can be done by a model which struggles to count or solve a sudoku without external tools, but that's where the technology has been for years now and no paper I have read has shown that LLMs can overcome that to any scalable degree. You can push further with training data, but the limitations remain, albeit less noticeable. Any externalities, be it tools (self-scripted or called by the model), external memory solutions of all shapes and sizes, etc. I personally also feel cannot be required for or lead towards AGI as intelligence may be better leveraged by such externalities, but should never require them, so much of your suggestion I feel shouldn't be considered even if one believes LLMs can yield intelligence. I will admit that I am very extreme here though, this is not a position held by everyone for good reason. At the end I will always point towards the "extraordinary claims require extraordinary proof" of it all and that LLMs, in the face of any doubt, should be viewed akin to how Stockfish can play better than any grandmaster, but that does not mean intelligence, at least in my world.
If your definition for intelligence does not require basic arithmetics or an understanding of ones own knowledge gaps, then maybe LLMs can achieve that, but I'd push back on that truly rising to the AGI moniker. Maybe a more comprehensive or even my definition of AGI is possible whilst keeping the autoregressive nature after all, but there is no evidence supporting that by itself and quite a few things that haven't even begun to be overcome before something of that magnitude could be honestly considered.
It's akin to "let's colonise Mars by 2020 or 2030 or 2040 for sure, then terraform it" proposals. If that were possible, wouldn't we see a lot of these methods applied on earth and in a moon base long before (as in, we'd have had a permanent moon base in the early 2000s)? Same with LLMs, if they can truly yield AGI, we'd see some of the major deficiencies dealt with long before. The fact that we neither are terraforming earth, nor have any permanent off world colonies, nor have solved some of the listed, inherent limitations with LLMs by their design, that's what informs my skepticism that both are reasonably achievable in the timelines some industry "experts" (read hype merchants) propose on the regular. You tend to see some progress, a path toward solving actionable problems long before full implementation, at least in the real world...
Did I even mention AGI? All I’m saying is that we’re hitting a plateau with how good models are while harnesses are untapped potential. And with Pi, you can swap models like electricity companies. Yes for now, the electricity is better with some companies but this will stabilize.
And no I came up with the metaphor all on my own, send me the chat of you getting the LLM to come up with it. Why not argue based on merit instead of strawman and ad hominem attacks?
grey-area 13 hours ago [-]
I’m afraid it’s a terrible metaphor, starting with the fact that LLMs are nothing like electricity, and the relation of harnesses to them is nothing like that of electronics to electricity, save perhaps one is a prerequisite of the other.
Harnesses (and the concept of agents before them) presuppose competence in LLMs which simply doesn’t exist.
rasputin243 12 hours ago [-]
“Just as electricity transformed almost everything 100 years ago, today I actually have a hard time thinking of an industry that I don’t think AI will transform in the next several years” - Andrew Ng
Avicebron 12 hours ago [-]
Is it just a coincidence that he works in the space and will directly benefit if this is true?
theturtletalks 12 hours ago [-]
I didn’t come with the electricity idea, it was Sam Altman saying it will be like a utility down the line and metered[0]. What would the “electronics” be in your opinion?
Altman is a salesman selling flimflam to people who should know better.
His idea of metering is predicated on the thing he’s selling being AGI, it is not, and all his predictions have turned to dust.
Also that isn’t how metaphors work - they illuminate by comparison, if the comparison is not close they are not useful.
theturtletalks 11 hours ago [-]
If it is metered and like a utility, Sam Altman will not benefit alone. All the models will have plateaued and you can swap for any of them. Then the only differentiator is the harness.
I don’t believe in AGI, but that doesn’t mean I don’t find AI useful. I just understand that the correct harness can take them to the next level.
qarl2 11 hours ago [-]
> presuppose competence in LLMs which simply doesn’t exist.
Then how do you explain the wild success at using them for development?
grey-area 9 hours ago [-]
Guided by humans, code generators which have ingested the worlds’ code and can recognise and generate patterns can be useful tools. I wouldn’t personally qualify it as a wild success as we are early and there are significant downsides.
That doesn’t make them intelligent agents which think independently.
qarl2 7 hours ago [-]
Would you like to see some of my own examples of wild success?
I have a system that entirely reverse engineers old arcade games. Creates semantic symbol mappings that were considered impossible just a couple years ago.
Granted, it took me a couple weeks to build the system.
From impossible to a couple weeks in just a couple years.
Would you like to see it or continue to pretend these things don't exist? Your call.
(It's finding the coolest stuff - the anti-tampering hacks they put into the old machines is fascinating.)
sph 12 hours ago [-]
> Sadly, many people have bought into the cult that LLMs will lead to AGI
You can never tell if the goomba opinion of the forum will agree we have reached AGI (seen that happen on a few threads lately) or will readily call that a ludicrous proposition.
PepegaRoach 12 hours ago [-]
[dead]
jacobgold 13 hours ago [-]
> ...once that settles, the harnesses will be the actual value providers.
The words "once that settles" are doing historic levels of work here.
No human on earth has a clear idea whether model technology will settle tomorrow or 100 years from now.
There's every reason to expect architectural breakthroughs will keep being discovered and causing nuclear blasts of forward progress.
jrflo 13 hours ago [-]
I've never used Pi but I don't see why you can't use stock codex or claude code for the same purpose, what makes Pi special? I've built plenty of custom harnesses on top of claude code and codex using custom skills or simple markdown instructions and subagents. Never had any issues or limitations with that approach.
I do agree that harnesses are going to extend AI capabilities a lot in the next year, but after reading Pi's page I don't see anything that makes it particularly special in terms of functionality, other than being more provider-agnostic.
throwup238 13 hours ago [-]
For one you can ask Pi to create a TUI extension, so along with the agent interface you can add whatever custom TUI you need, such as portfolio stock tickers, alerts, whatever you want.
Many of my harnesses eventually turn into customized UIs around the chat interface.
pests 5 hours ago [-]
I was doing something similar months ago with openclaw. I had skills/scripts that replaced my todo list, expense tracker, habits, whatever etc and then would create a minimal web ui. Then a deploy skill that wires it up to my docker/traefik setup. This eventually led to a custom chat dashboard with those wired up as widgets.
ni10c 13 hours ago [-]
Author here. I think our website could be much clearer - but Pi is fundamentally easier to mold than other harnesses. It’s not magic but it strikes the balance well of letting you shape it extensively without letting you break it.
gritzko 12 hours ago [-]
A harness is the bottom layer of a pie that gets fed into the model. In my project, I count 7 more layers on top of it https://replicated.live/blog/wiki
They all affect consistency, coherence, token efficiency. Probably we need some broader term. Like "information architecture", "knowledge architecture"? It's not just shoveling Markdown to nvidias, after all.
lebek 13 hours ago [-]
Codex and Claude historically had more bloat in their system prompt and tools. Pi is minimal by design so more adaptable. But to be fair Claude Code is moving in the Pi direction with a small system prompt.
Aardwolf 12 hours ago [-]
If LLMs are oxen, harnesses are... the harnesses
qarl2 12 hours ago [-]
Yeah. This is pretty clearly the origin of the usage.
The harness facilitates the work animal doing work for you.
Not climbing harnesses to keep you safe.
GodelNumbering 13 hours ago [-]
This is a plug, but relevant. I recently added a 'build native tools on the fly' functionality to Dirac (https://github.com/dirac-run/dirac) that works like:
1. You can use the '/new-tool' and tell what kind of tool you want (including whether it should be task-scoped, workspace-scoped, or global), the model builds it, the harness runs validation and other tests until the tool is ready
2. The model decides that in such and such task, it would be helpful to have a tool like this, it can build a task-scoped tool.
In either scenario, the tool catalog is rebuilt, and the new tool is instantly available in the next turn.
_superposition_ 12 hours ago [-]
This type of modification of the harness on the fly to fit the need is the future.
The only thing left after that is the mobile front. I think static app store type software as we know it is a thing of the past. You'll only ever need one self modifying app.
Kinda ironic that the most popular extensions turn pi back into claude.
faangguyindia 3 hours ago [-]
No, DSH is not built on top of pi
grim_io 12 hours ago [-]
I don't think so.
What I can see is a world where we end up with a Chromium-shaped harness, a fully featured standard implementation everyone builds against, because doing every single thing yourself would be crazy.
The antithesis to Pi, if you will.
jbstack 8 hours ago [-]
I disagree with this. Unlike training models (which requires huge compute), harness development is available to anyone with an editor and ideas. That means that solo devs and small startups can still make meaningful progress.
Also, having only a "standard implementation" makes no sense for a harness. A standard implementation would need to try to be as good as possible at all things. But you'd often want a specialised harness designed for exactly your use case.
grim_io 7 hours ago [-]
I don't see us tinkering with Pi in 5 years.
Some standard solution will emerge, which will be amplified by models being trained specifically to work with it.
12 hours ago [-]
wyre 3 hours ago [-]
It is way too early to tell. If we are comparing to web browsers we are in the early 90s with Netscape, IE, Firefox, etc. I don't even think we are at the point where agent's have a metaphorical JavaScript, we are that early.
The thing that makes everyone build against Chromium is because web browsers are very hard and it is well supported by dev tools like Electron and Playwright.
Harnesses are so easy compared to a web browser, I'm curious what in this world you see that would make building your own harness seem crazy, because I don't see it.
sejje 13 hours ago [-]
What did you bring over from prime-agent? (I use prime-agent as my daily since it launched)
I primarily like how it manages sessions, and how agents can easily reference other sessions.
theturtletalks 13 hours ago [-]
Something like this since Prime Intellect uses RLM under the hood:
I want to move from Claude Desktop to Pi, but I found it a little unfriendly. Any tips to set it up?
theturtletalks 13 hours ago [-]
Pi doesn’t have a UI like Claude Desktop. It also doesn’t work with the Claude subscription, only API key and pricing.
So if you do want to use it, use the Codex sub. Once you install it, run Pi and /login and you’ll get login with ChatGPT. From there, Pi can tweak it’s settings if you ask. Check out their extensions (or ask Pi) and that will take you most of the way there.
What hiccups were you having?
goosejuice 12 hours ago [-]
> It also doesn’t work with the Claude subscription, only API key and pricing.
Not out of the box, but you can add agent sdk. I'm not sure how great the results will be though.
zukzuk 13 hours ago [-]
I haven’t tried it myself yet but I’m under the impression that Hermes Agent might be what you’re looking for?
ni10c 13 hours ago [-]
Can you be more specific regarding unfriendliness?
Topfi 13 hours ago [-]
Please tell me this is satire, it reads like straight from the depths of LinkedIn where a while loop is seen as the second coming…
hliyan 13 hours ago [-]
Are human HN commenters now starting to speak in a dialect of Claudish?
cyanydeez 13 hours ago [-]
what have you built other than a harness?
theturtletalks 13 hours ago [-]
I built a software factory and am now building a stock trader using opencandle extension[0] and a custom extension. For inspiration for how to tweak Pi, check out OMP, Prime Intellect, and Deepseek harnesses.
looking at the website. i can't really tell if they have benchmarks and measuremnts on how all that improves capablities over just using regular agent withtout all that
epolanski 12 hours ago [-]
Both Claude and codex are unappealing, crap, generic agents that you have 0 control over.
Don't understand what people see in them.
irishcoffee 13 hours ago [-]
How does Pi compare to vscode? Admittedly that is the only “agent/harness” I’ve ever used.
lebek 13 hours ago [-]
The harness is just another codebase for the model to write and optimize. The value is still very much in the model.
dominotw 13 hours ago [-]
i think its the opposite. claude code apparently removed hundreds of lines of system prompt because its not relavent anymore with newer models.
also i think its hard to build general harnesses if they were trained on specific harness architecture.
theturtletalks 13 hours ago [-]
Yes but Pi has had a minimal system prompt since inception. Skills and Pi extensions let you make a hyper specific harness for specific use cases. For general conversation, harnesses are overkill most times.
There’s evidence of harnesses making a smaller, weaker model perform better than SOTA and some benchmarks ban harnesses because it becomes too easy.
tokai 13 hours ago [-]
All agentic editors/frameworks have skills and extensions and plugins?
jascha_eng 13 hours ago [-]
The ai hype word for 2026 after agent in 2025 for any LLM powered application.
Well kind of, I wouldn't be surprised to see that some things marketed as agents are actually good old deterministic software.
sph 12 hours ago [-]
It's really funny (and a bit obnoxious) to watch the vocabulary from the outside. In 2023 everybody learned the word moat, then it's been agent(ic), from last year there's more talking about harnesses than at a bungee jumping convention. The mot du jour is frontier.
It truly proves like there's a handful of thought leaders on Twitter that everybody follows blindly and start to copy down to the lexicon and parrot everywhere else.
phforms 9 hours ago [-]
My guess is that everything "reliable" in LLM/agentic-coding comes down to either calls to reliable/deterministic tools or providing well-defined success criteria (such as loads of unit tests) for the LLM to throw its stuff at in "agentic loops" until something sticks.
wangii 2 hours ago [-]
`harness` is a term invented/borrowed to describe a vague vision of applying llm in a more controlled manner, suitable for semi organized, complex work. the sad part is then people shift the attention from the real problem but seeking inspiration from the terminology.
the real problem: is there a general framework to allow llm with limited context size to approach problems, just like a capable newbie integrated into a chaotic code base? how does it work?
I think it's an epistemic problem, and my fork of pi: pie (github.com/wangii/pie)
childofhedgehog 13 hours ago [-]
Clear, relevant, and easy to understand. Thank you for writing this up, I’ll be sharing this link with all my non-tech friends!
phforms 10 hours ago [-]
I second that. Not using agents myself but trying to get an idea on how this stuff works, so I always wondered what an "harness" even is, since anyone seems to assume that this is common knowledge. Now it is really clear to me!
ni10c 13 hours ago [-]
Author here. Thanks - I appreciate the feedback
tosh 13 hours ago [-]
i also like the backpack analogy
the harness is what you take with you on a trip/task
whatever you take with you is not free (system prompt, tools, skills …)
some models are really good even if you bring almost no skills, tools or system prompt
the harness is the complement to the model
the better the model the more minimal the harness can be
harnesses like pi [0] and smol [1]are on the more minimal end of things
Not a bad analogy because the bigger your backpack the slower you walk. With models a big context and tool set degrades performance. So you want the smallest harness/backpack that can do the job.
FinnLobsien 8 hours ago [-]
I feel like harnesses will become massively important for enterprise AI agents.
Right now every tool is shipping some kind of AI agent, but I can’t help but feel that AI agents in large companies will eventually be some kind of internal app with internal MCPs, CLIs, APIs etc.
There might be different harnesses for different use cases that different people have different levels of access to.
This would make sense for the platform/infrastructure engineers who can build a modular harness that a person or team can get access to.
You could have agents team members use locally that have memory enabled for personalization and then agents that anyone can use to ask questions about company context, which wouldn’t personalize things.
profsummergig 5 hours ago [-]
When SE's cannot agree on the meaning of a tool, it indicates the tool is a placeholder for desire.
alexjurkiewicz 4 hours ago [-]
Do you think there is disagreement about the meaning of "harness" in AI? I would say there is almost unanimous consensus.
Revanche1367 2 hours ago [-]
Comments on this very thread would indicate otherwise.
ggangsir 2 hours ago [-]
It is a loop of action-thinking-action based on users'input and LLM response.
tingletech 13 hours ago [-]
A harness is "the code that gives a model an operational environment" according the the framing in https://github.com/shareAI-lab/learn-claude-code -- which is a build your own python claude code tutorial shareAI made after the code leak a few months back.
stacktraceyo 12 hours ago [-]
To me, Before agentic programming a harness was like a mini framework in the app. Like for testing mostly. You’d set up the harness and configure it for your test and it would take care of boilerplate setup / optional reporting / benchmarks ect. Still works for both - but yea need a new word I guess
asim 12 hours ago [-]
From a naming terms yes, the closest equivalent of the past is the term "framework". Having written a Go service framework that's how I perceived it and as I started to work on agents, anything related to that became an "agent harness". I guess naming and terms change with different paradigms.
JBAnderson5 12 hours ago [-]
I have a similar mental model to the climbing harness. I think of LLMs as horses and harnesses as the saddle, reins, etc that you put on your horse. You might configure your harness for an individual rider or you might hook together several horses to pull a carriage.
kmansm27 10 hours ago [-]
From these comments, it seems like people still don't understand what harnesses are... The point is you shouldn't build a harness, you should use a harness and change its system prompt, the tools it has, MCPs it has, give it skills, etc, to make it work for your usecase. You aren't "building a harness on top of pi" if all you're doing is the above. You're just using the harness to connect different things to the LLM.
tedggh 8 hours ago [-]
I feel I’ve been left behind because all I use is Agents.md / Claude.md and Project.md. No skills, no tools. Everything I want my agent to do is specified in those two files. I also have a particular directory and file structure for things like scripts and sources which is indexed in the mds.
alansaber 56 minutes ago [-]
You don't need to. Underlying agent operations (how it compacts, utilises memory, etc) is not user navigable.
anon7000 7 hours ago [-]
Fairly sure that your agent is using tools like “edit file” behind the scenes. The post is describing the fundamentals of how agent harnesses work, models don’t exactly have a way to edit files just sending an API request to and forth the provider in a chatbox.
freepiai 11 hours ago [-]
I think a harness is kind of anything around the intelligence that allows the intelligence to be applied towards something, some sort of task. A great (if off-color) example I remember hearing was how Steven Hawking was brilliant, but really needed that computer setup to be able to apply his intelligence. It really stands out to me as such a clear visual example of what a harness actually is.
Anyway I've been building my own harness on top of pi- www.freepi.ai (it's based on Pi, but now I have an OpenAI compatible endpoint so I'm thinking of it more like free-api :-) ). Basically ad+training supported so I can offer completely free inference. It's really important to me that we don't have harnesses and intelligence trapped in a "have and have not" world. If we don't all have access to intelligence we will end up in a dark place.
Thats again where the visual of Steven Hawking and the wheelchair really stand out in my mind. It's not enough to have the raw intelligence, we need a really good wheelchair too.
_pdp_ 8 hours ago [-]
A harness is all that plus compensating for bad model behaviour - which unfortunately few harnesses actually do
thelastgallon 13 hours ago [-]
Is harness load-bearing?
0x457 7 hours ago [-]
No, its a smoking gun seam.
cyode 11 hours ago [-]
Great example of writing about AI that maintains a human voice. Starting off the post with a picture of the author + nod to real-world experience (climbing) is a reasonably strong “this is not slop” signal.
apparently its an easy way to get on HN, seems like a great blogspam target
phs318u 6 hours ago [-]
My 2c. A lot of what’s being discussed here may boil down to experience (or lack thereof) of writing good functional (and non-functional) requirements. Some developers are always thinking about implementation (or at least, implementation approach), and as a result they are very technically prescriptive. There’s nothing wrong with that per se if you’re actively driving towards a specific architecture (for your own reasons - which could be specified as non-functional reqs).
What others are describing is the BA skill of capturing good _business_ requirements ie. specifying outcomes and qualities rather than methods.
We should be clear about who’s being the BA, the architect and the developer here. At one end, a human vibe coder is acting as BA and the AI as architect, designer and engineer. At the other end, the human is a senior dev closely instructing a AI junior dev. All approaches are valid depending on what you need and the AI tools available (eg. SOTA model vs small self-hosted open-weights model).
tannerr_dev 6 hours ago [-]
super cool site design
bitwize 7 hours ago [-]
A harness is the thing that turns sloppotron spew into action. That's... pretty much it. LLMs are still "just" token prediction algorithms, but you can coax them into outputting things that look like commands. The harness figures out which bits are supposed to be commands, executes them, and feeds the output back into the context of the LLM. They also have prompts of their own to guide LLM behavior.
dwheeler 12 hours ago [-]
Another AI harness is Goose, which is OSS and housed under the Linux Foundation (LF) Agentic AI Foundation (AAIF):
Full disclosure: I work at the LF, but not the AAIF.
jpitz 12 hours ago [-]
Harnesses are hands.
myegorov 12 hours ago [-]
Written using a harness? Too verbose to be read.
webdoodle 13 hours ago [-]
I thought this was the next evolution of the smartphone. One so smart that it does all the thinking for you. You don't even have to be conscious, you just do whatever it tells you too. Oh wait, that's what they do already.
We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents.
We paired this with skills initially, but found that the way folks built skills was often too prescriptive and limited to the authors own specific function in the company. A 2k line long skill suffers from the same gaps as we do, if an agent is just following a laundry list it’s less likely to reason about the request it’s doing.
So we instead asked ourselves: what if we just _let_ the agent reason about the work to be done and only provided the tools + guardrails to gather context and perform accounting work?
Turns out frontier models are GOOD at what they do, they outperformed our highly prescriptive skills and were able to work across a larger set of tasks even without instruction on how to do those tasks.
It’s a breath of fresh air from the decade of CRUD I’ve worked on, harness engineering is very neat.
1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY
Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom
Multi-agent / org (core/agent_radio/, core/fleet_orchestration/): - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551, Zou Lab — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — 37k agents, CSO->scientists->reviewer->re-delegation, Merck external validation of B7-H3 design. Basis for VFS + hierarchy. - Debate or Vote (Choi & Li) — NeurIPS 2025 — https://arxiv.org/abs/2508.17536 — MAD gains = majority vote, not debate (why we didn't build debate)
Sandbox / eval: - DABstep — arXiv:2506.23719 — https://arxiv.org/abs/2506.23719 — 450 real Adyen tasks, justifies code-interpreter + sandbox isolation - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 — provenance delimiters cut injection ASR 50% -> <2%
Governance: - OWASP Top 10 for Agentic Applications 2026 — globally peer-reviewed by 100+ experts, Dec 2025 — https://genai.owasp.org/resource/owasp-top-10-for-agentic-ap... — HIGH. Atom maps 1:1 (Goal Hijack -> match-confidence + oracle, Tool Misuse -> sandbox whitelist/caps, Privilege Abuse -> capability bindings, Memory Poisoning -> verified-episode graduation, etc.) docs/marketing/RESEARCH_NOTES.md:130 - NIST AI Agent Standards Initiative — Feb 17 2026, NIST CAISI — https://www.nist.gov/artificial-intelligence/ai-agent-standa... + RFI summary May 2026 https://www.nist.gov/publications/summary-analysis-responses... — HIGH (US gov standard). Defines the 4 enterprise minimums Atom implements: identification, authorization, access delegation, logging. - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551 — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — CSO -> 4 divisions -> 8 scientists -> reviewer -> re-delegation, no debate, no SFT — HIGH (Stanford Zou lab + Merck external validation). Basis for Atom's fleet hierarchy core/agent_radio/ and why maturity is routing not security. - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 — HIGH — provenance delimiters <provenance type="tool_output"> cut indirect injection ASR 50% -> <2%, used in core/provenance.py:10 - IntentGuard — https://arxiv.org/abs/2512.00966 + OpenReview — HIGH — intent tracing ASR 100% -> 8.5% on AgentDojo/Mind2Web, basis for sandbox egress allowlist + core/sandbox_tripwire.py
As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the model, it fills the context with conflicting instructions, it reduces the ability of the agent to respond to novel situations (and really in real life most situations are going to be novel). If you want to follow a process or a checklist you probably shouldn't use an LLM, or you should use it for some sub-tasks in the checklist/process but something more deterministic to work through the list.
How do you handle security?
Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?
Not always. In my case LLM goes to grafana mcp, pulls metrics/traces/cpu profiles. Figures out what is slow and proposes a solution.
I do this constantly.
As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgrading the quality of their work.
In a perfect world everything is covered by distributed tracing and the problems are only in your application code and the agent just needs to find the data
In reality the data is often missing or misleading. “Your observability sucks”? Yeah, but that’s life
You could start by asking your AI "help me add better observability to our stack"
Definitely not true and like everyone else is saying, shows how people still underestimate these models.
I have been working on a simple vite + react app lately and commonly ask Gemini/Antigravity to just "improve speeds", "x is running slow, check it out" and have no complaints.
If you want outlier performance from these models it is best to just ask in the most high level prompt of the most minimal harness and let them loose.
Any extra information reduces their performance.
However, as often as these models output masterpieces, they also produce utter garbage so our current choice is for them to have a process to follow that can be reviewed by humans and LLMs.
The native app I'm building on top, which I hope people who are less technical (or not technical at all) will use, is even more interesting because it's not just supposed to shell out to the CLI for everything and needs its own state.
I find your description intriguing but I'd like to see it to make sure I understand it.
I will however say that this page alone does a pretty good job of illustrating what an agent harness might look like: https://docs.agno.com/tools/overview
* System prompt
* Tool calls
* Model definition
Everything else (guards / etc) can just exist as code abstractions between the agent layer and the tool layer.
Could you give an example of an accounting guardrail you created?
Usually I’m returning TSV as a default format and I add a `help-all` subcommand to list every available command at once when needed. Another thing that helps is adding just-in-time context-sensitive hints, such as: user has just run a list query with at least one result. Add a one-liner to the response explaining the command shape for getting the detail view of the first response.
In terms of skill files, I like to have my CLI generate them dynamically at runtime by walking their own current command tree and then feeding that through a text template.
Examples from a public project: https://github.com/radiusmethod/gitlab-kiosk/blob/main/skill...
As for an example: if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed, if it attempts to do so without the requisite information we deny the tool call and ask the agent to escalate back to the client for proof of receipt.
Often times this results in the agent not doing the work and instead sending a message back to the client asking for proof of the transaction.
For humans on our platform there may be valid situations where we’d want to allow this, but for our agent this is a hard guardrail thus why it’s not just standard validation for any JE posting on our platform.
https://docs.agno.com/tools/overview
When I say handoff, I mean:
Does such a thing exist?I used to think that a PR would be a good place to centralize all this. Who cares what IDE, or developer, or location. But, now I feel like an agent harness might contain that better.
Why do I want handoff? I keep losing context of where my harness is running. Sometimes I am inside an isolated VM. Sometimes I'm on my laptop, sometimes I'm on my home machine with the big GPU for local models. If I could spin up a harness that could identify itself inside my tailscale network, then I could probably have a single web UI which allows me to keep all that context straight.
I'm tempted to experiment with Pi to configure such a thing. But, perhaps there are patterns out there already with a harness I have not considered.
Open core is here: https://github.com/gofixpoint/amika
We're designing it like a meta-harness actually. The goal is to let users put any agent(s) (Claude, Codex, Pi, OpenCode) on a sandbox on any computer (local, K8S pod, cloud VPS, sandbox provider, etc.)
And then be able to interact with the agent(s) from anywhere: terminal, phone, web, or load them into the Codex, Claude, or Cursor apps
Under the hood, I think of part of our tech like mashing together Tailscale and Firecracker.
The edges are rough right now, but if you kick the tires, I appreciate any feedback! (my email is dylan@amika.dev)
And besides what we're building, there are cool projects like Amp Orbs, YC's QM harness, etc.
Not sure how others do it, but opencode stores sessions in a sqlite db and you can extract them and share them as needed.
https://opencode.ai/docs/cli/#export
Pro-tip: Building your own extremely minimal harness takes about 15m and is both fun and enlightening. Agents are unsurprisingly quite good at it, but ask them to walk you through it step by step.
Let's assume handoff happens when one "agent" finishes its work on one task, i.e. "submit a PR".
At that point you want to exit the agent/clear context etc (any context the next actor needs should be in the handoff artifact).
And the orchestrator calls the next agent with the artifact.
Claude can do this with subagents. If you want to get more serious, I'd look at "durable workflows" and check out what the pi people have to say: https://earendil-works.github.io/absurd/ https://earendil-works.github.io/absurd/patterns/pi-ai-agent...
you should also look at dbos https://www.dbos.dev/
And then do a search for these terms on HN and get some idea of their shortcomings vs a 'real' orchestration tool like Airflow or Dagster
harness = chassis, model = engine, fuel = tokens, agent = car
I’m curious what y’all might think and whether that analogy carries more explanatory power
I guess you could say that tokens = hay, and agent = horse and cart, from there? Not sure how useful the hay part is but you could observe from the second that there are many different things you could harness a horse to (also a plough, or a coach, or just a saddle) based on your goal.
Saying the harness is like a car's chassis doesn't work as well for me because the chassis isn't as configurable as a climbing harness for as little work.
Getting deeper into the climbing analogy you can even swap out the harnesses themselves for wildly different climbs. Like using Claude Code with a bunch of agents for medical software (climbing K2 where that extra padding comes in super handy) and pi.dev with a local model for a respectable web project (sport route where you'll be back in a few hours and it's safe to be a little more exposed).
I'm glad your article made HN, and thank you for pi!
I've been building coding harnesses since 2021 and believed in their value for a long time. Harnesses matter a lot, look at what claude did for Anthropic.
And Pi is the best harness because of the amazing extension system. You can build extensions that turn Pi into a stock trader, software factory, anything. I tried switching to another harness but none have extension functionality as good as Pi.
Even if there is a new harness or agent project, I tell Pi to dig into the codebase and then make me an extension that brings that functionality into Pi. I did it with Prime Intellect’s and Deepseek’s harnesses and those are built on Pi.
Don't get ahead of yourself. Harnesses are not exactly rocket science and will be a commodity.
The real value providers here are the hardware, then the LLM as a distant second, and at a much larger distance the harness.
Labs are now post-training models with Harness so that Harness now gets absorbed into the weights.
Would you rather hire an engineer that can adopt to your org's prefered tooling, or hire an engineer that can only perform well with their own favorite tools? It's the same thing.
Solar goes all the way up => power is commodity.
Some hyperscaler goes bankrupt => hardware is commodity.
Models get real good => output is a commodity, no profitable problems to solve anymore.
Open source models get good => models are commodity.
I really though this comment was a satire ...
This entire forum is infested with shameless hype chasers and biological linkedin bots.
I’m thinking of how in cyberpunk, people are replacing their cybernetic enhancements all the time. You could alternatively bioengineer your own body towards the desired outcomes, but that’s more constrained by the trajectory your body has already taken, whereas the promise of cybernetic parts is that they are more independently replaceable. (Probably an illusion in practice, but I’m talking about the fictional ideal.)
As another analogy, monolithic software tends to quickly become hard to change significantly, whereas a plugin architecture tends to be more flexible and modular, and people can share and combine their various plugins.
They probably used an LLM to come up with this bizarre metaphor.
- They can already reason better than many humans and are still improving all the time
- Harnesses are improving all the time
- We're already exploring things like long term memory, long term goals, and other things that humans have which LLMs traditionally lack
- An AI agent can read and reason about every piece of AI research ever published, including looking for insights that humans may have missed. A team of humans could never do this even if they dedicated their whole lives to it.
- They can design and execute experiments on a mass scale to determine what does and doesn't work
- Large AI labs have more than sufficient resources and motivation to throw at the problem, and are in fact doing this.
If you provide what you'd consider AGI, we may not agree on that definition, but I and other skeptics could at least discuss with you whether A.) that seems reasonably achievable given LLMs inherent limitations and B.) whether any of what you'd listed is actually likely to get us there.
As it stands, neither is possible without knowing what you believe AGI to be, but for what it's worth, coming from someone who both does see LLMs as valuable tools but whose definition for AGI also contains, among other things, reliable self-assessment of factual uncertainty [1] and basic counting and grade school maths [0][2] without tools or eternally scaling training data, I have yet to read any evidence that LLMs can achieve my, rather strict, metric for AGI.
These models are amazing tools, their ability to leverage massive amounts of high quality training data to further sciences truly awe inspiring, but that does not mean intelligence, at least in my definition that requires some internals these models have never been proven to possess. It's nuts that solving Erdos problems can be done by a model which struggles to count or solve a sudoku without external tools, but that's where the technology has been for years now and no paper I have read has shown that LLMs can overcome that to any scalable degree. You can push further with training data, but the limitations remain, albeit less noticeable. Any externalities, be it tools (self-scripted or called by the model), external memory solutions of all shapes and sizes, etc. I personally also feel cannot be required for or lead towards AGI as intelligence may be better leveraged by such externalities, but should never require them, so much of your suggestion I feel shouldn't be considered even if one believes LLMs can yield intelligence. I will admit that I am very extreme here though, this is not a position held by everyone for good reason. At the end I will always point towards the "extraordinary claims require extraordinary proof" of it all and that LLMs, in the face of any doubt, should be viewed akin to how Stockfish can play better than any grandmaster, but that does not mean intelligence, at least in my world.
If your definition for intelligence does not require basic arithmetics or an understanding of ones own knowledge gaps, then maybe LLMs can achieve that, but I'd push back on that truly rising to the AGI moniker. Maybe a more comprehensive or even my definition of AGI is possible whilst keeping the autoregressive nature after all, but there is no evidence supporting that by itself and quite a few things that haven't even begun to be overcome before something of that magnitude could be honestly considered.
It's akin to "let's colonise Mars by 2020 or 2030 or 2040 for sure, then terraform it" proposals. If that were possible, wouldn't we see a lot of these methods applied on earth and in a moon base long before (as in, we'd have had a permanent moon base in the early 2000s)? Same with LLMs, if they can truly yield AGI, we'd see some of the major deficiencies dealt with long before. The fact that we neither are terraforming earth, nor have any permanent off world colonies, nor have solved some of the listed, inherent limitations with LLMs by their design, that's what informs my skepticism that both are reasonably achievable in the timelines some industry "experts" (read hype merchants) propose on the regular. You tend to see some progress, a path toward solving actionable problems long before full implementation, at least in the real world...
[0] https://logicalintelligence.com/blog/energy-based-model-sudo...
[1] https://arxiv.org/html/2607.19367v1
[2] https://arxiv.org/html/2605.02028v2
And no I came up with the metaphor all on my own, send me the chat of you getting the LLM to come up with it. Why not argue based on merit instead of strawman and ad hominem attacks?
Harnesses (and the concept of agents before them) presuppose competence in LLMs which simply doesn’t exist.
0. https://www.businessinsider.com/sam-altman-ai-utility-electr...
His idea of metering is predicated on the thing he’s selling being AGI, it is not, and all his predictions have turned to dust.
Also that isn’t how metaphors work - they illuminate by comparison, if the comparison is not close they are not useful.
I don’t believe in AGI, but that doesn’t mean I don’t find AI useful. I just understand that the correct harness can take them to the next level.
Then how do you explain the wild success at using them for development?
That doesn’t make them intelligent agents which think independently.
I have a system that entirely reverse engineers old arcade games. Creates semantic symbol mappings that were considered impossible just a couple years ago.
Granted, it took me a couple weeks to build the system.
From impossible to a couple weeks in just a couple years.
Would you like to see it or continue to pretend these things don't exist? Your call.
(It's finding the coolest stuff - the anti-tampering hacks they put into the old machines is fascinating.)
You can never tell if the goomba opinion of the forum will agree we have reached AGI (seen that happen on a few threads lately) or will readily call that a ludicrous proposition.
The words "once that settles" are doing historic levels of work here.
No human on earth has a clear idea whether model technology will settle tomorrow or 100 years from now.
There's every reason to expect architectural breakthroughs will keep being discovered and causing nuclear blasts of forward progress.
I do agree that harnesses are going to extend AI capabilities a lot in the next year, but after reading Pi's page I don't see anything that makes it particularly special in terms of functionality, other than being more provider-agnostic.
Many of my harnesses eventually turn into customized UIs around the chat interface.
The harness facilitates the work animal doing work for you.
Not climbing harnesses to keep you safe.
1. You can use the '/new-tool' and tell what kind of tool you want (including whether it should be task-scoped, workspace-scoped, or global), the model builds it, the harness runs validation and other tests until the tool is ready
2. The model decides that in such and such task, it would be helpful to have a tool like this, it can build a task-scoped tool.
In either scenario, the tool catalog is rebuilt, and the new tool is instantly available in the next turn.
https://pi.dev/packages?type=extension
What I can see is a world where we end up with a Chromium-shaped harness, a fully featured standard implementation everyone builds against, because doing every single thing yourself would be crazy.
The antithesis to Pi, if you will.
Also, having only a "standard implementation" makes no sense for a harness. A standard implementation would need to try to be as good as possible at all things. But you'd often want a specialised harness designed for exactly your use case.
Some standard solution will emerge, which will be amplified by models being trained specifically to work with it.
The thing that makes everyone build against Chromium is because web browsers are very hard and it is well supported by dev tools like Electron and Playwright.
Harnesses are so easy compared to a web browser, I'm curious what in this world you see that would make building your own harness seem crazy, because I don't see it.
I primarily like how it manages sessions, and how agents can easily reference other sessions.
https://github.com/manojlds/pi-rlm
So if you do want to use it, use the Codex sub. Once you install it, run Pi and /login and you’ll get login with ChatGPT. From there, Pi can tweak it’s settings if you ask. Check out their extensions (or ask Pi) and that will take you most of the way there.
What hiccups were you having?
Not out of the box, but you can add agent sdk. I'm not sure how great the results will be though.
0. https://github.com/Kahtaf/OpenCandle
looking at the website. i can't really tell if they have benchmarks and measuremnts on how all that improves capablities over just using regular agent withtout all that
Don't understand what people see in them.
also i think its hard to build general harnesses if they were trained on specific harness architecture.
There’s evidence of harnesses making a smaller, weaker model perform better than SOTA and some benchmarks ban harnesses because it becomes too easy.
Well kind of, I wouldn't be surprised to see that some things marketed as agents are actually good old deterministic software.
It truly proves like there's a handful of thought leaders on Twitter that everybody follows blindly and start to copy down to the lexicon and parrot everywhere else.
the real problem: is there a general framework to allow llm with limited context size to approach problems, just like a capable newbie integrated into a chaotic code base? how does it work?
I think it's an epistemic problem, and my fork of pi: pie (github.com/wangii/pie)
the harness is what you take with you on a trip/task
whatever you take with you is not free (system prompt, tools, skills …)
some models are really good even if you bring almost no skills, tools or system prompt
the harness is the complement to the model
the better the model the more minimal the harness can be
harnesses like pi [0] and smol [1]are on the more minimal end of things
[0] https://github.com/earendil-works/pi
[1] https://github.com/smol-env/smol
Right now every tool is shipping some kind of AI agent, but I can’t help but feel that AI agents in large companies will eventually be some kind of internal app with internal MCPs, CLIs, APIs etc.
There might be different harnesses for different use cases that different people have different levels of access to.
This would make sense for the platform/infrastructure engineers who can build a modular harness that a person or team can get access to.
You could have agents team members use locally that have memory enabled for personalization and then agents that anyone can use to ask questions about company context, which wouldn’t personalize things.
Anyway I've been building my own harness on top of pi- www.freepi.ai (it's based on Pi, but now I have an OpenAI compatible endpoint so I'm thinking of it more like free-api :-) ). Basically ad+training supported so I can offer completely free inference. It's really important to me that we don't have harnesses and intelligence trapped in a "have and have not" world. If we don't all have access to intelligence we will end up in a dark place.
Thats again where the visual of Steven Hawking and the wheelchair really stand out in my mind. It's not enough to have the raw intelligence, we need a really good wheelchair too.
FYI, the pic is Royal Robbins (https://en.wikipedia.org/wiki/Royal_Robbins), who was kind of a hero to some of my climbing buddies back in the 70's.
What others are describing is the BA skill of capturing good _business_ requirements ie. specifying outcomes and qualities rather than methods.
We should be clear about who’s being the BA, the architect and the developer here. At one end, a human vibe coder is acting as BA and the AI as architect, designer and engineer. At the other end, the human is a senior dev closely instructing a AI junior dev. All approaches are valid depending on what you need and the AI tools available (eg. SOTA model vs small self-hosted open-weights model).
https://github.com/aaif-goose/goose
Full disclosure: I work at the LF, but not the AAIF.