ReAct Agent Orchestrator

I wanted to build a basic harness application to gain an appreciation of the different implementation issues, as well as fully understand the flow of managing tool calls, state, context, etc. I chose LangGraph as the framework, TogetherAI for open source model calls, and Tavily for web search functionality.

I decided to initially tilt the tools and functionality towards research, and in doing so created a tool that can execute concurrent internet research agents, with the option of calling a single research agent if the task was simple enough. In addition to some basic filesystem tools for content search and discovery, I added the ability to generate and save reports based on the user’s research requests. I now actually use this as a useful research agent, particularly for initial, high-level first passes on topics of interest.

Interesting Lessons

1. Model providers and subtle compatibility issues.

   I discovered that TogetherAI’s LangChain adapter, while seemingly returning all the required model response data, quietly dropped the crucial reasoning_content or reasoning fields from streamed responses. This is a big problem, as some reasoning models keep their thinking traces in fields outside of the main content field. Thus, when conversation history was replayed on subsequent turns, models that didn’t return thinking traces in the content field would be presented with an incomplete record of their previous responses. In a nutshell, they couldn’t see their previous thinking traces, and this appeared to degrade their performance substantially. This was something I only discovered through vibes-based testing, when the agent would sometimes make strings of consecutive tool calls that were unnecessary, repetitive, or just plain illogical given the context. The worst part was this was a silent bug that was not very obvious and only present with some types of reasoning models.

   Finding the cause involved some luck, but my strong sense that something was fundamentally wrong pushed me to dig further and consult an LLM, which eventually helped pinpoint the issue. After I extended TogetherAI’s ChatTogether client by overriding two methods in order to preserve the reasoning fields in streamed responses and subsequent requests, the performance for these models normalized.

2. Tools should be as non-overlapping as possible.

   As my approach tends to be trying to get things to work as well as possible on a relatively weak model, it’s possible that the issue I’m about to describe was partly a function of limited model capacity. Nevertheless, I found that if I gave the agent a task and a set of tools that allowed for more than one way to complete it, with option A being the efficient path (a single tool call) versus option B the less efficient path requiring three separate tool calls, option B stood a surprisingly good chance of being selected.

   More concretely, if I wanted to search a file in a particular directory, instead of calling the search_file_contents tool with arguments specified as the full file path (which I gave to it), the agent would call get_working_directory, then list_directory, then change_directory, etc., meandering through far too many calls before actually executing the task at hand. Rewriting the tools to strip out redundant or overlapping functionality, as much as was practical, was key to making the agent more efficient.

3. Proper context management is a huge cost/token saver for some capabilities.

   I used a fan-out fan-in pattern for the parallel research sub-agents that would, by necessity, contribute a large amount of text to the conversation history via the research tool call, meaning that subsequent turns would repeatedly spend compute processing information that was no longer needed in full. Furthermore, if I wanted to write a report to disk after it was generated (and displayed onscreen), I was effectively feeding the entire report back through the model simply to have it output the same text again, word for word. It was a ridiculously wasteful and expensive round-trip.

   My solution was to:

   - Enlist a task-dedicated, throwaway sub-agent to consume and synthesize each research sub-agent’s output, generating both a compact summary and a full report in one synthesis pass.

   - Persist the full report outside the graph instead of keeping it in context.

   - Replace the original tool message using the same message ID and keep only the compact report summary in the orchestrator’s context.

   - Discard all sub-agents once their jobs were finished.

   This allowed for writing full research reports to disk while keeping the high-level research findings in the orchestrator agent’s memory through the compact summaries. Thus, the token cost of every subsequent turn was reduced without losing access to the underlying report content, which could be searched or read in full again with another tool call, if needed.

There’s a lot more to be done with the harness, but it takes care of all the high level necessities and includes logging and telemetry. Adding new functionality is also relatively straightforward: simply add new tools accessible to the tool node in the graph.

In a nutshell, I think my biggest takeaways, other than the fact that compatibility at the API or interface level does not always guarantee semantic compatibility (i.e. dropping reasoning metadata), are the efficiency gains to be had from using parallel sub-agents, so long as one manages downstream concurrency issues appropriately (such as locks on shared writes). Equally important is the potential for significant token savings through good context management hygiene, so long as keeping things lean doesn’t sacrifice accuracy in downstream usage. Also, I found that adding a planning stage for the agent can also really improve the richness and depth of research queries by fleshing out a user’s initial instructions before the research begins.