Experimented with a barebones agent that forced the model to reply only in Python, and then automatically executed the generated python. I tried it with Qwen-2.5-Coder, Gemma4 E4B, and Qwen3-4B-Instruct.
The idea was to remove the need for a bash/powershell tool, and have the agent write complex tool call chains in a single program output.
The first approach extracted python code blocks from the reply and executed them. The second approach exposed python as a regular tool.
The first approach seemed to improve the overall ability of Qwen-2.5-Coder, because this model was now able to answer questions by writing simple programs (whereas it failed previously due to poor tool-calling ability). It answered questions about my PC and folders, and made web calls when I asked it to (using the requests module in python).
To give it a feedback loop, I fed back the error messages as tool outputs. I also tried feeding the output of the program, but that didn’t work well. The model wasn’t able to plan ahead in steps, tried to one-shot the answer anyway, and would go into an endless loop over the answer.
Smaller models would also occasionally glitch-out, by not wrapping the code in code blocks.
Qwen3-4B-Instruct seemed to perform better, but was more adamant about refusing things like making web calls, even though Qwen-2.5 and Gemma4 happily used the requests module to fetch stuff.
I got significantly better results with the second approach, i.e. by making python a regular “tool”. I had to coerce the model (in the system prompt) to use python, not refuse, and tell it that “everything is possible with python” :) Qwen3 took that to heart, and was quite good.
This also fixed a lot of the message-loop problems that I experienced in the previous approach.
This version of the agent had the following tools: run_python, read_file, edit_file, write_file, grep and glob.
Models are pretty well-trained to use tools now, so it looks like we’ll just have to align with that direction for now. Bypassing the “tools” system didn’t work well for me. Would’ve been nice, because it would’ve opened up smaller/faster models that have poor tool-calling ability (but good python skills).