How to Build CLI Agents with Python & Ollama

Intro “terminal” or “shell”) is a text-based user interface used to view, manage, and interact with the computer by typing specific commands. CLI AI Agents are the latest step in the evolution of developer tools. They combine the power of LLMs with direct access to a computer’s terminal, files, and external tools. Just imagine using the smartest AI models directly on your terminal, without needing a web interface. The release of ChatGPT in 2022 changed how people interacted with software, as users could write natural language instead of typing commands. In 2023, the tech world had a major breakthrough with the tool calling function introduced by OpenAI. Instead of only producing text, models could also perform actions by sending structured requests to applications, getting an output, and continuing reasoning. Finally, in 2025, Super CLI arrived, the first AI Agent explicitly built for running on your CLI without a web interface. Anthropic’s Claude Code and OpenAI’s Codex CLI followed right after. CLI Agents succeeded because they fit naturally into existing developer workflows. Instead of replacing the terminal, they augment it with reasoning. The CLI has evolved from a place where humans executed commands into a collaborative workspace where users define goals, and AI carries out much of the execution. Today, there are three types of CLI Agents: Cloud-native (i.e. Claude Code): the model runs in the cloud while the CLI securely accesses local tools. Open-source (i.e. Hermes): the orchestration framework is open-source and can use local or remote models. Fully-local (i.e. Ollama): both model and orchestration run on your own machine. In this tutorial, I’m going to show how to build a fully-local CLI Agent with Python and Ollama. I will present some useful code and walk through every line of code with comments so that you can replicate this example. Setup Let’s start by setting up Ollama (pip install ollama==0.6.2), the most famous library to run open-source LLMs locally. First of all, you need to download Ollama from the website. Then, on your CLI, use the command to download the selected LLM. I’m going with Alibaba’s Qwen as it’s both smart and light. After the download is completed, you can move on to Python and start writing code. import ollama llm = "qwen2.5" This Agent must be able to execute shell commands on the CLI, so we need to provide it with the appropriate Tool. First, we define the function that actually performs the action, then it must be mapped to Ollama schemas for tool calling. For the CLI commands, we are going to need the subprocess module, a built-in library used to run system commands directly from your Python code. import subprocess # 1. Define the actual tool function def execute_shell_command(command: str) -> str: """Executes a terminal command and returns stdout or stderr.""" try: result = subprocess.run(command, shell=True, capture_output=True, text=True, timeout=10) return result.stdout if result.returncode == 0 else result.stderr except Exception as e: return str(e) # 2. Map tool names to Python functions TOOL_MAP = { 'execute_shell_command': execute_shell_command } # 3. Provide schema representations for Ollama TOOLS_SCHEMA = [ { 'type': 'function', 'function': { 'name': 'execute_shell_command', 'description': 'Execute safe terminal shell commands on the local machine.', 'parameters': { 'type': 'object', 'properties': { 'command': { 'type': 'string', 'description': 'The exact bash or shell command to run.', } }, 'required': ['command'], }, }, } ] Agent In the interaction with a LLM chatbot, there are 3 types of roles: “role”:“system” — used to pass core instructions to the model on how the conversation should proceed “role”:“user”— used for user’s questions “role”:“assistant” — it’s the reply from the model When we give the very first instruction to the model, we have to specify that it’s the system prompt. messages = [ {"role": "system", "content": "You are a helpful local CLI assistant. You can inspect the system and run tasks using your tools."} ] In order to keep the chat with the AI alive, I will use a loop that starts with the user’s input, and then invokes the Agent to respond (which can be a text from the LLM or the activation of a Tool). import sys while True: try: user_input = input("🙂 >") if user_input.lower() in ['exit', 'quit']: break if not user_input.strip(): continue messages.append({"role": "user", "content": user_input}) # Request completion from Ollama with enabled tools response = ollama.chat( model=llm, messages=messages, tools=TOOLS_SCHEMA ) ### WE WILL ADD CODE HERE ### except KeyboardInterrupt: print("\nExiting.") sys.exit(0) If the model wants to use the Tool, the appropriate function needs to be executed with the input parameters suggested by the LLM in its response object. import json # Process potential tool calls requested by the model while response.get('message', {}).get('tool_calls'): messages.append(response['message']) for tool_call in response['message']['tool_calls']: tool_name = tool_call['function']['name'] arguments = tool_call['function']['arguments'] print(f"🔧 >[Executing Tool] {tool_name}({json.dumps(arguments)})") if tool_name in TOOL_MAP: # Execute tool and grab output string tool_result = TOOL_MAP[tool_name](**arguments) # Provide tool outcome back to the model context messages.append({ "role": "tool", "name": tool_name, "content": tool_result }) else: print(f"⚠️ >Unknown tool execution attempted: {tool_name}") # Re-submit history including tool logs for final evaluation response = ollama.chat( model=llm, messages=messages, tools=TOOLS_SCHEMA ) Since the CLI is a delicate tool, in the last lines of code, I asked the model to double check the conversation history and action parameters, right before the final answer. # Display final text answer to user res = response['message']['content'] print(f"👽 >{res}\n") messages.append({"role": "assistant", "content": res}) Run We can run and interact with the CLI Agent. I saved the code in a py file, so I shall run it from the terminal (python CLIagent.py). The model translates your natural-language request into shell commands. For example, it could inspect your system: “How much disk space I have left?” (df -h), or “What’s my OS version?” (sw_vers), or “How much memory is being used? What processes are eating my CPU?” (top, ps). Please note that anything phrased ambiguously around deletion or cleanup (“clean up my Downloads folder“) could do some real damage, so be careful. Conclusion This article has been a tutorial to demonstrate how to build your own fully-local CLI Agent. Now that the basics are clear, one could add logic and Tools (i.e. Excel, Python) to make a more complex AI, just like Hermes and Claude Code. I hope you enjoyed it! Feel free to contact me for questions and feedback, or just to share your interesting projects. 👉 Let’s Connect 👈 (All images are by the author unless otherwise noted)