Automating My Favorite Part of Software Engineering

I don’t usually write essays. Actually, I don’t think I have written many long-form things in my life, except code files. My journey with computers started with Microsoft Word. I was probably around 10 when I first saw WordArt. It was one of the coolest things I had seen. You could create these amazing (at least for a ten-year-old) pieces of text and make things move around the screen. I remember trying to replicate it myself, building all sorts of fun WordArts. It was a small thing, but it fascinated me. How did the computer do that? That fascination eventually led me to electronic engineering. In college, I participated in a competition where we had to build a robot that could follow a given track. We bought the parts, built a small robot on wheels, and I did the microcontroller coding. It was pretty simple, but I remember having a lot of fun building it and trying to make it actually follow the track. From there, I eventually landed at Microsoft. That’s where my real software engineering journey began. I spent over a decade there working on different teams, including Exchange Server and Outlook. One project I worked on at Microsoft was a room booking assistant. The idea was pretty simple: instead of finding and booking a room yourself, you could just select an option indicating that you need a room for the meeting. Before the meeting, the system would find one and book it for you. And if people were joining from different locations, it would book a room in each location. Then in 2023, I joined Docusign. ChatGPT had just come out and suddenly everyone was talking about AI and trying things with it. The very first feature I worked on at Docusign was a feature that used AI to summarize an agreement for the signer, so instead of going through pages of an agreement, they could quickly understand what they were signing. That was really my first experience working with AI on the job. I remember being genuinely excited about it. It felt new, and honestly none of us really knew how far this was going to go. In this whole journey, I realized that my joy for software engineering has always been solving problems. I loved writing code. I still write it, or at least I used to before AI took much of that over in recent years. But I have always enjoyed debugging things even more. Anyone who has spent enough time debugging knows the feeling. You have an issue that makes absolutely no sense. You try one thing. That’s not it. You stare at logs. You form another hypothesis. That’s wrong too. Hours later, you finally root-cause it. That feeling cannot be described and I loved that part of the job. Now AI has started to do more and more of this for me, and I definitely didn’t think that in just a few years, AI would change so much of how I do my own job and, increasingly, my relationship with code. Before we called everything an agent At Microsoft, I worked on Exchange Server. Sometimes we would get reports of issues that had happened to one particular email for one particular customer. Reproducing something like that could be almost impossible. We tried a lot of things to find what was going on, but it was one of those race conditions that happens occasionally, so consistent repro was not there. Eventually, we built a diagnostic tool. If I built the same thing today, I am pretty sure we would call it an agent. The idea was simple: for each request, we could register a condition, such as “when the user is Joe and the exception contains ‘InvalidOperationException,’ capture additional information and dump it in a secure log.” We didn’t need to reproduce the problem anymore. We waited for it to happen again and collected the information we needed. I had a lot of fun building that system. In the last year, though, things have started to change. I went to lunch I was working on a data transfer tool here at Docusign. I needed to understand a bunch of existing code and add support for some data and tables that weren’t being transferred. Sonnet 4 had just launched inside Copilot and it was definitely something different. I started with explaining my data transfer problem to Sonnet 4. It started going through the legacy code and putting together a plan. I went for lunch. When I came back, Copilot had already come up with a plan. I sat down and reviewed it. Trust in AI in those days was thin, so I fact-checked its assumptions. And it was mostly good. I made some minor changes and asked it to go ahead and implement the plan Something I had estimated at about a week (given the code base was new to me) was basically done in a day. It still took time to test things locally, get the PR reviewed by other engineers, and eventually deploy it to customers. But I remember that moment very clearly. That was probably the first time I realized AI was going to change software engineering much more, and much faster, than I had expected. And it did. I don’t write code at all anymore It’s true. I don’t write code at all anymore, at least not in the traditional way of “writing code.” That sounds strange, even to me. I plan things with AI. We go back and forth. I ask questions. It proposes something. I disagree. It investigates more. We change the plan. Sometimes we do that several times. (I’m just now realizing I’m calling this workflow “we”.) Once I am happy with our plan, I let it run. That’s how I am developing features these days. I am still making high-level decisions, fo the most part. But the part where I sit down and type out the implementation? That’s basically gone. Our coding elf At Docusign, a coworker and I built our cloud coding agent, aka our “coding elf.” It started as a hackathon project. Initially it was really just the two of us using it. And somewhat ironically, most of what we used it for was to write code for the coding elf itself. Then other engineers started trying it for simple changes like feature flag rollouts and bug fixes. Over time, more and more people started depending on it. The funny part is that now the coding elf is writing most of its own code. Sometimes I would send it a Slack message. Sometimes I would assign it a Jira task and give it what to build, and it would go off and come back with a PR. Slack made it ridiculously easy. I would send it tasks during the day. At night. While I was at my kids’ school. The coding elf could spit out a PR in something like 20 minutes with just a message from me. That started changing the way I thought about engineering work. But something that happened during one of our FixIt Days changed it even more. Non-engineers started writing code We have a quarterly FixIt Day where engineers get one day to fix something, or work on pretty much anything they want. A group of Docusign designers saw our coding elf and asked if it could help them fix a few UX issues. They weren’t asking me to help them fix it; they wanted to use it themselves, instead of creating backlog ticket items for us engineers. So they gave it a try. They found an issue — a simple bug — then asked the coding elf to convert that into a PR. During that FixIt Day, they sent around 80 PRs. More than 10 were merged the very next day. The remaining ones went through review and many were merged over time. Engineers reviewed the code and it went through our deployment process, but engineers didn’t need to context switch to write the implementation for the simple bugs. That’s the shift we are seeing across the industry now. Normally, a designer finds something that needs change and then there’s a handoff somewhere. An engineer has to translate that task/bug into code. Now, that boundary has started to disappear. Maybe AI isn’t only changing how engineers write software. Maybe it’s changing who gets to write software. Then I automated myself out of the loop The coding elf created another problem for me. Because colleagues across Docusign — engineers, designers, product folks — had started depending on it to write and update software, I needed to keep it running. I would monitor errors and exceptions from the coding elf service logs. If I saw a bug, I would investigate it and then ask the coding elf to fix itself. The coding elf was writing most of its own code. But I still had to spend time investigating any errors/issues and then tell the coding elf to fix it. In a way, I was the bridge between the error logs and the coding agent. So, during Docusign’s next hackathon, two engineers and I built an agentic platform called Krew. I created my first agent (we call them Krewmates) and named it Night Watch. Night Watch monitors the coding elf service logs. It looks for errors and problems, investigates them, and opens Jira tasks. The coding elf can then pick up those tasks and produce a PR. A code review agent can review the PR. So now, I can go to sleep and software engineering continues. The system can find a problem. An agent can investigate it. Another agent can make a change. Another agent can review that change. And I can wake up to a PR. There is some irony here: I spent much of my career loving debugging. That eureka moment when I finally figured out why something was broken was one of my favorite parts of being an engineer. Then I built an agent to do it while I was sleeping. I automated myself out of the loop. As an engineer, that is a strange feeling. Part of me loves it; I built something that can do work I used to spend hours doing myself. That feels “engineering-y” in the sense of automating things and making them better. But there is another part of me that wonders what happens when we automate parts that we enjoy. I am still not sure how I feel about that. “Claude said…” AI has changed things in the last year more than I expected. It has amazed me in many ways. It has also disappointed me. And one of the things I have started noticing is how much we trust it. I hear a new phrase from engineers now: “Claude said…” One actual conversation with an engineer was about a service. He responded, “It’s throwing a 404. I thought it was because the feature flag was off, but Claude said otherwise.” Claude had a different theory: the host app needed a corresponding route registered. It even pointed to the existing route and the source file where the related routes lived. It was a detailed, completely plausible technical explanation. Maybe Claude was right. But something about the wording caught my attention. Not, “I traced it and it looks like we’re missing a route.” Not, “I think we need a change in the host app.” It was, “Claude said…” End of sentence. I started seeing versions of this trust in AI elsewhere. Another real message: “Claude said that these fields are exposed with some API somehow connected with Zuora.” I love the “somehow” in that sentence. We have an answer. But do we understand the answer? This was probably my favorite version of a ‘Claude said’: “Copilot said this was 95% coverage. I’m sure it wouldn’t lie.” Obviously that person was joking. But there is something underneath the joke. We know these models can be wrong. Yet we are learning to trust them remarkably quickly. I am too. Those were the days when I fact-checked everything I mentioned earlier that when Sonnet gave me that data transfer plan, I carefully fact-checked it. I don’t do that for everything anymore. Partly because the models got better. Partly because after something gives you a good answer again and again and again, you naturally start trusting it. But I have also been burned by that trust. For one feature, I was researching capabilities in Azure Cosmos DB. I needed to understand something around the MongoDB API and change streams. AI told me a capability existed. It sounded completely reasonable. I started going down that path. Except things weren’t adding up. Eventually I went back to the MSDN documentation and started actually reading it myself. The capability wasn’t supported the way the AI had told me it was. It had sent me down a rabbit hole. That is what makes this tricky. If AI gave stupid answers all the time, this would be an easy problem. It doesn’t. It gives incredibly good answers most of the time. Good enough that I estimated a week of work and got it done in a day. Good enough that I stopped writing code. Good enough that non-engineering designers created 80 PRs in one day. And good enough that slowly, without really deciding to, we started trusting it. What happens when the AIs disagree? We recently had a situation where one LLM suggested a fix. The engineer implemented it. Then our code review agent looked at the change and said the fix was broken. Maybe the code review agent only noticed it now because those lines had been touched. So we asked a different LLM to look at it. It agreed with the code review agent. So now one LLM says one thing. The code review agent says it is wrong. Another LLM says the code review agent is right. It’s funny. But it’s also a preview of a problem I think we are going to deal with much more. We keep saying agents will review the work of other agents. Great. What happens when they disagree? Which AI do we believe? One? The other? Neither? And if a third AI is the tie-breaker, what happens when that one disagrees too? At some point, someone needs to know what is actually true. Or do they? Is this just Assembly all over again? A comparison I hear a lot is that this is just another abstraction layer. We used to write Assembly. Then higher-level languages came along. Most engineers today don’t care about the machine instructions generated underneath their code. And software engineering is better because of it. So perhaps AI is simply the next version of that. Maybe in the future, code is effectively Assembly. We express intent at a higher level and agents generate the implementation. Personally, I have a hard time agreeing with that comparison. At least today. The reason is the probabilistic nature of LLM output versus the deterministic nature of compilers. A compiler doesn’t occasionally decide that Azure Cosmos DB supports a feature it doesn’t support. It doesn’t confidently give you a plausible explanation for something that isn’t true. LLMs can. And now we are increasingly asking LLMs to check other LLMs. That feels different to me. But I also have to be careful here. Maybe this is just the old engineer in me resisting the next abstraction. Because my own behavior is already contradicting my argument. I don’t review all the code either I don’t like big PRs anymore. Actually, if anyone sends me a giant PR, I ain’t reviewing it. Break it down. And AI has made this problem worse. Our telemetry shows PR sizes going up with AI. That makes perfect sense. AI doesn’t get tired after writing 50 lines like human engineers do. Generating another 500 lines is cheap. But my attention isn’t. My attention span has probably gone down too, if I am being fair. So my review process has changed. I let AI review code in the first pass. I look at the comments AI added. If the comments make sense, I have them addressed. Then I decide how deeply I personally review the code based on what the code is doing. If it is customer-facing, I am going to review it much more thoroughly. Internal code is different. Our coding elf is probably the extreme example. I don’t review its code. The coding elf writes the code. The code review agent reviews it. If there are no comments and the testing output looks good, I let it go. This probably reflects badly on me. Except … does it? The software works. Tests pass. The review agent is happy. It is an internal system with a risk profile I understand. So what exactly would I gain by reading every line? A year ago I probably would have had a very strong answer to that question. Today I don’t. How do you teach something that won’t sit still? I see another side of this through the program I lead at Docusign focused on helping our engineers adopt AI. It is hard to keep up. It is even harder to keep everyone else up-to-date. At one point, I had a model and a harness that worked really well for me, and I thought I had found my setup. Then in six months, another frontier model took the lead. And models are only one part of it. The MCP server became a thing. Suddenly we were building MCP servers for everything. Then came skills, plugins, different agent harnesses, and different ways of managing context. What works well today might not be what I’m using six months from now. That creates a weird problem for engineering organizations. Our traditional technical learning model assumes knowledge has some reasonable shelf life. We create a learning session. We document the best practices. We teach engineers how to use a new technology. But what if the material is out of date by the time everyone has learned it? I increasingly think we need to rethink technical learning and enablement for engineers. Maybe teaching the current tool matters less. Maybe teaching engineers how to evaluate the next tool matters more. Our code is in Git. Our knowledge isn’t There is another problem I don’t think better models alone will solve: context. Legacy software contains knowledge that isn’t actually in the software. Some of it is in Jira. Some is in Slack. Some is buried in an old design document. And a lot of it is sitting in the minds of engineers who have worked on the system for years. Why does this weird code exist? Why didn’t we do the obvious thing? What broke the last time somebody changed this? Which customer depends on behavior that looks completely accidental? An experienced engineer might know those answers. An agent looking at the repository doesn’t. If agents are going to do more of the work, I think that legacy knowledge needs to come down from engineers’ minds and into files. Not necessarily just text files. But into durable context that agents can consume. And once we have many agents, we need to get much better at context and memory between them. What did my Night Watch agent learn while investigating the issue? What context did the coding elf use when it implemented the fix? Does the code review agent know the same history? Should it? How do we know where a piece of information came from? How do we know it is still true? I suspect we are going to spend a lot more time building some kind of “shared memory” where agents can learn from what happened in their past sessions and share those learnings with other agents when they are relevant. Maybe that’s one way we solve this. I’m sure there will be others. But the more agents we have doing real work, the more important this shared context is going to become. And maybe we end up spending a lot more time tuning that context. And less time writing code. So … do I still need to understand the code? This is the question I keep asking myself. And I don’t have an answer. The old me — the debugging wonk — wants to cling onto understanding code. That is what being an engineer has meant to me for most of my life. I liked knowing how things worked. I liked debugging. I liked finally finding the root cause. I still don’t want an engineer to tell me, “Claude said so.” I want to ask: “But do you understand why?” Except then I have to look at myself. I don’t write code at all anymore. I let AI plan with me and then implement. I let AI review AI-generated code. For the coding elf, if the code review agent is happy and the tests are good, I don’t even read all the code. I built Night Watch so an agent can discover a problem and send it to another agent without me. So maybe I need to ask myself the uncomfortable version of the question. If five years from now an agent can understand a requirement, investigate the codebase, implement the change, test it, have another agent review it, deploy it safely, monitor production and roll it back when something goes wrong… …and if that process turns out to be more reliable than the way we build software today… …do I really care whether the engineer understands every line of code? The old me says yes. But maybe I should let go of that. Maybe code really does become an implementation detail. Maybe the thing an engineer needs to understand moves up a level. The problem. The intent. The architecture. The constraints. The evidence that tells us the system is behaving correctly. The system of agents that produces the code. Maybe that becomes engineering. Or maybe we are slowly giving up something important and won’t realize what it was until we need it. I honestly don’t know. Which is funny and scary at the same time, and not that different from the wider sentiment towards AI in this zeitgeist. That curiosity took me into electronics, microcontrollers, Microsoft, Exchange Server, Docusign, and a career spent building and debugging software. Now, after all of that, I am watching software get to a point where I may no longer need to understand exactly how it works. The ten-year-old version of me who was enamored with WordArt would probably think what we can do today is magic. The current engineer in me still wants to look behind the curtain. I am just no longer sure that I need to. And maybe learning when to look behind the curtain, and when to let go, is going to be one of the hardest parts of being a software engineer in this next era.