The Agent on the Night Shift

How we used Bob to work through vulnerability debt while we slept

In my last post, I covered what it takes to run IBM Bob in a container on a schedule. In this post, I'll cover what we built once that infrastructure was in place, and why I think it was worth building.

The Problem

IBM teams manage a lot of repositories. Our automated tooling opens vulnerability fix PRs at scale, which sounds great until you think about what happens next. Someone still needs to review those PRs. Someone needs to check that the dependency bump didn't break anything, and that the repo has unit tests to catch it if it did. When a CI build fails because of a flaky external service, someone needs to retry it before it sits for a week.

In practice, those PRs pile up. Everyone is busy, and nobody has time to babysit automation.

Bob can do this work, and IBM developers already have access to it. Most of us just use it interactively: open a session, ask something, close the tab. In between sessions, the backlog keeps growing. I wanted to put Bob to work on that backlog while nobody was at their keyboard.

What the Job Does

The vulnerability job runs every night as an OpenShift CronJob, working through up to ten repositories per run.

First, it queries our Data Lake for repositories with critical or high severity vulnerabilities due this quarter. This gives Bob a prioritized list, focused on the repos that need attention now instead of every repo we own.

Next, Bob decides what to do with each repository: fix the vulnerabilities, or generate unit tests first. There's not much value in patching a repo with no test coverage, since you can't tell whether the fix broke something else. Bob assesses the repo's test quality and picks the right task before touching any code.

From there, Bob opens PRs for the work it completed and reviews PRs that our existing automation already opened, approving or flagging them based on whether the changes look safe. Once all the new PRs are open, Bob checks CI status across all of them and attempts fixes for any failing checks. Finally, it posts a Slack summary of everything that happened during the run.

Letting Bob Make the Call

What makes this different from a script is that Bob makes decisions along the way. A script runs the same commands in the same order every time. Bob has to recognize when a repo needs tests before a vulnerability fix means anything, tell a transient CI failure from a real one, and read a PR opened by another tool to decide whether it's safe to approve.

These are the small judgment calls that pile up in a developer's queue. Bob works through them overnight, and every decision ends up in the Slack summary. The work still needs a human review, but now there's something ready to review in the morning.

Skills Make It Reliable

The job only works because Bob knows how we do things. Out of the box, Bob doesn't know which vulnerability tool our projects use, how our Artifactory credentials work, what a passing CI pipeline looks like for us, or what our SonarQube coverage gate requires.

Skills fill that gap. Before Bob starts on a repository, it reads the relevant skill files. The vulnerability remediation skill walks through the audit, fix, and verify workflow I described in How to Remediate a Vulnerability, with the specific tools and flags our projects expect. The unit test generation skill defines what "done" means for us: not just that tests exist, but that they pass and coverage meets the gate.

Skills were helpful when I ran Bob interactively, but they matter even more unattended. When I'm watching, I can catch a wrong guess. Overnight, nobody is there to catch it.

Results

The job has been running for a few weeks now, and the backlog is moving. PRs that would have sat unreviewed for weeks get opened, checked, and often merged before anyone gets online. Repos that had no unit tests now have a baseline, and transient CI failures get retried automatically.

It's not perfect. Bob occasionally flags a PR that a human would approve outright, and some repos have quirks the skills don't cover yet. Still, every skill I add and every prompt I refine makes the next run a little better.

What's Next

Right now, the job handles vulnerabilities and unit tests. Next, I plan to broaden what it covers: picking up stale dependency updates, flagging repos with outdated base images, and opening an issue with a diagnosis when Bob finds a CI failure it can't fix.

Thanks to the infrastructure from the last post, adding these is straightforward. Each one is a new prompt wired into the orchestrator as another phase. Bob does the work overnight, and we review what it found in the morning. If you have access to Bob, it's worth thinking about what it could be doing while you're away from your keyboard.