For this project, I used IBM Bob. Bob is an AI-powered development partner that helps write and manage code. This is my first experience using AI powered coding tools, and I'd like to share what I've learned. If you're new to using AI at work, you might find these tips useful in your projects too.

Start with learning about the project

The most important thing to tackle early is understanding the project you're working on. Since I was new to the team, I didn't have any context around the history of the projects, and I didn't even know what most of these services did. To make matters worse, most of the projects were lacking any documentation. Before changing any code, I asked Bob to document the project so I could begin to understand what the service did and how it fit in the overall stack.

Here's the real prompt I used during this stage:

Task 1: Documentation

Requirements

The repository must have a concise, newcomer-friendly README.md that includes:
 • What this repository does (1–2 paragraphs)
 • Technologies used
 • High-level architecture (use Mermaid diagrams)
 • How to run locally (prerequisites + commands)
 • How tests are run
 • How this service is deployed (if applicable)

If the README is missing or incomplete:
 • Create or rewrite it
 • Keep it concise and skimmable
 • Assume the reader has never seen this repo before

This resulted in documentation that I could easily consume and get an overall understanding of the project. This understanding helps guide the follow-up steps.

Plan mode is your friend

Plan mode is available on many AI tools now, and I found it to be incredibly useful for this project. By using plan mode, you give the AI opportunity to think through all changes before just jumping into coding. This also helped me think through exactly what I wanted to accomplish. Practically, using plan mode means you're less likely to find the AI gets confused or misinterpreted your prompt.

During this project, I followed a workflow:

1. Craft what I thought was a great prompt and ask the AI to create a phased plan for implementation. Write the plan to markdown.
2. In a loop, as the AI to implement the phases and update the plan with status. For each implementation phase, I started a new task to clear context.
3. Commit the changes in between tasks to save state.

If the plan needed adjusting, there was opportunity to do so between tasks. However, if things got too off track, I found it easier to scrap the implementation in progress and go back to planning from the beginning.

Don't overwhelm your reviewers

The same best practices we used before AI generated code are still important. For this project, that meant splitting my changes into multiple pull requests so I could keep the size of the changes small. I decided to organize the changes by level of risk. First, documentation only changes, testing improvements next, followed by code changes. This makes for two pull requests that should be very easy to review, and a third that is now well documented and tested to aid reviewers.

At IBM, we don't have any AI tools specifically created for reviewing pull requests yet, but these would help. I've experimented with using the same tool that writes code to review it, but there is high risk with this approach: using the same model could mean making the same mistakes. If available, a tool for reviewing AI generated code could improve the bottleneck of pull requests.

Don't boil the ocean

I had goals at the beginning of the project to improve documentation and testing, and I had to restrain myself from rewriting entire services. AI tools can make it so easy to quickly make large changes, which feels so powerful in the moment. However, in my case, these are production services and any changes require careful reviews. Even though it is easy to make sweeping changes, I'm not an expert in the project and reviews would be lengthy. Instead, set targeted goals and stick to them.

In the end, using generative AI didn’t change the fundamentals of good engineering. Clear goals, thoughtful planning, small and reviewable changes, and a solid understanding of the codebase still mattered just as much as before. Tools like Bob helped me move faster and build confidence as I improved unfamiliar services, but the real value came from using AI deliberately and not letting it run unchecked. If you’re new to AI-powered coding tools, start small, stay disciplined, and treat them as a partner.

1. Start with learning about the project
2. Set standards early
3. Plan mode is your friend
4. A consistent prompt makes for maintainable projects
5. Don't overwhelm your reviewers
6. Before changing any code, add tests
7. Use AI to look for inconsistencies
8. Don't boil the ocean

Leadership Behaviors

Dinesh Nirmal kicked off the event with an opening session focused on key behaviors of a technology leader. I found this session insightful, especially as I compare with the behaviors described in The Geek Way, which is constantly being referenced by our senior leadership.

Here are the six key behaviors Dinesh highlighted:

• Stay Curious
• Move Toward the Unknown
• Collaborate at Speed
• Simplify Relentlessly
• Lead With Data
• Persist With Purpose

Of these, Lead With Data stands out especially. There's strong overlap with the science norm discussed in The Geek Way here. Dinesh points out that we should be gathering data in order to make better decisions, but we can go further too. It's not just what the data says, but it's what we do with it.

"AI Assistants" Are the Past - How Are "Agents" Shaping the Future?

This session was the one I found to be the most exciting. The talk focused on new capabilities recently added to watsonx Orchestrate, particularly the agentic AI features. These agents are powered by the think-act-observe loop and have the ability to interact with other systems, making them much more dynamic than a chat bot. In addition, the team has released the Agent Developer Kit (ADK) that can be used for building and deploying tools and agents to Orchestrate. There's great potential now for a CI/CD pipeline to build, test and deploy agents to Orchestrate with the ADK.

Many of the other sessions were also focused around AI and ways to improve our workflows. I'm looking forward to experimenting with the ADK and seeing where agent-based workflows could add value for my team.

By weaving security into each stage of your pipeline, you can catch issues early, reduce risk, and build trust in every release. Here, I'll share some of what I learned from building secure pipelines for tens of thousands of repositories.

Security Scans

Security scanning is one of the easiest and most impactful improvements you can make. These scans will find issues and vulnerabilities in your code that you can then either fix or improve. There are many different types of scans, each with their own focus. Here are the most important types to consider.

Software Composition Analysis

Software composition analysis (SCA) focuses on the open source dependencies included in your project. These scanners can identify dependencies and their vulnerabilities, giving insight into areas that might need updates. In addition, these tools can create a software bill of materials (SBOM), which will be very useful for the Security Gates section.

In my experience, I've implemented this type of scan in two ways: directly scanning the application after installing dependencies and scanning the built container image. The container image will give a more complete scan, including vulnerabilities coming from any base image in use. A scan earlier in the pipeline will be more focused, giving a high confidence that any findings are a direct result of the application.

Trivy is a great open source scanning tool in this area.

Detecting Secrets

Sometimes developers accidentally add secrets to their GitHub repository. I use an open source tool called Detect Secrets to help discover when this has happened as part of the pipeline. It isn't ideal, because if this tool finds anything then the secret has already leaked, but it can identify the problem so that it can be resolved immediately.

Static Analysis

Static analysis is a scan of your source code that can recommend improvements as well as find vulnerabilities present. This should be run before the artifact is built, and can result in a variety of recommendations. The results of this scan can be output in the SARIF format, which is a machine readable output that can be persisted for analysis.

Dynamic Analysis

Dynamic analysis is another scanning option, but this scan runs against your deployed application. It does not have access to the source code of the application, and instead will attempt to find vulnerabilities by probing the actual application. Therefore, dynamic analysis should be run after deployment only in non-production environments.

Zed Attack Proxy is an open source option for running dynamic security testing.

Signing Build Artifacts

All build artifacts should be signed with a key that is only available to your CI system. This will prove that any build artifact with this signature was created by your secure build system, and included all of the security testing that is expected.

While all build artifacts should be signed, if you're creating container images cosign is a great open source tool for signing the image.

Signature Verification

While not part of the CI/CD pipeline itself, your deployment environments should be configured to validate the signature of everything deployed before it is run. Without this step, signing the images is a nice addition but not enforced for protection.

Audit Readiness

With all of these new security tasks added to your pipeline, you have a lot of data about the applications being built. This includes all of the dependencies, vulnerability data, versions, and information about the specific changes made. Saving this data can make audits run more smoothly, giving confidence that vulnerabilities are taken seriously and quickly resolved.

Your results from SCA will be among the most valuable. These should result in an SBOM, and using a standard format will make further analysis more straightforward. CycloneDX is a popular format for this requirement.

With this information saved, you can quickly show your developers which dependencies are responsible for vulnerabilities, even if they are not direct dependencies.

Security Gates

The final improvement to your pipelines is to apply policies to prevent risky changes from deploying. For example, if you would like to block all releases with a critical vulnerability, you can do so by running a policy task after the scans are completed. A great open source tool for applying policy checks is the Open Policy Agent.

These additions take your pipeline from simple CI/CD to an advanced pipeline ready to securely deploy changes.

CI/CD Essentials

First, the basics. CI/CD is all about giving fast feedback to developers, enabling more frequent deployments to production. To facilitate this, a collection of basic CI tasks can be especially helpful.

Dependency Install

As part of CI, you should be installing dependencies (preferably from a lock file) to validate the correct configuration. Because dependencies are often external to the codebase, it is likely necessary to install these dependencies for the other CI tasks.

Compilation

If your language supports compilation (like Java, Go, or Rust), it is one of the most straightforward tests available. Running compilation tasks as part of CI will ensure the code builds cleanly.

Unit Testing

Every CI pipeline needs to have unit testing to give quick feedback to your developers and detect any regressions in their code changes. Unit tests aren't perfect – they'll only catch the problems you're looking for – but they can be a great tool for establishing a baseline quality.

Linting

A lint task in the CI pipeline will be useful if there are code style standards to maintain. Not every application will require this, but it is a great addition to catch when someone is adding tabs to a codebase that uses spaces.

Versioning

Each build needs a unique version to support traceability, debugging and rollback. If appropriate for the language, Semantic Versioning is a great way to manage versions. The CI system should generate one automatically if developers do not set a version explicitly.

Packaging

A packaging step gives developers an opportunity to move files in preparation for publishing the build artifact. This simple step ensures the artifact has only the essentials included.

Publishing

The culmination of the CI pipeline results in a build artifact being published. These days, this is typically a container image pushed to a registry, but it could also be a JAR file published to a remote repository, or even just a collection of files ready to be moved to a server.

Deploying

There are a multitude of different options for deployments, but I like to deploy feature branches to dedicated test environments and deploy the main branch to a persistent non-production environment followed by production. This allows developers to test their changes in an isolated environment before they merge, plus protects the production environment from any changes that do not pass tests in preprod.

Deployment Verification

Finally, once the deployment is complete it is useful to run some tests to validate the change. These could include end-to-end or smoke tests, but should run quickly to determine if there are any issues with the application. This could be as simple as using curl to validate the deployment, or running Cypress for more advanced tests.

With these building blocks in place, your CI/CD pipeline becomes a reliable foundation for shipping software. Next, we’ll look at how to level up — with security scans, image signatures, and policy enforcement.

Local Environment Setup

In order to configure your local development environment, you’ll first need to install a few tools. 

• Python: You probably already have Python, but a fairly modern version is required. If you need to install Python, I have found pyenv to be very helpful. I’m using Python version 3.11 locally.
• Poetry: Poetry is a tool that helps manage Python dependencies and automatically manages virtual environments. I’m using Poetry version 2.1.1.
• Java: Spark requires Java in order to run, and I needed to update to a newer version for this application. I used home-brew to update to openJDK 17.

From this point, there’s just a bit more additional setup required. I like to use the poetry shell command to run a subshell and activate the correct virtual environment for the project, but Poetry does not ship with this plugin installed. We can add it by running Poetry self add poetry-plugin-shell. If your project already uses Poetry, you can install the dependencies with the Poetry install command. If you’re starting a new project, you can run Poetry new <project> and be sure to add PySpark as a dependency.

Next, if you updated your Java version, make sure that version is on the PATH properly. I ran this command:

echo 'export PATH="/usr/local/opt/openjdk@17/bin:$PATH"' >> ~/.zshrc

Managing Iceberg Dependencies

In order to use Iceberg with Spark, you’ll need to add the correct JAR file when running your spark commands. This documentation from Iceberg outlines which JAR should be used and how to configure Spark correctly. In addition, this page documents which of the available JARs are still supported.

When running locally, it is important to pass in the JAR files when running the script. You can do this with either the --packages or --jars flag. The packages flag has the ability to automatically download (and cache) the specified JAR and can be easier to use. Alternatively, you can download the Iceberg JAR yourself and pass it in with the jars flag.

Once you’ve added the JAR files, you also need to configure the Iceberg catalog. You can do this either though the command line options or by adding config options when you initialize your Spark session. This Iceberg documentation describes the configuration options available. For my local testing, I’ve used these options:

spark.sql.catalog.local = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.local.type = hadoop
spark.sql.catalog.local.warehouse = spark-warehouse/iceberg

Environment Variables

When running locally, it is important to make sure you also set some environment variables correctly. These are documented here. Of these, these are the ones that I've found are most important:

With this configuration done, you should now be able to use Iceberg with Spark locally.

Creating Tables

The best way to test this configuration is to try to create some tables for use locally. Start by initializing Spark using the Iceberg configurations:

# Initialize Spark session with Iceberg configurations
spark:SparkSession = SparkSession.builder \
.appName(os.getenv('APP_NAME')) \
.config('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2') \
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.local.type", "hadoop") \
.config("spark.sql.catalog.local.warehouse", "spark-warehouse/iceberg") \
.getOrCreate()

You can use this object to run SQL commands. spark.sql(sqlQuery) will run whatever SQL command you pass in. When working with Iceberg, you can reference this page for the valid DDL. Pay special attention to the Spark to Iceberg type conversion that will automatically take place as well.

Try creating a table with:

spark.sql(f"CREATE TABLE {APP_NAME}.db.sample (id bigint NOT NULL COMMENT 'unique id', data string) USING iceberg;")

To write data to the table, you can also use the spark.sql() function. The options for writing to the Iceberg table are outlined here. For a simple test, you can try to directly insert into the table with:

spark.sql("INSERT INTO local.db.sample VALUES (1, 'a'), (2, 'b')");

You can also verify that everything is working correctly by checking your filesystem to confirm the warehouse was created with the correct tables.

Append data using a DataFrame when you’re ready to write actual data. You can read more about that here.

Querying Tables

To double check that your data is being written properly, you can try to query the Iceberg table. Again, you can query with simple SQL commands using the spark.sql() function. You can check for the data you’ve just written by running:

spark.sql("SELECT * FROM local.db.sample;").show();

This will print the rows that you’ve written to the table so far. When you’re ready, you can also query using DataFrames.

Summary

Now you can create local Iceberg tables with PySpark, as well as read and write data to the tables. If you’re using Iceberg in production, this will help you learn about how it works, as well as allow you to test all changes locally.