Putting Amazon Q to the Test: Can AI Really Build AI?
If AI can write code, could it one day build entire systems, or even other AIs? That question has been sitting in the back of my mind as AI tools continue to evolve. So I decided to find out the practical way. Using Amazon Q Developer, AWS’s generative AI development assistant, I gave it a real assignment: design, build, and deploy an analytics application using modern infrastructure and DevOps tooling, with as little human input as possible.
This wasn’t about writing a demo or spinning up a “Hello World” project. I wanted to treat Q like a genuine teammate and see what it could do under realistic conditions.
The Challenge: AI-Only Development
To keep the test fair, I treated the project like a real client engagement. I opened Q’s chat interface and toggled on the agentic coding feature with Claude 4.0, provided clear prompts, and stayed hands-off unless something broke. My goal was simple: see if AI could handle the end-to-end build, from architecture and infrastructure to code and deployment, without me stepping in.
Over the course of development, Q delivered a working MVP that spanned ten AWS services, including S3, Lambda, and DynamoDB. Infrastructure was built and deployed using AWS SAM, giving me reproducible, serverless architecture-as-code. The application logic was written in Python, infrastructure definitions in YAML, and deployment scripts in Bash, all generated or scaffolded by Q with minimal direction.
For CI/CD, Q configured GitHub Actions pipelines with secure credentials, testing steps, and automated deployments on every push to the main branch. The setup mirrored what we’d expect in a modern, production-grade workflow.
Architecture Diagram
Strengths and Surprises
across multiple languages, configured IAM roles correctly, and demonstrated a surprisingly coherent understanding of how AWS services fit together. In most cases, it produced usable code that required little to no rewriting. It was especially adept at scaffolding: spinning up Lambda functions, wiring them to DynamoDB, and creating the IAM permissions to make it all run smoothly.
But things didn’t always go according to plan. A few times, Q went off the rails, rewriting working handlers, misplacing files, or introducing logic that belonged to a completely different use case. In one case, it generated duplicate IAM roles with conflicting permissions; in another, it renamed key files without updating imports. Each time, I had to step in, diagnose the issue, and override the commit.
In total, the project saw twenty-nine commits. Twenty-five were fully AI-generated. Four were manual corrections I made to keep the project on track. That ratio tells a clear story: Q is capable of delivering meaningful progress on its own, but still depends on human judgment to spot when it’s heading in the wrong direction.
The core takeaway? Amazon Q Developer is immediately helpful even for experienced AWS engineers, but it’s not yet self-sufficient. Deploying complex architecture successfully requires patience, context, and experience. Q Developer can accelerate strong developers, but it’s not ready to replace them. For early-career engineers or non-technical users, it still demands too much debugging intuition and architectural awareness to be used independently.
Lessons from the Lab
This experiment wasn’t about speed. I wanted to test the limits of what generative tools like Q can actually do in a real-world setting. And on balance, the experience was encouraging. Q handled most of the heavy lifting, and when it failed, the fixes were straightforward. Still, it was clear that a purely AI-driven approach wasn’t the most efficient path.
If I’d worked with Q instead of trying to let it run solo, using it as a collaborative partner rather than a pilot, I likely would have saved time, avoided regressions, and improved the overall quality of the build. I’d give the experience a 7.5 out of 10: impressive output, occasionally frustrating, but absolutely useful.
What This Means for the Future
Tools like Q aren’t replacing developers, they’re changing what development looks like. Engineers are shifting from authors to reviewers, from typing code to orchestrating systems that write it for them. For tasks like scaffolding infrastructure, building CRUD workflows, or configuring pipelines, tools like Q are already delivering measurable value.
At Red Oak Strategic, we’re watching this space closely. Not as novelty, but as serious tooling that will shape how we build, deploy, and maintain solutions across our DevOps and data science practices. The future isn’t AI versus developers, it’s AI with developers. And the teams who learn to collaborate with these tools effectively will move faster, ship cleaner, and spend more time solving real problems.
If you’re curious what AI-assisted delivery could look like for your organization, or want to see what happens when you hand the keyboard to an AI, let’s talk.