Your Infrastructure Is Lying to You

Somewhere between the third hotfix and the fifth manual config change, your production environment stopped matching what your team thinks it looks like. The Terraform state says one thing. The running cluster says another. The engineer who made the change at 2 a.m. during an incident forgot to commit it, and now nobody is quite sure which version of the truth is correct.

This is the problem GitOps was designed to solve — not with more tooling for its own sake, but with a foundational shift in how infrastructure changes flow. It is a workflow that treats your Git repository as the single, authoritative source of truth for everything running in your environment. When it works well, it does not just automate deployments. It keeps your infrastructure honest.

What GitOps Actually Is (and What It Is Not)

GitOps is a set of practices where the desired state of your infrastructure and applications is declared in code, stored in Git, and continuously reconciled by an automated agent. If someone changes something in the live environment without updating the repository, the system notices — and corrects the drift.

The term was coined by Weaveworks in 2017, but the principles underneath it are older than the label. Declarative infrastructure, version-controlled configuration, automated reconciliation — these ideas have been building in the DevOps community for years. GitOps gives them a coherent framework.

A few things GitOps is not:

• It is not just CI/CD. A CI/CD pipeline pushes changes forward. GitOps adds a pull-based reconciliation loop that also detects and corrects unauthorized changes. The distinction matters.
• It is not limited to Kubernetes. Kubernetes is the most common target, but the pattern applies anywhere you can declare desired state and reconcile against it.
• It is not a specific product. Tools like Flux CD and Argo CD implement GitOps principles, but adopting a tool without adopting the workflow underneath it gives you complexity without the payoff.

The Four Principles That Make It Work

GitOps rests on four grounded principles. Understanding them matters more than choosing between Flux and Argo.

1. Declarative Infrastructure

Every component — from cluster configuration to application manifests to network policies — is defined declaratively. You describe what should exist, not the steps to get there. This is foundational. If your infrastructure is not declarative, you cannot meaningfully reconcile it.

In practice, this means YAML or HCL files checked into a repository. No snowflake servers. No configurations that only exist as tribal knowledge in someone's head.

2. Git as the Source of Truth

The desired state of your system lives in Git. Not in a dashboard. Not in a wiki. Not in the memory of the person who set it up two years ago. Git gives you versioning, audit trails, branching, and code review — all the discipline software engineering has spent decades building, applied to your infrastructure.

Every change goes through a pull request. Every pull request gets reviewed. Every merge triggers reconciliation. The commit history becomes your change log, your audit trail, and your rollback mechanism in one place.

3. Automated Reconciliation

This is where GitOps diverges from traditional CI/CD. An agent running inside your environment continuously compares the live state against the declared state in Git. When they diverge — whether because of a new commit or because someone made a manual change — the agent brings the environment back in line.

Flux CD handles this with a set of controllers that watch your Git repositories and automatically apply changes to your cluster. You commit a manifest update, and Flux reconciles it within minutes, without a separate deploy step.

4. Continuous Observation

The reconciliation loop is not a one-time sync. It runs continuously, surfacing drift and ensuring that what is declared is what is running. This is where the honesty comes in. Your environment cannot quietly deviate from its declared state without the system noticing.

Why This Matters More Than You Think

If you are managing cloud infrastructure at any real scale — multiple services, multiple environments, a team of more than one — the benefits of this workflow compound quickly.

Drift Detection Is Built In

Configuration drift is one of the most common sources of production incidents. An engineer applies a quick fix directly to a running environment. It works. They move on. Three months later, a deploy overwrites that fix because it was never captured in code, and now the team is debugging an issue they thought was already solved.

GitOps eliminates this category of problem. The reconciliation agent treats any undeclared change as drift and corrects it. If the change was intentional, it belongs in the repository. If it was not, it gets reverted. Either way, the declared state remains authoritative.

Rollbacks Become Trivial

When every change is a Git commit, rolling back is a git revert. No hunting through deployment logs. No trying to reconstruct what changed. The previous known-good state is right there in the commit history, and the reconciliation agent will apply it just as reliably as it applied the change that broke things.

Audit Trails Are Automatic

For teams operating under compliance requirements — SOC 2, HIPAA, PCI DSS — the question of who changed what, when, and why is not optional. GitOps answers it by default. The pull request history captures the change, the author, the reviewer, and the reasoning. No separate change management process required.

Developer Experience Improves

Engineers already know how to use Git. They already know how to open pull requests and review code. GitOps does not ask them to learn a new deployment tool or memorize a new set of CLI commands. The deploy workflow is the same workflow they use for application code. This lowers friction and reduces the chance of mistakes during high-pressure moments.

The Trade-Offs Nobody Mentions in the Sales Pitch

GitOps is not free. The workflow has real costs and real friction points, and pretending otherwise does a disservice to anyone evaluating it seriously.

Repository Structure Requires Thought

How you organize your Git repositories matters enormously. A monorepo with application code and infrastructure manifests mixed together creates coupling that slows teams down. Separate repos for app code and deployment manifests provide cleaner separation of concerns but add coordination overhead.

There is no universally correct answer. The right structure depends on your team size, your release cadence, and how many environments you manage. Getting this wrong early creates friction that compounds over months.

Secret Management Needs a Real Solution

Secrets do not belong in Git, even encrypted. GitOps requires a deliberate approach to secret management — tools like Sealed Secrets, SOPS, or external secret operators that reference a vault rather than storing sensitive values in the repository. This is solvable, but it is not solved by GitOps itself. Teams that adopt GitOps without addressing secrets end up with either insecure repositories or a workflow that breaks every time a credential rotates.

The Learning Curve Is Real

For teams accustomed to imperative deployments — running scripts, clicking buttons in a console, applying changes manually — the shift to declarative, pull-based reconciliation requires a genuine change in mental model. The technology is not the hard part. The discipline is.

Multi-Tenancy and Multi-Cluster Add Complexity

GitOps at the single-cluster level is relatively straightforward. Scaling it across multiple clusters, multiple regions, and multiple teams introduces questions about repository access, reconciliation scope, and environment promotion strategies that do not have simple answers. Flux CD handles multi-tenancy through its tenant isolation model, but designing that architecture well requires experience with the failure modes.

Where Flux CD Fits

Flux CD is a CNCF graduated project that implements GitOps natively on Kubernetes. It runs as a set of controllers inside your cluster, each responsible for a specific part of the reconciliation loop — watching Git sources, building Kustomize overlays or Helm releases, and applying the results.

A few things make Flux particularly well-suited for production GitOps workflows:

• It is pull-based by design. Flux runs inside the cluster and pulls changes from Git, which means your CI system never needs direct access to your cluster API. This is a meaningful security boundary.
• It handles dependency ordering. Flux lets you define dependencies between resources, so your database migration runs before your application deployment, and your namespace exists before anything tries to deploy into it.
• It integrates with notification systems. Flux can send alerts to Slack, Teams, or webhook endpoints when reconciliation succeeds, fails, or detects drift. This feeds directly into your observability stack.
• It supports progressive delivery. Combined with Flagger, Flux can manage canary deployments and automated rollbacks based on metrics — turning your GitOps workflow into a deployment safety net.

Getting Started Without Overengineering

The biggest mistake teams make with GitOps is trying to adopt everything at once. A more grounded approach:

1. Start with one environment. Pick your staging or development cluster. Get the workflow right before touching production.
2. Declare what you already have. Export your existing manifests, commit them, and let the reconciliation agent manage them. Do not try to redesign your architecture and adopt GitOps simultaneously.
3. Establish the pull request discipline. No changes without a PR. No exceptions. The value of GitOps collapses the moment you allow bypass.
4. Add observability early. Set up alerts for reconciliation failures and drift detection from day one. A GitOps agent that fails silently is worse than no agent at all.
5. Expand deliberately. Once the workflow is proven in one environment, extend it to production, then to additional clusters, then to policy enforcement.

This is soil work — not glamorous, but it is the foundation that lets everything above it thrive.

The Deeper Principle

GitOps is ultimately about integrity in the engineering sense of the word: the state of being whole, consistent, and structurally sound. Your declared state matches your running state. Your history is complete. Your changes are traceable. Your rollbacks are reliable.

That kind of integrity does not happen by accident. It is engineered, deliberately, by teams that value getting the foundation right before building upward.

Build the Foundation Right

If your infrastructure has drifted from what your team believes is running — or if you are scaling into multiple environments and want to architect a GitOps workflow that holds under real pressure — that is exactly the kind of problem Figtree Development was built to solve. Jason brings over fifteen years of experience designing and managing cloud environments, and every engagement starts with a Discovery and Strategy process to understand what you actually need before recommending what to build.

Book a free 20-minute discovery call to talk through where your infrastructure stands today and what a scalable, declarative foundation could look like for your team.