In partnership with

🚀 Day 66 - Agentic AI x LinkScopic Data Stacks

Fixing Elasticsearch Growth Before It Breaks the System

Today was one of those days where the infrastructure pushes back.

While running our normal workflows inside the LinkScopic data stacks, we started noticing duplication issues and search instability inside our Elasticsearch cluster.

Shortly after that, things escalated.

The ES cluster flipped to YELLOW status, showing 99 unassigned replica shards and causing Kibana search failures across the environment.

When you're dealing with massive product datasets and continuous ingestion pipelines, these kinds of issues can snowball quickly if they aren't addressed at the root level.

So today became a deep infrastructure day.

🧠 Codex CLI vs Claude Code

One thing became very clear during this process.

Codex CLI absolutely shines for deep engineering work.

When we needed to debug infrastructure, identify growth vectors, and implement a full remediation plan, Codex CLI proved incredibly effective.

Claude Code is still excellent, but its strength is different.

Claude Code is fantastic for:

• business process logic

• documentation workflows

• structured analysis

• operational planning

But when it comes to low-level engineering tasks, Codex CLI simply moves faster and thinks more structurally about the problem space.

Having both tools available has become a huge advantage.

⚠️ The Elasticsearch Issue

The cluster status changing to YELLOW indicated that Elasticsearch could not allocate replica shards properly.

The result:

• 99 unassigned replica shards

• search failures in Kibana

• cluster instability during ingestion

If left unresolved, this kind of issue can eventually cascade into:

• degraded query performance

• ingestion bottlenecks

• potential data loss risk

🔎 Root Cause Analysis

After digging into the system, we identified four separate growth vectors contributing to the issue.

These were causing disk pressure and shard allocation failures inside Elasticsearch.

Rather than applying temporary fixes, we decided to implement a comprehensive ES disk management and cleanup plan.

⚙️ Infrastructure Fixes Implemented

The remediation plan focused on several core areas:

Disk cleanup

Removing stale and duplicated data that had accumulated during ingestion testing.

Shard management

Reducing excessive shard counts and ensuring replicas could properly allocate across the cluster.

Duplication prevention

Implementing safeguards inside the ingestion pipelines to prevent duplicate records from inflating dataset size.

Long-term disk strategy

Creating a plan to ensure disk growth stays predictable as the datasets continue expanding.

Each fix was implemented with atomic acceptance criteria, ensuring every change could be validated before moving forward.

🧠 What Days Like This Actually Mean

Days like today are a reminder that building data infrastructure isn’t just about adding features.

Sometimes it's about:

• protecting system stability

• preventing silent data corruption

• managing scale before it becomes a problem

Because once your infrastructure grows to the size we're operating at, small issues can become massive ones very quickly.

🔮 Moving Forward

With the cluster stabilized and disk management improvements in place, the LinkScopic data stacks are now positioned to continue scaling safely.

And as our datasets continue expanding across retailers and marketplaces, these kinds of infrastructure improvements become just as important as the intelligence layers built on top of them.

Because great AI systems are only as strong as the data infrastructure beneath them.

Support our newsletter by checking out the sponsor below. Your click helps keep our subscription fees as low as possible.

When it all clicks.

Why does business news feel like it’s written for people who already get it?

Morning Brew changes that.

It’s a free newsletter that breaks down what’s going on in business, finance, and tech — clearly, quickly, and with enough personality to keep things interesting. The result? You don’t just skim headlines. You actually understand what’s going on.

Try it yourself and join over 4 million professionals reading daily.

Keep Reading