How I Fixed a Performance Bug No One Believed Existed

The Bug That Wasn't Supposed to Be There

"It's probably your network."

That's what I heard for three weeks straight. Our API was slow—painfully slow—but only for certain users, only sometimes, and only in production. The kind of bug that makes you question your sanity.

I was the only one who believed it was real.

The Hook: When Fast Code Runs Slow

Our dashboard loaded in 200ms for most users. But for about 5% of our customers, it took 8-12 seconds. Same code. Same infrastructure. Wildly different results.

The support tickets were piling up. My manager suggested we "monitor it for another sprint." The senior engineer blamed client-side caching. DevOps pointed at AWS. Everyone had a theory. Nobody had proof.

I decided to find out why.

The Investigation: Following the Breadcrumbs

Clue #1: The Pattern That Wasn't Random

I started logging everything. Request IDs, user IDs, timestamps, response times. After two days of data collection, I noticed something:

Slow requests always came from users with account IDs ending in certain digits.

Not random. Not network issues. Something in our code was treating these users differently.

Clue #2: The Cache That Lied

We used Redis for caching user permissions. The cache hit rate was 99.8%—excellent, right?

I added more granular logging:

const cacheKey = `permissions:${userId}`;
const startTime = Date.now();
const cached = await redis.get(cacheKey);
const cacheTime = Date.now() - startTime;

logger.info({ userId, cacheTime, hit: !!cached });

The slow users? Their cache lookups took 6-8 seconds. The fast users? 2-5 milliseconds.

Same Redis instance. Same network. Different keys.

Clue #3: The Key That Grew

I examined the actual cached data:

redis-cli --bigkeys

Most permission objects were 2-5KB. But some were 45MB.

Forty. Five. Megabytes.

For a permissions object.

The Root Cause: Death by a Thousand Permissions

Here's what was happening:

Our permissions system allowed users to have granular access controls. Most users had 50-200 permissions. But some enterprise customers had created hundreds of thousands of individual permission entries—one for each file, folder, and resource they managed.

When we cached these permissions, we serialized the entire object:

// The problematic code
async function getUserPermissions(userId) {
  const cacheKey = `permissions:${userId}`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    return JSON.parse(cached); // Parsing 45MB of JSON = 💀
  }

  const permissions = await db.permissions.findAll({
    where: { userId }
  });

  await redis.set(cacheKey, JSON.stringify(permissions));
  return permissions;
}

The problem wasn't the cache miss—it was the cache hit. Deserializing 45MB of JSON took 6+ seconds. We'd optimized for the wrong thing.

The Fix: Restructuring the Cache Strategy

I implemented a three-tier approach:

1. Paginated Permission Checks

Instead of loading all permissions, we only loaded what we needed:

async function checkPermission(userId, resourceId, action) {
  const cacheKey = `perm:${userId}:${resourceId}:${action}`;
  const cached = await redis.get(cacheKey);

  if (cached !== null) {
    return cached === '1';
  }

  const hasPermission = await db.permissions.exists({
    where: { userId, resourceId, action }
  });

  // Cache individual permission checks (tiny payload)
  await redis.set(cacheKey, hasPermission ? '1' : '0', 'EX', 300);
  return hasPermission;
}

2. Bloom Filters for Quick Negatives

For users with massive permission sets, I added a Bloom filter:

async function mightHavePermission(userId, resourceId) {
  const bloomKey = `bloom:${userId}`;
  const exists = await redis.bf.exists(bloomKey, resourceId);

  if (!exists) {
    return false; // Definitely doesn't have permission
  }

  // Might have permission, check database
  return checkPermission(userId, resourceId);
}

3. Lazy Loading with Streaming

For admin dashboards that needed to show all permissions, I implemented streaming:

async function* streamPermissions(userId) {
  const batchSize = 100;
  let offset = 0;

  while (true) {
    const batch = await db.permissions.findAll({
      where: { userId },
      limit: batchSize,
      offset
    });

    if (batch.length === 0) break;

    yield batch;
    offset += batchSize;
  }
}

The Results: From 8 Seconds to 80 Milliseconds

After deploying the fix:

• P95 latency: 8.2s → 120ms
• P99 latency: 12.4s → 180ms
• Cache memory usage: Down 87%
• Support tickets: Zero in the following month

The "slow" users were now the fastest, because their permission checks were cached individually and hit Redis more efficiently.

The Lessons: What I Learned

1. Caching Can Make Things Slower

We assume caching always helps. But if you're caching the wrong thing (or too much), you're just moving the bottleneck. Deserializing a 45MB JSON blob is slower than a targeted database query.

2. Averages Hide Problems

Our average cache hit rate was 99.8%. Our average response time was 250ms. Both metrics looked great. But averages obscure outliers—and outliers are where users suffer.

Always look at P95, P99, and max values.

3. Production Data Tells Stories

I couldn't reproduce this locally. Our test data had 50 permissions per user. Production had users with 500,000.

The bug only existed at scale. Synthetic tests missed it entirely.

4. Trust Your Instincts (But Verify)

Everyone said it was network issues or client problems. I felt it was something else. But feelings aren't enough—I needed data. The logging I added was the difference between a hunch and a fix.

5. Granularity Matters

The original code treated all users the same. The fix recognized that different users have different needs:

• Small permission sets: Cache everything
• Large permission sets: Cache individual checks
• Admin views: Stream data

One size doesn't fit all.

The Takeaway: Be the Detective

Performance bugs are mysteries. You need:

• Curiosity: Why is this happening?
• Patience: Collect data, even when others doubt you
• Creativity: The solution isn't always obvious
• Humility: Your first theory will probably be wrong

The bug no one believed in was real. It was costing us customers. And it was fixable—once someone took the time to investigate.

Sometimes the best code you write isn't clever algorithms or elegant abstractions. It's the logging statement that reveals the truth.

---

What performance mysteries have you solved? The ones where everyone said "it's probably just X" but you knew better? I'd love to hear your detective stories.

P.S. — After this fix shipped, the senior engineer who blamed caching sent me a Slack message: "I was wrong about the cache. Nice work." Sometimes that's all the validation you need.