5 MongoDB Mistakes That Cost Me $10K in AWS Bills

How I Learned Query Optimization the Hard Way (So You Don't Have To)

I'll never forget the morning I opened my AWS billing dashboard and saw $10,247.83 for a single month. My coffee went cold as I stared at the screen, convinced there had been a mistake.

There wasn't.

My MongoDB deployment had spiraled out of control, and I had no one to blame but myself. What started as a promising SaaS product with growing user adoption had turned into a financial nightmare—all because I made five critical mistakes that any developer could make.

The good news? I fixed them, cut my bills by 87%, and learned lessons that transformed how I approach database optimization. Let me share these mistakes with you so you can avoid the same expensive education I got.

Table of Contents

1. Missing Indexes on High-Traffic Queries
2. Using $lookup Without Understanding the Cost
3. Fetching Entire Documents When I Only Needed Fields
4. Ignoring Connection Pool Configuration
5. Not Monitoring Slow Queries Until It Was Too Late

---

1. Missing Indexes on High-Traffic Queries

The Mistake That Started It All

My application had a simple user dashboard that displayed recent activities. The query looked innocent enough:

db.activities.find({ 
  userId: "12345", 
  createdAt: { $gte: new Date("2024-01-01") } 
}).sort({ createdAt: -1 }).limit(20);

This query ran thousands of times per hour. Without proper indexes, MongoDB was performing collection scans on a table with 50 million documents. Each query took 3-5 seconds and consumed massive amounts of CPU and memory.

The Fix

I created a compound index that matched my query pattern:

db.activities.createIndex({ 
  userId: 1, 
  createdAt: -1 
});

The Impact

Key Lessons

• Always index your query filters: If you're filtering or sorting by a field, it probably needs an index
• Compound indexes matter: The order matters—put equality matches first, then sort fields
• Use explain(): Run .explain("executionStats") on your queries to see if they're using indexes

// Check if your query uses an index
db.activities.find({ userId: "12345" })
  .sort({ createdAt: -1 })
  .explain("executionStats");

// Look for "IXSCAN" (good) vs "COLLSCAN" (bad)

---

2. Using $lookup Without Understanding the Cost

When Joins Become Expensive

Coming from a SQL background, I loved that MongoDB supported joins via $lookup. I built a feature that displayed user profiles with their associated posts, comments, and likes—all in one aggregation pipeline.

db.users.aggregate([
  { $match: { _id: userId } },
  {
    $lookup: {
      from: "posts",
      localField: "_id",
      foreignField: "authorId",
      as: "posts"
    }
  },
  {
    $lookup: {
      from: "comments",
      localField: "_id",
      foreignField: "userId",
      as: "comments"
    }
  },
  {
    $lookup: {
      from: "likes",
      localField: "_id",
      foreignField: "userId",
      as: "likes"
    }
  }
]);

This single query was costing me $2,800/month in compute resources.

Why $lookup Is Expensive

• Each $lookup performs a separate query operation
• Without indexes on foreign keys, it scans entire collections
• Multiple lookups multiply the cost exponentially
• Memory usage spikes when joining large datasets

The Better Approach

I redesigned my data model to embed frequently accessed data and made separate, optimized queries:

// Denormalize: Store post count directly on user document
db.users.updateOne(
  { _id: userId },
  { $inc: { postCount: 1 } }
);

// Make separate, indexed queries when needed
const user = await db.users.findOne({ _id: userId });
const recentPosts = await db.posts
  .find({ authorId: userId })
  .sort({ createdAt: -1 })
  .limit(10);

Cost Comparison

---

3. Fetching Entire Documents When I Only Needed Fields

The "Just Get Everything" Anti-Pattern

I was lazy. Instead of specifying which fields I needed, I'd just fetch entire documents:

// Bad: Fetching 2MB documents when I only need 2 fields
const users = await db.users.find({ status: "active" });

My user documents contained profile pictures (stored as base64), full activity histories, and nested preference objects. Each document averaged 2.1 MB. I was transferring gigabytes of unnecessary data across the network.

The Simple Fix

Use projection to fetch only what you need:

// Good: Fetch only required fields
const users = await db.users.find(
  { status: "active" },
  { projection: { name: 1, email: 1, _id: 1 } }
);

Real Numbers

For a query that returned 1,000 users:

Pro Tips for Projections

// Exclude heavy fields instead of including light ones
db.users.find(
  { status: "active" },
  { projection: { profileImage: 0, activityHistory: 0 } }
);

// Use projections in aggregation pipelines
db.users.aggregate([
  { $match: { status: "active" } },
  { $project: { name: 1, email: 1 } }
]);

// Create covering indexes (index contains all queried fields)
db.users.createIndex({ status: 1, name: 1, email: 1 });

---

4. Ignoring Connection Pool Configuration

The Connection Pool Disaster

My Node.js application was creating new MongoDB connections for every request. Under load, I had thousands of open connections, each consuming memory and resources on both the application and database servers.

// Bad: Creating new connection each time
async function getUser(id) {
  const client = await MongoClient.connect(uri);
  const user = await client.db().collection('users').findOne({ _id: id });
  await client.close();
  return user;
}

MongoDB Atlas was auto-scaling to handle the connection load, costing me $1,200/month in unnecessary cluster upgrades.

The Right Way

Configure a proper connection pool once at application startup:

// Good: Reuse connection pool
const client = new MongoClient(uri, {
  maxPoolSize: 50,
  minPoolSize: 10,
  maxIdleTimeMS: 30000,
  serverSelectionTimeoutMS: 5000,
  socketTimeoutMS: 45000,
});

await client.connect();

// Reuse throughout your application
async function getUser(id) {
  return await client.db().collection('users').findOne({ _id: id });
}

Optimal Pool Size Formula

I use this formula to determine pool size:

maxPoolSize = (Number of App Instances × Expected Concurrent Requests) / 2

For my setup:

• 4 application instances
• ~100 concurrent requests per instance
• maxPoolSize = (4 × 100) / 2 = 200

Connection Pool Best Practices

---

5. Not Monitoring Slow Queries Until It Was Too Late

Flying Blind

For the first six months, I had zero visibility into my database performance. I didn't know which queries were slow, which indexes were unused, or where my money was going.

The wake-up call came when users started complaining about timeouts, and by then, the damage was done.

The Monitoring Stack I Should Have Had from Day One

1. Enable MongoDB Profiler

// Profile queries slower than 100ms
db.setProfilingLevel(1, { slowms: 100 });

// Check slow queries
db.system.profile.find()
  .sort({ ts: -1 })
  .limit(10)
  .pretty();

2. Use MongoDB Atlas Performance Advisor

Atlas's built-in Performance Advisor identified:

• 12 missing indexes
• 8 inefficient query patterns
• 3 unused indexes consuming resources

3. Set Up Custom Alerts

// Example: Alert on slow queries using MongoDB Atlas API
{
  "eventTypeName": "OUTSIDE_METRIC_THRESHOLD",
  "metricName": "QUERY_TARGETING_SCANNED_OBJECTS_PER_RETURNED",
  "operator": "GREATER_THAN",
  "threshold": 1000,
  "notifications": [
    {
      "typeName": "EMAIL",
      "emailAddress": "alerts@myapp.com"
    }
  ]
}

Monitoring Metrics That Matter

Tools I Now Use Daily

1. MongoDB Atlas Monitoring: Built-in dashboards and alerts
2. Datadog MongoDB Integration: Custom dashboards and anomaly detection
3. Custom Query Logger: Logs all queries > 50ms to CloudWatch

// Simple query performance logger
const originalFind = Collection.prototype.find;
Collection.prototype.find = function(...args) {
  const start = Date.now();
  const cursor = originalFind.apply(this, args);

  cursor.toArray = async function() {
    const results = await originalToArray.apply(this);
    const duration = Date.now() - start;

    if (duration > 50) {
      console.warn(`Slow query detected: ${duration}ms`, {
        collection: this.namespace.collection,
        filter: args[0]
      });
    }

    return results;
  };

  return cursor;
};

---

FAQ

How much can proper MongoDB optimization really save?

In my case, I reduced costs by 87%—from $10,247 to $1,331 per month. Most applications can expect 60-80% savings with proper indexing, query optimization, and connection pooling alone.

What's the fastest way to identify missing indexes?

Use MongoDB's explain() method on your most frequent queries. Look for COLLSCAN in the execution plan—that's a red flag. MongoDB Atlas Performance Advisor also automatically suggests missing indexes.

Should I always avoid $lookup operations?

Not always, but use them sparingly. For frequently accessed data, denormalization is often better. Reserve $lookup for admin dashboards or reports where real-time joins are necessary and query frequency is low.

How do I know if my connection pool is sized correctly?

Monitor your connection pool metrics. If you're consistently hitting maxPoolSize, increase it. If most connections are idle, decrease it. A healthy pool uses 40-70% of available connections under normal load.

What's the single most impactful optimization I can make today?

Add indexes to your most frequent queries. Run this command to see your slowest operations: db.currentOp({ "secs_running": { "$gt": 1 } }). Index the fields those queries are filtering and sorting on.

---

Key Takeaways

• Index everything you query: Missing indexes were responsible for 40% of my costs. Use compound indexes that match your query patterns exactly.

• Avoid $lookup in hot paths: Denormalize data for frequently accessed relationships. Save joins for admin features and reports.

• Use projection religiously: Fetching only needed fields reduced my data transfer costs by 95%. Never fetch entire documents unless you need them.

• Configure connection pools properly: One shared connection pool per application instance. Size it based on your concurrency needs, not arbitrary numbers.

• Monitor from day one: Enable slow query logging, set up alerts, and review performance weekly. Problems caught early cost pennies; problems caught late cost thousands.

• Use explain() liberally: Before deploying any query to production, run .explain("executionStats") to verify it's using indexes efficiently.

• Think in MongoDB, not SQL: MongoDB isn't a relational database. Embrace denormalization, embedded documents, and document-oriented design patterns.

---

Conclusion: The $10K Lesson

That $10,247 AWS bill was painful, but it taught me more about database optimization than any tutorial ever could. The mistakes I made aren't unique—they're the same ones I see developers make every day.

The difference between an expensive MongoDB deployment and an efficient one isn't about scale or complexity. It's about understanding how your queries execute, monitoring what matters, and making small optimizations that compound over time.

Today, my application serves 10x more users than it did during my expensive month, but my database costs are 87% lower. Every query is indexed, monitored, and optimized. My connection pools are tuned. My data model embraces MongoDB's strengths instead of fighting them.

You don't need to learn these lessons the hard way. Start with proper indexing, add monitoring, and optimize incrementally. Your future self (and your AWS bill) will thank you.

What MongoDB mistakes have cost you? I'd love to hear your stories and lessons learned. Drop a comment below or reach out—we're all learning together.

---

Last updated: January 2025 | Word count: 1,687