FastAPI and MongoDB: ObjectId object is not iterable error

The Problem

I’m building a calendar application with FastAPI, using MongoDB for data storage. The endpoint I’ve written for event creation takes data based on an Event model I’ve defined and should store that data in the MongoDB database and then return it to the user. However, at the moment, calls to this endpoint fail with the following errors:

ValueError: [TypeError("'ObjectId' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

Here’s my application code:

from fastapi import FastAPI
from pymongo import MongoClient
from pydantic import BaseModel
from datetime import datetime

app = FastAPI()

class Event(BaseModel):
    title: str
    description: str
    start_time: datetime
    end_time: datetime

def store_in_mongo(collection: str, event: dict):
    client = MongoClient("mongodb://localhost:27017")
    db = client["calendar_db"]
    collection = db[collection]
    result = collection.insert_one(event)
    return result.acknowledged

@app.post("/events/")
async def create_event(event: Event):
    event_dict = event.dict()
    result = store_in_mongo("events", event_dict)

    if result:
        return event_dict
    else:
        return {"message": "Failed to create event."}

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)

Why am I getting these errors and how can I fix them?

The Solution

This error occurs because PyMongo’s insert_one method mutates event_dict by adding an '_id' key with a MongoDB ObjectID object as a value. We can see this in action by adding print statements to create_event before and after store_in_mongo is called:

@app.post("/events/")
async def create_event(event: Event):
    event_dict = event.dict()
    print(event_dict)
    result = store_in_mongo("events", event_dict)
    print(event_dict)

    if result:
        return event_dict
    else:
        return {"message": "Failed to create event."}

If we call the endpoint now, we will see lines similar to the following in our console output:

{'title': 'Sunday Lunch', 'description': 'Lunch with family.', 'start_time': datetime.datetime(2024, 6, 30, 8, 25, 19, 294000, tzinfo=TzInfo(UTC)), 'end_time': datetime.datetime(2024, 6, 30, 14, 25, 19, 294000, tzinfo=TzInfo(UTC))}
{'title': 'Sunday Lunch', 'description': 'Lunch with family.', 'start_time': datetime.datetime(2024, 6, 30, 8, 25, 19, 294000, tzinfo=TzInfo(UTC)), 'end_time': datetime.datetime(2024, 6, 30, 14, 25, 19, 294000, tzinfo=TzInfo(UTC)), '_id': ObjectId('6683bc742368f131075b745d')}

The dictionary on the second line has an additional '_id' key. The application throws an exception when attempting to return this dictionary because FastAPI’s internal methods for converting Python dictionaries to JSON objects don’t know what to do with PyMongo’s ObjectIds.

There are different ways to fix this error, depending on the requirements of your application. The following solutions are presented in ascending order of complexity.

Solution 1: Remove the ID

If you don’t need to return the database event ID to users, remove the '_id' key from the dictionary before returning it. For example:

@app.post("/events/")
async def create_event(event: Event):
    event_dict = event.dict()
    result = store_in_mongo("events", event_dict)
    event_dict.pop('_id', None) # remove key from dictionary if it exists

    if result:
        return event_dict
    else:
        return {"message": "Failed to create event."}

This solution hides the internal details of our data storage from users. Depending on the use cases and audience for your application, you may wish to hide event IDs from users or create your own, separate from the MongoDB document IDs.

Solution 2: Define a Custom JSONEncoder Class

We can define a custom JSONEncoder class that knows how to handle ObjectIds. For example:

import json
from bson import ObjectId # bson = binary JSON, the data format used by MongoDB

class MyJSONEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, ObjectId):
            return str(o) # this will return the ID as a string
        return json.JSONEncoder.default(self, o)

To use this encoder, make the following change to create_event:

@app.post("/events/")
async def create_event(event: Event):
    event_dict = event.dict()
    result = store_in_mongo("events", event_dict)

    if result:
        return MyJSONEncoder().encode(event_dict) # use custom JSONEncoder
    else:
        return {"message": "Failed to create event."}

This solution is the most complex, but also the most flexible, and could be useful if we need to use different JSONEncoders for different endpoints. This approach can be used to fix JSON encoding errors for any custom object.