Chess Scan: from a photo to playing Stockfish

Chess Scan was born right after my internship: I had just learned a bunch of new technologies and wanted to bring them together into a project of my own — and, while I was at it, about something I really enjoy: chess. The idea is easy to describe and fun to solve — you take a photo of a physical board and the app rebuilds the position on a virtual board so you can keep playing against the Stockfish engine, which always replies with the best possible move.

Under the hood there is more going on than it seems: a computer-vision pipeline to find the board, a deep-learning model to classify every piece, a REST service that orchestrates everything, and a Flutter app as the visible face. I'll walk through it in the same order I built it.

The architecture at a glance

The system lives in two worlds: the mobile app (Flutter) and a Python backend that does the heavy lifting. The app sends the photo, the backend processes it and returns the position; and to play, the app sends the current position and the backend replies with the opponent's move.

App (Flutter)  ──►  Google Cloud Run  ──►  Docker container
                                              └─►  Flask backend (Python)
                                                     ├─ OpenCV     → board cropping
                                                     ├─ TensorFlow → piece classification
                                                     └─ Stockfish  → best move

1. Finding and cropping the board

The first thing the server does with the incoming photo is isolate the board from the rest of the image. A real photo is never perfect: there's perspective, shadows, the phone isn't perfectly perpendicular… so I applied a chain of OpenCV transformations until only the board was left, seen head-on:

• Convert to grayscale and blur to remove noise.
• Canny edge detection to keep just the lines.
• Hough transform to extract the horizontal and vertical lines.
• Clustering the intersections of those lines to locate the board's real corners.
• Perspective transform (four_point_transform) to "straighten" the board.
• And finally, splitting it into 64 identical squares.

Original → Grayscale + Blur → Canny → Hough → Clustering → Corners → Crop

2. The dataset and piece classification

With the cropping service working, I could generate my own dataset: 8,357 images of squares, split into 13 classes — the 6 pieces of each color plus the empty square. Labeling that many images by hand is tedious, but it was the foundation for the model to learn well.

The TensorFlow model (transfer learning)

To classify each square I trained a model with TensorFlow and Keras, but not from scratch. I used transfer learning starting from MobileNet, a network already pretrained on ImageNet that knows how to recognize everyday objects. The idea: reuse everything that network already "knows how to see" and retrain only the last layer so it tells my chess pieces apart. The key steps:

• Data augmentation: rotations, zooms and distortions to simulate imperfect real photos.
• An 80% training / 20% validation split to measure accuracy honestly.
• Freezing the MobileNet base and training only the final classification layer.
• 200 epochs of training (several hours of compute).

The result: around 93% validation accuracy (and ~98% on training). The trained model is exported to an .h5 file with everything needed to predict in production.

3. The REST service with Flask

The backend exposes a Flask service with two endpoints. The first, POST /upload, receives the image in base64, crops it, classifies the 64 squares and returns the encoded position (e.g. wp = white pawn, bk = black king, em = empty square):

@app.route("/upload", methods=["POST"])
def reciveImage():
    data = request.json
    image = base64.b64decode(data["image"])
    image = cv2.imdecode(np.frombuffer(image, np.uint8), -1)
    positions = Functions.predict_image(image)
    return str(positions)

The second, POST /play, receives the current position in FEN notation and asks Stockfish for the opponent's best move (I set its strength to 1500 ELO to make it a demanding but playable rival):

@app.route("/play", methods=["POST"])
def reciveFen():
    data = request.json
    fen = data["fen"]
    game = ChessGame(fen)
    game.d_engine.set_elo_rating(1500)
    move = game.d_engine.get_top_moves(1)[0]["Move"]
    return move

4. The Flutter app

The visible face is a Flutter app (my first contact with the framework). The user flow is straightforward: from the menu you can Play directly or Scan. When scanning, you pick a photo from the camera or gallery, crop it to leave only the board, and send it to the server. The app shows a confirmation screen of the detected position and, with one tap, you start playing against Stockfish by dragging your pieces until someone wins.

5. Deployment: Docker and Google Cloud Run

To make the app work from anywhere, I packaged the Flask backend into a Docker container and deployed it on Google Cloud Run, which keeps the service listening for requests without me having to manage servers. This very pattern —container + Cloud Run— is the one I still use today in my projects.

6. A second life: from mobile app to web

Chess Scan could have stayed as the project I wrapped up my internship with, but I was fond of it and wanted anyone to be able to open it without installing anything. So I recently rebuilt it entirely for the browser, keeping the same philosophy —photo → crop → detection → correction → play— but rethinking every layer with what I've learned since 2022.

The core decision was clear: the brain that recognizes the pieces stays untouched. The backend is still Flask + TensorFlow + OpenCV in a Docker container and reuses the exact same trained model (modelo_sipr_v7_200_epocas.h5) from the original app. What changed completely was everything else:

• The front end went from Flutter to the web: React 19 + TypeScript + Vite + Tailwind, with touch-friendly photo cropping (react-easy-crop) that works the same on phone and desktop.
• Stockfish now runs in your own browser (compiled to WebAssembly), not on the server. The opponent lives in your tab: zero network latency to think each move.
• The model's confidence is no longer a hidden number: the web returns confidence per square and highlights the doubtful ones so you can review them at a glance. That's where the human-machine collaboration shows: the model does 95%, you fix the 5% it struggles with.
• A real board editor: drag and drop pieces, a palette, right-click to clear. If the photo came out so-so, you fix the position in seconds before playing.
• And a full game: adjustable difficulty, an evaluation bar, move history with PGN export, take-back, and a hint arrow, with chess.js validating every move.

The contrast is fun to see: the same .h5 model that in 2022 lived inside a phone app today answers requests from a website, and the chess engine that used to query the server now runs entirely on the client. The project didn't change its idea; it changed its shape.

7. Where it is today

The web version is in production and open to anyone at chessscan.com (with the AI backend at api.chessscan.com). Like the rest of my projects, it's deployed on Google Cloud Run and scales to zero: it costs nothing at rest and spins up on its own when someone opens it.

I like that Chess Scan went from being "the project I learned with" to something anyone can try with a photo of their own board. It's the best way to show what I do: not to tell you, but to let you use it.

What I took away from the project

Chess Scan was my first end-to-end project: from computer-vision research and training an AI model, to building an app in a brand-new language and deploying everything to the cloud. Touching that many different pieces and watching them fit together is exactly what I love most about this job — and, three years later, rebuilding it for the web confirmed that the original foundation was well thought out.

What in 2022 I noted as "ideas for the future" —a web version, playing the engine more comfortably, being able to fix the position— is done today. The next notes: a multiplayer mode and sharpening detection even further on tricky photos.