Peridot

Gameplay Engineer · Niantic · 2023

Peridot is a cutting-edge AR mobile pet-sim game played via camera passthrough. Our primary goal was to use AR to make the pet feel like a real creature in the physical world.

To do so, we used AI to build a 3D mesh of the world in realtime via a depth estimation algorithm. As a Unity engineer on the team, I was responsible for making it so the pet used this 3D data to do things like hop up on your table, be occluded when it walks behind a wall, and pathfind realistically around your environment.

The pet appearing on a park bench and settling onto the slats at the height and angle of the seat — 3D meshing allows the pet to interact realistically with the world. They can be occluded behind real objects and...

The pet standing on a cafe counter beside a latte and an open book — ...hop up onto the counter next to you!

The pet trotting ahead of the camera down a tree-lined sidewalk — Players wanted to be able to walk with their pet "dog walking" feature I contributed to.

Another major gameplay feature I helped with was the semantic segmentation (basically object recognition) gameplay. When you point your phone camera at the ocean, a semantic segmentation model detects it and has your Peridot bring you back a kelp gift. When your phone is viewing a person, the Peridot can run up to them and ask for a scratch behind the ears.

A circle traced onto a lawn, after which the pet digs there and returns a grass-specific find — The "foraging" gameplay feature. We used semantic segmentation to assess the kind of surface the player was foraging and to provide them with an appropriate reward. Flowers for grass, kelp for water, etc.

Outside of the AR features, the game had to... be a game! I implemented features like friending, gifting, questing, notifications, streaks, news, and telemetry. I think the strongest engineers are those who understand the entire system they're working with, and to that end I took on server, internal tooling, and tech art work where I could.

The quest log with daily quests completing, showing their countdown timers and Sundrop payouts — The daily quests system which assigned users a series of tasks based on conditional probability tables built by our game designer

The daily login panel awarding day one, then scrolling the full thirty-day reward ladder — To encourage players to return to the game day after day, we added a login streak mechanic.