Nexus
Intelligent Lost Item Recognition and Management Platform
The main compenent of Nexus is its modular kiosk that can be easily deployed to a wide range of public facilities. The kiosk is equipped with a human interface and a camera in the item scanning chamber to recognize and automatically log the item to our database as well as an expandable storage system.
This graph documented how users will interact with each component of the Nexus platform. Since Nexus is a platform consisting of several products, this was an important piece of document that laid out how every part of the platform works together to solve different user's problems.
System Architecture
As the system architect of the project, I laid out the system block diagram for the platform. This is the final version in which Raspberry Pi was chosen as the main hardware platform.
I also laid out the Engineering Validation Test plan for the kiosk hardware. I built several prototypes using different hardware architectures shown here to finalize our engineering parameters and options. The following section shows the prototypes I built during the process.
Similarily, I also made the EVT plan for our software platform, outlining different options for developing the software services.
Hardware Prototyping
The various stages of prototyping was crucial in turning the direction of the project from a single product to a platform that integrtes several products. The prototypes I developed helped showcase the feasibility and the benefit of having the hardware, software and services all working seamlessly.
This first generation prototype use an iPad app to test the feasibility and options of the two biggest technical uncertainties:
- Object Recognition: I tested using genAI APIs to recognize items stored in the kiosk.
- Control Circuit: In order for my iPad prototype app to be able to control the circuit for the locking mechanism on the kiosk, I used an ESP32 microcontroller that is connected to iPad via BLE wirelessly to control the circuit.
This video shows the app controlling a pin on the microprocessor that is connected to the multimeter.
As the first gen prototype proved it's very feasible to use a genAI API to perform object recognition, the next step for me is then to try migrating the application to a Raspberry Pi. The benefits of using Raspberry Pi compared to an iPad are:
- A Raspberry Pi combined with a touch screen is half the price of an iPad.
- Raspberry Pi provides the pinout that can be used to control the circuit of the locking mechanism.
This video also shows that the application on Raspberry Pi is capable of uploading the object recognition result to our Nexus database.
This video shows the application of Gen 2 prototype integrated with the control circuit tested by a multimeter.
Besides the electronics part of the hardware, I also designed the concept hardware of the kiosk and the mechanical design of an early candidate for our build plan.
This is a CAD model designed in Fusion that incorporates an iPad as the main device and use aluminum frame as the main structure.
UI Prototyping
I performed extensive evaluations on the operation flow of all the user interfaces across different product in the platform. For example, I created several different flow for validating identification on the kiosk and also various ways to modify the result of object recognition. Then, user tesing was conducted to decide the final flow.
These were several candidates for different operation flow for authenticating owner of lost items. I built the Figma prototypes to perform user testing and eventually decided to use the passcode flow.
Learn more
This project was funded by Google. I am grateful to have a wonderful team working on other aspects of the project. Heather Hsieh implemented the Nexus website, ChunKai Hsu helped fine tune the prompt for Google Gemini LLM, they also shot and edited the main demo video. Alan Chen helped building the final prototype hardware and YuJie Huang helped implemented the database in Notion.