Students don’t normally enter hackathons designed for enterprise professionals.
But that didn’t stop M&T freshmen Ria Saheta and Crystal Yang from joining the AWS and NVIDIA Agentic AI Hackathon. As dual majors in management and technology, Saheta and Yang split their time between Penn’s Wharton School of Business and the School of Engineering and Applied Sciences. They got inspired to build a sensor-based hardware project while attending a GRASP Lab doctoral student dissertation defense.
Having learned that one in four older adults aged 65 and older report falling each year, they wanted to see if they could use modern AI to make care more proactive, not by replacing people, but by giving caregivers earlier, clearer signals. With little time to spare, they headed to the Detkin Lab, testing out available sensors and integrating them with software from AWS and NVIDIA, designing a finished product that incorporates both software and hardware.
Their efforts and quick turnaround paid off. Out of 1,900+ applicants, Crystal and Ria won the hackathon, with one of the prizes being a new NVIDIA GPU worth $7K.
Their project also addresses a rapidly growing issue in developed countries: there are disproportionately more elderly people than caregivers to take care of them,” Saheta and Yang explained. The WHO projects a 13.5 million caregiver shortage by 2040. Furthermore, falls remain the leading cause of injury-related deaths in the elderly population (CDC, 2023).
By incorporating both the camera and various sensors into their system, Nomi fills in the gaps left for seniors who do not have 24-hour caregiver assistance. This continuous tracking takes into account that seniors forget to eat meals or miss doses of their medication without any compromise of privacy. Nomi also uses the vital signs alongside the camera feed to monitor seniors for falls.
To ensure seniors take the correct medication dosage, Nomi comes equipped with “a thin pressure film sensor placed under a pillbox to detect when it’s opened,” Saheta and Yang explain. “Combined with time tracking in DynamoDB, Nomi infers whether medication was taken or missed, including this in its AI reasoning summary.”
When it comes to tracking eating activity, Saheta and Yang placed “pressure sensors under utensils or trays to detect eating events. Nomi tracks patterns across days,” Saheta and Yang explained, “identifying skipped meals and alerting caregivers to subtle changes before they become health risks.”
To alert caregivers about falls, Nomi comes equipped with a fall and posture detector. Nomi incorporates an “OpenCV + MediaPipe model to analyze posture locally on-device,” Saheta and Yang explain, adding that when a “fallen or inactive posture is detected, Nomi triggers an immediate email alert to the caregiver.”
As for monitoring vital signs, Nomi’s pulse sensor “tracks heart rate and oxygen proxies, while temperature humidity sensors monitor environment,” Saheta and Yang explain, adding, data streams through AWS Lambda → DynamoDB → FastAPI → React, updating live charts in the dashboard.
If emergencies arise, elders who use the product will detect abnormal vitals or falls, utilizing “AWS SNS or Gmail SNTP to instantly email caregivers with event time, vitals, and recommendations,” Saheta and Yang add.
Beyond ensuring seniors take their correct medicine, eat, check vitals, and respond to emergencies, Nomi provides AI health insights. “NVIDIA NIM (Llama-3.1-Nemotron-8B) interprets sensor data like a virtual health coach-summarizing patterns,” Saheta and Yang note. Further, Nomi responds to concerns over constant monitoring. “All video-based detection happens locally using OpenCV and MediaPipe.” In other words, “only anonymized event labels reach the cloud.”
Currently, Nomi remains in the prototype phase. Saheta and Yang have tested the sensors and models on themselves. “For the OpenCV model, we’ve experimented with different camera angle setups to see what it looks like if it detects poses correctly,” Saheta says, explaining this includes whether “they’re sitting, lying down, or eating.”
Next steps? “We’re working towards building a launch-ready product,” Yang explains.
Their initiative speaks volumes. “We competed in a hackathon designed for experienced professionals,” Yang says. “And we successfully built a full-stack system with hardware, the cloud, AI, and a dashboard.”
In doing so, they created a product that moves from “reactive to proactive care,” Saheta emphasizes.
When they’re not working, Saheta enjoys crocheting, mountain biking, learning about blockchain, and watching drone racing. Yang enjoys discovering the Bing daily wallpaper, playing the piano & viola, and playing pickleball.
For more information on Nomi: https://devpost.com/software/nomi-aoim58