Is it possible for Slack to alert you about your coffee?

August 13, 2024

Blogs

A big part of the working culture at DeepArt Labs involves participation in various hackathons. These events foster creativity, keep the team updated with industry trends, and offer a productive yet fun environment for innovation. Recently, during our internal hackathon, Nexothon, my team delved into how emerging technologies like facial recognition can enhance existing processes and day-to-day activities.

Background and Inspiration

Personalization has been a significant trend in IT for a while now, offering valuable enhancements to user experience. Envision waiting in line at a restaurant and instantly knowing the availability of your favorite dishes or having the waiter suggest 'your usual' based on your order history. This level of personalization through automated systems can significantly improve customer satisfaction and operational efficiency.

Armed with this inspiration, we identified an existing issue in our office — people often forget their coffee mugs at the coffee machine. This usually led to the coffee getting cold or mess within the shared office space. Our solution was to develop a system that would notify individuals if they left their coffee mug unattended.

Defining the Problem and the Solution

The problem was outlined as follows: if a person makes a coffee but doesn't take it within two minutes, they should receive a direct notification to minimize disruptions to others. Traditionally, this would be resolved by someone yelling out, "Who left their coffee?!" Our task was to replace this archaic method with an automated notification system using facial recognition.

System Overview

Our system employed a Raspberry Pi 4 equipped with Raspbian OS, several peripherals including a PC webcam, an ultrasonic distance sensor, an LED board, and a button. The Raspberry Pi sends notifications via Slack, making it a robust and versatile platform for both development and deployment.

Hardware Components

Raspberry Pi 4: The central unit managing all operations and communications.
PC Webcam: For capturing images and performing facial recognition.
Ultrasonic Sensor: To detect the presence of a coffee cup on the pedestal.
Button: To allow user consent for capturing their photo.
LED Board: To provide visual feedback on system operations.

Implementation Using Python

We opted for Python due to its rapid prototyping capabilities and an extensive library ecosystem. The project was structured into various modules, each responsible for specific tasks.

Project Structure

The project was divided into several logical modules:

camera.py: Manages image capturing and processing, including facial recognition.
button.py: Detects when the button is pressed, indicating user consent.
distance.py and coffee_detector.py: Handle input from the ultrasonic sensor to detect a coffee cup.
lcd.py: Displays notifications on an external LCD connected to the Raspberry Pi.
nexoboard.py: Sends messages to our internal motivation system, Nexoboard.

Facial Recognition

We used the OpenCV library for facial recognition, a popular computer vision library that leverages machine learning algorithms. The process was segmented into three phases:

Data Gathering: A training module captures 30 images of a person's face and creates a dataset.
Training: OpenCV uses the dataset to generate a trained model, stored in a trainer.yml file.
Recognition: The system uses the trained model to identify faces in new images.

We incorporated a Haar Cascade classifier, a machine learning approach trained with a plethora of positive and negative images to detect faces accurately.

Slack Notifications

The Slack module was designed to send notifications to the recognized person or to the #random channel if the face was not in the database. This required a few steps to set up:

Creating an application on the Slack App Management Page.
Adding OAuth scopes to allow the app to view user information and send direct messages.
Using a security token generated by adding the app.

Slack messages included pictures captured by the webcam, alerting the coffee owner directly or broadcasting it to a general channel.

State Machine Implementation

We used a state machine model to manage the system states efficiently. The states and transitions were defined as follows:

Idle State: No cup detected.
Waiting for Action: System awaits the user to press the button for consent.
Waiting for Coffee: System waits to detect a coffee cup after getting user consent.
Known Face: Recognizes the faced and waits for the pickup within 2 minutes.
Unknown Face: Unrecognized faces trigger general notifications.

Python Code Example

The following Python snippet illustrates the state machine implementation:

def state_machine(coffeeDetector): global state global t1 global t2 P = coffeeDetector.consent_button_pressed() K = coffeeDetector.coffee_on_tray() switcher = { 1: waiting_for_action, 2: waiting_for_consent, 3: waiting_for_coffee, 4: known_face_and_wait_for_coffee_done, 5: wait_for_pickup, 6: unknown_face_and_wait_for_coffee_done, } machine_step = switcher.get(state) machine_step(P, K) print("State:", state, machine_step.__name__, "t1:", t1, "t2:", t2)

Testing and Performance

The Raspberry Pi 4 handled all components smoothly, facilitating real-time facial recognition and fast Slack message delivery. The entire process completes within a few seconds, ensuring timely notifications to prevent cold coffee cups!

Summary and Future Applications

This hackathon project showcased how AI and facial recognition could be used for practical, everyday applications. While this prototype focused on notifying users about forgotten coffee cups, the same technology could be adapted for targeted advertising in retail stores or personalized service recommendations in restaurants.

Our system was initially equipped with a single camera and proximity sensor, but it could be scaled or modified based on different scenarios. Creating this prototype was an engaging and insightful experience for the team.

Final Thoughts and Board Design Changes

Post-hackathon, we modified how the LED diode was controlled, transitioning from a hardware to a software-controlled system. This adjustment provided greater control and flexibility in managing the diode's behavior.

After finalizing the design and debugging the hardware, we ordered new boards from the factory. A few weeks later, we received our brand new boards, ready for deployment in more exciting projects.