GPSpeak

GPSpeak is a mobile app that uses voice technologies like Siri and Google Assistant. The core idea is that cyclist or joggers want to make voice notes while working out. The voice notes are then tied to a GPS location and rendered on a map.

Going from vague idea to a concrete plan

R&D Phase

Being an avid cyclist, often ideas and thoughts come to the client while on his bike, maybe something related to a project, or he discovers new fun routes or discovers a fun place that he would like to visit later. He wanted to make a voice note about this to process later, so he could optimize his cycling routes based on these notes. So the key concept, as the name suggests, was linking voice notes with GPS locations.

After exploring various design ideas and refining the initial offering together with the client, we started looking into the technical solutions. Initially we liked the idea of having a custom wake word, and played around with technologies like porcupine and spokestack - which seemed very promising, but because of expensive pricing models that would kick in when growing the app, we decided to take another route.

Since most modern phones are equipped with voice assistants such as Siri and Google Assistant, we decided that practically it would be a better idea to hook into those systems. The downside would be that we couldn't use a custom wake word, but the upside would be better performance.


The power and flexibility of React Native

Development Setup & Voice Commands

The development phase of our project leveraged the power and flexibility of React Native, allowing us to create a cross-platform mobile application that could seamlessly integrate with both iOS and Android devices. This choice of framework enabled us to maintain a single codebase while targeting multiple platforms, significantly streamlining our development process.

One of the key components in our implementation was the integration with voice assistants, particularly Siri for iOS devices. We utilized the react-native-siri-shortcut package, which provided a bridge between our React Native app and Apple's SiriKit framework. This integration allowed users to trigger our app's functionality through voice commands, making it possible to create location-tagged voice notes without manually opening the app.

The implementation process involved creating custom intents and configuring them to work with our app's specific functions. We defined a custom shortcut that would activate our voice note recording feature, associating it with a phrase like "GPSpeak" This allowed users to simply say, "Hey Siri, GPSpeak" to initiate the recording process.

On the Android side, we explored similar integrations with Google Assistant, utilising the Actions on Google platform to create custom voice commands that would interact with our app. While the implementation details differed slightly from iOS, the overall concept remained the same – providing users with a hands-free way to capture their thoughts and associate them with their current location.

Functionality

Storage, searching and tagging

To handle the GPS functionality, we incorporated the react-native-geolocation library, which allowed us to accurately capture the user's location at the time of recording. This data was then paired with the voice note in our app's database. Because of the personal nature of this app, we didn't develop a backend for this one. All data is stored locally on the device.

We also built included speech to text technology, based on react-native-voice - which would transcribe the speech to text, this made it much easier to search for specific terms in the notes. We also leveraged google APIs to map locations to a specific city, so notes are automatically tagged by city.

Apart from the city tags, we also provided some other tags to organize the notes, such as interesting thoughts, route information, point of interest, .. which users can customize for their needs. All this leads up to a system that makes it very easy to find the note you are looking for.


Refining the user experience

Finetuning and challenges

As we progressed through the development, we continually refined the user experience, focusing on making the voice note creation process as seamless and intuitive as possible. This involved extensive testing and iteration, particularly around the reliability of voice command recognition and the accuracy of location tagging.

One of the challenges here was to activate app logic while the app was in the background or even put to sleep, when the OS usually restricts app resource usage. We used react-native-background-actions to tackle this problem. This did mean the user would have to start a session, so that we could keep the app running in the background, waiting for voice notes, during the user's sporting activity.

Another challenge is being on the bike often creates a lot of background noise, so this really required a good wearable with voice assistant support built in, such as apple airpods.


The next steps

Reception and next steps

The client is still evaluating the app internally and looking for marketing ideas to bring this to the public. One thing is for sure, this was a super interesting project with a lot of technical challenges where we learned a lot.

Team

Team that worked on GPSpeak

  • RobbyEngineering Team Lead
  • IvanFrontend Developer
  • AlexLead Designer

Other interesting projects

iFLAG

Client iFlag aims to enhance internal reporting, making it easier to surface misconduct. This fosters team confidence and helps mitigate operational risks for your company.

Gloow

Nebulae is also developing in-house software products. We want to help people organise their thoughts, ideas and information in a more effective way. That's why we built Gloow. A tool for organising and visualising your information using knowledge graphs.

Like what you see?

Get in touch and we will be happy to discuss your project.