Solution 1 :
The way you want to solve this is basically not bad.
But in my opinion ELK is not precisely you want.
The SQS + Lambda is a good idea to aggregate and filter the lots of public requests, but have some issues with security, or efficiency (or maybe cost).
And the architecture of that will not be simple at all.
However, if I may recommend sentry.io, which is exactly the tool you need there as I understand the problem correctly.
And sentry have solution for mobile, android and iOS too.
I did use sentry for different things, and have a clearly readable interface to work with. With minimal code assigns you can tag the events, throw good Exceptions for filtering, and sentry solves the rest.
Solution 2 :
Here a good article that explain many solutions to do remote logging for mobile apps.
The best way to send your log to elastic stack is to use Hyperlog.
Hyperlog is a free and open source library to send your logs to ELK or
RequestBin, service for inspecting HTTP payloads.ELK goes for Elasticsearch, Logstash, Kibana – services for storing,
collecting and monitoring logs. Despite they are mostly used for web
apps, we can send logs from our apps to ELK as well. Deployment and
configuring of the ELK stack is out of scope of this article, but you
can find details here. You can also choose one of the ELK stack
providers, there are quite a lot of them.
You can also take a look to Bugfender and Loggly that do not require the entire elastic stack to monitor and troubleshoot your apps.
Problem :
I am developing a mobile application on IOS and Android. Currently I use Firebase Crashlytic to track application crash log.
I am not quite satisfied with what I can do with Crashlytic. For example when a user reported an issue and had a video record an app crash at a specific time, I would like to see the log from the device around that time, but it’s not easy with Crashlytic.
A solution popup in my mind to have the mobile application send the crash log to my AWS SQS queue and somehow pass it to Elasticsearch so that I can filter the log using Kibana.
I would like to implement something like this
- The mobile app keep log of everything into a rotatable temporary file.
- When meet following conditions, send the log from file to the SQS queue.
- Before showing any error popup
- After any app crash event
- When an API did not response after X seconds
- If found any error in the process of sending log in 2., set flag
retry_send_error=truein memory. - On any possible application event, if found that
retry_send_error==truein memory, try send the log again. - Create a lambda listening the SQS queue and send the log to LogStash or Elasticsearch.
I have been searching on the Internet to find some reference example and could not find any good one. So I doubt my solution could have something wrong.
Please help suggest if you know some good example with architecture similar to this, or if you know the reason why this solution is not so popular.
Comments
Comment posted by Hyperlog
Have you tried using
Comment posted by Val
Any feedback from the link I shared above?
Comment posted by asinkxcoswt
@Val thank you. I have reviewed the library but it seems not quite popular enough and the project is not active. I would love to tried it in my personal project but it is not enough to entrust customers to believe that using ELK for mobile is a common practice because the popularity of this solution is far beind other options.
Comment posted by Val
The mere fact that you can’t find any existing project doesn’t necessarily mean that doing so is not popular. I’ve done such things in private projects and I’m most probably not the only one, the thing is that most customers don’t like to have their code published on Github or elsewhere. But what you’ve described is definitely one way of doing it. I’ve seen this done in multiple ways, via MQTT, web sockets, firebase, direct HTTP calls, etc
Comment posted by Jordan Stewart
I think it’s more natural to stream logs from a web-server to Elasticsearch. Streaming logs from a mobile apps isn’t ideal due to limited bandwidth, and network connection. I think pushing logs on need works around that, which you suggested. You could also have more tests in the development environment for the mobile app to try to limit errors.
Comment posted by sentry.io/vs/logging
Sentry is a good tool when it comes to crashes or error tracking but it’s not necessarily a longing framework. You might lose the historic events of the same error type similar to in every crash reporting tool like firebase etc. They also mentioned it in their website.
