The following information stems from a conversation with HOT and their fAIr lead and AI engineer Omran Najjar about the challenge and the outcomes:
Q: What was the best part of running an Omdena AI Innovation Challenge?
A: The diversity of knowledge coming from Omdena’s collaborators is amazing! For instance, the various team members all have different techniques within deep learning and machine learning techniques in their “toolbox”. Different models – new models ourselves, and we are trying RAMP model.
The efficiency of Omdena’s coordination is the second. I am amazed at how organized and delivery-oriented they are. Normally projects don’t work this well! It has all been very professional.
The 8 weeks intensive AI challenge with 50 AI engineers gave wide insights into the problem statement which is something organizations find hard to achieve. A feeling of something might be missed or an area of potential improvement is hidden somewhere, Omdena’s engagement helped significantly in mitigating these feelings.
Q: Another AI challenge was conducted a few years back (the Open Cities challenge). Aren’t we just repeating what has already been done?
A: No, we are actually building on top of it. The output from the Open Cities challenge had an approximately 86% intersection over the union score, which is good, but not great. This was before my time, so I am not sure, but I guess there was a hope that someone would pick up this model and build upon it. What did happen was that a student (now a HOT volunteer), Christopher Chan, reached out because he wanted to do some research on AI-assisted mapping. This research was just published actually (see this overview). What we learned from Chris’ research was pretty amazing: it proved that small, local datasets actually increased the accuracy of the models, hence the local AI models predict better buildings and roads than a global model! (explained below) So now, during this Omdena challenge, we have: i) the knowledge of Chris’ research and ii) the training datasets (explained below):
Source: OpenAerialMap
For training datasets, imagine a 5-year-old child who has never seen satellite imagery before and you challenge them to tell you what are the buildings in the following image:
Source: OpenAerialMap
Initially, they don’t know how a building looks from the top, they might struggle to know what is a shadow of a building and what the ground is in the image! To teach them, you create a couple of pages of the book as a curriculum, where you put images and highlight what is a building. Then you train the child to enhance their ability to detect buildings. Then maybe extend the curriculum to train them to detect a road, a tree, a swimming pool, etc. Here is one page of that book:
Source: OpenAerialMap
The child is an AI model and the curriculum is a training dataset. The curriculum seems to be like OpenStreetMap data, right?
Q: Oh, love this analogy! Ok, so to nail it down: why exactly have you been doing this AI Challenge with Omdena?
A: In short, we want to ensure that fAIr ultimately provides as accurate and precise models as possible for AI-assisted mapping in our communities and Omdena’s Innovation Challenge approach is a community-driven approach of engineers as collaborators. To get there, we teamed up with Omdena, and are now working together with a diverse team of 50+ AI engineers who support the decision on which models to use in our fAIr service. To get a little more technical; we had a baseline model before the challenge started. The Omdena team has been using other open AI models (including the Open Cities model and RAMP model) to re-train/fine-tune the models and together discuss and advise on the direction to go.
Q: This is such a good example of the power of collective intelligence – very cool! What were the biggest struggles before working with Omdena?
A: Before working with Omdena, we struggled to decide which approach or model to select for further fine-tuning, the collaboration work by Omdena’s challenge participant has widened the research, supported the decision, and opened new doors for development.
Q: Can you share a few insights from the outcomes of the Challenge?
A: The project team tested several architectures of Convolutional Neural Networks (CNN) during the challenge. As the provided input data was limited, the final approach was to apply the pre-trained model from the RAMP project. This deep neural network was trained with large amounts of satellite images and was used for transfer learning. The RAMP model achieved the best prediction results during fine-tuning with an average accuracy of around 94 % and an Intersection over Union score (IoU) of around 84 % across all test regions with an average inference time of ~1.7 seconds.
One major outcome of the challenge was the development of the HOTLib library as a reference implementation of the planned workflow. This Python library supports HOT in the preprocessing of the input data, the inference step based on the RAMP model as well as the post-processing of the prediction results.
Final presentation
