Author: Anne-Kristin Fischer | Peer review: Anju Mercian
Chatbots are on the rise – and so are chatbots designed for refugees. Together with our partners Art For Impact and Tech Equalized, as part of their Voices Of Venezuela initiative, an AI engineering team from Omdena improved a chatbot for Venezuelan refugees to access crucial information using AI.
Context
The integration process for immigrants in a new country is not an easy task. Even with hard work and the desire to fit in, immigrants face many challenges, including accessing basic information or understanding some of their basic rights in the country of residence. In this 8-week challenge, a global team of more than 30 AI engineers developed a contextual assistant as a solution for a scalable refugees helpline.
With the help of Omdena Collaborators, Art For Impact wants to take its goal of building a bridge of understanding between cultures and improving equitable access to information a step further by utilizing cutting-edge technology.
Problem Statement
Currently, most conversations for immigrants are being carried out by human agents and multi-option menus. The majority of the questions are repetitive. The goal for this project is to reduce the time, agents spend on simple and repetitive questions, by building an Intelligent Chatbots; those that are able to hold intuitive and natural conversions to provide the services and answer those questions while maintaining engagement.
Approach
The very first step towards an intelligent chatbot with human-like conversations is a well-structured and comprehensive data mapping of realistic behavior. Art for Impact provided the first set of data that were really useful during this project. To reach our goal we extended this data using web scraping and new data from different pages and links. This task created a bridge to have updated information on different government programs and information sources. The findings then provide predetermined answers, generate support in the training process and complement the database information. All project data in its different formats is centralized in a database which was generated explicitly for this project. It serves as input for the model training. In order to exploit all the benefits of our chatbot framework with the development of Natural Language Understanding (NLU) and Natural Language Generation (NLG) solutions, data preparation is the key step. This task receives as inputs the collected data from the database and the web scraping outputs.
Furthermore, the data preparation task benefited from the findings of Exploratory Data Analysis (EDA). The objective of the EDA task was to gain first insights in our data. Relevant chatbot intents, entities, and responses could be determined to then help with the creation of chatbot stories. The outputs of the data preparation task were useful for the chatbot modeling. To build our custom chatbot we used RASA (an open source tool that automates conversation AI) in combination with Python for (NLU) and dialog management. We integrated into our pipeline the insights from the EDA task, the data from the data collection and preparation teams, and the determined emergencies for actions. In the scope of emergency detection, we identified utterances and messages related to emergencies and grouped them by intent.
Furthermore, we highlighted keywords in emergency messages as entities and developed some stories as conversation archetypes. In a final step, we deployed our chatbot to make it available for use.
Fig. 1: Project Pipeline
Data sources
Within the scope of the project, a database with the project data was created including 300+ links of useful data on different government programs and information sources. Additionally, several datasets in different formats such as .csv and .xlsx were used for the model training. Art for Impact’s own material was invaluable first-hand sources for likely topics, questions, and answers. For instance, the video guides featured on their Facebook page proved very useful in two ways: Firstly, questions could be derived based on the video transcripts, to cover the content of the video. In addition, the public chats following the videos provided with real examples of questions in its original wording used. Both methods were used to collect examples for training the chatbot.
Methods and tools
Fig. 2: Tools used by the different task channels
Let’s check out some of the methods and corresponding tools used more in detail.
Labeling workflow
Step 1: CSV2TextLine
We took tabular datasets and converted them into textLine data. We had a columnar plain text file and we converted it to .txt files. The first file has one column for utterances and another with the corresponding intent. The .txt file was then composed of one utterance per line. The CSV files are transformed to text lines using the textLine_creator module.
Note: that this step is not mandatory since a TextLine file can also be created directly from raw data
Fig. 3: Conversion of tabular datasets into textLine data
Step 2: Labeling
Labeling was done in doccano, a well-known open source tool. Using doccano’s color coding user-friendly interface we created an “Intent Detection and Slot filling” project and assigned each of the whole utterances as intents. Additionally, we were able to link the entities to the set list of intents just by highlighting parts of the utterance text.
Fig. 4: Labeling with doccano (Source: doccano)
The platform allows the collaboration of multiple raters if it is deployed online as a service. The results could then be exported in the desired format (jsonl). If a doccano instance is configured, new utterances can be added and labeled constantly.
Step 3: JSONL parser
We downloaded the jsonl file from the doccano instance and parsed it into nlu.yml in Rasa 3.x format. This conversion was done using the parse_doccano module.
Stories implementation
Finally, we wrote some rules and stories. Rules were mainly used for greetings, goodbyes, and for handling the fallback with a “please rewrite your message” response. The stories allow you to put the designed conversations into Rasa. We implemented simple stories composed of conversations with one question and one answer. The utterances were written in the domain file.
Chatbot modeling
As stated before, the chatbot was implemented in Rasa.
Rasa is an open source platform used for creating artificial intelligence assistants and chatbots. Rasa enables developers to create personalized, automated customer interactions at scale. Rasa’s Open Source libraries supply the building blocks for creating virtual assistants. Rasa Open Source provides three main functions, Natural Language Understanding (NLU), dialogue management, and integrations.
NLU is a branch of artificial intelligence (AI) that uses computer software to understand input made in the form of sentences in text or speech format. The NLU steps were implemented in the pipeline section of the config file. The heart of our NLU implementation is the pre-trained word embeddings model es_core_news_sm and the DIET architecture. Es_core_news_sm by spaCy, as well as other pre-trained word embeddings were helpful as they already encode some linguistic knowledge. Moreover, DIET model is a deep learning architecture based on transformers which we used for intent classification.
Fig. 5: Chatbot modeling structure
As metrics, we performed a 5-fold cross validation. The confusion matrices for intent classification and entity recognition are presented below:
Fig. 6: Entity Confusion Matrix
Fig. 7: Intent Confusion Matrix
Natural language generation (NLG) was implemented in the policies section of the config file. This section defines steps that decide which actions take place in a conversation. It was not reviewed in depth, only Memorization Policy and Rule Policy were set up for looking for matches between stories and real conversations and for identifying fixed behaviors (rules) respectively.
Deployment
The structure of the Rasa system is based on microservices. For the deployment, we tried to work with each one of them.
In a first approach, the Rasa NLU and Rasa Actions services were created in Docker containers connecting to each other. Therefore a VM instance and Debian 11 operating system were configured as a service.
Fig. 8: Rasa system
Testing is done by enabling the API with ´rasa run –enable-api´ and using insomnia. Requests are sent to the docker instances path, managing to establish communication with the server and its actions.
In a second step, it is tested using Telegram since it is a widely used application and can be easily used by end users. Initially, a `rasa_es` bot was created with the BotFather assistant. This telegram bot defines a name and token that serves as an input to configure the credentials.yml file in the main folder of the Rasa project and that, together with webhook_url, allows communication between these two platforms.
We found that one of the requirements of the webhook url field is that the link must be with the https protocol. Thus, for the local tests and to be able to go to the cloud, ngrok was used as a tool to make the connection with a port forwarding. In the cloud and not having a domain with the certificates, it is decided to use this tool again to connect with the bot.
Fig. 9: Architecture of the first attempt for deployment
At this point, it could be considered that it is possible to have a chat available to the public that, fed with the model developed in task 5, could provide an answer to the problem posed. Even though, to reach this result and for it to be repeatable, the company would require staff with sufficient knowledge to be able to maintain or update it.
For this reason, the use of Rasa-X is explored, which contains some more layers of services, which allow the administration of the processes that are carried out in Rasa in an iterative way, such as labeling, managing the flow of conversations, configuring other services, training, and model generation, version management, among others.
Fig. 10: Rasa-X architecture
The implementation was tested with Docker-compose which is documented for specific versions of Rasa and Rasa X and not for the latest versions. This implementation launches different services which are connected to each other through the internal networks that are created with docker, and are taken out using forwarding of the ports that are required. As Rasa X versions are being released later than corresponding Rasa Open Source versions, the latest version of Rasa X (1.1.X) does not yet work properly due to its very recent release.
In a fourth approach, the `Rasa X Helm Chart` deployment method is used, which can be installed on top of Kubernetes. To do this, a yaml file is configured with all the characteristics in which the deployment is to be made, such as a version of rasa for training or production, a version of Rasa X, a Database for the DB tracker, among others. With a line of code and with the appropriate configuration of the work environment, the different services and layers related to the creation of the environment for a chatbot can be deployed and released to production.
Working in Spanish and with the es_core_news_sm model, it was necessary to create an image with the latest Rasa version, SpaCy, and the model in question. For this, a Docker file was created with the necessary libraries and packages. Subsequently, the image is uploaded to the Google Cloud Artifact Register. The Yaml document that creates or updates the application is configured with this repository, otherwise the app won’t work.
With the implementation of Rasa X, the process of creating conversation-based bots is improved, allowing you to annotate, review, test, track and manage conversations, all within an interactive and easy-to-use web application.
Results & insights
Web scraping
In the context of the web scraping task over 4000 links were found. The algorithm spent around 4 hours to reach its goal. The data links were then filtered into “useful” and “not useful” with the result of more than 300 new links of useful data. This data was gathered in a pre-database with SQLite with two tables in the initial database: webpages and records_dataset.
Exploratory Data Analysis (EDA)
In the scope of Exploratory Data Analysis several approaches were gathered to gain first insights into the available data for determining relevant chatbot intents, entities and responses and ultimately helping with the creation of stories. Some approaches and their results proved to be more insightful than others for this specific project. This summary orders approaches from more insightful outcomes to less insightful.
Word clouds, n-grams, knowledge graphs
One of the most revealing approaches turned out to be the creation of word clouds. Essential keywords were gathered, based on the number of appearances, across the available data to identify and visualize topics of major relevance. Equally important results were obtained with the visualization of unigrams as a histogram. To go beyond the mere count of words, a knowledge graph was created to identify relations between words and concepts (specifically but not only related to the concept “necesito”, Spanish for “I need” as center node). The results of these 3 approaches were used to create a keyword list for Spanish and English, ordered alphabetically as well as by language. This keyword list can be considered the major deliverable in the scope of this task and contains around 240 meaningful entries.
Fig. 11: Word cloud
Fig. 12: Unigram (histogram)
Fig. 13: Knowledge graph
Sentiment analysis, correlation heatmaps
Furthermore, sentiment analysis was applied to the data classifying all analyzed messages as neutral. Thus, it can be deducted that the user target group is seeking information in a rather objective and rational manner. No significant relations could be determined with the creation of correlation heatmaps. The heatmap intended to analyze the nodes [date], [send], [person], [number] and [content]. It can be deduced that the content of a message is neither linked particularly to a specific date or person nor to the status whether it was sent or not.
Fig. 14: Sentiment analysis, correlation heatmap
Fig. 15: Correlation heatmap
Other
Particular importance revealed a word count across the datasets for the term “estatuto temporal” (Spanish for “temporary status”) which in this case is relevant for a new 10 year visa program being implemented in Colombia called the “Estatuto Temporal de Protección para Migrantes Venezolanos (ETPV).”
Fig. 16: Word and conversation count for “estatuto temporal”
Initial attempts were also made of applying EDA with TensorFlow, topic modeling with LDA and TF-IDF.
Chatbot modeling
The chatbot was developed and trained with the data provided by Art For Impact. The pipeline consists of SpaCy Tokenizer, SpaCy Featurizer, Regex Featurizer, Lexical Syntactic Featurizer, CountVectorsFeaturizer, and DIET classifier. This configuration was chosen after trying multiple ones with the data, and this has proven to be the best with NLU so far.
Fig.17: Modeling pipeline
Fig. 18: NLU test results
Multiple simple stories were added and tested.
Fig. 19: Examples for simple stories
Deployment
A series of services based on the Rasa X architecture were implemented in Kubernetes of Google Cloud Platform which is scalable in resources according to connections, services, or the number of users, easy to update in architecture, and useful for developing conversational models. With this, the company will be able to upload or create bot models, in Spanish or English, and connect them to different mass-use services that will allow them to connect with end users easily.
Conclusion
The goal of this project was to build a more intelligent chatbot for use in a refugee integration helpline. The work included several major pivots, each with its corresponding deliverables.
The aim of the web scraping task was to collect information relevant to the needs of refugees from the web and identify the new information among all the search results.
The database design and set up task gathered all project data in one place. With data annotation and preparation we identified relevant intents and entities, collected and annotated training examples for them, and created stories with conversation flows.
Within the framework of exploratory data analysis (EDA) we gained first insights into the data. Various EDA approaches such as word clouds, topic modeling, and knowledge graphs were helpful in identifying relevant chatbot intents, entities, and responses.
In the context of the model training task, NLU (natural language understanding) and NLG (natural language generation) models were built in Rasa.
A task for handling emergency cases were identified, utterances, and messages related to emergencies and grouped them by intent. Keywords were highlighted in emergency messages as entities and some stories were developed as conversation archetypes.
Finally, the deployment task made the chatbot available for use.
Working with Spanish, the chatbot’s main language, proved to be a particularly challenging . Issues were encountered with encoding, atypical characters such as “¿”, accentuation, word inflection, and verb conjugation. The latter is of major importance for understanding the meaning of the verbs (e.g. future, past, intention), and to rule out the removal of accentuation or the use of spell-checking and other pre-processing tools. The sentences had to be taken in their entirety – a tedious process. Also dealing with colloquial speech and online speech, like slang, misspelling, and emojis, turned out to be no neglectable task. Tools like Alexa Utterance Generator came in handy to combine the multiple word forms and inflections for generating varied combinations to be used for training.
0 Comments