Yesterday I aggregated some data from Twitter that referenced KFC, Nandos, Debonairs or McDonalds and sat with the Mobenzi pilot participants as we answered two simple questions about each tweet.

1. Was the message positive, negative or neutral in reference to the brand?
2. If it was negative, was it due to customer service, taste, health or some other reason?

The work was entertaining for the participants, they completed tasks efficiently and the results seem to be very accurate.

About sentiment analysis

With the growing use of online services like Twitter, blogs and forums, there is a vast amount of publicly available information generated by everyday people about millions of different topics (companies, products, movies etc.). Knowing the sentiment of messages (e.g. whether they are positive or negative) can be extremely valuable to the people or organisations involved, especially when monitoring trends over time.

Sentiment analysis or opinion mining refers to a broad area of natural language processing, computational linguistics and text mining. Generally speaking, it aims to determine the attitude of a speaker or a writer with respect to some topic.

The rise of social media such as blogs and social networks has fuelled interest in sentiment analysis. With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.

Find out more about sentiment analysis on Wikipedia

This kind of work is well suited to Mobenzi agents

Sentiment analysis seemed like a very appropriate type of task for processing by Mobenzi agents on their phones as tweets are very short (only 140 characters). We also felt that there would be a demand for an efficient human sentiment rating service since computer algorithms face many difficulties in trying to understand the tone of human messages.

Twitter includes a lot of slang, humour, Textese and other informal language that makes automated analysis especially difficult. In a multi-cultural country like South Africa, many tweets also combine words from a variety of local languages which would make analysis very challenging to a computer.

Example Tweets that reference take-out brands

These were some of the messages included in our sample set of data from Twitter.

"Damn you debonairs" in this context is meant in jest and is not negative.

The sentiment is not obvious in this message.

The tone in this tweet changes totally at the end of the message.

The results were ‘positive’

The focus of this study was to assess issues relating to the completion of tasks. We only looked at a small sample of tweets, and could have been a lot more scientific in our approach, so the sentiment results themselves should not be taken too seriously.

There were six participants (including myself) and we each stepped through the analysis of Twitter messages that mentioned KFC, Nandos, Debonairs or McDonalds. Each task took only a few seconds to complete and the team found the work interesting and engaging. None of the participants (except myself) use Twitter themselves, but they were all very familiar with the concept and frequently use Mxit which is similar in some aspects.

One of the measurements we look at to gauge the accuracy of results, is the agreement between different participants for the same task. All six participants rated the sentiment of each tweet, so we were able to look at where our answers differed. It was very encouraging to see that most answers had 100% agreement (Especially if we exclude where participants stated that they were unsure of the sentiment). There are only a few cases where we disagreed on whether a particular tweet was positive or negative. In these cases, the majority was correct and in some cases the disagreement actually helped to balance the rating where the sentiment was ambiguous.

The summary across all brands came out at 48% positive, 35% negative and 17% neutral or unclear. Of the negative tweets, 29% were service related, 16% to do with taste, 9% health related and the rest for other reasons.

These charts show the summary of the sentiment analysis for all four brands.

In the following results, we excluded tweets that were either neutral or unclear with regard to sentiment. Out of the four brands, Nandos was clearly the favourite with 80% of tweets being rated as positive.

Breakdown of Positive versus Negative sentiment for each brand

To have a look at what people are saying right now about these brands, simply go to www.twitter.com and search for #Nandos, Debonairs, #KFC or #Mcdonalds.

Interestingly, a quick analysis of these keywords on Tweetsentiments.com (A service that attempts to automate the analysis of tweets) returns fairly similar results in terms of rank, but with some significant variations in the actual sentiment rating. Nandos: 68% positive, Debonairs 59% positive, Mcdonalds 56% positive and KFC 52% positive. The ranking of the brands is the same as our result, except that Mcdonalds moved in front of KFC with the automated analysis. This may have to do with the fact that we only looked at tweets in English and other South African dialects. Perhaps English speaking people are the least positive about Mcdonalds? Looking at some of the tweets in their data sets, I would trust our result over the automated one. Try the service out yourself at  http://tweetsentiments.com/analyze

It is no coincidence that we ended up having Nandos and Debonairs for lunch.

The possibility of having Mobenzi agents convert English content into local languages (and vice versa) was one which really excited us.

As with all Mobenzi tasks, we split up a large, complex challenge into small, discrete tasks which can be performed in a few minutes.

Obviously, there are limitless reasons people require translation services. One of our own immediate applications for translation is for surveys conducted using our Mobile Researcher platform. We frequently have clients embarking on multilingual studies who need questions to be available in several local languages (in South Africa alone we have 11 official languages); and need qualitative responses given in other languages available in English for analysis. Normally, surveys are designed, reviewed, debated and eventually finalised in English before they are sent for translation and back-translation. Our objective is to crowdsource this activity instead – making near real-time translations available to survey designers in Mobile Researcher at the click of a button.

In our first trial we took a client’s survey questions which we already had in English and professionally translated Zulu and converted them into two sets of translation tasks: from English to Zulu and vice versa.

As most people don’t speak Zulu (and mine isn’t exactly fluent either), I’ll use a Zulu-to-English example taken from the actual exercise:

Original English (not visible to Mobenzi agent): What is the household’s average monthly income?
Professionally Translated Zulu (shown to Mobenzi agent): Ungayilinganisela kumalini isiyonke imali engenayo ngenyanga?
Mobenzi Agent Translation (from Zulu to English): How much is the monthly income for the household?

Now, clearly the ideal would be for the original phrase to match the version coming back from the Mobenzi agent (and this did happen frequently) but in most cases, it does prove that, for the most part, crowdsourced translation works to the extent that the concept is sufficiently conveyed.

Some of the key challenges and limitations of crowdsourced translation tasks which our agents helped us identify were:

• Lack of context. Just as computer translation algorithms struggle without context – so do people. We noticed that, on several occasions, what agents had responded with made sense if read in isolation, but not within the context of the broader scope. It has to be said that this problem is exacerbated by our approach of segmenting text for translation.
• Language formality. The mobile communication culture is almost exclusively an informal one. Some agents seemed to struggle to snap out of this mode and became frustrated having to use formal language.
• Input mode. Although an obvious observation, translation requires a lot more typing than other types of tasks. Somewhat foolishly, we didn’t enable predictive text on during the first day of our translation trial which didn’t help matters.
• Multiple alternatives. Just as there are a number of ways to phrase a question in English, translation is not a one-to-one mapping. This is not a problem inherent to mobile crowdsourced translation however – two professional translators may also come up with different ways of phrasing something based on experience and personal preference.

An interesting observation is that the value in using crowdsourced translation is not only that one might not necessarily need to use traditional translation services but more that crowdsourced translation can give insight into how “normal” people understand a question in their mother tongue.

Another example:

Original English (not visible to Mobenzi agent): What is the household’s average monthly income?
Professionally Translated Zulu (shown to Mobenzi agent): Ungayilinganisela kumalini isiyonke imali engenayo ngenyanga?
Mobenzi Agent Translation (from Zulu to English): Estimate how much money do you earn a month?

In this case, the agent has incorrectly translated the question back into English by missing that the question wants to know the cumulative income of the household – not just that of the respondent. Now, this might be a mistake by the agent – but a mistake is not, in itself valueless. In this case, it can help a survey designer identify where confusion or areas of potential misunderstanding might occur in the real study. The response from the agent can guide the designer, in collaboration with their translator (professional or otherwise) in refining the question to ensure optimal wording which will make sense to the actual respondents.

We were rather pleased by the quality of translation provided by agents without any kind of moderation, statistical or other methods being applied but clearly there is still a lot of room for improvement in this area. We’ll continue working on some ideas before our main pilot takes place next year.

Like many other tech enthusiasts, I am a big fan of Moleskine notebooks & diaries. I decided to write each sentence of this article on a new page of my pocket size, yellow note book. For the sake of saving some very small trees, I will try to keep this article short. I think it will be very interesting to see how long it takes to type up each page. We will calculate the average time taken, per word & per character, to type (capture) this content. Each participant will type up each handwritten page so that we have a bit more data to assess time and quality.

Each sentence was hand-written on a separate page.

Results

The above text was typed up into a form on a mobile phone by three of the participants. There were a few minor errors (double spaces and some miss-typed characters) but the quality was decent. At least one of the three participants got each of the sentences correct without errors.

There were 8 sentences including the title with a total of 145 words or 777 characters (with spaces). It took participants just over 12 minutes on average to complete the 8 tasks. The average speed for the tasks was therefore between 12 and 13 words per minute (WPM).

One study of average computer users, the average rate for transcription was 33 words per minute, and only 19 words per minute for composition. In the same study, when the group was divided into “fast”, “moderate” and “slow” groups, the average speeds were 40 wpm, 35 wpm, and 23 wpm respectively. Wikipedia

This result (in terms of typing speed) is not very impressive (almost half the speed of a ‘slow’ typist) but we have to consider a few points.

• Participants had to capture META data about the task (the task number) and read an instruction about the task for each sentence. These steps could be avoided.
• The form used to capture the text did not make use of predictive text. Predictive text could definitely improve typing speed.
• The convenience of being able to type on your phone while you are travelling on public transport, walking, lying in bed etc. must definitely be taken into account in comparing typing on a personal computer to typing on a mobile phone.
• The types of HCR tasks we have been discussing do not involve typing up long documents, but rather short strings of text like those used by Recaptcha.

In summary, the accuracy was pretty good but I had hoped that the speed would have been better. We will have to conduct some further testing in this regard if we seriously consider targeting Human Character Recognition (HCR) as a business service to offer through Mobenzi. This is just one of many applications.

Siyanda brought together an excellent group of people who I really enjoyed working with today. Mbongwa (featured in the title image who calls himself Kingdom), Ayanda, Nobuhle, Nieh and Bonga are all between the ages of 20 and 26 and are all currently seeking part or full time employment. Their first language is isiZulu but they all speak English fluently.

The team of mentors who will help support the extra 20 pilot participants who will join us next week.

I had fairly high expectations for how easily the participants would pick up the concept and would be able to process tasks. We had discussed last week whether an introductory training session was necessary – to explain how the mobile application works, how to skip between questions and complete the various question types etc. But based on my interactions with youths from the area, I decided to try and see what progress the participants could make without any training at all.

I started the session by introducing myself to the team and giving them a brief overview of Mobile Researcher and how we created Mobenzi to try and leverage the platform for completing tasks. This took about 15 minutes and the team really picked the idea up quickly and were eager to get started. I handed out the 5 new Nokia 3120 classic phones and told them there was a shortcut to the application on the main screen.

The Nokia 3120 classic that we installed the Mobenzi application on for the pilot.

I was very encouraged to hear discussion about the make and model of the phone and it’s various features without me saying anything about it. In their community a person’s phone is a hugely significant status symbol and everyone seems to know about each others phones (it took some convincing to get them to agree to hand the phones back after each session).

Without any instruction, most of the participants had the application open and simply started completing tasks. Although I had high expectations, I still thought there would be many questions and a fairly slow start. But within half an hour of me arriving at the venue, the participants had their heads down and were completing tasks. A few questions popped up during the day, but none that the other participants couldn’t answer themselves.

It is difficult for us to understand how central a phone is to youth in communities like Kwanyuswa. Their familiarity with the technologies made the transition to ‘working’ on their phone completely natural.

I jotted down some notes from our discussions that illustrate how important phones are to them.

Everyone uses Mxit around here. Even our parents.

Mxit is a South African instant messaging system that millions of people use for cheap, quick communication on their phones. One of the guys said that he installed Mxit on his mom’s phone so that he could chat to her from home about what to buy when she goes shopping in town.

I installed Opera mini on my phone and at one stage used to spend over 8 hours a day browsing the internet and using applications. I used to spend at least R100 per week on airtime, but it was still cheaper than the internet cafes. I would only go to the internet cafes if I needed to print.

This was a quote from Kingdom who really flew through tasks today. I am definitely expecting experience with services like Mxit to play a huge role in how easily new Mobenzi agents can get started and how productive they are in their work.

The tasks themselves involved structuring free text sms messages by answering a series of questions about the sms. The participants varied in the time taken to complete each task, averaging at around 2 minutes (for about 5 questions per task). I did a brief analysis of the quality and I was very pleased to see almost 100% accuracy on the small set of tasks that I looked at.

I found the first day of the pilot incredibly interesting and I am now even more excited for the future of Mobenzi.

Many organisations receive thousands of inbound SMS messages

SMS text messaging is the most widely used data application on the planet, with 2.4 billion active users (Wikipedia). A growing number of organisations are using text messages to communicate with mobile phone users. Although most communications involve distributing information to end users, many organisations receive and process thousands of text messages from end users.

Once an SMS is received by an organisation, its content needs to be analysed and understood to enable reporting or features such as sending a relevant reply SMS. Dealing with a large volume of inbound SMS messages requires automated processing (efficient sorting of messages) to save time and money. Processing these SMS messages manually is often not feasible.

It is difficult for computers to understand messages written by people

Using computer programs to automatically sort sms messages involves either extremely advanced natural language processing (NLP) or end users adopting a specified syntax in their messages (e.g. Sms your NAME, followed by your AGE and then the KEYWORD…) so that the messages can be more easily understood by a computer.

Natural language processing is extremely complex

Natural language processing (NLP) is a field of computer science and linguistics concerned with the interactions between computers and human (natural) languages. Natural language understanding systems convert samples of human language into more formal representations such as parse trees or first-order logic structures that are easier for computer programs to manipulate. (Wikipedia )

Although there are many services like Google SMS and WolframAlpha that make use of NLP concepts, I don’t believe that we will see a generic natural language understanding solution that we can use for SMS messages in the near future, especially one that would cater for the many languages in developing countries. Such systems would still require extensive customisation in order for a computer to understand the meaning of an SMS within a particular context.

Structured syntax is not easy to use

People have invested significantly in making operating systems, computer programs, websites and hardware easy and intuitive to use. For many people however, plain text sms messages are the only interface they will have to online services such as banking, classifieds and social networks for the next few years.

The process of interfacing to a software system via sms is often difficult. The common approach of using structured syntax messages (requiring users to adopt a set of words and rules in their messages) is comparable to entering instructions at the DOS command prompt like many of us used to do 20 years ago. Although the command line as an interface is having somewhat of a comeback (“The Web Browser Address Bar is the New Command Line“),  it is usually only expert users who really make use of most commands.

Many people make use of structured syntax commands with applications like Twitter (“RT” or “@username” ect.), Google (“1USD in ZAR“)  and other services like IRC. But with all of these applications, understanding the structured syntax is not required for new users or the majority of interactions. With many sms systems however, users are required to understand the syntax right from their first interaction. I think it is quite unnatural, unintuitive and difficult for many people to use fixed syntax commands, especially via SMS which is predominantly used for communication between people.

Organisations also need to communicate the rules and language of the system to mobile phone users so that they understand what words they can use and how to structure their messages. It is very difficult to do this via SMS due to limitations in the number of characters in an SMS and the cost of each message, especially if a sequence of messages is required.

Where there are only a few commands or keywords that need to be learned, using structured syntax can work very well. But when SMS systems start offering more interactive services, they will become much more difficult to design and to use.

Perhaps the biggest challenge with any computer based processing of SMS messages is that users often expect that a person will read and understand their message and therefore use informal language (Textese), ask questions or otherwise make spelling and grammatical errors.

People can help computers understand SMS messages

Certain processing tasks, such as understanding an SMS as described above, are still performed better and faster by humans than by computers. But manually processing inconsistently large volumes of such tasks is not feasible for internal staff at many organisations.

This problem inspired us to create Mobenzi, a service that allows organisations to outsource these kinds of tasks to a distributed team of workers who could share the load of many organisations’ processing requirements.

Mobenzi is a software service that empowers people to be rewarded for completing simple tasks on their mobile phones.

We are currently piloting the system and are confident that we will be able to take some live projects on within the next few months. Although there are several business applications we are addressing with Mobenzi, sorting of SMS data is our initial focus.

The concept of human intelligence tasks (simple tasks that computers find difficult) was first popularised by Amazon with their Mechanical Turk web service. We also recently came across Textonic that is attempting to leverage Mechanical Turk for tagging SMS messages in a similar way to Mobenzi. We hope that Mobenzi will be very well suited to processing SMS data since the processing itself can be done on a mobile phone by a Mobenzi agent who speaks the same language as those people sending the SMS messages. We plan on making it very easy to set up new teams of Mobenzi agents in new locations so that the agents will understand the local languages and colloquial terms that are included in the data that needs to be processed.

Mobenzi will allow computer programs that receive SMS messages to seamlessly interface to real people who can interpret free text messages and return information such as categories, tags, and other structured data as part of an automated process. Once an SMS is received by an organisation, the message can be submitted to Mobenzi for processing (via an API call for example). A Mobenzi agent would then be sent the original free text SMS with an associated form (see example below) to extract and structure the relevant information contained in the message – obvious to a human observer but inaccessible to computer systems.

This task illustrates how a simple natural language sms can be processed by a Mobenzi agent to extract structured data from the message.

Find out more about how Mobenzi works.

As part of the current pilot phase, we are assessing the cost and quality issues involved in the completion of tasks. We hope to make the processing service very affordable, especially considering that the requesting system can save costs in the long run through learning from the results generated by Mobenzi.

We hope that Mobenzi will help make many more online services available to people whose only interface to the internet is via SMS.

If you have any specific requirements you’d like to explore using Mobenzi for, please contact us.