During a press event we held recently, we got a chance to get qualitative feedback from some of our agents about Mobenzi and what it means to them. At the time the majority of the tasks they were completing involved analysing the sentiment of Tweets about a few prominent South African brands. We were quite surprised to get so much feedback about how the nature of work itself seems to have a positive impact in agents’ lives.

Nokhuthula Njoko said that she feels empowered by being involved with Mobenzi.

“It helps me focus on the work I’m doing, knowing that for each task I complete I will be paid. I’ve learnt to be a critical thinker and enjoy the challenge of learning new words and abbreviations in different languages and in business terms. I’ve even made myself a book where I write down all the words I don’t understand, then later find their meanings. Mobenzi has given me employment, empowerment and an education in the business world”.

Trevor Ngcobo said being a Mobenzi agent enables him to study and still work at his own leisure.

“I never have to worry about transport problems, being late for work or not having time to attend college. I can make money, study and even do my Mobenzi tasks in a taxi on my way to lectures. It’s helped me in more ways than I thought when I first started”.

Msizi Phewa relishes in the fact that he can tell his peers that he ‘works on the internet and analyses data’.

“It makes me feel so important when I tell people that I work with analysing information from social networking sites like Twitter and Facebook. And because it’s something you can do in your spare time and you are paid for it, your mind is not focused on distractions of drugs, alcohol and hanging out with unproductive and negative people on the streets. I want the world to eventually be plugged into Mobenzi so that we can have an entire planet of productive people”.

It was great to hear such positive and interesting feedback.

Our first pilot project for Mobenzi ended on December 4^th 2009 and on the final afternoon we assigned a survey to the participants’ phones to find out information about them as well as their thoughts on the pilot.

Although we had 25 participants in the pilot, 2 members of the team were not present on the Friday afternoon. The following statistics are therefore based on the remaining 23 team members who completed the self-administered survey using the Nokia 3120 mobile phones we provided for the pilot.

Age, Gender and Language

The 25 pilot participants were all from the local community of Kwanyuswa. The average age of the team members was 24 and there was an even gender split. Each of the participants had completed grade 12 and could speak fairly good English. Their first language is isiZulu but each of them studied English as a second language.

Employment history

17 of the participants (70%) had never had a full time job at the time of running the pilot. A few participants had part time jobs but were able to make the 5 hour sessions each morning.

Household information

The 17 participants that were willing to answer questions about their households have on average 7 people living permanently at home. 16 homes had stoves (94%), 14 had running water (82%), 14 had a television (82%), only 10 owned a fridge (60%) and none of the households owned a motor vehicle.

Mobile phone usage

19 of the 23 participants (82%) owned their own mobile phone (53% Nokia, 21% Samsung, 16% LG). Most participants (60%) had used MXIT (a mobile instant messaging client) in the month preceding the pilot. 9 team members (40%) had used their phones within the last month to browse the web and download pictures, music or games. The average airtime expenditure per person over the preceding 3 months was R100 per month.

Demand for mobile tasks

If employed full time in another position, the participants expressed on average that they would probably like to do Mobenzi tasks for about 3.5 hours per weekday to subsidise other income. If working only part time in another position, the desired commitment increased to 5.5 hours. Over weekends the average expected commitment was 10 hours (Including Saturday and Sunday). This works out at between 27 and 37 hours per week. 5.5 hours of concentrated work is probably the ceiling for how much time someone could spend doing Mobenzi tasks in a single day.

Everyone agreed that most Mobenzi tasks would be completed at their homes, but most participants also mentioned they would probably complete tasks while on public transport (buses and taxis) and while walking around the local community.

Thoughts on Mobenzi

The major reason the participants noted for what they liked about Mobenzi was that the work was interesting and entertaining. Only one person answered that the work was boring. The biggest challenge the team raised was that some classification tasks were ambiguous and deciding on the most appropriate answer was sometimes very difficult.

Fatigue was a problem for some participants who mentioned that their hands started hurting by the end of the day or they battled to concentrate for so long (We ran the pilot for about 5 hours each day with short breaks every hour and a longer break for lunch).

The participants were generally very excited about Mobenzi. Some of their comments are included in a related article: Feedback from pilot participants about mobile tasks

Yesterday I aggregated some data from Twitter that referenced KFC, Nandos, Debonairs or McDonalds and sat with the Mobenzi pilot participants as we answered two simple questions about each tweet.

1. Was the message positive, negative or neutral in reference to the brand?
2. If it was negative, was it due to customer service, taste, health or some other reason?

The work was entertaining for the participants, they completed tasks efficiently and the results seem to be very accurate.

About sentiment analysis

With the growing use of online services like Twitter, blogs and forums, there is a vast amount of publicly available information generated by everyday people about millions of different topics (companies, products, movies etc.). Knowing the sentiment of messages (e.g. whether they are positive or negative) can be extremely valuable to the people or organisations involved, especially when monitoring trends over time.

Sentiment analysis or opinion mining refers to a broad area of natural language processing, computational linguistics and text mining. Generally speaking, it aims to determine the attitude of a speaker or a writer with respect to some topic.

The rise of social media such as blogs and social networks has fuelled interest in sentiment analysis. With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.

Find out more about sentiment analysis on Wikipedia

This kind of work is well suited to Mobenzi agents

Sentiment analysis seemed like a very appropriate type of task for processing by Mobenzi agents on their phones as tweets are very short (only 140 characters). We also felt that there would be a demand for an efficient human sentiment rating service since computer algorithms face many difficulties in trying to understand the tone of human messages.

Twitter includes a lot of slang, humour, Textese and other informal language that makes automated analysis especially difficult. In a multi-cultural country like South Africa, many tweets also combine words from a variety of local languages which would make analysis very challenging to a computer.

Example Tweets that reference take-out brands

These were some of the messages included in our sample set of data from Twitter.

"Damn you debonairs" in this context is meant in jest and is not negative.

The sentiment is not obvious in this message.

The tone in this tweet changes totally at the end of the message.

The results were ‘positive’

The focus of this study was to assess issues relating to the completion of tasks. We only looked at a small sample of tweets, and could have been a lot more scientific in our approach, so the sentiment results themselves should not be taken too seriously.

There were six participants (including myself) and we each stepped through the analysis of Twitter messages that mentioned KFC, Nandos, Debonairs or McDonalds. Each task took only a few seconds to complete and the team found the work interesting and engaging. None of the participants (except myself) use Twitter themselves, but they were all very familiar with the concept and frequently use Mxit which is similar in some aspects.

One of the measurements we look at to gauge the accuracy of results, is the agreement between different participants for the same task. All six participants rated the sentiment of each tweet, so we were able to look at where our answers differed. It was very encouraging to see that most answers had 100% agreement (Especially if we exclude where participants stated that they were unsure of the sentiment). There are only a few cases where we disagreed on whether a particular tweet was positive or negative. In these cases, the majority was correct and in some cases the disagreement actually helped to balance the rating where the sentiment was ambiguous.

The summary across all brands came out at 48% positive, 35% negative and 17% neutral or unclear. Of the negative tweets, 29% were service related, 16% to do with taste, 9% health related and the rest for other reasons.

These charts show the summary of the sentiment analysis for all four brands.

In the following results, we excluded tweets that were either neutral or unclear with regard to sentiment. Out of the four brands, Nandos was clearly the favourite with 80% of tweets being rated as positive.

Breakdown of Positive versus Negative sentiment for each brand

To have a look at what people are saying right now about these brands, simply go to www.twitter.com and search for #Nandos, Debonairs, #KFC or #Mcdonalds.

Interestingly, a quick analysis of these keywords on Tweetsentiments.com (A service that attempts to automate the analysis of tweets) returns fairly similar results in terms of rank, but with some significant variations in the actual sentiment rating. Nandos: 68% positive, Debonairs 59% positive, Mcdonalds 56% positive and KFC 52% positive. The ranking of the brands is the same as our result, except that Mcdonalds moved in front of KFC with the automated analysis. This may have to do with the fact that we only looked at tweets in English and other South African dialects. Perhaps English speaking people are the least positive about Mcdonalds? Looking at some of the tweets in their data sets, I would trust our result over the automated one. Try the service out yourself at  http://tweetsentiments.com/analyze

It is no coincidence that we ended up having Nandos and Debonairs for lunch.