During a press event we held recently, we got a chance to get qualitative feedback from some of our agents about Mobenzi and what it means to them. At the time the majority of the tasks they were completing involved analysing the sentiment of Tweets about a few prominent South African brands. We were quite surprised to get so much feedback about how the nature of work itself seems to have a positive impact in agents’ lives.

Nokhuthula Njoko said that she feels empowered by being involved with Mobenzi.

“It helps me focus on the work I’m doing, knowing that for each task I complete I will be paid. I’ve learnt to be a critical thinker and enjoy the challenge of learning new words and abbreviations in different languages and in business terms. I’ve even made myself a book where I write down all the words I don’t understand, then later find their meanings. Mobenzi has given me employment, empowerment and an education in the business world”.

Trevor Ngcobo said being a Mobenzi agent enables him to study and still work at his own leisure.

“I never have to worry about transport problems, being late for work or not having time to attend college. I can make money, study and even do my Mobenzi tasks in a taxi on my way to lectures. It’s helped me in more ways than I thought when I first started”.

Msizi Phewa relishes in the fact that he can tell his peers that he ‘works on the internet and analyses data’.

“It makes me feel so important when I tell people that I work with analysing information from social networking sites like Twitter and Facebook. And because it’s something you can do in your spare time and you are paid for it, your mind is not focused on distractions of drugs, alcohol and hanging out with unproductive and negative people on the streets. I want the world to eventually be plugged into Mobenzi so that we can have an entire planet of productive people”.

It was great to hear such positive and interesting feedback.

On May 26th we invited some representatives from the press to the launch of the next phase of our Mobenzi pilot project in KwaNyuswa (Valley of a thousand hills, KZN).

During our two week trial run in December 2009, agents had completed Mobenzi tasks using our company owned phones, under supervision and together at a central location. Since May 26th however, a group of agents have been working independently as private contractors to Mobenzi.

These are some of the major factors that make the launch of this phase of the pilot a significant step forward.

1. Agents are now working in their own time, requesting batches of tasks whenever they have a few minutes spare.
2. They complete tasks while at home, travelling on public transport or even between lectures at college.
3. Most of the agents are using their own mobile phones after having installed the Mobenzi application from a link we sent to them.
4. With each task that agent’s complete, associated credit is built up in their account. Once credit reaches a certain thresh-hold, funds are disbursed electronically to their phones using FNB’s SendMoney platform. Although some agents had to borrow our company phones, many have already earned enough income from Mobenzi to purchase their own, brand new compatible Nokia phones.

These changes in the way the pilot is being run are allowing us to test the scalability of the concept. We can now manage recruitment of new agents, assignment of tasks, monitoring of quality and disbursement of funds all from our central office.

With this platform in place, it is only the demand from businesses for the services of our agents that will slow the growth of Mobenzi.

One of the first examples to demonstrate how Mobenzi can be used in a commercial context will be through the launch of a new vertical solution for researchers called SMS Insight.

Traditionally, when it comes to gathering data via SMS, researchers have two rather ungainly options if they wish to collect anything more than the simplest one word response:

1. Require that respondents structure their feedback in some way (e.g. comma delimited) or;
2. Solicit answers one SMS at a time, with responses still needing to be confined to a reasonably limited vocabulary.

But what if you could ask a question (or several questions) in an advert, SMS or display and allow respondents to answer in free text – but still be able to analyse the data using quantitative techniques? That’s precisely what SMS Insight offers: the ability to derive structured results from free text responses. Here’s how:

1. Pose the question

Your message can be communicated via SMS, on TV or radio, in print ads or outdoor displays, on product packaging etc.

e.g. An advert asks: “What would you change about our company? SMS your name with your ideas to 35xxx”

2. Solicit feedback in plain text

Respondents simply reply via SMS – in their own words – with no fixed formats, keywords or codes required.

e.g. “I wish your staff would be more friendly. Niki”

3. Use human intelligence to crunch the responses

Our panel of mobile workers use their innate human ability to structure, classify and quantify each response by answering simple questions based on your exact analysis requirements.

e.g. “Is the response about our products, prices or service?”; “What is the respondent’s name and if possible, their gender?”

4. Integrate and act on the results

What was once raw, unusable SMS data is transformed into rich, structured information that can be used to inform business strategy, evaluate performance and identify opportunities.

e.g. “14% Products, 12% Price, 74% Service”

Interested in conducting research using SMS Insight? Please get in touch with us to discuss your requirements.

About Mobenzi

Mobenzi is a software service that empowers people to be rewarded for completing simple tasks on their mobile phones. These tasks involve certain types of problems that are difficult for a computer to solve without assistance from a real person – even someone without expert knowledge of the problem.

Find out more about how Mobenzi works

Purpose of the pilot project

For two weeks we equipped pilot participants with the Mobenzi software application installed on standard mobile phones to assess whether they could effectively complete simple business tasks using only their phones.

These were some of the guiding questions we were attempting to answer during the pilot.

• Is the concept easy to understand?
• Is the technology easy to use?
• What types of tasks are feasible?
• What types of people are most suitable for doing Mobenzi tasks?
• What is the best way to present a given task to an agent?
• How long does it take to complete different types of tasks?
• What quality should be expected in the results of completed tasks?
• What issues are involved that may affect attrition rates (fatigue, boredom etc)?
• Could the service grow through viral expansion (Can participants teach each other)?
• Based on other findings, what are the financial implications with regard to agent remuneration and the cost of the service to organisations?

Project location and venue

We ran the pilot project from the Light Providers community centre in KwaNyuswa. The area lies on the outskirts of urban development, west of the Inanda Dam, about 40 minutes outside of Durban in KwaZulu Natal, South Africa. It is one of the largest of the various tribal authorities that make up the Valley of a Thousand Hills.

Due to the gross unemployment rates in the region, and our close proximity to the area (Only 14km from our office), we selected KwaNyuswa as the location for our pilot project.

Format of the pilot

We started the first week of the pilot with 5 participants who would later act as mentors when 20 new recruits joined them for the second week. We spent the first week testing out various types of human intelligence tasks and discussing issues surrounding understanding the use of the mobile application as well as the various types of tasks themselves.

During the second week we had more participants to help work through large sets of tasks. We assigned participants various types of tasks and recorded completion times and responses for all participants so that we could crunch the data to assess what factors affect quality and efficiency.

We focused on text-based human intelligence tasks

We decided to focus on “Text to Form” tasks for the pilot project. These types of tasks involve extracting structured data from free-text.

Some examples of this type of task include:

• Categorising SMS survey responses into reportable data.
• Sentiment Rating of “Tweets” (Messages on Twitter).
• Classifying text based job and product advertisements.

For all of these tasks, we displayed a short instruction for the task, followed by the content (such as an SMS or a tweet) and then a series of questions about the content (Such as whether the SMS included a person’s name). The participant worked through each task one step at a time.

Find out about other types of human intelligence tasks

Results of the first phase of our pilot project

One of the critical factors affecting the feasibility of Mobenzi is whether or not the mobile application is easy to use for people who have had little exposure to the internet and other software applications. A quote from the summary of the first day of the pilot shows how easily the participants understood both the concept of doing work on their phones as well as how to use the application itself:

Without any instruction, most of the participants had the application open and simply started completing tasks. Although I had high expectations, I still thought there would be many questions and a fairly slow start. But within half an hour of me arriving at the venue, the participants had their heads down and were completing tasks. A few questions popped up during the day, but none that the other participants couldn’t answer themselves.

Using the software to complete tasks came very naturally and required almost zero training. From the participant comments, it is also clear that there would be a huge demand for Mobenzi tasks. I believe we could easily find thousands of Mobenzi agents who already own compatible phones within just half an hour’s drive of our offices in Hillcrest, let alone the rest of South Africa and the world.

We have not yet done much analysis on the quality or efficiency of the completed tasks, but initial assessments are very positive. Over the next few weeks we will be crunching the data to help answer some more of the questions we outlined at the start of the pilot.

The results so far have exceeded our expectations and at this stage I would guess that our biggest challenge in moving forward will be to generate a sufficient supply of tasks to keep Mobenzi agents busy.

Scaling up the pilot in April 2010

This pilot was a short 2 week project to get an early feel for what to expect. In April next year we will scale our efforts up and take on a much larger group of participants to pilot the concept further. Until then we will be tweaking the software and preparing the systems to handle the logistics of a much larger project.

We are very open to suggestions if you have any ideas for types of tasks or even real world data that we could get Mobenzi agents to process during our pilot later this year.

Workin wit mobenzi ws great n hp w’l start soon. Al d best:-)

This is a great example of Textese (’SMS language’ involving abbreviations and slang). This comment translates to regular English as ‘Working with Mobenzi was great and I hope We’ll start soon. All the best. (said with a smile)’
[61 vs 93 characters = 34% compression].

Mobenzi is a good program/organisation which will bring many job opportunities to people, its interesting and entertaining and at the same time its challenging you to think before answering each question. Last but not least it will improve English language for many people who work with mobenzi because most of time it all about English

This was a very positive comment from one of the participants. Internally, we had discussed the potential impact Mobenzi work could have on education (such as English comprehension), but we certainly never expected participants to pick that up as a benefit during a short pilot project (it’s becoming very clear that we should stop underestimating participants).

Establish marketing strategies for mobenzi to ensure availability of tasks and more employment.

This participant seemed eager to see us succeed and offered some business advice.

I would like to work for Mobenzi.!!!

No comments, it will be a previlage working at mobenzi.

It was fun ,challenging and informative about the world that we live in.

No comment everything is new and perfect I enjoy mobenzi.

Mobenzi is very interesting and it challenges my knowledge in English and makes you think. But mostly it’s going to give us some sort of employment. THUMBS UP MOBENZI!!

This final comment sums up the sentiment of the team. I don’t think we could have expected a more positive reception to the project from the participants themselves.

Yesterday I aggregated some data from Twitter that referenced KFC, Nandos, Debonairs or McDonalds and sat with the Mobenzi pilot participants as we answered two simple questions about each tweet.

1. Was the message positive, negative or neutral in reference to the brand?
2. If it was negative, was it due to customer service, taste, health or some other reason?

The work was entertaining for the participants, they completed tasks efficiently and the results seem to be very accurate.

About sentiment analysis

With the growing use of online services like Twitter, blogs and forums, there is a vast amount of publicly available information generated by everyday people about millions of different topics (companies, products, movies etc.). Knowing the sentiment of messages (e.g. whether they are positive or negative) can be extremely valuable to the people or organisations involved, especially when monitoring trends over time.

Sentiment analysis or opinion mining refers to a broad area of natural language processing, computational linguistics and text mining. Generally speaking, it aims to determine the attitude of a speaker or a writer with respect to some topic.

The rise of social media such as blogs and social networks has fuelled interest in sentiment analysis. With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.

Find out more about sentiment analysis on Wikipedia

This kind of work is well suited to Mobenzi agents

Sentiment analysis seemed like a very appropriate type of task for processing by Mobenzi agents on their phones as tweets are very short (only 140 characters). We also felt that there would be a demand for an efficient human sentiment rating service since computer algorithms face many difficulties in trying to understand the tone of human messages.

Twitter includes a lot of slang, humour, Textese and other informal language that makes automated analysis especially difficult. In a multi-cultural country like South Africa, many tweets also combine words from a variety of local languages which would make analysis very challenging to a computer.

Example Tweets that reference take-out brands

These were some of the messages included in our sample set of data from Twitter.

"Damn you debonairs" in this context is meant in jest and is not negative.

The sentiment is not obvious in this message.

The tone in this tweet changes totally at the end of the message.

The results were ‘positive’

The focus of this study was to assess issues relating to the completion of tasks. We only looked at a small sample of tweets, and could have been a lot more scientific in our approach, so the sentiment results themselves should not be taken too seriously.

There were six participants (including myself) and we each stepped through the analysis of Twitter messages that mentioned KFC, Nandos, Debonairs or McDonalds. Each task took only a few seconds to complete and the team found the work interesting and engaging. None of the participants (except myself) use Twitter themselves, but they were all very familiar with the concept and frequently use Mxit which is similar in some aspects.

One of the measurements we look at to gauge the accuracy of results, is the agreement between different participants for the same task. All six participants rated the sentiment of each tweet, so we were able to look at where our answers differed. It was very encouraging to see that most answers had 100% agreement (Especially if we exclude where participants stated that they were unsure of the sentiment). There are only a few cases where we disagreed on whether a particular tweet was positive or negative. In these cases, the majority was correct and in some cases the disagreement actually helped to balance the rating where the sentiment was ambiguous.

The summary across all brands came out at 48% positive, 35% negative and 17% neutral or unclear. Of the negative tweets, 29% were service related, 16% to do with taste, 9% health related and the rest for other reasons.

These charts show the summary of the sentiment analysis for all four brands.

In the following results, we excluded tweets that were either neutral or unclear with regard to sentiment. Out of the four brands, Nandos was clearly the favourite with 80% of tweets being rated as positive.

Breakdown of Positive versus Negative sentiment for each brand

To have a look at what people are saying right now about these brands, simply go to www.twitter.com and search for #Nandos, Debonairs, #KFC or #Mcdonalds.

Interestingly, a quick analysis of these keywords on Tweetsentiments.com (A service that attempts to automate the analysis of tweets) returns fairly similar results in terms of rank, but with some significant variations in the actual sentiment rating. Nandos: 68% positive, Debonairs 59% positive, Mcdonalds 56% positive and KFC 52% positive. The ranking of the brands is the same as our result, except that Mcdonalds moved in front of KFC with the automated analysis. This may have to do with the fact that we only looked at tweets in English and other South African dialects. Perhaps English speaking people are the least positive about Mcdonalds? Looking at some of the tweets in their data sets, I would trust our result over the automated one. Try the service out yourself at  http://tweetsentiments.com/analyze

It is no coincidence that we ended up having Nandos and Debonairs for lunch.

Many organisations receive thousands of inbound SMS messages

SMS text messaging is the most widely used data application on the planet, with 2.4 billion active users (Wikipedia). A growing number of organisations are using text messages to communicate with mobile phone users. Although most communications involve distributing information to end users, many organisations receive and process thousands of text messages from end users.

Once an SMS is received by an organisation, its content needs to be analysed and understood to enable reporting or features such as sending a relevant reply SMS. Dealing with a large volume of inbound SMS messages requires automated processing (efficient sorting of messages) to save time and money. Processing these SMS messages manually is often not feasible.

It is difficult for computers to understand messages written by people

Using computer programs to automatically sort sms messages involves either extremely advanced natural language processing (NLP) or end users adopting a specified syntax in their messages (e.g. Sms your NAME, followed by your AGE and then the KEYWORD…) so that the messages can be more easily understood by a computer.

Natural language processing is extremely complex

Natural language processing (NLP) is a field of computer science and linguistics concerned with the interactions between computers and human (natural) languages. Natural language understanding systems convert samples of human language into more formal representations such as parse trees or first-order logic structures that are easier for computer programs to manipulate. (Wikipedia )

Although there are many services like Google SMS and WolframAlpha that make use of NLP concepts, I don’t believe that we will see a generic natural language understanding solution that we can use for SMS messages in the near future, especially one that would cater for the many languages in developing countries. Such systems would still require extensive customisation in order for a computer to understand the meaning of an SMS within a particular context.

Structured syntax is not easy to use

People have invested significantly in making operating systems, computer programs, websites and hardware easy and intuitive to use. For many people however, plain text sms messages are the only interface they will have to online services such as banking, classifieds and social networks for the next few years.

The process of interfacing to a software system via sms is often difficult. The common approach of using structured syntax messages (requiring users to adopt a set of words and rules in their messages) is comparable to entering instructions at the DOS command prompt like many of us used to do 20 years ago. Although the command line as an interface is having somewhat of a comeback (”The Web Browser Address Bar is the New Command Line“),  it is usually only expert users who really make use of most commands.

Many people make use of structured syntax commands with applications like Twitter (”RT” or “@username” ect.), Google (”1USD in ZAR“)  and other services like IRC. But with all of these applications, understanding the structured syntax is not required for new users or the majority of interactions. With many sms systems however, users are required to understand the syntax right from their first interaction. I think it is quite unnatural, unintuitive and difficult for many people to use fixed syntax commands, especially via SMS which is predominantly used for communication between people.

Organisations also need to communicate the rules and language of the system to mobile phone users so that they understand what words they can use and how to structure their messages. It is very difficult to do this via SMS due to limitations in the number of characters in an SMS and the cost of each message, especially if a sequence of messages is required.

Where there are only a few commands or keywords that need to be learned, using structured syntax can work very well. But when SMS systems start offering more interactive services, they will become much more difficult to design and to use.

Perhaps the biggest challenge with any computer based processing of SMS messages is that users often expect that a person will read and understand their message and therefore use informal language (Textese), ask questions or otherwise make spelling and grammatical errors.

People can help computers understand SMS messages

Certain processing tasks, such as understanding an SMS as described above, are still performed better and faster by humans than by computers. But manually processing inconsistently large volumes of such tasks is not feasible for internal staff at many organisations.

This problem inspired us to create Mobenzi, a service that allows organisations to outsource these kinds of tasks to a distributed team of workers who could share the load of many organisations’ processing requirements.

Mobenzi is a software service that empowers people to be rewarded for completing simple tasks on their mobile phones.

We are currently piloting the system and are confident that we will be able to take some live projects on within the next few months. Although there are several business applications we are addressing with Mobenzi, sorting of SMS data is our initial focus.

The concept of human intelligence tasks (simple tasks that computers find difficult) was first popularised by Amazon with their Mechanical Turk web service. We also recently came across Textonic that is attempting to leverage Mechanical Turk for tagging SMS messages in a similar way to Mobenzi. We hope that Mobenzi will be very well suited to processing SMS data since the processing itself can be done on a mobile phone by a Mobenzi agent who speaks the same language as those people sending the SMS messages. We plan on making it very easy to set up new teams of Mobenzi agents in new locations so that the agents will understand the local languages and colloquial terms that are included in the data that needs to be processed.

Mobenzi will allow computer programs that receive SMS messages to seamlessly interface to real people who can interpret free text messages and return information such as categories, tags, and other structured data as part of an automated process. Once an SMS is received by an organisation, the message can be submitted to Mobenzi for processing (via an API call for example). A Mobenzi agent would then be sent the original free text SMS with an associated form (see example below) to extract and structure the relevant information contained in the message – obvious to a human observer but inaccessible to computer systems.

This task illustrates how a simple natural language sms can be processed by a Mobenzi agent to extract structured data from the message.

Find out more about how Mobenzi works.

As part of the current pilot phase, we are assessing the cost and quality issues involved in the completion of tasks. We hope to make the processing service very affordable, especially considering that the requesting system can save costs in the long run through learning from the results generated by Mobenzi.

We hope that Mobenzi will help make many more online services available to people whose only interface to the internet is via SMS.

If you have any specific requirements you’d like to explore using Mobenzi for, please contact us.