How can companies assign monotonous AI tasks to optimize employee performance and help businesses scale

Spam, scammers, prohibited goods and services, funny or incorrect ads will scare any user away from the service or application. A business that wants to expand its customer base does everything to ensure that only relevant and necessary information reaches the user. 

The heart of the IT products of the Kolesa Group are ads for the sale and purchase of cars, real estate, goods and services. Every day, the company's users submit 83 thousand ads that need to be checked for correctness. Who processes such a huge flow of information and how?

We talked with Indira Aldenova, head of the moderation department at the Kolesa Group. The department consists of 39 people. Every year, the flow of ads increases, the business grows, and the moderation department grows numerically minimally. How do they manage to scale up and deal more effectively with incorrect ads without inflating the staff?Taking her experience as an example, she will talk about the introduction of AI (artificial intelligence) into moderation and what results it brought them. Further from her words.

Everything that comes to the service is checked: ads, reviews, comments, cabinets and applications from specialists.

Manual moderation works according to the classical scheme.

1. The ad gets checked by the moderator. He's asking questions:

- Is there anything in this photo that shouldn't be visible?

- Is there any forbidden words or private information in this message? etc.

2. If violations are found, the moderator can decide how to correct or reject them.

It would seem that manual moderation is easier and cheaper, but it's not about economic benefits. The main thing in manual moderation is the accumulation of historical data and building a system for working with this data, which will form the basis of a future ML model.

Tip 1 Collect data in data sets from the very beginning. Because you can build ML models based on them. We recommend collecting at least 10,000 data points to start working on ML. A data point is a photo, text, or any other unit of data.

Thanks to the well—structured work with manual moderation, we do not build the self-moderation process from scratch - it goes in parallel.

We mark up data for ML models. Why: the more the model "sees" examples of what we think is right or wrong, the better it will learn. And he will be able to take a significant part of the work on himself in the future.

Self-moderation is carried out by text and photo. 

Text self-moderation is updated with a list of stop words and a selection of parameters. For example:

- One model determines whether the category is indicated correctly in the ad so that spare parts are not supplied to the car;

- Another model is responsible for checking the text for unwanted words in the ad;

- The third model is responsible for the adequacy of the price in the offer. After all, scammers often hide behind cheap ads. Etc.

Photomoderation consists of 8 different ML models: people, screenshots, duplicates, weapons, etc. The logic of its work:

1. The user submits an ad.

2. Auto-moderation downloads all photos of this ad.

3. Determines the order of the model to check the photos.

4. If at some stage the model finds an error in the photo, then further verification is stopped and the result is given.

Possible answers: 

- publish

- submit for manual moderation.

Self-moderation does not reject photos on its own.

But the AI doesn't always understand what to do with the ad. In this case, the ad is sent for manual moderation. Self-moderation takes over 85% of the entire ad stream.

Tip 2 Make the suggestions from self-moderation comprehensive and understandable. Thanks to these hints, for example, "Animals detected", we understand exactly what confused the AI during the check. The more informative the self-moderation will be to send ads for manual moderation, the more effective the data collection process will be for retraining models.

Let's look at the work of self-moderation using the example of a product Kolesa.kz . The main facaps of self-moderation are in the photo:

Case 1 — UFOs

Recently we had an "alien invasion". Users submitted comic ads, and photos were posted on social networks. The reason is photos that the AI cannot recognize.

Case 2 — AI recognizes spare parts as animals, weapons.

We collect such cases and take them to retrain the model. 

The main focus of the implementation of self—moderation is the release of team resources for: 

• More thorough verification of suspicious ads to protect users from scammers;

• improve the quality of ads;

• faster and better processing of requests from users.

Thanks to automation, moderators were able to focus on more intellectual work. Exactly:

1. They help to analyze moderation indicators to analyze cases and improve efficiency. 

2. They conduct master classes, workshops, and exchanges of experience in order to have as many high-quality ads as possible.

3. Increase the speed of checking ads

4. Participate in growth and development programs. This gives them the opportunity to grow into team leaders or make a career transition to another IT direction.

The quality of manual moderation is 99.9%

The quality of self—moderation is 99.4%.

There are 60+ models in the products of the Kolesa Group. Work on retraining ML models, as well as improving the quality and efficiency of self-moderation is ongoing.

1. Reduced the time for checking ads from 15 minutes to a couple of seconds.

2. Reduced the time of manual moderation from 15 to 10 minutes.

3. Reduced the load on moderators — self-moderation takes over 85% of the text and photos.

4. By automating routine processes, we were able to reallocate resources to other tasks. 

1. Improving the quality of self-moderation.

2. Error analysis, retraining of the model by marking up images / text.

3. Training on the main triggers with ads from scammers, send them to manual moderation for additional verification.

When implementing AI into your processes, it is important to remember that maintaining its effectiveness is a constant process of analyzing and working through errors.