LP Agency

How we developed a specialist marketplace

with automated database filling

About the project

Type:
Web platform / Marketplace
Direction:
Specialist search services
Format:
Web + PWA for iOS and Android
Stack:
Laravel, JavaScript
Appendix 1:
Appendix 2:

Workers.an online platform for finding specialists and craftsmen in different regions. The user can find a suitable artist, study his profile, portfolio, rating and reviews, and then contact him.

At the same time, one of the key tasks of the project was not just to create a platform, but to quickly fill it with its own database of specialists.

To do this, we have developed a system for automated data collection from several external resources, processing the information received and then creating profiles of specialists on the platform.

Thus, the project combined several components at once: a marketplace of services, a regional catalog, a PWA application and a system for automatically generating a database of specialists.

rabotniki1

Task

It was necessary to create our own Russian-language platform for finding specialists with the possibility of further use on mobile devices.

Main tasks:

  • develop a catalog of specialists;
  • implement a search by region and category;
  • create full-fledged wizard profiles;
  • Add reviews, ratings, portfolios, and favorites;
  • implement personal accounts of users and specialists;
  • provide a mobile work scenario through PWA;
  • create an administrative part for managing the platform;
  • create an initial base of specialists automatically.

The last task was especially important: the new site could not work effectively without a sufficient number of proposals from the performers.

Decision

Specialist Search Platform

We have developed a full-fledged marketplace where the user can find a specialist in the required category and region.

The platform has implemented:

  • catalog of specialists;
  • search;
  • filtering;
  • regionality;
  • categories of services;
  • specialist profiles;
  • portfolio;
  • ratings;
  • reviews;
  • favourites;
  • applications;
  • orders;
  • personal accounts;
  • the opportunity to contact a specialist.

The regionality is built according to logic:

city / region → category → list of specialists → specialist profile.

This takes into account not only the specialists working in a particular region, but also the regions of their departure.

Specialist profile

The profile of the wizard has become one of the key entities of the platform.

Some of the information could be automatically generated based on data collected from external sites, after which the specialist had the opportunity to independently supplement and change the information.

The profile could be edited:

  • information about yourself;
  • categories of services;
  • prices;
  • portfolio;
  • regions of work and departure;
  • other profile data.

This made it possible to use the automatically collected information as the basis of the profile, rather than as an immutable copy of the data from a third-party site.

Automatic database generation

One of the most difficult parts of the project was the task of filling the new platform.

To do this, we have developed a system for parsing three external resources.:

  • poisk-pro.ru;
  • poisk-mastera.ru;
  • ru.

Each site had its own data structure, so instead of a universal parser, a single system with separate modules for each source was implemented.

Available data on specialists was collected:

  • name;
  • description;
  • experience;
  • rating;
  • region;
  • portfolio;
  • education;
  • reviews;
  • regions of departure;
  • contact information.

After receiving the information, the system performed post-processing and prepared the data for import into the Employees.online structure.

The scheme of work

External platforms → parsers → data processing → normalization → profile creation → database.online

However, data from different sources could differ in structure and format. The system adapted such differences, and the missing values could be formed in accordance with the logic of the platform.

After creating the profile, the specialist could make the necessary changes on his own.

rabotniki3

The parser management system

A separate administrative part of the parsers has been developed to manage data collection.

It allowed us to manage the process of obtaining information from external sources, and the main admin panel of the platform was already the specialists themselves, users and the contents of the service.

As a result, the system was divided into two levels.:

Parsers → generate and update data

The platform admin panel → manages the ready-made database

This made it possible not to mix the technical data collection process with the usual marketplace management.

Automatic activation of specialists

After forming the profile, it was necessary not only to add a specialist to the database, but also to attract him to a new site.

To do this, we have developed automatic sending of authorization data via WhatsApp using the Green API.

The scenario looked like this:

specialist found → data collected → profile created → account created → login and password sent to specialist → specialist logs on to platform → completes profile.

In other words, automation solved two tasks at once: filling the catalog and initially attracting performers to the site.

PWA application

The web platform was originally designed with mobile usage in mind.

Based on it, we implemented a PWA application for iOS and Android.

A Flutter wrapper was used for publishing, so that the web application could be distributed through mobile stores without developing two completely independent mobile applications.

The PWA implemented:

adaptive interface;

push notifications;

offline scenarios;

load optimization;

the main user functions of the platform.

The app has been published on Google Play and the App Store.

rabotniki2

The administrative system

An administrative part has been developed to manage the platform.

The administrator got the opportunity to work with:

  • by users;
  • by specialists;
  • by regions;
  • by categories;
  • profiles;
  • content;
  • reviews and ratings;
  • applications;
  • by orders;
  • other platform data.

A separate parser management system was provided, responsible for automatically receiving and processing data from external sources. resources.

Result

As a result, a full-fledged platform for searching and interacting with specialists was created, working simultaneously as a web service and a mobile PWA application.

Main results:

  • A marketplace of specialists has been developed;
  • The regional catalog structure has been implemented;
  • Performer profiles with portfolios, reviews, and ratings have been created.;
  • applications, orders, favorites, and personal accounts have been implemented;
  • A PWA version has been developed for iOS and Android.;
  • The apps are published in the App Store and Google Play;
  • An automatic data collection system has been created from three external sites;
  • Separate parsing modules have been developed for different source structures;
  • the processing and normalization of the received data has been implemented;
  • automated creation of specialist profiles;
  • a separate admin panel has been created to manage the parsers;
  • automatic sending of authorization data via WhatsApp has been implemented.

We have not just developed another catalog of specialists. We have created a mechanism for quickly launching a marketplace with an already formed database of offers.

Instead of manually adding each specialist, the system could independently go from searching for a performer on an external resource to creating his profile and sending him login information.

It is this bundle — marketplace + automatic data collection + normalization + profile creation + WhatsApp activation — that is the strongest technical feature of the project.