How we migrated a million products to WordPress

The team faced several challenges:

• migrate an existing website to WordPress;
• save the directory structure and contents;
• transfer about 1 million items;
• save the full information for each card;
• to adapt the category structure to the customer’s requirements;
• implement a convenient search through a huge catalog;
• ensure an acceptable website speed;
• adapt the visual concept of the original solution to the new version of the site.

However, there was no need to use a standard online store. The catalog was supposed to work as an independent system, so it was implemented on its own WordPress framework without WooCommerce.

The existing customer’s website was used as the basis, but the new version was not copied literally.

According to the customer’s requirements, the visual concept was changed: the color scheme was based on the style of another corporate resource, and individual interface elements were adapted to the new structure.

We have independently developed and implemented the layout of the main pages and sections of the catalog.

The final structure includes:

• main page;
• product catalog;
• Product categories;
• product cards;
• section of polymers;
• reference books;
• search;
• information pages;
• the application submission form.

Special attention was paid to the product card, since with such a large number of items, it is the main point of user interaction with the catalog.

The most extensive part of the project was data migration.

It was impossible to get a million product cards manually, so we developed our own parser system to automatically collect information from the source resource.

The parsers collected complete information on the products:

• title;
• article number;
• description;
• specifications;
• images;
• category;
• data of a specific variation;
• other information present in the source card.

After the data was collected, it was processed and compared with the new category structure.

The overall process was as follows:

source directory → parsers → data processing → category mapping → import → WordPress card generation.

This made it possible to automate the transfer of a large amount of information and eliminate the need for manual creation of cards.

As a result, a database of about 1 million product cards was created.

The usual online store model was redundant in this case, so we developed our own structure for the catalog.

This made it possible to separate the CMS tasks from the tasks of searching and working with a large amount of product data.

WordPress was used as the main content management platform, and the catalog was adapted to the specifics of industrial products and a large number of variations.

Pagination, category structure, and display of search results were implemented separately.

After transferring a large amount of data, a key technical problem arose.

The standard WordPress search was not designed for such a large catalog.

With heavy search queries, the system could generate results within 3-5 minutes.

For the user, this actually meant that it was impossible to use the catalog normally.

The problem was especially evident on broad requests.

For example, about 800 thousand product items were suitable for the request «pipes».

It was necessary to make sure that having hundreds of thousands of suitable results did not turn the search into a multi-minute operation.

To solve the problem, we moved the search to a separate search engine — Elasticsearch.

At the same time, WordPress continued to serve as a CMS and data source, and Elasticsearch was directly responsible for quick catalog search.

The scheme of work was as follows:

User → site search → Elasticsearch → index search → results → pagination on the site.

This made it possible not to force WordPress to perform heavy search operations directly on the main database.

At the same time, the user saw a regular catalog: the search results were displayed in pages, despite the fact that the potential number of matches could be hundreds of thousands of positions.

Elasticsearch wasn’t the only change.

When working with a catalog of this scale, several levels of the system have been optimized.:

• database structure;
• SQL queries;
• caching;
• catalog operation;
• pagination;
• processing of search results;
• images;
• the server configuration.

The main task was not just to transfer a million items, but to make it possible to work with them normally.

As a result, heavy search queries that previously could have taken 3-5 minutes were reduced to about 5-8 seconds.

A separate task was to compare the existing product lines with the categories of the new catalog.

The category structure was provided by the customer, so when importing, it was necessary to determine the appropriate category for each item.

This was especially important because of the large number of similar products and variations.

As a result, the user has the opportunity to move through the catalog both through the category structure and through the search.

As part of the project, we:

• migrated an existing website to WordPress;
• we have developed our own B2B catalog structure;
• about 1 million product cards were transferred;
• we have developed our own parsers for automatic data collection;
• automated the processing and import of the product base;
• we have saved the full information on the cards;
• implemented the category structure;
• We have developed product cards;
• implemented a huge search the catalog;
• integrated Elasticsearch;
• optimized the database and SQL queries;
• caching and pagination have been implemented;
• optimized the images and the server side;
• reduced the time of heavy search from 3-5 minutes to 5-8 seconds.

The project has shown that WordPress can be used not only for small corporate websites, but also as the basis for a large B2B catalog with hundreds of thousands and millions of product items — provided the architecture is correct, a separate search layer and comprehensive optimization.

The main value of the project is not the migration to WordPress itself, but the solution to the productivity problem when working with a catalog of ~ 1 million cards.

This volume is already closer to our normal cases. And I would continue to keep this structure as standard: short block → introductory → task → design/interface → development → individual complex technical tasks → functionality → optimization → result.