Magento 2: import products from external sources

Magento is an e-commerce solution, i.e. more focused on the sale of products than on the accompanying sales of warehouse, logistics or financial accounting. For a companion, other applications are better suited (for example, ERP systems). Therefore, quite often in the practice of using Magento, the problem arises of integrating the store with these other systems (for example, with 1C).

By and large, integration can be reduced to data replication by:

• catalog (products, categories);
• inventory data (stock balances and prices);
• customers;
• orders;

Magento for data manipulation in the database offers a separate class of objects - repositories . Due to the specifics of Magento, adding data to the database via repositories is easily encoded, but it happens, let's say, not fast. In this publication, I look at the main stages of programmatically adding a product to Magento 2 in a “classical” way - using repo classes.

Customers and orders are usually replicated in the other direction - from Magento to external ERP systems. Therefore, with them easier, on the side of Magento, you just need to select the relevant data, and then - " from our side the bullets flew out ."

Principles of data recording in the database

At the moment, the creation of objects stored in the database in a programmatic way in Magento is done through the Factory :

function __construct (\Magento\Cms\Model\BlockFactory $blockFactory) { $this->blockFactory = $blockFactory; } /** @var \Magento\Cms\Model\Block $block */ $block = $this->blockFactory->create(); 

and writing to the database via the Repository :

 function __construct (\Magento\Cms\Api\BlockRepositoryInterface $blockRepo) { $this->blockRepo = $blockRepo; } $this->blockRepo->save($block); 

The approach using "Factory" and "Repository" can be used for all major models in the Magento 2 subject area.

Basic Product Information

I am considering a data structure that corresponds to the version of Magento 2.3. The most basic information about the product is in the catalog_product_entity table (product registry):

 entity_id attribute_set_id type_id sku has_options required_options created_at updated_at 

I confine type_id='simple' one product type ( type_id='simple' ), a set of attributes by default ( attribute_set_id=4 ) and ignore the attributes has_options and required_options . Since the attributes entity_id , created_at and updated_at generated automatically, then, in essence, for us to add a new product, it is enough to set sku . I do this:

 /** @var \Magento\Catalog\Api\Data\ProductInterfaceFactory $factProd */ /** @var \Magento\Catalog\Api\ProductRepositoryInterface $repoProd */ /** @var \Magento\Catalog\Api\Data\ProductInterface $prod */ $prod = $factProd->create(); $prod->setAttributeSetId(4); $prod->setTypeId('simple'); $prod->setSku($sku); $repoProd->save($prod); 

and get an exception:

 The "Product Name" attribute value is empty. Set the attribute and try again. 

I add the product name to the request and I get a message that the Price attribute is missing. After adding the price, the product falls into the base:

 $prod = $factProd->create(); $prod->setAttributeSetId(4); $prod->setTypeId('simple'); $prod->setSku($sku); $prod->setName($name); $prod->setPrice($price); $repoProd->save($prod); 

The product name is stored in the varchar-attribute table of the product ( catalog_product_entity_varchar ), the price is in the catalog_product_entity_decimal table. Before adding a product, it is desirable to explicitly indicate that we are using an administrative storefront to import data:

 /** @var \Magento\Store\Model\StoreManagerInterface $manStore */ $manStore->setCurrentStore(0); 

Optional Attributes

Processing additional product attributes with Magento is a pleasure. The EAV data model for basic entities (see table eav_entity_type ) is one of the key features of this platform. Simply add the appropriate attributes to the product model:

 $prodEntity->setData('description', $desc); $prodEntity->setData('short_description', $desc_short); // или $prodEntity->setDescription($desc); $prodEntity->setShortDescription($desc_short); 

and while saving the model through the repo object:

 $repoProd->save($prod); 

additional attributes will also be saved in the corresponding database tables.

Inventory data

In a simple way - the amount of product in stock. In Magento 2.3, the structures in the database that describe the format for storing inventory data differ significantly from what was previously. However, adding a quantity of a product in stock through a product model is not much more difficult than adding other attributes:

 /** @var \Magento\Catalog\Model\Product $prodEntity */ /** @var \Magento\Catalog\Api\ProductRepositoryInterface $repoProd */ $inventory = [ 'is_in_stock' => true, 'qty' => 1234 ]; $prodEntity->setData('quantity_and_stock_status', $inventory); $repoProd->save($prodEntity); 

Media

As a rule, media support for a product for a customer in a store (e-commerce) is different from media support for the same product for an employee in an internal accounting system (ERP). In the first case, it is desirable to show the "product face", in the second - enough to give a general idea of ​​the product. However, transferring at least the primary product image is a fairly common case when importing data.

When adding an image through the admin panel, the image is first stored in the temporary directory ( ./pub/media/tmp/catalog/product ) and only when the product is saved is moved to the media catalog ( ./pub/media/catalog/product ). Also, when adding an image through the admin small_image , image , small_image , thumbnail , swatch_image .

 /** @var \Magento\Catalog\Api\ProductRepositoryInterface $repoProd */ /** @var \Magento\Catalog\Model\Product\Gallery\CreateHandler $hndlGalleryCreate */ /* $imagePath = '/path/to/file.png'; $imagePathRelative = '/f/i/file.png' */ $imagePathRelative = $this->imagePlaceToTmpMedia($imagePath); /* reload product with gallery data */ $product = $repoProd->get($sku); /* add image to product's gallery */ $gallery['images'][] = [ 'file' => $imagePathRelative, 'media_type' => 'image' 'label' => '' ]; $product->setData('media_gallery', $gallery); /* set usage areas */ $product->setData('image', $imagePathRelative); $product->setData('small_image', $imagePathRelative); $product->setData('thumbnail', $imagePathRelative); $product->setData('swatch_image', $imagePathRelative); /* create product's gallery */ $hndlGalleryCreate->execute($product); 

For some reason, the media is tied up only after first saving the product and retrieving it from the repository again. And you need to specify the label attribute when adding an entry to the media gallery of the product (otherwise we get the exception Undefined index: label in .../module-catalog/Model/Product/Gallery/CreateHandler.php on line 516 ).

Categories

Often the structure of store categories and backend applications or the placement of products in them can vary considerably. Strategies for transferring data about categories and products in them depend on many factors. In this example, I stick to the following:

• categories of backend and store are matched by name;
• if a category is imported that is not in the store, then it is created under the root category ( Default Category ) and its further positioning in the store catalog is assumed to be manually;
• the binding of a product to a category occurs only when it is created in a store (first import);

Basic information about the category is in the catalog_category_entity table (category catalog). Creating a category in Magento:

 /** @var \Magento\Catalog\Api\Data\CategoryInterfaceFactory $factCat */ /** @var \Magento\Catalog\Api\CategoryRepositoryInterface $repoCat */ $cat = $factCat->create(); $cat->setName($name); $cat->setIsActive(true); $repoCat->save($cat); 

Binding of a product to a category is carried out by category ID and product SKU:

 /** @var \Magento\Catalog\Model\CategoryProductLinkFactory $factCatProdLink */ /** @var \Magento\Catalog\Api\CategoryLinkRepositoryInterface $repoCatLink */ $link = $factCatProdLink->create(); $link->setCategoryId($catMageId); $link->setSku($prodSku); $repoCatLink->save($link); 

Total

Writing code to add products to Magento 2 programmatically is quite easy. All of the above, I brought in the demo module " flancer32 / mage2_ext_demo_import ". There is only one fl32:import:prod console command in the fl32:import:prod that imports the products described in the " ./etc/data/products.json " JSON file:

 [ { "sku": "...", "name": "...", "desc": "...", "desc_short": "...", "price": ..., "qty": ..., "categories": ["..."], "image_path": "..." } ] 

Pictures for import are in the directory ./etc/data/img .

Import time 10 products in a similar way is about 10 seconds on my laptop. If you develop this idea further, it is easy to come to the conclusion that about 3,600 products can be imported per hour, and it can take about 30 hours to import 100K products. Replacing the laptop on the server allows you to smooth out the situation somewhat. Maybe even at times. But not by orders of magnitude. Maybe this speed slowness is to some extent one of the reasons for the emergence of the magento / async-import project.

A direct solution to the database can be a cardinal solution for increasing the speed of import, but in this case all the “bugs” related to Magento extensibility are lost - you have to do everything “extended” yourself. However, it is worth it. If it works out, then I will consider the approach with direct writing to the database in the next article.