February 25, 2015 at 17:03One and a half years of work with SAP hybris: flight is normal. The most important thing you need to know about developing on eCommerce platforms
It so happened that the last year and a half I have been working closely with SAP hybris . In Russia, there is great interest in ecommerce platforms, so I decided to tell this article simply and readily about this topic based on my experience.
Foundation . To create any complex software, you need to prepare a “foundation” that will subsequently make the development manageable, and the entire system visible and understandable for the architect. This foundation is a "multi-layer cake" of universal, standard and well-established technologies and products. The correct choice of this set largely determines the development of the system for the coming years. Examples of the components of such a “pie” - ORM , CMS , PCM , Search Engine, from specific technologies - Hadoop , Apache ServiceMix , NodeJSother. The set of these technologies is often determined by the experience of the development team, and not just and not so much by the needs of the business, so you can often find systems on Scala , Erlang or Haskell . In eCommerce platforms, such a zoo of technologies is often found - Java, C ++, Perl, C #. It turns out that when a vendor acquires various components from third companies or the entire company. We were lucky with the platform - there is only Java.
Thus, the eCommerce platform is an organic, prepared, customized, debugged, packaged and documented set of such technologies. For many typical tasks in e-commerce, the vendor has developed ready-made blocks that require only a small “fit” for the task, and some are implemented at an abstract level and require “file cutting”. The more organically interconnected the technologies, the more thoughtful the architecture, the easier it will be to expand it to fit your tasks in the coming years.
Will we write everything ourselves?Of course, you can develop a store without an industrial platform, collecting the “pie” yourself. The question is how long it will take and whether it will be possible to maintain good architecture, scalability, performance, security, extensibility, reliability, and documentation in a year or two. A good and rare example when it was possible for large e-commerce is Amazon.com . Experience shows that with today's requirements for quality, security, functionality and the pace of development, writing everything from scratch ultimately comes out very expensive, long and risky.
Do you need an eCommerce platform?The owners of online stores before deciding on a platform should think about where they see their business in about five years. If in this future there are the words “multiregionality”, “multilingualism”, “multicurrency”, “huge range”, “big traffic”, “personalization”, “hundreds of warehouses”, “hundreds of employees in the process”, then today you need to look for a platform supporting all of the above.
When using the platform, the daunting task of building a large online store becomes quite visible and manageable. Features of business automation, integration with internal systems, specific business processes, user interface, features of goods and services come to the fore. The cost of development and implementation is a substantial part of these components.
The timing. You can’t even name the average time, but you can call the minimum time from your own experience. Our team has successful experience when the system is put into production 3 months after signing the contract. The first release in "production" was two months after the start of work - a catalog without the possibility of ordering goods. The team on such a project totaled six people, including me. There are other examples where the team and terms are longer.
For e-commerce, the project deadline is very important. If it is possible to increase the team, reducing the time - this should be done (as F. Brooks said , this is not always the case). While we are developing a new site, the development of an existing customer will never be stopped. Therefore, each additional month creates enormous risks that the project will not be launched.
Cost. You can roughly estimate the cost of such a project by multiplying the planned duration of the project, the average team size and the average cost of a man-day of specialists. The cost of licenses is usually lower than the opportunity cost of buying, developing and integrating into a single complex of software comparable in functionality.
In order not to be mistaken with the choice of eCommerce platform, you should pay attention to
I would not advise paying much attention to:
On the one hand, the use of the platform allows you to launch the store in a matter of months. On the other hand, not a single e-commerce platform is a ready-made store template that was bought today, the checkmarks in the admin panel clicked and started. That is, technically it is possible, of course ... But in 100% of cases, additional programming is required for features - like the client’s internal information systems, his processes, our market, our user experience, data migration automation, etc.
The eCommerce system includes interaction with the buyer through various channels - from the call center and web storefront to the mobile application and kiosks.
E-commerce master data management . This includes a set of software for managing Internet commerce master data - content, stocks and other important e-commerce objects. Some components of this block, such as goods management, can be used outside the online store as an independent system. The platform implementation project includes the configuration and expansion of this software.
Web showcase. On the one hand, an online store is a large but website. Design concept, design layouts, back-office, front-end - HTML-layout, javascript-automation, data exchange with internal systems, content - these are typical stages familiar to any website project.
Business processes and workflow . On the other hand, these are business processes for automating e-commerce. This includes working with a catalog of goods, and marketing, warehouses, logistics and a call center. When a lot of people participate in the sale of goods, it is important to ensure a transparent and reliable document management system.
Other customer interaction channels. The mobile application comes in the form of a working prototype and a set of APIs. The version for kiosks and the version for mobile devices are additional versions of a web showcase that use almost the same functionality and the same data as the main website, but in a different “wrapper”.
Special functionality . This includes such important topics as returns, defective goods, partial payment, loyalty systems, timing, cost and delivery options, trigger and mass mailings.
Search. In the field of e-commerce, search is one of the most important components of the system, as it directly affects the conversion of visitors to customers.
Each of the above topics has a “workpiece” in the platform of varying degrees of sophistication, it can be a module ready for connection, or it can be “semi-finished”. Ready-made modules have a minus in the fact that they can not always be easily changed for specific requirements, and it is more difficult to study them - the ambiguity and opacity of the logic requires more time to study and prototype. "Semi-finished products" do not have this problem, but they require more time to configure and refine. But they have a plus in that the system is slimmer, safer and more reliable due to a more seamless and scalable architecture.
On the basis of one “semi-finished product”, hundreds of different options for the implementation of functionality can be made, and only one of them can be implemented in a demo store, which is presented to customers at their first meetings. That is why you can’t call what you saw in the demo store “best practice”, since this is just one of the possible implementations of a very flexible mechanism.
The professionalism of the architect of the eCommerce system consists in understanding this flexibility, limitations and opportunities. We almost never tell customers “no, this cannot be done”, because on modern ecommerce systems absolutely everything can be done at one price or another.
The eCommerce platform includes a set of back office tools that automate the work of specialists involved in eCommerce processes from the online store - call center operators, marketer, content manager, product manager and others. Since many of these components are often refined for specific processes, ideally they should be implemented on a single technology, with common interfaces and approaches to expansion.
Large platforms were originally designed for large amounts of data, for complex business processes, for high traffic, productivity and availability. For example, clustering and caching are performed at the industrial level.
The platform has thousands of places where you can “wedge” a programmer with your own logic, redefine or expand the standard behavior of the system. The development of an online store is the design, development and testing of such modules separately and as part of a platform. As you can see, there are two boundary options here - to replace all business logic with your own or use the one supplied in the distribution kit.
The development uses well-known technologies such as JSP / Spring / Java, which simplifies the connection of programmers to the project without experience with a specific eCommerce platform.
From the point of view of the Internet buyer, a search is the receipt of goods or pages of a site in response to the keywords indicated by it. The closer the search results to his query, the higher the likelihood that he will buy from you, not from competitors. Therefore, all major online stores are continuously working on improving search.
You can write the search yourself from scratch. So does the vast majority of online stores with a small assortment and traffic. But with the growth of the product base and traffic, the search is slower, the online store is losing customers, and, starting at some point, it becomes clear that you need to completely rewrite the mechanism to a more complex one or connect an external “search engine” (search engine).
Simply put, the “search engine” is a specialized database that stores information about products and pages so that it can then be pulled out quickly and often, using fairly complex queries if necessary. This feature allows you to use the “search engines” not only for searching by keywords, but also for searching, for example, by the characteristics of goods or to display a list of goods of a selected category. Search with a gradual refinement of the request through a convenient customer interface - a must have for any major store. For non-platform stores, this functionality is one of the bottlenecks in terms of performance and flexibility.
Among the "search engines" in the field of e-commerce are respected Apache SOLR, ElasticSearch, Endeca, Sphinx. Connecting to the online store of the search engine can be quite a time-consuming procedure, if everything is done as it should. In e-commerce platforms, this issue is usually resolved with one of the products in the out of the box version.
For example, Apache SOLR is used in all our projects, and all product listings are also built on requests to it. This approach allows you to have a huge margin in performance.
A product catalog is the main thing that an online store begins with. Product data can be used, for example, on price tags in a store or in a printed catalog. It is good practice to store so-called “Enriched data on goods” - images, technical specifications (including dynamic), attached files - in a separate system, a specialized knowledge base for goods. Such systems may be a separate product, or may be delivered as part of platforms.
The common name for such systems is PIM (Product Information Management) or PCM (Product Content Management).
These systems do not have a direct relationship to the online store, as their purpose is to automate the management of information about products and their groups. If you plan to display them on a website, then this is already the case of the content management system (CMS), which I will discuss in the next section.
A typical project for developing an online store on an eCommerce platform includes setting up and implementing PIM / PCM, as well as integration with internal accounting systems. Typically, catalog development is the first thing a project starts with.
For example, in the online store RIV GOSHsome products have a unit of measure “grams”, there are options for color and volume, the product can be tied to several categories from different hierarchies of categories, some of which are not navigation. Products can be difficult to relate to each other, and some of the information can be downloaded from various sources automatically.
The content management system is responsible for the layout and display of pages. This is also a mandatory component of any eCommerce platform, since, as mentioned above, an online store is also just a big site. Tasks such as placing banners, adding a menu item, adding a page with information, personalizing the display of individual blocks and many others are performed in the CMS.
The eCommerce platform already includes many ready-made components ("widgets") that are very likely to be useful in any store, such as a "basket", "product listing", "product card" and others. Also, using CMS, they manage both trigger mailings and pages of the mobile version.
CMS also includes personalization of pages and components. The system will compose pages in different ways for different users, following exactly the logic. For a customer who likes the brand on Facebook, when they enter the site, a corresponding banner will be displayed, the main page will be branded, and a new item will appear in the menu.
A typical project for developing an online store on an eCommerce platform includes the development, processing, or customization of the widget components listed above.
For example, in Project 1, the Platform for the Technonikol CMS company produces different page templates and widget components for the mobile / desktop version, while the functionality of the pages and components is identical.
This is the most eCommerce part of the platform. After the order has been placed and possibly paid by the buyer, its processing chain is launched. In a sense, this is a document flow: an order can be divided into several, orders for specific warehouses should be formed for each, contingencies such as “oh, the goods are gone” or “the order is suspicious, must be checked before shipment” should be processed.
This is the most “abstract” part of the platform, containing a lot of what is almost impossible to show at the demonstration, because this will not work without integration. Here, like an iceberg, there are few visible user interfaces, but a very large and massive "underwater" part.
A typical project for developing an online store on an eCommerce platform includes designing or studying business processes, setting up order processing logic, integration with WMS, with payment gateways, uploading an order to ERP or an external order management system.
The most important thing about e-commerce platforms
Foundation . To create any complex software, you need to prepare a “foundation” that will subsequently make the development manageable, and the entire system visible and understandable for the architect. This foundation is a "multi-layer cake" of universal, standard and well-established technologies and products. The correct choice of this set largely determines the development of the system for the coming years. Examples of the components of such a “pie” - ORM , CMS , PCM , Search Engine, from specific technologies - Hadoop , Apache ServiceMix , NodeJSother. The set of these technologies is often determined by the experience of the development team, and not just and not so much by the needs of the business, so you can often find systems on Scala , Erlang or Haskell . In eCommerce platforms, such a zoo of technologies is often found - Java, C ++, Perl, C #. It turns out that when a vendor acquires various components from third companies or the entire company. We were lucky with the platform - there is only Java.
Thus, the eCommerce platform is an organic, prepared, customized, debugged, packaged and documented set of such technologies. For many typical tasks in e-commerce, the vendor has developed ready-made blocks that require only a small “fit” for the task, and some are implemented at an abstract level and require “file cutting”. The more organically interconnected the technologies, the more thoughtful the architecture, the easier it will be to expand it to fit your tasks in the coming years.
Will we write everything ourselves?Of course, you can develop a store without an industrial platform, collecting the “pie” yourself. The question is how long it will take and whether it will be possible to maintain good architecture, scalability, performance, security, extensibility, reliability, and documentation in a year or two. A good and rare example when it was possible for large e-commerce is Amazon.com . Experience shows that with today's requirements for quality, security, functionality and the pace of development, writing everything from scratch ultimately comes out very expensive, long and risky.
Do you need an eCommerce platform?The owners of online stores before deciding on a platform should think about where they see their business in about five years. If in this future there are the words “multiregionality”, “multilingualism”, “multicurrency”, “huge range”, “big traffic”, “personalization”, “hundreds of warehouses”, “hundreds of employees in the process”, then today you need to look for a platform supporting all of the above.
When using the platform, the daunting task of building a large online store becomes quite visible and manageable. Features of business automation, integration with internal systems, specific business processes, user interface, features of goods and services come to the fore. The cost of development and implementation is a substantial part of these components.
The timing. You can’t even name the average time, but you can call the minimum time from your own experience. Our team has successful experience when the system is put into production 3 months after signing the contract. The first release in "production" was two months after the start of work - a catalog without the possibility of ordering goods. The team on such a project totaled six people, including me. There are other examples where the team and terms are longer.
For e-commerce, the project deadline is very important. If it is possible to increase the team, reducing the time - this should be done (as F. Brooks said , this is not always the case). While we are developing a new site, the development of an existing customer will never be stopped. Therefore, each additional month creates enormous risks that the project will not be launched.
Cost. You can roughly estimate the cost of such a project by multiplying the planned duration of the project, the average team size and the average cost of a man-day of specialists. The cost of licenses is usually lower than the opportunity cost of buying, developing and integrating into a single complex of software comparable in functionality.
How to choose an ecommerce platform?
In order not to be mistaken with the choice of eCommerce platform, you should pay attention to
- western successful projects . Why to the west? Because they exist longer than Russian ones, and they have long passed the path that Russian e-commerce is going on today.
- the number of implementations in Russia over the last 2-3 years . Many platforms in the world have received a high rating because they have been developed for a long time, and have collected many implementations over the history of their existence. Others have appeared recently. A rich history can be a sign of both good accumulated experience and a large amount of “from the nineties” code. Why in Russia? eCommerce in Russia is very different from Western.
- “Openness” and accessibility of pie technology . It’s bad when the customer has no other alternatives than contacting the vendor or partner for support of the launched site. It’s good when there is a choice - to assemble and train your team or work with an experienced partner.
- the number of companies and specialists in the market . Assess the potential, how many will be given the existing pace of development in a year or two.
- Frequency of release of versions by a vendor . If the platform is updated only once a year or less, it is worth considering what its priority is among other vendor products.
- standard integration mechanisms . If the platform has an API, this greatly facilitates the work of integration.
- complexity and cost of scaling . Increase all your numbers by traffic and SKU volume every hundred and evaluate how much licenses will cost, how many servers will be needed, and whether the platform will cope with it at all. Be sure, not a hundred, but ten times your business should grow in terms of these indicators in five years. If there are no such ambitions, you hardly need a platform)
- quality of documentation . It is important that the documentation includes both a block for business users and programmers, reflects the difference in versions, is up-to-date, complete, accessible and well-structured.
- Gartner and Forrester ratings . These companies are recognized leaders who are trusted in the market.
I would not advise paying much attention to:
- Ready-made modules . eCommerce in Russia is so different and so dynamic that ready-made modules are more a headache than a blessing. It is better to look for ready-made “frameworks” for a task + documented prototypes for solving specific problems on them. For example, the integration module with payment systems “in general” with an example for one payment system is better than a set of disparate integration modules with any specific payment systems.
- Ready-made design templates . Appearance in large e-commerce is never an extension of demo stores, it is always replaced by a new, client-specific one. Taking standard visual templates as a basis is always more difficult, incorrect, and longer than re-creating them for a specific design and specific functional requirements. But this approach requires more experience than the completion of ready-made templates.
Fast start?
On the one hand, the use of the platform allows you to launch the store in a matter of months. On the other hand, not a single e-commerce platform is a ready-made store template that was bought today, the checkmarks in the admin panel clicked and started. That is, technically it is possible, of course ... But in 100% of cases, additional programming is required for features - like the client’s internal information systems, his processes, our market, our user experience, data migration automation, etc.
What is a project on an ecommerce platform?
The eCommerce system includes interaction with the buyer through various channels - from the call center and web storefront to the mobile application and kiosks.
E-commerce master data management . This includes a set of software for managing Internet commerce master data - content, stocks and other important e-commerce objects. Some components of this block, such as goods management, can be used outside the online store as an independent system. The platform implementation project includes the configuration and expansion of this software.
Web showcase. On the one hand, an online store is a large but website. Design concept, design layouts, back-office, front-end - HTML-layout, javascript-automation, data exchange with internal systems, content - these are typical stages familiar to any website project.
Business processes and workflow . On the other hand, these are business processes for automating e-commerce. This includes working with a catalog of goods, and marketing, warehouses, logistics and a call center. When a lot of people participate in the sale of goods, it is important to ensure a transparent and reliable document management system.
Other customer interaction channels. The mobile application comes in the form of a working prototype and a set of APIs. The version for kiosks and the version for mobile devices are additional versions of a web showcase that use almost the same functionality and the same data as the main website, but in a different “wrapper”.
Special functionality . This includes such important topics as returns, defective goods, partial payment, loyalty systems, timing, cost and delivery options, trigger and mass mailings.
Search. In the field of e-commerce, search is one of the most important components of the system, as it directly affects the conversion of visitors to customers.
Architecture
Each of the above topics has a “workpiece” in the platform of varying degrees of sophistication, it can be a module ready for connection, or it can be “semi-finished”. Ready-made modules have a minus in the fact that they can not always be easily changed for specific requirements, and it is more difficult to study them - the ambiguity and opacity of the logic requires more time to study and prototype. "Semi-finished products" do not have this problem, but they require more time to configure and refine. But they have a plus in that the system is slimmer, safer and more reliable due to a more seamless and scalable architecture.
On the basis of one “semi-finished product”, hundreds of different options for the implementation of functionality can be made, and only one of them can be implemented in a demo store, which is presented to customers at their first meetings. That is why you can’t call what you saw in the demo store “best practice”, since this is just one of the possible implementations of a very flexible mechanism.
The professionalism of the architect of the eCommerce system consists in understanding this flexibility, limitations and opportunities. We almost never tell customers “no, this cannot be done”, because on modern ecommerce systems absolutely everything can be done at one price or another.
The eCommerce platform includes a set of back office tools that automate the work of specialists involved in eCommerce processes from the online store - call center operators, marketer, content manager, product manager and others. Since many of these components are often refined for specific processes, ideally they should be implemented on a single technology, with common interfaces and approaches to expansion.
Large platforms were originally designed for large amounts of data, for complex business processes, for high traffic, productivity and availability. For example, clustering and caching are performed at the industrial level.
Basic principles of development
The platform has thousands of places where you can “wedge” a programmer with your own logic, redefine or expand the standard behavior of the system. The development of an online store is the design, development and testing of such modules separately and as part of a platform. As you can see, there are two boundary options here - to replace all business logic with your own or use the one supplied in the distribution kit.
The development uses well-known technologies such as JSP / Spring / Java, which simplifies the connection of programmers to the project without experience with a specific eCommerce platform.
Search
From the point of view of the Internet buyer, a search is the receipt of goods or pages of a site in response to the keywords indicated by it. The closer the search results to his query, the higher the likelihood that he will buy from you, not from competitors. Therefore, all major online stores are continuously working on improving search.
You can write the search yourself from scratch. So does the vast majority of online stores with a small assortment and traffic. But with the growth of the product base and traffic, the search is slower, the online store is losing customers, and, starting at some point, it becomes clear that you need to completely rewrite the mechanism to a more complex one or connect an external “search engine” (search engine).
Simply put, the “search engine” is a specialized database that stores information about products and pages so that it can then be pulled out quickly and often, using fairly complex queries if necessary. This feature allows you to use the “search engines” not only for searching by keywords, but also for searching, for example, by the characteristics of goods or to display a list of goods of a selected category. Search with a gradual refinement of the request through a convenient customer interface - a must have for any major store. For non-platform stores, this functionality is one of the bottlenecks in terms of performance and flexibility.
Among the "search engines" in the field of e-commerce are respected Apache SOLR, ElasticSearch, Endeca, Sphinx. Connecting to the online store of the search engine can be quite a time-consuming procedure, if everything is done as it should. In e-commerce platforms, this issue is usually resolved with one of the products in the out of the box version.
For example, Apache SOLR is used in all our projects, and all product listings are also built on requests to it. This approach allows you to have a huge margin in performance.
Product Management
A product catalog is the main thing that an online store begins with. Product data can be used, for example, on price tags in a store or in a printed catalog. It is good practice to store so-called “Enriched data on goods” - images, technical specifications (including dynamic), attached files - in a separate system, a specialized knowledge base for goods. Such systems may be a separate product, or may be delivered as part of platforms.
The common name for such systems is PIM (Product Information Management) or PCM (Product Content Management).
These systems do not have a direct relationship to the online store, as their purpose is to automate the management of information about products and their groups. If you plan to display them on a website, then this is already the case of the content management system (CMS), which I will discuss in the next section.
A typical project for developing an online store on an eCommerce platform includes setting up and implementing PIM / PCM, as well as integration with internal accounting systems. Typically, catalog development is the first thing a project starts with.
For example, in the online store RIV GOSHsome products have a unit of measure “grams”, there are options for color and volume, the product can be tied to several categories from different hierarchies of categories, some of which are not navigation. Products can be difficult to relate to each other, and some of the information can be downloaded from various sources automatically.
Content management system
The content management system is responsible for the layout and display of pages. This is also a mandatory component of any eCommerce platform, since, as mentioned above, an online store is also just a big site. Tasks such as placing banners, adding a menu item, adding a page with information, personalizing the display of individual blocks and many others are performed in the CMS.
The eCommerce platform already includes many ready-made components ("widgets") that are very likely to be useful in any store, such as a "basket", "product listing", "product card" and others. Also, using CMS, they manage both trigger mailings and pages of the mobile version.
CMS also includes personalization of pages and components. The system will compose pages in different ways for different users, following exactly the logic. For a customer who likes the brand on Facebook, when they enter the site, a corresponding banner will be displayed, the main page will be branded, and a new item will appear in the menu.
A typical project for developing an online store on an eCommerce platform includes the development, processing, or customization of the widget components listed above.
For example, in Project 1, the Platform for the Technonikol CMS company produces different page templates and widget components for the mobile / desktop version, while the functionality of the pages and components is identical.
Order fulfillment and business processes
This is the most eCommerce part of the platform. After the order has been placed and possibly paid by the buyer, its processing chain is launched. In a sense, this is a document flow: an order can be divided into several, orders for specific warehouses should be formed for each, contingencies such as “oh, the goods are gone” or “the order is suspicious, must be checked before shipment” should be processed.
This is the most “abstract” part of the platform, containing a lot of what is almost impossible to show at the demonstration, because this will not work without integration. Here, like an iceberg, there are few visible user interfaces, but a very large and massive "underwater" part.
A typical project for developing an online store on an eCommerce platform includes designing or studying business processes, setting up order processing logic, integration with WMS, with payment gateways, uploading an order to ERP or an external order management system.