Such a different online advertising. Three axes vs. AI, Ad Tech and DOOH

“The best minds of the generation are busy making people click on ads,” Jeff Hammerbacher, a Facebook engineer, told me. - The horror ... "- Ilon Musk (Tesla, SpaceX and the road to the future).

Our minds in IPONWEB are doing the same.

And here we have a smart business logic that decides which ads to show to the user, powerful clusters for analyzing big data, complex gluing and match on the fly, a learning algorithm that predicts clicks, conversions, and even a potential price, a complex budget calculation algorithm who is able to optimize the rates in real time, ... and the idea of ​​advertising technologies in RuNet still forms Azino 777. Do not be so, we thought, and decided to start talking about real advertising technologists on Habré x.

The area in which we have been operating since 2007 is called AdTech, “advertising technology.” It is very diverse and constantly evolving, it involves hundreds of thousands of companies around the world - from small “pocket” advertising agencies to the giants of the big three (Google, Facebook, Amazon). We specialize in programmatic advertising, which implies automated algorithmic purchase or sale of advertising. Today, these technologies are already implemented in satellite and digital television, on the streets in the form of digital billboards, in taxis, airplanes, on screens in shopping centers, in Internet radio and online games. In general, wherever there are people, there is advertising, and where there is advertising, you can use programmatic.

According to eMarketer , 86% of all digital advertising in the United States in 2020 will be purchased through automated platforms. If you "turn on Dudya" and translate it all into money, you get 65 billion dollars - and this is only in the USA. In Russia, the volume of the programmatic advertising market is 20 times less, but it also grows by 22% per year, despite crises.

The picture below shows the major players in the marketing and advertising market, some of which form the global programmatic advertising market (about 6800 companies in total) "from a bird's-eye view." If for some reason you want to consider each individually, then here is a picture in high quality of tyts .



Since 2010, the programmatic market began to grow rapidly, more and more money flowed there from traditional advertising channels - and the AdTech boom began, and RTB and that was it.

AdTech is one of the fastest growing industries in the world; one cannot stand still here because the market is rapidly becoming saturated with new players.

And where are we?

Today IPONWEB is developing software for several dozen companies: ( DSP and SSP ) that monetize advertising for their clients (advertisers, agencies, exchange networks, site owners) or directly for the same clients (among them, for example, Automattiс, WordPress developer .com), regardless of whether it is a matter of advertising to a specific user or of selling the advertising space most profitably, we make decisions for both the dark and the bright side for both sides of the market.

Each client has its own business model and unique tasks, they work with different companies, with different audiences, with different types of advertising (banners, videos, text, native advertising, interactive formats, billboards), traffic sources (websites, mobile applications, outdoor advertising, games, TV, lok-screens of smartphones) and in different markets, it is very important for us to understand the specifics at the technical and business levels and sharpen the system for a specific client.

Simplified, all interactions within the system are as follows (using the example of a platform that buys ads):

1. Decide what advertising to show right now and how much to pay for it.
2. Save information about what advertising was shown, process it for further use in machine learning and show it in the report.

Most of these interactions are built using the real-time openRTB protocol.

RTB is already on TV and on the streets

Online advertising is not at all limited to advertising on the site or in the application, it can be shown on any device that at least sometimes connects to the network to get information about the auctions played or the deals made. We are already doing projects to show personalized and automated advertising on cable and smart TV and in public places, this is called Digital Out Of Home, abbreviated DOOH.

With these displays, a range of new tasks emerges, because there are no user cookies, here the audience data needs to be obtained in other ways, and the impressions are measured. On the other hand, in public places, the security requirements for advertising are higher, thousands of people, including children, will immediately see unwanted advertising, so you need to set up a preliminary advertising verification system and sell it through private transactions.

Like programmatic-DOOH, programmatic-TV is far from mainstream, according to estimates by eMarketer in 2018 only 3 percent of television inventory is sold through automated auctions, most of the advertising is still sold in the form of traditional contracts concluded in the sales department of the channel, office or by e-mail, as planned in a spreadsheet in Excel. But, given the penetration of TV into the lives of people around the world, this segment of the programmatic market is very promising and is growing rapidly - in the United States over the past 5 years, the volume of advertising purchases through the programmatic-TV channel has increased 7 times, to two billion dollars, and in 2019 promises to show an increase of almost 100%.

And what is all this written on?

Of course, to solve such a range of tasks requires a lot of different technologies:

• The whole system is based on a highly loaded HTTP server written in C ++, it processes requests for advertisements, requests for bids in real time and writes logs. A lot of logs. Terabytes of logs per day.
• Business logic written in Lua. It determines how requests will be processed, which creatives will go to the auction, who will be targeted and what will be the rate, who will win.
• MongoDB / TokuMX is used as the user data storage.
• Logs are processed and transformed into aggregates using Hadoop / Spark, aggregates contain data calculated by unique keys and cleared of duplication for other components to work, for example, an algorithm and an application predicting the likelihood of an event that optimizes the magnitude of the rate. All this is done in our own ETL cluster written in Java and using Apache Spark.
• The rest of the offline work with logs, for example, pasting and matching records according to a certain algorithm, is done by an internal service written in Java / Scala.
• The user interface is a combination of Python / Django on the backend and Angular + TypeScript on the frontend.
  

There are still some number of services, monitoring systems, working with data, building metrics and reporting, but in this article I do not want to run a gallop around Europe, ask questions in the comments about specific components, and we will make more detailed information about them.