Aram Zucker-Scharff, a developer who works at Salon, was recently testing an ad unit when it broke the page he was on. He then went on Twitter and published an essay about his views of the ad tech industry today.
I reached out to Aram and asked if he could expound on some of his tweeted remarks because I thought they were insightful and because I’ve heard many similar conversations in recent months. Below is a lightly edited version of a conversation we had via Slack about the major issues facing publishers and suggested strategies for thinking about how to gain better control over programmatic ads.
I saw your Twitter series on ad tech and was wondering if you could talk a little bit about what inspired it.
I work with Salon now, but I have specialized in working with smaller publishers for a long time. Working with smaller publishers often means that I get the opportunity (and look for the opportunity) to work with all the silos, business, tech, editorial. So advertising tech has been on my mind for a long time now.
You started your tweets by saying “People underestimate just how impossible it is for publishers to use ad networks & avoid serving malicious ads. So little control.” Can you talk a little bit about that lack of control?
That started because I was testing an ad unit and it broke the page. But it isn’t just a single unit, I’ve worked with devs who have had to struggle with preventing ads from crashing pages and I’ve had to do so myself.
Basically the problem is that even when I’ve been in organizations where ad server control, tech development, and editorial all lives in the same office, there is very little that can be done when users start reporting getting hit by a bad advertisement. There’s no way for a developer to go on to a page and see all the ads that have been served, or to block a bad ad they spot in one click. And when a malicious ad gets past the loose controls on a network, it can hit many, many publishers.
I know a lot of people who read this column are on the editorial side of things and may not realize what the devs in their newsrooms are dealing with. Can you go into a little more detail about how this works?
Ok well, basically most ads on the average publisher are network ad slots. The New York Times may be able to have direct sales fill most of their units, but most of us use ad networks that live on ad servers like Doubleclick For Publishers (DFP, run by Google).
A significant portion of ads on most of the sites people visit aren’t placed individually. You have ad networks placing them programmatically, sometimes more than one. I’ve worked with publishers who run two or three. But you can run more, and some do.
When you open the console on your browser — right click, click inspect element, then go to the console tab — most sites will show a lot of events, and usually broken code running on those sites, sometimes continually.
So there aren’t checks in place on either end to ensure that malicious ads can’t get through?
Usually ad servers claim they run some sort of checks, but considering just how many malicious or badly formed ads get through, it is pretty apparent they don’t do much. The things that kill performance on a site aren’t even the malicious ads. They are the broken ones. Like, I went to Business Insider right now, and this is their front page for me, using a slightly off-standard browser (Opera).
That’s not a malicious ad, it’s just a broken one. Malicious ads need to work to be malicious.
It seems like many of the people making the decisions about ad tech in newsrooms may not have the technical chops to properly assess the partnerships they’re making.
I mean, yes and no. This was sort of what I was getting at [on Twitter]. When they started this process, it was a straightforward idea. You sell white space to a company like Google and they auction it off to the highest bidder. You make your direct sales and any leftover white space gets as much money as it can get. Then the newsroom doesn’t have to think about how to make online ads work.
Which was where the problems started. We didn’t want to have to think about how to make online ads work and we gave away our control in exchange for the promise of not just profit but also that places like Google’s DFP would be our partners! That they’d work with us for mutual benefit. That hasn’t turned out to be the case.
Were the wrong people in the room to make these decisions? It seems like sometimes the technical folks are left out of these business conversations. (And I realize we can’t assess every newsroom, but I’m wondering how we start to put the technical chops — or recognize what skills are needed — when having these conversations.)
I think that happens. But I think that has to do with publishers’ prioritization. Tech resources are usually pushed towards controlling and enabling editorial abilities and decisions. Tech is usually left out of the initial conversations and the sales conversations in many places, though not all of them. And the thing is that I’ve never met a sales team that was against having a technical person sit in.
I think some publishers see ad tech as something that is supposed to take care of technical problems for them, and so they don’t need tech people in the room for contracting out the solutions to those problems.
But many of these actors in ad tech are not acting in good faith. And they don’t see themselves that way.
How do they see themselves?
Like most tech companies they want to take over their market and they want to maximize their returns any way possible. I reported briefly on economics in 2007 and in many ways ad tech reminds me of the big banks problem. They don’t care about the economy, they are individuals looking to get as much money as possible and leave the problems to the next generation. As such, they don’t care about making good tech or a good community.
This is the ad tech market right now:
This is an evolution of about 150 companies to 3,874 in five years. five years!
That’s a lot!
At the very same time the market that they are supposedly building out a lot of their products for is basically in a decline.
Do you think the firewall in newsrooms contributes to these issues? In my newsrooms, the people thinking about this stuff sat on a different floor than the editorial folks.
Yes. Absolutely.
I have a story about this actually, from working at a previous employer. I prefer to sit in on sales calls for contracts with ad tech because of this very problem. And we had a problem with a major advertiser. The advertiser had run their ad through an agency and the agency was telling them that the ad had 10 percent viewability.
Meaning 10 percent of people had seen the ad?
Sort of. Viewability is a ridiculous metric. The IAB defines it as 50 percent viewable for at least one second. It is very easy to measure with open-source tools, but dozens of companies all offer different techniques to measure it. That was what was being measured, as opposed to impressions or loads.
So, the agency called us. And we talked to them and said they we could measure viewability on our own (which we could) and that wasn’t making any sense. So they put us on the phone with a major ad tech company that was measuring viewability for them. And the hard part to understand is just how abstract this company is from any of the people who are actually seeing these stats. As I understand it, agencies buy a license from these companies and upsell the products as part of packages to advertisers. Which is why one ad could have many pieces of ad tech on it, even measuring the same thing. The advertiser doesn’t have the technical expertise either. They get some numbers and we get some numbers.
So how many degrees are we talking now from the people sitting in your conference room?
This was a direct sales ad, which means our sales people contact the agency which contacted the advertiser and sold them their services including ad tech which ended up on our site, invisible to anyone on the publisher side until this moment.
Finally we are connected with the ad tech company and they have sent an engineer who is talking technical stuff at high speed with, in my opinion, the clear intent to confuse the people who are usually in this type of meeting — all sales people with little expertise in tech.
The thing is, I’m in the room and I’m just listening until he gets to the point which is ‘we don’t track Webkit browsers.’
At this time Webkit is the browser rendering engine for Chrome, Safari and a number of other smaller browsers.
This is a major ad tech company and they just said that they’re not tracking 70 percent of our traffic, but reporting it to the advertiser as if they were.
So I ask how are they getting their stats without Webkit?
And they say they are tracking 20 to 30 percent of our traffic — that’s generous, some people will block or fail to load the script — and “extrapolating.” A big chunk of which includes Internet Explorer which is infamously bad at executing JavaScript.
And so they have the pitch to solve our problem, which is to pay them $300 per month to put some of their code on our site and sign an NDA so we can’t talk to anyone about the code we are using. This is not a small company or a startup. This is a major ad tech company. And they’re giving us the shakedown.
If that meeting had been all sales people, they wouldn’t know better, they would just look at the numbers and say that $300 per month is less than they are making per month with the ad placement so they’d have just run it.
Wow, what did you do?
I shut it down. I told them we’d think about it and then I told the CTO and the VP of sales to tell them to go to hell, because in the end what it really meant was running their analytics on our site and we still would have had to argue with the advertiser about whose stats were right anyway.
Yeah, I was just going to say “Who analyzes these code snippets that they’re putting on sites?”
Yeah, no one. Devs hate dealing with it because all the code is awful and you aren’t allowed to change it anyway. Only recently have some companies started double checking this stuff.
Are news organizations pooling resources here?
Not really, because I’m sure as is the case with these folks, a lot of them are requiring NDAs. So devs talk on back channels and meetups and swap solutions. But a lot of the code is so badly written that it isn’t even minified or uglified, which are techniques for compressing and obscuring code. So you can see some of it in the wild.
I feel really depressed right now. Are we #%$#%$?
It’s not a good situation, no. This is the problem though, when people talk about large publisher sites or long loading times. It isn’t the publishers that are loading stuff, it is all ads. At a previous employer, around 75 percent of load time, on a good day, was ad units. Not even video ads.
Yeah, I saw a similar stat tossed around at the International Symposium on Online Journalism.
That’s almost all really awful JavaScript when it isn’t trying to track you it is just bad. Something like tracking viewability is so easy. But these ad tech companies are just throwing stuff out there.
So I’m taking off my “working in news” hat for a second and putting on my “I also enjoy reading the news” hat. Is the solution to install ad blockers? Does that force anything to change for the better?
Not really. The leader of the industry, the IAB, is leading the fight in blocking ad blockers, as opposed to fixing their practices. I mean, this is a pretty radical idea, but the first thing [I think] publishers have to do is abandon the IAB. Remove their members from its board, stop showing up to its conferences, stop advising them. They aren’t listening anyway.
The point of an organization that contains publishers, ad tech, and advertisers is that it is supposed to be a coalition to make the web better from everyone’s perspective. But last year the IAB reversed its stance on auto-play video ads. So it is pretty clear where that is going.
Where?
Not anywhere good for readers. And what’s bad for readers is bad for publishers, because then they stop reading.
So if you could start all of this over from scratch, what would you do?
The first thing would have been for publishers to give a shit about how their sites hosted advertisements. In the same way that content management systems’ development has been increasingly driven by publisher needs, the same case should have happened with ad tech. It hasn’t.
If we had started innovating on how ads worked on our sites at the same pace we worked towards building ways to Snowfall better, our entire industry would be a lot more healthy.
That graph with almost 4,000 marketing companies? The profit that drives their existence comes from digging into the advertising ecosystem and extracting the money from the process, and a big chunk of those profits are coming from publishers. Either directly in buying into their products, or indirectly in taking money from advertisers or agencies that would have gone towards publishers.
Publishers should have led the construction of an open source ad server in the same way they have been huge contributors to open source CMS like WordPress. Imagine the state of play if we had treated OpenX (what used to be an open source ad server) like WordPress? It’s almost impossible to picture.
Oh man. I can’t even conceive of what that would have led to.
Exactly. The same with our understanding of metrics if we had pushed forward co-developing an open source metrics platform like Piwik instead of becoming dependent on external operators to tell us how our own sites worked.
I wrote last week about all of the questions publishers should be asking before partnering with third party tools.
Yeah, I saw that! It is a good idea, it needs to happen at the sales level as well as the editorial level. Just an example of that: every publisher I know who runs Adobe Omniture and Google Analytics at the same time sees some degree of discrepancy. No one can tell you why. Both are running some sort of secret recipe as to what traffic is valid and what isn’t. We don’t know what it is or how valid it is. As a result, we are incredibly vulnerable to bot nets.
It seems like a black box that’s preventing publishers from making all sorts of decisions.
Very much so. And because we don’t know how the metrics are being measured we’re suckers for every ad tech company that comes along and says ‘we can measure this better’.
So what do we do?
We need to stop the arms race with ad blockers, that won’t lead anywhere good. We need to find a way to either move away from programmatic or a way to gain better control over it. We need to break down the walls between business, editorial and sales so we can all talk about how our sites work.
We need to seek, push, and co develop open source tools for metrics and advertisements.
And we need to seek more direct contact with advertisers. Agencies are basically relics. While there are good ones, the truth of the matter is that in a digital market, organizations whose existence was to connect advertisers with publishers make little sense. Unless their role is design, they are locking publishers out of needed conversations about how the technology should work for their own profit.
How do we do that when no one wants to share their stuff? It seems like the industry could work together but competition gets in the way.
I don’t think that’s the case. Look at the Coral Project [which is a collaboration between the New York Times, Washington Post and Mozilla to examine online commenting platforms.]
Do you think there could be a Coral Project for ad tech?
There should be. There needs to be. But funding something like that isn’t very sexy in the media-on-media press, and a hard thing to do with something like a non-profit for the obvious reason that it is all about making money. Oh, and we need to stop doing shit like suing users or user tools.
Keep in mind, publishers should be jumping all over this. This is huge direct savings. If you hosted all your own metrics, or didn’t have a network scraping profits off the top, that would be a
lot of money saved.
I want to ask one more question: How would you recommend learning more about this, for people in newsrooms who may not be dealing with the dev or business issues every day?
Talk to the devs working on ad tech? Talk to your sales people about the problems they face. Talk about what code is running on our site and why? Editorial tends to treat the problem of bad ads and bad ad tech as this insurmountable problem. They throw up their hands and say ‘this is the way it is.’ The companies that are and will be doing well don’t do that. They have conversations that include all units.
Oh, the other thing, the hardest thing, that publishers can do?
They need to restore trust with advertisers to get them to stop going to sketchy ad tech. That’s going to mean opening up their own analytics. Dunno who is going to do that, but it is the best way to cut out bad actors in ad tech.
If everything wasn’t a mystery, advertisers wouldn’t feel the need to load 20 scripts onto an ad to assure themselves it was being seen.
The other thing people can do to keep track is just watch the industry publications like AdAge, Digiday, etc…
BTW, useful stuff: This is everything that was a web article that I have read about ad tech since 2015. There are some great people to follow in that list.
That’s really useful. Thanks Aram. I think this is going to start a really useful (and much needed) conversation.
