Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Welcome to the cruel reign of highly efficient algorithms! Yay... In short, this is an excellent albeit very high-level overview of the most pressing techno-moral issues at the core of advancements in Machine Learning, AI, and the many obscure mathematical models quietly ruining running our lives. Be advised, those looking for mathematical exposition or in-depth explanations about the models mentioned herein will be better served elsewhere. As a data-science/machine-learning practitioner, I found O'Neil's case and her supporting material both edifying and deeply concerning. You see, I had heard stories of algos running amok, kicking asses and taking names in the all consuming search for optimizing ways to squeeze cents out of each byte of data comprising our cyber identities, but the extent of the chicanery employed by the companies and their analysts in their approach is just so deliciously evil that you would think they're secretly engineered by cats. While I found much of the book solidly researched and cogent in its underlying argument, from time to time I did find some minor quibbles with her points. For instance, early on in the text she recounts her time as a quantitative analyst at a high-caliber Wall Street hedge fund, where she ultimately came to the conclusion that it was the insidious power of math that engendered much of the chaos that resulted in the financial crisis of 2008. However, not five pages later, she mentions leaving said fund to go work for an investement risk consultancy firm, where her team's detailed analysis would go unheeded by the very same firms employing them (they just needed to look like they were being responsible by carrying out due diligence). So, it's not that the math was bad, or that the models failed to take into account this or that variable, it's just that the guys running the show knew the risks but decided to gamble on them anyways. This theme is repeated throughout, as her case studies expose a deep disregard on the part of the algo overlords to rectify unfair practices unless legally obliged to do so. One can see how deeply flawed this attitude is and where it may lead us, especially under the mercy of an arguably lethargic political system; random fact: in my home country there's a saying, "hecha la ley, hecha la trampa", which roughly translates to "by the time the law is written, a new snare is already in place". Traditional politicians will never keep up with the tech sector. Which brings me to the saddest part of the book, which is the author's attempt to lay down a blueprint for bringing much needed change. I can tell she deeply cares about the issues at the core of her argument, but I'm just not that convinced any of them could ever work without somehow making it simultaneously profitable to the companies involved. All in all, I think most of us would do well to give this read if only to get a sense of what's at stake here, and how we ultimately came to be unwilling participants in this curve-fitting, dot-connecting, profits-above-all game.

Fairness is a virtue which requires human culture to be preserved. Societies try to agree on rules which - at least on paper - help to uphold justice, impartiality and decency. Market forces, on the other hand, pursue hard numbers of growth and income. The blooming IT industry, crunching numerical data, excels at treating most challenges as optimization problems - and the optimized factor is how to get the most cash out of each customer with minimal effort. Fairness eludes computer models as it is difficult to define and measure - thus it is dropped by programmers, for the sake of algorithms’ effectiveness and “elegance”. In result, interactions between consumers and companies are driven more and more by heartless digital logic. Whether you apply for a loan, request insurance quotation or just visit a webpage, algorithms try to make the most of you, by adjusting costs or selecting most persuasive content. Models try to predict what will happen, based on knowledge from the past. Their power comes from generalisation - looking for similarities between different people, and looking for patterns how they behave. This allows for profiling and adaptation. However, since models are oftentimes constructed with incomplete and/or biased data, their judgements may be prejudiced towards unfortunate groups. More often than not this results in heightened exploitation of the most vulnerable (indigent, minorities)... The book presents many such examples. It is not only the commerce which employs faulty models, but the public sector as well, with particularly egregious misuses in the judiciary branch. Personally I believe that unchecked technological progress brings too much harm; Cathy clearly shares that sentiment. Markets do not self-regulate towards betterness of societies. It is the role of governments to moderate how mathematical models are built so they are more transparent and accountable. The book shows many examples of how current models are exploiting the vulnerable and how their logic reinforces social divides. It is not the technology which is bad per se, just the way we use it. This title is enjoyable, well organized and with clear message. I am taking one star off, as some parts of the book seem a bit too alarmist to me, but maybe Cathy just has a better grasp of how dire the things really are...

This is a thoughtful and very approachable introduction and review to the societal and personal consequences of data mining, data science, and machine learning practices which seem at times extraordinarily successful. While others have breached the barriers of this subject, Professor O'Neil is the first to deal with it in the call-to-action manner it deserves. This is a book you should definitely read this year, especially if you are a parent. It should be required reading for anyone who practices in the field before beginning work. I have a few quibbles about the book's observations based on its very occasional leaps of logic and some quick interpretations of history. For example, while I wholeheartedly deplore the pervasive use of e-scores and a financing system which confounds absence of information with higher risk (that is, fails to posit and apply proper Bayesian priors), the sentence "But framing debt as a moral issue is a mistake", while correct, ignores the widespread practice of debtors courts and prisons in the history of the United States. This is really not something new, only a new form. Perhaps it is more pervasive. For a few of the cases used to illustrate WMDs, there are other social changes which exacerbate matters, rather than abused algorithms being a cause. For instance, the idea of individual home ownership was not such a Big Deal in the past, especially for people without substantial means. These less fortunate individuals resigned themselves to renting their entire lives. Having a society and a group of banks pushing home ownership onto people who can barely afford it sets them up for financial hardship, loss of home, and credit. What will be interesting to see is where the movement to fix these serious problems will go. Protests are good and necessary but, eventually, engagement with the developers of actual or potential WMDs is required. An Amazon review is not a place to write more of this, nor give some of my ideas. Accordingly, I have written a full review at my blog (see the image) for the purpose. My primary recommendation is a plea for rigorous testing of anything which could become a WMD. It's apparent these systems touch the lives of many people. Just as in the case of transportation systems, it seems to me that we as a society have very right to demand these systems be similarly tested, beyond the narrow goals of the companies who are building them. This will result in fewer being built, but, as Dr O'Neil has described, building fewer bad systems can only be a good thing.

This book is written with great authority on a crucial and timely topic: the influence of big data driven algorithms on everyday people. The author, with her background in mathematics, describes several instances of what she calls WMDs (weapons of math destruction), where profit driven data models deeply and negatively affect people, especially poor people. WMDs have a tendency to be inherently unfair, keeping unfortunate victims in a vicious cycle of poverty/injustice. They are opaque and unaccountable and affect too many lives due to sheer scale of use. The book makes a strong case for accountability of such data driven algorithms, so that perhaps they can be used for social good. This is a must read for every data scientist as well as policy maker and anyone interested in the effect of the data driven world on common people.

To keep on with the analogy of its catchy title, Weapons of Math Destruction gave me the impression as I read it to have one and only purpose: to convince me that nuclear science is generally bad because it is used to produce bombs. And bombs are bad because they are built by cold-hearted generals who only think about winning a war, no matter how many people die as those bombs explode. Still, this is a book that has very good intentions, and equally conveys compelling messages about inequality, social injustice, and unfair treatment to the underprivileged. The problem, imho, is that most of these messages are not necessarily new and certainly not centered in the many controversial issues that involve technology, deep learning and other AI. So different from Shoshana Zuboff's The Age Of Surveillance Capitalism, where technology IS the problem (take, for example, Google using Street View cars to suck data from domestic Wi-Fi networks) here the main issues are, more often than not, related to injustices that are part and parcel of capitalism. Many modern societies are permeated with inequality based on race, gender and income. WMD is not wrong when it says that algorithms come to perpetuate and expand those discrepancies: when poor people search for education, fill out job applications, work as low-pay employees at Starbucks, even when they get auto insurance. But algorithms are just a new sauce for an old meal, and WMD seems to ignore that, focusing on the algorithms alone as if they were an ingredient that could simply be removed from the recipe. That makes the book sound, for the most part, either naive and narrow-minded or else completely biased by its own political agenda (which, of course, is not necessarily bad or wrong per se, but certainly weakens the power of its own message.) Central to many of the examples used in the book is the tradeoff between fairness and efficiency, which ends up statistically in balancing Type I and Type II errors. Surprisingly for a book written by someone in the field, this is clearly stated only in the Afterword (the harm of false positives and false negatives). It is also only in the Afterword that biased data used to train models is clearly described as one of the main problems with algorithms, something the book misses to discuss in the main section even when it (only marginally) talks about machine learning ("deep learning" is not mentioned at all). In the main section, WMD is more concerned about choosing variables or proxies that intentionally guide the model towards a certain result than problems with the data itself. Indeed, the book does not go much beyond selection bias and ill-built (or lack of) feedback loops to validate (and correct) models. Those are very crude and basic 'design mistakes' that can be described only as poor math dressed of rigor or grotesquely flawed modeling with no scrutiny. More than anything, though, the main caveat of WMD to me was the lack of more rigorous research (some statistical analysis would help) beyond anecdotal evidences. I was particularly surprised to discover that the whole discussion about Kyle Behm and his job personality tests was entirely extracted from an article published on the WSJ in 2014 (including the job application questions). From that discovery on, I noticed that apart from the narrative about Cathy O'Neil's own experiences at D.E. Shaw, RiskMetrics and Intent Media, the rest of the book is essentially a pout-porri of newspaper articles. That is even more evident when the book finally have a chance to talk about big tech. It almost embarrassingly falls short, not going much beyond trivial and shallow opinions reproduced from the news. That lack of depth and analytical rigor made me feel progressively less interested, if not frustrated, with the whole book, even more so when I finally got to its (rather messy and almost reckless) final chapters. The last page of the book reads it is "dedicated to all the underdogs". That is a noble and laudable cause and clearly and sincerely shows throughout the book, which makes me now think that maybe the problem with WMD is exactly its catchy title and the way it tries to sell itself. It is not math that matters here. It is simply us collectively, as an unfair, prejudiced and regressive society.

…and a book that delivers by delving into the numerous ways that mathematical algorithms impact our lives, sometimes very negatively and often without appeal. All too often the ones who formulate the algorithms view the injustices as just so much collateral damage, sacrificed on the altar of economic efficiency. At the end of the review, I’ll provide an example of my own. Cathy O’Neil has a PhD in math from Harvard, taught at Barnard, decided to make three times the money by working as a “quant” on Wall Street, specifically for the hedge fund D. E. Shaw. Of the numerous wry observations she makes in the book, she compares working at D.E. Shaw to the structure of Al Qaeda. Information was tightly controlled in individual “cells.” No one (probably even the big boys) understood the entire structure which prevented someone “walking” to a rival. The financial meltdown of 2008, when suddenly the quants, and others, realized that a strawberry picker named Alberto Ramirez, making $14,000 a year, really couldn’t afford the $720,000 he financed in Rancho Grande, CA,, and therefore the “Triple A” rating on the bonds issued based on the mortgage was phony, proved to be her “Saul on the road to Damascus moment,” which eventually led to this book. (She doesn’t make the point that the damage done by the quants, in terms of lost homes and jobs, to so many Americans, was far, far greater than Al Qaeda’s wildest aspirations.) In her book, O’Neil goes far beyond Wall Street to other segments of our society: colleges, the judicial system, insurance, advertising, employment, teacher evaluations, credit scores, and political campaigns and Facebook. Consider colleges. It was US News and World Report that dreamed up the idea of ranking colleges based on “objective” quantitative criteria. They convinced others to play along, in particular the colleges themselves. And so, from the perspective of a university President, “…they were at the summit of their careers dedicating enormous energy toward boosting performance in fifteen areas defined by a group of journalists at a second-tier news magazine.” A most important area was totally omitted: “value for money,” a standard criteria for most Amazon Vine reviews. And so, as she says, to meet these journalists’ criteria, the cost of higher education rose 500% between 1985 and 2013. She cites a couple of examples how colleges “gamed” the system. The most interesting was King Abdulaziz University in Saudi Arabia. Its math department had been around TWO years, in 2014, when it came in 7th place in the world, behind Harvard, but ahead of MIT and Cambridge! How? It searched the professional journals for professors with the most citations, one of the criteria in the algorithm, offered the professors $72,000 a year for three weeks of work as “adjunct faculty.” Voila. In public school teacher evaluations in the USA, O’Neil cites the example of a well-respected teacher who was fired for being in the bottom 10% in teacher evaluations. How? Apparently the teachers from the PREVIOUS year had falsified the students’ standardized testing results. The following year, when the well-respected teacher did not, it appeared to the algorithm that the students had declined. No appeal or common sense. She was fired. Insurance is a personal bugaboo with me. O’Neil confirmed what I learned the hard way. A MAJOR factor in determining the price of insurance is an algorithm that determines which customers are unlikely to switch insurance companies – and those customers are charged the most! When I finally figured this out, the hard way, a few years back, the company that famously proclaims that you can “save 15% or more” was actually willing to drop my insurance premium 30% because I was changing, which I still did, to another company that offered the same coverage for 50% less. (I’ll be changing from that company in a couple of years, of course.) (What a racket.) Another fascinating section is on how our on-line behavior is monitored, which changes not only the ads we see, but the very news. And how much effort is expended in political campaigns on those few undecided voters in Florida and Ohio. Wow. Truly calls for the abolition of the Electoral College. Finally, my own example. I once worked for the COO of the most famous hospital in the aforementioned Saudi Arabia. He called me in one day and asked if I could do standard deviations. Thanks to Bill Gates, et al., I assured him I could readily do them. “Then please do them on all the doctors’ salaries, per department”. Again, thanks to Bill, it was done in a day. Why, oh why? It was the COO’s own “algorithm.” When he met monthly with each department Chair, to discuss physician evaluations and salary increases, there would be nothing “personal” involved. He could point to this objective report, and express his concerns about the “standard deviation” of the salaries within the department. And depending on – hum – the circumstances, he could say: I think the standard deviation is “too high” (or, of course, “too low”). The “basis” for giving out a 2 ½% or 5% salary increase. “Clever.” As for O’Neil’s book, 5-stars, plus.

I was excited to read this book as soon as I heard Cathy O'Neill, the author, interviewed on EconTalk. O'Neill's hypothesis is that algorithms and machine learning can be useful, but they can also be destructive if they are (1) opaque, (2) scalable and (3) damaging. Put differently, an algorithm that determines whether you should be hired or fired, given a loan or able to retire on your savings is a WMD if it is opaque to users, "beneficiaries" and the public, has an impact on a large group of people at once, and "makes decisions" that have large social, financial or legal impacts. WMDs can leave thousands in jail or bankrupt pensions, often without warning or remorse. As examples of non-WMDs, consider bitcoin/blockchain (the code and transactions are published), algorithms developed by a teacher (small scale), and Amazon's "recommended" lists, which are not damaging (because customers can decide to buy or not). As examples of WMDs (many of which are explained in the book), consider Facebook's "newsfeed" algorithm, which is opaque (based on their internal advertising model), scaled (1.9 billion disenfranchised zombies) and damaging (echo-chamber, anyone?) I took numerous notes while reading this book, which I think everyone interested in the rising power of "big data" (or big brother) or bureaucratic processes should read, but I will only highlight a few: * Models are imperfect -- and dangerous if they are given too much "authority" (as I've said) * Good systems use feedback to improve in transparent ways (they are anti-WMDs) WMDs punish the poor because the rich can afford "custom" systems that are additionally mediated by professionals (lawyers, accountants, teachers) * Models are more dangerous the more removed their data are from the topic of interest, e.g., models of "teacher effectiveness" based on "student grades" (or worse alumni salaries) * "Models are opinions embedded in mathematics" (what I said) which means that those weak in math will suffer more. That matters when "American adults... are literally the worst [at solving digital problems] in the developed world." * It is easy for a "neutral" variable (e.g., postal code) to reproduce a biased variable (e.g., race) * Wall Street is excellent at scaling up a bad idea, leading to huge financial losses (and taxpayer bailouts). It was not an accident that Wall Street "messed up." They knew that profits were private but losses social. * Many for-profit colleges use online advertisements to attract (and rip off) the most vulnerable -- leaving them in debt and/or taxpayers with the bill. Sad. * A good program (for education or crime prevention) also relies on qualitative factors that are hard to code into algorithms. Ignore those and you're likely to get a biased WMD. I just saw a documentary on urbanism that asked "what do the poor want -- hot water or a bathtub?" They wanted a bathtub because they had never had one and could not afford to heat water. #checkyourbias * At some points in this book, I disagreed with O'Neill's preference for justice over efficiency. She does not want to allow employers to look at job applicants' credit histories because "hardworking people might lose jobs." Yes, that's true, but I can see why employers are willing to lose a few good people to avoid a lot of bad people, especially if they have lots of remaining (good credit) applicants. Should this happen at the government level? Perhaps not, but I don't see why a hotel chain cannot do this: the scale is too small to be a WMD. * I did, OTOH, notice that peer-to-peer lending might be biased against lender like me (I use Lending Club, which sucks) who rely on their "public credit models" as it seems that these models are badly calibrated, leaving retail suckers like me to lose money while institutional borrowers are given preferential access. * O'Neill's worries about injustice go a little too far in her counterexamples of the "safe driver who needs to drive through a dangerous neighborhood at 2am" as not deserving to face higher insurance prices, etc. I agree that this person may deserve a break, but the solution to this "unfair pricing" is not a ban on such price discrimination but an increase in competition, which has a way of separating safe and unsafe drivers (it's called a "separating equilibrium" in economics). Her fear of injustice makes me think that she's perhaps missing the point. High driving insurance rates are not a blow against human rights, even if they capture an imperfect measure of risk, because driving itself is not a human right. Yes, I know it's tough to live without a car in many parts of the US, but people suffering in those circumstances need to think bigger about maybe moving to a better place. * Worried about bias in advertisements? Just ban all of them. * O'Neill occasionally makes some false claims, e.g., that US employers offered health insurance as a perk to attract scarce workers during WWII. That was mainly because of a government-ordered wage freeze that incentivised firms to offer "more money" via perks. In any case, it would be good to look at how other countries run their health systems (I love the Dutch system) before blaming all US failures on WMDs. * I'm sympathetic to the lies and distortions that Facebook and other social media spread (with the help of WMDs), but I've gotta give Trump credit for blowing up all the careful attempts to corral, control and manipulate what people see or think (but maybe he had a better way to manipulate). Trump has shown that people are willing to ignore facts to the point where it might take a real WMD blowing up in their neighborhood to take them off auto pilot. * When it comes to political manipulations, I worry less about WMDs than the total lack of competition due to gerrymandering. In the 2016 election, 97 percent of representatives were re-elected to the House. * Yes, I agree that humans are better at finding and using nuances, but those will be overshadowed as long as there's a profit (or election) to win. * * * Can we push back on those problems? Yes, if we realize how our phones are tracking us, how GPA is not your career, or how "the old boys network" actually produced a useful mix of perspectives. * Businesses will be especially quick to temper their enthusiasm when they notice that WMDs are not nearly so clever. What worries me more are politicians or bureaucrats who believe a salesman pitching a WMD that will save them time but harm citizens. That's how we got dumb do not fly lists, and other assorted government failures. * Although I do not put as much faith in "government regulation" as a solution to this problem as I put into competition, I agree with O'Neill that consumers should own their data and companies only get access to it on an opt-in model, but that model will be broken for as long as the EULA requires that you give up lots of data in exchange for access to the "free" platform. Yes, Facebook is handy, but do you want Facebook listening to your phone all the time? Bottom Line: I give this book FOUR STARS for its well written, enlightening expose of MWDs. I would have preferred less emphasis on bureaucratic solutions and more on market, competition, and property rights solutions.

A Jr High Student Could Beat a PHD using index cards. I use spread sheets in place of index cards, and use none of the formulas of higher math included in the program. Few competent people use high level math to forecast. Set theory dominates successful forecasting. Since 1973 the vast majority of Americans are not wealthier than before. The information age cost serious money and does nothing to improve our lives. The cost of our new distopian society is included in our supposed wealth. The Epstein Class unlikely even own a pocket calculator between themselves. But they direct computer programmers to rule over US and bypass intermediate management completely with Artificial Intelligence. I could show a Jr High student how to beat all of them in a competitive forecasting competition. But they wouldn't believe it could be done, and the Epstein Class has other plans for Jr High School students. Don't mean to insult the author. She seems to understand our false beliefs. Maybe ¨your¨ false beliefs, not mine. Mega-data is just a shiny toy they give adults when they want to distract them. Mega-data simply does not work.

Pros: • Easy to read, but treat it more like a trivia book if you will. Cons: • Heavily americanized writing. Examples and analogies are mainly based on American context. • Gets political at times. • Made no effort to give a balanced views on ”WMD’s” - anecdotes focused only on the victims and not the overall utility provided. Overall, it’s a heavily biased book that constantly reminds you they hate “WMDs” and are only interested in telling you what’s bad about algorithms (and they really do just that!). I picked this book up expecting helpful examples to better understand ethical AI practices for my work, but left feeling like I’d just wasted $20 on a printed blog page.

Not sure why we need responsible AI? Please read this book. It is brilliantly written and explains that AI and data-driven systems aren't inherently bad - it's how we design and govern them that matters. There are so many cautionary tales. This is a must for every person working in AI, data or policy - particularly in government where decisions can impact citizens.

ビッグデータのアルゴリズムがこのように悪用され、人々の生活を脅かしているとは知りませんでした。アメリカの話なので必ずしも日本は良くも悪くもそこまで進んでいませんが、我々はこ脅威を認識し、良く手綱を握らないといけないと思いました。

The author describes current (American) trends in data mining and the recklessness shown by many of the organizations who sometimes even ruthlessly try to maximize their profits by screening and classifying people using overly simple models and methods. These algorithms can help us see things clearer, but if they are applied without careful doubt, the results can be inadvertently biased and really detrimental. Because most sophisticated statistical methods and data bases are proprietary, their verdict comes from a "black box" - but it must be true, because computers calculate correctly, don't they? Well, in a word: NO, garbage in - garbage out still holds true. Read this book and you get a better understanding why data science is so important and you will better understand both the opportunities and the pitfalls.

This is a pretty crucial read if you want to get your head around the less-than-rosy side of big data and algorithms. For someone like me, who’s always got their hands dirty with data, it really hammered home how these systems, even when they’re built with good intentions, can totally mess things up for people. The section on hiring algorithms was particularly eye-opening. It’s a stark reminder of how easily automated systems can just rule people out, not because they're not up to scratch, but because they don't fit some "ideal" profile the algorithm has been fed. It makes you seriously question how much trust we put in these invisible processes. This isn't a book you'll breeze through on a lazy Sunday. I read it over a long weekend and definitely found myself having to dip in and out to give my brain a bit of a break. But O'Neil does a solid job of breaking down some pretty complex ideas, making them understandable even if you’re not a whiz with machine learning. While it didn’t completely change my perspective – I’m already in this line of work, after all – it absolutely deepened my understanding and gave me some really valuable points to chew on. Honestly, I reckon everyone should read this book. It’s absolutely vital to grasp how data is increasingly shaping, and often manipulating, the world around us.