I just spent a week\nof vacation in Italy. One of my good friends from college was getting married,\nand I didn’t want to miss his wedding. It was a good time for Kelly (my wife)\nand I to also take a week of vacation prior to the wedding. We love Italy, and\na wedding was a good excuse for us to go there again.<\/p>\n\n\n\n
During vacations (and during long flights) I like to read a good book. During this trip, I had loaded Weapons of Math Destruction by Cathy O’Neil<\/a> on my Kindle. This was a fantastic read! When reading a book, I sometimes pause and read a passage to my wife. I know it’s a good book when she is intrigued by it and we can discuss that passage.<\/p>\n\n\n\n The main concept\nthroughout the whole book is the negative side-effect of big data models and AI\non certain parts of our society. The author calls the big data or AI models\nthat have those negative side-effects ‘Weapons of Math Destruction’ (WMD for\nshort). She does a really good job in talking through the actual cause and\neffect relationship behind these WMDs, and giving very personable examples in\neach of the different chapters. <\/p>\n\n\n\n When I finished the book I spent some time thinking about what I learned. And the more I thought about it, the more I actually wanted to share what I learned and provoke a discussion.<\/p>\n\n\n\n I love the expression ‘a rising tide lifts all boats’. If you think about ships on the sea, or even on a river, if the water level rises, all ships move upwards with the rising tide. I would love for this to be true for our society as well, and it certainly is in some cases. Literacy rates are going up all<\/a> around the world, child mortality is declining<\/a> and overall, less people are living in extreme poverty<\/a>.<\/strong><\/p>\n\n\n\n However, the author\nof this book makes a couple of excellent cases where this is not true. One very\nspecific example that struck me personally is the optimization of staff hours\nin retail stores and specifically the phenomenon of ‘clopening’. Clopening is a\nterm created for when an employee of a retail closes the store one evening and\nopens that same store the day after, with limited actual time of work.<\/p>\n\n\n\n This example was\nrelevant for my own personal situation. During my student years, I worked as a\nwaiter for about 7 years. I made less on an hourly basis than my friends did\nworking 9 to 5 desk jobs during summers, but I had the ability to make up for\nit with working extra hours (not by tips however. I am from Belgium, where\ntipping your waiter isn’t as big as in the US. I was happy with a 2% average\ntip at the end of the day). <\/p>\n\n\n\n All large retailers\nby now use scheduling software to plan employee hours. In my naivety, I would\nexpect a computer program to have a simple control loop that avoids clopenings\nfrom happening. At least, from having them happen frequently. But clopenings happen\nfrequently, and when a scheduling algorithm is optimized for other metrics,\nthis weird side effect creeps in. Even worse, the author mentions cases where\nlow-wage workers’ hours are optimized for them to stay below the ‘eligible for\nmedical insurance’-mark. Just imagine having to close a store at 11PM, being\nthere the next day at 7AM for the opening but not working enough hours during\nthe week to be eligible for medical insurance.<\/p>\n\n\n\n I understand from a\nbusiness owner perspective that personnel cost is a major cost for a business.\nAnd as a business owner and shareholder, your primary goal is to optimize for\nbusiness results. The ethical question you should ask yourself is whether this\nbusiness result is gained by outsmarting your competition or by selling more,\nor rather if it is gained by profiting from paying your minimum-wage employees\nas little as possible.<\/p>\n\n\n\n And if this is the case, I would argue that the rising isn’t lifting all boats at all. If a retail employee has to suffer both clopenings and working very irregular hours there is something wrong with your scheduling algorithm. And you could indeed call this a ‘weapon of math destruction’ optimized for a poor metric.<\/p>\n\n\n\n Data is the new oil.\nThe more data a business has, the more valuable that business can become. If\nyou want to train a machine learning algorithm, you need as much data as you\ncan get. Having the right data, and much of that, will enable you to make better\npredictions, and tailor your products and its pricing to the customers you are\nserving.<\/p>\n\n\n\n However, not all\ndata is created equally and not all data has the same value. There is already a\nlegislative framework that prohibits certain data to be part of a decision. A\nloan application shouldn’t be influenced by matters such as race or gender. The\nproblem with data is however, that some of those prohibited matters can be\nlearned from the data. <\/p>\n\n\n\n The book has a\ncouple of great examples about the usage of a ZIP code. ZIP codes can for\ninstance be correlated to crime rates, to default rates on mortgages and to\ncollege graduation rates. As a mortgage provider, it only makes sense for you\nto use this metric to optimize your business. If people living in a certain\nneighborhood are more likely to default on their mortgage, it might make sense\nfor you to ask for a premium to that population.<\/p>\n\n\n\n It’s even harder to break out of this spiral if you consider that your zip code can influence not only your mortgage, but also your car insurance rate (apparently even more than drunk driving history) and even your acceptance rates in colleges and universities. The book has a number of examples in those area, that sound reasonable from a certain perspective (a business wanting to make profit), but in the same time make you think about how a person could get out of this spiral.<\/p>\n\n\n\n I am an optimist. I\nbelieve that big data and AI will have a positive impact on society. To reach\nthis positive impact, we should pay close attention to how we use our models.<\/p>\n\n\n\n Let’s take the\nprevious example of the mortgage model trained on ZIP code data. In the\nexample, we discussed that the data tells us that people from certain ZIP codes\nare more likely to default on their mortgage. Instead of using this data and\nthis model to influence people’s mortgage rates, why don’t we use it to offer\nthem with better protection? This protection could be as simple as offering\nthose people with courses in financial literacy and regular follow-up to check\nup on their finances, or even offering them an additional insurance that would\nprotect them.<\/p>\n\n\n\n That’s one place where the book in my point of view could have been a bit more prescriptive. There are only a couple of example of positive models being used, or models being used differently to have a positive impact. The book felt very negative in that regard, showing a lot of negative example, with only a couple of empowering positive impacts. As an optimist, I took those couple of positive examples at heart and want to make sure that we implement the technology the right way to have a positive impact.<\/p>\n\n\n\n The main conclusion here is I highly recommendation you to read this book and form your own opinions. It certainly has given me a different perspective, and I hope it will do the same for you. <\/p>\n\n\n\n From a practical lens, I got a couple of learnings from this book. One is to consider multiple perspectives when thinking about or even designing an AI algorithm. The second is to make sure that the model doesn’t reinforce itself and creates downwards spirals. Finally, and that’s the most positive learning, is to consider how the technology can be used to create WIN-WIN situations, that can benefit all stakeholders.<\/p>\n","protected":false},"excerpt":{"rendered":" I just spent a week of vacation in Italy. One of my good friends from college was getting married, and I didn’t want to miss his wedding. It was a good time for Kelly (my wife) and I to also take a week of vacation prior to the wedding. We love Italy, and a wedding […]<\/p>\n","protected":false},"author":1,"featured_media":620,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[47],"tags":[45,65,64,44],"class_list":["post-619","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","tag-ai","tag-big-data","tag-book","tag-data-science"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/12\/2019-12-10-10_34_44-Window.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=619"}],"version-history":[{"count":1,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/619\/revisions"}],"predecessor-version":[{"id":621,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/619\/revisions\/621"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/620"}],"wp:attachment":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Does this rising tide lift all boats?<\/h2>\n\n\n\n
How can someone get out of a reinforcing downward spiral?<\/h2>\n\n\n\n
This logic creates a problem. This problem boils down to creating a downwards spiral that is hard to impossible to get out of. Sticking with the mortgage example, people likely live in a neighborhood because of their social background. This background doesn\u2019t only influence their living location, but also their school choices and often job choices. If this impacts those people’s mortgage rate, and makes their mortgage more expensive, it is harder for them to first make their payments and secondly to potentially move out of that neighborhood to a ‘better’ area. If it’s harder for a person to make mortgage payments, they are more likely to miss payments. This in turn will reinforce the machine learning algorithm with more data that a certain population misses payments. So, even more people of your neighborhood will get higher interest rates. And this spiral is hard to impossible to get out of.<\/p>\n\n\n\nThere is a silver lining, and we should pay close attention to how we use AI models.<\/h2>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
![less people are living in extreme poverty<\/a>.<\/strong><\/p>\n\n\n\nHowever, the author\nof this book makes a couple of excellent cases where this is not true. One very\nspecific example that struck me personally is the optimization of staff hours\nin retail stores and specifically the phenomenon of ‘clopening’. Clopening is a\nterm created for when an employee of a retail closes the store one evening and\nopens that same store the day after, with limited actual time of work.<\/p>\n\n\n\nThis example was\nrelevant for my own personal situation. During my student years, I worked as a\nwaiter for about 7 years. I made less on an hourly basis than my friends did\nworking 9 to 5 desk jobs during summers, but I had the ability to make up for\nit with working extra hours (not by tips however. I am from Belgium, where\ntipping your waiter isn’t as big as in the US. I was happy with a 2% average\ntip at the end of the day). <\/p>\n\n\n\nAll large retailers\nby now use scheduling software to plan employee hours. In my naivety, I would\nexpect a computer program to have a simple control loop that avoids clopenings\nfrom happening. At least, from having them happen frequently. But clopenings happen\nfrequently, and when a scheduling algorithm is optimized for other metrics,\nthis weird side effect creeps in. Even worse, the author mentions cases where\nlow-wage workers’ hours are optimized for them to stay below the ‘eligible for\nmedical insurance’-mark. Just imagine having to close a store at 11PM, being\nthere the next day at 7AM for the opening but not working enough hours during\nthe week to be eligible for medical insurance.<\/p>\n\n\n\nI understand from a\nbusiness owner perspective that personnel cost is a major cost for a business.\nAnd as a business owner and shareholder, your primary goal is to optimize for\nbusiness results. The ethical question you should ask yourself is whether this\nbusiness result is gained by outsmarting your competition or by selling more,\nor rather if it is gained by profiting from paying your minimum-wage employees\nas little as possible.<\/p>\n\n\n\nAnd if this is the case, I would argue that the rising isn’t lifting all boats at all. If a retail employee has to suffer both clopenings and working very irregular hours there is something wrong with your scheduling algorithm. And you could indeed call this a ‘weapon of math destruction’ optimized for a poor metric.<\/p>\n\n\n\nHow can someone get out of a reinforcing downward spiral?<\/h2>\n\n\n\nData is the new oil.\nThe more data a business has, the more valuable that business can become. If\nyou want to train a machine learning algorithm, you need as much data as you\ncan get. Having the right data, and much of that, will enable you to make better\npredictions, and tailor your products and its pricing to the customers you are\nserving.<\/p>\n\n\n\nHowever, not all\ndata is created equally and not all data has the same value. There is already a\nlegislative framework that prohibits certain data to be part of a decision. A\nloan application shouldn’t be influenced by matters such as race or gender. The\nproblem with data is however, that some of those prohibited matters can be\nlearned from the data. <\/p>\n\n\n\nThe book has a\ncouple of great examples about the usage of a ZIP code. ZIP codes can for\ninstance be correlated to crime rates, to default rates on mortgages and to\ncollege graduation rates. As a mortgage provider, it only makes sense for you\nto use this metric to optimize your business. If people living in a certain\nneighborhood are more likely to default on their mortgage, it might make sense\nfor you to ask for a premium to that population.<\/p>\n\n\n\n This logic creates a problem. This problem boils down to creating a downwards spiral that is hard to impossible to get out of. Sticking with the mortgage example, people likely live in a neighborhood because of their social background. This background doesn\u2019t only influence their living location, but also their school choices and often job choices. If this impacts those people’s mortgage rate, and makes their mortgage more expensive, it is harder for them to first make their payments and secondly to potentially move out of that neighborhood to a ‘better’ area. If it’s harder for a person to make mortgage payments, they are more likely to miss payments. This in turn will reinforce the machine learning algorithm with more data that a certain population misses payments. So, even more people of your neighborhood will get higher interest rates. And this spiral is hard to impossible to get out of.<\/p>\n\n\n\nIt’s even harder to break out of this spiral if you consider that your zip code can influence not only your mortgage, but also your car insurance rate (apparently even more than drunk driving history) and even your acceptance rates in colleges and universities. The book has a number of examples in those area, that sound reasonable from a certain perspective (a business wanting to make profit), but in the same time make you think about how a person could get out of this spiral.<\/p>\n\n\n\nThere is a silver lining, and we should pay close attention to how we use AI models.<\/h2>\n\n\n\nI am an optimist. I\nbelieve that big data and AI will have a positive impact on society. To reach\nthis positive impact, we should pay close attention to how we use our models.<\/p>\n\n\n\nLet’s take the\nprevious example of the mortgage model trained on ZIP code data. In the\nexample, we discussed that the data tells us that people from certain ZIP codes\nare more likely to default on their mortgage. Instead of using this data and\nthis model to influence people’s mortgage rates, why don’t we use it to offer\nthem with better protection? This protection could be as simple as offering\nthose people with courses in financial literacy and regular follow-up to check\nup on their finances, or even offering them an additional insurance that would\nprotect them.<\/p>\n\n\n\nThat’s one place where the book in my point of view could have been a bit more prescriptive. There are only a couple of example of positive models being used, or models being used differently to have a positive impact. The book felt very negative in that regard, showing a lot of negative example, with only a couple of empowering positive impacts. As an optimist, I took those couple of positive examples at heart and want to make sure that we implement the technology the right way to have a positive impact.<\/p>\n\n\n\nConclusion<\/h2>\n\n\n\nThe main conclusion here is I highly recommendation you to read this book and form your own opinions. It certainly has given me a different perspective, and I hope it will do the same for you. <\/p>\n\n\n\nFrom a practical lens, I got a couple of learnings from this book. One is to consider multiple perspectives when thinking about or even designing an AI algorithm. The second is to make sure that the model doesn’t reinforce itself and creates downwards spirals. Finally, and that’s the most positive learning, is to consider how the technology can be used to create WIN-WIN situations, that can benefit all stakeholders.<\/p>\n","protected":false},"excerpt":{"rendered":"I just spent a week of vacation in Italy. One of my good friends from college was getting married, and I didn’t want to miss his wedding. It was a good time for Kelly (my wife) and I to also take a week of vacation prior to the wedding. We love Italy, and a wedding […]<\/p>\n","protected":false},"author":1,"featured_media":620,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[47],"tags":[45,65,64,44],"class_list":["post-619","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","tag-ai","tag-big-data","tag-book","tag-data-science"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2019\/12\/2019-12-10-10_34_44-Window.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=619"}],"version-history":[{"count":1,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/619\/revisions"}],"predecessor-version":[{"id":621,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/619\/revisions\/621"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/620"}],"wp:attachment":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}](\"https:\/\/ourworldindata.org\/uploads\/2019\/11\/Extreme-Poverty-projection-by-the-World-Bank-to-2030.png\"){kind=link}