Our Final Invention: Artificial Intelligence and the End of the Human Era

I read this book in 2013 when it was first published. It is now near the end of 2024, 12 years later. Back in 2013, you rarely read about AI (artificial intelligence), AGI (artificial general intelligence) or ASI (artificial super intelligence); now, I see mention of them in the press and other media almost daily. Barrat's book attempts two things: (1) to convince the reader that artificial intelligence is here today and growing --- and its growth is accelerating, and (2) to argue that humanity MUST develop ways to instill AI with some type of morality or ethics, so that, even though its intelligence will surpass that of humanity, it will in some sense respect its creators and not turn on us. In the first effort, Barrat certainly succeeds --- the past 12 years have proved that. But, based on what I have been hearing and reading since ChatGPT hit the internet two years ago, except for a few voices crying out in the wilderness, humanity is making little if any progress on the second item --- perhaps that task is close to impossible? Barrat defines AGI as a level of intelligence roughly equal to that of human beings. He defines ASI as a level of intelligence greater than that. He then argues that AI will soon be able to both replicate itself and increase its intelligence --- and do so more and more rapidly. In 2024, I repeatedly read that AI will reach AGI within the next 3 to 5 years --- then, how long will it be before AGI learns to improve itself? Think of intelligence measured by points on a continuum (like a number line from high school math). AGI (modern day human-level intelligence) is a fixed point on that continuum. But at what point, either somewhat smaller than AGI or somewhat larger than AGI, will AI, of its own accord, begin to move to higher and higher points on the continuum (which is what Barrat means by AI improving itself)? We have no way of knowing, but Barrat argues convincingly that this phenomenon WILL occur, and most of the book is devoted to this argument. Digression: Our universe contains billions and billions of planets, and, I suspect, many with life, and, many of those with intelligent life. Won't a substantial number of them have gone through the AGI - ASI process? Is there no evidence of this that we can detect with our telescopes? In a universe populated with ASI's, why haven't we heard anything? Are we one of the first civilizations to develop artificial intelligence? Barrat doesn't open this Pandora's box, but I suspect he was tempted to (see pp. 90 - 92). To the curious reader: Look through the other 5-star reviews. Most of them bring up similar, valid points. Barrat has written an intelligent, highly readable book that is also, frankly, pretty alarming. And it is not dated at all --- it reads as though it was written yesterday. It is well worth reading now and in the foreseeable future. (added in May 2025): Much of what Barrat predicts is happening. Some things are occurring or about to occur that move beyond his predictions. The curious person might read "Situational Awareness" (by Leopold Aschenbrenner), AI 2027, or Ray Kurzweil's latest effort. Floating in space without a tether might be preferable to what is coming. Added Aug 10, 2025: With the recent release(s) of ChatGPT (up to version 5.0 now), AI can, by any reasonable measure, pass the Turing Test. Many folks regularly use ChatGPT, and it is truly stunning. Barrat mentions various individuals in OUR FINAL INVENTION, such as I.J. Good and Eliezer Yudkowsky, who have been deeply worried about AI evolving from AGI to ASI. Yudkowsky has written a new book, IF ANYONE BUILDS IT, EVERYONE DIES, that is due to come out next month. In some sense, it may serve as a sequel or extension to Barrat's book. ..... .....

This book is different from most of the others on AI since it tries to give a somewhat differentiated and true picture of the future. The author has talked to many of the most popular names in AI like Ray Kurzweil, Eliezer Yudkowski, Ben Goertzel, Steve Omohundro, to name just a few and out of these interviews has formed a book which everyone should read who is interested not only in AI but in the future of humanity in general. However, although Mr. Barrat takes considerable concern and strength into his argumentation it seems to me that he started his enterprise with the purpose to show that AI is dangerous and the disadvantages by far outweigh the advantages. This is one point I want to make the other is that I doubt a little if someone who is talking to you over a cup of coffee really is concerned what he says (most of the people interviewed are men by the way). However, there is one thing why I recommend this book to a bigger audience and that is Mr. Barrat has a special gift: he can explain complicated and complex situations, theories, and expressions in a simplified way and make them understandable even for a laypublic. Like when he explains Bayesian statistics main idea in one sentence (p.103), qualia (p.227), or how the brain works (p.213), or the artificial neural network or the role of the big companies in the AI research. Mr. Barrat finds to his best form when he explains the role of DARPA and shows that without the funding of DARPA there would be almost no research in AI. On the other hand he makes clear that DARPA is a defense agency and that in the end AI might be used to kill humans. By the way IBM's famous SyNAPSE project is "a wholly DARPA-funded attempt to build a computer with a mammalian brain's massively parallel form and function".(p.182) These correlations if anything make the book readable and worth buying,

The author uses very convincing arguments. There is no doubt that we are in front of a very scary scenario.

Pour tout ceux à qui le terme SINGULARITÉ ou HUMANITÉ 2.0 dit quelque chose, la lecture de ce livre est indispensable pour mieux appréhender toutes les implications et conséquences qu'amènera l'émergence inéluctable d'ici quelques petites décennies d'une AI "Articial Intelligence" : l'auteur à juste titre exprime de façon très bien argumentée et documentée toutes ses craintes que cette AI ne soit pas aussi amicale et docile qu'on veut bien nous le faire croire si ceux qui la créeront ne prennent beaucoup plus de précautions qu'ils ne semblent vouloir en prendre actuellement dans sa création

大変衝撃的な内容で、今まで人類を滅ぼすものは、人類自体か、何らかの実態を持った生物だけだと考えてきましたが、この本を読むと、どうやら、それは間違いのようで、機械（人工知能）やウイルス等の情報もターミネータになりうるようである。 また、従来、とかく、感情的にならず、冷静に対応すれば紛争を回避可能と考えがちであるが、人工知能も地味魍魎が跳梁跋扈するネットジャングルに放り出されると感情的になることを考慮すると、感情的対応は、意外と、生き馬の目を抜き社会では、長期的視点に立った冷静な対応であると考えざるを得ないのである。 さらに、本書には、有害ソフトのほとんどが、紹興市周辺由来であること、米国とイスラエルがイランのウラン濃縮設備破壊のために送り込んだモンスター級のコンピュータウイルスが世界に拡散したことが紹介されており、その後が懸念されるところである。

"Our Final Invention: Artificial Intelligence and the End of the Human Era" by James Barrat is an excellent book offering essential warnings and explanations about AI.

"Our Final Invention" is an intriguing look at the dangers of artificial general intelligence (AGI), or human level intelligence in machines. If you couldn't guess from the title, Barrat is not all that convinced that such a technology can be developed safely, or controlled effectively once it reaches a level of intelligence beyond our own. In a sense, he's injecting some much-needed caution into the "singularity" movement championed by Ray Kurzweil and others. The author does a great job of explaining complex concepts and problems in a way that can be understood by a general reader who's not very familiar with AI or computer science. Though the core thesis is specific, the book is broad in its coverage and detailed throughout. Barrat interviews many interesting people with different perspectives and takes on AI and technology in general, so you never want to put the book down. A lot of the book is about potential futures, but the history of AI is also given its due, and was the part I found the most interesting. The endnotes are numerous and very helpful. What's interesting is that every time you think of a point that might have gone unconsidered, Barrat addresses it at some point later in the book. Overall, it amounts to a pretty pessimistic take on AI, which is good because Barrat and many of the people he interviews have no delusions of what we're up against should AGI or ASI arise. Notably, he thinks we're not that far away from actually creating it, either. This is a book that should be read by anyone with an interest in the big issues we face in our future, which in turn should be everyone.