“And if I ever manage to spend enough tokens to boil an entire lake with the datacenter’s heat, I will spend it with the greatest pleasure in the world. It’s my personal revenge against whoever forced me to use paper straws that dissolve in your mouth for all these years.”

I confess: it was deliberate provocation, written already knowing I was going to explain it here today. Because behind the joke there’s a serious thesis.

Last week I took apart here the AI fear propaganda from OpenAI, Anthropic, and NVIDIA. Today I want to use the same lens on a much older and much more profitable hysteria: the climate one. And there’s an irony in the middle: what unmasked this hysteria was AI itself, by accident.

The thesis is simple. The moment the datacenter race tightened for real, every “green” commitment from governments and big techs evaporated. The same Microsoft that promised to be carbon negative is restarting Three Mile Island. The same Germany that shut down its nuclear fleet now has a chancellor admitting it was a strategic mistake. What took down the climate agenda was another agenda, one with more money.

And before anyone labels me, my position in full: yes, there is a human effect on the climate. No, the world is not ending, and the distance between those two sentences is where the hysteria lives. What I dispute is the industrialized panic and the misallocation of trillions it justifies.

Every decade has its apocalypse

I grew up under the nuclear apocalypse of the Cold War. We lived with the certainty that the world could end in atomic mushrooms at any moment. When the USSR fell in 1989, that fear drained away, and the hole in the ozone layer filled the void. That one, by the way, is recovering decades after the 1987 Montreal Protocol, which phased out CFCs and friends: proof that a real problem with a cheap solution needs no hysteria.

Then came the Y2K bug. Then “global warming”. And when “warming” got hard to sustain in every headline, the agenda changed its name to “climate change”.

Notice the trick. Orwell called it Newspeak in “1984”: capture the words to capture the thinking. “Global warming” was a testable hypothesis: either it warmed or it didn’t. “Climate change” accommodates everything. Heat, cold, drought, flood, snow, lack of snow. Any event confirms the thesis, and a thesis that everything confirms explains nothing.

When the vocabulary is captured, the question becomes a crime. Whoever asks for evidence becomes a “denier”. A scientist funded by obscure interests becomes an “expert”, and “expert” gets sold as a monopoly on truth. The covid hysteria was the worst fruit of this way of thinking, but covid is for another article. The climate case is enough.

The human cost of this machine is measured. A 2019 Washington Post/Kaiser Family Foundation survey of American teenagers found 57% afraid and 52% angry because of the climate. A global UN poll that collected about 9.7 million votes asked about priorities: “action on climate change” came in last among 16 topics in the global total, and even in the poorest countries it sat at the bottom of the list. The people want hospitals and jobs. Whoever ranks the apocalypse first is whoever has their comfort guaranteed.

Follow the money: who funds the fear

About Greta Thunberg, one fact bothers both sides: she herself keeps none of the money. Her father told Politico she never received a cent, and she donated the entire 1-million-euro Gulbenkian prize to climate projects. The character is real and, by all accounts, unsalaried.

The girl and the startup

The machine around her is another story. The school strike started on August 20, 2018, a Monday. The Swedish startup We Don’t Have Time, founded by advertising man Ingmar Rentzhog, was still building its climate social network. On the day of the strike, Rentzhog posted the girl’s photo on his personal Facebook and the company account tweeted the same day, tagging Al Gore’s Climate Reality Project. Three days after the strike began, on Thursday, August 23, the family book, “Scenes from the Heart”, hit bookstores.

The startup’s fundraising prospectus, from November 2018, cited Greta 11 times, without the family’s knowledge; that round raised ~13 million Swedish kronor, and the company said it had raised 23 million in total by early 2019. She was a junior advisor to the company’s foundation from October 2018 to January 2019 and left. The product was the startup; the girl was the marketing.

The salaried protest industry

Then came the industrialization of protest. The Climate Emergency Fund, founded in 2019, funds Just Stop Oil, Extinction Rebellion, Scientist Rebellion, and friends. The one who opened the vault with $500,000 was Aileen Getty, heiress to the Getty oil fortune, mind you. Hollywood director Adam McKay came in with $4 million and a board seat.

By April 2022 the fund had already distributed $1.7 million to activists in 25 countries; the full-year tax filings point to ~$5 million in grants. Just Stop Oil alone received almost $1 million, helping pay the salaries of some 40 organizers. That soup on the Van Gogh canvas is salaried work, funded by oil inheritance and movie money.

Billionaire foundations and governments

Above the protest funds sit the big foundations. Bloomberg Philanthropies alone totals $1 billion in the Beyond Carbon campaign, and the New York Times called Bloomberg perhaps the biggest individual funder of climate activism in the world. The Hewlett Foundation committed $600 million over five years, Rockefeller announced in 2023 it would invest more than $1 billion in climate over five years, and ClimateWorks has already distributed more than $2 billion. That’s an industry, with the budget of a small country’s GDP.

And governments fund the pressure against their own political opponents. The EU pays €15.6 million a year in operating grants to some 30 green NGOs through the LIFE program, and in January 2025 the European Parliament held a formal debate about Commission-funded NGOs lobbying the Commission itself, in which the budget commissioner admitted it was “inappropriate” for parts of the Commission to sign agreements obliging NGOs to lobby specific MEPs. In Germany, Correctiv received €2.5 million in public money since 2016, the “Demokratie leben!” program had a budget of €182 million in 2024, and the CDU filed 551 formal questions about the political neutrality of state-funded organizations.

And who profits from the policies? Enron’s internal memo from December 1997, written from Kyoto, was already celebrating: “this agreement will be good for Enron stock!!”. The climate-industrial complex was born together with the treaty. Global revenue from carbon pricing, taxes and emission markets combined, hit a record $104 billion in 2023, and a 2008 Point Carbon study for the WWF estimated between €23 and €71 billion in windfall profits for generators in five EU countries in phase II of the European carbon market alone. And the biggest beneficiary of Western green targets has a name and an address: China controls more than 80% of every stage of the solar panel manufacturing chain and produced almost 75% of the world’s electric cars in 2025. The West subsidizes demand; Beijing sells the supply.

If you don’t know the Enron case, take a minute and watch:

Who wins

So who benefits from the fear narrative? Billionaire foundations, consultancies, subsidized industries, governments buying social peace with grants, and the West’s biggest industrial competitor. Greta walked away without a dime. The machine kept the megaphone.

The datacenter race

Some of the numbers I’m about to show come from the research I did for my talk at Tropical Ruby 2026, about energy and AI. The IEA data tells the story:

• Datacenters consumed 415 TWh in 2024, about 1.5% of the world’s electricity.
• In 2025: ~485 TWh, 17% growth in one year; AI-focused datacenters alone grew 50%.
• Central projection for 2030: ~945 TWh, the consumption of all of Japan, close to 3% of the planet.

In the US, datacenters account for almost half of all electric demand growth through 2030. And the money followed: big tech capex went from $400 billion in 2025 to ~$715 billion guided for 2026, more than the annual investment of the entire American energy sector. Cumulative datacenter investment between 2026 and 2030 is expected to hit $3.9 trillion.

And even that doesn’t cut it. The bottleneck stopped being chips and became copper and concrete:

• PJM, the largest grid operator in the US, closed for the first time in history a capacity auction below the reliability requirement, in December 2025: ~6.6 GW came up short. The capacity price jumped to $333.44 per MW-day, more than 11 times the level of the 2024/2025 auction, hitting the ceiling allowed by FERC.
• High-power transformers have a two-to-three-year waiting list, and the biggest models run past that. The American interconnection queue has more than 2,000 GW waiting, about 1.6 times the country’s entire installed capacity.
• Of the ~16 GW of large datacenters Sightline Climate forecast globally for 2026 (the American slice is ~12 GW), only ~5 GW were under construction at the start of the year, and the estimate is that 30% to 50% of that announced pipeline will slip.

The queue in the market now is for GPUs, power, and datacenters.

Remember the electric car, the green agenda’s great salvation? The entire global EV fleet consumed ~250 TWh in 2025, about 1% of the world’s electricity. Datacenters consumed almost double, already last year. And the projection tells the rest: from 415 TWh in 2024 to ~945 TWh in 2030, that’s +530 TWh of new load on the grid, almost three times the electricity of the entire 2024 global EV fleet (~180 TWh).

The entire decade of “green economy” was swallowed in two years of AI race. And the comic detail: in March 2026, Bernie Sanders and Alexandria Ocasio-Cortez announced the AI Data Center Moratorium Act, an attempt to ban new datacenters in the US. The green agenda spent decades telling you to consume less energy. Faced with AI, the coherent exit left over was trying to ban the new consumer.

The return of nuclear power

If the climate agenda were serious, its greatest ally was always nuclear power: the cleanest, densest, most reliable source there is. The same agenda that preached apocalypse treated nuclear as taboo for fifty years. Here’s what AI’s hunger for energy did in one year:

Big tech’s nuclear race

• Microsoft signed a 20-year contract to restart Three Mile Island’s Unit 1, 835 MW. Read that again: Three Mile Island, the ultimate symbol of American nuclear fear since 1979, resurrected as the Crane Clean Energy Center to feed datacenters.
• Meta locked in up to 6.6 GW in nuclear agreements with Vistra, TerraPower, and Oklo.
• Google ordered 500 MW in modular reactors from Kairos.
• Amazon expanded its contract with the Susquehanna plant to 1.9 GW through 2042 and invested in X-energy.

The technical reason is simple. In the US, nuclear runs at a capacity factor around 91%, and stayed above 92% on average for the decade through 2023; wind sits at ~34% and solar at ~23%. Datacenters want baseload, 24 hours a day. Sun and wind don’t deliver that without a mountain of batteries nobody has.

If you don’t know the 1979 Three Mile Island incident, watch:

The German case

And Germany? Germany shut down its last three plants on April 15, 2023 and became a net importer of electricity for the first time since 2002, including importing French nuclear. The Energiewende cost in the range of hundreds of billions of euros, German industrial electricity costs about double the American or more, and the German grid emits around 350 g of CO2 per kWh, against 22 g from nuclearized France.

Well then. In January 2026, Chancellor Friedrich Merz admitted in public: “It was a serious strategic mistake to phase out nuclear energy”, “we are now undertaking the most expensive energy transition in the entire world”. The head of the IEA itself, Fatih Birol, went public agreeing with him. Since May 2025 Germany stopped sabotaging nuclear France in the EU, and in June 2026 former bosses of the closed plants formally asked for the reactors to be reactivated.

Let me put my opinion on record here: closing the nuclear fleet was one of the biggest mistakes in Germany’s modern history. They impoverished their own industry, got hooked on Russian gas, went back to burning coal in the 2022 crisis, and today buy nuclear electricity from the neighbor. All of it in the name of an agenda they themselves are abandoning.

Fear versus the numbers

About the fear itself, the official numbers say otherwise. Deaths per TWh produced, counting accidents and pollution: coal kills 24.6; oil, 18.4; gas, 2.8. Nuclear: 0.03, less than wind and almost equal to solar. Chernobyl killed 30 workers directly, and the Chernobyl Forum and WHO projection for eventual cancers sits at up to 4,000 among the 600,000 most exposed, while Greenpeace was selling 93,000. The same report concluded the biggest public health problem was the psychological damage: people got sick from fear of radiation, not from radiation. Fukushima: zero radiation deaths among the public (Japan recognized a single radiation-linked death: a worker with lung cancer, in 2018); 2,313 deaths linked to the panicked evacuation. Fear killed more than the accident.

And the waste? All the used fuel from America’s commercial reactors fits on a football field, stacked less than 10 yards high, about 9 meters. France has been reprocessing its own at La Hague for decades. Finland built the world’s first deep geological repository for used nuclear fuel, Onkalo, already in the final stretch of licensing. A technical problem solved long ago; what was missing was political courage, and it arrived in the form of a datacenter power bill.

The man who’s been pointing this out for twenty years

The person who has organized this argument with rigor for two decades is the Dane Bjorn Lomborg. A political scientist by training, a statistics professor for years at the University of Aarhus, former visiting professor at Copenhagen Business School, fellow at Stanford’s Hoover Institution, founder of the Copenhagen Consensus Center, a think tank that works with hundreds of economists, Nobel laureates included. He blew up in 2001 with “The Skeptical Environmentalist” and became persona non grata of the green establishment, despite repeating the same sentence since forever:

“Global warming is real, but it is not the end of the world. It is a manageable problem.”

Take note: Lomborg accepts the scientific consensus on the existence of warming and its human cause. He defends a carbon tax and investment in green R&D. What he contests is the panic and the math of the policies. His 2020 book, “False Alarm”, is a catalog of numbers the headline never tells you:

The numbers the headline doesn’t tell you

• Deaths from climate disasters (drought, flood, storm, wildfire, extreme temperatures; Lomborg’s compilation over the EM-DAT database): from ~500,000 per year in the 1920s to fewer than 20,000 per year today, a 96% drop. Since the population quadrupled, individual risk fell 99%.
• In Lomborg’s synthesis of the Stanford Energy Modeling Forum models, complying with the Paris Agreement costs between $1 and 2 trillion per year, the most expensive pact in human history. And the effect of fulfilling every 2030 promise, measured in the MAGICC model, the same one used by the IPCC, is to reduce 2100 warming by 0.048°C, in his paper in Global Policy. Stretching every promise to 2100: 0.17°C. Every dollar spent on Paris avoids 11 cents of damage.
• The cost of warming with no action at all, in the damage function of Nobel laureate William Nordhaus: about 3 to 4% of global GDP in 2100. Lomborg applies this to the IPCC income scenarios and translates: instead of having 4.5 times today’s income, we’ll have 4.34 times. A manageable problem.
• New Zealand calculated that its net-zero costs between 5% and 17% of 2050 GDP, against the previous target of cutting 50% of emissions. Lomborg converts the country’s slice of emissions, by his math, into a three-week postponement of global warming.
• Tuvalu, the poster country of “it’s going to vanish into the sea”, grew 2.9% in area over four decades with the sea rising at double the global average. And the “187 million displaced by 2100” headline comes from a study for up to 2 meters of sea level rise with no adaptation; the same study, with dikes and realistic adaptation, calculates between 41,000 and 305,000 displaced.
• A study in The Lancet that analyzed 74 million deaths in 13 countries attributed 7.3% of them to cold and 0.4% to heat: in the current climate, cold kills some 17 times more.

The math nobody lets you do

Now the comparison that matters. The Copenhagen Consensus’s “Best Things First” calculated: $35 billion per year on twelve cheap solutions (treating tuberculosis, malaria bed nets, child nutrition, vaccines) would save ~4 million lives per year. Treating tuberculosis costs $6.2 billion a year and saves ~600,000 lives a year this decade, a return of ~$46 per dollar. Malaria alone killed 610,000 people in 2024, 95% in Africa, most of them children under 5. A delivered treated net costs about six dollars; saving a life with them costs around $5,500.

International climate aid alone, a quarter of all development aid, would be enough to fund tuberculosis, malaria, contraception, and nutrition in the entire world and still leave change. That’s the real choice on the table. Nobody lets you do this math out loud without being called a monster.

His 2005 TED talk, “Global priorities bigger than climate change”, is exactly this cost-benefit exercise, done on stage, twenty years ago. Worth the 16 minutes:

And since the comments will aim at the messenger, let me get the rap sheet out of the way early. About Lomborg you’ll hear that a Danish “scientific dishonesty” committee declared his first book objectively dishonest in 2003. True. And Denmark’s Ministry of Science overturned the decision at the end of the same year, over flaws in the process. You’ll also hear about Scientific American’s eleven-page hostile dossier. All on record. And none of it touches the numbers, which come out of the IPCC, the WHO, the FAO, and EM-DAT.

Thirty years of summits, and the score is zero

Before China, it’s worth looking at the full track record of the process that produced these policies. Three decades of treaties, each announced as historic, each with the same result:

The treaties

• Rio, 1992. The UNFCCC is born, with the written goal of “stabilizing greenhouse gas concentrations in the atmosphere”. Result: global fossil CO2 went from 22.5 gigatonnes in 1992 to 37.4 in 2024, +66%. At Mauna Loa, atmospheric CO2 went from 356 ppm to 427 ppm, with no visible dent in the growth curve.

• Kyoto, 1997. Target: industrialized countries would cut 5.2% below 1990 by 2012. The US Senate preemptively rejected the idea 95 to 0 before the final text even existed. Canada left the treaty in 2011 when it became clear it would miss the target. Global emissions rose 43% while it was in force. The countries that “complied” complied for free: the post-Soviet industrial collapse and the 2008 recession did the work. And Kyoto’s second period, the Doha one, covered 37 countries and ~14% of global emissions and entered into force on December 31, 2020: the last day of the period it governed.

• Kyoto’s credit mechanism, the CDM, became a fraud factory: industries in China and India produced HFC-23, a greenhouse gas 14,800 times more potent than CO2, just to destroy it and sell credits, a €2.6 billion scheme. The Öko-Institut, auditing for the European Commission itself, found that 85% of projects had a low likelihood of integrity. A 2024 study in Nature Communications calculated that fewer than 16% of the carbon credits analyzed correspond to real reductions.

• Copenhagen, 2009. The goal was the binding successor to Kyoto. It collapsed live: the final text was taken note of, not adopted, drafted on the last night by the US, China, India, Brazil, and South Africa. What survived was the promise of $100 billion a year for the poor world, met for the first time in 2022, two years late.

• Paris, 2015. The target system is self-assigned and the targets are not legally binding. Of the 197 parties, only 58 had economy-wide targets in domestic law or policy, and only 17 with ambition equal to or greater than what was promised on paper (Grantham Institute, via Lomborg). Nature wrote in 2017 that every major industrialized country was falling short. The US left, came back, left again (effective January 2026) and, in January 2026, announced its withdrawal from the UNFCCC itself, the 1992 convention, something no country had ever done. Since Paris, emissions rose from 35.4 Gt to 37.4 Gt, every year, except for the covid dip.

• Belém, 2025 (COP30). The house was ours, and the result was a portrait of the state of the process. Before it opened: delegations from Africa and the Pacific formally complaining about the lodging crisis, with stratospheric hotel rates and part of the delegations sleeping on cruise ships. During: a fire in the blue zone evacuated the delegates in the middle of the final negotiation, with 13 people treated for smoke inhalation. It was the first COP without an American delegation in 30 years, and Trump mocked the event on Truth Social, the highway through the rainforest and all. Attendance: ~42,000 in person, half of what Dubai had. And what came out of it? A text called “Global Mutirão” (mutirão is Portuguese for a collective work effort) with the fossil fuel phase-out plan ripped out after pressure from Saudi Arabia and allies, despite more than 80 countries backing it, and a promise to triple adaptation finance already born weakened, from 2030 to 2035. The bill for the Brazilian taxpayer: ~R$4.7 billion in the federal package alone (Union Budget, BNDES, and Itaipu), not counting what the state and the city spent on their own and never consolidated in public. One more circus paid for with our money, with nothing to show at the end.

The COP circus

And there’s the circus itself, which isn’t just ours. COP28 in Dubai had 83,884 in-person participants, including 2,456 fossil fuel lobbyists, which perhaps explains the final texts. Glasgow had 118 private jets; COP27, some 315 executive jet trips. And the cherry on top of our edition: Avenida Liberdade, 13 km of road cutting through Belém’s Environmental Protection Area, with 68 hectares of vegetation cleared (the project was the state’s and predates the COP, but the cut in a protected area is fact).

The scoreboard of 32 years of treaties: +68 ppm of CO2 in the atmosphere, zero deceleration. The emissions curve doesn’t know the treaties exist.

Patient zero of the hysteria: Al Gore

Before the treaty machine, the propaganda had a founding father. Al Gore lost the 2000 election and reinvented himself with “An Inconvenient Truth” (2006): an Oscar, the 2007 Nobel Peace Prize shared with the IPCC, and the template for every hysteria that came after. Practically every apocalyptic prediction you hear to this day premiered in that film.

The one who dissected the film claim by claim was Roy Spencer: PhD meteorologist, former senior climate scientist at NASA, co-creator with John Christy of the UAH satellite temperature dataset, the independent reference for global temperature. His position is the “lukewarmer” one: CO2 warms a little, the theory is solid up to there; the catastrophe is the exaggeration. In 2017 he published “An Inconvenient Deception” (Kindle on Amazon), which I read for this article. Gore’s catalog of busted prophecies, with Spencer’s data:

The catalog of busted prophecies

• “Ten years to the point of no return” (2006). Ten passed, almost twenty passed. The trend measured by Spencer’s own satellite: ~0.16°C per decade, through June 2026.
• Arctic summer ice “could be gone as early as 2014”: the line is from his Copenhagen speech, December 2009, citing a modeler who later disowned the number. It didn’t vanish.
• Kilimanjaro’s snows by 2016: still there.
• Stronger and more frequent hurricanes after Katrina: what came was a drought of almost 12 years without a single major hurricane touching the US, from Wilma in October 2005 to Harvey in August 2017, the longest since 1850. NASA researchers calculated that a gap like that happens once every 270 years. And the global cyclone count has no upward trend since the 1970s.
• “Up to 20 feet of sea level rise”: the measured global rate today is ~3.4 mm per year, about 1.4 inches per decade. The Battery Park tide gauge in Manhattan measures 1.16 inches per decade since 1856, and part of that is the land sinking. At this rate, 20 feet takes more than a millennium.
• The drowning polar bears: the film’s animated scene was born from a short note in Polar Biology about four bears found dead after a storm in 2004. The estimated global population sits between 22,000 and 31,000, probably larger than in the 1970s, when no reliable estimate even existed and the real threat was unregulated hunting, solved by the 1973 treaty. Twenty years later, “the bears are dying” still circulates as fact.
• The Syria drought, “the worst in 900 years”, said to have caused the civil war: rainfall data for the west of the country, Aleppo, Damascus, Homs, stayed above average; the strong 2007/08 drought concentrated in the northeast. What broke the countryside was the government tripling the price of diesel in 2007-2008, which made pumping well water unviable.
• The flooding of the 9/11 memorial by hurricane Sandy, presented in the sequel as “I told you so”: in the original film he talked about gradual sea level rise, never storm surge. Spencer calls it explicit deception.
• The tabletop experiment with Bill Nye, heating CO2 in glass jars in the Climate Reality Project’s “Climate 101” video (2011): Anthony Watts’s analysis showed the experiment doesn’t work and the result was faked in post-production. According to Spencer, the video stayed on Gore’s site for years afterward.
• The crops: the 42 climate models used by the UN predicted summer warming in the American Corn Belt; NOAA data shows no significant warming there since 1960.

The cast of the prophecies is going strong. The bears, too:

The character

And the character: Spencer documents that Gore got rich on “green” investments (he’s a cofounder of Generation Investment Management), keeps mansions and an industrial-sized carbon footprint. And even the establishment didn’t swallow the film whole: in 2007, the British courts found nine errors in “An Inconvenient Truth” and called the sea level part “distinctly alarmist”. Spencer’s word for Gore’s technique is “wordsmithing”: writing in a way that leads the audience to conclude what he never stated in so many words.

The audience, by the way, had already tuned out: in the first weeks of the 2017 sequel, the screening Spencer attended in a 750-seat theater had three people, and one left halfway through.

If this subject got to you, the two bedside books are Spencer’s, above, and Lomborg’s “False Alarm”, which I used in the previous section. One dissects the propaganda; the other dissects the bill.

Spencer’s rap sheet also comes ready-made in the comments: he’s one of the favorite skeptics of Republicans in Congress, the UAH dataset had famous corrections in the 90s and 2000s (published and incorporated decades ago), and a 2011 paper of his in Remote Sensing was so contested that the journal’s editor resigned, saying the article had slipped through by mistake. All real. And none of it changes the Battery Park tide gauge or the calendar of expired prophecies. Criticize the numbers, if you can. The messenger is just the messenger.

The arithmetic of insignificance

Let’s get to your electric car and your paper straw. China accounted for 32% of global fossil CO2 in 2024, more than the US, the EU, and India combined. China started construction on 94.5 GW of new coal plants in 2024 alone, a ten-year record, and burns more than half of all the planet’s coal every year since 2011.

Now your individual actions, with the numbers:

• The entire global EV fleet displaced ~1.3 million barrels of oil per day in 2024, something like 0.5% of the world’s fossil CO2. That’s the total size of the impact of the biggest green policy of the decade.
• Lomborg calculated in 2020: an average subsidy of $10,000 per EV to save ~8 tonnes of CO2, worth $48 in the RGGI, the carbon market of the American Northeast states, at the time. You paid ten grand for 48 dollars of carbon.
• Lomborg’s math, with IEA data from the time: an EV emitted ~26 tonnes of CO2 over its lifetime against ~34 for a gasoline car, a 24% cut. And it depends on the grid: the Volvo XC40 study calculated the battery payback at 146,000 km driven on the global average electric mix (84,000 km on the European mix, 47,000 km on wind power). On a coal-powered grid, it’s worse.
• Paper straws: even if every estimated straw on the coasts ended up in the sea in one year, they’d make ~0.03% of the annual plastic entering the ocean.
• Household recycling: of all the plastic that became waste in history through 2015, only 9% was recycled; 12% incinerated; 79% went to landfill or into nature (Geyer et al., Science Advances 2017). And plastic doesn’t recycle in a loop: each cycle degrades the polymer, it becomes a lower-quality product, and it postpones the landfill by one step. The worst part: NPR showed that the plastics industry knew since the 70s that recycling didn’t pay and sold the symbol anyway, so you’d keep buying guilt-free. The exception that proves the rule is aluminum: recycling uses 5% of the energy of producing from scratch, and Brazil is the world champion at it, with 97.3% of cans recycled in 2024. Recycling works when the physics and the economics close. When they don’t, it’s kitchen liturgy.

Your paper straw doesn’t show up in this math. Neither does your EV. The math is dominated by whoever builds 94 GW of coal plants per year.

These gestures are liturgy, and this is where their real damage lives. The paper straw gives that feeling of “at least I’m helping”, and that feeling numbs. The person makes the micro-sacrifice, feels part of saving the world, and closes their eyes to the actually big problems, like the children dying of malaria in Africa I showed two sections back. Objectively, the gesture moves nothing. Subjectively, it buys the conscience. It’s worse than useless.

And Lomborg’s complete thought is even more brutal: if the rich world zeroed all its emissions and stayed at zero for the rest of the century, the temperature increase in 2100 would drop 0.7°F, less than half a Celsius degree, in his math with the MAGICC model. The West’s total sacrifice is a rounding error.

The world doesn’t fit in your head

Part of the hysteria exists because most people have no sense of the planet’s physical scale. Let’s fix that real quick, in human language.

The Sun. The Earth continuously intercepts ~173,000 terawatts of solar energy. All of civilization consumes ~20 TW. The ratio is more than 8,000 to 1, and the Earth captures only one part in 2.2 billion of what the Sun emits. A single large coronal mass ejection carries billions of tonnes of plasma with energy in the range of 10^25 joules, something like fifty million Tsar bombs. In 1859, the Carrington Event set telegraphs on fire and drew auroras in the sky over Cuba. No human control panel participated.

The Moon. Tides are the Moon deforming the ocean: a bulge of water on its side, another on the opposite side. The lunar effect on the tide is ~54 cm; the Sun’s, ~25 cm. In the Bay of Fundy, the tide swings 16 meters, and each flood tide brings in ~160 billion tonnes of water, more than all the rivers in the world combined in the same period. Meanwhile, the Moon drifts away 3.8 cm per year and the Earth’s day gets longer by some 2 milliseconds per century. Some 620 million years ago, the day was almost 22 hours.

El Niño. Normally the trade winds push the Pacific’s warm water west. Every 2 to 7 years they weaken, the warm water slides east, and the global average temperature rises 0.1 to 0.2°C for a year or two, on its own, no cars, no industry. La Niña does the opposite. When you see a “hottest year in history” headline, check whether there was an El Niño: there almost always was.

New York and Lisbon. New York sits at 40.7°N. Lisbon, at 38.7°N; Madrid, at 40.4°N. Same latitude. New York buries itself in snow every winter; Lisbon and Madrid have mild winters by comparison. The explanation: the westerly winds throw moderate air across the Atlantic onto Europe, while the American east coast receives freezing continental air from the Arctic, with the Atlantic transporting ~1.2 petawatts of heat north via the meridional current of which the Gulf Stream is part. Iceland, at 64°N, has winter averages comparable to New York’s. Latitude rules far less than current and wind.

The Sahara fertilizes the Amazon. Every year, ~182 million tonnes of dust leave Africa crossing the Atlantic; ~28 million tonnes fall on the Amazon basin, carrying ~22,000 tonnes of phosphorus, almost exactly what the rain leaches from the forest’s soil each year. The world’s biggest desert feeds the world’s biggest forest, and it’s been happening for millions of years before you were born.

And if you zoom out the time scale, it gets even more absurd:

The Sahara was once green. Between ~11,000 and ~5,000 years ago, the Sahara was savanna with lakes, hippos, and crocodiles, and the Cave of Swimmers, in Egypt, holds rock art of people swimming in what is now desert. The driver was the orbital precession cycle, and the flip to desert took decades to a few centuries, depending on the region. And it comes back: the next orbital summer maximum arrives in some 10,000 years.

The Earth was once a snowball. Twice between ~717 and ~635 million years ago, the Sturtian and Marinoan glaciations took ice all the way to the tropics: there are glacial deposits at tropical latitudes to prove it. The Earth got out of it on its own: the global ice shut down the process that pulls CO2 out of the air, volcanoes stacked CO2 for millions of years up to hundreds of times the current level, and the greenhouse effect beat the ice.

The Mediterranean once dried up. Between 5.97 and 5.33 million years ago, the Mediterranean Sea nearly evaporated and left layers of salt more than a kilometer thick on its bed. When the Atlantic broke through at Gibraltar, the flood filled the sea back up with 90% of the volume in months to two years, at a flow estimated at a thousand times the Amazon’s.

Sharks in Kansas. In the Cretaceous, with the sea level 100 to 200 meters above today’s, a shallow sea split North America in two, from the Gulf of Mexico to the Arctic. Mosasaurs and sharks swam where Kansas is today.

The plants almost starved. At the last glacial maximum, CO2 fell to ~180 ppm, dangerously close to the ~150 ppm range where C3 plants start to fail. Glacial-era trees preserved in the La Brea tar show physiological carbon starvation. Plants evolved in a world with much more CO2, and today the planet is greening: the Zhu et al. 2016 study, publicized by NASA, measured greening across 25 to 50% of vegetated areas between 1982 and 2009, with ~70% attributed to CO2 fertilization.

CO2 became a synonym for a dangerous gas that must be fought at any cost. For plants, it’s food. When CO2 rises, green life tends to rise with it. If CO2 were poison, there would be no plants.

Rainforest in Antarctica. In 2020, Nature published the discovery of intact roots of a temperate forest that grew at 82°S, some 900 km from the South Pole, ~90 million years ago, with a mean annual temperature of 12 to 13°C. New Zealand climate, in Antarctica.

The climate has always changed

“Climate change” is an oxymoron. Changing is what the climate does, since forever, without asking permission.

For most of Earth’s history there was no permanent ice at the poles. The planet’s normal condition is greenhouse, no polar ice. In the Cambrian, CO2 was 10 to 20 times the pre-industrial level. At the peak of the Cretaceous, the global average was some 10°C above today’s, or more. Antarctica only gained permanent ice 34 million years ago.

And the great die-offs, which the hysteria loves to invoke, happened five times before any industry:

• ~444 million years ago (Ordovician): ~85% of species, glaciation.
• ~372 million (Devonian): ~75%, ocean anoxia.
• ~252 million (Permian, “the Great Dying”): ~96% of marine life, volcanism.
• ~201 million (Triassic): ~80%, volcanism.
• ~66 million (K-Pg): ~76%, the dinosaur asteroid. That one’s just the most famous.

On the more recent scale: the current ice age began 2.58 million years ago, with ice cycles every ~100,000 years in the last million of them. 20,000 years ago, hundreds of meters of ice covered where Manhattan is today, and the sea level was 120 meters lower. 11,500 years ago, at the end of the Younger Dryas, Greenland warmed ~10°C in about a decade. Then came the Medieval Warm Period, the Little Ice Age with the Thames freezing until 1814, and the “year without a summer” of 1816, courtesy of the Tambora eruption. All of it without a single coal plant.

To be clear about what I’m saying: none of this proves the current warming doesn’t exist, and the current speed of the CO2 rise is indeed faster than the natural precedents. Lomborg himself says so. What this scale destroys is the phrase “end of the world”. A planet that has already been 10°C warmer with the sea 120 meters lower is not in its death throes. The serious discussion is about the cost-benefit of each policy, which is exactly the discussion the hysteria forbids.

The hysteria died of natural causes: money

Fear doesn’t save the planet, doesn’t help the environment, and doesn’t improve anyone’s life. Fear sells narrative, funds consulting, elects politicians, and stuffs foundations with money. The cost of this era is right there: trillions torched on gestures that don’t move the thermometer, while 4.9 million children died in 2024 before turning 5 from diseases the developed world forgot exist.

And then AI came along and closed the subject by accident. Notice how nobody talks about COPs in the headlines anymore. The conversation now is datacenter gigawatts, 20-year nuclear contracts, gas turbines. The same CEOs who signed climate letters are fighting over power plants. The same governments that imposed paper straws approve reactor subsidies. The climate agenda left the stage without being refuted, evicted by a bigger agenda.

If the end of the world were really on their agenda, you wouldn’t be watching Three Mile Island being restarted to train chatbots. You’re watching it. It was theater, and the theater closed when a more profitable show opened in the room next door.

Fear doesn’t save the planet. Fear sells. And whoever bought the fear is now buying megawatts.

This post has two parts. First, the news: Microsoft’s Majorana 2 announcement and why the physicists in the field remain with one foot (both, actually) behind. Then, the educational part: a decent explanation of Shor’s algorithm, because a lot of people repeat “quantum computers break RSA” without the slightest idea of the mechanism, and the mechanism is too beautiful to leave out.

Majorana 2: the announcement

On June 2, 2026, at the Build conference in San Francisco, Microsoft revealed Majorana 2, the second generation of its topological quantum chip. The headlines: qubit stability 1,000 times better than Majorana 1’s (announced some 15 months earlier), and the timeline for a scalable quantum computer shortened from 2033 to 2029.

The central number in the technical paper: Z-parity lifetimes above 20 seconds (more precisely 22 ± 1 seconds) in an InAs device with lead, against 1 to 12 milliseconds in Majorana 1’s aluminum devices. Three orders of magnitude of improvement. The main hardware change was swapping the superconductor from aluminum to lead, whose superconducting gap is about four times larger, which doubled the reported topological gap and drastically dropped the parity error rate.

These materials engineering numbers are real and measurable, and even the critics acknowledge the progress on that front. The problem lies in what the paper does NOT show.

The history the announcement would rather not remember

Microsoft’s topological quantum program carries almost two decades of history, and part of it is embarrassing. In 2018, a Microsoft-funded team at TU Delft published in Nature evidence of quantized Majorana conductance. In 2021 the paper was retracted, after physicists Sergey Frolov and Vincent Mourik demonstrated that the data had been selectively presented. TU Delft’s scientific integrity committee concluded the data had been “unnecessarily corrected”. A second paper from the same group was retracted in 2022.

When Majorana 1 was announced in February 2025 as “the world’s first topological qubit”, the accompanying Nature paper said less than the press release: Nature’s own reviewers wrote that the published measurements did not represent evidence for the presence of Majorana zero modes in the devices. When Microsoft presented additional data at the APS meeting in March 2025, Frolov (University of Pittsburgh) publicly called the data “just noise”, and physicist Henry Legg (St Andrews) raised similar objections, including criticism of the Topological Gap Protocol itself, the method Microsoft uses to identify the topological phase.

And did Majorana 2 answer those criticisms? Within hours the same physicists answered no. Legg, to Science News: “Nothing in this preprint resolves the fundamental issues. Nothing in the presented data proves the existence of a topological qubit or Majoranas in these devices.” Frolov, to Scientific American, was more acid: “When Microsoft is mentioned these days, physicists and quantum computing specialists just chuckle or raise their eyebrows.”

Marin Ivezic’s detailed analysis at PostQuantum, which I recommend reading in full, lists what’s missing with precision:

• No X measurements. A functional qubit requires two complementary types of measurement: Z (parity of a single wire) and X (joint parity of the tetron’s two wires). The Majorana 2 paper only presents Z. And the X measurement was exactly the point of contention with Majorana 1. Without it, what was demonstrated is a long-lived parity state in a superconducting wire. A qubit, not yet.
• No demonstration that the states are topological. The measurements are consistent with Majorana zero modes, but they are also potentially consistent with trivial Andreev states that mimic the same signatures. The paper itself admits the ambiguity.
• No logic gates, no entanglement, no algorithms. It’s a characterization experiment, with no computation at all. Chetan Nayak, who leads the program, says he has unpublished data on qubit control. Until he publishes, it’s a claim, without evidence.
• No peer review. The paper came out on Microsoft’s website and on arXiv. Given the program’s history of retractions, the absence of peer review at launch deserves extra scrutiny.
• No demonstrated reproducibility. The 22 seconds come from a single wire of a single tetron of a single chip. No ensemble of identical devices or independently fabricated chips.

My summary of the opera: the materials engineering genuinely advanced, and that counts. But the trillion-dollar question (“is this a topological qubit?”) remains exactly as open as it was before the announcement. And shortening the public timeline from 2033 to 2029 on top of a result that doesn’t even demonstrate a qubit is a marketing decision, with the scientific rigor left for the future.

Shor: why factoring numbers becomes period hunting

Now the fun part. Everybody repeats that quantum computers break RSA. The algorithm responsible is Peter Shor’s, from 1994, and its central trick is one of the most elegant ideas in computing. This Computerphile video explains it very well, and I recommend watching it before continuing:

RSA’s security depends on a simple fact: multiplying two giant primes is easy, but taking the result N and figuring out which primes they were is brutally hard. The best known classical algorithm takes sub-exponential time, which in practice means that with large keys the Sun burns out before you finish.

Shor’s insight was to prove that factoring N is reducible to another problem: finding the period of a function. You pick a number a coprime to N and look at the function:

f(x) = a^x mod N

This function is periodic: there’s an r such that the values repeat every r steps. Once you find that period r, a bit of classical number theory (which any ordinary computer solves in an instant) hands you the prime factors of N. The problem is that, classically, finding the period for a giant N is as hard as the original problem.

This is where the quantum computer comes in, in four movements:

1. Prepare a superposition of many values of x at the same time.
2. Evaluate f(x) in superposition (the famous quantum parallelism: one evaluation over all the x at once).
3. The resulting state now encodes the function’s periodicity.
4. Apply the Quantum Fourier Transform (QFT), which uses constructive and destructive interference to amplify exactly the frequency components of the period, and measure.

The detail almost everybody gets wrong: you don’t “read all the answers at once” (measurement collapses the superposition and gives you ONE value). The genius is in using interference to make the wrong answers cancel out and let the global information you want (the period) survive in the probability distribution. The QFT turns a global property of the function, which no individual measurement reveals, into a local, measurable feature. That’s why factoring drops from sub-exponential to polynomial time on a quantum computer. And that’s why post-quantum cryptography exists as a field.

So quantum computers are good with “periodic functions”?

That’s the hasty conclusion I want to undo. Periodicity and Fourier-style analysis are indeed a strong suit, and not only in Shor: quantum phase estimation (QPE), hidden subgroup problems and a good part of physical system simulations live in that territory. But the correct formulation is different: quantum computers shine when the problem has exploitable mathematical structure, something that can be mapped onto superposition, entanglement and interference.

There are quantum advantages that have nothing to do with periods. Grover’s algorithm does unstructured search with a quadratic gain (√N instead of N). The simulation of quantum systems (molecules, materials) is probably the most promising application of all, because there the problem’s Hilbert space is naturally the machine’s space. There’s HHL for linear systems, optimization techniques, some machine learning methods. The common thread is always the same: there’s structure (symmetry, groups, geometry) that quantum interference exploits better than any known classical method.

And the reverse question: can you decompose any classical algorithm into periodic functions and gain quantum speedup? No. A quantum computer can simulate any classical algorithm (in the field’s notation, BQP contains P), so technically everything runs there. But running is different from accelerating: without exploitable structure, you pay all the quantum overhead (error correction, qubit coherence, cryogenic temperatures) to arrive at the same result, probably slower.

Sorting, arithmetic, most graph problems, the bulk of what a computer does day-to-day: none of it has any known superpolynomial quantum speedup. For those tasks, the classical computer is faster, cheaper and more reliable, and will keep being so.

A quantum computer is an accelerator for problems with specific structure. A better universal machine, it is not.

Conclusion: put the hype back in the closet

Put the two halves of this post together and the picture becomes clear. On one side, Shor’s algorithm is real, is beautiful, and is the reason the entire post-quantum cryptography race exists. On the other, the hardware needed to run Shor against a real RSA key requires millions of physical qubits with error correction, and the state of the art announced with fanfare in 2026 is… a 22-second parity state in a single wire, which the experts don’t even agree is a qubit.

In practice, in the short term, quantum computing remains a very niche, very limited, very impractical solution. The real applications that arrive first will be in chemistry and materials simulation, inside research labs and datacenters, not in your life. Don’t expect to use a quantum computer yourself any time soon. Follow the news the way I follow it: with genuine curiosity for the physics, and a raised eyebrow for the press releases.

I can probably pinpoint why a certain argument is wrong. But I am very incompetent the other way around: given a proper data collection, how to do proper exploratory data analysis? What are the correct methodologies for any given scenario? And of course, all the math involved.

I just posted 2 articles to shed some light on exactly that. Most developers’ “common sense” on dealing with data is primitive aggregations such as sums and averages. Start talking about “standard deviation” and you lost half of the developer population. Start talking about binomial or Poisson distributions and you lost the remaining half. Now get into linear regression and bayesian statistics, and almost everybody left the room already.

We live in the 21st century. Every 60 seconds, Facebook receives an extra 3.3 million new posts, YouTube receives 400 hours of videos, Instagram receives 55,555 uploaded photos, and WhatsApp exchanges 44.4 million messages. Even e-mail: more than 206 million of them are fired off. By the time you finish reading this post, you can multiply that amount by 5 to 10.

“We are drowning in information,

but we are starved for knowledge”

– Various authors, original probably John Naisbitt

I will spend a few weeks diving into R. Yes, many people will talk about Julia, but we can’t deny the amazing body of knowledge, the experience, and the robust and extensive set of packages available for R, including learning material. There’s a great tool called RStudio, which just saw its 1.0 release.

And within the many materials I gathered, one stood out just for its introduction, and I still have to review the book as a whole. It’s called “Learning Statistics with R”, by Daniel Navarro from the University of Adelaide, Australia. I find it interesting because it’s a psychology professor teaching proper statistics through the basics of R, which is exactly what I wanted. You can purchase a hardcopy or download the free PDF.

I like the introduction so much that I wanted to share a few paragraphs to motivate you to join me in learning better statistics. So let’s dive right into it:

The cautionary tale of Simpson’s paradox

The following is a true story. In 1973, the University of California, Berkeley got into some trouble over its admissions of students into postgraduate courses. Specifically, the thing that caused the problem was that the gender breakdown of their admissions looked like this …

         Number of applicants     Percent admitted
Males           8442                    44%
Females         4321                    35%

… and they got sued. Given that there were nearly 13,000 applicants, a difference of 9% in admission rates between males and females is just way too big to be a coincidence. Pretty compelling data, right? And if I were to say to you that these data actually reflect a weak bias in favor of females, you’d probably think that I was either crazy or sexist.

Oddly, it’s actually sort of true … after Berkeley got sued, people started looking very carefully at the admissions data (Bickel, Hammel, & O’Connell, 1975). And remarkably, when they looked at it on a department by department basis, it turned out that most of the departments actually had a slightly higher success rate for female applicants than for male applicants. The table below shows the admission figures for the six largest departments (with the names of the departments removed for privacy reasons):

                      Males                          Females
Department   Applicants   Percent admitted   Applicants   Percent admitted
A                825             62%            108              82%
B                560             63%             25              68%
C                325             37%            593              34%
D                417             33%            375              35%
E                191             28%            393              24%
F                272              6%            341               7%

Remarkably, most departments had a higher rate of admissions for females than for males! Yet the overall rate of admission across the university for females was lower than for males. How can this be? How can both of these statements be true at the same time?

Here’s what’s going on. Firstly, notice that the departments are not equal to one another in terms of their admission percentages: some departments (e.g., engineering, chemistry) tended to admit a high percentage of the qualified applicants, whereas others (e.g., English) tended to reject most of the candidates, even if they were high quality.

So, among the six departments shown above, notice that department A is the most generous, followed by B, C, D, E and F in that order. Next, notice that males and females tended to apply to different departments. If we rank the departments in terms of the total number of male applicants, we get A > B > D > C > F > E (the “easy” departments are in bold).

On the whole, males tended to apply to the departments that had high admission rates. Now compare this to how the female applicants distributed themselves. Ranking the departments in terms of the total number of female applicants produces a quite different ordering C > E > D > F > A > B.

In other words, what these data seem to be suggesting is that the female applicants tended to apply to “harder” departments.

And in fact, if we look at all Figure 1.1 we see that this trend is systematic, and quite striking. This effect is known as Simpson’s paradox. It’s not common, but it does happen in real life, and most people are very surprised by it when they first encounter it, and many people refuse to even believe that it’s real. It is very real. And while there are lots of very subtle statistical lessons buried in there, I want to use it to make a much more important point . . . doing research is hard, and there are lots of subtle, counterintuitive traps lying in wait for the unwary. That’s reason #2 why scientists love statistics, and why we teach research methods. Because science is hard, and the truth is sometimes cunningly hidden in the nooks and crannies of complicated data.

Before leaving this topic entirely, I want to point out something else really critical that is often overlooked in a research methods class. Statistics only solves part of the problem. Remember that we started all this with the concern that Berkeley’s admissions processes might be unfairly biased against female applicants. When we looked at the “aggregated” data, it did seem like the university was discriminating against women, but when we “disaggregate” and looked at the individual behavior of all the departments, it turned out that the actual departments were, if anything, slightly biased in favor of women.

The gender bias in total admissions was caused by the fact that women tended to self-select for harder departments. From a purely legal perspective, that puts the university in the clear. Postgraduate admissions are determined at the level of the individual department (and there are very good reasons to do that), and at the level of individual departments, the decisions are more or less unbiased (the weak bias in favor of females at that level is small, and not consistent across departments). Since the university can’t dictate which departments people choose to apply to, and the decision making takes place at the level of the department it can hardly be held accountable for any biases that those choices produce.

That was the basis for my somewhat glib remarks earlier, but that’s not exactly the whole story, is it? After all, if we’re interested in this from a more sociological and psychological perspective, we might want to ask why there are such strong gender differences in applications. Why do males tend to apply to engineering more often than females, and why is this reversed for the English department? And why is it the case that the departments that tend to have a female-application bias tend to have lower overall admission rates than those departments that have a male-application bias? Might this not still reflect a gender bias, even though every single department is itself unbiased? It might.

Suppose, hypothetically, that males preferred to apply to “hard sciences” and females prefer “humanities”. And suppose further that the reason why the humanities departments have low admission rates is because the government doesn’t want to fund the humanities (Ph.D. places, for instance, are often tied to government funded research projects). Does that constitute a gender bias? Or just an unenlightened view of the value of the humanities? What if someone at a high level in the government cut the humanities funds because they felt that the humanities are “useless chick stuff”. That seems pretty blatantly gender biased. None of this falls within the purview of statistics, but it matters to the research project.

If you’re interested in the overall structural effects of subtle gender biases, then you probably want to look at both the aggregated and disaggregated data. If you’re interested in the decision making process at Berkeley itself then you’re probably only interested in the disaggregated data.

In short there are a lot of critical questions that you can’t answer with statistics, but the answers to those questions will have a huge impact on how you analyze and interpret data. And this is the reason why you should always think of statistics as a tool to help you learn about your data, no more and no less. It’s a powerful tool to that end, but there’s no substitute for careful thought.

Download the Book

Did you get intrigued? So download the book and let’s study some statistics beyond the mere basics. I am not saying this is the best book, just one that seems interesting, and if you happen to know a good book teaching statistics for novices through the use of R without any prior knowledge, let me know in the comments section below.

I’m guilty too, addicted to arguing about serious subjects and irrelevant ones alike. It’s part of the entertainment, I guess, but I’ll admit that sometimes I just run out of patience. Anyone who watched my more recent talks will remember the part where I say this: people look for “validation” of their ideas and arguments. It seems that the more people agree, the more valid the idea supposedly is, the so-called “democratic” process where 50% + 1 guarantees victory.

There are several problems with this, and I already discussed part of the topic in Mea Culpa: Democratic Organizations Don’t Work. Although I’m not a formal, working mathematician, my foundation is mathematics. One of the good things I remember from my time at the Institute of Mathematics and Statistics at USP was precisely the heavy mathematical foundation I was put through. To this day I bring what I learned into my daily life.

Laws Are Laws

The first big thing I learned is this: given a set of axioms, and given that a certain theorem is proven, I can use that tool with the guarantee that it will always work within the stipulated premises.

So, given that natural numbers exist, given x belonging to the naturals and given the algebraic properties of the naturals, I know that y = x + x and that y will be in the naturals. With that, I can say with full certainty that 2 + 2 is 4 and will always be 4.

That matters because I don’t need a majority’s “validation” to know it. It doesn’t matter if someone proposes a vote and 1 million people say 2 + 2 is 5. Quantity of opinions doesn’t undo Algebra. Certain areas of scientific knowledge have been fully proven, and they don’t depend on the consensus of “opinions.”

Majority and authority mean nothing within that context.

This gives me several certainties. There’s an entire body of scientific knowledge across the most different areas, from mathematics to physics, biology, psychology, sociology, and medicine, that is trustworthy and at our disposal. And using it doesn’t depend on consensus, on voting, or on the “validation” of any group.

It also gives me the certainty that just because I don’t know something, that doesn’t mean someone else doesn’t already know it, and it raises my esteem for proper research into that body of knowledge. The error is being too lazy to do that research and assuming that because some majority said something, or because someone with apparent authority said something, it “must be true.”

To me, by definition, everything is wrong until it’s rigorously proven to be true, by the same methods that guaranteed that body of scientific knowledge.

First World Problems

In the self-help world there are thousands of “formulas,” or “recipes.” Recipes to lose weight, for success, to be happy, and so on. It’s an enormous industry that, at least by 2010, was moving around USD 11 billion. It won’t stop and should grow.

We move along Maslow’s pyramid. Our parents’ and grandparents’ generation still had to worry about the most basic survival: food, health. In this new generation, a huge portion of the population lives much better, and that base is no longer the biggest priority.

But human beings will always look for problems to worry about, the famous “first world problems”. Sometimes it seems we’re a species built to find problems even where they don’t exist.

That’s why the self-help market grows so much, and why problems that seemed not to exist a few centuries ago now seem so much bigger: depression, stress. Medications that were never so needed 2 or 3 centuries ago are now manufactured by the ton: Zoloft, Prozac, Paxil, Celexa.

For the record, I’m not belittling these problems. A huge portion is quite real, and thankfully modern medicine has the instruments to treat it. Still, it’s undeniable that a lot of people make bad use of that circumstance, since unlike a broken leg, there’s no physical evidence of this kind of problem.

I bring it up because it’s already part of our culture to assume this kind of pathology is a common thing. With each new generation, it seems everyone is made of crystal and any small disruption already breaks them. The impression is that if this generation were magically transported to the 17th century or earlier, no one would survive.

The Effect of Self-Help

All that said, here comes the part that bothers me most: the magic formulas, the silver bullets. Within the self-help market, I don’t mind that much. After all, what harm is there in someone reading “Who Moved My Cheese” to entertain themselves?

The problem is when pseudo-science gets taken as real science.

It’s what I call “because it worked for X it must work for Y,” or what plenty of people have heard me repeat ad nauseam: “correlation is not causation”.

All self-help is one or more formulas, duly wrapped in heavy marketing and sold as universal solutions.

“Just think positive.”

On their own, statements like that are harmless, but they quickly unfold into biblical proportions.

What’s the problem with formulas? Again I go back to my mathematical foundation. A mathematical formula, on its own, is useless: it’s undefined, incomplete. To be completely defined, it needs at least two more elements, a Domain and an Image. For example, look at the formula:

f(x) = 1/x

Immediately someone could say “but this formula is wrong, because if x is zero the calculation doesn’t work.” It’s a good observation, but it starts from a premise: that x can be any number that exists.

But if I define it correctly and say:

given x belonging to the domain of real numbers, and given x different from zero, f(x) = 1/x, will result in the image of real numbers

Now this formula, in the field of Algebra, is well defined and undisputable. Notice what I did with this simple example: I gave an incomplete formula and immediately generated a discussion about it. That’s exactly what happens in every social media discussion and every self-help topic.

To repeat: all self-help is one or more formulas that don’t define the domain or the image. Each person who reads that incomplete formula tries to complete it with their own domain, “for me it works” or “for me it doesn’t work,” and we’ve just found the simplest origin of every flame war, every hater, and every internet troll.

On the other hand, we also find the first way to distinguish an invalid formula from a “possibly” valid one. With the invalid one, you’ll have enormous difficulty defining the domain. We always start from the largest domain, the one that would work for everyone in every scenario. The formula breaks on the spot, so we start reducing that domain and, invariably, it shrinks until it reaches a single case that once worked.

Complex Systems, Biographies, and the Fallacy of “Evidence”

And normally the formulas really do reduce to a single individual. That’s the case with every biography: the sequence of events a certain individual went through, only they went through, and it doesn’t repeat.

I’ve been talking about complex systems for a long time, since at least 2009 the subject has caught my attention. I’ve fallen into the trap of deriving theories about the concept, but I’ll admit that so far I haven’t succeeded. In fact, the whole field of sociology and network studies is still in its infancy in this kind of knowledge, with little in the way of postulates and laws. So handle this kind of subject with far more care than usual.

Quantum physics has also been unduly exploited in the self-help world. Look at calamities like “What the Bleep Do We Know” to understand how to turn a serious subject into a joke (that plenty of people think is serious).

At the risk of being superficial, I’ll summarize the relevant part like this: in the real world, we’re all part of a complex adaptive system. An action I take can trigger several others, which in turn trigger several more, and many of them interact with each other, creating a network of incalculable nodes and connections.

It’s practically impossible to determine the end of a chain of events in a network with countless variables. I say “practically” because we still don’t know how to determine all the variables of the equation, and for all practical purposes that makes any prediction impossible.

Of course, given certain limits of space, time, and error margins, we can predict plenty, enough to send rockets to Jupiter with precision. On the other hand, there are many cases whose error margins are still hard to control, and meteorology itself is an example.

That’s where the idea comes from that a butterfly flaps its wings in China and a tsunami hits Rio de Janeiro. The butterfly didn’t cause the tsunami. It was only an initial condition of an intricate chain of events that happened to result in it.

Now, given a certain result, in many cases we can look backward and trace the chain of events back to the initial conditions. Anyone who watched CSI or some detective show knows we can collect evidence and use deduction to connect events back to a point of origin.

That’s how all “formulas” are created. The chain of events has already happened and won’t change anymore, because within our Universe the past is immutable and time only has one direction.

George Santayana once said “those who cannot remember the past are condemned to repeat it.” If we take what I’ve said so far literally, we’ll conclude that History rarely repeats itself. Just because the initial conditions and the result resemble those of another event, the chain in between is rarely the same. Reading it as “History repeating itself” is coincidence and a series of tricks our brain plays all the time, including confirmation bias.

I know this stretch can get complex, but the point is this: just because something happened once, that doesn’t mean the same chain will happen again.

The initial conditions and the chain of events are that restricted “domain” of the “formula” I mentioned above. And as Duncan Watts would say, everyone fools themselves all the time with the concept of “obvious.” Today it seems obvious to say Facebook is a success. But in 2002 no one could have predicted that in 2012 there would be this Facebook thing and that its IPO would open with a valuation of USD 104 billion. After the fact, we can try to trace the events backward and find the initial conditions and the chain that made it possible. Never before.

Empirically, out of everything I’ve seen offered as a silver bullet, the “best formula” looks very good, until you start to question it: “it looks like a Formula that leads to a good result (Image), but what are the necessary initial conditions? What is the Domain of this formula?”

And when you start to question that, the theory doesn’t hold up for 1 second. Take Lean Startup, the tech world’s fashion. Look at Eric Ries’ initial conditions: you’ll find examples of Groupon, Intuit, and HP, companies that are anything but “small startups,” but the book is evangelized to every small entrepreneur. There’s a clear contradiction of initial conditions, and the scientific method is very different from what the book suggests.

“But there are several good examples of small startups that followed Lean Startup and did well.” And here’s another lesson anyone who watched CSI knows how to spot: a contaminated crime scene is useless. In none of those cases can you say the “cause” of success was using Lean Startup. There may be a “correlation,” and a small one, because we’re talking about a complex adaptive system.

Did the butterfly “cause” the tsunami? The probability is very small. But then marketing comes in. As Joseph Goebbels, Nazi Germany’s propaganda minister, would say, “the bigger the lie, the more they’ll believe it,” or the derivative of that, “repeat a lie enough times and people will believe it’s true.”

As Carl Sagan would say, “extraordinary claims require extraordinary evidence.” And that depends on exhaustive work of testing to break the hypothesis. A medication, made seriously, takes years, sometimes decades, to reach pharmacies, but silver bullets go viral in a matter of hours or days. Depending on who speaks and how, it gains the status of “unquestionable truth” very fast. Any prosperity-gospel preacher can tell you that.

Just because it “seems” to have worked for 1, 2, 10 cases, that isn’t enough to make something true. Charles Darwin’s “Theory” of Evolution of species has tons of concrete, solid evidence, has been questioned, attacked, and meticulously torn apart for more than 150 years, and for all practical purposes is already close to an absolute truth. Even so, it still hasn’t earned the status of “Law.” If it hasn’t, why would something far less tested deserve it?

Conclusion

In practice, unless it’s part of the body of scientific knowledge already recognized and exhaustively documented, everything else is questionable, should be questioned, and should never earn the dogmatic status of “truth,” by definition.

Self-help is pseudo-science, by definition.

Formulas that guarantee a certain “result” (Image) without a Domain are incomplete and undefined, and that will necessarily split opinions, because each person has their own domain. And just because a group is bigger than whoever is questioning it, the formula doesn’t become true. The opposite would be following Goebbels’ doctrine.

And what actually is proven science needs no quorum, majority, or “validation.” 2 + 2 will always be 4, no matter who wants to argue otherwise.

A tip: for any subject that crosses your path, use Google. Put the term next to the word “debunked” or “skeptic,” “fake,” or “scam,” and you’ll find the answer. For example, search for “The Secret Debunked” and see how much has already been published taking down the so-called “Law of Attraction”. I do this all the time. Nothing is unquestionable, no matter who’s saying it.

Since childhood I’ve been a strong defender of destroying pseudo-science, because it causes more harm than almost anything else. Remember that coincidences happen, and they aren’t so rare.

“We’re putting together an excellent new product that will solve your problems, would you use it?” says the validation survey of a new startup. Result: 70% of the 1,000 people who answered said yes. So the conclusion is clear: the product is valid and can’t go wrong, right?

Wrong. Those numbers don’t mean a thing. On their own they carry no validity, and they support no conclusion at all. “But it’s 1,000 people! How can you say that?” This article isn’t trying to be statistically rigorous, just to point out a few of the many places where almost everyone gets it wrong when running a survey these days.

First of all, what’s your product’s target audience? Let’s say it’s the population of the interior of the southern states, classes C and D, ages 50 to 60, with little experience using computers. Now imagine the question above was part of a survey done online, shared on Facebook and Twitter.

Who answered it? Plenty of people from classes A through C for sure, concentrated in the southeastern capitals, ages 16 to 25, on a computer every day. You asked the wrong question to the wrong people.

For the sake of the example, let’s say you took care to pick the right population and dodged what’s known as “Biased Sampling.” Even so, the question is still bad, and it should be obvious why. It’s what we call a “Loaded Question”: the wording already nudges the respondent toward a certain answer.

For starters, it takes for granted that the product is “excellent” and that it “will solve” the person’s problems. A less loaded form would be: “We’re building a product with features X; assuming you have problems Y, do you think this would help solve them?”

There are plenty of techniques for steering clear of baseless premises, like actually explaining the value proposition instead of stuffing the question with rhetoric. It’s very easy to build a fallacious question that pulls a false answer out of people.

It gets worse when you take those 1,000 people, single out the 700 who said “yes” to the rigged question asked of the wrong audience, and start drawing conclusions from it. “Oh, we noticed that of these 700, half have a credit card, shop online at least once a month, and say they’d share the service with at least 2 friends on social media.” And it gets worse still when the ones who said “no” happen to be the audience closest to the target you want to reach.

I also consulted Ricardo Couto* who explained the issue. “Knowing the respondent, their habits and motivations is much more valuable for the business than having a high percentage of favorable responses about the product or service in an online questionnaire. By the way, calling a form put together in 5 minutes on Google Docs or SurveyMonkey a ‘survey’ is, at best, naïve. What you have there is a questionnaire. Questionnaires can be useful if used well. But in general, they’re just sets of questions around a subject. A survey can even make use of that tool, but not every questionnaire is a survey.”

Statistics, probability, and research in general are the work of professionals who master this field. It’s no trivial matter, and it’s not something to handle carelessly. This example is just one of the many ways to fool yourself with numbers.

Mark Twain popularized the phrase “There are three kinds of lies: lies, damned lies, and statistics.” The subject has been discussed and rebutted so widely that it produced a famous 1954 book by Darrell Huff, “How to Lie with Statistics,” which catalogs the intentional and unintentional errors in interpreting statistics.

Ricardo also added: “If we think of an online questionnaire as a serious research tool, first you have to know how research is done. I really doubt that the people who say they ‘are doing’ a survey and send you a link to a questionnaire took into account the order effect, the observer effect, or avoided leading questions – factors well known to anyone who does real research. And I’m not even getting into statistical validity, which you only reach by applying statistical methods (a simple percentage isn’t descriptive statistics, okay? you have to know what a p-value is), or into selecting respondents suited to the target audience you intend to reach. Or do I only want, as users, customers, or consumers, professionals in my age bracket, my income range, and my online shopping and social media behavior?”

So the next time you go to create an account on one of those dozens of online survey services, check whether the tool meets the bare minimum of a well-done survey and walks you through the right steps: defining the audience, collecting, processing, and analyzing the data rigorously. Without that, all you’re left holding is a pile of opaque numbers, with no meaning and no conclusion to draw from them.

*Ricardo Couto is a Consultant in User Experience and Research, Mentor at Aceleratech, President of the Interaction Designers Association SP (IxDA SP), and Member of the Brazilian Information Design Society. He studied Cognitive Psychology (master's) at the Federal University of Pernambuco and specializations in Information Design, Higher Education Teaching, and Distance Education Technology. He was a university professor at undergraduate and graduate level for 7 years.

I decided to kick off 2013 with an article I hope reads as a positive message for the year. The message is simple:

The world today is good, very good, and it's getting better and better.

I get frustrated every time I see theoretically “educated” people who prefer to believe the world is getting worse, that we’re worse off today than yesterday. Today we’re “impersonal,” we don’t play in the street anymore, we lost solidarity, we burn our forests, every corporation is a villain, people suffer more.

Respectfully, I disagree with all those ideas that we’re getting worse. One symbolic reason to write this today is that we made it through 2012, one of the last known “End Times” prophecies, as if it were nothing (which, by the way, it always was).

Anyone who reads the news knows we have increasingly overcrowded metropolises, violence flaring up all the time in various parts of the world, wars in the Middle East, religious intolerance and terrorism, natural disasters that “seem” to be the result of the infamous “Global Warming.”

All of that is true. And none of it is a symptom that the world is getting worse.

The World Is Better

Despite all the cataclysmic prophecies, let me sum it up fast: the world is better today than it was 50 years ago, 100 years ago, or millennia ago. Fortunately there are people in science, doing serious research and work, who piled up data year after year, and today we can compare it item by item. One of these organizations is Gapminder, which puts all of this data out there so you can see it with your own eyes.

I recommend watching the TED talks by Hans Rosling reading that data. He shows how over recent decades we raised life expectancy for most of the world, improved quality of life, and sharply cut poverty, malnutrition, and inequality. The vast majority of people today live better, with more comfort, and for longer.

Violence is dropping too, as this TED talk by Steven Pinker shows. The book Abundance, by Peter H. Diamandis, gives you even more data on this era of abundance we’re in.

And even the dreaded Global Warming gets constantly questioned. Plenty of scientists aren’t convinced by the data we have. And it hardly matters whether Global Warming is “real”: there are clearly people raking in money and fame off the “fear” of this possible natural phenomenon. Watch the documentary Cool It, where researcher Bjorn Lomborg debunks, one by one, the things you, a reader of this blog, probably never stopped to think about.

Example: if the Kyoto Protocol had succeeded the way the lobbyists wanted, it would have lowered the world’s temperature by all of 0.008 degrees Fahrenheit by 2100. And the European Union’s climate policy would drop the temperature by a mere 0.1 degrees Fahrenheit, at the ridiculous cost of USD 250 billion a year for the next century. With half that money you could wipe out poverty, hunger, and disease worldwide.

If everyone in the United States switched to a Prius, it wouldn’t cover even 0.5% of the carbon that technically should be cut by mid-century. And if everyone in the United States used nothing but energy-saving lightbulbs, it wouldn’t cover even 0.2% of the carbon to be cut.

Comedian George Carlin explained this in the clearest way I’ve ever seen:

Getting it?

Who Does This Benefit?

No “small act,” like recycling an aluminum can, makes much difference on its own today (don’t worry, it doesn’t hurt by itself either). What really makes a difference are the steady advances of science. Disciplined experimentation and the buildup of knowledge one day led us to agriculture, keeping famine at bay even with the world’s overpopulation.

That achievement has been sliding back thanks to fads like “organic food”, which ignores years of progress in agronomic technology, the same progress that made sure our generation exists at all.

The reason for this article is to question the obvious: who does all this benefit? The easy answer is the big corporations that know how to cash in on it. But they aren’t the problem. The problem is the people who make it easy for the corporations, turning every discussion into a moral judgment.

Today, if I say I don’t help any “social” cause, I somehow become guilty of someone’s misfortune. A story that bugs me to this day came from a friend I respect a lot. He’s a successful businessman, and he feels a little guilty for having more than others. So when he can, he offers alms and that sort of thing. One day, if I remember right, one of those beggars snubbed the alms, and my friend didn’t know if he was even allowed to feel as frustrated as he was.

That’s my long discussion about the Rights of Man. And to be crystal clear, I’ll spell it out: I have no obligation to help anyone, just as nobody has any obligation to help me.

The key word here is “obligation.” I always have the choice to help or not, but never the obligation. Exercising that choice is an individual call that nobody gets to judge.

The world is simple: everything I produced, by my own effort, is mine and no one else’s. If I want to donate it, that’s my choice. If I want to throw it away, that’s another perfectly valid choice of mine. Nobody has a right to something of mine, and nobody gets to judge what I do with it. The reverse holds too: I’ll never have a right to anything of anyone’s for free, and I should never judge what others do with their own possessions. It’s simple.

Alms and donations are the same thing. I’ll give to whomever I want, when I want, and if I want. And I don’t judge anyone who donates or who doesn’t. Now, it really gets to me when someone has the nerve to think they can judge me for what I do with something entirely mine.

Go back to my story: why does that friend of mine need to feel “guilty” for something he’s not guilty of? Everything he earned came from his own effort and sweat, so it’s his. On top of that, his work made lots of jobs possible, and even let other companies exist, spreading indirect wealth to plenty of people. Why should he feel guilty for not giving away for free something that’s his?

This is the prostitution of the mind that working people suffer daily. The more they work, the harder they push, the more they earn, the more guilt they feel and the more judged they feel. And who are these “others”?

This is the real target of my article. I want to expose the people who benefit from this orgy of fake altruism and ignorance. Charismatic types who talk pretty, with inspiring speeches (especially the ones who invoke justice, inspiration, and motivation), who pile on the hyperbole and present almost no data from a credible source.

They all run the same speeches, the same pompous, contentless routine. They show videos of the world falling apart, apathetic people, little kids, repeating the same old “We must…”, “We need to…”:

"We need to unite," "We need to be more supportive," "We need to care more," "We need to think about our neighbor," "We need to put down the phone and really connect," blah, blah.

BULLSHIT!

Anyone who says a lot of “We need to,” always in the first person plural “we,” clearly wants to be the dictator of a group. Someone who on his own has little to offer, so he rallies more people to borrow the credibility of the masses. And past a certain headcount, whatever he says starts to seem valid.

This is only possible because we let it happen. Because we don’t look at the data, we don’t dig for the facts, we get moved by any slickly edited video with animals, landscapes, and children, especially children in some miserable stretch of the African interior. We applaud every one of these demagogues.

And you know what demagogues like that do once they have masses behind them? They make everyone who is actually working, creating, and producing wealth in the world feel guilty. They point the crowd at these people and pressure them to “share” their property, making them feel lousy if they don’t. And they do it in a “cheerful,” almost festive way: they park the property-owner in the middle of the crowd and say things like “guys, wouldn’t this guy be super cool if he donated what he has to our cause?”, clapping and smiling sarcastically while they shame and corner the producer, leaving no way out.

This, ladies and gentlemen, is called fraud. It’s a fraudulent, criminal, and plainly offensive act. Let’s be clear: I won’t put myself through the abuse of criminal acts. And neither should you.

The lesson here is simple. If someone comes to talk to you about another person or group, ask: “who does this benefit?” If someone tells you “such-and-such company must be going bankrupt, since it doesn’t even have money to donate to this very important social cause,” why does the person passing that along care? Who does it benefit? Then flip the question: why would a company not donating necessarily mean it’s in bad financial shape? The most obvious answer, “the company has no interest in donations,” seems “morally too ugly” to consider. And that’s exactly the likeliest one. Except we’ve reached such a degrading point in our morality that this choice is almost treated as a crime.

Don’t Fall for the Talk

So pay attention: the world today is better than yesterday. Don’t doubt it. Donating and taking part in social causes are valid things. But judging others for not taking part is a crime.

Any company, any worker runs one very simple operation: trades. A worker trades the value of what they produced for money. A company trades a product for money. And someone who takes part in social causes trades their own work capacity for another kind of currency: conscience. Every donation is a transaction of your money or possessions for “you feeling good about yourself.” It’s a commercial exchange like any other. After all, have you ever seen anyone donate with the goal of feeling bad about it afterward?

This gets more obvious when you run into the infamous weekend do-gooders. People who want to buy positive karma because they’re frustrated, depressed, unhappy with themselves. So they jump into “save the world” movements, donate money, donate time, donate work. But the moment they get bored, they’re back at the beach, at a barbecue playing soccer, out by the pool. Except now they’re “morally in the black” and put themselves in a spot to judge everyone else with an “I helped and you didn’t.”

Just to be clear, I’m not talking about everyone who’s charitable. Plenty take it dead seriously and, in most cases, stay anonymous, because they’re not in it for attention, they’re in it to help. But the type I’m talking about is in the business of “feeling good,” and that needs an audience, needs telling everyone what good people they are, how everyone should copy them to feel good the same way. Like addicts pushing drugs because the drugs make them feel good.

And they’re worried about the little children? See the trauma they’re putting them through by telling false stories of the end of the world:

The very existence of people like this is one more piece of evidence of how much better our times are. At the start of the 20th century, no one would spare a thought for “social causes” like today: many didn’t even have enough to eat, had no clothes, died from common diseases.

Except this crowd keeps getting more irritating and more tiresome. In case it wasn’t clear, I’m fully in favor of serious social causes, and I can’t stand the weekend do-gooder hipsters. There are serious movements that deserve respect being stained by these groups. Want to really give yourself to a cause? Join organizations like the Red Cross, work on it full-time, and never force anyone, physically or morally, to make the same choice you did.

This goes for social causes and for any kind of “cause.” Be sharp for once: there are plenty of clever people who want you doing the dirty work for them. Be suspicious of anyone who speaks for “us,” for the “good of all,” or any cheap demagoguery of that sort.

The world today is better, check the data, and it keeps getting better. All we have to do is work and produce more and more wealth. Producing is how the world got here, and producing is how it keeps improving. Taking wealth from others, what plenty of people euphemistically call “sharing,” was never the way.

The motivation comes from the 11th principle of the Agile Manifesto:

“The best architectures, requirements, and designs emerge from self-organizing teams.” – Principles behind the Agile Manifesto

Agility requires self-organization. But this concept is alien to most people and organizations. The 12th principle further adds:

“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” – Principles behind the Agile Manifesto

For this you need to understand the concept of a Learning Organization. Or better: understand the dynamic moment of learning and continuous improvement called the Edge of Chaos. Implementing Agility, in physics terms, means the following:

Removing the organization, which is a complex system, from its state of dynamic equilibrium, increasing the overall entropy of the system, forcing it toward chaos. At the ideal point, at the edge of chaos, self-organization happens and the organization becomes a learning organization.

Remember: “learning” also includes exploring the unknown, incorporating new information. Doing what you’ve always done means you’re not learning anything, by definition. You need to do things differently, make mistakes, and improve in order to learn.

Let’s get into the concepts now.

Translation: Chaos Theory in Organizational Development

from Wikipedia

Self-organization, as opposed to natural or social selection, is a dynamic change within the organization where system changes are made by recalculating, re-inventing and modifying its structure in order to adapt, survive, grow, and develop.

Self-organization is the result of re-invention and creative adaptation due to the introduction of, or being in a constant state of, perturbed equilibrium. One example of an organization which exists in a constant state of perturbation is that of the learning organization, which is “one that allows self-organization, rather than attempting to control the bifurcation through planned change.” (Dooley, 1995)

Being “off-balance” lends itself to regrouping and re-evaluating the system’s present state in order to make needed adjustments and regain control and equilibrium. By understanding and introducing the element of punctuated equilibrium (chaos) while facilitating networks for growth, an organization can change gears from “cruise” to “turbo” in regard to speed and intensity of organizational change. While maintaining an equilibrial state seems to be an intuitively rational method for enabling an organization to gain a sense of consistency and solidarity, existing on the edge of a chaotic state remains the most beneficial environment for systems to flourish, develop and grow.

For instance, two competing organizations that differ in regard to their levels of homeostasis will not be in competition for long. Generally speaking, the organization with the less-stable structure will come out ahead while the constant stability of the latter will eventually lead to its own demise. Although quite similar, small differences in homeostasis levels are enough to make a tremendous difference in future outcomes for each organization. The notion of similarity in origin vs. dissimilar results comes to fruition with the emergence of bifurcation.

The concept of bifurcation cannot be explained without discussion of the term frequently labeled “sensitivity to initial conditions.” Sensitivity to initial conditions refers to the high level of importance of primary conditions from which the future path and direction of a system stems. This sensitivity to initial conditions is commonly referred to as the Butterfly Effect, in which a butterfly flaps its tiny wings in one end of the world which results in a typhoon or hurricane somewhere else on the globe. While this is an entertaining notion, sensitivity to initial conditions remains in reality a very abstract concept without the presence of bifurcation, which is mathematically labeled as the actual splitting point of two near-identical entities which, due to the sensitivity of initial conditions, tend to take two very distinct paths and result in two totally different geographically or even evolutionary places.

The primary goal of an organizational development (OD) consultant is to initiate, facilitate, and support successful change in an organization. Using chaos theory as the sole model for change may be far too risky for any stakeholder buy-in. The concept of uncertainty on which chaos theory relies is not an appealing motive for change compared to many alternative “safer” models of organizational change which entail less risk. By careful planning and management of disorder a successful intervention is possible, but only with a truly dedicated arsenal of talented and creative resources. By permitting or actively forcing an organization to enter a chaotic state, change becomes inevitable and bifurcation imminent; but the question remains, “Will the new direction be the one intended?” In order to account for the direction of the new thrust, most planning attention should be focused on attractors instead of the initiation of disorder.

Although chaos eventually gives way to self-organization, how can we control the duration, intensity, and shape of its outcome? It seems that punctuating equilibrium and instilling disorder in an organization is risky business. Throwing an organization off balance could possibly send it in a downward spiral towards dissemination by ultimately compromising the structural integrity (i.e. identity) of the system to the point of no return. The only way to reap the benefits of chaos theory in OD while maintaining a sense of security is to adjust the organization towards a state of existence which lies on the edge of chaos.

By existing on the edge of chaos, organizations are forced to find new, creative ways to compete and stay ahead. Good examples of such learning organizations are found throughout the field of technology as well as the airline industry, namely organizations such as Southwest Airlines, which used re-invention not just for survival, but also to prosper in an otherwise dismal market. In contrast, there are organizations which, due to extended periods of equilibrium, find themselves struggling for survival. Telephone companies, for instance, were once solid and static entities that dominated the communication market. While the rest of the world was developing new communication technology, telephone companies did not creatively grow at the same rate. The result is an organization that is battling to stay alive unless they embrace the element of chaos due to crisis, and allow creative adaptability to function freely so that self-organization and re-invention can occur.

While organizations existing on the edge of chaos are known to be the most creative and adaptive of organizations, how do their members feel about constant evolution and re-invention? Is it possible to identify with, and stay loyal to, an organization that constantly changes shape? The short answer is yes. As long as the organization does not change its core essence, its identifiable, shared purpose, its members will still experience the organization as a developing system that changes shape but retains the same familiar face.

Perhaps the safest way to use chaos theory in OD is not in the instigation of organizational change, but in the use of its principles in dealing with issues that arise within the organization. By embracing organizational phenomena previously seen as dysfunctional, such as interpersonal conflict, and using it as a source for transformational change by applying principles found in chaos theory (Shelton, 2003), an organization can make “lemonade out of lemons” and become more responsive to change agents while continuously moving ahead and growing from the inside out without the fear of complete chaos.

How Business Is a Lot Like Life: The 4 Principles

Fast Company Magazine, March 2001. How Business is a Lot Like Life

• Equilibrium is a precursor to death. When a living system is in a state of equilibrium, it is less responsive to the changes that are occurring around it. It is most at risk when it feels most secure.

• When threatened or when galvanized by a compelling opportunity, living things move toward the edge of chaos. This condition evokes higher levels of mutation and experimentation and is more likely to result in fresh new solutions.

• As living things move closer to the edge of chaos, they have a tendency to self-organize, and new forms emerge from the turmoil. This property of life, called “self-organization and emergence,” is a major source of innovation, creativity, and evolution.

• Living systems cannot be directed along a linear path. Unforeseen consequences are inevitable. The challenge is to learn how to disturb them in a manner that approximates the desired outcome and then to correct the course as the outcome unfolds.

Definitions

from Wikipedia

Reductionism can either mean (a) an approach to understanding the nature of complex things by reducing them to the interactions of their parts, or to simpler or more fundamental things or (b) a philosophical position that a complex system is nothing but the sum of its parts.

In physics and systems theory, the superposition principle, also known as superposition property, states that, for all linear systems, the net response at a given place and time caused by two or more stimuli is the sum of the responses which would have been caused by each stimulus individually. So that if input A produces response X and input B produces response Y, then input (A + B) produces response (X + Y).

In mathematics, a nonlinear system is a system (…) that does not satisfy the superposition principle, or whose output is not proportional to its input. Less technically, a nonlinear system is any problem where the variable(s) to be solved for cannot be written as a linear combination of independent components. (…) Nonlinear problems are of interest to physicists and mathematicians because most physical systems are inherently nonlinear in nature. Nonlinear equations are difficult to solve and give rise to interesting phenomena such as chaos. The weather is famously nonlinear, where simple changes in one part of the system produce complex effects throughout.

Emergent Behavior: Thriving on the Edge of Chaos

by Chris Rollins, January 2009.

The cathedral termite, found in parts of Australia, is capable of creating mounds for the colony well over 10 feet high. Individual cathedral termites are just standard-looking bugs: head, thorax, abdomen, legs, and so on, with a tiny little primitive brain. But when combined with others of its species, the cathedral termite is capable of constructing a huge, complex hive to house the colony. Unlike human building projects, however, there is no foreman, no plan, and it’s unlikely that any termite even knows what it is helping to create.

How is this possible?

The answer lies in the fact that sometimes, a system can provide more complexity than the sum of its parts, leading to what scientists call “emergent behavior.”

Emergent behavior, or the spontaneous creation of order, is present all around us. Insects are a good example because they are familiar to us and manage to undertake massive building projects which we can appreciate. Other parts of the animal kingdom also display autonomous order: fish organize themselves into schools that move in concert; birds and pack animals flock or herd in a similar manner. And nonliving examples also abound: natural magnets align themselves into a common North-South orientation and crystals can form from liquids, showing a spontaneous increase in order despite the lack of a more “intelligent” force.

There’s a major thermodynamic problem with all of this, of course: entropy, a measure of disorder, is supposed to continually increase. The universe tends toward chaos.

How, then, can spontaneous order arise, especially in a purely physical system?

Entropy still must increase, even when crystals form or birds flock, but the important distinction is in where the entropy increases or decreases. It turns out that nature will allow entropy to decrease in certain areas provided that it increases elsewhere to compensate. For instance, when a sugar solution begins forming crystals those crystals have a net lower energy level than the free-floating molecules of sugar in the solution. When the sugar enters the structure, it loses energy that is transferred to the water in the form of heat: the most disorganized form of energy.

Therefore, the crystalline portion of the solution has now decreased in entropy while the total system, including the water and the crystal, has had a net increase in entropy.

Why Gaussian Statistics Are Almost All Wrong for Organizational Strategies

Power Law phenomena exhibit Pareto distribution rather than Gaussian (normal). The fundamental difference lies in the premise about the correlation of events. In Gaussian distribution, events are assumed to be independent. Independent events generate normal distributions, which are at the heart of modern statistics. When events are interdependent, normality in distributions is not the norm. Instead, Pareto distribution dominates because extreme events occur more frequently than the normal distribution, with its Gaussian bell curve, would lead us to expect. Physical, biological, ecological, social, and industrial systems exhibit a remarkable variety of fractal structure (Kaye, 1993). Many scholars now believe that power laws are the best analytical framework for describing the origin and form of most natural objects. Given the ubiquity of these findings and the nature of scale-free theory, we think they are equally ubiquitous phenomena in organizations, though unknown and underappreciated (Andriani, 2003).

Extremes vs. Averages

Linear thinking is encrusted in our mentality. Scientific and mathematical models are based on the concepts of equilibrium and linearity. Linearity means two things: (1) proportionality between cause and effect, and (2) that the dynamic of a system can be reconstructed by summing up the effects of single causes acting on single components (Nicolis and Prigogine, 1989), which allows efficient causality to operate, equations to be solved, and predictive modeling. Economics, for instance, is almost theistic in its (scarcely verified) assumption that economic phenomena trend toward “general equilibrium” (Mirowski, 1989). However, this assumption allows linear equations and analytical simplicity.

By focusing on systems in equilibrium, statisticians implicitly accept that the number of possible states a system can reach is limited (and computable) and that the search time following the onset of instability (i.e., an exogenous shock) is short compared to the equilibrium time. For this to be true, the many elements making up a system need to be independent data points by assumption; otherwise we could have interdependence, possible mutual causalities, and the occurrence of possible extreme events.

If we take 100 companies approximately of the same size belonging to the same sector and assume independence, and plot a variable, say profit, we expect most events to pack around the mean, exhibiting the classic bell curve. The bell shaped distribution is by far the most studied statistical distribution; it is assumed to correctly characterize much of our discoveries about the natural and social worlds. In real life, however, the crux of the point is whether all events are independent. In real life, for example, these companies could: benchmark against each other, imitate those perceived as successful, exchange information, organize cartels, pursue mergers and acquisitions, compete for limited resources, etc. In a word, they are most likely interdependent, not independent. The statistical distribution that governs interconnected agents does not yield a bell curve, but a power law: the Pareto distribution.

Gaussian and Paretian distributions differ radically. The Gaussian distribution is reliably characterized by its stable mean and finite variance (Greene, 2002). A Paretian distribution doesn’t show a well-behaved mean and variance. A power law, therefore, has no “average” that can be assumed to represent the typical features of the distribution and no finite variance upon which to base confidence intervals (Moss, 2002). There are two major implications.

The dream of social science, of building robust frameworks that allow prediction, is shattered by the absence of statistical regularities in phenomena dominated by persistent interconnectivity. Absent stable mean and finite variance, the probabilistic assessment of individual outcomes becomes much more difficult. This point reflects the more pervasive and structural issue of nonlinearity and emergence in complex systems. Linearity assumes the divisibility of systems into modules whose dynamics can be studied independently of context.

Paretian tails decay more slowly than those of normal distributions. These fat tails affect systems’ behaviors in significant ways. For instance, Buchanan (2004) reports that a 10% drop in the financial market in a single day should happen once every 500 years according to the normal distribution. Mandelbrot (Mandelbrot and Hudson, 2004) shows that, instead, such crashes happen roughly once every five years. Extreme events, that in a Gaussian world could be safely ignored, are not only more common than expected but also of vastly larger magnitude and consequence.

The “no time” excuse people use to skip writing tests also serves to skip testing hypotheses. Most don’t even realize they should be experimenting. I’m talking about building proofs of concept, pieces of what you want to develop that will potentially be thrown away.

The “thrown away” part is what makes programmers and managers bristle. “But that’s wasted time, work down the drain! Unacceptable!” That thinking leads people to believe every line of code written must necessarily end up in the application.

It’s the old mistake of assuming we have to get everything right on the first try, the culture that treats trial and error as wrongdoing. My point is the opposite: the mistake is assuming we’ll always nail it on the first try. In most cases, we’ll get it wrong every single first time.

Cargo Cult

After World War II, native tribes on South Pacific islands built replicas of airplanes and military landing strips, hoping to summon back the “god planes” that had brought them so much wonderful cargo during the war. Most programmers do exactly that: as if in a ritual, they drop programming structures into the code without really understanding why, only because they “should.” They’re the ones who force Design Patterns where none are needed, who plaster comments over code that already explains itself and, more recently, who practically kneel before “Dependency Injection” without understanding the reason. (source: Wikipedia)

Following fashion gets an unfair bad rap; what ruins people is applying things, new or old, without understanding the reason. The main job of education should be teaching people to reason, but in a culture of rote memorization people accept everything they memorize without knowing why they memorized it: some “authority” said it was true and therefore it must be applied.

Remember the discussions about why certifications are harmful? This is one of the reasons. For those who already reason, it makes no difference at all. For the large mass who don’t, the end result is pure cargo culting.

Most people read tutorials, a few books, attend some workshops, and already consider themselves ready for the task. In practice they run the good old mental “copy and paste” and dump into the application every piece of code they ever learned. I’ve seen people write something like this:

if ( a == b ) {  
 return true;  
} else {  
 return false;  
}  

Nothing wrong with that, but it’s amazing how many people are surprised when I tell them this one line does the same thing:

return (a == b);  

The first snippet, on its own, is harmless. The subtlety is that the programmer doesn’t know why he wrote it; he only knows he “has to.” Want a worse case? I’ve seen, in several languages, programmers do the following:

$dbname="meu_banco";
$chandle = mysql_connect("localhost", 'root', 'root') or die("Falhou");  
$query1 = "select * from tabela";  
$result = mysql_db_query($dbname, $query1) or die ("Falhou");  
while ($row = mysql_fetch_row($result)) {  
 $field = mysql_fetch_field($result, 1);  
 if ( $field == 'foo' ) {  
 echo "encontrei!";  
 break;  
 }  
}

That’s a textbook What the F*ck!?

Again, code that “runs.” In some languages it compiles with no complaint at all. Anyone who can’t spot the deadly problem in this code needs to go back to their freshman year.

Blindness

“Repeat a lie long enough and soon it becomes the truth.”

There are plenty of bad books, bad tutorials, bad teachers, and a whole rabble of bad actors spreading bad practices. Even so, the greatest blame lies with those who let themselves be fooled. Whoever swallows everything they hear, without a shred of skepticism, is guilty of their own ignorance.

That’s why newspapers still print horoscopes instead of a science column: there are far more readers interested in being fooled than in learning boring things, like reality.

In technology it’s the same:

• Windows must be better because it’s the market leader (ignoring that Apple, even with a “measly 8%” of the market, ranks among the most profitable companies in the industry, and that Firefox already has a huge share of users in Brazil)

• Java must be better because it’s the market leader, or PHP must be better because big sites use it (no need to even discuss it)

• Threads are the best way to write concurrent code (ignoring, out of laziness, the advantages of functional programming)

• The best way to manage data is a relational database (ignoring, again out of laziness, the whole range of non-relational databases gaining ground)

• Static languages are better because you can compile them (ignoring, here out of pure stupidity, the enormous productivity advantages of dynamic languages)

• Object-oriented programming is the pinnacle of programming technique (ignoring that it’s just one of dozens of paradigms, functional programming among them)

• Rails doesn’t scale (seriously? anyone still repeating that proves they’re an amateur)

Everyone carries hundreds of preconceived ideas like these. Things heard from someone or read somewhere, usually from a source of dubious reputation, that the person now repeats with no argument to back the belief. Or rather: they think the dubious source they got it from is basis enough to keep repeating it.

In programming or in any other field: if you hold any belief you can’t argue for and defend, research it better. If it doesn’t survive counterarguments, throw it out, because it’s useless to you. Everyone fancies themselves “open-minded”; I disagree, most people are quite closed. Doubt me? Rethink everything you believe and argue against yourself. Done properly, you’ll notice most of what you believe has no foundation at all.

Scientific Method

A careful person practices the basics of the scientific method every day. Our daily life is a sequence of decisions, some small, some huge, and decisions made on preconceived ideas are where mistakes live.

This doesn’t rule out intuition. Intuition is a fast conclusion built on experience. If your daily experiences have been methodically rational for a long time, your intuition tends to be solid. If they were built on superstition, lucky charms, pseudoscience, cargo culting, and preconceived ideas, sorry to say: your intuition is garbage.

Read this Wikipedia definition. It’s long, thorough, and asks for real reflection. What interests me most is summarized here.

Like any good process, this one is also iterative: it expects you to loop back through steps to refine what you know.

The steps are simple, and they can run fast or slow and detailed. What matters is this: faced with a question, run these steps at least mentally. It’s the bare minimum for an educated decision.

• Ask the question. This phase matters: the wrong question leads to the wrong answer. People have been wasting time chasing irrelevant answers since forever because the question was wrong. Treat the question like a User Story in an agile backlog: check whether it’s a priority, check whether it’s needed. Don’t spend time answering irrelevant questions.

• Do research. Before anything else, research the subject. Don’t burn days on it; sometimes a few minutes on Google is enough. For me, the value of this step is the “stop, think, continue.” Most people rush too much; this is the moment to pause for a second and gain more knowledge before moving on.

• Build a hypothesis. When you ask a question, you usually already have one or more possible answers. When you framed the question, don’t cling to preconceived ideas. Consider that what most people call “truth” is, in reality, a set of hypotheses. A hypothesis is merely a suggested explanation.

• Test with an experiment. This is the most important step. Experiments must be repeatable and have very well-defined criteria: if two people run the same experiment, for the hypothesis to hold up the results have to match. Note that I said “hold up,” not “be true.” Truth is too strong a word; I rarely consider myself near any truth. Most of the time I just accept that my hypothesis hasn’t been falsified yet. The crucial part, in programming: build proofs of concept, pieces of code written only to test the hypothesis, which can simply be thrown away afterward. Do it outside your project’s codebase, in a separate environment. Don’t mix the two.

• Draw conclusions. From everything you did above, you’ll either confirm or demolish your hypothesis. Accept from the start that you may well prove your own hypothesis false. In that case, go back to step two, frame a new hypothesis, and try again. That’s what open-minded means: proving yourself wrong and immediately setting out to find another answer.

• Communicate your results. The application I’m describing here is informal, the minimum for you to have some grounding, probably not all of it, in what you’re doing. If the subject is more complex, with more time invested and more detail, it may interest more people. Share your results, at least with your colleagues. If you invested that much time, the answer is probably important, and then it’s worth giving others the chance to try to demolish it. It hardly matters how many people reach the same conclusion; headcount means nothing. What matters far more is whether someone manages to demolish yours. In that case, throw the hypothesis out and start over.

People

Most people can be described like this:

• They hold preconceived ideas. Heard from other people who, they believe, have credibility. Just because someone has a credential, is a celebrity, speaks well, or dresses well doesn’t mean they know the whole truth. Quite the opposite: that person may be stuffed with preconceived ideas. Do listen to what they have to say, but file it all in a mental box labeled “to verify” and move on.

• They hate being proven wrong. Nobody likes admitting they’re wrong: first it kills the ego, then it destroys self-esteem. So the ignorant person clings to their private lie to the bitter end. These are people with weak foundations. Build your foundation on half-truths and, when one falls, the whole thing falls. That’s the biggest risk.

• They hate wasting time. And “wasting time” here is entirely relative. Most people practice lousy economics: testing and experimenting, in other words “not doing anything immediately,” gets treated as wasted time. I call it “saving future time”: a little more care now so I don’t have to scramble tomorrow. It’s a matter of balance. It makes no sense to prepare for 15 days on a 20-day project. But do the minimum: stop for a second, think, and if in doubt, experiment before you proceed.

• They hate new things. There’s another wrong notion of “cost” at play. Many people think that because they invested time studying a subject, that investment can’t be lost and they must stick with it. It’s what I once wrote about in The Sunk Cost Fallacy. If the loss already exists, the dumbest move is to insist on it. Far smarter to write the cost off, change direction, and move on.

• They only do what they’re told. If nobody orders tests, nobody tests. If nobody orders a proof of concept, nobody builds one. If nobody tells them to care about security, the code ships full of holes. If nobody tells them to automate tests, nobody automates. The number of Lemmings on projects is staggering. First, because everything that should be mandatory goes undone; second, because if the boss orders them to shoot their own foot, they shoot. Once again, a failure to reason.

• They do the unnecessary. Sounds like a paradox, but it complements the previous item. Because of fixed preconceived ideas, people waste time doing totally unnecessary things because someone they consider “credible” said it should be done. It’s a behavior I still haven’t figured out, I see it all the time, and it contradicts everything I said above: someone picks an idiotic technology for some inexplicable reason, burns ages away from the project’s real priorities, builds no proof of concept at all, and starts coding on faith that it’ll work. In the end, the project is late and full of code headed for the trash because it’s good for nothing.

I think all of this comes down to people’s lack of practice at reasoning. It’s irrational behavior, needlessly complicated, full of basic mistakes, that nobody perceives as wrong.

I remember a documentary by the great Richard Dawkins in which he tells the story of a scientist who spent years studying a theory, I don’t recall which. A young scientist proved him wrong. The older man looked at the kid and said something like “thank you very much.” That’s rational behavior. Proving something true is very hard; proving it false is far simpler. If someone proves you wrong, say thanks: that person just saved you from wasting time down the road, and that’s worth gold.

And whenever you join a new project, don’t assume you know what needs to be done, or even that you “need” to know. Assume you don’t know. Build hypotheses, discuss, experiment, gain real confidence, and only then do what needs to be done. There’s nothing wrong with that. The problem is assuming you know what you’re going to do and, after wasting everyone’s time, finally making it obvious you were wrong all along.

Don’t waste other people’s time!

References:

• Scientific method
• Steps of the Scientific Method
• Deductive Reasoning
• Cargo Cult Programming
• Cargo Cult Science
• Fallacy
• Due Diligence
• Prejudice
• Hypothesis

I repeat over and over that I don’t know what the truth is. I answer “I don’t know” every time someone asks me a question expecting an absolute answer. “Will Rails take off?” Forget that kind of question: predicting the future is hard. If someone hands you a prediction about the future, ignore it. Odds are the person has no idea what they’re talking about.

Behind that kind of question sits the behavior I described above: the person thinks they must always be right and hates being proven wrong. Worst case, the path you walked to prove yourself wrong gave you more knowledge and experience. That’s worth more than finding a truth.

I’ve written about Reason in a few other posts:

• Be Arrogant
• Enemies of Reason
• 100% pure Object-Oriented: The Fallacy

Conclusions

My goal in this post is to reinforce the importance of experimentation. I don’t know what the truth is; anyone can post a comment here with dozens of counterarguments. Whoever does will be missing the point, and their “arguments” will be citations of preconceived ideas. Picking at details in the text to point out errors is beside the point. What matters is the professional understanding that age, years on the road, certifications, and waving credentials solve nothing. In the end we’re all amateurs and, as such, we need to go back to zero and review our hypotheses.

A practical example. When I spoke with John Straw (YellowPages.com) in San Francisco, he described something not every team does: extensive due diligence. They spent 22 months building the original YellowPages.com in Java. They decided to stop for 4 months to build prototypes, proofs of concept, and test hypotheses: which framework, Rails, Django, or Seam? Which architecture, SOA, EJBs? How long would the team take to get comfortable with the new technology? Only after they gained confidence in what they were doing did they start implementing for real. And it took just 4 more months to finish. See it? The first 4 months weren’t wasted time: they were insurance, the thing that kept the following 4 months from turning into 20.

As I said last year: “Be arrogant, for real!” Be arrogant with yourself to the point of genuinely questioning yourself and winning. You can fool others; fooling yourself buys you nothing. There is no better inquisitor for you than yourself. Wrong? Excellent: one less wrong path. Find another and start over.

Repeating: it wasn’t preconceived ideas that got us to the Moon.

Nassim Taleb and Benoît Mandelbrot must be watching this crisis, tragically, through very different eyes from ours. Some people are probably tired of hearing it, but I’ll repeat my recommendation: read both.

Take this excerpt from The Black Swan, published in 2007, by Nassim Taleb:

Globalization creates interlocking fragility, while reducing volatility and giving the appearance of stability. In other words it creates devastating Black Swans. We have never lived before under the threat of a global collapse. Financial institutions have been merging into a smaller number of very large banks. Almost all banks are now interrelated. So the financial ecology is swelling into gigantic, incestuous, bureaucratic banks — when one falls, they all fall. The increased concentration among banks seems to have the effect of making financial crisis less likely, but when they happen they are more global in scale and hit us very hard. We have moved from a diversified ecology of small banks, with varied lending policies, to a more homogeneous framework of firms that all resemble one another. True, we now have fewer failures, but when they occur … I shiver at the thought.

The government-sponsored institution, Fannie Mae, when I look at its risks, seems to be sitting on a barrel of dynamite, vulnerable to the slightest hiccup. But not to worry: their large staff of scientists deemed these events “unlikely.”

No, Nassim is not a prophet, a fortune teller, or a guru. He’d actually hate being called a guru. He just made an observation based on what all of us ignore: the human machine is lousy at dealing with abstractions, like Randomness.

Don’t get me wrong: this is no argument against globalization, nor any pseudo-socialist nonsense. The important part is the second paragraph of the quote.

It’s Happened Before

Long-Term Capital Management was the darling of the financial world: an American hedge fund formed in 1994 that had among its founders not one, but two Nobel laureates in Economics, Robert C. Merton and Myron Scholes. They believed they had the mathematics (cough Gaussian cough) to predict any kind of event.

In 1998, LTCM lost USD 4.6 billion. The Russian crisis destroyed their Gaussian theories, which completely ignored and underestimated Black Swans.

And you think people learned the lesson? Unfortunately we’re more stubborn than that: Scholes and Merton’s theories are still taught in economics programs to this day.

This is no exclusive trait of the economic world. Plenty of other fields accept and employ absurd theories, with no foundation and no results, and still treat them as the next great revolution. Ordinary people let themselves be fooled by names and credentials. If credentials were worth anything, a Nobel destroyed in 1998 should mean something.

I often say this: companies and charlatans who sell methodologies (of any kind: financial, human resources, management) have the easiest job in the world. All they need is to be good salespeople.

If the client succeeds after implementing the methodology: “See? You succeeded because you implemented our revolutionary methodology.” And that client becomes a “success story” in the portfolio.

If the client fails: “Of course it failed. You didn’t implement it exactly as we said, your teams lacked commitment. There’s nothing wrong with our methodology; the problem is you.” That client will never show up in the portfolio, since nobody likes to publicize failure.

Fooling people is very easy. And don’t look sideways: you’re implementing one of these methodologies today. I know it.

Falsifiability

People have the nasty habit of asking the wrong question: “how do we know if a theory is true?” That’s how they get the wrong answers, which lead to even worse decisions.

As I’ve said several times in previous articles, we are built to be fooled. Worse: we consciously refuse to exercise our skeptical muscles.

Every time we see a “success story,” we automatically accept the theory as “true.” Or, in an even more twisted version: “I’ve never heard of a case where this methodology failed, so it must be valid.” That’s what most “managers” and “executives” conclude. How many times do we need to repeat this?

Absence of evidence is not evidence of absence.

Today we use a lot of statistical data to make decisions. Statistics is very useful when used the right way. Used the wrong way, it’s an enormous disaster.

The reasoning is always the same: “for the last 3 years we’ve been growing 2% every month, so we can conclude with certainty that we’ll keep growing 2% in the coming months.” Everyone has done this. Until a Black Swan shows up and the excuse changes: “I don’t know what happened, it was an accident, because according to the data this shouldn’t have happened.”

Let’s be crystal clear: a rare event is called rare precisely because it doesn’t happen all the time. And it’s exactly this kind of rare, random, unpredictable event that usually delivers the billion-dollar losses or the billion-dollar gains. It depends on whether you’re the Gaussian type or the Paretian type.

But back to the original problem: is past statistical data enough to guarantee a theory’s validity?

Obviously NOT.

The most that past data can do is guarantee a theory’s falsifiability. As Nassim explains in Fooled by Randomness: with past data we can prove a theory is invalid, but we can never prove it is valid.

Today we know Newton’s classical physics is invalid, because Einstein’s Theory of Relativity overturned it. But we know in which contexts classical theory can still be used and when we need relativity. Good scientific theories are the ones that provide criteria to judge their falsifiability, never their validity.

Astrology and other pseudo-sciences are dogmatic. Anything dogmatic should be automatically discredited, since it blocks any attempt to check its falsifiability. That’s the signature of charlatans.

The argument I described in the previous section is exactly what charlatans do: if it worked, credit the theory; if it didn’t, you applied it wrong. The prediction failed despite Mars being aligned with Saturn because Mars was “a tiny bit” off position. Otherwise it would have worked.

Nassim gave another great example: “I have just completed a thorough statistical examination of the life of President Bush. For 55 years, close to 16,000 observations, he did not die once. I can hence pronounce him as immortal, with a high degree of statistical significance.”

Fallacies

Lately I’ve given several tips on things to avoid. If the subject is management or control, especially of people, and the theory doesn’t take Pareto into account, throw it out.

If a theory leaves no room to evaluate its falsifiability, it’s charlatanism. Throw it out.

Self-help books work like that: they offer flowery theories, covered in honey, attractively packaged. But unlike scientific theories, they never let anyone try to prove they don’t work. They only assert that they work, cite “success stories,” and hide every failed attempt.

Since the beginning of last month I’ve been traveling every week to give talks, and I always stop by the airport or bus station bookstore. Paying closer attention, the featured books are almost all of this kind: cheap charlatanism.

The “beautiful” theory only exists today because it hasn’t met a Black Swan on its path yet. “Ah, it’ll never happen, because it never happened before.” And that’s exactly why it may have even better odds of happening: because it never has!

Watch out for the following fallacies:

• Induction Fallacy: the fallacy where induction goes wrong. Induction means trying to find general principles from known facts.

• Narrative Fallacy: the creation of a post-hoc story so that the event seems to have had an identifiable cause.

• Regressive Statistical Fallacy: believing that the probability of future events is predictable by examining occurrences of past events.

• Ludic Fallacy: believing that the structured randomness found in games resembles the unstructured randomness found in life. It’s the problem of confusing the map (model) with the territory (reality).

The last one takes into account:

• it’s impossible to have all the information
• very small variations in the data can lead to enormous impact (the Butterfly Effect, and yes, it happens all the time)
• theories and models based on empirical data are flawed, since events that haven’t happened yet can’t be taken into account

It’s the example Taleb explains: you’re drawing colored balls from a covered box, without seeing inside. You pull 5 white balls and 5 red balls and, from this empirical data, conclude that "always 1 red ball will come out for every 2 balls drawn." Little do you know there’s a hole under the table with a boy hiding in it. Hearing your claim, he starts handing you more white balls than red ones. That’s reality.

We exercise our skepticism far too little. Nobody needs to become paranoid, but we could all evaluate things a little better than the mediocre way we do today.

Emotions

As Malcolm Gladwell says in Blink, we do make decisions in the blink of an eye. Of course, the decision will be as good as our experience, knowledge, and skills.

Taleb describes the case of a person who needed brain surgery because of a tumor, and the doctors had to remove the part of the brain responsible for our emotions. Everything else remained intact.

“Excellent!” someone might think. Here is a 100% rational person, who won’t let emotions get in the way of reason, and who should be making smart decisions all the time.

Surprise: this person became completely incapable of making any decision. They could barely decide to get out of bed. The studies show that our decisions are driven far more by the emotional part than by the rational one, contrary to what we imagine.

Anyone who has worked with artificial intelligence concludes that we humans need an approximation mechanism, because it’s simply impossible to take every variable into account. Evaluating everything we know would take so long that predators would have wiped us out millennia ago.

Deciding in the blink of an eye, or worse, trying to be “rational,” has advantages and disadvantages. Open-minded, highly studious people, experienced in many fields, will probably make many right decisions very fast (some by luck, some wrong). Closed-minded, mediocre people will make many wrong ones.

An example of a wrong decision: trusting pseudo-science salesmen.

Karl Popper used to say one should not take science too seriously. It makes sense: science allows itself to be wrong and refines itself over time. If you take everything too seriously, you risk betting on theories that haven’t been truly put to the test, and your next decision may be the very Black Swan that falsifies them.

Conclusion

Be very, very careful with experts. Without denigrating every kind of expert, because many are genuinely good: experts in abstract, intangible things like “methodologies” and “economics” should always be viewed with suspicion. A good credential doesn’t make a theory better or worse. It’s simply irrelevant.

Don’t let your decisions be biased by non-scientific theories: pseudo-science, superstition, astrology, homeopathy.

Again, read Taleb: what he says is obvious, but for some reason we all ignore it. Few people truly understand randomness. Stop frantically watching Bloomberg and refreshing your browser every 5 seconds to check financial indices. None of that will help you: you already ignored the Black Swan, you already lost.

Just a hint about Agile philosophy: I’ve always found it smart that Agile methodologies insist so much on short Sprints and iterations. They know predicting the long-term future is impossible, so they prioritize what really matters and plan only the short term, what’s effectively possible. Agile methodologies look tailor-made to defend against the Black Swans that still haunt traditional software development teams, as I explained in my previous article.

Everyone only ever saw white swans in the past and infers that Black Swans don’t exist. That’s where the danger lies!

Today I read two articles discussing exactly that. Both are right, and if you stop to think for 30 seconds, it was obvious I should never have used those numbers. Me, of all people, who has spoken more than once about the dangers of misused statistics! Let’s understand why.

So, how is the TIOBE index formed?

It’s a percentage, with no adjustment whatsoever, of the number of hits on 5 search engines for the terms +"[language] programming". Let’s look at examples with absolute numbers from Google alone:

• Java: 3.4 million hits
• C: 2.2 million hits
• C++: 1.7 million hits
• PHP: 1.5 million hits
• Perl: 0.9 million hits
• Python: 584 thousand hits
• C#: 558 thousand hits
• Ruby: 363 thousand hits
• Smalltalk: 28.4 thousand hits
• Groovy: 15.7 thousand hits

Now let’s see Yahoo!:

• C: 12.3 million hits
• Java: 9.6 million hits
• C++: 4.9 million hits
• PHP: 4.9 million hits
• Perl: 2.8 million hits
• Python: 2.2 million hits
• C#: 1.9 million hits
• Ruby: 1.6 million hits
• Smalltalk: 103 thousand hits
• Groovy: 31.3 thousand hits

There are also exception lists for grouping or splitting terms. C#, for example, is also searched as “CSharp”, “C-Sharp”, “C# 3.0”, and the results of all of them are summed as “C#”. In the example above I simply searched for “C# programming”.

After that, they take each language’s total results and divide by the total of the top 50. In my example, with the 10 in the list, C# comes out at 558 thousand / 11,249,100 = 4.9%. The same is done for Yahoo and the other search engines. The final percentages are summed and divided by 5. Here’s what you get in my smaller example, using none of the exceptions and only the 2 major search engines:

• Java: 27%
• C: 25%
• C++: 13.6%
• PHP: 12.7%
• Perl: 7.47%
• Python: 5.3%
• C#: 4.87%
• Ruby: 3.6%
• Smalltalk: 0.3%
• Groovy: 0.1%

There you go: we just saw the easy way to manufacture indices. This methodology is absolutely broken, and you don’t need to be a statistics expert to say so. It can’t even be compared to a methodology considered “serious”, like an IBOPE poll, which is controversial anyway.

The Problems

Some of the problems that I, a statistics layman, can point out:

• Blind trust in third parties: the search engines are presumed to do a good job.

• Duplicates count. The same article that gains notoriety gets re-published on hundreds of blogs (many of us bloggers are notorious for being mere echoes of primary sources). What should count as 1 suddenly becomes 100. Google App Engine, just launched, must be inflating Python’s index, for example.

• All 5 search engines weigh the same, because the index is just the simple average across all of them. There is no adjustment factor. And Yahoo!, in my experience, does a terrible job at search: “C programming” returned 2.2 million hits on Google and 12.3 million on Yahoo!. I doubt Yahoo’s database is 10 times larger; most likely it’s 10 times worse at filtering out noise. I wouldn’t be surprised if it counted “Objective C programming” as a hit for “C programming”, something at the level of the basic select * from table where text like "C programming".

• The hit count depends on the exact search string. “Ruby programming” brought a mere 363 thousand hits on Google. But “Ruby Rails” brings 8.1 million hits (!). I browsed all the way to page 63 and every link was genuinely about Ruby and Rails (no gemstones, no train tracks, so very little noise). With the same string, Yahoo returned 3.9 million, well above the previous 1.6 million. It becomes clear how hard it is to arrive at a reliable number for each platform.

The complaint from both authors of the posts I linked is this: even with the results we saw, why did Python become last year’s language of the year? Why did Ruby become the language of the year before that? By this method it’s clear that Java, C, C++, Perl, and PHP are an order of magnitude above.

My assessment: the TIOBE Index assumes too much and lets too many errors slip through. In a measurement situation like this, the margin of error is larger than the number being measured. Drawing any conclusion from it is practically impossible.

Want an equally ridiculous methodology? Let’s “assume” that a person who receives a political candidate’s flyer and doesn’t throw it on the street likes the candidate. We could elect a president by counting the flyers thrown on the street: whoever has the fewest flyers on the ground is the most liked and, therefore, the winner.

I’ve talked about this in another article: We Are Mathematically Illiterate. Unfortunately, unlike first-year algebra, statistics is still a very poorly understood field. I studied statistics at the Institute of Mathematics and Statistics at USP and I still know very little about it. People are constantly forced to “swallow” the numbers TV shows them, without knowing how to judge, at minimum, whether the criteria are adequate.

My recommendation: when in doubt, ignore the indices. I’m no Dijkstra, but I’d say “Statistics Considered Harmful”. As long as people don’t understand what statistics are, showing them indices is just trying to lie with more substance. In the article I linked above, Tim quotes Mark Twain:

“There are three kinds of lies: lies, damned lies, and statistics.”

For my part, I don’t intend to mention TIOBE ever again in any article or presentation as an argument that Ruby is growing. In practice, I believe there is no reliable index today to evaluate that.

Especially if, even if only once in a while, you try to explain purely random events as “mysterious coincidences” that “must” have some mystical or supernatural explanation.

Even among us programmers, many skipped the subject normally taught in the first year of Computer Science. One practical result of this kind of ignorance: Birthday Attacks, a type of cryptographic attack named that way because it exploits the mathematics behind the Birthday Paradox, which is explained in the article I translate below.

Given a function f (example: authentication), the goal of the attack is to find two inputs x1 and x2 (example: passwords) such that f(x1) = f(x2). Such a pair x1 and x2 is called a collision. And we always size up this problem the wrong way.

An interesting read is The Psychology of the Psychic, by David Marks and Dr. Richard Kammann, a skeptical analysis of paranormal claims. One non-obvious definition is what Marks called “Koestler’s Fallacy”: people assume that two strange hits in random events cannot happen by chance alone (cough Evolution cough).

This article by Bruce Martin, from the excellent Skeptical Inquirer magazine, demonstrates how two “coincident” events in a universe of random alternatives are more common than we think. It shows how our intuition, or common sense, around this type of analysis is totally flawed, and how we are completely ignorant when assigning probabilities to events.

“This project has a 90% chance of succeeding.” “This soccer team has a 60% chance of winning the championship.” “I have a 45% chance of passing the entrance exam.” All of this is nothing more than a guess, an arbitrary number with no rational grounding in the events it tries to predict.

When you try to predict something and miss, you say “how strange, I’m usually right” and everyone ignores it and forgets. If you happen to get a prediction right, you say “aha, I knew I was right, I always get it right!” and everyone is surprised and remembers only that one. Our minds work this way: we don’t store everything, we store only what suits us. And this influences our (bad) future decisions.

Truth #1: nobody can “predict” the future with precision. Truth #2: random events happen more frequently than we imagine. Truth #3: among many completely random events we will always find patterns, and we will memorize those patterns and try, futilely, to give them “meaning.” But these “coincidences” are just that: entirely calculable random events.

Note: I highly recommend reading the excellent Skeptical Inquirer magazine. Likewise I recommend staying informed about CSICOP, the “Committee for Skeptical Inquiry”, an organization that encourages critical investigation of pseudoscientific and paranormal claims from a responsible and scientific standpoint, and disseminates factual information and results of this kind of inquiry to the scientific community and the public (more about CSI).

Every day someone somewhere in the world is actively thinking about how to deceive us. Every day we deceive ourselves without noticing. Every day this kind of lie leads us to wrong and harmful decisions. That’s why we need to police ourselves all the time to be “Skeptical Inquirers.”

Not knowing the answer to some question today doesn’t mean we won’t unravel it tomorrow. Hasty conclusions are the greatest enemy of progress, because we stagnate investing in lies. I hope this article serves as an alarm that we deceive ourselves more often than we’d like.

Think you can “predict” the results of a sports championship? Think you can predict stock market results? Think you can predict the results of successive spins of a roulette wheel in a casino? Think again. I’ve had this article saved for a long time and wanted to translate it, but only now decided to put it here. This text is long and maybe tiring for most people, but I recommend reading to the end. On to the article:

Coincidences: Remarkable or Random?

Most improbable coincidences likely result from play of random events. The very nature of randomness assures that combing random data will yield some pattern.

Bruce Martin

“You don’t believe in telepathy?” My friend, a sober professional, looked askance. “Do you?” I replied. “Of course. So many times I’ve been out for the evening and suddenly became worried about the kids. Upon calling home, I’ve learned one is sick, hurt himself, or having nightmares. How else can you explain it?”

Such episodes have happened to us all and it’s common to hear the words, “It couldn’t be just coincidence.” Today the explanation many people reach for involves mental telepathy or psychic stirrings. But should we leap so readily into the arms of a mystic realm? Could such events result from coincidence after all?

There are two features of coincidences not well known among the public. First, we tend to overlook the powerful reinforcement of coincidences, both waking and in dreams, in our memories. Non-coincidental events do not register in our memories with nearly the same intensity. Second, we fail to realize the extent to which highly improbable events occur daily to everyone. It is not possible to estimate all the probabilities of many paired events that occur in our daily lives. We often tend to assign coincidences a lesser probability than they deserve.

However, it is possible to calculate the probabilities of some seemingly improbable events with precision. These examples provide clues as to how our expectations fail to agree with reality.

Coincident Birthdates

In a random selection of twenty-three persons there is a 50 percent chance that at least two of them celebrate the same birthdate. Who has not been surprised at learning this for the first time? The calculation is straightforward. First find the probability that everyone in a group of people have different birthdates (X) and then subtract this fraction from one to obtain the probability of at least one common birthdate in the group (P), P = 1 - X. Probabilities range from 0 to 1, or may be expressed as 0 to 100%. For no coincident birthdates a second person has a choice of 364 days, a third person 363 days, and the nth person 366 - n days. So the probability for all different birthdates becomes:

For two people: X2 = (365*364)/365^2

For three people: X3 = (365*364*363)/365^3

For n people: Xn = (365*364* … *(366-n))/365^n  
 Xn = 365! / [365^n * (365-n)!]  

With its factorials the last equality is not especially useful unless one possesses the capability of handling very large numbers. It is instructive to use a spreadsheet or a loop in a computer language to calculate Xn from the first equality for successive values of n. When n = 23, one finds X = 0.493 and P = 0.507. A plot of the probability of at least one common birthdate, P, versus the number of people, n, appears as the right-hand curve of circles in Figure 1. The curve shows that the probability of at least two people sharing a common birthdate rises slowly, at first passing just less than 12% probability with ten people, rising through 50% probability at the open circle corresponding to twenty-three people, then flattening out and reaching 90% probability in a group of forty-one people. This means that on the average, out of ten random groups of forty-one persons, in nine of them at least two persons will celebrate identical birthdates. No mysterious forces are needed to explain this coincidence.

Figure 1: Probabilities of Coincident Birthdates: The right-hand curve of circles represents the probability that in a random group of people at least two celebrate the same birthdate. As indicated by the open circle just above the horizontal line at 0.50 probability, an even 50% chance is achieved at just 23 people. The left-hand curve represents the probability that in a random group of people at least two share a birthdate within one day of each other. The 50% chance for this three-day coincidence occurs with just 14 people.

Note that the probability of coincident birthdays for 23+23=46 people is not 100%, as some might suppose, but 95% as shown by the right-hand curve in Figure 1. Extension of the curve beyond the limit of Figure 1 reveals that fifty-seven people produce a 99% probability of coincident birthdays.

The same principle may be used to calculate the probability that at least two people in a random group possess birthdates within one day (same and two adjacent days, e.g., the first person’s is the 20th and the second’s between the 19th and 21st). This condition is less restrictive than the former, and 50% probability is passed with just fourteen people. The left-hand curve in Figure 1 shows a plot for the probabilities of within-one-day birthdates.

Delving a little deeper into some aspects of the probabilities of identical birthdates provides additional insight. Note that we said several times “at least two people” sharing a common birthdate. As the group size increases the chances for multiple coincidences also increase.

• The descending curve at the left of Figure 2 represents the probability of no coincidences (NC) of birthdates, identical to the Xn values calculated above.

• The first curve with a maximum plots the probability of only one pair (1P) sharing an identical birthdate. The maximum occurs at twenty-eight people with a probability of almost 0.39.

• As the group becomes larger the probability of other coincidences increases as well. The second curve with a maximum represents the probability of exactly two pairs (2P) sharing an identical birthdate. Its maximum occurs at thirty-nine people with a probability of 0.28.

• The last, rising curve in Figure 2 plots the total probabilities of all remaining coincidences (>2P), consisting of three pairs, triplets, etc. For all numbers of people, the probabilities of all four curves total 1.00.

Figure 2: Probabilities of Multiple Coincident Birthdates: The descending curve at the left represents the probability for no shared birthdates, no coincidences (NC). The first curve with a maximum plots the probability of only one pair (1P) with an identical birthdate. The second curve with a maximum represents the probability of exactly two pairs (2P) sharing identical birthdates (different date for each pair). The ascending curve at the right plots the probability of all other coincidences (>2P), three pairs, triplets, etc. For any number of people the probabilities of the four curves total 1.00.

Figure 2 shows that for twenty-three people the probabilities are 0.36 for one pair, 0.11 for two pairs, and 0.03 for the total of all other coincidences for a probability sum of 0.50. We have broken down the 0.50 probability for at least one coincidence discussed above for twenty-three people into component contributions. For twenty-three people the probability of no coincidences is also 0.50, as shown in the descending curve (NC) of Figure 2.

There is an almost triple intersection at thirty-eight people where the chance of 1 identical pair, 2 identical pairs, and the total of all other coincidences is 28-29%. For thirty-eight or more people the total of all other coincidences becomes greater than the exactly one and two pair possibilities, and passes through 50% chance at forty-five people. In a random group of more than forty-five people there is a better than even chance that there are more than two coincidental birthdates.

What this series of calculations boils down to is this: If coincident birthdates are so much more common than we would have guessed, isn’t it likely that many of those other striking coincidences in our lives are the outcome of probability as well? We should not multiply hypotheses: the principle of Occam’s Razor states that the simplest explanation is to be preferred.

Presidential Coincidences

Consider the birth and death dates of American presidents to see how this reasoning works in real cases. There have been forty-one presidential births, and Figure 1 indicates a 90% probability that at least two presidents should have been born on the same day. There is one such coincidence: James K. Polk and Warren G. Harding were both born on November 2. The result appears in Table 1.

With forty-one cases there is a 66% chance of a second coincidence, but none has yet occurred. [The result may be obtained by adding the probabilities for forty-one persons for the 2P (0.28) and >2P (0.38) curves of Figure 2.] Perhaps the next president’s birthdate will coincide with one of the previous forty-one. (The birthdates of neither Albert Gore nor Colin Powell do so.)

Of the thirty-six dead presidents Figure 1 indicates an 83% probability that at least two should have died on the same date. The results also appear in Table 1. Both Millard Fillmore and William Howard Taft died on March 8.

With 36 cases there is a 51% chance of a second coincidence.

In what seems an astounding coincidence, three early presidents died on July 4, as listed in Table 1. Both John Adams and Thomas Jefferson died in the same year, 1826, on the fiftieth anniversary of their signing the Declaration of Independence. Adams’s final words, that his long-time rival and correspondent Jefferson “still lives,” were mistaken, as Jefferson had died earlier that same day. James Monroe died on the same date five years later.

Presidential scholars suggest that the former early presidents made an effort to hang on till July 4. James Madison rejected stimulants that might have prolonged his life, and he died six days earlier on June 28 (in 1836). It seems evident that for the deaths of several presidents July 4 is not a random date. Only one president, Calvin Coolidge, was born on July 4.

An important point in all of the preceding is that no birthdate was specified in advance. Table 2 lists the crowd sizes for 50% probabilities. The first entry restates what we already know: a group of twenty-three suffices for at least two to possess the same unspecified birthdate.

If we specify a particular birthdate, such as today, a crowd of 253 people is required to have an even chance for even one person with that birthdate.

For at least two persons to possess a specified birthdate, the 50% probability is not reached until there is a mob of 613 people. This huge difference of twenty-three versus 613 for 50% probability of at least two persons with a common birthdate is due to the fact that the date is unspecified for the group and specified for the mob.

That some improbable event will occur is likely; that a particular one will occur is unlikely. If we look at our personal coincidences, we see that they were rarely predicted in advance.

(Akita’s note: what we mean is this: coincidences happen. Saying “unlikely” is not saying “impossible”: 0.000001% is still not zero. The difference is that it’s almost never possible to predict a particular event in advance, but almost never is not never. Hitting 1 prediction out of thousands is not impossible. Hitting all of them is highly unlikely. That’s the nature of coincidences: they happen, however unlikely they are, that’s a fact, and there’s no supernatural agent handling it, it’s just a random event that is mathematically describable. Predicting that a certain event will happen “and” being right is also absolutely random and there’s no mystical or intuitive power involved. It’s purely like rolling dice and, by chance, hitting a number. No: you are not a psychic. :-)

Abraham Lincoln and John Kennedy

It is always possible to comb random data to find some regularities. A well-known qualitative example is the comparison of coincidences in the lives of Abraham Lincoln and John Kennedy, two presidents with seven letters in their last names, and elected to office 100 years apart, 1860 and 1960. Both were assassinated on Friday in the presence of their wives, Lincoln in Ford’s theater and Kennedy in an automobile made by the Ford motor company. Both assassins went by three names: John Wilkes Booth and Lee Harvey Oswald, with fifteen letters in each complete name. Oswald shot Kennedy from a warehouse and fled to a theater, and Booth shot Lincoln in a theater and fled to a barn (a kind of warehouse). Both succeeding vice-presidents were southern Democrats and former senators named Johnson (Andrew and Lyndon), with thirteen letters in their names and born 100 years apart, 1808 and 1908.

But if we compare other relevant attributes we fail to find coincidences. Lincoln and Kennedy were born and died in different months, dates, and states, and neither date is 100 years apart. Their ages at death were different, as were the names of their wives. Of course, had any of these features corresponded for the two presidents, it would have been included in the list of “mysterious” coincidences. For any two people with reasonably eventful lives it is possible to find coincidences between them. Two people meeting at a party often find some striking coincidence between them, but what it is, birthdate, hometown, etc., is not predicted in advance.

(Akita’s note: coincidences are much more common than we think. We’re not as different from each other as people believe.)

Bridge Hands

In the card game bridge there are a possible 635,013,559,600 different thirteen-card hands. This number of hands could be realized if all the people in the world played bridge for a day. For an individual it would take several million years of continuous playing to be dealt each of these hands. Yet any given hand held by a player is equally probable, or rather, equally improbable, as its probability is 1/635,013,559,600 or a little better than one part in a million million. Any hand is just as improbable as thirteen spades. Bridge hands are an example of the daily occurrence of very improbable events, but of course, the hands are not specified in advance.

Consider a group of just 10 or more students in a classroom of a college that draws students from several states. During school session, numerous such classrooms exist each day. Yet the odds against predicting the exact make up of any classroom ten years in advance (all the students and teacher born by then) are truly astronomical. This is another example of the daily occurrence of highly improbable events.

Runs of Heads and Tails

What sequence of heads (H) and tails (T) might you expect in random tossing of a coin? Not all heads nor all tails, nor even the alternating sequence (HTHTHTHT), as this series is obviously regular and not random. In a random sequence we expect runs of both heads and tails. We can simulate progressions of coin tosses from a random sequence of numbers.

So far as is known, the decimal digits of the irrational number π, which multiplies the diameter of a circle to obtain the circumference, are random. This does not mean that every time π is calculated a different result is obtained, but rather that the value of any single digit is not predictable from preceding digits. An example of a pattern leading to predictability is the sequence of decimal digits in the fraction 1/7 = 0.142857142857142857…, where there is an obvious repeat every six digits.

The decimal digits of π have been calculated to hundreds of millions of digits by high-speed computers, but we list only the first 100 digits in four rows of 25 digits:

3.141592653589793238462643
38327950288419716939937510
58209749445923078164062862
089986280348253421170679

There are fifty-one even digits and forty-nine odd digits. There is almost an even distribution when the first 100 decimal digits are divided in another way: forty-nine digits from 0 to 4 and fifty-one digits from 5 to 9.

Since the decimal digits of π are random, we may simulate a random sequence of heads and tails in coin tossing by assigning even digits to heads and odd digits to tails. The sequence of heads and tails in 100 tosses with 25 tosses per line becomes (T = tails and H = heads):

THTTTHHTTTHTTTTHTHHHHHHT
THTHTTTHHHHHTTTTHTTTTTTT
THTHHHTTHTHHTTHTHTHTHHHHHH
HHHHTTHHHHHTHHHTTHHTTTHHTT

Combing the random sequence we find some regularities, such as the alternating sequence of eight tosses from 62-69 (in bold). The probability of an alternating sequence of 8 tosses is once in 2^7 = 128 tosses. There are some long runs of all heads and all tails. There are two runs of 5 heads, one run of 6 heads, one run of 8 tails, and a surprising run of 10 heads. The π decimal digits 69-78 are all even. A run of ten even digits should occur only once in 2^10 = 1,024 digits. Yet such a run occurs within the first eighty digits.

So what have we here? A proof that the decimal digits of π are not random? No, what we have instead is a demonstration of how it is always possible to comb random data and find regularities not specified in advance. Since ten even digits occur within the first 100 decimal digits of π, we might (mistakenly) think we are on to something, and that such a run might occur frequently. In fact a run of ten even digits does not occur again in the first 1,000 decimal digits of π. In the first 1,000 digits a single run of ten odd digits occurs from 411-420.

The point is that the very nature of randomness assures that combing random data will yield some pattern. But what that pattern is cannot be specified in advance. If someone finds a pattern combing random data, he or she may use it as a hypothesis for investigation of more data but should never make a general conclusion from it. In our example we discovered (but did not predict) ten even digits within the first 100 digits but not again in the next 900 digits. For confirmation of a trend, the target data must be stated in advance of data inspection. If an unexpected pattern does emerge during inspection after the data is obtained, the pattern can be used as a hypothesis for obtaining and inspecting an entirely new set of data.

The heads and tails sequence may be applied in other ways. Consider a football quarterback who completes 50% of his passes or a basketball player who makes 50% of his or her free throws. Assign heads (H) to a pass completion or made free throw and tails (T) to a miss, and then one expects long runs of completions and misses as shown in the HT sequence above. Most hot and cold streaks in sports are just the consequence of randomness. The “hot hand” is most often an illusion of significance that appears in data sets that are random.

We may utilize the random sequence of π decimal digits to find likely streaks for a .300 hitter in baseball. For example, assign the digits 0, 2, and 4 to hits and the other seven digits to outs. Then, out of the first 100 decimal digits there are 30 hits and 70 outs. If we divide the sequence of 100 digits into successive groups of four, a representative number of bats per game, we obtain the results for twenty-five games. Our .300 hitter then goes hitless in four games (three in succession for a “slump”), strokes one hit in thirteen games, two hits in seven games, three hits in one game, and has no game in which he gets four hits. Astonishingly, the batter gets at least one hit in the last thirteen games, considered enough to be a real “streak.” But this “streak” arises out of the random sequence of π decimal digits. A batter’s slump or hitting streak is likely just the result of randomness in play.

Clearly, unspecified improbable coincidences occur daily to everyone, and these coincidences are most likely the result of randomness. If the data set is large enough, coincidences are sure to appear, as demonstrated with the first 100 decimal digits of π. The chance of tossing five straight heads is only 3 percent, but for 100 tosses the chance becomes 96 percent. Though applied in a different context, Ramsey theory (Scientific American, July 1990) states that “Every large set of numbers, points, or objects necessarily contains a highly regular pattern.” It is not necessary to posit mysterious forces to explain coincidences.

Figure 3: Stock Market Simulation: Daily stock market action presented as price for 109 trading days generated from the random decimal digits of π. The representative “head and shoulders top” is shown to be consistent with random play of the market. Of course, the number of days is flexible, one decimal digit may represent any fraction or number of days. See Note for a description of how the price action was generated.

Random Prices in the Stock Market

Given the current fascination with the long bull market in stocks, we can generate an even more interesting result from the random decimal digits of π. Let us plot on the x-axis the number of the decimal digit and on the y-axis a price value that is generated from the decimal digits as described in the Figure 3 caption and Note so that there is an arbitrary and equal balance between the up and down directions for price.

For the first 108 decimal digits of π the entire plot is in positive territory. Starting at zero the plot works its way haltingly to increasingly positive values, attaining a plateau from the 48-71 decimal digits before it begins to work its way down, almost returning to zero on the 99th digit, and crossing into negative territory after the 108th decimal digit. To a stock market technician this plot represents a head and shoulders top in a plot of a stock price or stock market price versus time. It is all there in Figure 3, a top and shoulders on both sides of the top. Yet this plot was generated from the first 109 random decimal digits of π!

The maximum value of 65 on the y-axis is reached three times in the plateau region and is more than 7 times greater than the maximum single move of 9. Therefore, we may conclude that a head and shoulders top in stock or commodity prices may represent nothing more than random play in the markets. (Over the longer term there is a rising trend in stock market averages.)

A recent sweepstakes received in the mail offered a grand prize of $5,000,000. The fine print stated the chances of winning this prize as one in 200,000,000. Out of this large population some one person will win the sweepstakes. With such incredibly unfavorable odds each person must decide for him or herself whether it is worth the time and the first class postage to return the entry. The sure big winner appears to be the postal service, which garners more than ten times the grand prize amount in postage.

So, the next time you hear, “It couldn’t be just coincidence,” you will be fully justified in answering, “Why not?”

CSICOP Presidential Coincidences Contest

Back in 1992, the Skeptical Inquirer held a Spooky Presidential Coincidences Contest, in response to Ann Landers printing “for the zillionth time” a list of chilling parallels between John F. Kennedy and Abraham Lincoln. The task was for readers to come up with their own list of coincidences between other pairs of presidents. There were two contest winners, Arturo Magidin of Mexico City, and Chris Fishel, a student at the University of Virginia. Magidin came up with sixteen stunning coincidences between Kennedy and former Mexican President Alvaro Obregón, while Fishel managed to come up with lists of coincidences between no fewer than twenty-one different pairs of U.S. presidents.

A few examples from Magidin’s list: Both “Kennedy” and “Obregón” have seven letters each; each was assassinated; both their assassins had three names and died shortly after killing the president; Kennedy and Obregón were both married in years ending in 3, each had a son who died shortly after birth, and both came from large families and died in their forties.

Fishel came up with dozens of coincidences; here are a few between Thomas Jefferson and Andrew Jackson. Both men served two full terms; both their wives died before they became president; each had six-letter first names; both were in debt at the time of their deaths; each had a state capital named after him, and both their predecessors refused to attend their inaugurations. [For more information and the full lists, see SI Spring 1992, 16(3); and Winter 1993, 17(2).]

Acknowledgment

I am indebted to Professor Russell N. Grimes of the University of Virginia for discussions of expressions leading to Figure 2 and Table 2.

Note

The price action was generated in a positive direction when the preceding digit is odd (except for 5) and in a negative direction when the preceding digit is even, with the magnitude of the direction given by the value of the digit. Thus preceding odd digits 1 + 3 + 7 + 9 = 20 generate a positive direction and preceding even digits 0 + 2 + 4 + 6 + 8 = 20 a negative direction. The sum in the two directions is the same. For the left over digit 5 the direction is up or down depending upon whether the previous digit is odd or even, respectively. In the first 108 decimal digits there are eight 5s, half each generating positive and negative directions. Therefore we have a perfectly arbitrary and equal balance between the positive and negative directions.

References

• Epstein, Richard A. 1967. The Theory of Gambling and Statistical Logic. New York: Academic Press.

• Falk, Ruma. 1981. On Coincidences. Skeptical Inquirer 6 (2): Winter: 18-31.

• Graham, Ronald L., and Joel H. Spencer. 1990. Ramsey Theory. Scientific American 263 (1): 112-117 (July).

• Paulos, John Allen. 1988. Innumeracy. New York: Random House.

• Weaver, Warren. 1963. Lady Luck. Garden City, NY: Anchor Books.

About the Author

Bruce Martin is Professor Emeritus of Chemistry at the University of Virginia, Charlottesville, Virginia.

Akita’s closing notes: just to be clear: “hits” in random events, finding “patterns,” also known as “coincidences,” simply happen. Especially in a large universe of random events, or mutually exclusive ones.

The problem is when we decide to identify these patterns (easy) and try to give “meaning” to these randomly coincident sequences. Worse still: when we decide to attribute “cause and effect” meanings to these events.

Suppose someone took a certain substance “H” they saw in some folk remedy or something similar. A few days later, they were free of the ailment. Of course they’ll say: “thanks to substance H, I got better!” and everyone will believe it, even more so if several people report the same thing. But none of them actually investigated the case: they drew conclusions after the fact. That’s the wrong way.

The right way: if we formulated the hypothesis that such substance “H” works, we would then need to experimentally obtain more data sets with results that exceed the results that would easily be obtained through randomness alone, as we saw above. In the birthday case, you only need 23 people to find 2 people with the same birthday out of a universe of 365 different days in a year!!

People will only remember that, coincidentally, a thousand people “were cured” by substance “H.” But they ignore the other ten thousand, with the same ailment, treated with the same substance, who showed no improvement at all. We prefer to ignore the ten thousand and only look at the “magnificent” thousand positives. As you can see, numbers out of context are absolutely useless.

Carl Sagan needs no introduction. I’ve always found this passage inspiring. It’s good to give people the right motivation, and it’s one more reason I recommend Cosmos to every kid: a legacy that outlives generations. (More recommended videos at the end of the article.)

This excerpt from Pale Blue Dot was inspired by an image taken, at Sagan’s suggestion, by Voyager 1 on February 14, 1990. As the spacecraft was leaving our planetary neighborhood for the fringes of the solar system, engineers turned it around for one last look at our planet.

Voyager 1 was about 6.4 billion kilometers away, roughly 32 degrees above the ecliptic plane, when it captured this portrait of our world. Caught in the center of scattered light rays, an effect of shooting so close to the Sun, the Earth shows up as a tiny point of light: a crescent only 0.12 pixels in size.

by Carl Sagan
Co-founder of The Planetary Society
1994

Look again at that dot. That’s here. That’s home. That’s us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, every corrupt politician, every “superstar,” every “supreme leader,” every saint and sinner in the history of our species lived there—on a mote of dust suspended in a sunbeam.

The Earth is a very small stage in a vast cosmic arena. Think of the rivers of blood spilled by all those generals and emperors so that, in glory and triumph, they could become the momentary masters of a fraction of a dot. Think of the endless cruelties visited by the inhabitants of one corner of this pixel on the scarcely distinguishable inhabitants of some other corner, how frequent their misunderstandings, how eager they are to kill one another, how fervent their hatreds.

Our posturings, our imagined self-importance, the delusion that we have some privileged position in the Universe, are challenged by this point of pale light. Our planet is a lonely speck in the great enveloping cosmic dark. In our obscurity, in all this vastness, there is no hint that help will come from elsewhere to save us from ourselves.

The Earth is the only world known so far to harbor life. There is nowhere else, at least in the near future, to which our species could migrate. Visit, yes. Settle, not yet. Like it or not, for the moment the Earth is where we make our stand.

It has been said that astronomy is a humbling and character-building experience. There is perhaps no better demonstration of the folly of human conceits than this distant image of our tiny world. To me, it underscores our responsibility to deal more kindly with one another, and to preserve and cherish the pale blue dot, the only home we’ve ever known.

Carl Sagan, Pale Blue Dot, 1994

How to explain the Origins of the Universe …

And, let us never forget Sagan’s legacy …

In that sense, religions and superstitions do have their problems. That doesn’t mean religious people can’t be excellent programmers. They are! There’s a mistaken view that atheists like me, when we talk about religion, only want to slander, badmouth, ridicule. Slow down, we’re not that bad. Just as we don’t judge anyone by their color, we don’t judge anyone by their belief either.

The problem is when devaluing evidence in favor of superstition leads to real harm, like the setback in stem cell research. Or when someone drops a serious medical treatment for a glass of homeopathic water.

It’s interesting how many discussions in the tech world have a large irrational component: my language is better than yours, my process is better than yours, my Linux is better than your Mac, my Ruby is better than your Java. People assume there’s an absolute Right and Wrong: for one side to win, the other has to lose. If mine is obviously the Right one, the other guy’s is, obviously, the Wrong one. As Einstein would say (in another context, I know), everything is relative.

Readers of my older articles have noticed that I don’t root for any team. I’m a Ruby programmer, but I have no problem opening Visual Studio and writing some C#. I own several Macs, but I optimize a Windows or a Linux box just as comfortably. Anything that improves my knowledge is welcome. What’s the practical point of limiting yourself?

The motivation to write about this came from two videos from England’s Channel 4: “Enemies of Reason,” a two-episode documentary that could be watched in full on Google Video (episodes 1 and 2). The host is one of my favorite non-tech authors: Dr. Richard Dawkins. Biologist, fierce defender of evolution, and the man who coined the term meme, first introduced in the excellent The Selfish Gene.

He’s also the author of my recent favorite book: The God Delusion, released in Brazil by Companhia das Letras as Deus, um Delírio. That book spawned another Channel 4 documentary, which I also highly recommend: The Root of All Evil?, also available on Google Video (here and here). The second episode is titled “The Virus of Faith.”

Dawkins belongs to a much-criticized new generation of atheists known as the New Atheists. The same group includes authors and scholars like Sam Harris, author of “The End of Faith,” and philosopher Daniel Dennett, author of “Breaking the Spell: Religion as a Natural Phenomenon.”

The biggest problem is fanaticism. There are non-fanatical religious people (fortunately, the majority). But the fanatical minority is dangerous, and many of them sit in the halls of power, shaping everyone’s future. We already went through an era of religious alienation known as the Middle Ages. We’re lucky to be a post-Renaissance generation. Regressing out of sheer lack of reason makes no sense.

Politics and culture aside, what personally bothers me is the lack of self-worth. I’m not defending the arrogance of someone who thinks he knows everything without knowing anything. But the simple “Thank God” is an unconscious brake. If you achieved something, the credit belongs to you and your collaborators. Luck helps, but it’s just a statistical event. Nothing is impossible: some things are only probabilistically more unlikely.

If you didn’t achieve it, the responsibility is entirely yours too. I feel many people have a hard time owning their responsibilities, for the hits and the misses alike. The credit and the blame for your actions are exclusively yours. Live with it.

Another thing that bothers me: all the material I mentioned here will be heavily criticized. Maybe I’ll get a flood of angry comments, as usual, or maybe I’ll just be ignored. Either way, prejudice will make a lot of people simply refuse to look.

That’s another problem: the path of learning demands reason and logical reasoning built on solid evidence. It’s been almost a century and a half since Charles Darwin published his Theory of Evolution, and nobody in the sciences was arrogant enough to upgrade it to a Law of Evolution. The same goes for technology: nobody can claim “my technology is superior.” It’s all relative, and nobody here has the authority to create a Law proving such a claim. What we need to keep in mind, to build a consciously investigative environment, is: “if mine isn’t the best, what are the alternatives? what’s the evidence? what are the circumstances?”

Joseph Campbell would say societies need mythologies, heroes, beliefs. It’s a way of explaining what we don’t know and of communicating what counts as Right or Wrong in a given period, since those concepts shift across space and time. As we evolve, we kill one god at a time: Ra, Isis, Osiris, Apollo, Artemis, Poseidon, and many others. We explained where the Sun rises, where it sets, what’s in the sky, where we come from, where we’re going, and a ton of other natural events.

As Richard Dawkins said, we miss Carl Sagan. For me, he produced two of the most inspiring works there are: the “Cosmos” series and the book The Demon-Haunted World: Science as a Candle in the Dark. Every child should be encouraged to watch the whole series and learn to think for themselves, free of dogmas and superstitions.

As Dawkins shows in the documentary, astrology books and other superstition titles vastly outsell science books. And even among documentaries we get superstitious productions peddling fairy tales with science as the pretext, like the dreadful “What the Bleep Do We Know?”

In a competitive world, with plenty of problems to solve, like diseases, conflicts, global warming, and resource scarcity, we need the largest number of skeptical, non-superstitious brains of all time. Only then will we move forward instead of stagnating in the magic tricks of the past. And even setting the big questions aside, as individuals, we cannot and must not become enemies of reason.

Definition of Science: from the Latin “scientia,” or “knowledge.” According to Webster’s New Collegiate Dictionary: “knowledge attained through study or practice,” or “knowledge covering general truths of the operation of general laws, especially as obtained and tested through the scientific method and concerned with the physical world.”

Science refers to a system for acquiring knowledge. This system uses observation and experimentation to describe and explain natural phenomena. The term also refers to the organized body of knowledge people have gained using that system.

Science is an elegant system because the scientific method is simple and coherent (observation/research, hypothesis, prediction, experimentation, and conclusion), yet rigorous enough to be reliable, mainly because, by definition, it has self-correcting mechanisms. It is independent of personal opinion, belief, or point of view. The Law of Gravity is the same for everyone, regardless of race, culture, ideology, age, or social condition.

No hypothesis earns the status of “Law” without solid enough evidence for it. Besides, every theory (or hypothesis, defined as a suggested explanation) can be discarded if solid contrary evidence shows up.

Definition of superstition: (1) an irrational belief that an object, action, or circumstance not logically related to a course of events influences its outcome. (2) (a) a belief, practice, or rite irrationally maintained by ignorance of the laws of nature or by faith in magic or chance. (b) a fearful or abject state of mind resulting from such ignorance or irrationality. (c) idolatry.

Finally, the definition of dogma: (a) something held as an established opinion; especially: a definite authoritative tenet. (b) a code of such tenets (pedagogical dogma). (c) a point of view or tenet put forth as authoritative without adequate grounds. (d) it cannot be disputed or doubted.