'Why doesn't the immune system recognize tumor cells as abnormal and destroy them?'

This answer briefly explains

• Immunosurveillance, the theoretical basis for the immune system to recognize and react to tumor cells.
• How the history of spontaneous tumor remissions suggests immune system involvement in tumor elimination.
• How lifestyle factors better explain cancer rate increase compared to heritable factors. Focus then turns to how such lifestyle factors might influence immune system function and cancer risk.

Bigger issue is one that's as-yet poorly researched, namely, how the immune systems of those with and without cancer are different. Even the characteristics of a healthy immune system remain undefined till date. Key to understanding why some people develop clinically diagnosed tumors while others don't will likely depend on understanding how Microbiota - Wikipedia changes increase or minimize cancer risk through their effects on immune system function.

Immunosurveillance, the theoretical basis for the immune system to recognize and react to tumor cells

According to the immunosurveillance theory, the immune system indeed must recognize and destroy tumor cells, not as abnormal but as antigenically different. Immunosurveillance emerges as an inevitability when considered from the theoretical standpoint of how the immune system ought to function. Such a theoretical basis was first clearly outlined in the early days of immunology,

• In 1957 by Frank Macfarlane Burnet - Wikipedia (see below from 1, emphasis mine),

'It is by no means inconceivable that small accumulations of tumour cells may develop and because of their possession of new antigenic potentialities provoke an effective immunological reaction, with regression of the tumour and no clinical hint of its existence’

• In 1959 by Lewis Thomas - Wikipedia (2) who similarly argued protection from neoplastic disease was a major function of the immune system.

A more modern interpretation of the immunological consequence of tumors expressing new antigens would propose that

• T and B cells would be unlikely to have encountered and become tolerized to such antigens during their development in the thymus and bone marrow, respectively.
• Such untolerized new tumor antigens should draw the attention of at least some T and B cells expressing receptors that could specifically bind them.
• Such binding should trigger the clonal proliferation of such cells and lead to the development of strong and sustained immune responses which should eventually eliminate such tumors.

The history of spontaneous tumor remissions suggests immune system involvement in tumor elimination

Long before Burnet or Thomas provided a theoretical underpinning for the notion that ridding of tumors would or should be a major function of the immune system, history notes apocryphal stories of miraculous tumor regressions. One of the most famous is the patron saint of cancer patients himself, Peregrine Laziosi - Wikipedia, whose cancer of the tibia regressed completely and he lived to the age of 85 with no record of tumor recurrence (3) and even some indication to suggest that the regression coincided with some type of acute infection, extant records noting how shortly before the miraculous regression, the stench given off by Peregrine's severely infected tumor tissue was so powerful that no one could bear to sit near him (4).

In the late 19th century, the British surgeon Campbell De Morgan - Wikipedia noticed a link between tumor remissions and regressions, and post-operative infections, especially with the bacterial infection Erysipelas - Wikipedia (5).

Some years later, the new York city-based surgeon, William Coley - Wikipedia, rediscovered this acute infection-spontaneous tumor regression link when he stumbled onto an intriguing connection between the history of febrile infections and spontaneous tumor remissions (6, 7).

A talented surgeon, Coley was spurred by his inability to save a young sarcoma patient despite his best efforts to carefully operate and remove as much of it as was then possible (8). Investigating all sarcoma case histories then available in the records of the New York Cancer Hospital (today's Memorial Sloan-Kettering), Coley found the curious case of a patient diagnosed with the exact same cancer, round cell sarcoma, which disappeared entirely shortly after the patient contracted a severe Erysipelas infection (9).

Erysipelas being usually caused by Streptococcus bacteria drove Coley to empirically test whether deliberately inoculating sarcoma patients with S. pyogenes would induce similar tumor regressions. Testing various combinations of bacterial treatments, Coley eventually came up with a concoction he called mixed bacterial vaccines (MBV), today better known as Coley's toxins - Wikipedia.

Though Coley's daughter, Helen Coley-Nauts, over many decades painstakingly tracked down nearly every one of Coley's patients (~1200) and recorded their quite impressive long-term outcome, and periodic surveys of MBV found it to be quite effective (57% for example in one such survey), Coley and his idea faded away in the years after his death in 1936 for largely two reasons that had little to do with treatment efficacy and more with commercial potential (8).

• Marketed as breakthroughs, radiotherapy and chemotherapy emerged and remain oncology mainstays even today.
• The thalidomide tragedy in 1961 spurred tightening of regulations on new treatments. By then, MBV had been used for ~70 years but was still classified as a new treatment which meant expensive studies were needed to gain its licensing. MBV was considered a natural substance and was thus not patentable. Predictably, investment and research interest in exploring MBV as an anti-cancer treatment dried up.

Long before immunotherapy became the lucrative and intensively researched topic in cancer treatment that it is today, treatment with BCG vaccine - Wikipedia, the live attenuated vaccine against tuberculosis, became and stays a reliable mainstay for treating superficial urinary bladder cancer (10), even as how it actually works remains an active field of study (11, 12).

Meanwhile the recent explosion in microbiota research bolsters, not weakens, the need to understand how febrile infections could instigate or mediate tumor remissions and regressions (13, 14).

Some epidemiological data suggest a mechanistic connection in terms of antigenic overlap between microbe and tumor. Some years back the European Organization for Research and Treatment of Cancer (EORTC) established and tasked the Febrile Infections and Melanoma (FEBIM) working group to examine how prior infectious diseases and vaccines influenced melanoma risk. Such epidemiological studies suggest BCG, Vaccinia and Yellow fever vaccines provide some protection against melanoma (15, 16, 17, 18).

How to mechanistically explain how a bacterial toxin concoction such as Coley's toxins could help get rid of established sarcomas and other cancers or how an attenuated bacterium (BCG) or two unrelated viruses (Vaccinia and Yellow Fever) could help protect against developing melanoma? The link might turn on a foundational concept of how T and B cells operate, Cross-reactivity - Wikipedia, that such toxins or organisms might express certain antigens structurally similar to those expressed by tumors, and that once triggered, T and B cells specific for such bacterial or viral antigens attack the tumor as well.

Lifestyle factors better explain cancer rate increase compared to heritable factors

Many epidemiological studies have shown heritable factors account for minuscule proportions of cancer mortality (19, also see below a figure from 20 that summarizes data from multiple large long-term epidemiological studies), meaning cancer risk turns more on exposure to environmental factors.

OTOH, epidemiological studies suggest industrialized lifestyle factors such as alcohol, tobacco, highly processed food, sedentariness and attendant obesity go a long way in explaining cancer rates increases (21). In the coming years, research on microbiota changes will likely explain much of how such lifestyle changes increase cancer risk through their influence on immune system function.

Bibliography

1. Burnet, Macfarlane. "Cancer—A Biological Approach: III. Viruses associated with neoplastic conditions. IV. Practical applications." British medical journal 1.5023 (1957): 841. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1973618/pdf/brmedj03150-0013.pdf

2. Thomas, L., and H. S. Lawrence. "Cellular and humoral aspects of the hypersensitive states." New York: Hoeber-Harper (1959): 529-32.

3. Krone, Bernd, Klaus F. Kölmel, and John M. Grange. "The biography of the immune system and the control of cancer: from St Peregrine to contemporary vaccination strategies." BMC cancer 14.1 (2014): 595. https://bmccancer.biomedcentral.com/track/pdf/10.1186/1471-2407-14-595

4. Jackson, Robert. "Saint Peregrine, OSM--the patron saint of cancer patients." Canadian Medical Association Journal 111.8 (1974): 824. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1947908/pdf/canmedaj01592-0076.pdf

5. Grange, John M., John L. Stanford, and Cynthia A. Stanford. "Campbell De Morgan's ‘Observations on cancer’, and their relevance today." Journal of the Royal Society of Medicine 95.6 (2002): 296-299. http://journals.sagepub.com/doi/pdf/10.1177/014107680209500609

6. Jessy, Thomas. "Immunity over inability: The spontaneous regression of cancer." Journal of natural science, biology, and medicine 2.1 (2011): 43. http://www.jnsbm.org/temp/JNatScBiolMed2143-8286679_230106.pdf

7. Cann, SA Hoption, J. P. Van Netten, and C. Van Netten. "Dr William Coley and tumour regression: a place in history or in the future." Postgraduate medical journal 79.938 (2003): 672-680. http://pmj.bmj.com/content/postgradmedj/79/938/672.full.pdf

8. Kienle, Gunver S. "Fever in cancer treatment: Coley's therapy and epidemiologic observations." Global advances in health and medicine 1.1 (2012): 92-100. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3833486/pdf/gahmj.2012.1.1.016.pdf

9. Coley, William Bradley. "The treatment of malignant tumors by repeated inoculations of erysipelas: with a report of ten original cases." Am. J. Med. Sci. 5 (1893): 487-511.

10. B. C. G. Vaccine: Tuberculosis Cancer: Sol Roy Rosenthal: 9780884162131: Amazon.com: Books

11. Biot, Claire, et al. "Preexisting BCG-specific T cells improve intravesical immunotherapy for bladder cancer." Science translational medicine 4.137 (2012): 137ra72-137ra72.

12. Gan, Christine, et al. "BCG immunotherapy for bladder cancer—the effects of substrain differences." Nature Reviews Urology 10.10 (2013): 580. https://www.researchgate.net/profile/Hugh_Mostafid2/publication/256665424_BCG_immunotherapy_for_bladder_cancer_-_The_effects_of_substrain_differences/links/54d167530cf28959aa7af117.pdf

13. Oikonomopoulou, Katerina, et al. "Infection and cancer: revaluation of the hygiene hypothesis." Clinical Cancer Research 19.11 (2013): 2834-2841. http://clincancerres.aacrjournals.org/content/clincanres/19/11/2834.full.pdf

14. Wong, S., and R. A. Slavcev. "Treating cancer with infection: a review on bacterial cancer therapy." Letters in applied microbiology 61.2 (2015): 107-112.

15. Krone, Bernd, et al. "Protection against melanoma by vaccination with Bacille Calmette-Guerin (BCG) and/or vaccinia: an epidemiology-based hypothesis on the nature of a melanoma risk factor and its immunological control." European Journal of Cancer 41.1 (2005): 104-117.

16. Grange, John M., Bernd Krone, and John L. Stanford. "Immunotherapy for malignant melanoma–tracing Ariadne’s thread through the labyrinth." European Journal of Cancer 45.13 (2009): 2266-2273.

17. Mastrangelo, G., et al. "Does yellow fever 17D vaccine protect against melanoma?." Vaccine 27.4 (2009): 588-591.

18. Mastrangelo, Giuseppe, et al. "Yellow fever vaccine 17D administered to healthy women aged between 40 and 54 years halves breast cancer risk: an observational study." European Journal of Cancer Prevention (2017). Yellow fever vaccine 17D administered to healthy women aged ... : European Journal of Cancer Prevention

19. Roos, Leonie, Timothy D. Spector, and Christopher G. Bell. "Using epigenomic studies in monozygotic twins to improve our understanding of cancer." Epigenomics 6.3 (2014): 299-309. https://www.futuremedicine.com/doi/pdfplus/10.2217/epi.14.13

20. Rappaport, Stephen M. "Genetic factors are not the major causes of chronic diseases." PLoS One 11.4 (2016): e0154387. http://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0154387&type=printable

21. Stewart, B. W. K. P., and Christopher P. Wild. "World cancer report 2014." Health (2017).

Thanks for the R2A, Andrew Atkins.