Ronald Lewin's 1982 book, THE AMERICAN MAGIC: CODES, CIPHERS AND THE DEFEAT OF JAPAN, illustrates how "Magic", the United States' remarkable code-breaking operation, played a key role in the Allied defeat of Japan during W.W.II and ultimately saved thousands of American soldiers' lives. Although Magic did not officially come into existence until the significant event of September 25, 1940-- when the U.S. penetration of the Japanese cipher system enabled the first complete text of a message to be decoded--its origins can be traced back as far as 1919.
MAY 1919: HERBERT YARDLEY CREATES THE BLACK CHAMBER.
Herbert Yardley, the man who first penetrated the Japanese code system on a massive scale, was just a young clerk in the State Department when he began his career as a code breaker by privately cracking the President's own enciphered telegrams. After the U.S. entered the war in 1917, he was made head of the MI 8 (the code-breaking section of the Military Intelligence Division). He then went on to lead a similar unit within the American delegation to the Peace Conference at Versailles.
In May of 1919, he persuaded the Departments of State and War to take the unprecedented step of funding a permanent organization devoted to the goal of penetrating other countries' signal traffic. Its prime mission: breaking Japan's diplomatic codes and ciphers.
1921: THE OFFICE OF NAVAL INTELLIGENCE ACCELERATES.
For quite awhile, ONI had been dismissed as simply a "post office" for passing on reports from elsewhere. However, in 1921, Captain Andrew Long takes over as director, and transforms ONI with his energy and determination causing it to accelerate and grow--the field of Navy cryptanalysis had suddenly entered the twentieth century. Two good examples of this are events that occurred during 1921, but whose consequences ultimately had an effect on the Pacific war. 1) Long significantly increased the amount of naval intelligence and language student officers at the U.S. Embassy in Tokyo and, 2) According to then Lieutenant Commander Ellis Zacharias (who later became a rear admiral), counterespionage experts broke into the office of a Japanese consulate and retrieved a photocopy of the Japanese Navy's current "fleet code". The benefits of the retrieval of the "Red Book", as it came to be called, were not immediate (it would take several years to translate), but it did cause the Navy to lay the foundations of a system which would expand and grow.
NOVEMBER 28, 1921: THE BLACK CHAMBER BREAKS A CRUCIAL SIGNAL FROM TOKYO TO THE JAPANESE AMBASSADOR.
The Japanese were, because of serious economic problems at home, in a complicated position during the time of the Washington Naval Conference: should they push for their claim to be allowed to build warships up to 70 percent of the British and American total, or settle, out of fear of pushing the U.S. too far, for 60 percent and agree to maintain the "status quo" of Pacific defenses? The U.S. had called the Conference mainly to prevent the British and French from doing the same thing, to break the long-standing treaty of British-Japanese alliance, and to hopefully advance a desire among nations to limit armaments by mutual agreement. It was during this conference that Yardley first penetrated the Japanese code system on a large scale, and even before the Conference opened, he and the Black Chamber were already breaking the codes used for Japan's diplomatic signals. On November 28, one such signal informed the U.S. that the Japanese were going to attempt to maintain a "middle attitude" and so, Charles Evans Hughes, the American Secretary of State, merely had to "outstare" the Japanese negotiators until December 10 when another signal from Tokyo communicated that the Japanese were willing to settle. Thus, the Four Power Pact was signed on December 13, and the U.S. came away from the conference feeling that it had achieved a major diplomatic triumph.
JANUARY 1924: LIEUTENANT LAURENCE L. SAFFORD ARRIVES IN THE NAVY'S CODE AND SIGNAL SECTION, CREATING THE NAVY'S COMMUNICATIONS INTELLIGENCE ORGANIZATION.
Laurence Safford, an Annapolis man, "put a bomb under the old regime" in January of 1924 when he took over the Research desk, and begins the start of the system that the 1920 retrieval of the Red Book had laid the foundation for. First, Safford ordered that a regular team of cryptanalysts be set up, and soon the team began work on the translation of the Red Book in Room 2646, a heavily protected room in a wing of the old Navy building. Neither their work or even the existence of the Red Book was known about by the army. In 1925. a radio surveillance post was set up on Guam in order to supply the code breakers with signals by virtually "picking them out of the air". Without the acquisition of a minimal amount of signals in a particular code or cipher, it is almost impossible to identify the particular characteristics of the traffic, and often the characteristics become obvious only after the extensive studying of many signals. Thus, the Guam post was of critical importance, as were the two posts in the Philippines and the more obscure post on the fourth floor of the American consulate in Shanghai that soon followed.
OCTOBER 1925: LIEUTENANT JOSEPH ROCHEFORT IS POSTED TO THE SAFFORD SECTION.
Joseph Rochefort had made his way up from the ranks of the Naval Reserve, and had been serving in the intelligence office of a battleship when he was posted to the Safford section in October of 1925. Rochefort, outstanding as an operator, became one of the best of the many excellent officers produced by the American system to handle signal intelligence during W.W.II, and was considered by his peers to be the most effective cryptanalyst in the Navy. Soon after being posted, he was sent to Japan for a three-year language course--a point of vital significance. When history refers to officers of the Pacific war as a language specialist or a Japanese translator, what is often overlooked is the fact that from the 1920's on, a small but steady flow of Marine, Army, and Navy officers were selected to be given an opportunity for a thorough "grounding" in Japanese. This was an extremely well-calculated policy that reaped many benefits. Another profitable policy was that of the U.S. Navy's insistence on certain key intelligence officers serving sea time--a policy which became vitally important during wartime when having intelligence officers and cryptanalysts that knew how things go on at sea first hand was critical. According to Lewin, "This interweaving of language study, sea time and understanding of signal intelligence is a good instance of the sensible interwar training and nursing-along of selected officers which, when Pearl Harbor exploded, left the U.S. Navy better equipped than has been realized--for reasons that have not been fully appreciated.".
1929: SECRETARY OF STATE STIMSON PUTS THE BLACK CHAMBER OUT OF BUSINESS.
The father of the Black Chamber, Herbert Yardley, also turned out to be "the millstone around the Army's neck" despite his earlier success during the Washington Naval Conference. In 1929, the Chamber was a complete, expensive disaster, doing no research in cryptanalysis whatsoever and consisting only of six people, with approximately 37% of the total payroll going to Yardley alone. Meanwhile, Yardley was devoting most of his time to private enterprises like commercial code compilation, real estate brokerage, and consulting commercial firms in code matters. His motto was "let joy be unconfined" and he was doing so--at the Government's expense. His obsession with poker, mistresses and the life of pleasure led to what Lewin calls "one of the most bizarre betrayals since the homosexual Colonel Redl" (an Austrian staff officer who, in 1913, was discovered to have been marketing military secrets to the Russians in order to maintain his standard of life). A 1929 investigation by the Signal Corps uncovered all of this, but before they had a chance to straighten things out, Secretary of State Stimson became aware of the interception and code-breaking aspects of the Black Chamber and shut it down with his famous statement 'Gentlemen do not read one another's mail' (according to Lewin, this statement is actually part of American mythology--it is not absolutely certain that Stimson made at the time). Thus, the Black Chamber was out of business and its employees were left with just a small gratuity and no pension. All of them loyally and silently accepted their unfortunate fate. Everyone, that is, except Yardley.
1931: YARDLEY'S REVENGE.
When Yardley's revealing book hit the stands in 1931 it shocked those still devoted to working in the field of signal intelligence and infuriated the Japanese. Japan's foreign minister, looking back to the Washington Naval Conference, accused the U.S. of a "breach of faith". Also, there is evidence that even before Yardley's book appeared, he had sold to the Japanese for $7,000 all he knew about the penetration of their codes. Thus, the true reason the Japanese were so angry may have been the fact that the "secrets" they had just paid for were now available for all the world to read about in Yardley's best seller. As for Yardley, he never recovered from this unforgivable breach of trust. When he attempted to publish a second book, JAPANESE DIPLOMATIC SECRETS, the U.S. government suppressed it. When the Canadians wanted him as cryptanalyst in 1940, the British banned him. When he attempted entrance into the OSS, General Donovan refused him. Finally, Yardley ended up doing his war work as an enforcement agent in the Office of Price Administration.
The Japanese Purple machine was one of the Jade "big" machines (National Cryptologic Museum)
September 25, 1940: MAGIC IS BORN.
After the appearance of Yardley's book in 1931, the Japanese naturally sought a way to improve the security of their signals. The solution to their security problem was a radical one: they decided to abandon using code and began instead to encipher their most confidential and secret messages on a machine. Ironically, it was this decision that eventually enabled the U.S. to read Japanese diplomatic messages with great ease. The enciphered machine they used was developed by the special signals unit of the Navy, under captain Ito. It was this machine that became known as Purple to the Americans, and of all the many enciphering machines used during World War II, only it and the German machine, Enigma, attained 'star quality'.
According to Lewin, the reason both these machines failed their masters was simple: the Japanese and the Germans had complete confidence throughout the war that their machines would protect their secrets. Both countries were so confident, in fact, that they believed their machines to be infallible while in actuality both ciphers were cracked at a sufficiently early stage and then broken consistently until the end. The man usually credited with the breaking of Purple was William F. Friedman, a highly skilled cryptanalyst of great experience. Since the Stimson shutdown of the Black Chamber, a new unit had been established for army intelligence, called the Signal Intelligence Service. Its chief signal officer was General Gibbs and its director was Friedman, whose staff consisted of only six people. Interchanges between this group and Safford's group allowed the Army and Navy teams to pool much of their knowledge, but the battle of the breaking of Purple was, nevertheless, harsh and prolonged. The first break was recorded as having occurred in August of 1940, and the first complete text of a message as having been established on September 25.. According to the U.S. Department of Defense's THE MAGIC BACKGROUND OF PEARL HARBOR, " By the fall of 1940 United States Government cryptanalysts had solved some of the Japanese foreign offices highest grade cryptographic systems. The interception, decryption and translation on a current basis of secret Japanese worldwide diplomatic messages then began." This marked the birth of Magic--the remarkable code-breaking operation that ultimately saved thousands of American soldiers' lives and contributed greatly to the Allied victory over Japan.
Other useful references: