CAVEAT MILITES: SOLDIERS, BEWARE — LEARNING FROM PAST ARMY EXPERIMENTS
(WARGAMING ROOM)

These wargames bring together experts from across the Army to wrestle with important questions while representing military action against a thinking adversary in a future context.

Experiments and wargames can provide important insights on future warfighting requirements, but if their design is flawed these insights may be precisely the wrong lessons to learn. As participants in multiple Army future concept experiments, the authors have seen well designed events result in solid feedback to improve concept development. Alternatively, poorly designed events have been “much less productive than they could have been.” A quick review of best practices in experimentation can help the Army as it is embarked on another exploration of the future.

In August 2021 and again in late January 2022, members of the Army Modernization Enterprise (AME), along with several joint and coalition partners, converged at Carlisle Barracks for a series of two-week long experiments to inform the development of the Army’s concept for warfighting in 2035 and beyond. Led by the Army Future Command’s (AFC) Director of Concepts (DoC), the U.S. Army Future Studies Program (FSP) is the latest instantiation of the Army’s Title 10 wargame series and the Army’s main venue to evaluate and examine questions about the future. Each service uses its Title 10 wargame to help understand the future environment, build future concepts, and identify future requirements. These wargames help senior enterprise levels consider future capabilities across the design elements in DOTMLPF-P, especially organization, materiel, and policy.

These events help the Army, and the other services, determine how to best equip, organize, and operate in the future. These wargames bring together experts from across the Army to wrestle with important questions while representing military action against a thinking adversary in a future context. Important insights will eventually bear fruit in modernized capabilities fielded to Army soldiers and tested in future battles. The recent FSP event was an important tool in analyzing future warfighting concepts and the Army must ensure future iterations of FSP and other warfighting experiments are properly designed, built, and executed to ensure the most useful outcomes.

The current AFC-led concept development effort builds on a legacy of past efforts. A Google search on “lessons learned from Army concept wargaming” yielded 490,000 results covering many decades. A key series of wargames and experiments that still resonate today were part of the Army After Next (AAN) effort that ran from 1996-1999 and focused on the far future. Lessons from AAN shaped the Army’s subsequent Transformation Campaign and its two main experiment and analysis paths of an Interim Force and an Objective Force. The Interim force resulted in the Army’s Stryker Brigade Combat Teams which remain in use today and an Objective Force centered on the Future Combat System which was canceled in 2009.  A review of lessons from these previous efforts can identify best practices and point out potential hazards to avoid as Army leaders continue the campaign of learning and make investment decisions based on results from these concept wargames.

Based on a review of these past efforts, and on our own experiences working in some of them, we developed a list for Army leaders to consider as they move forward and design the experiments to test and inform development of concepts, doctrine, organizations, materiel, and policies.

Design the event to inform your questions.

Experiments must have a purpose that is clear and achievable. Experimenting to discover what you don’t know or to obtain data to inform decisions about what you do know are both useful endeavors for planners and concept writers. A key to success is to have a set of effective questions that give guidance to the game designers and players. Creating good questions is not an easy task and requires energy and thought. Time is always in short supply, so leaders must ensure these wargame events focus on the most important questions about the future concepts. Experimental design matters and designers need to follow best practices. Often the game will be one in a series, as was the case for the recent FSP, so the experimental questions can adapt over time. One challenge in Army wargames and experiments is in how to best portray joint operations and effects. Too much joint play and the details needed to test Army concepts become lost. Too little joint play and the experiment becomes unrealistic and likely results in invalid conclusions.

Ask questions that you can answer.

Write hypotheses to confirm or deny the main assumptions included in future concepts. Write concepts in clear language using accepted joint terminology and avoid jargon.  Hypotheses about the concept must be testable so that clear answers can inform concept refinement. Characteristics of good hypotheses are well known as are the best uses of experiments and wargames to test those hypotheses. The FSP had an extensive set of questions and hypotheses that were not all fully answered and thus will need detailed review in preparation for the next series of concept wargames.

Control what you can and isolate variables of interest.

While experiments should allow for free play, some controls are required to actually test the hypotheses. Variables of interest should be explicitly identified, measured, and isolated so spurious correlation does not enable bad conclusions. A current important example of concept development is in the use Artificial Intelligence/Machine Learning (AI/ML). Since there are few existing true AI capabilities, concept developers must be particularly clever in how they test for the effects of AI/ML. Visioning future warfare enabled by AI/ML is not the same as determining in what use cases AI/ML is beneficial and which are not.

Use the proper model for the experiment.

Large Army experiments use models, many of them automated and computerized. One cannot test the operational impact of future technologies in the real-world if those technologies do not yet exist. Exercise designers should remember that “All models are wrong, some are useful.”  DoD has developed best practices for validating, verifying and accrediting models for their intended uses and the Army needs to follow them.

Participant experience matters.

To efficiently complete the game’s objectives, participants need experience both in playing the specific game and in warfighting at the appropriate level. This does not mean the players must be experts, in fact non-experts can often develop new, innovative solutions. For FSP, a collection of individual experts came together to play a newly created entity-level, computer assisted, operational level wargame. The team struggled as much to work through the game mechanics as it did in adapting new ideas for a future war fight against a peer enemy. Understanding the anticipated capabilities and uses of new technologies of interest was a challenge. Keeping a team together for future experiments, while difficult, will be helpful. At a minimum, the Army should establish a core nucleus of players and experimenters that can educate new players on game mechanics, past outcomes, and desired future outcomes. Additionally, joint concepts and capabilities in land force experiments require significant preparation of both players and game rules. Each military service updates their own concepts and doctrine and coordination with them is necessary to ensure the Army accurately portrays their contributions to gain the correct joint operational lessons.

A “World Class OPFOR” adds value to your experiment.

Include a skilled, unscripted, free playing opposing force and a well-resourced Red Team.

A thinking adversary allows real testing of future concepts. They should be trained in adversary tactics, but allowed to think freely and adapt their procedures to gain advantage and highlight issues with the concept. A “World Class OPFOR” adds value to your experiment. The 2002 exercise Millennium Challenge is a good example of a tough, thinking OPFOR in free play mode that challenged the proposed operational concepts. In addition to the OPFOR, the experiment director should staff and empower a red team to identify shortcomings in the experiment, to root out group think, and to ensure the experiment is achieving all intended outcomes.

Incorporate analysts and a collection plan.

Effective experiments require rigorous analysis and well-designed collection plans. Answering the posed questions and confirming/denying hypotheses are challenging to do well. Players must focus on fighting the adversary, and analysts can focus on collecting and analyzing data in order to answer the questions. Without dedicated analysts and a collection plan, observations are scattered and might not answer the question. FSP had a robust collection effort and this will need to continue as these experiments proceed. Resourcing the people, time, and cost for the collection and analysis is a challenge, but is critical to success.

Run excursions and repeat.

A point estimate of the future based on one run of an experiment will be precisely wrong; a range of scenarios is more useful and more likely to include the actual future.  Repetition will be critical to success of these experiments. Lessons derived from one iteration must be further examined in subsequent experiments. Interesting concepts must be both analyzed in smaller, more detailed pieces and also combined in larger, broader ranging Army and joint experiments. Only then will leaders see the best ideas worthy of investment and development.

While this list is important for any future concept experiment, two differences from past efforts complicate the current effort. The Army lacks recent practice conducting large, campaign-level, future concept wargames and a relatively new 4-star command is leading the effort. These differences mean the Army must iterate and try new methods of experimentation to both train the experimenters and find the best answers to the experimental questions.

We hope the points discussed above help the Army make the right choices about the future of war. If the Army fails to follow these lessons its hypotheses may become assertions, its assumptions may be untested, and it won’t prepare appropriately for the future.

The Army is well served to think about the future. Innovating in peacetime is particularly difficult. Even following all the best practices and lessons, experiments won’t describe the exact future. Thinking about how to think and taking some simple steps in improving experiments will ensure the best outcomes. The findings from these efforts will at least be in the ballpark, and work done today in peacetime experiments can better prepare the Army for future battlefields.

Bob Bradford is a retired U.S. Army Colonel and the Professor of Defense and Joint Processes in the U.S. Army War College’s Department of Command, Leadership, and Management.

Fred Gellert is a retired U.S. Army Colonel and the Professor of Science and Technology at U.S. Army War College’s Center for Strategic Leadership.

The views expressed in this article are those of the author and do not necessarily reflect those of the U.S. Army War College, the U.S. Army, or the Department of Defense.

Photo Credit: Fire vector created by vectorpocket – www.freepik.com